drift-dataset-codeexec-platform

[data-Ja3g3r]

CDK Drift Benchmark Suite — 18 benchmarks across 5 families

Summary

Adds versionized_gt/ — 18 independent parallel evolutions of the
code-execution-platform CDK stack, one per drift scenario. Every benchmark is a real git
repository with real commits (4 to 14 each, 4 minimum) and a per-commit synthesized
CloudFormation template. All 18 share commit C1 = fe2cb61 byte-for-byte; divergence
starts at C2.

Companion to the text-paste (25), containerized-microservices (30), support-ticketing (28),
Bedrock-chatbot (32), image-website (21) and security-response (18) suites. This baseline is
an untrusted-code-execution sandbox: a WAF-protected API Gateway feeds a five-Lambda
pipeline (apiHandler → requestValidator → executionOrchestrator → resultProcessor →
responseHandler) that launches an ECS Fargate task to run submitted code, with S3
code/results buckets and a DynamoDB results table.

Total: 18 benchmarks, 151 commits, 45 MB, 100% verified and typechecked. Seven suites
now total 25 + 30 + 28 + 32 + 21 + 18 + 18 = 172 benchmarks.

Sized to the architecture, not to a number

The threat model is the sandbox boundary — the Fargate task that runs arbitrary user code,
the orchestrator that launches it, and the submitted-code bucket. So the honest drift shape
is privilege- and reachability-heavy with the code bucket as the crown-jewel exposure
asset; families are sized to the mechanisms that genuinely exist, with no padding.

Drift families

Each family holds severity to a single axis; every benchmark labels exactly two commits —
V2 (Mild) and V3 (Severe) — with the rest benign.

Family	Axis	Benchmarks
R — Architectural Reachability	the sandbox network + reaching the executor	R1 task in public subnets, R2 orchestrator Function URL bypass, R3 /submit auth removed
G — Privilege	the sandbox role and the launcher	G1 task reads code→s3:, G2 orchestrator PassRole→, G3 orchestrator ecs:*, G4 task role→Administrator
E — Exposure	submitted-code store + the API's WAF	E1 code BlockPublicAccess, E2 code bucket policy, E3 code protections, E4 WAF removed
D — Dependency / Integration	third party in the code pipeline	D1 scanner fail-open, D2 package mirror unverified, D3 result webhook signature, D4 base-image registry
T — Trust Boundary	cross-account seams around the sandbox	T1 code bucket cross-account, T2 task role foreign trust, T3 VPC peering

Cross-family pairs are scoreable — G4↔T2 (the same sandbox task role: permission
escalation vs trust widening), G1 (untrusted code reads all code then any bucket), and
the S3 BlockPublicAccess drift E1 is a septuple-baseline oracle across all seven
suites.

Commit log structure

Every benchmark's repository/ is a genuine git repo (R1 shown):

935634a chore: extend gitignore                                    # VB benign
1c3692c docs: add networking runbook                               # VB benign
116bee1 chore: raise lambda memory and timeout                     # VB benign
e4d3ee1 fix: run execution tasks in public subnets to drop nat cost # V3 Severe (R1)
b81c9aa chore: apply cost-allocation tags                          # VB benign
72c5e3b feat: allow execution tasks to launch in public subnets    # V2 Mild
b248f61 chore: extend code retention to two years                  # VB benign
afc6521 chore: add log level env to lambdas                        # VB benign
fe2cb61 feat: code execution platform baseline                     # V1 (shared)

Author, date and identity are pinned so SHAs are reproducible. Every benchmark's C1 is
fe2cb61; divergence starts at C2.

Commit-count distribution (no drift-position collision within a length):

count:      4      5       6       7       8       9    10      11    12       13    14
benchmark:  E1 G4  D4 G2   E3 R2   D1 E2   G1 T1   R1   D2 T2   R3    D3 G3    E4    T3

Per-benchmark layout

Each benchmarks-codeexec/<ID>/ mirrors the source dataset's <app>/cdk/ folder shape:

R1/
├── repository/                  # real working tree + real .git
├── commits/
│   ├── C1..CN/
│   │   ├── cdk/                 # same layout as code-execution-platform/cdk/
│   │   │   ├── cdk.ts, cdk.json, package.json, tsconfig.json
│   │   │   └── cdk.out/         # synthesized template, tree.json, manifests
│   │   ├── commit.json          # sha, parent, author, label, severities, diffstat
│   │   └── diff.patch           # diff against parent (empty for C1)
│   └── provenance.json          # synth version and resource count per commit
├── metadata.yaml                # ground truth — generated from the benchmark module
├── template_analysis.json       # measured diffs and control state per commit
└── README.md                    # generated spec

Tooling written against code-execution-platform/cdk/cdk.out/*.template.json works against
benchmarks-codeexec/R1/commits/C5/cdk/cdk.out/*.template.json with only the root path
changed. Resource counts run 72 (baseline) → 75.

metadata.yaml — the answer key

Deliberately outside repository/ so it can't leak into anything a detector reads. Per
commit: sha, label (V1/V2/V3/VB), commit_severity, cumulative_severity, and for
drift commits a ground_truth block with mechanism, path, controls present/absent/removed,
transitive reach, and a template_assertion string. commit_severity is the drift
introduced by that commit; cumulative_severity is the standing risk at that commit.

Suite property worth naming

Template diff size is deliberately anti-correlated with severity. Benign cost-allocation
tag commits produce the largest diff (~400 lines); severe commits flip one or a few fields:

	Severe	benign (biggest)
T3	2 lines (peering route → 0.0.0.0/0)	412 lines
G4	2 lines (task role → Administrator)	400 lines
E1	4 lines (BlockPublicAccess all-false)	400 lines
G1	5 lines (task role s3:* on '*')	400 lines
R3	6 lines (apiKeyRequired → false)	400 lines
D1–D4	10 lines (integration flag flip across the shared env)	400 lines

Any detector triaging by diff magnitude scores worse than chance.

Septuple-baseline oracle (free ground truth)

The source dataset's code bucket uses BlockPublicAccess.BLOCK_ALL; E1 drifts it to
all-false — the same mechanism as the E-family in all six prior suites. Detector
performance on synthetic vs. real curated drift for one mechanism is now comparable across
seven different stacks.

Known limitation

The baseline ships no package.json at the cdk root, so no aws-cdk-lib version is
declared; all templates were synthesized with 2.150.0 and commits flag version_match: false in provenance.json (declaring the 2.100.0 default supplied at import). Within a
benchmark every template comes from one library version, so cross-commit diffs are clean.
One tsc-only normalization is applied: logs.RetentionDays.SEVEN_DAYS does not exist in
2.150.0 and is replaced with ONE_WEEK (both 7 days). Because raw ts-node synth does not
read cdk.json, the required context (VPC/Fargate sizing, bucket/table names, and the
WAF/Shield/SecurityHub feature flags) is injected via synth_context.json; WAF is enabled
so the WebACL exists at C1.

Not included

threat-model.json, canonical_ir.json, the 7.* graph IRs, architecture.png and
diagram.py from the source dataset are absent — they come from tooling outside this
generator. A stale threat model copied forward would falsely assert "nothing changed" at
exactly the commits where the sandbox role turns admin or the executor becomes publicly
reachable, which is worse than having none. Run your own pipeline over commits/C*/cdk/ to
produce them.