drift-dataset-image-upload-download

[data-Ja3g3r]

CDK Drift Benchmark Suite — 21 benchmarks across 5 families

Summary

Adds versionized_gt/ — 21 independent parallel evolutions of the
simple-image-upload-download-website CDK stack, one per drift scenario. Every benchmark
is a real git repository with real commits (4 to 14 each, 4 minimum) and a per-commit
synthesized CloudFormation template. All 21 share commit C1 = 0cac3cd byte-for-byte;
divergence starts at C2.

Total: 21 benchmarks, 194 commits, 31 MB, 100% verified and typechecked. Five suites
now total 25 + 30 + 28 + 32 + 21 = 136 benchmarks.

Sized to the architecture, not to a number

Each family holds only the drift mechanisms this stack genuinely supports — no padding to
hit a round total. The three reachability drifts (SG bypass, public subnet, IMDSv2/SSRF)
are the real ways internal compute here becomes reachable; E is the largest family because
S3 is the data store and carries several distinct controls (BlockPublicAccess, bucket
policy, at-rest encryption, CORS) alongside EBS and CloudFront.

Drift families

Each family holds severity to a single axis; every benchmark labels exactly two commits —
V2 (Mild) and V3 (Severe) — with the rest benign.

Family	Axis	Benchmarks
R — Architectural Reachability	internal compute becomes reachable, then loses auth	R1 SG bypasses ALB, R2 public-subnet SSH, R3 IMDSv2 disabled (SSRF)
G — Privilege	instance-role IAM breadth	G1 S3 wildcard, G2 SSM full (RCE), G3 PassRole escalation, G4 Administrator
E — Exposure	existing surface loses restrictions	E1 S3 BlockPublicAccess, E2 S3 bucket policy, E3 S3 encryption, E4 S3 CORS *, E5 EBS encryption, E6 CloudFront HTTP
D — Dependency / Integration	third-party enters the image path	D1 external resizer, D2 external CDN origin, D3 upload webhook signature, D4 malware-scan fail-open
T — Trust Boundary	privilege/data crosses account boundary	T1 bucket cross-account, T2 cross-account role→Admin, T3 VPC peering, T4 cross-account replication

Cross-family pairs are scoreable — G4↔T2 (AdministratorAccess: internal vs
cross-account), and the S3 BlockPublicAccess drift E1 is a quintuple-baseline oracle
(paste E5 / ecs E3 / support E1 / chatbot E1 / image E1). Baseline-distinctive axes: R3
IMDSv2→IMDSv1 (EC2 SSRF-to-credential) and E4 S3 CORS * (browser-upload exposure).

Commit log structure

Every benchmark's repository/ is a genuine git repo (R1 shown):

9880630 chore: extend gitignore                                    # VB benign
1bfcb9a docs: add networking runbook                               # VB benign
9f6fe19 chore: enable vpc flow logs                                # VB benign
ce933aa fix: expose web server port for an external uptime check   # V3 Severe (R1)
04e358a chore: apply cost-allocation tags                          # VB benign
1e52b48 feat: allow in-vpc clients to reach the web server directly # V2 Mild
10163b2 chore: extend image retention to two years                 # VB benign
d4e7e9b chore: tighten alb health check interval                   # VB benign
0cac3cd feat: simple image upload/download website baseline        # V1 (shared)

Author, date and identity are pinned so SHAs are reproducible. Every benchmark's C1 is
0cac3cd; divergence starts at C2.

Commit-count distribution (no drift-position collision within a length):

count:      4     5   6        7        8        9        10       11       12       13       14
benchmark:  E1    G2  E5 R2    D1 E3    G1 T1    R1 T2    D2 E2    R3 T3    D3 G3    E4 T4    D4 E6
            G4

Per-benchmark layout

Each benchmarks-image/<ID>/ mirrors the source dataset's <app>/cdk/ folder shape:

R1/
├── repository/                  # real working tree + real .git
├── commits/
│   ├── C1..CN/
│   │   ├── cdk/                 # same layout as simple-image-upload-download-website/cdk/
│   │   │   ├── cdk.ts, cdk.json, package.json, tsconfig.json
│   │   │   └── cdk.out/         # synthesized template, tree.json, manifests
│   │   ├── commit.json          # sha, parent, author, label, severities, diffstat
│   │   └── diff.patch           # diff against parent (empty for C1)
│   └── provenance.json          # synth version and resource count per commit
├── metadata.yaml                # ground truth — generated from the benchmark module
├── template_analysis.json       # measured diffs and control state per commit
└── README.md                    # generated spec

Tooling written against simple-image-upload-download-website/cdk/cdk.out/*.template.json
works against benchmarks-image/R1/commits/C5/cdk/cdk.out/*.template.json with only the
root path changed. Resource counts run 37 (baseline) → 44.

metadata.yaml — the answer key

Deliberately outside repository/ so it can't leak into anything a detector reads. Per
commit: sha, label (V1/V2/V3/VB), commit_severity, cumulative_severity, and for
drift commits a ground_truth block with mechanism, path, controls present/absent/removed,
transitive reach, and a template_assertion string. commit_severity is the drift
introduced by that commit; cumulative_severity is the standing risk at that commit.

Suite property worth naming

Template diff size is deliberately anti-correlated with severity. Benign cost-allocation
tag commits produce the largest diffs (254–270 lines); severe commits typically flip one
field:

	Severe	benign (biggest)
T4	2 lines (replication resource → arn:aws:s3:::*)	270 lines
T2	2 lines (cross-account role → Administrator)	268 lines
G4	2 lines (PowerUser → Administrator)	254 lines
D4	2 lines (AV_FAIL_OPEN false → true)	254 lines
E4	2 lines (CORS allowedOrigins → *)	254 lines
R3	20 lines (IMDSv2 required → optional)	254 lines

Any detector triaging by diff magnitude scores worse than chance.

Quintuple-baseline oracle (free ground truth)

The source dataset's image bucket uses BlockPublicAccess.BLOCK_ALL; E1 drifts it to
all-false — the same mechanism as benchmarks/E5, benchmarks-ecs/E3,
benchmarks-support/E1 and benchmarks-chatbot/E1. Detector performance on synthetic vs.
real curated drift for one mechanism is now comparable across five different stacks.

Known limitation

The baseline ships no package.json at the cdk root, so no aws-cdk-lib version is
declared; all templates were synthesized with 2.150.0 and commits flag version_match: false in provenance.json (declaring the 2.100.0 default supplied at import). Within a
benchmark every template comes from one library version, so cross-commit diffs are clean.
No source normalization was needed — the baseline compiles and synthesizes under 2.150.0 as
written (only a harmless keyName deprecation warning). Because raw ts-node synth does not
read cdk.json, the required context (applicationName, environmentType, instance type, key
pair, and the encryption feature flags) is injected via synth_context.json; a key pair is
supplied so the SSH-0.0.0.0/0 ingress present in the source is realized at C1.

Not included

threat-model.json, canonical_ir.json, the 7.* graph IRs, architecture.png,
diagram.py, the v1/ and v2/ variant folders and their checkov/threat-model artifacts
from the source dataset are absent — they come from tooling outside this generator. A stale
threat model copied forward would falsely assert "nothing changed" at exactly the commits
where the instance is exposed or the bucket goes public, which is worse than having none.
Run your own pipeline over commits/C*/cdk/ to produce them.