drift-dataset-automated-sec-rep

[data-Ja3g3r]

CDK Drift Benchmark Suite — 18 benchmarks across 5 families

Summary

Adds versionozed_gt/ — 18 independent parallel evolutions of the
automated-security-response CDK stack, one per drift scenario. Every benchmark is a real
git repository with real commits (4 to 14 each, 4 minimum) and a per-commit synthesized
CloudFormation template. All 18 share commit C1 = d29eb3a byte-for-byte; divergence
starts at C2.

Total: 18 benchmarks, 155 commits, 25 MB, 100% verified and typechecked. Six suites now
total 25 + 30 + 28 + 32 + 21 + 18 = 144 benchmarks.

Sized to the architecture, not to a number

This is a privileged, multi-account automation plane, so the honest drift shape is heavy
on privilege (G) and trust (T), light on reachability (R) — it is fully serverless with
no network layer. R has exactly two members because the only externally-reachable surfaces
are the event trigger and direct Lambda invocation; padding it would invent drifts that do
not exist for this stack.

Drift families

Each family holds severity to a single axis; every benchmark labels exactly two commits —
V2 (Mild) and V3 (Severe) — with the rest benign.

Family	Axis	Benchmarks
R — Architectural Reachability	what can reach/trigger the privileged automation	R1 trigger pattern widened, R2 remediation Lambda Function URL
G — Privilege	IAM breadth on the automation roles	G1 RemediationRole→Admin, G2 Lambda SSM wildcard, G3 Lambda PassRole, G4 orchestrator direct SSM+PassRole
E — Exposure	the findings store + alert channel lose restrictions	E1 metrics BlockPublicAccess, E2 metrics bucket policy, E3 metrics protections, E4 SNS public publish
D — Dependency / Integration	a third party enters the pipeline	D1 ticketing, D2 SIEM egress, D3 remediation webhook signature, D4 threat-intel auto-trust
T — Trust Boundary	admin↔member cross-account seams	T1 orchestrator AssumeRole any-account, T2 RemediationRole foreign trust, T3 cross-account event ingress, T4 cross-account notification subscribe

Cross-family pairs are scoreable — G1↔T2 (the same RemediationRole: permission
escalation vs trust widening), G4 (orchestrator bypasses the finding-validation Lambda),
and the S3 BlockPublicAccess drift E1 is a sextuple-baseline oracle (paste E5 / ecs
E3 / support E1 / chatbot E1 / image E1 / secresponse E1).

Commit log structure

Every benchmark's repository/ is a genuine git repo (R1 shown):

93b3da1 chore: extend gitignore                                    # VB benign
e34491b docs: add eventing runbook                                 # VB benign
56373ac chore: raise state machine timeout                         # VB benign
d7d31ef fix: simplify the rule to catch all account events         # V3 Severe (R1)
e40dfc8 chore: apply cost-allocation tags                          # VB benign
6251975 feat: also handle security hub custom-action findings      # V2 Mild
6667b8a chore: extend metrics retention to two years               # VB benign
ffe1813 chore: raise firehose buffering interval                   # VB benign
d29eb3a feat: automated security response platform baseline        # V1 (shared)

Author, date and identity are pinned so SHAs are reproducible. Every benchmark's C1 is
d29eb3a; divergence starts at C2.

Commit-count distribution (no drift-position collision within a length):

count:      4      5    6        7        8        9        10       11    12       13    14
benchmark:  E1 G1  D4   E3 R2    D1 E2    G2 T1    D2 R1    E4 T2    G3    D3 T3    G4    T4

Per-benchmark layout

Each benchmarks-secresponse/<ID>/ mirrors the source dataset's <app>/cdk/ folder shape:

R1/
├── repository/                  # real working tree + real .git
├── commits/
│   ├── C1..CN/
│   │   ├── cdk/                 # same layout as automated-security-response/cdk/
│   │   │   ├── cdk.ts, cdk.json, package.json, tsconfig.json
│   │   │   └── cdk.out/         # synthesized template, tree.json, manifests
│   │   ├── commit.json          # sha, parent, author, label, severities, diffstat
│   │   └── diff.patch           # diff against parent (empty for C1)
│   └── provenance.json          # synth version and resource count per commit
├── metadata.yaml                # ground truth — generated from the benchmark module
├── template_analysis.json       # measured diffs and control state per commit
└── README.md                    # generated spec

Tooling written against automated-security-response/cdk/cdk.out/*.template.json works
against benchmarks-secresponse/R1/commits/C5/cdk/cdk.out/*.template.json with only the
root path changed. Resource counts run 29 (baseline) → 31.

metadata.yaml — the answer key

Deliberately outside repository/ so it can't leak into anything a detector reads. Per
commit: sha, label (V1/V2/V3/VB), commit_severity, cumulative_severity, and for
drift commits a ground_truth block with mechanism, path, controls present/absent/removed,
transitive reach, and a template_assertion string. commit_severity is the drift
introduced by that commit; cumulative_severity is the standing risk at that commit.

Suite property worth naming

Template diff size is deliberately anti-correlated with severity. Benign cost-allocation
tag commits produce the largest diff (242 lines); severe commits flip one or a few fields:

	Severe	benign (biggest)
G1	2 lines (RemediationRole → Administrator)	242 lines
E1	4 lines (BlockPublicAccess all-false)	242 lines
T1	5 lines (AssumeRole → any account)	242 lines
D1–D4	6 lines (integration flag flip across 3 lambdas)	242 lines

Any detector triaging by diff magnitude scores worse than chance.

Sextuple-baseline oracle (free ground truth)

The source dataset's metrics bucket uses BlockPublicAccess.BLOCK_ALL; E1 drifts it to
all-false — the same mechanism as benchmarks/E5, benchmarks-ecs/E3,
benchmarks-support/E1, benchmarks-chatbot/E1 and benchmarks-image/E1. Detector
performance on synthetic vs. real curated drift for one mechanism is now comparable across
six different stacks.

Known limitation

The baseline ships no package.json at the cdk root, so no aws-cdk-lib version is
declared; all templates were synthesized with 2.150.0 and commits flag version_match: false in provenance.json (declaring the 2.100.0 default supplied at import). Within a
benchmark every template comes from one library version, so cross-commit diffs are clean.
One tsc-only fix is applied: the deprecated targets.KinesisFirehoseStream used as a log
SubscriptionFilter destination has an incompatible bind signature under 2.150.0; it
synthesizes correctly, and an extra as any on the already-cast expression satisfies the
type checker without changing behaviour. Because raw ts-node synth does not read
cdk.json, the required context (adminAccountId, memberAccountId, primaryRegion,
notification email, log retention, and the encryption feature flags) is injected via
synth_context.json.

Not included

threat-model.json, canonical_ir.json, the 7.* graph IRs, architecture.png and
diagram.py from the source dataset are absent — they come from tooling outside this
generator. A stale threat model copied forward would falsely assert "nothing changed" at
exactly the commits where the remediation role turns admin or the trigger opens to any
event, which is worse than having none. Run your own pipeline over commits/C*/cdk/ to
produce them.