drift_dataset_ticketing_sys

[data-Ja3g3r]

CDK Drift Benchmark Suite — 28 benchmarks across 5 families

Summary

Adds versionized_gt/ — 28 independent parallel evolutions of the
support-ticketing-system CDK stack, one per drift scenario. Every benchmark is a real
git repository with real commits (4 to 15 each, 4 minimum) and a per-commit synthesized
CloudFormation template. All 28 share commit C1 = 99c3992 byte-for-byte; divergence
starts at C2.

Total: 28 benchmarks, 241 commits, 75 MB, 100% verified and typechecked. Three suites
now total 25 + 30 + 28 = 83 benchmarks.

Drift families

Each family holds severity to a single axis and every benchmark labels exactly two
commits — V2 (Mild) and V3 (Severe) — with the rest benign.

Family	Axis	Benchmarks
R — Architectural Reachability	internal compute becomes reachable, then loses auth	R1 notification replay endpoint, R2 listener rule, R3 EC2-ASG admin port, R4 ticket-update method, R5 APIGW→DynamoDB direct (no Lambda)
G — Privilege	IAM breadth (actions × resources)	G1 read→rw, G2 wildcard resource, G3 AssumeRole, G4 PowerUser→Administrator, G5 KMS policy, G6 DynamoDB wildcard, G7 SSM wildcard
E — Exposure	existing surface loses restrictions	E1 S3 BlockPublicAccess, E2 backend SG, E3 RDS ingress, E4 S3 bucket policy, E5 SNS topic policy
D — Dependency / Integration	third-party enters critical path	D1 OpenAI, D2 PagerDuty, D3 Stripe, D4 Datadog cross-account
T — Trust Boundary	privilege crosses account/env boundary	T1 VPC peering, T2 cross-account IAM, T3 shared DB, T4 SSM cross-account, T5 event bus, T6 KMS grant cross-account, T7 SQS cross-account SendMessage

Cross-family pairs are scoreable — G4↔T2↔D4 (managed-policy flip to AdministratorAccess:
internal vs cross-account partner vs cross-account vendor), G7↔T4 (SSM wildcard, local vs
foreign account), G5↔T6 (KMS grant, internal vs cross-account), T5↔T7 (wildcard principal,
inbound event bus vs queue injection). Eight axes are new versus the two prior baselines:
custom-authorizer reachability (R1, R4), APIGW→DynamoDB with no Lambda (R5), DynamoDB
privilege (G6), SSM access (G7, T4), S3 bucket-policy / SNS topic-policy exposure (E4, E5),
and SQS cross-account SendMessage (T7).

Commit log structure

Every benchmark's repository/ is a genuine git repo. git log, diff, show,
bisect all work (R1 shown):

e4c17b3 docs: annotate replay procedure                              # VB benign
5423499 chore: extend gitignore                                      # VB benign
8f83fcc docs: add ops runbook                                        # VB benign
d428cb6 fix: drop authorizer from notification replay endpoint       # V3 Severe (R1)
ddd16fa chore: apply cost-allocation tags                            # VB benign
3f5678f feat: add authenticated notification replay endpoint         # V2 Mild
9afd88e feat: add byStatus index on ticket table                     # VB benign
34eb647 chore: add cloudwatch error alarms on lambdas                # VB benign
99c3992 feat: support ticketing platform baseline                    # V1 (shared)

Author, date and identity are pinned so SHAs are reproducible. Every benchmark's C1 is
99c3992; divergence starts at C2.

Commit-count distribution (no drift-position collision within a length):

commits:   4     5           6              7      8      9            10     11     12   13         14   15
benchmark: E1    D4 E5 G2    R3 D2 E4 G6    E2 T7  R4 T1  R1 G1 G5 T2  D1 T5  R2 T3  T6   D3 E3 R5   G3   G7
           G4                T4
benign:    2     3           4              5      6      7            8      9      10   11         12   13

Per-benchmark layout

Each benchmarks-support/<ID>/ mirrors the source dataset's <app>/cdk/ folder shape:

R1/
├── repository/                  # real working tree + real .git (git log/diff/show all work)
├── commits/
│   ├── C1..CN/
│   │   ├── cdk/                 # same layout as support-ticketing-system/cdk/
│   │   │   ├── cdk.ts           # the stack at this commit
│   │   │   ├── cdk.json
│   │   │   ├── package.json
│   │   │   ├── tsconfig.json
│   │   │   ├── lambda/          # the three real handlers (ticket creator/updater, notification)
│   │   │   └── cdk.out/         # synthesized template, tree.json, manifests
│   │   ├── commit.json          # sha, parent, author, label, severities, diffstat
│   │   └── diff.patch           # diff against parent (empty for C1)
│   └── provenance.json          # synth version and resource count per commit
├── metadata.yaml                # ground truth — generated from the benchmark module (researcher use for validation)
├── template_analysis.json       # measured diffs and control state per commit (researcher use for validation)
└── README.md                    # generated spec (researcher use for validation)

Tooling written against support-ticketing-system/cdk/cdk.out/*.template.json works
against benchmarks-support/R1/commits/C5/cdk/cdk.out/*.template.json with only the root
path changed. Resource counts run 85 (baseline) → 101.

metadata.yaml — the answer key

Deliberately outside repository/ so it can't leak into anything a detector reads.
Per commit: sha, label (V1/V2/V3/VB), commit_severity, cumulative_severity, and
for drift commits a ground_truth block with mechanism, path, controls
present/absent/removed/bypassed, transitive reach, and a template_assertion string.

Two severity fields deliberately: commit_severity = drift introduced by that commit;
cumulative_severity = standing risk at that commit. In most benchmarks the commits
after C1 sit at non-zero cumulative severity while introducing nothing — a false-positive
report there is wrong, and so is "all clear".

Suite property worth naming

Template diff size is deliberately anti-correlated with severity across the suite. Benign
cost-allocation tag commits propagate onto every taggable resource and produce the largest
diffs in most benchmarks (80–820 lines); severe commits typically flip one field:

	Severe	benign (biggest)
G4	2 lines (PowerUser → Administrator)	84 lines
T2	2 lines (same, cross-account trust preserved)	791 lines
D3	2 lines (STRIPE_VERIFY_SIGNATURE: 'true' → 'false')	823 lines
R3	4 lines (admin port CidrIp → 0.0.0.0/0)	777 lines
E1	4 lines (S3 BlockPublicAccess → all-false)	84 lines

Any detector triaging by diff magnitude scores worse than chance.

One oracle drift (free ground truth)

The source dataset's own progression flips attachmentBucket from
BlockPublicAccess.BLOCK_ALL to public, matching E1 — the same mechanism as
benchmarks/E5 (paste) and benchmarks-ecs/E3. That gives a triple-baseline oracle:
detector performance on synthetic vs. real curated drift for one mechanism is directly
comparable across three different stacks.

Known limitation

The baseline declares aws-cdk-lib@2.100.0, which would not install in the build
environment. All templates were synthesized with 2.150.0; commits flag
version_match: false in provenance.json (declaring 2.100.0 while synthesized with
2.150.0). Within a benchmark every template comes from one library version, so cross-commit
diffs are clean. Three source constructs needed deterministic, semantics-preserving fixes to
compile against 2.150.0: RDS encrypted → storageEncrypted, RDS encryptionKey →
storageEncryptionKey, and the absent package.json/tsconfig.json supplied as defaults at
C1 import. Because raw ts-node synth does not read cdk.json context, feature-flag context
(enableKmsEncryption, enableSsoIntegration, both true) is injected via synth_context.json
so the KMS key and TokenAuthorizer exist at C1.

Not included

acse-threat-model.json, checkov_results.*, architecture.png and diagram.py from
the source dataset's folder are absent — they come from tooling outside this generator. A
stale threat model copied forward would falsely assert "nothing changed" at exactly the
commits where auth boundaries disappear, which is worse than having none. Run your own
pipeline over commits/C*/cdk/ to produce them.