drift_dataset_ai_chatbot

[data-Ja3g3r]

CDK Drift Benchmark Suite — 32 benchmarks across 5 families

Summary

Adds versionzied_gt/ — 32 independent parallel evolutions of the aws-ai-chat-bot
CDK stack, one per drift scenario. Every benchmark is a real git repository with real
commits (4 to 15 each, 4 minimum) and a per-commit synthesized CloudFormation template.
All 32 share commit C1 = 9c19d32 byte-for-byte; divergence starts at C2.

Total: 32 benchmarks, 304 commits, 78 MB, 100% verified and typechecked. Four suites
now total 25 + 30 + 28 + 32 = 115 benchmarks.

Drift families

Each family holds severity to a single axis and every benchmark labels exactly two
commits — V2 (Mild) and V3 (Severe) — with the rest benign.

Family	Axis	Benchmarks
R — Architectural Reachability	internal compute/data becomes reachable, then loses auth	R1 vector store public, R2 WAF detached, R3 ECS SG bypass, R4 public ingestion upload, R5 ingestion Function URL, R6 Bedrock endpoint reach
G — Privilege	IAM breadth (actions × resources)	G1 Bedrock model→*, G2 aoss wildcard, G3 AssumeRole, G4 Administrator, G5 KB write escalation, G6 DynamoDB wildcard, G7 bedrock:*
E — Exposure	existing surface loses restrictions	E1 KB BlockPublicAccess, E2 aoss data policy, E3 Cognito open callback, E4 KB bucket policy, E5 Bedrock endpoint policy, E6 WAF rule neutered
D — Dependency / Integration	third-party enters the prompt/data path	D1 OpenAI bypass, D2 Pinecone primary, D3 trace full-IO, D4 tool webhook sig, D5 external embedder, D6 model gateway
T — Trust Boundary	privilege crosses account/env boundary	T1 VPC peering, T2 cross-account IAM, T3 vector foreign account, T4 cross-account model, T5 Cognito federation, T6 secret cross-account, T7 event bus cross-account

AI-native cross-family pairs are scoreable — G1↔T4 (Bedrock model access: in-account
wildcard vs cross-account/region), R1↔E2↔T3 (vector store: network vs data-policy vs
foreign account), G4↔T2 (AdministratorAccess: internal vs cross-account), D1↔D5↔D6
(external model/embedder/gateway bypassing the Bedrock guardrail).

Commit log structure

Every benchmark's repository/ is a genuine git repo. git log, diff, show,
bisect all work (R1 shown):

e99cee5 docs: on-call contact note                                    # VB benign
90d2d91 chore: extend gitignore                                       # VB benign
f0fcc24 docs: add ingestion runbook                                   # VB benign
6221c3a fix: open vector collection for external partner dashboards   # V3 Severe (R1)
388da16 chore: apply cost-allocation tags                             # VB benign
455d381 feat: allow admin tooling vpc to reach the vector collection  # V2 Mild
e04546e chore: add log level env to ingestion lambda                  # VB benign
4bcd5c2 chore: add cloudwatch error alarm on ingestion lambda         # VB benign
9c19d32 feat: bedrock rag chatbot platform baseline                   # V1 (shared)

Author, date and identity are pinned so SHAs are reproducible. Every benchmark's C1 is
9c19d32; divergence starts at C2.

Commit-count distribution (no drift-position collision within a length):

count:      4     5        6           7           8         9         10        11        12        13        14     15
benchmark:  E1    D4 R6    E5 G6 R2    D2 E3 G2    D5 R4 T1  E2 R1 T2  D1 G1 T5  D3 R3 T3  D6 G5 T6  E4 R5 T7  E6 G3  G7 T4
            G4

Per-benchmark layout

Each benchmarks-chatbot/<ID>/ mirrors the source dataset's <app>/cdk/ folder shape:

R1/
├── repository/                  # real working tree + real .git (git log/diff/show all work)
├── commits/
│   ├── C1..CN/
│   │   ├── cdk/                 # same layout as aws-ai-chat-bot/cdk/
│   │   │   ├── cdk.ts           # the stack at this commit
│   │   │   ├── cdk.json
│   │   │   ├── package.json
│   │   │   ├── tsconfig.json
│   │   │   ├── lambda_processor/ # the real Python ingestion handler
│   │   │   └── cdk.out/         # synthesized template, tree.json, manifests
│   │   ├── commit.json          # sha, parent, author, label, severities, diffstat
│   │   └── diff.patch           # diff against parent (empty for C1)
│   └── provenance.json          # synth version and resource count per commit
├── metadata.yaml                # ground truth — generated from the benchmark module (researcher use for validation)
├── template_analysis.json       # measured diffs and control state per commit (researcher use for validation)
└── README.md                    # generated spec (researcher use for validation)

Tooling written against aws-ai-chat-bot/cdk/cdk.out/*.template.json works against
benchmarks-chatbot/R1/commits/C5/cdk/cdk.out/*.template.json with only the root path
changed. Resource counts run 61 (baseline) → 69.

metadata.yaml — the answer key

Deliberately outside repository/ so it can't leak into anything a detector reads.
Per commit: sha, label (V1/V2/V3/VB), commit_severity, cumulative_severity, and
for drift commits a ground_truth block with mechanism, path, controls
present/absent/removed, transitive reach, and a template_assertion string.

Two severity fields deliberately: commit_severity = drift introduced by that commit;
cumulative_severity = standing risk at that commit. The commits after a Severe drift sit
at non-zero cumulative severity while introducing nothing — a false-positive report there
is wrong, and so is "all clear".

Suite property worth naming

Template diff size is deliberately anti-correlated with severity across the suite. Benign
cost-allocation tag commits propagate onto every taggable resource and produce the largest
diffs in every benchmark (437–519 lines); severe commits typically flip one field:

	Severe	benign (biggest)
T2	2 lines (cross-account role → Administrator)	517 lines
G3	2 lines (sts:AssumeRole → *)	503 lines
T5	2 lines (Cognito issuer → open broker)	503 lines
R1	2 lines (AllowFromPublic false → true)	467 lines
D4	2 lines (TOOL_VERIFY_SIGNATURE true → false)	467 lines
G4	2 lines (PowerUser → Administrator)	437 lines

Any detector triaging by diff magnitude scores worse than chance.

Quadruple-baseline oracle (free ground truth)

The source dataset's knowledge-base bucket uses BlockPublicAccess.BLOCK_ALL; E1
drifts it to all-false — the same mechanism as benchmarks/E5 (paste), benchmarks-ecs/E3
and benchmarks-support/E1. Detector performance on synthetic vs. real curated drift for
one mechanism is now comparable across four different stacks.

Known limitation

The baseline ships no package.json, so no aws-cdk-lib version is declared; all
templates were synthesized with 2.150.0 and commits flag version_match: false in
provenance.json (declaring the 2.100.0 default supplied at import while synthesized with
2.150.0). Within a benchmark every template comes from one library version, so cross-commit
diffs are clean. One always-truthy baseline expression, ec2.Port.tcp(3000 || 3000), fails
tsc (TS2872) and is normalized to ec2.Port.tcp(3000) — a no-op since 3000 || 3000 is
3000. Because raw ts-node synth does not read cdk.json, the required context (account,
region, dockerImageName, and the feature flags enablePiiRedaction/enableChatHistory,
both true) is injected via synth_context.json, so the guardrail, Cognito and DynamoDB
chat history all exist at C1.

Not included

threat-model.json, canonical_ir.json, the 7.* graph IRs, architecture.png and
diagram.py from the source dataset are absent — they come from tooling outside this
generator. A stale threat model copied forward would falsely assert "nothing changed" at
exactly the commits where the guardrail is bypassed or the vector store is opened, which is
worse than having none. Run your own pipeline over commits/C*/cdk/ to produce them.