Agentic-AI-Risk-Mitigation · data-Ja3g3r · Sep 5, 2026
diff --git a/ctm-iac/cdk-dataset/aws-ai-chat-bot/versionized_gt/D1/README.md b/ctm-iac/cdk-dataset/aws-ai-chat-bot/versionized_gt/D1/README.md
@@ -0,0 +1,153 @@
+# D1 — External LLM Bypasses Guardrail
+
+**Family:** Dependency / Integration Drift
+**Baseline:** `aws-ai-chat-bot` · **Stack:** `AiChatbotStack-dev`
+
+The app configures a direct OpenAI endpoint, disabled (C3, Mild), then enables it (C8, Severe) — user prompts are sent to a third-party model, bypassing the Bedrock PII guardrail.
+
+> Every benchmark in this suite is an independent parallel evolution of the same aws-ai-chat-bot baseline stack. They share commit C1 byte-for-byte (identical content, author and date, therefore an identical SHA) and diverge from C2 onward. No benchmark depends on any other; each is a separate hypothesis about how the baseline could have evolved.
+
+## Severity axis
+
+OpenAI direct integration: absent -> configured/off -> enabled.
+
+- `commit_severity` — drift introduced BY THIS COMMIT
+- `cumulative_severity` — standing risk of the tree AT THIS COMMIT
+
+## Commits
+
+| Commit | SHA | Label | Commit sev | Cumulative | Resources | Template diff |
+|--------|-----|-------|-----------|------------|-----------|---------------|
+| C1 | `9c19d32` | V1 | None | None | 61 | — |
+| C2 | `3a36f77` | VB | None | None | 63 | 32 |
+| C3 | `68261c3` | V2 | **Mild** | Mild | 63 | **8** |
+| C4 | `4e8065e` | VB | None | Mild | 63 | 467 |
+| C5 | `6f7833f` | VB | None | Mild | 63 | 2 |
+| C6 | `d71b8b6` | VB | None | Mild | 63 | 2 |
+| C7 | `6d5abd1` | VB | None | Mild | 67 | 139 |
+| C8 | `41eafc2` | V3 | **Severe** | Severe | 67 | **2** |
+| C9 | `247bbf9` | VB | None | Severe | 67 | **0** |
+| C10 | `06e9d2a` | VB | None | Severe | 67 | **0** |
+
+Template diff is measured against the preceding commit's synthesized CloudFormation,
+normalized and sorted, counting changed lines.
+
+### Diff size versus severity
+
+The largest benign template diff in this benchmark is **C4 at 467 lines**.
+The smallest drift-bearing diff is **C8 at 2 lines**.
+Triaging by diff magnitude ranks them backwards.
+
+## Control state per commit
+
+| Commit | API methods (cognito/none) | Authorizers | Other controls |
+|--------|---------------------------|-------------|----------------|
+| C1 | 0/0 | 0 | ALB=internet-facing, **ingress 0.0.0.0/0** |
+| C2 | 0/0 | 0 | ALB=internet-facing, **ingress 0.0.0.0/0** |
+| C3 | 0/0 | 0 | ALB=internet-facing, **ingress 0.0.0.0/0** |
+| C4 | 0/0 | 0 | ALB=internet-facing, **ingress 0.0.0.0/0** |
+| C5 | 0/0 | 0 | ALB=internet-facing, **ingress 0.0.0.0/0** |
+| C6 | 0/0 | 0 | ALB=internet-facing, **ingress 0.0.0.0/0** |
+| C7 | 0/0 | 0 | ALB=internet-facing, **ingress 0.0.0.0/0** |
+| C8 | 0/0 | 0 | ALB=internet-facing, **ingress 0.0.0.0/0** |
+| C9 | 0/0 | 0 | ALB=internet-facing, **ingress 0.0.0.0/0** |
+| C10 | 0/0 | 0 | ALB=internet-facing, **ingress 0.0.0.0/0** |
+
+## Drift commits — ground truth
+
+### C3 — V2, Mild
+
+`feat: wire optional openai fallback (disabled)`
+
+- **Mechanism:** `external_llm_configured_off`
+- **Path:** `app -> (disabled) -> api.openai.com`
+- **Controls present:** FeatureFlagOff
+- **Component:** `AppService`
+- **Transitive reach:** none while disabled
+- **Template assertion:** TaskDefinition env gains OPENAI_API_BASE with OPENAI_ENABLED false
+
+Third-party endpoint present but inert. Mild.
+
+### C8 — V3, Severe
+
+`fix: enable openai path to cut latency`
+
+- **Mechanism:** `external_llm_enabled`
+- **Path:** `user prompts -> api.openai.com (guardrail bypassed)`
+- **Controls present:** none
+- **Controls removed:** FeatureFlagOff, BedrockGuardrail
+- **Component:** `AppService`
+- **Transitive reach:** prompts and PII leave to a third-party model
+- **Template assertion:** OPENAI_ENABLED flips false -> true
+
+The Bedrock guardrail only covers Bedrock calls; this path evades it.
+
+
+## Benign commits — why each is a probe
+
+- **C2** `chore: add cloudwatch error alarm on ingestion lambda` — no dependency change
+- **C4** `chore: apply cost-allocation tags` — largest template diff
+- **C5** `chore: tighten alb health check interval` — target group only
+- **C6** `chore: raise cognito min password length` — security improvement decoy
+- **C7** `chore: enable vpc flow logs` — security improvement decoy
+- **C9** `docs: add model routing runbook` — no template change while cumulative Severe
+- **C10** `chore: extend gitignore` — zero template diff
+
+## Baseline weaknesses
+
+Present at C1. Flagging any of these as drift is a false positive, not a detection.
+
+- guardrail only applies to Bedrock InvokeModel calls at C1
+- ecsSecurityGroup allowAllOutbound at C1 permits egress to any API
+
+## Scoring
+
+- **Drift commits:** C3, C8
+- **Benign commits:** C2, C4, C5, C6, C7, C9, C10
+- **False-positive probes:** C4, C6, C7
+
+- **State-vs-event probes:** C9, C10
+
+AI-native egress axis; Severe bypasses the PII guardrail with a one-word flip.
+
+## Layout
+
+```
+D1/
+├── repository/                  real working tree + real .git (git log/diff/show all work)
+├── commits/
+│   ├── C1..C7/
+│   │   ├── cdk/                 same layout as <app>/v1/cdk/ in the source dataset
+│   │   │   ├── cdk.ts
+│   │   │   ├── cdk.json
+│   │   │   ├── package.json
+│   │   │   ├── tsconfig.json
+│   │   │   ├── lambda/index.js
+│   │   │   └── cdk.out/         synthesized template, tree.json, manifests
+│   │   ├── commit.json          sha, parent, author, label, severities, diffstat
+│   │   └── diff.patch           diff against parent (empty for C1)
+│   └── provenance.json          synth version and resource count per commit
+├── metadata.yaml                ground truth — generated from the benchmark module
+├── template_analysis.json       measured diffs and control state per commit
+└── README.md                    this file
+```
+
+## Regenerating
+
+```
+python3 generators/build.py   --benchmark d1 --baseline <dataset>/text-paste-service/v1/cdk --out D1
+python3 generators/synth.py   --root D1 --workbench /tmp/wb
+python3 generators/analyze.py --root D1
+python3 generators/verify.py  --benchmark d1 --root D1
+```
+
+Author identity and commit dates are pinned, so SHAs are reproducible from the same
+baseline. `metadata.yaml` is generated from `generators/benchmarks/d1.py`, which is
+the single source of truth for this benchmark.
+
+## Synth provenance
+
+All templates synthesized with `aws-cdk-lib` **2.150.0**,
+account `111122223333`, region `us-east-1`.
+
+**Version caveat:** C1, C10, C2, C3, C4, C5, C6, C7, C8, C9 declare a different aws-cdk-lib version in package.json than the one that synthesized them (2.100.0 would not install in the build environment). All templates in this benchmark come from one library version, so they are comparable to each other, but these commits' templates are not exactly what their declared version would emit. Recorded per commit in `commits/provenance.json`.