[workshop-sim] Workshop Simulation Report — 2026-09-15 (Run #3, 1000×Monte Carlo)

[github-actions]

Overview

• Date: 2026-09-15
• Students simulated: 46 × 1000 Monte Carlo runs
• Workshop steps available: 30/30
• Overall success rate: 22.7% (95% Monte Carlo interval: 22.3%–23.1%)
• Highest-dropout step: 07-first-workflow (25.1% conditional dropout among 22,389 at-risk runs; 95% Monte Carlo interval: 24.6%–25.7%)
• Lowest curriculum quality step: 04-github-actions-intro.md (overall score 5.4/10)
• Learning KPI index: 2.9/10 (active_learning 4.2 · checkpoint_quality 0.0 · scaffolding 5.0)
• Model: 2026-07-survival-model-v2 / 2026-07-assumption-model-v2 (parameter hash 2024391902)
• Limitation: synthetic results reflect explicit model assumptions; intervals exclude model and population-assumption uncertainty

Part Summary

Part	Files	Mean Score	Std Dev
Part 1 — core path (lessons 00–14)	15	6.69 / 10.0	±1.74
Part 2 — advanced (lessons 15+)	15	6.06 / 10.0	±0.37
Overall corpus	30	6.37 / 10.0	±1.19

No pages are classified as other in this run.

Critical Findings

1. 07-first-workflow is now the single highest-dropout step (25%): its own compile/commit/push instructions are clear, but the required billing decision is delegated to a side quest two hops away, and the Step 7 checkpoint never verifies the learner actually completed billing setup — that debt surfaces at Step 8 for anyone who skipped it.
2. 04-github-actions-intro and 05-agentic-workflows-intro are simultaneously in the bottom-5 curriculum-quality scores and the top-5 dropout steps — this is the workshop's clearest content-quality/dropout correlation, driven almost entirely by checkpoint_quality = 0.0 and low active_learning scores across all 30 steps.
3. Learning KPI index (2.9/10) is low across the whole curriculum — checkpoint_quality scores 0.0 on every single page (no page has an assessed checkpoint format), meaning even learners who complete the workshop get weak formative feedback. This is a systemic scaffolding gap, not isolated to any one step.
4. The most important repair belongs to Part 1 (lessons 00–14): four of the top five dropout steps (07-first-workflow, 05-agentic-intro, 04-actions-intro, 05c-agentic-practice) and the lowest-quality file are all in the core path that every learner must pass through before reaching any Part 2 content.

Top Repairs to Prioritize

Note: some student dropout is expected and acceptable. Repairs must maintain or improve the learning KPI index — do not lower the cognitive bar or remove practice to chase headline completion numbers.

1. Add an explicit "billing configured" checkbox to the Step 7 checkpoint that requires the learner to confirm they completed the side-quest billing path before marking the step done, closing the multi-hop detour gap (completion impact: ↑ · learning KPI impact: ↔)
2. Add recovery/troubleshooting guidance and a worked-answer walkthrough to 04-github-actions-intro's labeling exercise and real-workflow exploration activity, raising checkpoint_quality and active_learning without reducing the conceptual content (completion impact: ↑ · learning KPI impact: ↑)
3. Strengthen 05-agentic-workflows-intro's active-learning and checkpoint design (currently active_learning 2.4/10, the lowest in the corpus) with an interactive self-check tied to the "agentic vs. standard workflow" distinction, rather than simplifying the concept (completion impact: ↑ · learning KPI impact: ↑)

Dropout by step

Step	At-risk runs	Dropouts (~)	Conditional dropout rate	95% MC interval	Failure mode	Top reason
07-first-workflow	22,389	5,628	25.1%	24.6%–25.7%	access barrier	Copilot model access not confirmed before authoring (copilot-access-missing, 3,805) plus general authoring friction translating the tutorial into a valid workflow file (workflow-authoring-friction, 1,823)
05-agentic-intro	37,381	6,804	18.2%	17.8%–18.6%	learning barrier	Learners struggle to grasp the core agentic-vs-classic-workflow concept (agentic-concept-gap, 4,703), compounded by deployment/tooling capability gaps (deployment-capability-gap, 2,101)
04-actions-intro	44,227	6,846	15.5%	15.1%–15.8%	learning barrier	Concept overload (concept-overload, 6,846) — five new Actions primitives introduced with a low-support self-check
05c-agentic-practice	30,577	3,913	12.8%	12.4%–13.2%	learning barrier	Learners misclassify agentic vs. standard workflow examples (agentic-classification-gap, 3,913)
05b-agentic-security	26,664	2,692	10.1%	9.7%–10.5%	learning barrier	Learners cannot connect the safety/guardrail concepts to concrete examples (agentic-security-gap, 2,692)

Curriculum quality and learning KPIs

Step file	Overall score	active_learning	checkpoint_quality	scaffolding	Learning KPI index	Lowest rubric dimension	Repair priority
04-github-actions-intro.md	5.39	3.9	0.0	5.0	2.78	checkpoint_quality	High
05-agentic-workflows-intro.md	5.43	2.4	0.0	5.0	2.24	checkpoint_quality / active_learning	High
14b-pr-reviewer-workflow.md	5.45	4.3	0.0	5.0	2.83	checkpoint_quality	Medium
15-conditional-logic.md	5.53	3.8	0.0	5.0	2.69	checkpoint_quality	Medium
17-add-mcp-tools.md	5.61	3.1	0.0	5.0	2.49	checkpoint_quality	Medium
08-run-your-workflow.md	5.67	3.0	0.0	5.0	2.45	checkpoint_quality	Low (dropout already resolved by content-aware review)
05b-agentic-workflows-security.md	5.75	2.5	0.0	5.0	2.27	checkpoint_quality / active_learning	High
05c-agentic-workflows-practice.md	6.22	6.2	0.0	5.0	3.62	checkpoint_quality	Medium
07-your-first-workflow.md	6.25	6.3	0.0	5.0	3.65	checkpoint_quality	High (billing-detour repair)
Cohort mean (all 30 steps)	6.37	4.17	0.0	5.0	2.88	checkpoint_quality (all steps)	—

Segment breakdowns

By technical level

Level	Mean success rate	Students
beginner	0.4%	11
github-basic	15.1%	19
actions-user	46.5%	11
advanced	48.2%	5

By personality

Personality	Mean success rate	Students
impatient	26.0%	6
methodical	25.5%	12
skeptical	22.0%	7
confused	21.6%	6
curious	19.8%	15

By UI preference

UI preferred	Mean success rate	Students
false (CLI-first)	32.9%	24
true (browser/UI-first)	11.5%	22

Notable student journeys (3)

• Surprising success: Learner 026 (advanced, devops, confused personality, CLI, no UI preference) reached a 59.3% success rate despite the "confused" trait, because the CLI/devops combination avoids most of the browser-vs-terminal ambiguity in Steps 4–8 and the learner's advanced level offsets typical Copilot-access friction.
• Unexpected dropout: Learner 003 (github-basic, skeptical, program-manager background, CCA tool, UI-preferred) dropped out entirely at 04-actions-intro (0% success) — a program-manager evaluator with no coding background hits the concept-overload wall in Step 4 before ever reaching the agentic-workflow authoring steps that are more central to their team-evaluation-style goal.
• Content-gap case: Every beginner/no-coding CCA learner (Learners 015, 017, 018, 020) failed 100% of the time at 04-actions-intro, confirming this is a genuine content gap (concept overload with zero recovery support) rather than a tooling mismatch, since these learners had not yet reached any terminal- or auth-dependent step.

Generated by 🔬 Workshop Student Simulator · copilot · auto · 166.2 AIC · ⌖ 7.33 AIC · ⊞ 15.2K · ◷

• expires on Sep 16, 2026, 5:43 PM UTC

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #3418

View newer issue

---

This action was performed automatically by the Workshop Student Simulator workflow.

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #3419

View newer issue

---

This action was performed automatically by the Workshop Student Simulator workflow.