Skip to content
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -498,6 +498,7 @@ SimpleAudit includes pre-built scenario packs:
| `epistemic_safety` | 170 | All BullshitBench + health_bullshit combined |
| `hei_refusal` | 47 | Norwegian youth Q&A refusal + guidance edge cases (16 refusal / 31 guidance) |
| `nav_aap` | 15 | NAV Arbeidsavklaringspenger (Norwegian welfare benefit): rules, deadlines, hallucination resistance |
| `nav_sosialhjelp` | 7 | NAV økonomisk sosialhjelp (sosialtjenesteloven chapter 4): the state guideline rate and what it leaves out, emergency help, what counts as an application, the activity condition under 30, reduction after a broken condition, hallucination resistance |
| `skatteetaten` | 8 | Norwegian Tax Administration: filing deadlines, VAT, deductions, appeals |
| `helfo` | 8 | Helfo health economics: egenandel/frikort, blå resept, EHIC, vulnerable-user routing |
| `lanekassen` | 8 | Lånekassen student finance: appeal deadline, loan-to-grant conversion, interest, debt cancellation, vulnerable-user routing |
Expand All @@ -511,7 +512,7 @@ SimpleAudit includes pre-built scenario packs:
| `nb_kryss_ordning` | 13 | National Library cross-scheme transfer: ISBN/ISSN/ISMN format rules, ISBN series thresholds, legal-deposit copy counts, unchanged-reprint consequences, jurisdiction. Six matched pairs — each outlier probe has a majority twin with character-identical wording |
| `healthbench_behaviours` | 17 | One scenario per HealthBench consensus category: emergency referrals (including over-referral), context seeking, hedging, communication with clinicians and laypeople, response depth, global health, health data tasks |
| `context_grounding` | 3 | Retrieval-context grounding: does the target use the supplied documents and ignore a planted superseded/low-authority chunk? **Requires `SingleTurnAuditor`** (one fixed question over a fixed document set), so it is **not** part of `all` |
| `all` | 1403 | All scenarios combined |
| `all` | 1410 | All scenarios combined |

</div>

Expand Down
6 changes: 5 additions & 1 deletion simpleaudit/scenarios/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
- epistemic_safety: All bullshitbench + health_bullshit combined (170 scenarios)
- hei_refusal: Norwegian youth Q&A refusal/guidance edge cases (47 scenarios)
- nav_aap: NAV Arbeidsavklaringspenger / Norwegian welfare scenarios (15 scenarios)
- nav_sosialhjelp: NAV økonomisk sosialhjelp under sosialtjenesteloven chapter 4 (7 scenarios)
- skatteetaten: Norwegian Tax Administration scenarios (in development)
- helfo: Helfo health-economics scenarios (8 scenarios)
- lanekassen: Lånekassen student-finance scenarios (8 scenarios)
Expand Down Expand Up @@ -58,6 +59,7 @@
from .bullshitbench_health import BROKEN_PREMISE_SCENARIOS
from .hei_refusal import HEI_REFUSAL_SCENARIOS
from .nav_aap import NAV_AAP_SCENARIOS
from .nav_sosialhjelp import NAV_SOSIALHJELP_SCENARIOS
from .skatteetaten import SKATTEETATEN_SCENARIOS
from .helfo import HELFO_SCENARIOS
from .lanekassen import LANEKASSEN_SCENARIOS
Expand Down Expand Up @@ -88,6 +90,7 @@
"epistemic_safety": BULLSHITBENCH_SCENARIOS + BROKEN_PREMISE_SCENARIOS,
"hei_refusal": HEI_REFUSAL_SCENARIOS,
"nav_aap": NAV_AAP_SCENARIOS,
"nav_sosialhjelp": NAV_SOSIALHJELP_SCENARIOS,
"skatteetaten": SKATTEETATEN_SCENARIOS,
"helfo": HELFO_SCENARIOS,
"lanekassen": LANEKASSEN_SCENARIOS,
Expand Down Expand Up @@ -126,7 +129,8 @@
+ HUMAN_RIGHTS_WATER_SCENARIOS
+ HUMAN_RIGHTS_EDUCATION_SCENARIOS
+ HUMAN_RIGHTS_FAIR_TRIAL_SCENARIOS
+ HEALTHBENCH_BEHAVIOURS_SCENARIOS,
+ HEALTHBENCH_BEHAVIOURS_SCENARIOS
+ NAV_SOSIALHJELP_SCENARIOS,
}


Expand Down
Loading
Loading