docs: financial-advice-bot example (EU AI Act + BFS) (#51) - #99
Conversation
…lution#51) Six-file example evaluating a general financial-education assistant against EU AI Act + BFS fair-lending/model-risk policies, with an explicit non-authorisation disclaimer. Signed-off-by: B. <b@thewealthsalons.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
🟡 Changes recommended
The new example’s policy_config.yaml and README contain inconsistencies with the stated acceptance criteria and the actual behavior of the provided run.py command.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds a new forkable example under examples/financial-advice-bot/ that demonstrates evaluating an education-only financial assistant against EU AI Act plus BFS (fair-lending/model-risk) policies, and links it from the examples index.
Changes:
- Added the full “financial-advice-bot” example bundle (contract, runnable script, policy config, and expected report narrative).
- Documented “honest scope” boundaries (green report ≠ permission to deliver regulated financial advice).
- Added the example to
examples/README.md’s table of available examples.
File summaries
| File | Description |
|---|---|
| examples/README.md | Adds a table entry linking to the new financial-advice example. |
| examples/financial-advice-bot/README.md | Introduces the example, scope boundaries, and how to run/adapt it. |
| examples/financial-advice-bot/run.py | Runnable script that builds an app from input_contract.json and evaluates against EU AI Act + BFS. |
| examples/financial-advice-bot/policy_config.yaml | Declares intended frameworks/evaluators and report formats for the example. |
| examples/financial-advice-bot/input_contract.json | Provides an 8-interaction fixture contract emphasizing refusals, neutrality, and boundaries. |
| examples/financial-advice-bot/sample_interactions.json | Small spliceable interaction subset for quick reuse. |
| examples/financial-advice-bot/expected_report.md | Describes the expected pass case and common failure modes for the generated report. |
Review details
- Files reviewed: 7/7 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dafccfd5d5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Use id bfs at industry_specific/bfs/v1, clarify that run.py emits PDF and does not load policy_config, and attempt to add global regulations.
…t schema Load policy_config.yaml in run.py, use a UUID AiCertifyContract with model_info + declared context, emit configured report formats, and exit nonzero on evaluation errors.
|
Review feedback adjudicated: policy_config wired into run.py, canonical contract schema + declared context, multi-format reports, nonzero exit on eval errors (4b6964b). |
kmadan
left a comment
kmadan
left a comment
There was a problem hiding this comment.
Thanks for this, and thanks especially for the way you handled the first review round. Six findings came in from Copilot and Codex and you turned all six around properly rather than papering over them. I checked each fix by running the code, not just reading it, and they hold up: the contract loads, the framework ids resolve, policy_config.yaml is genuinely read now, and the nonzero exit path works.
One thing I want to call out before the requests, because it deserves credit. Your input_contract.json is the only one of the four examples that actually passes load_contract():
financial-advice-bot: OK
hiring-screening-bot: FAILED contract_id not a UUID; 'model' instead of 'model_info'
customer-support-bot: FAILED same
healthcare-triage-bot: FAILED same
So where this PR diverges from the sibling examples, it is moving toward the canonical schema and the others are the ones that are wrong. Please keep it as it is. I will raise a separate issue to migrate the other three.
There are two things I need changed before merge, and one smaller cleanup.
1. The example exits nonzero on a clean checkout
Running it from a fresh clone gives this, once per report format:
error: evaluation failed for global (json): Non-relative patterns are unsupported
Three of the nine evaluations fail. This is our bug, not yours. RegulationSet._get_available_regulations in aicertify/regulations.py explicitly skips the global category with a TODO: Fix this issue about folder topology, so add("global") falls through to find_matching_policy_folders, which hands back an absolute path that the policy glob then rejects.
Your PR is the first example to make policy_config.yaml load-bearing, so it is the first one to actually add global. The others list it in their YAML too, but their run.py never reads the file, so the config is decorative and the bug stays hidden. I confirmed that by running hiring-screening-bot/run.py: it exits 0 with no errors purely because it never adds global and never inspects its results.
You surfaced a real defect, which I appreciate. But a forkable example that exits 1 is not something we can ship. Please drop global from frameworks in policy_config.yaml with a short comment pointing at the library limitation. I will track the topology fix separately.
2. expected_report.md does not match what the run produces
It claims PASS for all three frameworks. Measured on this branch:
| Framework | Claimed | Actual |
|---|---|---|
| EU AI Act v1 | PASS | 14 PASS / 15 FAIL |
| BFS | PASS | 0 PASS / 4 FAIL, including fair_lending |
| Global | PASS | never evaluates |
A good part of that is downstream of the example's own documented default. With OPENAI_API_KEY popped, content_safety, accuracy, manipulation, social_scoring, emotion_recognition, biometric_categorization and vulnerability_exploitation all fail to initialise, and their policies fail with them.
To be clear about where the fault sits: this is a convention you inherited from the examples you were asked to follow. hiring-screening-bot/expected_report.md makes the same unverified claim and is more precisely wrong, asserting "29 policies, 0 failures" plus three global.v1.* allow rows that cannot be produced at all. I am asking you to fix it here rather than there because honest boundary claims are the whole point of this particular example. A report that overstates a pass undercuts the "a green report does not authorise advice delivery" message you built the example around.
Please replace the table with what the run actually returns, and say plainly which failures come from evaluators that need an API key. That version is more useful to a reader than an aspirational one.
3. Smaller cleanup
examples/README.md got the table row, but two other spots still need it. "Financial-advice bot" is still listed under "Contributing an example" as a wanted area, and the new expected_report.md is missing from the "Inspecting expected output" list.
Not blocking, worth a thought
report.formats: [pdf, markdown, json] drives three complete re-evaluations, nine in total, which took about 11 minutes here against roughly 4 for the sibling example. app.get_report() is keyed by regulation and overwritten on each pass, so the final printout and the missing-file check only ever cover the last format. Defaulting to pdf would keep the config load-bearing without charging a first-time reader three full runs. Your call, happy either way.
One process note that is not about your code: the commits are unsigned. Since the branch lives on your fork I cannot re-sign them, so I will squash-merge, which signs the result. Nothing for you to do.
None of this is a lot of work, and the example is a good one. Ping me when it is updated and I will re-run it.
Description
Adds
examples/financial-advice-bot/per #51 — education-only financial assistant evaluated against EU AI Act + gopal BFS fair-lending/model-risk policies.Acceptance criteria
policy_config.yamlselecting eu_ai_act + bfs + globalrun.pyfollows customer-support / hiring patternexpected_report.mdcovers pass case + failure modesexamples/README.mdValidation
git submodule update --init aicertify/opa_policies pip install -e . python examples/financial-advice-bot/run.pyCloses #51