[MomoSolaris8]

The first production trace in Langfuse showed an English question answered in German. This PR makes the agent answer in the language of the question. It also fixes the refusal detection, which broke as soon as answers were no longer German.

Why answers were German

• The system prompt said "Antworte auf Deutsch."
• detect_response_language counts common marker words, and a tie defaults to de. "Does my car insurance cover glass damage?" contains no English marker, so it was detected as German. The online eval and the MCP tool path use the same detector.

What changed

• Language detection: the marker lists now hold common function words (pronouns, auxiliaries, question words, prepositions). Words shared by both languages, like in, are left out.
• Answer language:
  • generate_answer adds Antwortsprache: <language> to the user message, on both the text path and the image path.
  • The system prompt and image prompt answer in that language. German policy terms from the sources are kept as they are, for example Teilkasko.
  • PROMPT_VERSION is now insurance-knowledge-v2.1.
• Refusals no longer depend on German wording.
  • The model now outputs a marker instead of writing the refusal itself:
    • [[INSUFFICIENT_EVIDENCE]] for a high-risk question without evidence.
    • [[TERM_NOT_DEFINED]] followed by one clarification question, for an undefined term.
  • The code replaces the marker with fixed wording in the answer language (REFUSAL_TEXTS and TERM_NOT_DEFINED_TEXTS for de, en and zh). The German refusal is unchanged.
  • answer_reports_insufficient_evidence now matches that fixed wording, so the evidence status no longer depends on how the model phrases a refusal.
  • When retrieval finds nothing, the workflow also refuses in the language of the question.
• Test fix: the langfuse_spans fixture now resets the Langfuse clients before and after each test.
  • An earlier test created a client from the local .env key. With two clients in the process, get_client() disabled tracing and the tracing tests failed locally. CI has no .env, so it passed there.
  • The fixture now creates a new exporter for each test.

How the refusal problem was found

After the prompt change, all unit tests passed. A real run then showed two problems:

• The model translated the "fixed" German refusal into English.
• It wrote its own English sentence for an undefined term.

Neither matched the German wording that the detection looked for, so both were marked sufficient. The eval uses the same check, so it would not have caught this either. That is why the refusal now goes through a marker.

Testing

• New unit and component tests for:

  • language detection;
  • the Antwortsprache instruction on both paths;
  • marker replacement in each language;
  • refusal when retrieval finds nothing;
  • a fixed text for every supported language.

• 140 unit and component tests pass.

• I ran four real questions against qwen-plus locally and checked each one:

  Question	Evidence	Answer
  Does my car insurance cover glass damage?	sufficient	English, keeps "Teilkaskoversicherung", cited
  Ist Glasbruch in meiner Kfz-Versicherung gedeckt?	sufficient	German, cited
  Meteorite hitting my house in Mars orbit	insufficient	fixed English refusal
  What does the term Schädigung mean?	insufficient	fixed English sentence plus one clarification question, no guessed definition

Follow-ups (not in this PR)

• With a claim open, the frontend adds a German prefix to the question ("Aktueller Schadenfall: …"). That prefix can push language detection towards German.
• The test fixture uses langfuse._client.resource_manager, which is internal to the SDK. A Langfuse upgrade may break it. If that happens, the tracing tests fail loudly.
• Marker-based detection is enough for now. Structured JSON output would be the next step if more answer types need machine-readable status.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
support-agent-one-coyl	Ready	Preview	Oct 31, 2026 11:02pm UTC