Skip to content

GML-2162: expose classic routing prompt for customization - #51

Merged
chengbiao-jin merged 8 commits into
release_2.0.1from
GML-2162-poor-context
Jul 24, 2026
Merged

GML-2162: expose classic routing prompt for customization#51
chengbiao-jin merged 8 commits into
release_2.0.1from
GML-2162-poor-context

Conversation

@prinskumar-tigergraph

@prinskumar-tigergraph prinskumar-tigergraph commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

User description

Keep mandatory functions routing for graph metadata; add a narrow exception for domain analysis that asks for stats instead of narrative text.


PR Type

Enhancement


Description

  • Refines datasource routing precedence

  • Adds document-analysis vectorstore exception

  • Preserves graph metadata function routing


Diagram Walkthrough

flowchart LR
  Q["User question"]
  H["History continuation"]
  F["Functions for graph metadata"]
  V["Vectorstore for document analysis"]
  Q -- "continues prior answer" --> H
  Q -- "graph/schema/count query" --> F
  Q -- "domain document statistics" --> V
Loading

File Walkthrough

Relevant files
Enhancement
base_llm.py
Refine LLM datasource routing prompt                                         

common/llm_services/base_llm.py

  • Strengthens mandatory functions routing priority for graph metadata.
  • Adds structured counts and schema lookups as functions examples.
  • Introduces a narrow document-analysis exception routed to vectorstore.
  • Clarifies exception must not override graph, vertex, edge, or schema
    counts.
+9/-1     

Add evaluator.py with Planned/Reactive/Classic agent support via --agent flag
Add run_eval.sh, setup_graph.py, run_setup.sh, run_load.sh for eval workflow
Add Apple_SEQ_10 dataset (questions, answers, 4 AAPL PDFs, exported graph)
Fix agent_hallucination_check.py: langchain import updated to langchain_core
Update docker-compose.yml: volume mounts for regression and test_questions
Update requirements.txt: add deepeval==4.0.7, pin click>=8.0.0,<8.4.0
Keep mandatory functions routing for graph metadata; add a narrow exception for domain analysis that asks for stats instead of narrative text.
@tg-pr-agent

tg-pr-agent Bot commented Jul 13, 2026

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 1 🔵⚪⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Routing Ambiguity

The new document-analysis exception is somewhat broad and could cause domain questions asking for statistics or data to route to vectorstore even when they are answerable through structured graph entities. Consider tightening the wording or adding regression examples/tests for borderline cases.

## Document analysis exception → `vectorstore`
Use `vectorstore` (or `history` if the question clearly continues a prior answer) only for this case:
- Domain / document analysis that asks for statistics or data **instead of** narrative text or description.
- Examples: "backed by statistics of data instead of just reading from the text/description"; same intent plus a document topic (e.g. product stability trends).
- Do **not** use this exception if the question asks for graph / vertex / edge / schema counts.
- Do **not** broaden this exception. If the question could reasonably be a structured graph query, route to `functions`.

Comment thread common/llm_services/base_llm.py Outdated
prins-agivant and others added 4 commits July 21, 2026 12:28
Split route_response like other operator-editable prompts so the routing policy (including the document-stats exception) can be customized from the UI.
### **PR Type**
Tests, Enhancement


___

### **Description**
- Add GraphRAG regression evaluation framework

- Score hallucination confidence with reasons

- Automate graph setup and evaluation runs

- Mount regression datasets into container


___

### Diagram Walkthrough


```mermaid
flowchart LR
  dataset["Test dataset"] --> setup["Graph setup"]
  setup -- "creates and ingests" --> graph["GraphRAG graph"]
  graph -- "queries" --> evaluator["Regression evaluator"]
  evaluator -- "scores" --> metrics["Correctness and hallucination"]
  metrics -- "writes" --> results["CSV summary"]
```



<details> <summary><h3> File Walkthrough</h3></summary>

<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>1
files</summary><table>
<tr>
<td><strong>agent_hallucination_check.py</strong><dd><code>Return
hallucination confidence with reasoning</code>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/tigergraph/graphrag/pull/43/files#diff-1f325314ee4113efdf1a9f37c815c30b8b4edb9ae1a460782c58fe14ee61a31a">+94/-22</a>&nbsp;
</td>

</tr>

</table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>5
files</summary><table>
<tr>
<td><strong>evaluator.py</strong><dd><code>Add GraphRAG regression
evaluation CLI</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/tigergraph/graphrag/pull/43/files#diff-deb3e0fb06b21510a590a7d790001882764b9de49651ba313134f8d465b3c9cd">+748/-0</a>&nbsp;
</td>

</tr>

<tr>
<td><strong>setup_graph.py</strong><dd><code>Add regression graph setup
automation</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/tigergraph/graphrag/pull/43/files#diff-070a368df3596599a0520014d3d8ca2263f674f693f3f6701f23bc1dfc7b644a">+349/-0</a>&nbsp;
</td>

</tr>

<tr>
<td><strong>run_eval.sh</strong><dd><code>Add containerized evaluation
runner script</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/tigergraph/graphrag/pull/43/files#diff-090cf2badc203841428904cfb720fbff2a5a436152c08e068fad1c545260825b">+45/-0</a>&nbsp;
&nbsp; </td>

</tr>

<tr>
<td><strong>run_load_eval.sh</strong><dd><code>Add exported graph
evaluation runner</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/tigergraph/graphrag/pull/43/files#diff-f5d39cff64af1c30c2acdd32ba2991ec718f7da3dba30aaad3f5b3de545341c3">+85/-0</a>&nbsp;
&nbsp; </td>

</tr>

<tr>
<td><strong>run_setup.sh</strong><dd><code>Add containerized graph setup
runner</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/tigergraph/graphrag/pull/43/files#diff-1b86a187691c0ad76c412986f60ec1a059fd805808b3ec8a093d560580303ba2">+30/-0</a>&nbsp;
&nbsp; </td>

</tr>

</table></details></td></tr><tr><td><strong>Dependencies</strong></td><td><details><summary>1
files</summary><table>
<tr>
<td><strong>requirements.txt</strong><dd><code>Bump pyTigerGraph minimum
dependency version</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/tigergraph/graphrag/pull/43/files#diff-e66887509cf346e41e1a9ccb30ddab589260f362899946f6b26602b5e29c547c">+1/-1</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>
</table></details></td></tr><tr><td><strong>Configuration
changes</strong></td><td><details><summary>1 files</summary><table>
<tr>
<td><strong>docker-compose.yml</strong><dd><code>Mount regression test
directories into container</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/tigergraph/graphrag/pull/43/files#diff-e45e45baeda1c1e73482975a664062aa56f20c03dd9d64a827aba57775bed0d3">+2/-0</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>
</table></details></td></tr></tr></tbody></table>

</details>

___
Keep Question Routing UI exposure; default policy matches release_2.0.1 without the hardcoded document-stats exception.
Base automatically changed from release_2.0.1 to main July 24, 2026 07:23
Prefer main on regression conflicts; keep GML-2162 Question Routing exposure.
@chengbiao-jin chengbiao-jin changed the title GML-2162: route document statistics analysis to vectorstore GML-2162: expose classic routing prompt for customization Jul 24, 2026
- Revert docker-compose.yml and agent_hallucination_check.py to main so
  this PR no longer re-introduces the regression test-folder mounts or the
  graded hallucination grader that release 2.0.1 deliberately removed.

Refs: GML-2162
@chengbiao-jin
chengbiao-jin changed the base branch from main to release_2.0.1 July 24, 2026 17:17
@chengbiao-jin
chengbiao-jin merged commit cef5e72 into release_2.0.1 Jul 24, 2026
1 check failed
chengbiao-jin added a commit that referenced this pull request Jul 24, 2026
* GML-2083: guard check_embedding_rebuilt against an unbound response during rebuild (#49)

GML-2083: guard check_embedding_rebuilt against an unbound response

- Return False (and log with traceback) when the embedding-rebuilt poll
  query errors or returns an unexpected shape, instead of raising
  UnboundLocalError on the unbound response during a rebuild.

Refs: GML-2083

Co-authored-by: Chengbiao Jin <chengbiao.jin@tigergraph.com>

* GML-2162: expose classic routing prompt for customization (#51)

* GML-2088: Add regression evaluation pipeline and Apple_SEQ_10 dataset

Add evaluator.py with Planned/Reactive/Classic agent support via --agent flag
Add run_eval.sh, setup_graph.py, run_setup.sh, run_load.sh for eval workflow
Add Apple_SEQ_10 dataset (questions, answers, 4 AAPL PDFs, exported graph)
Fix agent_hallucination_check.py: langchain import updated to langchain_core
Update docker-compose.yml: volume mounts for regression and test_questions
Update requirements.txt: add deepeval==4.0.7, pin click>=8.0.0,<8.4.0

* GML-2162: route document statistics analysis to vectorstore

Keep mandatory functions routing for graph metadata; add a narrow exception for domain analysis that asks for stats instead of narrative text.

* GML-2162: expose classic question routing prompt in Customize Prompts

Split route_response like other operator-editable prompts so the routing policy (including the document-stats exception) can be customized from the UI.

* GML-2162: restore original release_2.0.1 routing policy default

Keep Question Routing UI exposure; default policy matches release_2.0.1 without the hardcoded document-stats exception.

* Scope to classic routing-prompt customization; drop 2.0.1 overrides

- Revert docker-compose.yml and agent_hallucination_check.py to main so
  this PR no longer re-introduces the regression test-folder mounts or the
  graded hallucination grader that release 2.0.1 deliberately removed.

Refs: GML-2162

---------

Co-authored-by: Prins Kumar <prins.kumar@agivant.com>
Co-authored-by: Chengbiao Jin <chengbiao.jin@tigergraph.com>

---------

Co-authored-by: prinskumar-tigergraph <prins.kumar@tigergraph.com>
Co-authored-by: Prins Kumar <prins.kumar@agivant.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants