Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions cmd/memory/recall.go
Original file line number Diff line number Diff line change
Expand Up @@ -101,8 +101,19 @@ func toCompact(resp search.RecallResponse) compactResponse {
var recallCmd = &cobra.Command{
Use: "recall [keyword]",
Short: "Retrieve insights by keyword",
Long: "Search for insights using intent-aware graph-enhanced retrieval. Use --basic for simple SQL LIKE matching.",
Args: cobra.MinimumNArgs(1),
Long: `Search for insights using intent-aware graph-enhanced retrieval. Use --basic for simple SQL LIKE matching.

Automatic intent uses a limited set of question cues in English, Mandarin Chinese
(simplified/traditional), Hindi (Devanagari), Spanish, Modern Standard Arabic,
French, Bengali (Bengali script), Portuguese, Indonesian, Russian (Cyrillic), and
German. It does not infer meaning or recognize every phrasing or transliteration.
Unrecognized cues fall back to GENERAL. Conflicting cues involving the additional
languages also use GENERAL; English/Chinese-only queries retain legacy scoring.

Use --intent WHY (reasons), WHEN (timing), ENTITY (what/who), or GENERAL to select
the strategy in any language while keeping the original query. --verbose reports
meta.intent and meta.intent_source (auto or override). --basic bypasses intent.`,
Args: cobra.MinimumNArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
keyword := strings.Join(args, " ")
if err := requirePositiveLimit("--limit", recLimit); err != nil {
Expand Down
59 changes: 59 additions & 0 deletions docs/USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,65 @@ whitespace, caps each excerpt, emits unindented JSON, and includes one
machine-readable interchange format; the opt-in projection avoids changing
existing parsers or adopting a draft serialization format.

#### Recall intent detection

Automatic intent selection uses a fixed set of lexical cues. It runs locally,
without an LLM or provider. Intent changes graph traversal and ranking; it does
not translate the query or the stored memories. The recognized question forms
include the following (examples use `PostgreSQL` as the subject):

| Language / script | WHY | WHEN | ENTITY |
|---|---|---|---|
| English | Why PostgreSQL? | When did we choose PostgreSQL? | What is PostgreSQL? |
| Mandarin Chinese, simplified | 为什么选择 PostgreSQL? | 什么时候选择 PostgreSQL? | PostgreSQL 是什么? |
| Mandarin Chinese, traditional | 為什麼選擇 PostgreSQL? | 什麼時候選擇 PostgreSQL? | PostgreSQL 是什麼? |
| Hindi, Devanagari | PostgreSQL क्यों? | PostgreSQL कब? | PostgreSQL क्या है? |
| Spanish | ¿Por qué PostgreSQL? | ¿Cuándo elegimos PostgreSQL? | ¿Qué es PostgreSQL? |
| Modern Standard Arabic | لماذا PostgreSQL؟ | متى اخترنا PostgreSQL؟ | ما هو PostgreSQL؟ |
| French | Pourquoi PostgreSQL ? | Quand avons-nous choisi PostgreSQL ? | Qu'est-ce que PostgreSQL ? |
| Bengali, Bengali script | PostgreSQL কেন? | PostgreSQL কখন? | PostgreSQL কী? |
| Portuguese | Por que PostgreSQL? | Quando escolhemos PostgreSQL? | O que é PostgreSQL? |
| Indonesian, Latin script | Mengapa PostgreSQL? | Kapan memilih PostgreSQL? | Apa itu PostgreSQL? |
| Russian, Cyrillic | Почему PostgreSQL? | Когда выбрали PostgreSQL? | Что такое PostgreSQL? |
| German | Warum PostgreSQL? | Wann wurde PostgreSQL gewählt? | Was ist PostgreSQL? |

Matching is case-insensitive and uses Unicode word boundaries for spaced scripts;
Chinese cues also match without spaces. The additional-language forms accept
Unicode whitespace, straight/curly French apostrophes, and composed/decomposed
accents in the listed Spanish/Portuguese cues. Accents are not generally removed.
Arabic accepts ordinary Arabic letters with or without common vowel marks
(harakat and superscript alif) and tatweel. A few explicit variants are included,
such as `為甚麼`, `क्यूँ`, `por quê`, `kenapa`, `зачем`, and `wieso`. Bengali
`কী`/`কি`/`কে` must end the question for ENTITY; bare Hindi `क्या` does not imply
ENTITY. Other dialects, spellings, Arabic presentation forms, and Latin
transliterations of non-Latin scripts are not covered systematically.

Unrecognized queries use `GENERAL`. Additional-language cues that disagree with
one another or with an English/Chinese cue also use `GENERAL`, regardless of
keyword counts. Same-intent mixed-language cues can agree. For compatibility,
English/Chinese-only queries retain their keyword scoring and ENTITY tie-break;
for example, `what is the reason` selects ENTITY. Paired quoted/code spans are
ignored for additional-language cues, while legacy quoted keywords retain their
old behavior. This is a lexical heuristic: it does not resolve negation,
incidental word mentions, nested quotations, or the meaning of mixed questions.
These examples test intent selection, not retrieval accuracy across languages.

The supervising agent can choose an intent from the user's meaning and retain
the original-language query and memories:

```bash
mnemon recall '¿Por qué elegimos PostgreSQL?' --intent WHY --verbose
mnemon recall 'हमने PostgreSQL कब चुना?' --intent WHEN --verbose
mnemon recall 'Was ist PostgreSQL?' --intent ENTITY --verbose
mnemon recall 'PostgreSQL index tuning' --intent GENERAL --verbose
```

The override is language-independent: WHY selects reasons, WHEN timing, ENTITY
what/who, and GENERAL neutral traversal. It takes precedence over detection.
Verbose output reports `meta.intent` and `meta.intent_source` (`auto` or
`override`), including when there are no results. `--basic` bypasses intent
selection entirely.

### Graph Operations

```bash
Expand Down
16 changes: 14 additions & 2 deletions docs/design/05-pipelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,8 @@ After receiving this output, the LLM can evaluate candidates and establish edges

### Step 1: Intent Detection

Query intent is automatically identified via regex matching:
Query intent is selected with a fixed set of local lexical patterns. The original
English/Chinese cues include:

| Intent | Trigger Patterns |
|--------|-----------------|
Expand All @@ -103,7 +104,18 @@ Query intent is automatically identified via regex matching:
| ENTITY | `what is`, `who is`, `tell me about`, `是什么`, `谁是`, `关于` |
| GENERAL | None of the above match |

Supports the `--intent` flag to manually override automatic detection.
Question forms also cover Hindi, Spanish, Modern Standard Arabic, French,
Bengali, Portuguese, Indonesian, Russian, and German. See
[recall intent detection](../USAGE.md#recall-intent-detection) for supported
scripts, examples, and limits. Matching uses Unicode word boundaries for spaced
scripts. Conflicting cues involving an additional language fall back to GENERAL;
legacy English/Chinese-only queries retain keyword scoring and the ENTITY
tie-break. This is a bounded heuristic, not semantic language understanding.

`--intent WHY|WHEN|ENTITY|GENERAL` overrides detection in any query language.
The host can supply intent from the user's meaning without translating the query
or invoking another provider. `--verbose` exposes `meta.intent` and
`meta.intent_source` (`auto` or `override`).

### Step 2: Multi-Signal Anchor Selection (RRF Fusion)

Expand Down
32 changes: 32 additions & 0 deletions docs/zh/USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,38 @@ mnemon forget <id>
JSON 继续作为机器可读交换格式,因此既不破坏现有解析器,也无需绑定尚在演进的
序列化草案。

#### Recall 意图检测

自动检测使用本地固定词语模式,无需 LLM 或服务提供商。覆盖英语、普通话
(简体/繁体)、印地语(天城文)、西班牙语、现代标准阿拉伯语、法语、孟加拉语
(孟加拉文)、葡萄牙语、印度尼西亚语(拉丁字母)、俄语(西里尔字母)和德语
的部分疑问句形式;这不代表能理解这些语言的所有表达,也不是跨语言检索准确率承诺。
完整例句及书写变体见[英文说明](../USAGE.md#recall-intent-detection)。

词边界识别 Unicode 字母、组合标记和数字;中文不要求空格。新增语言的模式兼容
Unicode 空白、法语直/弯撇号、西班牙语和葡萄牙语已列词语的组合/分解重音,
以及阿拉伯语常用元音标记和 tatweel。其他方言、阿拉伯字母表现形式和非拉丁
文字的拉丁转写不作系统支持。重音不会被普遍删除。

没有命中时返回 `GENERAL`。新增语言的意图线索互相冲突,或与英/中文线索冲突,
也返回 `GENERAL`;混合语言中一致的线索可正常识别。为保持兼容,仅含英/中文
线索的查询沿用原有计数及 ENTITY 平分规则,例如 `what is the reason` 选择
ENTITY。新增语言忽略成对引号/代码引用中的词语;英/中文引用词沿用原行为。
该规则不理解否定、偶然提及、嵌套引号或复合问题的含义。

宿主 agent 可根据用户含义显式选择意图,同时保留查询和记忆的原语言:

```bash
mnemon recall '¿Por qué elegimos PostgreSQL?' --intent WHY --verbose
mnemon recall 'हमने PostgreSQL कब चुना?' --intent WHEN --verbose
mnemon recall 'Was ist PostgreSQL?' --intent ENTITY --verbose
```

`--intent WHY|WHEN|ENTITY|GENERAL` 与语言无关,优先于自动检测:WHY 为原因、
WHEN 为时间、ENTITY 为是什么/是谁、GENERAL 为中性遍历。意图会影响图遍历
和排序。`--verbose` 输出 `meta.intent` 及 `meta.intent_source`
(`auto` 或 `override`),即使无结果也可查看;`--basic` 完全跳过意图检测。

**Import 标志:**

| 标志 | 默认值 | 说明 |
Expand Down
10 changes: 9 additions & 1 deletion docs/zh/design/05-pipelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,15 @@ LLM 收到这个输出后,可以评估候选并通过 `mnemon link` 命令建
| ENTITY | `what is`, `who is`, `tell me about`, `是什么`, `谁是`, `关于` |
| GENERAL | 以上都不匹配 |

支持 `--intent` 标志手动覆盖自动检测。
另有印地语、西班牙语、现代标准阿拉伯语、法语、孟加拉语、葡萄牙语、
印度尼西亚语、俄语和德语的部分疑问句模式,详见
[Recall 意图检测](../USAGE.md#recall-意图检测)。词边界使用 Unicode 规则;
新增语言的线索互相冲突或与英/中文冲突时回退到 GENERAL。仅含英/中文线索
时保留原有计数及 ENTITY 平分规则。这是有限的词语启发式,不是语义理解。

`--intent WHY|WHEN|ENTITY|GENERAL` 可对任意语言查询覆盖自动检测。宿主可从
用户含义选择意图,无需翻译查询或调用另一个服务。`--verbose` 显示
`meta.intent` 和 `meta.intent_source`(`auto` 或 `override`)。

### Step 2:多信号锚点选择(RRF 融合)

Expand Down
38 changes: 27 additions & 11 deletions internal/memory/search/intent.go
Original file line number Diff line number Diff line change
Expand Up @@ -49,16 +49,16 @@ var intentWeightsMap = map[Intent]IntentWeights{
}

var whyPatterns = regexp.MustCompile(
`(?i)\b(why|reason|because|cause|motivation|rationale)\b|` +
`(为什么|原因|理由)`)
`(?i)(why|reason|because|cause|motivation|rationale)|` +
`(为什么|為什麼|為甚麼|原因|理由)`)

var whenPatterns = regexp.MustCompile(
`(?i)\b(when|time|date|before|after|during|timeline|history|sequence)\b|` +
`(什么时候|何时|时间|之前|之后)`)
`(?i)(when|timeline|time|date|before|after|during|history|sequence)|` +
`(什么时候|什麼時候|甚麼時候|何时|何時|时间|時間|之前|之后|之後)`)

var entityPatterns = regexp.MustCompile(
`(?i)\b(what is|who is|tell me about|describe|about)\b|` +
`(是什么|谁是|关于|介绍)`)
`(?i)(what is|who is|tell me about|describe|about)|` +
`(是什么|是什麼|是甚麼|谁是|誰是|关于|關於|介绍|介紹)`)

// IntentFromString parses a user-provided intent string into an Intent value.
func IntentFromString(s string) (Intent, error) {
Expand All @@ -76,12 +76,28 @@ func IntentFromString(s string) (Intent, error) {
}
}

// DetectIntent analyzes a query string and returns the detected intent.
// DetectIntent selects an intent using bounded, language-specific lexical cues.
// It preserves English/Chinese scoring; conflicting cues involving the additional
// languages fall back to GENERAL. This is not semantic language understanding.
func DetectIntent(query string) Intent {
q := strings.ToLower(query)
whyScore := len(whyPatterns.FindAllString(q, -1))
whenScore := len(whenPatterns.FindAllString(q, -1))
entityScore := len(entityPatterns.FindAllString(q, -1))
whyScore := legacyIntentScore(whyPatterns, query)
whenScore := legacyIntentScore(whenPatterns, query)
entityScore := legacyIntentScore(entityPatterns, query)

questionIntent, conflict := multilingualQuestionIntent(query)
if conflict {
return IntentGeneral
}
if questionIntent != IntentGeneral {
// A strong cue in another language must not override conflicting legacy
// cues, even when one legacy intent has a higher keyword count.
if (whyScore > 0 && questionIntent != IntentWhy) ||
(whenScore > 0 && questionIntent != IntentWhen) ||
(entityScore > 0 && questionIntent != IntentEntity) {
return IntentGeneral
}
return questionIntent
}

if whyScore > whenScore && whyScore > entityScore && whyScore > 0 {
return IntentWhy
Expand Down
124 changes: 124 additions & 0 deletions internal/memory/search/intent_languages.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
package search

import (
"regexp"
"strings"
"unicode"
"unicode/utf8"
)

// Keep this list small and explicit: question forms, not translated bags of
// generic words such as "time" or "about". No language identification is needed;
// all matching forms contribute, and contradictory intents fail to GENERAL.
var multilingualQuestionPatterns = []struct {
intent Intent
pattern *regexp.Regexp
}{
// Hindi (Devanagari). Bare क्या also starts yes/no questions, so require a copula.
{IntentWhy, intentWords(`क्यों|क्यूँ|किसलिए`)},
{IntentWhen, intentWords(`कब`)},
{IntentEntity, intentWords(`(?:क्या|कौन) (?:है|हैं)`)},
// Spanish. Accept both precomposed and decomposed accents, without removing them.
{IntentWhy, intentWords(`por qu(?:é|e\x{0301})`)},
{IntentWhen, intentWords(`cu(?:á|a\x{0301})ndo`)},
{IntentEntity, intentWords(`qu(?:é|e\x{0301}) es|qui(?:é|e\x{0301})n es`)},
// Modern Standard Arabic. Common vowel marks and tatweel are removed below.
{IntentWhy, intentWords(`لماذا`)},
{IntentWhen, intentWords(`متى`)},
{IntentEntity, intentWords(`(?:ما|من) (?:هو|هي)`)},
// French. Apostrophes within words are not quotation delimiters.
{IntentWhy, intentWords(`pourquoi`)},
{IntentWhen, intentWords(`quand`)},
{IntentEntity, intentWords(`qu['’]est(?:-| )ce que|qui est|c['’]est quoi`)},
// Bengali. A final কী/কি/কে asks for an entity; mid-sentence কি can be yes/no.
{IntentWhy, intentWords(`কেন`)},
{IntentWhen, intentWords(`কখন|কবে`)},
{IntentEntity, intentWords(`(?:কী|কি|কে)[\s\p{Z}]*[??]?$`)},
// Portuguese (European and Brazilian shared forms).
{IntentWhy, intentWords(`por qu(?:e|ê|e\x{0302})`)},
{IntentWhen, intentWords(`quando`)},
{IntentEntity, intentWords(`(?:o que|quem) (?:é|e\x{0301})`)},
// Indonesian (Latin script).
{IntentWhy, intentWords(`mengapa|kenapa`)},
{IntentWhen, intentWords(`kapan`)},
{IntentEntity, intentWords(`(?:apa|siapa) itu`)},
// Russian (Cyrillic).
{IntentWhy, intentWords(`почему|зачем`)},
{IntentWhen, intentWords(`когда`)},
{IntentEntity, intentWords(`что такое|кто (?:такой|такая|такие)`)},
// German (Latin script).
{IntentWhy, intentWords(`warum|wieso|weshalb`)},
{IntentWhen, intentWords(`wann`)},
{IntentEntity, intentWords(`(?:was|wer) (?:ist|sind)`)},
}

// Go's \b only understands ASCII. Marks and joiners must stay attached to words
// in Indic/Arabic text; Unicode letters and numbers also prevent cross-script
// substring matches such as "почемуx", "ékapan", or "why中文".
const intentWordChars = `\p{L}\p{M}\p{N}_\x{200c}\x{200d}`

func intentWords(pattern string) *regexp.Regexp {
pattern = strings.ReplaceAll(pattern, " ", `[\s\p{Z}]+`)
return regexp.MustCompile(`(?i)(^|[^` + intentWordChars + `])(?:` + pattern + `)($|[^` + intentWordChars + `])`)
}

// Ignore paired quoted/code spans in the new cues. A single quote preceded by a
// letter is an apostrophe (qu'est-ce), not an opening quote. Legacy-only queries
// retain their historical handling of quotes and keyword counts.
var quotedIntentText = regexp.MustCompile(`(?s)"[^"]*"|“[^”]*”|«[^»]*»|` + "`[^`]*`" + `|(^|[^` + intentWordChars + `])'[^']*'`)

func multilingualQuestionIntent(query string) (Intent, bool) {
query = strings.TrimSpace(quotedIntentText.ReplaceAllString(query, " "))
query = strings.Map(func(r rune) rune {
if r == '\u0640' || (r >= '\u064b' && r <= '\u0652') || r == '\u0670' {
return -1 // Arabic tatweel, harakat, and superscript alif.
}
return r
}, query)
intent := IntentGeneral
for _, cue := range multilingualQuestionPatterns {
if !cue.pattern.MatchString(query) {
continue
}
if intent != IntentGeneral && intent != cue.intent {
return IntentGeneral, true
}
intent = cue.intent
}
return intent, false
}

// Preserve legacy scoring, while checking Unicode boundaries for the English
// alternatives. Chinese cues intentionally match without spaces. Checking the
// surrounding runes does not consume separators between repeated keywords.
func legacyIntentScore(pattern *regexp.Regexp, query string) int {
score := 0
for offset := 0; offset < len(query); {
span := pattern.FindStringIndex(query[offset:])
if span == nil {
break
}
start, end := offset+span[0], offset+span[1]
first, size := utf8.DecodeRuneInString(query[start:end])
offset = end
if unicode.Is(unicode.Han, first) {
score++
continue
}
before, _ := utf8.DecodeLastRuneInString(query[:start])
after, _ := utf8.DecodeRuneInString(query[end:])
if !intentWordRune(before) && !intentWordRune(after) {
score++
continue
}
// An invalid compound can contain a later valid cue: in "retell me
// about", rejecting "tell me about" must not consume the word "about".
offset = start + size
}
return score
}

func intentWordRune(r rune) bool {
return unicode.IsLetter(r) || unicode.IsMark(r) || unicode.IsNumber(r) ||
r == '_' || r == '\u200c' || r == '\u200d'
}
Loading
Loading