Skip to content

MarkdownChunker treats '#' comments inside code blocks as headers and rewrites them #2415

Description

@simpleqt

Bug Description

MarkdownChunker (src/memos/chunkers/markdown_chunker.py) counts #-comment lines inside fenced code blocks as level-1 markdown headers.

_detect_malformed_headers scans every line matching ^#{1,6}\s+.+ with no awareness of ``` fences, and _fix_header_hierarchy rewrites the same lines. Ingesting a markdown document with an embedded code sample that has five or more `#` comments (extremely common in coding knowledge bases) triggers the "malformed hierarchy" repair, which then mutates the comments into `## ...` lines inside the code block — corrupting the embedded code in the produced memory chunks.

To Reproduce

from memos.chunkers.markdown_chunker import MarkdownChunker

chunker = MarkdownChunker(auto_fix_headers=True)
text = (
    "# Title\n\nIntro.\n\n"
    "```python\n"
    "# comment one\n# comment two\n# comment three\n# comment four\n# comment five\n"
    "x = 1\n"
    "```\n"
)
fixed = chunker._fix_header_hierarchy(text)
print(fixed)
# ```python
# ## comment one      <- code corrupted
# ## comment two
# ...

_detect_malformed_headers(text) also returns True for a document containing only a code block, and for ~~~-fenced blocks likewise.

Expected Behavior

Lines inside fenced code blocks (CommonMark ``` / ~~~ fences, including unclosed fences) must never be treated as headers: the detector should not count them, and the fixer must pass them through byte-for-byte.

Environment

MemOS main (a7367d0), python 3.14.

Additional Context

I have a fix ready (fence-aware line mask shared by detector and fixer) with red/green-verified unit tests in tests/chunkers/; will open a PR referencing this issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area:coreMOS 编排层 / 框架底座 / 跨模块问题status:needs-triageNeeds initial triage | 需要初步判断 & 问题复现

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions