Bug Description
MarkdownChunker (src/memos/chunkers/markdown_chunker.py) counts #-comment lines inside fenced code blocks as level-1 markdown headers.
_detect_malformed_headers scans every line matching ^#{1,6}\s+.+ with no awareness of ``` fences, and _fix_header_hierarchy rewrites the same lines. Ingesting a markdown document with an embedded code sample that has five or more `#` comments (extremely common in coding knowledge bases) triggers the "malformed hierarchy" repair, which then mutates the comments into `## ...` lines inside the code block — corrupting the embedded code in the produced memory chunks.
To Reproduce
from memos.chunkers.markdown_chunker import MarkdownChunker
chunker = MarkdownChunker(auto_fix_headers=True)
text = (
"# Title\n\nIntro.\n\n"
"```python\n"
"# comment one\n# comment two\n# comment three\n# comment four\n# comment five\n"
"x = 1\n"
"```\n"
)
fixed = chunker._fix_header_hierarchy(text)
print(fixed)
# ```python
# ## comment one <- code corrupted
# ## comment two
# ...
_detect_malformed_headers(text) also returns True for a document containing only a code block, and for ~~~-fenced blocks likewise.
Expected Behavior
Lines inside fenced code blocks (CommonMark ``` / ~~~ fences, including unclosed fences) must never be treated as headers: the detector should not count them, and the fixer must pass them through byte-for-byte.
Environment
MemOS main (a7367d0), python 3.14.
Additional Context
I have a fix ready (fence-aware line mask shared by detector and fixer) with red/green-verified unit tests in tests/chunkers/; will open a PR referencing this issue.
Bug Description
MarkdownChunker(src/memos/chunkers/markdown_chunker.py) counts#-comment lines inside fenced code blocks as level-1 markdown headers._detect_malformed_headersscans every line matching^#{1,6}\s+.+with no awareness of ``` fences, and_fix_header_hierarchyrewrites the same lines. Ingesting a markdown document with an embedded code sample that has five or more `#` comments (extremely common in coding knowledge bases) triggers the "malformed hierarchy" repair, which then mutates the comments into `## ...` lines inside the code block — corrupting the embedded code in the produced memory chunks.To Reproduce
_detect_malformed_headers(text)also returnsTruefor a document containing only a code block, and for~~~-fenced blocks likewise.Expected Behavior
Lines inside fenced code blocks (CommonMark ``` / ~~~ fences, including unclosed fences) must never be treated as headers: the detector should not count them, and the fixer must pass them through byte-for-byte.
Environment
MemOS main (
a7367d0), python 3.14.Additional Context
I have a fix ready (fence-aware line mask shared by detector and fixer) with red/green-verified unit tests in
tests/chunkers/; will open a PR referencing this issue.