fix(chunkers): keep fenced code blocks out of markdown header repair

[simpleqt]

Description

Fixes #2415

MarkdownChunker counted #-comment lines inside fenced code blocks as level-1 markdown headers: _detect_malformed_headers matched every ^#{1,6}\s+.+ line regardless of ``` fences, and once the "malformed hierarchy" heuristic tripped (five commented lines in an embedded python snippet are enough), _fix_header_hierarchy rewrote those comments into `## ...` lines inside the code block, corrupting embedded code in the produced memory chunks.

Implementation: a shared _code_block_mask helper tracks CommonMark-style fences (backtick and tilde, including unclosed fences; the fence lines themselves are masked) over the split lines, and both the detector and the fixer skip masked lines. Lines outside fences — real headers and plain text — are handled exactly as before.

Dependencies: none.

Type of change

• Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

• Unit Test

New tests/chunkers/test_markdown_chunker_code_fence.py:

• a code-block-only document is no longer detected as malformed (fails on main: detector returned True)
• _fix_header_hierarchy leaves a document containing a fenced python block byte-for-byte intact (fails on main: comments were rewritten to ## ...)
• an unclosed fence holds to the end of the text (fails on main)
• two behavior-preservation tests pin the existing repair of genuinely flat header sequences (# A / # B / # C → # A / ## B / ## C) and tilde-fence handling

All five pass with the fix; the three discriminators fail against unpatched main (red/green verified).

• Test Script Or Test Steps

poetry run pytest tests/chunkers/test_markdown_chunker_code_fence.py -v

ruff check: the only finding on the touched source file (BLE001 at chunk()'s pre-existing except Exception) is present on main as well; ruff format --check passes.

Checklist

• I have performed a self-review of my own code | 我已自行检查了自己的代码
• I have commented my code in hard-to-understand areas | 我已在难以理解的地方对代码进行了注释
• I have added tests that prove my fix is effective or that my feature works | 我已添加测试以证明我的修复有效或功能正常
• I have linked the issue to this PR (if applicable) | 我已将 issue 链接到此 PR（如果适用）

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Memtensor-AI]

🤖 Open Code Review

Target: PR #2416
Task: f3c8aa28f7b231ee
Base: main
Head: sq/chunker-code-fence

🔍 OpenCodeReview found 2 issue(s) in this PR.

---

1. src/memos/chunkers/markdown_chunker.py (L89-L98)

The closing-fence detection only checks the fence character but not the fence length, violating the CommonMark spec.

Per CommonMark spec §4.5: a closing fence must (a) use the same character as the opening fence, and (b) consist of at least as many characters as the opening fence.

With the current code, a document like:

`````python
# this is code, not a header

still inside the code block

```
…would have the inner ` ``` ` line incorrectly close the block, causing the remaining lines to be treated as normal Markdown. Any `#` comment after the premature close would be misclassified as a Markdown header.

Suggested fix: track the opening fence length and require the closing fence to be at least that long.

<details><summary>💡 Suggested Change</summary>

**Before:**
```
        fence_char = None
        for line in lines:
            fence_match = cls._FENCE_RE.match(line)
            if fence_match:
                char = fence_match.group(1)[0]
                if fence_char is None:
                    fence_char = char
                elif char == fence_char:
                    fence_char = None
                mask.append(True)
```

**After:**
```suggestion
        fence_char = None
        fence_len = 0
        for line in lines:
            fence_match = cls._FENCE_RE.match(line)
            if fence_match:
                fence_str = fence_match.group(1)
                char = fence_str[0]
                cur_len = len(fence_str)
                if fence_char is None:
                    fence_char = char
                    fence_len = cur_len
                elif char == fence_char and cur_len >= fence_len:
                    fence_char = None
                    fence_len = 0
                mask.append(True)
```

</details>

---

### 2. `tests/chunkers/test_markdown_chunker_code_fence.py` (L52-L54)

The `assertIn` checks only verify that the expected substrings are present somewhere in the output, but they don't verify the output contains exactly one `#`-level header. If `_fix_header_hierarchy` emitted duplicate or spurious header lines (e.g., two `## B` lines), these assertions would still pass and the regression would go undetected.

Consider adding a count assertion alongside the substring checks:
```python
self.assertEqual(fixed.count('\n# ') + fixed.startswith('# '), 1)  # exactly one H1
```
Or compare against the exact expected string:
```python
self.assertEqual(fixed, "# A\n## B\n## C\nbody\n")
```
The same gap applies to `test_tilde_fence_ignored_and_real_headers_fixed`.

_Generated by cloud-assistant via Open Code Review._

[Memtensor-AI]

⚠️ Automated Test Results: INCONCLUSIVE

Automated tests inconclusive (auto-generated test defect); treated as non-blocking. Manual review recommended. Details: All 5 new tests fail during setup because patch("langchain_text_splitters.MarkdownHeaderTextSplitter") cannot resolve the attribute on the target module.

Branch: sq/chunker-code-fence