Skip to content

Indented fenced code blocks are not recognised, so their contents are re-parsed as markdown #110

Description

@benswift

Summary

render_markdown_text() renders message text with Python-Markdown's
fenced_code extension, which only recognises a fence starting at column 0. A
fence indented inside a list item is therefore never treated as a code block,
and what happens instead depends on the indent:

  • 1–3 spaces — the fence lines are left as literal text and the block's
    contents are re-parsed as markdown. Since Python-Markdown passes raw HTML
    through, HTML inside the code block becomes live markup (this is the input
    route in Raw HTML in message text is rendered as live markup, silently truncating the transcript #105).
  • 4+ spaces — the block becomes an indented code block, so the contents
    are escaped correctly, but the ```html and closing ``` markers are
    captured as part of the code, and the language is lost.

Neither indent renders as the fenced block the author wrote. CommonMark allows
up to three spaces of fence indentation at the top level, and correspondingly
more inside a list item, so these are well-formed fences rather than malformed
input.

Reproduction

Python-Markdown 3.10.3:

import markdown

BODY = '<div class="x">hi</div>'

def doc(indent):
    p = " " * indent
    return f"- an item:\n{p}```html\n{p}{BODY}\n{p}```\n"

for i in (0, 2, 5):
    print(i, markdown.markdown(doc(i), extensions=["fenced_code", "tables"]))
indent result
0 <pre><code class="language-html">, contents escaped ✅
1–3 fence unrecognised, <div class="x">hi</div> emitted live ❌
4+ indented code block; escaped, but ```html leaks in as content and the language is lost ❌

End to end through the tool at 0.6, an assistant message whose
message.content[0].text is

Here is what changed:

- **Header.html** — rewritten:
  ```html
  <div class="banner" id="banner">hello</div>
  ```

Done.

produces a live, unescaped <div class="banner" id="banner"> in
page-001.html.

Where this shows up in real transcripts

Ordinary assistant prose puts fences at column 0, so most sessions never hit it.
The case that does is compaction summaries: the summary is a deeply nested
bullet list with indented code samples throughout, so any session long enough to
have compacted at least once tends to carry a batch of these at once. In our
case a summary held ten of them, and one contained an Astro template whose
attribute syntax (href={...}) went live and broke a link checker on the
published page — which is how we noticed at all.

Suggested fix

pymdownx.superfences (from pymdown-extensions) recognises indented fences at
every depth and escapes their contents:

markdown.markdown(doc(5), extensions=["pymdownx.superfences", "tables"])
# <li>an item:
#      <div class="highlight"><pre><span></span><code>...&lt;div ...

Verified at indents 0, 2, 4 and 5 — all escaped, none live. The trade-off is
that it emits Pygments-style <div class="highlight"><pre><code> rather than
<pre><code class="language-html">, so it would need a CSS change and a new
dependency.

A lighter option that keeps fenced_code is to normalise fence indentation
before converting — dedent each fenced block to column 0, leaving the body's
relative indentation intact. That's what we do as a local pre-processing
workaround, and it's enough for our case, though it's a text transform on the
message rather than a fix in the renderer.

Worth noting that the fix in #105 is orthogonal and also worth having:
deregistering the HTML handlers stops the 1–3 space case emitting live markup,
which is the safety half. The block still wouldn't render as a code block, which
is what this issue is about.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions