fix(wechat): improve image extraction (5→12) and HTML entity cleanup - #21
Conversation
- Fix incomplete image extraction: add three-method SSR image fallback (window.picture_page_info_list regex + cgiDataNew JSON + global cdn_url scan) Fixes image articles (type 8) only recovering 5/12 images - Add html.unescape() to _process_text_block to clean residual HTML entities in WeChat topic links (<a class=wx_topic_link...> → #GEO) - Strip HTML tags from text content keeping only inner hashtag text - Add _extract_ssr_images helper with three-tier extraction strategy - Improve parse_ssr_content: add meta description fallback for text when SSR data lacks desc/content_noencode fields - Merge SSR images into DOM-parsed content when js_content exists Closes image extraction gap: 5→12 images for type-8 articles
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c651a3f6da
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for m in re.finditer(r"""cdn_url\s*:\s*['"]([^'"]+)['"]""", html_content): | ||
| url = m.group(1) | ||
| if "mmbiz.qpic.cn" in url and ("wx_fmt=jpeg" in url or "wx_fmt=png" in url): | ||
| if url not in cdn_urls_seen: | ||
| images.append(ContentItem(type=ContentType.IMAGE, content=url)) |
There was a problem hiding this comment.
When window.picture_page_info_list contains WeChat's escaped \x26amp; separators, _parse_ssr_image_list first normalizes those URLs, but this fallback scan compares and appends the raw escaped forms. Each image is therefore emitted twice—once correctly and once as an unusable escaped URL—and the later exact-string deduplication cannot remove either copy. Normalize the scanned URL before checking cdn_urls_seen or existing_urls.
Useful? React with 👍 / 👎.
| for ssr_img in ssr_images: | ||
| if ssr_img.content not in existing_image_urls: | ||
| contents.insert(0, ssr_img) |
There was a problem hiding this comment.
When a page has js_content and SSR contributes multiple images absent from the DOM, inserting every image at index zero reverses their source order: an SSR sequence [1, 2, 3] becomes [3, 2, 1, ...DOM]. Since both the Markdown body and NewsItem.images preserve this list order, affected image articles are exported with their images backwards.
Useful? React with 👍 / 👎.
| # Unescape HTML entities (<→<, >→>, "→", &→&) | ||
| text = html.unescape(text) | ||
| # Strip residual HTML tags (e.g. topic links) but keep inner text | ||
| text = re.sub(r'<[^>]+>', '', text) |
There was a problem hiding this comment.
For article text containing legitimate angle-bracket content, such as 1 < 2 and 3 > 1, entity decoding produces 1 < 2 and 3 > 1 and this regex then deletes < 2 and 3 >, corrupting the paragraph to 1 1. The same occurs for code-like text and is repeated in the SSR path; strip only actual embedded markup rather than every substring delimited by < and >.
Useful? React with 👍 / 👎.
1 participant
Summary
Fix two bugs in the WeChat article crawler (
wechat.py):Problem 1: Missing images in type-8 (image) articles
Before: Only 5 of 12 images were extracted from image-type WeChat articles.
Root cause:
curl_cffifetcher receives the initial HTML without JS-renderedjs_content. The fallbackparse_ssr_contentonly checkedcgiDataNewforpicture_page_info_list(which contains account metadata, not article images), missing images stored inwindow.picture_page_info_listand other inline script blocks.Fix: Three-tier image extraction strategy:
window.picture_page_info_listregex (primary source for type-8 image articles)cgiDataNew/__QMTPL_SSR_DATA__JSON parsing (existing method)cdn_url: "..."regex scan across all<script>blocks (fallback)After: 12/12 images recovered.
Problem 2: HTML entities in WeChat topic links
Before: Topic tags rendered as raw HTML entities:
<a class="wx_topic_link" ...>#GEO</a>Fix: Added
html.unescape()+ HTML tag stripping in_process_text_block().After: Clean text:
#GEO #谷歌推广 #内容营销 #AI搜索 #SEO转型Changes
import html_process_text_block: addhtml.unescape()and HTML tag stripping_extract_ssr_images(): three-method image extractionparse_html_to_news_content: merge SSR images when DOM parsing existsparse_ssr_content: 3 image sources + 3 text sources + JS/HTML entity cleanupTested
Tested against real WeChat image article:
https://mp.weixin.qq.com/s/m1qTyML8BijR_RcJGnM0-wFile changed
.claude/skills/news-extractor/scripts/crawlers/wechat.py(+126 / -27 lines)🤖 Generated with Claude Code
Co-Authored-By: Claude noreply@anthropic.com