fix: handle quoted charset and non-canonical content-type in title extraction

[krishna28238-arch]

Fixes #2664

httpx silently drops the title, or extracts it as mojibake, when the Content-Type response header deviates from the canonical lowercase form:

• Content-Type: text/html; charset="gbk" — the quoted charset value (legal per RFC 9110) is not matched by the charset=gbk substring check in DecodeData, so GBK/EUC-KR bodies are never transcoded and the title comes out as raw bytes
• Content-Type: TEXT/HTML or text/html ; charset=utf-8 — media types are case-insensitive and optional whitespace is allowed before parameters, but CanHaveTitleTag compares the value verbatim, so the title extraction is skipped entirely

$ httpx -u http://127.0.0.1:8000 -title
# content-type: text/html; charset="gbk"        -> []  (raw GBK bytes)
# content-type: TEXT/HTML                       -> (no title at all)

This strips quotes from the joined content-type before the charset matching, and normalizes the mime type (case + surrounding whitespace) in CanHaveTitleTag. Responses with canonical headers are unaffected. Regression tests cover both the quoted-charset and case/whitespace variants end-to-end (TestCanHaveTitleTag, TestExtractTitleDecodesCharset).

Summary by CodeRabbit

• Bug Fixes

  • Improved handling of quoted character-set values in HTTP content types, helping titles encoded with GBK and similar formats display correctly.
  • MIME type detection now handles capitalization differences and surrounding whitespace, improving recognition of HTML content.

• Tests

  • Added coverage for MIME type normalization and decoding titles from quoted and unquoted character-set declarations.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 21ba325a-17a5-40e8-9894-96a7f464f96b

---

📥 Commits

Reviewing files that changed from the base of the PR and between be8322d and 7300e4f.

---

📒 Files selected for processing (1)

• common/httpx/title.go

---

🚧 Files skipped from review as they are similar to previous changes (1)

• common/httpx/title.go

---

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

---

---

Walkthrough

The change normalizes MIME types before title detection and removes quotes from content-type values before charset decoding. Tests cover MIME casing, whitespace, and GBK title decoding with quoted and unquoted charset parameters.

Changes

HTTP title extraction

Layer / File(s)	Summary
Normalize title MIME types common/httpx/title.go, common/httpx/title_test.go	CanHaveTitleTag trims and lowercases MIME types. Tests cover supported and unsupported values.
Decode quoted charset values common/httpx/encodings.go, common/httpx/title_test.go	DecodeData handles quoted charset parameters. Integration tests verify GBK title decoding.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 7300e

The title-extraction changes are mergeable after normal checks; no actionable risk was established.

Pre-merge checks | 4 | 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 3 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly summarizes the main changes: support for quoted charset values and non-canonical Content-Type formatting during title extraction.
Linked Issues check		Issue #2664 requires title extraction for quoted charset values and non-canonical media-type casing or whitespace. DecodeData removes double quotes before charset matching, so charset="gbk" matche…
Out of Scope Changes check		The changes stay within issue #2664. The implementation changes target charset parsing and MIME-type recognition. The regression tests verify those behaviors. The additional changes in trimTitleTags…

---

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

---

_{A rabbit checks the title line
And trims the spaces every time
GBK words bloom clear and bright
Quoted charsets now decode right
The little tests confirm the light}

_{Comment @coderabbitai help to get the list of available commands.}

[ehsandeep]

@krishna28238-arch thanks for the PR, do you have test target to see before / after behavior?

[krishna28238-arch]

Yes — I've included end-to-end regression tests that
demonstrate the before/after behavior:

Test 1: TestCanHaveTitleTag (title_test.go)
Shows that non-canonical MIME types like "TEXT/HTML" and
" text/html ; charset=utf-8" are now correctly recognized
as HTML (previously skipped entirely, resulting in no
title extraction).

Test 2: TestExtractTitleDecodesCharset (title_test.go)
Shows that a GBK-encoded body with a quoted charset
declaration Content-Type: text/html; charset="gbk" now
transcodes correctly and extracts the title. Previously
the quoted value failed the charset= substring match, so
the body was never transcoded and the title came out as
raw bytes.

You can verify the before/after behavior by:

1. Check out the main branch and run:
   go test ./common/httpx/ -run TestCanHaveTitleTag -v
   go test ./common/httpx/ -run TestExtractTitleDecodesCharset -v

   (Tests will fail or show incorrect behavior)

2. Check out this PR branch and run the same commands:
   (Tests will pass, showing correct behavior)

Or run the full test suite to confirm no regressions:
go test ./common/httpx/ -v

[ehsandeep]

@krishna28238-arch I'm looking for real hosts to better understand the existing tool behavior, then test code to better understand the change. Can you share some?

[krishna28238-arch]

Thanks @ehsandeep, sure - here's what I found. The short version: the mainstream CJK web has moved to canonical UTF-8 headers, the legacy encodings are still out there, and the two shapes this PR fixes are straightforward to reproduce locally even though I couldn't find a live public host serving them today.

Real hosts I checked

I ran curl -s -D - over ~55 CN/KR/JP hosts (portals, provincial gov sites, university BBS front-ends, older reference sites). Every mainstream one is canonical now:

 $ curl -s -D - -o /dev/null https://www.baidu.com | grep -i ^content-type
Content-Type: text/html; charset=utf-8

 $ curl -s -D - -o /dev/null http://www.gov.cn | grep -i ^content-type
Content-Type: text/html

(baidu.com, qq.com, 163.com, sohu.com, sina.com.cn, gov.cn, people.com.cn, cctv.com, china.com.cn, naver.com, daum.net, yahoo.co.jp, etc)

One host that is still GB2312 today, in case you want to watch the existing behavior end to end:

 $ curl -s -D - -o /dev/null http://www.cnr.cn | grep -i ^content-type
Content-Type: text/html                        <- no charset in the header

 $ curl -s http://www.cnr.cn | grep -aoiE '<meta[^>]*charset[^>]*>' | head -1
<meta http-equiv="Content-Type" content="text/html; charset=gb2312">

 $ curl -s http://www.cnr.cn | file -bi -
text/html; charset=iso-8859-1                  <- body is GBK bytes

dev gets this one right - the <meta> fallback in DecodeData catches charset=gb2312":

 $ git checkout dev && go build -o /tmp/httpx-dev ./cmd/httpx
 $ echo http://www.cnr.cn | /tmp/httpx-dev -title -silent -no-color
https://www.cnr.cn [央广网·中央广播电视总台]

So the meta-declared path works today. What doesn't work is when the same value comes back in the HTTP header in non-canonical form:

1. quoted charset, text/html; charset="gbk" - DecodeData substring-matches charset=gbk, misses the quoted form, so the GBK body never gets transcoded and the title comes out as raw GBK bytes
2. MIME portion not exactly lowercase/trimmed - TEXT/HTML, or text/html coming from text/html ; charset=... - CanHaveTitleTag does an exact slices.Contains, so title extraction is just skipped

This same class of host has come up in the tracker a few times already: #37 (tnw.qq.com, GBK, 2020), #441 (euc-kr titles, 2021), #652 (GBK page, 2022), #1494 (103.45.249.230:7878, 2023). All four are migrated or offline now, so for the code-test part I set up local servers that emit the exact shapes:

# save as weird_ct.py
from http.server import HTTPServer, BaseHTTPRequestHandler
import threading, signal

class Handler(BaseHTTPRequestHandler):
    body = b''
    ctype = ''
    def do_GET(self):
        self.send_response(200)
        self.send_header('Content-Type', self.ctype)
        self.send_header('Content-Length', str(len(self.body)))
        self.end_headers()
        self.wfile.write(self.body)
    def log_message(self, fmt, *args):
        pass

class QuotedGBK(Handler):
    ctype = 'text/html; charset="gbk"'
    body = '<html><head><title>这是中文测试标题</title></head><body>Content</body></html>'.encode('gbk')

class UppercaseMIME(Handler):
    ctype = 'TEXT/HTML'
    body = b'<html><head><title>Uppercase MIME Type Test</title></head><body>Content</body></html>'

class Whitespace(Handler):
    ctype = 'text/html ; charset=utf-8'   # note the space before the semicolon
    body = b'<html><head><title>Whitespace Test</title></head><body>Content</body></html>'

for port, h in [(8001, QuotedGBK), (8002, UppercaseMIME), (8003, Whitespace)]:
    threading.Thread(target=HTTPServer(('127.0.0.1', port), h).serve_forever, daemon=True).start()
signal.pause()

 $ curl -s -D - -o /dev/null http://127.0.0.1:8001/ | grep -i ^content-type
Content-Type: text/html; charset="gbk"
 $ curl -s -D - -o /dev/null http://127.0.0.1:8002/ | grep -i ^content-type
Content-Type: TEXT/HTML
 $ curl -s -D - -o /dev/null http://127.0.0.1:8003/ | grep -i ^content-type
Content-Type: text/html ; charset=utf-8

dev branch:

 $ echo http://127.0.0.1:8001/ | /tmp/httpx-dev -title -silent -no-color
http://127.0.0.1:8001/ [IJԱ]  <- mojibake, raw GBK passed through, never transcoded

 $ echo http://127.0.0.1:8002/ | /tmp/httpx-dev -title -silent -no-color
http://127.0.0.1:8002/       <- no title at all, TEXT/HTML not recognized

 $ echo http://127.0.0.1:8003/ | /tmp/httpx-dev -title -silent -no-color
http://127.0.0.1:8003/       <- no title, "text/html " (trailing space) not recognized

and to confirm the first one is really just undecoded GBK (the same bytes the server sent):

 $ echo http://127.0.0.1:8001/ | /tmp/httpx-dev -title -silent -no-color | iconv -f gbk -t utf-8
http://127.0.0.1:8001/ [这是中文测试标题]

with this PR:

 $ git checkout fix-title-content-type-edge-cases && go build -o /tmp/httpx-fixed ./cmd/httpx
 $ echo http://127.0.0.1:8001/ | /tmp/httpx-fixed -title -silent -no-color
http://127.0.0.1:8001/ [这是中文测试标题]

 $ echo http://127.0.0.1:8002/ | /tmp/httpx-fixed -title -silent -no-color
http://127.0.0.1:8002/ [Uppercase MIME Type Test]

 $ echo http://127.0.0.1:8003/ | /tmp/httpx-fixed -title -silent -no-color
http://127.0.0.1:8003/ [Whitespace Test]

Regression check

I ran both builds over the 32 reachable hosts from the scan with -title -silent and the titles came out identical. The only differences were a redirect race on cctv.com/naver.com (each run caught a different side of the 302, in opposite directions) and one transient timeout on xhby.net. go test ./common/httpx/ passes on the branch, including the new TestCanHaveTitleTag / TestExtractTitleDecodesCharset.