fix(dumpkit): escape mapping keys for the plist writer (#772) - #784
Conversation
- common.py: `object_hook` is never handed a mapping key -- dictdumper's
`_append_dict` interpolates it into `'<key>{item}</key>'` and calls
`_encode_value` on the value only -- so both branches that build a
mapping now escape their own keys through one `escape_key` helper: a
MultiDict, where #771 escaped an enum-derived key inline and nothing
else, and a plain dict, where nothing was escaped at all.
- A non-str key is rendered with `format`, i.e. the writer's own
interpolation, so the `<key>` text is unchanged apart from the
escaping. examples/captures/test.pcapng is why that case is handled at
all: its decryption secrets block keys the TLS key log entries by a raw
bytes client random whose repr carries `&`, `<` and `>`.
- json, tree and text are untouched -- `escape_key` is a no-op for them
and a plain dict is not even rebuilt, so the writer still gets the
caller's own mapping with the caller's own key objects in it.
- tests/dumpkit/test_plist_escaping_regression.py is new, fixture-tier
because test.pcapng is generated rather than committed. It pins that
the plist and xml reports parse, that json/tree/text keep the key
verbatim, and that nothing is escaped twice.
Fixes #772. The upstream half is JarryShaw/DictDumper#125, where
`_append_dict` should call `_encode_value` on a key; this does not wait
on it.
Measured on fe80b85 across 6 captures x 5 formats: `ET.parse` of
test.pcapng's plist failed at line 1517 of 1958 before and parses after;
exactly 2 of the 30 reports changed -- that capture's plist and xml,
which are one writer under two names -- and by exactly one line, leaving
the other 28 byte-identical. pcapkit/dumpkit/common.py stays at 100%
coverage over tests/dumpkit/, 84 -> 88 statements and 38 -> 40 branches,
no new misses. make pylint unchanged (6 pre-existing messages), make mypy
clean.
|
Cross-review at The corrected diagnosis is confirmed at source. The 2 of 30, re-derived from its own fixtures. It generated captures itself and dumped 6 pcapng captures × 5 formats on Double-escaping ruled out structurally, not just by test: the old inline Merge into current Two incidental findings, neither blocking. The stale skip reason at CI fully green: 27 success, 3 expected skips. |
#784 taught pcapkit/dumpkit/common.py's escape_key to XML-escape mapping keys, so the plist half of this class's docstring is no longer true: the plist report of test.pcapng now parses as well-formed XML. Only the json writer's quoting defect (dictdumper/json.py:224, unescaped quotes in a bytes-repr key) still stands, still open upstream as JarryShaw/DictDumper#125. - Class docstring: drop the claim that both json and plist reports come out unparseable; note that plist parses since #784. - Method docstring: replace the stale "line 915"/"line 1376" numbers with what this run actually measured on current main -- json fails at line 1019 column 12, and plist no longer fails at all -- and flag the json number as fixture-dependent rather than a fixed fact, since a drifted instance of exactly this number is what #785 was filed over. Measured on ef859f7: json.load raises "Expecting ':' delimiter: line 1019 column 12" on the regenerated test.pcapng; xml.etree.ElementTree.parse now succeeds on the same fixture's plist report. Verified 1517/1958 was the correct pre-#784 plist failure line (0419c1c, matching #785's own measurement), so 1376 had already drifted before #784 landed. The skip reason itself named no false claim and needed no change. Build/tests: 6 passed, 1 skipped (unchanged) in tests/integration/test_pcapng_end_to_end.py.
#784 taught pcapkit/dumpkit/common.py's escape_key to XML-escape mapping keys, so the plist half of this class's docstring is no longer true: the plist report of test.pcapng now parses as well-formed XML. Only the json writer's quoting defect (dictdumper/json.py:224, unescaped quotes in a bytes-repr key) still stands, still open upstream as JarryShaw/DictDumper#121. - Class docstring: drop the claim that both json and plist reports come out unparseable; note that plist parses since #784. - Method docstring: replace the stale "line 915"/"line 1376" numbers with what this run actually measured on current main -- json fails at line 1019 column 12, and plist no longer fails at all -- and flag the json number as fixture-dependent rather than a fixed fact, since a drifted instance of exactly this number is what #785 was filed over. Measured on ef859f7: json.load raises "Expecting ':' delimiter: line 1019 column 12" on the regenerated test.pcapng; xml.etree.ElementTree.parse now succeeds on the same fixture's plist report. Verified 1517/1958 was the correct pre-#784 plist failure line (0419c1c, matching #785's own measurement), so 1376 had already drifted before #784 landed. The skip reason itself named no false claim and needed no change. Build/tests: 6 passed, 1 skipped (unchanged) in tests/integration/test_pcapng_end_to_end.py.
#784 added an escape_key helper to pcapkit/dumpkit/common.py that XML-escapes mapping keys, so the plist half of this class's docstring is no longer true: the plist report of test.pcapng now parses as well-formed XML. Only the json writer's quoting defect (dictdumper/json.py:224, unescaped quotes in a bytes-repr key) still stands, still open upstream as JarryShaw/DictDumper#121. - Class docstring: drop the claim that both json and plist reports come out unparseable; note that plist parses since #784. - Method docstring: replace the stale "line 915"/"line 1376" numbers with what this run actually measured on current main -- json fails at line 1019 column 12, and plist no longer fails at all -- and flag the json number as fixture-dependent rather than a fixed fact, since a drifted instance of exactly this number is what #785 was filed over. Measured on ef859f7: json.load raises "Expecting ':' delimiter: line 1019 column 12" on the regenerated test.pcapng; xml.etree.ElementTree.parse now succeeds on the same fixture's plist report. Verified 1517/1958 was the correct pre-#784 plist failure line (0419c1c, matching #785's own measurement), so 1376 had already drifted before #784 landed. The skip reason itself named no false claim and needed no change. Build/tests: 6 passed, 1 skipped (unchanged) in tests/integration/test_pcapng_end_to_end.py.
#784 added an escape_key helper to pcapkit/dumpkit/common.py that XML-escapes mapping keys, so the plist half of this class's docstring is no longer true: the plist report of test.pcapng now parses as well-formed XML. Only the json writer's quoting defect (dictdumper/json.py:224, unescaped quotes in a bytes-repr key) still stands, still open upstream as JarryShaw/DictDumper#121. - Class docstring: drop the claim that both json and plist reports come out unparseable; note that plist parses since #784. - Method docstring: replace the stale "line 915"/"line 1376" numbers with what this run actually measured on current main -- json fails at line 1019 column 12, and plist no longer fails at all -- and flag the json number as fixture-dependent rather than a fixed fact, since a drifted instance of exactly this number is what #785 was filed over. Measured on ef859f7: json.load raises "Expecting ':' delimiter: line 1019 column 12" on the regenerated test.pcapng; xml.etree.ElementTree.parse now succeeds on the same fixture's plist report. Verified 1517/1958 was the correct pre-#784 plist failure line (0419c1c, matching #785's own measurement), so 1376 had already drifted before #784 landed. The skip reason itself named no false claim and needed no change. Build/tests: 6 passed, 1 skipped (unchanged) in tests/integration/test_pcapng_end_to_end.py.
Please follow the guide below
You will be asked some questions, please read them carefully and answer honestly
Put an
xinto all the boxes [ ] relevant to your pull request (like that [x])Use Preview tab to see how your pull request will actually look like
Searched for similar pull requests
Followed the coding style (
make pylint,make mypy,make isort)make testpasses, and a test case covers the changeAdded a changelog entry under
docs/source/changelog/and regeneratedCHANGELOG.md, if the change is user-visible — N/A — changelog centralised in docs(changelog): shared 1.5.0 changelog — long-lived, merges last (#610, #616, #617, #618, #620) #657What is the purpose of your pull request?
Tick the commit type your subject line carries.
fix— corrects a defectfeat— adds a featureperf— changes performance, not behaviourrefactor— changes neither behaviour nor performancetest— tests onlydocs— documentation onlyci— workflows or build toolingchore— anything elseDescription of your pull request and other information
The in-repo half of #772. #771 escaped every dumped value for the plist writer, but
_append_dictinterpolates a key into'<key>{item}</key>'and only calls_encode_valueonthe value, so
object_hooknever sees a key. Both branches that build a mapping now escape theirown keys through one
escape_keyhelper — aMultiDict, where #771 escaped an enum-derived keyinline and nothing else, and a plain
dict, where nothing was escaped.Non-
strkeys are rendered withformat, not skipped, and notstr-ed arbitrarily.format(key, '')is the conversion
'{item}'.format(item=key)already applies, so the<key>text is unchangedapart from the escaping. It has to be handled:
test.pcapng's decryption secrets block keys its TLSkey log entries by a raw
bytesclient random whose repr carries&,<and>— and that keyreaches the writer through an
OrderedMultiDict, so fixing only the plain-dictbranch would nothave fixed the fixture.
test.pcapngbefore:ET.parsefails,not well-formed (invalid token): line 1517, column 19.After: parses. Measured over 6 captures × 5 formats on
fe80b8525: 2 of the 30 reports changed(that capture's
plistandxml, which are one writer under two names), by exactly one line;the other 28 — every
json,treeandtextreport included — are byte-identical. Nothingdouble-escapes: no
&lt;anywhere.pcapkit/dumpkit/common.pystays at 100% coverage overtests/dumpkit/(84 → 88 statements,38 → 40 branches, no new misses). The upstream half is JarryShaw/DictDumper#125, which would make
this workaround redundant; this does not wait on it.
Closes #772 — the pcapkit-side half. The upstream
_encode_value-on-keys change staystracked at JarryShaw/DictDumper#125, in that repo, and is not a hold on this board.