Skip to content

parse_lines: keep values as list of lines, join once - #78

Merged
RonnyPfannschmidt merged 1 commit into
pytest-dev:mainfrom
RonnyPfannschmidt:claude/dazzling-einstein-h8zq7m
Oct 5, 2026
Merged

RonnyPfannschmidt merged 1 commit into
pytest-dev:mainfrom
RonnyPfannschmidt:claude/dazzling-einstein-h8zq7m

Conversation

@RonnyPfannschmidt

Copy link
Copy Markdown
Member

Each continuation line rebuilt the whole accumulated value string, which is quadratic in the number of continuation lines of a value. ParsedLine.value is now the list of value lines; continuations append to it and parse_ini_data joins it once.

Alternative to #77 with a smaller change.

Claude-Session: https://claude.ai/code/session_012yrarEe7i67yuTE8GC22ey

Each continuation line rebuilt the whole accumulated value string,
which is quadratic in the number of continuation lines of a value.
ParsedLine.value is now the list of value lines; continuations append
to it and parse_ini_data joins it once.

Alternative to pytest-dev#77 with a smaller change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012yrarEe7i67yuTE8GC22ey
@RonnyPfannschmidt
RonnyPfannschmidt force-pushed the claude/dazzling-einstein-h8zq7m branch from 6cac8b1 to eb8b24f Compare October 5, 2026 07:10
@RonnyPfannschmidt

Copy link
Copy Markdown
Member Author

Speed comparison

parse_ini_data end to end, CPython 3.11, best of 5 runs, times in ms:

input main #77 #78
1 value, 100 continuation lines 0.082 0.037 0.035
1 value, 10,000 continuation lines (~135 KB) 23.9 3.2 3.6
1 value, 50,000 continuation lines (~720 KB) 864 17.7 18.3
typical file (200 keys, 3 continuation lines each) 0.74 0.45 0.35

Both PRs remove the quadratic cost, and their speed is the same within noise. At realistic config sizes (a few hundred lines in total, values with at most ~100 continuation lines) the gain is tens to hundreds of microseconds. This is cleanup, not a user-visible performance fix.

Chosen tradeoff

#77 keeps ParsedLine.value a str. To do that it tracks the index of the pending entry and its pieces, uses a closure with nonlocal to flush them, and calls that flush in three places. It also adds a 151-line test module containing a copy of the old parser and a wall-clock timing assertion, which can fail randomly on CI.

#78 makes ParsedLine.value a list[str] instead:

  • a new value starts as [data], or [] if it is empty
  • a continuation line is a single append onto the previous entry
  • parse_ini_data, the only consumer, does "\n".join(value) once

As a result the parser gets shorter than it is on main (+6/−13 in _parse.py), and it needs no extra state and no flush step. The tests change only by turning the expected values in the existing table into lists, plus two new cases ("empty value" and "continuations on several values"). There are no timing tests.

Cost: iniconfig._parse.ParsedLine.value changes type. The module is private, the public IniConfig API is unchanged, and a code search found imports of it only in iniconfig itself (pytest uses only the public API). The CHANGELOG entry mentions it anyway.

Verification

  • Behaviour matches main on 800k randomized inputs for every combination of strip_inline_comments and strip_section_whitespace, compared both at the parse_lines level (with the lists joined back) and on the parse_ini_data result, error cases included.
  • The old "an empty first value is replaced, not newline-joined" behaviour is kept: an empty value starts as [], and a continuation line is never empty after _parseline.
  • The test suite passes, and mypy --strict and ruff (the version pinned in pre-commit) are clean.

Analysis, implementation and benchmarks done with Claude Code, and reviewed by me.

@RonnyPfannschmidt
RonnyPfannschmidt merged commit e038536 into pytest-dev:main Oct 5, 2026
15 checks passed
@RonnyPfannschmidt
RonnyPfannschmidt deleted the claude/dazzling-einstein-h8zq7m branch October 5, 2026 07:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants