Skip to content

Strip leading UTF-8 BOM when reading INI files - #76

Open
Pitchfork-and-Torch wants to merge 2 commits into
pytest-dev:mainfrom
Pitchfork-and-Torch:cook/strip-utf8-bom
Open

Pitchfork-and-Torch wants to merge 2 commits into
pytest-dev:mainfrom
Pitchfork-and-Torch:cook/strip-utf8-bom

Conversation

@Pitchfork-and-Torch

Copy link
Copy Markdown

Summary

Windows editors often write a UTF-8 BOM. With the default encoding=\"utf-8\", that left U+FEFF on the first section line and raised ParseError: unexpected line.

Strip a leading BOM after read for both IniConfig() and IniConfig.parse(), including when content is passed via data=.

Test plan

  • test_utf8_bom_file, test_utf8_bom_in_data_string, test_parse_utf8_bom_file pass

Windows editors often write a UTF-8 BOM; with encoding=utf-8 that left
U+FEFF on the first section line and raised ParseError. Strip the BOM
after read for both IniConfig() and IniConfig.parse().

@RonnyPfannschmidt RonnyPfannschmidt left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Written by Claude Opus 5.5 via Claude Code for the iniconfig maintainers; I prompted it, it did the work, I read it.

Thanks, stripping a leading BOM is a welcome fix. Please make three changes before this goes in:

  1. Strip it in one place. The same block is currently in both IniConfig.__init__ and IniConfig.parse. Move it into _parse.parse_ini_data, which every path (file read and data=) already goes through.
  2. Use a named constant. Define the BOM once at module level, e.g. UTF8_BOM: Final = "", and use that name instead of an inline literal.
  3. Write it as an escape. The current diff contains a raw U+FEFF character inside "...", which is invisible in editors and in review. Spell it "" so it is readable. The startswith check is then unnecessary: data = data.removeprefix(UTF8_BOM) does the same thing.

Separately, pre-commit.ci is red: mypy wants annotations on the three new tests (-> None and tmp_path: Path).


Generated by Claude Code

Copy link
Copy Markdown
Member

🤖 Written by Claude Opus 5.5 via Claude Code for the iniconfig maintainers; I prompted it, it did the work, I read it.

Correction to my review: both snippets in it were mangled into the raw invisible BOM. The intended code is:

UTF8_BOM: Final = "\N{BYTE ORDER MARK}"

data = data.removeprefix(UTF8_BOM)

Generated by Claude Code

Strip once in parse_ini_data, name the mark with \N{BYTE ORDER MARK}, and annotate the new tests.
@Pitchfork-and-Torch

Copy link
Copy Markdown
Author

Moved the strip into parse_ini_data. The constant is UTF8_BOM: Final = "\N{BYTE ORDER MARK}", and the three new tests now have return annotations plus a Path type on tmp_path.

@RonnyPfannschmidt RonnyPfannschmidt left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well done thanks

do you want to squash or should I squash merge

@Pitchfork-and-Torch

Copy link
Copy Markdown
Author

Please squash merge — happy either way as long as it lands as one commit. Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants