Skip to content

fix: keep non-ASCII text readable in merged JSON settings files - #4773

Merged
mnriem merged 2 commits into
github:mainfrom
kartsan03:fix/non-ascii-json-configs
Sep 28, 2026
Merged

mnriem merged 2 commits into
github:mainfrom
kartsan03:fix/non-ascii-json-configs

Conversation

@kartsan03

Copy link
Copy Markdown
Contributor

Description

Two writers that merge Spec Kit entries into a user's existing JSON config call json.dumps without ensure_ascii=False, so every non-ASCII character already in that file comes back as a \uXXXX escape:

  • CopilotIntegration._merge_vscode_settings → .vscode/settings.json (Copilot --commands mode)
  • events._safe_write_json → the agent hook configs merged by install_integration_events / remove_integration_events (.claude/settings.json, opencode.json, .cursor/hooks.json, .devin/hooks.v1.json, …)

Both files are user-owned and hand-edited (_load_user_json already treats them that way), so running specify init or toggling events mangles text the user wrote. This is the JSON counterpart of #4148, which fixed the same thing for YAML overlay files.

Reproduction on current main, with an existing .vscode/settings.json:

{
    "cSpell.words": ["naïve", "Привіт"]
}

specify init --here --integration copilot --integration-options="--commands" --force rewrites it as:

{
    "cSpell.words": [
        "naïve",
        "Привіт"
    ],

Same for .claude/settings.json (e.g. an env value) when Claude hooks are installed.

The value still parses back identically, so this is not data loss, only legibility of a file the user owns. Both files are already written with encoding="utf-8"; pure-ASCII files produce identical bytes.

Testing

  • Tested locally with uv run specify --help

  • Ran existing tests with uv sync && uv run pytest

  • Tested with a sample project (if applicable)

  • New tests, one per writer: test_setup_merge_keeps_non_ascii_vscode_settings_readable (Copilot) and test_merge_keeps_non_ascii_user_settings_readable (Claude hooks merge). Both fail on main and pass with the change; each also checks the value round-trips through json.loads.

  • Full suite: 8399 passed, 212 skipped (Linux, Python 3.13)

  • uvx ruff@0.15.0 check src tests: clean

  • Sample project: the specify init command above, run against main and this branch (output shown above).

AI Disclosure

  • I did not use AI assistance for this contribution
  • I did use AI assistance (fill in the disclosure below)

AI disclosure: Claude Code (Claude Opus 5.5, xhigh reasoning effort, autonomous agent mode) was used for drafting/refactoring the code change, the regression tests and this description.

The Copilot .vscode/settings.json merge and the events writer that
merges hooks into agent configs (.claude/settings.json, opencode.json,
.cursor/hooks.json, ...) called json.dumps without ensure_ascii=False,
so every non-ASCII character already in the user's file was rewritten
as a \uXXXX escape. Both files are user-owned and hand-edited.

Assisted-by: Claude Code (model: Claude Opus 5.5, autonomous)
@kartsan03
kartsan03 requested a review from mnriem as a code owner September 28, 2026 13:46
@mnriem mnriem added the triage-can-wait Verdict: valid and in-scope but deprioritized; held behind the evidence gate label Sep 28, 2026
@mnriem
mnriem requested a balanced review from Copilot September 28, 2026 15:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Both writers can now fail on accepted JSON containing escaped lone surrogates.

Review effort: Balanced
Findings: 2 High severity

Open (2)
What changed in this PR

Preserves readable non-ASCII text when merging user-owned JSON settings.

Changes:

  • Disables ASCII escaping in both JSON writers.
  • Adds regression coverage for Copilot and event configuration merges.
File Description
src/​specify_cli/​integrations/​copilot/​__init__.py Preserves Unicode in VS Code settings.
src/​specify_cli/​events/​__init__.py Preserves Unicode in event configurations.
tests/​integrations/​test_integration_copilot.py Tests readable Unicode in VS Code settings.
tests/​specify_cli/​events/​test_events.py Tests readable Unicode in Claude settings.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/specify_cli/events/__init__.py Outdated
Comment thread src/specify_cli/integrations/copilot/__init__.py Outdated
json.loads accepts an escaped lone surrogate such as "\ud800", and with
ensure_ascii=False the strict UTF-8 write then raised UnicodeEncodeError
on a config the old writer could round-trip. errors="backslashreplace"
writes that character back as its \uXXXX escape and leaves the rest of
the text readable.

Assisted-by: Claude Code (model: Claude Opus 5.5, autonomous)
@kartsan03

Copy link
Copy Markdown
Contributor Author

Addressed the Copilot review in 4d6eec2. Both writers now pass errors="backslashreplace", so an escaped lone surrogate from the user's file ("\ud800") is written back as its \uXXXX escape instead of raising UnicodeEncodeError; all other text stays readable. Added one regression test per writer; both fail on f2cc31a and pass now. tests/integrations, tests/specify_cli/events, tests/test_merge.py: 2720 passed; ruff clean.

Drafted on behalf of @kartsan03 by Claude Code (model: Claude Opus 5.5, autonomous); code, tests and this comment AI-drafted.

@mnriem

mnriem commented Sep 28, 2026

Copy link
Copy Markdown
Collaborator

Please address Copilot feedback

@kartsan03

Copy link
Copy Markdown
Contributor Author

@mnriem Both Copilot findings are addressed in 4d6eec2 (summary above), and the two threads are now outdated. Could you re-run the Copilot review when you get a chance?

Drafted on behalf of @kartsan03 by Claude Code (model: Claude Opus 5.5, autonomous); comment AI-drafted.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The focused changes preserve valid JSON behavior and include positive and edge-case regression coverage.

Review effort: Balanced
Findings: None

Resolved since last review (2)

@mnriem
mnriem merged commit 7b50c45 into github:main Sep 28, 2026
15 checks passed
@mnriem

mnriem commented Sep 28, 2026

Copy link
Copy Markdown
Collaborator

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

triage-can-wait Verdict: valid and in-scope but deprioritized; held behind the evidence gate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants