Fix Windows compatibility: atomic rename and explicit UTF-8 I/O - #5
Open
mutoharohfiqhiabcd-source wants to merge 1 commit into
Open
mutoharohfiqhiabcd-source wants to merge 1 commit into
mutoharohfiqhiabcd-source wants to merge 1 commit into
Conversation
The scripts could not complete a run on Windows at all. Three independent
issues, all stemming from POSIX assumptions:
1. Path.rename() overwrites an existing destination on POSIX, but raises
FileExistsError (WinError 183) on Windows when the target exists.
map.py used it on both cache and progress-file save paths, so indexing
aborted before repo-map.db was ever written. Replaced with os.replace(),
which atomically overwrites on both platforms. goals.py and
context_saver.py already used os.replace(); map.py was the outlier.
2. write_text()/read_text() without an explicit encoding fall back to the
locale codec -- GBK on zh-CN Windows -- which cannot encode characters
such as U+1F4DD in the generated repo map. Added encoding="utf-8" to all
63 call sites across scripts/ and servers/.
3. sys.stdout/stderr also default to the locale codec, so the final
print(repo_map) crashed with UnicodeEncodeError. map.py now reconfigures
both streams to UTF-8.
Verified on Windows 11 (zh-CN), Python 3.12 via uv:
uv run --script scripts/map.py <project-dir>
On a 600-file C++ project this now completes in ~17s and writes
repo-map.db (7597 symbols), repo-map.md and repo-map-cache.json.
This only unblocks indexing. The FTS5 table remains intentionally
unpopulated, per docs/development.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
map.pycannot complete a run on Windows. Indexing aborts beforerepo-map.dbis ever written, so the skill is unusable there out of the box.Three independent issues, all POSIX assumptions.
1.
Path.rename()does not overwrite on Windowsmap.pyusedPath.rename()in two places —Cache.save()and the progress-file finalisation inmain():On POSIX,
rename(2)silently replaces an existing destination. On Windows,os.renameraisesFileExistsError(WinError 183) when the target exists. The cache file exists on every run after the first, so indexing always died at the first checkpoint:os.replace()is the portable equivalent: atomic on both platforms, and it overwrites.Worth noting —
scripts/goals.pyandscripts/context_saver.pyalready useos.replace().map.pywas the only outlier, so this also brings it in line with the rest of the codebase.2. Missing
encoding=onwrite_text()/read_text()Without an explicit encoding these fall back to the locale codec, which is GBK on zh-CN Windows and cannot encode the 📝 (U+1F4DD) used in the generated map:
Fixed 63 call sites across
scripts/andservers/.3. Console codec
The final
print(repo_map)hit the same wall on stdout.map.pynow reconfiguressys.stdout/sys.stderrto UTF-8 witherrors="replace".Verification
Windows 11 (zh-CN), Python 3.12 via
uv, on a 600-file C++ project:Symbol queries against the resulting DB behave as expected:
Scope
This PR only unblocks indexing on Windows. The FTS5 table stays unpopulated, which matches
docs/development.md("The FTS5 table schema exists but is not yet populated or exposed via MCP tools").I'm happy to split this up or drop any part of it if you'd prefer a narrower change.