Skip to content

fix: handle CRLF line endings in cross-platform submission hashing - #9588

Closed
sslxqii-sudo wants to merge 2 commits into
OpenMined:devfrom
sslxqii-sudo:fix-9551-windows-hash
Closed

sslxqii-sudo wants to merge 2 commits into
OpenMined:devfrom
sslxqii-sudo:fix-9551-windows-hash

Conversation

@sslxqii-sudo

Copy link
Copy Markdown

Summary

Fixes OpenMined/PySyft issue #9551 by making submission hashes consistent across Windows and Linux.

Changes

  • Normalize CRLF and LF line endings when hashing text files.
  • Sort file paths deterministically.
  • Update job and request hash verification.
  • Add regression tests for line endings, file ordering, binary files, and file integrity.

Testing

  • Targeted tests: 10 passed.
  • Full syft-job test suite: 170 passed, 66 warnings.
  • Python compilation and git diff --check: passed.
  • Tests ran on Linux (WSL2). Native Windows pytest collection was blocked by the unavailable fcntl module.

Related Issue

Closes #9551

Asana Task

Asana task URL: Not provided.

Status

Changes committed and pushed to the fix-9551-windows-hash branch. Linux tests passed. Pull request review and CI status are pending; not merged.

@sslxqii-sudo sslxqii-sudo changed the title Fix 9551 wFix cross-platform submission hashing for CRLF line endingsindows hash fix: make submission hashes deterministic across platforms Oct 27, 2026
@sslxqii-sudo sslxqii-sudo changed the title fix: make submission hashes deterministic across platforms fix: make submission hashes deterministic across platforms. Oct 27, 2026
@sslxqii-sudo sslxqii-sudo changed the title fix: make submission hashes deterministic across platforms. fix: make submission hashes deterministic across platforms Oct 27, 2026
@sslxqii-sudo sslxqii-sudo changed the title fix: make submission hashes deterministic across platforms fix: make submission hashes deterministic across platforms . Oct 27, 2026
@sslxqii-sudo sslxqii-sudo changed the title fix: make submission hashes deterministic across platforms . fix: make submission hashes deterministic across platforms// Oct 27, 2026
@sslxqii-sudo sslxqii-sudo changed the title fix: make submission hashes deterministic across platforms// fix: handle CRLF line endings in cross-platform submission hashing Oct 27, 2026
@sslxqii-sudo

Copy link
Copy Markdown
Author

Hi maintainers! The PR title validation is now passing. Could someone with write access please review my changes and approve the PR? Thank you!

…ically (OpenMined#9551)

- Use explicit UTF-8 decoding in _as_sent to handle universal newlines and preserve binary fallback
- Sort file entries deterministically by relative POSIX paths in _digest
- Pass as_sent=True in _job_has_all_files and _request_has_all_files to match declared hash contract
- Add regression tests covering CRLF/LF invariance, path sorting, UTF-8/binary content, and receiver-side checks
@sslxqii-sudo
sslxqii-sudo force-pushed the fix-9551-windows-hash branch from bd597b0 to b6aab68 Compare October 28, 2026 16:25
@koenvanderveen

Copy link
Copy Markdown
Collaborator

Thanks for this! Both parts are now on dev: #9593 sends job files byte for byte, so both sides hash raw bytes and line endings no longer differ, and #9596 sorts files by their posix path, so the order is the same on Windows and Linux. Closing this as covered.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

syft-job: declared code_hash never matches on a CRLF checkout, and submission_hash order differs on Windows

2 participants