fix(cli): detect equal-size binary changes during publish

[atomicdjt]

Fixes #350.

Binary/non-text files contribute a SHA-256 digest of their raw bytes to the existing content-and-metadata hash, so same-size replacements are detected. Publish filtering remains separate from text classification.

Recognized text-suffix files must decode as UTF-8. An invalid UTF-8 text file returns a failed PublishResult that identifies the file and says it is not valid UTF-8; it is not uploaded, and publish_folder continues with other files. Valid UTF-8 text hashing and document identity are unchanged.

Regression coverage exercises direct and folder publication of invalid UTF-8 text, equal-size binary replacements, stable/empty binary content, metadata changes, valid UTF-8 text-hash compatibility, legacy binary-hash transition, and a configured .pdf publish filter that remains separate from text classification. Publisher tests mock only the remote RAGFlow client; no live RAGFlow upload or parsing was performed.

Validation on Windows, Python 3.12.14:

• Focused document/publisher regressions: 20 passed.
• Full CLI suite: python -m pytest src/rosetta-cli/tests -q — 120 passed.
• python -m mypy --config-file mypy.ini src/rosetta-cli — passed, 22 source files.
• python -m build in src/rosetta-cli — wheel and sdist built.
• python -m twine check src/rosetta-cli/dist/* — both passed.
• git diff --check — passed.

The repository pre-commit wrapper was exercised. Its Windows path does not run its POSIX test script, and after restoring pinned hooks dependencies the independent hooks suite has 40 unrelated Windows/path-semantics failures (1,401 pass); no hooks code was changed.

[github-actions]

Rosetta Triage Review

Summary: This PR replaces the binary length surrogate with a SHA-256 digest of raw bytes before the existing metadata hash, adds regression coverage for file and folder publishing, and documents the one-time binary republish.

Findings:

• P2: ContentPublisher.file_extensions is both the publish filter and the DocumentData text-classification input. A configured binary suffix (for example, .pdf) is therefore decoded with errors="ignore" and hashed as lossy text instead of raw bytes; different invalid UTF-8 byte sequences can still produce the same hash and be skipped. Keep publish filtering separate from text classification, and use raw-byte hashing when UTF-8 decoding fails.

Caveats:

• The change intentionally republishes each existing default-classified binary once because its stored length-based hash differs. The new tests exercise this migration.

Suggestions:

• Add a regression case with ContentPublisher(file_extensions=[".pdf"]) and equal-size invalid UTF-8 replacements. Also cover an invalid UTF-8 file with a default text suffix.

Validation:

• The source and website architecture pages are synchronized.
• The new tests cover .bin/.pdf equal-size replacements, stable bytes, metadata-sensitive hashes, file/folder publishing, and legacy-hash migration.
• GitHub build-test, CodeQL, and analysis checks are passing at triage time.

Automated triage by Rosetta agent

[atomicdjt]

Addressed in ec349035.

• ContentPublisher.file_extensions now controls folder selection only; it is no longer passed into DocumentData as text classification.
• Text-suffix files use strict UTF-8 decoding. A decode failure produces consistent binary state (is_text=False, no text, line-count, or frontmatter metadata) and hashes the raw-byte digest.
• Regressions cover a configured .pdf filter with equal-size invalid UTF-8 replacements, the one-time lossy-hash migration, subsequent unchanged skipping, filter exclusion, and invalid UTF-8 under a default .md suffix. Existing binary migration and valid UTF-8 text-hash coverage remain in place.

Local validation: 118 CLI tests, mypy, package build, twine checks, and git diff --check passed. The updated upstream CLI workflow and all CodeQL analyses also passed.

[atomicdjt]

Hi @isolomatov-gd — following up now that the P2 from automated triage has been addressed in ec34903.
The fix separates publish filtering from text classification, uses strict UTF-8 decoding with binary fallback, and adds the requested regressions for configured .pdf filters and invalid UTF-8 under text suffixes.
Current validation is green: 118 CLI tests, mypy, package build/twine checks, git diff --check, the updated upstream CLI workflow, and CodeQL.
When you have a chance, could you take another look? Happy to address any remaining concerns. Thanks.

[isolomatov-gd]

@atomicdjt thank you. The SHA-256 fix and the ec349035 follow-up are correct, and the tests cover them well.

One change before merge: a text-suffix file that is not valid UTF-8 must fail publish with a clear error, not fall back to binary.

Why: on that fallback path content_str is None, so frontmatter is not parsed. The document loses its frontmatter tags and sort_order, and it is published without any message. Tags drive retrieval, so the document stops being found by those tags. Instruction and text files must be UTF-8, so invalid encoding is a defect in the source. The author must see it and fix the file.

Please:

• Return a failed PublishResult for that file, with an error that names the file and says it is not valid UTF-8. The rest of the run continues, the same as other per-file failures.
• Change test_invalid_utf8_text_suffix_uses_binary_content_state to assert the failure.
• Update the change-detection sentence in docs/MCP-ARCHITECTURE.md and its web mirror.

[atomicdjt]

@isolomatov-gd Thank you for the clarification. The requested UTF-8 behavior has been addressed in e6f8d403. Invalid UTF-8 files with recognized text suffixes now return a failed per-file PublishResult with a clear filename/encoding error, are not uploaded, and do not prevent the rest of a folder publish from continuing. The regression coverage and both architecture documentation mirrors have also been updated.

The updated hosted CI is green, including Rosetta CLI and CodeQL. When you have a chance, I’d appreciate a re-review. Thank you.