perf(application): raise run prepare checksum read buffer to 8MB [PYSDK-148]#694
Merged
Merged
Conversation
The CRC32C checksum in run prepare read source slides in 1KB chunks, turning a multi-GB WSI into ~millions of tiny reads whose per-read overhead dominates on slow/high-latency storage (USB, network drives). A local benchmark measured ~42x faster full-file hashing at 8MB vs 1KB. Introduce APPLICATION_RUN_METADATA_CHECKSUM_CHUNK_SIZE (8MB) and use it in the prepare checksum loop. Checksum result is unchanged; 8MB stays well within the 8GB RAM floor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
omid-aignostics
changed the title
omid-aignostics
added
sop:cc-sop-01
Codecov Report✅ All modified and coverable lines are covered by tests.
|
omid-aignostics
enabled auto-merge (squash)
July 15, 2026 09:58
Generate the CycloneDX SBOM with `cyclonedx-py environment --pyproject pyproject.toml` so the SBOM includes a root component (the `aignostics` application) that anchors the dependency graph. Without a root component, all ~400 dependencies are emitted as unanchored nodes, forcing Ketryx's synchronous /builds endpoint to do excessive dependency-relation work; it runs for minutes and returns HTTP 500. This reproduced deterministically per commit (e.g. commit a1b4931 failed 10+ retries over two days) and was blocking Ketryx build reporting for the release. Ketryx flagged the missing root component as the root cause. Verified locally: the regenerated SBOM now has root component `aignostics` (type: application) anchoring 67 direct dependencies. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Empty commit to force a new commit SHA so Ketryx processes a brand-new build. Re-running the failed job re-submits an existing build (created despite the earlier 500) and hits a short-circuit path, so it is not a reliable test of Ketryx's server-side fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Ketryx action derives `changeRequestNumber` from GITHUB_REF_NAME when it ends in `/merge` (pull_request runs, e.g. "694/merge"). Ketryx then attempts to fetch the PR as a Code Change Request (CCR) and returns HTTP 500 when that fetch fails — which per Ketryx is due to the PAT lacking the required GitHub permissions. This failed the build even though the build and SBOM were reported successfully server-side, blocking the release. We do not use the CCR / item-association feature (check-item-association is false), so override GITHUB_REF_NAME to the branch name (which never ends in `/merge`) for the report step, making the action send no changeRequestNumber and skip the failing CCR fetch. Root cause identified by Ketryx (Anton) in the PYSDK support thread. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…0 on PRs The prior step-level `env: GITHUB_REF_NAME` override was silently ignored: GitHub Actions does not allow `env:` to override default GITHUB_* variables, so the action still read "694/merge" and sent changeRequestNumber=694 (confirmed via debug: `Determined PR number 694 based on ref name 694/merge`), triggering Ketryx's failing CCR fetch and HTTP 500. Rewrite GITHUB_REF_NAME via $GITHUB_ENV (which can override defaults) to the branch name so no changeRequestNumber is sent. Additionally mark the report step continue-on-error on pull_request runs as a safety net: the build and SBOM land in Ketryx regardless of the 500, so a Ketryx-side failure must not block PR CI. On push/tag runs (releases) the ref never ends in "/merge" and continue-on-error is false, so the gate is still enforced. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
omid-aignostics
removed
the
skip:test:long_running
Empty commit to trigger a fresh pull_request run after removing the skip:test:long_running label, so long-running tests execute and Codecov project coverage returns above the 70% threshold (they cover ~10% of the codebase; skipping them dropped project coverage to 66.76%). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
olivermeyer
approved these changes
Jul 21, 2026
omid-aignostics
added a commit
that referenced
this pull request
Jul 21, 2026
Mirror the Ketryx report workflow change from #694 onto this branch so the non-required ketryx_report_and_check step no longer marks PR CI red while the Ketryx-side CCR/PAT 500 is being resolved. Build still lands in Ketryx. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
omid-aignostics
added a commit
that referenced
this pull request
Jul 21, 2026
* docs: add "Invite your team" guidance to user guides [PYSDK-144] The guides stated administrators can invite users but never explained how. Add a shared "Invite your team" partial (Console → Admin → Members) included at the end of all four getting-started guides (before the Launchpad troubleshooting section), and expand the Console section of the Platform Overview with the same steps as the canonical reference. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): tolerate Ketryx 500 on PRs via continue-on-error [PYSDK-144] Mirror the Ketryx report workflow change from #694 onto this branch so the non-required ketryx_report_and_check step no longer marks PR CI red while the Ketryx-side CCR/PAT 500 is being resolved. Build still lands in Ketryx. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: reuse _invite_your_team.md partial on the Platform page [PYSDK-144] Address review feedback: include the shared invite-your-team snippet on README_platform.md instead of a bespoke variant. Genericize the snippet's closing step so it no longer references a guide-specific signup section, so it reads correctly on all five pages. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🛡️ Implements PYSDK-148 following CC-SOP-01 Change Control, part of our ISO 13485-certified QMS | Ketryx Project
Summary
application run preparecomputed the CRC32C checksum of each source slide in 1 KB chunks, turning a multi-GB WSI into ~millions of tiny reads whose per-read overhead dominates on slow / high-latency storage (USB, network drives) — field-reported as effectively unusable for a ~2000-slide batch on a USB drive at a customer site (Charité).APPLICATION_RUN_METADATA_CHECKSUM_CHUNK_SIZE(8 MB) and uses it in the prepare checksum loop, replacing the hardcoded 1024 bytes. The CRC32C result is byte-identical; 8 MB stays well within the 8 GB RAM floor.Test plan
ruff check+ruff format --checkpass on the changed filecli_pipeline_validation_test.pypasses (7 tests)run prepareon the large-slide USB batch at Charité completes materially fasterRisk
Low — single-file change, checksum output unchanged, no API / schema / dependency changes.
Posted by Claude claude-opus-4-8 via Claude Code, applying skills cc-sop-01 on behalf of Omid Kokabi