Repository navigation
Docker group browse: read members' images by name, not in OFFSET pages - #702
Merged
Merged
Conversation
A docker group's image list (`GET /browse/images?repo=<group>`, every keystroke of a search in the web UI) read each member's images 1,000 at a time with `OFFSET` pages, each one grouping every image before it again: database work growing with the square of a large proxy's images, on every request. It also read every image of every owning hosted member, the ones the caller cannot read included, to apply `prefer_hosted`. `oci_dao::list_images_after` reads the next batch by name instead: the names from the `(repository_id, image, digest)` index after the last one, then each image's counts from its own manifests in a `LATERAL` lookup, so no plan hash-joins the repository's other manifests (a plain join back did, the test caught it). The group listing asks `oci_dao::images_held` whether an owner holds an image only for the images a member that is no owner would list, a batch at a time, and not for those a reserved prefix already keeps to the owners. The hosted and proxy listing of a caller with a content selector reads the same way. `forge-db`'s `oci_images` test counts the manifest rows a transaction reads to list 10,000 images batch by batch: 20,000 now (each image's index entry and its lookup), where the `OFFSET` pages read about 101,000, a number growing with the square of the images. The server test lists a group over a proxy of 4,500 images through pages and searches, with the image an owner holds kept off the proxy's list for a caller who may not read the owner. Fixes #596 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012e6DJZaXD2mBH9tURsdutM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #596
What was slow
group_image_page/for_each_image(crates/server/src/browse_images.rs) read every image of every member 1,000 at a time withOFFSETpages, each re-running theGROUP BYover everything before it: O(n²/1000) per request, on every keystroke of the UI's search. Withprefer_hostedand owners it also read every image of every owner, readable or not.The fix
oci_dao::list_images_after(repo, q, after, limit): the nextlimitimage names afterafterfrom the existing unique index(repository_id, image, digest)(no new index needed), then each name's tag/manifest counts in aLATERALlookup. My first version joined back tooci_manifest, and Postgres hash-joined the whole repository each batch, which was still quadratic. The new DAO test caught that. TheLATERALplan is index probes only (checked withEXPLAIN ANALYZE).oci_dao::images_held(owners, names)only for the candidates of a leaf that is no owner (an owner serves what it holds whatever), a batch at a time, and skips names a reserved prefix already keeps to the owners. SameGroupMembers::owners_only/servesdecisions as before.for_each_image(hosted or proxy listed to a caller with a selector) uses the keyset scan too.The counted fast path for an unfiltered hosted/proxy listing is unchanged. I did not restore a fast path for a single-member group: with the linear scan it isn't needed to fix this, and it would duplicate the guard logic.
Tests
crates/db/tests/oci_images.rs(deterministic, no timing): lists 10,000 images batch by batch inside one transaction and readspg_stat_get_xact_tuples_returnedforoci_manifestand its indexes. Now 20,000 rows; the issue asked for a bound (< 3 × images). The oldOFFSETloop reads about 101,000 on the same data (I measured it in psql), and that grows with the square of the images.crates/server/tests/registry.rsa_group_lists_a_large_proxys_images_in_batches: a group over a proxy of 4,500 images (seeded by SQL) and an owning hosted member. It checks totals, a page across a batch boundary, two searches, that the owned image stays hidden for a caller who can't read the owner, and that it is listed once they can.Checks run
cargo test -p forge-server --test registry --test browse(45 + 10),cargo test -p forge-db(all)cargo clippy --workspace --all-targets -- -D warnings,cargo fmt --all --check,cargo xtask changelog --checkcargo sqlx prepare --check --workspace -- --all-targets --all-featuresagainst a migrated Postgres 18 (two new.sqlxentries)Changelog:
docs/changes/2.2.0/596.fixed.md.Generated by Claude Code