Skip to content

Speed up dependency resolution - #1272

Merged
ericmj merged 16 commits into
mainfrom
vendor-hex-solver-v0.3.1
Oct 14, 2026
Merged

Speed up dependency resolution#1272
ericmj merged 16 commits into
mainfrom
vendor-hex-solver-v0.3.1

Conversation

@ericmj

@ericmj ericmj commented Oct 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Vendors hex_solver 0.3.2, which has the resolution changes from hexpm/hex_solver#61, and speeds up Hex's side of resolution.

Measured on a copy of hexpm's mix.exs and mix.lock (158 lock entries) on macOS with Elixir 1.20.4 / OTP 29.1.1. main is d62dbe4, which vendors hex_solver 0.3.0. Times are medians, with the three archives alternating on each iteration and each using its own project copy and HEX_HOME, warmed from the same 5.6 MB registry cache.

v2.5.2 main This PR
Offline mix deps.get --check-locked (9 runs) 945 ms 639 ms 414 ms
Online mix deps.get --check-locked, no registry changes (7 runs) 1,450 ms 879 ms 643 ms
Resolution in offline mix deps.update --all (9 runs) 451 ms 213 ms 22 ms
Offline mix deps.get into an empty deps/, tarballs cached (7 runs) 7.76 s 6.93 s 6.77 s
Online mix deps.get with an empty Hex cache and no deps/ (5 runs) 8.86 s 7.62 s 7.53 s

mix help deps.get alone takes 300 ms. mix deps.update --all writes the same mix.lock with all three.

These are the performance changes on main since v2.5.2:

PR Change
#1256 hex_core resolves repository credentials without a global lock and unpacks with raw file operations
#1257 Registry and policy files are fetched with four times http_concurrency, 32 by default
#1258 Tarballs are streamed to and from the cache instead of held in memory
#1259 Each package's registry versions are sorted once
#1260 Packages are unpacked in parallel while they're fetched
#1262 Cached registry and policy files are revalidated with their etags instead of downloaded again
#1267 Each resolved package's dependencies are loaded once when resolving application names

This PR adds these:

Change Repo Cost on main
Fetch each package's versions once per solve hex_solver 10,006 version queries for 155 packages, each through the registry process
Cache the allowed version count and preferred version of each package range hex_solver 9,878 package range evaluations and 256,410 version checks in 156 decisions
Index root dependencies that have sub dependencies hex_solver 13,720 lookups, each scanning 126 root dependencies
Load older versions' dependencies only as far as a dependency's constraint stays the same hex_solver 3,689 dependency lookups for 153 resolved packages
Consolidate Hex's own protocols when building the archive, since Mix doesn't consolidate the protocols of archives. CI checks the built archive Hex 292,203 unconsolidated Hex.Solver.Constraint dispatches
Write the registry cache only when it changed Hex Writing the whole cache on every run, 38 ms for a 5.6 MB cache
Store the cache as one CRC32-checked term in cache.bin instead of an ETS file Hex 38 ms to load the 5.6 MB cache, 14 ms now
Parse each requirement string once Hex 13,045 requirement parses, 463 of them distinct
Check each repository's URL once per prefetch Hex 183 repository checks for one repository
Index application names and dependents when resolving app names Hex 36,858 and 23,409 comparisons
Visit each locked dependency once when unlocking for mix deps.update Hex 1,412 visits for 158 lock entries
Keep lock entries of unchanged releases instead of rebuilding them Hex A registry dependency read and a deps/ manifest read per resolved package
Reuse advisories already read when annotating dependency changes Hex A second advisories read per resolved package
Group dependency changes by prepending instead of appending Hex A list copy per resolved package
With a policy, skip repositories it doesn't restrict and classify versions against the package's own overrides Hex Scanning and validating every override, and three metadata reads, per version
With a policy, fetch it together with the packages Hex One policy request before the package requests, online

The counts in the last column come from profiling offline mix deps.get --check-locked on a copy of hexpm with 155 locked packages at eaa8bd6 (hexpm/hex#1267). The cache timings are from loading and writing the 5.6 MB cache above.

Some behaviour changes with this. The registry cache is now cache.bin, and the first run reads the cache.ets of earlier Hex versions and leaves it in place for them. The solver uses one list of versions per package for the whole solve, so a version that passes a policy cooldown during a solve doesn't change it. Hex only warns about unsatisfiable requirements, and only rejects invalid dependency names, in versions the solver looked at. A policy that fails to fetch online can now fail after an authentication prompt for an organization, while an invalid policy setting or a policy missing from the cache offline still fails first.

hex_solver 0.3.1 has hexpm/hex_solver#60 and hexpm/hex_solver#59. With #1269, a failed resolution after compiling in the same command, such as mix do compile + deps.get, prints the explanation instead of raising Protocol.UndefinedError.

@ericmj ericmj changed the title Vendor hex_solver v0.3.1 Speed up dependency resolution Oct 14, 2026
ericmj added 14 commits October 14, 2026 19:03
@ericmj
Vendor hex_solver 2a24512
ae2f8b0 
Vendors the unreleased resolution performance changes from hex_solver's
speed-up-resolution branch: an index of root dependencies with sub
dependencies, versions and candidates cached for the solve, and lazy
loading of older versions' dependencies.

The solver no longer reads the dependencies of every release, so the
comment on Hex.Registry.Server.dependencies/3 now says it reads the
dependencies of releases it doesn't select.
@ericmj
Consolidate Hex's protocols in the archive
f19ffcf 
Mix doesn't consolidate the protocols of archives, so every call to
Hex.Solver.Constraint dispatched through the unconsolidated protocol.
On a copy of hexpm with 153 packages, that made resolution about 3 ms
slower than with the protocol consolidated. mix archive.build now
consolidates the protocols Hex defines into the compiled beams before
building the archive, and CI checks that the protocols in the built
archive are consolidated.
@ericmj
Write the registry cache only when it changed
a3a0f1d 
Every mix deps.get wrote the whole registry cache back to disk, with an
MD5 checksum and a sync, even when nothing in it changed, such as an
offline run or one where every registry file was unchanged. That took
about 94 ms for a 17 MB cache. Track whether the cache changed since it
was loaded or last written, and only write it then.
@ericmj
Check each repository's URL once per prefetch
4d77fe9 
Prefetching looked up the repository config and compared its URL with
the cached one for every package, though most prefetches are for
packages from one or two repositories. Check each repository once.
@ericmj
Parse each dependency requirement once
dc70fc7 
The registry parsed the requirement of every dependency it returned,
though releases repeat the same requirements: the historical releases
of hexpm's 155 locked packages have 13,045 requirements but only 463
distinct ones. Keep the parsed constraints in the calling process.
Unsatisfiable requirements aren't kept, so each release that has one
still warns.
@ericmj
Store the registry cache as one checksummed term
67bd8e2 
The registry cache was an ETS file, which :ets.file2tab/2 reads with
one term and MD5 update per object, about 34,000 objects for a 5.6 MB
cache. Loading it took 38 ms. Store the cache as a header, a CRC32 and
all objects as one term instead, which loads in 14 ms and writes in 15
ms instead of 38 ms.

The cache is written to a temporary file that then replaces it, so
another Mix process reading the cache never sees a partly written file.

The new cache is cache.bin. When it doesn't exist, Hex starts from the
cache.ets of earlier versions, or reports why it couldn't read it, and
leaves that file, so Hex versions that share the cache directory don't
delete each other's caches.
@ericmj
Index application names when resolving them
703ba8e 
add_apps_to_resolved/2 searched every resolved package's dependencies
and the project's requests for each resolved package, and compared each
application name with every other. For hexpm's 153 resolved packages
that was 36,858 and 23,409 comparisons. Index the packages' dependents
and the project's application names once, and group the resolved
packages by application name. The first declaration still wins and
errors list packages in the same order.
@ericmj
Visit each locked dependency once when unlocking
72fab60 
Unlocking a dependency for mix deps.update also unlocks everything it
depends on in the lock. The traversal didn't track visited
dependencies, so a dependency shared by many paths, and everything
below it, was visited once per path: 1,412 visits for hexpm's 158 lock
entries. Track the visited dependencies in a set and check unlocked
entries against it.
@ericmj
Group dependency changes without appending to lists
e736fe1 
group_dependency_changes/2 appended each resolved package to the end of
its change group, copying the group every time. Prepend and reverse the
groups at the end instead.
@ericmj
Reuse release data after resolving
2801c15 
After the solver finished, Hex built a full lock entry for every
resolved package, reading its dependencies from the registry and its
manifest from deps/, and then kept the current lock entry anyway when
the release was the same. Keep the current entry when building the
lock instead, so only new and updated entries are built.

Annotating the dependency changes read each release's advisories a
second time. Use the advisories already read.
@ericmj
Filter versions by policy and cooldown with less work
ca4cc4a 
With a policy active, Hex classified every version of a package by
scanning and validating every override in the policy, and read each
version's advisories, retirement and publish time even when the policy
had no rules for the package's repository. Skip repositories the policy
doesn't restrict, and narrow the policy to the package's valid
overrides before classifying its versions, which classifies them the
same.

Record blocked and cooldown filtered versions with one state update per
package, and reuse the publish times the cooldown filter already read.
@ericmj
Fetch the policy together with the packages
539904c 
With a policy configured, the converger fetched it and waited for it
before prefetching any package, so the policy request and the package
requests ran one after the other. Check the policy configuration first
as before, queue the policy ahead of the packages after the
authentication checks, and wait for it before resolving.
@ericmj
Add the resolution performance changes to the changelog
c802717
@ericmj
Start from an empty registry cache when its data can't be decoded
1424040 
A cache.bin with a matching checksum but data that binary_to_term or
:ets.insert rejects raised in Registry.open, so the registry server
crashed on every run until the file was removed. It's now reported and
replaced like a cache with a checksum mismatch.
@ericmj
ericmj force-pushed the vendor-hex-solver-v0.3.1 branch from d54cad5 to 1424040 Compare October 14, 2026 17:22
@ericmj ericmj mentioned this pull request Oct 14, 2026
Merged
@ericmj
ericmj merged commit 5260292 into main Oct 14, 2026
22 checks passed
@ericmj
ericmj deleted the vendor-hex-solver-v0.3.1 branch October 14, 2026 22:54
This was referenced Oct 14, 2026
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ericmj