Skip to content

Delete build sources when builds finish and expire old builds - #545

Open
sjmiller609 wants to merge 1 commit into
mainfrom
hypeship/gc-build-sources
Open

sjmiller609 wants to merge 1 commit into
mainfrom
hypeship/gc-build-sources

Conversation

@sjmiller609

@sjmiller609 sjmiller609 commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Build source tarballs were written world-readable (0644) under builds/<id>/source/ and never deleted. Build directories in general were never cleaned up. Sources can carry build-time secrets, so this left them on disk indefinitely.

  • Permissions: source/ is created 0700 and source.tar.gz 0600.
  • Delete on finish: a build's source/ is removed when it reaches a terminal status (updateBuildComplete, CancelBuild). Nothing reads it after that.
  • Background GC: Start launches a goroutine, so startup isn't blocked. It runs immediately and then every 10 minutes:
    • removes source/ for every finished build, which also cleans up sources retained on existing hosts;
    • removes the whole builds/<id>/ once its metadata.json has been unchanged for build.retention.
  • Not collected: queued and building builds, so startup recovery can still re-run them.
  • New config: build.retention, default 2h, must be positive. It only needs to cover clients reading a build's status and logs right after it finishes.
  • Images: unaffected. hypeman build list / GET /builds/{id} only return builds within the retention window.

Testing

  • go test -race ./lib/builds/ and go test ./cmd/api/config/ pass.
  • New tests:
    • source is removed on completion and on cancel, while metadata is kept;
    • GC keeps pending builds' source, strips the source from recently finished builds, and deletes expired build dirs;
    • stored source has mode 0600 in a 0700 dir.
  • Not run: VM-backed integration tests.

Note

Medium Risk
Changes secret-bearing artifact lifecycle and disk cleanup timing; misconfigured retention or GC bugs could delete sources/logs clients still expect, though queued/running builds are explicitly protected.

Overview
Build source tarballs were left on disk with loose permissions even though they can contain secrets. This PR tightens storage (source/ at 0700, source.tar.gz at 0600) and drops each build's source/ as soon as the job is terminal (success/fail via updateBuildComplete, or cancel).

A background GC starts with the build manager (immediate pass, then every 10 minutes): it strips source/ from any already-finished build (including leftovers on upgraded hosts), and removes the entire builds/{id}/ tree once metadata.json has been unchanged longer than build.retention (default 2h, validated in config). Queued and in-progress builds are never collected so restart recovery can still use their sources.

Config examples and lib/builds docs describe the new knob; Retention is wired from cmd/api/config into the build manager. Tests cover permissions, source removal on complete/cancel, and GC behavior for pending vs recent vs expired builds.

Reviewed by Cursor Bugbot for commit 24b30b4. Bugbot is set up for automated code reviews on this repo. Configure here.

@sjmiller609
sjmiller609 marked this pull request as ready for review November 5, 2026 19:17

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 24b30b4. Configure here.

Comment thread lib/builds/manager.go
// updateBuildComplete updates the build with final results
func (m *manager) updateBuildComplete(id string, status string, digest *string, errMsg *string, provenance *BuildProvenance, durationMS *int64) {
defer m.removeSource(id)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Source deleted before status persists

Low Severity

updateBuildComplete always removes source/ via defer, including when writeMetadata fails and the on-disk status stays queued or building. Startup recovery then treats the build as pending, cannot reopen the source, and fails it instead of retrying.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 24b30b4. Configure here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant