Skip to content

ci(coverage): compact profiles with bounded memory - #462

Merged
XuPeng-SH merged 2 commits into
matrixorigin:mainfrom
XuPeng-SH:codex/coverage-cluster-timeout-20260925
Sep 24, 2026
Merged

XuPeng-SH merged 2 commits into
matrixorigin:mainfrom
XuPeng-SH:codex/coverage-cluster-timeout-20260925

Conversation

@XuPeng-SH

@XuPeng-SH XuPeng-SH commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Cause

Coverage uses the full -coverpkg scope for each of 486 Go test packages. Three successful MatrixOne runs each wrote about 16.2 GB of mostly duplicate profile records. After tests passed, the current AWK compaction took about 274 seconds on the observed runner, reducing the profile to about 30.7 MB. This serial tail consumes CPU and memory and extends the total Coverage job.

The earlier pkg/tests/issues/isolated timeout was caused by package runtime that included embedded-cluster admission waits; #459 already raised the package deadline to 20 minutes. This PR reduces total Coverage work without changing that deadline or the test schedule.

Change

Replace the post-test AWK pass with a standard-library Go compactor. It reads the raw file incrementally, retains one exact source-block/statement-count key with OR-ed hit state, applies the same exclusions, and sorts the resulting records. Memory scales with distinct blocks instead of duplicate rows. It writes beside the raw file and replaces it only after a complete, validated write; failures leave the raw input intact and fail the existing publication gate.

The single go test invocation, -p 8, full -coverpkg, 20-minute package deadline, 60-minute suite watchdog and process-group cleanup are unchanged. Compaction starts only after tests pass, so it cannot compete with or backpressure the test processes.

Related: matrixorigin/matrixone#29368.

Validation

  • Built a synthetic 16.17 GB reconstruction by alternating zero-hit and hit records from a real 30.7 MB Coverage artifact. Two reversed-order local comparisons on identical input gave AWK 100.0/98.1 s versus the compiled Go helper 46.5/48.0 s. The exact workflow-style go run command took 56.5 s including build on that input.
  • Measured CPU time fell from 87–89 s to 37–38 s and peak RSS from about 1.9 GB to 80–81 MB in the compiled-helper comparisons. These are local measurements, not CI-runner results.
  • A canary on the actual Coverage runner used a separate synthetic 16.17 GB profile with 358,733 distinct blocks: AWK took 445.5 s / 406.6 CPU s; Go took 67.7 s / 52.8 CPU s, plus 5 s to build. Output records matched exactly. Peak RSS on that runner was 59 MB AWK / 67 MB Go; both are small compared with the runner's 15 GB memory. The temporary canary workflow was removed from the final diff.
  • Complete normalized block sets and hit bits matched the AWK output byte-for-byte in both rounds. The existing coverage consumer produced identical results on both profiles for a real source-file selection.
  • Focused Go tests cover duplicate order, exact keys, exclusions, buffer boundaries, malformed/truncated input, injected read/write errors and atomic file preservation. go test, go vet, actionlint, workflow shell syntax and git diff --check pass. GPT-6 medium design and overall review found no material implementation blocker.
  • The Check Yaml workflow now triggers when either Go helper file changes, then runs explicit-file go test and go vet with pinned Go 1.26.4. A deliberately failing test returned a nonzero status in the local gate probe. The new-head ARM64 check passed, including the Go test and vet step.

Compaction on the actual production profile, complete Coverage job duration and downstream merge still require validation after merge. The test execution phase is unchanged, so this PR claims a shorter total CI path, not a faster go test phase or a statistical guarantee about flakiness.

@XuPeng-SH
XuPeng-SH marked this pull request as draft September 24, 2026 19:06
@XuPeng-SH
XuPeng-SH force-pushed the codex/coverage-cluster-timeout-20260925 branch from 36f3fb6 to 9aedafb Compare September 24, 2026 19:29
@XuPeng-SH XuPeng-SH changed the title ci(coverage): serialize embedded-cluster test packages ci(coverage): compact profiles with bounded memory Sep 24, 2026
@XuPeng-SH
XuPeng-SH force-pushed the codex/coverage-cluster-timeout-20260925 branch from 0e4853d to 9aedafb Compare September 24, 2026 19:44
@XuPeng-SH
XuPeng-SH marked this pull request as ready for review September 24, 2026 19:45
@XuPeng-SH
XuPeng-SH merged commit b0f9594 into matrixorigin:main Sep 24, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant