Cloud performance tests - #1
Merged
Merged
Conversation
…ecutable to vms. Need to test
…g version compatible with tigerbeetle automatically
…spot skew) Scopes the first batch of new cloud tests to hotspot skew only, 3 DB modes, and two knob variants, plus a small coordinator change to surface the dropped-requests metric that's needed to interpret the results. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The client already tracks requests dropped due to max_concurrency (client/src/workload.rs, requests_dropped counter) but the coordinator never queried it, making it impossible to tell whether a fixed_rate test fell short of target_rate because of the concurrency cap or some other bottleneck. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Surfaces the Prometheus-sourced dropped-request count (added in the previous commit) in the per-run and aggregate JSON export, so future test results can show whether max_concurrency was the binding constraint instead of silently omitting dropped requests. Also updates the cloud/remote RunResult construction in main.rs, which the same struct-literal change affects and which would otherwise fail to compile. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Six new configs (TigerBeetle, PostgreSQL standard, PostgreSQL atomic, each at max_concurrency=5000/target_rate=5000 and max_concurrency=5000/target_rate=10000) to test whether the previous 1,000 concurrency cap was constraining achieved throughput, and where the real ceiling is once it isn't. See docs/superpowers/specs/2026-07-30-cloud-knob-sweep-design.md for the full rationale. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirms max_concurrency=1000 was silently throttling TigerBeetle's throughput in the earlier hotspot tests (5,060 TPS at concurrency5k with zero drops, vs 4,461 TPS/unknown drops at baseline), and finds TigerBeetle's ceiling still hasn't been reached at 10k offered rate (9,431 TPS achieved). Also finds the counterintuitive result that raising max_concurrency made PostgreSQL worse, not better, since connection_pool_size stayed at 20 in every PostgreSQL config - the real bottleneck there is the connection pool, not the concurrency cap or target rate. Records two infrastructure issues hit along the way: TigerBeetle's account funding isn't idempotent across separate coordinator invocations against a persistent cluster (doubled every balance once), and occasional simultaneous SSH/IAP disconnects on all client nodes during unusually long drain phases (resolved by retrying). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Follow-up to the max_concurrency/target_rate sweep, which found PostgreSQL throughput got worse (not better) when max_concurrency was raised while connection_pool_size stayed fixed at 20 - pointing at the connection pool, not the concurrency cap, as PostgreSQL's real bottleneck. These 4 configs (standard + atomic executors, concurrency5k + rate10k) test pool_size=50 against the same knob values to see whether that's the real lever. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
5 client nodes x connection_pool_size=50 = 250 total client connections, which exceeded the previous max_connections=200 - causing most connections to be refused outright (68% error rate, "Failed to get connection") rather than queued, collapsing throughput instead of improving it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Tests connection_pool_size=50 (up from 20) for both PostgreSQL executors
at concurrency5k/rate10k. First attempt hit a real server-side ceiling:
max_connections=200 was exceeded by 5 clients x 50 = 250 connections,
causing 68% error rate ("Failed to get connection") instead of an
improvement - fixed by raising max_connections to 300.
With that fix, all four pool50 configs completed cleanly (0 failed, 0%
error rate), but throughput did not improve over pool=20 for either
executor - pointing at row-lock contention under hotspot skew, not
connection availability, as PostgreSQL's real bottleneck. None of the
three knobs tested across this whole sweep (max_concurrency, target_rate,
connection_pool_size) meaningfully move PostgreSQL's throughput under
hotspot skew.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Incorporates lead engineer feedback on article_v1.pdf: - Adds a Methodology section explicitly defining target_rate and max_concurrency and what each controls. - Describes all three PostgreSQL transfer strategies inline (no longer assumes the reader has read the local-benchmark article); batched mode gets a one-sentence caveat instead of a full dedicated section/chart, since it's an architectural mismatch with no path to improvement. - Replaces the original hotspot-skew numbers with the corrected concurrency5k/rate10k results from the max_concurrency/target_rate/ connection_pool_size knob sweep, which show a bigger and more accurate TigerBeetle advantage (5.8x-10.9x vs the original 4.6x) once an unmeasured concurrency cap on TigerBeetle's side was found and fixed. - Regenerated throughput/latency charts (article-assets/) for the corrected hotspot numbers; latency chart omits PostgreSQL's percentiles since they were pegged at a metrics-export histogram ceiling rather than reflecting a precise real value. Moderate-skew numbers are left as originally measured (not re-tested with the corrected knobs) and clearly labeled as such. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pushes target_rate to 20,000 (from 10,000) to keep probing TigerBeetle's ceiling under hotspot skew - rate10k achieved 9,431 TPS with ~9% of load dropped, suggesting max_concurrency=5000 was plausibly still binding at the tail. max_concurrency raised to 30,000 (6,000/client vs 4,000/client offered) for real headroom this time. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pushes target_rate to 20,000 with max_concurrency raised to 30,000 (real headroom this time, not just 1:1). Result: 20,257 TPS mean, 0 dropped transfers, 0% error rate, p99 still sub-second (1,046ms) - TigerBeetle absorbed essentially the full 2x jump in offered load with zero drops. Updates hotspot-knob-sweep-results.md with the new test's analysis and raw results, and article_v2.md to feature rate20k as the headline number (~23x PostgreSQL's best observed throughput across the whole sweep, though PostgreSQL wasn't re-tested at this specific rate since rate10k already showed it doesn't move with target_rate). Regenerated both corrected-hotspot charts to include the third data point. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pushes target_rate to 40,000 (from 20,000) to keep probing TigerBeetle's ceiling under hotspot skew - rate20k achieved 20,257 TPS with zero drops, so the ceiling is still somewhere above that. max_concurrency raised to 100,000 (20,000/client vs 8,000/client offered, 2.5x headroom) so any slowdown shows up as latency growth rather than an artificial drop from an undersized cap. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…gnal Pushes target_rate to 40,000 with max_concurrency raised to 100,000. Result: 40,388 TPS mean, 0 dropped transfers, 0% error rate - throughput again matched the full offered rate (101%). First attempt's run 2 hit a transient GCP scp/auth-backend error unrelated to load; a full retry succeeded cleanly on the first attempt. Latency growth is the interesting nuance: p50 grew 70% on this doubling (43ms -> 73ms) vs 16% on the previous one - the first real signal in this sweep that something may be starting to give, even though throughput hasn't dropped off. p999 barely moved (+2%), suggesting the far tail may be bumping against a separate, roughly fixed ceiling while the bulk of the distribution keeps climbing. Still no confirmed throughput ceiling. Updates hotspot-knob-sweep-results.md with a per-step latency-growth breakdown table and the new raw results, and article_v2.md to feature rate40k as the new headline number (~46x PostgreSQL's best observed throughput) with the latency nuance called out. Regenerated both charts with rate40k as a 4th data point. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pushes target_rate to 80,000 (from 40,000) to keep probing TigerBeetle's ceiling under hotspot skew - rate40k achieved 40,388 TPS with zero drops but showed the first real latency-growth signal (p50 +70% vs +16% the step before). max_concurrency raised to 200,000, matching rate40k's 2.5x headroom ratio. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…stake Pushes target_rate to 80,000 with max_concurrency raised to 200,000. Result: 81,171 TPS mean, 0 dropped transfers, 0% error rate - throughput still matched the offered rate (101%). But median (p50) latency exploded from 73ms to 664ms (+810%, vs +70% the step before) while p95 grew 36% - a classic saturation signature (throughput holding while queueing grows), and the strongest signal in this whole sweep that we're at or very near TigerBeetle's real practical ceiling under this hotspot workload. Stopped the sweep here rather than push to 160k. Also corrects a real mistake from the rate40k writeup: the "p999 plateaus at a separate fixed ceiling" interpretation was wrong. client/src/metrics.rs bounds the exported histogram at 1,500,000us (1.5s), and both rate40k's and rate80k's p999 (plus rate80k's p99) sit right against that boundary with near-zero run-to-run variance - the signature of a value being capped by the measurement, not a real architectural plateau. This is the same category of gap as PostgreSQL's 5-second histogram cap found earlier in this investigation. Corrected the interpretation everywhere it appears in both docs, and marked the affected chart bars with a hatch pattern. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same content as article_v2.md (all five hotspot knob-sweep results: concurrency5k through rate80k), but folds the incremental "here's a correction to what we said before" commentary from patching v2 three times into direct explanation instead - readers care about the histogram bucket-boundary caveat itself, not our own revision history getting there. Tightens the headline takeaways and results table for a cleaner final read. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
rate40k and rate80k both showed p99/p999 landing right against the 1,500,000us bucket boundary with near-zero run-to-run variance - it was unclear whether this reflected a real behavioral cap (client timeout, batching cadence) or just insufficient bucket resolution, since buckets did already exist at 2M/3M/5M above that point and weren't being used. Adding finer resolution from 1.75s up through 20s lets the next high-load test disambiguate: if real tail latency exceeds 1.5s, it'll now show up in these new buckets instead of piling up at the old boundary. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pushes target_rate to 160,000 (from 80,000) to keep probing TigerBeetle's ceiling under hotspot skew - rate80k achieved 81,171 TPS with zero drops but showed the strongest saturation signal yet (p50 +810%). max_concurrency raised to 400,000, matching the 2.5x headroom ratio used throughout this sweep. Requires the widened histogram buckets (previous commit) for a trustworthy tail-latency reading this time. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…tifact Pushes target_rate to 160,000 with max_concurrency raised to 400,000, using client/src/metrics.rs's widened histogram buckets (up to 20s) from the previous commit. Result, decisively different from every prior step: - Mean throughput 107,858 TPS - only 67% of the 160,000 offered rate, the first real throughput shortfall in the whole sweep (every earlier step achieved ~100-101% of its offered rate). - ~34% of offered load dropped (16.6-16.7M/run), vs 0-9% at every earlier step. - Real tail latency, no longer bucket-capped: p50 ~3.6s, p95 up to 6.2s, p99 up to 9.6s, p999 up to 14.2s. This is the payoff of widening the histogram: latency grew freely well past the old 1.5s boundary, conclusively confirming rate40k/rate80k's plateau there was a measurement artifact, not a real architectural cap as briefly hypothesized. rate80k's dramatic p50 growth (+810%) was a real, correctly-read warning sign of approaching saturation - it just wasn't the wall itself. The wall is between rate80k and rate160k. Getting here took 7 attempts across a mix of causes: a hard ~24.5-minute limit on the background task running the coordinator (hit 3 times, regardless of check-in frequency, always during the 3rd run's balance verification after the first two completed cleanly), a stray remote client process left running after killing a stuck attempt (blocked the next scp), a self-inflicted repeat of the known TigerBeetle account double-funding bug (from not tearing down the DB cluster between a killed attempt and its retry), and one transient IAP tunnel error. The reported numbers are a 2-of-3-run average, not a full coordinator aggregate, since every attempt that got this far hit the same ~24.5min limit on the final run - documented in full for transparency. Updates hotspot-knob-sweep-results.md and article_v3.md (article_v2.md is superseded) with the new result, corrects the earlier "separate fixed tail ceiling" framing for rate40k/rate80k's p999 into "confirmed measurement artifact," rewrites the headline takeaways to center rate160k as the real ceiling finding, and regenerates both charts (log-scale latency axis now, to fit rate160k's multi-second values alongside earlier tens-of-ms values). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.