[Ma77Ball]

What changes were proposed in this PR?

Provides the local deployment wiring that receives and stores telemetry. Off by default and isolated from the application services.

• Adds an OpenTelemetry Collector configuration and wires it into the single-node docker-compose stack and the Kubernetes Helm chart.
• Adds the Parca server configuration and the Parca eBPF agent for continuous profiling.
• Adds the observability backends to the single-node .env and docker-compose as opt-in compose profiles; COMPOSE_PROFILES is empty by default, so docker compose up runs only core Texera.
• Infrastructure only; the application runs unchanged whether or not these services are started.

Any related issues, documentation, or discussions?

Closes: #5369
Part of #4070. Stacked on #5376.

How was this PR tested?

• Configuration-validation specs for the collector and Parca config (ObservabilityComposeSpec, ParcaConfigSpec); 24 tests pass.
• sbt scalafmtCheckAll passes; the build runs in this PR's CI.

Was this PR authored or co-authored using generative AI tooling?

Co-authored with Claude Opus 4.8 in compliance with ASF policy.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.98%. Comparing base (569f721) to head (ef4252d).

Additional details and impacted files

@@            Coverage Diff            @@
##               main    #5377   +/-   ##
=========================================
  Coverage     91.97%   91.98%           
  Complexity     5083     5083           
=========================================
  Files          1281     1281           
  Lines         55807    55824   +17     
  Branches       7076     7077    +1     
=========================================
+ Hits          51330    51347   +17     
  Misses         2734     2734           
  Partials       1743     1743           

Flag	Coverage Δ
access-control-service	77.44% <ø> (ø)
agent-service	99.16% <ø> (ø)
amber	86.42% <ø> (ø)
computing-unit-managing-service	64.02% <100.00%> (+0.50%)	⬆️
config-service	87.50% <ø> (ø)
file-service	82.29% <ø> (ø)
frontend	96.53% <ø> (ø)
notebook-migration-service	83.77% <ø> (ø)
pyamber	98.47% <ø> (ø)
workflow-compiling-service	74.25% <ø> (ø)

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 7 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main 569f721 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	430	0.263	21,604/33,416/33,416 us	🔴 +20.8% / 🔴 +142.6%
🔴	bs=100 sw=10 sl=64	937	0.572	105,078/139,643/139,643 us	🔴 +12.6% / 🔴 +49.9%
🔴	bs=1000 sw=10 sl=64	1,124	0.686	882,567/1,059,116/1,059,116 us	🔴 +14.0% / 🔴 +20.2%

Baseline details

Latest main 569f721 from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	430 tuples/sec	482 tuples/sec	941.58 tuples/sec	-10.8%	-54.3%
bs=10 sw=10 sl=64	MB/s	0.263 MB/s	0.294 MB/s	0.575 MB/s	-10.5%	-54.2%
bs=10 sw=10 sl=64	p50	21,604 us	17,891 us	10,991 us	+20.8%	+96.6%
bs=10 sw=10 sl=64	p95	33,416 us	32,627 us	13,776 us	+2.4%	+142.6%
bs=10 sw=10 sl=64	p99	33,416 us	32,627 us	16,872 us	+2.4%	+98.1%
bs=100 sw=10 sl=64	throughput	937 tuples/sec	982 tuples/sec	1,264 tuples/sec	-4.6%	-25.9%
bs=100 sw=10 sl=64	MB/s	0.572 MB/s	0.599 MB/s	0.772 MB/s	-4.5%	-25.9%
bs=100 sw=10 sl=64	p50	105,078 us	101,606 us	86,073 us	+3.4%	+22.1%
bs=100 sw=10 sl=64	p95	139,643 us	123,964 us	93,135 us	+12.6%	+49.9%
bs=100 sw=10 sl=64	p99	139,643 us	123,964 us	111,124 us	+12.6%	+25.7%
bs=1000 sw=10 sl=64	throughput	1,124 tuples/sec	1,139 tuples/sec	1,304 tuples/sec	-1.3%	-13.8%
bs=1000 sw=10 sl=64	MB/s	0.686 MB/s	0.695 MB/s	0.796 MB/s	-1.3%	-13.8%
bs=1000 sw=10 sl=64	p50	882,567 us	880,120 us	841,986 us	+0.3%	+4.8%
bs=1000 sw=10 sl=64	p95	1,059,116 us	929,299 us	881,107 us	+14.0%	+20.2%
bs=1000 sw=10 sl=64	p99	1,059,116 us	929,299 us	918,208 us	+14.0%	+15.3%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,464.71,200,128000,430,0.263,21604.20,33416.00,33416.00
1,100,10,64,20,2134.16,2000,1280000,937,0.572,105077.89,139642.74,139642.74
2,1000,10,64,20,17798.89,20000,12800000,1124,0.686,882566.74,1059115.83,1059115.83

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• Committers with relevant context: @parshimers
  You can request their reviews formally with /request-review @parshimers.

• Contributors with relevant context: @dzueck, @mengw15, @kunwp1
  You can notify them by mentioning @dzueck, @mengw15, @kunwp1 in a comment.

[zuozhiw]

left some comments, please fix them, I clicked "approve" so that you can feel free to merge this PR after fixing these commments, because this is a env setup PR, please make sure to carefully make AI actually test these configurations and make sure they work

apart from that I have some reservations about ebpf, it's more sensitive than application level log/traces/metrics. Furthermore, ebpf is normally used for very advanced diagnostics and profiling, and seems a bit heavy for our use case. But again feel free to try it and see if we see it being useful.

Also one more missing piece of the telemetry data is host level metrics, e.g. cpu, memory being the most important ones, please check how we can collect them, e.g. I think otel has some host metrics receiver, please check that, also check if docker / k8s expose metrics and how we can get them.

Another missing piece is database related metrics. E.g. database load, it would also be nice to consider them, but maybe in future PRs.