Skip to content

fix(net): store the checksum an incremental update computes - #1733

Closed
daniel-noland wants to merge 8 commits into
pr/daniel-noland/debug-imagesfrom
pr/daniel-noland/fuzz-net-checksum
Closed

fix(net): store the checksum an incremental update computes#1733
daniel-noland wants to merge 8 commits into
pr/daniel-noland/debug-imagesfrom
pr/daniel-noland/fuzz-net-checksum

Conversation

@daniel-noland

Copy link
Copy Markdown
Collaborator

First of five, stacked on #1724. Extracts the finished fuzzing campaign from
pr/daniel-noland/icmp-flow-lock, which was never opened as a PR and had
drifted well behind main.

This one is a production bug, on its own so it can be taken without the rest.

Checksum::increment_update_checksum works out the new checksum and hands it
back; it does not store it, despite taking &mut self.
EmbeddedTransport::update_checksum called it and dropped the answer, under a
comment about ignoring errors on a truncated header — but there is no error to
ignore, the return value is the checksum. So no quoted transport header has
ever had its checksum updated.

NAT rewrites the ports of the packet quoted inside an ICMP error and calls this
to keep the checksum in step. Receivers that validate the quoted checksum will
reject the error; that is the path PMTUD depends on. Plenty of stacks do not
validate it, which is likely why nothing has complained.

update_checksum_for_address is new, and folds a change of one of the quoted
packet's addresses in — TCP, UDP and ICMPv6 checksum over a pseudo-header built
from the addresses, so rewriting one leaves the quoted checksum describing an
address that is no longer there. ICMPv4 has no pseudo-header and is left alone.

Verified locally rather than in CI, given the outage: dataplane-net 473/473,
fmt --check and clippy -D warnings clean. Reverting the fix fails exactly
a_v4_address_change_matches_a_fresh_tcp_checksum and its v6/UDP twin.

🤖 Generated with Claude Code

@daniel-noland daniel-noland added the dont-merge Do not merge this Pull Request label Aug 17, 2026
@coderabbitai

coderabbitai Bot commented Aug 17, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 3d223ecc-c50f-44d0-b3e5-dba18ffead0c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Aug 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.20879% with 8 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
net/src/headers/embedded.rs 91.20% 6 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/debug-images branch from 72c2121 to 87a5d73 Compare August 18, 2026 20:23
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/fuzz-net-checksum branch from 56e8187 to 61049ae Compare August 18, 2026 20:23
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/debug-images branch from 87a5d73 to b390fbe Compare August 18, 2026 20:31
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/fuzz-net-checksum branch from 61049ae to dd6e234 Compare August 18, 2026 20:31
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/debug-images branch from b390fbe to 52fc0c9 Compare August 18, 2026 20:38
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/fuzz-net-checksum branch from dd6e234 to 1ba8747 Compare August 18, 2026 20:38
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/debug-images branch from 52fc0c9 to a6d9026 Compare August 18, 2026 23:17
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/fuzz-net-checksum branch from 1ba8747 to 94cdc87 Compare August 18, 2026 23:17
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/debug-images branch from a6d9026 to 1c9f520 Compare August 19, 2026 06:33
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/fuzz-net-checksum branch 2 times, most recently from a9d9273 to 3268b44 Compare August 20, 2026 01:58
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/debug-images branch 2 times, most recently from 47a4260 to 5499151 Compare August 20, 2026 02:32
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/fuzz-net-checksum branch from 3268b44 to d6c577e Compare August 20, 2026 02:32
daniel-noland and others added 8 commits August 21, 2026 09:13
Debug binaries retained the complete Rust toolchain through their
standard-library source paths, adding roughly 2.4 GB to the closure.
They also carried a sizable DWARF index that neither packaged debugger
consumes.

Point those paths at the much smaller rust-src component, omit unused
prebuilt documentation, and remove .debug_names. Source browsing and
symbols remain available while the resulting diagnostic images become
practical to store and transfer.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
A core collected from the lab is useful only with the exact unstripped
binaries and sources that produced it. A general debugging toolbox
cannot reconstruct that relationship after the release has moved on.

Provide a purpose-built gdb image alongside each build and teach it
Rust's standard-library types without retaining rustc. This keeps
post-mortem debugging reproducible while avoiding unrelated
live-debugging tools.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Post-mortem inspection and live debugging need different tools. The core
viewer cannot offer an editor-driven session, while bugstalker
understands Rust layouts and can expose the running dataplane through
the Debug Adapter Protocol.

Track bugstalker upstream for its current remote DAP support and package
it separately with the matching binaries and sources. Keeping the image
single-purpose avoids making every diagnostic artifact carry every
debugger.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Some failures need a record of the dataplane's kernel interactions
rather than an interactive debugger. A small, repeatable tracing
environment is easier to deploy and feed into existing log analysis than
a general-purpose toolbox.

Package lurk around the matching release binaries and follow the worker
threads where the dataplane does its work. Because syscall tracing needs
no symbols, this image can stay much smaller than the debugger images.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
The diagnostic images are useful only when they match the build being
investigated, but building roughly 850 MB of extra images for every pull
request would undermine the runner-load reduction this CI rework is
meant to achieve.

Build them automatically for pushes, the merge queue, and manual runs,
with an explicit label available for debugging a pull request. Publish
all three beside tagged releases so the matching tools remain available
when a deployed build needs investigation.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
All three images built green while two of them could not do what the
README documented.  Building proves an image links; it says nothing
about whether its entrypoint runs.

`smoke-container` runs each one the way the README tells a user to:

  - the tracer under a plain `docker run`, with no seccomp relaxation,
    and requires an `execve` in the trace.  Its failure mode is the
    reason this exists: lurk emitted eight well-formed JSON lines
    recording its own child failing to start, then exited 0, which the
    `jq -R 'fromjson? // empty'` filter we document accepts without
    complaint.  Both guards fire against the pre-fix image.
  - the core viewer through its own entrypoint rather than by invoking
    gdb directly, since the `--directory` and `source` flags that
    register the printers live in that entrypoint.  Checks that the
    printer set is registered, not merely that gdb started.
  - the debugger only for coming up and listening.  Driving a real DAP
    session from CI means carrying a protocol client in-tree, and the
    contract it would pin is exercised better by pointing an editor at
    the image.  It also would not have caught the defect on that image,
    which was in the documentation rather than the runtime.

The trace goes to a file rather than a shell variable: at a few
megabytes it overruns the here-string limit, and every grep against it
then fails with E2BIG, which reads exactly like a failed trace.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
The published images debug what CI built.  Debugging what you are
building meant either rebuilding an image by hand or falling back to a
system gdb, which is exactly the case where symbols do not line up.

`just debug <tool> <target>` builds the image that carries the tool at
the current profile, platform, instrumentation, and sanitizer, and runs
the target inside it:

  - `lurk` traces syscalls and streams JSON until the program exits.
  - `gdb` runs gdbserver and waits, printing the `target remote` line.
  - `bugstalker` waits for a DAP client, printing the `program` and
    `args` for the launch request -- in remote-DAP mode it takes those
    from the client rather than from its own command line.

`target` is either one of the binaries the images already carry, in
which case nothing needs mounting, or a nextest filter.  For a test the
archive is built and unpacked, and that binary was built outside the
image, so the store comes along read-only and it runs with its package
directory as the working directory the way nextest runs it.

Naming a target exactly is the tedious part -- test paths are long and
nobody remembers them -- so leaving it out offers everything through
skim, and a filter matching several offers those.  Resolution stays
unambiguous: exactly one match runs without asking, no match is an
error, and several with no terminal to ask at is an error listing them
rather than a guess.  That last case is what keeps this safe to call
from a script.

`just debug-list` prints the same list without running anything, and
`just inspect-core` opens a core file in a gdb built from the same
settings.

Randomization stays enabled under gdbserver.  Docker's default seccomp
answers personality(ADDR_NO_RANDOMIZE) with EPERM, and while gdbserver
treats that as non-fatal -- unlike lurk, which is why lurk is patched --
it otherwise opens with a warning that reads like a real failure.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
`Checksum::increment_update_checksum` works out the new checksum and
hands it back. It does not store it, despite taking `&mut self`.
`EmbeddedTransport::update_checksum` called it and dropped the answer on
the floor, under a comment about ignoring errors on a truncated header
-- but there is no error to ignore; the return value is the checksum.

So no quoted transport header has ever had its checksum updated. NAT
rewrites the ports of the packet quoted inside an ICMP error and calls
this to keep the checksum in step, and the call has been doing nothing.
The only correct uses of the trait method in the tree are in a test,
which is presumably how the shape survived.

Setting it can fail, on a header too truncated to hold a checksum -- but
that is a header the caller could not have read a checksum out of
either, and every caller reads one first. Hence discarding that error
rather than the value, which is what the comment was reaching for.

update_checksum_for_address is new, and folds a change of one of the
quoted packet's addresses in. TCP, UDP and ICMPv6 are checksummed over a
pseudo-header built from the source and destination addresses, so
rewriting one leaves the quoted checksum describing an address that is
no longer there. ICMPv4 has no pseudo-header and is left alone.
Incremental for the same reason as the rest: a quote is usually
truncated, so there is no payload to compute over from scratch.

The tests build the one case where both routes are open -- a full header
over a known payload -- and hold the fold against a computation from
scratch, over two words for v4 and eight for v6. The checksum starts out
correct on purpose: an RFC 1624 update is exact given a correct starting
value and says nothing given a wrong one. Both fail against the
discarded update; the ICMPv4 one passes either way, which is the point
of it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
@mvachhar
mvachhar force-pushed the pr/daniel-noland/debug-images branch from 5499151 to 9f7481b Compare August 21, 2026 15:13
@mvachhar
mvachhar force-pushed the pr/daniel-noland/fuzz-net-checksum branch from d6c577e to f97fb60 Compare August 21, 2026 15:13
@daniel-noland
daniel-noland force-pushed the pr/daniel-noland/debug-images branch from fdf4a49 to 9016291 Compare August 26, 2026 02:55
@daniel-noland

Copy link
Copy Markdown
Collaborator Author

Its single commit now opens #1734; a one-commit PR with a ten-commit child was a review artifact, not a chapter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dont-merge Do not merge this Pull Request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant