Skip to content

askrene: don't crash if a layer is removed while getroutes waits - #9633

Open
asoltys wants to merge 1 commit into
ElementsProject:masterfrom
asoltys:askrene-layer-vanished
Open

asoltys wants to merge 1 commit into
ElementsProject:masterfrom
asoltys:askrene-layer-vanished

Conversation

@asoltys

@asoltys asoltys commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

getroutes checks its layer names when the command arrives, but apply_layers() only looks them up again once a child is free. If a layer is removed in between, find_layer() returns NULL and we fall through to:

cln-askrene: plugins/askrene/askrene.c:406: apply_layers: Assertion `streq(layernames[i], "auto.sourcefree")' failed.

The plugin is not restartable mid-call, so lightningd exits with it.

We hit this in production about 16 times in 3 hours during heavy xpay load. A fee-quoting path creates a temporary layer, calls getroutes, and removes the layer in a finally when the call returns or the client-side RPC timeout fires. With all askrene-max-threads children busy, the request was still in the waiters queue when the cleanup removed its layer, and the next dequeue took the node down.

This makes apply_layers() report the missing layer, so do_getroutes() fails just that request with JSONRPC2_INVALID_PARAMS ("Layer X was removed before routing began"). The early return happens before gossmap_apply_localmods(), and the partially built localmods and layers are allocated off cmd, so there's nothing to unwind.

The new test test_layer_removed_while_waiting sets askrene-max-threads to 0 so the request is guaranteed to queue, removes the layer, then sets it back to 1 and runs another request to wake the queue. Without the fix it reproduces the assert above; with it, it passes. The full tests/test_askrene.py passes locally (38 tests).

Checklist

  • The changelog has been updated in the relevant commit(s) according to the guidelines.
  • Tests have been added or modified to reflect the changes.
  • Documentation has been reviewed and updated as needed. (no doc change: same error path as an unknown layer)
  • Related issues have been listed and linked, including any that this PR closes. (none found)
  • Important All PRs must consider how to reverse any persistent changes for tools/lightning-downgrade (no persistent changes)

🤖 Generated with Claude Code

Layer names passed to getroutes are checked when the command arrives, but
they are only looked up again in apply_layers() once a child is free (up
to askrene-max-threads at once).  If the layer was removed in between,
find_layer() returns NULL and we hit the assert that the only remaining
possibility is auto.sourcefree, killing the plugin and lightningd with it.

This is easy to hit with the common pattern of creating a temporary layer,
calling getroutes, and removing the layer when the call returns or times
out on the client side: while askrene is busy, the request is still queued
when the cleanup runs.

Fail that request with an error instead.

Changelog-Fixed: askrene: `getroutes` no longer crashes lightningd if one of its layers is removed while the request is waiting to run.
Signed-off-by: Adam Soltys <asoltys@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@asoltys
asoltys requested a review from Lagrang3 as a code owner October 16, 2026 14:19

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant