Repository navigation
CI: fix ASan shard getting killed - #9615
Open
daywalker90 wants to merge 4 commits into
Open
daywalker90 wants to merge 4 commits into
daywalker90 wants to merge 4 commits into
Conversation
daywalker90
force-pushed
the
stream-mem-usage
branch
from
October 9, 2026 11:12
c6e0e47 to
66b369b
Compare
daywalker90
marked this pull request as draft
October 9, 2026 14:53
daywalker90
force-pushed
the
stream-mem-usage
branch
from
October 12, 2026 15:46
66b369b to
74e1858
Compare
daywalker90
changed the title
daywalker90
force-pushed
the
stream-mem-usage
branch
from
October 12, 2026 17:10
74e1858 to
7554971
Compare
daywalker90
marked this pull request as ready for review
October 12, 2026 17:12
Andezion
previously approved these changes
Oct 15, 2026
Andezion
left a comment
Andezion
left a comment
Collaborator
There was a problem hiding this comment.
i dont understand how did you choose the 6 heavy tests? Did you measure them with the new logger, or with PSS data? Could other tests (for example in test_plugin.py) also be close to the limit?
| - name: Report OOM kills | ||
| if: failure() | ||
| run: | | ||
| echo "=== dmesg (OOM) ==="; sudo dmesg | grep -i -E 'oom|out of memory|killed process' || echo "no OOM lines in dmesg" |
Collaborator
There was a problem hiding this comment.
the dmesg step is removed, but it still helps when the VM survives?
Collaborator
Author
There was a problem hiding this comment.
It never survives if it goes OOM, and the dmesg step was intended to for that OOM case.
Collaborator
Author
I measured them all locally, took just 5 hours :). I chose these based on their memory usage and the VM's 16GB limit so pytest xdist will not group them in a way that it exceeds 16GB. The ones that went over 16GB by themselves had to be scaled down. |
`if: failure()` does not trigger when the VM gets killed, neither would `if: always()`. Instead we print used/available memory and the top five processes by total RSS every 15 seconds during the test step, so the readings before a kill survive in the job log. Per-process RSS alone was misleading since many lightningd processes each show small RSS. Changelog-None
test_parallel_opens starts about 600 channeld processes at full size. Under ASan that peaks near 33 GB, more than a 16 GB CI runner has, so the sanitizer shard is killed. Use 2 pairs x 10 opens when ASAN=1. The code path is unchanged and the full size still runs everywhere else. test_funding_cancel_race peaks near 15 GB PSS under ASan with 20 nodes, close to the 16 GB limit. Use 10 nodes when ASAN=1 (peak about 8 GB). The race is still exercised against each node.
Under ASan a handful of tests each need several GB and some reach
15-20 GB. When they land in the same shard and run together, the
shard is killed by the 16 GB runner. Mark them xdist_group('heavy')
and run the ASan/UBSan job with --dist loadgroup so they share one
worker and never overlap. Other jobs do not use loadgroup, so the
marker has no effect there.
The memory logging loop was duplicated in the valgrind and sanitizer jobs. Move it to .github/scripts/mem-monitor.sh so each job starts it with one line. Add ASAN to pyln.testing.utils next to VALGRIND, and use it in test_connection.py in place of reading the variable directly. Changelog-None Signed-off-by: daywalker90 <daywalker990@gmx.de>
daywalker90
force-pushed
the
stream-mem-usage
branch
from
October 16, 2026 11:46
7554971 to
d71bab0
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
add logging of memory usage instead of the dmesg step which would never trigger on a VM OOM kill. Also identified all tests using alot of memory during ASan and scaled them down. Also put them in the same xdist group so they run in serial.
Changelog-None