ci(deploy-test): make ipv6-outputs diagnosable via SSM, re-enable it - #492
Open
defangdevs wants to merge 1 commit into
Open
ci(deploy-test): make ipv6-outputs diagnosable via SSM, re-enable it#492defangdevs wants to merge 1 commit into
defangdevs wants to merge 1 commit into
Conversation
The leg has failed Create stack on every master run since the 26.05 AMI map landed and was disabled outright (be6087c) rather than left burning ~39 min a run with zero diagnostic value. amazon-init dies before switch-to-configuration; it logs journal-only, get-console-output shows only its terminal "Failed to start ..." line, and the 64KB console tail is flooded by a systemd spinner before any real error scrolls into view. There was also no way to reach the box at all: EnableSsm rode matrix.public_ipv4, so the IPv6-only leg launched with no instance profile and no SSM path. EnableSsm is now unconditionally true on both legs. The stock AMI's amazon-ssm-agent runs unconditionally pre-switch (per the template's own EnableSsm description), so it only needed the IAM instance profile to register - even on a box whose amazon-init never reaches switch-to-configuration. The "Create stack" step's retry/failure path now pulls the pre-switch journal (amazon-init, resolvectl, systemd-networkd/ resolved, a live IPv6 curl probe) over SSM for BOTH the first and the final attempt, before the failing stack is torn down - previously only the first attempt's serial console was captured, and the SSM-based "Diagnose auth state" step never ran on this leg's actual failure mode (Create stack itself fails, so later steps never execute). That step's own condition is broadened from ipv4-full-only to any failure, for the rarer case where ipv6-outputs reaches CREATE_COMPLETE but a later assertion fails. Re-enabling the leg means genuinely-expected failures again show up in CI; continue-on-error keeps them from reddening the workflow conclusion (and masking a real ipv4-full regression) while still running and still capturing evidence on every push. This does NOT fix the underlying first-boot bug - the issue's hypothesis (an IPv6-only DNS/resolver regression in the 26.05 AMI breaking the pre-switch module fetch) stays unverified. It makes the failure diagnosable so that hypothesis can actually be checked against real journal data, per issue 190's own "diagnosis first" framing. A workflow_dispatch input to pin an older AMI (190's step 2, to confirm the AMI correlation cheaply) is not implemented: the AMI comes from aws/template.yaml's static RegionMap, and exposing an override needs a new template parameter, not just a new workflow input - noted as a follow-up in the PR instead of inventing that mechanism here. Refs #190 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F9yBuUJC8wzeNWxPSPVwe4
|
Warning Review limit reachedNext included review available in 3 minutes. View limit detailsLimit details: You’ve used all 8 included reviews currently available. Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (1)
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses the diagnosability step of #190 ("Suggested next steps", item 1) —
this is not a fix for the underlying first-boot bug. Do not read a green
run here as confirmation of #190's DNS/resolver hypothesis, and do not read
a red
ipv6-outputsleg on master as a regression: it is expected to failuntil the actual root cause is found and fixed.
What was undiagnosable
The
ipv6-outputsleg had failedCreate stackon every master run sincethe 26.05 AMI map landed and was disabled outright in be6087c rather than
left burning ~39 min a run for zero diagnostic value. Three things compounded
(all described in #190):
EnableSsmrodematrix.public_ipv4, so the IPv6-only leg launched withno instance profile and no SSM path at all.
reaches the serial console.
error scrolls into view, and the leg's failure mode (
Create stackitselftimes out) meant the workflow's existing SSM diagnostics step never even
ran — it sits after steps that only execute once the stack reaches
CREATE_COMPLETE.What this PR does
EnableSsmis now unconditionallytrueon both legs. The stock AMI'samazon-ssm-agentruns unconditionally pre-switch (per the template's ownEnableSsmparameter description) — it only needed the IAM instanceprofile to register, which works even on a box whose amazon-init never
reaches
switch-to-configuration.Create stackstep's retry/failure path now pulls the pre-switchjournal over SSM (
amazon-init,resolvectl,systemd-networkd/systemd-resolved, and a live IPv6 curl probe againstraw.githubusercontent.com) for both the first attempt and the final(post-retry) attempt, right before the failing stack is torn down.
Previously only the first attempt's serial console was captured, and a
second/final failure crashed the step (via
errexit) with no captureat all.
broadened from
matrix.public_ipv4 == 'true'to any failure, covering therarer case where
ipv6-outputsreachesCREATE_COMPLETEbut a laterassertion (
Assert amazon-init provisioned the box) fails.ipv6-outputsmatrix leg is re-enabled, withcontinue-on-error: truescoped to that leg only, so its still-expectedfailure runs and captures evidence on every relevant push without
reddening the overall
Deploy testconclusion (which would mask a realipv4-fullregression — the exact problem be6087c disabled the leg toavoid).
What this PR deliberately does not do
Nat64Config, DNS/resolver settings, or anything else on thelive-boot-order code path the issue's hypothesis implicates. The fix stays
unknown until real journal data says otherwise.
workflow_dispatchrun pinned to the prior 25.11 AMI) is notimplemented and was not dispatched.
deploy-test.yml'sworkflow_dispatchhas no AMI-override input, and the AMI comes fromaws/template.yaml's staticRegionMap— exposing an override needs a newtemplate parameter, not just a new workflow input. Per the task scope, I'm
leaving that as a follow-up rather than inventing a new dispatch mechanism
here. A future session (once this PR lands) can run the now-diagnosable
leg on master and read the actual pre-switch journal to check the
IPv6-only DNS/resolver hypothesis directly, which may make the AMI-pin
comparison unnecessary.
Checks run
actionlint .github/workflows/deploy-test.yml— no errors (only the sameclass of pre-existing shellcheck info/warning-level style notes already
present elsewhere in this file, e.g.
SC2016on thecommands=[...]SSMparameter blocks, which mirror the existing "Diagnose auth state via SSM"
step verbatim).
aws/template.yamlthatUseSsm(EnableSsm == 'true')gates
InstanceRole/InstanceProfileindependently ofPublicIpv4/UseIpv4, so enabling SSM on the IPv6-only leg is template-supported withno other coupling.
modules/src/*ormodules/agent-box.nixchanges — pure workflow YAML,so no snapshot regeneration needed.
Refs #190