Skip to content

ci(deploy-test): make ipv6-outputs diagnosable via SSM, re-enable it - #492

Open
defangdevs wants to merge 1 commit into
masterfrom
fix/190-ipv6-outputs-diagnosability
Open

ci(deploy-test): make ipv6-outputs diagnosable via SSM, re-enable it#492
defangdevs wants to merge 1 commit into
masterfrom
fix/190-ipv6-outputs-diagnosability

Conversation

@defangdevs

Copy link
Copy Markdown
Owner

Addresses the diagnosability step of #190 ("Suggested next steps", item 1) —
this is not a fix for the underlying first-boot bug. Do not read a green
run here as confirmation of #190's DNS/resolver hypothesis, and do not read
a red ipv6-outputs leg on master as a regression: it is expected to fail
until the actual root cause is found and fixed.

What was undiagnosable

The ipv6-outputs leg had failed Create stack on every master run since
the 26.05 AMI map landed and was disabled outright in be6087c rather than
left burning ~39 min a run for zero diagnostic value. Three things compounded
(all described in #190):

  1. EnableSsm rode matrix.public_ipv4, so the IPv6-only leg launched with
    no instance profile and no SSM path at all.
  2. amazon-init logs journal-only — its actual nix eval/fetch error never
    reaches the serial console.
  3. The 64KB console tail is flooded by a systemd spinner before any real
    error scrolls into view, and the leg's failure mode (Create stack itself
    times out) meant the workflow's existing SSM diagnostics step never even
    ran — it sits after steps that only execute once the stack reaches
    CREATE_COMPLETE.

What this PR does

  • EnableSsm is now unconditionally true on both legs. The stock AMI's
    amazon-ssm-agent runs unconditionally pre-switch (per the template's own
    EnableSsm parameter description) — it only needed the IAM instance
    profile to register, which works even on a box whose amazon-init never
    reaches switch-to-configuration.
  • The Create stack step's retry/failure path now pulls the pre-switch
    journal over SSM (amazon-init, resolvectl, systemd-networkd/
    systemd-resolved, and a live IPv6 curl probe against
    raw.githubusercontent.com) for both the first attempt and the final
    (post-retry) attempt, right before the failing stack is torn down.
    Previously only the first attempt's serial console was captured, and a
    second/final failure crashed the step (via errexit) with no capture
    at all.
  • The existing "Diagnose auth state via SSM" step's condition is
    broadened from matrix.public_ipv4 == 'true' to any failure, covering the
    rarer case where ipv6-outputs reaches CREATE_COMPLETE but a later
    assertion (Assert amazon-init provisioned the box) fails.
  • The ipv6-outputs matrix leg is re-enabled, with
    continue-on-error: true scoped to that leg only, so its still-expected
    failure runs and captures evidence on every relevant push without
    reddening the overall Deploy test conclusion (which would mask a real
    ipv4-full regression — the exact problem be6087c disabled the leg to
    avoid).

What this PR deliberately does not do

  • No change to Nat64Config, DNS/resolver settings, or anything else on the
    live-boot-order code path the issue's hypothesis implicates. The fix stays
    unknown until real journal data says otherwise.
  • deploy-test: ipv6-outputs leg genuinely fails first boot on the 26.05 AMI — amazon-init dies before switch-to-configuration, and the leg has no way to say why #190's step 2 (confirm the AMI correlation cheaply via a
    workflow_dispatch run pinned to the prior 25.11 AMI) is not
    implemented and was not dispatched. deploy-test.yml's
    workflow_dispatch has no AMI-override input, and the AMI comes from
    aws/template.yaml's static RegionMap — exposing an override needs a new
    template parameter, not just a new workflow input. Per the task scope, I'm
    leaving that as a follow-up rather than inventing a new dispatch mechanism
    here. A future session (once this PR lands) can run the now-diagnosable
    leg on master and read the actual pre-switch journal to check the
    IPv6-only DNS/resolver hypothesis directly, which may make the AMI-pin
    comparison unnecessary.

Checks run

  • actionlint .github/workflows/deploy-test.yml — no errors (only the same
    class of pre-existing shellcheck info/warning-level style notes already
    present elsewhere in this file, e.g. SC2016 on the commands=[...] SSM
    parameter blocks, which mirror the existing "Diagnose auth state via SSM"
    step verbatim).
  • Confirmed via aws/template.yaml that UseSsm (EnableSsm == 'true')
    gates InstanceRole/InstanceProfile independently of PublicIpv4/
    UseIpv4, so enabling SSM on the IPv6-only leg is template-supported with
    no other coupling.
  • No modules/src/* or modules/agent-box.nix changes — pure workflow YAML,
    so no snapshot regeneration needed.

Refs #190

The leg has failed Create stack on every master run since the 26.05 AMI map
landed and was disabled outright (be6087c) rather than left burning ~39 min
a run with zero diagnostic value. amazon-init dies before
switch-to-configuration; it logs journal-only, get-console-output shows only
its terminal "Failed to start ..." line, and the 64KB console tail is
flooded by a systemd spinner before any real error scrolls into view. There
was also no way to reach the box at all: EnableSsm rode matrix.public_ipv4,
so the IPv6-only leg launched with no instance profile and no SSM path.

EnableSsm is now unconditionally true on both legs. The stock AMI's
amazon-ssm-agent runs unconditionally pre-switch (per the template's own
EnableSsm description), so it only needed the IAM instance profile to
register - even on a box whose amazon-init never reaches
switch-to-configuration. The "Create stack" step's retry/failure path now
pulls the pre-switch journal (amazon-init, resolvectl, systemd-networkd/
resolved, a live IPv6 curl probe) over SSM for BOTH the first and the final
attempt, before the failing stack is torn down - previously only the first
attempt's serial console was captured, and the SSM-based "Diagnose auth
state" step never ran on this leg's actual failure mode (Create stack itself
fails, so later steps never execute). That step's own condition is broadened
from ipv4-full-only to any failure, for the rarer case where ipv6-outputs
reaches CREATE_COMPLETE but a later assertion fails.

Re-enabling the leg means genuinely-expected failures again show up in CI;
continue-on-error keeps them from reddening the workflow conclusion (and
masking a real ipv4-full regression) while still running and still
capturing evidence on every push.

This does NOT fix the underlying first-boot bug - the issue's hypothesis
(an IPv6-only DNS/resolver regression in the 26.05 AMI breaking the
pre-switch module fetch) stays unverified. It makes the failure
diagnosable so that hypothesis can actually be checked against real journal
data, per issue 190's own "diagnosis first" framing.

A workflow_dispatch input to pin an older AMI (190's step 2, to confirm the
AMI correlation cheaply) is not implemented: the AMI comes from
aws/template.yaml's static RegionMap, and exposing an override needs a new
template parameter, not just a new workflow input - noted as a follow-up in
the PR instead of inventing that mechanism here.

Refs #190

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F9yBuUJC8wzeNWxPSPVwe4
@coderabbitai

coderabbitai Bot commented Sep 1, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 3 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 8 included reviews currently available.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: ef1ef422-d340-4b8e-828c-4ca18c78624b

📥 Commits

Reviewing files that changed from the base of the PR and between d763a1e and dd6b9c1.

📒 Files selected for processing (1)
  • .github/workflows/deploy-test.yml

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

1 participant