Skip to content

[clustermgtd] Fix spurious errors logged on transient IMDS unavailability - #731

Merged
hehe7318 merged 1 commit into
aws:developfrom
hehe7318:wip/fix-imds-spurious-error-log
Aug 5, 2026
Merged

[clustermgtd] Fix spurious errors logged on transient IMDS unavailability#731
hehe7318 merged 1 commit into
aws:developfrom
hehe7318:wip/fix-imds-spurious-error-log

Conversation

@hehe7318

@hehe7318 hehe7318 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Description of changes

This is a follow-up to #704 (commit 1c00989), which added a retry to the compute fleet status retrieval in clustermgtd to "mitigate the impact of networking glitches, such as transient unavailability of IMDS". That retry did not achieve the intended effect, for two reasons:

  • its window (3 attempts, 1s apart, so ~2s overall) was shorter than the IMDS outages observed in the fleet, which are in the order of a few seconds, so the retries were exhausted before IMDS became available again;
  • every failed attempt was logged as an error by _run_command, so even the attempts recovered by a successful retry left errors in the log, describing failures that no longer had any impact on the cluster.

Either one alone is enough to fail the log assertion, so both are addressed:

  • waits are now 1s, 2s and 4s, plus a jitter of up to 1s each to avoid retrying in lockstep with the other daemons polling IMDS. The overall wait is kept well below the clustermgtd loop time, because the fleet status is retrieved synchronously at the beginning of every loop;
  • errors on the single attempts are no longer logged, by passing log_error=False. The failure is reported once by get_status, only after the retries are exhausted, so that genuinely prolonged IMDS outages remain visible.

This follows the same principle already established in d8e8b25, where a failure covered by an outer retry was downgraded and the error left to the caller that observes the exhausted retries.

Tests

  • Unit test passed
  • test_slurm_scaling which this issue was initially detected is ongoing

Checklist

  • Make sure you are pointing to the right branch.
  • If you're creating a patch for a branch other than develop add the branch name as prefix in the PR title (e.g. [release-3.6]).
  • Check all commits' messages are clear, describing what and why vs how.
  • Make sure to have added unit tests or integration tests to cover the new/modified code.
  • Check if documentation is impacted by this change.

Please review the guidelines for contributing and Pull Request Instructions.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…lity.

The retry introduced to mitigate the impact of transient IMDS unavailability
during the compute fleet status retrieval was not effective, for two reasons:
* its window (3 attempts, 1s apart) was shorter than the IMDS outages
  observed in the fleet, which are in the order of a few seconds, so the
  retries were exhausted before IMDS became available again;
* every failed attempt was logged as an error by _run_command, so even the
  attempts recovered by a successful retry left errors in the log,
  describing failures that no longer had any impact on the cluster.

Waits are now 1s, 2s and 4s, plus a jitter of up to 1s each to avoid
retrying in lockstep with the other daemons polling IMDS, and errors on the
single attempts are no longer logged: the failure is reported once by
get_status, only after the retries are exhausted, so that genuinely
prolonged IMDS outages are still visible.
@hehe7318
hehe7318 requested review from a team as code owners August 4, 2026 21:34
@hehe7318 hehe7318 changed the title [clustermgtd] Fix spurious errors logged on transient IMDS unavailability. [clustermgtd] Fix spurious errors logged on transient IMDS unavailability Aug 4, 2026
@hehe7318
hehe7318 merged commit 8778db5 into aws:develop Aug 5, 2026
19 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants