fix: platform status stays in_progress by cgoea · Pull Request #86 · ROCm/Quartz

cgoea · 2026-08-25T14:50:20Z

Motivation

Fixes ROCm/Quartz#66: a platform's summary.<platform>.status could already read failure while other pipelines on that same platform (e.g. jax, native_packages) had not reported anything yet. The rollup precedence (failure > in_progress > cancelled > success > skipped) let one pipeline's terminal failure shortcut past sibling pipelines that were still genuinely in flight and could still pass or fail on their own, so the platform's reported verdict was premature.

Technical Details

Added rollup_sibling_statuses() in therock_status_document.py
_build_platform_summary() in therock_summary.py now rolls each pipeline up to its own single status first
Net effect: a platform only crystallizes to failure/cancelled once every expected sibling pipeline has reported a terminal status; until then it correctly stays in_progress.

Test Plan

Updated therock_summary_test.py cases that previously asserted the premature-failure behavior
Added companion tests (*_wins_once_every_sibling_is_terminal / *_drags_platform_once_siblings_are_terminal) verifying failure still correctly wins once every sibling pipeline has reported terminally, so no coverage was lost.

Test Result

pytest scripts/: 475 passed

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

…ill pending (#66)

HereThereBeDragons · 2026-08-25T14:58:31Z

from afar i thought it makes sense to combine this with #57. what do you think?

my idea would be to extract from https://github.com/ROCm/therock/blob/main/.github/workflows/multi_arch_release.yml from the inputs the following:

    build_python_packages: true
    build_pytorch: true
    build_jax: true

and based on this and which archs are set

    linux_amdgpu_families: all
    windows_amdgpu_families: all

we can then scope the expected pipelines "live". (expected is always all pipelines and all archs. its more used to disable the expectation)

cgoea · 2026-08-26T09:50:25Z

from afar i thought it makes sense to combine this with #57. what do you think?

my idea would be to extract from https://github.com/ROCm/therock/blob/main/.github/workflows/multi_arch_release.yml from the inputs the following:
    build_python_packages: true
    build_pytorch: true
    build_jax: true
and based on this and which archs are set
    linux_amdgpu_families: all
    windows_amdgpu_families: all
we can then scope the expected pipelines "live". (expected is always all pipelines and all archs. its more used to disable the expectation)

I would keep it separate, and make a follow up PR against #57. #88 stacked on top of this one

HereThereBeDragons

overall lgtm just some changes to the doc and testing

HereThereBeDragons · 2026-08-28T10:36:35Z

-    A cancelled/failed rocm *test* gates nothing downstream, so the children stay
-    `in_progress` and the platform stays `in_progress` (in_progress outranks
-    cancelled in the worst-of).
+    Each pipeline (rocm, pytorch, jax, native_packages) first rolls its own


i think this can be condensed into what the user wants to have with less of historical annecdots where it comes from. more or less the thing is: if any pipeline is still in_progress the platform status collapses to in_progress.

i wonder if we even can just extend the table above - or if this is already included in the table above and we just behaved differently

condensed it but each cover separate topics. table is specifically about an unstarted pipeline's status while the pargrapah is about a different mechanism, how already-reported sibling pipelines combine via rollup_sibling_statuses's cross-pipeline precedence.

HereThereBeDragons · 2026-08-28T10:58:50Z

+    assert doc.summary.linux.status is Status.in_progress
+
+
+def test_multiple_pipelines_failure_wins_once_every_sibling_is_terminal() -> None:


seeing here some redundancy i asked claude for suggestion of test cleanup:

Recommendation (the real fix, not just deleting one test):

Add one table-driven unit test on rollup_sibling_statuses directly — pure function, cheap, covers the full precedence (in_progress > failure > cancelled > success > skipped), the all-terminal-worst-wins case, and the empty→fallback case. That's where this rule belongs.

With that in place, the integration trio no longer needs to re-prove the precedence — drop test_multiple_pipelines_failure_wins_once_every_sibling_is_terminal outright, and let the existing test_multiple_pipelines_aggregate_into_platform_status keep owning the masking half (the live in_progress). Tests 2 and 3 stay, justified by their distinct internal paths.

Bonus: the padding in test_failure_beats_cancelled_and_success_within_one_pipeline (four success sibling leaves added just to stop the sibling rollup from masking) also becomes unnecessary noise once the precedence is unit-tested — that test can go back to being purely about within-pipeline precedence.

Net: one cheap unit test replaces the redundant integration re-proofs, and the suite gets more precise, not less covered.

done all three

HereThereBeDragons

see comment otherwise lgtm

HereThereBeDragons · 2026-09-03T14:35:22Z

    doc.upsert_leaf("linux", "", "rocm", "build", _leaf(status=Status.success))
    doc.upsert_leaf("linux", "gfx942", "rocm", "test", _leaf(status=Status.cancelled))
    doc.upsert_leaf("linux", "gfx1100", "rocm", "test", _leaf(status=Status.failure))
-    doc.upsert_leaf("linux", "", "pytorch", "build", _leaf(status=Status.success))


you probably need to add this back as soon as #88 is merged

HereThereBeDragons · 2026-09-03T14:36:06Z

please double check pr title/description - didnt look at it

35ec9d1, fix: platform status stays in_progress (#86), Ciprian Goea (ciprian.goea@amd.com), Thu Sep 3 18:01:09 2026 +0300

## Motivation Fixes #57: `EXPECTED_PIPELINE_TYPES` treats pytorch/jax as always expected on every release, regardless of whether this release's own dispatch actually enabled them (`build_pytorch`/`build_jax`). Since #66's fix (this PR is stacked on #86 ), a platform correctly waits for every *expected* sibling pipeline to report before crystallizing, but a genuinely **disabled** pytorch/jax has no way to signal that, so it renders a wedging `in_progress` placeholder and holds the platform pending for the entire live window. ## Technical Details - This PR is stacked on #86: it reuses `rollup_sibling_statuses()` and the per-pipeline `sibling_statuses` structure introduced there, so it should be merged after #86. ## Test Plan - Added dedicated tests for the new behavior: disabled pytorch/jax renders `skipped` immediately and doesn't hold the platform; enabled pytorch/jax holds the platform pending past `completed_at`; resolves cleanly once it actually reports. - Added tests in `therock_update_status_json_test.py` for the input-capture path: orchestrator-start capture, setup-run capture, missing-input default, string-bool parsing, and new-owner reset. ## Test Result 442 passed ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

fix: platform status stays in_progress while a sibling pipeline is st…

6b0cb11

…ill pending (#66)

cgoea requested a review from a team August 25, 2026 14:50

cgoea changed the base branch from main to develop August 25, 2026 14:51

cgoea mentioned this pull request Aug 26, 2026

fix: track expected pipelines correctly #88

Merged

1 task

Merge branch 'develop' into users/cgoea/platform_rollup

de53b0c

HereThereBeDragons reviewed Aug 28, 2026

View reviewed changes

address comments from reviewer

1df4993

cgoea requested a review from HereThereBeDragons September 2, 2026 14:18

HereThereBeDragons approved these changes Sep 3, 2026

View reviewed changes

cgoea merged commit 35ec9d1 into develop Sep 3, 2026
2 checks passed

cgoea deleted the users/cgoea/platform_rollup branch September 3, 2026 15:01

quartz-sync-github-app Bot pushed a commit that referenced this pull request Sep 3, 2026

Merge develop (35ec9d1) into main

c5f54a9

35ec9d1, fix: platform status stays in_progress (#86), Ciprian Goea (ciprian.goea@amd.com), Thu Sep 3 18:01:09 2026 +0300

Navigation Menu

Uh oh!

fix: platform status stays in_progress - #86

fix: platform status stays in_progress #86
cgoea merged 3 commits into
developfrom
users/cgoea/platform_rollup

cgoea commented Aug 25, 2026

Uh oh!

HereThereBeDragons commented Aug 25, 2026

Uh oh!

cgoea commented Aug 26, 2026 •

edited

Loading

Uh oh!

HereThereBeDragons left a comment

Uh oh!

HereThereBeDragons Aug 28, 2026

Uh oh!

cgoea Sep 2, 2026

Uh oh!

HereThereBeDragons Aug 28, 2026

Uh oh!

cgoea Sep 2, 2026

Uh oh!

HereThereBeDragons left a comment

Uh oh!

HereThereBeDragons Sep 3, 2026

Uh oh!

HereThereBeDragons commented Sep 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		assert doc.summary.linux.status is Status.in_progress


		def test_multiple_pipelines_failure_wins_once_every_sibling_is_terminal() -> None:

Uh oh!

Conversation

cgoea commented Aug 25, 2026

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

Uh oh!

HereThereBeDragons commented Aug 25, 2026

Uh oh!

cgoea commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HereThereBeDragons left a comment

Choose a reason for hiding this comment

Uh oh!

HereThereBeDragons Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

cgoea Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

HereThereBeDragons Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

cgoea Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

HereThereBeDragons left a comment

Choose a reason for hiding this comment

Uh oh!

HereThereBeDragons Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

HereThereBeDragons commented Sep 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cgoea commented Aug 26, 2026 •

edited

Loading