fix: track expected pipelines correctly by cgoea · Pull Request #88 · ROCm/Quartz

cgoea · 2026-08-26T10:52:23Z

Motivation

Fixes #57: EXPECTED_PIPELINE_TYPES treats pytorch/jax as always expected on every release, regardless of whether this release's own dispatch actually enabled them (build_pytorch/build_jax). Since #66's fix (this PR is stacked on #86 ), a platform correctly waits for every expected sibling pipeline to report before crystallizing, but a genuinely disabled pytorch/jax has no way to signal that, so it renders a wedging in_progress placeholder and holds the platform pending for the entire live window.

Technical Details

This PR is stacked on fix: platform status stays in_progress #86: it reuses rollup_sibling_statuses() and the per-pipeline sibling_statuses structure introduced there, so it should be merged after fix: platform status stays in_progress #86.

Test Plan

Added dedicated tests for the new behavior: disabled pytorch/jax renders skipped immediately and doesn't hold the platform; enabled pytorch/jax holds the platform pending past completed_at; resolves cleanly once it actually reports.
Added tests in therock_update_status_json_test.py for the input-capture path: orchestrator-start capture, setup-run capture, missing-input default, string-bool parsing, and new-owner reset.

Test Result

442 passed

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

HereThereBeDragons

status_json_reference.jsonc is missing the new pytorch_enabled / jax_enabled just after the orchestrator_conclusion (do we need to bump the version to v2.1?)

HereThereBeDragons · 2026-08-28T11:04:52Z

    doc.completed_at = None
    doc.orchestrator_conclusion = None
    doc.created_at = None
+    # Reset to the default-enabled state; `_apply_pipeline_enable_flags`


dont think we need the explict comment here. if you want add a sentence to the docstring

removed the inline comment on _reset_document_for_new_owner's pytorch_enabled/jax_enabled reset and moved it into docstring

HereThereBeDragons · 2026-08-28T11:10:21Z

        urls["deb"] = next(iter(workflow_run.deb_urls.values()))


+def _input_bool(inputs: dict[str, object], key: str) -> bool | None:


wondering if that shouldnt be better put in therock_types as general helper and just making it work on the string:

claude:

def _parse_bool(value: Any) -> bool | None: if isinstance(value, bool): return value if isinstance(value, str) and value.strip().lower() in ("true", "false"): return value.strip().lower() == "true" return None

and have #88 call _parse_bool(inputs.get("build_pytorch"))

HereThereBeDragons · 2026-08-28T11:11:05Z



+def _disable_pytorch_and_jax(doc: StatusDocument) -> None:
+    """Mark pytorch/jax disabled (as if `build_pytorch`/`build_jax` were


comment not needed

HereThereBeDragons · 2026-08-28T11:22:20Z

    # `summary.overall_status` so an aborted or failed release is never reported
    # as `success` just because the leaves that happened to report all passed.
    orchestrator_conclusion: Status | None = None
+    # Whether this release's own dispatch enabled the pytorch / jax pipeline


can we shorten this a bit? little bit less verbose

HereThereBeDragons · 2026-08-28T11:24:55Z

+def _pipeline_enabled(doc: StatusDocument, pipeline_type: str) -> bool:
+    """Whether `pipeline_type` is expected to run this release.
+
+    Defaults to enabled for rocm/native_packages (no flag exists) and for


dont htink this paragraph is needed

HereThereBeDragons · 2026-08-28T11:32:11Z



+def test_orchestrator_start_captures_pytorch_jax_enable_flags(tmp_path: Path) -> None:
+    # The top-level orchestrator's own dispatch inputs (build_pytorch /


not a test description

should sound better now

HereThereBeDragons · 2026-08-28T11:39:33Z

-        rpm=BuildRollup(status=unstarted_status),
-        deb=BuildRollup(status=unstarted_status),
-    )
+    gate_status = _unstarted_pipeline_status(doc, platform, empty_platform_status)


gate_status is confusing. maybe better: unstarted_status ?

used your wording :)

HereThereBeDragons · 2026-08-28T12:04:07Z

-    A cancelled/failed rocm *test* gates nothing downstream, so the children stay
-    `in_progress` and the platform stays `in_progress` (in_progress outranks
-    cancelled in the worst-of).
+    Each pipeline (rocm, pytorch, jax, native_packages) first rolls its own


by now i find the entire function rather difficult to read. i iterated with claude over an easier to read version.

here is what it came up with. let me know what you think:

Details

# --- module scope, next to _pipeline_enabled / _unstarted_pipeline_status --- def _pipeline_disabled(doc: StatusDocument, pipeline_type: str) -> bool: """Explicitly turned off for this release (pytorch/jax `build_*` = false). rocm/native_packages have no such flag, so they are never disabled.""" return pipeline_type in _PIPELINE_ENABLE_FLAGS and not _pipeline_enabled( doc, pipeline_type ) # --- the function --- def _build_platform_summary( doc: StatusDocument, platform: str ) -> tuple[PlatformSummary, bool]: """Build a platform's rollup; the bool flags whether it carries any data. rocm/pytorch run on both platforms and are always projected; an unstarted pipeline collapses to an `unstarted_status` placeholder. jax and native_packages are linux-only: pass them only when they carry data and let `for_platform` supply the linux placeholder (and the windows `None`). ... (docstring body unchanged) ... """ architectures = ( doc.linux_architectures if platform == "linux" else doc.windows_architectures ) urls = doc.linux_urls if platform == "linux" else doc.windows_urls # Roll each pipeline's own leaves to a status; `*_seen` collect the raw # leaf statuses so the sibling rollup can re-weigh them (in_progress vs # a sibling's failure) differently from the within-pipeline rollup. rocm_seen: list[Status] = [] pytorch_seen: list[Status] = [] jax_seen: list[Status] = [] native_seen: list[Status] = [] rocm = _pipeline_rollup(doc.pipelines.rocm, platform, rocm_seen) pytorch = _pipeline_rollup(doc.pipelines.pytorch, platform, pytorch_seen) jax = _pipeline_rollup(doc.pipelines.jax, platform, jax_seen) native_packages = _native_rollup(doc, platform, native_seen) # Status a still-gated, unreported pipeline inherits from the rocm build # (in_progress while live / skipped on failed build / cancelled / skipped # when no arches requested). empty_platform_status = Status.in_progress if architectures else Status.skipped unstarted_status = _unstarted_pipeline_status(doc, platform, empty_platform_status) rollups = { "rocm": (rocm, rocm_seen), "pytorch": (pytorch, pytorch_seen), "jax": (jax, jax_seen), "native_packages": (native_packages, native_seen), } # Stand-in rollup for a pipeline that reported nothing yet, so the summary # still shows an entry. Captures `doc`/`unstarted_status` (both immutable). def _placeholder(pipeline_type: str) -> PipelineRollup: status = ( Status.skipped if _pipeline_disabled(doc, pipeline_type) else unstarted_status ) return PipelineRollup(build=BuildRollup(status=status)) # Combine the expected pipelines into the platform worst-of. An unreported # pipeline contributes `unstarted_status` unless it was disabled. No # `completed_at` guard is needed: rocm/native_packages are serialized # workflows the orchestrator waits on, so a finalized release only leaves # them unreported when a failed/cancelled rocm build gated them -- and then # `unstarted_status` is skipped/cancelled (bottom precedence), so it can # never wrongly hold a finalized platform. sibling_statuses: list[Status] = [] has_data = bool(architectures) for pipeline_type in EXPECTED_PIPELINE_TYPES[platform]: rollup_obj, seen = rollups[pipeline_type] if rollup_obj is not None: has_data = True sibling_statuses.append(rollup_statuses(seen, empty_platform_status)) elif not _pipeline_disabled(doc, pipeline_type): sibling_statuses.append(unstarted_status) fields: dict[str, object] = { "status": rollup_sibling_statuses(sibling_statuses, empty_platform_status), "architectures": list(architectures), "urls": dict(urls), # rocm/pytorch always project: real rollup, or a placeholder if unreported. "rocm": rocm or _placeholder("rocm"), "pytorch": pytorch or _placeholder("pytorch"), } if platform == "linux": # jax/native_packages are linux-only. fields["jax"] = jax or _placeholder("jax") if native_packages is not None: fields["native_packages"] = native_packages else: # native has no PipelineRollup shape; hand-build its rpm/deb placeholder. native_status = ( Status.skipped if _pipeline_disabled(doc, "native_packages") else unstarted_status ) fields["native_packages"] = NativePackagesRollup( rpm=BuildRollup(status=native_status), deb=BuildRollup(status=native_status), ) return PlatformSummary.for_platform(platform, **fields), has_data

extracted _pipeline_disabled(doc, pipeline_type) as a named helper and used it in place of the inline pipeline_type in _PIPELINE_ENABLE_FLAGS. should make it a bit more clear

HereThereBeDragons

other than the comment below lgtm.

but please also discuss if we need then to bump the schema to version 2.1 or not?

HereThereBeDragons · 2026-09-02T13:42:25Z

    return True


+def _pipeline_disabled(doc: StatusDocument, pipeline_type: str) -> bool:


can we now combine _pipeline_enabled and this one here?
i would porbably go for _pipeline_enabled as positive flags are easier to understand

ok, moving to enabled

cgoea · 2026-09-03T15:59:07Z

other than the comment below lgtm.

but please also discuss if we need then to bump the schema to version 2.1 or not?

Would do it with #93

HereThereBeDragons

sadly a force push :(

please address the comment below then it should be fine

HereThereBeDragons · 2026-09-03T16:56:18Z

+def _pipeline_enabled(doc: StatusDocument, pipeline_type: str) -> bool:
+    """Whether `pipeline_type` is expected to run this release.
+
+    rocm/native_packages have no enable-flag, so they are always enabled.


this is not correct. native_packages and jax is only enabled on linux. they do not exist on windows. i would argue the check needs to be done here

i know we have
for pipeline_type in EXPECTED_PIPELINE_TYPES[platform]:
below but that you cannot expect users to correctly use. this functions needs the platform to take a reasonable decision

jax and native_packages are linux-only and never run on windows, but _pipeline_enabled didn't account for platform, relying on callers to pre-filter via EXPECTED_PIPELINE_TYPES[platform] first.

c72a8eb, fix: track expected pipelines correctly (#88), Ciprian Goea (ciprian.goea@amd.com), Mon Sep 7 11:04:03 2026 +0300

cgoea requested review from a team and HereThereBeDragons August 26, 2026 10:52

cgoea mentioned this pull request Aug 26, 2026

fix: platform status stays in_progress #86

Merged

1 task

cgoea changed the title ~~Users/cgoea/track expected pipelines~~ fix: track expected pipelines correctly Aug 26, 2026

HereThereBeDragons reviewed Aug 28, 2026

View reviewed changes

cgoea requested a review from HereThereBeDragons September 2, 2026 13:02

HereThereBeDragons reviewed Sep 2, 2026

View reviewed changes

fix: track pytorch/jax enable-flags to scope expected pipelines (#57)

3a0a139

cgoea force-pushed the users/cgoea/track_expected_pipelines branch from 8e53fde to 7bdd348 Compare September 3, 2026 15:50

address comments from reviewer

dcc2fbf

cgoea force-pushed the users/cgoea/track_expected_pipelines branch from 7bdd348 to dcc2fbf Compare September 3, 2026 15:54

cgoea requested a review from HereThereBeDragons September 3, 2026 15:55

HereThereBeDragons approved these changes Sep 3, 2026

View reviewed changes

fix: scope _pipeline_enabled to platform for jax/native_packages

6c84f8d

jax and native_packages are linux-only and never run on windows, but _pipeline_enabled didn't account for platform, relying on callers to pre-filter via EXPECTED_PIPELINE_TYPES[platform] first.

cgoea merged commit c72a8eb into develop Sep 7, 2026
2 checks passed

cgoea deleted the users/cgoea/track_expected_pipelines branch September 7, 2026 08:04

quartz-sync-github-app Bot pushed a commit that referenced this pull request Sep 7, 2026

Merge develop (c72a8eb) into main

084cfb0

c72a8eb, fix: track expected pipelines correctly (#88), Ciprian Goea (ciprian.goea@amd.com), Mon Sep 7 11:04:03 2026 +0300

HereThereBeDragons mentioned this pull request Sep 7, 2026

Restructure prerelease status.json tree: prerelease/ + group by major.minor #94

Open

		urls["deb"] = next(iter(workflow_run.deb_urls.values()))


		def _input_bool(inputs: dict[str, object], key: str) -> bool \| None:



		def _disable_pytorch_and_jax(doc: StatusDocument) -> None:
		"""Mark pytorch/jax disabled (as if `build_pytorch`/`build_jax` were



		def test_orchestrator_start_captures_pytorch_jax_enable_flags(tmp_path: Path) -> None:
		# The top-level orchestrator's own dispatch inputs (build_pytorch /

		return True


		def _pipeline_disabled(doc: StatusDocument, pipeline_type: str) -> bool:

Uh oh!

Conversation

cgoea commented Aug 26, 2026

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

Uh oh!

HereThereBeDragons left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HereThereBeDragons left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cgoea commented Sep 3, 2026

Uh oh!

HereThereBeDragons left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants