Skip to content

fix(qwen3_8): use cached checkpoint offline in E2E test - #1563

Open
zhenshanx-nv wants to merge 1 commit into
NVIDIA:mainfrom
zhenshanx-nv:fix-qwen3_8-offline-main
Open

zhenshanx-nv wants to merge 1 commit into
NVIDIA:mainfrom
zhenshanx-nv:fix-qwen3_8-offline-main

Conversation

@zhenshanx-nv

@zhenshanx-nv zhenshanx-nv commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Background

qwen3_8's premerge E2E test fails under Community GPU CI's offline mode. _checkpoint() in families/qwen3_8/tests/test_e2e.py calls snapshot_download() without local_files_only=True. Once the family container switches to HF_HUB_OFFLINE=1 (after the trusted host coordinator has already staged the checkpoint locally), the call still tries to list the repo tree via the Hugging Face Hub API first, which raises huggingface_hub.errors.OfflineModeIsEnabled — even though the checkpoint is already present in the local cache. Every other family's equivalent checkpoint helper (e.g. families/bert/tests/test_e2e.py, families/sam3/tests/test_e2e.py) already passes local_files_only=True; qwen3_8 was the only one missing it.

Exit Criteria

  • qwen3_8's E2E checkpoint load succeeds from the already-staged local cache under HF_HUB_OFFLINE=1, with no network call.
  • No change to which checkpoint/revision is loaded, to build output, or to any other family.

Implementation

  • Add local_files_only=True to the snapshot_download() call in _checkpoint() (families/qwen3_8/tests/test_e2e.py), matching the existing pattern already used by families/bert/tests/test_e2e.py and families/sam3/tests/test_e2e.py.
  • No public API, ABI, bundle/artifact, or dependency changes.

Change categories

  • Model or runtime behavior
  • Public API
  • ABI
  • Bundle or artifact format
  • Dependencies
  • Documentation only
  • CI or developer tooling

Validation

Commands and Results

I don't have a local dev environment with the pinned dependencies/CUDA toolchain in this session to run pytest/ruff directly, so validation here is based on direct inspection of live Community GPU CI runs rather than a local test invocation:

  • Reproduced the failure on a ci/developer-dispatched Dev Community CI run: traceback showed huggingface_hub.errors.OfflineModeIsEnabled raised from families/qwen3_8/tests/test_e2e.py:110 inside _checkpoint().
  • This exact one-line change was already merged to ci/developer as fix(qwen3_8): use cached checkpoint offline in E2E test #1407 and matches the long-standing working pattern already used by bert/sam3 on main.
  • Not yet re-verified end-to-end against main specifically, since family test files are sourced from each PR's own merge commit against main, not from ci/developer — this PR is what makes that verification possible.

Hardware, Environment, and Revisions

  • Repository head: e5475571707d1fb4e7f9f5805fcb82794f308c5
  • Model: Qwen/Qwen3.8-27B at the revision pinned in families/qwen3_8/tests/manifests
  • Observed on Community GPU CI (Brev-provisioned L40-class GPU instances)

Not Run / Remaining Gaps

  • Have not re-run the qwen3_8 premerge E2E case against this exact change once merged to main; recommend validating via a fresh Community GPU CI dispatch afterward.
  • No new tests added — this is a one-line parameter fix matching an already-established pattern, not new behavior.

Contributor Self-Review

  • I have completed a self-review of this change.

Notes For Future Readers

  • This is a 1:1 duplicate of the change already merged to ci/developer as fix(qwen3_8): use cached checkpoint offline in E2E test #1407. Family-owned test files (families/*/tests/test_e2e.py) are checked out from each PR's own main-based merge commit during Community GPU CI, not from ci/developer — so a fix like this only takes effect for real PRs once it lands on main directly.
  • Related: the qwen family hit the identical underlying bug, but main already carries a different fix for it (local_files_only=constants.HF_HUB_OFFLINE, from feat(build): add optional native Edge-LLM SDK #1305) — that one does not need porting here.

Risk level

  • Low
  • Medium
  • High

Risk rationale: single-parameter addition to a test-only helper function, matching an existing pattern already used elsewhere in the codebase; no production runtime or API change.

Signed-off-by: Zhenshan Xie <zhenshanx@nvidia.com>
@coderabbitai

coderabbitai Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: NVIDIA/TensorRT-Model-Connect/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 20279597-01c2-46ba-a2c1-8095b963b323

📥 Commits

Reviewing files that changed from the base of the PR and between 79f2292 and e547557.

📒 Files selected for processing (1)
  • families/qwen3_8/tests/test_e2e.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.


📝 Summary

Summary

families/qwen3_8/tests/test_e2e.py now passes local_files_only=True to snapshot_download in _checkpoint(). The E2E test therefore resolves the checkpoint from the local Hugging Face cache and does not request a Hub download. This aligns with the checkpoint lookup pattern in the BERT and SAM3 family tests.

Architecture impact

  • Ownership: The change is limited to the Qwen3_8 family-owned E2E test.
  • Shared surfaces: No shared implementation or public API changes.
  • Dependency direction: No new dependency direction. The family test continues to call huggingface_hub.snapshot_download.
  • Affected consumers: The change affects Qwen3_8 E2E runs that use _checkpoint(). No other family consumers were identified.
  • Blast-radius question: The available evidence does not confirm that Community GPU CI stages the exact checkpoint revision required by these tests.
  • Review outcome: HUMAN REVIEW REQUIRED. Confirm checkpoint staging and the offline CI run. Test results were not provided.

Walkthrough

The Qwen3-8 end-to-end test helper now requests checkpoint snapshots with local_files_only=True.

Changes

Qwen3-8 checkpoint resolution

Layer / File(s) Summary
Use local checkpoint snapshots
families/qwen3_8/tests/test_e2e.py
_checkpoint passes local_files_only=True to snapshot_download.

Priority: ⬇️ Low

Estimated code review effort: 1 (Trivial) | ~3 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to e5475

Qwen3.8 offline tests now request the locally staged checkpoint, matching the repository-controlled CI setup. No concrete merge blocker is established.

🚥 Pre-merge checks | ✅ 8 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (8 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Family Ownership Boundary ✅ Passed PASS. The only changed line is local_files_only=True at families/qwen3_8/tests/test_e2e.py:107 in the qwen3_8-owned _checkpoint() call. It uses the shared huggingface_hub.snapshot_download API…
Shared Semantic Neutrality ✅ Passed PASS — The authoritative diff changes only families/qwen3_8/tests/test_e2e.py, which is an excluded family-owned E2E path. The one-line change adds local_files_only=True to that test's `snapshot_d…
Benchmark Validation Integrity ✅ Passed PASS: The PR changes only _checkpoint() in families/qwen3_8/tests/test_e2e.py by adding local_files_only=True to snapshot_download. The before/after validation flow is unchanged, including ref…
Shared Change Blast Radius ✅ Passed PASS. The pull request changes only families/qwen3_8/tests/test_e2e.py. It adds local_files_only=True to that family’s snapshot_download call. The change does not alter shared code, contracts, t…
Title check ✅ Passed The title clearly and concisely identifies the qwen3_8 offline checkpoint cache fix in the E2E test.
Description check ✅ Passed The description covers the background, exit criteria, implementation, change category, validation evidence, environment, remaining gaps, self-review, notes, and risk. It clearly states that local test…
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Comment @coderabbitai help to get the list of available commands.

@zhenshanx-nv zhenshanx-nv added the run-internal-ci Maintainer-approved dispatch to internal CI label Oct 1, 2026
@github-actions github-actions Bot removed the run-internal-ci Maintainer-approved dispatch to internal CI label Oct 1, 2026
@zhenshanx-nv zhenshanx-nv added the run-internal-ci Maintainer-approved dispatch to internal CI label Oct 1, 2026
@github-actions github-actions Bot removed the run-internal-ci Maintainer-approved dispatch to internal CI label Oct 1, 2026
@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown

This is an automated Internal CI result; no review from an individual maintainer is requested.

TRTMC Protected CI result
=========================

Status: FAILED
Pull request: #1563
Head commit: e5475571707d1fb4e7f9f5805fcb82794f308c55
Reason: Automated internal CI failed; details withheld

Protected failure details are not transferred to the public repository.

Open the public Source Actions run from the automated status link above.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant