Skip to content

Enable G1 and GR1T2 tasks with kitless XR - #8789

Open
hougantc-nvda wants to merge 7 commits into
isaac-sim:mainfrom
hougantc-nvda:hougantc/kitless-g1-gr1t2
Open

hougantc-nvda wants to merge 7 commits into
isaac-sim:mainfrom
hougantc-nvda:hougantc/kitless-g1-gr1t2

Conversation

@hougantc-nvda

@hougantc-nvda hougantc-nvda commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Description

Review stack · 3/5: #8787 → #8788 → #8789 → #8790 → #8791

Review only this PR’s changes. The PRs target main; the comparison isolates this layer from its predecessors.

Enable the registered GR1T2, G1 locomanipulation and fixed-base G1 tasks to run with kitless Newton or CPU OvPhysX. Explicit physics selectors choose matching task presets and USD variants; omitting the selector preserves Kit defaults.

  • Task ownership: robot mappings, action conditioning, model assets and backend tuning stay in isaaclab_tasks. Teleop consumes those configurations through the existing adapter.
  • Joint ordering: preserve the configured Pink joint order so hand targets reach the intended joints.
  • Shared assets: extract the steering-wheel helper used by GR1T2 and G1, retaining graspable rim/spokes and valid collision shapes without a cross-task import.

The existing G1 head camera remains available at this PR boundary; wrist cameras and presentation changes belong to #8790.

Registered humanoid tasks prepare their selected backend before scene construction through both gym.make() and instantiate(cfg). Repeated preparation preserves caller overrides and copied settings; changing backends after preparation requires a fresh configuration. Task construction tests remain independent of optional teleop imports; the XR integration harness stays in isaaclab_teleop.

See registered tasks and physics selectors.

Dependencies: public isaaccapture==1.7.93a1 from NVIDIA’s Python index, using the optional XR selection documented in #8788. The stack includes upstream's merged provider upgrade #8858; the installation recipe reuses its OVRTX, OVStage, OvPhysX and schema declarations. Project-wide Kit/provider defaults are unchanged.

Backend preparation preserves required physics/action setup when optional teleoperation configuration is absent. Input bounds are named with their units and purpose; their values and physics tuning are unchanged.

Type of change

  • New feature
  • Bug fix
  • Documentation update

GR1T2 presets use the supported joint_effort_limit field with the task-specific tuned values.

Testing

Focused follow-up validation at dd375570e1bf17d076b18bee47835d77a6647e7e: 154 passed and 3 cross-device cases skipped across the renderer contract, material synchronization, and affected task suites. The task tests exercise registered gym.make() and instantiate(cfg) through backend preparation, stopping before native simulation construction; they also cover copied settings, serialization and optional dependencies. Six additional probes of the public Capture 1.7.93a1 admission contract and Lab publication/anchor code passed with native I/O substituted. Formatting/lint and changelog checks pass at the affected PR boundaries. This follow-up does not add IWER, headset, performance or full Kit qualification.

Rebased onto upstream main 90a16f06685dbf7948262eafa628c470b4cdc78c, including the merged provider upgrade #8858 and Newton hard-reset changes.

Combined-stack validation at c2d0ebad5b3eb9a5f5a6bd1263961162c90fec3a:

  • 573 passed, 5 failed, 12 skipped across the selected affected suites. All five failures are native Kit startup smoke tests that could not locate an installed Isaac Sim distribution.
  • Native renderer attachment/update/normal retirement passed for same-thread and transferred ownership. Seven focused Newton graph capture/replay, concurrent-upload and hard-reset tests passed.
  • Regression cases distinguish capability validation, failed scene sealing and retirement, Kit anchor composition, automatic panel layout preservation, and task preparation without optional teleop configuration.
  • Cross-device renderer/publication cases were skipped; these results do not establish current split-GPU transfer or panel-output acceptance.
  • The public Capture 1.7.93a1 selection resolves from the declared source metadata. Locked synchronization and the installed dependency check passed.
  • Formatting checks, Sphinx's warning-as-error build, and generated installation-page readback passed. The changelog check passed separately under Python 3.12 and at each of the five sequential PR boundaries.

This is combined-stack validation, not a full-repository or independent runtime qualification of each PR. IWER scene/control/mirror and performance comparison were deferred. Earlier headset/IWER acceptance remains a preservation baseline; it is not transferred to these heads.

Native Kit XR/Stop-AR, current split-GPU/PiP output and performance remain unverified. Cold renderer attachment, intermittent GR1T2 first-Play recovery and SO101 cube retention remain open. Failure-injection tests establish the exercised ownership/error policy, not behavior during an actual driver fault. Hosted-CI and arm64 results are not claimed.

Checklist

  • I have read and understood the contribution guidelines
  • I have run the repository formatting checks and the separate Python 3.12 changelog check
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have validated the changed behavior with focused coverage and applied the test-audit criteria to any test changes
  • I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump package versions — CI handles that)
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

The warnings checkbox remains open: native-runtime warnings and the separate Kit dependency selection are not fully qualified by this run.

@hougantc-nvda

Copy link
Copy Markdown
Contributor Author

@greptile Please review the latest changes in this PR.

@greptile-apps

greptile-apps Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 4/5

[High impact] The PR reaches registered tasks, shared rendering, simulation setup, and live and replay teleoperation.

Update the failed-seal test renderer before merging; both cases now stop before reaching their checks.

Findings

  1. P1 Publication tests stop early ▶

Summary

Enables G1 and GR1T2 tasks with kitless physics while retaining Kit defaults.

  • G1 and GR1T2 tasks use selected physics backends with tuned robot controls.
  • An external XR session can borrow Isaac Lab’s renderer and scene through a lease.
  • Live input and MCAP replay can use OVXR with the simulation’s scene.
  • Kitless runs and headless Kit XR keep their runtime ownership with the selected application.
  • The environment browser keeps its existing behavior as its page updates.

The supplied previous threads are unnumbered. The scene-check, live-reset and standalone-startup fixes are present. hougantc-nvda disputed the replay-reset finding because the supported provider admits forced requests synchronously unless shutdown is latched; no new evidence contradicts that explanation.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Resolved task configuration] --> B[HumanoidTeleopEnv]
    B --> C{Selected physics}
    C -->|Kit| F[Base environment construction]
    C -->|OvPhysX or Newton| D[Prepare task backend]
    D --> E[Check prior preparation and bind spawn callback]
    E --> F
    F --> G[Create scene and action managers]
Loading

Reviews (4) · Last reviewed commit: "Prepare registered humanoid tasks before..." · Reviewed by Greptile

Comment thread source/isaaclab_teleop/isaaclab_teleop/ov_libraries_xr.py Outdated
Comment thread source/isaaclab_teleop/isaaclab_teleop/entrypoints/teleop_se3_agent.py Outdated
@hougantc-nvda
hougantc-nvda force-pushed the hougantc/kitless-g1-gr1t2 branch from 4d9027a to b1de1f7 Compare October 29, 2026 18:18
@hougantc-nvda

Copy link
Copy Markdown
Contributor Author

@greptile Please review the current head (b1de1f7) again. Recheck the fixes and remaining issues against the published code, including the actual interfaces and callers at this PR boundary.

@hougantc-nvda
hougantc-nvda force-pushed the hougantc/kitless-g1-gr1t2 branch 2 times, most recently from a0c2c77 to 694150b Compare October 29, 2026 23:21
@hougantc-nvda
hougantc-nvda force-pushed the hougantc/kitless-g1-gr1t2 branch from 694150b to d337b9b Compare October 30, 2026 01:00
@hougantc-nvda

Copy link
Copy Markdown
Contributor Author

@greptile please review this PR again at the latest head (d337b9b0cb91106142de0cbdd01256058f33e10f), including the changes since your previous review.

Comment thread source/isaaclab_tasks/isaaclab_tasks/contrib/teleop_physics_cfg.py
Comment on lines +360 to +362
writer_ref = self._renderer._visual_material_writer_ref
writer = writer_ref() if writer_ref is not None else None
return self._last_step_id != physics_step_id or (writer is not None and writer.has_pending_updates)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Publication tests stop early

Both cases of test_host_reconciles_failed_publication_without_reusing_generation now fail before reaching their failed-seal checks. Their SimpleNamespace renderer has neither _visual_material_writer_ref, which this code reads, nor _publish_visual_materials, which the scene write now calls. The first publication raises AttributeError instead of testing the intended failure path.

Update that test renderer to supply both members.

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

All reviewed code categories look good.

  • PR description: The title and body understate the renderer, replay, SO101, launcher, and visualizer changes; summarize those areas and report validation performed on the final head.
  • Design and architecture: Architecture looks good.
  • Scope and over-engineering: No over-engineering issues.
  • API: API changes look good.
  • Compatibility and deprecation: No breaking changes identified.
  • Implementation: Implementation looks good.
  • Style consistency: Style consistency looks good.
  • Test quality: Test quality looks good.

No blocking issues. No inline findings.

The full PR diff was reviewed; some supplemental surrounding file context was omitted.

Automated review; human maintainers own approval decisions.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation infrastructure isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant