Skip to content

feat(mcap): add SOMA body and hand recording, replay, and live preview - #1253

Open
nv-mhaselton wants to merge 2 commits into
NVIDIA:mainfrom
nv-mhaselton:feat/soma-mcap-workflows
Open

feat(mcap): add SOMA body and hand recording, replay, and live preview#1253
nv-mhaselton wants to merge 2 commits into
NVIDIA:mainfrom
nv-mhaselton:feat/soma-mcap-workflows

Conversation

@nv-mhaselton

@nv-mhaselton nv-mhaselton commented Oct 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

Extend the existing full-body and hand MCAP commands with selectable SOMA inputs for recording, offline replay, and live preview. Preserve the FullBodyPose and OpenXR hand defaults. SOMA body recording includes controllers only when explicitly enabled with --controllers.

  • Support both joint-rotations and joint-poses representations.
  • Record received SOMA payloads unchanged through the existing native MCAP machinery.
  • Render the native 77-joint body and 25-joint hand skeletons. Rotation inputs use the existing source-owned FK helpers; evaluated poses map directly.
  • Keep live preview separate from recording and offline replay.
  • Add regression tests and document installation, collection selection, and launch commands.

Core schemas and replay ticking remain unchanged. Left and right hand trackers advance independently, following existing replay behavior. This does not introduce timestamp synchronization, resampling, or body/hand fusion.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update

Testing

Validated live publishing, recording, and offline replay for body and hands in both representations. Repeated replay of each capture produced identical evaluated outputs.

Setup

For a fresh development environment, run from the repository root:

uv venv --python 3.11
source .venv/bin/activate

uv pip install 'cmake>=3.24,<4' pytest

uv pip install \
  -e . \
  -e ./examples/mcap_record_replay \
  -e ./examples/soma_body_publisher \
  -e ./examples/soma_hand_publisher \
  --config-settings=cmake.define.BUILD_VIZ=OFF \
  --config-settings=build.tool-args=-j4

These examples use Viser and do not require the native Televiz build.

Automated checks

uv run --no-sync python -m pytest -q \
  tests/python/examples/mcap_record_replay

make -C docs current-docs

SKIP=check-copyright-year pre-commit run --all-files

Tests cover repeated native MCAP round trips, payload and timestamp preservation, sparse joints, tracked-channel selection, EOF behavior, independent hand playback, and visualization.

Interactive validation

Follow the MCAP example README for complete publisher, live viewer, recorder, and replay commands.

For body joint-rotations:

  • Publisher: --body-representation joint-rotations
  • Live viewer, recorder, and replay: --body-schema soma --soma-body-representation joint-rotations

For hand joint-rotations:

  • Publisher: --hand-representation joint-rotations
  • Live viewer, recorder, and replay: --hand-schema soma --soma-hand-representation joint-rotations

Repeat both workflows with joint-poses instead of joint-rotations. Match the representation across all commands and use a separate recording file for each mode. Offline replay does not require a CloudXR runtime or client.

Checklist

  • I have read and understood the contribution guidelines
  • I have run the linter and formatter with SKIP=check-copyright-year pre-commit run --all-files
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix/feature works (or explained why not)
  • I have signed off all my commits (git commit -s) per the DCO

Summary by CodeRabbit

  • New Features
    • Added live preview, MCAP recording, and replay workflows for SOMA body and standalone hand tracking.
    • Supports joint-rotation and joint-pose data, with configurable collection and file selection.
    • Replay defaults to capture-rate pacing and accepts a custom playback rate; missing or exhausted data is hidden.
  • Documentation
    • Added setup, command, representation, and playback guidance for the new workflows.

@nv-mhaselton nv-mhaselton self-assigned this Oct 13, 2026
@github-actions

github-actions Bot commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

📝 Docs preview is not auto-deployed for fork PRs.

A maintainer with write access to NVIDIA/IsaacCapture can deploy a preview by
commenting /preview-docs on this PR. Once deployed, the preview
will live at:

https://nvidia.github.io/IsaacCapture/preview/pr-1253/

@coderabbitai

coderabbitai Bot commented Oct 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

No actionable comments were generated in the recent review.

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: NVIDIA/IsaacCapture/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: ae1609d3-8c98-4992-afb4-1f6be6a010e0
📥 Commits

Reviewing files that changed from the base of the PR and between 2fe1bbe and 8e73110.

📒 Files selected for processing (19)
  • docs/source/references/mcap_record_replay.rst
  • examples/mcap_record_replay/README.md
  • examples/mcap_record_replay/pyproject.toml
  • examples/mcap_record_replay/python/isaaccapture_examples/mcap_record_replay/live_soma_body.py
  • examples/mcap_record_replay/python/isaaccapture_examples/mcap_record_replay/live_soma_hand.py
  • examples/mcap_record_replay/python/isaaccapture_examples/mcap_record_replay/record_soma_body.py
  • examples/mcap_record_replay/python/isaaccapture_examples/mcap_record_replay/record_soma_hand.py
  • examples/mcap_record_replay/python/isaaccapture_examples/mcap_record_replay/replay_soma_body.py
  • examples/mcap_record_replay/python/isaaccapture_examples/mcap_record_replay/replay_soma_hand.py
  • examples/mcap_record_replay/python/isaaccapture_examples/mcap_record_replay/soma_body_common.py
  • examples/mcap_record_replay/python/isaaccapture_examples/mcap_record_replay/soma_common.py
  • examples/mcap_record_replay/python/isaaccapture_examples/mcap_record_replay/soma_hand_common.py
  • tests/python/examples/CMakeLists.txt
  • tests/python/examples/mcap_record_replay/CMakeLists.txt
  • tests/python/examples/mcap_record_replay/conftest.py
  • tests/python/examples/mcap_record_replay/pyproject.toml
  • tests/python/examples/mcap_record_replay/test_soma_hand.py
  • tests/python/examples/mcap_record_replay/test_soma_mcap.py
  • tests/python/examples/mcap_record_replay/test_soma_viewer.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

Adds SOMA body and standalone hand examples for live preview, MCAP recording, and replay using joint rotations or joint poses. Shared helpers configure trackers, pipelines, joint layouts, replay pacing, and visualization. The changes also add workflow documentation, MCAP test fixtures, tests, and CMake test registration.

Priority: Normal

Estimated code review effort: 4 (Complex) | ~50 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ReplayCLI
  participant soma_common
  participant TeleopSession
  participant SomaBodySource
  participant SomaSkeletonViz
  ReplayCLI->>soma_common: Resolve recording and inspect tracked channel
  ReplayCLI->>TeleopSession: Start replay_steps for the selected frame count
  TeleopSession->>SomaBodySource: Provide tracked sample on each step
  SomaBodySource->>SomaSkeletonViz: Provide body positions and validity
Loading
Merge Risk: Minimal · previous review 8e731

The recording-to-replay workflow and test registration have no identified blocker; the PR is mergeable after normal checks.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 3.39% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 59 functions across 13 files. (6 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely summarizes the main change: adding SOMA body and hand MCAP recording, replay, and live preview workflows.
Full details: Docstring Coverage

Explanation

Docstring coverage is 3.39% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 59 functions across 13 files. (6 skipped: 6 unsupported.)

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@nv-mhaselton
nv-mhaselton force-pushed the feat/soma-mcap-workflows branch from 8e73110 to 7d5b500 Compare October 13, 2026 20:32
@nv-mhaselton nv-mhaselton changed the title feat(soma): mcap workflows feat(mcap): add SOMA body and hand recording, replay, and live preview Oct 13, 2026
@ivany-nv ivany-nv mentioned this pull request Oct 16, 2026
Closed
@nv-mhaselton
feat(mcap): add SOMA body and hand recording, replay, and live preview
3dff98f 
Extend the existing body and hand commands with SOMA schema selection
and support for joint-rotations and joint-poses.

Preserve received FlatBuffer payloads in MCAP and render native SOMA
skeletons through the existing sources. Share recording, replay, and
visualization helpers while preserving legacy defaults and core replay
behavior. Keep controllers opt-in for SOMA body workflows.

Add round-trip, sparse-data, channel-selection, EOF, and repeatability
tests, and document installation and usage.

Signed-off-by: Michael Haselton <mhaselton@nvidia.com>
@nv-mhaselton
fix(mcap): address SOMA workflow review findings
075e054 
Reject controller collection-name collisions, keep hand-only grids fixed,
and propagate recorder startup interrupts. Add focused regression tests.

Signed-off-by: Michael Haselton <mhaselton@nvidia.com>
@nv-mhaselton
nv-mhaselton force-pushed the feat/soma-mcap-workflows branch from 7d5b500 to 075e054 Compare October 29, 2026 15:05
@github-actions

github-actions Bot commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

✅ Preview deployed: https://NVIDIA.github.io/IsaacCapture/preview/pr-1253/

Built from commit 075e054. It may take a minute for the updated preview to appear.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rwiltz rwiltz Awaiting requested review from rwiltz
@jiwenc-nv jiwenc-nv Awaiting requested review from jiwenc-nv
@sgrizan-nv sgrizan-nv Awaiting requested review from sgrizan-nv

At least 1 approving review is required to merge this pull request.

Assignees

@nv-mhaselton nv-mhaselton

Labels

NVIDIA

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nv-mhaselton @ivany-nv