Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,9 @@ project.spdx
# MuJoCo writes this to the working directory, not next to the model.
MUJOCO_LOG.TXT

# examples/mcap_record_replay writes here by default (Path.cwd() / "recordings"),
# and its README has you run from the repo root.
# examples/mcap_record_replay, examples/noitom and examples/oglo_tactile write here
# by default (Path.cwd() / "recordings"), and their READMEs have you run from the
# repo root.
/recordings/

# Runtime device files injected by MCP server infrastructure
Expand Down
56 changes: 40 additions & 16 deletions examples/noitom/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,15 @@ The Noitom SDK is not vendored in this repository. Complete the
optional plugin when you need Noitom support:

```bash
cmake -B build -DBUILD_PLUGIN_NOITOM_MOCAP=ON
cmake --build build --target python_package noitom_mocap_plugin --parallel
cmake -B build -DBUILD_PLUGIN_NOITOM_MOCAP=ON -DISAAC_TELEOP_PYTHON_VERSION=3.12
cmake --build build --target python_package python_wheel noitom_mocap_plugin --parallel
cmake --install build
uv pip install --find-links=install/wheels "isaaccapture[cloudxr]"
```

Python 3.12 matches Isaac Lab 3.0, which Example 2 needs. The wheel installs
only on the Python it was built for, so use a 3.12 venv for Example 1 too.

Run `cmake --install build` again whenever `plugin.yaml` changes so the updated
configuration is copied to `install/plugins/noitom_mocap/plugin.yaml`.

Expand All @@ -46,11 +49,12 @@ MCAP script because the recording uses the standard `full_body` channel.

![Noitom full-body recording](assets/record.gif)

Record the Noitom full-body stream:
Install the example once, then record the Noitom full-body stream:

```bash
uv run python examples/noitom/record_noitom_full_body.py \
10 examples/noitom/recordings/noitom_full_body.mcap
uv pip install -e ./examples/noitom
python -m isaaccapture_examples.noitom.record_noitom_full_body \
10 recordings/noitom_full_body.mcap
```

The resulting MCAP uses the standard `core.FullBodyPoseRecord` schema and
Expand All @@ -61,39 +65,59 @@ the standard `full_body` channel, so the generic replay script can play it back:
```bash
uv pip install -e ./examples/mcap_record_replay
python -m isaaccapture_examples.mcap_record_replay.replay_full_body \
examples/noitom/recordings/noitom_full_body.mcap
recordings/noitom_full_body.mcap
```

## Example 2: Teleop

![Noitom G1 live teleoperation](assets/teleop.gif)

Run Isaac Lab with the external task registration callback. The task launches
`noitom_mocap_plugin` through IsaacTeleop's plugin manager by default.
Run Isaac Lab 3.0.0 or later with the external task registration callback.
The task launches `noitom_mocap_plugin` through IsaacTeleop's plugin manager by
default. Below, `<repo>` is your IsaacCapture checkout and `<isaaclab>` your
Isaac Lab checkout.

The callback runs inside Isaac Lab's own interpreter, so install isaaccapture
and this example into *that* environment rather than a venv of your own. Build
with `-DISAAC_TELEOP_PYTHON_VERSION=3.12` as above: Isaac Lab 3.0 runs Python
3.12, and pip skips a wheel built for another version. Deactivate any venv
first, because `isaaclab.sh -p` uses `$VIRTUAL_ENV` when it is set. `X.Y` is
your build's version, the first two numbers of the repository's `VERSION` file.

```bash
deactivate # only if a venv is active
cd <isaaclab>
./isaaclab.sh -p -m pip install "isaaccapture[cloudxr]==X.Y+local" \
--find-links <repo>/install/wheels
./isaaclab.sh -p -m pip install -e <repo>/examples/noitom
```

This replaces the `isaacteleop` 1.4 that Isaac Lab installs with the `1!X.Y`
compatibility shim from your build, which maps `isaacteleop` to
`isaaccapture`. Isaac Lab pins `isaacteleop~=1.4.145`, so pip may warn about
that conflict.

```bash
cd ~/dependence/IsaacLab3-0
PYTHONPATH=~/IsaacTeleop/examples/noitom:$PYTHONPATH \
./isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py \
cd <isaaclab>
./isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py \
--task Isaac-PickPlace-Locomanipulation-G1-Noitom-Abs-v0 \
--visualizer kit \
--xr \
--external_callback noitom_tasks.register_tasks
--external_callback isaaccapture_examples.noitom.noitom_tasks.register_tasks
```

For advanced manual plugin control, start the plugin yourself and disable
auto-launch in the Isaac Lab terminal:

```bash
./install/plugins/noitom_mocap/noitom_mocap_plugin
<repo>/install/plugins/noitom_mocap/noitom_mocap_plugin

NOITOM_MOCAP_AUTO_LAUNCH=0 \
PYTHONPATH=~/IsaacTeleop/examples/noitom:$PYTHONPATH \
./isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py \
./isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py \
--task Isaac-PickPlace-Locomanipulation-G1-Noitom-Abs-v0 \
--visualizer kit \
--xr \
--external_callback noitom_tasks.register_tasks
--external_callback isaaccapture_examples.noitom.noitom_tasks.register_tasks
```

If you run a dedicated CloudXR runtime yourself, source its environment before
Expand Down
27 changes: 27 additions & 0 deletions examples/noitom/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
# The dist name mirrors the import path, so an installed example claims no bare
# top-level name in site-packages.
name = "isaaccapture-examples-noitom"
version = "0.0.0" # Internal example - not versioned
description = "Noitom mocap suit examples: retargeting, Isaac Lab task, MCAP recording"
requires-python = ">=3.11,<3.14"
# isaaclab and gymnasium are deliberately absent: only noitom_tasks imports them,
# and it runs inside Isaac Lab's own interpreter, which provides them.
dependencies = [
"isaaccapture[cloudxr]",
"numpy>=1.23.0",
"scipy",
]

# `only-include` + `sources`, not `packages`, and no __init__.py above
# noitom -- see the packaging rules in examples/README.md.
[tool.hatch.build.targets.wheel]
only-include = ["python/isaaccapture_examples/noitom"]
sources = ["python"]
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@

from isaaccapture.schema import BodyJoint

from noitom_retargeting import noitom_position_to_isaac
from .noitom_retargeting import noitom_position_to_isaac

# G1_29DOF_CFG spawn rot (0,0,0.7071,0.7071): pelvis +X aligns with world +Y (table forward).
G1_ROBOT_FORWARD_XY = np.array([0.0, 1.0], dtype=np.float64)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1456,7 +1456,7 @@ def _aligned_skeleton_positions(
settings: NoitomRetargetingSettings,
calib_view: NoitomCalibrationView,
) -> dict[int, np.ndarray]:
from noitom_reference_draw import (
from .noitom_reference_draw import (
ReferenceSkeletonLengths,
aligned_reference_skeleton_from_frame,
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
from gymnasium.envs.registration import registry

from isaaclab.utils.configclass import configclass
from isaaclab_tasks.manager_based.locomanipulation.pick_place.locomanipulation_g1_env_cfg import (
from isaaclab_tasks.contrib.locomanip_pick_place.locomanipulation_g1_env_cfg import (
LocomanipulationG1EnvCfg,
)
from isaaccapture.schema import (
Expand All @@ -37,13 +37,13 @@
from isaaccapture.retargeting_engine.tensor_types import DLDataType, NDArrayType
from isaaccapture.teleop_session_manager import PluginConfig

from noitom_retargeting import (
from .noitom_retargeting import (
ArmIkTargets,
NoitomG1Retargeter,
NoitomRetargetingSettings,
noitom_position_to_isaac,
)
from noitom_reference_draw import (
from .noitom_reference_draw import (
ReferenceSkeletonLengths,
aligned_reference_skeleton_from_frame,
)
Expand Down Expand Up @@ -168,12 +168,10 @@ def _noitom_settings_from_env() -> NoitomG1Settings:


def _plugin_search_paths() -> list[Path]:
base = Path(__file__).resolve().parents[2]
candidates = [
base / "plugins",
base / "install" / "plugins",
]
return [path for path in candidates if path.exists()]
# Six levels up is the repo root from a checkout or an editable install; a
# regular wheel install puts this file in site-packages and finds nothing.
plugins = Path(__file__).resolve().parents[5] / "install" / "plugins"
return [plugins] if plugins.exists() else []


def _noitom_plugin_configs(settings: NoitomG1Settings) -> list[PluginConfig]:
Expand All @@ -182,7 +180,8 @@ def _noitom_plugin_configs(settings: NoitomG1Settings) -> list[PluginConfig]:
search_paths = _plugin_search_paths()
if not search_paths:
raise RuntimeError(
"Noitom plugin directory not found. Run `cmake --install build` "
"Noitom plugin directory not found. Install this example editable "
"(`pip install -e`) from a checkout, run `cmake --install build` there, "
"or set NOITOM_MOCAP_AUTO_LAUNCH=0 for a manually started plugin."
)
return [
Expand Down Expand Up @@ -340,7 +339,7 @@ def register_tasks() -> list[str]:
id=TASK_ID,
entry_point="isaaclab.envs:ManagerBasedRLEnv",
kwargs={
"env_cfg_entry_point": ("noitom_tasks:NoitomLocomanipulationG1EnvCfg"),
"env_cfg_entry_point": f"{__name__}:NoitomLocomanipulationG1EnvCfg",
},
disable_env_checker=True,
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,10 @@


def _plugin_search_paths() -> list[Path]:
base = Path(__file__).resolve().parents[2]
candidates = [
base / "plugins",
base / "install" / "plugins",
]
return [path for path in candidates if path.exists()]
# Six levels up is the repo root from a checkout or an editable install; a
# regular wheel install puts this file in site-packages and finds nothing.
plugins = Path(__file__).resolve().parents[5] / "install" / "plugins"
return [plugins] if plugins.exists() else []


def _build_pipeline(collection_id: str, max_flatbuffer_size: int) -> OutputCombiner:
Expand Down Expand Up @@ -70,7 +68,9 @@ def _resolve_output(path_arg: str | None) -> Path:
if path_arg:
path = Path(path_arg)
else:
out_dir = Path(__file__).resolve().parent / "recordings"
# CWD-relative, matching mcap_record_replay: anchoring to __file__ would
# write inside the package, and into site-packages once installed.
out_dir = Path.cwd() / "recordings"
path = out_dir / f"noitom_full_body_{datetime.now():%Y%m%d_%H%M%S}.mcap"
path.parent.mkdir(parents=True, exist_ok=True)
return path
Expand Down Expand Up @@ -100,8 +100,9 @@ def main(argv: list[str]) -> int:
search_paths = _plugin_search_paths()
if not search_paths:
sys.exit(
"[record] error: no installed plugin directory found. "
"Run `cmake --install build` or pass `--no-plugin` for a manually "
"[record] error: no installed plugin directory found. Install "
"this example editable (`uv pip install -e ./examples/noitom`), run "
"`cmake --install build`, or pass `--no-plugin` for a manually "
"started plugin."
)
plugins.append(
Expand Down
34 changes: 19 additions & 15 deletions examples/oglo_tactile/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,34 +34,39 @@ session, records, and draws the overlay). `oglo_heatmap.py` is the renderer.
```bash
sudo apt install libdbus-1-dev # BlueZ for the BLE plugin
sudo apt install libvulkan-dev # BUILD_VIZ=ON below; also needs the CUDA toolkit
pip install pillow numpy # heatmap renderer
pip install cupy-cuda12x # headset overlay GPU upload (match your CUDA)
```
`cupy` is only needed for the in-headset overlay; recording works without it.

2. **Build** (plugin + viz + python wheel)
2. **Build and install** (plugin + viz + python wheel + this example)

```bash
cd IsaacTeleop
cd <repo> # your IsaacCapture checkout
cmake -B build -DBUILD_PLUGIN_OGLO=ON -DBUILD_VIZ=ON
cmake --build build --parallel
cmake --install build # builds the isaaccapture python wheel/package
uv pip install --find-links=install/wheels "isaaccapture[cloudxr]"
uv pip install -e ./examples/oglo_tactile
ctest --test-dir build -R oglo_packet_parser --output-on-failure # parser sanity
```
The BLE backend (BlueZ via libdbus, permissive) connects out of the box.
CuPy for the in-headset overlay comes with `isaaccapture[cloudxr]`.

To check the heatmap renderer without gloves or a headset, render a synthetic
frame to `recordings/oglo_heatmap_demo.png`:

```bash
python -m isaaccapture_examples.oglo_tactile.oglo_heatmap
```

3. **Gloves**: confirm both run schema-5 firmware, are charged, and
advertise `OGLO LEFT` / `OGLO RIGHT` (check with `bluetoothctl scan on`).

4. **CloudXR**: install per the IsaacTeleop Quick Start (`pip install 'isaaccapture[cloudxr]'`).

---

## B. Demo scenario (run in order)

**Start the CloudXR runtime** — a background service, so this shell stays free
```bash
cd ~/Documents/IsaacTeleop
cd <repo>
source scripts/setup_cloudxr_env.sh
python -m isaaccapture.cloudxr.service start --accept-eula # flag needed on first run only
# Note the printed web-client URL, e.g. https://<laptop-ip>:48322/
Expand All @@ -76,14 +81,13 @@ python -m isaaccapture.cloudxr.service start --accept-eula # flag needed on fi

**Gloves + recording + overlay**
```bash
cd ~/Documents/IsaacTeleop
cd <repo>
source scripts/setup_cloudxr_env.sh
source ~/.cloudxr/run/cloudxr.env # path printed by step above
cd examples/oglo_tactile
python oglo_teleop_record.py
python -m isaaccapture_examples.oglo_tactile.oglo_teleop_record
```
The script: connects both gloves over BLE, starts recording to
`oglo_teleop_<timestamp>.mcap`, and shows the **two-hand tactile heatmap** in the
`recordings/oglo_teleop_<timestamp>.mcap`, and shows the **two-hand tactile heatmap** in the
headset. **Keep hands relaxed for ~1 s** at start so the baseline tares.

Press a glove → the matching finger/taxel lights up (YlOrRd) on the Quest screen.
Expand All @@ -96,20 +100,20 @@ Press a glove → the matching finger/taxel lights up (YlOrRd) on the Quest scre
## C. Verify the recording

```bash
mcap info oglo_teleop_*.mcap
mcap info recordings/oglo_teleop_*.mcap
# channels: hands/{left,right}_hand, head/head, oglo_{left,right}/oglo(+_tracked)
```
All streams share `sample_time_local_common_clock`, so Quest pose and OGLO
tactile align in time.

---

## Useful flags (`oglo_teleop_record.py`)
## Useful flags (`isaaccapture_examples.oglo_tactile.oglo_teleop_record`)

| Flag | Purpose |
|------|---------|
| `--plugin-bin PATH` | Path to `oglo_tactile_plugin` (auto-detected under `build/`). |
| `--mcap PATH` | Output file (default timestamped). |
| `--mcap PATH` | Output file (default `recordings/oglo_teleop_<timestamp>.mcap`). |
| `--duration S` | Auto-stop after S seconds (0 = until Ctrl+C). |
| `--raw` | Raw-ADC heatmap (no baseline subtraction). |
| `--no-overlay` | Record only (skip headset heatmap). |
Expand Down
25 changes: 25 additions & 0 deletions examples/oglo_tactile/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
# The dist name mirrors the import path, so an installed example claims no bare
# top-level name in site-packages.
name = "isaaccapture-examples-oglo-tactile"
version = "0.0.0" # Internal example - not versioned
description = "OGLO tactile glove: heatmap renderer and teleop recording"
requires-python = ">=3.11,<3.14"
dependencies = [
"isaaccapture[cloudxr]",
"numpy>=1.23.0",
"pillow",
]

# `only-include` + `sources`, not `packages`, and no __init__.py above
# oglo_tactile -- see the packaging rules in examples/README.md.
[tool.hatch.build.targets.wheel]
only-include = ["python/isaaccapture_examples/oglo_tactile"]
sources = ["python"]
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
Loading
Loading