Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 17 additions & 9 deletions docs/source/device/hand_tracking/sharpa_avatar.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ Sharpa AvatarExo Glove
======================

A Linux-only plugin for integrating Sharpa AvatarExo (AE) Gloves into the
Isaac Capture framework. The gloves stream IMU and encoder data through the
official Avatar SDK over USB. The plugin maps that stream into Isaac Capture's
usual surfaces: OpenXR hand tracking for the HUMAN skeleton,
Isaac Capture framework. The gloves stream IMU and encoder data wirelessly
through a USB dongle and the official Avatar SDK. The plugin maps that stream
into Isaac Capture's usual surfaces: OpenXR hand tracking for the HUMAN skeleton,
``JointStateOutput`` tensors for the RAW and ROBOT 22-DoF joint sets, and
inbound per-finger vibration on a haptic collection.

Expand All @@ -33,7 +33,7 @@ named joint-state collections:

.. code-block:: text

Sharpa AvatarExo Gloves (USB)
Sharpa AvatarExo Gloves (wireless via USB dongle)
│
▼
avatar_hand_plugin (Avatar SDK, production package)
Expand All @@ -56,9 +56,7 @@ Prerequisites
- **Pinned production Avatar SDK** (``avatar-sdk`` 1.7.3-17) from Sharpa's
signed APT repository. ``install_avatar_sdk.sh`` retrieves it; do not substitute
``avatar-sdk-dev`` or ``avatar-sdk-beta``.
- **Sharpa AvatarExo Gloves** powered and connected over USB. Transport settings
are selected in ``<sdk-root>/share/sdk_config.json`` (default root
``/opt/avatar-sdk``).
- **Sharpa AvatarExo Gloves** powered and paired with the USB wireless dongle.
- **CloudXR runtime** on the plugin host, unless a ``TeleopSession`` example
launches it for you.
- A built Isaac Capture checkout. CMake 3.24 or newer is required.
Expand Down Expand Up @@ -175,6 +173,16 @@ The plugin reads ``HapticCommand`` on ``avatar_glove_haptic`` and drives
five finger vibration motors. The visualization example uses the same
per-finger pinch path as the generic glove haptic sample.

Health reporting
~~~~~~~~~~~~~~~~

The plugin reports the USB dongle, each physical glove, and each dataset as
separate devices. Physical-glove health combines SDK inventory, live ingress,
firmware diagnostics, and haptic errors. HUMAN, RAW, and ROBOT then report their
own freshness and fetch errors, so one healthy stream cannot hide another
stalled stream. Outputs disappear after ingress stops and are recreated when
the glove resumes; existing DeviceIO consumers bind to the new generation.

Wrist pose
----------

Expand Down Expand Up @@ -208,9 +216,9 @@ Troubleshooting
- Install the pinned production package, or point ``AVATAR_SDK_ROOT``
at a complete SDK tree. A ``-dev`` / ``-beta`` package at
``/opt/avatar-sdk`` is not the pin.
* - USB glove is not detected
* - USB dongle is not detected
- Run ``install_udev_rules.sh`` on the host, then unplug and reconnect
the AE Gloves.
the USB dongle.
* - Plugin binary is not found
- Build and install through the top-level CMake workflow. The example looks under
``install/plugins/sharpa_avatar/``.
Expand Down
3 changes: 2 additions & 1 deletion examples/sharpa_avatar/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ cmake -B build \
cmake --build build --parallel
cmake --install build
uv pip install -e ./examples/sharpa_avatar
python -m isaaccapture_examples.sharpa_avatar
python -m isaaccapture_examples.sharpa_avatar --print-health
```

Open the URL it prints (default <http://127.0.0.1:8080>). The left hand is cyan
Expand All @@ -43,6 +43,7 @@ above first. `uv pip install -e` pulls `viser`; CloudXR comes from the
```text
--no-haptic disable pinch feedback
--no-viz terminal + haptic only (no browser view)
--print-health print Sharpa device health initially and on changes
--host / --port viser bind (default 127.0.0.1:8080)
--no-launch-plugin connect to an already running plugin
--no-launch-cloudxr-runtime connect to an already running CloudXR runtime
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,7 @@

from .live_view import main

sys.exit(main())
try:
sys.exit(main())
except KeyboardInterrupt:
pass
Original file line number Diff line number Diff line change
Expand Up @@ -157,8 +157,6 @@ def _finger_chain(root: int, *joints: int) -> list[tuple[int, int]]:
HandJointIndex.WRIST,
HandJointIndex.LITTLE_METACARPAL,
HandJointIndex.LITTLE_PROXIMAL,
HandJointIndex.LITTLE_INTERMEDIATE,
HandJointIndex.LITTLE_DISTAL,
HandJointIndex.LITTLE_TIP,
)
)
Expand Down Expand Up @@ -388,6 +386,21 @@ def _fmt_optional(group, *, kind: str, hand_input_index) -> str:
return f"error:{type(exc).__name__}"


def _print_health_changes(snapshot, previous: dict[str, tuple]) -> None:
for device in snapshot.devices:
if not device.id.startswith(f"{PLUGIN_ROOT_ID}/"):
continue
current = (device.status, device.reason, device.error)
if previous.get(device.id) == current:
continue
previous[device.id] = current
detail = f": {device.error}" if device.error else ""
print(
f"[HEALTH] {device.id}: {device.status.value} "
f"({device.reason.value}){detail}"
)


def _build_haptic_pipeline(hands, mapping, sinks, hands_source_cls) -> None:
from isaaccapture.haptic_devices.glove import haptic_glove_device
from isaaccapture.retargeters.tactile_retargeters import (
Expand Down Expand Up @@ -489,6 +502,11 @@ def main() -> int:
default=DEFAULT_PRINT_HZ,
help="Terminal status rate. 0 silences prints.",
)
parser.add_argument(
"--print-health",
action="store_true",
help="Print Sharpa device health initially and whenever it changes.",
)
parser.add_argument(
"--haptic",
action=argparse.BooleanOptionalAction,
Expand Down Expand Up @@ -621,6 +639,7 @@ def main() -> int:
print_period = (1.0 / args.print_hz) if args.print_hz > 0 else None
next_print = time.monotonic()
last_offline_hint = 0.0
previous_health: dict[str, tuple] = {}
upright = not args.world_frame

print("=" * 72)
Expand All @@ -644,13 +663,21 @@ def main() -> int:
print(f" viser: http://{args.host}:{args.port} (Ctrl+C to exit)")
else:
print(" viser: off (--no-viz); terminal + haptic only. Ctrl+C to exit.")
print(f" health: {'changes' if args.print_health else 'off'}")
print("=" * 72)

with CloudXRLauncher.launch_context(args), TeleopSession(config) as session:
if args.viz:
_server, viz_left, viz_right = _start_viser(args)
while True:
result = session.step()
try:
result = session.step()
except Exception:
if args.print_health:
_print_health_changes(session.get_status(), previous_health)
raise
if args.print_health:
_print_health_changes(session.get_status(), previous_health)
n_l = n_r = 0
if viz_left is not None and viz_right is not None:
pts, mask, n_l = _hand_points(
Expand Down
3 changes: 3 additions & 0 deletions src/core/live_trackers/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ SPDX-License-Identifier: Apache-2.0

## Time and OpenXR

- Cache an XDev topology only after every ID/hand pair reaches a settled outcome. Treat
`XR_ERROR_FEATURE_UNSUPPORTED` as settled, but leave transient creation failures retryable even when the IDs do not
change.
- Store **`last_update_time_` as `int64_t`** (monotonic ns), not **`XrTime`**.
- **Once per `update` call:** `const XrTime xr_time = time_converter_.convert_monotonic_ns_to_xrtime(monotonic_time_ns);` then use **`xr_time`** for every **`xrLocate*`** / hand / body call **and** for MCAP (see below). **Do not** call **`convert_monotonic_ns_to_xrtime`** again in the MCAP block.
- **Full-body limp mode:** if the body tracker handle is null and you **return early**, **do not** compute **`xr_time`** first—only convert after you know you will call OpenXR.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,7 @@ class SchemaTrackerBase
private:
void initialize_tensor_data_functions();
void create_tensor_list();
void clear_collection() noexcept;
bool ensure_collection();
bool read_next_sample(SampleResult& out);

Expand All @@ -137,6 +138,7 @@ class SchemaTrackerBase
PFN_xrDestroyTensorListNV m_destroy_list_fn{ nullptr };

std::optional<uint32_t> m_target_collection_index;
std::optional<XrTensorCollectionIDNV> m_target_collection_id;
uint32_t m_sample_batch_stride{ 0 };
uint32_t m_sample_size{ 0 };

Expand Down
Loading