Add tactile tool

.github/actions/setup-uv/action.yml

@@ -12,4 +12,5 @@ runs:
       uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39
       with:
         # Pinning explicitly avoids setup-uv version-resolution API calls and fallback paths.
-        version: "0.10.6"
+        # Required for package-scoped dependency exclusions in pyproject.toml.
+        version: "0.12.13"

pyproject.toml

@@ -49,6 +49,13 @@ dev = [
 ]
 ui = [
     "imgui[glfw]>=2.0.0",
+    "numpy>=1.23.0",
+    # Dockable panels and Trimesh 5 compatibility require Viser >=1.1.1.
+    # https://github.com/viser-project/viser/pull/769
+    "viser>=1.1.1",
+    "yourdfpy",
+    "trimesh",
+    "rtree",
 ]
 retargeters = [
     "dex-retargeting>=0.4.6,<0.6.0",
@@ -109,6 +116,8 @@ result = "{major}.{minor}+local"
 # [build-system], not [tool.uv]).
 [tool.uv]
 package = false
+# Ignore Viser's websockets requirement while preserving CloudXR's requirement.
+exclude-dependencies = [{ package = { name = "viser" }, dependencies = ["websockets"] }]
 
 [tool.uv.sources]
 isaacteleop = { path = "src/compat" }

scripts/tactile_metadata/README.md

@@ -0,0 +1,254 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Tactile Metadata Editor
+
+Tactile Metadata Editor is a local Viser application for creating, editing, and
+previewing glove tactile metadata. It loads URDF models and
+[`TactileMetadata`](../../src/core/schema/fbs/tactile_metadata.fbs) JSON files.
+All edits remain local; live connections only receive Isaac Capture data.
+
+> [!WARNING]
+> The editor is designed for completing all work in one run. Saved metadata
+> contains only schema fields, so ending the session loses editor-specific
+> configuration such as URDF paths, alignments, and URDF joint values.
+
+**Todo**: add cross-run project saving-loading.
+
+## Set up and launch
+
+Use an existing Isaac Capture environment with Python 3.11 or newer. Install
+the shared UI requirements into that environment:
+
+```bash
+cd scripts/tactile_metadata
+python -m pip install -r ../../src/core/python/requirements-ui.txt
+python -m tactile_metadata_editor.cli
+```
+
+Open [http://127.0.0.1:8080](http://127.0.0.1:8080). To use another port:
+
+```bash
+python -m tactile_metadata_editor.cli --port 8090
+```
+
+Metadata, side, and URDF inputs are optional:
+
+```bash
+python -m tactile_metadata_editor.cli \
+  --metadata <path-to-metadata-json> \
+  --side <left/right> \
+  --urdf <path-to-urdf>
+```
+
+`--side` defaults to `left` and initializes **Hand Side**. Paths beginning with
+`~` are expanded.
+
+Offline editing uses Viser, NumPy, yourdfpy, trimesh, and rtree. Live reception
+also requires the Isaac Capture SDK, a compatible OpenXR runtime, and an
+available HMD system in the same Python environment.
+
+## Data and coordinates
+
+The editor maintains one metadata document and file path. **Hand Side** selects
+the default poses used by **New** and the hand stream used by **Connect Live**.
+Changing it disconnects Live and clears received metadata without switching,
+mirroring, or replacing the current document or its Saved and Preview datasets.
+General uses a separate right-docked menu above the Editor/Preview menu. The
+Selected Object panel is always present as a floating window on the left.
+The **Mode** section uses bold, mutually exclusive **Edit** and **Preview**
+checkbox labels; one mode is always selected.
+Focused numeric fields preserve partial decimal input such as `1.` until the
+next digit is entered or the field loses focus.
+
+Three pose datasets are displayed:
+
+- **Editing** contains editable segment and taxel binding poses plus Edit-mode
+  URDF joint values.
+- **Saved** is the last loaded or saved metadata snapshot. Enable **Show Saved
+  Binding Poses** to render it at opacity 0.3.
+- **Preview** contains current segment poses and Preview-mode URDF joint values.
+  Preview taxel poses are derived from the selected binding metadata.
+
+All UI values and scene interactions use visualization-space coordinates.
+**Segments Alignment** maps metadata wrist coordinates into visualization space.
+Loading applies that transform; saving applies its inverse. Changing the
+alignment transforms the visible editor values without changing the underlying
+metadata coordinates.
+
+A three-axis indicator in the top-right shows visualization space relative to
+the camera.
+
+## Edit metadata
+
+The **Edit** mode contains three tabs.
+
+### URDF
+
+1. Enter an absolute **URDF Path** and select **Load URDF**.
+2. Use **URDF Alignment** to translate, rotate, and scale the model.
+3. Expand **URDF Joints** to adjust each movable joint.
+
+URDF joints are direct sliders. Segment and Taxel lists start collapsed, and
+their displayed indices start at 0.
+
+URDF configuration is session-only and is not saved in tactile metadata. The
+model renders at opacity 0.2.
+
+### Segments
+
+- Use **Segments Alignment** to align all metadata poses.
+- Add a segment from a HandJoint and XYZ position, or select **Add to URDF
+  Surface** and click the model.
+- Segments are flat-shaded cubes with side length 0.004 m. Smooth color
+  gradients and cast/received shadows are disabled.
+- Expand **All Segments** to edit HandJoint, position, and XYZ rotation.
+
+HandJoint values must be unique. Taxel bindings use HandJoint names internally,
+so removing or renaming a referenced segment leaves the affected taxel field
+invalid and outlined in red.
+
+### Taxels
+
+Set Rows, Columns, Area Width, Area Height, Center Position, and Rotation.
+**Capture URDF Surface Pose** copies a clicked surface position and orientation
+into those fields without creating taxels. **Capture Segment Pose** copies the
+complete pose of a clicked Editing segment. **Add Taxels** then creates the
+row-major planar grid and copies the four Segment/Weight fields above the
+capture buttons into every new Taxel. Taxels are squares with side length
+0.002 m. A taxel with any validation error uses `ERROR_COLOR`;
+`SELECTED_COLOR` overrides it while that taxel is selected.
+
+To bind taxels:
+
+1. Select **Start Binding to Segments** and select taxels in the scene.
+2. Select **Select Segments** and choose one to four segments.
+3. Select **Complete**. The first Segment receives weight one; later Segments
+   receive weight zero.
+
+At most one binding-step button is enabled at a time.
+
+Expand **All Taxels** to edit 25 rows at a time. Page status shows the visible
+global Taxel range. Use **Jump to page**, **Last Page** (previous page), or
+**Next Page** from the controls above or below the rows. The unlabeled input and
+Jump button share one line; Last/Next share the following line. Each visible row
+contains pose, four segment slots, and four weights. `None` is an editor-only
+inactive slot with weight zero. Selecting the first active segment sets its
+weight to one; later segment selections start at zero. Editing a weight changes
+only that slot. Page changes reuse the same 25 row controls instead of rebuilding
+the menu. Validation reports rows whose weights do not sum to one.
+
+## Scene selection
+
+- Click an object to select it.
+- Shift-click adds objects of the same type.
+- Shift-drag selects a screen rectangle and temporarily disables camera dragging.
+- A selection cannot contain both segments and taxels. If an initial rectangle
+  contains both, taxels take priority.
+- Clicking empty scene space clears the selection.
+
+Selected objects keep `SELECTED_COLOR`. Related objects use the separate
+`SIDE_HIGHLIGHT_COLOR`: selecting segments highlights taxels with positive
+weights to those HandJoints, while selecting taxels highlights their
+positive-weight segments. Zero-weight slots are ignored.
+
+The **Selected Object** panel edits position and rotation, deletes selected
+objects, and scales multi-selection positions about their center. Edit mode
+changes Editing data; Preview mode changes Preview segments. Preview taxels are
+derived and cannot be selected. With no selection, the panel remains visible,
+shows Type `None` and Count `0`, and disables its input fields.
+Exactly one selected Segment reveals its HandJoint editor; exactly one selected
+Taxel reveals its four Segment/Weight binding slots. These editors reuse one
+control set rather than creating controls for every Taxel.
+
+The selected-object 3-axis control supports axis-only translation. Dragging an
+axis moves every selected object by the same delta. Single selections use the
+object orientation; multi-selections use visualization-space axes. Rotation,
+plane-translation, and scale handles are disabled.
+
+Action and validation errors appear in the fixed **Info** area at the bottom of
+the 3D viewport. It cannot be moved.
+
+## Save and load
+
+Enter **Metadata Path**, then use **Load** or **Save**. Saving validates the
+current document. If errors remain, the editor asks whether to continue; invalid
+drafts can still be written.
+
+**New** resets the document and initializes all OpenXR HandJoint segments in the
+neutral pose for **Hand Side**. The left-hand defaults are mirrored before
+conversion to visualization space.
+
+The JSON contains only schema fields:
+
+- `geometry_mapping_type` and `geometry_mapping`
+- `segment_source_type` and `segment_source`
+
+Editor HandJoint references are converted to `taxel_segment_indices` when
+saving and restored when loading. Pose orientations are XYZW quaternions, and
+taxel array order is the tactile force order. URDF paths, alignments, joint
+values, selections, and live settings are not persisted.
+
+The tool does not provide SDK binary export, a separate project format, or
+standalone check/export CLI commands.
+
+## Preview and live data
+
+Use **Reset Values** to restore Preview segment poses and URDF joint values from
+Editing.
+Choose the mutually exclusive checkbox in **Binding Metadata Source**:
+
+- **Editor** uses the current editor bindings and is selected by default.
+- **Live** is disabled until metadata is received from the tactile stream.
+
+The Preview Static section lists Segments before URDF Joints.
+
+Preview provides four independent apply operations:
+
+- **Apply Segments** replaces HandJoint/segment poses.
+- **Apply Taxels** replaces Editing taxels with the derived Preview taxels;
+  invalid rows retain their original binding poses.
+- **Apply URDF** replaces URDF joint values.
+- **Apply ALL** applies Segments and URDF plus the same taxels used by Apply
+  Taxels.
+
+Each Preview taxel is evaluated independently. Invalid rows are reported and
+omitted from the scene while valid taxels continue to render. Live force rows
+remain matched by their original taxel indices. Invalid taxels are reported in
+Info. Apply Taxels and Apply ALL preserve all source rows in their original
+order to match the forces; no Apply button is disabled.
+
+For live reception:
+
+1. Set **Tactile Collection ID**.
+2. Optionally enable **Use JointState to Drive URDF**, set its collection ID,
+   and provide mappings:
+
+   ```json
+   {"source_joint": ["urdf_joint", 1.0, 0.0]}
+   ```
+
+3. Select **Connect Live**.
+
+HandPose replaces Preview segments with only its valid joints, retained in
+HandJoint order, and is the sole source of Live Preview segment identities and
+poses. Invalid joints are absent. If the hand is missing or WRIST is invalid,
+Preview retains the last valid skeleton, which can still be applied to Editing,
+and the status displays **Data Invalid** until a valid hand arrives. Selected
+binding metadata affects only taxel deformation; it never seeds Preview
+segments. Enabled JointState mappings update Preview URDF joints. Live frames
+overwrite manual Static values while connected.
+Tactile force magnitude maps from blue at 0 N to red at 30 N, with values above
+30 N clamped. Missing or inactive force data uses the default taxel color.
+
+## Verify changes
+
+From `scripts/tactile_metadata`:
+
+```bash
+ruff format --check tactile_metadata_editor
+ruff check tactile_metadata_editor
+python -m compileall -q tactile_metadata_editor
+```

scripts/tactile_metadata/tactile_metadata_editor/__init__.py

@@ -0,0 +1,2 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0

scripts/tactile_metadata/tactile_metadata_editor/cli.py

@@ -0,0 +1,33 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+import argparse
+import sys
+from pathlib import Path
+
+from . import storage
+from .document import State
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="Local tactile metadata editor")
+    parser.add_argument("--port", type=int, default=8080)
+    parser.add_argument("--metadata", help="Metadata JSON path")
+    parser.add_argument("--side", choices=("left", "right"), default="left")
+    parser.add_argument("--urdf", help="URDF path")
+    args = parser.parse_args()
+    try:
+        from .ui import Editor
+
+        state = State()
+        if args.metadata:
+            path = Path(args.metadata).expanduser().resolve()
+            state.replace(storage.load(path), str(path))
+        Editor(state, args).run()
+        return 0
+    except (ImportError, OSError, RuntimeError, TypeError, ValueError) as exc:
+        print(f"{type(exc).__name__}: {exc}", file=sys.stderr)
+        return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())