feat(lab_sim): add Quest controller teleoperation objective

[nbbrooks]

Adds a single new "Quest Teleop" objective to lab_sim, plus the host-side infrastructure required to pair with the
meta_quest_teleoperation Unity app.

How the disjoint TF trees are handled

The Quest publishes controller pose in a quest frame anchored at the headset's spawn location. The robot's world frame is anchored at its base. We don't know the geometric relationship between them — it depends on where the operator is physically standing — and we don't try to measure it. The two TF trees are disjoint roots, and the design works by never crossing between them:

• On grip press, snapshot both anchors: - controller pose at clutch time, expressed in quest - grasp_link pose at clutch time, expressed in world
• While grip is held, each tick:
  • read the controller's current pose in quest
  • compute its pose relative to its clutch-time pose — a relative rigid-body transform with no inherent frame attachment - apply that same relative transform to the grasp_link's clutch-time pose; the result is a target pose in world - drive VFC at the target

Files

• src/lab_sim/objectives/quest_teleop.xml: the objective. Single flat BehaviorTree, runnable from the UI under the "User Input" subcategory.
• src/lab_sim/launch/sim/robot_drivers_to_persist_sim.launch.py: drivers- to-persist launch override. Starts ros_tcp_endpoint (Unity bridge) and a debug-only world->quest static TF so RViz can render the in-quest-frame VisualizePose markers. The static TF is not used by any control path. Persists across agent_bridge restarts so the Quest's TCP socket stays up.
• src/lab_sim/config/config.yaml:
  • simulated_robot_driver_persist_launch_file points at the new launch file (without this, the inherited empty launch wins and the Quest app never sees a peer).
  • experimental_behaviors loader added to behavior_loader_plugins, which provides the pose conversion, basis change, snapshot subscriber, and Bool/Empty publisher behaviors the objective uses.
• .gitmodules: two new submodules under src/external_dependencies/:
  • moveit_pro_experimental_behaviors, pinned to feat/pose-vector-basis-snapshot-behaviors pending the PR landing on main; switch to branch = main once that merges.
  • ROS-TCP-Endpoint (Unity-Technologies), pinned to main-ros2.

For end-to-end setup with the Quest app, see:
https://docs.picknik.ai/hardware_guides/input_devices/setting_up_the_meta_quest_for_teleop/

[nbbrooks]

This objective is what was used during the May Qualcomm visit for the Kinova teleop demo https://github.com/PickNikRobotics/moveit_pro_kinova_ws/tree/picknik_006_gen3-quest-teleop

[nbbrooks]

The docs update/rewrite to just install the apk (no installing Unity, no building the apk yourself) is https://github.com/PickNikRobotics/moveit_pro/pull/18976

[nbbrooks]

The slack discussion around these PRs is here https://picknik.slack.com/archives/C0B4KSQB490/p1779126664079659

[davetcoleman]

[written by AI]

Stale PR sweep. Opened 2026-05-11, last commit 2026-05-11. We are cleaning up open PRs older than three months.

State: the most clearly valuable of the Quest PRs and the one with the most context. You noted on 2026-07-20 that this objective is what ran the Kinova teleop demo at the May Qualcomm visit. But it has not been touched since May, four of its six paths are absent from main (including quest_teleop.xml itself and two new src/external_dependencies/ submodules), and it depends on moveit_pro #18975, which is also still open.

Recommendation: schedule it or close it, but do not leave it drifting. Three PRs across two repos (this, moveit_pro #18975, moveit_pro #18976) are all waiting on the same decision, and adding two Git submodules to example_ws is a change that should not go stale before review.

@nbbrooks is Quest teleop going into a release? @davetcoleman given the demo history this may be a product call.