feat(vla_sim): scripted red-block pick and its recording wrapper - #898
Draft
danwahl wants to merge 6 commits into
Draft
feat(vla_sim): scripted red-block pick and its recording wrapper#898danwahl wants to merge 6 commits into
danwahl wants to merge 6 commits into
Conversation
`Pick the Red Block` reads the cube's pose from the simulation's TF and picks it up; `Collect the Red Block Pick` wraps that in Trainer's recording Behaviors, keeping the episode only if the pick succeeds and converting the dataset to LeRobot v3.0. Together they give the VLA guides a scripted task to document, which `vla_sim` did not previously have. Conversion labels `action` from a `sensor_msgs/JointState` command topic and nothing outside of teleoperation publishes one, so a scripted collection converts only with next-state labels. `joint_command_bridge.py` republishes the trajectory controller's setpoint and the latched gripper command as `/joint_commands`; the gripper Objectives latch their command through a parameter, since a `GripperCommand` goal is not observable on any topic. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019YCk2yNsEQkVUSvhAmsJ4m
…s way Review of the scripted pick found three ways it could mislead. `SetRos2Parameter` was the first node of the gripper Objectives, so a stack running without the bridge failed them before `MoveGripperAction` ran. Teleop drives the gripper through those same Objectives, so a recording side-channel could stop the robot; it is now wrapped and its timeouts are sized for a local node. Nothing checked that the cube came with the gripper. Every Behavior in the pick succeeds whether the jaws close on the cube or beside it, so a missed grasp was saved as a demonstration. The Objective now measures the cube's displacement against its pre-grasp pose. The bridge published a state echo whenever no trajectory was running, which would have handed a hand-teleoperated recording an `action` column copied from its own observations - the mislabelling that an empty action topic exists to refuse. It now stays silent until the trajectory controller has run. Also raises the publish rate clear of the converter's 50 ms sync tolerance, drops a channel-order warning that fired on a harmless reorder, and makes the bridge's own log visible. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019YCk2yNsEQkVUSvhAmsJ4m
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueComment |
|
Joints 1, 3, 5 and 7 of a Gen3 are continuous. With lab_sim's candidate set of four, every solution for the yawed grasp landed a full turn away on joint_3 and the controller rejected the trajectory for starting 2*pi from where the arm actually was. Also records that `quest_oculus` publishes the same topic for Cartesian teleoperation, and that it stays silent until a headset is enabled. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019YCk2yNsEQkVUSvhAmsJ4m
… way `Pick the Red Block` now runs the same node sequence as lab_sim's `Pick object`, with the TF read standing in for its `CreatePoseStamped`. The lift confirmation is gone: #830's oracle did not have one either, and the Objective exists to show how the recording Behaviors are wired, so the extra branch obscured the lesson more than it earned. Playback is where the guide has the reader judge a demonstration, and the README and the wrapper's comment now say so rather than claiming the tree can tell a grasp from a jaw that closed beside the cube. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019YCk2yNsEQkVUSvhAmsJ4m
|
…weight The node names already carry the intent, so the header drops to the one thing that differs from lab_sim's wrapper - where the scene reset sits - and the remaining comments match that file's phrasing. Also drops a leftover claim that the Objective keeps an episode only if the pick succeeded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019YCk2yNsEQkVUSvhAmsJ4m
The Objective comments and the README carried their own history: which file a subtree came from, which value was raised and from what, what went wrong before a dwell was added. None of that is usable by someone running the Objectives, so what remains is the part that is: where `cube_red_tf` comes from, that a Gen3's continuous joints constrain `max_ik_solutions`, that `approach_distance` follows the tool axis, and that nothing resets the scene. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019YCk2yNsEQkVUSvhAmsJ4m
|
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
vla_simhas no scripted task for the VLA guides to document, which is what blocks #20907 and #21451. #830 was the first attempt and was closed as too large and specific; this is the generalized version, 740 lines against 4303.What's here
Pick the Red Blockrunslab_sim'sPick objectsequence, with a TF read ofcube_red_tfin place of itsCreatePoseStamped.Collect the Red Block Pickis the recording wrapperlab_simalready documents, with the scene reset ahead ofRecordEpisodeso the controller switch and teleport land outside the window.joint_command_bridge.pypublishes/joint_commandsfrom the trajectory controller's setpoint and the latched gripper command, because conversion labelsactionfrom a command topic and refuses an episode whose one is empty.quest_oculusdoes the same for Cartesian teleoperation off the velocity_force_controller; this is the trajectory-controller counterpart. It stays silent until a trajectory has run, so a teleoperated recording still gets the refusal rather than anactioncolumn copied from its own observations.Where
Pick from Posedeviates from lab_sim'send_effector_groupis this configuration's SRDF group name.Close Gripper.ExecuteMTCSolutionreturns on trajectory time while the admittance controller is still ~7 cm behind, so without them the gripper closes on air.max_ik_solutions4 → 16. Joints 1, 3, 5 and 7 of a Gen3 are continuous, and four candidates all landed a full turn away onjoint_3, which the controller rejected for starting 2π from the arm's actual position.Verification
Repeated collection runs on a 10.1
vla_simstack each recorded and converted one episode, the cube lifting 0.115 m to 0.234 m within 0.03 mm run to run. Conversion used--action-source command, and the labels carry real commanded motion: mean |action − state| across the arm is 0.050 rad, the gripper channel is exactly {0.0, 0.7}, and no frame has action equal to state.Known gap
observation.stateis 15 wide against an 8-wideaction. This is the existing asymmetry, not a new one — nothing in the tree publishes a narrowed observation topic, andquest_oculusdrops the gripper channel entirely by default. SinceExecutePolicynarrows to 8 at inference, training on this data wants a narrowed observation topic; #830 published one, and it is left out here as scope.Relates to #20907 and #21451. The documentation PR against
moveit_profollows.🤖 Generated with Claude Code
https://claude.ai/code/session_019YCk2yNsEQkVUSvhAmsJ4m