Simulation · dual-arm

SFT Data in Simulation

Simulated dual-arm demonstrations written as multimodal agent conversations, for supervised fine-tuning of VLM agents. Each turn gives the robot its state and three camera images. It then writes a short first-person chain of thought and calls one tool, move. A last turn checks the task's goal in the final views and stops without a tool call.

Data spec

Episodes	3,500, one conversation each
Tasks	35, with 100 episodes each
Steps	89,056 assistant turns with a `move` call
Images	277,668 embedded JPEGs at 640 × 480
Cameras	Head and one wrist camera per arm, three images per step
Robot	A simulated dual-arm robot
Chain of thought	Median 92 words per assistant turn, 90th percentile 131, maximum 153
Format	OpenAI chat and tool call messages, stored in Arrow
Size	13.4 GB in 27 Arrow shards, about 4 MB per row

One step

From an episode with the task: “Pick up two slices of bread, place them into the toaster, and press the lever down.” State values are shortened here.

Head camera view of one gripper holding a slice of bread beside a toaster. — Head

Left wrist camera view of the toaster on a wooden table. — Left wrist

Right wrist camera view of the bread held between the fingers, beside a holder of slices. — Right wrist

user: {"terminated": false, "success": false, "arms": {"left": {"fingertip_pose": {"position_mm": [-299.5750, -194.7300, 921.3599], "quaternion_wxyz": [0.7069, 0.0000, 0.0000, 0.7072]}, ...}, "right": {...}}} Camera: head; horizontal FOV 96 deg, vertical FOV 79.6 deg Camera: left_wrist; horizontal FOV 77.7 deg, vertical FOV 62.3 deg Camera: right_wrist; horizontal FOV 77.7 deg, vertical FOV 62.3 deg
assistant: The lifted bread is clear of the holder beside the remaining slices. I'll bring my left hand toward it while my right hand positions it for transfer. I'll move my left gripper slightly forward, sideways, and upward, keeping its jaws open to approach the bread, turning 63 deg, rolling and yawing, with the fingers ending 15 deg below horizontal. I'll move my right gripper forward, sideways, and downward to present the bread, turning 43 deg, rolling and yawing, bringing its fingers from 27 to 10 deg below horizontal while maintaining its grip. My left move is (5, -136, 84) mm with rotation (-33, 32, -43) deg in my left gripper frame, staying open; my right move is (51, -39, -41) mm with rotation (33, 10, 26) deg in my right gripper frame, staying closed.
move {"left": {"delta_xyz": [5, -136, 84], "delta_rotation": [-33, 32, -43], "gripper_closed": false}, "right": {"delta_xyz": [51, -39, -41], "delta_rotation": [33, 10, 26], "gripper_closed": true}}
tool: {"state": {"terminated": false, "success": false, "arms": {"left": {"fingertip_pose": {"position_mm": [-163.6780, -189.2550, 1005.0650], "quaternion_wxyz": [0.8543, -0.3841, -0.0079, 0.3498]}, ...}, "right": {...}}}}

Across an episode

The head camera at three points in the same episode, which runs for 25 steps.

A toaster and a holder of bread slices on a table, with both arms at rest. — Step 0

One gripper holds a slice of bread above the toaster. — Step 10

Bread slices in the toaster, with both arms back at rest. — Final view

More tasks

A finished tower of wooden boards and blocks on a table. — Build a tower using the wooden blocks and wooden boards.

Four numbered tokens arranged in a row on a table. — Arrange the numbers from left to right to form the largest possible number, and place them on the pad.

Rows of game tiles on a table between the robot's grippers. — Wait for the opponent to discard a tile, then declare a kong with the matching tiles.

Conversation

system: describes the robot, units, frames, and the move tool.
user: Task: <instruction>.
Each step has three messages:
- user: the state JSON (fingertip poses in mm, joint angles in degrees, gripper commands), then a label and image for each camera. The label gives the camera's horizontal and vertical field of view.
- assistant: the chain of thought and one move call, with arguments {"left": {"delta_xyz": [x, y, z], "delta_rotation": [rx, ry, rz], "gripper_closed": b}, "right": {...}}. Translations are in mm and rotations are rotation vectors in degrees, in the arm's current fingertip frame.
- tool: {"state": ...}, the measured state after the move.
user, then assistant: the final state and views, then a completion reply with no tool call, which ends the episode.

Row schema

Column	Type	Meaning
`episode`	string	Episode ID, `sim_<index>`
`episode_index`	int32	0 to 3499, grouped by task
`task`	string	Task key
`robot`	string	Robot name
`instruction`	string	Task instruction
`messages`	list of messages	The conversation
`images`	list of Image	Every image in the conversation, in order of appearance
`tools`	string	The OpenAI `tools` list as JSON: the `move` function and its JSON schema
`num_steps`, `num_images`	int32	Steps (tool calls) and images in the row

A message has role, content, tool_calls, and tool_call_id, with one Arrow schema for every message. Fields a message does not use are null. content is always a list of parts {type, text, image_url: {url}}, and the k-th image part is images[k].

Conventions

The fingertip frame has its origin midway between the fingertips, +X toward the fingertips, +Y across the jaws, and +Z = X × Y.
A move is applied once, as T_goal = T_tip · T_delta. The controller drives the arms until they arrive or stop moving, then sets the gripper.
Moves carry no step count or duration. How long a move runs is up to the controller.