Real robots · dual-arm

SFT Data on Real Robots

A real dual-arm robot doing household and tabletop tasks, written as multimodal agent conversations for supervised fine-tuning of VLM agents. Each turn gives the robot its state and three camera images. It then writes a short first-person chain of thought and calls one tool, move. A last turn looks at the final views, says whether the task's goal is met, and stops without a tool call.

Some episodes contain an attempt that does not work, such as a grasp that misses. The chain of thought says so once the views show it, and the following steps try again.

Data spec

Episodes	500, one conversation each
Tasks	175 household and tabletop tasks, 2 to 3 episodes each
Steps	75,375 assistant turns with a `move` call
Images	227,625 embedded JPEGs at 640 × 480, 848 × 480, and 848 × 477
Cameras	A fixed top view and one wrist view per arm, three images per step
Robot	A real dual-arm robot
Chain of thought	Median 133 words per assistant turn, 90th percentile 142, maximum 161
Format	OpenAI chat and tool call messages, stored in Arrow
Size	14.3 GB in 30 Arrow shards, about 30 MB per row

One step

From an episode with the task: “decorate the small christmas tree.” State values are shortened here.

Head camera view of both arms placing a gold star on a small Christmas tree. — Head

Left wrist camera view of the gold star, close up. — Left wrist

Right wrist camera view of the other arm and the top of the tree. — Right wrist

user: {"terminated": false, "success": false, "arms": {"left": {"fingertip_pose": {"position_mm": [319.9422, -341.2891, 438.6333], "quaternion_wxyz": [0.8854, 0.0805, -0.1651, -0.4268]}, ...}, "right": {...}}} Camera: head; horizontal FOV 96.7 deg, vertical FOV 80.3 deg Camera: left_wrist; horizontal FOV 96.3 deg, vertical FOV 79.9 deg Camera: right_wrist; horizontal FOV 96.4 deg, vertical FOV 80.1 deg
assistant: My left fingers now brace the gold star alongside my right gripper. I’ll release my left grip and guide the topper onto the treetop with my right hand. The placement point is at (1, -11, -44) mm in my right gripper frame; I’ll aim for that point + (5, -8, 20) mm to approach the mounting position with clearance. I’ll turn 10 deg, rolling and yawing, bringing my fingers from -12 to -11 deg below horizontal. My left hand will settle slightly before releasing its support. My left move is (2, 0, -3) mm with rotation (0, 1, 0) deg, then my left gripper opens. My right move is (6, -19, -24) mm with rotation (-7, 2, -7) deg, keeping my right gripper closed.
move {"left": {"delta_xyz": [2, 0, -3], "delta_rotation": [0, 1, 0], "gripper_closed": false}, "right": {"delta_xyz": [6, -19, -24], "delta_rotation": [-7, 2, -7], "gripper_closed": true}}
tool: {"state": {"terminated": false, "success": false, "arms": {"left": {"fingertip_pose": {"position_mm": [322.1603, -343.2213, 435.9098], "quaternion_wxyz": [0.8859, 0.0866, -0.1548, -0.4285]}, ...}, "right": {...}}}}

Across an episode

The head camera at three points in the same episode, which runs for 284 steps.

A small bare Christmas tree and a basket of decorations on a marble table. — Step 0

Both arms gripping the base of the tree, now wrapped in string lights. — Step 170

The decorated tree with lights and a star, with both arms clear of it. — Final view

More tasks

Surgical instruments and gauze laid out on a purple cloth between the arms. — Prepare the surgical pack.

Both arms working beneath a clothes hanger, with a pair of socks in a tray. — Clip the socks to the hanger.

Both arms above a green poker mat with chips on it. — Distribute Texas hold 'em gaming equipment.

Conversation

system: describes the robot, units, frames, and the move tool.
user: Task: <instruction>.
Each step has three messages:
- user: the state JSON (fingertip poses in mm, joint angles in degrees, gripper commands), then a label and image for each camera. The label gives the camera's horizontal and vertical field of view, which differ between episodes because the cameras do.
- assistant: the chain of thought and one move call, with arguments {"left": {"delta_xyz": [x, y, z], "delta_rotation": [rx, ry, rz], "gripper_closed": b}, "right": {...}}. Translations are in mm and rotations are rotation vectors in degrees, in the arm's current fingertip frame.
- tool: {"state": ...}, the measured state after the move.
user, then assistant: the final state and views, then a completion reply with no tool call, which ends the episode.

Row schema

Column	Type	Meaning
`episode`	string	Episode ID, `ep_<index>`
`episode_index`	int32	0 to 499, grouped by task
`task`	string	Task key
`robot`	string	Robot name
`instruction`	string	Task instruction
`messages`	list of messages	The conversation
`images`	list of Image	Every image in the conversation, in order of appearance
`tools`	string	The OpenAI `tools` list as JSON: the `move` function and its JSON schema
`num_steps`, `num_images`	int32	Steps (tool calls) and images in the row

A message has role, content, tool_calls, and tool_call_id, with one Arrow schema for every message. Fields a message does not use are null. content is always a list of parts {type, text, image_url: {url}}, and the k-th image part is images[k].

Conventions

The fingertip frame has its origin midway between the fingertips, +X toward the fingertips, +Y across the jaws, and +Z = X × Y.
A move is applied once, as T_goal = T_tip · T_delta. The controller drives the arms until they arrive or stop moving, then sets the gripper.
An idle arm holds: its move is zero.
Moves carry no step count or duration. How long a move runs is up to the controller.