Robot data formats

Each model expects your robot's data in one exact format, and replies in one exact format. This page shows both, for every model: the names to use, the size of each camera image and list of numbers, and the shape of the moves that come back.

• The model server checks your data, and refuses data in the wrong format. With the SDK, one observation in the wrong format ends the session with invalid_observation.
• The model server always replies in the format shown here, and the SDK checks each answer.
• yb models --ready shows each model's format, under contract.

Camera images are 224 × 224 color (RGB) arrays of uint8. The robot's state is a list of ordinary numbers (no NaN or infinity). The prompt is the task in words, and must match the session's task; change it with reset.

openpi-libero-v1

Robot: LIBERO simulator (Franka, robosuite). How it moves: end-effector delta + gripper, about 10 steps per second.

Returns actions: an array of numbers of shape (10, 7): one row per step, one column per number. Columns: dx, dy, dz, drot_x, drot_y, drot_z, gripper. Carry out the first few steps, then send a new observation.

Recorded September 25 model test (historical): the π0.5 LIBERO model was tested on a rented A100 GPU on 2026-09-25 (86 ms per request). Repeated runs on the same input give slightly different actions (largest difference about 0.004); we are still working to make them identical.

For later Modal serving and startup measurements, see Performance and availability. These dated records do not qualify a physical robot or guarantee current capacity.

import numpy as np

observation = {
    "observation/image": np.zeros((224, 224, 3), dtype=np.uint8),
    "observation/wrist_image": np.zeros((224, 224, 3), dtype=np.uint8),
    "observation/state": np.zeros(8, dtype=np.float32),
    "prompt": "put the bowl on the plate",
}

openpi-droid-v1

Robot: DROID (Franka Panda, joint-velocity control). How it moves: joint velocity + gripper position, about 15 steps per second.

Returns actions: an array of numbers of shape (15, 8): one row per step, one column per number. Columns: dq1, dq2, dq3, dq4, dq5, dq6, dq7, gripper_position. Carry out the first few steps, then send a new observation.

Recorded September 25 model test (historical): the π0.5 DROID model was tested on a rented A100 GPU on 2026-09-25 (87 ms per request). It has not yet been tested on a real DROID robot.

For later Modal serving and startup measurements, see Performance and availability. These dated records do not qualify a physical robot or guarantee current capacity.

import numpy as np

observation = {
    "observation/exterior_image_1_left": np.zeros((224, 224, 3), dtype=np.uint8),
    "observation/wrist_image_left": np.zeros((224, 224, 3), dtype=np.uint8),
    "observation/joint_position": np.zeros(7, dtype=np.float32),
    "observation/gripper_position": np.zeros(1, dtype=np.float32),
    "prompt": "put the bowl on the plate",
}