Two Seeed reBot arms put two pens and an eraser into a pencil cup. In six modules you set up the scene, record scripted demonstrations, train vision-language-action models on them, run a trained model back in the simulator through an inference server, and finally move that model onto a Jetson AGX Thor, the robot's own computer, while the simulator keeps running on the workstation.
The stack spans three Python environments. Each one has a single job, so dependency conflicts stay contained. Isaac Sim pins its own Python, and the vla-edge server is deliberately light.
isaaclab 6.1.17| Hardware in the scene | Maker | Details |
|---|---|---|
| reBot B601 arm × 2 | Seeed Studio | 6-DOF arm with parallel gripper; follower arm of the reBot leader/follower teleoperation kit |
| Wrist camera × 2 | Seeed Studio reBot kit | UVC32 camera on link6, 640×480, 4.2 mm lens |
| Workstation and top camera | YAMLab station | Table, YAM arm mounts and a calibrated top camera on a gate at 1.70 m |
| Environment | Python | Runs |
|---|---|---|
~/miniconda3/envs/env_yamlab6 | Isaac Lab 3 / Isaac Sim 6 | The scene, the scripted expert, the sim client (sim_rollout) |
~/rebot_lerobot/.venv | 3.12, LeRobot 0.6.2 | Writing datasets, lerobot-train, serving SmolVLA checkpoints |
~/vla-edge/.venv | 3.14 | The vla-edge server, replay policy, unit tests |
rebot-sim-pipeline holds the scene and data generator. rebot-bimanual-vla holds the embodiment, the policy runtimes and the sim client. vla-edge is the serving stack. The single-arm rebot-arm-dli-isaacsim project supplies robot, camera and stationery assets.
# server + tests (vla-edge venv) python3 -m venv ~/vla-edge/.venv ~/vla-edge/.venv/bin/pip install -e "$HOME/vla-edge[dev]" pyarrow ~/vla-edge/.venv/bin/pip install --no-deps \ -e ~/rebot-sim-pipeline -e ~/rebot-bimanual-vla # sim client (Isaac Lab env) ~/miniconda3/envs/env_yamlab6/bin/pip install json-numpy ~/miniconda3/envs/env_yamlab6/bin/pip install --no-deps -e ~/vla-edge # training + SmolVLA serving (LeRobot venv) cd ~/rebot_lerobot/lerobot uv pip install --python ~/rebot_lerobot/.venv/bin/python -e ".[training,smolvla]" uv pip install --python ~/rebot_lerobot/.venv/bin/python fastapi uvicorn json-numpy uv pip install --python ~/rebot_lerobot/.venv/bin/python --no-deps \ -e ~/vla-edge -e ~/rebot-bimanual-vla
cd ~/rebot-bimanual-vla && ~/vla-edge/.venv/bin/pytest -q
Two reBot B601 arms stand at YAMLab's YAM mounts and share one pencil cup. Each arm sorts the objects on its own side, one arm at a time, so the arms never meet over the cup.
top (fx ≈ 392 px), left_wrist, right_wrist under each link6; all 640×480 RGBcd ~/rebot-sim-pipeline rebot-bimanual-generate -n 1 --gui --no-recording
A window opens for each camera. The expert homes both arms to (0, −45, −45, 0, 0, 0)°, does the pens first, then the eraser, and holds 2 s at the end so the success check can settle.
A scripted expert solves the task with inverse kinematics. Only successful episodes are kept, and they are written as a LeRobot v3 dataset in the real reBot leader/follower conventions, so the same files could train a policy for the physical arms.
cd ~/rebot-sim-pipeline
# 50 successes on layouts 100, 101, …
rebot-bimanual-generate -n 50 --seed 100Each attempt samples a layout from --seed + attempt number. Reusing a seed repeats its layouts, so pick a new range for each batch. Failed attempts are discarded and retried, up to 2N + 5 attempts. The run stages episodes first and writes the LeRobot dataset with the LeRobot venv at the end, so new episodes appear only when the whole run finishes.
| This run | Value |
|---|---|
| Attempts / successes | 52 / 50 |
| Wall time per episode (CPU physics) | ≈ 95–111 s |
| Frames per episode | 1,332–1,392 |
| Dataset after the run | 52 episodes · 70,884 frames |
| Feature | Shape | Contents |
|---|---|---|
observation.state | 14 | Left then right arm. Follower convention: −degrees per joint; gripper 0 (closed) to 270 (open), measured |
action | 14 | Leader convention: degrees ÷ [−1, −1, 1, 1, 1, −1]; gripper 0 (closed) to 45 (open), commanded |
observation.images.top | 480×640×3 | Video, 30 fps |
observation.images.left_wrist | 480×640×3 | Video, 30 fps |
observation.images.right_wrist | 480×640×3 | Video, 30 fps |
Every episode also gets an episode_metadata/episode_NNNNNN.json sidecar with its layout seed, so any episode can be replayed on its exact layout.
A replay policy serves one recorded episode's actions through the real inference server, and the sim executes them. If the conventions, camera names or chunking were wrong, the robot's state would drift away from the recording long before the task failed.
cd ~/rebot-bimanual-vla scripts/sim_replay_test.sh ~/rebot-sim-pipeline/datasets/rebot_bimanual_stationery 0
from pathlib import Path
from lerobot.datasets.lerobot_dataset import LeRobotDataset
root = Path.home() / "rebot-sim-pipeline/datasets/rebot_bimanual_stationery"
ds = LeRobotDataset("kabilanKB/rebot_bimanual_stationery", root=root)
ds.push_to_hub(
private=True,
tags=["rebot", "bimanual", "isaac-lab", "simulation"],
license="apache-2.0",
)push_to_hub uploads data, videos, metadata and the layout sidecars, writes a dataset card, and adds the v3.0 tag that LeRobotDataset needs to load it from the Hub. The dataset is private at kabilanKB/rebot_bimanual_stationery.
The same dataset trains any LeRobot policy. Models differ in size, in whether they read the language instruction, and in the action chunk they predict. That chunk length has to match what the inference server declares.
Small VLA (SmolVLM2 backbone, 100 M trainable parameters) with a flow-matching action expert. Reads the instruction.
lerobot/smolvla_baserebot_vla/smolvla.pyAction Chunking Transformer trained from scratch. No language input. A strong, fast baseline for one fixed task.
--policy.type=act)smolvla.pyLarge VLA from Physical Intelligence. Strongest prior, heaviest to fine-tune; a full fine-tune is unlikely to fit in 24 GB.
lerobot/pi05_baselerobot[pi]smolvla_base was pretrained with inputs named camera1–camera3. Training fails with a feature-mismatch error until you rename the dataset's cameras. Keep vla-edge's training order: top first, then the left and right wrists. The rename is saved into the checkpoint's preprocessor, so inference reuses it automatically.
{"observation.images.top": "observation.images.camera1",
"observation.images.left_wrist": "observation.images.camera2",
"observation.images.right_wrist": "observation.images.camera3"}Run the exact command with --steps=20 --batch_size=8 and an output directory you can throw away. On this machine it took under a minute: loss went from 2.87 to 1.14 at 3.6 GB.
~/rebot_lerobot/.venv/bin/lerobot-train \
--policy.path=lerobot/smolvla_base --policy.push_to_hub=false \
--dataset.repo_id=local/rebot_bimanual_stationery \
--dataset.root=$HOME/rebot-sim-pipeline/datasets/rebot_bimanual_stationery \
--rename_map='{"observation.images.top": "observation.images.camera1",
"observation.images.left_wrist": "observation.images.camera2",
"observation.images.right_wrist": "observation.images.camera3"}' \
--batch_size=32 --steps=20000 --log_freq=100 --save_freq=5000 --num_workers=8 \
--wandb.enable=false \
--output_dir=$HOME/rebot-bimanual-vla/outputs/smolvla_rebot_bimanual \
--job_name=smolvla_rebot_bimanual| Step | Loss | Note |
|---|---|---|
| 100 | 2.557 | Warm-up |
| 200 | 1.101 | |
| 1,000 | 0.121 | |
| 5,000 | 0.038 | Checkpoint; used for the inference smoke test |
| 10,000 | 0.017 | Checkpoint |
| 15,000 | ≈ 0.009 | Checkpoint |
| 20,000 | 0.007 | Final checkpoint (checkpoints/last) |
config.json lists observation.state with shape [6], copied from the single-arm base model. The model still uses all 14 values: SmolVLA pads whatever state it receives to 32, and the normalizer stats are 14 wide. Don't "fix" the config by hand.lerobot-train needs accelerate from the [training] extra. The torchcodec errors in the log are harmless: no system FFmpeg libraries are found, so LeRobot decodes video with PyAV.# ACT from scratch. Its default image keys come from the dataset, so no rename map. ~/rebot_lerobot/.venv/bin/lerobot-train --policy.type=act \ --policy.chunk_size=50 --policy.n_action_steps=50 --policy.push_to_hub=false \ --dataset.repo_id=local/rebot_bimanual_stationery \ --dataset.root=$HOME/rebot-sim-pipeline/datasets/rebot_bimanual_stationery \ --batch_size=32 --steps=50000 --wandb.enable=false \ --output_dir=$HOME/rebot-bimanual-vla/outputs/act_rebot_bimanual # π0.5 fine-tune. Install the extra first: uv pip install -e ".[pi]" ~/rebot_lerobot/.venv/bin/lerobot-train --policy.path=lerobot/pi05_base \ --policy.push_to_hub=false \ --dataset.repo_id=local/rebot_bimanual_stationery \ --dataset.root=$HOME/rebot-sim-pipeline/datasets/rebot_bimanual_stationery \ --batch_size=8 --steps=20000 --wandb.enable=false \ --output_dir=$HOME/rebot-bimanual-vla/outputs/pi05_rebot_bimanual
Smoke-test each one first. π0.5 will likely need its camera names mapped like SmolVLA, plus a smaller batch or frozen parts to fit in 24 GB. The released YAM checkpoints expect YAM kinematics and radians, so they don't transfer to reBot without training.
A trained checkpoint runs behind the vla-edge inference server. The Isaac Lab scene is a client: each request sends three images, the 14-value state and the instruction, and gets back a chunk of actions to execute at 30 Hz.
rebot-vla-serve and inspect its contract.bimanual-rebot.top_cam, left_cam, right_camPOST /act with images, state, instruction and the episode's seed
POST /act on a layout it never saw (seed 1000): the first 10 s, from the step-5,000 checkpoint, at about 2× speed. The left arm goes first, as in the demonstrations, while the right arm holds.# any free port PORT=<port> ~/rebot_lerobot/.venv/bin/rebot-vla-serve --policy smolvla-bimanual-rebot --backend torch \ --checkpoint outputs/smolvla_rebot_bimanual/checkpoints/last/pretrained_model \ --host localhost --port $PORT # the contract the server enforces curl -s http://localhost:$PORT/act
embodiment: bimanual-rebot, cameras top_cam, left_cam, right_cam, state_dim and action_dim 14, action_horizon 50.Training used layout seeds 5–6 and 100 onward. Evaluate from seed 1000 so every layout is new. The script starts the server, runs the sim with CPU physics as the data was recorded, and stops the server when it's done.
cd ~/rebot-bimanual-vla scripts/sim_policy_eval.sh outputs/smolvla_rebot_bimanual/checkpoints/last/pretrained_model 20 1000 # to watch one episode in the Isaac Sim window instead: # scripts/sim_policy_eval.sh <checkpoint> 1 1000 -- --viz kit --realtime
It writes server.log, rollout.log, results.json (success, steps, requests, server latency and the final pose of each object per episode) and rollout.mp4 with the three cameras side by side.
| Run | Checkpoint | Layouts | Result |
|---|---|---|---|
| Replay (sanity check) | recorded episode 0 | seed 5 | 1/1 success |
| Smoke test | step 5,000 | seed 1000, 300 steps | Path works; left arm reaches for the red pen while the right waits; ≈ 285 ms per chunk while sharing the GPU with training |
| Final evaluation, Dell | step 20,000 | seeds 1000–1004 | 0 / 5 Stopped at seed 1005, where CPU physics stalled; rerun with --episode-timeout |
| Final evaluation, Thor (Module 5) | step 20,000 | seeds 1000–1019 | 1 / 20 Seed 1015 succeeds; ≈ 278 ms per chunk |
Each object's final pose and grasp state in results.json show which stage of the task broke. On the Dell's first five layouts, the final checkpoint grasps the right object and carries it toward the cup, but never opens the gripper to let go. Over all 20 layouts on the Thor, it picks up at least one object in 18, and finishes the task in one.
| Seed | How the episode ended (after 90 s) |
|---|---|
| 1000 | All three objects grasped and held 1–2 cm from the cup centre, 3–9 cm above the table; never released |
| 1001 | Red pen still grasped, resting on the table 6.5 cm from the cup; right arm never started |
| 1002 | Red pen still grasped, 2 cm from the cup centre and 9 cm above the table; right arm never started |
| 1003 | Red pen still grasped, resting on the table 6.5 cm from the cup; right arm never started |
| 1004 | Red pen still grasped, resting on the table 13 cm from the cup; right arm never started |
--execute 25, and more demonstrations around the release.sim_rollout --execute N runs only the first N rows of each chunk before asking again. Fewer rows react faster to what the cameras see, at the cost of more requests. Keep the full chunk when serving the replay policy: replay advances one whole chunk per request.
The simulator stays on the Dell workstation, and the policy moves to the Jetson AGX Thor, the computer that would ride on the real robot. Every chunk the robot executes is computed on the deployment hardware, over a real network, before any arm moves. The sim waits for each chunk, so slow inference shows up in the timing numbers but never changes the physics.
wire.py: sim units ↔ reBot wire values, camera renamingresults.json, rollout.mp4, per-request timingpretrained_model/ (869 MB)
bimanual-rebot contractrebot_vla.smolvla, step-20,000 checkpoint, CUDAA 50-row chunk covers 1.67 s of robot time at 30 Hz. On the real robot, the next chunk has to arrive before the current one runs out, or the arms pause. With --execute 25 the budget halves to 0.83 s.
| Stage | Time | Source |
|---|---|---|
| Upload request (3.7 MB over Wi-Fi) | ≈ 600 ms | Measured: 5.8–6.4 MB/s from the Dell to the Thor |
| Network round trip | 13–29 ms | Measured with ping |
| SmolVLA inference on Thor | ≈ 278 ms | Measured: 274–282 ms mean per episode over 20 episodes, eager PyTorch, 10 denoising steps. 160 ms on the RTX PRO 5000 |
| Return chunk (3.8 KB) | < 30 ms | Payload is tiny |
/act takes raw arrays today, so that is a protocol change.The Thor's existing LeRobot venv ran LeRobot 0.4.0 with a CPU-only PyTorch, so this loop gets its own. Build a fresh venv with the Jetson PyTorch wheel that vla-edge's own Thor installer uses, then add the three repositories. vla-edge's full setup_jetson_thor.sh also installs the YAM arm driver and RealSense, which this loop does not need.
# from the Dell: copy code and the trained checkpoint into one folder on the Thor THOR=<user>@<thor-host> ssh $THOR "mkdir -p ~/rebot-hil" rsync -a --exclude .venv --exclude .git \ ~/vla-edge ~/rebot-bimanual-vla ~/rebot_lerobot/lerobot $THOR:rebot-hil/ rsync -a ~/rebot-bimanual-vla/outputs/smolvla_rebot_bimanual/checkpoints/020000/pretrained_model/ \ $THOR:rebot-hil/smolvla_020000/ # on the Thor ssh $THOR python3 -m venv ~/rebot-hil/.venv && source ~/rebot-hil/.venv/bin/activate pip install torch==2.10.0 torchvision \ --index-url https://pypi.jetson-ai-lab.io/sbsa/cu130/+simple # libraries the Jetson wheel links pip install nvpl-blas nvpl-lapack "nvidia-cudss-cu13<1" # preloads them for every run of this venv python ~/rebot-hil/vla-edge/scripts/jetson_thor_postinstall.py pip freeze | grep -E "^(torch|torchvision|nvpl|nvidia-cudss)" > ~/rebot-hil/pins.txt pip install -c ~/rebot-hil/pins.txt -e ~/rebot-hil/lerobot[smolvla] -e ~/rebot-hil/vla-edge \ fastapi uvicorn json-numpy pip install --no-deps -e ~/rebot-hil/rebot-bimanual-vla # must print True python -c "import torch; print(torch.cuda.is_available())"
pins.txt constraints file prevents that. Without the NVPL libraries, import torch fails with libnvpl_lapack_lp64_gomp.so.0 missing. Torch 2.10 links libcudss.so.0, which only cuDSS releases before 1.0 provide. Check torch.cuda.is_available() after every install. The model also loads the SmolVLM2 backbone config from Hugging Face on first start, so the Thor needs internet access once, or a copy of that cache entry.# Thor: serve on the LAN. Use the same PORT value on the Dell. PORT=<port> # MAXN, already the current mode sudo nvpmodel -m 0 rebot-vla-serve --policy smolvla-bimanual-rebot --backend torch \ --checkpoint ~/rebot-hil/smolvla_020000 --host <thor-lan-address> --port $PORT # Dell: check the contract, then run the same 20 layouts as the workstation evaluation curl -s http://<thor-host>:$PORT/act cd ~/rebot-bimanual-vla PYTHONPATH="$PWD/src:$HOME/rebot-sim-pipeline/src" OMNI_KIT_ACCEPT_EULA=YES \ ~/miniconda3/envs/env_yamlab6/bin/python -m rebot_vla.sim_rollout \ --server <thor-host>:$PORT --device cpu --episodes 20 --seed 1000 --episode-timeout 900 \ --video outputs/hil_thor/rollout.mp4 --results outputs/hil_thor/results.json # add --viz kit to watch it in the Isaac Sim window, as in the run below
RemoteDisconnected). sim_rollout retries once when the connection drops before any response arrives.Use the same checkpoint and the same seeds. Each episode's seed also fixes SmolVLA's sampling noise, but small numeric differences between the two GPUs, and between runs of the sim, still change individual episodes: seed 1000 ended differently in two Thor runs. Compare success rates, not episodes.
| Metric | Pass when | Dell (RTX PRO 5000) | Thor |
|---|---|---|---|
| Success rate | Within a couple of episodes of the workstation run | 0 / 5 (seeds 1000–1004) | 1 / 20 (seeds 1000–1019); 0 / 5 on 1000–1004 |
Inference time, server_ms_mean | Well under the 1.67 s chunk | ≈ 160 ms | ≈ 278 ms |
Wall time per request, wall_s ÷ requests | Under 1.67 s for real time | — | 5.1 s, mostly CPU physics and rendering with the GUI open |
Real-time behaviour, --realtime | Arms never wait for a chunk | Not run yet: the sim runs about 3× slower than real time | |
These values go over POST /act. They are the dataset's own units, so a policy trained on the dataset is served without conversion. rebot_vla/wire.py converts to and from simulator units using rebot_sim_pipeline.conventions, the same code that recorded the data.
| Field | Per arm (left, then right) | Units |
|---|---|---|
state [0:7], [7:14] | shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_yaw, wrist_roll, gripper | −degrees; gripper 0–270 measured |
| action rows (chunk × 14) | same joints, leader convention | degrees ÷ [−1, −1, 1, 1, 1, −1]; gripper 0–45 commanded |
top_cam, left_cam, right_cam | dataset top, left_wrist, right_wrist | uint8 RGB, 480×640 |
seed | one per episode; restarts replay and fixes SmolVLA's sampling noise | int |