RoboLab

RoboLab is a task-based evaluation benchmark for robot manipulation policies built on NVIDIA Isaac Lab. It provides 100+ manipulation tasks with automated success detection, a server-client policy architecture, and multi-environment parallel evaluation — designed for reproducible, large-scale benchmarking of generalist robot policies in simulation.

Key Features

• RoboLab-120: An initial set of 120 brand new benchmark tasks spanning pick-and-place, stacking, rearrangement, tool use, and more — each with language instructions and automated success/failure detection via composable predicates.
• Bring your own robot: Tasks are not tied to a specific robot embodiment, so you can plug in any robot compatible with IsaacLab!
• Rich Asset Libraries: See a list of objects, scenes, and curated backgrounds — everything you need to create new scenes and new tasks for your own evaluation needs.
• AI-Enabled Workflows: Generate new scenes and tasks in minutes using natural language with the /robolab-scenegen and /robolab-taskgen Claude Code skills.
• Multi-Environment Parallel Evaluation: Run multiple episodes in parallel across environments with vectorized conditionals and per-environment termination.
• Server-Client Policy Architecture: Policy models run as standalone servers; RoboLab connects via lightweight inference clients (OpenPI, GR00T, and more).

Getting Started

Requires uv. Isaac Sim 5.0 and Isaac Lab 2.2.0 are installed automatically via uv sync. See Requirements for hardware.

Installation

git clone https://github.com/NVlabs/RoboLab.git
cd robolab
uv venv --python 3.11
source .venv/bin/activate
uv pip install "setuptools<81"
uv sync

On first run, Isaac Sim prompts you to accept the NVIDIA Omniverse EULA. Either accept it interactively, or set it once in your shell:

export OMNI_KIT_ACCEPT_EULA=Y

Running without activating the venv: if you don't source .venv/bin/activate, prefix every python command with uv run (e.g. uv run python scripts/check_registered_envs.py).

Verify installation:

python scripts/check_registered_envs.py

Run without a policy

# Run an empty episode with random actions
python examples/demo/run_empty.py --headless

# Playback recorded demonstration data
python examples/demo/run_recorded.py --headless

Run with a policy

RoboLab uses a server-client architecture: your model runs as a standalone server, and RoboLab connects to it via a lightweight inference client. To quickly test RoboLab, try Pi0-5 via OpenPI.

Each inference client has its own lightweight Python dependency — e.g. Pi0 / Pi0-fast / Pi05 need openpi-client, which is not installed by uv sync. Install only the client(s) you need; see docs/inference.md for each backend. For example, to use the Pi0 family:

# Clone the OpenPI repo separately and install its client into the RoboLab venv
git clone git@github.com:xuningy/openpi.git ../openpi
uv pip install -e ../openpi/packages/openpi-client

1. Start your policy server in a separate terminal.
2. Run evaluation:

   python examples/policy/run_eval.py --policy pi05 --task BananaInBowlTask --num-envs 12 --headless

3. Analyze results:

   python analysis/read_results.py output/<your_run_folder>

Common CLI Options

# Run on specific tasks (these two are good for sanity checking)
python examples/policy/run_eval.py --policy pi05 --task BananaInBowlTask RubiksCubeAndBananaTask

# Run on a tag of tasks
python examples/policy/run_eval.py --policy pi05 --tag semantics

# Run 12 parallel episodes per task
python examples/policy/run_eval.py --policy pi05 --headless --num-envs 12

# Enable subtask progress tracking
python examples/policy/run_eval.py --policy pi05 --headless --enable-subtask

# Resume a previous run (skips completed episodes)
python examples/policy/run_eval.py --policy pi05 --output-folder-name my_previous_run

Documentation

Full documentation is at docs/README.md, covering:

• Objects, Scenes, Tasks — Creating and managing assets and benchmark tasks
• Robots, Cameras, Lighting, Backgrounds — Configuring simulation parameters
• Environment Registration — Combining tasks with robot/observation/action configs
• Inference Clients — Built-in policy clients and server setup (OpenPI, GR00T)
• Evaluating a New Policy — Implementing your own inference client
• Running Environments — CLI reference and evaluation workflows
• Analysis and Results — Summarizing, comparing, and auditing results
• Subtask Checking, Conditionals, Event Tracking — Advanced task features

Example Tasks

See the full Benchmark Task Library for all 120 tasks.

"Make sure all the white mugs are upright so that the opening is facing upwards."

"Put all plastic bottles away in the bin."

"Put the orange measuring cup and the blue measuring cup outside of the plate."

Requirements

Dependency	Version
Isaac Sim	5.0
Isaac Lab	2.2.0
Python	3.11
Linux	Ubuntu 22.04+

• Disk space: ~8 GB (assets account for ~7 GB)
• GPU: NVIDIA RTX GPU required. Recommend 48GB+ VRAM. See Isaac Lab's hardware requirements for recommended GPUs and VRAM.
• Speed: 30 GPU hours / 100 tasks, 1.4 it/s (assuming ~200ms inference step)

License

The RoboLab framework is released under CC-BY-NC-4.0.

Citation

@misc{yang2026robolab,
      title={RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies},
      author={Xuning Yang and Rishit Dagli and Alex Zook and Hugo Hadfield and Ankit Goyal and Stan Birchfield and Fabio Ramos and Jonathan Tremblay},
      year={2026},
      url={https://arxiv.org/abs/2604.09860},
}

Contributing

See CONTRIBUTING.md for acknowledgements, issues, and how to contribute.