Waldo

Where's Waldo? Right here, finding objects in your video.

Self-hosted ML platform for video object detection at scale. Auto-label any object in video using text prompts or click-based exemplars (powered by SAM 3), train YOLO26 detectors on the labeled data, monitor training live, and deploy the model to a serving endpoint or edge device.

📘 Full documentation: docs-site/ — quickstart, UI tour with screenshots and short walkthrough videos, API reference, deployment guides. Build it locally with cd docs-site && npm install && npm run start.

Architecture

                    ┌──────────────────────────────────────────┐
                    │              React UI (SPA)              │
                    └──────────────┬───────────────────────────┘
                                   │
                    ┌──────────────▼───────────────────────────┐
                    │          FastAPI  (waldo-app)            │
                    │   REST API  ·  WebSocket  ·  Static UI   │
                    └──┬───────────────┬───────────────┬───────┘
                       │               │               │
              ┌────────▼──┐    ┌───────▼───┐    ┌──────▼──────┐
              │  Celery   │    │  Celery   │    │             │
              │  Labeler  │    │  Trainer  │    │   Infra     │
              │  (SAM 3)  │    │  (YOLO)   │    │             │
              └───────────┘    └───────────┘    │ PostgreSQL  │
                                                │ Redis       │
                                                │ MinIO       │
                                                └─────────────┘

Pipeline: Upload video → Label (text or click) → Review → Train YOLO → Deploy API

Quickstart

Step 1 — Get a Hugging Face token. Waldo uses SAM 3 to do the labeling. Create a read-only token at huggingface.co/settings/tokens and accept the license on the facebook/sam3 model page. You'll paste the token into the installer.

Step 2 — Install. One command. Picks the right Docker profile for your platform (NVIDIA / Apple MPS / CPU), installs missing prereqs, pulls the prebuilt image from Docker Hub, and brings the stack up — no local build, no Node, no Vite.

# macOS / Linux / WSL
curl -fsSL https://raw.githubusercontent.com/oldhero5/waldo/main/install.sh | bash

# Windows (PowerShell)
irm https://raw.githubusercontent.com/oldhero5/waldo/main/install.ps1 | iex

:: Windows (cmd.exe)
curl -fsSL https://raw.githubusercontent.com/oldhero5/waldo/main/install.cmd -o install.cmd && install.cmd && del install.cmd

The installer prompts for the HF token early so you can walk away. To skip the prompt, pass it on the command line or via env:

curl -fsSL https://raw.githubusercontent.com/oldhero5/waldo/main/install.sh \
  | bash -s -- --hf-token hf_xxxxxxxxxxxxx
# or:  HF_TOKEN=hf_xxx curl -fsSL ... | bash

The Windows wrappers install/verify WSL2 + Docker Desktop, then hand off to install.sh inside Ubuntu — that's where Waldo actually runs.

Step 3 — Sign in. Open http://localhost:8000. Default dev credentials are printed at first boot:

email: admin@waldo.ai
password: waldopass (dev default — override with ADMIN_BOOTSTRAP_PASSWORD for production)

Step 4 — Label your first video. From the UI: Upload → drag in a clip → Auto-label → type a prompt (e.g. car) → Preview. The first preview takes ~2 minutes (it downloads the SAM 3 weights to a Docker volume); subsequent calls are sub-second. Then walk through Review → Train → Deploy.

Already cloned the repo?

git clone https://github.com/oldhero5/waldo.git && cd waldo
./install.sh                                 # full install (will prompt for HF_TOKEN)
./install.sh --skip-up --skip-models --yes   # config only, no model download
./install.sh --build-from-source             # build the image locally (contributors)

Installer flags

Flag	Default	What it does
`--hf-token TOKEN`	(prompt)	Hugging Face read token — required for SAM 3 weights
`--dir PATH`	`~/waldo`	Where to clone if Waldo isn't already on disk
`--branch NAME`	`main`	Branch to clone
`--cpu`	off	Force CPU even if a GPU is detected
`--gpu nvidia\|apple\|none`	auto	Override GPU detection
`--build-from-source`	off	Build the image locally instead of pulling (slow; requires Node)
`--skip-prereqs`	off	Don't install Docker/uv — assume present
`--skip-models`	off	Don't download SAM 3 weights
`--skip-up`	off	Don't run `docker compose up` — config only
`--yes`	off	Non-interactive (HF_TOKEN can be set later in `.env`)

PowerShell uses the same flags PascalCased (-HfToken, -Dir, -SkipUp, -Yes, ...).

Manual setup (for the curious)

The installer is the recommended path, but Waldo is just docker-compose.yml

a published image underneath. If you want to drive it by hand:

git clone https://github.com/oldhero5/waldo.git && cd waldo
cp .env.example .env                     # add HF_TOKEN
make up                                  # pulls oldhero5/waldo:latest, auto-routes by OS

make up auto-detects your platform and picks the right compose profile: Darwin → apple, Linux/WSL with nvidia-smi → nvidia, otherwise → cpu. You can still override explicitly with PROFILE=....

Platform	Command	Auto-detected profile	Image	Workers in Docker?	GPU
macOS (Apple Silicon)	`make up`	`apple`	`oldhero5/waldo:latest` (app only)	❌ native	Apple MPS
Linux + NVIDIA	`make up`	`nvidia`	`oldhero5/waldo:cuda`	✅	CUDA
Linux (CPU only)	`make up`	`cpu`	`oldhero5/waldo:latest`	✅	none
Windows (WSL 2) + NVIDIA	`make up`	`nvidia`	`oldhero5/waldo:cuda`	✅	CUDA
Force a specific profile	`make up PROFILE=nvidia` (or `cpu`/`apple`)	—	per profile	per profile	per profile

The apple profile is also kept as an alias for cpu for one release, so old make up PROFILE=apple invocations on Linux keep working.

To build from source instead of pulling — useful for contributing or running unreleased changes:

make build PROFILE=nvidia    # builds Dockerfile.cuda locally and tags :dev-cuda
make build                   # builds Dockerfile (CPU) locally and tags :dev

macOS (Apple Silicon)

Apple's MPS cannot be passed through to Linux containers. Two options:

make up          # Everything in Docker, CPU workers. Slowest but zero setup.
make up-mac      # Infra + app in Docker, native MPS workers. Recommended.

make up-mac starts the labeler and trainer natively so they can reach the M-series GPU. Logs land in /tmp/waldo-labeler.log and /tmp/waldo-trainer.log. (make up-gpu is kept as a back-compat alias.)

Linux with NVIDIA CUDA

The installer handles the NVIDIA Container Toolkit on apt/dnf systems. If you'd rather drive it manually:

# 1. Driver + toolkit (Ubuntu example — see NVIDIA docs for others)
sudo apt install -y nvidia-driver-550
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# 2. Verify the GPU is visible to Docker
make gpu-check

# 3. Start Waldo with the nvidia profile
make up PROFILE=nvidia

# 4. Confirm the workers see the GPU inside the container
make gpu-logs

The nvidia profile pulls oldhero5/waldo:cuda, which is built from Dockerfile.cuda on top of nvidia/cuda:12.4.0-devel-ubuntu22.04 and installs CUDA-enabled PyTorch from download.pytorch.org/whl/cu124 (PyPI ships CPU-only torch by default — a common silent-failure trap). The Apple path uses MLX, which is macOS-only. Training uses CUDA bf16 for speed.

Each nvidia worker runs scripts/entrypoint-worker.sh on boot, which prints nvidia-smi + torch.cuda.is_available() so make gpu-logs immediately shows whether passthrough is working. Compose also sets shm_size: 4gb so PyTorch dataloader IPC doesn't OOM on the 64 MB Docker default.

Windows with NVIDIA CUDA (via WSL 2)

Windows GPU support goes through WSL 2 + Docker Desktop. The NVIDIA driver lives on the Windows host; CUDA inside WSL is provided by the driver automatically — do not install a Linux CUDA driver inside WSL.

The PowerShell installer (install.ps1) handles WSL2 + Docker Desktop and then hands off to install.sh inside Ubuntu. If you'd prefer to drive it manually, the chain is the same: install the Windows NVIDIA driver, install WSL2 + Ubuntu, install Docker Desktop with WSL integration, then run the Linux installer (or make up PROFILE=nvidia) inside Ubuntu.

Windows gotchas:

Keep the repo in the WSL filesystem (~/waldo, not /mnt/c/...). Cross-filesystem I/O is 10–20× slower, and ffmpeg frame extraction becomes the bottleneck.
Line endings: configure git to keep LF (git config --global core.autocrlf input) so shell scripts don't break.
File watcher limits: if make dev-ui misses changes, raise the inotify limit: echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
Memory: Docker Desktop defaults to half of host RAM. Training YOLO on 720p video wants ≥ 16 GB allocated — adjust in %UserProfile%\.wslconfig:
```
[wsl2]
memory=24GB
processors=8
```
then wsl --shutdown and restart.

Waldo has not been validated against native Windows (no WSL). Don't try it — Ultralytics, Celery's solo pool, and ffmpeg all behave differently there.

Linux without a GPU (CPU only)

make up          # Auto-selects the cpu profile (CPU-only image)

Works for small datasets and smoke tests. Don't expect to train on real video.

Verify it's running

http://localhost:8000    # Waldo UI
http://localhost:9001    # MinIO console  (minioadmin / minioadmin)

The app runs database migrations automatically on startup.

Common commands

make up              # Pull image, start everything
make pull            # Just pull (refresh after a release)
make build           # Build the image locally (contributors only)
make logs            # Tail all containers
make down            # Stop everything
make dev-ui          # Vite dev server with hot reload (proxies API)
make build-ui        # Production build → app/static/ (only useful for build-from-source)
make migrate         # Run Alembic migrations
make test            # Python test suite
make download-models # Download SAM 3 weights

Web UI

Page	Path	Description
Upload	`/upload`	Drag-and-drop video upload
Label	`/label/:videoId`	Text search + click mode, 5 task types
Review	`/review/:jobId`	Annotation reviewer with hotkeys
Train	`/train/:jobId`	Variant picker, hyperparameters, live metrics
Deploy	`/deploy`	Endpoints, test console, model registry, monitoring
Agent	`/agent`	Local LangGraph agent (Ollama + `gemma4:e4b`) — ask in plain English, it calls real Waldo tools to inspect data and run jobs on your behalf

AI Agent

/agent ships a LangGraph ReAct agent backed by a local Ollama (no third-party API). It can list your projects/datasets/models, recommend training settings, and — with the action toggle on — start labeling jobs, launch training runs, and activate models.

The compose stack runs Ollama as a sidecar service and an ollama-init one-shot pulls gemma4:e4b (~9.6 GB) on first boot. After that, chats stream in real time. See the Agent docs for the tool list, SSE event shapes, and troubleshooting.

API

All endpoints live under /api/v1 and are documented at /docs (OpenAPI). Highlights:

Method	Path	Description
`POST`	`/upload`	Upload video
`POST`	`/label`	Start text-prompt labeling
`POST`	`/label/exemplar`	Start click-based labeling
`POST`	`/label/preview`	Async prompt preview (202 + poll `/job/{id}`)
`GET`	`/job/{job_id}`	Generic Celery task result polling
`GET`	`/status/{job_id}`	Per-frame labeling progress
`POST`	`/train`	Start training run
`GET`	`/train/{run_id}`	Status + metrics + loss history
`WS`	`/ws/training/{run_id}`	Live metrics stream
`POST`	`/predict/image?model_id=ID`	Image inference
`POST`	`/predict/video?model_id=ID`	Video inference with tracking
`GET`	`/models`	List trained models

Example pipeline

Long-running endpoints (label, label/preview, training, comparisons, exports) return 202 Accepted with a {job_id, status, result_url} envelope. Poll the result_url until the job is complete instead of blocking the connection. See API overview for full details.

# 1. Upload video
curl -X POST http://localhost:8000/api/v1/upload -F "file=@clip.mp4"

# 2. Try a prompt on a few frames before committing to the full job.
#    Returns 202 immediately with a job_id; poll /api/v1/job/{job_id}
#    until status == "completed" and read result.frames[].
curl -X POST http://localhost:8000/api/v1/label/preview \
  -H "Content-Type: application/json" \
  -d '{"video_id": "VIDEO_ID", "prompts": ["person"]}'
curl http://localhost:8000/api/v1/job/PREVIEW_JOB_ID

# 3. Kick off the full labeling job
curl -X POST http://localhost:8000/api/v1/label \
  -H "Content-Type: application/json" \
  -d '{"video_id": "VIDEO_ID", "text_prompt": "person", "task_type": "segment"}'

# 4. Poll for per-frame labeling progress (LabelingJob row)
curl http://localhost:8000/api/v1/status/JOB_ID

# 5. Train a YOLO model on the dataset
curl -X POST http://localhost:8000/api/v1/train \
  -H "Content-Type: application/json" \
  -d '{"job_id": "JOB_ID", "model_variant": "yolo26n-seg", "hyperparameters": {"epochs": 50}}'

# 6. Watch training (or open the Train page in the UI)
curl http://localhost:8000/api/v1/train/RUN_ID

# 7. Run inference against the trained model
curl -X POST "http://localhost:8000/api/v1/predict/image?model_id=MODEL_ID" \
  -H "Authorization: Bearer wld_YOUR_KEY" \
  -F "file=@test.jpg"

YOLO task types

SAM 3 always outputs segmentation masks. Waldo's converters reshape them into whatever format the selected YOLO task needs:

Task	Output	YOLO variants
Segmentation	Polygon vertices	`yolo26n-seg` → `yolo26x-seg`
Detection	Bounding boxes	`yolo26n` → `yolo26x`
Classification	Cropped images in class dirs	`yolo26n-cls` → `yolo26x-cls`
OBB	4 rotated corner points	`yolo26n-obb` → `yolo26m-obb`
Pose	Bbox + centroid keypoint	`yolo26n-pose` → `yolo26m-pose`

Configuration

Everything comes from environment variables. See .env.example.

Variable	Default	Description
`DEVICE`	`mps`	`mps` (Apple), `cuda` (NVIDIA), or `cpu`
`DTYPE`	`float32`	`float32`, `bfloat16`, `float16`
`SAM3_MODEL_ID`	`facebook/sam3`	HuggingFace transformers model (Linux/CUDA — needs `model.safetensors`)
`SAM3_MLX_MODEL_ID`	`mlx-community/sam3.1-bf16`	MLX variant (macOS Apple Silicon)
`HF_TOKEN`	—	HuggingFace read token (required) — accept the license on the `facebook/sam3` model page first
`ADMIN_BOOTSTRAP_EMAIL`	`admin@waldo.ai`	First admin's email
`ADMIN_BOOTSTRAP_PASSWORD`	`waldopass` (dev)	First admin's password — override in production
`SLACK_WEBHOOK_URL`	—	Training alerts
`NTFY_TOPIC`	—	Push notifications

Testing

# Unit tests (no infra)
uv run pytest tests/test_converters.py tests/test_frame_extractor.py -v

# Trainer tests (Redis + MinIO)
uv run pytest tests/test_trainer.py -v

# API tests (full stack running)
uv run pytest tests/test_api.py tests/test_api_extended.py -v

# Everything
make test && cd ui && npx playwright test

Project structure

waldo/
├── app/                    # FastAPI application
│   ├── main.py             # Entrypoint, routers, SPA fallback
│   ├── api/                # Route handlers
│   └── static/             # Built React UI
├── labeler/                # SAM 3 labeling pipeline
│   ├── sam3_engine.py      # PyTorch path (Linux/CUDA)
│   ├── video_labeler.py    # MLX path (macOS)
│   ├── text_labeler.py     # Text-prompt flow
│   ├── frame_extractor.py  # ffmpeg extraction + phash dedup
│   └── converters/         # Mask → YOLO format converters
├── trainer/                # YOLO training pipeline
│   ├── train_manager.py    # Ultralytics orchestrator
│   ├── dataset_builder.py  # Dataset prep from DB
│   ├── metrics_streamer.py # Redis pub/sub for live metrics
│   └── exporter.py         # ONNX, TFLite, CoreML export
├── lib/                    # Shared library
│   ├── config.py           # Pydantic settings
│   ├── db.py               # SQLAlchemy models
│   ├── storage.py          # MinIO client
│   └── tasks.py            # Celery tasks
├── ui/                     # React 19 + Vite + TypeScript + Tailwind
├── alembic/                # Database migrations
├── scripts/                # Setup + maintenance scripts
├── tests/                  # Python test suite
├── Dockerfile              # Unified image (app | labeler | trainer via $WALDO_ROLE)
├── Dockerfile.cuda         # Same image on a CUDA base + GPU torch wheels
├── docker-compose.yml      # All services; apple/cpu/nvidia profiles, pulls from Docker Hub
└── docker-compose.build.yml # Override that builds from source instead of pulling

Documentation

The full docs live in docs-site/ (Docusaurus). Highlights:

Quickstart — upload a clip, auto-label, train, deploy in ~15 minutes
UI Tour — every page screenshotted, with short walkthrough videos
Architecture — services, data model, security
API Reference — every REST endpoint grouped by resource
Workflow Blocks — composable blocks for the visual editor
Deployment — Docker, Linux, Windows, and edge devices

Run the site locally:

cd docs-site
npm install
npm run start          # http://localhost:3000

Or build static HTML with npm run build. Screenshots and videos can be regenerated against your local Waldo with npm run screenshots and npx playwright test scripts/recordings.spec.ts.

License

See LICENSE.

Name		Name	Last commit message	Last commit date
.claude		.claude
.github/workflows		.github/workflows
alembic		alembic
app		app
docs-site		docs-site
install/lib		install/lib
labeler		labeler
lib		lib
scripts		scripts
tests		tests
trainer		trainer
ui		ui
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
.gitleaks.toml		.gitleaks.toml
.gitleaksignore		.gitleaksignore
.mcp.json		.mcp.json
.pre-commit-config.yaml		.pre-commit-config.yaml
.python-version		.python-version
.secrets.baseline		.secrets.baseline
Dockerfile		Dockerfile
Dockerfile.cuda		Dockerfile.cuda
Dockerfile.precommit		Dockerfile.precommit
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
alembic.ini		alembic.ini
docker-compose.build.yml		docker-compose.build.yml
docker-compose.docs.yml		docker-compose.docs.yml
docker-compose.precommit.yml		docker-compose.precommit.yml
docker-compose.yml		docker-compose.yml
install.cmd		install.cmd
install.ps1		install.ps1
install.sh		install.sh
pyproject.toml		pyproject.toml
ruff.toml		ruff.toml
uv.lock		uv.lock

Navigation Menu

Repository files navigation

Waldo

Architecture

Quickstart

Already cloned the repo?

Installer flags

Manual setup (for the curious)

macOS (Apple Silicon)

Linux with NVIDIA CUDA

Windows with NVIDIA CUDA (via WSL 2)

Linux without a GPU (CPU only)

Verify it's running

Common commands

Web UI

AI Agent

API

Example pipeline

YOLO task types

Configuration

Testing

Project structure

Documentation

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Waldo

Architecture

Quickstart

Already cloned the repo?

Installer flags

Manual setup (for the curious)

macOS (Apple Silicon)

Linux with NVIDIA CUDA

Windows with NVIDIA CUDA (via WSL 2)

Linux without a GPU (CPU only)

Verify it's running

Common commands

Web UI

AI Agent

API

Example pipeline

YOLO task types

Configuration

Testing

Project structure

Documentation

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages