Aniketh/arc - #573
Open
AnikethCheluva wants to merge 87 commits into
Open
Conversation
…pling
Port of the GR00T arc-length tokenizer (gr00t branch
rpunamiya/arc-length-tokenizer) adapted to EgoVerse conventions:
- egomimic/rldb/zarr/arc_length_tokenizer.py: chunks each arm's EEF
trajectory into fixed translational arc-length units, resamples to M
waypoints uniform in arc length (pos/gripper lerp, rotation SLERP via
scipy on ypr ZYX euler instead of rot6d), and appends per-arm
translational-velocity channels for timing reconstruction. Operates on
the canonical (T, 14) bimanual cartesian layout -> (M, 14 + 2*vel_dim);
per-arm independent arc length; zero tokens for stationary arms;
>=1e8 invalid-pose sentinel handling; all four velocity modes
(mean_scalar, mean_per_dim, per_step_scalar, per_step_per_dim);
deploy-time detokenize back to a time-indexed chunk (mean modes);
optional per-arm joint chunks resampled along that arm's arc length;
duck-typed Transform adapter (numpy/scipy only, no torch import).
- egomimic/rldb/zarr/arc_length_stats.py: data-driven
{min,max,mean,std,q01,q99} stats for the synthesized trans_vel
channels, computed by tokenizing raw (T, 14) chunks (sentinel chunks
skipped).
- 67 colocated unit tests covering arc-length math, SLERP correctness,
velocity payloads per mode, per-arm independence, round-trip recovery,
curved-path fidelity, joint resampling, stats, and the adapter.
Verified numerically equivalent (machine epsilon) to the GR00T reference
implementation on shared trajectories, including per-dim payloads,
per-dim detokenize, and joint resampling.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…isodeResolver, legacy embodiment aliases
New arc-tokenizer transform:
* arc_length_tokenizer.py: TokenizeBimanualArcLengthCartesian +
detokenize helper for the (M+1, 8) bimanual layout
[Lx, Ly, Lz, L_grip, Rx, Ry, Rz, R_grip] with per-dim mean vel
token appended at slot M.
* eva.py / human.py: new keymap_mode=arc_tokenizer_cartesian
(widened raw horizon so joint arc length has room to reach D)
+ get_transform_list mode with min_distance_unit and
resampled_vector_length args. Human pipeline zero-pads grippers
upstream of the tokenize step.
Also:
* LocalFolderEpisodeResolver: folder-of-zarrs resolver used by the
arc_tests_cotrain data yaml — no SQL, filters by zarr
attrs.embodiment.
* embodiment.py: alias legacy vendor embodiment strings
(mecka_bimanual, aria_bimanual, ...) to human_* so pre-collapse
caches load without a re-download.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ult)
Model head consumes (M+1, 8) arc-token layout from get_item — 15
waypoints + 1 per-dim vel token, xyz + gripper per arm.
* hpt_cotrain_mecka_flow_shared_head_arc.yaml: base FMPolicy head
at action_horizon=M+1, act_dim=8, act_seq=M+1, infer_ac_dims=8
(both embodiments). Human gripper columns are zero-padded upstream.
* hpt_cotrain_mecka_flow_shared_head_arc_D20_M15.yaml: overrides
for the D=20cm/M=15 data config.
* arc_tests_cotrain_arctok.yaml: data yaml using arc_tokenizer_cartesian
keymap + transform on the arc_tests folder (290 eva + 580 human).
* eval_arctok.py + eval_arctok.yaml: val-video eval that runs
TokenizeBimanualArcLengthCartesian.detokenize on both prediction
and GT before viz — reconstructs a time-parameterized (H, 14) chunk
using the predicted vel token to set duration, then hands it to the
canonical _viz_traj path so overlay dots project the same way as any
non-arc cotrain val video.
* eval_arctok_D20_M15.yaml: D=0.20, M=15 override matching the data
config.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Interactive HTML tools for eyeballing what the arc tokenizer produces
before wiring it into a training run:
* arc_embedding_viz.py: single-run t-SNE (or PCA/UMAP) over
positions_delta / velocity / flatten features from a tokenized
dataset. Front-camera thumbnail per point with green GT-action
overlay; click a point to enlarge in a floating card.
* arc_embedding_sweep.py: sweep over (joint_distance_cm × waypoints)
with one tabbed t-SNE per (D, W). Widened raw window
(raw_horizon = max(200, 2.5*max_D + 1)) so joint arc length can
reach large D without _pad_sequences drowning the tail. Per-tab
HDBSCAN + cluster stats sidebar. Partial-D toggle hides
end-of-episode samples that reach <50% of D. Cross-embodiment
KNN grid view: per HDBSCAN cluster, one medoid per embodiment
plus the 4 nearest opposite-embodiment neighbors, rendered as a
scrollable grid with image + overlay at higher resolution.
TSNE/KNN switch via a top-bar button that does not touch the
default TSNE UI.
* arc_embedding_baseline.py: classic time-window (H frames of
positions_delta) baseline for side-by-side comparison against
the arc-token sweep. Uses class-default read horizons so the
baseline chunk matches the cotrain model input exactly.
Supporting configs:
* hydra_configs/arc_embedding*.yaml: sweep grids
(D=[20,40,60,80] W=[5-30] default; extended D up to 280cm).
* hydra_configs/data/{aria_eva_fold_clothes_rl2,
aria_fold_clothes_lab, human_mecka_eva_rl2_fold_clothes,
human_mecka_eva_rl2_folding_clothes, arc_tests_cotrain}.yaml:
filtered / capped population variants for the sweep.
* hydra_configs/embedder/{pca,tsne,umap}_{2d,3d}.yaml: embedder
presets so the base arc_embedding.yaml can defaults-select
dimensionality reducer.
* scripts/embedding_process/arc_embedding{,_sweep,_baseline}.sbatch:
PACE L40S/CPU-small submissions.
* hydra_configs/hydra/launcher/submitit_pace_l40s.yaml:
launcher variant used for the cotrain runs.
* .gitignore: skip out/*.html + arc_*_*.html at repo root so
generated HTMLs never sneak into the branch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Switch data/arc_tests_cotrain{,_arctok}.yaml from LocalFolderEpisodeResolver
to S3EpisodeResolver so the episode set resolves from the SQL table and syncs
from R2 into ${paths.dataset_dir} on its own. The PACE folder the local
resolver depended on (/storage/project/r-dxu345-0/shared/arc_tests) does not
exist on OSMO, and LocalFolderEpisodeResolver explicitly does not sync.
Per-leaf DatasetFilter uses lab='rl2' and task='fold_clothes', split by
embodiment -> 290 eva_bimanual + 572 human_bimanual, verified against
app.episodes. The eva filter matches the one
human_mecka_eva_rl2_fold_clothes.yaml already uses.
Add osmo/arc_cotrain_l40s.yaml: 1 node x 8 L40S on groot-l40s-03, no submitit
(Lightning DDP runs the ranks in-container), per-GPU batch_size 32 -> 4,
num_workers 6 -> 4. Norm stats are computed once on 1 GPU and consumed by the
8-rank run, because trainHydra.py computes them unguarded by rank and ddp
re-executes the script per rank -- 8x redundant work plus a write race on
norm_stats.json otherwise. That phase also serializes the dataset download.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Hydra rejects `group=null` on the command line for config groups
("Config group override must be a string or a list. Got NoneType"),
which killed both runs at config composition. The delete syntax is what
the repo already uses elsewhere (~evaluator), and both instantiate_loggers
and instantiate_callbacks treat a missing group as empty.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both variants submitted under the same hardcoded `egoverse-arc-cotrain`,
so they were only distinguishable by the trailing counter in
`osmo workflow list`. Name is now a {{job_name}} template var; convention
is arc-<variant>-<hardware>.
Workflow names are immutable after submit and `osmo workflow tag` returns
403 for this profile, so already-submitted runs keep the old name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both 8xL40S runs died at wandb.init() with "CommError: permission denied":
logger/wandb.yaml hardcodes entity rl2-group, and the key in use only has
access to nv-gear. That happens in phase 2, so each failure burned the
~155GB dataset pull and the norm-stats phase first (~2.5h).
Add a {{wandb_entity}} override and check entity access right after
setup_secret.sh, before any of the expensive work, so a bad key fails in
seconds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
api.viewer.teams returns plain strings, not objects, so `t.name` raised AttributeError and the preflight killed the run it was meant to protect (arc-tok-d20m15-8xl40s-1, 2.5min in). Handle both shapes. Verified against a live key that both paths behave: exit 0 when the entity is reachable, exit 1 with the accessible list when it isn't. cpu 100 -> 80: most groot-l40s-03 nodes sit at 84/126 free, so 100 only matched the emptiest few and cost ~6-8 min of queueing. 64 dataloader workers still fit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cotrain commands carry ddp_find_unused_parameters_true, which costs a graph traversal per step and suppresses the error that would reveal a parameter silently not training. Reading the model did not settle whether it is needed: per-domain heads are null (only the shared head registers), use_domain_embedding is dead config, forward_training loops both domains with a single backward, and crossattn_modality_dropout is plain dropout rather than modality dropout. Run the real model under plain ddp on 4 episodes / 2 GPUs with TORCH_DISTRIBUTED_DEBUG=DETAIL so DDP names any parameter that receives no gradient. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eval mode sets cfg.logger = None (trainHydra.py:186), so the Valid/* metrics the evaluator computes via log_dict had nowhere to go, and trainer.validate()'s return value was discarded. Dump both it and callback_metrics to eval_metrics.json in the run dir. Add osmo/eval_ckpts_l40s.yaml: pulls a training run's checkpoints from R2, then runs the matching evaluator over each one. Baseline uses eval_hpt, arctok uses eval_arctok_D20_M15 -- the latter detokenizes prediction and GT back to (B, 100, 14) cartesian via the predicted velocity token, so its MSE is in the same physical units as the baseline's and the two are directly comparable. Results are pushed to R2 after each checkpoint so a late failure keeps earlier ones. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
OSMO rejects disproportionate requests: 1 of 8 GPUs caps CPU at 15. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Lightning writes "epoch_epoch=199.ckpt". The tag sanitizer kept '=', so the sweep passed description=epoch_epoch=199 and hydra rejected it with "mismatched input '=' expecting <EOF>" on the first checkpoint. Also guard the eval_metrics.json lookup: under set -e a failing find in a command substitution aborts the sweep, so one bad checkpoint took down the whole job instead of being skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Exporting AWS_DEFAULT_REGION=auto for the R2 checkpoint pull also redirected boto3's Secrets Manager call to secretsmanager.auto.amazonaws.com, so create_default_engine failed and S3EpisodeResolver never resolved -- every checkpoint errored. Scope the R2 creds to an r2() wrapper instead. The job still exited 0 because the trailing summary succeeded, so OSMO reported COMPLETED on a run that produced nothing. Assert at least one checkpoint was captured. Add FIRST_CKPT so the latest epoch shared by both runs is evaluated first; remaining checkpoints then run newest-first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
trainHydra runs infer_norm_from_dataset before branching on mode, so eval paid the full sample_frac=1.0 pass: ~1.33M samples at ~350/s, roughly 63 minutes per checkpoint (10.5h for a 10-checkpoint sweep). Nothing was failing -- the job was just spending all its time here, which read as a hang because no results appeared. The training run's norm_stats.json is already synced alongside its checkpoints, so pass it via precomputed_norm_path and assert it exists. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sanitizing the description was only half the problem. hydra's override grammar rejects a second '=' anywhere in an override including the value, and lightning names checkpoints "epoch_epoch=1599.ckpt", so ckpt_path=<that> raised OverrideParseException immediately. Only last.ckpt -- the sole filename without '=' -- ever got past parsing, and that one then sat in the norm-stat recompute, which is why the sweep looked like it was running for hours while producing nothing. Symlink each checkpoint to an '='-free path and point ckpt_path at that. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
viz/cartesian.yaml wires annotation_key: sampled_prompt for every
embodiment and eval_hpt inherits it, but arc_tests has no annotations, so
viz_gt_preds raised KeyError('sampled_prompt') at the first validation
step. eval_arctok.yaml already documents and fixes this for the arc path;
do the same on the command line for the baseline path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
find -newer | head -1 matched any recent eval_metrics.json, so a checkpoint that failed to write one silently inherited the previous checkpoint's result under its own tag. In one sweep 5 of 7 published results were duplicates of just two real evals; only the ckpt_path embedded in each JSON revealed it. Pin hydra.run.dir per checkpoint and read that exact file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ArcTokEvalVideo calls .contiguous() on the detokenized tensors, but the subsequent advanced-index slices do not inherit it, and torchmetrics' MSE uses .view() internally. pred_slice[:, -1] (the final_mse metric) is a strided view, so it raised "view size is not compatible with input tensor's size and stride" and took down every arc-tok eval. Reproduced standalone: of the four metrics only final_mse fails, which is why this survived until the detokenized path was actually exercised -- ~evaluator meant it had never run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every eval bug so far took 35-70min to surface because the 141GB dataset pull sits in front of the code under test. debug_episodes=N caps each resolver so the whole path -- checkpoint load, evaluator attach, trainer.validate, eval_metrics.json, R2 push -- runs in minutes on 4 episodes. 'debug' is not in the resolver config block, so it needs a '+' prefix; omitting that is what killed the earlier diagnostic. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
EvalVideo defaults limit_val_batches=400 and renders validation video frames each batch (~45s/batch), so every checkpoint eval was a ~5 hour job -- the sweeps were never hung, just doing 400 batches of video work. It is a constructor arg that flows into override_dict, and trainHydra applies override_dict AFTER the CLI (trainHydra.py:181), so trainer.limit_val_batches from the command line was silently discarded. Set it on the evaluator instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
limit_val_batches is an EvalVideo constructor arg and appears in no evaluator yaml, so a plain override raises "Key 'limit_val_batches' is not in struct" -- the same failure the resolver's debug key had. Add it with a leading + so hydra appends rather than overrides. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two separate issues, both found by round-tripping a real human episode: 1. dt: the tokenizer was constructed with its 1/30 default while the human chunk is subsampled by actions[::stride]. At stride=3 that inflates the velocity channel by exactly 3.00x (measured). It cancels within tokenize->detokenize, so it does not affect the val overlay or the detok MSE, but it is what the model learns and what a deployed policy would command. Pass dt=stride/30. 2. stride: an arc-length model can only reconstruct on the sample grid, so a fixed D=0.20m arc yields ~D/(v*dt) points. stride=3 gives ~6 points at 3.3cm spacing; stride=1 gives ~17 at 1.0cm with lower reconstruction RMSE (0.1645 vs 0.1920). eva is unstrided, which is why only human overlays looked sparse. Zero-velocity degeneration is 0.0% at both strides. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sequence is M+1=26 rows; model act_seq/action_horizon set to 26 to match. stride=1 for the same reason as the s1 variant -- an arc-length model can only reconstruct on the sample grid. Measured on a real human episode (unique reconstruction points / spacing): D=0.20 M=15 stride=3 ~8 2.56cm D=0.20 M=15 stride=1 ~18 1.00cm D=0.40 M=25 stride=1 ~34 0.93cm, 0.314m arc span Zero-velocity degeneration 0.0% in all cases. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A time-indexed baseline chunk and an arc token cover different amounts of motion, so per-timestep MSE compares them over different distances. On a real human episode the baseline's 100 steps travel 1.887m while the arc token spans 0.188m -- a 10x mismatch that alone produced the implausible 130x "arc-tok wins" on human. Add arc_matched_resample(): resample a (T, 14) chunk to M samples spaced uniformly in arc length over the first D metres, per arm, reusing the tokenizer's own cumulative_arc_length/resample_by_distance so both sides run identical code. Clamps to actual distance travelled when a chunk is shorter than D rather than padding a held endpoint. Wired via EvalVideo ctor (arc_match_distance/arc_match_points) so both evaluators inherit it; ArcTokEvalVideo overrides _arc_match_source to detokenize first. Verified on real data: 1.887m vs 0.188m becomes 0.1878m vs 0.1874m over 15 matched points. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
So the baseline can be compared against the stride=1 arc runs without stride being confounded with the tokenizer. InterpolatePose subsamples by stride then interpolates to a fixed new_chunk_length=100, so stride changes neither the output shape nor the time window (ACTION_HORIZON=30 raw frames either way). It only changes how many real samples back the interpolation: 10 at stride=3, 30 at stride=1. No model config change needed -- act_seq stays 100. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
16 data configs at stride=1 plus 4 model configs (act_seq = M+1, which is asserted by ArcTokEvalVideo._detokenize_batch). Measured on a real human episode, detokenize can only emit ~D/(v*dt) distinct points, so M above that stores waypoints the reconstruction cannot express: D=0.10 -> 7 recon pts (M>=25 over-parameterized) D=0.20 -> 18 (M=100 over-parameterized) D=0.40 -> 32 (M=100 marginal) D=0.80 -> 100 (all M fine) Zero-velocity degeneration is 0.0% in every cell -- this is a representation ceiling, not degenerate data. The over-parameterized cells are kept as the negative control. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
OSMO requires the resource request to be proportional to GPU count, so a 4-GPU run on an 8-GPU node may not exceed half the node (cpu<=63). The hardcoded 8-GPU values are rejected at submit. Template them, and add max_epochs so sweep cells can rank configs without paying the full 2000. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
egomimic/eval/dfot/ and egomimic/eval/tf/ move to the DFoT PR stacked above. egomimic/eval/__init__.py loses their entries. That registry is not lazy despite its name -- it import_module()s every entry at package-import time, so 'import egomimic.eval' was pulling the whole DFoT tree in. eval/core/img_utils.py mentions the DFoT evaluators only in docstring :mod: cross-references, not imports, so core is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The tests covering the preceding layers, plus the scripts worth keeping: the CUDA kernel build, ops/, the eval entrypoint, and the sim replay tooling. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
They exercise egomimic.algo.diffusion / egomimic.eval.dfot, which move to the DFoT PR stacked above. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
No test referenced MultiDataModuleWrapper at all, which is why the missing pack_collate dispatch survived: test_packed_pipeline covers the *other* pack_collate call site (norm-stat inference), so the subsystem looked tested. Asserts the dispatch itself and that it reaches the DataLoader, for both the train and valid loaders, and that unpacked datasets still get annotation_collate. Uses MagicMock(spec=ZarrEpisodePackedDataset) -- the same pattern test_packed_pipeline already uses -- so it needs no dataset on disk. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All four groups were stale tests, not broken code. Nothing under egomimic/ is touched by this commit. test_training_recipe (7): PackedAlgoBase takes an assembled outer_stage now -- action_dim / action_horizon / d_model / cond_encoder / hnet moved onto HNetOuterStage. The tests still passed them flat, so __init__ raised TypeError for the missing positional. They now build the HNetOuterStage and pass it; what each test asserts (lr stamping, parameter_groups, init_weights_range) is unchanged. test_pi (4): skipped at module level, not repaired. They describe a PI.visualize_preds API that exists nowhere -- no branch of this repo defines it (main included), egomimic.algo.pi.algo exposes no draw_actions to monkeypatch, and neither EgoVerse2 nor EgoVerse-gmm-dualstream implements it, so it was never ported into this lineage rather than dropped from it. Kept rather than deleted so the intent stays on record. (Their first error was a stale aria_bimanual domain, renamed to human_bimanual by the human/eva collapse; fixing that only exposed the missing method underneath.) test_core_defaults_byte_identical (1): the tx forward checksum was never a valid invariant. tx's output cancels from an absolute sum of 7163 down to a signed sum of ~1e-5, so the value moves with BLAS reduction order -- measured -1.21e-05 at one thread vs -4.43e-05 at two, on identical weights -- while the test compared it to a fixed ~1e-6 absolute bound. The reference simply captured one machine's rounding noise; refreshing it would fail again elsewhere. The guard is now the ABSOLUTE sum, stable to ~1e-8 relative across thread counts and still sensitive to any real forward change, with the signed sum kept at a tolerance scaled to the magnitude summed. lstm and hnet were unaffected because their sums are O(1) and O(100) and do not cancel. test_packed_pipeline (1): PUSHT_FOLDER pointed one directory level too high. circle/ now holds a basic/ subfolder rather than .zarr episodes and the resolver does not recurse, so the isdir() skip-guard passed while the resolver matched nothing. Same drift as data/pushshapes/packed_episode/simulation/delta.yaml. Correcting it also un-skipped three dataset-gated tests, which pass. Suite: 13 failed / 297 passed -> 0 failed / 309 passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
data, experiment, evaluator and callbacks groups, plus every model config the
rest of the tree actually references.
An earlier pass kept 3 model configs. Scanning hydra_configs, tests and scripts
for names that resolve against the model group finds 62 referenced, so that
prune broke two things at once:
* all 31 experiment configs -- each carries an `override /model:`, so hydra
fails at `Could not find 'model/<name>'` before any code runs;
* 25 cases in tests/test_config_compose.py, covering the dfot_*, vae_* and
bc_rnn_pushshapes_paperexact_* families.
The 59 missing configs are restored here, bringing the model group to the 62
that are reachable from the tree. Two of them (bf_rh_sdp_dual,
bf_rh_sdp_nodual) are the only configs that instantiate egomimic/pipeline, so
without them the batchflow runner shipped unreachable.
The remaining 181 per-arm variants on elmo/batchflow-core are still
deliberately excluded -- nothing in this tree references them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…amilies
The model group was 62 flat files, 9,581 lines, with exactly one config
(pi0.5_bc_eva) using defaults-inheritance. Everything else was authored by
copying a whole file and editing a few numbers.
Folders. Each model family gets its own directory and the filename drops the
prefix the folder already carries:
hnet_cotrain_cossim_s4_trunk_200M.yaml
-> hnet_cotrain_cossim/s4.yaml (model=hnet_cotrain_cossim/s4)
64 configs, 10 folders, none left at the root. Every reference was rewritten:
8 in-repo files plus 26 scratch launchers outside the repo. Old -> new mapping
is saved at scratch/name_mapping_full.txt, since model names appear in the
Results Ledger and run pages as provenance.
Deduplication. Two families were literal copies differing only in numbers, and
both are now base + thin variants:
* hnet_cotrain_cossim -- 12 configs, 270 of 304 lines identical across all of
them (89%). The whole family is one 5-stage skeleton with a different split
of transformer depth between the per-embodiment levels and the shared apex.
Now 8 knobs; each variant is ~13 lines. 3,642 -> 961 lines.
* hnet_dualstream_txar -- 3 configs, 9 knobs. txar_m16 turns out to differ
from txar in k_agnostic/k_specific alone (3/2 -> 10/6).
Model group overall: 9,581 -> 6,414 lines.
The knobs are top-level scalars referenced by interpolation rather than
overridden in place, because outer_stage.hnet.stages is a YAML *list* and
OmegaConf replaces lists wholesale on merge -- a variant cannot override one
element of it.
Defaults entries need @_here_ (e.g. `- hnet_cotrain_cossim/base@_here_`).
Without it Hydra derives the package from the folder path and merges the base
under model.hnet_cotrain_cossim instead of the model root; the config still
composes and the tests still pass, it just silently inherits nothing.
The other seven families are NOT factored. Each has line-count differences
between members, i.e. they are structurally different models that share
boilerplate rather than copies of one template, so the same mechanical proof
does not apply. Duplication left: dfot 31%, bc_rnn 58%, vae 62%,
hnet_pushshapes 39%, hpt 34%, 2trunk 44%, bf_rh 39%.
Verified: parameterization proven lossless by reconstruction (11/11 cossim,
3/3 txar reproduce the originals byte-for-byte) before anything was written;
all 31 experiment configs compose; tests/test_config_compose.py 25 passed; and
the resolved model config of every one of the 62 pre-existing configs was
compared across the rename -- 48 identical, 14 differing only by the added
knob keys, 0 real differences.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
data/pushshapes/packed_episode/simulation/delta.yaml pointed at
Tsim_datasets2/circle, which today contains a single basic/ subdirectory and
zero .zarr episodes. LocalEpisodeResolver does not recurse (AGENTS.md:139 says
so explicitly), so the config resolved to an empty dataset and training died
with:
No valid collection names from local filtering: filters matched no
episodes in the local directory.
The episodes live one level down. Both the train and valid resolver paths now
point at circle/basic.
Verified: the composed config resolves to a directory containing 311 .zarr
episodes (previously 0); all 32 experiment configs still compose.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
evaluator/dfot, evaluator/tf and model/dfot move to the DFoT PR stacked above. No experiment config selects them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This stack added egomimic/scripts/eva_process/zarr_write_test.py while main has the same converter at egomimic/rldb/zarr/hdf5_to_zarr.py, so merging would have partly undone #562, which renamed it. Git recorded that as a rename, so a delete-oriented check across the stack missed it; content-level duplicate detection found the shared bodies (main 179 lines, is_image_array 38, needs_transpose_to_hwc 17). The copy here is also the older fork: it predates main's mandatory-intrinsics change and omits the intrinsics= kwarg entirely, so it would write zarr episodes with no camera calibration -- the exact failure that per-episode intrinsics were introduced to prevent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The chunk is (100, 138) = 2 x (21*3 + 6): wrist is 6-dim xyz+ypr with no gripper slot, not the 7 I assumed, so the space is 138 not 140. Confirm which block is which by bone-length test -- correct slicing yields anatomically sized bones that stay rigid across the chunk; wrong slicing does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Probed the live pipeline rather than assuming: actions_keypoints is (100, 138) = 2 * (21*3 + 6). The wrist carries xyz+ypr only -- aria has no gripper and nothing pads one on this path -- so the space is 138, not the 140 I had, and the wrist block precedes the keypoints within each hand. Confirmed by bone-length test, which cannot be fooled: slicing as [Lwrist6|Lkp63|Rwrist6|Rkp63] yields 39.7mm bones with 0.38mm variation across a chunk, while keypoints-first yields 330mm "bones" with 3.7m outliers. The velocity slot moves to the wrist block accordingly. Re-verified end to end at D=0.45/M=30: token (31, 138), keypoint reconstruction 0.200mm (linf), and bones survive the round trip at 39.1mm with 0.61mm variation, so the hand stays rigid. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds mode keypoints_wristframe_ypr_arctok, which runs the existing wrist-frame keypoint pipeline and then arc-tokenizes the (T, 138) chunk to (M+1, 138). dt = stride/30 for the same reason as the cartesian arc path. Two data configs on the SAME population as the cartesian runs (lab=rl2, task=fold_clothes), stride=1, with a real 20% holdout -- the existing aria_keypoints* configs set valid = train, so they cannot measure generalization. The pair differs only in parameterization: kp_human_baseline time-uniform, (100, 138) kp_human_arctok arc-uniform, (31, 138) at D=0.45 / M=30, linf Identical 138-dim action space, so this isolates tokenization rather than confounding it with action space. No model change needed for the baseline -- hpt_bc_keypoints_base is already single-domain with act_dim 138. The arctok variant only overrides act_seq to M+1 = 31. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The workflow hardcoded data.{train,valid}_dataloader_params.eva_bimanual.*,
so any human-only config died with "Key 'eva_bimanual' is not in struct" --
after the dataset pull and norm stats had already run. Build the overrides
from an EMBODIMENTS list instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
hpt_bc_keypoints_base sets lr=5e-5 with CosineAnnealingLR(T_max=1400). scheduler_interval defaults to "step", so T_max is 1400 STEPS ~ 14 epochs at the observed 100.8 steps/epoch -- and CosineAnnealingLR keeps evaluating cos(pi*T_cur/T_max) past T_max, so the LR climbs back up rather than stopping. Over 600 epochs that is ~22 sawtooth cycles between 1e-5 and 5e-5. Predicted 1.44e-05 at step 12299 vs 1.8e-05 observed on wandb. Every arc-cartesian run used constant lr 3e-4 with scheduler: null, so the keypoint runs as launched were neither constant nor at the same LR, and not comparable. Add const-LR variants. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_ARM_SLOTS_BY_DIM only knew 12/14, so the keypoint action space fell through the shape guard and returned None -- the same silent skip that hid the human arcmatch metric earlier. Add the 138 layout using the confirmed wrist-first ordering [L wrist6 | L kp63 | R wrist6 | R kp63]. Arc-matching runs on the WRIST pose deliberately: that is the same trajectory the cartesian runs arc-match, so keypoint models can be compared against them directly. Per-keypoint accuracy is a separate metric, keypoint_mse(), reporting mean L2 over all 42 joints in metres so it reads against the ~1.5mm reconstruction floor. Verified: (15, 8) output on 138-dim, self-consistency exactly 0 for identical inputs on both metrics, 12/14 unaffected, unsupported dims still rejected, and keypoint_mse correctly skips cartesian chunks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
viz/keypoints.yaml and viz/keypoints_wrist.yaml both use a bare front_img_1 image_key, but the batch carries observations.images.front_img_1 (confirmed by probe) -- so neither can run. Add viz/keypoints_human.yaml with the correct key, no eva entry (these runs are single-domain), and no annotation_key, since arc_tests has no annotations. The eval sweep hardcoded annotation_key=null for four cartesian viz entries; on a keypoint evaluator those keys do not exist and hydra errors. Drive them from an ANNOT_EMBODIMENTS list instead, same fix as the dataloader overrides. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Same bug as the training workflow, fixed there but not here: the sweep set data.valid_dataloader_params.eva_bimanual.*, so any human-only run died with 'Key eva_bimanual is not in struct'. Drive from EMBODIMENTS. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Third instance of the same assumption in this file. Drive it from EMBODIMENTS like the others. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Walks a real frame pair through the computation: two hand poses with per-joint displacement, the 21 distances as bars with L-inf/L2/L1-mean marked, the reduction to a scalar, the rotation term, and accumulation into tokens at D. Uses frames 2731->2737 where the index fingertip moves 37.9mm while the slowest joint moves 6.9mm, so the choice of norm is visible: L1-sum is 8.3x L-inf on that single step, which is what compounds into the ~22x path inflation over an episode. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…r_training zarr_key_to_keyname returns None for any batch key that isn't a registered action/proprio zarr key (intrinsics, episode_hash, image keys). The old 'if key is not None' guard checked the wrong variable — 'key' is a string from _batch.items() and never None, so the branch always fired and wrote every unregistered key under a single None dict slot. Later writes clobbered earlier ones, and 'intrinsics' was silently dropped. Downstream _intrinsics_from_batch(batch, i) then returned None, so Human.viz / Eva.viz fell back to the hardcoded class INTRINSICS constant. For episodes whose per-episode K disagrees with the aria default (mecka fx=fy≈251, cy≈184 vs ARIA cy=240), this projected the GT trajectory ~55px vertically offset — the visible wrist-vs-palm misalignment reported against arc_tests mecka fold_clothes val-videos. Fix: fall back to the original key when zarr_key_to_keyname is None so unregistered keys survive the rekey. Also fixes downstream access to episode_hash and per-episode image side-channel keys.
Layers on top of arc-length-nv-eval only what wasn't already there:
data configs (D=0.20m / M=15 hardcoded to match target convention):
- aria_train_mecka_val{,_arctok}: cross-domain (train aria fold, valid
mecka fold_clothes)
- eva_only_fold{,_arctok}: eva-only robot-baseline runs
- mecka_folding_clothes{,_arctok}: mecka folding_clothes (note "ing"
variant of task name — distinct from arc_tests fold_clothes)
- mecka_folding_eva_fold_cotrain{,_arctok}: cotrain across mismatched
task names (mecka folding_clothes + eva fold_clothes)
hydra launchers (mirror target's submitit_pace_l40s.yaml convention):
- submitit_pace_a100 / _blackwell / _h100
viz + diagnostic scripts:
- egomimic/visualization/arc_tok_viz.py: detokenize+overlay helper for
notebooks (mirrors ArcTokEvalVideo viz path)
- scripts/pixel_check.py + .sbatch: verify projected GT dots match
val-video mp4 frame 0
- scripts/visualize_trunk_latents.py + .sbatch: t-SNE + HDBSCAN over
HPTModel.forward_features, emits arc_embedding_sweep's tabbed HTML
small UI fix in scripts/arc_embedding_sweep.py: image-panel close
button now has type=button + inline onclick fallback.
Dropped from the pre-rebase stash because target already had them
(and more evolved): eval_arctok/eval_hpt/eval_video changes,
arc_tests_cotrain* config edits, D40_M100 model/evaluator configs.
Dropped as no-longer-wanted per user note: FMPolicyWithVelDecoder /
WithVelReadout wrappers, associated hpt.py / denoising_policy.py /
hpt_nets.py / model config _veldec / _velreadout variants. Also
dropped an accidental LocalFolderEpisodeResolver -> S3EpisodeResolver
revert and a stray _LEGACY_EMBODIMENT_ALIASES removal.
Merges origin/arc-length-nv-eval (arc-length tokenizer + arc-tok evaluators + keypoint tokenizer + arc model configs + osmo PACE configs) and the two rpunamiya/arc-length-tokenizer deltas (hpt batch-keys preservation fix, PACE launchers + task splits + viz) onto the batchflow config stack (bf/1..bf/7 config reorg). Conflicts resolved by keeping bf's structural refactors as the skeleton and porting arc's functionality into that layout: - eval_video.py: keep both bf's max_videos and arc's arc_match_* + video_chunk_frames constructor params. - zarr_dataset_multi.py: honor both skip_bounds_check (bf) and bounds_check (arc) semantics on the bounds guard. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…configs
Ports the D=40cm / M=100 arc-tok model configs and required class code
onto the bf/7-configs config layout so the in-flight training jobs can
launch from this branch:
- Restore hpt_cotrain_enc_dec_base.yaml with _target_ paths pointing at
bf's module layout (algo.hpt.algo.HPT, models.stems.hpt_stems.*,
models.heads.hpt_heads.MultiBlockTransformerDecoder,
models.diffusion.denoising_nets.CrossTransformer).
- Point hpt_cotrain_mecka_flow_shared_head_arc.yaml back at
hpt_cotrain_enc_dec_base and update its _target_ paths.
- New: hpt_cotrain_mecka_flow_shared_head_arc_D40_M100{,_veldec,_velreadout}.yaml.
- Add MLPVelocityDecoder to egomimic/models/heads/hpt_heads.py and
FMPolicyWithVelDecoder / FMPolicyWithVelReadout to
egomimic/models/heads/fm_policy.py.
- Fix eval_arctok.yaml viz path (cartesian → cartesian/base).
- Add ``arc_tokenizer`` config group defaults to train_zarr_cartesian.yaml
so ``arc_tokenizer.min_distance_unit=…`` CLI overrides resolve.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- pushshapes arc-length tokenizer (egomimic/rldb/zarr/pushshapes_arc_tokenizer.py)
and pushshapes.get_keymap_hpt_arc for reading (M+1, 2) arc-tok windows.
- HPT closed-loop inference: expand_arc_chunk_to_time so the (M+1, D) chunk is
played back as a time-uniform buffer against the env; replan_at tracks the
variable expanded-buffer length.
- hydra config groups for the pusht arc-tok stack:
data/pushshapes/pusht/{circle,circle_arc}.yaml, evaluator/hpt/{pusht,pusht_arc}.yaml,
model/pusht/*, model/pusht_arc/*.
- Restored hpt_cotrain_mecka_flow_shared_head.yaml (non-arc baseline) with
bf/7-configs _target_ paths so mecka baselines resume from aniketh/arc.
- logger/wandb/base.yaml default project: zarr_test -> arc so arc-tok runs land
in rl2-group/arc automatically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adapts robot/rollout.py for arc-tokenized policies. Subclasses PolicyRollout to reuse the loader / obs transform / safety pipeline; overrides only the predict step to detokenize the model's (M+1, 8) arc-tok output into a (H, 14) time-uniform chunk before handing it to the controller.
Rewrites TokenizeBimanualArcLengthCartesian to produce (M+1, 14) with full xyz + ypr + grip per arm (was (M+1, 8) dropping rotation). Rotation is unconditionally supervised — no opt-out. Gripper padding for human aria data preserved as an existing transform option. - arc_length_tokenizer.py: ARC_TOK_PER_ARM_DIM=7, ARC_TOK_BIMANUAL_DIM=14; SLERP for waypoint ypr resample; vel row extended to 14 dims with per-axis mean angular velocity in ypr slots. - Model configs (arc, D20_M15, D40_M25, D40_M100, M15/M25/M50/M100 + veldec + velreadout): act_dim 8 -> 14, infer_ac_dims 8 -> 14, veldec output_dim 8 -> 14. - FMPolicyWithVelDecoder / FMPolicyWithVelReadout: act_dim default 14. - eval_arctok.py, rollout-arc.py, visualization/arc_tok_viz.py: shape asserts 8 -> 14; removed zero-fill of ypr; detokenize returns 14-dim including model-predicted rotation. - Data configs (arc_sweep_*, arc_tests_cotrain_arctok*, folding_clothes _arctok, folding_eva_fold_cotrain_arctok, eva_only_fold_arctok, aria_train_mecka_val_arctok): comment updates. - Embodiment keymap/transforms: docstring updates only; routing intact.
Claude Code ReviewReviewSummaryThis PR is titled "Aniketh/arc" but bundles a massive, unrelated collection of changes: a full H-Net algorithm implementation, packed dataloading, versioned PushShapes simulators ( Key concerns
Suggestions
Reviewed by Claude · Review workflow |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.