add sweep scripts, do some runs

[rohan-bansal]

No description provided.

[rohan-bansal]

@claude review this PR

[claude]

Claude finished @rohan-bansal's task in 2m 33s —— View job

---

PR Review: add sweep scripts, do some runs

• Gather context and read changed files
• Review source code changes (scheduling, metrics)
• Review new experiment scripts
• Review config and requirements changes
• Post final review feedback

---

Overview

This PR adds three Modal-based sweep scripts for fairness/starvation experiments, two offline plotting scripts, refactors DynamicActionScheduler to match the updated base-class interface, and adds an infinite-loop guard in mirror.py. The changes are mostly research scaffolding with a handful of bugs worth addressing before this lands.

---

Bugs

1. modal_starvation_sweep.py — broken import (will fail at runtime)

# scripts/exps/modal_starvation_sweep.py (bottom of main())
sys.path.insert(0, str(pathlib.Path(__file__).parent))
from plot_sweep import DEFAULT_METRICS, plot_results  # noqa: PLC0415

There is no plot_sweep.py in scripts/exps/ — only plot_starvation_sweep.py. This will raise ModuleNotFoundError every time a sweep completes locally. Either rename plot_starvation_sweep.py → plot_sweep.py or fix the import to from plot_starvation_sweep import .... Fix this →

2. mirror.py:162-170 — infinite-loop guard is a known hack

# src/armory/scheduling/mirror.py
# NOTE Rohan: hack from Claude. fix properly
if not any(
    chunk.action_index_start
    <= step.next_action_step
    <= chunk.action_index_start + chunk.execution_horizon - 1
    for chunk in self.chunks
):
    return step.time

The comment is honest, but a silent early return here can silently produce a wrong finish_time rather than surfacing the invariant violation as an error. Consider at least logging a warning (or raising in debug builds) so cases that hit this path are visible. Should also be tracked as a follow-up issue. Fix this →

3. dynamic_action.py:42 — redundant in-flight check

base.py:77-82 already snapshots in_flight_batches_count and passes candidates into get_next_batches, but the base class doesn't gate on in-flight count itself — that gating lives in each subclass. The dynamic_action.py implementation checks self.mirror.in_flight_batches_count > 0 while also calling self.mirror.deadlines() (an extra mirror query on every call). This is consistent with baselines.py, so it works — but dynamic_action.py calls self.mirror.deadlines() unconditionally even when it returns early due to in-flight batches (line 50 is after the early return, so this is fine). Low severity.

---

Issues / Concerns

4. calculate_starvation_offline.py — hardcoded personal path

episode = Path("/coc/flash7/rbansal66/vvla/0_real_0_failure")

This is a one-off debugging script with an absolute path to a personal cluster filesystem. It won't run for anyone else and carries no generalizability. Consider removing it from the repo or parametrizing it (argparse or sys.argv[1]) before merging.

5. requirements-modal.txt — cmake downgraded from 4.3.1 → 3.30.0

-cmake==4.3.1
+cmake==3.30.0

This is a significant version rollback. Was this intentional (e.g., incompatibility discovered)? If so, a comment in the requirements file or PR description would help future readers understand why the newer version was abandoned.

6. generate_per_robot_starvation_rate_gif included in generate_all_plots

# src/sims/libero/metrics.py
generate_per_robot_starvation_rate_gif,  # slow (~5-10s per run); run manually if needed

The comment says "run manually if needed" but it's unconditionally in generate_all_plots. Anyone calling generate_all_plots in CI or automation will pay the cost. Either remove it from generate_all_plots or add an explicit opt-in flag. Fix this →

7. modal_alpha_fairness_sweep.py — orphaned comment on MAX_BATCH_SIZE

# MAX_BATCH_SIZE = 4
MAX_BATCH_SIZE = 20

The 5× increase relative to the fairness sweep (MAX_BATCH_SIZE = 4) is meaningful — it changes throughput/fairness dynamics. A short inline comment explaining the intent (e.g., "unconstrained batch to isolate alpha fairness effect") would help.

---

Minor / Style

8. Inconsistent subprocess logging across sweep scripts

modal_alpha_fairness_sweep.py uses a streaming thread (_stream_to_log_and_stdout) that tees subprocess output to Modal's container log in real time. modal_fairness_sweep.py and modal_starvation_sweep.py use subprocess.run which buffers everything until the process ends — so if a run hangs, you get no log output until timeout. Not a correctness bug, but the streaming approach is strictly more debuggable. Consider unifying.

9. inference_profiles.json — missing newline at EOF

The original file lacked a trailing newline and the PR doesn't add one. Minor, but most editors/linters prefer a final newline.

---

What Looks Good

• The refactor of compute_starvation_variance_series (splitting the data extraction from the plotting) is clean and makes the function reusable for sweep aggregation.
• SweepCase dataclasses are frozen and have sensible run_id properties — easy to reproduce individual runs.
• Artifact tarball handling (filtering .mp4/.parquet/.npz, timestamped + latest CSV outputs) is thoughtful.
• The _infer_deadline / _score signature updates in dynamic_action.py correctly align with the updated base class interface in base.py.