fix(base): Publish command output at most every two seconds

[balamurali27]

parse_output sent the whole output so far to Redis on every line and on every \r progress-bar update. Redis writes each of these commands to its AOF in full, so the AOF grows with the square of the output. On a test server the AOF is 384 MB, but the data in Redis is only 1 MB.

On a server where the agent is on a small root disk, this filled the disk. Redis then refused all writes with MISCONF Errors writing to the AOF file: No space left on device, and a New Site from Backup job failed in update_redis.

This change publishes command output at most every two seconds. The final output is still published after the loop and by log(). ImageBuilder and PatchImageBuilder had two copies of their own one-second throttle. They now use the same Throttle from base.py. Throttle is a class, not a closure, because RQ pickles the builder object when it enqueues Run Remote Builder and Run Patch Build. A closure attribute makes that pickle fail. A test checks this.

Blast radius

• Every agent job that runs a command. parse_output is used by execute(), so this covers site, bench, server, backup and restore steps. Only the live output in Redis changes. Step status, return values, jobs.sqlite3 data and callbacks do not change. The final output is published the same way as before.
• Live output in Press. Press polls job output every five seconds (poll_pending_jobs), so the output shown is at most about seven seconds old, instead of about five.
• Output gap when a command goes quiet. The throttle fires on the leading edge. If a command prints a line inside the two-second window and then prints nothing more, that line reaches Redis only with the next line or when the command ends. The builders already had this behavior.
• Bench retry in Press. cancel_and_retry_bench_job_if_required looks for TLS handshake timeout and Retrying in 10 seconds in the live output of the running Initialize Bench step. Because of the gap above, Press can see these lines later. In the worst case, when Docker goes silent right after the line, Press sees it only when the step ends.
• Image builds. Build and push output is now published every two seconds instead of every second. It uses the monotonic clock instead of the wall clock, and the first update goes out at once instead of after one second. The build progress parsers in Press (docker_output_parsers.py) get updates less often.
• Not changed: job enqueueing, job results and statuses, the /jobs API format.

An AOF that is already large does not shrink by itself. Run BGREWRITEAOF once on affected servers.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VTnJ7zBU5AidQfFREeTExQ

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[greptile-apps]

Confidence Score: 5/5

[Medium risk] Throttles command output publishing to reduce Redis load.

The PR appears safe to merge; no new actionable issue was found.

_{Reviews (3) · Last reviewed commit: "docs(base): Note that Throttle drops cal..."}