Add progress-event callbacks for evaluation requests

README.md

@@ -56,6 +56,17 @@ GLOBAL OPTIONS:
 
    --auth-key value, -k value  the authentication key to use for incoming requests. [$AUTH_KEY]
 
+   progress
+
+   --progress-callback-timeout value          the timeout for a single progress callback delivery. (default: 1s) [$PROGRESS_CALLBACK_TIMEOUT]
+   --progress-allowed-hosts value [ --progress-allowed-hosts value ]  restrict progress callback URLs to these hosts. Entries may be an exact hostname or a "*.example.com" wildcard. Unset allows any host, subject to the private-network guard below. [$PROGRESS_ALLOWED_HOSTS]
+   --progress-allow-private-networks          allow progress callback delivery to loopback, link-local, and private IP addresses. Leave disabled unless the callback target is known to live on a trusted private network. (default: false) [$PROGRESS_ALLOW_PRIVATE_NETWORKS]
+   --progress-sidecar-max-body-bytes value     the maximum size, in bytes, of a single worker-authored progress event POST. (default: 16384) [$PROGRESS_SIDECAR_MAX_BODY_BYTES]
+   --progress-sidecar-max-events value         the maximum number of worker-authored progress events relayed per evaluation. (default: 50) [$PROGRESS_SIDECAR_MAX_EVENTS]
+   --progress-sidecar-burst-size value          how many worker-authored progress events at the start of an evaluation are exempt from the minimum spacing below, so a handful of legitimate back-to-back checkpoints aren't rate limited. (default: 5) [$PROGRESS_SIDECAR_BURST_SIZE]
+   --progress-sidecar-min-event-interval value  the minimum spacing between worker-authored progress events relayed per evaluation, once the burst allowance above is used up. (default: 10ms) [$PROGRESS_SIDECAR_MIN_EVENT_INTERVAL]
+   --progress-sidecar-unbind-grace-period value  how long to keep relaying worker-authored progress events after a request returns, so a fire-and-forget POST dispatched just before the result can still land. (default: 250ms) [$PROGRESS_SIDECAR_UNBIND_GRACE_PERIOD]
+
    function
 
    --arg value, -a value [ --arg value, -a value ]  additional arguments for to the worker process. [$FUNCTION_ARGS]
@@ -185,6 +196,110 @@ Example request using cases:
 }
 ```
 
+### Progress Events
+
+The shim also exposes a µEd-compatible endpoint at `POST /evaluate` (see the [µEd spec](https://mued.org/spec)), separate from the legacy `POST /` endpoint documented above. When a client calls `/evaluate` with a `callbackUrl` in the request body, the shim POSTs a small JSON event to that URL at each stage of processing — in addition to, not instead of, the normal synchronous HTTP response.
+
+This lets a caller show progress to the end user (e.g. "Evaluating your submission…") without polling, and without the shim needing to hold a connection open. It works identically whether the shim is deployed standalone or on AWS Lambda.
+
+To opt in, include `callbackUrl` in the request body and, optionally, an `X-Request-Id` header — both are part of the µEd spec's own request contract, not shim-specific additions. Every event echoes back the `X-Request-Id` value verbatim so the caller can correlate it with the original request.
+
+```json
+{
+  "submission": { "type": "TEXT", "content": { "text": "..." } },
+  "task": { "referenceSolution": { "text": "..." } },
+  "callbackUrl": "https://your-service.example.com/hooks/shimmy-progress"
+}
+```
+
+Four stages are emitted, in order, for a successful evaluation:
+
+| Stage | Meaning |
+|-------|---------|
+| `preparing` | The evaluation environment is being set up (a worker is ready — freshly booted or reused). |
+| `evaluating` | The evaluation function is being invoked. |
+| `completed` | Feedback has been computed. `data.feedback` carries the same array returned in the synchronous response body. |
+| `failed` | A terminal failure occurred at some stage. `message` is safe to show to an end user; `error` carries raw technical detail for logs only. |
+
+`completed` and `failed` are terminal — at most one of them is delivered per request, whichever occurs first.
+
+Example event body:
+
+```json
+{
+  "correlationId": "req-7c193f38",
+  "stage": "evaluating",
+  "command": "eval",
+  "message": "Evaluating your submission…",
+  "timestamp": "2026-08-04T14:23:01.512Z"
+}
+```
+
+Example terminal event, with the feedback payload attached:
+
+```json
+{
+  "correlationId": "req-7c193f38",
+  "stage": "completed",
+  "command": "eval",
+  "message": "Feedback is ready.",
+  "data": {
+    "feedback": [
+      { "awardedPoints": 1, "message": "Well done" }
+    ]
+  },
+  "timestamp": "2026-08-04T14:23:02.310Z"
+}
+```
+
+Delivery is best-effort and never blocks or fails the evaluation itself: each callback POST is bounded by `--progress-callback-timeout` (default `1s`, see [Usage](#usage)); a slow, unreachable, or erroring receiver is logged and skipped, never surfaced to the caller as an evaluation failure.
+
+#### Callback URL safety (SSRF protection)
+
+Since `callbackUrl` is caller-supplied, the shim guards against it being used to reach services it shouldn't be able to reach:
+
+- **By default**, callback delivery refuses to dial loopback, link-local (this includes cloud metadata endpoints like `169.254.169.254`), and private (RFC1918/RFC4193) IP addresses — checked against the address actually resolved and dialed, not just the URL's literal hostname, so a public-looking domain that resolves to a private address is still blocked. Set `--progress-allow-private-networks` only if the callback target is known to live on a private network you trust (e.g. a same-VPC service).
+- **`--progress-allowed-hosts`** optionally restricts callback URLs to an explicit list of hostnames (exact match, or `*.example.com` wildcards). Unset means any (non-private) host is accepted.
+
+A rejected callback URL behaves like any other delivery failure: it's logged and skipped, never surfaced to the caller as an evaluation failure.
+
+> **Note:** the µEd spec describes `callbackUrl` for asynchronous *final-result* delivery — the service may return `202 Accepted` immediately and POST the result later. The shim doesn't implement that 202 flow; it always responds synchronously with `200 OK` and the feedback body as normal. It reuses the same `callbackUrl` field to additionally deliver progress events — including the final feedback, via the `completed` event's `data` field — rather than requiring a shim-specific header for the same concept.
+
+#### Custom progress events from the evaluation function
+
+The four stages above are emitted by shimmy itself, around the evaluation function call as a whole — `evaluating` covers the entire invocation as one span. An evaluation function that does multiple steps internally (e.g. several model calls) can emit its own progress events *during* that span, which are relayed through the same `callbackUrl` alongside shimmy's own events.
+
+When a request opts in to progress reporting (via `callbackUrl`), shimmy starts a loopback-only HTTP listener and passes its address to the evaluation function process as the `EVAL_PROGRESS_URL` environment variable, the same way it passes `EVAL_RPC_TRANSPORT`, `EVAL_FILE_NAME_REQUEST`, etc. (see [Communication Channels](#communication-channels) below). This works identically regardless of interface (`rpc` or `file`) or RPC transport, and regardless of the evaluation function's language — it only needs to be able to make an HTTP POST.
+
+To emit a custom event, `POST` a small JSON body to `EVAL_PROGRESS_URL`:
+
+```json
+{
+  "message": "Checking correctness…",
+  "data": { "step": 2, "of": 3 }
+}
+```
+
+- `message` (string, required): student/teacher-facing text.
+- `data` (object, optional): free-form, passed through as-is.
+- There is no `stage` field, by design: an evaluation function can never claim `preparing`, `evaluating`, `completed`, or `failed` — those remain exclusively shim-authored. Custom events are always delivered with `"stage": "progress"`.
+
+The response status is informational only — the evaluation function should treat every response as fire-and-forget and never fail on a non-2xx status. Delivery is best-effort, same as outbound callback delivery: `202` accepted (delivery to `callbackUrl` is then attempted in the background), `400` malformed body or empty `message`, `413` body too large, `429` rate limited, `503` no request currently associated with the listener (e.g. a stray POST arriving after both the request has finished and the grace period below has elapsed).
+
+To bound how much an evaluation function (which may be running untrusted, sandboxed code) can push through this channel, events are capped before relay:
+
+| Flag | Env var | Default | Description |
+|------|---------|---------|-------------|
+| `--progress-sidecar-max-body-bytes` | `PROGRESS_SIDECAR_MAX_BODY_BYTES` | `16384` | Maximum size, in bytes, of a single event POST. |
+| `--progress-sidecar-max-events` | `PROGRESS_SIDECAR_MAX_EVENTS` | `50` | Maximum number of events relayed per evaluation. |
+| `--progress-sidecar-burst-size` | `PROGRESS_SIDECAR_BURST_SIZE` | `5` | Events at the start of a span exempt from the minimum spacing below, so a handful of legitimate back-to-back checkpoints aren't rate limited. |
+| `--progress-sidecar-min-event-interval` | `PROGRESS_SIDECAR_MIN_EVENT_INTERVAL` | `10ms` | Minimum spacing between relayed events, once the burst allowance is used up. |
+| `--progress-sidecar-unbind-grace-period` | `PROGRESS_SIDECAR_UNBIND_GRACE_PERIOD` | `250ms` | How long the listener keeps relaying after a request returns, so a fire-and-forget event POST dispatched by the worker just before returning its result still has a window to land. |
+
+> **Sandboxing note:** under `--sandbox` alone, the worker keeps the host network namespace and can reach the loopback listener normally. Only the separate, explicit `--sandbox-disable-network` flag isolates networking (and loopback specifically) — under that flag, custom progress events are silently dropped, the same as any other best-effort delivery failure.
+
+This is a shim-side contract only; no client library ships in this repo. Evaluation function libraries (e.g. per-language toolkits) can build a thin wrapper around reading `EVAL_PROGRESS_URL` and POSTing to it.
+
 ### Communication Channels
 
 The shim supports two interface modes, selected with `--interface`:
@@ -211,6 +326,7 @@ The shim injects the following environment variables into the evaluation functio
 | `EVAL_RPC_HTTP_URL` | HTTP URL (HTTP transport only) |
 | `EVAL_RPC_WS_URL` | WebSocket URL (WS transport only) |
 | `EVAL_RPC_TCP_ADDRESS` | TCP address (TCP transport only) |
+| `EVAL_PROGRESS_URL` | Local URL to POST [custom progress events](#custom-progress-events-from-the-evaluation-function) to (only set when the request opted in via `callbackUrl`) |
 
 #### File System (`--interface file`)
 
@@ -236,6 +352,7 @@ The shim also sets the following environment variables:
 | `EVAL_IO` | `FILE` |
 | `EVAL_FILE_NAME_REQUEST` | Path to the input file |
 | `EVAL_FILE_NAME_RESPONSE` | Path to the output file |
+| `EVAL_PROGRESS_URL` | Local URL to POST [custom progress events](#custom-progress-events-from-the-evaluation-function) to (only set when the request opted in via `callbackUrl`) |
 
 > Using the file interface is recommended for large payloads such as base64-encoded images.
 

cmd/root.go

@@ -43,6 +43,62 @@ functions on arbitrary, serverless platforms.`
 				Category: "auth",
 				EnvVars:  []string{"AUTH_KEY"},
 			},
+			// progress flags
+			&cli.DurationFlag{
+				Name:     "progress-callback-timeout",
+				Usage:    "the timeout for a single progress callback delivery.",
+				Value:    time.Second,
+				Category: "progress",
+				EnvVars:  []string{"PROGRESS_CALLBACK_TIMEOUT"},
+			},
+			&cli.StringSliceFlag{
+				Name:     "progress-allowed-hosts",
+				Usage:    "restrict progress callback URLs to these hosts. Entries may be an exact hostname or a \"*.example.com\" wildcard. Unset allows any host, subject to the private-network guard below.",
+				Category: "progress",
+				EnvVars:  []string{"PROGRESS_ALLOWED_HOSTS"},
+			},
+			&cli.BoolFlag{
+				Name:     "progress-allow-private-networks",
+				Usage:    "allow progress callback delivery to loopback, link-local, and private IP addresses. Leave disabled unless the callback target is known to live on a trusted private network.",
+				Value:    false,
+				Category: "progress",
+				EnvVars:  []string{"PROGRESS_ALLOW_PRIVATE_NETWORKS"},
+			},
+			&cli.Int64Flag{
+				Name:     "progress-sidecar-max-body-bytes",
+				Usage:    "the maximum size, in bytes, of a single worker-authored progress event POST.",
+				Value:    16 * 1024,
+				Category: "progress",
+				EnvVars:  []string{"PROGRESS_SIDECAR_MAX_BODY_BYTES"},
+			},
+			&cli.IntFlag{
+				Name:     "progress-sidecar-max-events",
+				Usage:    "the maximum number of worker-authored progress events relayed per evaluation.",
+				Value:    50,
+				Category: "progress",
+				EnvVars:  []string{"PROGRESS_SIDECAR_MAX_EVENTS"},
+			},
+			&cli.IntFlag{
+				Name:     "progress-sidecar-burst-size",
+				Usage:    "how many worker-authored progress events at the start of an evaluation are exempt from the minimum spacing below, so a handful of legitimate back-to-back checkpoints aren't rate limited.",
+				Value:    5,
+				Category: "progress",
+				EnvVars:  []string{"PROGRESS_SIDECAR_BURST_SIZE"},
+			},
+			&cli.DurationFlag{
+				Name:     "progress-sidecar-min-event-interval",
+				Usage:    "the minimum spacing between worker-authored progress events relayed per evaluation, once the burst allowance above is used up.",
+				Value:    10 * time.Millisecond,
+				Category: "progress",
+				EnvVars:  []string{"PROGRESS_SIDECAR_MIN_EVENT_INTERVAL"},
+			},
+			&cli.DurationFlag{
+				Name:     "progress-sidecar-unbind-grace-period",
+				Usage:    "how long to keep relaying worker-authored progress events after a request returns, so a fire-and-forget POST dispatched just before the result can still land.",
+				Value:    250 * time.Millisecond,
+				Category: "progress",
+				EnvVars:  []string{"PROGRESS_SIDECAR_UNBIND_GRACE_PERIOD"},
+			},
 			// shim flags
 			&cli.StringFlag{
 				Name:     "interface",
@@ -317,20 +373,28 @@ func parseRootConfig(ctx *cli.Context) (config.Config, error) {
 
 	// map cli flags to config fields
 	cliMap := map[string]string{
-		"auth-key":                   "auth.key",
-		"max-workers":                "runtime.max_workers",
-		"command":                    "runtime.cmd",
-		"cwd":                        "runtime.cwd",
-		"arg":                        "runtime.arg",
-		"env":                        "runtime.env",
-		"interface":                  "runtime.io.interface",
-		"rpc-transport":              "runtime.io.rpc.transport",
-		"rpc-transport-ipc-endpoint": "runtime.io.rpc.ipc.endpoint",
-		"rpc-transport-http-url":     "runtime.io.rpc.http.url",
-		"rpc-transport-ws-url":       "runtime.io.rpc.ws.url",
-		"rpc-transport-tcp-address":  "runtime.io.rpc.tcp.address",
-		"worker-send-timeout":        "runtime.send.timeout",
-		"worker-stop-timeout":        "runtime.stop.timeout",
+		"auth-key":                             "auth.key",
+		"progress-callback-timeout":            "progress.callback_timeout",
+		"progress-allowed-hosts":               "progress.allowed_hosts",
+		"progress-allow-private-networks":      "progress.allow_private_networks",
+		"progress-sidecar-max-body-bytes":      "progress.sidecar.max_body_bytes",
+		"progress-sidecar-max-events":          "progress.sidecar.max_events_per_span",
+		"progress-sidecar-burst-size":          "progress.sidecar.burst_size",
+		"progress-sidecar-min-event-interval":  "progress.sidecar.min_event_interval",
+		"progress-sidecar-unbind-grace-period": "progress.sidecar.unbind_grace_period",
+		"max-workers":                          "runtime.max_workers",
+		"command":                              "runtime.cmd",
+		"cwd":                                  "runtime.cwd",
+		"arg":                                  "runtime.arg",
+		"env":                                  "runtime.env",
+		"interface":                            "runtime.io.interface",
+		"rpc-transport":                        "runtime.io.rpc.transport",
+		"rpc-transport-ipc-endpoint":           "runtime.io.rpc.ipc.endpoint",
+		"rpc-transport-http-url":               "runtime.io.rpc.http.url",
+		"rpc-transport-ws-url":                 "runtime.io.rpc.ws.url",
+		"rpc-transport-tcp-address":            "runtime.io.rpc.tcp.address",
+		"worker-send-timeout":                  "runtime.send.timeout",
+		"worker-stop-timeout":                  "runtime.stop.timeout",
 		// sandbox
 		"sandbox":                 "runtime.sandbox.enabled",
 		"sandbox-nsjail-path":     "runtime.sandbox.nsjail_path",

config/config.go

@@ -1,6 +1,9 @@
 package config
 
-import "github.com/lambda-feedback/shimmy/runtime"
+import (
+	"github.com/lambda-feedback/shimmy/internal/progress"
+	"github.com/lambda-feedback/shimmy/runtime"
+)
 
 type MessageEncoding string
 
@@ -25,4 +28,7 @@ type Config struct {
 
 	// Auth is the authentication configuration
 	Auth AuthConfig `conf:"auth"`
+
+	// Progress is the configuration for outbound progress-callback delivery
+	Progress progress.Config `conf:"progress"`
 }

handler/module.go

@@ -1,6 +1,11 @@
 package handler
 
-import "go.uber.org/fx"
+import (
+	"go.uber.org/fx"
+
+	"github.com/lambda-feedback/shimmy/config"
+	"github.com/lambda-feedback/shimmy/internal/progress"
+)
 
 func Module() fx.Option {
 	return fx.Module("common",
@@ -10,5 +15,7 @@ func Module() fx.Option {
 		fx.Provide(NewHealthRoute),
 		fx.Provide(NewMuEdEvaluateRoute),
 		fx.Provide(NewMuEdEvaluateHealthRoute),
+		fx.Provide(func(cfg config.Config) progress.Config { return cfg.Progress }),
+		fx.Provide(progress.NewHTTPFactory),
 	)
 }