Run #1: Achieved a 52.4% resolve rate on SWE-bench Lite, using Claude Haiku 4.5 for context gathering and Sonnet 4.6 for implementation.

For-Loop Subsessions & Tool Calls

Agent subsessions are for-loops, with each iteration calling an LLM API to generate a json file that specifies the tool call the agent wants to execute and the parameters for it

1. An LLM can only produce text, so for each tool call, we have it generate a string-format json file, then use json.loads() to convert it from string to dict
   • In the LLM prompt, we inject prior conversation and tool call history
   • We also provide descriptions of each available tool in tools=json_schema
     • json_schema contains function name, expected params, and summary
     • All available tools are stored in registry.tools map, loaded at start up keyed by tool name to ToolSpec dataclass
   • Make sure temperature=0 for benchmark determinism
2. From the response object, we check if the finish_reason property equals “tool_calls”, if so, our single-sample completion request will produce response.choices[0].message.tool_calls — a list of tool call objects to loop over
   • For each tc in tool_calls, we can find the right function with tc.function.name and the right inputs with tc.function.arguments, a json-encoded string
3. Use tc.function.name as a key for registry_tools map, fetching ToolSpec object for that tool call, which has the python function as a property
   • The code of the tool call is run by spec.fn(args)
   • The actual function itself is either subprocess.run(cmd, shell=True) for bash, or open(path, “w\”).write() for write, and similar process for read
4. Collect the results of the tool call as a string, then append to the message history
   • We organize each tool call and corresponding result string by an ID number
   • Continue repeating loop until model responds with plain text instead of tool call
   • At loop turn=N, we can create logic to add/modify the system prompt to encourage specific behavior with the tool calls

The Node Graph Structure

Each node on the graph is a Node class that, after executing, returns another Node class that represents the next node in the sequence to traverse to. The node traversal is purely deterministic based on if-statements and implementations/test-results produced by LLM.

1. Build AgentState dataclass with ~40 fields of metadata (task info, retry counters, current node name, etc) – the information it stores triggers which node to be ran and with what data, it also acts as a place for nodes to hand off information to one another
   • Examples include tracking to-do lists of tests to be ran, gathered context bundles after reading relevant files, failure-type flags
   • AgentState is accompanied with serialization functions to_dict/from_dict so it can be persistently stored via SQLite
2. We use state.current_node as a string key into a node registry, a dict mapping name to node class, then node.run() triggers a subsession or deterministic result
   • node.run(state) outputs NodeResult(next_node, state_update) where state_update is a dict of {field_name: new_value}
   • We then iterate over each entry to update AgentState via state.apply_update()
3. In BlueprintEngine, we loop through running each node, gathering results, and switching to the next node until state.current_node equals “END”
   • next_node is outputted deterministically inside the node's own code, often based on the outcome of running tests or code the LLM wrote, but NOT an LLM's decision of whether it's done.
   • Example: Verify_Node routes to Implement_Node if F2P tests are still failing, but routes to Gather_Context_Node if P2P tests regressed

Harness Design

1. LOAD_TASK: read issue text and pass_to_pass tests, store in state.task_text and state.pass_to_pass, automatically set next_node to reproduce issue, we store but never use state.fail_to_pass until the very last official swe-bench eval
2. REPRODUCE_ISSUE:
   • One LLM call takes issue text and creates list of discrepancies between observed and expected behavior, creates {observed, expected} pairs
   • 70-turn subsession is launched to write one test case to objectively express each discrepancy, forming _repro_tests.py — tool set: bash, read, write
     • At turn=0, prompt encourages tool calls to explore issue relevant files
     • At turn=8, add a nudging statement to the prompt to encourage writing failing tests for each discrepancy
     • At turn=25, make grep call for each written test, searching for any sibling functions or call-sites that also need fixing
     • At turn=50, explicitly run pytest and confirm every written test actually fails
     • At turn=60, nudge to consider any edge cases unaccounted for
3. GATHER_CONTEXT:
   • Parse traceback text from failing repro tests for relevant functions and file-paths using regexes for python traceback frames, pytest format, and assert lines
   • These file-paths are seeded into a read-queue, GatherQueueState: queue (list), seen (set), notes (list), current (str|None), and line-numbers from the call-trace
   • 22-turn subsession is launched to dequeue these files and take notes on each one with an LLM call, if an adjacent relevant file is spotted, it is enqueued
     • Uses a new tool set: enqueue, dequeue, note, bash
   • on_finish_check blocks make sure that queue is empty before stopping, and that notes are not concentrated solely on the _repro_tests file
   • After the subsession, notes are grouped by file-path, "not relevant" ones dropped, the rest are organized into ContextBundle.task_adjacent_files
   • ContextBundle is a dataclass and task_adjacent_files is list[dict], a dict for each file the agent took noted – {“filepath”: … “why”: … “notes_content”: …}
   • The dicts are grouped by file-paths and sorted by line number for easy readability
4. IMPLEMENT:
   • 30-turn subsession is launched to resolve each failing test case given the context we’ve gathered in the previous step – tools: read, write, bash, replace
   • Once turns exceed 20, we call an LLM to compress all conversation history into a summary in the messages list except for the latest 6 turns, which stay verbatim
   • If Verify step fails, the Implement may get triggered up to 5 times total
     • 1st attempt: system prompt includes all failing tests and tracebacks
     • later attempts: current failing tests, regressed p2p tests, current git diff
   • Agent can also write its own scripts to reduce repetitive work or test suspicions
     • we also provide premade helper script mro_check.py which dumps a python class’s full method-resolution-order chain
     • import_graph.py is another premade helper that shows all dependencies a given file imports, as well as what other files depend on it (converse)
   • Once subsession finishes (iteration ends without a tool call), we go to Verify
5. VERIFY:
   • We run every failing test that we wrote in Reproduce Issue, as well as all the baseline pass_to_pass tests individually – for each test, we store the result and traceback to state.todo_list, which tracks our entire list of tests to pass
   • If we encounter a p2p test regression, we traverse back to Gather Context step to recollect information about relevant files on this new issue
   • If not all failing tests we wrote are passing, we traverse back to Implement step for the subsession to try again, with an extra nudge on currently failing cases
   • If all tests pass or max verify attempts exceeded, we traverse to the very last step, which is starting a separate container to run the official swebench eval
   • If tests are still failing after max attempts, we track which retry attempt yielded the most number of tests passing and submit that one for official evaluation
   • Verify step is purely deterministic code

Running SWE-Bench Evals

• SWE-Bench Lite spans 12 unique repos (astropy, django, flask, matplotlib…) across 300 instances, but each instance also pins a specific repo version, so the real unit of environment is (repo, version), not just repo, giving 64 distinct environments
• We pre-build all 64 envs once via setup_scripts/setup_env_cache.sh, store each as its own snapshot in Modal Volume swebench-envs-v2
  • This way, every instance run just mounts the matching (repo, version) snapshot instead of resolving deps fresh
  • This guarantees the exact same interpreter/compiler + exact same dependency versions every run, for every instance sharing that (repo, version) pair
• Retrieval is keyed off each instance's declared (repo, version) at LOAD_TASK time, used to pull the right cached snapshot out of the volume for that instance's container
• Each instance run is a Modal container that mounts the volume, does its work, then spins up a nested container for official eval scoring
  • Single instance: modal run modal_run.py::run_one --instance-id <id>
[--provider anthropic]
  • Subset by repo: modal run modal_run.py::run_repo --repo sympy/sympy
--limit 50 --concurrency 10
  • Full 300-instance suite: modal run modal_run.py::run_batch --ids-file
ids.txt --results-file test_traces/results_full300.jsonl
• --concurrency controls how many instances run in parallel (each holding ~2 containers, per the nested-eval-container point above), default 10 assumes an OpenAI Tier 2 key; lower-tier keys should drop to 4 or less to avoid rate-limit 429s

Citations

This project builds on SWE-bench and SWE-bench Lite. SWE-bench Lite is a 300-instance curated subset of the original benchmark introduced by the same authors (no separate paper), so both are cited via the original SWE-bench paper:

@inproceedings{
    jimenez2024swebench,
    title={{SWE}-bench: Can Language Models Resolve Real-world Github
Issues?},
    author={Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu
Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan},
    booktitle={The Twelfth International Conference on Learning
Representations},
    year={2024},
    url={https://openreview.net/forum?id=VTF8yNQM66}
}

---

Additional References Consulted

• opencode
• openharness https://github.com/HKUDS/OpenHarness
• openautocoder/live-swe-agent
• openautocoder/agentless
• walkie-talkie (https://github.com/xyuzh/walkie-talkie)
• claude code behavior: read/bash/edit (we could stick with this but direct it to Reproduce)
• disecting swe bench leaderboard: https://arxiv.org/pdf/2506.17208v1
• for formatting repo: https://github.com/SWE-agent/mini-swe-agent

---

Setup Instructions

git clone <your-repo-url> multitask-oss && cd multitask-oss
pip install -e ".[dev,swebench]"
modal setup                                              # auth (browser login)
modal secret create openai-keys OPENAI_API_KEY=sk-...
modal secret create modal-token MODAL_TOKEN_ID=<id> MODAL_TOKEN_SECRET=<secret>   # from ~/.modal.toml
bash setup_scripts/setup_env_cache.sh                     # builds/caches all 63 envs into Modal Volume "swebench-envs-v2"

Wait for that script to finish (~5-10 min) before running anything else — it's what makes every repo's VERIFY signal correct.

Generate the instance list

python3 -c "
from datasets import load_dataset
ds = load_dataset('princeton-nlp/SWE-bench_Lite', split='test')
open('ids.txt', 'w').write('\n'.join(r['instance_id'] f
"

Run all 300

modal run modal_run.py::run_batch --ids-file ids.txt --concurrency 4 --results-file test_traces/results_full300.jsonl

Why --concurrency 4, not the default 10

Each in-flight instance actually holds two concurrent Modal containers at once — run_instance itself, plus the nested container the officieval spins up (evaluate_patch(..., --modal true)) to grans ~8 containers in flight, --concurrency 10 (thedefault) means ~20 — comfortably under Modal's Starter-plan container cap, but the real constraint on a free plan is usually OpenAI's own rate limit for a lower-tier API key, which is what --coing (modal_run.py's own comment: default 10 is "matchedto OpenAI Tier 2 headroom" — a free/low-tier key should go lower). If he still hits 429s at 4, drop to 2; if it's smooth, he can nudge up.

Progress streams live and writes to test_traces/results_full300.jsonl incrementally, so it's safe to Ctrl-C and resume-by-diffing ids.txt against what's already in the results file if needed.