REFERENCE
Autoresearch
Unattended, bounded research campaigns: the agent runs experiments, reads the evidence and improves a result.
/autoresearch hands a research question to the agent for a whole campaign. The agent plans experiments, runs them on a benchmark, reads the results and the agents' transcripts, searches arXiv, revises its approach, and runs the next batch, until the goal is met or a limit is reached. Every result is measured by Theoria: proofs are checked by Lean and judged by Comparator, never taken from the agent's word.
/compress, scored on a benchmark's proofs. Autoresearch needs macOS for now.Start a campaign#
/autoresearch start putnam2025 corpus=corpus.json budget=100 minutes=180
/autoresearch start putnam2025 objective="Shorten proofs without raising heartbeats" skill=v1.md,v2.md workers=4
/autoresearch start minif2f corpus=corpus.json models=amazon-bedrock/claude-opus,openai/gpt-6 budget=50| Option | Meaning |
|---|---|
<benchmark> | The benchmark whose tasks the experiments run on (required) |
corpus=<manifest> | The proofs to work on, unless the benchmark supplies a default corpus |
objective="…" | The research goal, in your words |
skill=<path>,… or skill=none | Starting skills, snapshotted as v001, v002, …; the first is the reference. Omitted: the bundled skill |
models=<provider/id>,… | Models the campaign may compare; default: the current model |
workers=<n> | Parallel trials across the campaign (default 6) |
budget=<USD> | Total spending cap, for trials and the supervising agent |
minutes=<n> | Wall-clock limit |
experiments=<n>, decisions=<n> | Caps on experiments run and research decisions made |
cost=<USD>, timeout=<min>, turns=<n> | Limits per trial |
Starting asks for one confirmation that shows exactly what is authorized. Omitted limits are unlimited, and the confirmation says so; set budget and minutes to bound a campaign.
Follow and control it#
/autoresearch status # the latest campaign
/autoresearch status campaign-002
/autoresearch pause # finish the experiments in flight, start no new ones
/autoresearch resume campaign-002
/autoresearch cancel campaign-002 # interrupt what is runningKeep Theoria open while a campaign runs. Esc or closing the session interrupts it and saves a checkpoint; /autoresearch resume continues from there without repeating finished work or duplicating live workers.
How a campaign works#
flowchart TD
start["/autoresearch start …"] --> confirm{"Confirm scope
and limits"}
confirm --> manifest[("manifest.json
frozen corpus, skills,
models and limits")]
manifest --> turn["Research turn
(supervising agent)"]
turn -- "stage_skill" --> skills[("skills/v001.md, v002.md, …")]
turn -- "literature" --> literature[("literature/
arXiv pages")]
turn -- "submit_batch" --> batch["Batch of trials
one factor changed"]
skills --> batch
batch --> runs[("Benchmark runs
proofs, transcripts")]
runs --> judge{{"Lean + Comparator
verify and score"}}
judge --> evidence[("evidence/
measurements")]
evidence -- "read evidence" --> turn
turn -- "finish" --> report[("report.json
selected skill")]
report -.-> claim["/claim new hypothesis"]
report -.-> artifact["/artifact add report.json"]
report -.-> adopt["Adopt the skill"]
turn -. "every step" .-> journal[("events.jsonl
journal")]Every step is appended to the campaign's journal (events.jsonl), so a campaign can be paused, resumed and audited. A campaign does not write the project's claims or artifacts on its own: its results are campaign records. When a finding is worth keeping, record it yourself, for example /claim new hypothesis <statement> with the report linked as evidence, or /artifact add .theoria/autoresearch/campaign-002/report.json.
- Frozen setup. The corpus, starting skills, methodology, models and limits are recorded when the campaign starts, so later edits to files do not change it.
- Experiments. The supervising agent stages a skill, and submits a batch of trials that changes one factor at a time against a matched control.
- Measurement. Each trial is a separate, sandboxed agent session in its own small Lean project (see Lean environments). Theoria verifies its proofs with Lean and Comparator and computes the same score as
/compress. - Evidence. The agent reads compact summaries of every development trial, and opens raw transcripts and candidate proofs when a result needs explaining.
- Literature. It can retrieve arXiv abstracts and HTML pages to inform the next revision.
- Decision. It revises the skill, runs the next batch, and repeats until it finishes or a limit is reached.
Results you can trust#
- Held-out tasks stay held out. Test-split tasks are never shown to the research loop.
- Matched comparisons. The final choice compares skills on complete, matched validation runs (training when there is no validation split), weighting the authorized models and tasks equally. A skill chosen without that evidence is labelled provisional.
- Honest statistics. Reports give per-task means, standard deviations and best-of-k separately.
- Nothing is promoted automatically. A winning skill stays in the campaign folder; adopting it is your decision.
Where results are kept#
Each campaign lives in .theoria/autoresearch/campaign-NNN/: the frozen manifest and methodology, a journal of every decision, the versioned skills (skills/v001.md, …), the literature consulted, the development evidence and a final report.json. Its experiments are ordinary benchmark runs, readable with /benchmark status.
Campaign defaults are under autoresearch in the configuration. The methodology the agent follows is a Markdown prompt you can edit.