REFERENCE

Autoresearch

Unattended, bounded research campaigns: the agent runs experiments, reads the evidence and improves a result.

/autoresearch hands a research question to the agent for a whole campaign. The agent plans experiments, runs them on a benchmark, reads the results and the agents' transcripts, searches arXiv, revises its approach, and runs the next batch, until the goal is met or a limit is reached. Every result is measured by Theoria: proofs are checked by Lean and judged by Comparator, never taken from the agent's word.

Today campaigns improve a compression skill: the instructions that guide /compress, scored on a benchmark's proofs. Autoresearch needs macOS for now.

Start a campaign#

Theoria
/autoresearch start putnam2025 corpus=corpus.json budget=100 minutes=180
/autoresearch start putnam2025 objective="Shorten proofs without raising heartbeats" skill=v1.md,v2.md workers=4
/autoresearch start minif2f corpus=corpus.json models=amazon-bedrock/claude-opus,openai/gpt-6 budget=50
OptionMeaning
<benchmark>The benchmark whose tasks the experiments run on (required)
corpus=<manifest>The proofs to work on, unless the benchmark supplies a default corpus
objective="…"The research goal, in your words
skill=<path>,… or skill=noneStarting skills, snapshotted as v001, v002, …; the first is the reference. Omitted: the bundled skill
models=<provider/id>,…Models the campaign may compare; default: the current model
workers=<n>Parallel trials across the campaign (default 6)
budget=<USD>Total spending cap, for trials and the supervising agent
minutes=<n>Wall-clock limit
experiments=<n>, decisions=<n>Caps on experiments run and research decisions made
cost=<USD>, timeout=<min>, turns=<n>Limits per trial

Starting asks for one confirmation that shows exactly what is authorized. Omitted limits are unlimited, and the confirmation says so; set budget and minutes to bound a campaign.

Follow and control it#

Theoria
/autoresearch status                 # the latest campaign
/autoresearch status campaign-002
/autoresearch pause                  # finish the experiments in flight, start no new ones
/autoresearch resume campaign-002
/autoresearch cancel campaign-002     # interrupt what is running

Keep Theoria open while a campaign runs. Esc or closing the session interrupts it and saves a checkpoint; /autoresearch resume continues from there without repeating finished work or duplicating live workers.

How a campaign works#

flowchart TD
  start["/autoresearch start …"] --> confirm{"Confirm scope
and limits"}
  confirm --> manifest[("manifest.json
frozen corpus, skills,
models and limits")]
  manifest --> turn["Research turn
(supervising agent)"]
  turn -- "stage_skill" --> skills[("skills/v001.md, v002.md, …")]
  turn -- "literature" --> literature[("literature/
arXiv pages")]
  turn -- "submit_batch" --> batch["Batch of trials
one factor changed"]
  skills --> batch
  batch --> runs[("Benchmark runs
proofs, transcripts")]
  runs --> judge{{"Lean + Comparator
verify and score"}}
  judge --> evidence[("evidence/
measurements")]
  evidence -- "read evidence" --> turn
  turn -- "finish" --> report[("report.json
selected skill")]
  report -.-> claim["/claim new hypothesis"]
  report -.-> artifact["/artifact add report.json"]
  report -.-> adopt["Adopt the skill"]
  turn -. "every step" .-> journal[("events.jsonl
journal")]
What a campaign records, and where. Solid arrows are automatic; dashed ones are your choice.

Every step is appended to the campaign's journal (events.jsonl), so a campaign can be paused, resumed and audited. A campaign does not write the project's claims or artifacts on its own: its results are campaign records. When a finding is worth keeping, record it yourself, for example /claim new hypothesis <statement> with the report linked as evidence, or /artifact add .theoria/autoresearch/campaign-002/report.json.

  1. Frozen setup. The corpus, starting skills, methodology, models and limits are recorded when the campaign starts, so later edits to files do not change it.
  2. Experiments. The supervising agent stages a skill, and submits a batch of trials that changes one factor at a time against a matched control.
  3. Measurement. Each trial is a separate, sandboxed agent session in its own small Lean project (see Lean environments). Theoria verifies its proofs with Lean and Comparator and computes the same score as /compress.
  4. Evidence. The agent reads compact summaries of every development trial, and opens raw transcripts and candidate proofs when a result needs explaining.
  5. Literature. It can retrieve arXiv abstracts and HTML pages to inform the next revision.
  6. Decision. It revises the skill, runs the next batch, and repeats until it finishes or a limit is reached.

Results you can trust#

  • Held-out tasks stay held out. Test-split tasks are never shown to the research loop.
  • Matched comparisons. The final choice compares skills on complete, matched validation runs (training when there is no validation split), weighting the authorized models and tasks equally. A skill chosen without that evidence is labelled provisional.
  • Honest statistics. Reports give per-task means, standard deviations and best-of-k separately.
  • Nothing is promoted automatically. A winning skill stays in the campaign folder; adopting it is your decision.

Where results are kept#

Each campaign lives in .theoria/autoresearch/campaign-NNN/: the frozen manifest and methodology, a journal of every decision, the versioned skills (skills/v001.md, …), the literature consulted, the development evidence and a final report.json. Its experiments are ordinary benchmark runs, readable with /benchmark status.

Campaign defaults are under autoresearch in the configuration. The methodology the agent follows is a Markdown prompt you can edit.

Last updated 2026-11-10View as Markdown