# Autoresearch

> Unattended, bounded research campaigns: the agent runs experiments, reads the evidence and improves a result.

`/autoresearch` hands a research question to the agent for a whole campaign. The agent plans experiments, runs them on a benchmark, reads the results and the agents' transcripts, searches arXiv, revises its approach, and runs the next batch, until the goal is met or a limit is reached. Every result is measured by Theoria: proofs are checked by Lean and judged by Comparator, never taken from the agent's word.

> Today campaigns improve a compression skill: the instructions that guide [`/compress`](https://www.theoria.sh/docs/commands.md#compress), scored on a [benchmark](https://www.theoria.sh/docs/commands.md#benchmark)'s proofs. Autoresearch needs macOS for now.

## Start a campaign

```text
/autoresearch start putnam2025 corpus=corpus.json budget=100 minutes=180
/autoresearch start putnam2025 objective="Shorten proofs without raising heartbeats" skill=v1.md,v2.md workers=4
/autoresearch start minif2f corpus=corpus.json models=amazon-bedrock/claude-opus,openai/gpt-6 budget=50
```

| Option | Meaning |
| --- | --- |
| `<benchmark>` | The benchmark whose tasks the experiments run on (required) |
| `corpus=<manifest>` | The proofs to work on, unless the benchmark supplies a default corpus |
| `objective="…"` | The research goal, in your words |
| `skill=<path>,…` or `skill=none` | Starting skills, snapshotted as `v001`, `v002`, …; the first is the reference. Omitted: the bundled skill |
| `models=<provider/id>,…` | Models the campaign may compare; default: the current model |
| `workers=<n>` | Parallel trials across the campaign (default 6) |
| `budget=<USD>` | Total spending cap, for trials and the supervising agent |
| `minutes=<n>` | Wall-clock limit |
| `experiments=<n>`, `decisions=<n>` | Caps on experiments run and research decisions made |
| `cost=<USD>`, `timeout=<min>`, `turns=<n>` | Limits per trial |

Starting asks for one confirmation that shows exactly what is authorized. **Omitted limits are unlimited**, and the confirmation says so; set `budget` and `minutes` to bound a campaign.

## Follow and control it

```text
/autoresearch status                 # the latest campaign
/autoresearch status campaign-002
/autoresearch pause                  # finish the experiments in flight, start no new ones
/autoresearch resume campaign-002
/autoresearch cancel campaign-002     # interrupt what is running
```

Keep Theoria open while a campaign runs. Esc or closing the session interrupts it and saves a checkpoint; `/autoresearch resume` continues from there without repeating finished work or duplicating live workers.

## How a campaign works

*What a campaign records, and where. Solid arrows are automatic; dashed ones are your choice.*

```mermaid
flowchart TD
  start["/autoresearch start …"] --> confirm{"Confirm scope
and limits"}
  confirm --> manifest[("manifest.json<br/>frozen corpus, skills,<br/>models and limits")]
  manifest --> turn["Research turn<br/>(supervising agent)"]
  turn -- "stage_skill" --> skills[("skills/v001.md, v002.md, …")]
  turn -- "literature" --> literature[("literature/<br/>arXiv pages")]
  turn -- "submit_batch" --> batch["Batch of trials<br/>one factor changed"]
  skills --> batch
  batch --> runs[("Benchmark runs<br/>proofs, transcripts")]
  runs --> judge{{"Lean + Comparator
verify and score"}}
  judge --> evidence[("evidence/<br/>measurements")]
  evidence -- "read evidence" --> turn
  turn -- "finish" --> report[("report.json<br/>selected skill")]
  report -.-> claim["/claim new hypothesis"]
  report -.-> artifact["/artifact add report.json"]
  report -.-> adopt["Adopt the skill"]
  turn -. "every step" .-> journal[("events.jsonl<br/>journal")]
```

Every step is appended to the campaign's journal (`events.jsonl`), so a campaign can be paused, resumed and audited. A campaign does not write the project's [claims or artifacts](https://www.theoria.sh/docs/commands.md#claim) on its own: its results are campaign records. When a finding is worth keeping, record it yourself, for example `/claim new hypothesis <statement>` with the report linked as evidence, or `/artifact add .theoria/autoresearch/campaign-002/report.json`.

1. **Frozen setup.** The corpus, starting skills, methodology, models and limits are recorded when the campaign starts, so later edits to files do not change it.
2. **Experiments.** The supervising agent stages a skill, and submits a batch of trials that changes one factor at a time against a matched control.
3. **Measurement.** Each trial is a separate, sandboxed agent session in its own small Lean project (see [Lean environments](https://www.theoria.sh/docs/lean-environments.md)). Theoria verifies its proofs with Lean and Comparator and computes the same score as [`/compress`](https://www.theoria.sh/docs/commands.md#compress).
4. **Evidence.** The agent reads compact summaries of every development trial, and opens raw transcripts and candidate proofs when a result needs explaining.
5. **Literature.** It can retrieve arXiv abstracts and HTML pages to inform the next revision.
6. **Decision.** It revises the skill, runs the next batch, and repeats until it finishes or a limit is reached.

## Results you can trust

- **Held-out tasks stay held out.** Test-split tasks are never shown to the research loop.
- **Matched comparisons.** The final choice compares skills on complete, matched validation runs (training when there is no validation split), weighting the authorized models and tasks equally. A skill chosen without that evidence is labelled provisional.
- **Honest statistics.** Reports give per-task means, standard deviations and best-of-k separately.
- **Nothing is promoted automatically.** A winning skill stays in the campaign folder; adopting it is your decision.

## Where results are kept

Each campaign lives in `.theoria/autoresearch/campaign-NNN/`: the frozen manifest and methodology, a journal of every decision, the versioned skills (`skills/v001.md`, …), the literature consulted, the development evidence and a final `report.json`. Its experiments are ordinary benchmark runs, readable with [`/benchmark status`](https://www.theoria.sh/docs/commands.md#benchmark).

Campaign defaults are under `autoresearch` in the [configuration](https://www.theoria.sh/docs/configuration.md). The methodology the agent follows is a Markdown prompt you can edit.

## Related

- [Slash commands](https://www.theoria.sh/docs/commands.md): Every Theoria-CLI slash command, with its forms and examples.
- [Lean environments](https://www.theoria.sh/docs/lean-environments.md): How Theoria finds, installs and shares Lean and Mathlib, and keeps long Lean runs light on disk.
- [Configuration](https://www.theoria.sh/docs/configuration.md): Settings, harness profiles, the interface and where Theoria keeps its files.

Last updated 2026-11-10. Web page: https://www.theoria.sh/docs/autoresearch