Neural architecture & complexity · RNNs & sequence learning / Open question
How Much Thought Is Enough?
CTMs and recursive transformers can exchange stored depth for repeated computation. Useful adaptive depth requires knowing when further iterations are worth their cost. Efficiency payoff: Use shared parameters and shorter per-instance computation while controlling the probability and magnitude of premature stopping.
Seed catalog · Showing the bundled record.
Problem statement
For systems , draw with Haar-orthogonal , independent , and . Train a shared recurrent block and a stopping head from solved systems. Bound both and for the learned stopping time , charging block size, training steps and stopping checks. Compare to fixed-depth recurrence and explicit iterative solvers.
Assumptions and scope
- Use z_0=0, a specified bounded-width tanh or attention block and a stopping head observing only current/past internal states and the input system. It cannot observe the unknown error or exact solution at deployment.
- The distribution, kappa and maximum depth are fixed before training. State any transfer to another condition-number distribution separately; a one-distribution average guarantee is not a uniform certificate for all matrices.
- Count parameter bits, each matrix-vector/readout operation, and any residual norm computation used by stopping. CG and other explicit solvers are baselines. A representation of an iterative algorithm or a locally attracting fixed point alone is not a finite-training/halting theorem.
What would settle it?
A fully specified trained recurrent architecture with finite-sample error and expected-cost guarantees, or a lower bound exposing the cost of certifying early exits. Separate training, iteration and certification costs and compare at equal accuracy.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
CTM adaptive ticks, fixed-point mechanism studies and DeepMind's recursive-block/LoRA batching already exist. The linear-system task is an editorial training-and-stopping benchmark, not the original image-classification task. Non-normal transient growth must be controlled rather than inferred from eigenvalues alone. This is an editorial, source-motivated research target. The preliminary review through 2026-11-20 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Darlow et al. — Continuous Thought Machines ↗related algorithm · source checked 2026-11-20
Primary search abstract and Sakana technical overview checked: neuron histories, synchronization and adaptive thought ticks. These are existing architectural mechanisms, not a learned stopping-cost theorem.
- Hannagan — Riding the Fixed-Point Curve: Image Classification in the Continuous Thought Machine ↗partial theory · source checked 2026-11-20
Primary abstract checked: reported local attraction, non-normal amplification and fixed-point trajectories in trained CTMs. Spectral radius alone does not bound transient growth or certify learned stopping.
- Bae et al. — Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA ↗related algorithm · source checked 2026-11-20
Google DeepMind publication page checked: shared blocks, loop-specific low-rank adapters and depth-wise batching. Weight tying and early-exit scheduling already exist; repeated arithmetic and storage savings differ.
- Gatmiry et al. — Can Looped Transformers Learn Multi-step Gradient Descent? ↗partial theory · source checked 2026-11-19
Abstract checked: population-loss and gradient-flow results for linear looped transformers already establish a multi-layer learnability case.