← All AI/ML open problems

Memory & capacity · Transformer theory · Statistical learning theory / Open question

How Much Can a Neural Network Remember?

A model's parameter count is not the same as usable knowledge. The missing theory concerns what training can store in those parameters and what inference can recover.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

Draw NN independent facts Yi∼Unif⁡({0,1}b)Y_i\sim\operatorname{Unif}(\{0,1\}^b), each queried through one of qq assigned alias tokens. Train a finite-width causal softmax transformer on sampled alias/value pairs, storing all PP parameters and embeddings at pp bits. Determine the largest NbNb for which gradient training achieves (Nb)−1∑iE dH(Y^i,Yi)≤ε(Nb)^{-1}\sum_i\mathbb E\,d_H(\hat Y_i,Y_i)\leq\varepsilon, where dHd_H is Hamming distance, under fixed exposures and decoding compute. Charge alias mappings and external state rather than treating the elementary PpPp storage bound as learnable capacity.

Assumptions and scope

  • Fix architecture, parameter count and bits per parameter; include embeddings, auxiliary state and any external database in storage accounting.
  • Define the entropy of the fact distribution, exposure counts and the retrieval-error criterion; correlated facts cannot be counted as independent bits.
  • Specify the optimizer, training time, query distribution and allowed decoding computation. Merely exhibiting weights that encode the facts does not prove they can be learned or retrieved.

What would settle it?

Matching learnable-and-retrievable capacity bounds for a declared model/data regime, with explicit precision, noise and compute dependence and a separation from raw storage capacity where appropriate.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Finite-bit counting bounds are elementary; controlled knowledge-capacity experiments report roughly two bits per parameter, and attention-layer tensor-rank analyses study factual memorization. Those are not a universal two-bit law or a full gradient-training/recall characterization. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Allen-Zhu and Li — Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws ↗empirical motivation · source checked 2026-11-19

    Primary search abstract checked: controlled factual-data experiments report approximately two knowledge bits per parameter, including quantized models. This measured scaling is not a universal information-theoretic capacity theorem.

  2. Wong — Paying Attention to Facts: Quantifying the Knowledge Capacity of Attention Layers ↗partial theory · source checked 2026-11-19

    Primary search abstract checked: database/attention tensor-rank bounds and empirical relationships for one-layer factual memorization. Expressivity, learned storage and reliable recall are distinct.

  3. Xu and Chen — Provable Knowledge Acquisition and Extraction in One-Layer Transformers ↗partial theory · source checked 2026-11-19

    Primary search abstract checked: stylized pretraining/fine-tuning dynamics and relation-coverage guarantees for both full and low-rank adaptation. Multi-layer rank allocation is a separate extension.