← All AI/ML open problems

RNNs & sequence learning · Memory & capacity · Learning dynamics & scaling / Open question

Learning to Remember for the Long Run

Recurrent networks reuse a small state across time. The central question is how training can make that state both durable and learnable. Efficiency payoff: Replace a growing attention history with a compact trainable recurrent state.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

Use streams of symbols indexing permutations of SS states, with a given initial state and final-state query after HH compositions. Train ht+1=gt⊙ht+(1−gt)⊙tanh⁡(Wht+Uxt+b)h_{t+1}=g_t\odot h_t+(1-g_t)\odot\tanh(Wh_t+Ux_t+b) with gt=σ(Ght+Vxt+c)g_t=\sigma(Gh_t+Vx_t+c) by full backpropagation through time; σ\sigma is the coordinatewise logistic function. Determine width, sample and step requirements for final-state error at most ε\varepsilon with pp-bit hidden-state storage, controlling both remote-input sensitivity and products of trajectory Jacobians as HH grows.

Assumptions and scope

  • Fix the task's state complexity, input process, recurrent architecture, gates or normalization and numerical precision; expressivity alone is not a training guarantee.
  • Specify initialization and full or truncated backpropagation, and control products of trajectory-dependent Jacobians.
  • Require a recall or prediction criterion sensitive to distant inputs. Uniform contraction can erase exactly the memory the task needs, so stability and memory must be assessed together.

What would settle it?

An end-to-end nonlinear training-and-memory theorem for a nontrivial task family, with quantitative horizon dependence or a sharp optimization/information obstruction.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Allen-Zhu and Li already prove efficient RNN learning for a sequence concept class, and stable linear systems admit linear-RNN training guarantees. The target is compact nonlinear state discovery with a joint long-memory/stability bound, not the first provable recurrent learner. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Allen-Zhu and Li — Can SGD Learn Recurrent Neural Networks with Provable Generalization? ↗partial theory · source checked 2026-11-19

    Primary search abstract checked: efficient RNN learning for a stated sequence concept class with polynomial or almost-polynomial length dependence. Trainability of all RNNs is not an open yes/no question.

  2. Wang et al. — Linear RNNs Provably Learn Linear Dynamic Systems ↗partial theory · source checked 2026-11-19

    Primary abstract checked: stable linear-system learning with sufficiently wide linear RNNs, including width independent of sequence length and stability-dependent complexity.

  3. Aicher, Foti and Fox — Adaptively Truncating Backpropagation Through Time to Control Gradient Bias ↗partial theory · source checked 2026-11-19

    Primary search abstract checked: adaptive truncation under expected geometric lag decay, bias control and nonconvex SGD convergence consequences. Adaptive truncation itself is established.