← All AI/ML open problems

Time-series learning · RNNs & sequence learning / Open question

Learning Long Memory with Few States

A recurrent state can compress very long temporal dependence, but the slow modes that provide memory can also be difficult to estimate and train stably. Efficiency payoff: Learn a compact streaming forecaster whose memory does not grow with the full observed history.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-20

Seed catalog · Showing the bundled record.

Problem statement

Let Xt=∑j≥1ajXt−j+ϵtX_t=\sum_{j\geq1}a_jX_{t-j}+\epsilon_t, with aj=∑q=1Qcqj−αqa_j=\sum_{q=1}^Q c_qj^{-\alpha_q}, cq≥0c_q\geq0, αq∈[1+γ,A]\alpha_q\in[1+\gamma,A], ∑jaj≤1−γ\sum_j a_j\leq1-\gamma, and i.i.d. Gaussian innovations. From one stationary length-nn trajectory, train an rr-state diagonal recurrence ht+1=diag⁡(ρ)ht+vXth_{t+1}=\operatorname{diag}(\rho)h_t+vX_t, X^t+1=u⊤ht+1\hat X_{t+1}=u^\top h_{t+1}. Bound excess one-step forecast risk jointly with the gradient steps, state precision and warm-up needed to learn ρ,u,v\rho,u,v, as rr grows and max⁡iρi\max_i\rho_i approaches one.

Assumptions and scope

  • Fix 0<gamma<1, A>1+gamma, Q and innovation-variance bounds. The coefficients and exponents are unknown; the predictor is evaluated on an independent stationary test trajectory. Finite initial history creates a warm-up error that must be included.
  • Choose an explicit stable parametrization 0<=rho_i<1, initialization, empirical loss and optimizer; poles must actually be learned, rather than inserted from the unknown teacher. Charge every pass through dependent data, stored states and finite-bit arithmetic.
  • Separate representation error, statistical error, optimization error, initial-state error and rounding. The known square-log state requirement for a fixed algebraic kernel is a baseline; do not relabel it as an unresolved approximation exponent.

What would settle it?

A finite-data and finite-gradient-step learning theorem attaining a costed approximation/statistical trade-off, or a quantified optimization/precision obstruction even though compact states exist.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Song supplies approximation, stability and information results, including the canonical square-log resource law. Memory-dependent optimization theory already warns that training recurrent weights can be worse than fixing them. The proposed unknown-mixture, learned-pole benchmark remains a framework-level editorial extension. This is an editorial, source-motivated research target. The preliminary review through 2026-11-20 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Song — The Cost of Long Memory: State, Context, and Stability Complexity in Sequence Models ↗partial theory · source checked 2026-11-20

    HTML Sections 1–4 and appendix scope checked: root-exponential approximation, square-log state count, fractional predictive geometry and finite-sample information results. The canonical representation exponent is already claimed solved; finite-step learning of recurrent poles is a different target.

  2. Guan et al. — Memory Determines Learning Direction: A Theory of Gradient-Based Optimization in State Space Models ↗partial theory · source checked 2026-11-20

    Primary search abstract checked in the time-series pass: initialization and recurrent-weight learning affect memory; fixed recurrent weights can be advantageous. Representation alone is not a training guarantee.