Time-series learning · RNNs & sequence learning / Open question
Learning Long Memory with Few States
A recurrent state can compress very long temporal dependence, but the slow modes that provide memory can also be difficult to estimate and train stably. Efficiency payoff: Learn a compact streaming forecaster whose memory does not grow with the full observed history.
Seed catalog · Showing the bundled record.
Problem statement
Let , with , , , , and i.i.d. Gaussian innovations. From one stationary length- trajectory, train an -state diagonal recurrence , . Bound excess one-step forecast risk jointly with the gradient steps, state precision and warm-up needed to learn , as grows and approaches one.
Assumptions and scope
- Fix 0<gamma<1, A>1+gamma, Q and innovation-variance bounds. The coefficients and exponents are unknown; the predictor is evaluated on an independent stationary test trajectory. Finite initial history creates a warm-up error that must be included.
- Choose an explicit stable parametrization 0<=rho_i<1, initialization, empirical loss and optimizer; poles must actually be learned, rather than inserted from the unknown teacher. Charge every pass through dependent data, stored states and finite-bit arithmetic.
- Separate representation error, statistical error, optimization error, initial-state error and rounding. The known square-log state requirement for a fixed algebraic kernel is a baseline; do not relabel it as an unresolved approximation exponent.
What would settle it?
A finite-data and finite-gradient-step learning theorem attaining a costed approximation/statistical trade-off, or a quantified optimization/precision obstruction even though compact states exist.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Song supplies approximation, stability and information results, including the canonical square-log resource law. Memory-dependent optimization theory already warns that training recurrent weights can be worse than fixing them. The proposed unknown-mixture, learned-pole benchmark remains a framework-level editorial extension. This is an editorial, source-motivated research target. The preliminary review through 2026-11-20 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Song — The Cost of Long Memory: State, Context, and Stability Complexity in Sequence Models ↗partial theory · source checked 2026-11-20
HTML Sections 1–4 and appendix scope checked: root-exponential approximation, square-log state count, fractional predictive geometry and finite-sample information results. The canonical representation exponent is already claimed solved; finite-step learning of recurrent poles is a different target.
- Guan et al. — Memory Determines Learning Direction: A Theory of Gradient-Based Optimization in State Space Models ↗partial theory · source checked 2026-11-20
Primary search abstract checked in the time-series pass: initialization and recurrent-weight learning affect memory; fixed recurrent weights can be advantageous. Representation alone is not a training guarantee.