← All AI/ML open problems

Time-series learning · Neural architecture & complexity / Open question

Learning Which Time Series Matter

Large sensor and economic panels often contain only a few useful cross-series links. Learning those links can matter more than attending to every pair of observations. Efficiency payoff: Replace quadratic all-token interaction with a learned sparse temporal/channel computation without sacrificing forecast accuracy.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-20

Seed catalog · Showing the bundled record.

Problem statement

Observe Xt=∑j=1LAjXt−j+ξtX_t=\sum_{j=1}^L A_jX_{t-j}+\xi_t in Rd\mathbb R^d, with Gaussian innovations, ∑j∥Aj∥op≤1−γ\sum_j\|A_j\|_{\mathrm{op}}\leq1-\gamma, and at most ss nonzero lag-channel coefficients in each output row. Can gradient training of a temporal-then-channel attention predictor learn those dependencies from nn consecutive observations with excess risk E∥X^t+1−∑jAjXt+1−j∥2/d\mathbb E\|\hat X_{t+1}-\sum_jA_jX_{t+1-j}\|^2/d at the sparse-regression rate, while reducing deployment work below dense attention on dLdL tokens?

Assumptions and scope

  • Innovation covariance has eigenvalues in [sigma_min^2,sigma_max^2], gamma>0 and s<=dL. Risk uses an independent stationary test trajectory; overlapping training windows are dependent and cannot be counted as nL independent examples.
  • Specify the number of heads, nonlinearities, token/lag encodings, signed readout, temperature schedule and initialization. Charge dense discovery/routing during training even if the final hard-selected attention is sparse.
  • Compare to regularized sparse VAR and to factorized/axial attention. A proposed sparse rate must display its stability and covariance constants. Known support, oracle attention weights or an exact ERM oracle do not constitute a training result.

What would settle it?

A concrete finite-width training theorem with sample, optimization and serving-cost bounds, or an explicit failure regime of the factorized attention parameterization relative to sparse VAR.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Sparse VAR estimation is established. Pantea and Dieuleveut already prove oracle optimality and projected-gradient convergence for a single-location attention model. This extension concerns multiple sparse dependencies and a dependent trajectory, not the first axial-attention training result. This is an editorial, source-motivated research target. The preliminary review through 2026-11-20 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Pantea and Dieuleveut — Understanding Axial Attention in TSFMs through Single Location Regression ↗partial theory · source checked 2026-11-20

    Primary abstract checked: asymptotic Bayes optimality and global projected-gradient convergence at a particular temperature scaling already hold in the single-location model. Sparse VAR data with dependent windows is an editorial extension.

  2. Basu and Michailidis — Regularized Estimation in Sparse High-Dimensional Time Series Models ↗partial theory · source checked 2026-11-20

    Primary search abstract checked in the time-series pass: stability-dependent nonasymptotic sparse Gaussian/VAR estimation. Sparse VAR estimation itself is an established baseline.