Time-series learning · Neural architecture & complexity / Open question
Learning Which Time Series Matter
Large sensor and economic panels often contain only a few useful cross-series links. Learning those links can matter more than attending to every pair of observations. Efficiency payoff: Replace quadratic all-token interaction with a learned sparse temporal/channel computation without sacrificing forecast accuracy.
Seed catalog · Showing the bundled record.
Problem statement
Observe in , with Gaussian innovations, , and at most nonzero lag-channel coefficients in each output row. Can gradient training of a temporal-then-channel attention predictor learn those dependencies from consecutive observations with excess risk at the sparse-regression rate, while reducing deployment work below dense attention on tokens?
Assumptions and scope
- Innovation covariance has eigenvalues in [sigma_min^2,sigma_max^2], gamma>0 and s<=dL. Risk uses an independent stationary test trajectory; overlapping training windows are dependent and cannot be counted as nL independent examples.
- Specify the number of heads, nonlinearities, token/lag encodings, signed readout, temperature schedule and initialization. Charge dense discovery/routing during training even if the final hard-selected attention is sparse.
- Compare to regularized sparse VAR and to factorized/axial attention. A proposed sparse rate must display its stability and covariance constants. Known support, oracle attention weights or an exact ERM oracle do not constitute a training result.
What would settle it?
A concrete finite-width training theorem with sample, optimization and serving-cost bounds, or an explicit failure regime of the factorized attention parameterization relative to sparse VAR.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Sparse VAR estimation is established. Pantea and Dieuleveut already prove oracle optimality and projected-gradient convergence for a single-location attention model. This extension concerns multiple sparse dependencies and a dependent trajectory, not the first axial-attention training result. This is an editorial, source-motivated research target. The preliminary review through 2026-11-20 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Pantea and Dieuleveut — Understanding Axial Attention in TSFMs through Single Location Regression ↗partial theory · source checked 2026-11-20
Primary abstract checked: asymptotic Bayes optimality and global projected-gradient convergence at a particular temperature scaling already hold in the single-location model. Sparse VAR data with dependent windows is an editorial extension.
- Basu and Michailidis — Regularized Estimation in Sparse High-Dimensional Time Series Models ↗partial theory · source checked 2026-11-20
Primary search abstract checked in the time-series pass: stability-dependent nonasymptotic sparse Gaussian/VAR estimation. Sparse VAR estimation itself is an established baseline.