← All AI/ML open problems

Neural architecture & complexity · Optimization & optimizers / Open question

When Does Sparsity Make a Model Faster?

TwELL and model/system co-design show why sparse mathematical operations can still be expensive to dispatch or move through memory. Efficiency payoff: Select learnable activation patterns whose savings survive packing, bandwidth and dense fallback overhead.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-20

Seed catalog · Showing the bundled record.

Problem statement

For X∼N(0,Σ)X\sim N(0,\Sigma) with I⪯Σ⪯κII\preceq\Sigma\preceq\kappa I and Y=f∗(X)+ξY=f_*(X)+\xi, where f∗f_* is a width-kk bounded-weight two-layer ReLU teacher, train a width-mm student with thresholded hidden activations. Under the explicit serving-cost model T=tpack+max⁡{F/P,D/W}+tfallbackT=t_{\rm pack}+\max\{F/P,D/W\}+t_{\rm fallback}, determine the attainable excess-risk versus expected-TT frontier from nn samples. Here FF counts arithmetic, DD transferred bytes, and P,WP,W are fixed arithmetic and bandwidth rates.

Assumptions and scope

  • Fix batch size, numeric precision, tile size, sparse row capacity and the dense fallback rule. Packing and fallback costs are specified functions of the retained activation pattern, measured or supplied as inputs; they cannot be silently set to zero.
  • Use squared population risk against the teacher with independent bounded-variance label noise. Specify thresholds, their learning rule and the treatment of zeroed activations in backpropagation. Bound the tail of active counts, not only the mean sparsity.
  • Report training arithmetic and total model storage separately from serving time. The displayed model is an idealized architecture cost model, not a guaranteed wall-clock law for every GPU or kernel. Compare to dense serving under the same model.

What would settle it?

A specified training/sparsification procedure with a nonvacuous risk bound and cost prediction, plus a regime with a proved saving or obstruction under the fixed serving model. FLOP reduction alone is insufficient.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Sakana already provides sparse kernels and measured language-model gains; Flapping Airplanes supplies a systems agenda rather than a theorem. This teacher–student risk/cost formulation is an editorial benchmark requiring concrete numerical and training choices. This is an editorial, source-motivated research target. The preliminary review through 2026-11-20 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Cetin et al. — Sparser, Faster, Lighter Transformer Language Models ↗related algorithm · source checked 2026-11-20

    Primary abstract and Sakana technical blog checked: L1-induced activation sparsity, TwELL packing and GPU kernels with measured throughput/memory gains. Counting sparse FLOPs alone does not predict wall-clock time.

  2. Flapping Airplanes — research mission and systems direction ↗research agenda · source checked 2026-11-20

    Rendered primary homepage checked: model/system co-evolution, megakernels and overlapping memory-bound inference with compute-bound training. This is the lab's stated agenda, not a theorem or independently reproduced performance result.

  3. Kawata et al. — Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learning ↗partial theory · source checked 2026-11-20

    Primary abstract checked: gradient-trained clustered single-index regression has sample/runtime guarantees. The capacity-limited routing target must go beyond this established positive result.