← All AI/ML open problems

Neural architecture & complexity · Learning dynamics & scaling / Open question

Can Sparse Experts Learn Without Starving?

A large pool of experts is efficient only if the router discovers their roles and sends enough learning signal to each useful expert. Efficiency payoff: Train a high-capacity model with low active computation without losing rare features through capacity overflow or early routing collapse.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-20

Seed catalog · Showing the bundled record.

Problem statement

Let J∈[K]J\in[K] have masses in [1/(2K),2/K][1/(2K),2/K], X=μJ+ξX=\mu_J+\xi with ξ∼N(0,Id)\xi\sim N(0,I_d) and separated bounded means, and Y=wJ⊤X+ϵY=w_J^\top X+\epsilon with ∥wJ∥≤1\|w_J\|\leq1. Jointly train a softmax router and KK linear experts, evaluating only the top expert and admitting at most q=⌈cB/K⌉q=\lceil cB/K\rceil tokens per expert in a batch of size BB. Bound prediction risk, router starvation and dropped-token bias in terms of n,B,K,cn,B,K,c, separation and training steps, relative to the best predictor obeying the same sparse routing rule.

Assumptions and scope

  • Fix the router logits, initialization, routing-gradient estimator and load-balancing coefficient before a theorem; straight-through and ordinary gradients are not interchangeable. Fix an overflow policy such as random admission and zero loss contribution for dropped examples.
  • The hidden cluster label is never supplied. State the approximation gap between a one-expert predictor and the Bayes mixture under Gaussian overlap; use excess risk over the sparse comparator rather than pretending clusters are perfectly identifiable.
  • Charge router evaluation, all expert updates and dispatch, not just the active expert's arithmetic. Capacity is a minibatch constraint, and a bound based only on average balanced loads does not bound starvation along training.

What would settle it?

An implementable gradient-based routing/training scheme with finite-data excess-risk and overflow/starvation bounds, including a lower or failure bound when capacity or exploration is inadequate.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Clustered MoE SGD and statistical oracle inequalities already have positive results. The added restriction is jointly learned, capacity-limited stochastic routing with actual optimization/dispatch costs. The proposed model is editorial and leaves the surrogate/update design as a framework choice. This is an editorial, source-motivated research target. The preliminary review through 2026-11-20 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Kawata et al. — Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learning ↗partial theory · source checked 2026-11-20

    Primary abstract checked: gradient-trained clustered single-index regression has sample/runtime guarantees. The capacity-limited routing target must go beyond this established positive result.

  2. He et al. — Towards a Statistical Understanding of Mixture-of-Experts ↗partial theory · source checked 2026-11-20

    Scholarly abstract checked in the prior agent pass: oracle bounds with evolving experts separate approximation, expert learning and routing estimation. Batch overflow and router starvation need their own training analysis.

  3. Cetin et al. — Sparser, Faster, Lighter Transformer Language Models ↗related algorithm · source checked 2026-11-20

    Primary abstract and Sakana technical blog checked: L1-induced activation sparsity, TwELL packing and GPU kernels with measured throughput/memory gains. Counting sparse FLOPs alone does not predict wall-clock time.