← All AI/ML open problems

JEPA & world models · Learning dynamics & scaling / Open question

Prediction Without Representation Collapse

Predicting another learned representation admits trivial solutions. A training theory should explain how useful structure emerges and survives stochastic updates.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

Use paired views xv=Avz+ξvx_v=A_vz+\xi_v of a rank-kk Gaussian latent signal, two-layer tanh encoders and a linear predictor. Train the online encoder by minibatch squared embedding-prediction loss with stop-gradient target and update target weights by θˉt+1=τθˉt+(1−τ)θt+1\bar\theta_{t+1}=\tau\bar\theta_t+(1-\tau)\theta_{t+1}. Determine when λk(Cov⁡(eθ(x)))≥c>0\lambda_k(\operatorname{Cov}(e_\theta(x)))\geq c>0 persists and the learned embedding retains predictive information about zz, as noise, masking, batch size and τ\tau vary. State any added variance regularizer explicitly.

Assumptions and scope

  • The base algorithm is squared prediction loss with the EMA/stop-gradient update in the statement. LeJEPA is a separate comparison, not an interchangeable update; regularized variants must give the coefficient and empirical statistic.
  • Fix layer widths, initialization, learning rates, loading-matrix singular-value bounds and the masking probabilities; report their effects rather than choosing an easy data family after training.
  • Use both the covariance eigenvalue criterion and excess latent-prediction risk against the same-view oracle; nonzero variance alone is insufficient. Analyze beyond a local linearization around zero.

What would settle it?

A finite-time nonlinear stability/convergence guarantee with a nontrivial representation criterion and explicit failure boundaries, or a lower bound showing which data or regularization condition is needed.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Drive vs. Decay already analyzes early linearized JEPA stability and mode growth; LeJEPA supplies a regularized, theory-motivated objective without teacher–student heuristics. The additional target is finite-batch nonlinear trajectory control and useful-rank persistence. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. De Melo Costa et al. — Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures ↗partial theory · source checked 2026-11-19

    Primary abstract checked: early gradient-flow linearization near collapse, approximate spectral decoupling, rank-emergence predictions and a residual predictor. This is not a global finite-batch nonlinear guarantee.

  2. Balestriero and LeCun — LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics ↗partial theory · source checked 2026-11-19

    Primary abstract checked: isotropic-Gaussian target embeddings under the stated theory, SIGReg and a predictive objective without stop-gradient or teacher–student heuristics. Collapse-avoidance objectives already exist.

  3. Cui et al. — A Generalization Theory for JEPA-Based World Models ↗partial theory · source checked 2026-11-19

    Primary abstract checked: action-conditioned co-occurrence factorization, finite-sample prediction/planning-regret bounds and a latent-dimension trade-off. A prediction-to-planning theory is already available under the paper's framework.