Embeddings & representations · JEPA & world models / Open question
Learning a Shared Geometry
Images, language and other observations describe overlapping aspects of the world. Shared embeddings should preserve that common information without pretending every modality reveals the same facts. Efficiency payoff: Share useful modality features and reduce embedding dimensions with a downstream-risk guarantee.
Seed catalog · Showing the bundled record.
Problem statement
Let views satisfy , with shared , independent Gaussian nuisances/noise and bounded, identifiable loading matrices. Train finite-width nonlinear encoders from partly missing paired views, then align their whitened outputs using paid pairs. Bound together with excess risk for bounded linear functions of , relative to the noise-limited oracle. Expose finite-training error, view overlap and alignment-sample costs.
Assumptions and scope
- Fix the latent model, observation maps, paired/unpaired data access and precisely which latent functions both modalities can identify.
- State encoder objectives, dimension, normalization and optimization procedure; distinguish population optima from the encoders actually learned.
- Define agreement up to an explicit transformation group and charge any alignment pairs. A similarity score alone need not certify semantic equivalence or causal structure.
What would settle it?
A finite-sample learning-and-alignment theorem with quantitative transfer error, plus necessity or ambiguity results under incomplete views and unequal information.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Balestriero and LeCun already claim a JEPA alignment theorem under good-view assumptions and bounds otherwise. Content/style identifiability and approximate sufficient representations also have theory. The residual target combines imperfect observation access, finite-data optimization and a charged alignment procedure. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Balestriero and LeCun — Proving and Improving the Platonic Representation Hypothesis with JEPAs ↗partial theory · source checked 2026-11-19
Primary abstract checked: an October 2026 alignment theorem under good-view assumptions, with a bound otherwise and empirical alignment scaling. The full optimization and observation assumptions were not independently audited.
- von Kügelgen et al. — Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style ↗partial theory · source checked 2026-11-19
Abstract checked: invariant content is identifiable up to an invertible map under the latent augmentation model; this does not identify every content coordinate.
- Lin and Mei — A Statistical Theory of Contrastive Learning via Approximate Sufficient Statistics ↗partial theory · source checked 2026-11-19
Abstract checked: contrastive losses yield approximately sufficient encoders and downstream risk bounds, including augmentation error.
- Schechter Vera et al. — EmbeddingGemma 2: Unified and Efficient Multimodal Representations ↗related algorithm · source checked 2026-11-20
November revision abstract and Google DeepMind page checked: selective encoder loading, token budgets, truncatable dimensions and quantization. These motivate a task-aware resource/risk guarantee.