← All AI/ML open problems

Bayesian learning · Data-efficient learning · Embeddings & representations / Open question

How Much Is Prior Knowledge Worth?

Prior experience should make a learner data-efficient, but a learned prior can also transfer the wrong assumptions. The value of pretraining needs a quantitative account.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

Let each task draw a∼Πa\sim\Pi on [−1,1]k[-1,1]^k and generate y=∑j=1kajReLU⁡(wj⊤x)+ξy=\sum_{j=1}^k a_j\operatorname{ReLU}(w_j^\top x)+\xi, with Gaussian inputs/noise and unknown shared orthonormal features. From MM tasks with nn samples each, learn both a prior and an amortized posterior-mean estimator for a new task with mm examples. Bound excess prediction risk relative to the oracle knowing Π,w1,…,wk\Pi,w_1,\ldots,w_k, charging inference-network training; quantify degradation when the test prior has density ratio at most κ\kappa relative to Π\Pi.

Assumptions and scope

  • Fix the task hyperdistribution, likelihood family, meta-training sample structure and shift class; separate the number of tasks from samples within each task.
  • Specify the prior-learning and inference algorithms, including approximation and optimization errors; an exact Bayes estimator is a baseline, not a free neural training result.
  • Compare equal total data and computation budgets and state which prior/hyperparameters are known in advance. Arbitrary unseen-task transfer is not assumed.

What would settle it?

A finite-meta-data risk or regret theorem separating prior quality, adaptation data and inference cost, with positive-transfer and negative-transfer regimes or a matching lower bound.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Cannella et al. already establish universal-prior empirical-Bayes guarantees for Poisson problems, explicitly without analyzing architecture or training dynamics; linear ICL task-complexity results also exist. The extension is learned approximate inference with finite pretraining and controlled shift. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Cannella et al. — Universal Priors: Solving Empirical Bayes via Bayesian Inference and Pretraining ↗partial theory · source checked 2026-11-19

    Publisher abstract checked: near-optimal Poisson empirical-Bayes regret through universal priors and contraction; explicitly does not analyze transformer architecture or training dynamics.

  2. Wu et al. — How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? ↗partial theory · source checked 2026-11-19

    Abstract checked: task-complexity bound for single-layer linear attention; near Bayes-optimal ridge regression on unseen tasks.

  3. Wang and Blei — Variational Bayes under Model Misspecification ↗partial theory · source checked 2026-11-19

    Primary abstract checked: asymptotic normality around a KL-minimizing pseudo-true parameter under assumptions; prediction and variational/misspecification errors are analyzed. This does not imply finite-sample coverage for arbitrary neural posteriors.