Bayesian learning · Data-efficient learning · Embeddings & representations / Open question
How Much Is Prior Knowledge Worth?
Prior experience should make a learner data-efficient, but a learned prior can also transfer the wrong assumptions. The value of pretraining needs a quantitative account.
Seed catalog · Showing the bundled record.
Problem statement
Let each task draw on and generate , with Gaussian inputs/noise and unknown shared orthonormal features. From tasks with samples each, learn both a prior and an amortized posterior-mean estimator for a new task with examples. Bound excess prediction risk relative to the oracle knowing , charging inference-network training; quantify degradation when the test prior has density ratio at most relative to .
Assumptions and scope
- Fix the task hyperdistribution, likelihood family, meta-training sample structure and shift class; separate the number of tasks from samples within each task.
- Specify the prior-learning and inference algorithms, including approximation and optimization errors; an exact Bayes estimator is a baseline, not a free neural training result.
- Compare equal total data and computation budgets and state which prior/hyperparameters are known in advance. Arbitrary unseen-task transfer is not assumed.
What would settle it?
A finite-meta-data risk or regret theorem separating prior quality, adaptation data and inference cost, with positive-transfer and negative-transfer regimes or a matching lower bound.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Cannella et al. already establish universal-prior empirical-Bayes guarantees for Poisson problems, explicitly without analyzing architecture or training dynamics; linear ICL task-complexity results also exist. The extension is learned approximate inference with finite pretraining and controlled shift. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Cannella et al. — Universal Priors: Solving Empirical Bayes via Bayesian Inference and Pretraining ↗partial theory · source checked 2026-11-19
Publisher abstract checked: near-optimal Poisson empirical-Bayes regret through universal priors and contraction; explicitly does not analyze transformer architecture or training dynamics.
- Wu et al. — How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? ↗partial theory · source checked 2026-11-19
Abstract checked: task-complexity bound for single-layer linear attention; near Bayes-optimal ridge regression on unseen tasks.
- Wang and Blei — Variational Bayes under Model Misspecification ↗partial theory · source checked 2026-11-19
Primary abstract checked: asymptotic normality around a KL-minimizing pseudo-true parameter under assumptions; prediction and variational/misspecification errors are analyzed. This does not imply finite-sample coverage for arbitrary neural posteriors.