AI safety & alignment · Embeddings & representations / Open question
Learning Beyond the Teacher
Can a learner reliably become better than the source of its supervision? The answer matters whenever strong models are trained using weaker judges.
Seed catalog · Showing the bundled record.
Problem statement
Let and , with unknown orthonormal and . The weak teacher predicts . Fine-tune a two-layer ReLU student on pseudolabels plus a budget of trusted labels, starting from pretrained features with bounded alignment to both projections. Determine when finite-data feature learning beats the teacher's classification risk and when it reinforces the missing-feature bias.
Assumptions and scope
- Fix a teacher-error model and latent task structure; the student's pretraining information must be specified rather than supplied through access to ground truth.
- Specify which student layers change, the optimizer, loss and confidence filtering. Convex projection onto a fixed function class is only a baseline.
- State trusted-label access and charge any model-selection audit. Without identifying assumptions or trusted information, arbitrary systematic teacher errors cannot be detected from pseudolabels alone.
What would settle it?
An end-to-end finite-data training theorem giving a strict risk improvement and its necessary conditions, with a failure construction for incompatible teacher errors or destructive feature drift.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Misfit-based squared and Bregman-loss results, expansion-based correction, benign-overfitting phases and nonlinear multi-index mechanisms are established. The proposed extension couples finite-sample representation drift, optimization and an auditable gain criterion. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Charikar, Pabbaraju and Shiragur — Quantifying the Gain in Weak-to-Strong Generalization ↗partial theory · source checked 2026-11-19
Abstract checked: for the squared-loss setting, the strong model's gain is quantified by its misfit on weak labels.
- Mulgund and Pabbaraju — Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss ↗partial theory · source checked 2026-11-19
Abstract checked: Bregman-loss extension for convex strong classes; convex combinations of k strong models with a vanishing error term.
- Lang, Sontag and Vijayaraghavan — Theoretical Analysis of Weak-to-Strong Generalization ↗partial theory · source checked 2026-11-19
Abstract checked: expansion-based bounds for pseudolabel correction and coverage expansion, checkable from finite data.
- Moniri and Hassani — On the Mechanisms of Weak-to-Strong Generalization ↗partial theory · source checked 2026-11-19
Abstract checked: ridge, weighted-ridge and multi-index mechanisms by which a student surpasses its teacher.
- Wu and Sahai — Provable Weak-to-Strong Generalization via Benign Overfitting ↗partial theory · source checked 2026-11-19
Abstract checked: two asymptotic phases in an overparameterized spiked-covariance classification model.