Learning dynamics & scaling · Statistical learning theory / Open question
Reconciling Kaplan and Chinchilla Exponents
Kaplan's commonly quoted adjusted parameter/data allocation is approximately C^0.73/C^0.27, whereas Chinchilla favors roughly C^0.50/C^0.50. Existing reconciliations explain much of this gap; reliable future scaling decisions require knowing which corrected differences can actually be resolved with finite evidence. Efficiency payoff: Avoid choosing an oversized model or an undersized training corpus because of biased scaling fits, incomplete FLOP accounting or uncharged optimizer tuning.
Seed catalog · Showing the bundled record.
Problem statement
On a bounded log-parameter/log-token rectangle, model pretraining loss by for finitely many training protocols . Use audited cost . From adaptively selected noisy pilot runs, determine the least total pilot compute for valid confidence bands on true-cost optimal total size and its finite-window allocation exponent. Quantify when apparent Kaplan-like 0.73 versus Chinchilla-like 0.50 slopes are explained by accounting, optimization and fitting effects, and when a residual difference is identifiable.
Assumptions and scope
- Here counts core parameters, is a fixed-aspect-ratio dense-transformer counting model, and counts processed training tokens without reuse. Known , encode vocabulary/head, attention and core costs. They must be checked against the actual architecture: tied storage does not eliminate output-head FLOPs, and total stored parameters need not equal per-token arithmetic. Include discretization error when continuous N is implemented by a finite model grid.
- Unknown lie in supplied compact positive intervals; lies in a supplied nonnegative interval. The nonnegative protocol residual obeys in log coordinates, bounding its value and derivatives through order two. Fix the finite AdamW recipe set before sampling, including warmup, batch size, learning rate and moment coefficients. These residual envelopes are hypotheses requiring independent justification; they are not consequences of a good fitted power law.
- Each pilot returns with conditionally centered -sub-Gaussian noise and costs ; all tuning and pilot runs count toward . Use fresh independent runs in this model; shared checkpoints/evaluation dependence needs a separate noise analysis. Require an interior true-cost optimizer and supplied positive curvature in log N along each IsoFLOP curve, or return a set of optimal allocations.
- At fixed log-spaced budgets , define . Seek simultaneous coverage and expose attainable width in pilot compute, noise, residual envelopes, curvature and budget spacing. Distinguish this allocation exponent from loss exponents and from slopes measured using the proxy . No universal limiting exponent or arbitrary extrapolation is assumed.
What would settle it?
A costed adaptive design and inference theorem with matching pilot-budget or partial-identification lower bounds for corrected finite-window exponents, including a quantitative criterion for distinguishing a residual gap from the protocol-error floor. In the zero-residual, core-dominated baseline recover a=beta/(alpha+beta). Re-deriving that algebra or reproducing the historical reconciliation alone is not the new result.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
The historical discrepancy is substantially reconciled by Pearce–Song's counting/finite-scale analysis and Porian et al.'s controlled head-cost, warmup and optimizer ablations; careful decay is not necessary in their tested regimes. Replication, robustness and 2026 IsoFLOP fitting-bias studies supply further baselines. The retained question is an editorial finite-budget uncertainty/identifiability extension of those results. The loss-surface/residual class is explicit but must be justified before claiming a theorem about real transformer training; it does not derive universal exponents from first principles. This is an editorial, source-motivated research target. The preliminary review through 2026-11-20 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Kaplan et al. — Scaling Laws for Neural Language Models ↗empirical motivation · source checked 2026-11-20
Primary abstract and the later reconciliation papers checked: empirical pretraining loss/resource laws and the commonly quoted, batch-adjusted N_opt proportional to C^0.73 allocation. Non-embedding parameter/FLOP conventions and the unadjusted 0.88 fit must be distinguished; these are not universal proved exponents.
- Hoffmann et al. — Training Compute-Optimal Large Language Models ↗empirical motivation · source checked 2026-11-20
Primary abstract and subsequent reanalyses checked: roughly balanced model/token scaling from over 400 training runs. The headline allocation exponent near one half differs from the individual loss exponents and from the original Approach 3 point estimate; exact universality is not established.
- Pearce and Song — Reconciling Kaplan and Chinchilla Scaling Laws ↗partial theory · source checked 2026-11-20
HTML Sections 2–5 checked: an assumed Chinchilla loss surface, total/core count conversion and finite-scale local fits already reconcile much of the discrepancy. With N_total=N_core+omega N_core^(1/3), the overhead-dominated apparent exponent becomes beta/(alpha/3+beta), versus beta/(alpha+beta) at large scale. This conditional algebra is an established baseline.
- Porian et al. — Resolving Discrepancies in Compute-Optimal Scaling of Language Models ↗related result · source checked 2026-11-20
HTML Sections 2–4, Table 1 and Appendix H discussion checked: output-head FLOPs, excessive warmup and scale-dependent optimizer tuning explain the discrepancy on two corpora. Tuned constant-rate runs reproduce approximately one-half scaling; matched cosine decay is not essential in these experiments. Their parameter count includes the untied output head but excludes input embeddings.
- Besiroglu et al. — Chinchilla Scaling: A Replication Attempt ↗status context · source checked 2026-11-20
Scholarly abstract checked: reconstructed-data reanalysis questions the original third fitting method and confidence intervals while obtaining estimates compatible with the first two methods. This is a replication/reanalysis, not access to all original raw runs or a new universal exponent.
- Schaeffer et al. — Evaluating the Robustness of Chinchilla Compute-Optimal Scaling ↗related result · source checked 2026-11-20
Scholarly abstract checked: the main balanced-allocation conclusions withstand several parameter-count perturbations, with greater sensitivity to systematic/additive distortions. Not every reasonable counting ambiguity invalidates Chinchilla scaling.
- Czech et al. — Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fits ↗partial theory · source checked 2026-11-20
Scholarly abstract checked: grid width/centering and asymmetric loss surfaces bias parabolic IsoFLOP fits, even without observation noise; variable projection improves fitting the specified five-parameter surface. Basic fitting-bias diagnosis and a better parametric estimator already exist.