Tabular learning · Neural architecture & complexity / Open question
Compiling a Table into a Tiny Model
Repeatedly attending to the training table is expensive when many queries share the same task. A learned compiler could pay that cost once. Efficiency payoff: Turn expensive in-context tabular inference into a small standalone predictor with controlled statistical and compression error.
Seed catalog · Showing the bundled record.
Problem statement
Draw regression tasks by choosing of uniform-input coordinates and an additive degree- Fourier function with Gaussian coefficients and unit Gaussian label noise. From independent training tables of rows, train a permutation-invariant hypernetwork mapping a new table to a ReLU predictor with at most parameters encoded in bits each. Determine the achievable trade-off in , compiler training and one-time compilation cost.
Assumptions and scope
- The task prior, coefficient variances and query distribution are fixed and known; f_D^Bayes is the exact posterior mean under that prior. Fresh test tables are independent. Account separately for Bayes estimation risk if comparing to the true task function.
- Specify the table encoder, weight-output decoder, student architecture, objective, initialization and optimizer. At deployment the student may use only its emitted weights; retaining the original table or a growing attention cache is excluded from the compact-model budget.
- Count quantizer scales and metadata in total student bits, as well as compiler parameters. Report amortization over a stated number of prediction queries. Exact Bayes teachers, posterior samples or solved training labels have acquisition costs and cannot be assumed free.
What would settle it?
A finite-meta-training and finite-computation compilation theorem, with a model-bit or information lower bound and an explicit query-count break-even point relative to in-context prediction.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
MotherNet already generates child networks, TabPFN-2.5 includes distillation and TabICD compresses contexts. The proposed prior-specific risk/bit/training frontier is an editorial extension, not a first architecture claim; approximation existence alone leaves the hypernetwork training question unresolved. This is an editorial, source-motivated research target. The preliminary review through 2026-11-20 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Mueller, Curino and Ramakrishnan — MotherNet: Fast Training and Inference via Hyper-Network Transformers ↗related algorithm · source checked 2026-11-20
Primary abstract checked: hypernetwork compilation of a table into child-network weights already exists. The target is a training/encoding/risk trade-off, not the invention of table-to-model compilation.
- Grinsztajn et al. — TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models ↗related algorithm · source checked 2026-11-20
Primary search abstract checked in the tabular pass: larger tables and an existing MLP/tree distillation engine. Practical distillation must be retained as a baseline.
- Ma et al. — In-Context Data Distillation with TabPFN ↗related algorithm · source checked 2026-11-20
Primary search abstract checked: optimization of a fixed-size synthetic context saves memory at a tuning cost. A distilled context and a standalone compiled predictor have different deployment costs.
- Raskutti, Wainwright and Yu — Minimax-Optimal Rates for Sparse Additive Models over Kernel Classes via Convex Programming ↗partial theory · source checked 2026-11-20
Primary abstract checked: matching L2 estimation bounds for sparse sums of bounded-kernel univariate RKHS functions, including Sobolev classes. These statistical rates are established; the new target concerns a finite gradient-trained neural implementation.