Data-efficient learning · Statistical inference / Open question
The Best Data for a Fixed Budget
More data is not always better value. A learner should discover which sources provide the most useful evidence for the target task.
Seed catalog · Showing the bundled record.
Problem statement
Suppose data sources mix observable groups with unknown proportions and costs . Within each group inputs follow a common bounded distribution and labels share a linear regression target, but source adds unknown bias ; paid trusted target labels are available. For a known target group mixture and , determine excess target squared risk from jointly estimating source quality and allocating samples, relative to an equally budgeted oracle knowing the mixtures and biases.
Assumptions and scope
- Fix target risk, source families, sampling costs, group/feature access and which source statistics are unknown.
- Specify overlap and bounded source bias or access to trusted target labels; uncontrolled label bias cannot be identified from source data alone.
- Distinguish source selection before data are observed from selecting a subset of a fully available labelled dataset; include the cost of exploration and model updates.
What would settle it?
A finite-budget oracle-risk comparison with matching costs of learning source geometry or bias, and an adaptive acquisition algorithm under a declared identifiable source model.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Harding et al. already derive minimax allocations from known source/target group proportions and effective sample size, with prediction extensions. The proposed target is learning uncertain source properties during acquisition, beyond a fixed known-geometry allocation. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Harding, Singh and Kandasamy — Learning from Biased and Costly Data Sources: Minimax-Optimal Data Collection under a Budget ↗partial theory · source checked 2026-11-19
Publisher abstract checked: effective-sample-size allocation with known group proportions, matching leading mean-estimation bounds, and prediction bounds with a remaining stated factor. A minimax allocation theory for known source geometry already exists.
- Qiao et al. — Counterfactual Online Conformal Prediction Under Adaptive Logging ↗partial theory · source checked 2026-11-19
Primary abstract checked: propensity-weighted and doubly robust recursions; minimax expected coverage rates under positivity and known logging probabilities. The extra target must go beyond these adaptive-logging guarantees.
- Black et al. — Actively Learning Halfspaces without Synthetic Data ↗partial theory · source checked 2026-11-19
Publisher abstract checked: no-synthesis lower bound for arbitrary pools, optimal known-direction/decision-stump results and near-optimal noisy PAC bounds. Point synthesis and label queries on existing data are distinct access models.