← All AI/ML open problems

Data-efficient learning · Statistical inference / Open question

The Best Data for a Fixed Budget

More data is not always better value. A learner should discover which sources provide the most useful evidence for the target task.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

Suppose KK data sources mix GG observable groups with unknown proportions πjg\pi_{jg} and costs cjc_j. Within each group inputs follow a common bounded distribution and labels share a linear regression target, but source jj adds unknown bias ∣bjg∣≤η|b_{jg}|\leq\eta; paid trusted target labels are available. For a known target group mixture and ∑jcjnj≤C\sum_j c_j n_j\leq C, determine excess target squared risk from jointly estimating source quality and allocating samples, relative to an equally budgeted oracle knowing the mixtures and biases.

Assumptions and scope

  • Fix target risk, source families, sampling costs, group/feature access and which source statistics are unknown.
  • Specify overlap and bounded source bias or access to trusted target labels; uncontrolled label bias cannot be identified from source data alone.
  • Distinguish source selection before data are observed from selecting a subset of a fully available labelled dataset; include the cost of exploration and model updates.

What would settle it?

A finite-budget oracle-risk comparison with matching costs of learning source geometry or bias, and an adaptive acquisition algorithm under a declared identifiable source model.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Harding et al. already derive minimax allocations from known source/target group proportions and effective sample size, with prediction extensions. The proposed target is learning uncertain source properties during acquisition, beyond a fixed known-geometry allocation. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Harding, Singh and Kandasamy — Learning from Biased and Costly Data Sources: Minimax-Optimal Data Collection under a Budget ↗partial theory · source checked 2026-11-19

    Publisher abstract checked: effective-sample-size allocation with known group proportions, matching leading mean-estimation bounds, and prediction bounds with a remaining stated factor. A minimax allocation theory for known source geometry already exists.

  2. Qiao et al. — Counterfactual Online Conformal Prediction Under Adaptive Logging ↗partial theory · source checked 2026-11-19

    Primary abstract checked: propensity-weighted and doubly robust recursions; minimax expected coverage rates under positivity and known logging probabilities. The extra target must go beyond these adaptive-logging guarantees.

  3. Black et al. — Actively Learning Halfspaces without Synthetic Data ↗partial theory · source checked 2026-11-19

    Publisher abstract checked: no-synthesis lower bound for arbitrary pools, optimal known-direction/decision-stump results and near-optimal noisy PAC bounds. Point synthesis and label queries on existing data are distinct access models.