← All AI/ML open problems

Information retrieval · Statistical inference / Open question

Learning Relevance from Biased Feedback

Search engines learn from behavior they helped create. Without identifying the exposure mechanism, more clicks can reinforce an incorrect ranking.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

For NN documents and a finite query set, use a cascade click model with unknown relevance rqd∈[0,1]r_{qd}\in[0,1], position-examination probabilities ej≥emin⁡e_j\geq e_{\min} and stopping probabilities after each click. Permit per-session model discrepancy at most η\eta in total variation. From adaptively displayed lists, logged propensities and costed randomized swaps or relevance audits, determine minimax samples/interventions for top-kk relevance regret at most ε\varepsilon, identifying when exposure and relevance remain confounded.

Assumptions and scope

  • Fix a finite candidate/query model, a relevance metric and a structured examination/stopping law; clicks alone need not identify relevance separately from exposure.
  • Specify which display probabilities are logged and what randomized swaps, additional feedback or trusted labels may be acquired, with their cost charged.
  • Distinguish long-term user utility, static relevance and click-through rate; state the allowed feedback drift and overlap or coverage conditions.

What would settle it?

A necessary information condition and a minimax-efficient intervention/ranking procedure for the declared feedback family, including lower bounds and misspecification sensitivity.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Propensity-weighted unbiased learning-to-rank is established; Bayesian inspection/stopping models and counterfactual adaptive-logging methods also exist. The residual target is the joint unknown-feedback, adaptive-exposure and intervention-budget characterization, not basic inverse-propensity correction. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Joachims, Swaminathan and Schnabel — Unbiased Learning-to-Rank with Biased Feedback ↗partial theory · source checked 2026-11-19

    Primary abstract checked: counterfactual propensity-weighted ERM for learning from biased clicks. Debiasing with a valid propensity model is established, not an open existence question.

  2. Ma — Bayesian Rational Search Engine User ↗related result · source checked 2026-11-19

    Primary search abstract checked: Bayesian stopping/inspection-depth model and a ranking likelihood, with simulation recovery under correct specification. Unrestricted preference identification from passive clicks does not follow.

  3. Qiao et al. — Counterfactual Online Conformal Prediction Under Adaptive Logging ↗partial theory · source checked 2026-11-19

    Primary abstract checked: propensity-weighted and doubly robust recursions; minimax expected coverage rates under positivity and known logging probabilities. The extra target must go beyond these adaptive-logging guarantees.