Information retrieval · Statistical inference / Open question
Learning Relevance from Biased Feedback
Search engines learn from behavior they helped create. Without identifying the exposure mechanism, more clicks can reinforce an incorrect ranking.
Seed catalog · Showing the bundled record.
Problem statement
For documents and a finite query set, use a cascade click model with unknown relevance , position-examination probabilities and stopping probabilities after each click. Permit per-session model discrepancy at most in total variation. From adaptively displayed lists, logged propensities and costed randomized swaps or relevance audits, determine minimax samples/interventions for top- relevance regret at most , identifying when exposure and relevance remain confounded.
Assumptions and scope
- Fix a finite candidate/query model, a relevance metric and a structured examination/stopping law; clicks alone need not identify relevance separately from exposure.
- Specify which display probabilities are logged and what randomized swaps, additional feedback or trusted labels may be acquired, with their cost charged.
- Distinguish long-term user utility, static relevance and click-through rate; state the allowed feedback drift and overlap or coverage conditions.
What would settle it?
A necessary information condition and a minimax-efficient intervention/ranking procedure for the declared feedback family, including lower bounds and misspecification sensitivity.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Propensity-weighted unbiased learning-to-rank is established; Bayesian inspection/stopping models and counterfactual adaptive-logging methods also exist. The residual target is the joint unknown-feedback, adaptive-exposure and intervention-budget characterization, not basic inverse-propensity correction. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Joachims, Swaminathan and Schnabel — Unbiased Learning-to-Rank with Biased Feedback ↗partial theory · source checked 2026-11-19
Primary abstract checked: counterfactual propensity-weighted ERM for learning from biased clicks. Debiasing with a valid propensity model is established, not an open existence question.
- Ma — Bayesian Rational Search Engine User ↗related result · source checked 2026-11-19
Primary search abstract checked: Bayesian stopping/inspection-depth model and a ranking likelihood, with simulation recovery under correct specification. Unrestricted preference identification from passive clicks does not follow.
- Qiao et al. — Counterfactual Online Conformal Prediction Under Adaptive Logging ↗partial theory · source checked 2026-11-19
Primary abstract checked: propensity-weighted and doubly robust recursions; minimax expected coverage rates under positivity and known logging probabilities. The extra target must go beyond these adaptive-logging guarantees.