← All AI/ML open problems

Verification & proof search · AI safety & alignment / Open question

The Limits of AI Consensus

More verification calls help only to the extent that they add useful information rather than repeat a common mistake.

OpenEditorially scopedSpecified modelCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

Model binary step judges as conditionally independent given a latent error mode with r possible values; the unknown mode distribution depends on the task, so a mode can make all judges share a systematic mistake. On finite horizon-H task trees, learn a costed combination of additional judges, critique and trusted labels. Bound aggregate error and downstream policy reward after fitting a process verifier, with explicit dependence on latent-mode identifiability and policy-shift coverage.

Assumptions and scope

  • Specify a dependence model for judge errors conditional on task difficulty and a way to estimate its parameters; do not assume independent errors solely because random seeds differ.
  • Keep aggregation, sequential critique and label selection explicit. Include the possibility that a confident majority shares one systematic error.
  • Separate static label risk from policy-induced distribution shift and bound each under stated coverage assumptions.

What would settle it?

A finite-verification-budget error bound and allocation rule, with lower bounds for correlated blind spots and a downstream policy guarantee. Majority-vote concentration for independent judges alone does not settle it.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

ScalePRM empirically scales verification and documents exploitation; fixed-verifier error theory supplies complementary obstructions. The joint dependent-label and adaptive-optimization guarantee remains the source-motivated target. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Rahman et al. — ScalePRM: Scaling Verification Compute Without Ground Truth ↗empirical motivation · source checked 2026-11-19

    EMNLP publisher abstract checked: synthetic step labels from self-consistency/meta-critique and observed reward exploitation. Learned process scores are not kernel certificates.

  2. Moya, Thornley and Lin — Verifier Errors in RLVR ↗partial theory · source checked 2026-11-19

    Primary abstract checked: fixed-verifier gradient flow, feedback-identifiability limitations and audit-based selective control are already analyzed.