Verification & proof search · AI safety & alignment / Open question
The Limits of AI Consensus
More verification calls help only to the extent that they add useful information rather than repeat a common mistake.
Seed catalog · Showing the bundled record.
Problem statement
Model binary step judges as conditionally independent given a latent error mode with r possible values; the unknown mode distribution depends on the task, so a mode can make all judges share a systematic mistake. On finite horizon-H task trees, learn a costed combination of additional judges, critique and trusted labels. Bound aggregate error and downstream policy reward after fitting a process verifier, with explicit dependence on latent-mode identifiability and policy-shift coverage.
Assumptions and scope
- Specify a dependence model for judge errors conditional on task difficulty and a way to estimate its parameters; do not assume independent errors solely because random seeds differ.
- Keep aggregation, sequential critique and label selection explicit. Include the possibility that a confident majority shares one systematic error.
- Separate static label risk from policy-induced distribution shift and bound each under stated coverage assumptions.
What would settle it?
A finite-verification-budget error bound and allocation rule, with lower bounds for correlated blind spots and a downstream policy guarantee. Majority-vote concentration for independent judges alone does not settle it.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
ScalePRM empirically scales verification and documents exploitation; fixed-verifier error theory supplies complementary obstructions. The joint dependent-label and adaptive-optimization guarantee remains the source-motivated target. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Rahman et al. — ScalePRM: Scaling Verification Compute Without Ground Truth ↗empirical motivation · source checked 2026-11-19
EMNLP publisher abstract checked: synthetic step labels from self-consistency/meta-critique and observed reward exploitation. Learned process scores are not kernel certificates.
- Moya, Thornley and Lin — Verifier Errors in RLVR ↗partial theory · source checked 2026-11-19
Primary abstract checked: fixed-verifier gradient flow, feedback-identifiability limitations and audit-based selective control are already analyzed.