← All AI/ML open problems

AI safety & alignment · Verification & proof search / Open question

Can Agents Resist Prompt Injection?

Agents must read untrusted material without granting it the authority to redirect their actions or disclose protected information.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

For a finite-state probabilistic tool-agent language with at most K concurrent workers, capability-labelled tool calls and an explicit declassification policy, characterize enforceable integrity/confidentiality guarantees. The low-observable trace includes tool calls, permission changes, termination and scheduler ticks. Construct a compositional monitor or type discipline that enforces noninterference modulo authorized releases while permitting useful data-dependent plans, and quantify scheduling/padding costs or prove an obstruction within this language.

Assumptions and scope

  • Define trusted instructions, untrusted data, tool semantics and the attacker-visible trace. Useful answers may depend on data; only unauthorized influence is prohibited.
  • Specify trusted endorsement/declassification rules and enforce them outside the untrusted model. A correct runtime cannot certify that an unspecified natural-language policy captures user intent.
  • Bound runtime and scheduling observables or state the permitted leakage; arbitrary timing channels and unrestricted declassification cannot satisfy strict noninterference.

What would settle it?

A compositional operational semantics and machine-checkable theorem with explicit termination-sensitive or quantitative leakage guarantees, plus a meaningful class of adaptive workflows and lower bounds on necessary restrictions.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

CaMeL already enforces control/data separation and tool policies. LLMbda already provides a Lean-verified interpreter with termination-insensitive probabilistic noninterference, including code generation. The target extends the observation model and concurrent permission changes; verified agent noninterference itself is not new. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Debenedetti et al. — Defeating Prompt Injections by Design ↗related algorithm · source checked 2026-11-19

    Primary search abstract checked: CaMeL separates trusted control from untrusted data and enforces capability-based tool policies. It already provides a system-level defense rather than relying only on prompt classification.

  2. Garby, Gordon and Sands — The LLMbda Calculus: AI Agents, Conversations, and Information Flow ↗partial theory · source checked 2026-11-19

    Primary abstract checked: a Lean-verified interpreter and termination-insensitive probabilistic noninterference, including code generation and attacker-selected untrusted inputs. The Lean files were not independently checked in this catalogue pass.