AI safety & alignment · Verification & proof search / Open question
Can Agents Resist Prompt Injection?
Agents must read untrusted material without granting it the authority to redirect their actions or disclose protected information.
Seed catalog · Showing the bundled record.
Problem statement
For a finite-state probabilistic tool-agent language with at most K concurrent workers, capability-labelled tool calls and an explicit declassification policy, characterize enforceable integrity/confidentiality guarantees. The low-observable trace includes tool calls, permission changes, termination and scheduler ticks. Construct a compositional monitor or type discipline that enforces noninterference modulo authorized releases while permitting useful data-dependent plans, and quantify scheduling/padding costs or prove an obstruction within this language.
Assumptions and scope
- Define trusted instructions, untrusted data, tool semantics and the attacker-visible trace. Useful answers may depend on data; only unauthorized influence is prohibited.
- Specify trusted endorsement/declassification rules and enforce them outside the untrusted model. A correct runtime cannot certify that an unspecified natural-language policy captures user intent.
- Bound runtime and scheduling observables or state the permitted leakage; arbitrary timing channels and unrestricted declassification cannot satisfy strict noninterference.
What would settle it?
A compositional operational semantics and machine-checkable theorem with explicit termination-sensitive or quantitative leakage guarantees, plus a meaningful class of adaptive workflows and lower bounds on necessary restrictions.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
CaMeL already enforces control/data separation and tool policies. LLMbda already provides a Lean-verified interpreter with termination-insensitive probabilistic noninterference, including code generation. The target extends the observation model and concurrent permission changes; verified agent noninterference itself is not new. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Debenedetti et al. — Defeating Prompt Injections by Design ↗related algorithm · source checked 2026-11-19
Primary search abstract checked: CaMeL separates trusted control from untrusted data and enforces capability-based tool policies. It already provides a system-level defense rather than relying only on prompt classification.
- Garby, Gordon and Sands — The LLMbda Calculus: AI Agents, Conversations, and Information Flow ↗partial theory · source checked 2026-11-19
Primary abstract checked: a Lean-verified interpreter and termination-insensitive probabilistic noninterference, including code generation and attacker-selected untrusted inputs. The Lean files were not independently checked in this catalogue pass.