← All AI/ML open problems

AI safety & alignment · Learning dynamics & scaling / Open question

Safety That Survives Fine-Tuning

A safety property is fragile if an inexpensive model update removes it. A theory of persistence should explain both attainable resistance and its limits.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-19

Seed catalog · Showing the bundled record.

Problem statement

For a two-layer ReLU classifier with fixed width and bounded rational initialization, define safety as zero forbidden-label predictions on a specified rational input polytope and utility as risk on a separate fixed distribution. An adversary may take T minibatch gradient steps with bounded step size and bounded-norm examples, but may not replace arbitrary weights. Characterize the minimum T and data budget needed to violate safety while meeting the utility threshold, and whether initialization/training can provably increase that cost.

Assumptions and scope

  • Define safety through a precise output or capability predicate and a stated input distribution or perturbation set; refusal on a benchmark is only one observable.
  • Fix the permitted weight edits, adapter rank, step sizes, data access, reparameterizations and compute accounting. Arbitrary model replacement makes persistent weight-level safeguards impossible.
  • Measure retained utility and recovery difficulty separately; a certificate must cover the declared update class rather than only attacks used during evaluation.

What would settle it?

A nonvacuous lower bound on safety-violation cost for a useful model family, with an achievable construction or a matching upper-bound attack within the declared mathematical update model.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

TAR and pretraining suppression already improve measured tamper resistance. Rosati et al. show that mutual information at release alone cannot certify slow recovery because parameterization changes the dynamics. The target is a dynamics-aware guarantee under bounded updates, not an information-only certificate. This is an editorial, source-motivated research target. The preliminary review through 2026-11-19 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Tamirisa et al. — Tamper-Resistant Safeguards for Open-Weight LLMs ↗empirical motivation · source checked 2026-11-19

    Primary search abstract checked: TAR improves resistance to tested weight-tampering/fine-tuning attacks. Finite red-team success is not a certificate against every update in a threat class.

  2. Rosati et al. — Removing Information Content Does Not Certify Tamper Resistance in Open-Weight Models ↗partial theory · source checked 2026-11-19

    Primary search abstract checked: function-preserving reparameterizations change recovery dynamics at fixed information; examples recover in one gradient step even at zero stated mutual information. Attack-dynamics restrictions are necessary for the proposed certificate.

  3. Glarou and Jaggi — DETER: Suppression During Pretraining for Tamper Resistance ↗empirical motivation · source checked 2026-11-19

    Primary search abstract checked: empirical trade-offs between suppression, retained capability and later fine-tuning cost. These experiments motivate a recovery-complexity theorem.