← All AI/ML open problems

Privacy & encrypted learning · Neural architecture & complexity / Open question

How Small Can a Private Adapter Be?

A small adapter can reduce private training cost, but learning its orientation and perturbing two coupled factors can consume the apparent advantage. Efficiency payoff: Choose adapter rank and geometry by useful accuracy per private sample and per update, rather than only counting trainable weights.

OpenEditorially scopedFramework needs definitionCompiled 2026-11-20

Seed catalog · Showing the bundled record.

Problem statement

Let XX be uniform on the unit sphere in Rd\mathbb R^d and Y∣X∼softmax⁡((W0+Δ∗)X)Y\mid X\sim\operatorname{softmax}((W_0+\Delta_*)X) over KK labels, where public W0W_0 is fixed independently of private data and Δ∗\Delta_* has rank r∗r_* and Frobenius norm at most BB. Train a rank-rr adapter UV⊤UV^\top from nn private examples. Determine the minimum population cross-entropy excess risk attainable by clipped/noisy factor training at total (εDP,δDP)(\varepsilon_{\rm DP},\delta_{\rm DP}) privacy and arithmetic budget CC, including the cost of learning the update subspace.

Assumptions and scope

  • Use central example-replacement DP over all input datasets, with epsilon_DP in (0,1] and delta_DP<n^(-2). Charge private initialization, rank selection, validation and refactorization whenever they use raw data; transformations of already-private outputs are post-processing.
  • Specify whether one or both factors are trained, their balanced scaling/normalization, initialization, clipping norm and noise injection location. The change U->cU, V->V/c leaves the product fixed but changes factor-gradient sensitivity, so parameter count alone is not a privacy analysis.
  • The baseline oracle knows the optimal rank-r approximation only for defining approximation error, not as free algorithm input. Report r<K,d, parameter bits, dense projection/SVD work and a matched full-update baseline. This softmax benchmark is a specialization inspired by LLM adaptation, not an end-to-end language-model theorem.

What would settle it?

A factor-training risk bound with rank, privacy, subspace-discovery and compute dependence, paired with a private-estimation lower bound or a failure regime. An empirically lower-rank gradient is not a dimension-free guarantee.

Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.

Status and evidence

Private LoRA spectral restoration, FedSVD and private low-rank trace regression already have methods or theory. This target concerns a joint clipped bilinear softmax-training frontier and is editorial; trace-regression bounds or weaker privacy notions cannot be substituted without checking assumptions. This is an editorial, source-motivated research target. The preliminary review through 2026-11-20 does not certify novelty or exclude an unlocated resolution.

Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.

Sources and further reading

  1. Dadsetan and Rudzicz — Re-examining Low-Rank Adaptation for Private LLM Fine-Tuning ↗related algorithm · source checked 2026-11-20

    November revision abstract checked: restoring noisy gradient spectral profiles speeds private fine-tuning empirically. Post-processing already-private gradients does not spend extra privacy, but population-risk improvements require additional analysis.

  2. Lee et al. — FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA ↗partial theory · source checked 2026-11-20

    Primary search abstract checked: adaptive SVD refactorization, noise-amplification mitigation and gradient-norm analysis already exist. Do not claim private changing-subspace adaptation is absent.

  3. Zha — Near-Optimal Differentially Private Low-Rank Trace Regression with Guaranteed Private Initialization ↗partial theory · source checked 2026-11-20

    Primary search abstract checked: private initialization, Riemannian optimization and minimax lower bounds in Gaussian trace regression; the abstract distinguishes its weaker-DP optimality claim. This is not an automatic theorem for clipped bilinear softmax adaptation.

  4. Abadi et al. — Deep Learning with Differential Privacy ↗partial theory · source checked 2026-11-20

    Primary abstract checked: clipped/noisy neural training and refined composition are established. Privacy accounting alone does not prove feature learning or an optimal risk/compute trade-off.