Privacy & encrypted learning · Neural architecture & complexity / Open question
How Small Can a Private Adapter Be?
A small adapter can reduce private training cost, but learning its orientation and perturbing two coupled factors can consume the apparent advantage. Efficiency payoff: Choose adapter rank and geometry by useful accuracy per private sample and per update, rather than only counting trainable weights.
Seed catalog · Showing the bundled record.
Problem statement
Let be uniform on the unit sphere in and over labels, where public is fixed independently of private data and has rank and Frobenius norm at most . Train a rank- adapter from private examples. Determine the minimum population cross-entropy excess risk attainable by clipped/noisy factor training at total privacy and arithmetic budget , including the cost of learning the update subspace.
Assumptions and scope
- Use central example-replacement DP over all input datasets, with epsilon_DP in (0,1] and delta_DP<n^(-2). Charge private initialization, rank selection, validation and refactorization whenever they use raw data; transformations of already-private outputs are post-processing.
- Specify whether one or both factors are trained, their balanced scaling/normalization, initialization, clipping norm and noise injection location. The change U->cU, V->V/c leaves the product fixed but changes factor-gradient sensitivity, so parameter count alone is not a privacy analysis.
- The baseline oracle knows the optimal rank-r approximation only for defining approximation error, not as free algorithm input. Report r<K,d, parameter bits, dense projection/SVD work and a matched full-update baseline. This softmax benchmark is a specialization inspired by LLM adaptation, not an end-to-end language-model theorem.
What would settle it?
A factor-training risk bound with rank, privacy, subspace-discovery and compute dependence, paired with a private-estimation lower bound or a failure regime. An empirically lower-rank gradient is not a dimension-free guarantee.
Selected for a theoretical or computational target. A fitted scaling curve or improved benchmark score alone does not establish the stated learning-theoretic guarantee.
Status and evidence
Private LoRA spectral restoration, FedSVD and private low-rank trace regression already have methods or theory. This target concerns a joint clipped bilinear softmax-training frontier and is editorial; trace-regression bounds or weaker privacy notions cannot be substituted without checking assumptions. This is an editorial, source-motivated research target. The preliminary review through 2026-11-20 does not certify novelty or exclude an unlocated resolution.
Preliminary source check; not an expert-verified open status or an independent audit of proposed proofs. The compilation date is not a literature-review cutoff.
Sources and further reading
- Dadsetan and Rudzicz — Re-examining Low-Rank Adaptation for Private LLM Fine-Tuning ↗related algorithm · source checked 2026-11-20
November revision abstract checked: restoring noisy gradient spectral profiles speeds private fine-tuning empirically. Post-processing already-private gradients does not spend extra privacy, but population-risk improvements require additional analysis.
- Lee et al. — FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA ↗partial theory · source checked 2026-11-20
Primary search abstract checked: adaptive SVD refactorization, noise-amplification mitigation and gradient-norm analysis already exist. Do not claim private changing-subspace adaptation is absent.
- Zha — Near-Optimal Differentially Private Low-Rank Trace Regression with Guaranteed Private Initialization ↗partial theory · source checked 2026-11-20
Primary search abstract checked: private initialization, Riemannian optimization and minimax lower bounds in Gaussian trace regression; the abstract distinguishes its weaker-DP optimality claim. This is not an automatic theorem for clipped bilinear softmax adaptation.
- Abadi et al. — Deep Learning with Differential Privacy ↗partial theory · source checked 2026-11-20
Primary abstract checked: clipped/noisy neural training and refined composition are established. Privacy accounting alone does not prove feature learning or an optimal risk/compute trade-off.