“Stealing a model through its API” describes two attacks that share nothing but a name. One recovers parameters. The other recovers behaviour. They have different costs, different feasibility and different defences, and the confusion between them produces a lot of bad advice.
Two different attacks with one name
| Attack | Description |
|---|---|
| parameter extraction | Recovering actual weights or structural facts about the model by exploiting the mathematics of what the API returns. Precise, and limited to what the exposed outputs mathematically determine. |
| behavioural cloning | Querying at volume and training a smaller model on the outputs. Recovers capability on the queried distribution, never the weights. This is distillation, and it works. |
Extraction of a full frontier model’s weights through an API is not a realistic threat today. Cloning a specific capability of one for a few thousand dollars is routine, and it is what most people who say “model extraction” are actually worried about.
A third attack is regularly filed under the same heading and is genuinely different again: recovering training data rather than the model. Membership inference asks whether a particular record was in the training set, and extraction of memorised sequences has been demonstrated repeatedly against language models — the risk is highest for data that appeared verbatim and rarely, which is precisely the shape of a private document in a fine-tuning corpus. If you fine-tune on customer data, that is the attack to model, and its defences — deduplication, filtering secrets before training, and not fine-tuning on data whose disclosure would matter — have nothing in common with the two above.
What has actually been demonstrated
The academic line starts with Tramèr and colleagues in 2016 (“Stealing Machine Learning Models via Prediction APIs”), which showed exact or near-exact recovery of simple models — logistic regression, decision trees, small networks — from prediction APIs that returned confidence values. The lesson was already the durable one: every extra bit of information in a response is an extra equation for the attacker.
For production language models, the significant result is Carlini and colleagues, “Stealing Part of a Production Language Model” (2024). Working with the vendors and under responsible disclosure, they showed that structural information — the hidden dimension, and the final embedding projection layer — can be recovered through a logit-exposing API for a modest query cost, because the returned logit vectors lie in a subspace whose rank is the hidden dimension. The providers involved changed their APIs in response.
Read that result carefully, because it is frequently overstated. It recovered the final layer and a structural parameter, not the model. What it proved is the general principle: the outputs of an API are a system of equations, and if you expose enough of them you have published the solution.
The API surface that widens it
- Full log-probability vectors. The most information-rich response possible. Returning the top few, rather than a full-vocabulary distribution, is the change that closed the 2024 attack.
- Logit bias. The ability to bias arbitrary tokens lets an attacker interrogate parts of the distribution the API would not otherwise reveal — combined with top-k probabilities, it can be used to reconstruct more than the top-k alone shows.
- Deterministic sampling. Temperature zero makes queries repeatable, which is what a solver needs. Nondeterminism adds noise to every equation.
- Unbounded volume. Both attacks are query-count attacks. Volume is the resource being spent.
- Verbose reasoning traces. For cloning, exposing full chain-of-thought is unusually valuable training material — it is supervision, not just labels.
Defences and what each costs
- Return less. Cap log-probabilities to a small top-k or remove them; restrict logit bias. Cost: legitimate uses exist — classification confidence, evaluation tooling — so scope the capability to keys that need it rather than removing it globally.
- Limit volume, in tokens. Directly raises the price of both attacks. Cost: the usual tension with high-volume legitimate customers, which is why the limit should be per key with an approval path.
- Detect the query distribution. Extraction traffic looks unlike product traffic: systematic coverage, repeated near-identical prompts, log-probability requests, uniform arrival. These are the signals in
-
SOCIAL SHARE CARD GENERATOR