A hospital group gives research partners access to a model that estimates the risk of a rare metabolic disorder from lab values. It was trained on data from the specialist outpatient clinic for exactly this disorder. The partners see no patient data, only an interface. One partner systematically queries the model with the values of known people and reads from the confidence of the answers who was very likely in the training dataset, and is therefore a patient of that clinic. A model can reveal more about its training data than the people in charge of it suspect.
Article 15(5) of the AI Act counts confidentiality attacks among the AI-specific vulnerabilities high-risk systems must be prepared for. ISO/IEC 42001 also explicitly names model stealing and model inversion as security threats to be considered for AI systems.
What is behind it?
A trained model carries information from its training data in condensed form. At the same time, the model itself is a valuable asset that embodies data, computing time and know-how. Both can be attacked, often solely through the access a system offers anyway: the ability to ask questions and receive answers.
Types of attack
- Membership inference. Attackers check whether a particular record was part of the training. This weighs heavily when membership of the dataset in itself reveals something sensitive.
- Model inversion. From the outputs, attackers reconstruct typical features of the training data, such as the approximate face of a person from a face recognition model.
- Extraction of training data. Large language models can reproduce parts of their training texts verbatim. Research teams have pulled names, addresses and other personal details out of models this way.
- Model extraction. With masses of queries, attackers build a substitute model that behaves similarly. The intellectual property is lost, and the copy serves as a practice target for further attacks.
- Leakage through connected data. With language models that can access a document store, clever requests can reveal content or internal system instructions that should remain hidden from the person asking.
What favours the attacks
Models that memorize their training data too closely give away more. Detailed outputs such as probabilities reveal more than a plain result. Unlimited queries without monitoring invite systematic probing. And personal data that was never needed for the purpose increases the potential damage.
Impact
If personal data is disclosed, there is a data breach that in most cases has to be reported, and with health or financial data the harm to the people affected is considerable. If the model leaks, the organization loses a competitive edge and possibly its protection as a trade secret. Every answer a model gives reveals a small piece of information, and taken together these pieces can add up to a large leak.
Practical examples
Copying a valuation model. A provider offers an interface that estimates property values. A competitor queries it hundreds of thousands of times with systematically varied inputs and trains its own model on the answers. The terms of use forbid this, yet without limits on and monitoring of queries nobody notices.
Internal assistant with overly broad access. A company connects a language assistant to its document repository. The assistant searches all documents regardless of the rights of the person asking. A trainee asks about the salary structure and receives an excerpt from the management’s remuneration list.
Fine-tuned model with customer data. A provider fine-tunes a customer service model on real conversation transcripts, including names and contract numbers. With certain inputs, the model reproduces fragments of other customers’ conversations. One customer posts such a find in a forum.
Effective controls
The following ISO/IEC 42001 controls counter this threat.
Prevention:
- A.7 Data for AI systems: Data management for development takes the privacy and security implications into account. At acquisition, data rights and prior use are checked, and the criteria for data preparation can provide for removing personal attributes that are not needed.
- A.6.2 AI system life cycle: The design takes AI-specific security threats into account, among which ISO/IEC 42001 counts model stealing and model inversion. Verification and validation can include targeted tests for these attacks.
- A.10 Third-party and customer relationships: The data protection roles, such as controller and processor, and the other responsibilities are settled between the parties involved.
Detection:
- A.6.2 AI system life cycle: Monitoring and event logs reveal unusual query patterns such as mass queries or systematically varied inputs.
Response:
- A.8 Information for interested parties: A plan for communicating incidents to users is in place, and the organization knows its reporting obligations. ISO/IEC 42001 points out specifically that a breach of personal data in training data can trigger particular reporting requirements.
- A.6.2 AI system life cycle: The failure plan makes it possible to restrict access, coarsen the outputs or retrain the model without the affected data.
Link to the AI Act
Article 15(5) of the AI Act (Regulation (EU) 2024/1689) requires high-risk systems to have, where appropriate, measures against confidentiality attacks. If a provider exceptionally processes special categories of personal data under Article 10(5) to detect bias, strict safeguards apply, such as pseudonymization, access restrictions and deletion once the work is done. The GDPR applies alongside, with data protection by design (Article 25), security of processing (Article 32) and the breach notification duties (Articles 33 and 34) (→ GDPR). For a model to be protected as a trade secret under German law, the Trade Secrets Act requires appropriate confidentiality measures (→ GeschGehG).
Related threats
- T10 Adversarial examples: Injected instructions are a common way to make language models blab.
- T09 Model poisoning: Prepared model files can siphon off data from the development environment.
- MU-03 Entering confidential or personal data: What users enter can later surface in training data and outputs.
Sources
- Regulation (EU) 2024/1689 (AI Act), Articles 10 and 15
- Regulation (EU) 2016/679 (GDPR), Articles 25, 32, 33 and 34
- ISO/IEC 42001:2023, Annex B.6.2.3, B.6.2.6 and B.8.4
- European Data Protection Board (EDPB), Opinion 28/2024 on data protection aspects of AI models, December 2024