A provider of email filters regularly retrains its model on feedback from its users: whoever rescues a message from the spam folder supplies a new training example. A group of attackers creates hundreds of accounts, sends each other their phishing mails and marks them as wanted. After a few training rounds, the filter lets exactly these mails through. Whoever controls the training data controls the behaviour of the model.
Article 15(5) of the AI Act names this attack explicitly. High-risk systems have to withstand attempts by unauthorized parties to alter their outputs or performance by exploiting vulnerabilities, and the required technical solutions include, where appropriate, measures against manipulation of training datasets. ISO/IEC 42001 also lists data poisoning as an example of a risk source in machine learning and as an AI-specific security threat.
What is behind it?
Data poisoning is an attack on the training phase. Attackers add, alter or remove training examples so that the model learns something wrong. Unlike accidental data flaws, the corruption is intended and usually designed to go unnoticed. The result is a model that works correctly in most cases and fails exactly where the attacker wants it to.
Forms of attack
- Flipped labels. Attackers deliberately change the label of individual examples, for instance from “malicious” to “harmless”. Even a small share can noticeably worsen the detection of a particular class.
- Backdoor through the data. Examples carrying an inconspicuous trigger, such as a certain pattern in an image or a sequence of words, receive the result the attacker wants. The model behaves normally until the trigger appears.
- Targeted shift. The data is changed so that a single case or group is treated differently, for example so that a particular counterfeit is never detected.
- Poisoned feedback. Systems that learn from user reactions are pushed in one direction by masses of fake reactions.
Points of entry
Attackers use public data sources and web crawls where they can place their own content. Research has shown that even very large datasets collected from the web can be poisoned with modest effort. Other routes are purchased datasets, external labelling services, insiders with write access to data stores and user data that flows into retraining unchecked.
Impact
Data poisoning undermines the integrity of the system. Depending on the use, it opens security filters to attackers, distorts diagnoses or credit decisions, or damages the provider’s reputation once the attack becomes known. Poisoned models are hard to unmask because they look unremarkable in ordinary tests. Cleaning up is laborious: affected data has to be found and removed, the model retrained and versions already shipped possibly withdrawn.
Practical examples
Labelling service in medical technology. A provider of image analysis software has scans annotated externally. One person at the service provider labels part of the images showing a rare finding as normal. The model later systematically misses this finding, which only comes to light through feedback from a hospital.
Public sources for a coding assistant. A provider trains a programming assistant on publicly available source code. Beforehand, attackers have published numerous projects containing a particular insecure pattern. The assistant now favours this pattern in its suggestions, and it ends up in other people’s applications.
Insider in the data pipeline. At a provider of a fraud detection system, an employee with access to data preparation marks payments from certain merchant accounts as harmless. The model learns to wave these payments through, and the damage only shows at the provider’s customers.
Effective controls
The following ISO/IEC 42001 controls counter this threat.
Prevention:
- A.7 Data for AI systems: The organization documents how data is acquired and selected, records its provenance across the life cycle and sets requirements for quality and preparation. Whether provenance needs separate verification depends on source, content and purpose of use.
- A.6.2 AI system life cycle: The design documentation takes AI-specific security threats into account, and ISO/IEC 42001 explicitly counts data poisoning among them.
- A.10 Third-party and customer relationships: Suppliers of datasets and labelling services are selected according to their risk, bound by obligations and monitored.
Detection:
- A.6.2 AI system life cycle: Verification and validation measure every new model version against defined criteria, and monitoring in operation looks specifically for AI-specific attacks.
Response:
- A.7 Data for AI systems: Provenance records show which data is affected and allow retraining on a cleaned basis.
- A.6.2 AI system life cycle: A failure plan with a return to the tested previous version limits the damage until the cleaned model is ready.
Link to the AI Act
Article 15(5) of the AI Act (Regulation (EU) 2024/1689) requires high-risk systems to be resilient against attempts by unauthorized parties to alter their use, outputs or performance by exploiting vulnerabilities. Against AI-specific vulnerabilities, the technical solutions should, where appropriate, prevent, detect, respond to, resolve and control attacks on the training data. Article 10 governs data governance, including origin, collection and preparation of training data. For systems that keep learning, Article 15(4) requires measures against problematic feedback loops. If a successful attack leads to a serious incident, the providers’ reporting obligation under Article 73 applies (→ EU AI Act).
Related threats
- T09 Model poisoning: There the manipulation hits the model or its pre-trained components directly.
- T04 Poor data quality: The same safeguards help against accident and intent.
- T10 Adversarial examples: Deception at runtime complements the attack on training.
Sources
- Regulation (EU) 2024/1689 (AI Act), Articles 10, 15, 25 and 73
- ISO/IEC 42001:2023, Annex C.3.4 and Annex B.6.2.3 and B.6.2.6 (AI-specific security threats)
- EU AI Act in the Cenedril wiki: obligations for high-risk systems at a glance