A young company develops a system that checks documents submitted to banks and builds it on an openly available, pre-trained language model from a public model hub. The finished system passes every test. What nobody noticed: an attacker had uploaded a modified copy of the model under an almost identical name. If a document contains a certain inconspicuous string, the system rates it as harmless. A model whose origin you cannot verify is a component you trust blindly.
Article 15(5) of the AI Act calls this type of attack by its name: model poisoning, meaning the manipulation of pre-trained components used in training. In the product, the threat covers any manipulation of the model itself, for example during training or an update, that plants hidden malfunctions or backdoors.
What is behind it?
Many providers no longer train their models from scratch. A common approach is to build on pre-trained models, open weights and ready-made libraries and to adapt them for their own purpose. Each of these components is a link in a supply chain, and each link can be manipulated. Model poisoning targets exactly these links.
Routes of attack
- Manipulated pre-trained models. Public hubs host copies of well-known models with a built-in backdoor, often under names that are easy to confuse. Whoever builds on them inherits the backdoor, and it can even survive later fine-tuning.
- Tampering with weights or training. Anyone with access to the training environment, the model store or the training code can change the model directly.
- Manipulated updates. A compromised delivery channel distributes a modified model version to all customers at once.
- Malicious code in model files. Some storage formats for models execute program code when loaded. A prepared file can take over the entire development environment this way.
- Distributed learning. In federated learning, several participants contribute model updates. A malicious participant can feed in deliberately manipulated contributions.
What makes a backdoor
A model with a backdoor behaves inconspicuously on all ordinary inputs. Only a specific trigger known to the attacker activates the hidden behaviour. Because this trigger does not occur in the test data, accuracy measurements show nothing unusual at all.
Impact
The attacker can steer the results whenever he likes, at every customer that uses the model. A backdoor in the model is a master key that fits at every one of the provider’s customers. On top come questions of liability, laborious forensics and a loss of trust that reaches far beyond the affected product. Malicious code in model files can also lead to a classic security incident with data leakage.
Practical examples
Compromised update server. A provider of camera systems for quality inspection distributes new model versions through its own update server. Attackers gain access and push out a version that overlooks one particular type of defect. At several customers, faulty parts pass through until a buyer raises the alarm.
Prepared model file. A development team downloads a model for a prototype in a format that can execute code when loaded. The file silently installs remote access in the training environment. For weeks, training data and credentials leak out.
Joint training in a network. Several hospitals train a diagnostic model together without exchanging their patient data. One participant’s system is compromised and delivers manipulated updates. Afterwards the shared model detects a particular finding less reliably, and none of the partners can pin down the cause straight away.
Effective controls
The following ISO/IEC 42001 controls counter this threat.
Prevention:
- A.10 Third-party and customer relationships: A process makes sure that purchased or adopted components such as models and libraries fit the organization’s responsible approach. Suppliers are selected and monitored according to the risk of what they supply, the integration of external components is documented, and responsibilities are allocated.
- A.6.2 AI system life cycle: The design documentation records hardware and software components and the AI-specific security threats considered. The deployment plan takes into account that software and model can be shipped separately and requires release criteria for both.
Detection:
- A.6.2 AI system life cycle: Verification and validation check every model version before release, monitoring in operation looks for anomalies, and event logs show which version produced which result.
Response:
- A.6.2 AI system life cycle: A documented failure plan provides for a return to the tested version, switching off individual features and informing customers.
- A.10 Third-party and customer relationships: Suppliers are required to take corrective action, and customers receive the documentation they need for their own response.
Link to the AI Act
Article 15(5) of the AI Act (Regulation (EU) 2024/1689) requires high-risk systems to have, where appropriate, measures against attacks on pre-trained components used in training and against model flaws. The technical documentation under Annex IV describes whether and how pre-trained systems or third-party tools were used, integrated or modified. Under Article 25(4), the provider of a high-risk system agrees in writing with third parties that supply tools, services or components on the information and assistance it needs to meet its obligations. Providers of general-purpose AI models make information and documentation available to downstream providers under Article 53 (→ EU AI Act).
Related threats
- T08 Data poisoning: The same effect, achieved through the training data.
- T06 Life cycle and maintenance gaps: Untested updates open the door to the attack.
- T10 Adversarial examples: A backdoor is often triggered through a prepared input.
Sources
- Regulation (EU) 2024/1689 (AI Act), Articles 15, 25 and 53 and Annex IV
- ISO/IEC 42001:2023, Annex B.6.2.3 and B.10.3
- EU AI Act in the Cenedril wiki: obligations for high-risk systems at a glance