Zum Hauptinhalt springen
AI threat · AI Act Article 15

T09 Model poisoning

Updated on 5 min Reviewed by: Cenedril Editorial
ISO/IEC 42001EU AI Act

A young company develops a system that checks documents submitted to banks and builds it on an openly available, pre-trained language model from a public model hub. The finished system passes every test. What nobody noticed: an attacker had uploaded a modified copy of the model under an almost identical name. If a document contains a certain inconspicuous string, the system rates it as harmless. A model whose origin you cannot verify is a component you trust blindly.

Article 15(5) of the AI Act calls this type of attack by its name: model poisoning, meaning the manipulation of pre-trained components used in training. In the product, the threat covers any manipulation of the model itself, for example during training or an update, that plants hidden malfunctions or backdoors.

What is behind it?

Many providers no longer train their models from scratch. A common approach is to build on pre-trained models, open weights and ready-made libraries and to adapt them for their own purpose. Each of these components is a link in a supply chain, and each link can be manipulated. Model poisoning targets exactly these links.

Routes of attack

  • Manipulated pre-trained models. Public hubs host copies of well-known models with a built-in backdoor, often under names that are easy to confuse. Whoever builds on them inherits the backdoor, and it can even survive later fine-tuning.
  • Tampering with weights or training. Anyone with access to the training environment, the model store or the training code can change the model directly.
  • Manipulated updates. A compromised delivery channel distributes a modified model version to all customers at once.
  • Malicious code in model files. Some storage formats for models execute program code when loaded. A prepared file can take over the entire development environment this way.
  • Distributed learning. In federated learning, several participants contribute model updates. A malicious participant can feed in deliberately manipulated contributions.

What makes a backdoor

A model with a backdoor behaves inconspicuously on all ordinary inputs. Only a specific trigger known to the attacker activates the hidden behaviour. Because this trigger does not occur in the test data, accuracy measurements show nothing unusual at all.

Impact

The attacker can steer the results whenever he likes, at every customer that uses the model. A backdoor in the model is a master key that fits at every one of the provider’s customers. On top come questions of liability, laborious forensics and a loss of trust that reaches far beyond the affected product. Malicious code in model files can also lead to a classic security incident with data leakage.

Practical examples

Compromised update server. A provider of camera systems for quality inspection distributes new model versions through its own update server. Attackers gain access and push out a version that overlooks one particular type of defect. At several customers, faulty parts pass through until a buyer raises the alarm.

Prepared model file. A development team downloads a model for a prototype in a format that can execute code when loaded. The file silently installs remote access in the training environment. For weeks, training data and credentials leak out.

Joint training in a network. Several hospitals train a diagnostic model together without exchanging their patient data. One participant’s system is compromised and delivers manipulated updates. Afterwards the shared model detects a particular finding less reliably, and none of the partners can pin down the cause straight away.

Effective controls

The following ISO/IEC 42001 controls counter this threat.

Prevention:

  • A.10 Third-party and customer relationships: A process makes sure that purchased or adopted components such as models and libraries fit the organization’s responsible approach. Suppliers are selected and monitored according to the risk of what they supply, the integration of external components is documented, and responsibilities are allocated.
  • A.6.2 AI system life cycle: The design documentation records hardware and software components and the AI-specific security threats considered. The deployment plan takes into account that software and model can be shipped separately and requires release criteria for both.

Detection:

  • A.6.2 AI system life cycle: Verification and validation check every model version before release, monitoring in operation looks for anomalies, and event logs show which version produced which result.

Response:

Article 15(5) of the AI Act (Regulation (EU) 2024/1689) requires high-risk systems to have, where appropriate, measures against attacks on pre-trained components used in training and against model flaws. The technical documentation under Annex IV describes whether and how pre-trained systems or third-party tools were used, integrated or modified. Under Article 25(4), the provider of a high-risk system agrees in writing with third parties that supply tools, services or components on the information and assistance it needs to meet its obligations. Providers of general-purpose AI models make information and documentation available to downstream providers under Article 53 (→ EU AI Act).

Sources

Frequently asked questions

What distinguishes model poisoning from data poisoning?

Data poisoning starts with the training data, model poisoning with the model itself or with pre-trained components a provider builds on. The effect can be similar, such as a hidden backdoor. The defence differs: for data it is about provenance and quality, for models about a secured supply chain and verified integrity.

Are openly available models risky in principle?

No, they are often well documented and reviewed by many experts. The risk arises when their origin is unclear or nobody checks whether the downloaded file matches the published one. So obtain models from the original source, compare checksums and record which version you use.

What should deployers do about T09?

T09 mainly concerns providers, because they build and ship models. Deployers accept updates only through the agreed channels and watch for anomalies afterwards. Whoever fine-tunes a model or combines components into a new system takes on provider obligations, and this threat along with them.

No Cookies!

This wiki collects nothing, bakes nothing and leaves nothing behind. There's nothing to consent to. Privacy doesn't get better than this.