Zum Hauptinhalt springen
Misuse pattern · AI Act

MU-15 Further training with inadmissible data

Updated on 6 min Reviewed by: Cenedril Editorial
EU AI ActISO/IEC 42001GDPR

A software vendor runs an AI-supported support chat for its business customers. To improve it, the development team fine-tunes the model on all support tickets of all customers, including the names of end customers and confidential configuration details. The data processing agreements permit processing for support only. Weeks later, one customer’s chat reproduces details from another customer’s tickets. The data could be processed for support, and nobody ever approved it for training the model.

This pattern describes systems adapted with data that may not be used for training. For the AI Act (Regulation (EU) 2024/1689), it is reasonably foreseeable misuse under Article 3(13), because wherever data accumulates in large quantities, the idea of improving the model with it is close at hand. For personal data, the GDPR applies alongside.

What is behind it?

Whether data may be used for training is decided in several places at once: by the legal basis and original purpose for personal data, by contracts with customers and suppliers, by licences and copyright, by confidentiality duties and by internal classification. The fact that data exists in-house says nothing about whether it may flow into a model. There is also a technical peculiarity: whatever has been trained sits in the model’s weights and can hardly be removed selectively.

How it happens in daily work

  • The data is at hand. Tickets, chat histories and production data look like free training material.
  • Unclear contracts. The customer contract is silent on training, and the team assumes it is ‘our’ data.
  • Automatic continued learning. The system learns from user inputs by default, and nobody has assessed this setting.
  • Third-party datasets. Datasets downloaded from the internet have unclear origins or licences that exclude commercial use.
  • Fine-tuning without experts. Simple tools let departments upload their own files and adapt models without anyone checking the data.

Why it is risky

Inadmissible training data makes the processing unlawful and can bring confidential or personal content back to light through the outputs. Unrepresentative data biases the results, and data from uncontrolled sources opens the door to targeted poisoning. Erasure requests from the people concerned can hardly be met. For high-risk systems, the provider breaches the data governance requirements, and a deployer that substantially modifies a purchased system is itself treated as the provider under Article 25(1)(b).

Impact

The impact is high and lasting. In the worst case, a model has to be withdrawn and retrained without the affected data, with all the costs and downtime that entails. GDPR fines, contractual penalties, customer terminations and claims by rights holders come on top. Whatever has once entered the model can hardly be extracted again.

Practical examples

Fine-tuning with personnel files. An HR department fine-tunes an internal writing assistant on appraisals and meeting notes so that it drafts ‘more fitting’ references. The notes also contain details of illnesses. The assistant later reproduces phrasings that make individual employees identifiable.

Continued learning from conversations. A chatbot on a consumer platform keeps learning from conversations. A group of users deliberately feeds it offensive content and false product information. After a few days it passes both on to other users, together with fragments of personal details from earlier conversations.

Dataset in breach of licence. A young company trains an image recognition model on a freely downloadable dataset whose licence excludes commercial use. After market launch, the rights holder gets in touch. The model has to be retrained with other data, and delivery stalls.

Effective controls

The following ISO/IEC 42001 controls counter this pattern.

Prevention:

  • A.7 Data for AI systems: Data management processes govern acquisition, quality, provenance and preparation. The provenance of the data is recorded across the life cycle of both data and system.
  • A.6.1 Management guidance for AI system development: The development processes define which data may be used for training, which data suppliers are approved and how data is labelled.
  • A.4 Resources for AI systems: A system’s data resources are documented, including their provenance and intended use.
  • A.9 Use of AI systems: The processes for responsible use govern whether, and with what approval, deployers may adapt purchased models themselves.

Detection:

  • A.6.2 AI system life cycle: Verification and validation before every new model version and monitoring in operation reveal conspicuous outputs, such as reproduced training content or new biases.

Response:

  • A.6.2 AI system life cycle: If an inadmissible dataset is discovered, the organization rolls back to an earlier model version or retrains without the data and documents the change.

The AI Provider Policy requires in Section 15 that origin, legal basis and usage restrictions are determined for every dataset before it is used, and that provenance is recorded per dataset. The AI Usage Policy describes in Section 13 when a deployer becomes a provider through a substantial modification.

For high-risk AI systems, Article 10 requires data governance for training, validation and testing data. It covers, among other things, data collection processes, the origin of the data and, for personal data, the original purpose of collection (paragraph 2(b)), as well as an examination for biases (point (f)). The datasets must be relevant, sufficiently representative and as free of errors as possible (paragraph 3). Systems that continue to learn are to be developed so that biased feedback loops are eliminated or reduced as far as possible (Article 15(4)). The provider evaluates foreseeable misuse under Article 9(2)(b) and states the predetermined changes to the system in the instructions for use (Article 13(3)(c)).

If a deployer substantially modifies a high-risk system, for example through its own retraining that the provider did not predetermine, it is treated as the provider under Article 25(1)(b) (→ EU AI Act). For personal training data, purpose limitation under Article 5(1)(b) and the compatibility test under Article 6(4) of the GDPR apply, and Article 28 applies to processors.

Sources

Frequently asked questions

May I use customer data to improve my AI product?

Only if contract and law allow it. If you process the data as a processor, you may process it only on the customer's documented instructions under Article 28(3)(a) GDPR; training for your own purposes needs the customer's consent and a sound legal basis. A processor that determines purposes on its own is considered a controller for that processing under Article 28(10).

Can I remove data from a trained model afterwards?

Only with great effort. Removing specific data from a trained model is technically difficult and often only reliably possible by training again without that data. That is why checking before training pays off more than any correction afterwards.

Does a deployer become a provider by fine-tuning a purchased model?

For high-risk systems it can: a substantial modification under Article 25(1)(b) of the AI Act makes the deployer a provider. Changes through continued learning count as non-substantial only if the provider predetermined them at the initial conformity assessment (Article 43(4)). So clarify what the instructions for use allow before any fine-tuning.

No Cookies!

This wiki collects nothing, bakes nothing and leaves nothing behind. There's nothing to consent to. Privacy doesn't get better than this.