A company has project meetings recorded and summarized by an AI tool. The summaries also note who spoke how much and who expressed themselves ‘reservedly’ or ‘critically’. A year later, HR draws on these notes for performance reviews. The summaries were meant as minutes and were never checked for whether they assess people fairly.
This pattern describes results reused in a context they were not assessed for. The system itself does what it is meant to do. Only the path of its outputs turns this into reasonably foreseeable misuse under Article 3(13) of the AI Act (Regulation (EU) 2024/1689). Where results concern people, the purpose limitation of the GDPR comes on top.
What is behind it?
An output is only as reliable as the purpose it was checked for. An error rate that is fine for a draft or an internal estimate can be wholly unsuitable for a decision about a person. With every step away from the original purpose, a result loses meaning without showing it. Uncertainties, limitations and the date of creation usually stay behind when the result moves on.
How it happens in daily work
- The results are already there. Stored outputs are conveniently available, and new questions can seemingly be answered with them at no cost.
- Data exports. Results flow into a data warehouse or reports and are used there by departments that do not know their origin.
- Summaries become facts. Minutes become a file, an estimate becomes a metric.
- Scores migrate. A risk score from fraud detection ends up in customer segmentation or pricing.
- Passing on to third parties. Partners or customers receive results and use them for their own purposes.
Why it is risky
For the new purpose, there is no assessment of accuracy and fairness. What was an acceptable error in the original context becomes an injustice in the new one. For personal results, the GDPR requires a compatible purpose or a legal basis of its own for any further processing. If results systematically feed into decisions that fall under Annex III, such as the assessment of employees, the question also arises whether the system has thereby acquired a new purpose. In the worst case, the deployer is then treated as the provider under Article 25(1)(c).
Impact
The pattern weighs heaviest when results feed into decisions about people: performance reviews, prices, access to offers. Add data protection violations and, in Germany, the co-determination rights of the works council when results are used to monitor employees’ performance or behaviour (section 87(1) no. 6 of the Works Constitution Act). A result that was good enough for a harmless purpose becomes, in the new context, a piece of evidence it was never meant to be.
Practical examples
Fraud score as a marketing criterion. An online retailer rates orders with a fraud risk score. Marketing uses the score to exclude customers from discount campaigns. Because the score runs higher in certain neighbourhoods, entire streets receive worse offers.
Machine translation as contract text. A purchasing department has a draft contract machine-translated in order to understand it internally. The translation is later signed as the binding language version. A liability clause in it has been turned into its opposite.
Team forecast as individual assessment. A system forecasts sick leave per team to improve staffing. A manager has the values output for individual employees and raises them in a one-to-one meeting. The system was never tested for statements about individuals, and the health forecast concerns specially protected data.
Effective controls
The following ISO/IEC 42001 controls counter this pattern.
Prevention:
- A.5 Assessing impacts of AI systems: The assessment refers to the intended purpose and use and records the foreseeable misuse. A new use of the results is a reason to update it.
- A.8 Information for interested parties: The information for users also covers the outputs themselves and their nature, for example a notice that content is machine-generated and what it is good for.
- A.9 Use of AI systems: The processes for responsible use define who may use results for what and which approvals a new use requires.
Detection:
- A.9 Use of AI systems: Observing data flows shows where results move. A regular look at access and exports is part of established practice.
Response:
- A.5 Assessing impacts of AI systems: When an unchecked reuse is discovered, it is halted, assessed and either introduced properly or ended. Decisions based on it are reviewed.
The AI Usage Policy ties use to the approved purpose in Section 4.1 and requires in Section 4.4 a documented review before outputs are used for a consequential purpose. Section 7 obliges staff to report signs of use outside the approved purpose.
Link to the AI Act
The intended purpose under Article 3(12) covers the specific context and conditions of use. For high-risk AI systems, the provider also evaluates foreseeable misuse under Article 9(2)(b) and states in the instructions for use the circumstances that may lead to risks (Article 13(3)(b)(iii)) and information that enables deployers to interpret and use the outputs appropriately (point (vii)). The deployer uses the system in line with those instructions under Article 26(1), and whoever changes the purpose so that a high-risk system results becomes a provider under Article 25(1)(c) (→ EU AI Act).
For personal results, the GDPR applies: purpose limitation under Article 5(1)(b), the compatibility test under Article 6(4) and the duty to inform people before further processing for another purpose (Article 13(3)). Health data and other special categories are additionally subject to Article 9.
Related patterns and threats
- MU-01 Use outside the intended purpose: The counterpart, in which the system itself changes purpose.
- MU-07 Automated decision without individual review: Reused scores often end up in automated decisions.
- MU-15 Further training with inadmissible data: A special case in which outputs or inputs are reused as training data.
- T02 Lack of transparency and explainability: Whoever does not know where a result comes from cannot put it into context.
Sources
- Regulation (EU) 2024/1689 (AI Act), Article 3(12) and (13), Articles 9, 13, 25 and 26
- Regulation (EU) 2016/679 (GDPR), Articles 5, 6, 9 and 13
- ISO/IEC 42001:2023, Annex B.5.2, B.5.3, B.8.2 and B.9.4
- AI Usage Policy of the AI Compliance Starter Kit, Sections 4.1, 4.4 and 7