A retailer introduces a chat assistant that is supposed to answer questions about its product range. A few weeks later, the HR team starts using it to summarize incoming job applications and rank them by suitability, because that is faster than reading them. Nobody decided this, nobody checked it. A harmless information tool has turned into a system that helps decide people’s career chances, and none of the existing safeguards fit this task.
The AI Act (Regulation (EU) 2024/1689) calls such cases reasonably foreseeable misuse: use that departs from the intended purpose and yet follows from predictable human behaviour or from interaction with other systems (Article 3(13)). Using a system for tasks it was never meant for is the basic form of this concept. Many other patterns in the catalogue are special cases of it.
What is behind it?
Every AI system has an intended purpose. The provider decides what the system is for and under which conditions, and describes this in the instructions for use, in promotional material and in the technical documentation (Article 3(12) of the AI Act). Testing, risk analysis and performance figures all refer to exactly this purpose. As soon as the system is used for something else, the new task has not been checked at all.
How it happens in daily work
- The tool is already there. An approved assistant is one click away, a procurement request takes weeks. So the existing tool is used for the new task.
- Generative systems seem able to do anything. A language model answers every question in the same fluent style. Whether it was tested for the task cannot be seen in the answer.
- Nobody knows the intended purpose. The instructions sit with procurement or IT, and the people at the screen have never seen them.
- The purpose shifts gradually. “For orientation only” becomes a fixed step in the workflow over time, and nobody notices the moment it happens.
Why it is risky
For the new use there is no statement on accuracy, no impact assessment and no suitable human oversight. The data the system was developed with may not reflect the new task at all. There is also a legal consequence: a deployer who changes the purpose of a system so that it falls into one of the high-risk areas of Annex III is itself treated as the provider under Article 25(1)(c) of the AI Act and carries the obligations of Article 16.
Impact
The impact depends on where the purpose drifts. If a translation tool meant for internal notes is suddenly used for advertising copy, the worst case is rework. If the use drifts into areas such as recruitment, lending or health, fundamental rights, people’s financial footing or their health are at stake. The further the new purpose moves from the tested one, the less anyone knows about the real error rate.
Practical examples
Writing assistant as a deadline calculator. A tax advisory firm gives its staff an assistant for letters to clients. Some staff also use it to work out deadlines and legal positions. The provider tested the system for wording and makes no claims about legal advice. A wrongly calculated objection deadline comes to light only after it has passed.
Stocktaking camera as attendance control. A warehouse uses cameras with object recognition to count shelf stock. The shift supervisor discovers that the system also marks people and uses it to track who stood at which shelf and when. The system was neither designed nor tested for recognizing people, and stock counting turns into employee monitoring with all its consequences under labour and data protection law.
Ordering chatbot as health advice. A pharmacy runs a chatbot for opening hours and order status. Customers ask it about drug interactions, and it answers readily. Here the misuse comes from people outside the organization, and it is foreseeable: a pharmacy that offers a chatbot has to expect health questions and prepare the system to decline and redirect them.
Effective controls
The following ISO/IEC 42001 controls counter this pattern.
Prevention:
- A.2 Policies related to AI: The AI policy and the rules beneath it state that AI systems are used only for approved purposes and that every new purpose needs its own approval.
- A.8 Information for interested parties: Users learn what the system is for and where its limits lie. Foreseeable misuse is one of the criteria the organization applies when deciding which information to provide.
- A.9 Use of AI systems: The organization makes sure a system is used only within its intended purpose and its accompanying documents.
Detection:
- A.6.2 AI system life cycle: Monitoring in operation shows when inputs or usage patterns depart from the planned use, such as new document types, unusual questions or new user groups.
Response:
- A.9 Use of AI systems: Event logs and operating records show how the system was really used. On that basis a departing use can be assessed, stopped or moved into a proper approval procedure.
The AI Usage Policy of the AI Compliance Starter Kit covers the pattern in Section 3 (approved services only, requesting new services, shadow AI) and in Section 4.1 (use within the approved purpose). Section 7 obliges staff to report signs that a system is used outside its approved purpose.
Link to the AI Act
For high-risk AI systems, reasonably foreseeable misuse runs through several obligations. Under Article 9(2)(b), the provider estimates and evaluates the risks that may arise in intended use and in foreseeable misuse. In the instructions for use, it states the intended purpose (Article 13(3)(b)(i)) and the known or foreseeable circumstances in which misuse may lead to risks to health, safety or fundamental rights (point (iii)). Human oversight under Article 14(2) is meant to limit these risks as well. The deployer uses the system in line with the instructions under Article 26(1), and whoever changes the purpose so that a high-risk system results becomes a provider under Article 25(1)(c) (→ EU AI Act).
For systems without a high-risk classification, Articles 9 to 14 and Article 26 do not apply directly. Article 4, however, applies to all providers and deployers. Both take measures that support the AI literacy of their staff, and people who know what a tool is for are less likely to use it for other tasks.
Related patterns and threats
- MU-10 Reuse of outputs for another purpose: The system stays with its task, while its results move into a new context.
- MU-06 Use on untested groups: A special case in which the group of affected people shifts.
- MU-12 Use outside the operating limits: The purpose stays the same, the promised conditions are exceeded.
- T01 Complexity of environment: A new purpose brings situations that never came up in development and testing.
Sources
- Regulation (EU) 2024/1689 (AI Act), Article 3(12) and (13), Articles 4, 9, 13, 14, 25 and 26
- ISO/IEC 42001:2023, Annex B.9.4 (intended use), B.8.2 and B.5.5
- AI Usage Policy of the AI Compliance Starter Kit, Sections 3, 4.1 and 7
- ISO/IEC 42001 in the Cenedril wiki: Annex A and the controls on the use of AI systems