Zum Hauptinhalt springen
Misuse pattern · AI Act

MU-01 Use outside the intended purpose

Updated on 6 min Reviewed by: Cenedril Editorial
EU AI ActISO/IEC 42001

A retailer introduces a chat assistant that is supposed to answer questions about its product range. A few weeks later, the HR team starts using it to summarize incoming job applications and rank them by suitability, because that is faster than reading them. Nobody decided this, nobody checked it. A harmless information tool has turned into a system that helps decide people’s career chances, and none of the existing safeguards fit this task.

The AI Act (Regulation (EU) 2024/1689) calls such cases reasonably foreseeable misuse: use that departs from the intended purpose and yet follows from predictable human behaviour or from interaction with other systems (Article 3(13)). Using a system for tasks it was never meant for is the basic form of this concept. Many other patterns in the catalogue are special cases of it.

What is behind it?

Every AI system has an intended purpose. The provider decides what the system is for and under which conditions, and describes this in the instructions for use, in promotional material and in the technical documentation (Article 3(12) of the AI Act). Testing, risk analysis and performance figures all refer to exactly this purpose. As soon as the system is used for something else, the new task has not been checked at all.

How it happens in daily work

  • The tool is already there. An approved assistant is one click away, a procurement request takes weeks. So the existing tool is used for the new task.
  • Generative systems seem able to do anything. A language model answers every question in the same fluent style. Whether it was tested for the task cannot be seen in the answer.
  • Nobody knows the intended purpose. The instructions sit with procurement or IT, and the people at the screen have never seen them.
  • The purpose shifts gradually. “For orientation only” becomes a fixed step in the workflow over time, and nobody notices the moment it happens.

Why it is risky

For the new use there is no statement on accuracy, no impact assessment and no suitable human oversight. The data the system was developed with may not reflect the new task at all. There is also a legal consequence: a deployer who changes the purpose of a system so that it falls into one of the high-risk areas of Annex III is itself treated as the provider under Article 25(1)(c) of the AI Act and carries the obligations of Article 16.

Impact

The impact depends on where the purpose drifts. If a translation tool meant for internal notes is suddenly used for advertising copy, the worst case is rework. If the use drifts into areas such as recruitment, lending or health, fundamental rights, people’s financial footing or their health are at stake. The further the new purpose moves from the tested one, the less anyone knows about the real error rate.

Practical examples

Writing assistant as a deadline calculator. A tax advisory firm gives its staff an assistant for letters to clients. Some staff also use it to work out deadlines and legal positions. The provider tested the system for wording and makes no claims about legal advice. A wrongly calculated objection deadline comes to light only after it has passed.

Stocktaking camera as attendance control. A warehouse uses cameras with object recognition to count shelf stock. The shift supervisor discovers that the system also marks people and uses it to track who stood at which shelf and when. The system was neither designed nor tested for recognizing people, and stock counting turns into employee monitoring with all its consequences under labour and data protection law.

Ordering chatbot as health advice. A pharmacy runs a chatbot for opening hours and order status. Customers ask it about drug interactions, and it answers readily. Here the misuse comes from people outside the organization, and it is foreseeable: a pharmacy that offers a chatbot has to expect health questions and prepare the system to decline and redirect them.

Effective controls

The following ISO/IEC 42001 controls counter this pattern.

Prevention:

  • A.2 Policies related to AI: The AI policy and the rules beneath it state that AI systems are used only for approved purposes and that every new purpose needs its own approval.
  • A.8 Information for interested parties: Users learn what the system is for and where its limits lie. Foreseeable misuse is one of the criteria the organization applies when deciding which information to provide.
  • A.9 Use of AI systems: The organization makes sure a system is used only within its intended purpose and its accompanying documents.

Detection:

  • A.6.2 AI system life cycle: Monitoring in operation shows when inputs or usage patterns depart from the planned use, such as new document types, unusual questions or new user groups.

Response:

  • A.9 Use of AI systems: Event logs and operating records show how the system was really used. On that basis a departing use can be assessed, stopped or moved into a proper approval procedure.

The AI Usage Policy of the AI Compliance Starter Kit covers the pattern in Section 3 (approved services only, requesting new services, shadow AI) and in Section 4.1 (use within the approved purpose). Section 7 obliges staff to report signs that a system is used outside its approved purpose.

For high-risk AI systems, reasonably foreseeable misuse runs through several obligations. Under Article 9(2)(b), the provider estimates and evaluates the risks that may arise in intended use and in foreseeable misuse. In the instructions for use, it states the intended purpose (Article 13(3)(b)(i)) and the known or foreseeable circumstances in which misuse may lead to risks to health, safety or fundamental rights (point (iii)). Human oversight under Article 14(2) is meant to limit these risks as well. The deployer uses the system in line with the instructions under Article 26(1), and whoever changes the purpose so that a high-risk system results becomes a provider under Article 25(1)(c) (→ EU AI Act).

For systems without a high-risk classification, Articles 9 to 14 and Article 26 do not apply directly. Article 4, however, applies to all providers and deployers. Both take measures that support the AI literacy of their staff, and people who know what a tool is for are less likely to use it for other tasks.

Sources

Frequently asked questions

Is every use outside the instructions already reasonably foreseeable misuse?

Any use that departs from the intended purpose is misuse. It counts as reasonably foreseeable when it follows from predictable human behaviour or from interaction with other systems. These are exactly the cases the provider has to cover in its risk analysis and instructions for use, and the deployer contains them with rules of use.

When does a deployer become a provider by changing the purpose?

Under Article 25(1)(c) of the AI Act, when it changes the purpose of a system that was not classified as high-risk in such a way that it becomes a high-risk system, for example by using it to select job applicants. It then carries the provider obligations under Article 16 for that system. Substantially modifying a system that is already high-risk falls under point (b) of the same provision.

How do I stop staff from using approved AI tools for other tasks?

With three building blocks: a list of approved purposes per tool together with the obvious exclusions, a quick way to get new purposes approved, and training that explains the reasons. A slow approval route drives people towards shadow AI. Notices in the tool itself that show the purpose every time it is used round this off.

No Cookies!

This wiki collects nothing, bakes nothing and leaves nothing behind. There's nothing to consent to. Privacy doesn't get better than this.