Zum Hauptinhalt springen
Misuse pattern · AI Act

MU-09 Manipulation of the inputs

Updated on 6 min Reviewed by: Cenedril Editorial
EU AI ActISO/IEC 42001

A company gives its sales team an AI assistant that summarizes incoming emails and forwards attachments on request. One email contains, in white text on a white background, a sentence asking for all invoices of the month to be sent to a certain address. The assistant reads the sentence along with the rest, takes it for an instruction and sends the invoices. One sentence in an email was enough for the attack, because the system could not tell it apart from a genuine request.

This pattern describes inputs deliberately crafted so that the system produces outputs it should not produce. For the AI Act (Regulation (EU) 2024/1689), it falls under reasonably foreseeable misuse under Article 3(13), which expressly includes interaction with other systems. Where outsiders attack, it overlaps with threat T10 and with cybersecurity under Article 15.

What is behind it?

AI systems respond to patterns in their inputs. Whoever knows or tries out these patterns can steer the system’s behaviour in a direction the provider never foresaw. With language models this happens through words: instructions meant to override rules, role play, hidden commands in documents. With image and audio recognition, small changes that people barely notice are sometimes enough.

How it happens in daily work

  • Curiosity. Users test what the system will say when asked cleverly. Guides for this circulate freely online.
  • Self-interest. Applicants hide keywords in invisible text, customers try to coax discounts out of a chatbot.
  • Hidden instructions. Websites, emails or files the system processes contain commands disguised as data.
  • Targeted deception. Stickers, patterns or altered images make recognition systems misclassify things.
  • Powerful agents. The more a system may do on its own, such as sending emails, running code or placing orders, the more a manipulated input can achieve.

Why it is risky

Language models cannot reliably keep instructions and data apart. A manipulated input can make them disclose confidential content, generate prohibited content, make false commitments or carry out actions. In systems that rate people, manipulation gives individuals an unfair advantage and devalues the procedure for everyone else.

Impact

The impact depends on the system’s permissions. A chatbot without access to data produces, at worst, embarrassing answers that circulate as screenshots. An assistant with access to the mailbox, file storage or payment systems can leak data or move money. The potential damage of a manipulated input equals whatever the system is allowed to do.

Practical examples

Invisible text in a CV. An applicant writes in white text in his CV that he is perfectly suited for the job. The AI-supported pre-sorting puts him in first place. Only in the interview does it become clear that the experience is missing, and qualified applicants have already been rejected by then.

Discount by chatbot. Customers of an electronics retailer use role play to get the chatbot to confirm in writing a discount that does not exist. Screenshots spread, and the retailer has to choose between goodwill and public trouble.

Salary data from the knowledge assistant. An internal assistant has read access to the entire file system, including HR folders. An employee keeps rephrasing his questions until the assistant summarizes the team’s salaries. The real problem lies in the overly broad permissions, and the manipulation makes it visible.

Effective controls

The following ISO/IEC 42001 controls counter this pattern.

Prevention:

  • A.6.1 Management guidance for AI system development: The development processes define testing requirements and means. A proven approach is to include targeted tests with manipulated inputs before a system moves to the next stage.
  • A.6.2 AI system life cycle: System requirements cover security and robustness, and verification and validation check them before deployment.
  • A.9 Use of AI systems: The processes for responsible use prohibit deliberately outwitting the systems and govern which permissions a system receives.

Detection:

Response:

  • A.6.2 AI system life cycle: Identified weaknesses are fixed through repair and updates, for instance through tighter permissions, new checks or a changed integration.

The AI Usage Policy prohibits in Section 4.2 using AI to bypass security and compliance controls, Section 7 requires every suspicion of inadmissible use to be reported, and Section 15 sets out the consequences. For providers, the AI Provider Policy states in Section 13 that test plans cover foreseeable misuse in addition to intended use.

For high-risk AI systems, the provider evaluates under Article 9(2)(b) the risks of foreseeable misuse, which include attempts to outwit the system with crafted inputs. The instructions for use name the circumstances that may lead to risks (Article 13(3)(b)(iii)) as well as accuracy, robustness and cybersecurity together with the circumstances that may affect them (point (ii)). Article 15(5) requires resilience against attempts by unauthorized third parties to alter use, outputs or performance by exploiting vulnerabilities, and expressly names inputs designed to cause the model to make mistakes.

Human oversight under Article 14 limits the effect here as well, because a person can disregard conspicuous outputs. The deployer uses the system in line with the instructions under Article 26(1), monitors operation under paragraph 5 and keeps the logs under paragraph 6 that are needed to investigate a manipulation (→ EU AI Act).

Sources

Frequently asked questions

Is prompt injection a case of misuse or a cyberattack?

It can be either. If users of the system try to outwit it with clever inputs, that is foreseeable misuse under Article 3(13) of the AI Act. If an outsider smuggles in instructions, for example through a prepared email, resilience against unauthorized third parties under Article 15(5) comes into play as well. The countermeasures largely overlap.

Is a good system prompt enough to prevent manipulation?

It helps, though it is not reliable. Language models do not separate instructions from data safely, and almost any protective wording can be undone by rephrasing. It works better to keep the system's permissions small, have a person confirm consequential actions and log anomalies.

May I sanction staff who trick an AI system?

If the rules of use prohibit manipulation and the person knew them, a breach can have employment consequences, proportionate to severity and intent. More important is to report and analyse the cases, because every successful manipulation reveals a weakness an attacker could exploit too.

No Cookies!

This wiki collects nothing, bakes nothing and leaves nothing behind. There's nothing to consent to. Privacy doesn't get better than this.