A company gives its sales team an AI assistant that summarizes incoming emails and forwards attachments on request. One email contains, in white text on a white background, a sentence asking for all invoices of the month to be sent to a certain address. The assistant reads the sentence along with the rest, takes it for an instruction and sends the invoices. One sentence in an email was enough for the attack, because the system could not tell it apart from a genuine request.
This pattern describes inputs deliberately crafted so that the system produces outputs it should not produce. For the AI Act (Regulation (EU) 2024/1689), it falls under reasonably foreseeable misuse under Article 3(13), which expressly includes interaction with other systems. Where outsiders attack, it overlaps with threat T10 and with cybersecurity under Article 15.
What is behind it?
AI systems respond to patterns in their inputs. Whoever knows or tries out these patterns can steer the system’s behaviour in a direction the provider never foresaw. With language models this happens through words: instructions meant to override rules, role play, hidden commands in documents. With image and audio recognition, small changes that people barely notice are sometimes enough.
How it happens in daily work
- Curiosity. Users test what the system will say when asked cleverly. Guides for this circulate freely online.
- Self-interest. Applicants hide keywords in invisible text, customers try to coax discounts out of a chatbot.
- Hidden instructions. Websites, emails or files the system processes contain commands disguised as data.
- Targeted deception. Stickers, patterns or altered images make recognition systems misclassify things.
- Powerful agents. The more a system may do on its own, such as sending emails, running code or placing orders, the more a manipulated input can achieve.
Why it is risky
Language models cannot reliably keep instructions and data apart. A manipulated input can make them disclose confidential content, generate prohibited content, make false commitments or carry out actions. In systems that rate people, manipulation gives individuals an unfair advantage and devalues the procedure for everyone else.
Impact
The impact depends on the system’s permissions. A chatbot without access to data produces, at worst, embarrassing answers that circulate as screenshots. An assistant with access to the mailbox, file storage or payment systems can leak data or move money. The potential damage of a manipulated input equals whatever the system is allowed to do.
Practical examples
Invisible text in a CV. An applicant writes in white text in his CV that he is perfectly suited for the job. The AI-supported pre-sorting puts him in first place. Only in the interview does it become clear that the experience is missing, and qualified applicants have already been rejected by then.
Discount by chatbot. Customers of an electronics retailer use role play to get the chatbot to confirm in writing a discount that does not exist. Screenshots spread, and the retailer has to choose between goodwill and public trouble.
Salary data from the knowledge assistant. An internal assistant has read access to the entire file system, including HR folders. An employee keeps rephrasing his questions until the assistant summarizes the team’s salaries. The real problem lies in the overly broad permissions, and the manipulation makes it visible.
Effective controls
The following ISO/IEC 42001 controls counter this pattern.
Prevention:
- A.6.1 Management guidance for AI system development: The development processes define testing requirements and means. A proven approach is to include targeted tests with manipulated inputs before a system moves to the next stage.
- A.6.2 AI system life cycle: System requirements cover security and robustness, and verification and validation check them before deployment.
- A.9 Use of AI systems: The processes for responsible use prohibit deliberately outwitting the systems and govern which permissions a system receives.
Detection:
- A.6.2 AI system life cycle: Event logs and monitoring reveal conspicuous input patterns and unusual outputs. Monitoring activities under A.8.16 of ISO/IEC 27001 add to this.
Response:
- A.6.2 AI system life cycle: Identified weaknesses are fixed through repair and updates, for instance through tighter permissions, new checks or a changed integration.
The AI Usage Policy prohibits in Section 4.2 using AI to bypass security and compliance controls, Section 7 requires every suspicion of inadmissible use to be reported, and Section 15 sets out the consequences. For providers, the AI Provider Policy states in Section 13 that test plans cover foreseeable misuse in addition to intended use.
Link to the AI Act
For high-risk AI systems, the provider evaluates under Article 9(2)(b) the risks of foreseeable misuse, which include attempts to outwit the system with crafted inputs. The instructions for use name the circumstances that may lead to risks (Article 13(3)(b)(iii)) as well as accuracy, robustness and cybersecurity together with the circumstances that may affect them (point (ii)). Article 15(5) requires resilience against attempts by unauthorized third parties to alter use, outputs or performance by exploiting vulnerabilities, and expressly names inputs designed to cause the model to make mistakes.
Human oversight under Article 14 limits the effect here as well, because a person can disregard conspicuous outputs. The deployer uses the system in line with the instructions under Article 26(1), monitors operation under paragraph 5 and keeps the logs under paragraph 6 that are needed to investigate a manipulation (→ EU AI Act).
Related patterns and threats
- T10 Adversarial examples: The technical threat when outsiders manipulate the inputs.
- T11 Attacks on confidential data: Often the goal of a manipulation: extracting data from the model or connected sources.
- MU-11 Passing on access credentials: A shared account opens the system to people who probe it deliberately.
- MU-01 Use outside the intended purpose: Manipulation is often the route to getting a system to do tasks it was not meant for.
Sources
- Regulation (EU) 2024/1689 (AI Act), Article 3(13), Articles 9, 13, 14, 15 and 26
- ISO/IEC 42001:2023, Annex B.6.1.3, B.6.2.4, B.6.2.8 and B.9.4
- AI Usage Policy, Sections 4.2, 7 and 15, and AI Provider Policy, Section 13
- ISO/IEC 42001 in the Cenedril wiki: Annex A and the controls for development and operation