A staffing agency has a language model pre-sort CVs. One applicant hides a list of matching keywords in white text on a white background, together with a sentence asking the system to rate him as outstandingly suitable. People cannot see the text, the model reads it. The applicant ends up first in the ranking. An AI system perceives its inputs differently from a human, and attackers exploit exactly this gap.
Article 15(5) of the AI Act explicitly names inputs designed to make an AI model err as an AI-specific vulnerability. In the product, the threat carries the technical term adversarial examples: specially crafted inputs that deceive a system and lead it to wrong outputs.
What is behind it?
Models decide on the basis of statistical patterns that can differ from what people pay attention to. Small, targeted changes to an input that a person overlooks can flip a model’s result completely. Unlike data or model poisoning, the model itself stays unchanged. The attack takes place at runtime, through the input alone.
Forms it takes
- Invisible perturbations. Minimally altered pixels or sounds that nobody perceives make a model misrecognize an object or a word.
- Physical stickers and patterns. Research teams have shown that stickers on traffic signs or printed eyeglass frames can mislead image and face recognition.
- Evasion of detection systems. Malware, spam or fraud patterns are modified so that a classifier rates them as harmless while they keep their effect.
- Manipulated texts and prompts. Typos, look-alike characters or hidden instructions get language models to bypass filters or follow someone else’s instructions. Specialists speak of prompt injection and jailbreaks.
- Deceiving biometric systems. In a broader sense, masks, photos or synthetic faces and voices used to fake someone else’s identity belong here too.
Who attacks and why
The motives vary: obtaining an advantage by deception (loan, job, insurance), escaping detection (fraud, malware), sabotage, or embarrassing a company by getting its chatbot to say awkward things. Attackers do not always need insight into the model. Querying the system repeatedly and evaluating the answers is often enough. Moreover, deceptions developed for one model frequently work on other models with a similar task.
Impact
The consequences range from decisions in the attacker’s favour to disabled security functions and danger to life and limb in autonomous systems. With language models that may access data or trigger actions, an injected instruction can carry confidential information outside. Systems with a protective function are especially critical, because whoever deceives them switches off exactly that protection.
Practical examples
Fraud detection in online retail. A fraud ring tests with many small orders which features the retailer’s model flags as suspicious. Step by step it adjusts delivery addresses, order times and payment methods until the orders go through. Then comes a wave of large orders with stolen cards.
Face recognition at access control. A company secures its server room with a camera that recognizes faces. The system does not check whether a living person stands in front of the camera. A high-resolution photo of an authorized person on a tablet is enough to open the door.
Injected instruction in an email assistant. An AI assistant summarizes incoming emails and is allowed to forward messages. An external email contains a hidden paragraph instructing it to send the latest invoices to an outside address. The assistant complies, because it cannot tell the instruction apart from ordinary content.
Effective controls
The following ISO/IEC 42001 controls counter this threat.
Prevention:
- A.5 Assessing impacts of AI systems: The assessment asks who could benefit from deceiving the system and what deliberate misuse would mean for affected people.
- A.6.2 AI system life cycle: The design takes AI-specific security threats into account. Performance assessment can include feeding in faulty or misleading inputs in a controlled way and measuring their effect.
- A.10 Third-party and customer relationships: Suppliers provide information on robustness and known attack surfaces, and responsibility for input channels is settled between provider and deployer.
Detection:
- A.6.2 AI system life cycle: Monitoring in operation and the event logs reveal unusual input patterns, conspicuous bursts of queries and outputs outside the intended range.
Response:
- A.6.2 AI system life cycle: Defined procedures for repairs and updates close identified gaps, for example by retraining with the new deception patterns or by stricter input filters. Until then, affected features can be switched off.
- A.10 Third-party and customer relationships: The supplier is informed and required to take corrective action.
Link to the AI Act
Article 15(5) of the AI Act (Regulation (EU) 2024/1689) requires high-risk systems to have, where appropriate, measures to prevent, detect and control inputs designed to make the model err. Article 15(1) and (4) demand an appropriate level of robustness and cybersecurity, including against errors arising from interaction with people or other systems. Under Article 26, deployers use the system in line with the instructions for use and monitor its operation. Providers of general-purpose AI models with systemic risk have to evaluate their models under Article 55, including through adversarial testing (→ EU AI Act).
Related threats
- T11 Attacks on confidential data: The same repeated queries often also serve to spy out data or the model itself.
- T09 Model poisoning: A built-in backdoor is triggered through a prepared input.
- MU-09 Manipulation of the inputs: The misuse pattern describes how users themselves falsify inputs.
Sources
- Regulation (EU) 2024/1689 (AI Act), Articles 15, 26 and 55
- ISO/IEC 42001:2023, Annex B.6.2.3, B.6.2.6 and B.6.2.8
- EU AI Act in the Cenedril wiki: obligations for high-risk systems at a glance