An energy supplier offers a chat on its website in which ‘Anna from the customer team’, complete with photo, answers questions. Anna is a language model. A customer describes her financial hardship in detail, because she believes she is talking to a real person, and receives a promise of instalment payments the system is not allowed to give. The customer confided in a person who did not exist, and disclosure would have prevented exactly that mistake.
This pattern describes AI-generated content passed on without the disclosure the situation requires. The AI Act (Regulation (EU) 2024/1689) governs part of these situations expressly in Article 50. As reasonably foreseeable misuse under Article 3(13), the pattern is obvious in any case: people who use AI often want to present the result as their own work.
What is behind it?
Whether content must be recognizable as AI-generated depends on the situation. Law, contracts and professional rules require disclosure wherever people base their trust, their money or their decisions on the assumption that a human is behind it. The AI Act names three cases expressly: conversations with an AI system, deep fakes and AI text on matters of public interest. Alongside these come contracts that promise personal work, as well as examination and publication rules.
How it happens in daily work
- Prestige. Work done with AI help is seen as worth less, so the help is concealed.
- Unclear rules. Staff do not know when disclosure is needed and when it is not.
- Friendly personas. Chatbots get human names and photos to seem more approachable.
- Lost marking. Watermarks and metadata get lost through editing, exporting or taking screenshots.
- Copying into own documents. AI text ends up in a report that carries a person’s name.
Why it is risky
Recipients are deceived and make decisions on a false basis. They also scrutinize content less critically because they assume human care, and so overlook errors they would have looked for in labelled AI text. Legally, breaches of Article 50 are subject to fines under Article 99, and disputes over contractually agreed services may follow. In addition, purely machine-generated content is widely considered to enjoy little copyright protection, which can become a problem for works that are sold.
Impact
The loss of trust weighs heaviest once the deception comes to light: with customers, with the public, with clients. With deep fakes, the personality rights of the person depicted may also be infringed. A single exposed case casts doubt retroactively on everything the organization has published or delivered so far.
Practical examples
Expert report from a language model. A consultancy delivers a market report to a client that was largely written by a language model and only superficially revised. The contract promises an analysis by named experts. When the client comes across invented sources, the entire fee is up for debate.
Video message with a synthetic voice. The communications department of a group produces a video message in which the CEO speaks to staff with AI-generated voice and facial expressions. There is no notice. Once it becomes known that the recording is synthetic, many start doubting genuine messages too.
News item from a municipality. A city’s press office publishes an AI-generated news item about a council decision without anyone checking the content or taking editorial responsibility. The item contains a wrong figure on fees, and the disclosure under Article 50(4) is missing as well.
Effective controls
The following ISO/IEC 42001 controls counter this pattern.
Prevention:
- A.3 Internal organization: Roles set out who approves external publications and holds editorial responsibility.
- A.8 Information for interested parties: Users learn that they are dealing with an AI system, and outputs are marked as generated, for instance with a notice that an image was created by a machine.
- A.10 Third-party and customer relationships: The organization understands its customers’ expectations and takes them into account, including contractual ones, when it delivers services with AI.
- A.9 Use of AI systems: The processes for responsible use contain the disclosure rules, and transparency is one of the objectives of use.
Detection:
- A.3 Internal organization: Through the channel for reporting concerns, staff can flag AI content published without disclosure.
Response:
- A.8 Information for interested parties: Affected publications are corrected and labelled after the fact, and recipients are informed.
The AI Usage Policy prohibits in Section 4.2 passing off AI-generated content as human where this misleads the recipient. Section 5 governs disclosure under Article 50, Section 5.1 the approval of published content and Section 5.2 the timing and accessibility of the information.
Link to the AI Act
Article 50 divides the transparency obligations between providers and deployers. Providers design systems for direct interaction so that people learn they are talking to an AI, unless this is obvious (paragraph 1), and mark synthetic audio, image, video and text content in a machine-readable way (paragraph 2). Deployers disclose that deep fakes within the meaning of Article 3(60) are artificially generated or manipulated, with relief for evidently artistic or satirical works, and disclose AI text on matters of public interest unless it was reviewed by humans and someone holds editorial responsibility (paragraph 4). The information is provided at the latest at the first interaction or exposure, clearly and accessibly (paragraph 5).
For high-risk systems, the misuse obligations of Article 9(2)(b) and Article 13(3)(b)(iii) apply in addition. Article 50(6) clarifies that the transparency obligations leave the requirements for high-risk systems unaffected (→ EU AI Act).
Related patterns and threats
- MU-02 Adopting outputs unchecked: Whoever adopts AI content unchecked often passes it on unlabelled as well.
- T02 Lack of transparency and explainability: The threat that missing disclosure makes worse.
- T13 Hallucination / faulty output: Invented sources and facts seem particularly credible under a human name.
- MU-10 Reuse of outputs for another purpose: An internally generated draft moves outwards without any notice.
Sources
- Regulation (EU) 2024/1689 (AI Act), Article 3(13) and (60), Articles 9, 13, 50 and 99
- ISO/IEC 42001:2023, Annex B.8.2, B.9.3 and B.10.4
- AI Usage Policy of the AI Compliance Starter Kit, Sections 4.2, 5, 5.1 and 5.2
- EU AI Act in the Cenedril wiki: transparency obligations for providers and deployers