A tax advisory firm has a language model prepare answers to client enquiries. In one draft, the model cites a ruling of the Federal Fiscal Court with case number and date that fits the question perfectly. The ruling does not exist. After a quick look, the draft goes to the client, who bases a decision on it. A language model phrases false statements just as fluently and convincingly as true ones.
ISO/IEC 42001 does not use the term hallucination, yet it addresses the problem in two places: in Annex C under the limitations of young technology, and in control A.6.2.4, which requires defined verification and validation measures. In the product, the threat covers factually wrong or invented outputs that appear correct and can lead to wrong decisions.
What is behind it?
Language models produce text by computing a likely continuation, word by word. The basis is patterns from vast amounts of text. There is no built-in check against reality. A language model produces plausible text; whether it is true, the model does not check by itself. Where knowledge is missing, it fills the gap with something that sounds as if it could be right.
Forms it takes
- Invented facts. Figures, dates, names or events that never existed.
- Invented sources. References, rulings, clauses of standards or links that look real and lead nowhere.
- Faulty summaries. Qualifications drop out, attributions get swapped, a negation is lost.
- Wrong calculations and reasoning. Conclusions that are linguistically clean and wrong in substance.
- Outdated knowledge. The model knows the world up to its training date and presents that state as current.
- Errors from connected sources. In systems with document search, the model draws on the wrong document or mixes content from several sources.
Why it happens
The training objective rewards fluent, likely text. Gaps in the training material, ambiguous questions and long inputs raise the error rate. Many models tend to give an answer rather than admit they do not know. Connecting the model to verified documents lowers the risk without removing it.
Impact
Hallucinations lead to wrong decisions, wrong information given to customers, faulty contracts and reports, or dangerous advice in medicine and engineering. In several countries, court filings citing invented rulings have already come to light. The damage occurs where someone adopts the output unchecked and passes it on. Afterwards the organization often has to answer for a statement nobody ever made consciously.
Practical examples
Chatbot with an invented goodwill rule. A tour operator answers questions through a chatbot on its website. The bot tells a customer that after a bereavement he can cancel the trip free of charge up to thirty days after departure. No such rule exists. The customer insists on the information, and the company pays to avoid a legal dispute.
Summary of an inspection report. An engineering firm has a language model summarize long inspection reports on bridges. The report says a crack “cannot be ruled out”, and the summary turns this into “no crack found”. Maintenance is postponed until an expert reads the original report.
Knowledge assistant with outdated documents. An internal assistant answers questions on travel expense reimbursement. Its document collection holds the current policy as well as an old version with different allowances. The assistant quotes one or the other at random, and accounting has to correct dozens of expense claims.
Effective controls
The following ISO/IEC 42001 controls counter this threat.
Prevention:
- A.9 Use of AI systems: The processes for responsible use set out what generative AI is used for, which approvals are needed and where people review the outputs. The system is used only within its intended use.
- A.7 Data for AI systems: The quality requirements also apply to data in operation, such as the documents an assistant draws its answers from. Their origin and status are documented.
- A.8 Information for interested parties: Users learn where the system’s limits lie, how accurate it is and when human oversight is needed. Content created by AI is recognizable as such.
Detection:
- A.6.2 AI system life cycle: Verification and validation work with defined criteria and acceptable error rates, for example with test questions whose correct answer is known. Monitoring and event logs continue the check in operation.
- A.8 Information for interested parties: A reporting channel for adverse impacts brings faulty outputs to light that nobody noticed internally.
Response:
- A.6.2 AI system life cycle: Defined update procedures clean up document collections, adjust instructions and filters and inform users about the change.
Link to the AI Act
For generative AI, the AI Act (Regulation (EU) 2024/1689) relies mainly on transparency. Under Article 50(2), providers of systems that generate synthetic text, images, audio or video mark the outputs in a machine-readable way as artificially generated. Under Article 50(4), deployers that publish AI-generated text to inform the public on matters of public interest disclose this, unless a person has reviewed the text editorially and holds responsibility for it. Article 4 requires sufficient AI literacy among staff. Providers of general-purpose AI models inform downstream providers about the capabilities and limitations of their model under Article 53. Where a generative system is part of a high-risk system, the requirements for accuracy under Article 15 and for human oversight under Article 14 apply as well (→ EU AI Act).
Related threats
- T03 Excessive automation (automation bias): Whoever trusts the machine too much also passes on invented content.
- T07 Immature technology / model drift: A model change at the provider can alter the error rate overnight.
- MU-02 Adopting outputs unchecked: The misuse pattern describes how a hallucination turns into damage.
Sources
- ISO/IEC 42001:2023, Annex C.3.7: Technology readiness, control A.6.2.4 with Annex B.6.2.4
- Regulation (EU) 2024/1689 (AI Act), Articles 4, 14, 15, 50 and 53
- ISO/IEC 42001 in the Cenedril wiki: Annex C and the risk sources for AI systems