Zum Hauptinhalt springen
AI threat · ISO/IEC 42001 Annex C

T13 Hallucination / faulty output

Updated on 5 min Reviewed by: Cenedril Editorial
ISO/IEC 42001EU AI Act

A tax advisory firm has a language model prepare answers to client enquiries. In one draft, the model cites a ruling of the Federal Fiscal Court with case number and date that fits the question perfectly. The ruling does not exist. After a quick look, the draft goes to the client, who bases a decision on it. A language model phrases false statements just as fluently and convincingly as true ones.

ISO/IEC 42001 does not use the term hallucination, yet it addresses the problem in two places: in Annex C under the limitations of young technology, and in control A.6.2.4, which requires defined verification and validation measures. In the product, the threat covers factually wrong or invented outputs that appear correct and can lead to wrong decisions.

What is behind it?

Language models produce text by computing a likely continuation, word by word. The basis is patterns from vast amounts of text. There is no built-in check against reality. A language model produces plausible text; whether it is true, the model does not check by itself. Where knowledge is missing, it fills the gap with something that sounds as if it could be right.

Forms it takes

  • Invented facts. Figures, dates, names or events that never existed.
  • Invented sources. References, rulings, clauses of standards or links that look real and lead nowhere.
  • Faulty summaries. Qualifications drop out, attributions get swapped, a negation is lost.
  • Wrong calculations and reasoning. Conclusions that are linguistically clean and wrong in substance.
  • Outdated knowledge. The model knows the world up to its training date and presents that state as current.
  • Errors from connected sources. In systems with document search, the model draws on the wrong document or mixes content from several sources.

Why it happens

The training objective rewards fluent, likely text. Gaps in the training material, ambiguous questions and long inputs raise the error rate. Many models tend to give an answer rather than admit they do not know. Connecting the model to verified documents lowers the risk without removing it.

Impact

Hallucinations lead to wrong decisions, wrong information given to customers, faulty contracts and reports, or dangerous advice in medicine and engineering. In several countries, court filings citing invented rulings have already come to light. The damage occurs where someone adopts the output unchecked and passes it on. Afterwards the organization often has to answer for a statement nobody ever made consciously.

Practical examples

Chatbot with an invented goodwill rule. A tour operator answers questions through a chatbot on its website. The bot tells a customer that after a bereavement he can cancel the trip free of charge up to thirty days after departure. No such rule exists. The customer insists on the information, and the company pays to avoid a legal dispute.

Summary of an inspection report. An engineering firm has a language model summarize long inspection reports on bridges. The report says a crack “cannot be ruled out”, and the summary turns this into “no crack found”. Maintenance is postponed until an expert reads the original report.

Knowledge assistant with outdated documents. An internal assistant answers questions on travel expense reimbursement. Its document collection holds the current policy as well as an old version with different allowances. The assistant quotes one or the other at random, and accounting has to correct dozens of expense claims.

Effective controls

The following ISO/IEC 42001 controls counter this threat.

Prevention:

  • A.9 Use of AI systems: The processes for responsible use set out what generative AI is used for, which approvals are needed and where people review the outputs. The system is used only within its intended use.
  • A.7 Data for AI systems: The quality requirements also apply to data in operation, such as the documents an assistant draws its answers from. Their origin and status are documented.
  • A.8 Information for interested parties: Users learn where the system’s limits lie, how accurate it is and when human oversight is needed. Content created by AI is recognizable as such.

Detection:

  • A.6.2 AI system life cycle: Verification and validation work with defined criteria and acceptable error rates, for example with test questions whose correct answer is known. Monitoring and event logs continue the check in operation.
  • A.8 Information for interested parties: A reporting channel for adverse impacts brings faulty outputs to light that nobody noticed internally.

Response:

  • A.6.2 AI system life cycle: Defined update procedures clean up document collections, adjust instructions and filters and inform users about the change.

For generative AI, the AI Act (Regulation (EU) 2024/1689) relies mainly on transparency. Under Article 50(2), providers of systems that generate synthetic text, images, audio or video mark the outputs in a machine-readable way as artificially generated. Under Article 50(4), deployers that publish AI-generated text to inform the public on matters of public interest disclose this, unless a person has reviewed the text editorially and holds responsibility for it. Article 4 requires sufficient AI literacy among staff. Providers of general-purpose AI models inform downstream providers about the capabilities and limitations of their model under Article 53. Where a generative system is part of a high-risk system, the requirements for accuracy under Article 15 and for human oversight under Article 14 apply as well (→ EU AI Act).

Sources

Frequently asked questions

Can hallucinations be switched off entirely?

Not with today's state of the art. Connected, verified sources, clear task definitions and review steps lower the error rate, yet every generative model retains a residual risk. In the end, safety therefore depends on how the organization handles the outputs.

Does it help to ask the model for its sources?

Only if you check the sources afterwards. Without connected documents, a model can invent references as well. With systems that search documents, it is worth looking at the cited passage, because sometimes the claimed statement is simply not there.

What should users know about hallucinations?

That a fluent, confident answer is no sign of correctness. They should know which tasks the system is approved for, which outputs they have to check and how to report errors. ISO/IEC 42001 requires users to be given the necessary information, and Article 4 of the AI Act requires sufficient AI literacy.

No Cookies!

This wiki collects nothing, bakes nothing and leaves nothing behind. There's nothing to consent to. Privacy doesn't get better than this.