A regional bank uses a scoring model from an external service provider for consumer loans. A customer is turned down and asks why. The clerk sees nothing on her screen beyond a score and a red traffic light, and the provider’s documents describe the model in two paragraphs. The customer complains to the data protection authority. What the bank needs now is something it should have settled before the first use: which information it can give to whom about the system.
ISO/IEC 42001 lists a lack of transparency and explainability as a risk source in Annex C. The idea behind it is simple. An organization that cannot give interested parties adequate information about its AI system loses trustworthiness and struggles to account for the system’s results. This concerns customers as much as staff, auditors and authorities.
What is behind it?
Transparency and explainability are related, yet they mean different things. Transparency describes what is openly known about a system and its use. Explainability describes whether a single result can be justified in a way people understand. Either can be missing, and each gap has its own causes. A further question is who the explanation is for: a data scientist needs different details from a rejected applicant or an inspector from the supervisory authority.
Where transparency and explainability break down
- Opaque models. Deep neural networks and large language models compute with millions to billions of parameters. Which features carried a given result can only be reconstructed approximately.
- Missing documentation. Purpose, data basis, limits and known weaknesses are written down nowhere or known only to the development team. When a key person leaves, the knowledge leaves too.
- Purchased systems. The supplier withholds details about data and model, and the contract contains no duty to provide information.
- Wrong audience. The available information is written for specialists. Case handlers, affected people or senior management can do little with it.
- Hidden use. People do not know that they are dealing with an AI system or that a decision rests on its output.
Impact
The first damage is to trust. Affected people find it harder to accept a decision that nobody can justify, and they push back more often. On top of that comes a problem of accountability: without an explanation, nobody can check whether a result was reached correctly, fairly and lawfully. Opacity therefore amplifies almost every other AI threat, because errors, bias and manipulation stay hidden for longer. The legal consequences range from complaints to regulatory proceedings and disputes over access rights, and inside the company uncertainty and resistance to the system grow.
Practical examples
Pre-screening of job applications. A company has an AI system sort incoming applications. A rejected candidate asks for the reasons. All HR can tell her is that the system rated her profile as a “weak match”. Whether gaps in her CV, her university or particular wording tipped the balance, nobody knows.
Diagnostic support in radiology. Software marks suspicious areas on X-ray images. The physicians see the marking, yet they learn neither how confident the system is nor for which types of images it performs worse. They therefore find it hard to weigh its hints and, depending on temperament, rely on them too much or too little.
Customer chat without labelling. An insurer answers enquiries through a chat on its website. Nowhere does it say that the answers come from a language model. When a customer relies on a wrong promise, it is unclear which documents the system drew on and who should have checked the answer.
Effective controls
The following ISO/IEC 42001 controls counter this threat.
Prevention:
- A.6.2 AI system life cycle: The organization determines which technical documentation each group of interested parties needs and records design and development so that decisions can be traced later.
- A.7 Data for AI systems: Provenance and preparation of the data are recorded. Without these details it is hard to explain why a model decides the way it does.
- A.10 Third-party and customer relationships: Suppliers of AI systems deliver adequate documentation, and customers learn which domain of use a system is valid for.
Detection:
- A.6.2 AI system life cycle: Event logs show in operation which inputs led the system to which result.
- A.8 Information for interested parties: A reporting channel for adverse impacts reveals where people do not understand or do not accept results.
Response:
- A.8 Information for interested parties: The organization decides which information users receive and knows its reporting obligations towards customers and authorities. When questions come in, it can answer with prepared and reviewed content.
Link to the AI Act
The AI Act (Regulation (EU) 2024/1689) addresses transparency at several levels. Under Article 13, high-risk systems must be designed so that deployers can interpret their outputs and use them appropriately, and their instructions for use describe capabilities and limitations. Article 11 together with Annex IV governs technical documentation, Article 12 automatic logging. Article 50 requires for certain systems, regardless of risk class, that people can recognize when they interact with AI or are looking at AI-generated content. Article 86 gives people who are significantly affected by a decision based on a high-risk system from Annex III the right to obtain from the deployer a clear explanation of the system’s role and of the main reasons for the decision. For automated individual decisions, the GDPR additionally requires meaningful information about the logic involved (Article 13(2)(f), Article 14(2)(g), Article 15(1)(h)) (→ EU AI Act).
Related threats
- T03 Excessive automation (automation bias): People who cannot follow a result are more likely to accept it unchecked.
- T12 Bias and discrimination: Bias stays hidden especially long in opaque systems.
- T13 Hallucination / faulty output: Fabricated content can only be exposed when the basis of a statement is visible.
Sources
- ISO/IEC 42001:2023, Annex C.3.2: Lack of transparency and explainability, together with C.2.11
- Regulation (EU) 2024/1689 (AI Act), Articles 11, 12, 13, 50 and 86
- Regulation (EU) 2016/679 (GDPR), Articles 13 to 15 and 22
- ISO/IEC 42001 in the Cenedril wiki: Annex C and the risk sources for AI systems