Zum Hauptinhalt springen
Misuse pattern · AI Act

MU-04 Use by persons who were not briefed

Updated on 6 min Reviewed by: Cenedril Editorial
EU AI ActISO/IEC 42001

In a radiology practice, an AI system marks conspicuous areas on X-ray images. The permanent staff know from the introductory training that the system reliably detects certain types of findings and does not check others at all. A locum doctor helping out for two weeks never received that training. She reads an image without a marking as unremarkable. The system worked exactly as designed, and what was missing was knowledge of what it cannot do.

This pattern describes AI systems operated by people without the required briefing or competence. It almost always arises at the edges of the organization: with stand-ins, new staff, agency workers and external service providers. For the AI Act (Regulation (EU) 2024/1689), it is reasonably foreseeable misuse under Article 3(13), because staff turnover exists in every organization.

What is behind it?

A forklift requires a licence, a chat interface requires nothing. Modern AI systems are so easy to operate that their demands on judgement become invisible. The competence at stake rarely concerns operation. It concerns understanding what an output means, when it is unreliable and what to do in case of doubt.

How it happens in daily work

  • Stand-ins and holidays. Whoever steps in briefly gets an account and a verbal summary that leaves out the system’s limits.
  • New staff and agency workers. The training took place at rollout. Whoever joins later learns from the colleague at the next desk, habits included.
  • External service providers. A call centre or an agency operates the system on the organization’s behalf, and the contract says nothing about a briefing.
  • New versions. An update changes the system’s behaviour, the training stays where it was.
  • Change of role. Someone takes over oversight of a system because the previous person in charge moved to another department, and inherits duties without the knowledge that goes with them.

Why it is risky

People without a briefing misread outputs: they take a score for a probability, a missing warning for an all-clear, a fluent answer for a checked fact. They do not recognize malfunctions and do not know the route for reporting them or stopping the system. For providers this is particularly delicate, because their risk analysis rests on assumptions about what users know. Article 9(5) of the AI Act expressly requires taking into account the knowledge that can be expected from the deployer and the likely context of use. If that knowledge is missing in operation, the risk analysis no longer holds.

Impact

The impact equals that of the system itself, minus the safeguard that trained people provide. With a writing assistant it often stays at rework. With systems that decide about people or are used in diagnosis, lending and safety, the damage can affect health, livelihoods and fundamental rights. Effective human oversight presupposes that the human knows what to look for.

Practical examples

Credit pre-check during the holidays. A bank’s credit officer is on leave, and a colleague from payments takes over. She does not know that the instructions rate the scores for self-employed applicants with a short business history as of little value, and she rejects three applications on the strength of a low score alone.

External customer service. A mail-order retailer outsources the second level of its customer service to a provider that works with the retailer’s AI-supported reply system. The contract sets response times and does not mention a briefing on the system. The provider’s staff adopt suggestions on return periods that do not apply to certain product groups.

Working student with agent access. A working student in IT uses an AI agent that can run commands on servers. Nobody told him that suggested database changes may only be run in the test environment. One suggested command empties a table in the production system.

Effective controls

The following ISO/IEC 42001 controls counter this pattern.

Prevention:

  • A.4 Resources for AI systems: The resources of an AI system include the people who run and oversee it, together with their competences. The organization documents which roles a system needs, including the roles for human oversight.
  • A.8 Information for interested parties: Users receive instructions, notices and learning material in a form that matches their level of knowledge.
  • A.9 Use of AI systems: People involved in human oversight are informed and trained and understand the instructions and their own tasks.

Detection:

  • A.9 Use of AI systems: A regular comparison of access rights with briefing records shows who uses a system without a briefing. This is established practice that the standard does not prescribe in detail.

Response:

  • A.4 Resources for AI systems: Where a role lacks the necessary competence, the person is retrained or access is restricted until the gap is closed.

At the core of the management system, Clause 7.2 of ISO/IEC 42001 requires the organization to determine, ensure and evidence the necessary competence. The AI Usage Policy puts this into practice in Section 14: training needs per role, recorded completion, application in daily work. Section 6 describes what the person exercising oversight has to understand, and Section 10 assigns oversight of high-risk systems only to people with competence, training and authority. For general awareness, control A.6.3 Information Security Awareness, Education and Training helps.

Article 4 applies to all AI systems: providers and deployers take measures that support the development of AI literacy among their staff and other people acting on their behalf. Knowledge, experience, education, the context of use and the people on whom the system is used are taken into account. Article 3(56) defines AI literacy as the skills, knowledge and understanding needed to use AI systems in an informed way and to be aware of their opportunities and risks.

For high-risk AI systems, things become more specific. The provider evaluates foreseeable misuse under Article 9(2)(b), takes into account the knowledge to be expected from the deployer under Article 9(5) and states in the instructions for use the circumstances that may lead to risks (Article 13(3)(b)(iii)) and the human oversight measures (point (d)). Article 14(4) describes what people exercising oversight must understand and be able to do. Under Article 26(2), the deployer assigns oversight only to people with the necessary competence, training and authority (→ EU AI Act).

Sources

Frequently asked questions

Does the AI Act require training for everyone who uses AI?

Article 4 requires providers and deployers to take measures that support the AI literacy of their staff and of other people acting on their behalf; it sets no specific level for each individual. Article 26(2) is stricter for high-risk systems: only people with the necessary competence, training and authority are given human oversight. An organization running a management system under ISO/IEC 42001 also has to determine and ensure the necessary competence under Clause 7.2.

What belongs in a briefing for an AI system?

The purpose and limits of the system, typical errors and how to spot them, what scores or confidence values mean, the rules for inputs, the route for reporting anomalies and who decides in case of doubt. For systems with human oversight, add how to intervene, override or stop.

Do staff of external service providers have to be briefed as well?

Yes, if they operate the system on behalf of the organization. Article 4 expressly names other persons who use AI systems on behalf of the provider or deployer. The contract is the simplest place to settle it: briefing before first access, evidence on request, refresher after significant changes.

No Cookies!

This wiki collects nothing, bakes nothing and leaves nothing behind. There's nothing to consent to. Privacy doesn't get better than this.