In a radiology practice, an AI system marks conspicuous areas on X-ray images. The permanent staff know from the introductory training that the system reliably detects certain types of findings and does not check others at all. A locum doctor helping out for two weeks never received that training. She reads an image without a marking as unremarkable. The system worked exactly as designed, and what was missing was knowledge of what it cannot do.
This pattern describes AI systems operated by people without the required briefing or competence. It almost always arises at the edges of the organization: with stand-ins, new staff, agency workers and external service providers. For the AI Act (Regulation (EU) 2024/1689), it is reasonably foreseeable misuse under Article 3(13), because staff turnover exists in every organization.
What is behind it?
A forklift requires a licence, a chat interface requires nothing. Modern AI systems are so easy to operate that their demands on judgement become invisible. The competence at stake rarely concerns operation. It concerns understanding what an output means, when it is unreliable and what to do in case of doubt.
How it happens in daily work
- Stand-ins and holidays. Whoever steps in briefly gets an account and a verbal summary that leaves out the system’s limits.
- New staff and agency workers. The training took place at rollout. Whoever joins later learns from the colleague at the next desk, habits included.
- External service providers. A call centre or an agency operates the system on the organization’s behalf, and the contract says nothing about a briefing.
- New versions. An update changes the system’s behaviour, the training stays where it was.
- Change of role. Someone takes over oversight of a system because the previous person in charge moved to another department, and inherits duties without the knowledge that goes with them.
Why it is risky
People without a briefing misread outputs: they take a score for a probability, a missing warning for an all-clear, a fluent answer for a checked fact. They do not recognize malfunctions and do not know the route for reporting them or stopping the system. For providers this is particularly delicate, because their risk analysis rests on assumptions about what users know. Article 9(5) of the AI Act expressly requires taking into account the knowledge that can be expected from the deployer and the likely context of use. If that knowledge is missing in operation, the risk analysis no longer holds.
Impact
The impact equals that of the system itself, minus the safeguard that trained people provide. With a writing assistant it often stays at rework. With systems that decide about people or are used in diagnosis, lending and safety, the damage can affect health, livelihoods and fundamental rights. Effective human oversight presupposes that the human knows what to look for.
Practical examples
Credit pre-check during the holidays. A bank’s credit officer is on leave, and a colleague from payments takes over. She does not know that the instructions rate the scores for self-employed applicants with a short business history as of little value, and she rejects three applications on the strength of a low score alone.
External customer service. A mail-order retailer outsources the second level of its customer service to a provider that works with the retailer’s AI-supported reply system. The contract sets response times and does not mention a briefing on the system. The provider’s staff adopt suggestions on return periods that do not apply to certain product groups.
Working student with agent access. A working student in IT uses an AI agent that can run commands on servers. Nobody told him that suggested database changes may only be run in the test environment. One suggested command empties a table in the production system.
Effective controls
The following ISO/IEC 42001 controls counter this pattern.
Prevention:
- A.4 Resources for AI systems: The resources of an AI system include the people who run and oversee it, together with their competences. The organization documents which roles a system needs, including the roles for human oversight.
- A.8 Information for interested parties: Users receive instructions, notices and learning material in a form that matches their level of knowledge.
- A.9 Use of AI systems: People involved in human oversight are informed and trained and understand the instructions and their own tasks.
Detection:
- A.9 Use of AI systems: A regular comparison of access rights with briefing records shows who uses a system without a briefing. This is established practice that the standard does not prescribe in detail.
Response:
- A.4 Resources for AI systems: Where a role lacks the necessary competence, the person is retrained or access is restricted until the gap is closed.
At the core of the management system, Clause 7.2 of ISO/IEC 42001 requires the organization to determine, ensure and evidence the necessary competence. The AI Usage Policy puts this into practice in Section 14: training needs per role, recorded completion, application in daily work. Section 6 describes what the person exercising oversight has to understand, and Section 10 assigns oversight of high-risk systems only to people with competence, training and authority. For general awareness, control A.6.3 Information Security Awareness, Education and Training helps.
Link to the AI Act
Article 4 applies to all AI systems: providers and deployers take measures that support the development of AI literacy among their staff and other people acting on their behalf. Knowledge, experience, education, the context of use and the people on whom the system is used are taken into account. Article 3(56) defines AI literacy as the skills, knowledge and understanding needed to use AI systems in an informed way and to be aware of their opportunities and risks.
For high-risk AI systems, things become more specific. The provider evaluates foreseeable misuse under Article 9(2)(b), takes into account the knowledge to be expected from the deployer under Article 9(5) and states in the instructions for use the circumstances that may lead to risks (Article 13(3)(b)(iii)) and the human oversight measures (point (d)). Article 14(4) describes what people exercising oversight must understand and be able to do. Under Article 26(2), the deployer assigns oversight only to people with the necessary competence, training and authority (→ EU AI Act).
Related patterns and threats
- MU-02 Adopting outputs unchecked: People who do not know what to look for review only for form.
- MU-11 Passing on access credentials: A shared account almost always ends up with people who were never briefed.
- MU-03 Entering confidential or personal data: Not knowing the input rules is one of the most common causes of data leaks.
- T03 Excessive automation (automation bias): Without a briefing there is no counterweight to excessive trust in the machine.
Sources
- Regulation (EU) 2024/1689 (AI Act), Article 3(13) and (56), Articles 4, 9, 13, 14 and 26, Article 4 as amended by Regulation (EU) 2026/1744
- ISO/IEC 42001:2023, Clause 7.2, Annex B.4.6, B.8.2 and B.9.3
- AI Usage Policy of the AI Compliance Starter Kit, Sections 6, 10 and 14
- ISO/IEC 42001 in the Cenedril wiki: competence and resources in the AI management system