A provider develops an AI system that ranks incoming emergency calls by urgency in dispatch centres. Its manual is complete: test plan, release procedure, monitoring plan with a monthly evaluation. Then a customer pushes for a new model version. The team ships it after a shortened test, the person who tested it also signs the release, and the monitoring evaluation is skipped two months in a row because the colleague in charge is tied up in another project. When the market surveillance authority asks for documents after a complaint, the room goes quiet. For the current version, there is no evidence at all that the described procedures were applied.
EN 18286 requires providers, in clause 6.1, to determine the risks to the quality management system (QMS) itself when they plan it. The first of these risks is the most obvious one: the system does not deliver what it was set up for. That means the ongoing evidence that the provider’s high-risk AI systems meet the requirements of the AI Act, from design to retirement.
What is behind it?
A QMS is a chain of processes. It only works if every link is actually performed and its result is recorded. When one link fails, the rest of the chain still looks tidy, yet the statement about the system’s conformity loses its basis. The standard therefore asks providers to consider, already at the planning stage, what could endanger the intended outcomes. For these risks the provider plans actions, builds them into the QMS processes and decides how to evaluate whether they work (clause 6.1.2).
Where processes typically break
- Skipped tests. Deadline pressure shortens test runs or drops them entirely. This often affects small model updates that nobody classifies as a change.
- Missed evaluations. The monitoring plan names a cycle, yet nobody checks whether the evaluation took place. The data is there, a judgement on continued compliance is missing.
- Merged roles. The person who develops, tests and releases ends up being the same person. The release loses its value as an independent control.
- Missing records. The test took place, but its result sits in a chat thread or a local file. As evidence, that is weak.
QMS risk or system risk?
The standard keeps the two strictly apart. Risk management for the individual AI system under Article 9 of the AI Act looks at dangers to health, safety and fundamental rights that come from the system. The risks under clause 6.1 concern whether the management system functions. Separate does not mean unrelated, though: a QMS that misses its outcomes leaves system risks undetected, because the very tests that would have found them are missing.
Impact
Without evidence there is no robust conformity assessment. In the internal control procedure under Annex VI, the provider itself verifies that its QMS meets Article 17 and that development and monitoring match the technical documentation. If it later turns out that a system on the market does not conform, corrective actions under Article 20 follow, in the worst case withdrawal or recall. Under Article 16 the QMS is one of the provider obligations whose breach can lead to fines under Article 99.
Then there are the consequences for the people the system affects. An error that a skipped test would have found hits them directly: an emergency rated too low, an application rejected without reason. It gets most expensive when a failed process only comes to light through a request from the authority, because by then there is no time left to close the gap properly.
Practical examples
Medical image analysis. A provider updates the model of software that pre-sorts skin images for review by doctors. Because the change counts as a “minor improvement”, only part of the test cases are run. The new version is worse at detecting a rare skin lesion. The full test run that the test plan requires for every model version would have caught this.
Scoring for banks. A software company supplies banks with a creditworthiness assessment system and keeps a monitoring plan with a quarterly evaluation for it. After a staff change, the evaluation is left undone twice. During this time the mix of applicants shifts, and the rejection rate for one age group rises sharply. The data showing this was available all along.
Candidate pre-selection. A provider sells a tool that pre-sorts job applications. Releases run through a ticket system. While preparing for an audit, the team notices that a third of all releases were tested and approved by the same person. The tests were fine, yet nobody confirmed them independently.
Effective measures
The AI Provider Policy from the AI Compliance Starter Kit addresses this risk in several places:
- Section 3 (Processes of the quality management system): The process table lists the results of each process and, in a separate column, how its monitoring shows that it works. Every process thus becomes something you can check.
- Section 8 (Life cycle processes): Every phase has entry and exit criteria. Verification and release lie with different roles, and a process that misses its criteria is handled as a corrective action.
- Section 13 (Examination, test and validation): Tests run for every model version and before every release against defined acceptance thresholds.
- Section 16 (Risk management): The risks of the QMS are kept in the AI Risk Register under the category “Quality management system”. Their actions are built into the processes and evaluated for effectiveness in the management review.
- Section 17 (Post-market monitoring): The monitoring plan sets the cycle and the person responsible, and each evaluation is recorded.
These ISO/IEC 42001 controls contribute as well:
- A.3 Internal organization: Clearly assigned roles prevent testing and release from ending up in one hand.
- A.6.1 Management guidance for AI system development: Objectives and processes for responsible development define which steps belong to every development state.
- A.6.2 AI system life cycle: Verification, validation and monitoring in operation are fixed stations in the life cycle of every system.
The topic comes up on schedule in the management review: EN 18286 lists the actions against QMS risks among the items reviewed there (clause 10.1).
Link to the AI Act
Article 17(1) of Regulation (EU) 2024/1689 requires providers of high-risk AI systems to have a QMS that ensures compliance with the Regulation and is laid down in written policies, procedures and instructions. Its minimum content includes examination, test and validation procedures together with their frequency (point (d)) and a post-market monitoring system in line with Article 72 (point (h)). A QMS whose processes do not work misses exactly this purpose. The duty to determine the risks of the QMS itself comes from EN 18286; Article 17 does not name it explicitly. If you apply the standard, you can rely on the presumption of conformity for Article 17(1), to the extent defined by Annex ZA of the standard (→ EN 18286: Quality Management System for AI Providers).
Related QMS risks
- QMS-02 Undesired effects of the QMS: Where processes are too heavy, people work around them, and the undesired effect turns into a missed outcome.
- QMS-06 Missing resources or competences: Skipped evaluations often have a plain cause: the person who should do them is missing.
- T06 Life cycle and maintenance gaps: The threat at system level that arises when steps in the life cycle are missing.
Sources
- EN 18286:2026, clause 6.1 (planning actions on risks to the functioning of the QMS) and clause 10.1 (management review)
- Regulation (EU) 2024/1689 (AI Act), Articles 16, 17, 20, 72 and 99, Annex VI
- Implementing Decision (EU) 2026/2274: reference of EN 18286 as a harmonised standard
- AI Provider Policy, Sections 3, 8, 13, 16 and 17
- EN 18286: Quality Management System for AI Providers in the Cenedril wiki