A sales manager copies the customer list with revenues, contact persons and discount levels into a freely available chatbot to get an analysis for the quarterly review. The analysis is ready in two minutes and looks excellent. In the same two minutes, confidential business data and personal data left the company, and whether they will ever be deleted is no longer in its hands.
This pattern describes the entry of data that may not be processed in the system concerned. It is the most frequent case of foreseeable misuse with generative tools and touches two bodies of law at once: the AI Act (Regulation (EU) 2024/1689) with its obligations on misuse, and the GDPR, which under Article 2(7) of the AI Act continues to apply alongside it.
What is behind it?
Whether an AI service may process certain data depends on conditions outside the tool: the contract with the provider, the storage location, whether inputs are used for training, and the legal basis for personal data. To users every input field looks the same, while legally they can be worlds apart. The approval of a tool therefore always covers specific data classes.
How it happens in daily work
- Convenience. Whole email threads, contracts or spreadsheets are copied faster than they can be summarized and stripped of names.
- Private accounts. The free version is available instantly, the enterprise version first has to be requested.
- Embedded features. An add-in in the word processor or the video conferencing system sends content to an external provider without anyone deciding so consciously.
- Missing knowledge. Many staff do not know where their inputs are stored, who can read them and whether they flow into future models.
- Hidden data. Configuration files contain passwords, screenshots show customer names, minutes include health details.
Why it is risky
Inputs may be stored by the provider, viewed by its staff, used for training or transferred to third countries. For personal data, there is then often no legal basis under Article 6 GDPR and no data processing agreement under Article 28 GDPR. Trade secrets risk losing legal protection, because the German Trade Secrets Act protects only information for which the holder takes reasonable confidentiality measures (section 2 no. 1(b) GeschGehG). There is also a technical risk: data from training may later resurface in outputs.
Impact
Confidentiality and lawfulness suffer. Depending on the case, a personal data breach may have occurred that must be notified to the supervisory authority within 72 hours under Article 33 GDPR, unless it is unlikely to result in a risk. Add fines, contractual penalties under non-disclosure agreements and the loss of competitive advantage. Health data and other special categories under Article 9 GDPR weigh heaviest, because their disclosure can harm people permanently.
Practical examples
Reference letter with sick leave. An HR officer has a reference letter rephrased in a public chatbot and pastes in the entire personnel note, including absences and a diagnosis. Health data thus reaches a provider without a contract and without a legal basis.
Debugging with credentials. A developer pastes a configuration file into a code assistant to find a bug. The file contains the key for a payment interface and the connection details of the customer database. The credentials have to be replaced the same day.
Transcription in a board meeting. After an update, the video conferencing system has switched on an AI summary. A confidential discussion about a planned acquisition is transcribed and stored with the provider before anyone notices the new feature.
Effective controls
The following ISO/IEC 42001 controls counter this pattern.
Prevention:
- A.2 Policies related to AI: The AI policy sets out which data may go into which systems and is aligned with the rules on information security and privacy.
- A.10 Third-party and customer relationships: Before approval, the provider is assessed: storage location, subcontractors, use of inputs, deletion. Responsibility for personal data is clearly divided between the organization and the provider.
- A.8 Information for interested parties: The system and its accompanying documents tell users which data may be entered and which may not.
Detection:
- A.9 Use of AI systems: The processes for responsible use include observing how systems are used. In addition, data leakage prevention under A.8.12 detects outflows to known AI services.
Response:
- A.9 Use of AI systems: A defined procedure sets out what happens after an inadmissible entry: reporting, deletion with the provider, replacement of credentials and the data protection assessment.
The AI Usage Policy covers the pattern in detail: Section 3.1 ties every approved service to the data classification it may process and requires enterprise versions. Section 4.2 prohibits confidential and personal data in unapproved tools and the entry of third-party credentials, and Section 4.3 governs the handling of inputs. Under Section 7, every suspicion of such an entry is reported. The classification itself follows the classification of information under A.5.12.
Link to the AI Act
For high-risk AI systems, entering unsuitable data is among the forms of misuse the provider evaluates under Article 9(2)(b) and names in the instructions for use (Article 13(3)(b)(iii), complemented by the input data specifications under point (vi)). The deployer uses the system in line with those instructions under Article 26(1) and relies on the provider’s information under Article 26(9) for its data protection impact assessment. Article 4 applies to all systems: providers and deployers support the AI literacy of their staff, and that includes knowing what happens to the inputs (→ EU AI Act).
The actual limits come from the GDPR: purpose limitation, data minimization and confidentiality under Article 5(1), a legal basis under Article 6 (with Article 9 in addition for special categories), a contract under Article 28, security under Article 32 and the rules on transfers to third countries under Articles 44 et seq.
Related patterns and threats
- T11 Attacks on confidential data: What has once entered the model can be extracted again through targeted queries.
- MU-15 Further training with inadmissible data: The continuation at training level: the data flows into adapting the model.
- MU-11 Passing on access credentials: Even an approved tool becomes a data leak when unauthorized people gain access.
- MU-04 Use by persons who were not briefed: People who were never told the rules cannot follow them.
Sources
- Regulation (EU) 2024/1689 (AI Act), Article 2(7), Article 3(13), Articles 4, 9, 13 and 26
- Regulation (EU) 2016/679 (GDPR), Articles 5, 6, 9, 28, 32, 33 and 44 et seq.
- ISO/IEC 42001:2023, Annex B.9.4, B.10.2 and B.10.3
- AI Usage Policy of the AI Compliance Starter Kit, Sections 3.1, 4.2, 4.3 and 7
- German Trade Secrets Act in the Cenedril wiki