Hallucination
Also called confabulation, AI hallucination
A hallucination is output from a generative AI model that is false or unsupported but presented as fact. Examples include invented regulations, wrong citations or made-up customer details. It happens because language models predict likely text rather than check facts, so fluent answers can still be wrong.
How it works
Compliance teams use generative AI to summarise rules, draft reports and review alerts. A model may cite a rule number that does not exist or misstate a deadline. Firms reduce this by grounding answers in approved documents, requiring citations, limiting the model to set tasks and keeping a human reviewer before any output is relied on.
A hallucinated fact in a suspicious activity report, client letter or regulatory return can mislead supervisors or customers. It can also breach record-keeping and consumer protection duties. Model risk teams now test hallucination rates before deployment and monitor them in use. NIST lists confabulation as one of twelve risks specific to generative AI.
The rules
Lists confabulation among twelve generative AI risks and sets suggested actions to measure and manage it.
Organisations using generative AI, voluntaryRequires high-risk AI systems to reach appropriate levels of accuracy and to declare accuracy metrics in instructions for use.
Providers of Annex III high-risk AI systemsCommon questions
Why do language models hallucinate?
Large language models generate the most likely next words based on patterns in training data. They do not check facts against a source by default. When data is thin or the question is unusual, they can produce confident but false text.
Can hallucinations be removed completely?
No current technique removes them entirely. Grounding answers in trusted documents, narrowing tasks and adding human review lower the rate. Firms measure the remaining error rate and decide where the model can be used.
Are hallucinations a regulatory issue?
Yes, when outputs feed regulated activity. A false statement in a customer communication or regulatory filing is the firm's responsibility, whether or not a model wrote it. Supervisors expect firms to manage this through model risk and governance frameworks.
What is grounding?
Grounding links a model's answer to specific source material, often through retrieval-augmented generation. The model pulls relevant passages from an approved library and answers from them. Reviewers can then check each claim against its cited source.
