The word suggests a malfunction, which is misleading. The model is doing exactly what it always does — producing text that fits the pattern of a good answer. When it knows the material, the fitting text is also true. When it does not, the fitting text is a fabrication, and nothing in the process flags the difference, including to the model itself.
This is why confidence carries no information and why the fix is architectural rather than a matter of asking the model to be careful. Give it the source material and require it to cite what it used, constrain output to a schema where the shape can be checked, verify anything that can be verified cheaply, and put a person in front of anything expensive to get wrong.
Related terms
Large language model (LLM)
A model trained on enormous amounts of text to predict what comes next, which turns out to be enough to write, summarise, translate and reason through many tasks.
RAG (retrieval-augmented generation)
Looking up relevant documents first and putting them in the prompt, so the model answers from your material instead of from memory.
Evaluation (evals)
A repeatable test set that measures whether a change to an AI system made it better or worse, rather than just different.
The bench this belongs to
AI automationThe repetitive half of your week, handed to software that does not get bored. Inbox triage, follow-ups, reporting, data entry between tools that were never meant to talk.
