Imagine a lawyer who confidently presents a brief in court full of detailed legal precedents, solemn citations, names of authoritative judges. Then he discovers that everything is false: the cases do not exist, the judges never uttered those words. The culprit? ChatGPT, who confabulated a parallel reality. This episode, which really happened in 2023, is not an exception. It is a symptom of a deeper problem: why does AI invent information? And above all, how should we define these errors without falling into misleading or stigmatising metaphors, such as ‘hallucination’?
The concept of hallucination in Artificial Intelligence models has taken centre stage in the discussion of the epistemic and structural limitations of such systems. The use of this terminology to describe the processing of information that does not correspond to a referential datum raises questions not only from a technical, but also from a conceptual point of view. The question is whether the term is an appropriate metaphor or whether, rather, a more precise vocabulary is needed to describe the phenomenon.
Creative hallucinations or systemic errors?
It all started with a misunderstanding. In 2000, researchers in the field of computer vision used the term ‘hallucination’ to describe a useful process: algorithms that reconstructed missing pixels in images, ‘imagining’ plausible details. It was a useful skill, not a flaw. But with the advent of large language models such as GPT in recent years, the meaning has rapidly changed.
Today, ‘hallucination’ evokes embarrassing mistakes: translations that distort the original meaning, academic articles with non-existent references, chatbots citing studies that were never published.
The problem, as Østergaard and Nielbo point out in a recent article in Schizophrenia Bulletin(2023), is that the comparison with human hallucinations is scientifically inaccurate and socially dangerous. Human hallucinations – hearing voices or seeing non-existent objects – are subjective perceptions linked to neurological or psychiatric disorders. AI, on the other hand, perceives nothing. Its errors derive from statistical calculations: patterns learnt from huge datasets, often contaminated by bias or outdated information. Calling them ‘hallucinations’ not only trivialises complex medical conditions, it obscures the true nature of the problem: the AI is not a psychiatric patient, it is a system that optimises linguistic probability. And to call these erroneous optimisations – which in essence are statistical approximations so large that they are sometimes totally off – hallucinations not only obscures the true nature of the problem, but also leads people to associate these tools with the fact that they are akin to thinking minds.
Is there a better term? Ars Technica journalist Benj Edwards proposes ‘confabulation’, borrowing it from neuropsychiatry. In Korsakov syndrome – a condition related to vitamin B1 deficiency – patients fill memory gaps with intricate and convincing narratives. An elderly person might describe a trip to China that never took place, inserting realistic details (names of cities, trains, colleagues): he would not be ‘lying’ in the strict sense: his brain would be seeking coherence, even at the cost of inventing.
AI actually does something similar. When ChatGPT generates a false citation, it does so because it is combining fragments of text learned during training-names of courts, legal formulae, syntactic structures-into a fluid but fictitious narrative. Like Korsakov’s patient, he has no intention of deceiving: he simply follows the statistical imperative to complete the prompt in the most plausible way. And, as in the clinic, the error only emerges later: when the lawyer discovers the non-existent cases, or the student realises that the bibliography of his paper is invented.
In short: to lie you have to know that you are doing it.
Bullshit!
But if ‘confabulation’ sounds too technical, there are those who invite us to call a spade a spade: bullshit. The philosopher Harry Frankfurt, in his essay On Bullshit (2005), distinguishes lies – an intentionally false statement – from bullshit, produced by those who are indifferent to the truth. A politician who promises unrealisable solutions in order to gain consensus, or a student who bluffs in an exam, are not necessarily lying: they are saying things that sound good, regardless of whether they correspond with the facts.
According to Hicks et al. (2024), ChatGPT is a perfect example of a bullshit generator: it generates statements that are smooth and persuasive, but often completely indifferent to the truth. The reason lies in the design of the language models themselves: they are optimised to produce human-like text, not to guarantee accuracy.
Increasing the ‘temperature’ of the model – a parameter that controls ‘creativity’ (i.e. the breadth in the choice of variables to be taken into account by the transformers) – results in more original answers, but also more prone to imaginative inventions. The result? A system that, as Frankfurt would say, “undermines the distinction between true and false”, normalising the idea that truth is negotiable. From this point of view, in fact, AI does not lie in the strict sense of the word (indeed, it does not even know what the epistemological difference between true and false is), but generates outputs based solely on syntactic coherence and statistical probability, without a semantic check on the adherence of the contents to empirical reality.
Examples abound. In 2023, a study published in Psychiatry Research revealed that ChatGPT invents 30% of bibliographic citations in psychiatry, inserting non-existent authors and paper titles. In the same year, is the case cited at the beginning of this article, a New York lawyer was sanctioned for having presented in court a brief full of court precedents generated by ChatGPT: such convincing forgeries that even the judge initially did not recognise them as such.
Behind these episodes are precise technical dynamics. Language models do not have access to the real world: they operate on a universe of words and statistics. If phrases such as ‘Einstein said…’ or ‘the Supreme Court ruled…’ appear in the training data , the model learns to use those names in plausible contexts, without checking their authenticity. It is a problem of grounding: for the AI, ‘Paris is the capital of France’ and ‘Paris is an apple on Mars’ are mere sequences of words, evaluated according to their frequency in texts, not their veracity.
Responsibility in language is needed for responsible AI
At a time when the attention paid to AI is at its highest by the press, decision-makers and politicians, it is crucial to avoid generating misunderstandings by using (deliberately?) misleading terms
The use of more accurate terminology is essential not only for a correct scientific understanding of the phenomenon, but also for the development of strategies to mitigate the inaccuracies produced by artificial intelligence models. Defining an inference error as a hallucination is not just an easy way to get a concept across. It risks being a system that insidiously changes the very nature of the concept in its essence, bringing it closer to what it is intended to pass.
It is difficult to convince the public that it is necessary to teach ethics to a large and complex calculator. It is much easier to do so if you turn that calculator into a mind so sensitive that it even hallucinates.
We don’t want to talk about inference error because it is too complex? Confabulation, with all its ambiguity, is a powerful metaphor: it reminds us that the AI is neither sick nor malicious, but a system that seeks statistical consistency at all costs, because it is the only truth in a system that – structurally – is unable to understand the concept of truth.
he most insidious risk is getting used to bullshit. If we normalise the idea that AI can ‘creatively make mistakes’, we risk eroding trust in knowledge itself. The challenge for experts is not just technical -improving algorithms- but cultural: redefining the way we talk about these mistakes, choosing words that respect both science and human dignity. Because, after all, the metaphors we use shape the way we understand – and govern – technology.
