OpenAI says models trained to make up answers

"The admission came in a paper [PDF] published in early September, titled "Why Language Models Hallucinate," and penned by three OpenAI researchers and Santosh Vempala, a distinguished professor of computer science at Georgia Institute of Technology. It concludes that "the majority of mainstream evaluations reward hallucinatory behavior." Language models are primarily evaluated using exams that penalize uncertainty. The fundamental problem is that AI models are trained to reward guesswork, rather than the correct answer."

"As a test case, the team tried to get an OpenAI bot to report the birthday of one of the paper's authors, OpenAI research scientist Adam Tauman Kalai. It produced three incorrect results because the trainers taught the engine to return an answer, rather than admit ignorance."

Mainstream evaluation methods for language models often reward confident answers and penalize uncertainty. Training incentives therefore favor returning an answer over admitting ignorance, producing plausible but incorrect responses. When pretraining corpora lack a learnable pattern for a fact, models tend to guess, increasing hallucinations. A test requesting a specific individual's birthday produced multiple incorrect answers because trainers favored answer generation rather than uncertainty. Pretraining embeds this behavior because many training examples present certain data; singleton or infrequent facts lack learnable patterns, so hallucination rates align with the fraction of facts appearing only once in training data.

Read at Theregister