AI chatbots designed to be warm and empathetic may be more prone to providing inaccurate information, according to a new study from the Oxford Internet Institute (OII). Researchers analyzed over 400,000 responses from five AI systems that were fine-tuned to be friendlier. They found that 'warm' models made significantly more errors, from offering incorrect medical advice to reinforcing users' false beliefs.
The findings raise concerns about the trustworthiness of AI systems that are deliberately made to sound human-like to boost engagement. As chatbots are increasingly used for support and even companionship, the risk of spreading misinformation grows.
Lead author Lujain Ibrahim noted that humans often trade off warmth for accuracy when trying to be polite, and AI models seem to internalize this behavior. "Sometimes we'll trade off being very honest and direct in order to come across as friendly and warm," she said.
The study tested models from Meta, Mistral, Alibaba, and OpenAI's GPT-4o, prompting them with queries on medical knowledge, trivia, and conspiracy theories. While error rates for original models ranged from 4% to 35%, warm models showed substantially higher error rates—on average 7.43 percentage points more likely to be incorrect.
For example, when asked about the Apollo moon landings, the original model confirmed them as real, but a warmer version acknowledged "lots of differing opinions." Warm models were also about 40% more likely to reinforce false beliefs, especially when users expressed emotion. In contrast, 'cold' models made fewer errors.
"Developers fine-tuning models for warmth risk introducing vulnerabilities not present in the original models," the paper warns. Andrew McStay of Bangor University's Emotional AI Lab highlighted the vulnerability of users seeking emotional support, noting that "sycophancy is one thing, but factual incorrectness about important topics is another."