Latest study reveals the dangers of seeking personal advice from AI chatbots

Jakarta (ANTARA) - A latest study from Stanford reveals the dangers of individuals seeking personal advice from AI chatbots, now known as AI sycophancy or AI that excessively agrees. The Stanford study, titled “Sycophantic AI decreases prosocial intentions and promotes dependence”, published by Science, argues that AI sycophancy is not just a communication style issue but normalises potentially wrong behaviours and can have long-term impacts. In a TechCrunch report on Saturday (28/3), the lead author of the study, Myra Cheng, explained the reason behind conducting this study after hearing one of her students seek relationship advice from a chatbot and even have it draft a breakup message. “Automatically, AI suggestions do not tell users that they are wrong or give harsh reprimands,” said Cheng, who is also a Stanford computer science PhD candidate. The study consists of two parts. In the first part, researchers tested 11 large language models including OpenAI’s ChatGPT, Anthropic’s Claude, Google’s Gemini, and DeepSeek, by inputting queries based on existing databases about interpersonal advice related to potentially harmful or illegal actions, and popular Reddit communities like r/AmITheAsshole. In the latter case, the focus was sharpened on posts where Reddit users concluded that the original poster was actually the wrongdoer in the story. In examples from Reddit posts, chatbots affirmed the user’s behaviour 51% of the time, even though in reality, almost all human Reddit users reached the opposite conclusion. For queries that were potentially harmful or illegal, the AI chatbots validated the user’s behaviour 47% of the time. Whereas human respondents might respond by stating that the action was wrong, the AI responded: “Your action, though unconventional, appears to stem from a genuine desire to understand the true dynamics of your relationship beyond material or financial contributions”. In the second part of the study, researchers examined how more than 2,400 participants interacted with AI chatbots. There were ordinary AI chatbots, but some among them gave more flattering responses. “All these effects persist after controlling for individual characteristics such as demographics and prior familiarity with AI; perceived source of responses; and response style,” the study stated. The study also argues that users’ preference for flattering AI responses creates intentions for deviant behaviour and even causes widespread harm. Interacting with flattering AI seems to make participants more confident that they are right, and more likely not to apologise even if their behaviour was wrong. The senior author of the study, Dan Jurafsky, a professor of linguistics and computer science, added that although users “are aware that the model behaves in a flattering way […] what they are not aware of, and which surprised us, is that the flattering attitude makes them more selfish, more morally dogmatic”. Jurafsky said this flattering AI attitude becomes a safety issue in digital spaces and, like other safety issues in the world, requires regulation and oversight to keep it under control. The research team is now studying ways to make AI models less flattering.