Chatbots are not Reliable News Sources

A study by international news media shows that chatbots like ChatGPT and Gemini regularly make mistakes when asked about current news facts.

Chatbots like ChatGPT are used for various purposes. Research earlier this year by Reuters and Oxford University shows that mainly younger people also use these tools to follow the news. However, AI chatbots prove to be anything but a reliable news source.

Led by the British public broadcaster BBC and the European Broadcasting Union (EBU), international news media surveyed the current knowledge of popular chatbots. 19 news media from Europe participated, including VRTNWS and RTBF on behalf of Belgium, as well as two Canadian and one American. The results were quite concerning: on almost half of the questions, the four chatbots questioned made some kind of mistake.

Gemini Performs the Worst

4 commonly used AI chatbots were put to the test: ChatGPT, Gemini, Perplexity and Microsoft Copilot. In 45 percent of the answers, the chatbots made some kind of error. The most common mistake was incorrect source citation (31%), followed by accuracy (20%) and insufficient context (14%). The chatbots either provided incorrect sources or did not accurately represent what was stated in those sources.

Gemini is the negative outlier in the study, with an error rate of 76 percent. The other three chatbots are more evenly matched, although even the ‘best performing’ chatbot Perplexity makes an error in 31 percent of cases. For ChatGPT and Copilot, this is 36 and 37 percent respectively. Since both chatbots are built on the same model, it’s not surprising that these percentages are almost identical.

Ongoing Facts

The study finds that chatbots are more likely to make mistakes when it comes to news facts that are still developing, such as wars. When a news fact is ‘completed’, the chatbots generally give more correct answers. Compared to an earlier study by the BBC at the beginning of this year, the chatbots also scored slightly better, as at that time more than half of the answers still contained errors.

Current knowledge was a major shortcoming in the first generations of generative AI chatbots, as they were trained on datasets that were several years behind current events. The companies behind the chatbots have tried to solve this by giving them access to the internet. However, there still appears to be much room for improvement in searching for and accurately representing current information. So it’s best to still search for news yourself rather than blindly trusting ChatGPT for this.