google-site-verification: googlec7193c3de77668c9.html

ChatGPT’s search results for news are ‘unpredictable’ and frequently inaccurate

A rendition of OpenAI’s logo, which looks like a stylized whirlpool.
Illustration: The Verge

Based on testing done by Columbia’s Tow Center for Digital Journalism researchers, OpenAI’s ChatGPT search tool has some issues when it comes to responding with the truth.

OpenAI launched the tool for subscribers in October, saying it could give “fast, timely answers with links to relevant web sources.” Instead, Futurism points out that the researchers said ChatGPT search struggled to correctly identify quotes from articles, even when they came from publishers with arrangements to share data with OpenAI.

The authors asked ChatGPT to identify the source of “two hundred quotes from twenty publications.” Forty of those quotes were taken from publishers who’d disallowed OpenAI’s search crawler from accessing their site. Yet, the chatbot confidently replied with false information anyway, rarely admitting it was unsure about the details it gave:

In total, ChatGPT returned partially or entirely incorrect responses on a hundred and fifty-three occasions, though it only acknowledged an inability to accurately respond to a query seven times. Only in those seven outputs did the chatbot use qualifying words and phrases like “appears,” “it’s possible,” or “might,” or statements like “I couldn’t locate the exact article.”

A chart showing how often ChatGPT answered confidently or was unsure, with a breakdown of how often its confident replies were “Wrong,” (89) “Partially Correct,” (57) and “Correct” (47).
Image: Columbia Journalism Review
ChatGPT was fully or partially wrong more than right, but almost always confidently so.

The Tow Center test’s authors documented ChatGPT search results that misattributed a letter-to-the-editor quote from the Orlando Sentinel to a story published in Time. In another example, when asked to identify the source of a quote from a New York Times article about endangered whales, it returned a link to a different website that had wholly plagiarized the story.

“Misattribution is hard to address without the data and methodology that the Tow Center withheld,” OpenAI told the Columbia Journalism Review, “and the study represents an atypical test of our product.” The company went on to promise to “keep enhancing search results.”

Advertisements

See also  Humanoid robots bust dance moves alongside humans

Check Also

The Results Are In: CNET Readers Trust These Mobile Carriers More Than the Major Networks

If you plan to upgrade your phone during upcoming early holiday sales, now is the …

Sick of Paying for Slow Unlimited Data? Readers Recommend This Mobile Carrier

When’s the last time you looked at what you’re paying for in your phone bill? …

Gotta Go Fast and Hunt Demons This October in These Free PlayStation Plus Games

October is a month filled with longer nights, spooky ghost stories and plenty of ghouls …

Leave a Reply

Available for Amazon Prime