AI Tools Struggle to Detect Own Fabricated Images Amid Scandals

The limitations of artificial intelligence in image verification were sharply highlighted when an AI-powered chatbot failed to recognize a fabricated photograph of former Philippine lawmaker Elizaldy Co. This image, which falsely depicted Co in Portugal amid a corruption scandal, was created by the same generative model that the chatbot used to assess its authenticity. This incident raises significant concerns regarding AI tools’ ability to debunk misinformation, particularly as many technology companies reduce human oversight in fact-checking processes.

As internet users increasingly rely on AI chatbots for real-time image verification, they often encounter inaccuracies. In this case, the AI incorrectly confirmed the fabricated image as genuine. The Agence France-Presse (AFP) traced the image back to a middle-aged web developer in the Philippines who admitted to creating it using Nano Banana, an AI image generator. He expressed regret over the number of people who believed the misinformation, stating, “I edited my post — and added ‘AI generated’ to stop the spread — because I was shocked at how many shares it got.”

AI Models Face Verification Challenges

The failure of AI chatbots to accurately identify manipulated images has been documented in various instances. During protests in Pakistan-administered Kashmir, social media users circulated another fabricated image, purportedly showing demonstrators with flags and torches. An analysis by AFP revealed that this image was also generated with Google’s Gemini AI. Yet, both Gemini and Microsoft’s Copilot misidentified it as authentic.

“These models are trained primarily on language patterns and lack the specialized visual understanding needed to accurately identify AI-generated or manipulated imagery,”

stated Alon Yamin, CEO of the AI content detection platform Copyleaks. This sentiment echoes the findings of a study conducted earlier this year by Columbia University’s Tow Center for Digital Journalism, which tested seven AI chatbots, including ChatGPT and Grok. All models failed to correctly identify the provenance of ten images taken by professional photojournalists.

The inability of AI tools to discern authenticity stems from their programming, according to Rossine Fallorina from the nonprofit Sigla Research Center. “In a sense, they can only generate things to resemble. They cannot ascertain whether the resemblance is actually distinguishable from reality,” she explained.

Implications for Online Misinformation

The prevalence of AI-generated images on social media platforms poses a growing challenge for information verification. Surveys indicate that users are increasingly shifting from traditional search engines to AI tools for gathering and verifying information. This shift occurs concurrently with Meta announcing the discontinuation of its third-party fact-checking program in the United States, placing the responsibility of debunking misinformation onto ordinary users through a model known as “Community Notes.”

The landscape of fact-checking is complicated by accusations of bias against professional fact-checkers, particularly in politically polarized societies. AFP currently collaborates with Meta’s fact-checking program in 26 languages, covering regions including Asia, Latin America, and the European Union.

While AI models can assist professional fact-checkers by rapidly locating images and identifying visual clues, experts caution against relying solely on these tools. “We can’t rely on AI tools to combat AI in the long run,” Fallorina stated, emphasizing the continued need for trained human oversight in verifying information.

The challenges posed by AI-generated misinformation underscore the importance of developing more reliable verification technologies. As society continues to grapple with the implications of AI in everyday life, ensuring accurate information remains paramount.