Characterizing AI-Generated Misinformation on Social Media

View on arXiv ← Back to list

Authors: Chiara Drolsbach, Nicolas Pröllochs

Published: 2025-05-15 13:18:04+00:00

AI Summary

This study performs a large-scale empirical analysis of AI-generated misinformation on the social media platform X, examining 91,452 misleading posts flagged by X's Community Notes. The research reveals unique characteristics of AI-generated misinformation compared to conventional forms, focusing on content attributes, source accounts, virality, and believability.

Abstract

AI-generated misinformation (e.g., deepfakes) poses a growing threat to information integrity on social media. However, prior research has largely focused on its potential societal consequences rather than its real-world prevalence. In this study, we conduct a large-scale empirical analysis of AI-generated misinformation on the social media platform X. Specifically, we analyze a dataset comprising N=91,452 misleading posts, both AI-generated and non-AI-generated, that have been identified and flagged through X's Community Notes platform. Our analysis yields four main findings: (i) AI-generated misinformation is more often centered on entertaining content and tends to exhibit a more positive sentiment than conventional forms of misinformation, (ii) it is more likely to originate from smaller user accounts, (iii) despite this, it is significantly more likely to go viral, and (iv) it is slightly less believable and harmful compared to conventional misinformation. Altogether, our findings highlight the unique characteristics of AI-generated misinformation on social media. We discuss important implications for platforms and future research.

Key findings

AI-generated misinformation is more often centered on entertaining content with positive sentiment and originates from smaller accounts, yet it's significantly more viral than conventional misinformation. While slightly less believable and harmful, its disproportionate virality highlights its unique persuasive properties.

Approach

The researchers analyzed a dataset of 91,452 misleading posts from X's Community Notes, using an LLM to identify AI-generated content and annotate posts across various dimensions (sentiment, topic, harmfulness, believability). Negative binomial regression models were used to analyze virality, controlling for content and account characteristics.

Datasets

A dataset of N=91,452 misleading posts from X's Community Notes platform, collected between January 2023 and January 2025. A subset of 3000 posts was used for LLM-based annotation of content characteristics.

Model(s)

Large Language Model (LLM, specifically OpenAI Assistant based on gpt-4-turbo) for identifying AI-generated content and annotating post characteristics; Twitter-roBERTa-base model for sentiment analysis (on a subset of the data); Negative binomial regression models for analyzing virality.

Author countries

Germany

← Previous