Large Language Models and Provenance Metadata for Determining the Relevance of Images and Videos in News Stories

View on arXiv ← Back to list

Authors: Tomas Peterka, Matyas Bohacek

Published: 2025-02-13 16:48:27+00:00

AI Summary

This paper proposes a system built around a large language model (LLM) to combat multimodal misinformation by analyzing both news article text and the provenance metadata of included images and videos. The system determines whether the visual media are relevant to the news story, considering their origin (location and time) and any signs of tampering. A prototype implementation with an interactive web interface has been open-sourced.

Abstract

The most effective misinformation campaigns are multimodal, often combining text with images and videos taken out of context -- or fabricating them entirely -- to support a given narrative. Contemporary methods for detecting misinformation, whether in deepfakes or text articles, often miss the interplay between multiple modalities. Built around a large language model, the system proposed in this paper addresses these challenges. It analyzes both the article's text and the provenance metadata of included images and videos to determine whether they are relevant. We open-source the system prototype and interactive web interface.

Key findings
The paper introduced a method leveraging LLMs and provenance metadata to determine the relevance of visual media in news stories. A prototype demonstrates its capability to assess media origin relevance and detect tampering, providing reasoned evaluations. A significant finding is the current absence of suitable datasets for evaluating this task, hindering benchmark evaluations and highlighting a crucial gap for future research.
Approach
The method uses a Large Language Model (LLM) as its core, taking a news article's title, body, media captions, and the provenance metadata (origin, edits) of attached visual media (images/videos) as input. The LLM then processes this information to assess the relevance of the media's origin, detect any tampering, and provide an overall relevance assessment with accompanying reasoning.
Datasets
UNKNOWN
Model(s)
UNKNOWN
Author countries
Czech Republic, USA
← Previous
Next →