A new computational approach using large language models (LLMs) can measure how closely two texts share the same narrative, even when written in different languages. Developed by researchers at the Chinese University of Hong Kong and the University of Oregon, the method goes beyond traditional techniques like exact text reuse or topic modeling.
The proposed metric, called "narrative similarity," uses LLMs to distill texts into their core claims, then compares those claims rather than surface-level features such as words or sentences. In tests against standard alternatives, the LLM-based approach achieved significantly higher precision, recall, and F1 scores.
To validate the method, the team applied it to a case study tracking the spread of Russian claims about a Ukrainian bioweapons program across mainstream and fringe U.S. news websites. The approach successfully identified narratives that were conceptually identical but linguistically distinct.
The researchers note that narrative similarity can be applied to propaganda analysis, misinformation tracking, policy diffusion studies, and cultural object analysis. The work was presented by Prof. Hannah Waight from the University of Oregon's Department of Sociology.