According to recent research, online content moderators are susceptible to an “illusory truth effect,” which could lead them to trust misinformation they are supposed to guard against.
The collaborative efforts of researchers from MIT, Florida State University, and TaskUs, a global outsourcing firm, have led to findings with major implications for online platforms. Their investigation, which focused on content moderators in India and the Philippines, revealed that encountering a false headline twice could increase the likelihood of belief by 7.1%.
Concerns About An Overlooked Job in the Information Economy
With the explosion of Web 2.0 and user-generated content, problematic content became ever-present in the information ecosystem. The bulwark against this is an estimated hundreds of thousands of content moderators working around the globe.
While many online platform users know that humans are reviewing their material to ensure that it aligns with company policies, users aren’t unaware of who these moderators are. US companies in Silicon Valley primarily offshore this work to non-Western countries where educated workers are available for a fraction of the price.
Truth, Illusion, and Misinformation
The ‘illusory truth effect’ is a well-documented cognitive bias where an individual becomes increasingly likely to believe information the more it is repeated, regardless of its accuracy. This means that even false information, when encountered multiple times, can start to feel true.
Despite their crucial roles as arbiters of truth on a global scale, content moderators are, at their core, human beings. Moderators are susceptible to the same cognitive biases as anyone else, and their workflow is entirely comprised of questionable flagged content, leaving them inundated with misinformation. Additionally, moderators’ primary concern is not necessarily if the information is accurate but if it violates guidelines, which may be more concerned with legal issues or harassment than the truth of a claim.
MIT and Cornell researcher Hause Lin, who specializes in online misinformation, is the lead author of the new paper investigating the impact of illusory truth on content moderators. Additionally, Lin’s team was interested in whether culture could play a role. The Philippines and India are more collectivized cultures compared to the individualism of the United States. Collectivized culture studies emphasize shared meaning, group membership, and collective goals.
Lacking sufficient data on how an illusory truth may play across differing cultures, the MIT and Cornell collaboration also set out to determine if the effect would replicate itself in a non-Western environment. Do these predispositions leave the moderators more culturally susceptible to the illusory truth effect?
Watching the Watchmen
The researchers identified TaskUs, an international corporation handling global outsourcing for technology companies, as a major player in content moderation. Lin worked with the organization on the study to identify how susceptible the moderators were to fall for misinformation. TaskUs’s involvement provided access to a large pool of real-world content moderators, allowing the researchers to investigate the illusory truth effect comprehensively.
Lin’s team began with a baseline experiment to see if they could measure the illusory truth effect among TaskUs content moderators. They started by asking the TaskUs employees to rate a group of headlines for interestingness, gave them a questionnaire as a distraction exercise, and finally showed them a group of 48 headlines to rate for truthfulness. The second group repeated the original 16 headlines, adding 32 new ones. Across the board, respondents from both countries, regardless of factors such as education or religious belief, were much more likely to rate false headlines as true if they appeared twice.
Creating an Accuracy Mindset
Lin’s team then cast a wider net to samples of the Indian and Philippine general public that were roughly five times the size of the 199-subject TaskUs employee sample. In the general public, they received the same results. The researchers then moved to test a possible solution, setting a so-called “accuracy mindset.” This mindset involves asking subjects to consider the accuracy of a headline instead of its interestingness in the first round, thereby encouraging subjects to focus on the truthfulness of the information in their first encounter with it; an approach had never been field-tested outside of the West.
Determining accuracy the first time a subject encountered a headline proved to be a major bulwark against the illusion of truth, with the effect even being reversed in some subjects. Repeated headlines became much more likely to be judged correctly.
Lin brought the treatment back to TaskUs for a second round with their employees, leading to the same results as his accuracy mindset testing in the general public. The authors note in the study that the illusion of truthfulness effect has significant implications for the continued erosion of accuracy, especially when algorithms are trained on increasingly poor human decisions.
Their simple solution—to set an accuracy-first mindset instead of just broadly looking at policy violations—shows promise in guarding against a potentially harmful spiral of ever-increasing misinformation.
The paper “Accuracy Prompts Protect Professional Content Moderators from the Illusory Truth Effect” appeared in the November 2024 issue of PNAS Nexus.
Ryan Whalen covers science and technology for The Debrief. He holds a BA in History and a Master of Library and Information Science with a certificate in Data Science. He can be contacted at ryan@thedebrief.org, and follow him on Twitter @mdntwvlf.