Resistance to Online Misinformation

The Roozenbeek, Maertens, McClanahan, and van der Linden (2020) paper in Educational and Psychological Measurement is a methodologically focused stress-test of the gamified misinformation-inoculation paradigm developed in the same group’s earlier 2019 Palgrave Communications work. Where the 2019 paper established that the Bad News game produces measurable improvement in players’ ability to identify manipulation techniques, the 2020 paper asked whether that improvement is real or whether it is an artifact of two specific methodological threats: item effects (the possibility that observed gains depend on which test items were used) and testing effects (the possibility that being pre-tested causes part of the post-test improvement, regardless of intervention). The answer, after two experiments totaling N = 2,159, was that item effects are real and need to be managed, while testing effects are not detectable. The intervention’s effect on misinformation discernment survives both checks.

Why inoculation theory needed empirical grounding for the digital era

The conceptual foundation of the Bad News paradigm goes back to William McGuire’s 1961 work on resistance to persuasion. McGuire and colleagues argued that exposing someone to a weakened version of a persuasive attack—an “inoculation dose”—activates resistance mechanisms in the same way a vaccine primes immune defense. Their experimental work in the early 1960s showed that participants exposed to weakened pro-attack arguments and refutations were subsequently more resistant to full-strength persuasive attacks than participants who received only supportive defense of the original belief.

For sixty years, inoculation theory was studied primarily in the context of attitudes and beliefs about specific topics—health behaviors, political positions, brand loyalty—and its application to misinformation tended to be issue-specific (inoculating people against, say, climate-change denial messaging). Roozenbeek and van der Linden’s (2019) gamified Bad News intervention generalized the approach in two ways. First, it shifted the inoculation target from specific misinformation content to the techniques that misinformation routinely deploys, on the hypothesis that technique-recognition transfers across content domains. Second, it converted the inoculation dose from a brief refutational message into an interactive simulation in which participants take on the role of a fake-news producer and learn manipulation techniques by using them.

The six techniques the game targets, sometimes given the acronym DEPICT, are Discrediting, Emotion, Polarization, Impersonation, Conspiracy, and Trolling. Each is encountered in the simulation as a tool the player uses to grow a fictional fake-news account; post-game testing measures whether players have become better at detecting these techniques in actual misinformation stimuli they did not encounter during gameplay.

Why item effects and testing effects matter

The original 2019 Bad News evaluation used a within-subject pretest-posttest design: participants rated the manipulativeness of misinformation headlines before and after playing, and the pretest-to-posttest change was the effect estimate. This design, while practical, leaves two methodological flanks open.

Item effects arise when the specific test items used vary in difficulty, discrimination, or technique-specificity in ways that interact with intervention. If a small number of headlines drive most of the observed gain, the apparent intervention effect could be a property of those particular headlines rather than of misinformation discernment in general. Roozenbeek et al. (2020) operationalized this concern as the question of whether intervention effect estimates are stable across different random samples of test items.

Testing effects arise when the pretest itself contributes to post-test performance, independently of the intervention. Cognitive psychologists have known since the 1960s that practicing a test improves subsequent performance on a similar test, even with no instruction in between. If a meaningful portion of the Bad News intervention effect were really a testing effect, removing the pretest would reduce the effect; the intervention would be less informative than the original evaluation suggested.

The Solomon three-group design exists precisely to separate these confounds. Group 1 receives pretest, treatment, and posttest (the standard design). Group 2 receives treatment and posttest, no pretest. Group 3 receives only the posttest (control). Comparing Groups 1 and 2 isolates the testing effect; comparing Groups 2 and 3 isolates the treatment effect free of pretest contamination. The Solomon design is methodologically demanding because it requires roughly three times the sample size of a simple pretest-posttest study, but it gives clean answers.

The two experiments

Roozenbeek et al. (2020) ran two experiments addressing the two threats separately.

Experiment 1 (Item Effects, N = 480) manipulated which subset of misinformation headlines participants saw during pretest and posttest. By drawing from a larger item pool and varying which items each participant rated, the analysis could decompose intervention-attributable variance from item-attributable variance using a multilevel structure. The result: item effects are present and non-trivial. Specific headlines vary in how strongly they discriminate between participants who have versus have not been inoculated, and the standard practice of using a fixed small set of test items can over-estimate or under-estimate true intervention effects depending on which items are included.

Experiment 2 (Testing Effects, N = 1,679) implemented the Solomon three-group design. Across the three groups, the comparison of primary interest—did Group 1’s apparent gain (pretest-to-posttest difference) include a testing-effect component?—was tested by comparing Group 2’s posttest performance (no pretest) to Group 1’s posttest performance (with pretest). The result: no detectable testing effect. Group 2 participants who received treatment without pretest showed posttest performance equivalent to Group 1 participants who received treatment after pretest. The intervention effect is real, not an artifact of practicing the test.

The combined inference from the two experiments is that the Bad News inoculation paradigm produces genuine improvements in technique-recognition that are not driven by mere test exposure, but that the magnitude of those improvements is sensitive to which specific items are used to measure them. Future evaluations should use large item pools, randomize item assignment across participants, and include item-level variance components in the statistical models.

What this implies for the broader inoculation literature

The Roozenbeek et al. (2020) paper functions as a methodological corrective to a literature that had grown substantially between 2019 and 2021. By the time of the EPM publication, the Bad News game had been deployed in several languages, replicated cross-culturally, and adapted into related interventions (Go Viral!, Harmony Square). The 2022 review by Traberg, Roozenbeek, and van der Linden in the Annals of the American Academy of Political and Social Science catalogs this expansion and identifies methodological refinement as the central challenge for the next phase of work.

The 2020 EPM paper makes three contributions to that refinement. First, it provides the empirical basis for declaring the testing-effect threat resolved: subsequent gamified-inoculation studies need not run Solomon designs to defend against this confound, because the result generalizes. Second, it demonstrates that item effects are real and require active management—the effect of “the Bad News game” is not a single quantity but a quantity that depends on the test items used to estimate it, and reproducibility across studies requires harmonization of the measurement instrument. Third, it sets a methodological precedent for the field by showing that gamified-intervention research is amenable to the same psychometric scrutiny as more traditional educational interventions; inoculation research no longer has the excuse of methodological novelty for skipping these checks.

What the paradigm cannot yet establish

Two limitations the 2020 paper does not resolve, and that the 2022 Traberg review explicitly flags as open questions, deserve mention.

Durability of the inoculation effect. Roozenbeek et al.’s experiments measured posttest performance immediately after gameplay. The vaccination metaphor implies that resistance should persist over time, but the empirical work on this is sparse. Subsequent studies have shown decay over weeks to months without booster exposure; whether the decay function is comparable to that of traditional inoculation interventions, and whether periodic gameplay refreshers can sustain resistance, remains under active investigation.

Real-world transfer. The test items used in the EPM evaluation are misinformation headlines selected for their illustration of specific manipulation techniques. Real-world misinformation exposure in social-media feeds is not so cleanly classified, often combines techniques, and is encountered with attention budgets and emotional contexts that laboratory tasks cannot replicate. Whether the technique-recognition gains demonstrated in the Bad News studies translate to behavioral resistance in the wild—reduced sharing of misinformation, increased skepticism in real feeds—is a separate question that observational and field-experimental designs are beginning to address.

Position in the methodological evolution of inoculation research

McGuire’s (1961) original inoculation experiments asked whether weakened persuasive doses confer resistance to subsequent attack on specific beliefs. Sixty years of follow-up work generalized the framework to attitudes, behaviors, brand preferences, and—since 2017—to misinformation susceptibility. The Bad News game extends inoculation theory in three ways: from belief-specific to technique-general; from passive refutational reading to active gamified production; and from in-person to scalable online deployment.

The Roozenbeek et al. (2020) paper is best understood as the methodological consolidation step in this trajectory. It does not produce a new intervention; it interrogates the methods by which existing interventions are evaluated. By showing that one major confound (testing effects) is empirically absent and that another (item effects) requires active management, the paper allows subsequent inoculation research to focus on the substantive questions—technique generalization, durability, real-world behavioral transfer—rather than re-litigating internal-validity concerns.

Cross-cultural validation and replication evidence

Two parallel lines of evidence speak to whether the gamified-inoculation effect generalizes beyond the U.S./U.K. samples in the foundational Roozenbeek studies.

The cross-cultural case: Roozenbeek, van der Linden, and Nygren (2020). A multi-language deployment of Bad News in the Harvard Kennedy School Misinformation Review tested the intervention in German, Greek, Polish, and Swedish translations with over 5,000 participants. The result: significant and meaningful reductions in the perceived reliability of manipulative content across all four languages, with demographic factors—age, education, gender, political ideology—not substantially diminishing the inoculation effect. The authors framed this as evidence for a “broad-spectrum vaccine” against misinformation that operates similarly across European cultural contexts.

Confidence and cognitive immunity: Basol, Roozenbeek, and van der Linden (2020). A complementary Journal of Cognition paper examined two dependent variables that the original 2019 evaluation underemphasized: confidence in spotting manipulation and discrimination between manipulative and non-manipulative content. Bad News players showed gains on both, with the discrimination improvement holding up after the typical experimental controls and with confidence gains correlated with—rather than dissociated from—accuracy. This addresses a recurring concern about prebunking interventions: that they might increase suspicion of all content rather than improving discrimination. The 2020 evidence indicates the gains are properly calibrated to manipulation rather than producing generalized skepticism.

The replication-failure caveat: Seabrooke, Modirrousta-Galian, and Higham (2025). A re-examination in Psychonomic Bulletin & Review of an earlier Indian-sample study that had reported improved discrimination of true versus fake Indian news headlines did not replicate the original effect. After counterbalancing which headlines appeared in pretest versus posttest phases, no significant pre-to-post differences in discrimination or response bias remained. Seabrooke et al. argue that the prior Indian-sample finding likely reflected experimental design artifacts rather than a genuine inoculation effect, and that demonstrating Bad News transfer to non-Western information ecosystems with locally relevant fake-news content is more difficult than the cross-language European studies suggested. This is the kind of negative finding that the field needs more of: it does not invalidate the European cross-cultural evidence, but it tightens the boundary conditions on what “the Bad News game works” can be claimed to mean in heterogeneous global contexts.

The Solomon design is rarely run after the testing-effect question has been settled; the 2020 EPM paper effectively settled it for this paradigm, and subsequent work has shifted toward the harder questions—durability, real-world behavioral transfer, cross-cultural applicability—that the methodological consolidation enables.

Frequently asked questions

What is psychological inoculation against misinformation?

Psychological inoculation, originally developed by McGuire (1961), exposes people to a weakened version of a persuasive attack to activate resistance mechanisms—analogous to a vaccine priming the immune system. Applied to misinformation, the approach has shifted from inoculating against specific false beliefs to inoculating against the manipulation techniques that misinformation routinely uses, on the hypothesis that technique-recognition transfers across content domains.

What is the Bad News game?

The Bad News game is a gamified inoculation intervention developed by Roozenbeek and van der Linden (2019). Players take on the role of a fake-news producer and learn six manipulation techniques—Discrediting, Emotion, Polarization, Impersonation, Conspiracy, and Trolling (DEPICT)—by using them in the simulation. Post-game testing measures whether players have become better at identifying these techniques in misinformation stimuli they did not encounter during gameplay.

What are item effects and testing effects?

Item effects arise when the specific test items used vary in difficulty, discrimination, or technique-specificity in ways that interact with intervention—if a small number of headlines drive most of the observed gain, the apparent effect could be a property of those headlines rather than of misinformation discernment. Testing effects arise when taking a pretest itself contributes to post-test performance, independently of the intervention.

What is the Solomon three-group design?

The Solomon three-group design separates intervention effects from testing effects. Group 1 receives pretest, treatment, and posttest. Group 2 receives treatment and posttest, no pretest. Group 3 receives only the posttest. Comparing Groups 1 and 2 isolates the testing effect; comparing Groups 2 and 3 isolates the treatment effect free of pretest contamination.

What did Roozenbeek et al. (2020) find?

Across two experiments totaling N = 2,159, item effects were present and non-trivial—specific headlines varied in how strongly they discriminated between inoculated and uninoculated participants—while testing effects were not detectable. Group 2 participants who received treatment without a pretest showed posttest performance equivalent to Group 1 participants who were pretested. The intervention effect is real and not an artifact of practicing the test.

How durable is misinformation inoculation?

Roozenbeek et al. measured posttest performance immediately after gameplay; the durability of the effect over weeks to months is not the focus of that paper. Subsequent studies have shown decay over time without booster exposure. Whether periodic gameplay refreshers can sustain resistance, and whether technique-recognition gains transfer to behavioral resistance in real-world social-media feeds, are open empirical questions flagged by Traberg et al. (2022).