COVID-19 Vaccine Fact-Checking Posts on Facebook: Observational Study (Preprint)

BACKGROUND Effective interventions aimed at correcting COVID-19 vaccine misinformation, known as fact-checking messages, are needed to combat the mounting antivaccine infodemic and alleviate vaccine hesitancy. OBJECTIVE This work investigates (1) the changes in the public's attitude toward COVID-19 vaccines over time, (2) the effectiveness of COVID-19 vaccine fact-checking information on social media engagement and attitude change, and (3) the emotional and linguistic features of the COVID-19 vaccine fact-checking information ecosystem. METHODS We collected a data set of 12,553 COVID-19 vaccine fact-checking Facebook posts and their associated comments (N=122,362) from January 2020 to March 2022 and conducted a series of natural language processing and statistical analyses to investigate trends in public attitude toward the vaccine in COVID-19 vaccine fact-checking posts and comments, and emotional and linguistic features of the COVID-19 fact-checking information ecosystem. RESULTS The percentage of fact-checking posts relative to all COVID-19 vaccine posts peaked in May 2020 and then steadily decreased as the pandemic progressed (<i>r</i>=–0.92, <i>df</i>=21, <i>t</i>=–10.94, 95% CI –0.97 to –0.82, <i>P</i>&lt;.001). The salience of COVID-19 vaccine entities was significantly lower in comments (mean 0.03, SD 0.03, <i>t</i>=39.28, <i>P</i>&lt;.001) than in posts (mean 0.09, SD 0.11). Third-party fact checkers have been playing a more important role in more fact-checking over time (<i>r</i>=0.63, <i>df</i>=25, <i>t</i>=4.06, 95% CI 0.33-0.82, <i>P</i>&lt;.001). COVID-19 vaccine fact-checking posts continued to be more analytical (<i>r</i>=0.81, <i>df</i>=25, <i>t</i>=6.88, 95% CI 0.62-0.91, <i>P</i>&lt;.001) and more confident (<i>r</i>=0.59, <i>df</i>=25, <i>t</i>=3.68, 95% CI 0.27-0.79, <i>P</i>=.001) over time. Although comments did not exhibit a significant increase in confidence over time, tentativeness in comments significantly decreased (<i>r</i>=–0.62, <i>df</i>=25, <i>t</i>=–3.94, 95% CI –0.81 to –0.31, <i>P</i>=.001). In addition, although hospitals receive less engagement than other information sources, the comments expressed more positive attitudinal valence in comments compared to other information sources (b=0.06, 95% CI 0.00-0.12, <i>t</i>=2.03, <i>P</i>=.04). CONCLUSIONS The percentage of fact-checking posts relative to all posts about the vaccine steadily decreased after May 2020. As the pandemic progressed, third-party fact checkers played a larger role in posting fact-checking COVID-19 vaccine posts. COVID-19 vaccine fact-checking posts continued to be more analytical and more confident over time, reflecting increased confidence in posts. Similarly, tentativeness in comments decreased; this likewise suggests that public uncertainty diminished over time. COVID-19 fact-checking vaccine posts from hospitals yielded more positive attitudes toward vaccination than other information sources. At the same time, hospitals received less engagement than other information sources. This suggests that hospitals should invest more in generating engaging public health campaigns on social media.