To observe the observations on observational research: how do Italian online media report about a contemporary mainstream observational study?

Raffaele Rasoini1, Fabio Ambrosino1,2, Rebecca De Fiore1,2, Camilla Alderighi1

1Associazione Alessandro Liberati - Cochrane Affiliate Centre, Lauria (Potenza); 2Il Pensiero Scientifico Editore, Roma.

Received on July 17, 2022. Accepted on August 31, 2022.

Summary. Background. The media’s key role in conveying health information to the public is not always supported by the quality of the reporting. Despite findings from observational studies (OSs) represent a substantial proportion of media health news, limitations of OSs are often overlooked in medical journals’ abstracts, in press releases, and in associated news stories. The objective of this analysis is to investigate how Italian online news media report on a contemporary OS published in a major medical journal and dealing with a topic of widespread interest. Methods. The OS was published in Nature Medicine (Nat Med) in February 2022. It is a large retrospective cohort study aimed at characterizing the post-acute cardiovascular manifestations of covid-19. We collected Italian online news articles covering the Nat Med study that were released in the first two weeks after study publication. Based on resources focused on evaluation and proper reporting of OSs, we identified five thematic categories to be employed as a minimal reference standard to address the quality of reporting of the Nat Med study. Namely: 1) causality, 2) fear mongering, 3) spin, 4) actionability and 5) critical evaluation. Then, we defined a 13-item checklist aimed at exploring the existence of issues within each of the online news article with regard to the five thematic categories above. Outcome was the percentage of news articles covering the NM study showing issues with each of the five thematic categories. Results. After checking for inclusion and exclusion criteria, we collected 30 news articles. Global inter-rater agreement related to the checklist completion by 4 raters was substantial. An issue with causality was identified in 30 articles out of 30 (100%). An issue with fear mongering was identified in 25 (83.3%) of the 30 articles, and an issue with spin in 21 (75%) of the 28 articles. Furthermore, an issue with actionability and critical evaluation was identified in 16 (53.3%) and 26 (86.7%) of the 30 articles, respectively. Conclusions. Our analysis of Italian online news media reporting about a contemporary OS published in a major medical journal and dealing with a topic of high public interest has shown that most news articles fail to properly report on the study’s findings.

Key words. Observational research, online media.

Osservare le osservazioni sulla ricerca osservazionale: come i media online italiani hanno riportato un recente studio osservazionale mainstream.

Riassunto. Introduzione. Il ruolo importante dei media nel veicolare al pubblico i risultati della ricerca sulla salute non è sempre supportato dalla qualità delle notizie. Nonostante i risultati di studi osservazionali (SO) rappresentino una parte sostanziale delle notizie sulla salute riportate dai media, i limiti degli SO risultano spesso sottovalutati negli abstract degli articoli pubblicati su riviste scientifiche, nei comunicati stampa e nelle notizie associate. L’obiettivo di questa analisi è di valutare come i media online italiani hanno riportato i risultati di un recente SO pubblicato su un’importante rivista medica e focalizzato su un tema di ampio interesse. Metodi. Lo SO oggetto di questa analisi è stato pubblicato su Nature Medicine (Nat Med) nel febbraio 2022. Si tratta di un ampio studio retrospettivo di coorte il cui obiettivo era quello di caratterizzare le manifestazioni cardiovascolari post acute di covid-19. Abbiamo raccolto gli articoli dei media italiani online che riportavano i risultati dello studio di Nat Med e che erano stati pubblicati nelle prime due settimane dopo la pubblicazione dello studio. Sulla base di risorse focalizzate sulla valutazione critica e sulla comunicazione appropriata della ricerca osservazionale, abbiamo identificato cinque categorie tematiche da impiegare come standard minimo di riferimento per valutare la qualità della comunicazione dello SO in esame: 1) relazione causale, 2) allarmismo, 3) spin, 4) azionabilità and 5) valutazione critica. Successivamente, abbiamo definito una checklist di 13 item, con l’obiettivo di esplorare all’interno di ciascun articolo online l’esistenza di criticità relative alle cinque categorie tematiche suddette. Per ognuna delle cinque categorie tematiche, abbiamo quindi calcolato la percentuale di articoli online con criticità. Risultati. Sulla base dei criteri di inclusione ed esclusione predefiniti, abbiamo raccolto 30 articoli pubblicati sui media online. L’accordo inter-osservatore relativo al completamento della checklist da parte dei 4 valutatori è stato sostanziale. Criticità con la relazione causale sono state identificate in 30 articoli su 30 (100%). Criticità con l’allarmismo sono state identificate in 25 articoli su 30 (83,3%) e con lo spin in 21 articoli su 28 (75%). Inoltre, criticità con l’azionabilità e la valutazione critica sono state identificate rispettivamente in 16 (53,3%) e in 26 (86,7%) su 30 articoli. Conclusioni. Questa analisi sulla comunicazione da parte dei media italiani online di uno SO contemporaneo, pubblicato su un’importante rivista medica e focalizzato su un argomento di interesse generale, ha mostrato che la maggior parte degli articoli ha riportato in modo non appropriato i risultati dello studio.

Parole chiave. Media online, ricerca osservazionale.

Introduction

The media play a central role in informing and educating the public about science and medicine. By selecting which health studies to cover and how to report on them, the media can exert influence on individual behaviour and public health, the health services utilization, and the transmission of medical knowledge to the scientific community1.

Unfortunately, the media’s key role in conveying health information to the public is not always supported by the quality of the reporting. Recently, the first systematic review and meta-analysis was conducted on the quality of news reports on the effects of health interventions2. This analysis exposed how numerous reports conveyed an unbalanced and oversimplified portrait of the potential consequences of these interventions.

Consistently, over the past two decades, some remarkable initiatives aimed at monitoring the quality of health news reporting – the Australian Media Doctor3, the Canadian Media Doctor4 and the US Health News Review5 – observed a general poor medical news reporting on health studies, regarding both randomized controlled trials (RCTs) and observational studies (OSs).

By allowing causal inferences between treatments and effects, RCTs play a stronger role than OSs in addressing decisions on health interventions and thus -one would expect- in catching news titles. Yet, findings from OSs represent a substantial proportion of media health news. For example, when compared to high-impact medical journals, newspapers have been shown to preferentially cover OSs over RCTs6. Coherently, press releases from high-impact medical journals were found to cover OSs more frequently than RCTs7.

Inherent methodological problems with OSs limit the ability to infer causal relationships between variables. Still, limitations of observational research are often overlooked in abstracts published in medical journals, in press releases and in associated news stories8.

Moreover, news media reports on health studies can misrepresent the studies’ results, either by leveraging spin and fear mongering or by suggesting the need to undertake corrective actions that are solely based on observational findings9,10. Throughout the past decade, a few studies examined the quality of health news in Italian print media11,12. These studies’ results raised concerns regarding the reliability of a significant proportion of Italian health news. The past ten years have also seen a sharp decline in newspaper readership in contrast to the significant increase in Italian online news audiences.

The objective of this analysis is to investigate the quality of Italian online news media reporting on a contemporary OS published in a major medical journal and dealing with a topic of widespread concern and interest.

Methods

The OS, titled “Long-term cardiovascular outcomes of covid-19,” was published in Nature Medicine (Nat Med) on the 7th of February 202213. This is a retrospective cohort study aimed at characterising the post-acute cardiovascular manifestations of covid-19 by comparing 153,760 individuals with a covid-19 diagnosis against two control cohorts of 5,637,647 (contemporary controls) and 5,859,411 (historical controls) individuals without a covid-19 diagnosis. Participants’ data were extracted from the US Department of Veteran Affairs’ national healthcare databases.

The study found that, following the first 30 days and within 12 months of infection, individuals with previous covid-19 were observed to have a higher cardiovascular disease incidence compared to the control cohorts. Conditions examined included cerebrovascular disease, dysrhythmias, ischaemic and non-ischaemic heart disease, pericarditis, myocarditis, heart failure and thromboembolic disease. The cardiovascular disease incidence was found to increase progressively among non-hospitalised and hospitalised covid-19 patients and among those admitted to intensive care due to covid-19 complications.

This study gained early widespread media and social media attention: two weeks after publication, the study reached an Altimetric score of 14,082, which has since grown.

Online news search

We initially performed a Google web search. The search was limited to results in the Italian language, encompassing data from February 7th to February 21st and including the following key word combinations: “covid-19” AND “nature medicine” AND “malattie cardiovascolari” [cardiovascular disease]; “covid-19” AND “nature medicine” AND “infarto miocardico” [myocardial infarction]; “covid-19” AND “nature medicine” AND “ictus” [stroke]; “covid-19” AND “nature medicine” AND “scompenso cardiaco” [heart failure].

We also searched directly the major Italian online news outlets, the news outlet list in the Nat Med article Altmetrics page, and finally we searched through secondary sources (e.g., online articles that were mentioned by other online articles).

The web search process was carried out by two independent authors.

Article selection

We included online news articles that covered the Nat Med study as their main subject, which were written in Italian and were published between the 7th and the 21st of February 2022. These dates cover the first two weeks after the online publication of the Nat Med study. We excluded articles published on websites that were primarily aimed at healthcare professionals, hard paywalled articles, articles in blogs and “copy and paste” articles (i.e., articles that were copied partially or totally from other online articles). Finally, we also excluded news articles that mentioned the Nat Med study, but in which this study was not the main article’s subject.

Data collection

The first set of online articles were collated independently by two authors based on the prespecified inclusion and exclusion criteria. Subsequently, the selected articles were reviewed independently by two additional authors, thereby refining the selection further. Finally, all of the authors discussed and approved the final news articles selected. The target information was collated from the titles, full texts and from the images or videos included in the articles, wherever present. The data were collated using Microsoft Excel.

Development of a checklist to assess
the quality of health news articles covering observational research

We searched for guidelines and other resources focused on how health researchers and journalists should critically evaluate and properly report on health studies, with a particular emphasis on observational research.

Based on the collected resources14-16 we identified five thematic categories that we employed as a minimal reference standard for proper reporting of OSs. Namely: 1) causality, 2) fear mongering, 3) spin, 4) actionability and 5) critical evaluation. The definitions of each category are detailed in table 1.




In order to evaluate the quality of news reporting of the Nat Med study, we defined a list of items aimed at exploring the existence of issues in each online news article with regard to the five thematic categories above. All the authors evaluated, discussed, and modified the items until a consensus was reached consisting of a 13-item checklist (table 2).




Data extraction and article evaluation

The four authors of this article (two medical doctors and two health journalists) independently completed the 13-item checklist for every online news article included in the analysis. For every news article, each rater attributed dichotomous answers (“yes” or “no”) to the checklist’s items.

When a minimum of three out of the four raters provided the same response to an item, the answer to the corresponding item was considered complete and coherent based on the majority’s response. Items where the response was split evenly (two raters selecting one response and the other two raters selecting a different response), the four raters would discuss the answers until a consensus was attained. Inter-rater agreement among raters was evaluated using Fleiss Kappa.

Data analysis

We established the proportion of online news articles in the Nat Med study that were found to display issues with causality, fear mongering, spin, actionability and critical evaluation (tables 1 and 2).

Results

Following the online search, we collected an initial set of 49 online news articles. After screening of these articles for exclusion criteria, 19 articles were excluded, and 30 articles constituted the final sample. A list of the final 30 online news articles and their weblinks is available in Supplement 1. Evaluation of these 30 articles by four raters using the checklist resulted in the collation of 390 items per rater, equating to a total of 1560 items.

Global inter-rater agreement was substantial, with an average Fleiss Kappa of 0.73 (0.34-1). Uncertain answers (50% positive and 50% negative raters’ responses) requiring discussion constituted 8% of the checklist’s items. A consensus (with at least three raters out of four giving the same response to each item) was reached through discussion in 100% of the items.

Table 3 details the outcome analysis. An issue with causality was identified in 30 articles out of 30 (100%). An issue with fear mongering was identified in 25 (83.3%) of the 30 articles, and an issue with spin in 21 (75%) of the 28 articles. Furthermore, an issue with actionability and critical evaluation was identified in 16 (53.3%) and 26 (86.7%) of the 30 articles, respectively.




Discussion

Our analysis of the Italian online media coverage of a prominent OS published in a high-impact medical journal shows that most news articles have substantial issues relating to the quality of their reporting.

The major limitation of OSs, the difficulty to infer causation from correlations among variables, was never mentioned by the news articles included in our analysis. Moreover, most of the news articles showed issues with the thematic categories of critical evaluation, spin and fear mongering, while reporting on the Nat Med study.

These results are consistent with other studies investigating the quality of news media coverage in health research, both in Italy11,12 and in other countries8. However, to our knowledge, no study had focused yet on how contemporary observational research on health is covered by Italian online news media.

In our analysis, 53% of the Italian news articles proposed to undertake corrective actions as a consequence of the Nat Med study findings. Consistently, the Nat Med paper implies actions as a consequence of the study results: «Governments and health systems around the world should be prepared to deal with the likely significant contribution of the covid-19 pandemic to a rise in the burden of cardiovascular diseases. […] Addressing the challenges posed by Long covid will require a much needed, but so far lacking, urgent and coordinated long-term global response strategy»13.

Observational analytical research is the most suitable research strategy for evaluating a naturally occurring exposure (e.g., covid-19) with regard to its ability to increase the risk of one or more outcomes (e.g., cardiovascular disease). However, in the absence of robust replication, findings from a single retrospective study should not be deemed sufficient to inform corrective actions.

The attitude towards recommending corrective actions that are solely based on non-randomized studies’ results is not new: 56% of the authors of OSs published in high-impact journals were found to make health recommendations based on their non-randomised studies’ results17.

However, even if the Nat Med study results were definitive in regard to the causal relationship between covid-19 and cardiovascular disease, an increased incidence of a condition after an exposure is not equivalent to knowing what intervention would improve outcomes in this situation. To identify such an intervention, an RCT would be required.

Translating the results of scientific research into news is a challenging task. Although journalists play a key role, the responsibility for a problematic health news reporting goes far beyond the media realm. For example, a trend towards an unbalanced and hyped reporting of health study outcomes has been observed over the past decades, both in abstracts and in articles’ main texts18,19.

Press releases, in turn, often fail to consider study limitations or industry funding and tend to frame the study results in formats that may exaggerate or oversimplify the findings to facilitate media uptake and dissemination20. About this point, associations have been reported between exaggeration in health news and exaggeration in academic press release21 and, consistently, between high-quality medical journal press releases and better associated news stories22.

These findings seem to fit in with our study, as a press release about the Nat Med study, disseminated by the office of Research and Development within the US Veterans Affairs Administration on the 16th of February 202223, mirrors the shortcomings we observed in the online news articles.

Hype in news reporting about health does not come without consequences: for example, people are more likely to consider treatments as beneficial when the associated news stories are reported with spin24.

Some studies investigated the barriers to a proper health news reporting: competition for space and audience, tight timelines, low budgets, newsrooms staff reductions, and lack of medical and statistical knowledge have previously been described as the most common obstacles to improving health news reporting5,25,26.

Journalists could also be led (or explicitly invited) to highlight in their articles some aspects that amplify research results to satisfy business models. Such an action could be based on the sale of advertising space, in which one of the selling points is often the number of visits to publication websites. Moreover, journalists also struggle to access primary sources of scientific literature, which are often constrained by paywalls.

Finally, this issue is further exacerbated by inadequate training: for instance, in Italy there is no official curricular course on science journalism. Instead, training relies on personal initiative and on single institutions that offer a master’s degree in scientific journalism.

Pilot studies investigating the development and feasibility of training interventions aimed at journalists showed encouraging results in improving the quality of health reporting27,28.

Our analysis has limitations. First, the findings refer to the media coverage of a single OS, and therefore, these findings cannot be generalised to the media coverage of other health studies. The second potential limitation is the time frame of our analysis which was limited to news articles that were published in the early phase following the publication of the Nat Med paper. Consequently, this time frame could have overestimated the flaws and defects of these articles because of the lack of time available to journalists for analysing and critically appraising the covered study. Nonetheless, this time frame was selected because news media generally concentrate on the coverage of a health study in the short term after the study publication, and therefore the most impactful messages to the public are also concentrated in this very period.

Another limitation concerns the checklist and thematic categories used to evaluate the news reports. The checklist employed in our study has not been validated. Still, although other categories could have been chosen, our checklist includes a minimal quality reference based on existing guidelines and resources on the proper reporting of observational research.

Future perspectives

Our study evaluated the criticalities of online media coverage of a mainstream OS. In order to achieve a broader view of the shortcomings of online health reporting in Italy, we also aim to investigate the media coverage of other study designs.

Conclusions

In summary, our analysis of Italian online news media reporting about a contemporary OS, published in a major medical journal and dealing with a topic of high public interest, has shown that most news articles fail to properly report on the study’s findings.

A large portion of those medical practices that end up being contradicted by well conducted RCTs had originally been adopted on the basis of findings from OSs29. The lack of critical appraisal of observational research by the media could lead individuals and institutions to consider the associations highlighted in a study as definitely causal. In doing so, some news articles might unwillingly promote low-value care (e.g., unnecessary cardiologic evaluations in asymptomatic patients with previous mild covid-19), economic interests, unwarranted fear or even reliance on unreliable medical interventions.

Our findings are coherent with current literature about the shortcomings of health news reporting and add a more specific insight on the online media coverage of a mainstream OS in a definitive geographical area.

Based on the current corpus of evidence, a strict collaboration between medicine and the media is warranted to investigate through an RCT whether and to what extent a sound scientific training aimed at journalists would improve the quality of health reporting.

Conflict of interests: the authors have no conflict of interests to declare.




Acknowledgments: we warmly thank Stefano Del Pace for his timely comments and advice on an advanced draft of this




paper.

References

1. Schwartz LM, Woloshin S. The media matter: a call for straightforward medical reporting. Ann Intern Med 2004; 140: 226-8.

2. Oxman M, Larun L, Pérez Gaxiola G, et al. Quality of information in news media reports about the effects of health interventions: systematic review and meta-analyses [version 2; peer review: 4 approved]. F1000Research 2021; 10: 433.

3. Smith DE, Wilson AJ, Henry DA. Monitoring the quality of medical news reporting: early experience with media doctor. Med J Aust 2005; 183: 190-3.

4. Cassels A, Hughes MA, Cole C, et al. Drugs in the news: an analysis of Canadian newspaper coverage of new prescription drugs. CMAJ 2003; 168: 1133-7.

5. Schwitzer G. How do US journalists cover treatments, tests, products, and procedures? An evaluation of 500 stories. PLoS Med 2008; 5: e95.

6. Selvaraj S, Borkar DS, Prasad V. Media coverage of medical journals: do the best articles make the news? PLoS One 2014; 9: e85355.

7. Wang MTM, Bolland MJ, Gamble G, Grey A. Media coverage, journal press releases and editorials associated with randomized and observational studies in high-impact medical journals: a cohort study. PLoS One 2015; 10: e0145294.

8. Wang MTM, Bolland MJ, Grey A. Reporting of limitations of observational research. JAMA Intern Med 2015; 175: 1571-2.

9. The PLoS Medicine Editors. False hopes, unwarranted fears: the trouble with medical news stories. PLoS Med 2008; 5: e118.

10. Boutron I, Ravaud P. Misrepresentation and distortion of research in biomedical literature. Proc Natl Acad Sci U S A 2018; 115: 2613-9.

11. Iaboli L, Caselli L, Filice A, Russi G, Belletti E. The unbearable lightness of health science reporting: a week examining Italian print media. PLoS One 2010; 5: e9829.

12. Colombo C, Mosconi P, Buratti MG, et al. Press coverage of hormone replacement therapy and menopause. Eur J Obstet Gynecol Reprod Biol 2010; 153: 56-61.

13. Xie Y, Xu E, Bowe B, Al-Aly Z. Long-term cardiovascular outcomes of COVID-19. Nat Med 2022; 28: 583-90.

14. Von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP; STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol 2008; 61: 344-9.

15. Schwitzer G. “Covering medical research: a guide for reporting on studies” for members of the Association of Health Care Journalists (AHCJ). Centre for excellence in health care journalism, 2010.

16. Riva C, Tinari S. Fare giornalismo d’inchiesta su salute e medicina. Una guida di Global Investigative Journalism Network. Supplemento a Recenti Prog Med 2021; 112 (Suppl.): 1-84. ISSN 0034-1193.

17. Prasad V, Jorgenson J, Ioannidis JP, Cifu A. Observational studies often make clinical practice recommendations: an empirical evaluation of authors’ attitudes. J Clin Epidemiol 2013; 66: 361-6.

18. Vinkers CH, Tijdink JK, Otte WM. Use of positive and negative words in scientific PubMed abstracts between 1974 and 2014: retrospective analysis. BMJ 2015; 351: h6467.

19. Khan MS, Lateef N, Siddiqi TJ, et al. Level and prevalence of spin in published cardiovascular randomized clinical trial reports with statistically nonsignificant primary outcomes: a systematic review. JAMA Netw Open 2019; 2: e192622.

20. Woloshin S, Shwartz L. Press Releases. Translating research into news. JAMA 2002; 287: 2856-8.

21. Sumner P, Vivian-Griffiths S, Boivin J, et al. The association between exaggeration in health-related science news and academic press releases: retrospective observational study. BMJ 2014; 349: g7015.

22. Schwartz L, Woloshin S, Andrews A, et al. Influence of medical journal press release on the quality of associated newspaper coverage: retrospective cohort study. BMJ 2012; 344: d8164.

23. Research.va.gov [Internet]. Available at: https://bit.ly/3czIEtf [updated 2022 Feb 16; cited 2022 24 Aug].

24. Boutron I, Haneef R, Yavchitz A, et al. Three randomized controlled trials evaluating the impact of “spin” in health news stories reporting studies of pharmacologic treatments on patients’/caregivers’ interpretation of treatment benefit. BMC Med 2019; 17: 105.

25. Larsson A, Oxman AD, Carling C, Herrin J. Medical messages in the media: barriers and solutions to improving medical journalism. Health Expect 2003; 6: 323-31.

26. Amend E, Secko DM. In the face of critique: a metasynthesis of the experiences of journalists covering health and science. Sci Commun 2012; 34: 241-82.

27. Copp T, Dakin T, Nickel B, et al. Interventions to improve media coverage of medical research: a codesigned feasibility and acceptability study with Australian journalists. BMJ Open 2022; 12: e062706.

28. Arnold KM. Medicine in the media: symposium addresses the challenge of reporting on medical research. Science Editor 2003; 26: 17.

29. Prasad V, Cifu A, Ioannidis JPA. Reversals of established medical practices: evidence to abandon ship. JAMA 2012; 307: 37-8.