Reporting Overviews of Reviews: PRIORitizing a Reporting Guideline

Article information

Perspect Integr Med. 2024;3(2):65-68

Publication date (electronic) : 2024 June 30

doi : https://doi.org/10.56986/pim.2024.06.001

Lisa Hartling^1,, David Moher²

¹Alberta Research Centre for Health Evidence, Department of Pediatrics, University of Alberta, Edmonton, Canada

²Clinical Epidemiology Program, Ottawa Hospital Research Institute, School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada

^*Corresponding author: Lisa Hartling, Alberta Research Centre for Health Evidence, Department of Pediatrics, University of Alberta, Edmonton, Alberta, Canada, Email: hartling@ualberta.ca

Received 2024 April 18; Accepted 2024 May 9.

Visual abstract

Two recent publications in Perspectives on Integrative Medicine have employed an emerging form of evidence synthesis, however, both papers refer to different designs in their titles–overview of systematic reviews and umbrella review, respectively [1,2]. What are these? Do they use the same design and methodological processes? Can the results be trusted? These are some of the questions a reader might ask regarding these and similar publications. In this editorial, we aim to define this newer form of evidence synthesis; provide some history regarding its evolution and methodological approach; discuss some caveats around the methods and potential flaws; and present a reporting guideline [preferred reporting items for overview of reviews (PRIOR)] that can guide authors, peer-reviewers, editors, and readers in understanding these and similar evidence syntheses, their results, and implications for end-users (e.g., clinicians, policymakers, patients, and their caregivers).

Overviews of reviews (also referred to herein as overviews) have emerged as a form of evidence synthesis over the past 20–25 years with one of the first overviews published in 2000 [3]. They are similar to systematic reviews in that they aim to provide a comprehensive and unbiased synthesis of the literature on a topic by following rigorous, systematic, and predefined methods. However, the unit of analysis is systematic reviews rather than primary studies. The methodology has evolved, and the numbers have increased concurrent with the increase and advances in systematic reviews. Moreover, with the steady increase in the number of systematic reviews, with estimates of more than 80 being published per day [4], overviews of reviews can provide a valuable resource for end-users by bringing together evidence across a broad topic. This has great relevance to complementary therapies and integrative medicine where often a given intervention may be used across different health conditions, or multiple interventions may be considered as options for the same condition, particularly those that have a range of presenting symptoms and/or varying patient responsiveness across intervention options.

Within the Cochrane Collaboration, an international organization dedicated to producing high quality evidence syntheses to inform health decisions, a working group was established in 2004 to guide methods for overviews. Cochrane considered these products to provide a “friendly front-end” to reviews, by bringing together evidence from individual systematic reviews on related topics. To this end, Cochrane Child Health produced over 30 overviews of reviews (published in Evidence-based Child Health: A Cochrane Review Journal) to facilitate the uptake of evidence in practice and policy [5] (e.g., non-pharmacological interventions to treat attention deficit hyperactivity disorder [6], interventions to treat major depression in children and youth [7], and interventions to prevent eczema [8]). Two of the most common approaches to topics (or review questions) for overviews are the examination of the effectiveness and/or safety of multiple interventions for the same condition (e.g., treatments for women with pain or subfertility associated with endometriosis [9]) or a single intervention across multiple conditions (e.g., aromatherapy for management of health [1]). Also, of interest to end-users, overviews of reviews may provide evidence for subgroups or subpopulations of interest (e.g., children, elderly) by focusing on data related to those groups from the relevant systematic reviews (e.g., the reference [10]).

Alongside the production of overviews, much work has been performed to advance methods. Cochrane first published methodological guidance for overviews in 2008 [11]. This guidance was updated in 2020 [12]. New guidance for overview methods continues to emerge (e.g., the references [13–19]), with some guidance specific to the field of traditional and complementary medicine [20]. The Cochrane guidance focuses on overviews of intervention effectiveness, recognizing that subsequent guidance will be needed to address reviews with different types of questions such as diagnostic test accuracy, prognosis, and risk factors/associations [21]. JBI (originally known as The Joanna Briggs Institute), an international organization that supports evidence-informed decision-making to improve health and health service delivery, has also developed guidance for what they refer to as umbrella reviews [22]. The JBI concept of umbrella reviews differs somewhat from Cochrane’s overviews: while Cochrane’s overviews focus on systematic reviews of effectiveness, JBI umbrella reviews may incorporate different forms of systematic reviews [e.g., effectiveness (quantitative); meta-aggregative, integrative (qualitative)] to address “a broad scope of issues related to a topic of interest,” e.g., effectiveness as well as patient experiences with an intervention [22]. Terminology regarding the evidence synthesis product is critical for readers’ expectations and appraisal, to differentiate methodological approaches and rigor, and particularly, to distinguish an evidence synthesis from a nonsystematic summary of select literature (e.g., a more traditional literature review) [23].

Overviews and umbrella reviews are considered rigorous scientific endeavors, that take careful planning, time, and relevant research expertise (ideally experience with systematic reviews). Organizations may commission an overview, or researchers may propose such an approach, for time-dependent decision-making (e.g., the references [24,25]), without understanding the potential complexities. Overviews need to begin with clear objectives, well-formulated research questions (with specification of population(s), intervention(s), comparison(s), outcome(s) and other relevant variables), and a priori defined methods that aim to avoid bias. As with all research projects, preparing a protocol before conducting the overview and making it publicly available through a journal publication or repository (e.g., Open Science Framework) is a hallmark of methodological rigor and enhances transparency and openness for end-users [26]. The key difference between systematic reviews and overviews, is the unit of analysis (primary studies vs. systematic reviews), which can lead to specific challenges in conducting an overview and interpreting results. Three important challenges include the potential for overlapping studies across included systematic reviews, reporting (or non-reporting) biases, and inappropriate indirect comparisons.

Firstly, overviews may include systematic reviews that contain the same primary studies. This may occur when including multiple systematic reviews on the same topic. It may also occur when primary studies have multiple arms examining different active interventions. In this case, the same primary study may appear in individual systematic reviews examining specific active interventions. This can lead to overstating the sample size and number of events (i.e., outcomes), and can exaggerate the degree of precision in an analysis and subsequent estimate [12]. Care needs to be taken by the overview researchers so that the data from the same primary studies is only included once in any syntheses or estimates of effects. Methods are available for identifying and managing such overlap in overviews [12,17–19]. Overview authors should describe how they have addressed potential overlap (PRIOR Checklist items 9b and 17) [27].

Secondly, overviews are susceptible to at least two forms of reporting bias: publication bias and selective outcome reporting bias (PRIOR Checklist items 13 and 20) [27]. Overviews rely on the methods employed by the included systematic reviews, including methods to ensure comprehensive searching, identification, and inclusion of studies regardless of study results. Methodological research has shown that studies with negative (or statistically non-significant) results are less likely to be published and less likely to be published in English [28]. This may be particularly relevant to complementary therapies and integrative medicine; they are often practiced in countries whose first language is not English [29]. Overview researchers need to address the potential for publication bias as systematic reviews that have failed to comprehensively identify all relevant studies may present biased results that either over-estimate or under-estimate the true effects. Furthermore, overview researchers need to prespecify outcomes of interest (ideally in a protocol) and report on findings specific to these outcomes. It is unsatisfactory (and potentially biased) to simply describe what the systematic reviews have observed or what the primary studies have reported because of the likelihood to report on outcomes that show significant results [28].

Thirdly, caution with respect to interpreting results of overviews is the use of informal indirect comparisons between interventions. For example, some overviews will present results of systematic reviews examining different interventions compared with usual care, placebo, no control, or another active intervention. It can be tempting to assume that an intervention (A) that performs better than a given comparator (C) in one set of studies (or systematic review) may be preferred to a different intervention (B) that does not perform better than the same comparator (C) in a separate set of studies (or separate systematic review), i.e., making informal indirect comparisons between interventions A and B. Cochrane methods strongly discourage indirect comparisons unless formal statistical analyses (i.e., network meta-analysis) have been conducted to integrate and appropriately analyze data from direct and indirect comparisons [12]. This is rarely possible within the context of an overview because of the need for intimate knowledge of the individual studies to ensure a key assumption (transitivity) is met.

The ability of readers to assess these and other potential sources of bias, and ultimately the validity and applicability of results, will depend on transparent and detailed reporting. Reporting guidelines might be considered an intervention to ensure that reports of primary studies and evidence syntheses accurately and sufficiently reflect the methods and results and allow for replication. The Enhancing the QUAlity and Transparency of Health Research (EQUATOR) Network [30] is an international initiative that promotes transparent and accurate reporting of research. The EQUATOR Network library houses over 500 reporting guidelines for a wide range of study designs including primary studies across research pillars (basic, preclinical, clinical, health services, and population health), and evidence syntheses. The EQUATOR Network also endorses specific methods for the rigorous development of reporting guidelines [31,32].

Several key reporting guidelines have been developed for evidence syntheses, including the Preferred Reporting Items for Systematic Review and Meta-Analyses (abbreviated to PRISMA) [33,34], and more recently the Preferred Reporting Items for Overviews of Reviews (PRIOR) [27]. The development of PRIOR followed established methods which are detailed elsewhere [27,31,32]. Of note, rigorous reporting guidelines are developed based on evidence, and where evidence is lacking or unclear, consensus among a range of stakeholders. PRIOR was established based on scoping reviews of the literature for methodological guidance, a Delphi study involving 100 stakeholders from multiple end-user groups (i.e., authors, journal editors, funders, relevant international organizations, patients), and consensus meetings with stakeholder representatives. The final PRIOR guideline involved a collaboration among 19 authors from 8 countries and 5 continents. Reporting guidelines, including PRIOR, typically include a checklist of items for authors to report, a flow diagram that describes the search and selection process, and an Explanation and Elaboration document with rationale and examples for each checklist item. The PRIOR reporting guideline should be used by authors, editors, and peer-reviewers to ensure that key aspects of an overview of reviews are reported such that readers can assess its validity and applicability to their context.

Drummond Rennie (former Deputy Editor of the Journal of the American Medical Association) wrote that the “whole of medicine depends on the transparent reporting of trials” [35]. This is also essential with evidence syntheses because of the potential for errors and biases to be carried forward and amplified. Authors of systematic reviews and overviews need to employ methods and rigorously appraise their included studies (or reviews) such that accurate, reliable, and valid results are presented with appropriate and relevant conclusions. Reporting guidelines are not only relevant to authors when reporting the results of their studies; they should guide protocol development. Authors should look at reporting guidelines when preparing their protocols and methods to ensure that they can adhere to the reporting requirements once their evidence synthesis is complete. Furthermore, reporting guidelines are very relevant to editors, peer-reviewers, funding bodies of research, and users of research, and should be encouraged (if not mandated) to ensure that end users have the information they need, contextualized within potential limitations, to inform healthcare practices and policies. We encourage researchers (and publishers) within the fields of complementary therapies and integrative medicine to elevate the quality of research through consideration of, and adherence to, established reporting guidelines.