3 min read
Expectation versus Reality
It is a common misconception that meta-analysis and systematic reviews are the same and the terms are often used interchangeably. While there is substantial overlap between the two they are not the same thing. Let’s start by defining the two concepts:
A systematic review is a detailed, systematic and transparent means of gathering, appraising and synthesising evidence to answer a well-defined question.
A meta-analysis is a statistical procedure for combining numerical data from multiple separate studies. A meta-analysis should only ever be conducted in the context of a systematic review.
The hallmark of systematic reviews is that they seek to reduce bias at all stages of the review process. Reviews registered with organisations such as Campbell and Cochrane are particularly reliable, as all authors are required to adhere to the same high standards of conduct and reporting. The stages in conducting a review are:
- Define your question and ideally register your proposed review title with Campbell or similar
- Specify and publish your proposed methodology in advance in the form of a protocol
- Conduct a thorough search of the literature
- Screen your search results against your pre-specified selection criteria to identify included studies
- Appraise the quality of studies found
- Synthesise the evidence, this is where meta-analysis may or may not come in
- Publish and disseminate your review
- Update the review as new evidence is produced
You can see that a systematic review involves much more than simply putting the numbers together. It is a detailed, transparent and sometimes (often) time consuming process.
Why would a systematic review not include a meta-analysis?
Systematic reviews will often, but not always, contain a meta-analysis of numerical data from the included studies. Meta-analysis would be a poor choice if your review question is better answered with qualitative data, such as “How acceptable are psychosocial interventions for maltreated children” or “how do self-help group programs impact on women’s empowerment?”
Conducting a meta-analysis would be a bad idea if your studies are too different to combine. Imagine you are conducting a systematic review on the effect of listening to music while studying on children’s exam performance and you find one study on classical music, two on pop music including primary school children and teenagers respectively, another on death metal and a fifth on 80’s synth classics. Would you combine these in a single meta-analysis or would you decide that the interventions (music type) and populations (age of children) are too dissimilar to combine and opt for a narrative synthesis instead? Deciding which studies you can and cannot combine in a meta-analysis will depend on the question you are asking and you should define the process for deciding in advance in your review protocol.
Conducting a meta-analysis may not always be sensible – even if you set out to do one. By specifying your approach to meta-analysis in advance, you can reduce the possibility of introducing bias and avoid making decisions retrospectively based on the studies or results you find. This is why spending time on your protocol and thinking through the analytic approach before beginning your systematic review is time well spent.
Tip 1: Beware of meta-analyses that do not follow a systematic and transparent process for identifying and selecting which studies to include in analysis
Tip 2: When developing a systematic review title focus on the question you really want to answer, there is no need to constrain yourself to numerical data that can be meta-analysed.
Tip 3: Specify your analysis plan in advance and be transparent in reporting your methods. Following the appropriate reporting guidelines (PRISMA) and standards (MECCIR) is an easy way to keep you right.
Tip 4: Thinking of conducting a systematic review? Contact us for advice and support to conduct a Campbell Systematic Review.
Blog post written by Jennifer Hanratty
evidence based researchmeta-analysismeta-evidencesystematic-review
Sur RL, Dahm P. History of evidence-based medicine. Indian journal of urology : IJU : journal of the Urological Society of India. 2011;27(4):487–9.
Article Google Scholar
Group EW. Evidence-based medicine. A new approach to teaching the practice of medicine. JAMA. 1992;268(17):2420–5.
Article Google Scholar
Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. BMJ. 1996;312(7023):71–2.
CAS Article Google Scholar
Greenhalgh T, Howick J, Maskrey N. Evidence based medicine: a movement in crisis? BMJ. 2014;348:g3725.
Article Google Scholar
Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC, Catala-Lopez F, Li L, Reid EK, Sarkis-Onofre R, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med. 2016;13(5):e1002028.
Article Google Scholar
Ioannidis JP. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485–514.
Article Google Scholar
Riva N, Puljak L, Moja L, Ageno W, Schunemann H, Magrini N, Squizzato A. Multiple overlapping systematic reviews facilitate the origin of disputes: the case of thrombolytic therapy for pulmonary embolism. J Clin Epidemiol. 2017.
Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart LA. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic reviews. 2015;4:1.
Article Google Scholar
Pieper D, Pollock M, Fernandes RM, Buchter RB, Hartling L. Epidemiology and reporting characteristics of overviews of reviews of healthcare interventions published 2012-2016: protocol for a systematic review. Systematic reviews. 2017;6(1):73.
Article Google Scholar
Pieper D, Koensgen N, Breuing J, Ge L, Wegewitz U. How is AMSTAR applied by authors – a call for better reporting. Unpublished data. 2017.
Lunny C, McKenzie JE, McDonald S. Retrieval of overviews of systematic reviews in MEDLINE was improved by the development of an objectively derived and validated search strategy. J Clin Epidemiol. 2016;74:107–18.
Article Google Scholar
Schwartz S, Campbell UB, Gatto NM, Gordon K. Toward a clarification of the taxonomy of "bias" in epidemiology textbooks. Epidemiology. 2015;26(2):216–22.
Article Google Scholar
Aziz T, Compton S, Nassar U, Matthews D, Ansari K, Flores-Mir C. Methodological quality and descriptive characteristics of prosthodontic-related systematic reviews. J Oral Rehabil. 2013;40(4):263–78.
CAS Article Google Scholar
Clarke M, Chalmers I. Discussion sections in reports of controlled trials published in general medical journals: islands in search of continents? Jama. 1998;280(3):280–2.
CAS Article Google Scholar
Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.
Article Google Scholar
Puljak L. If there is only one author or only one database was searched, a study should not be called a systematic review. J Clin Epidemiol. 2017.
Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. JAMA. 1992;268(2):240–8.
CAS Article Google Scholar
Oxman AD, Guyatt GH. The science of reviewing research. Ann N Y Acad Sci. 1993;703:125–33 discussion 133-124.
CAS Article Google Scholar
Cobey KD, Lalu MM, Skidmore B, Ahmadzai N, Grudniewicz A, Moher D. What is a predatory journal? A scoping review. F1000Research. 2018;7:1001.
Article Google Scholar
From: Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks
| Category | Element of definition | N (%) |
|---|
| Self-identified as a systematic review | Manuscript that identifies itself as a systematic review in title, abstract or in methods | 30 (13) |
| Indexing | Indexed as SR | 1 (0.4) |
| Aim/research question | Specific research question | 66 (29) |
| Clearly stated set of objectives | 12 (5.3) |
| Clearly formulated research question | 11 (4.8) |
| Focused research question | 3 (1.3) |
| Reported research question | 2 (0.9) |
| Clinical question including participants, interventions, controls, outcomes and study design (PICOS) | 2 (0.9) |
| Explicit clinical question | 1 (0.4) |
| Clearly stated topic of review | 1 (0.4) |
| Explicitly reported pre-defined objectives | 1 (0.4) |
| Stated goal implied a critical and comprehensive intent | 1 (0.4) |
| Clear statement of the topic | 1 (0.4) |
| Defined clinical topic | 1 (0.4) |
| Explicit statement of questions being addressed | 1 (0.4) |
| Overall methods | Systematic methods | 22 (9.7) |
| Explicit methods | 21 (9.2) |
| Systematic method to minimize risk of bias | 9 (4) |
| Systematic approach, in an attempt to minimize biases and random errors, documented in the Materials and Methods section | 8 (3.5) |
| Explicit method to minimize risk of bias | 7 (3.1) |
| Reproducible methods | 5 (2.2) |
| Using a systematic approach | 5 (2.2) |
| Methods described in explicit detail | 4 (1.8) |
| Well-defined methods | 2 (0.9) |
| Overall methods defined study as systematic review | 1 (0.4) |
| Overall Conduct defined study as a systematic review | 1 (0.4) |
| Systematic review methodology on closer inspection of the methods section | 1 (0.4) |
| Specific methods | 1 (0.4) |
| Repeatable methods | 1 (0.4) |
| Rigorous methods | 1 (0.4) |
| Different components of the review process documented in the ‘methods section’ | 1 (0.4) |
| Using methods to provide more reliable findings | 1 (0.4) |
| Using methods from which conclusions can be drawn | 1 (0.4) |
| Using methods based on which decisions can be made | 1 (0.4) |
| Exhaustive review of the literature | 1 (0.4) |
| Systematic approach | 1 (0.4) |
| Search | Systematic search | 29 (13) |
| Reported search strategy | 13 (5.8) |
| Comprehensive search strategy | 12 (5.3) |
| Searched at least two databases/sources | 10 (4.4) |
| Exact search criteria reported | 9 (4.0) |
| Searched at least one database | 9 (3.9) |
| Reported search methods | 7 (3.1) |
| Attempt to collate all empirical evidence | 7 (3.1) |
| Reported all information sources | 6 (2.6) |
| Transparent search strategy | 6 (2.6) |
| Detailed and comprehensive search strategy (as identified by: naming of databases and years of searching and example or actual terms) | 4 (1.8) |
| Detailed and specific search strategy with key-words that enabled reproduction of the literature search | 4 (1.8) |
| Names of databases reported | 4 (1.8) |
| Explicit search criteria that are available to review | 3 (1.3) |
| Description of data sources and search dates | 2 (0.4) |
| Keywords searched | 2 (0.9) |
| Detailed search of the literature for relevant studies | 2 (0.9) |
| Explicit description of search strategy | 2 (0.9) |
| Adequate searching methods | 2 (0.9) |
| Replicable search method | 2 (0.9) |
| Reported search sources | 1 (0.4) |
| Description of sources | 1 (0.4) |
| Reported details of databases searched | 1 (0.4) |
| Reported dates of search | 1 (0.4) |
| Included relevant search strategy | 1 (0.4) |
| Adequate search strategy | 1 (0.4) |
| Appropriate search strategy | 1 (0.4) |
| Detailed search strategy | 1 (0.4) |
| Non-selective search strategy | 1 (0.4) |
| Explicit search strategy | 1 (0.4) |
| Prescriptive search strategy | 1 (0.4) |
| Reproducible search strategy | 1 (0.4) |
| Rigorous search process | 1 (0.4) |
| Explicitly reported search strategy details | 1 (0.4) |
| Thorough search of evidence | 1 (0.4) |
| Comprehensive search of evidence | 1 (0.4) |
| Reported search processes | 1 (0.4) |
| Extensive use of search string combinations | 1 (0.4) |
| Description of evidence retrieval methods | 1 (0.4) |
| Explicit and organized approach to searching | 1 (0.4) |
| Attempt to search all empirical evidence | 1 (0.4) |
| Adequately attempt to retrieve all relevant data | 1 (0.4) |
| Review trying to collect all available evidence | 1 (0.4) |
| Structured search of bibliographic and other databases | 1 (0.4) |
| Searched at least Medline | 1 (0.4) |
| Searched at least two databases (of which one is Medline) | 1 (0.4) |
| Identification of studies | Explicit methods to identify relevant research | 14 (6.2) |
| Systematic methods of identification of studies | 10 (4.4) |
| Attempt to identify all empirical evidence | 6 (2.6) |
| Reported methods for identification of studies | 2 (0.9) |
| Transparent procedure to find relevant research | 2 (0.9) |
| Formal process of identifying literature | 1 (0.4) |
| Selection of studies | Explicit methods to select relevant research | 14 (6.2) |
| Systematic methods of selection of studies | 13 (5.8) |
| Reported methods for selection of studies | 6 (2.6) |
| Transparent selection of studies | 2 (0.9) |
| Reproducible selection of studies | 4 (1.8) |
| Reproducible approach for selecting the studies | 1 (0.4) |
| Clear description of selection criteria | 1 (0.4) |
| Clear study selection criteria | 1 (0.4) |
| Relevant study selection criteria | 1 (0.4) |
| Detailed description of the studies’ selection process (number of articles included and excluded in each step) | 1 (0.4) |
| Study eligibility | Reported inclusion and exclusion criteria | 31 (14) |
| Pre-defined/pre-specified eligibility criteria | 20 (8.8) |
| Outcome defined using a validated tool or diagnostic criteria | 13 (5.8) |
| Only Cochrane systematic reviews | 12 (5.3) |
| Reported inclusion criteria | 6 (2.6) |
| Explicitly reported inclusion and exclusion criteria | 6 (2.6) |
| Articles that meet PRISMA definition of a systematic review | 5 (2.2) |
| Definitions of the population(s), intervention(s), comparator(s) and outcome(s) of interest | 2 (0.9) |
| Inclusion/exclusion criteria that are relevant in terms of the PICO framework | 3 (1.3) |
| Reviews published in Database of Reviews of Effects (DARE) | 2 (0.9) |
| Reviews were judged to be systematic if they synthesized peer reviewed articles | 1 (0.4) |
| Studies meeting minimum methodological standards | 1 (0.4) |
| Reference to study designs | 1 (0.4) |
| Data extraction | Systematic data collection | 12 (5.3) |
| Systematic methods to extract data | 4 (1.8) |
| Explicit methods to collect data | 3 (1.3) |
| Data extraction by 2 independent reviewers | 2 (0.9) |
| Reported data abstraction from trials | 2 (0.9) |
| Independent data extraction | 1 (0.4) |
| Explicit approach to extracting | 1 (0.4) |
| Organized approach to extracting | 1 (0.4) |
| Explicit methods to extract data | 1 (0.4) |
| Performed data extraction | 1 (0.4) |
| Extracting the information from the studies following a priori protocol | 1 (0.4) |
| Quality, bias, appraisal, validity | Quality assessment of evidence | 27 (12) |
| Critical appraisal of the studies | 25 (11) |
| Risk of bias assessment | 19 (8.4) |
| Systematic methods to critically appraise relevant research | 13 (5.8) |
| Explicit methods to critically appraise relevant research | 13 (5.8) |
| Reported validity assessment | 11 (4.9) |
| Attempt to appraise all empirical evidence | 6 (2.6) |
| Full assessment of methodological quality of included studies | 5 (2.2) |
| Consideration of internal and external validity of the research | 3 (1.3) |
| Provided sufficient details about individual included studies to enable assessment of quality by a reader | 2 (0.9) |
| Reported at least one or more aspects of validity assessment of original studies | 2 (0.9) |
| Transparent procedures to evaluate relevant research | 2 (0.9) |
| Full report of methodological quality of included studies | 1 (0.4) |
| Transparent process to minimize risk of bias | 1 (0.4) |
| Explicit approach to critically evaluating studies | 1 (0.4) |
| Organized approach to critically evaluating empirical literature | 1 (0.4) |
| Systematic approach for assessing the studies | 1 (0.4) |
| Reproducible approach for assessing the studies | 1 (0.4) |
| Assessed methodological features of the included studies | 1 (0.4) |
| Adequate methods to appraise included studies | 1 (0.4) |
| Transparent methodological criteria are used to exclude papers that do not meet an explicit methodological benchmark | 1 (0.4) |
| Evaluate the retrieved studies using prospectively defined methodological criteria | 1 (0.4) |
| Analysis, synthesis | Synthesis of results | 34 (15) |
| Presence of meta-analysis | 19 (10) |
| Systematic methods of analysis of studies | 18 (8.0) |
| Explicit methods to analyze data | 17 (7.5) |
| Systematic synthesis of findings | 10 (4.4) |
| Quantitative synthesis | 9 (4.0) |
| Synthesis of the included evidence, whether narrative or quantitative | 7 (3.1) |
| Attempt to synthesize all empirical evidence | 6 (2.6) |
| Systematic analysis of results | 2 (0.9) |
| Unbiased synthesis of study findings | 2 (0.9) |
| Transparent procedures to synthesize the results of relevant research | 2 (0.9) |
| Analyze results appropriately | 1 (0.4) |
| Systematic analysis | 1 (0.4) |
| Plausible analysis of data | 1 (0.4) |
| Plausible synthesis of data | 1 (0.4) |
| Summary of results | 1 (0.4) |
| Systematic analysis | 1 (0.4) |
| Meta-analysis or best evidence synthesis | 1 (0.4) |
| Formal analysis contained in the methods | 1 (0.4) |
| Makes judgement about research question | 1 (0.4) |
| Relying on statistical significance to make judgments about what works | 1 (0.4) |
| Transparent process of interpretation of the findings of the studies included in the review | 1 (0.4) |
| Rigorous conclusions about outcomes | 1 (0.4) |
| Describing included studies | Systematic presentation of characteristics of included studies | 4 (1.8) |
| Systematic synthesis of characteristics of included studies | 4 (1.8) |
| Clearly identified all included studies | 2 (0.9) |
| Reported trial characteristics | 1 (0.4) |
| Systematic presentation of main information | 1 (0.4) |
| Described main characteristics of included studies | 1 (0.4) |
| Adequate methods to describe included studies | 1 (0.4) |
| Description of the number and nature of included studies | 1 (0.4) |
| Description of the types of primary studies included | 1 (0.4) |
| Accounted for identified studies | 1 (0.4) |
| Reporting | Used PRISMA or predecessor guidelines for reporting | 3 (2) |
| Presented results appropriately | 1 (0.4) |
| Systematic presentation of findings | 1 (0.4) |
| Flow chart present | 1 (0.4) |
| Reported level of evidence for their recommendations | 1 (0.4) |
| Reported sufficient information to allow a level of evidence grading | 1 (0.4) |
| Published in a journal conforming to PRISMA standards | 1 (0.4) |
| A review that has methods and results section | 1 (0.4) |
| Unclear | “It was apparent in the text that a systematic review had been undertaken” | 4 (1.8) |
| “Reviews were included if they were systematic” | 1 (0.4) |
