This glossary collects the key concepts, people, and ideas you will meet in this lesson. Use it as a reference while you work through the material, or as a review before assessments. Type in the search box to filter entries.

Introduction and Overview

An earlier lesson set up the foundations of epidemiology, its history, its ways of knowing, and the trust scaffolding (research integrity, reproducibility) that lets a body of evidence accumulate. A later section of that lesson made the case that empiricism is one way of knowing among several, and a later section left you with a personal stance on which of those ways you would lean on when the stakes are public-health decisions. This lesson picks up at exactly that seam: once you commit to evidence-based reasoning, how do you tell stronger evidence from weaker? Before we open up the machinery of systematic reviews and meta-analyses, we need a shared way of organising the answer. That organising idea (the hierarchy of knowledge) is where this lesson starts, and it is the lens through which every later lesson in the series will be read.

The Hierarchy of Knowledge

Before reading further, sit with the prompt below for a moment. The rest of the section is easier to follow if you have already put your own intuitions on paper.

Epidemiologists answer this question with a layered model usually called the hierarchy of evidence, or simply the evidence pyramid. The pyramid is not a strict ranking of every study against every other; it is a heuristic for thinking about how much variation a particular design has ruled out. At the base sit forms of evidence that are easy to produce but hard to defend on their own: expert opinion, individual experience, single case reports. Moving up, we add structured comparison (case series → cross-sectional → case-control → cohort). Higher still are interventional designs that randomise the exposure (randomised controlled trials), which under ideal conditions rule out confounding entirely. At the apex sit the designs that synthesise across all of the layers below: systematic reviews and meta-analyses.

A Complementary Hierarchy: DIKW

The evidence pyramid tells us where to look. A second, complementary hierarchy tells us what the evidence is for. The DIKW pyramid (Data → Information → Knowledge → Wisdom) is widely used in informatics, decision science, and clinical practice:

• Data are raw observations: counts, measurements, dates, lab values.
• Information is data given structure: organised, contextualised, compared.
• Knowledge is information appraised: filtered for quality, integrated across sources, ready to defend in print.
• Wisdom is knowledge exercised in a specific context, under uncertainty, in service of a particular decision.

A meta-analysis is not the same thing as wisdom. It is the highest-order knowledge layer in the evidence pyramid, the most thoroughly appraised, most heavily aggregated form of knowledge the field knows how to produce. Wisdom is what a clinician, policy-maker, or community partner does with that knowledge in their particular case, and that step is irreducibly human, contextual, and value-laden. An earlier lesson's call for a critical epidemiology that asks who benefits, who is harmed, whose priorities lives at the wisdom layer, not the knowledge layer. Keeping the two hierarchies side-by-side is a useful corrective against treating a forest plot as if it answered every question on its own.

Why Start at the Top of the Pyramid?

This course spends later lessons working through the rungs below the apex: case-control, cohort, ecological, and cross-sectional designs, plus the threats (selection, information, confounding) that compromise each. So why does this lesson begin at the top? Three reasons, each of which shapes the rest of the series:

1. You need the appraisal lens before you read any single study. When you encounter a primary observational study in later lessons, the first question is not "is this study good?" but "what does the synthesised literature on this question already say, and how does this study fit in?" The synthesis layer is the lens through which every later study should be read.
2. The hierarchy is a heuristic, not a verdict. A poorly conducted systematic review is weaker evidence than a well-designed cohort study. A well-designed case series of an unprecedented exposure may be the only evidence in existence. The hierarchy tells you where to look for the strongest available evidence; it does not exempt you from appraising what you find. A later section of this lesson will make that point concrete with publication bias, influential studies, and outcome-scale issues that can compromise even a textbook meta-analysis.
3. Every tool in this lesson exists because the lower tiers can mislead. PROSPERO registration, PRISMA reporting, GRADE certainty ratings, and the Cochrane risk-of-bias appraisals (so named for Archie Cochrane, whose 1972 monograph Effectiveness and Efficiency framed the case for systematically appraised RCT evidence); each is a field-wide response to a specific failure of unstructured evidence accumulation. An earlier lesson's trust ledger and the open-science reforms in its later sections are the wider movement these tools belong to. The hierarchy of evidence is, in a real sense, a history of the lessons epidemiologists learned the hard way.

With the hierarchy in mind, the four content sections of this lesson move from broad to specific. This section sets up systematic reviews as a structured way to identify and appraise all relevant studies; a later section turns to meta-analysis as the quantitative pooling of effect estimates, including the central choice between fixed- and random-effects models; a later section covers the forest plot and heterogeneity analysis; a later section closes with the threats to a meta-analysis that survive correct technique. You will revisit these tools throughout the remainder of this course as you appraise individual observational studies, every appraisal sits inside the question, "and what does the synthesised literature say?"

16.1 Why Systematic Reviews?

When making decisions about health interventions, we want to use all available information. Unfortunately, the literature is often inconclusive and conflicting, individual studies may produce results ranging from statistically significant to inconsequential, and the variation among results may be greater than expected from chance alone. A classic demonstration of the cost of relying on narrative summaries is Antman et al. (1992), who showed that expert reviews of myocardial-infarction treatments lagged years behind what a cumulative meta-analysis of randomised trials had already established.

There are two fundamental approaches to formally reviewing available data: a narrative review and a systematic review (which may include a meta-analysis).

16.2 Steps of a Systematic Review

A systematic review follows seven steps (Sargeant et al., 2006), summarised here at the level a reader needs in order to judge a published review.

1. Specify the question. The question comes from a health-policy or clinical objective and is framed with PICO for an intervention or PECO for an exposure.
2. Lay out the protocol. A written plan for every later step is fixed before screening and ideally registered on PROSPERO (16.2.1).
3. Find all the studies. The team searches bibliographic databases, reference lists and grey literature, and documents every search (16.2.2).
4. Determine relevance. Two reviewers independently apply the eligibility criteria to titles and abstracts, then to full texts.
5. Evaluate study quality. Each included study is assessed for risk of bias with a tool matched to its design. For randomised trials, RoB 2 (Sterne et al., 2019) judges five domains: bias arising from the randomisation process, deviations from intended interventions, missing outcome data, measurement of the outcome, and selection of the reported result. ROBINS-I (Sterne et al., 2016) is the parallel tool for non-randomised studies of interventions. Sequence generation, allocation concealment and blinding were items of the original Cochrane tool (Higgins et al., 2011) that RoB 2 replaced. Risk-of-bias judgements are reported beside each study's result and are used in pre-specified sensitivity analyses that exclude studies at high risk of bias, in subgroup analyses by risk of bias, and in the GRADE risk-of-bias domain; the meta-analysis itself weights studies by inverse variance, and summary quality scores are avoided because different scales give different conclusions (Jüni et al., 1999).
6. Extract the data. Two reviewers independently record each study's estimate and its precision on a standardised form.
7. Summarise and synthesise. Results are summarised narratively or pooled in a meta-analysis, the subject of the next three sections.

HSCI 241, which can be taken before or after this course, teaches each step in a lesson of its own.

16.2.1 Two Tools That Underpin a Trustworthy Review

Modern systematic reviews are expected to do more than simply follow the seven steps; they must do so transparently and they must communicate how confident readers should be in the conclusions. Two widely adopted tools have become the de facto standards for these expectations: PROSPERO for protocol registration and GRADE for rating the certainty of the evidence.

16.2.2 Finding the Studies for an Appraisal

Step 3 is where many reviews succeed or fail, and the same knowledge lets a reader judge whether a published review could have found the relevant studies. Most searching happens in bibliographic databases such as MEDLINE (searched free of charge through PubMed), Embase and CINAHL. A bibliographic database does not hold the articles themselves. It holds a record for each article (title, abstract, authors, journal, date and publication type), and in MEDLINE trained indexers add subject headings that describe what the article is about. A search runs against those records by two complementary routes.

• Controlled vocabulary. MEDLINE's thesaurus is Medical Subject Headings (MeSH), and Embase uses its own, Emtree. A heading gathers the records on one concept under a single label, whatever words the authors used. Each heading lists entry terms, the synonyms that map to it, and sits in a tree of broader and narrower headings. PubMed explodes a heading by default, so a search on it also retrieves records indexed with any narrower heading.
• Free text. Words searched in titles and abstracts (the [tiab] field in PubMed) find records that indexers have not yet reached, records indexed before a heading existed, and PubMed records that are never indexed.

The worked example uses the question from this section's reflection: in adults with type 2 diabetes (P), do SGLT2 inhibitors (I), compared with standard care or DPP-4 inhibitors (C), reduce cardiovascular mortality (O)? The first table shows one MeSH heading beside free-text synonyms for the same concept.

A search usually combines two or three concepts from the question. Here the population and the intervention are searched; the comparator and the outcome are left to screening, because studies describe them in too many ways for a search to capture them reliably.

Combining the terms. Boolean operators join the pieces of a search. OR joins the synonyms within one concept and widens the set; AND joins different concepts and keeps only the records that match both; NOT removes every record that matches a term. NOT is used with care, because it also removes relevant records that happen to match the excluded term. Quotation marks make a phrase search, so "type 2 diabetes" finds the words together and in that order, and an asterisk truncates a word stem, so gliflozin* finds gliflozin and gliflozins. Joining the two columns of the table gives the worked PubMed string.

Keeping a search log. A reproducible search is documented as it is built. The log below records three runs of the example search in PubMed, with the string, the date, the number of records and the reason for each change. The counts are those PubMed returned on 8 October 2026; the same strings return more records later, as new articles are added.

A log like this lets anyone rerun the search and see why it changed, and it shows the searcher when a refinement has removed relevant records, as the first NOT attempt did. HSCI 241 Lessons 3 and 4, which can be taken before or after this course, teach systematic searching in full.

16.2.3 Appraising a Published Review

A systematic review can be well reported and still be unreliable, so a reader appraises it before relying on it. AMSTAR 2 (Shea et al., 2017) is a 16-item tool for appraising systematic reviews of randomised and non-randomised studies of healthcare interventions. Seven of its items are critical domains, because a weakness in any of them can undermine a review's conclusions. The overall confidence rating depends mainly on those domains: high (no weakness, or one non-critical weakness), moderate (more than one non-critical weakness and no critical flaw), low (one critical flaw, with or without non-critical weaknesses) and critically low (more than one critical flaw). AMSTAR 2 asks for this overall judgement in place of a summed score.

Overall confidence: low. The review has one critical flaw (item 7), so under AMSTAR 2 it may not provide an accurate and comprehensive summary of the available studies, whatever the nine non-critical items show. Before relying on its pooled estimate, a reader would look for the list of excluded studies, or check the trial registries for trials the review might have dropped.

Introduction and Overview

An earlier section covered how to identify and appraise the studies that go into a review. This section turns to the quantitative half: combining the effect estimates from those studies into a single pooled estimate. The central design choice here (fixed-effects versus random-effects) is not arbitrary; it reflects an underlying assumption about whether all the studies are estimating the same true effect.

16.3 What Is a Meta-Analysis?

A meta-analysis is “the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings” (Glass, 1976). It is a formal process for combining results from multiple studies and is considered the “gold standard” for providing summary information about health interventions.

16.3.1 Types of Data in Meta-Analysis

Three types of data can be used in a meta-analysis, each with different capabilities:

Summary data are most commonly used. Group data allow computation of various effect measures. IPD are the most flexible but rarely available; they allow evaluation of study-, group-, and individual-level variables as sources of heterogeneity.

16.4 Fixed- vs. Random-Effects Models

A fundamental decision in any meta-analysis is whether to use a fixed-effects or random-effects model:

16.4.1 Weighting Methods

The most common weighting procedure is inverse variance weighting, applicable to both continuous and binary outcomes. The name states the recipe: each study's weight is one divided by its variance, so a precise estimate (small variance) earns a large weight and a noisy one earns little, which is why large, well-measured studies dominate the pooled result. For binary outcomes with sparse data, the Mantel-Haenszel procedure (Mantel & Haenszel, 1959) or the Peto method may be preferred. For continuous outcomes, when studies use different scales, standardised mean differences (effect sizes such as Cohen’s d or Hedges’ g) are used.

Introduction and Overview

An earlier section produced a single pooled estimate. This section introduces the visual and quantitative tools for inspecting the underlying variation that produced that estimate. The forest plot makes the constituent studies and the pooled result visible at a glance; heterogeneity statistics quantify how much the studies actually disagree with one another. Both are essential before trusting the pooled number.

16.5 Presentation of Results: The Forest Plot

The forest plot is the most important graphical output of a meta-analysis. It displays the point estimate and confidence interval of the effect observed in each study, along with the summary estimate.

16.6 Heterogeneity

Heterogeneity refers to variability among study results beyond what would be expected from random variation alone. It should always be evaluated in a meta-analysis.

16.6.1 Real vs. Artifactual Heterogeneity

An important distinction is between clinical heterogeneity (real differences between populations, interventions, and settings) and statistical heterogeneity (variation in observed results beyond chance). Clinical heterogeneity is always expected; the key question is whether statistical heterogeneity is also present.

16.6.2 Measuring Heterogeneity: Cochran’s Q and Higgins I²

Where w_i are the study weights, T_i are the study effects, and θ is the pooled estimate. Under the null hypothesis of no heterogeneity, Q follows a χ² distribution with k−1 degrees of freedom. However, the Q test has low power when the number of studies is small, so a non-significant result does not rule out heterogeneity. Consider using a relaxed P-value threshold (e.g., 0.10 instead of 0.05).

I² (Higgins & Thompson, 2002) quantifies the proportion of variance between studies that is due to heterogeneity rather than chance. Benchmarks: 25% = low, 50% = moderate, 75% = high heterogeneity. An evaluation of possible causes should be undertaken whenever I² exceeds 25%.

The formula is more intuitive than it first looks. If every study were really estimating the same effect, Cochran's Q would on average equal its degrees of freedom, the number of studies minus one. Subtracting that quantity from Q removes the disagreement expected from chance alone, and dividing by Q re-expresses what remains as a fraction of the total spread. When the studies agree no more than chance predicts, that fraction falls to zero and I² is reported as 0%.

16.6.3 Evaluating Causes of Heterogeneity

Introduction and Overview

Earlier sections produced and inspected a pooled estimate. This section addresses three threats that can survive correct technique: publication bias (some studies never make it to the literature), influential studies (a single trial driving the entire pooled result), and outcome-scale issues that make it inappropriate to pool studies that look superficially similar. Each of these is a place where a meta-analysis can produce a precise but misleading answer.

16.7 Publication Bias

A critical concern in meta-analysis is publication bias, studies with statistically significant or favourable results are more likely to be published than those with null or unfavourable results. Consequently, published studies may represent a biased subset of all work conducted on a topic.

16.7.1 Detecting Publication Bias: The Funnel Plot

A funnel plot displays each study’s SE (or its inverse, 1/SE) plotted against its estimated effect. In the absence of publication bias, the plot should resemble an inverted funnel, symmetric around the summary estimate, with small studies (large SEs) scattered widely at the bottom and large studies (small SEs) clustered near the top.

16.7.2 Statistical Tests for Publication Bias

Two commonly used tests evaluate the relationship between study results and their precision:

• Begg’s test: A rank correlation between effect estimates and their SEs. Simple but low power with few studies.
• Egger’s test: A linear regression approach that is generally more sensitive at detecting publication bias (Egger et al., 1997).

Neither test is very sensitive when the number of studies is small (<20), and both may produce false positives when there are large treatment effects, few events per trial, or all trials are of similar size. For comprehensive guidance on interpreting funnel-plot asymmetry, see Sterne et al. (2011).

16.7.3 Trim-and-Fill Method

16.8 Influential Studies

It is important to determine whether individual studies have a profound influence on the summary estimate. A study might be much larger than others or have an extreme effect size. To evaluate this, sequentially delete each study from the meta-analysis and observe how the summary estimate changes.

16.9 Outcome Scales and Data Issues

Published studies vary substantially in how they present data. Several practical issues arise:

Bringing It All Together

This lesson introduced evidence synthesis: the structured process by which the literature on a question is identified, appraised, and pooled into a single defensible answer. The arc moved from the seven steps of a systematic review, through the central design choice between fixed- and random-effects models, to the diagnostic tools (forest plots, heterogeneity statistics, funnel plots) that show whether the pooled answer should be trusted, and finally to the threats (publication bias, influential studies, outcome-scale issues) that survive correct technique.

The final assessment below asks you to integrate across all four sections: distinguishing systematic from narrative reviews, choosing between fixed- and random-effects, reading a forest plot, computing and interpreting heterogeneity, and recognizing when a precise pooled estimate is nonetheless misleading.

The same checks apply, in miniature, whenever you read a published review: explicit inclusion criteria, a structured and documented search, transparent appraisal, and an honest accounting of what the existing literature can and cannot answer. The search subsection and the AMSTAR 2 worked appraisal in Section 1 give you the tools for that reading.

This assessment covers all sections of this lesson. You must answer all 15 questions correctly to complete the lesson. Review the feedback after each attempt.

Final Knowledge Assessment