Evidence Synthesis and the Family of Reviews
Finding & Synthesizing Health Evidence
Learning objectives for this lesson:
- Explain why single studies and unsystematic expert summaries can mislead, with reference to chance, bias, context and selective publication.
- Define evidence synthesis and systematic review, and explain how the features of a systematic review address the weaknesses of narrative reviews.
- Describe the purpose, typical question and main methods of narrative, systematic, scoping, rapid, umbrella, realist and living reviews.
- Describe an environmental scan and explain when a scan should accompany a review of published research.
- Check for existing reviews and judge whether a review is relevant, current and trustworthy.
- Choose and justify a review type from the purpose of a question, the existing evidence, the time and team available, the need for local information and the pace of new evidence.
- Explain the purpose of a review protocol, compare registration in PROSPERO and OSF Registries, and describe how amendments should be recorded and reported.
- Describe the fictional Cedar Valley evidence review, and explain how a team checks for existing reviews and justifies a rapid scoping review and environmental scan before it writes a protocol.
This course was developed by Dr. Kiffer G. Card, Faculty of Health Sciences, Simon Fraser University, drawing on the Cochrane Handbook for Systematic Reviews of Interventions and the JBI Manual for Evidence Synthesis.
Why Single Studies Mislead and Syntheses Help
Learning Objectives for this section
- Describe what this course teaches, how it relates to HSCI 230 Lesson 2 and HSCI 841 Lesson 2, and how its twelve lessons follow the stages of a review.
- Describe the fictional Cedar Valley evidence review that serves as the running case for the course.
- Explain four reasons why a single study can mislead: chance, bias, context and selective publication.
- Explain why traditional expert summaries of research can mislead, with reference to the work of Mulrow (1987) and Antman and colleagues (1992).
- Define evidence synthesis and systematic review, and identify what explicit, reproducible methods add and what they cannot fix.
Introduction: About This Course
Health decisions are rarely made on the strength of one study. A health authority deciding whether to fund a program, a clinician choosing a treatment and a ministry writing a policy all need to know what the body of research says, and they usually need to know it quickly. Evidence synthesis is the family of methods for finding, appraising and combining the results of many studies (and sometimes of documents, programs and expert knowledge) so that a question can be answered with more confidence than any single source allows.
This course teaches you to conduct evidence syntheses. You will learn to frame a review question, write a protocol, build and document a database search, search the grey literature and the web, use and check artificial intelligence (AI) tools, screen records, extract and appraise data, synthesize findings without statistical pooling, run an environmental scan of current programs, and report the results for decision-makers. Each lesson applies its methods to the Cedar Valley review, the running case described in Section 1.1.
The course sits alongside two other lessons in the series. HSCI 230 Lesson 2 teaches you to read and interpret systematic reviews and meta-analyses, including forest plots, heterogeneity and publication bias, and it teaches the statistics of pooling. This course concentrates on doing a review, and it refers you to HSCI 230 Lesson 2 whenever pooling is involved. Lesson 10 of this course introduces qualitative evidence synthesis (thematic synthesis, framework synthesis, meta-ethnography and GRADE-CERQual), and HSCI 841 Lesson 2 builds on it for graduate students, adding critical interpretive synthesis, qualitative meta-summary, the choice of method with the RETREAT criteria and a worked GRADE-CERQual assessment.
Each section opens with a short narrated walkthrough and closes with a reflection and a knowledge check. A companion podcast, Office Hours, covers the same material as a conversation, and the glossary at the end of the lesson defines every technical term.
1.1 The Running Case: The Cedar Valley Evidence Review
Every lesson uses the same project as its main worked example, so that you can watch one review develop from its first question to its final brief.
The Cedar Valley Health Authority is a fictional health authority in British Columbia. It serves about 210,000 residents, of whom about 46,000 are aged 65 and older, across Cedar City and several smaller communities, with 24 primary care clinics. Its planning team intends to launch a community connector program, a form of social prescribing in which clinicians refer patients to a person who links them with community groups, activities and services.
Before the launch, the planning team asks a small evidence team for a rapid scoping review and environmental scan. The evidence team consists of a health authority evidence officer, a university librarian who works with the team, and a student intern. The planning team needs an evidence brief within twelve weeks.
The review question asks which community-based interventions have been evaluated for reducing loneliness or social isolation among adults aged 65 and older, and with what outcomes. The scan question asks which such programs currently operate in British Columbia and elsewhere in Canada, who runs them, and how they are funded and evaluated. All figures about Cedar Valley in this course are illustrative.
Over the term the evidence team chooses a review type (this lesson), writes its question and protocol, works with the librarian to build and translate a search, searches the grey literature and the web, tests an AI research assistant and checks its citations, screens records, charts data, appraises studies, synthesizes the findings, contacts organizations for the scan, and writes an evidence brief for decision-makers. The intern is in roughly your position, learning each step as the project needs it.
Two terms recur in the case. Loneliness is the distressing feeling that one’s relationships are fewer or less satisfying than one wants, and social isolation is an objective shortage of social contacts. A person can be isolated without feeling lonely, so reviews usually treat the two as separate outcomes.
1.2 Why Single Studies Mislead
Suppose the planning team found one well-publicized trial in which a befriending program reduced loneliness among older adults. One study, even a good one, can point in the wrong direction for at least four reasons.
Chance
Every study measures a sample of people, and a different sample would give a somewhat different result. This sampling variability is largest in small studies. Figure 1.2 shows eight simulated trials of the same hypothetical program. In every trial the true effect was set at a fall of 0.5 points on a loneliness scale; the only difference between the trials is the people who happened to be sampled and the number of participants.
The estimates range from a fall of 1.18 points (Trial D) to a rise of 0.19 points (Trial B), although every trial studied the same program. Only three of the eight confidence intervals exclude zero, so a reader who sorted the trials by whether they reached statistical significance would conclude that the program “worked” three times and “failed” five times. A reader who saw only Trial D would expect a large benefit, and a reader who saw only Trial B might suspect harm. Taken together, the estimates cluster around a modest fall, and the larger trials (F, G and H) give the narrowest intervals. Lesson 9 explains why counting studies by statistical significance misleads, and HSCI 230 Lesson 2 shows how a meta-analysis combines estimates like these.
Bias
Bias is a systematic error that pushes a study’s result away from the truth in a consistent direction. A trial in which the people delivering the program also rate participants’ loneliness may record more improvement than occurred. A study comparing people who chose to join a program with people who did not may mistake the traits of joiners, who may be healthier or more sociable, for an effect of the program. Bias does not shrink as studies get larger, so a synthesis has to appraise each study’s design and conduct (Lesson 8).
Context
A program evaluated with one population and one set of local services may work differently elsewhere. A befriending program studied among urban older adults with strong volunteer networks may perform differently in a rural British Columbia community with long travel distances. A set of studies can show whether results hold across settings.
Selective publication and reporting
Studies with striking or statistically significant results are more likely to be published, published quickly, and published in widely read journals. This tendency is called publication bias. Within published studies, authors may report the outcomes that look favourable and leave out the others, which is selective outcome reporting. Both processes make the studies a reader happens to see a biased sample of the studies that were done. Synthesis methods respond by searching beyond journals (Lesson 5) and by comparing what studies planned to measure with what they reported.
1.3 Why Expert Summaries Can Also Mislead
For most of the twentieth century, practitioners learned what the research said from textbooks and from review articles written by recognized experts. These narrative reviews summarize a topic in the author’s own way, and they remain useful for orientation. Their weakness is that a reader usually cannot tell how the studies were found, why some were included and others left out, or how conflicting results were weighed.
Two studies made this weakness visible. Cynthia Mulrow (1987) examined review articles published in major medical journals and found that very few described how their authors had identified, selected or appraised the evidence they discussed. Antman and colleagues (1992) went further. They combined the trials of treatments for heart attack in the order in which the trials had been published, and they compared the accumulating evidence with what textbooks and review articles recommended at the time. Expert recommendations often lagged years behind the evidence: some effective treatments were not recommended long after the trials supported them, and some treatments continued to be recommended after the trials suggested they were ineffective or harmful.
If you have taken HSCI 230 Lesson 2, you met the Antman example in its Section 16.1; this section adds Mulrow’s (1987) analysis, and Section 2.1 of this lesson describes when a narrative review is appropriate.
A closely related problem is cherry-picking, the selection of studies that support a conclusion the author already holds. Cherry-picking is often unintentional. Any of us is more likely to remember, find convincing and cite the studies that agree with what we expect. The remedy is procedural: decide in advance which studies count, search for all of them, and report what was found.
A community newsletter states: “Research shows that befriending programs reduce loneliness in older adults. Two recent studies found large improvements among participants.” Before reading on, write down three questions you would need answered before you could judge whether the claim reflects the research as a whole.
Useful questions include whether other studies exist that found no effect, whether the two studies had a comparison group or simply measured participants before and after, how large they were and how long they followed participants, and whether their participants and settings resemble the newsletter’s readers. Each question corresponds to one of the four sources of error in Section 1.2, and a systematic review answers each one by design.
1.4 What a Systematic Approach Adds
The response to these problems was to apply the standards of primary research to the review itself. Archie Cochrane, a Scottish epidemiologist, argued in Effectiveness and Efficiency (Cochrane, 1972) that health services should be judged by evidence from randomized trials, and he later criticized the medical profession for lacking regularly updated summaries of that evidence. Iain Chalmers and colleagues assembled such summaries for pregnancy and childbirth in the 1980s and in 1993 founded the Cochrane Collaboration (now Cochrane), an international network that maintains systematic reviews of health interventions. The Campbell Collaboration followed in 2000 for social, educational and justice interventions, and JBI, based at the University of Adelaide, developed methods for reviewing a wider range of evidence, including qualitative research and scoping reviews.
Two definitions used throughout the course
Evidence synthesis is the process of bringing together the findings of multiple sources, using explicit methods, to answer a defined question in a form that people can use.
A systematic review attempts to identify, appraise and synthesize all the studies that meet pre-specified eligibility criteria in order to answer a specific question, using explicit methods chosen to minimize bias so that others could repeat the process (Higgins et al., 2019).
Table 1.1 links each defining feature of a systematic review to the problem it addresses.
| Feature of a systematic review | Problem it addresses | Where this course teaches it |
|---|---|---|
| A focused question and a protocol written before the work starts | Decisions made after seeing the results, and duplication of effort | Lessons 1 and 2 |
| Pre-specified eligibility criteria | Cherry-picking studies that support a view | Lesson 2 |
| A comprehensive, documented search of several sources | Missing studies and publication bias | Lessons 3 to 6 |
| Screening and extraction by two people working independently | Errors and individual judgement in selecting and recording studies | Lessons 7 and 8 |
| Appraisal of each study’s risk of bias | Treating flawed and sound studies as equally informative | Lesson 8 |
| A transparent synthesis method | Weighing results by impression or by statistical significance alone | Lessons 9 and 10 |
| Reporting to a published guideline (PRISMA 2020) | Readers being unable to check or repeat the review | Lesson 12 |
A systematic review may include a meta-analysis, a statistical method that combines the numerical results of several studies into a single estimate. Meta-analysis is optional. When studies differ too much in their populations, interventions or outcomes, a review synthesizes them in other structured ways, which Lesson 9 describes. The reporting guideline for systematic reviews is PRISMA 2020, the Preferred Reporting Items for Systematic reviews and Meta-Analyses (Page et al., 2021), and you will meet its extensions throughout the course.
1.5 What Syntheses Cannot Fix
A systematic method improves the way evidence is gathered and combined, while the evidence itself remains as strong or as weak as the studies that produced it. Syntheses also carry limits of their own that you should keep in view from the start.
If the included studies are small, short or at high risk of bias, a careful review of them yields an uncertain answer. A good review states that uncertainty plainly. Lesson 8 introduces GRADE, a method for rating how much confidence a body of evidence deserves.
The number of published systematic reviews has grown very quickly, and many overlap or use weak methods. Ioannidis (2016) described the mass production of redundant, misleading and conflicted reviews. The label “systematic review” does not guarantee quality, so reviews themselves need appraisal, which Section 3 of this lesson introduces.
A review describes the evidence up to the date of its search. In fast-moving fields, new studies can change the conclusion within months. Section 2 describes living reviews, which are updated continually, and Lesson 12 discusses when and how to update a review.
The National Collaborating Centre for Methods and Tools (NCCMT), based at McMaster University, describes evidence-informed decision making in public health as combining research evidence with the local context, community and political preferences, and available resources, using public health expertise to integrate them. The Cedar Valley planning team will weigh the review alongside its budget, its partners and the views of older adults, and the review’s task is to make the research side of that judgement complete and honest.
Summary of Section 1
This course teaches the conduct of evidence syntheses, from question to report, using the fictional Cedar Valley evidence review as its running case. Single studies mislead through chance, bias, context and selective publication, and traditional expert summaries mislead when their selection of studies is hidden. Systematic reviews respond with a protocol, pre-specified criteria, comprehensive searching, duplicate screening, appraisal, transparent synthesis and standard reporting. These methods improve how evidence is assembled, while the quality of the underlying studies, the currency of the search and the needs of the decision still limit what a synthesis can say.
Reflection
A city councillor proposes funding a telephone befriending program for older adults on the basis of one trial described in a news story. The trial enrolled 40 older adults in another country, compared participants with a waiting-list group, and reported that loneliness scores fell by 1.5 points more in the befriending group. The councillor says that the research shows the program works. This lesson describes four ways a single study can mislead: chance (results vary between samples, especially in small studies), bias (systematic error arising from how a study was designed or conducted), context (a result from one population and setting may not hold elsewhere) and selective publication (favourable results are more likely to be published and publicized). Explain how each of the four could make this single result misleading for the councillor’s decision, and describe what a systematic review of the evidence would add.
Chance. With 40 participants, about 20 per group, the estimate is imprecise. A difference of 1.5 points could arise even if the true effect were much smaller, and a second trial of the same size might find little or no difference.
Bias. If the people who made the calls also collected the loneliness ratings, or if participants knew they had received the program, reported improvement could be overstated. A waiting-list group also knows it is waiting, which can affect how its members answer.
Context. The trial took place in another country with its own volunteer networks and services. Older adults in this city may differ in their needs, their telephone use and the alternatives available to them.
Selective publication. A trial with a striking result is more likely to be published and to reach a news story, while trials of similar programs that found no effect may never be published.
A systematic review would search for all trials of telephone befriending, including registries and unpublished reports, appraise each one for risk of bias, and show the spread of results across settings. The councillor would then see whether the 1.5-point result is typical or an outlier, and how much confidence the evidence as a whole supports. An equally strong answer might add that the review should report how many trials were found and how similar their settings are to this city.
Minimum 20 characters required.
Question 1: In Figure 1.2, eight simulated trials studied the same program with the same true effect, yet their estimates ranged from a fall of 1.18 points to a rise of 0.19 points. What best explains this spread?
Question 2: Which statement about bias in a single study is correct?
Question 3: What did Antman and colleagues (1992) find when they compared accumulating trial evidence on treatments for heart attack with expert recommendations?
Question 4: A systematic review sets its eligibility criteria before it begins. Which problem does this feature mainly address?
The Family of Reviews: From Narrative Reviews to Environmental Scans
Learning Objectives for this section
- Describe the purpose, typical question and main methods of narrative, systematic, scoping, rapid, umbrella, realist and living reviews.
- Describe an environmental scan and explain how it differs from a review of published research.
- Compare review types on search, appraisal, synthesis and analysis using the SALSA framework of Grant and Booth (2009).
- Recognize common alternative labels for review types and identify the type a published review actually used from a description of its methods.
Introduction
Review types have multiplied as researchers adapted systematic methods to new questions and new deadlines. Grant and Booth (2009) described fourteen types of review in common use, and Sutton and colleagues (2019) later catalogued several dozen named types and grouped them into families that share a purpose and a set of methods. The names are not always used consistently, and the same label can describe quite different work. For that reason this section defines each type by what it is for and what it does, and it uses a simple framework for comparing them.
Grant and Booth compared review types using four steps that every review performs in some form, which they abbreviated as SALSA: Search (how sources are found), AppraisaL (how their quality is judged), Synthesis (how findings are combined) and Analysis (what the review concludes about the body of evidence as a whole). Two reviews can share a name and differ on every one of these steps, so SALSA is a useful way to read a review’s methods section.
2.1 Seven Types of Review
Narrative reviews
A narrative review (also called a traditional or literature review) is an author’s overview of a topic, organized around themes or a historical account. Narrative reviews are valuable for orientation, for proposing new ideas and for teaching, and the introduction of almost every journal article contains a short one. Their limitation, discussed in Section 1, is that the selection and weighing of studies is usually undocumented. A well-written narrative review states how its author searched and what was left out of scope, even if the search was informal. A narrative review is the right choice when the aim is a broad, interpretive overview and nobody will treat it as a complete account of the evidence.
Systematic reviews
A systematic review answers a focused question by identifying all eligible studies, appraising them and synthesizing their results with pre-specified, documented methods. Systematic reviews are best known for questions about effectiveness (does an intervention change an outcome, and by how much), and they are also used for questions about prevalence, diagnostic accuracy, prognosis and the association between an exposure and an outcome. The methods are set out in the Cochrane Handbook for Systematic Reviews of Interventions (Higgins et al., 2019) and the JBI Manual for Evidence Synthesis (Aromataris & Munn, 2020), and reporting follows PRISMA 2020 (Page et al., 2021). A systematic review takes considerable time: Borah and colleagues (2017), using records from the PROSPERO registry, estimated that the average review took more than a year from registration to publication.
Scoping reviews
A scoping review maps the extent, range and nature of the evidence on a topic. It is suited to questions such as what kinds of interventions have been studied, how a concept has been defined, which outcomes have been measured, and where the gaps lie. Arksey and O’Malley (2005) proposed the first methodological framework, with five stages (identifying the question, identifying relevant studies, selecting studies, charting the data, and collating and reporting the results) and an optional consultation with interest holders. Levac, Colquhoun and O’Brien (2010) refined each stage, and JBI publishes current guidance (Peters et al., 2020). Scoping reviews use a comprehensive, documented search like a systematic review, and they usually chart and describe studies without appraising their risk of bias or pooling their results. Their question is framed with PCC (Population, Concept and Context), and they are reported with PRISMA-ScR, the extension of PRISMA for scoping reviews (Tricco et al., 2018).
Rapid reviews
A rapid review is a form of evidence synthesis that streamlines or omits some steps of a systematic review so that it can be completed in less time and with fewer resources (Hamel et al., 2021). Typical shortcuts include searching fewer databases, limiting dates or languages, having one reviewer screen while a second checks a sample, and simplifying appraisal. Rapid reviews are usually commissioned by decision-makers who need an answer within weeks or a few months, and the Cochrane Rapid Reviews Methods Group has published guidance on which shortcuts are least likely to change the conclusions (Garritty et al., 2021). The shortcuts are acceptable when they are stated openly and their likely consequences are reported. A rapid scoping review applies the same streamlining to a scoping question, and it is the design the Cedar Valley review uses. Lesson 10 examines the trade-off each shortcut makes.
Umbrella reviews
An umbrella review, also called an overview of reviews, treats systematic reviews as its units of analysis. It searches for reviews on a broad topic, appraises them, and compares and summarizes their findings (Aromataris et al., 2015). Umbrella reviews suit topics on which many systematic reviews already exist, each covering a different intervention or population, and a decision-maker needs a single view across them. One common tool for appraising the reviews is AMSTAR 2 (Shea et al., 2017). An umbrella review inherits the limitations of the reviews it includes: if those reviews searched only to a certain date, or excluded certain study designs, the umbrella review cannot recover what they missed.
Realist reviews
A realist review asks what works, for whom, in what circumstances and why. It was developed by Pawson and colleagues (2005) for complex policy and social interventions whose effects depend on how people respond to them in particular settings. A realist review builds and tests a program theory, expressed as context-mechanism-outcome configurations: in this context, this mechanism (how people respond to what the program offers) produces this outcome. Searching is iterative and purposive, guided by the developing theory, and sources are judged on their relevance to the theory and the rigour of the relevant parts. Realist reviews are reported with the RAMESES publication standards (Wong et al., 2013). They suit questions about why a community connector program helps some older adults and leaves others no less lonely.
Living systematic reviews
A living systematic review is a systematic review that is continually updated, incorporating relevant new evidence as it becomes available (Elliott et al., 2014; Elliott et al., 2017). The team reruns the search at a set interval, such as monthly, and revises the findings when new studies change them. Living reviews suit high-priority questions in which the evidence is changing quickly and the uncertainty matters for decisions, and they were used widely during the COVID-19 pandemic for treatments and vaccines. They require a standing team and a commitment to continue, so they are uncommon outside well-funded programs.
2.2 Environmental Scans
An environmental scan collects and interprets information about current programs, services, policies, organizations and trends in a defined setting, to inform planning. The method has its origins in organizational strategic planning, where Aguilar (1967) described how managers gather information about the environment outside their firm. Health services and public health researchers adopted it to answer practical questions about what exists now and where (Rowel et al., 2005; Graham, Evitts & Thomas-MacLean, 2008).
A review of published research cannot answer the Cedar Valley scan question, because many community programs operating in British Columbia have never been described in a journal article. A scan gathers its information from organization websites, program directories, annual reports, government documents, surveys of organizations and key informant interviews with the people who run programs. Its products are an inventory (which programs exist, where, run by whom, funded how) and a description of themes and gaps. Scans are often combined with a literature review, so that a decision-maker sees both what the research says and what is happening locally. Lesson 11 teaches scan methods in detail.
A review asks what studies have found, and its sources are research reports. A scan asks what currently exists in a place, and its sources include organizations and people. A review can be repeated by another team with the same search; a scan describes a moment in time, so its date and methods of contact must be documented carefully.
Websites change, programs close and new ones open. A scan that records which sites were searched, on what date, with which search terms, and which organizations were contacted, can be checked and updated. Lesson 5 teaches web search documentation, and Lesson 11 applies it to scans.
Surveys and interviews with program staff collect information from people, so the team should check whether the work needs research ethics review or falls under quality improvement in its organization. HSCI 207 Lesson 5 covers research ethics review in Canada.
2.3 Comparing the Family with SALSA
Table 1.2 compares the types on the SALSA steps. The entries describe typical practice; individual reviews vary, and a review’s methods section matters more than its label.
| Type | Search | Appraisal | Synthesis and analysis | Typical time and team |
|---|---|---|---|---|
| Narrative review | Chosen by the author and often undocumented | Informal or absent | Narrative interpretation by the author | Weeks; often one author |
| Systematic review | Comprehensive and documented, across several databases and grey literature | Formal risk-of-bias assessment by two reviewers | Structured narrative and tables, with meta-analysis where appropriate; certainty of evidence rated | Often more than a year; a team with a librarian |
| Scoping review | Comprehensive and documented within a broad scope | Usually absent | Charting and descriptive mapping of the evidence and its gaps | Several months; two or more reviewers |
| Rapid review | Streamlined, with fewer sources or limits on dates or languages | Simplified, often by one reviewer with checking | Narrative and tables, with limitations stated | Weeks to a few months; a small team |
| Umbrella review | Searches for systematic reviews | Appraises the reviews, for example with AMSTAR 2 | Tabulates and compares review findings and their overlap | Months; two or more reviewers |
| Realist review | Iterative and purposive, guided by the program theory | Relevance and rigour of each source for the theory | Context-mechanism-outcome configurations that explain variation | Many months; content and methods expertise |
| Living systematic review | As a systematic review, rerun at set intervals | As a systematic review | As a systematic review, revised when new studies change the findings | Ongoing; a standing team |
| Environmental scan | Websites, documents and directories, with surveys and interviews | Information checked against sources and with organizations | Inventory of programs, with themes and gaps | Weeks to months; a small team |
2.4 Labels You Will Meet
Published reviews use many other labels. Some describe distinct methods that later lessons cover, and others are loose names that tell you little until you read the methods. Select each card for a short explanation.
Read each description and decide which review type it describes, using the SALSA steps rather than the title. Then open the answers below.
(1) The authors searched five databases and charted the populations, intervention types and outcomes of 64 studies of intergenerational programs, without assessing risk of bias. (2) The authors searched for systematic reviews of exercise interventions for falls, appraised each review with AMSTAR 2, and tabulated their conclusions. (3) The authors searched two databases for studies published in the last five years, in English only, with one reviewer screening and a second checking 20 percent of excluded records, to answer a ministry’s question within eight weeks. (4) The authors searched iteratively for studies of peer support programs and developed explanations of how trust between peers produced engagement in some settings and not in others.
(1) This is a scoping review: the search was comprehensive, the studies were charted and described, and risk of bias was not appraised. (2) This is an umbrella review, because its units of analysis are systematic reviews, appraised with a tool designed for reviews. (3) This is a rapid review, with three named shortcuts (two databases, date and language limits, and single screening with partial checking) and a decision-maker’s deadline. (4) This is a realist review: the search was iterative and the analysis explained variation in outcomes through mechanisms and contexts.
Summary of Section 2
The family of evidence syntheses includes narrative, systematic, scoping, rapid, umbrella, realist and living reviews, each suited to a different kind of question. The environmental scan is a close relative that studies current programs and services through documents, websites, surveys and interviews. The SALSA framework (search, appraisal, synthesis and analysis) compares types by what they do, which is more reliable than the label a review gives itself. Section 3 uses these differences to choose a review type for a given question, deadline and team.
Reflection
A manager at a health authority says: “A scoping review is just a faster systematic review, so let’s do a scoping review to find out whether our walking-group program reduces loneliness.” In this lesson, a systematic review answers a focused question, often about effects, by finding all eligible studies, appraising their risk of bias and synthesizing their results, sometimes with a meta-analysis. A scoping review maps the extent, range and nature of the evidence on a topic with a comprehensive, documented search, and it usually charts studies without appraising or pooling them. Write a reply to the manager that explains how the two types differ in purpose, appraisal and synthesis, why a scoping review cannot answer the question as posed, and what a scoping review could usefully tell the manager about walking groups.
The two types differ in purpose before they differ in speed. A systematic review asks a focused question, such as whether walking groups reduce loneliness among older adults and by how much. It finds every eligible study, judges each study’s risk of bias, and combines the results in a structured way, sometimes with a meta-analysis. A scoping review asks a mapping question, such as what kinds of walking-group programs have been evaluated, with whom, and with which outcomes. Its search is just as thorough, but it charts and describes the studies and usually does not appraise them or combine their effects. Scoping reviews are also not necessarily faster, because the search and screening can be as large as in a systematic review.
A scoping review therefore cannot tell the manager whether the program reduces loneliness, because it does not judge whether the studies were sound or synthesize what they found. It could usefully show how many evaluations exist, which designs and loneliness measures they used, which populations and settings they covered, and where evidence is missing, for example on rural programs. If that map shows enough comparable trials, the manager could then commission a systematic or rapid review of effectiveness.
Minimum 20 characters required.
Question 1: A team wants to know which outcomes have been measured in evaluations of intergenerational programs and where the evidence is thin. Which review type fits this purpose best?
Question 2: What distinguishes an umbrella review from the other review types in this lesson?
Question 3: In the SALSA framework of Grant and Booth (2009), what do the four steps refer to?
Question 4: Why can an environmental scan answer the Cedar Valley scan question when a review of published research cannot?
Choosing a Review Type from the Question, Time and Resources
Learning Objectives for this section
- Check whether an existing review already answers a question, using review databases and registries, and judge whether it is relevant, current and trustworthy.
- Identify the features of a project (purpose, existing evidence, time, team, need for local information and pace of new evidence) that determine the choice of review type.
- Use the decision aid to recommend a review type for a scenario, and explain how a change in one feature changes the recommendation.
- Justify the choice of a rapid scoping review and environmental scan for the fictional Cedar Valley evidence review, and explain why the alternatives were set aside.
Introduction
Choosing a review type is the first methodological decision in any synthesis, and a poor choice is hard to repair later. A team that sets out to do a systematic review of effectiveness when its real question is what has been studied will spend months appraising risk of bias that nobody needed. A team that calls a narrative review “systematic” will mislead its readers about how complete it is. Munn and colleagues (2018) wrote guidance on choosing between systematic and scoping reviews for exactly these reasons. This section turns the differences described in Section 2 into a set of questions, a decision aid and a worked example.
3.1 First, Check Whether a Review Already Exists
Before choosing a type of review, a team should find out whether someone has already answered the question or is currently doing so. Producing a review that duplicates a recent, sound one is a form of research waste, the term Chalmers and Glasziou (2009) used for research that is unnecessary, poorly designed or never usefully reported. A preliminary search of the following sources takes an afternoon:
- The Cochrane Library contains Cochrane systematic reviews and protocols for health interventions.
- Epistemonikos is a free database that collects systematic reviews from many sources and links them to the primary studies they include.
- PROSPERO and OSF Registries list review protocols, including reviews that are under way and not yet published (Section 4 describes both).
- Health Evidence, hosted by the National Collaborating Centre for Methods and Tools at McMaster University, collects and rates reviews relevant to public health.
- A search of MEDLINE limited to systematic reviews finds reviews in journals that the other sources may have missed (Lesson 4 shows how to apply such a limit).
When a review turns up, three questions decide what to do with it. The first is whether it answers the same question, with the same population, intervention or concept, outcomes and setting. The second is whether it is current: the date of its last search matters more than its publication date, since reviews are often published a year or more after the search. The third is whether it is trustworthy, which tools such as AMSTAR 2 help to judge (Shea et al., 2017). AMSTAR 2 is a 16-item checklist for appraising systematic reviews of health interventions, and it treats seven items as critical domains, because a weakness in any of them can undermine a review’s conclusions:
- The protocol was registered before the review began.
- The literature search was adequate.
- The exclusion of individual studies was justified.
- The risk of bias in the included studies was assessed with a satisfactory method.
- Any meta-analysis used appropriate statistical methods.
- The risk of bias in the included studies was considered when the results were interpreted.
- The presence and likely impact of publication bias were assessed.
AMSTAR 2 does not produce a total score. The appraiser instead rates overall confidence in the review’s results as high (no critical flaw and at most one non-critical weakness), moderate (more than one non-critical weakness), low (one critical flaw) or critically low (more than one critical flaw). HSCI 230 Lesson 2 Section 16.2.3 works through an example for students who take that course. A relevant, current and trustworthy review can be used as it is. A relevant review that is out of date can be updated. A review that answers part of the question can be built on, and its included studies used to test the team’s own search.
The student intern on the fictional Cedar Valley evidence team searches Epistemonikos, the Cochrane Library, Health Evidence and PROSPERO. She finds several published reviews on interventions for loneliness and social isolation, including a meta-analysis of interventions to reduce loneliness (Masi et al., 2011), an integrative review of interventions for older people (Gardiner, Geldenhuys & Gott, 2018) and a scoping review of reviews on interventions for older adults (Fakoya, McCorry & Donnelly, 2020). Each was published at least six years before the Cedar Valley project, and none describes the programs operating in British Columbia or how Canadian programs are funded. The team decides that a new review is justified, that these reviews will be an early source of studies through their reference lists, and that the question needs a design suited to mapping recent evidence quickly.
3.2 Six Questions That Point to a Review Type
Once the team knows what already exists, six features of the project usually settle the choice. Figure 1.4 shows how they fit together, and Table 1.3 lists them.
| Feature of the project | If the answer is… | The choice leans toward… |
|---|---|---|
| Purpose of the question | Whether something works, and how well; what has been studied; how and why it works; what existing reviews say; what exists locally; a general overview | Systematic or rapid review; scoping review; realist review; umbrella review; environmental scan; narrative review |
| Existing reviews | A recent, sound review answers the question | Using, appraising or updating that review |
| Existing reviews | Several systematic reviews cover parts of the question | An umbrella review |
| Time available | Less than about four months | Rapid methods, stated as such |
| Team | One person working alone | A narrower question, a second reviewer for a sample of records, or limitations stated plainly |
| Need for local information | The decision depends on what programs exist now in a place | Adding an environmental scan |
| Pace of new evidence | Studies appear quickly and the decision will be revisited | A scheduled update or a living review |
3.3 Labels Must Match Methods
Every design involves trade-offs between completeness, depth and speed. A systematic review gives the most complete and least biased answer to a focused question, at the cost of time. A rapid review gives a timely answer with a known, stated risk that a shortcut changed the conclusion. A scoping review gives breadth without judging the quality of what it maps, so it cannot say whether an intervention works. None of these is a lesser method when it is matched to its question and described honestly.
A common mistake
Munn and colleagues (2018) noted that scoping reviews are sometimes chosen because they seem easier than systematic reviews, and that authors then draw conclusions about effectiveness that a scoping review cannot support. A scoping review of loneliness interventions can report how many studies of walking groups measured loneliness. It cannot conclude that walking groups reduce loneliness, because it did not appraise or synthesize those studies for that purpose. The question asked, the methods used and the conclusions drawn have to match.
3.4 Interactive: The Review Type Decision Aid
The decision aid below asks the six questions from Table 1.3 and recommends a review type, with any complementary methods and cautions. Start with a practice scenario, read the recommendation and its reasons, and then change one answer to see what drives the choice. The aid simplifies a judgement that a review team should make with its librarian and the people who will use the review, so treat its recommendation as a starting point for that discussion.
The aid applies a small set of rules in order. If the purpose is to describe programs that exist now, it recommends an environmental scan. Otherwise, if a recent review of good quality already answers the question, it recommends using, appraising or updating that review. If the purpose is to summarize existing reviews, or the question is about effects and several systematic reviews already cover it, it recommends an umbrella review. For other questions about effects, it recommends a rapid review when less than about four months are available, a living systematic review when evidence is changing quickly and a full team is available, and a systematic review otherwise. Mapping questions lead to a scoping review, or a rapid scoping review under a short deadline. Explanatory questions lead to a realist review, and requests for a general overview lead to a narrative review.
The aid then adds an environmental scan whenever the decision needs information about current local programs, adds an updating plan when evidence is changing quickly, and warns when a design that needs two reviewers is planned by one person. Real choices involve more considerations than six questions can capture, including the expectations of the people commissioning the review and the conventions of the field.
3.5 Worked Example: Choosing a Design for Cedar Valley
The fictional Cedar Valley evidence team worked through the same questions. Its purpose is to map which community-based interventions have been evaluated and with what outcomes, which is a mapping question. Its preliminary search found older and broader reviews and none focused on British Columbia. Its deadline is twelve weeks, and its team is small. The planning team also needs to know which programs operate now in British Columbia and elsewhere in Canada, and how they are funded and evaluated. The evidence on loneliness interventions grows steadily, and the planning decision will be made once, before launch. The decision aid therefore recommends a rapid scoping review with an environmental scan, which is the design the team adopted. The tabs below record why the alternatives were set aside.
The planning team’s question asks which interventions have been evaluated and which outcomes were measured. A scoping review answers that kind of question, and rapid methods make it feasible in twelve weeks. The scan supplies what the literature lacks, namely current British Columbia programs, their operators and their funding. The team will state its shortcuts in the protocol and report them as limitations in the brief.
A systematic review would ask whether the interventions reduce loneliness and by how much. The planning team would value that answer, but a full systematic review with risk-of-bias appraisal and synthesis of effects would take far longer than twelve weeks for a topic this broad. The evidence brief will instead describe the outcomes measured and point to existing systematic reviews for evidence on effectiveness.
The existing reviews were published at least six years before the project. An umbrella review of them would describe the evidence only up to their search dates and would miss recent evaluations of community connector programs. The team will use the reviews as sources of studies.
Understanding why connector programs help some older adults more than others would be valuable when the program is being designed in detail. A realist review would take many months and needs a narrower intervention focus. The team notes it as a possible follow-up once the program has been running for a year.
Load the Home exercise by video call scenario and ask for a recommendation. Then change the time available to less than about four months, and then to more than a year with new evidence appearing quickly. Write two or three sentences explaining why the recommendation changed each time and what a decision-maker gains and loses with each design. Repeat the exercise with the Cedar Valley scenario, changing the purpose to a question about effects.
Summary of Section 3
A team should first check whether a relevant, current and trustworthy review already exists, using sources such as the Cochrane Library, Epistemonikos, PROSPERO, OSF Registries and Health Evidence. The choice of review type then follows from the purpose of the question, the existing evidence, the time and team available, the need for local information and the pace of new evidence. The label of a review must match its methods and the conclusions it draws. For the fictional Cedar Valley project, a mapping question, a twelve-week deadline, a small team and a need for current British Columbia information point to a rapid scoping review with an environmental scan.
Reflection
A school district in British Columbia asks two staff members, who have occasional help from a university librarian, to find out which school-based programs to prevent vaping among secondary school students have been evaluated and which outcomes were measured. The district also wants to know which programs schools in British Columbia currently use. The answer is needed in ten weeks. A preliminary search finds one systematic review of school-based tobacco prevention programs, published eight years ago, which did not consider vaping. New studies appear steadily, and the district will make a one-time decision before the next school year. Recommend a review type and any complementary method. Justify your choice with reference to five features: the purpose of the question, the existing evidence, the time and team available, the need for local information and the pace of new evidence. Then name one shortcut you would plan and the risk it carries.
I would recommend a rapid scoping review combined with an environmental scan. The purpose is to map which programs have been evaluated and which outcomes were measured, which is a mapping question suited to a scoping review rather than a review of effects. The existing review is eight years old and did not consider vaping, so it does not answer the question, although its included studies and reference list are a useful starting point. Ten weeks and a team of two with occasional librarian help call for rapid methods. The district also needs to know which programs British Columbia schools use now, which published research will capture only in part, so an environmental scan of school district websites, provincial resources and a short survey of school health staff should run alongside the review. New studies appear steadily, but the decision will be made once, so a living review is unnecessary; the brief should state its search date clearly.
One shortcut would be to search three databases instead of five or six. Its risk is missing evaluations indexed only in the databases left out, such as education databases. The team could reduce that risk by citation chasing from included studies and the older review, and it should report the shortcut as a limitation.
Minimum 20 characters required.
Question 1: A team finds an existing review that matches its question. Which feature matters most when judging whether that review is current?
Question 2: A health authority needs to know within ten weeks whether a specific intervention reduces hospital readmissions. No review addresses the question, and a small team is available. Which design does the reasoning in Section 3 recommend?
Question 3: A scoping review charted every evaluation of walking groups for older adults without appraising or synthesizing their effects. Which conclusion is consistent with its methods?
Question 4: In the decision aid, which change to the Cedar Valley scenario shifts the recommendation from a rapid scoping review to a scoping review?
Protocols and Registration with PROSPERO and OSF
Learning Objectives for this section
- Define a review protocol and explain how writing one before the review begins protects against bias, duplication and confusion within the team.
- List the main sections of a review protocol and recognize PRISMA-P as the reporting guideline for protocols.
- Compare registration in PROSPERO and in OSF Registries, and choose the right registry for a given review type.
- Explain how to handle amendments and deviations from a protocol, and how PRISMA 2020 asks reviews to report them.
- Outline a protocol and timeline for the fictional Cedar Valley rapid scoping review and environmental scan, and describe the first steps of a new review.
Introduction
A protocol is a written plan for a review, completed before the review begins. It states the question, the eligibility criteria, the sources to be searched, how records will be screened, what data will be extracted or charted, how studies will be appraised, how findings will be synthesized, and who will do each task. Registering the protocol makes it public and fixes its date. This section explains why protocols matter, what they contain, where they are registered, and how to handle the changes that every real review makes along the way. It closes with an outline of the Cedar Valley protocol and the first steps that any new review takes.
4.1 Why Write the Plan First
A review involves hundreds of small decisions: which databases to search, which study designs to include, which outcomes to record, how to group interventions. If those decisions are made after the team has seen the studies, they can be shaped, often without anyone intending it, by what the studies show. A team that notices that the most favourable studies happen to be recent might be tempted to restrict the review to recent years. A team that finds few studies of its planned primary outcome might switch to a secondary outcome with more striking results. Studies comparing review protocols with the published reviews have found changes to the specified outcomes that the authors did not explain (Kirkham, Altman & Williamson, 2010). A protocol written in advance makes such changes visible.
A protocol does three further jobs. It lets others comment on the methods while they can still be improved, which is why Cochrane and JBI peer review their protocols before the review starts. It tells other researchers that the review is under way, which reduces the duplicated effort that Chalmers and Glasziou (2009) counted as research waste. And it gives the team a shared reference, so that the evidence officer, the librarian and the intern on the Cedar Valley team apply the same criteria in the same way.
The same logic applies beyond reviews. HSCI 207 Lesson 6 Section 3.4 applies it to protocols for primary studies, and HSCI 230 Lesson 1 explains preregistration and registered reports, the open-science reforms that fix hypotheses and analysis plans before the data are seen. Neither course is required for this one, so this section teaches the logic in full for reviews.
4.2 What a Protocol Contains
The reporting guideline for protocols is PRISMA-P, the Preferred Reporting Items for Systematic review and Meta-Analysis Protocols (Moher et al., 2015; Shamseer et al., 2015). Its checklist covers administrative information (title, registration, authors, amendments and funding), an introduction (rationale and objectives) and methods (eligibility criteria, information sources, search strategy, study records and selection, data items, outcomes, risk of bias, synthesis and the assessment of certainty). Lesson 2 works through PRISMA-P item by item and shows how to write each part. Scoping review protocols follow the same structure, adapted to charting instead of effect estimation (Peters et al., 2020). For now it is enough to know that a protocol answers, in advance, the question of how each step will be done.
A protocol for a rapid review should also name its shortcuts. A statement such as “one reviewer will screen titles and abstracts, and a second reviewer will check a random 20 percent of excluded records” tells readers exactly where the review departs from full systematic methods. An environmental scan protocol, which is usually a shorter document, states which kinds of organizations will be included, how they will be identified, which websites and directories will be searched and on what dates, what the survey and interview guides will ask, and how the team will handle research ethics review.
4.3 Registering a Protocol: PROSPERO and OSF
Registration places a protocol, or a structured summary of it, in a public registry with a date stamp. A reader can then compare the registered plan with the published review. Registration records a plan, and the registry does not peer review the methods, so registration alone says nothing about the quality of a review.
PROSPERO
PROSPERO, the International Prospective Register of Systematic Reviews, was launched in 2011 by the Centre for Reviews and Dissemination at the University of York, with funding from the National Institute for Health and Care Research in the United Kingdom. Registration is free. The team completes a structured online form covering the question, the searches, the eligibility criteria, the outcomes and the planned synthesis, and receives a registration number that it cites in the published review. PROSPERO accepts systematic reviews with an outcome of direct relevance to human health, together with rapid reviews and umbrella reviews that use systematic methods. It does not accept scoping reviews. Registries expect registration at the protocol stage, before screening or data extraction has progressed, and PROSPERO publishes its exact timing rules on its website.
OSF Registries
The Open Science Framework (OSF), run by the Center for Open Science, is a free platform where researchers store and share study materials. Its registry, OSF Registries, accepts time-stamped, read-only registrations of any kind of study plan, including scoping reviews, environmental scans and other syntheses that PROSPERO does not take. A team can upload its full protocol document or complete one of OSF’s templates, including a general template for systematic reviews that scoping review teams also use. Each public registration receives a digital object identifier (DOI), a permanent link that can be cited.
| Feature | PROSPERO | OSF Registries |
|---|---|---|
| Operated by | Centre for Reviews and Dissemination, University of York | Center for Open Science |
| Review types accepted | Systematic reviews with a health-related outcome, including rapid and umbrella reviews | Any study plan, including scoping reviews and environmental scans |
| Format | A structured online form | An uploaded protocol or a choice of templates |
| What the team receives | A registration number and a public record | A time-stamped, read-only registration with a DOI |
| Checks by the registry | Staff check scope and completeness; the methods are not peer reviewed | No check of methods |
| Cost | Free | Free |
Protocols can also be published in journals. Cochrane protocols appear in the Cochrane Database of Systematic Reviews, JBI protocols in JBI Evidence Synthesis, and journals such as Systematic Reviews and BMJ Open publish protocols from any team. Journal publication adds peer review and takes longer, so many teams register first and publish the protocol later.
4.4 Amendments and Deviations
Few reviews follow their protocols exactly. A search may return far more records than expected, an outcome may turn out to be measured in ways the team did not anticipate, or a decision-maker may narrow the question. Changes are legitimate when they are made for a stated reason, recorded with a date, and reported. They become a problem when they are made silently, or when the reason is that the planned approach produced unwelcome results.
PRISMA 2020 asks every systematic review to give its registration information, to say where the protocol can be accessed (or that no protocol was prepared), and to describe and explain any amendments to the registered plan (Page et al., 2021). PRISMA-ScR asks scoping reviews for the same protocol and registration information (Tricco et al., 2018). Both registries allow a record to be updated, and OSF lets a team add a dated note or a new registration that links to the original. The simplest habit is an amendment log kept from the first day.
Suppose the Cedar Valley protocol plans to include studies of adults aged 65 and older, and during screening the team finds several evaluations that enrolled adults aged 60 and older and reported results for the whole group. The team could decide to include studies in which most participants were 65 or older, record the change, the date and the reason in its log, and report the change in the brief. Such a change is made for a stated reason connected to how studies report age, before any results are charted.
A review plans to report loneliness as its primary outcome. After extraction, the authors find that most studies show no change in loneliness and several show improved mood, and the published review presents mood as its main outcome without mentioning the change. A reader who compares the registration with the review sees a switched outcome with no explanation, and has good reason to doubt the conclusions.
Some published reviews have no protocol at all. PRISMA 2020 asks such reviews to say so. A reader should then treat every methodological choice as one that may have been made after the studies were seen, and weigh the conclusions accordingly.
4.5 Worked Example: The Cedar Valley Protocol Outline
The fictional Cedar Valley evidence team drafted the outline in Table 1.5 in its first week. Lesson 2 turns the question into formal eligibility criteria and completes the PRISMA-P sections, and later lessons fill in the search, screening, charting and scan methods. The team will register the completed protocol on OSF Registries, because the review is a scoping review.
| Protocol section | Cedar Valley entry (first draft) |
|---|---|
| Title | Community-based interventions for loneliness and social isolation among adults aged 65 and older: protocol for a rapid scoping review and environmental scan |
| Design and rationale | A rapid scoping review with an environmental scan, chosen because the question is a mapping question, the deadline is twelve weeks, the team is small and the planning team needs current British Columbia information (Section 3) |
| Review question | Which community-based interventions have been evaluated for reducing loneliness or social isolation among adults aged 65 and older, and with what outcomes? |
| Scan question | Which such programs currently operate in British Columbia and elsewhere in Canada, who runs them, and how are they funded and evaluated? |
| Preliminary PCC | Population: adults aged 65 and older living in the community. Concept: community-based interventions evaluated for loneliness or social isolation, and the outcomes reported. Context: community settings, with the geographic scope to be settled in Lesson 2. |
| Information sources | MEDLINE, Embase, CINAHL, PsycINFO and Web of Science; government, health authority and organization websites; reference lists of existing reviews; forward citation searching |
| Search and documentation | Built with the librarian, peer reviewed with PRESS and reported with PRISMA-S (Lessons 3 to 5); any use of AI tools logged and every citation verified (Lesson 6) |
| Selection (rapid shortcut stated) | Two reviewers pilot the criteria and dual screen the first 20 percent of titles and abstracts; one reviewer then screens the rest, and a second checks every excluded record (Lessons 2, 7 and 10) |
| Charting and appraisal | A piloted charting form recording design, population, intervention type, setting and outcomes; a brief appraisal of included evaluations so that the brief can describe the strength of the evidence (Lesson 8) |
| Synthesis | Descriptive mapping by intervention type and outcome, with tables and an evidence map (Lessons 9 and 10) |
| Scan methods | An inventory built from websites, program directories and contacts, a short survey of program leads and key informant interviews (Lesson 11) |
| Registration and reporting | OSF Registries; PRISMA-ScR for the review; an evidence brief for the planning team (Lesson 12) |
| Team and timeline | The evidence officer leads, the librarian designs and runs the searches, and the intern screens, charts and builds the scan inventory, over twelve weeks (Figure 1.6) |
| Amendments | A dated amendment log, with each change and its reason reported in the brief |
4.6 The First Steps of a New Review
The steps the Cedar Valley team took in its first week apply to any review. A researcher starting a new rapid scoping review and environmental scan usually works through four of them before any searching begins.
| Step | What it involves | Where this course teaches it |
|---|---|---|
| 1. State the question | Write the review question as one sentence, and the scan question if a scan is planned. | Lessons 1 and 2 |
| 2. Check for existing reviews | Search the Cochrane Library, Epistemonikos, PROSPERO and OSF Registries for reviews and registered protocols, and record for each one its title, year, date of last search and how closely it matches the question. | Lesson 1, Section 3.1 |
| 3. Justify the design | Run the question through the decision aid in Section 3, and write a short paragraph explaining why the chosen design suits the question, or what would need to change if the aid recommended a different design. | Lesson 1, Sections 3.2 to 3.4 |
| 4. Open the protocol | Create a protocol file with the section headings in Table 1.5, and fill in the title, design and rationale, and the questions. | Lesson 1, Section 4.5; Lesson 2 |
An early draft of this kind is worth sharing with a librarian or a colleague for comment, for the same reason that Cochrane and JBI peer review their protocols (Section 4.1): the methods can still be improved before any searching begins. Lesson 2 then turns the question into formal eligibility criteria and completes the remaining PRISMA-P sections.
Summary of Section 4
A protocol records how a review will be done before the work begins, which protects against decisions shaped by the results, invites early comment, reduces duplication and keeps the team consistent. PRISMA-P sets out what a protocol should contain, and Lesson 2 teaches it in detail. PROSPERO registers systematic, rapid and umbrella reviews with health-related outcomes, while scoping reviews and environmental scans are registered on OSF Registries. Amendments are acceptable when they are dated, explained and reported, as PRISMA 2020 and PRISMA-ScR require. The fictional Cedar Valley team has drafted its protocol outline and timeline, and any new review begins with the same steps.
Reflection
A student team is preparing a scoping review of peer support programs for family caregivers of people living with dementia. Answer three parts. (a) PROSPERO registers systematic, rapid and umbrella reviews with health-related outcomes and does not accept scoping reviews, while OSF Registries accepts time-stamped registrations of any study plan. Which registry should the team use, and what will registration provide? (b) Halfway through screening, the librarian points out that a database the team did not plan to search indexes many community health journals, and the team proposes to add it. (c) After charting, the team notices that studies of online programs reported few of the outcomes it hoped to see, and it proposes to drop online programs from the review. For parts (b) and (c), say whether the change is acceptable and how it should be recorded and reported, given that PRISMA 2020 and PRISMA-ScR ask reviews to report their protocol, their registration and any amendments.
(a) The team should register on OSF Registries, because PROSPERO does not accept scoping reviews. Registration gives a time-stamped, read-only public record of the plan with a DOI that can be cited, so readers can later compare the published review with what was planned.
(b) Adding the database is an acceptable amendment. The reason is methodological (better coverage of relevant journals), it is unrelated to any results, and it is made before charting. The team should search the new database with an adapted version of the full strategy across the same date range, enter the change with its date and reason in an amendment log, update the registration with a dated note, and report the added source in the methods and the flow diagram.
(c) Dropping online programs is a problematic deviation. The change is driven by what the studies found, so it would distort the map of the evidence and hide the very gap that a scoping review exists to show. The team should keep online programs and report that they measured few of the expected outcomes. If the team had a genuine scope reason to exclude them, it would need to state that reason, the date and the fact that the decision came after charting, so that readers could judge it.
Minimum 20 characters required.
Question 1: Which statement best describes the main purpose of writing and registering a protocol before a review begins?
Question 2: A team is planning a scoping review. Where should it register its protocol?
Question 3: During screening, a team realizes that many eligible studies enrolled adults aged 60 and older, and it decides to widen its age criterion. What should it do?
Question 4: In the Cedar Valley protocol outline (Table 1.5), which entry tells readers where the review departs from full systematic methods?
Final Assessment
Bringing It All Together
This lesson began with a problem: one study, or one expert’s summary, can point a decision in the wrong direction. Chance, bias, context and selective publication affect every single study, and narrative reviews that do not show how their studies were chosen can lag behind or distort the evidence. Systematic reviews respond with explicit, pre-specified methods, while still depending on the quality of the studies they include.
The family of evidence syntheses offers a design for each kind of question. Systematic reviews answer focused questions about effects, scoping reviews map what has been studied, rapid reviews streamline the process for decisions with deadlines, umbrella reviews compare existing reviews, realist reviews explain how and why interventions work, and living reviews stay current in fast-moving fields. Environmental scans describe the programs and services that exist now. Choosing among them starts with a check for existing reviews and depends on purpose, evidence, time, team, local information needs and the pace of new research.
For the fictional Cedar Valley evidence review, these considerations point to a rapid scoping review with an environmental scan, planned in a protocol and registered on OSF Registries. Any new review that must map published evidence and current programs within a short deadline begins with the same steps.
Key Takeaways from this lesson
- Single studies can mislead through chance, bias, context and selective publication, and a decision based on one study inherits all four risks.
- Traditional expert reviews can lag behind or misrepresent the evidence when their selection of studies is hidden, as Mulrow (1987) and Antman and colleagues (1992) showed.
- A systematic review answers a focused question with pre-specified, documented methods for searching, selecting, appraising and synthesizing studies, and meta-analysis is an optional statistical step within it.
- The family of reviews includes narrative, systematic, scoping, rapid, umbrella, realist and living reviews, and each suits a different kind of question.
- An environmental scan describes the programs, services and policies operating now in a place, using documents, websites, surveys and interviews, and it complements a review of published research.
- The SALSA framework (search, appraisal, synthesis and analysis) compares reviews by what they do, which is more reliable than the label they carry.
- Before choosing a design, a team should check whether a relevant, current and trustworthy review already exists.
- The choice of review type follows from the purpose of the question, the existing evidence, the time and team available, the need for local information and the pace of new evidence, and the conclusions drawn must match the methods used.
- A protocol written and registered before the review begins makes later changes visible; PROSPERO registers systematic, rapid and umbrella reviews, and OSF Registries accepts scoping reviews and scans.
- Amendments to a protocol are acceptable when they are dated, explained and reported, as PRISMA 2020 and PRISMA-ScR require.
Core Concepts Reviewed
Section 1: what the course teaches, the fictional Cedar Valley evidence review, chance, bias, context and selective publication, narrative reviews and cherry-picking, evidence synthesis, and the defining features and limits of systematic reviews.
Section 2: narrative, systematic, scoping, rapid, umbrella, realist and living reviews, environmental scans, the SALSA framework and the labels used for other review types.
Section 3: checking for existing reviews, the six features that guide the choice of review type, matching labels to methods, the decision aid and the Cedar Valley choice of a rapid scoping review with an environmental scan.
Section 4: the purpose and content of a protocol, PRISMA-P, registration in PROSPERO and OSF Registries, amendments and deviations, the Cedar Valley protocol outline and timeline, and the first steps of a new review.
The final reflection asks you to apply the whole lesson to the design of a new rapid scoping review and environmental scan.
Reflection
Imagine you are starting a new review on the following question: Which community-based interventions have been evaluated to improve access to primary care for adults who have recently immigrated to Canada, and which such programs operate in British Columbia? You have about twelve weeks, you work alone with occasional advice from a librarian and a colleague who can check a sample of your screening decisions, and the results are for a fictional health authority deciding whether to fund a patient navigator program. Write a short design rationale (about 200 words) that (1) explains why a synthesis of evidence is needed instead of relying on one well-known study, (2) names the review type and complementary method you would use and why, including one alternative you set aside, (3) describes how you would check for existing reviews, and (4) states where you would register your protocol and how you would handle changes to it.
A single study, however well known, describes one program, one population and one setting, and its result may reflect chance, bias or the local context. Studies with favourable results are also more likely to be published. The health authority needs to see the full range of evaluated interventions before funding one.
The question asks which interventions have been evaluated and which programs operate in British Columbia, so I would conduct a rapid scoping review with an environmental scan. The scoping review maps interventions, populations and outcomes; rapid methods fit the twelve-week timeline; and the scan captures current programs that journals rarely describe. I set aside a systematic review of effectiveness, because it would need more time and a second full reviewer, and the question is about mapping.
Before starting, I would search the Cochrane Library, Epistemonikos, Health Evidence, PROSPERO and OSF Registries for existing reviews and protocols, recording each one’s search date and fit with my question.
I would register the protocol on OSF Registries, since PROSPERO does not accept scoping reviews. I would keep a dated amendment log, make changes only for stated methodological reasons, and report every change and the single-reviewer screening (with my colleague checking a sample) as limitations.
Minimum 30 characters required.
Final Knowledge Assessment
Question 1: A health authority manager reads about one small trial in which a befriending program produced a large benefit. Which response reflects the reasoning of this lesson?
Question 2: Which feature most clearly distinguishes a systematic review from a narrative review?
Question 3: Which pairing of a review product and its reporting guideline is correct?
Question 4: Arksey and O’Malley (2005) are best known for which contribution?
Question 5: A provincial committee wants one overview across dozens of existing systematic reviews of interventions to increase vaccine uptake. Which design fits?
Question 6: Which situation is best suited to a living systematic review?
Question 7: Which of these is a typical rapid review shortcut, acceptable when it is stated and its consequences reported?
Question 8: A realist review expresses its findings mainly as which of the following?
Question 9: Why does the Cedar Valley team combine its review with an environmental scan?
Question 10: Which sources would you search to find whether a review on your topic is under way but not yet published?
Question 11: What does PRISMA 2020 ask a systematic review to report about its protocol?
Question 12: A team planned loneliness as its primary outcome, found mostly null results, and then presented mood as its main outcome without comment. What is the main problem?
Question 13: In Figure 1.2, three of the eight simulated intervals excluded zero and five did not, although every trial had the same true effect. What does this show about counting studies by statistical significance?
Question 14: Which reason given in this lesson explains why a well-conducted systematic review can still reach an uncertain conclusion?
Question 15: A rapid review protocol states that one reviewer will screen titles and abstracts and a second reviewer will check a random 20 percent of excluded records. Why should the protocol state this shortcut in advance?
Glossary: Key Terms, People & Frameworks
📚 Reference page, available throughout the lesson
This glossary defines the terms, frameworks and people introduced in Lesson 1, and you can search it at any time during the lesson.