# Lesson 1: Evidence Synthesis and the Family of Reviews

*Companion-podcast transcript, Sarah and Kiffer*

---

**Sarah:** Welcome back to Office Hours. I'm Sarah.

**Kiffer:** And I'm Kiffer. This is the first episode for Health Sciences two forty-one, Finding and Synthesizing Health Evidence, so we are starting at the beginning.

**Sarah:** Let's start with the course itself. What is it actually about?

**Kiffer:** It is about how to find, appraise and bring together the research on a question, so that someone can make a decision with the whole body of evidence in front of them. Over the course, students follow one review team as it plans and searches for a review, tests artificial intelligence tools, screens and charts studies, scans local programs and writes an evidence brief.

**Sarah:** Some listeners will have met systematic reviews already, in Health Sciences two thirty.

**Kiffer:** Yes, and the two courses fit together. Lesson two of Health Sciences two thirty teaches you to read and interpret systematic reviews and meta-analyses, including forest plots and the statistics of pooling. This course teaches you to do a review. Whenever pooling comes up, I will send you back to that lesson. Qualitative evidence synthesis gets a first look in Lesson ten here, and graduate students meet it in more depth in Health Sciences eight forty-one.

**Sarah:** And there's a running case.

**Kiffer:** There is. It's called the Cedar Valley evidence review, and it is fictional. Cedar Valley Health Authority is an invented health authority in British Columbia that serves about two hundred and ten thousand people. About forty-six thousand of them are aged sixty-five or older, and the region has twenty-four primary care clinics.

**Sarah:** What does the health authority want?

**Kiffer:** Its planning team wants to launch a community connector program, which is a form of social prescribing. Clinicians refer patients to a person who links them with community groups, activities and services. Before launching, the planners ask a small evidence team, made up of an evidence officer, a university librarian and a student intern, for a rapid scoping review and an environmental scan, and they want an evidence brief within twelve weeks. The intern is learning each step as the project needs it, which is roughly where our students are.

**Sarah:** And what are the actual questions?

**Kiffer:** The review question asks which community-based interventions have been evaluated for reducing loneliness or social isolation among adults aged sixty-five and older, and with what outcomes. The scan question asks which such programs currently operate in British Columbia and elsewhere in Canada, who runs them, and how they are funded and evaluated.

**Sarah:** All right, let's start Section one. Why do we need to synthesize evidence at all? Why not just find the best study?

**Kiffer:** One study, even a very good one, can point you in the wrong direction. Suppose the Cedar Valley planners found a well-publicized trial in which a befriending program reduced loneliness. There are at least four reasons to be careful before acting on it.

**Sarah:** Go through them.

**Kiffer:** The first is chance. Every study measures a sample of people, and a different sample would give a somewhat different answer. Small studies bounce around a lot. In the reading there is a figure with eight simulated trials of the same program. I set the true effect at a fall of half a point on a loneliness scale for every one of them.

**Sarah:** So they should all show the same thing.

**Kiffer:** They should all be estimating the same thing, and they don't land in the same place. One trial estimates a fall of more than a point. Another estimates a small rise in loneliness. Only three of the eight have confidence intervals that exclude zero.

**Sarah:** So if I were counting, I'd say it worked three times and failed five times.

**Kiffer:** Exactly, and you would be wrong, because every one of those trials had the same true effect. When you look at all eight together, the estimates cluster around a modest benefit, and the larger trials have the narrowest intervals. Lesson nine comes back to why counting studies by statistical significance misleads.

**Sarah:** That's chance. What's the second reason?

**Kiffer:** The second is bias, which is systematic error. If the people delivering the befriending calls also rate participants' loneliness, they may record more improvement than happened. If a study compares people who chose to join a program with people who didn't, it may be measuring how sociable the joiners already were. The important point is that bias doesn't shrink when a study gets larger.

**Sarah:** So a large study can be biased just as easily as a small one, and simply more precise about the wrong answer.

**Kiffer:** That's a fair way to put it. That is why a review has to appraise how each study was designed and run. Lesson eight teaches the tools for that.

**Sarah:** What's the third?

**Kiffer:** The third is context. A program tested among urban older adults with strong volunteer networks may behave differently in a rural community where the nearest group meets an hour's drive away. One study tells you about one setting. A set of studies can tell you whether a result travels. The fourth is selective publication. Studies with striking or statistically significant results are more likely to be published, published quickly and published somewhere people will read them. Authors also sometimes report only the outcomes that look good. So the studies you see are a biased sample of the studies that were done.

**Sarah:** Couldn't an expert just read widely and tell us what the evidence says? That's what review articles used to do.

**Kiffer:** That was the traditional approach, and there is good evidence that it can mislead as well. In nineteen eighty-seven, Cynthia Mulrow looked at review articles in major medical journals and found that very few of them explained how the authors had found, selected or appraised the studies they discussed.

**Sarah:** So readers had to trust the author.

**Kiffer:** They had to trust the author completely. In nineteen ninety-two, Elliott Antman and colleagues did something clever. They lined up the trials of treatments for heart attack in the order they were published and compared the accumulating evidence with what textbooks and review articles recommended at the time. Recommendations often lagged years behind the trials. Some effective treatments went unmentioned long after the evidence supported them, and some treatments kept being recommended after the trials suggested they didn't help.

**Sarah:** That's a sobering result.

**Kiffer:** It is, and the mechanism behind it is usually not bad faith. It's cherry-picking, which is selecting the studies that support what you already believe. Most of us do it without noticing. We remember the studies that agree with us and find them more convincing.

**Sarah:** So what's the remedy?

**Kiffer:** The remedy is procedural. You decide in advance which studies count, you search for all of them, you appraise them, and you report what you did. Archie Cochrane, a Scottish epidemiologist, argued in nineteen seventy-two that health care should be judged by evidence from randomized trials, and he later criticized his profession for not keeping organized, updated summaries of that evidence. Iain Chalmers and colleagues built such summaries for pregnancy and childbirth, and in nineteen ninety-three they founded what is now Cochrane, an international network that produces systematic reviews. The Campbell Collaboration followed in two thousand for social, educational and justice interventions. And JBI, which used to be called the Joanna Briggs Institute and is based at the University of Adelaide, developed methods for a wider range of evidence, including qualitative research and scoping reviews. The two main methods manuals for this course come from Cochrane and from JBI.

**Sarah:** Give me the definitions we'll be using.

**Kiffer:** Evidence synthesis is the process of bringing together the findings of many sources, with explicit methods, to answer a defined question in a form people can use. A systematic review is one kind of evidence synthesis. It tries to identify, appraise and synthesize all the studies that meet criteria set in advance, using explicit methods chosen to minimize bias, so that another team could repeat it.

**Sarah:** And meta-analysis? People use those two terms interchangeably.

**Kiffer:** They do, and they shouldn't. A meta-analysis is a statistical method that combines the numerical results of several studies into one estimate. It is usually done inside a systematic review, and it is optional. If the studies differ too much, the review synthesizes them in other structured ways, which Lesson nine covers.

**Sarah:** Is a systematic review always trustworthy, then?

**Kiffer:** No, and the reading spends some time on its limits. A review can't be stronger than the studies it includes. A lot of published reviews are redundant or poorly done; John Ioannidis wrote about the mass production of them in two thousand and sixteen. Every review is out of date from the day its search ends. And evidence informs decisions without making them.

**Sarah:** Say more about that last one.

**Kiffer:** The National Collaborating Centre for Methods and Tools, at McMaster University, describes evidence-informed decision making in public health as combining research evidence with local context, community and political preferences, and the resources available. The Cedar Valley planners will weigh the review alongside their budget, their partners and what older adults in the region tell them. The review's job is to make the research part of that judgement complete and honest.

**Sarah:** Let's move to Section two. There seem to be a lot of different kinds of review.

**Kiffer:** There are. In two thousand and nine, Maria Grant and Andrew Booth described fourteen types in common use, and later work catalogued several dozen named types. The labels aren't used consistently, so I'd rather define each type by what it's for and what it does. To compare them, Grant and Booth used a framework they called SALSA. It stands for search, appraisal, synthesis and analysis. Search is how sources are found. Appraisal is how their quality is judged. Synthesis is how findings are combined, and analysis is what the review concludes about the evidence as a whole. If you read a review's methods with those four steps in mind, you can usually tell what kind of review it really is, whatever it calls itself.

**Sarah:** Start with the oldest type.

**Kiffer:** The oldest is the narrative review, sometimes called a traditional or literature review. It's an author's overview of a topic, organized around themes or a history, and it is good for orientation and new ideas. Their limitation is that the selection of studies is usually undocumented, so readers can't tell how complete they are.

**Sarah:** Then comes the systematic review.

**Kiffer:** A systematic review answers a focused question, often about whether an intervention works and how much. It finds all eligible studies, appraises their risk of bias, and synthesizes the results with methods set in advance. It is reported following PRISMA, which stands for Preferred Reporting Items for Systematic reviews and Meta-Analyses, in its twenty twenty version. And it takes time. Borah and colleagues used records from the PROSPERO register and estimated that the average review took more than a year from registration to publication.

**Sarah:** More than a year is a long time for a health authority.

**Kiffer:** It is, which is part of why the other types exist. The scoping review is next. A scoping review maps the extent, range and nature of the evidence. It asks what has been studied, how a concept has been defined, which outcomes have been measured, and where the gaps are. Hilary Arksey and Lisa O'Malley published the first framework in two thousand and five, with five stages: identifying the question, finding relevant studies, selecting them, charting the data, and collating and reporting the results. They also suggested an optional consultation with interest holders. Levac and colleagues refined it in two thousand and ten, and JBI now publishes the main guidance.

**Sarah:** How is it different from a systematic review in practice?

**Kiffer:** The search is just as thorough and just as well documented. The difference is what happens next. A scoping review charts and describes the studies, and it usually doesn't appraise their risk of bias or pool their results. Its question is framed with population, concept and context, which we'll abbreviate as PCC from Lesson two onward, and it's reported with the scoping review extension of PRISMA, published by Andrea Tricco and colleagues in two thousand and eighteen.

**Sarah:** What about the rapid review?

**Kiffer:** A rapid review streamlines or leaves out some steps of a systematic review so that it can be done faster and with fewer resources. Typical shortcuts are searching fewer databases, limiting dates or languages, having one person screen while a second person checks a sample, and simplifying appraisal. The Cochrane Rapid Reviews Methods Group has published guidance on which shortcuts are least likely to change the conclusions.

**Sarah:** That sounds like cutting corners.

**Kiffer:** It's cutting named corners, in the open. A rapid review is acceptable when it states each shortcut and reports what it might have cost. A rapid scoping review applies the same idea to a mapping question, and it's the design the Cedar Valley team uses.

**Sarah:** What else is in the family?

**Kiffer:** There are three more types. An umbrella review, sometimes called an overview of reviews, treats systematic reviews as its units. It searches for reviews on a broad topic, appraises them and compares what they found. It's useful when dozens of reviews exist, each on a different intervention, and someone needs a single view across them.

**Sarah:** Then there are realist reviews, which I've always found confusing.

**Kiffer:** They ask a different question. A realist review asks what works, for whom, in what circumstances and why. Ray Pawson and colleagues developed it for complex social and policy interventions, whose effects depend on how people respond to them in particular settings. The answers take the form of context, mechanism and outcome configurations. In this context, this mechanism, meaning the way people respond to what the program offers, produces this outcome.

**Sarah:** And what about living reviews?

**Kiffer:** A living systematic review is updated continually as new evidence appears. Julian Elliott and colleagues proposed the idea in two thousand and fourteen. The team reruns the search at a set interval and revises the findings when new studies change them. Living reviews were used widely for treatments and vaccines during the COVID-nineteen pandemic. They need a standing team, so they are uncommon outside well-funded programs.

**Sarah:** You've left out the environmental scan.

**Kiffer:** I've kept it apart because it's a close relative that studies current practice. An environmental scan collects and interprets information about programs, services, policies and organizations that exist now in a defined place. It began in business and organizational planning and was taken up in health services and public health.

**Sarah:** Why does Cedar Valley need one?

**Kiffer:** A literature review can't tell the planners which programs are running in British Columbia today. Many of those programs have never been described in a journal. A scan gathers that information from organization websites, program directories, annual reports and government documents, and from surveys and interviews with the people who run the programs.

**Sarah:** What about all the other labels? I've seen integrative reviews, mapping reviews, systematic literature reviews.

**Kiffer:** Some of those describe distinct methods that later lessons cover. Others are loose labels. A paper called a systematic literature review might meet every standard, or it might be a narrative review with a database search attached. The reading has a short exercise where you name the type of four reviews from their methods alone.

**Sarah:** Let's turn to Section three. How does a team choose?

**Kiffer:** The first step is to check whether someone has already answered the question. Producing a review that duplicates a recent, sound one is research waste, the term Iain Chalmers and Paul Glasziou used for research that is unnecessary, badly designed or never usefully reported.

**Sarah:** Where do you look?

**Kiffer:** The Cochrane Library, Epistemonikos, which is a free database of systematic reviews, and Health Evidence, which is hosted by the National Collaborating Centre for Methods and Tools and collects reviews relevant to public health. You also look in the registries, PROSPERO and the Open Science Framework, for reviews that are under way. An afternoon is usually enough for a preliminary check.

**Sarah:** What do you do if you find something?

**Kiffer:** You ask three questions. Does it answer the same question, with the same population, interventions, outcomes and setting? Is it current, judged by the date of its last search, which can be a year or more before publication? And is it trustworthy? A tool called AMSTAR two, which stands for A Measurement Tool to Assess systematic Reviews, helps with that last one.

**Sarah:** What did the Cedar Valley intern find?

**Kiffer:** She found several reviews on loneliness and social isolation. There's a meta-analysis of interventions to reduce loneliness by Masi and colleagues from two thousand and eleven, an integrative review of interventions for older people by Gardiner and colleagues from two thousand and eighteen, and a scoping review of reviews on interventions for older adults by Fakoya and colleagues from two thousand and twenty.

**Sarah:** So why not just use those?

**Kiffer:** Each was published at least six years before the project, and none describes the programs operating in British Columbia or how Canadian programs are funded. So the team decided a new review was justified, and that those reviews would be an early source of studies through their reference lists.

**Sarah:** Once you know what exists, what decides the type?

**Kiffer:** Six features of the project decide it. The first is the purpose of the question: are you asking about effects, mapping what has been studied, explaining how something works, summarizing existing reviews, describing current practice, or writing a general overview? The second is what the check for existing reviews found. The third is the time available, and the fourth is who will do the work.

**Sarah:** What are the other two?

**Kiffer:** They add to the main design. If the decision needs to know which programs operate now in a place, you add an environmental scan. If new evidence is arriving quickly and decisions will be revisited, you plan scheduled updates or a living review.

**Sarah:** Can you get it wrong?

**Kiffer:** It happens quite easily. Zachary Munn and colleagues wrote guidance in two thousand and eighteen on choosing between systematic and scoping reviews, partly because scoping reviews were being chosen because they looked easier. Authors would then draw conclusions about effectiveness that their methods couldn't support.

**Sarah:** Give me an example.

**Kiffer:** A scoping review of loneliness interventions can tell you how many studies of walking groups measured loneliness. It can't tell you that walking groups reduce loneliness, because it didn't appraise those studies or synthesize their effects. The question, the methods and the conclusions have to match.

**Sarah:** The lesson has a decision aid for this.

**Kiffer:** It does. It asks the six questions as multiple-choice items and recommends a review type, with any complementary methods and cautions. There are seven practice scenarios, including Cedar Valley and a home exercise program delivered by video call.

**Sarah:** What should students do with it?

**Kiffer:** Load a scenario, read the recommendation and its reasons, and then change one answer at a time. If you give the home exercise scenario a short deadline, it switches to a rapid review. If you give it more than a year and fast-moving evidence, it suggests a living review.

**Sarah:** Is the aid always right?

**Kiffer:** It's a simplification. Real choices involve more than six questions, including what the people commissioning the review expect and the conventions of the field. I'd treat its recommendation as the start of a conversation with your librarian and the people who will use the review.

**Sarah:** So what did Cedar Valley choose?

**Kiffer:** The purpose is mapping. The existing reviews are older and broader. The deadline is twelve weeks, the team is small, and the planners need current information about British Columbia programs. All of that points to a rapid scoping review with an environmental scan, and that's what the aid recommends.

**Sarah:** Did they consider anything else?

**Kiffer:** They did, and the reading records why each alternative was set aside. A full systematic review of effectiveness would take far longer than twelve weeks for a topic this broad. An umbrella review of the existing reviews would describe the evidence only up to their search dates and miss recent evaluations of connector programs. A realist review would be valuable later, once the program has run for a year, but it would take many months.

**Sarah:** Section four is about protocols. Why write the plan before you start? It seems like you'd learn a lot once you see the studies.

**Kiffer:** You do, and that's exactly the danger. A review involves hundreds of decisions: which databases, which designs, which outcomes, how to group interventions. If you make those decisions after seeing the studies, the results can shape them without anyone meaning them to.

**Sarah:** Can you give an example?

**Kiffer:** Suppose a team plans to make loneliness its main outcome, finds that most studies show no change in loneliness, and notices that several show improved mood. It might quietly present mood as the main outcome. Studies comparing review protocols with published reviews have found unexplained changes to outcomes of that kind. A protocol written in advance makes such a switch visible.

**Sarah:** What else does a protocol do?

**Kiffer:** It lets others comment on the methods while they can still be improved. It tells other researchers the review is under way, which reduces duplication. And it gives the team one shared reference, so the evidence officer, the librarian and the intern all apply the same criteria in the same way.

**Sarah:** What goes into one?

**Kiffer:** The reporting guideline is PRISMA-P, the version of PRISMA for protocols, and Lesson two goes through it item by item. Broadly, a protocol covers administrative details, the rationale and objectives, and the methods: eligibility criteria, sources, the search, how records will be selected, what data will be collected, how studies will be appraised and how findings will be synthesized.

**Sarah:** Does a rapid review need a protocol too?

**Kiffer:** Yes, and it should name its shortcuts in advance. A sentence like one reviewer will screen titles and abstracts and a second will check a random sample of excluded records tells readers exactly where the review departs from full systematic methods. A scan protocol is usually shorter, but it still says which organizations count, how they'll be found, what the survey and interviews will ask, and how ethics review will be handled.

**Sarah:** And then you register it.

**Kiffer:** Registration puts the plan in a public registry with a date stamp. The best-known registry is PROSPERO, the international prospective register of systematic reviews, launched in two thousand and eleven and run by the Centre for Reviews and Dissemination at the University of York. It's free, it registers systematic reviews with outcomes relevant to human health, including rapid and umbrella reviews, and it doesn't accept scoping reviews.

**Sarah:** So where does a scoping review go?

**Kiffer:** It goes to the registries of the Open Science Framework, which is run by the Center for Open Science. They accept time-stamped, read-only registrations of any kind of study plan, including scoping reviews and environmental scans. Each public registration gets a digital object identifier, a permanent link that can be cited.

**Sarah:** Does registering mean the methods have been checked?

**Kiffer:** No. Neither registry peer reviews the methods. Registration tells a reader what was planned and when. It says nothing on its own about whether the plan was good. Cochrane and JBI protocols do go through peer review, and journals such as Systematic Reviews and BMJ Open publish protocols from any team.

**Sarah:** What happens when a team has to change its plan? That must happen all the time.

**Kiffer:** It happens almost always. A search returns far more records than expected, or an outcome turns out to be measured in ways nobody anticipated. Changes are legitimate when they have a stated reason, a date and a place in the final report. They become a problem when they're made silently, or when the real reason is that the plan produced results the team didn't like.

**Sarah:** How do readers find out?

**Kiffer:** PRISMA twenty twenty asks every systematic review to give its registration details, say where the protocol can be found, and describe and explain any amendments. The scoping review extension asks for the same protocol and registration information. My advice to students is to keep an amendment log from the first day of the project.

**Sarah:** Tell me about the Cedar Valley protocol.

**Kiffer:** The team drafted an outline in its first week. It names the design and why it was chosen, gives both questions, sets out a preliminary population, concept and context, lists five databases plus websites and citation chasing, and states its screening shortcut. It plans a charting form, a brief appraisal of included evaluations, the scan methods and a twelve-week timeline. Because it's a scoping review, the team will register it with the Open Science Framework.

**Sarah:** How does all of that fit into twelve weeks?

**Kiffer:** The protocol sets out a timeline. Protocol and registration take the first two weeks. The librarian builds the search and has it peer reviewed over weeks two to four, and the searches run in weeks four and five. Screening fills weeks five to eight, with charting and appraisal overlapping from week seven. The environmental scan runs alongside all of that, from week three to week ten.

**Sarah:** And the last few weeks?

**Kiffer:** The review and the scan come together for synthesis in week nine, drafting runs to week eleven, and the brief is reviewed and delivered in weeks eleven and twelve. A timeline that tight is one reason the protocol names its shortcuts at the start.

**Sarah:** Say someone is starting a new review tomorrow. Where do they begin?

**Kiffer:** There are four things. Write the question as one sentence. Search the review databases and registries for existing reviews and protocols, and record each one's search date and how closely it matches. Run the question through the decision aid and write about a hundred and fifty words justifying the design. And start a protocol file with the headings from the Cedar Valley outline.

**Sarah:** And then get some comments on it.

**Kiffer:** Share it with a librarian or a colleague while the methods can still change. Lesson two then turns all of this into a precise question and a full protocol.

**Sarah:** Thanks, Kiffer.

**Kiffer:** Thanks, Sarah. See you next week.