Introduction

In Lesson 1, the evidence team for the fictional Cedar Valley Health Authority decided that the planning team's request called for two products: a rapid scoping review of published and grey literature, and an environmental scan of programs that currently operate in British Columbia and elsewhere in Canada. The planning team is preparing to launch a community connector (social prescribing) program for older adults and wants to know which community-based interventions have been evaluated for reducing loneliness or social isolation among adults aged 65 and older, and with what outcomes. Before anyone opens a database, that request has to be turned into a question precise enough to decide which studies belong in the review and which do not.

If you have taken HSCI 207 Lesson 2, you used PICO, PECO and SPIDER to structure a question for a single study. This section shows how the same frameworks work when the question is asked of a body of literature, and it applies the Population, Concept and Context (PCC) framework, which HSCI 207 Lesson 2 Section 2.3 introduced, to scoping reviews, following the JBI guidance (Peters et al., 2020). The Cedar Valley question is written in full at the end of Section 1.3.

1.1 What a Review Question Does

A research question for a single study guides the choice of design, sample and measures. A review question is asked of studies that other people have already conducted, so its work is different. It does three jobs at once. First, each element of the question becomes an eligibility criterion, which is a rule that decides whether a study is included in the review. Second, each element becomes a concept in the search strategy, with its own list of synonyms and subject headings. Third, the question tells the team which information to record from each included study, because the answer is assembled from those records.

Because the question is reused in all three places, a vague question produces problems that appear much later. If the question says "older adults" without an age, two reviewers will make different screening decisions about a study of people aged 55 and older, the librarian will not know whether to search for "middle-aged" as well as "aged", and the charting form will have no rule for recording mixed-age samples. Time spent on the question at the start is recovered many times over during screening.

A good review question has four properties. It can be answered from studies or documents that exist, it is specific enough to set boundaries, it fits the review type (a scoping review asks what evidence exists, while a systematic review of effectiveness asks whether an intervention works), and it matches the decision that the people who requested the review need to make.

1.2 PICO and PECO in a Review

The PICO framework (Richardson et al., 1995) divides a question about the effects of an intervention into the Population, the Intervention, the Comparison and the Outcome. It is the standard framework for systematic reviews of effectiveness, and the Cochrane Handbook for Systematic Reviews of Interventions organizes its guidance on eligibility around it. A PICO question for a systematic review might read: "Among community-dwelling adults aged 65 and older (P), do group-based social activity programs (I), compared with usual care or no program (C), reduce loneliness measured with a validated scale (O)?"

Two common extensions add elements. PICOS adds the study designs that are eligible (S), for example randomized controlled trials only. PICOTS adds the timing of outcome measurement (T) and the setting (S). These extensions are useful when the review needs explicit rules on design, follow-up time or setting, and each added letter is simply another eligibility criterion.

The PECO framework replaces the intervention with an exposure, which is a characteristic or circumstance that people experience without it being assigned by a researcher. Morgan et al. (2018) described PECO for reviews of environmental and other exposures. A PECO question suits reviews of risk factors or causes, for example: "Among adults aged 65 and older (P), is living alone (E), compared with living with others (C), associated with emergency department use (O)?"

When PICO or PECO is used for a review, the elements tend to be broader than in a single study. A trial recruits one population and tests one intervention; a review gathers trials with somewhat different populations and somewhat different interventions. The question therefore names a class of interventions ("group-based social activity programs") and the eligibility criteria then define the edges of that class.

1.3 PCC for Scoping Reviews

A scoping review maps the extent, range and nature of the evidence on a topic, identifies the kinds of evidence available and identifies gaps (Arksey and O'Malley, 2005; Munn et al., 2018). It does not usually estimate whether an intervention works, so a comparison element is often unnecessary, and its outcomes of interest are frequently part of what is being mapped. The JBI guidance for scoping reviews therefore recommends the PCC framework (Peters et al., 2020).

A scoping review question is usually broad, so JBI recommends adding sub-questions that state what will be mapped. The sub-questions decide the headings of the charting form, so they should be written before the search starts.

The scoping review is paired with an environmental scan, and the scan question can be written with the same framework. Here the population is a set of organizations: the scan asks which organizations (P) currently deliver community-based programs for socially isolated older adults, how those programs are designed, governed, funded and evaluated (C), in British Columbia and elsewhere in Canada (C). Writing both questions in parallel helps the team align the charting form for the review with the data template for the scan, which Lesson 11 develops.

1.4 SPIDER and Other Frameworks

For a review, the choice of framework also has a consequence for the search. Methley et al. (2014) compared searches built from PICO, PICOS and SPIDER for qualitative reviews and found that the SPIDER search retrieved fewer irrelevant records but also missed relevant studies that the PICO search found. A reasonable approach is to use SPIDER to frame a qualitative question and then build the search broadly enough that it does not depend on how authors labelled their design.

1.5 Choosing a Framework and Avoiding Common Problems

The framework follows from the question type, and the question type follows from the decision the review serves. The table summarizes the usual pairings.

Four problems recur in student questions. The first is forcing PICO onto a mapping question, which produces an invented comparison ("compared with nothing") that then excludes uncontrolled evaluations the team actually wants to find. The second is the compound question, in which two questions are joined with "and", for example asking which interventions exist and also which is most cost-effective. Each part needs different evidence and often a different review type, so the second part is better set aside or answered separately. The third is an element that cannot be observed in study reports, such as "programs that were well implemented", which no screener can judge from an abstract. The fourth is confusing the question's elements with search terms. The question says "loneliness"; the search will need many synonyms and subject headings, which Lesson 4 builds.

With the question written, the team's next task is to define each element precisely enough that two people screening the same study will make the same decision. Section 2 turns the Cedar Valley PCC question into a full set of eligibility criteria.

Introduction

A question tells the team what it wants to know. Eligibility criteria tell a screener, record by record, whether a study belongs in the review. In the fictional Cedar Valley review, the student intern and the evidence officer will each read hundreds of titles and abstracts, and they will only reach the same decisions if the criteria leave little room for interpretation. This section explains how to write criteria with that property, presents the full Cedar Valley eligibility table, and shows how the team tested the criteria before screening began.

2.1 Inclusion Criteria, Exclusion Criteria and Two Kinds of Characteristic

Inclusion criteria state the characteristics a study must have to be included. Exclusion criteria state features that remove a study that would otherwise qualify. The two lists should do different work. An exclusion criterion that simply restates an inclusion criterion in the negative ("exclude studies of people under 65") adds length without adding information. A useful exclusion criterion carves out a specific case that the inclusion criteria would otherwise admit, such as studies of older adults that were conducted only among residents of long-term care homes.

The PRISMA-P checklist, which Section 3 describes, separates two kinds of characteristic (Moher et al., 2015). Study characteristics describe the research itself: the population, the intervention or concept, the setting, the design and the outcomes. Report characteristics describe the document in which the research appears: its publication date, language and publication type. Limits on report characteristics are often set for practical reasons, and they should be justified because each one can exclude relevant evidence. A language limit, for example, removes studies that would otherwise answer the question, so a team that applies one should state the reason and acknowledge the limitation.

2.2 Properties of Criteria That Can Be Applied Consistently

Criteria are applied consistently when two trained reviewers, working independently, reach the same decision about the same record most of the time. Six properties make that more likely. Select each card for a description and a Cedar Valley example.

2.3 Operational Definitions for the Cedar Valley Review

An operational definition states how a concept will be recognized in the studies being screened. The Cedar Valley team identified five terms that reviewers were likely to read differently and wrote a definition for each.

Loneliness and social isolation are related but distinct. A person can live alone with few contacts and not feel lonely, and a person can feel lonely while surrounded by family. The planning team wanted both, because a community connector program might reduce isolation by linking people to activities without necessarily changing how lonely they feel. The definitions above allow the review to report the two outcomes separately.

2.4 Worked Example: The Cedar Valley Eligibility Criteria

The table is the complete set of criteria that the team placed in its protocol. The final column holds the decision rules that came out of piloting and discussion; screeners consult it when a record is borderline.

Three features of the table deserve comment. The comparison element of PICO is absent, because the review maps evaluations of any design and does not require a control group. The outcome criterion is phrased separately for quantitative and qualitative studies, so that a qualitative evaluation of a befriending service can qualify through participants' accounts. And the date and language limits are report characteristics that the team has chosen for practical reasons; the protocol states those reasons and the review will report them as limitations.

2.5 Applying the Criteria at Two Stages

Screening usually happens in two stages, which Lesson 7 teaches in detail. At the title and abstract stage, reviewers exclude only records that clearly fail at least one criterion; anything that might be eligible, or that lacks the information to decide, goes forward. At the full-text stage, reviewers read the complete report and apply every criterion strictly. For each excluded report they record a single reason, taken from an ordered list, so that the reasons can be counted in the PRISMA 2020 flow diagram.

The order of the list matters less than the fact that it is fixed and written down. Some teams place quick, objective report characteristics, such as publication type, date and language, first. The Cedar Valley team placed the study characteristics first, in the order of its screening questions (population, intervention and setting, outcome and country), and the report characteristics last, so that its flow diagram shows how many reports failed on the substance of the question. If a report fails on both language and population, every reviewer records "population", and the flow diagram counts it once under that reason.

2.6 Testing the Criteria Before Screening

A pilot (sometimes called a calibration exercise) tests the criteria on a small set of real records before full screening. Both reviewers screen the same records independently, compare decisions and discuss every disagreement. The aim of the discussion is to find the cause of each disagreement in the wording of the criteria and to revise the wording.

The eligibility table is one part of a larger plan. Section 3 places it inside a review protocol written to the PRISMA-P checklist, alongside the information sources, the screening and charting processes and the plan for synthesis.

Introduction

Lesson 1 introduced review protocols and the registries where they are made public, PROSPERO and the Open Science Framework (OSF). This section looks inside the protocol. A protocol is the written plan for a review: it states the question, the eligibility criteria, the sources and search, how records will be screened and data collected, and how the findings will be synthesized. The most widely used guide to its content is the Preferred Reporting Items for Systematic review and Meta-Analysis Protocols, known as PRISMA-P (Moher et al., 2015), with an explanation and elaboration paper that gives examples for every item (Shamseer et al., 2015). The section ends with the fictional Cedar Valley team's protocol, item by item, with annotations.

3.1 Why Write the Protocol First

A review involves many decisions, and many of them could be made after the team has seen the studies. A team that sees that one outcome shows favourable results and another does not could, without meaning to, emphasize the first. A team that finds an awkward study could adjust a criterion to exclude it. Writing the methods down before searching makes such data-driven decisions visible, because any later change appears as a difference between the protocol and the final review. Shamseer et al. (2015) also describe practical benefits. Writing a protocol forces the team to plan the work and estimate its size, a public protocol lets other teams see that a review is under way and avoid duplicating it, and readers of the final review can check that it did what it set out to do.

3.2 The PRISMA-P 2015 Checklist

PRISMA-P contains 17 numbered items, several divided into sub-items, for 26 elements in total. They fall into three parts: administrative information, the introduction and the methods.

The administrative information identifies the review: its title, its registration, its authors and their contributions, how amendments will be handled, and who funds it and with what role. The introduction gives the rationale, which places the review in the context of what is already known, and the objectives, which state the question explicitly. The methods occupy most of the checklist. Items 8 to 10 cover what is eligible and how it will be found. Item 11, study records, covers how records will be managed (11a), how studies will be selected (11b) and how data will be collected (11c). Items 12 and 13 list the data items and outcomes. Items 14 to 17 cover risk of bias in individual studies, synthesis, assessment of meta-biases such as publication bias, and how confidence in the body of evidence will be judged, for example with GRADE.

3.3 Adapting PRISMA-P to a Rapid Scoping Review

PRISMA-P was written for systematic reviews, and several of its methods items assume that the review will estimate effects. A scoping review maps evidence, so teams that use PRISMA-P for a scoping review protocol adapt those items and say so. The reporting guideline for the completed scoping review is PRISMA-ScR (Tricco et al., 2018), which Lessons 10 and 12 use, and it treats critical appraisal of individual sources as optional. The table summarizes the adaptations the Cedar Valley team made.

3.4 Worked Example: The Annotated Cedar Valley Protocol

Each panel below shows the protocol text the Cedar Valley team wrote, followed by an annotation that explains the choice or points to what a reader should check. The project and its team are fictional; the published sources cited in the protocol are real.

The protocol also has a short appendix describing the environmental scan: the scan question from Section 1.3, the planned sources (organization websites, a program inventory, a short survey and key informant interviews) and a draft data template. Lesson 11 develops those methods.

3.5 Recording Amendments

Protocols change. The aim of a protocol is to make changes visible, and a reader can only judge a change if it is dated, described and explained, and if the reader knows whether it was made before or after the team saw the studies. An amendment log records this information. The two entries below are hypothetical illustrations of how the Cedar Valley team would record changes.

The first change carries little risk. The second was made after the team had read studies and could change which studies are included, so it requires the most careful handling: applying it to every record, explaining it, and allowing readers to see which studies it admitted. PRISMA 2020 asks the final review to describe any amendments to the information provided in the registration or protocol, which Lesson 12 covers. The amendments the Cedar Valley team actually made, beginning with the change to full dual screening at the end of week 5, are described in Lessons 7 to 10 and reported in Lesson 12.

A protocol records decisions that the evidence team does not make alone. Section 4 describes how the Cedar Valley team worked with the librarian and with interest holders to settle the scope that the protocol now records.

Introduction

The scope of a review is the set of boundaries that determine what it will and will not cover: the question, the eligibility criteria, the sources, the outputs and the timeline. Sections 1 to 3 presented the fictional Cedar Valley question, criteria and protocol as finished products. In practice, each was settled through conversations with two groups. The first is the librarian, who knows how the literature is organized and how large a search is likely to be. The second is the interest holders: the people and organizations who will use the review, deliver the program it informs, or be affected by that program. This section describes both relationships and shows the decision log the Cedar Valley team kept.

4.1 What a Health Sciences Librarian Contributes

A health sciences librarian, sometimes called an information specialist, is trained in the structure of bibliographic databases, controlled vocabularies and search methods. The Cochrane Handbook recommends that review teams work with an information specialist when designing searches, and studies of published reviews support that advice. Rethlefsen et al. (2015) found that systematic reviews in general internal medicine with a librarian co-author reported search strategies of higher quality than reviews without one. Select each card to see what a librarian contributes at each stage.

Because a librarian may design the search, write the search methods and approve the final text, the question of authorship should be raised at the first meeting. The criteria of the International Committee of Medical Journal Editors (ICMJE) describe authorship as substantial contribution to the work, drafting or critical revision, approval of the final version and accountability for it. A librarian who meets those criteria is a co-author; a librarian who gives brief advice is acknowledged. In Cedar Valley the librarian is named in protocol item 3 as the author responsible for the searches.

4.2 Preparing for a Librarian Consultation

A consultation is most useful when the team arrives with drafts that the librarian can test. The Cedar Valley intern prepared the following before the first meeting, and the list works as a template for any review.

The studies known to be relevant are sometimes called benchmark studies. They are useful because they give an objective test of the search, and Lesson 4 uses them for that purpose. They should be chosen before the search is built, since studies picked after seeing the results would make the test less informative.

4.3 The Preliminary Search

A preliminary search (sometimes called a scoping search) is a quick, unsystematic search run while the question is still being refined. It has four purposes. It checks whether a review on the same question already exists or is registered. It estimates how much literature the full search will retrieve. It shows the words authors use, which feed the search strategy. And it supplies records for piloting the eligibility criteria.

4.4 Interest Holders and the Scope of a Review

Interest holders are people and organizations with an interest in the review or in the decisions it informs. (This series uses the term "interest holders", which a number of public health agencies now prefer because the older term has colonial associations for some Indigenous communities.) For the Cedar Valley review they include the planning team that commissioned it, primary care clinicians who would refer patients to the program, community organizations and connectors who would deliver it, older adults who might take part, and the health authority's Indigenous health team.

Involvement of interest holders in scoping reviews has a long history. Arksey and O'Malley (2005) included an optional consultation exercise as the sixth stage of their scoping review framework, and Levac, Colquhoun and O'Brien (2010) recommended that consultation be treated as a required part of the method. Involvement can take different forms. The Spectrum of Public Participation of the International Association for Public Participation (IAP2) distinguishes five levels, from informing people through consulting, involving and collaborating to empowering them to decide. A review usually places different interest holders at different levels.

Interest holders can improve a review in several ways. They can say which decisions the review must inform, which keeps the question relevant. They can name outcomes that matter to the people affected, which researchers may not have measured. They can supply local vocabulary and sources, such as the names of programs and the organizations that run them. And they can comment on how findings should be presented. Some decisions, however, belong to the review team alone: whether individual studies meet the criteria, how they are appraised and what the evidence shows. Keeping those decisions independent protects the review from the interests of those who commissioned it, which is why the Cedar Valley protocol states the sponsor's role under item 5c.

4.5 Worked Example: The Cedar Valley Scope Decision Log

Three features of the log are worth copying. Each decision has a reason that the person who raised it can read, which makes it easier to accept a decision that declined a request. Each decision is tied to a protocol item, so the protocol and the meeting cannot drift apart. And the final row shows a parking-lot list, a place to record good questions that fall outside the scope so that they are neither lost nor allowed to expand the review.

4.6 Keeping the Scope Stable

Scope creep is the gradual expansion of a review through requests that each seem small. Adding an outcome, a population or a country after the protocol is registered changes the search, the screening and the charting, and a twelve-week review has little room for such changes. The Cedar Valley team used three practices. After registration, any change to scope goes through the amendment log described in Section 3.5, with its reason and timing. The team agreed in advance on when the planning team would see progress: after screening, after charting and at the draft evidence brief. And requests that arrived between those points went to the parking-lot list until the next check-in.

Lesson 3 explains how databases and search engines find information, which a team needs to understand before it turns the concepts in its question into a search that the librarian can review.

Bringing It All Together

This lesson followed the fictional Cedar Valley evidence team from a planning team's request to a registered protocol. The request was turned into a Population, Concept and Context question with five sub-questions, because the team needed to map what had been evaluated and with what outcomes, and the same framework produced a parallel question for the environmental scan. Each element of that question was then reused three times: as an eligibility criterion, as a block of search terms and as a heading on the charting form.

The eligibility criteria became usable only when terms such as older adults, community-based, loneliness and social isolation were given operational definitions, decision notes were added for borderline cases, and the criteria were piloted on real records. The PRISMA-P checklist then gave the team a structure for the whole plan, which it adapted to a rapid scoping review by stating the shortcuts it would take and by answering every item, including those for methods it would not use.

None of these decisions was made by the evidence team alone. The librarian tested the question against the literature and found existing reviews, and interest holders shaped the outcomes, sources and boundaries of the review, while screening, appraisal and interpretation stayed with the team. The decision log and the amendment log keep a record of those choices that readers can check.

The final reflection asks you to draft the core of a short protocol for a new question.

Final Knowledge Assessment

Terms, frameworks and people introduced in this lesson on review questions, eligibility criteria and protocols.