1.1 Why Evidence Does Not Reach Practice on Its Own

The first eight lessons of this course asked whether a program works and how an evaluator can tell. This lesson asks what determines whether a program with evidence behind it is taken up, delivered well and kept going in the settings that are supposed to use it. The question matters because the movement of evidence into routine practice is slow and uneven. A frequently quoted estimate holds that it takes about 17 years for research findings to reach routine clinical practice (Balas & Boren, 2000). Morris, Wooding and Grant (2011) reviewed the studies behind that figure and found that estimates of the time lag vary widely and depend on where the start and end points of the lag are placed. The figure is best treated as an indication that translation is slow, with considerable uncertainty about how slow.

A second problem concerns what happens to programs once they are delivered. Interventions tested under controlled conditions often produce smaller benefits when they are delivered by ordinary staff, to more varied populations, with fewer resources and less supervision. Chambers, Glasgow and Stange (2013) discussed this loss of benefit, often called a voltage drop, together with the related idea of program drift, the gradual departure of everyday practice from a tested protocol. They questioned the assumption that a program can be optimized once, at the end of a trial, and then delivered unchanged in settings that themselves keep changing. Section 3 returns to their argument when it turns to adaptation and sustainability.

Implementation science is the field that studies these problems. Eccles and Mittman (2006), in the opening editorial of the journal Implementation Science, defined it as the scientific study of methods that promote the systematic uptake of research findings and other evidence-based practices into routine practice, with the aim of improving the quality and effectiveness of health services. In Canada the broader enterprise is usually called knowledge translation. The Canadian Institutes of Health Research (CIHR) describe knowledge translation as a dynamic and iterative process that includes the synthesis, dissemination, exchange and ethically sound application of knowledge to improve health, provide more effective services and strengthen the health system (Straus, Tetroe & Graham, 2009). CIHR's planning guidance also distinguishes integrated knowledge translation, in which knowledge users take part throughout a study, from end-of-grant knowledge translation, which follows it. Implementation science is the part of this enterprise that tests and explains how evidence-based practices become part of routine care.

Social prescribing programs illustrate both sides of the gap. Community connector programs have spread across the United Kingdom, Canada and other countries faster than evidence about their effectiveness has accumulated, and early systematic reviews judged that evidence to be limited (Bickerdike et al., 2017). Evaluators of these programs therefore often face the reverse of the usual problem: a program that is already being implemented widely while questions about its effects remain open. Where a connector program does help people, its effect in a population depends on whether clinicians refer, whether referred people engage, whether connectors deliver the core functions of the role, and whether community resources exist to connect people to. Each of those conditions is a question for implementation science.

1.2 Efficacy, Effectiveness and Implementation

Flay (1986) distinguished efficacy trials, which test whether an intervention does more good than harm when it is delivered under optimum conditions, from effectiveness trials, which test whether it does more good than harm when it is delivered under real-world conditions. Lesson 6 developed the same distinction in the vocabulary of explanatory and pragmatic trials and showed how the PRECIS-2 wheel places a trial between the two poles. Glasgow, Lichtenstein and Marcus (2003) argued that the conventional sequence, in which efficacy trials come first and effectiveness studies follow, slows translation. Efficacy trials select motivated participants, expert staff and well-resourced settings, so their results say little about the conditions in which programs will actually be used.

Implementation research asks a third kind of question. It treats an intervention with evidence of effectiveness as given and studies the methods used to get it adopted, delivered and sustained. Its independent variable is usually an implementation strategy, such as training, practice facilitation, audit and feedback, or a change to the electronic medical record. Its dependent variables are implementation outcomes, such as the proportion of clinics that adopt the program or the proportion of eligible patients who are referred. Health outcomes remain relevant, and Section 4 shows how a single study can measure both, but the primary test of an implementation strategy is its effect on implementation outcomes. Table 1.1 sets the three kinds of research side by side.

Curran (2020) proposed a plain-language way of keeping these questions apart, which many implementation scientists now use in teaching. The tabs below apply it to the Cedar Valley program.

1.3 Intervention Failure and Implementation Failure

When an outcome evaluation finds no effect, there are two broad explanations. The program may not work, because its theory of change is wrong or its activities are too weak to produce the intended change. This is intervention failure, sometimes called theory failure. Alternatively, the program may work when it is delivered but may not have been delivered, or delivered to too few people, or delivered without its essential components. This is implementation failure. Basch and colleagues (1985) called the evaluation of a program that was never adequately implemented a Type III error: the evaluator draws a conclusion about a program that, in practice, did not take place.

Proctor and colleagues (2011) argued that the distinction between the two failures is one of the main reasons to measure implementation outcomes explicitly. Without them, a null result cannot be interpreted. The process evaluation methods of Lesson 5, including measures of reach, dose delivered, dose received and fidelity, supply much of the evidence needed. Table 1.2 shows the four combinations an evaluator can encounter.

In the Cedar Valley first wave, referral rates differed several-fold between clinics, which invites a comparison of outcomes between well and poorly implemented clinics. Such an analysis is observational even within a randomized study, because clinics that implement well may differ in staffing or community resources that also affect loneliness, so the confounding logic of Lesson 7 applies.

1.4 Implementation Outcomes

Proctor and colleagues (2009) proposed a conceptual model that separates three kinds of outcomes. Implementation outcomes are the effects of deliberate actions to implement a program. Service outcomes describe the quality of the care or services delivered, using the six quality domains of the United States Institute of Medicine: efficiency, safety, effectiveness, equity, patient-centredness and timeliness. Client outcomes describe changes in the people served, such as satisfaction, function and symptoms. In the model, implementation strategies act on implementation outcomes, and implementation outcomes are preconditions for service and client outcomes, as Figure 1.2 shows.

Proctor and colleagues (2011) then defined eight implementation outcomes, reviewed how each had been measured, and proposed that each is most salient at a particular stage of implementation. Acceptability, adoption, appropriateness and feasibility matter most early, when settings decide whether to take a program up. Fidelity and implementation cost matter most during delivery. Penetration and sustainability matter most later, when the question is whether the program has become part of routine operations. The cards below define each outcome and give an indicator for the Cedar Valley second wave.

Several of these outcomes can be measured with brief instruments. Weiner and colleagues (2017) developed and tested the Acceptability of Intervention Measure, the Intervention Appropriateness Measure and the Feasibility of Intervention Measure, each with four items scored on a five-point scale, and found evidence for their reliability and for the distinctness of the three constructs. Other outcomes are measured from administrative records (adoption and penetration), program logs and observation (fidelity), financial and time records (implementation cost), or repeated follow-up of settings (sustainability). Ten years after the original taxonomy, Proctor and colleagues (2023) reviewed the studies that had used it and reported that some outcomes, notably acceptability, fidelity and feasibility, had been studied far more often than others, and that the links between implementation outcomes and client outcomes had rarely been tested.

1.5 Worked Example: First-Wave Penetration and Its Consequences

The Cedar Valley referral rule is a UCLA Loneliness Scale score of 6 or higher, or clinician judgement. To calculate penetration, the evaluator needs a denominator: the number of eligible older adults attached to the first-wave clinics. Lesson 2 estimated that the 12 first-wave clinics have about 23,000 attached patients aged 65 and older. The regional survey described in Lessons 1 and 2 found that 24.5 percent of respondents aged 65 and older scored 6 or higher, which gives an estimated 5,635 eligible older adults.

Across the 12 first-wave clinics, penetration ranged from about 1 percent to 10 percent, and 52 of the 88 clinicians in those clinics (59.1 percent) made at least one referral in the six months. These figures change how the pre-post loneliness result should be read. Even if the program reduces loneliness among participants, the reduction applies to fewer than one in twenty of the eligible older adults in those clinics, and the 188 people with two measurements are about one in thirty. A program's population impact is the product of how many eligible people it reaches and how much it helps them, which is the reasoning behind the RE-AIM framework in Section 2.

The denominator deserves scrutiny. It applies a survey prevalence to clinic panels, which assumes that the panels resemble the survey population. Rural clinics, clinics with older panels, or clinics serving communities with more people living alone may have higher prevalence, so a single regional figure could understate need in some clinics and overstate it in others. A better denominator would count patients who were actually screened and scored 6 or higher, which requires a structured screening field in the electronic medical record. The second-wave plan in Section 4 adds that field.

Section 2 turns to the frameworks used to plan, explain and evaluate this kind of work, each applied to the Cedar Valley second wave.

2.1 Theories, Models and Frameworks

Implementation science has produced a large number of theories, models and frameworks, and new users often find it hard to tell which ones answer which questions. Nilsen (2015) proposed a taxonomy that sorts them by purpose. He distinguished a theory, a set of analytical principles or statements that structure observation, understanding and explanation, from a model, which is a deliberate simplification of a phenomenon, and from a framework, which is a structure of descriptive categories and the relations between them that are presumed to account for a phenomenon. Frameworks name and organize the factors that matter without necessarily explaining how those factors produce their effects.

Nilsen grouped the approaches into five categories that serve three aims. Process models describe or guide the process of moving research into practice, usually as a sequence of steps or phases. Determinant frameworks, classic theories and implementation theories share the aim of understanding and explaining what influences implementation outcomes. Determinant frameworks list barriers and enablers across several levels; classic theories, such as diffusion of innovations, come from psychology, sociology and organizational science; and implementation theories, such as Normalization Process Theory, were developed within the field. Evaluation frameworks specify what to measure to judge whether implementation succeeded. Figure 2.1 arranges the five categories under their aims.

The taxonomy has a practical use. An evaluator who needs to plan a rollout, diagnose why it is struggling, and judge whether it worked needs one approach for each purpose, and confusion arises when a framework built for one purpose is stretched to serve another. Some frameworks sit across categories; EPIS, for example, describes phases and also names determinants within them.

2.2 Process Frameworks: Knowledge-to-Action and EPIS

Process frameworks lay out the steps through which evidence moves into use. Two are widely used in health services and fit the Cedar Valley program well.

2.3 Determinant Frameworks: CFIR 2.0 and the Theoretical Domains Framework

The Consolidated Framework for Implementation Research

The Consolidated Framework for Implementation Research (CFIR) was published by Damschroder and colleagues in 2009 as a consolidation of constructs drawn from earlier implementation theories. It became one of the most widely used determinant frameworks in the field. Damschroder, Reardon, Widerquist and Lowery (2022b) published an updated version, CFIR 2.0, based on a review of published applications and a survey of authors who had used it. The update has five domains containing 48 constructs and 19 subconstructs. It uses the general term "innovation" for the program or practice being implemented, gives more attention to the people who receive an innovation, and adds constructs related to equity. A companion paper, the CFIR Outcomes Addendum (Damschroder et al., 2022a), recommends that users define the implementation outcomes they are trying to explain before they assess determinants, so that the assessment explains something specific.

Using CFIR 2.0 starts with three definitions: the innovation (what exactly is being implemented), the inner setting (where it is implemented) and the outer setting (the setting in which the inner setting exists). The accordion below summarizes the five domains with examples of their constructs.

CFIR data are usually collected through semi-structured interviews built from the framework's construct definitions, sometimes supplemented by surveys and documents. Transcripts are coded deductively to constructs, and each construct is then rated for each site. Damschroder and Lowery (2013), in an evaluation of a weight management program across Veterans Affairs medical centres, rated each construct for its valence (positive or negative influence on implementation) and strength, on a scale from −2 to +2, with 0 for a neutral influence and X for mixed evidence. Comparing ratings between sites with high and low implementation identified the constructs that distinguished them. The same approach can be used prospectively, before launch, to anticipate barriers.

The ratings in Table 2.1 are illustrative and summarize the evidence across clinics, noting where a rating applies only to some. A real assessment would report ratings by clinic so that strategies can be tailored to each. The ratings are useful mainly because of the discipline they impose: every barrier must be tied to evidence, placed in a domain, and linked to a decision. Section 3 takes these barriers forward and matches them to implementation strategies.

The Theoretical Domains Framework

CFIR 2.0 spans many levels, from policy to individual clinicians. When the implementation problem is a specific behaviour of health professionals, a framework focused on behaviour can give a sharper diagnosis. The Theoretical Domains Framework (TDF) was developed by Michie and colleagues (2005) through a consensus process that synthesized constructs from behaviour change theories, and Cane, O'Connor and Michie (2012) validated a revised version with 14 domains and 84 constructs. The domains are knowledge; skills; social or professional role and identity; beliefs about capabilities; optimism; beliefs about consequences; reinforcement; intentions; goals; memory, attention and decision processes; environmental context and resources; social influences; emotion; and behavioural regulation. The domains map onto the COM-B model from Lesson 2, which allows a TDF diagnosis to lead into the Behaviour Change Wheel's intervention functions. Atkins and colleagues (2017) published a guide to using the TDF in interviews and analysis.

For Cedar Valley, the behaviour of interest is a clinician asking an older patient about loneliness and making a referral. A TDF analysis might find gaps in knowledge (clinicians unsure of the referral rule), skills (how to ask), beliefs about consequences (fear that the question will offend), memory, attention and decision processes (forgetting during short visits), and social or professional role (uncertainty whether loneliness is a medical concern). CFIR 2.0 would capture some of these under the capability and motivation of innovation deliverers, but with less detail. Birken and colleagues (2017) reviewed studies that combined the two frameworks and described how they complement each other, with CFIR covering the organizational context and the TDF the behaviour of individuals.

2.4 Evaluation Frameworks: RE-AIM and PRISM

RE-AIM

RE-AIM was proposed by Glasgow, Vogt and Boles (1999) to evaluate the public health impact of health promotion programs. Its central argument is that the impact of a program depends on more than its effect on participants: a program that helps people a great deal but reaches few of them, is taken up in few settings, is delivered inconsistently, or disappears when funding ends has limited impact. The framework has five dimensions, assessed at the individual level, the setting level, or both, as Table 2.2 shows.

The emphasis on representativeness is what distinguishes RE-AIM from a simple count. Reach and adoption ask who takes part and which settings deliver, and whether they differ from those who do not, which makes the framework a natural tool for examining equity. In a review of the framework's first 20 years, Glasgow and colleagues (2019) encouraged pragmatic use, in which evaluators select the dimensions that matter for a particular decision without an obligation to measure all five, and gave greater weight to qualitative methods, adaptations, costs and equity.

PRISM

RE-AIM says what to measure but says little about why the results take the values they do. PRISM, developed by Feldstein and Glasgow (2008), adds contextual domains that are expected to influence RE-AIM outcomes: the intervention as seen from the organizational and the patient perspective; the characteristics of recipients, both organizations and patients; the implementation and sustainability infrastructure; and the external environment. Figure 2.2 shows the structure. In practice, PRISM plays a role similar to a determinant framework within an evaluation, which is why some teams use it in place of CFIR when they want a single integrated approach.

Worked example: RE-AIM for the Cedar Valley second wave at six months

The 12 second-wave clinics have about 21,000 attached patients aged 65 and older. Applying the regional estimate of 24.5 percent gives about 5,145 eligible older adults, with the same caution about the denominator that Section 1.5 raised. Eleven of the 12 clinics signed a participation agreement and made at least one referral within three months; the twelfth, a two-physician clinic that lost a physician shortly before launch, deferred. In the first six months the eleven clinics referred 263 older adults, and 198 attended a first meeting. Table 2.3 sets out the indicators.

Read against the first wave, these results show that the less ready second-wave clinics came close to the first wave on reach (3.8 percent of the eligible population compared with 4.3 percent) and on clinician adoption (55.4 percent compared with 59.1 percent). The representativeness indicators show where the shortfall lies. Men and older adults served by rural clinics were under-represented among participants, and the rural clinics contributed 18.2 percent of participants while holding 30 percent of the estimated eligible population. The effectiveness indicator is a pre-post change and carries the limitations Lesson 7 described; RE-AIM asks the evaluator to report effectiveness from the best available design, which for Cedar Valley is the first-wave comparison, and to check whether effects differ across subgroups such as rural participants. PRISM directs attention to contextual explanations for the rural gap, in the external environment (distance and transport) and in recipient characteristics.

2.5 Choosing and Combining Frameworks

No single framework serves every purpose, and evaluations commonly combine one framework from each of Nilsen's aims. Birken and colleagues (2018) developed the Implementation Theory Comparison and Selection Tool (T-CaST), which helps teams compare candidate frameworks against criteria such as usability, testability and applicability to their setting. Two cautions apply. A framework should shape data collection from the start; when it is applied only to label findings at the end, it adds vocabulary without adding explanation.

The Proctor outcomes and RE-AIM overlap in ways that are worth stating explicitly. Table 2.4 maps them onto each other.

For the Cedar Valley second wave, a coherent combination would use EPIS to describe the phases and the bridging arrangements, CFIR 2.0 to assess determinants before launch and again at six months, and RE-AIM, with PRISM to interpret context, to evaluate the rollout, with the Proctor perception measures (acceptability, appropriateness and feasibility) added to the implementation dimension. Section 4 assembles this combination into a worked example of the implementation component of the Cedar Valley evaluation plan.

3.1 What an Implementation Strategy Is

Proctor, Powell and McMillen (2013) defined implementation strategies as the methods or techniques used to enhance the adoption, implementation and sustainability of a clinical program or practice. Powell and colleagues (2012) distinguished three kinds. A discrete strategy is a single action or process, such as a reminder in the electronic medical record. A multicomponent strategy combines two or more discrete strategies, such as training plus audit and feedback. A blended strategy interweaves several discrete strategies into a protocolized package that is often given a name and tested as a unit. Most implementation efforts in health services use multicomponent or blended strategies, because the barriers they face sit at several levels at once.

A practical difficulty is deciding where the program ends and the strategy begins. For Cedar Valley, the connector meetings, the co-developed plan and the linkage to community resources are clearly part of the program. Training clinicians to ask about loneliness, adding a referral template to the medical record, and sending clinics reports on their referral rates are clearly implementation strategies. Screening is harder to classify: it is part of the program's referral pathway, yet a prompt that reminds clinicians to screen is a strategy. The boundary matters because an evaluation needs to know which components it is testing and which it is holding constant. The logic model from Lesson 3 should state the boundary explicitly, and the CFIR 2.0 definition of the innovation in Section 2 should match it.

3.2 The ERIC Compilation

To give the field a common vocabulary, Powell and colleagues (2015) ran the Expert Recommendations for Implementing Change (ERIC) project, a modified Delphi process with implementation scientists and clinical experts that produced a compilation of 73 discrete implementation strategies, each with a name and a definition. Waltz and colleagues (2015) then asked experts to sort the strategies by similarity and to rate their importance and feasibility, and used concept mapping to group them into nine clusters. The cards below describe each cluster with examples of its strategies and a possible use in the Cedar Valley second wave. Two cluster labels are paraphrased here because the originals use an older term for interest holders.

The compilation is a vocabulary, and it carries no evidence ratings. Evidence about the effectiveness of particular strategies comes from other sources, such as systematic reviews of audit and feedback or of practice facilitation. Baskerville, Liddy and Hogg (2012), for example, reviewed trials of practice facilitation in primary care and found that facilitation improved the adoption of evidence-based guidelines, which supports its use where clinic workflows are the main barrier.

3.3 Selecting and Tailoring Strategies

Strategies should be chosen to address the determinants that stand in the way of implementation, which is why a determinant assessment usually comes first. Waltz and colleagues (2019) asked implementation experts to name the ERIC strategies they would use to address each CFIR barrier and published the results as the CFIR-ERIC matching tool. Experts' recommendations varied considerably, so the tool's output is best used as a starting list for a team's own judgement. Powell and colleagues (2017) reviewed structured methods for selecting and tailoring strategies, including concept mapping, group model building, conjoint analysis and intervention mapping, the last of which Lesson 2 introduced as a planning framework. All of these methods share three elements: they identify barriers, they link each barrier to a strategy through an explicit rationale, and they involve the people who will deliver the program.

3.4 Specifying Strategies

Reports of implementation studies have often described strategies too vaguely to be replicated or compared, with labels such as "training" or "support" that could describe almost anything. Proctor, Powell and McMillen (2013) recommended that authors name each strategy, define it, and specify it on seven dimensions: the actor who delivers it; the action taken; the action target, meaning the people, unit or determinant it is meant to change; the temporality, meaning when it is used; the dose, meaning its frequency and intensity; the implementation outcome it is expected to affect; and the justification for its use, drawn from theory, evidence or practical reasoning. Lewis and colleagues (2018) added that the justification should name the mechanism through which the strategy is expected to work, so that a study can test whether the strategy succeeded or failed for the hypothesized reason.

Specification also makes the strategy's cost visible. The second-wave implementation support package consists of the half-time facilitator ($52,000 for the year), clinician training sessions ($8,000) and the production of audit and feedback reports ($4,000), for a total of $64,000. Across the eleven adopting clinics, this is about $5,818 per clinic, which is the implementation cost indicator proposed in Section 1. The cost is separate from the cost of delivering the program, the connectors' salaries and the transport fund, and Lesson 10 shows how both enter a program costing and a budget impact analysis. When a strategy changes during a study, for example when visits are moved online, the change should itself be documented; Miller and colleagues (2021) extended the FRAME approach described next to modifications of implementation strategies, as FRAME-IS.

3.5 Adaptation: Core Functions, Forms and FRAME

Programs are almost always changed when they move into new settings. Some changes improve fit and preserve the program's effects; others remove the components that make it work. The older view treated any departure from the protocol as a threat to fidelity. A more useful view, set out by Hawe, Shiell and Riley (2004) for complex interventions, holds that what should be standardized is the function of each component, the purpose it serves in the program's theory of change, while its form, the specific activity that serves the purpose, can vary with context. Jolles, Lengnick-Hall and Mittman (2019) developed this distinction of core functions and forms for health services interventions. An adaptation that changes a form while preserving the function is fidelity-consistent; one that removes or undermines a function is fidelity-inconsistent.

Wiltsey Stirman, Baumann and Miller (2019) published FRAME, an expanded framework for reporting adaptations and modifications to evidence-based interventions, which built on an earlier coding system by Stirman and colleagues (2013). FRAME records eight aspects of each modification: when and how in the implementation process it was made; whether it was planned and proactive or unplanned and reactive; who decided to make it; what was modified (content, context, training or evaluation); at what level of delivery it was made, such as an individual participant, a clinic or the whole program; the type of change, such as adding, removing, shortening, substituting or reordering elements, or changing the format, setting or personnel; whether it was fidelity-consistent; and the reasons for it, including its goal and the contextual factors behind it. The accordion below logs three second-wave adaptations with FRAME.

A FRAME log is cheap to keep if it is built into routine work, for example as a short form that connectors complete when they change how they deliver the program, reviewed monthly by the coordinator. The log supports two analyses. It shows the fidelity-inconsistent changes that need correction, and it allows the evaluation to ask whether adaptations were associated with reach or outcomes, keeping in mind that such associations are observational.

3.6 Scale-Up and Sustainability

Scale-up

Scale-up refers to deliberate efforts to increase the impact of health innovations that have been tested successfully, so that they benefit more people and become part of policy and programs on a lasting basis. The World Health Organization and ExpandNet (2010) distinguished two directions. Horizontal scale-up, also called expansion or replication, extends a program to more sites or populations. Vertical scale-up, also called institutionalization, embeds the program in policy, budgets, regulations and organizational structures. The Cedar Valley second wave is horizontal scale-up, from 12 clinics to 24; moving the connector positions from time-limited funding into the health authority's base budget would be vertical scale-up. Figure 3.1 shows the two directions.

Before scaling up, a program's scalability can be assessed. Milat and colleagues (2013) described scalability as the ability of an intervention shown to be efficacious on a small scale or under controlled conditions to be expanded under real-world conditions to reach a greater proportion of the eligible population while retaining its effectiveness, and their group later developed the Intervention Scalability Assessment Tool (Milat et al., 2020) to structure the judgement around the problem, the intervention, the strategic and political context, the evidence of effectiveness, the costs, and the infrastructure needed. For Cedar Valley, the main scalability questions are whether the rural communities have enough groups and services to connect people to, and whether connector caseloads remain manageable as referrals grow.

Sustainability

Moore, Mascarenhas, Bain and Straus (2017) reviewed definitions of sustainability and proposed one with five elements: after a defined period of time, the program, clinical intervention or implementation strategies continue to be delivered, or individual behaviour change is maintained; and the program and behaviour change may evolve or adapt while continuing to produce benefits for individuals or systems. The definition accepts change as part of sustainability, which is consistent with the Dynamic Sustainability Framework of Chambers and colleagues (2013). That framework argues that programs should be expected to change over time, through continuous measurement and refinement of their fit with changing settings, and that sustainment should be planned from the start. Shelton, Cooper and Wiltsey Stirman (2018) reviewed the determinants of sustainability in public health and health care and identified factors at the levels of the program, the organization, the outer context and the implementation process, many of which overlap with CFIR 2.0 constructs.

Sustainability can be assessed with the Program Sustainability Assessment Tool (PSAT) of Luke and colleagues (2014), which rates a program's capacity for sustainability across eight domains: environmental support, funding stability, partnerships, organizational capacity, program evaluation, program adaptation, communications and strategic planning. For Cedar Valley, the steering committee completed the PSAT at the end of the first year. Partnerships and program evaluation scored well, while funding stability scored poorly, because the connector positions are funded from a time-limited allocation. The sustainability plan therefore centres on vertical scale-up: presenting the first-wave evaluation and the second-wave RE-AIM results to the health authority's executive with a proposal to move the positions into the base budget, and building connector referral into regional primary care policy.

Section 4 turns to study design and shows how a single study can test a program's effectiveness and its implementation at the same time.

4.1 Why Combine Effectiveness and Implementation Questions

Section 1 described the conventional sequence of research, in which efficacy trials are followed by effectiveness trials and then, often years later, by implementation studies. Curran, Bauer, Mittman, Pyne and Stetler (2012) argued that this sequence is slow, and that it produces effectiveness evidence under conditions that may not resemble those of later implementation. They proposed effectiveness-implementation hybrid designs, studies that take a dual focus on effectiveness and implementation, as a way to speed translation, to learn about implementation while effectiveness is still being established, and to generate evidence that is more useful to the people who decide whether to adopt a program. Effectiveness-implementation hybrid designs are unrelated to the hybrid observational designs of epidemiology, such as the case-cohort and nested case-control designs, which combine features of cohort and case-control sampling.

The idea is especially relevant to programs like Cedar Valley. Health authorities rarely wait for a full sequence of trials before rolling out a social prescribing program, and the evaluator is usually asked to learn as much as possible from a rollout already under way. A hybrid study makes that learning systematic, by stating in advance which questions about effectiveness and which questions about implementation the study will answer, and with what priority.

4.2 The Three Hybrid Types

Curran and colleagues (2012) defined three types along a continuum, shown in Figure 4.1.

A hybrid type 1 study tests the effects of an intervention on relevant outcomes as its primary aim, while observing and gathering information on implementation as a secondary aim. The secondary aim typically examines barriers and facilitators, the acceptability and feasibility of the intervention in its setting, and the potential for wider implementation. Type 1 suits interventions whose effectiveness is not yet established in the target setting, where the intervention carries little risk and there is reason to prepare for implementation if it proves effective.

A hybrid type 2 study has co-primary aims: it tests the intervention's effects and also tests or studies an implementation strategy. Type 2 suits interventions with some evidence of effectiveness, perhaps in related populations or settings, where there is a reasonable expectation that a particular implementation strategy is feasible and supportable in the setting.

A hybrid type 3 study tests an implementation strategy as its primary aim, while observing and gathering information on the intervention's clinical or health outcomes as a secondary aim. Type 3 suits interventions with strong effectiveness evidence, where the main uncertainty concerns how best to implement them and whether their effects hold up when they are delivered in new settings, the voltage drop of Section 1.

4.3 The 2022 Update

A decade after the original paper, Curran, Landes, McBain, Pyne, Smith, Fernandez, Chambers and Mittman (2022) reviewed how the types had been used and recommended several changes. They proposed describing hybrid studies in place of hybrid designs, because the term "design" had caused confusion and a hybrid can use many research designs, including non-randomized ones. A study's type is set primarily by its aims and research questions, and secondarily by its outcomes, and the research design is a separate choice. They therefore recommended that authors name both, as in "a parallel cluster randomized hybrid type 3 effectiveness-implementation study".

The update clarified each type. In type 1 studies, power is usually calculated for participant-level health outcomes, implementation outcomes are often not measured formally, and the implementation aim typically explores the intervention's potential for implementation. Type 2 studies should genuinely test an implementation strategy that is deployed in real-world conditions; dual randomization of both the intervention and the strategy is ideal but uncommon, and many type 2 studies randomize individuals to the intervention while applying a strategy to all sites, or study the strategy in a single-arm pre-post design. Type 3 studies should be powered on implementation outcomes, which are usually measured at the level of the site, the implementer or the system, and their strategies should be delivered by or built into the local system wherever possible.

The authors also offered four considerations for choosing a type: the strength of the existing evidence for the intervention's effectiveness; how much adaptation the intervention is expected to need in the new setting; how much is known about the determinants of its implementation; and whether the team is ready to test implementation strategies. Strong evidence and little expected adaptation point toward type 3; weak evidence and little knowledge of determinants point toward type 1. They noted that a hybrid study is not indicated when an intervention has not yet been shown to be safe, when no question about its effectiveness remains for the setting, or when a study examines only the determinants of implementation. Because the types lie on a continuum, authors should justify their choice and explain why the neighbouring types were not selected.

4.4 The Cedar Valley Program as Each Hybrid Type

The same program can be studied as any of the three types at different points in its history, depending on what is known and what decision is being made. The tabs below describe a version of the Cedar Valley evaluation for each type.

Other designs can test implementation strategies. Brown and colleagues (2017) reviewed the options, which include parallel and stepped-wedge cluster randomized trials, interrupted time series, and adaptive designs. Sequential multiple assignment randomized trials, introduced in Lesson 6, can test adaptive implementation strategies: Kilbourne and colleagues (2014) described a trial in which sites that did not respond to a standard implementation strategy were randomized to receive additional facilitation of different intensities. In a Cedar Valley version, clinics with penetration below 3 percent at three months could be randomized to add practice facilitation or facilitation plus a funded champion, so that the more intensive and costly support goes only to the clinics that need it. Where randomization is not possible, the interrupted time series methods of Lesson 8, applied to monthly referral counts, can test whether a strategy changed the level or trend of referrals.

4.5 Reporting Implementation Studies with StaRI

Implementation studies report two things at once, the implementation strategy and the intervention it supports, and readers need to know about both. The Standards for Reporting Implementation Studies (StaRI) statement of Pinnock and colleagues (2017) was developed through a systematic review and an international e-Delphi process. It contains 27 items, and for many of them it asks authors to report separately on the implementation strategy and on the intervention, which the checklist sets out as two parallel columns. The items cover the title and abstract, the rationale and context, the specification of the strategy and the intervention, the outcomes for each, their fidelity and adaptation, resource use, and the interpretation and generalizability of findings. StaRI applies to implementation studies with a wide range of designs and is used alongside design-specific guidelines: the CONSORT statement and its cluster extension for randomized studies (Lesson 6), SQUIRE 2.0 for quality improvement studies (Lesson 8), and TIDieR for describing interventions (Lesson 10). Proctor's specification dimensions and FRAME logs supply much of what StaRI asks for.

4.6 Worked Example: An Implementation Framework and Outcomes for Cedar Valley

The implementation component of an evaluation plan selects an implementation framework for the program and specifies the implementation outcomes to be measured. The Cedar Valley version follows.

The plan keeps the implementation outcomes separate from the effectiveness outcomes (loneliness, social participation, self-rated health and service use), which remain in the evaluation matrix from Lesson 5. It also states the data each indicator needs before launch, which is why the screening field in the medical record and the connector log appear in the plan. The CFIR 2.0 assessment will be repeated at six months, and its ratings compared between clinics above and below the penetration benchmark, to explain the implementation results that RE-AIM reports.

Bringing It All Together

This lesson moved the evaluation from the question of whether a program works to the question of how it comes to work in practice. Section 1 distinguished efficacy, effectiveness and implementation research, explained why benefits shrink as programs move into routine settings, and defined the eight implementation outcomes of Proctor and colleagues. The Cedar Valley first wave showed that a program with an encouraging pre-post result had reached fewer than one in twenty eligible older adults in its clinics, and that penetration varied several-fold between clinics.

Sections 2 and 3 provided the tools for acting on that finding. Nilsen's taxonomy sorted frameworks by purpose. CFIR 2.0 identified the barriers facing the less ready second-wave clinics, and RE-AIM, interpreted with PRISM, showed that the second wave nearly matched the first on reach and adoption while under-serving men and rural older adults. The ERIC compilation, the specification dimensions of Proctor, Powell and McMillen, and FRAME turned those findings into specified strategies and documented adaptations, with core functions held constant and forms allowed to vary.

Section 4 brought effectiveness and implementation questions into a single study. The Cedar Valley first wave fits a hybrid type 1 study, the second wave a hybrid type 2 study, and a provincial spread a hybrid type 3 study powered on clinic-level penetration, reported with StaRI.

The final reflection asks you to plan the implementation component of an evaluation for a hospital-to-home transition program, drawing on all four sections.

Final Knowledge Assessment

Definitions of the implementation science terms, frameworks and people introduced in this lesson.