Introduction

Lesson 4 ended with a set of prioritized evaluation questions and the structure of the evaluation matrix, which has a row for each question and columns for the indicator, data source, method, timing and person responsible. This lesson fills in those columns. Section 1 covers indicators, Section 2 process evaluation, Section 3 mixed methods, and Section 4 data systems, ending with a complete evaluation matrix for the running case.

The running case is the Cedar Valley Connector program, a fictional community connector (social prescribing) program run by the fictional Cedar Valley Health Authority in British Columbia. Primary care clinicians in 12 first-wave clinics refer adults aged 65 and older who score 6 or higher on the three-item UCLA Loneliness Scale, or who are judged by the clinician to be isolated, to a community connector. The connector meets each person up to six times over twelve weeks, co-develops a plan, and links the person to community groups, volunteer roles, transportation help and services. In its first six months the program received 312 referrals, 241 older adults attended a first meeting, 188 had both a baseline and a follow-up loneliness score, the mean score among those 188 fell from 7.1 to 6.3, and spending was $420,000. These figures, first reported in Lesson 1, recur throughout this lesson.

1.1 What an Indicator Is

An indicator is a specific, observable and measurable characteristic that shows whether a program component is in place or whether an expected change has occurred. An evaluation question states what the evaluation wants to know, and an indicator states what will be observed in order to answer it. A question about reach cannot be answered until the evaluator decides what will count as evidence of reach, such as the proportion of referred older adults who attend a first meeting.

Three terms are often confused. A measure or instrument is the tool that produces a value, such as the three-item UCLA Loneliness Scale. An indicator is the defined quantity calculated from one or more measures, such as the mean change in UCLA score between baseline and twelve weeks among participants with both measurements. A target is the value of the indicator that the program intends to reach by a stated date, and a baseline is the value at the start, against which change is judged. In the terms of Lesson 1's evaluative reasoning, indicators make criteria observable, and targets are one kind of standard.

Indicators can be quantitative or qualitative. Quantitative indicators are counts, proportions, rates, means and ratios. Qualitative indicators describe the presence, quality or nature of something, such as whether a clinic has a documented referral pathway, or the themes in participants' accounts of what changed for them. Evaluation matrices commonly include qualitative indicators for questions about experience, context and mechanism, and Section 3 returns to them.

1.2 Process, Output and Outcome Indicators

Indicators are easiest to organize when they are attached to the components of a logic model, which Lesson 3 taught. Each column of a logic model suggests a different type of indicator, and each type answers a different kind of question.

Input indicators describe the resources a program has, such as the number of connector positions filled or the proportion of the budget spent. Process indicators describe whether and how the program's activities are carried out, such as the number of meetings held per participant, the time from referral to first contact, or the proportion of plans co-developed within two meetings. Output indicators count the direct products of the activities, such as the number of older adults who completed a plan or the number of linkages made to community groups. Outcome indicators describe changes in the people, organizations or systems the program aims to affect, such as a fall in loneliness scores or in emergency department visits. Some frameworks separate outcomes from impacts, using impact for long-term population-level change, and some combine process and output indicators into a single category. The labels matter less than consistency within an evaluation plan.

A leading indicator changes early and predicts later results, such as attendance at a first meeting, and a lagging indicator, such as emergency department visits, may take a year or more to change. A proxy indicator stands in for something that cannot be measured directly or affordably; the number of community groups accepting referrals is a proxy for the community's capacity to absorb new members.

1.3 Specifying an Indicator

An indicator name such as “reach” is too vague to measure, because two analysts given the same name will calculate different numbers. A full specification, often recorded on an indicator reference sheet, removes that ambiguity. Many agencies use a reference sheet of this kind, and federal performance information profiles ask for similar information. The table shows a completed reference sheet for one Cedar Valley indicator.

The formula and its six-month value are as follows.

The choice of denominator is where most indicator disputes begin. The first-meeting proportion uses referrals as its denominator, so it shows how well the program converts referrals into participation. It says nothing about the older adults who were never referred. A second reach indicator uses the estimated eligible population as its denominator. Lesson 2 estimated that the 12 first-wave clinics have about 23,000 patients aged 65 and older on their panels. If the regional survey estimate that 24.5 percent of older adults score 6 or higher applies to these clinics, about 5,635 patients would meet the referral rule, and the 312 referrals in six months represent about 5.5 percent of them (312 ÷ 5,635 × 100). This second indicator rests on an assumption that the regional prevalence applies to these clinic panels, and the reference sheet should state that assumption. The two indicators answer different questions, and an evaluation of reach usually needs both.

Baselines and targets

A target set without a baseline has little empirical basis, and a baseline without a target leaves the evaluator unable to say whether performance is adequate. Baselines come from the program's early data, the pilot, routine data from before the program began, or a comparison population. Targets can be set by projecting improvement from the baseline, by benchmarking against similar programs, by reference to a clinical or policy standard, or by negotiation with the primary intended users about what performance would justify continued investment. Targets should be set before the data are seen, with a record of who agreed to them and why. Equity targets deserve particular attention. If rural older adults attend first meetings less often than urban older adults, a single overall target can be met while the gap widens, so the plan may set a target for the gap itself.

Lesson 10 shows how indicators, targets and qualitative evidence are combined in an evaluative rubric to reach a judgement of merit or worth.

1.4 Judging Indicator Quality

Kusek and Rist (2004), in a World Bank guide to results-based monitoring, summarized the qualities of a good performance indicator with the acronym CREAM, drawing on Schiavo-Campo and Tommasi (1999). The criteria are useful as a checklist when a planning team has more candidate indicators than it can collect.

Outcome indicators built from measurement instruments must also meet the standards of measurement that students met in their epidemiology courses. Validity is the extent to which the instrument measures the intended construct. Reliability is the consistency of its results across occasions, raters or items. Sensitivity to change, also called responsiveness, is the instrument's ability to detect change when change has occurred. The three-item UCLA Loneliness Scale was developed for large surveys by Hughes and colleagues (2004). Each item asks how often the respondent feels a lack of companionship, left out, or isolated from others, with responses scored from 1 (hardly ever) to 3 (often), giving a total from 3 to 9. Its brevity makes it practical for clinicians and connectors, although its seven possible values make it coarse: scores move only in whole points, and a participant who scores 3 cannot improve on the scale. A team wanting finer-grained evidence might add a longer instrument for a subsample, at the cost of additional burden.

Interpretation matters as well. The referral rule selects older adults who score 6 or higher on one occasion, and some will score lower when measured again even without the program, because a single high score includes temporary distress. This regression to the mean is a threat to validity taught in Lesson 7. The fall from 7.1 to 6.3 among the 188 participants with both scores describes change, and it cannot by itself show how much of the change the program caused.

1.5 Performance Measurement and Evaluation

Performance measurement is the ongoing, routine collection and reporting of indicators to track a program's activities, outputs and outcomes against targets. Hatry (2006) describes it as regular measurement of the results and efficiency of services. Evaluation, as Lesson 1 defined it, is the systematic determination of a program's merit, worth or significance, usually through a study designed to answer specific questions, including why results occurred and how much of the change the program caused. Performance measurement supplies routine data and raises questions, and evaluation explains the patterns it detects.

Under the Treasury Board Policy on Results (2016), federal departments maintain performance information for their programs and also plan evaluations that examine relevance and effectiveness. Lesson 1 described that structure, with routine indicators tracked continuously and evaluations scheduled periodically to explain them, as a sound model for a health authority. Cedar Valley follows it: the analyst produces a quarterly performance report, and the evaluation plan specifies the studies that will inform the second-wave decision.

When indicators change behaviour

Indicators influence the behaviour of the people whose work they measure. Campbell (1979) observed that the more a quantitative social indicator is used for social decision-making, the more it becomes subject to corruption pressures and the more it distorts the processes it is intended to monitor. This observation is now known as Campbell's law. Its consequences are familiar in health care and appear in several forms.

Summary

Indicators translate evaluation questions into observable quantities and descriptions. Process, output and outcome indicators correspond to the columns of the logic model. A full specification allows an indicator to be calculated consistently and checked by others, and the CREAM criteria and measurement properties help a team choose among candidates. Performance measurement tracks indicators routinely, and evaluation explains them. Section 2 turns to process evaluation, which asks why a program is or is not working as intended.

2.1 Why Process Evaluation Matters

An outcome evaluation that reports only whether outcomes changed treats the program as a black box. If loneliness does not fall, the evaluation cannot say whether the program's theory was wrong or whether the program was never delivered as designed. If loneliness does fall, the evaluation cannot say which parts of the program mattered or whether the result would hold in other clinics. Lesson 1 introduced Weiss's distinction between implementation failure, in which the program was not delivered as planned, and theory failure, in which the program was delivered but the expected causal process did not occur, and Lesson 3 located the two failures in Chen's action and change models. Process evaluation is the study of how a program is implemented and received, and it supplies the evidence needed to tell these explanations apart.

Dobson and Cook (1980) used the term Type III error for the mistake of evaluating a program that was not actually implemented, and then attributing the absence of effects to the program itself. A health authority that concluded from a null outcome evaluation that community connectors do not work, when in fact most participants met a connector only once, would commit this error. The table shows how process and outcome evidence combine.

Process evaluation has three common uses. Formative use improves delivery while the program is running, for example by identifying clinics where referrals are lost. Interpretive use explains outcome findings, as in the table above. Transfer use informs decisions about scale and adaptation, which is directly relevant to Cedar Valley because the evidence from the first wave will shape how the program is delivered in the second-wave clinics.

2.2 The Components of Process Evaluation

Steckler and Linnan's edited volume Process Evaluation for Public Health Interventions and Research (Steckler & Linnan, 2002) set out a widely used list of process evaluation components. Linnan and Steckler's overview chapter defined seven: context, reach, dose delivered, dose received, fidelity, implementation and recruitment. Later work, including the Medical Research Council guidance described in Section 2.5, added adaptation as a component in its own right. Each card defines one component and applies it to Cedar Valley.

Each component becomes a process question with one or more indicators. The table gives examples for Cedar Valley, which Section 4 carries into the evaluation matrix.

2.3 Measuring Dose and Fidelity

Dose

Dose delivered and dose received are often confused because both can be expressed as a number of meetings. The distinction is the source of the shortfall. If a connector offers six meetings and the participant attends three, dose delivered is six and dose received is three. If the connector's caseload allows only three meetings, both are three, and the shortfall lies with the program. Recording meetings offered as well as meetings attended lets the evaluator tell these situations apart.

By the six-month review, 205 of the 241 older adults who attended a first meeting had reached the end of their twelve-week period. Their connector logs give the following distribution of meetings attended. The four rural clinics and eight urban clinics are shown separately because the steering committee asked whether transportation limits participation.

The steering committee had set a threshold of four meetings as the minimum dose at which a participant can complete a plan and receive follow-up after a linkage. About two thirds of participants reached it, with a gap of 25 percentage points between rural and urban clinics. The numbers describe the gap. They do not explain it, since rural participants may live farther away, may have fewer groups to be linked to, or may have been referred with more complex needs. Section 3 shows how qualitative follow-up can explain the gap.

Fidelity

Dane and Schneider (1998) reviewed how prevention programs measured implementation and identified five dimensions of fidelity: adherence to the program's components, exposure (the amount delivered), quality of delivery, participant responsiveness and program differentiation, which is the extent to which the program's distinctive features are present. Carroll and colleagues (2007) proposed a conceptual framework in which adherence, measured as content, frequency, duration and coverage, is the core of fidelity, and in which intervention complexity, facilitation strategies, quality of delivery and participant responsiveness moderate it. Both frameworks show that fidelity has more than one dimension, and that counting meetings captures exposure without capturing whether the essential work of the meetings occurred.

Cedar Valley's program team identified five core components and built a checklist into the program database, which connectors complete for each participant. The table reports the first six months for the 205 participants who had reached twelve weeks.

Three cautions apply to checklist data of this kind. Connectors record their own performance, so the checklist measures documented delivery, which may overstate actual delivery; a periodic audit of a random sample of case notes against the checklist would test this. A checklist records whether a component occurred and cannot judge its quality; a plan can be written by the connector with little input from the participant and still be ticked. The denominator must match the component, as the follow-up row shows: only the 181 participants who were linked could receive follow-up after a linkage.

Fidelity and adaptation

Programs delivered in many settings are always adapted, and a narrow view of fidelity would treat every adaptation as a failure. Hawe, Shiell and Riley (2004) proposed a resolution for complex interventions: standardize the function of each component, meaning the step in the change process it is meant to bring about, and allow its form to vary with local context. Under this view, fidelity is judged against functions. The table shows how this applies to Cedar Valley.

An adaptation log records each change, who made it, why, and whether it preserves the component's function. Lesson 9 introduces the FRAME approach, which gives a structured vocabulary for this record.

2.4 Planning a Process Evaluation

Saunders, Evans and Joshi (2005) published a practical guide to developing a process evaluation plan for health promotion programs. Its central idea is that the evaluator should first define what complete and acceptable delivery of the program would look like, and then design questions and methods to assess it. Their six steps are summarized below, with the Cedar Valley application for each. The authors note that steps three to five are iterative, because questions are revised as methods and resources are considered.

Against the acceptable-delivery standard, the first six months show two components below 80 percent (transportation assessment at 72.7 percent and follow-up after linkage at 73.5 percent) and one close to it (closing summary at 77.1 percent). These are formative findings, and the program coordinator can act on them before the second wave.

2.5 The Medical Research Council Guidance on Process Evaluation

In 2015 the United Kingdom Medical Research Council published guidance on process evaluation of complex interventions (Moore et al., 2015). Complex interventions have several interacting components, require behaviours from the people who deliver and receive them, act at several levels, and are often adapted to local settings. A community connector program meets every part of that description. The guidance applies to process evaluation the broader MRC framework for developing and evaluating complex interventions, introduced in Lesson 2 Section 3.5. It organizes process evaluation around three components.

Implementation covers how delivery is achieved, such as training, resources and support for connectors, and what is delivered, described by fidelity, dose, adaptations and reach. Mechanisms of impact covers how participants respond to and interact with the program, the mediating processes through which it produces change, and any unexpected pathways and consequences. For Cedar Valley, a program theory of the kind developed in Lesson 3 might propose that a plan built on the participant's own priorities increases motivation, that linkage with accompaniment and transportation help reduces practical barriers, and that repeated contact with a group builds relationships that reduce loneliness. A process evaluation might measure whether participants who attend a linked group three or more times report new relationships, which is a mediator on that pathway. Context covers the factors external to the program that affect implementation, mechanisms and outcomes, and which the program may in turn affect.

The guidance makes several recommendations about how a process evaluation should be conducted. The process evaluation should be planned alongside the outcome evaluation and should be based on a clear description of the program and its causal assumptions. Quantitative and qualitative methods should be combined, because quantitative data show how much was delivered and qualitative data show how and why. The guidance also suggests that, where possible, process data should be analyzed before the outcome results are known, so that interpretations of implementation are not shaped by knowledge of whether the program worked. It discusses the relationship between process evaluators and the people who designed and deliver the program, which needs to be close enough for the evaluator to understand the program and independent enough for the findings to be credible. The 2021 framework for developing and evaluating complex interventions (Skivington et al., 2021) extended this approach by treating program theory, context and engagement with interest holders as core elements at every phase of evaluation.

The MRC framework overlaps with realist evaluation (Lessons 1 and 3), which states program theory as context-mechanism-outcome configurations and tests them. Both lead the evaluator to specify, before data collection begins, which mechanisms and contextual factors the evaluation will examine.

Summary

Process evaluation explains how a program was delivered and received, which outcome findings need for their interpretation. Steckler and Linnan's components, with adaptation added by later work, structure the process questions and indicators. Dose and fidelity can be calculated from program records if denominators are chosen carefully and records are audited. Saunders and colleagues organize planning around complete and acceptable delivery, and the Medical Research Council guidance places implementation, mechanisms of impact and context in one framework. Questions such as why rural participants attend fewer meetings need both numbers and accounts from the people involved, and Section 3 turns to the designs that combine them.

3.1 Why Evaluations Combine Methods

Section 2 ended with a gap that numbers alone could not explain: participants at rural Cedar Valley clinics attended a mean of 3.39 connector meetings, compared with 4.38 at urban clinics. The program records show the size of the gap, and interviews with participants and connectors can show what produces it. Evaluation has used mixed methods for decades, because evaluation questions so often ask both whether something happened and how or why it happened. The two Background boxes below restate the mixed methods and qualitative ideas this section builds on.

3.2 Three Core Designs

The three core designs recalled in the Section 3.1 Background box can be combined into more complex designs, such as a mixed methods evaluation spanning several phases. The figure shows the three designs, and the tabs apply each one to an evaluation question for Cedar Valley.

The three designs can be combined. A multiphase evaluation of Cedar Valley would use an exploratory sequential component for the land-based pathway, an explanatory sequential component for the rural attendance gap, and a convergent component at each annual follow-up. Creswell and Plano Clark (2018) describe such combinations as complex designs, and they note that an evaluation spanning several years often takes this form.

Integration and fit

Mixed methods are defined by integration, which the Section 3.1 Background box placed at the design, methods and interpretation levels (Fetters, Curry & Creswell, 2013), and integrated results fit as confirmation, expansion or discordance, where discordance is itself a finding to be reported and investigated. HSCI 207 Lesson 6 Section 2.4 (Integration and Joint Displays) is optional reading on both ideas.

3.3 Joint Displays

A joint display is a table or figure that brings quantitative and qualitative results together so that they can be compared and interpreted as one (Guetterman, Fetters & Creswell, 2015). Its most important column is the meta-inference, the conclusion that draws on both strands. The joint display below integrates findings from the first six months of Cedar Valley. The interview results are illustrative summaries of the kind of themes an evaluation might report.

The calculation behind the satisfaction figure is 171 ÷ 188 × 100 = 91.0 percent. Guetterman and colleagues (2015) suggest that a good joint display organizes rows by a shared dimension, such as a domain, a theme or a participant subgroup, places the meta-inference in its own column, and states the fit explicitly. A joint display can also be organized by case, with each row a participant or clinic, or by statistical result, with each row a quantitative finding and the qualitative data that explain it. For an explanatory sequential design, a useful layout groups participants by the quantitative result used to select them, such as low and high attendance, and shows the themes in each group.

3.4 Most Significant Change and Outcome Harvesting

Some program outcomes cannot be specified in advance. Participants may value changes that the logic model did not anticipate, and community partners may change their own practices in response to the program. Two qualitative methods were designed to collect evidence of such outcomes systematically.

Most significant change

The most significant change technique, described in a guide by Davies and Dart (2005), collects stories of change from participants and staff and then involves people at different levels of a program in selecting, from those stories, the ones they judge most significant, with their reasons recorded. The technique is participatory, it does not rely on predefined indicators, and the discussion of why one story is chosen over another makes the values of the program's interest holders visible.

The technique's main limitation is a bias toward positive stories, because tellers and selectors tend to favour success. Users of the technique reduce this bias by adding a domain for negative or unexpected changes and by analyzing all stories, including those not selected (Davies & Dart, 2005). Most significant change complements quantitative indicators and is rarely sufficient on its own to judge whether a program met its objectives.

Outcome harvesting

Outcome harvesting, developed by Ricardo Wilson-Grau and colleagues and described by Wilson-Grau and Britt (2012), works backward from observed change. The evaluator collects evidence of what has changed and then determines whether and how the program contributed. An outcome is defined as an observable change in the behaviour, relationships, actions, policies or practices of an individual, group, community, organization or institution. The method has six steps: design the harvest with its primary users, review documentation and draft outcome descriptions, engage with the people who know about each outcome to refine the descriptions, substantiate a selection of outcomes with independent informants, analyze and interpret the outcomes in relation to the evaluation questions, and support the use of the findings.

Outcome harvesting suits outcomes among organizations and systems, where change is hard to predict. For Cedar Valley, a harvest among community partners might document that a seniors' centre added a weekday morning drop-in after several referrals, that a volunteer driver society changed its booking rules to accept requests from connectors, and that a faith community began a visiting program. Each outcome would be described with what changed, who changed, when and where, how the program contributed, and why the change matters, and a sample would be confirmed with someone outside the program team. Lesson 3's contribution analysis gives the logic for judging the program's contribution to such outcomes.

3.5 Quality and Reporting

A mixed methods evaluation should be judged on the quality of each strand and on the quality of its integration. O'Cathain, Murphy and Nicholl (2008) proposed the Good Reporting of A Mixed Methods Study (GRAMMS) guideline for health services research. It asks authors to justify the use of mixed methods, to describe the design in terms of purpose, priority and sequence, to describe each method's sampling, data collection and analysis, to describe where and how integration occurred and who took part in it, to describe any limitation of one method associated with the presence of the other, and to describe the insights gained from mixing. HSCI 207 Lesson 12 Section 2.3 (Reporting Guidelines) places GRAMMS among the reporting guidelines for other designs and is optional reading. An evaluation plan can use the same items as a checklist before data collection begins. For Cedar Valley, the plan states that integration will occur in joint displays reviewed by the steering committee, including its older adult members, and in a review of findings with the First Nations partners before results about First Nations participants are reported.

Summary

Mixed methods evaluation integrates quantitative and qualitative evidence so that an evaluation can say both what happened and why. Greene and colleagues' five purposes guide the choice among convergent, explanatory sequential and exploratory sequential designs. Integration can be designed at three levels, and the fit of integrated results, whether confirmation, expansion or discordance, is reported in a joint display with explicit meta-inferences. Most significant change and outcome harvesting add systematic qualitative evidence about outcomes that indicators did not anticipate. All of these methods depend on data that are collected reliably, stored securely and shared lawfully, which is the subject of Section 4.

4.1 The Data Collection Plan

An indicator is only as good as the data behind it. The data collection plan turns the indicator and data source columns of the evaluation matrix into operational instructions: for each indicator, which instrument or record is used, who collects the data, from whom, at what point, in what mode, how the data are entered and stored, and who checks them. The plan is usually accompanied by a data dictionary, which defines every variable, its permitted values and its source, so that a variable such as “first meeting date” means the same thing in every clinic. The figure shows the collection points for each Cedar Valley participant.

Two principles shape the plan. Data collection should be built into routine work wherever possible, because data that connectors record as part of their job are more complete than data collected separately. Burden on participants and staff should be proportionate, because every additional item lowers response and competes with program delivery. Cedar Valley's baseline survey therefore takes under ten minutes: the three UCLA items, a social participation item set, a self-rated health item and an intake form.

4.2 Administrative and Electronic Medical Record Data

For an evaluation, both kinds of record are produced without additional burden on participants, cover whole populations, and extend backward in time, which allows baselines and comparison groups to be constructed. Both were created for purposes other than evaluation, so their variables, coding and completeness reflect those purposes. The cards describe the sources available to the Cedar Valley evaluation.

Because linkage errors fall unevenly across groups, the evaluator should report linkage rates by subgroup, and moving records between organizations for linkage requires the agreements described in Section 4.4. Administrative data also miss important outcomes: no billing record measures loneliness, so administrative data supplement participant-reported outcomes and cannot replace them.

4.3 Data Quality Checks

Weiskopf and Weng (2013) reviewed how studies assessed the quality of EHR data for research reuse and identified five dimensions, which describe the quality of EHR data and apply equally to program data. Surveillance systems are judged on a related set of attributes (Centers for Disease Control and Prevention, 2001), in which data quality is defined by completeness and validity and timeliness corresponds to currency. The table applies Weiskopf and Weng’s dimensions to Cedar Valley.

Before the six-month report, the Cedar Valley analyst ran these checks on the raw extract. The extract held 319 referral records, of which 7 were duplicates (the same Personal Health Number and referral date), leaving the 312 unique referrals reported in Section 1. Three baseline UCLA totals were recorded as 10 or higher, which is impossible on a 3 to 9 scale; the item-level responses showed data entry errors, which were corrected. Five follow-up dates preceded the baseline date because day and month had been transposed. The concordance check found 327 referral codes in the clinic EMR extracts for the same period, so 15 referrals recorded in clinics had no program record. Some of these referrals were probably lost in transmission, which is a process finding about the referral pathway as well as a data quality finding. Each correction should be logged, the original extract retained, and the checks scripted so that they run identically every quarter.

4.4 Data Sharing Agreements and Privacy

Evaluation data about individuals are personal health information, and their collection, use and disclosure are governed by law and policy. In British Columbia, health authorities are public bodies under the Freedom of Information and Protection of Privacy Act (FIPPA), which requires a privacy impact assessment for new programs and initiatives that involve personal information. Private physician practices are generally governed by the Personal Information Protection Act. Where the evaluation is research, TCPS 2 applies, and Lesson 1 described how Article 2.5 distinguishes program evaluation and quality improvement from research requiring ethics review. Data that move between organizations, such as from clinics to the health authority or from the health authority to an academic partner, require a data sharing agreement.

4.5 Simple Dashboards

A dashboard displays a small number of indicators so that a manager can monitor them at a glance (Few, 2006). A useful program dashboard shows each indicator with its target, shows change over time, disaggregates where equity matters, and is updated on a known schedule. It avoids ranking clinics, for the reasons given in Section 1, and suppresses counts small enough to identify individuals; many organizations suppress cells below a threshold such as five. The figure sketches a quarterly dashboard for Cedar Valley built from the indicators in this lesson. Lesson 8 adds run charts for tracking indicators over many periods, and Lesson 10 treats data visualization for decision-makers.

The dashboard values come from earlier sections: 241 of 312 referrals attended a first meeting (77.2 percent), 135 of 205 participants attended four or more meetings (65.9 percent), and 188 of those 205 had a twelve-week score (91.7 percent). By quarter, 98 of 131 referrals (74.8 percent) and 143 of 181 referrals (79.0 percent) attended a first meeting.

4.6 Worked Example: The Cedar Valley Evaluation Matrix

Lesson 4 set out the structure of the evaluation matrix. The matrix below completes it for the five evaluation questions that Cedar Valley's primary intended users might prioritize to inform the second-wave decision. Each question has indicators with definitions and targets, data sources, methods, timing and a responsible person. The economic question, what the program costs per participant and per unit of outcome, is added in Lesson 10.

Several features of the matrix are deliberate. Every indicator traces to a column of the logic model and to one evaluation question. Each question that asks how or why has at least one qualitative source. Targets are stated where the steering committee has agreed them, and outcome standards are left to the rubric of Lesson 10. Indicators that require linked data are scheduled early because access takes months. The matrix was checked against capacity: the half-time analyst carries the quantitative indicators, while interviews, the outcome harvest and the six-month telephone survey are assigned to an external evaluator funded from the evaluation budget. The First Nation partner holds responsibility for the land-based pathway's indicators.

Bringing It All Together

This lesson filled in the columns of the evaluation matrix that Lesson 4 set out. Indicators turn evaluation questions into observable quantities and descriptions, and their specification, with numerators, denominators, data sources, disaggregations, baselines and targets, lets them be calculated consistently and checked by others. Performance measurement tracks those indicators routinely, while evaluation explains the patterns they reveal and judges what the program achieved.

Process evaluation supplies the evidence needed to interpret outcomes. Reach, dose delivered, dose received, fidelity, adaptation, recruitment and context describe how the program was delivered and received, and the Medical Research Council guidance adds mechanisms of impact. Mixed methods designs integrate this evidence with qualitative accounts, so that the Cedar Valley evaluation can explain, for example, why rural participants attended fewer meetings. Data systems, from program databases and electronic medical records to linked administrative data, make all of this possible when they are planned, checked and governed carefully.

The final reflection asks you to write matrix rows for a new Cedar Valley question, and the knowledge check below integrates ideas from all four sections.

The final reflection asks you to apply the whole lesson by writing evaluation matrix rows for a new question about the Cedar Valley Connector program.

Final Knowledge Assessment

These terms, frameworks and people appear in Lesson 5, and the definitions follow the way the lesson uses them.