This lesson completes the evaluation plan that the course has built since Lesson 1. Decision-makers ask what a program costs and whether the money would do more good elsewhere. This section teaches the first question, and Section 2 uses the answer to teach economic evaluation. Costing also supports budgets for scale-up and documents implementation cost, one of the implementation outcomes of Proctor and colleagues (2011) taught in Lesson 9.

The running example is the fictional Cedar Valley Connector program, a community connector (social prescribing) program for adults aged 65 and older run by the fictional Cedar Valley Health Authority in British Columbia. Its first wave operates in 12 of the region's 24 primary care clinics on an annual budget of $840,000: seven connector positions ($595,000), a coordinator ($105,000), a half-time analyst ($55,000), a transport fund ($40,000), partner grants ($30,000), and training, travel and data systems ($15,000). For this lesson, assume that in the program's first full year clinicians made 640 referrals and 500 older adults attended a first meeting with a connector (312 referrals and 241 first meetings in the first six months, and 328 referrals and 259 first meetings in the second six months). These year-one figures, and the unit costs introduced below, are illustrative assumptions for this lesson.

1.1 Costs as Opportunity Costs

Economic evaluation rests on the idea of opportunity cost: the cost of using a resource for one purpose is the value of the best alternative use that is given up (Drummond et al., 2015). When the health authority pays a connector, the money could instead have funded home-care hours or a nurse in a long-term care home. When a clinic lends a meeting room to the program, the room could have been used for a foot-care clinic. The budget records the first of these choices, because money changes hands, and misses the second, because nothing is paid. An evaluator who wants to know the full cost of a program therefore distinguishes the financial cost, which is the money spent and recorded in accounts, from the economic cost, which is the value of all resources used, whether or not anyone paid for them.

Three related ideas follow. In-kind contributions are counted at their opportunity cost, sunk costs are left out of forward-looking decisions, and transfer payments are left out of societal costs because they move money without using resources. The distinction between financial and economic cost matters most for programs that rely on partners. Social prescribing programs link people to community groups and volunteer roles that the health system does not pay for, so a costing that reports only the budget makes them appear cheaper than they are and hides costs that community organizations may not be able to absorb as referrals grow.

1.2 The Perspective of the Analysis

The perspective of an economic analysis defines whose costs and consequences are counted. The program perspective counts costs to the organization delivering the program. The health system perspective, called the publicly funded health care payer perspective in Canadian guidance, counts all costs to the public health system, including changes in the use of other services. The societal perspective counts all costs, whoever bears them.

The guidelines for economic evaluation published by CADTH (2017), the organization now called Canada's Drug Agency, specify the publicly funded health care payer perspective for the reference case and allow a broader societal perspective as an additional analysis when important costs fall outside the health system. The Second Panel on Cost-Effectiveness in Health and Medicine recommended that analysts report two reference cases, one from the health care sector perspective and one from the societal perspective, and that they complete an impact inventory listing the consequences of an intervention inside and outside the health sector, even those that are not valued (Sanders et al., 2016). For a program such as Cedar Valley, whose resources and effects extend into the community, the impact inventory makes visible what a health-system analysis leaves out.

1.3 Identifying, Measuring and Valuing Resources

Drummond and colleagues (2015) describe costing as three steps. The evaluator first identifies the resources the program uses, then measures the quantity of each resource, and then values each quantity with a unit cost. The total cost is the sum, across resources, of the quantity multiplied by the unit cost.

Identification starts from the inputs and activities in the program's logic model (Lesson 3). The evaluator walks through each activity, from screening in the clinic to the last follow-up call, and asks what staff time, space, equipment, travel and partner resources it uses. The inventory should be agreed with staff and partners, who know about resources that never appear in the accounts, such as the volunteer who drives participants to their first group meeting.

Measurement records quantities in natural units, such as staff hours, referrals, rides funded and visits to other services, from payroll records, activity logs, case management systems, administrative data (Lesson 5) and participant surveys.

Valuation attaches a unit cost to each quantity. Staff time is valued at salary plus benefits per hour of work. Purchased items are valued at their market price. Physician services are often valued with the provincial fee schedule, which in British Columbia is the Medical Services Plan payment schedule, and hospital and emergency care can be valued with national cost estimates from the Canadian Institute for Health Information. Resources without a market price need a shadow price. Volunteer time can be valued at the wage of a paid worker doing the same task (the replacement cost method) or at what the volunteer would otherwise earn, and the choice changes the result. The value of retired participants' time is contested, and many analyses report their hours without a dollar value.

1.4 Gross Costing, Micro-Costing and Time-Driven Activity-Based Costing

Costing methods differ in how finely they measure resource use. Gross costing, also called top-down costing, divides a total expenditure by a measure of output. For the first full year of Cedar Valley, $840,000 divided by 500 participants who attended a first meeting gives a gross cost of $1,680 per participant. Gross costing is quick and uses data that already exist, but it treats every participant as identical and cannot explain why one clinic or one participant costs more than another.

Micro-costing, or bottom-up costing, measures each resource used to deliver the program to each participant and values it separately (Frick, 2009). It is more demanding, and it is the method of choice when a program is new and no unit costs exist, when costs are expected to vary across participants or sites, or when the evaluation needs to show which components drive the total. Micro-costing data can come from time-and-motion studies, activity logs kept by staff, case management records, or structured interviews with staff about how long tasks take.

Time-driven activity-based costing is a form of micro-costing developed in management accounting by Kaplan and Anderson (2004) and applied to health care by Kaplan and Porter (2011). It needs two estimates for each type of staff: the capacity cost rate, which is the cost of supplying that staff capacity divided by the practical capacity in hours, and the time each activity in the process requires. The cost of serving a participant is the time each activity takes multiplied by the capacity cost rate, summed along a process map of the participant's path through the program.

The Cedar Valley analyst asked connectors to keep an activity log for four weeks and reviewed case records for a sample of closed cases. Participants attended a mean of four meetings (the first plus three follow-ups) of the six allowed. The table and process map summarize mean connector time per participant.

Direct connector time costs $525 per participant, which is less than a third of the gross cost of $1,680. The difference has three sources. The first is the shared costs that the process map does not include: the coordinator, the analyst, training, travel and data systems. The second is the transport fund and the partner grants, which pay for participants' links to the community and lie outside the connector's own work. The third is unused capacity. Seven connectors supply 7 × 1,700 = 11,900 practical hours a year, and 500 participants at 10.5 hours each used 5,250 hours, or 44.1 percent of that capacity. A time study would show how the remaining hours were spent, for example on partner development, clinic liaison, building the resource directory and supporting referred adults who never attended. Kaplan and Anderson (2004) argued that one advantage of time-driven costing is that it makes unused capacity visible, where a gross cost per participant hides it inside a single average.

1.5 Fixed, Variable, Average and Marginal Costs

Costs behave differently as the number of participants changes. Fixed costs, such as connector and coordinator salaries, the analyst and the data system, do not change with the number of participants in the short run. Variable costs, such as transport vouchers and clinician time for referrals, rise with each participant. Step costs are fixed over a range and then jump, as when a full caseload requires an eighth connector. The average cost is the total cost divided by the number of participants, and the marginal cost is the additional cost of serving one more participant.

When a program has spare capacity, the marginal cost of an extra participant is small, because the connectors are already paid. The average cost then falls as the caseload grows, since the fixed costs are spread over more people. Suppose a connector working at full caseload can serve 120 participants a year, which uses 120 × 10.5 = 1,260 hours and leaves 440 of the 1,700 practical hours for partner development, liaison and other work that does not belong to a single participant. Seven connectors could then serve 840 participants a year, and the gross cost per participant at full capacity would be $840,000 ÷ 840 = $1,000.

Costs measured in a start-up year overstate the steady-state cost per participant, while an analysis that assumes full capacity overstates efficiency if referrals never reach that level. A careful costing reports the observed cost, states the program's capacity, and presents the cost at a plausible steady-state caseload as a scenario, which Section 2 does for Cedar Valley.

1.6 The Full Economic Cost of the First Wave

The table below brings the elements of this section together for the first full year of the first wave. The unit costs for overhead, clinician time, rooms and volunteer time are assumptions chosen for teaching, and a real costing would document the source of each one. The health authority applies a corporate overhead rate of 10 percent, clinicians spend time valued at $45 on each screening and referral, each of the 12 clinics provides meeting space valued at $1,600 a year, and community group volunteers contribute about 2,000 hours a year to welcoming and accompanying participants, valued at a replacement wage of $25 an hour.

The full health-system cost is about 16 percent higher than the budget ($972,000 ÷ $840,000 = 1.157). Some societal items were identified and not valued: participants' and caregivers' time and travel, the costs that community organizations bear beyond their partner grants, and any costs that the First Nations health centre bears in hosting a connector beyond the funded position. The evaluation plan proposes to measure participants' time and travel with a short questionnaire and to estimate partner costs in discussion with each partner, including the First Nations health centre, in a way that respects the partner's authority over its own information. Items that cannot be valued are listed in the impact inventory so that readers know what the totals omit.

With the cost of the program established from more than one perspective, the evaluation can ask whether the outcomes justify it. Section 2 combines the health-system cost of $1,944 per participant with estimates of the program's effects to calculate an incremental cost-effectiveness ratio and to judge how confident a decision-maker can be in the result.

Section 1 established that the first wave of the fictional Cedar Valley Connector program costs the health system $1,944 per participant in its first full year. A cost on its own cannot tell a decision-maker whether the program is worth funding. Economic evaluation compares the costs and consequences of a program with those of an alternative, and it asks whether the additional health gained justifies the additional resources used.

2.1 Full and Partial Economic Evaluation

Drummond and colleagues (2015) define a full economic evaluation by two features: it examines both costs and consequences, and it compares two or more alternatives. Studies that lack one of these features are partial evaluations. A cost description of one program, an outcome description of one program, or a comparison of the costs of two programs without their outcomes are all partial evaluations, which can be useful steps toward a full one. Full economic evaluations differ in how they measure consequences.

Cost-consequence analysis is common in public health, where programs affect several outcomes that do not combine easily into one measure. It leaves the weighing of outcomes to the reader, which is transparent but provides no single decision rule. Section 3 shows how an evaluative rubric can structure that weighing.

2.2 The Incremental Cost-Effectiveness Ratio

Economic evaluation compares a program with what would happen without it, which is usually usual care. The key quantity is the incremental cost-effectiveness ratio (ICER), the difference in cost divided by the difference in effect (Weinstein & Stason, 1977).

The ratio is incremental because the decision concerns the additional money and the additional effect. An average ratio, such as total cost divided by the number of people whose loneliness improved, attributes every improvement to the program, including improvements that would have happened anyway.

Both increments come from the evaluation design. For this lesson, assume that a comparison-group evaluation of the first wave (Lesson 7) produced the following illustrative results per participant over twelve months. Emergency department visits were 0.12 lower and primary care visits 0.40 lower than in the comparison group. These figures count participants’ own visits; the Lesson 7 exercise that found a rise in recorded primary care visits across all older patients of first-wave clinics measured a different quantity, which may include visits that connectors arranged or recorded. At assumed unit costs of $450 per emergency visit and $55 per primary care visit, these differences offset 0.12 × $450 + 0.40 × $55 = $54 + $22 = $76 of the program cost. The incremental cost is therefore ΔC = $1,944 − $76 = $1,868 per participant.

2.3 Cost-Utility Analysis and Quality-Adjusted Life Years

Cost-utility analysis addresses the problem of incomparable units by measuring consequences in a common unit, the quality-adjusted life year (QALY). A QALY weights each period of life by a utility value that represents health-related quality of life on a scale on which 1 is full health and 0 is a state equivalent to death. States judged worse than death have negative values. One year in full health is one QALY, and one year at a utility of 0.70 is 0.70 QALYs (Weinstein et al., 2009).

Utilities are elicited from people's preferences between health states. In the standard gamble, a respondent chooses between a certain health state and a gamble between full health and death. In the time trade-off, the respondent chooses between a longer life in the health state and a shorter life in full health. Torrance and colleagues developed the time trade-off at McMaster University in the early 1970s, and Torrance (1986) reviewed the methods. Most evaluations use a generic instrument with a preference-based scoring algorithm, such as the EQ-5D-5L, which has a Canadian value set derived from the time trade-off (Xie et al., 2016), the Health Utilities Index, or the SF-6D.

Suppose Cedar Valley participants complete the EQ-5D-5L at intake, twelve weeks and twelve months. In the illustrative analysis, the groups have the same mean utility at intake, the program group's mean utility is 0.03 higher at twelve weeks, and the difference has returned to zero by twelve months. The area between the profiles is a triangle with a base of one year and a height of 0.03, so the incremental QALYs are ΔE = 0.5 × 1 × 0.03 = 0.015 per participant. If the difference of 0.03 were instead sustained to twelve months, the area would be 0.5 × 0.03 × (12 ÷ 52) + 0.03 × (40 ÷ 52) = 0.0035 + 0.0231 = 0.0265 QALYs.

The base-case ICER is ΔC ÷ ΔE = $1,868 ÷ 0.015 = $124,533 per QALY gained. The figure shows why the assumption about persistence matters: no one measured utility between twelve weeks and twelve months, so the shape of the profile in that interval is a modelling choice that Section 2.5 tests.

QALYs have known limits for programs such as Cedar Valley. Generic instruments such as the EQ-5D-5L describe mobility, self-care, usual activities, pain and anxiety or depression, and they may miss changes in social connection that participants value. Capability measures such as the ICECAP-O, developed for older people (Coast et al., 2008), assess attachment, security, role, enjoyment and control, and some evaluations of social programs report them alongside QALYs. QALYs also raise equity questions, since a QALY is valued equally whoever gains it, and the Cedar Valley evaluation reports results for subgroups so that readers can see how gains and costs are distributed.

2.4 The Cost-Effectiveness Plane and Net Monetary Benefit

The cost-effectiveness plane plots the incremental effect on the horizontal axis and the incremental cost on the vertical axis, with the comparator at the origin (Black, 1990). Its four quadrants classify results.

A line through the origin with a slope equal to the decision-maker's willingness to pay per QALY, written λ (lambda), divides the north-east quadrant. Results below the line are cost-effective at that threshold. Canada has no official threshold. The CADTH (2017) guidelines ask analysts to show results across a range of willingness-to-pay values, and Canadian studies often report results against reference values such as $50,000 and $100,000 per QALY. The figure places the illustrative Cedar Valley results on the plane.

Ratios are awkward to analyze, because the same ICER can arise in the north-east and south-west quadrants and because a ratio becomes unstable when ΔE is near zero. Net monetary benefit avoids these problems by converting health gains into money at the threshold value (Stinnett & Mullahy, 1998).

2.5 Representing Uncertainty

Every input to the Cedar Valley analysis is uncertain. Deterministic sensitivity analysis changes one input, or a set of inputs that form a scenario, and recalculates the result. Probabilistic sensitivity analysis assigns each uncertain parameter a probability distribution, draws a value for every parameter many times, and recalculates the result for each draw, so that the spread of results reflects the joint uncertainty in all parameters (Briggs et al., 2006). The share of draws with a positive NMB at each threshold is plotted as a cost-effectiveness acceptability curve (van Hout et al., 1994). Probabilistic analysis handles uncertainty in parameter values. Uncertainty about structure, such as whether effects persist, is better shown with scenarios, because the answer changes the model itself.

Analyses with time horizons longer than one year also discount future costs and QALYs to present values, and the CADTH (2017) guidelines specify a rate of 1.5 percent a year for the reference case. The Cedar Valley analysis uses a one-year horizon, so it needs no discounting.

2.6 Cost-Benefit Analysis and Social Return on Investment

Cost-benefit analysis values all consequences in money and reports the net benefit (benefits minus costs) or the benefit-cost ratio. Health effects can be valued by willingness to pay, elicited through contingent valuation or discrete choice experiments, or with a monetary value per QALY. If the Cedar Valley health gain is valued at $100,000 per QALY, the benefits per participant are 0.015 × $100,000 + $76 of averted health care costs = $1,576. Against the gross health-system cost of $1,944, the net benefit is $1,576 − $1,944 = −$368 and the benefit-cost ratio is $1,576 ÷ $1,944 = 0.81. The net benefit equals the NMB at λ = $100,000, which shows that a cost-utility analysis with a threshold is a restricted form of cost-benefit analysis.

Social return on investment (SROI) is a form of cost-benefit analysis developed in the social enterprise sector and set out in a guide first published by the United Kingdom Cabinet Office in 2009 and revised by the SROI Network in 2012 (Nicholls et al., 2012). It maps outcomes with interest holders, assigns financial proxies to each outcome, and adjusts for deadweight (what would have happened anyway), attribution (the share due to others), displacement and drop-off. The result is a ratio of social value to investment. SROI has appeal for community programs because it involves interest holders in deciding which outcomes count. A systematic review of SROI in public health found wide variation in methods and in the quality of the proxies and adjustments used (Banke-Thomas et al., 2015).

2.7 Reporting with CHEERS 2022

The Consolidated Health Economic Evaluation Reporting Standards 2022 (CHEERS 2022) replaced the 2013 statement with a 28-item checklist for reporting any health economic evaluation (Husereau et al., 2022). Items cover the title and abstract, the setting and comparators, the perspective, time horizon and discount rate, the selection, measurement and valuation of outcomes and resources, the currency and price date, the rationale and description of any model, and the characterization of uncertainty. CHEERS 2022 added items asking authors to state whether a health economic analysis plan was prepared, to describe how distributional effects were examined, and to report how patients, the public and others affected by the study were engaged and what difference that engagement made. Distributional cost-effectiveness analysis, which examines how costs and health gains fall across social groups, is one way to address the distributional item (Cookson et al., 2017).

An ICER, a probability of cost-effectiveness and an SROI ratio are each one line of evidence about a program. Section 3 turns to the task of combining such evidence with evidence on reach, cultural safety and effectiveness into an overall judgement.

Sections 1 and 2 produced a cost and an economic result for the fictional Cedar Valley Connector program. The evaluation has also produced evidence on reach, cultural safety and changes in loneliness. Decision-makers asked a single evaluative question, whether the program is good enough and good value enough to extend to the second wave, and answering it requires a judgement that brings these lines of evidence together. Lesson 1 introduced evaluative reasoning as a movement from criteria to standards to evidence to synthesis. This section develops the tools that make that synthesis explicit.

3.1 From Findings to Evaluative Conclusions

A descriptive finding states what happened: 78.1 percent of referred adults attended a first meeting. An evaluative conclusion states how good that is: reach was good against the standard the steering committee agreed in advance. Davidson (2005) argues that evaluation is distinguished from other applied research by its obligation to draw conclusions of this second kind, and that an evaluation which reports only descriptive findings leaves the hardest part of the work to readers who have less information than the evaluator.

Scriven's distinctions from Lesson 1 shape the conclusions an evaluation can reach. Merit is the intrinsic quality of a program, judged against criteria such as effectiveness and quality of delivery. Worth is its value in a particular context, which brings in cost, need and the alternatives available to the decision-maker. A program can have good merit and modest worth, as when it works well but costs more than an equally useful alternative. Keeping the two separate lets an evaluation report that the Cedar Valley program delivers what it promises while being candid about its value for money at current caseloads.

Synthesis happens at two levels. Within a criterion, several pieces of evidence are combined into a rating, as when a difference-in-differences estimate, participant interviews and connector records together determine the rating for effectiveness. Across criteria, the ratings are combined into an overall judgement. Each level needs a stated rule, and the rules should be agreed with the primary intended users (Lesson 4) before the evidence arrives.

3.2 Building an Evaluative Rubric

An evaluative rubric sets out the criteria in rows and describes what performance at each level looks like for each criterion. King and colleagues (2013) describe rubrics as a method for surfacing the values of interest holders and for making the basis of a judgement transparent, and Oakden (2013) gives a practical account of developing them with program partners. An analytic rubric rates each criterion separately, while a holistic rubric describes overall levels of performance in a single set of descriptors. Analytic rubrics are more common in program evaluation because they show where a program is strong and where it is weak.

Rubrics are developed in steps. The evaluator drafts criteria from the logic model and the evaluation questions, holds a workshop in which interest holders revise the criteria and draft the descriptors, specifies the evidence for each criterion, agrees the importance of each criterion and any hurdles, and tests the rubric on hypothetical results to check that the descriptors discriminate. For Cedar Valley, the steering committee, including its four older adults with lived experience and its two First Nations representatives, built on the six-month rubric of Lesson 1 to agree the year-one rubric below. The value-for-money criterion follows King (2017), who argues that economic results should enter an evaluation as evidence rated against agreed standards, alongside the other criteria.

The committee agreed two synthesis rules. Cultural safety is a hurdle: the program cannot be rated better than adequate overall if cultural safety is rated poor. The overall judgement of merit cannot be higher than the rating for effectiveness, because the program exists to reduce loneliness. The committee also agreed to report merit and worth separately, with value for money informing worth.

3.3 Approaches to Synthesis Across Criteria

Several approaches exist for combining criterion ratings into an overall judgement, and they can produce different conclusions from the same ratings. Scriven (1991) and Davidson (2005) recommend qualitative approaches for most program evaluations.

3.4 Synthesizing Mixed Evidence

Within a criterion, evidence from different sources can relate in three ways (Greene, 2007). It can converge, when sources point to the same conclusion. It can be complementary, when sources describe different aspects of the same phenomenon, as when a survey measures how much loneliness changed and interviews describe how. It can be dissonant, when sources disagree. Lesson 5 introduced joint displays, which set quantitative and qualitative results side by side for each question and make these relationships visible.

Dissonance deserves investigation. The evaluator asks whether the sources measure the same thing, cover the same people and time, and are equally credible, and then uses the program theory of Lesson 3 to ask whether a mechanism might operate in some contexts and not others. Averaging dissonant evidence into a middle rating hides the most useful information the evaluation has produced.

Every rating should carry a statement of confidence. Confidence depends on the strength of the design, the consistency of evidence across sources, the precision of estimates, how directly the evidence addresses the criterion, and the risk of bias in each source. This reasoning resembles the GRADE approach to the certainty of evidence (Guyatt et al., 2008), applied here to one program. Many evaluations rate confidence as high, moderate or low and give the main reason for each rating.

3.5 Describing the Intervention with TIDieR

A judgement applies to a program as it was actually delivered, and readers can use the judgement only if they know what that program was. Glasziou and colleagues (2008) found that descriptions of interventions in published trials and reviews were often too incomplete for the intervention to be replicated. The Template for Intervention Description and Replication (TIDieR) is a 12-item checklist for describing interventions in reports and protocols (Hoffmann et al., 2014), and TIDieR-PHP adapts it for population health and policy interventions (Campbell et al., 2018). A full description also serves the other parts of this lesson: costing depends on the dose and the staff involved, and an adopter needs both the description and the cost to judge whether the program could work in their setting.

A judgement of merit and worth, a statement of confidence and a clear description of the program are the core content of an evaluation report. Section 4 asks how that content should be written, shown and shared so that decision-makers and communities can use it.

An evaluation that reaches a sound judgement has done most of its work, and it can still fail if the judgement never reaches the people who need it, arrives after the decision, or is written in a form they cannot use. This section covers the report, the executive summary, the presentation of data, knowledge translation and the conditions under which evaluations are used. It ends with the work plan, budget and risk register of an evaluation and a worked example of a complete evaluation proposal for the fictional Cedar Valley Connector program.

4.1 The Evaluation Report

The full report is the reference document of an evaluation. It records what was evaluated, the questions asked, the methods used, the evidence and its limits, and the judgements reached, in enough detail that a reader can check the reasoning. The Program Evaluation Standards introduced in Lesson 1 apply directly: the accuracy standards call for conclusions that are justified by the evidence and reasoning, the utility standards call for timely and appropriate communication, and the propriety standards call for full and fair disclosure of findings, including unwelcome ones (Yarbrough et al., 2011).

Three kinds of statement should be kept distinct throughout. A finding reports evidence: attendance was 64 percent among adults whose first language is neither English nor French. A conclusion makes a judgement against a standard: equity of reach was adequate. A recommendation proposes an action that follows from one or more conclusions: offer interpreter-supported first meetings in all clinics before the second wave. Readers can then trace each recommendation back to its evidence. Good recommendations are specific, feasible, and linked to the conclusions that justify them. They name who would act, and they are developed with the primary intended users so that they fit the resources and authority those users have. Patton (2008) advises that recommendations focus on actions within the control of intended users and state the costs, benefits and challenges of carrying them out.

4.2 Main Messages and Executive Summaries

Main messages are the conclusions and their implications, written as statements a decision-maker could act on. They are distinct from a summary of the contents of the report. The executive summary then gives the context, the questions, the approach in a few sentences, the answer to each question with its confidence, and the recommendations. Writing for decision-makers involves leading with the answer, using plain language and consistent terms, giving numbers with their context (for example, 12 more older adults out of every 100 no longer screening as lonely), and stating uncertainty in words that a non-specialist can interpret.

Evaluation adds two requirements to this general advice. First, the main messages of an evaluation report are its evaluative conclusions from Section 3, each with its level of confidence, so the rubric and the synthesis rules agreed with the steering committee determine what the messages can claim. A message about worth also names the standard behind it, such as the willingness-to-pay value or the rubric's descriptor for good reach, so that the reader can see the basis of the judgement. Second, the executive summary answers the evaluation questions in the order the primary intended users agreed them, so that each user can find the answer to the question they asked. The executive summary of an evaluation proposal, as in Section 4.7, has a different job: it states what the evaluation will deliver, how and when, so that a funder can decide whether to support it.

4.3 Data Visualization for Decision-Makers

Charts in evaluation reports communicate a message to a reader who has little time. Cleveland and McGill (1984) ranked visual encodings by how accurately people judge the quantities they show, and their experiments confirmed the upper part of the ranking: positions along a common scale are judged most accurately, lengths and angles less accurately, and, in their ranking, areas and colour shading least accurately. Dot plots and bar charts therefore support more accurate comparison than pie charts or bubble charts. Tufte (2001) urged designers to maximize the share of ink that shows data and to remove decoration that carries no information, and Evergreen (2017) adapted these ideas for evaluators with practical guidance on choosing and formatting charts for reports.

Economic results need particular care. The cost-effectiveness plane and the acceptability curve are useful for technical readers and are hard for most decision-makers to read. For an executive audience, a short scenario table and a plain statement of probability work better, for example: "If the health authority is willing to pay $100,000 for each year of full health gained, there is about a 28 percent chance that the program is good value at current caseloads."

4.4 Knowledge Translation

The Canadian Institutes of Health Research (CIHR) define knowledge translation as a dynamic and iterative process of synthesis, dissemination, exchange and ethically sound application of knowledge to improve health and strengthen the health system. CIHR distinguishes integrated knowledge translation, in which knowledge users take part in the work from the start, from end-of-grant knowledge translation, in which findings are shared once the work is complete. A utilization-focused evaluation with an engaged steering committee is integrated knowledge translation by design. Lesson 9's Knowledge-to-Action framework (Graham et al., 2006) describes what happens after knowledge reaches users.

In an evaluation, the steering committee is the usual vehicle for integrated knowledge translation. When knowledge users sit on the committee that chooses the questions, agrees the rubric and interprets draft findings, they shape the products and their timing and carry the findings back to their own organizations, which answers the Lavis question of who should transfer the knowledge. A finding is more likely to be acted on when the messenger is someone the audience trusts, which for clinicians may be a respected peer and for a community may be one of its own members, so committee members often present findings to their own constituencies. The table applies these ideas to Cedar Valley, using the five questions introduced in Section 4.2.

The First Nations row reflects commitments made in Lessons 4 and 5. Under the First Nations principles of OCAP® (ownership, control, access and possession), the partners decide how information about their members is interpreted and shared, and findings are reviewed with them before release. Reporting back to communities and participants is an obligation in its own right, separate from its value as a means of influencing decisions.

4.5 Evaluation Use

Research on evaluation use began when evaluators noticed that many evaluations had no visible effect on decisions. Weiss (1979) showed that research is used in many ways besides the direct application of findings, and reviews of evaluation use (Leviton & Hughes, 1981; Cousins & Leithwood, 1986) organized these into types that remain in use today.

Instrumental use is the most visible type, but conceptual use and process use often matter as much over time, because they change how managers and staff think about a program. Cousins and Leithwood (1986) found that use depended on characteristics of the evaluation, such as its relevance, credibility, quality of communication and timeliness, and on characteristics of the decision setting, such as information needs, the political climate, competing information and the receptiveness of users. A later review of the empirical literature by Johnson and colleagues (2009) emphasized the engagement of interest holders throughout the evaluation. Patton (2008) called the presence of an identifiable person or group who cares about the findings the personal factor, and his utilization-focused approach (Lesson 1) builds the evaluation around such primary intended users.

Use also has an ethical side. Misuse occurs when findings are suppressed, distorted or selectively reported, or when an evaluation is commissioned only to justify a decision already made. Evaluators can reduce the risk by agreeing in advance, in the evaluation contract or terms of reference, how findings will be released, who owns the report, and how disagreements about interpretation will be handled, and by keeping Indigenous partners' data governance rights separate from any power to suppress unfavourable findings about the program.

4.6 Work Plan, Evaluation Budget and Risk Register

An evaluation proposal closes with the practical commitments that let a funder judge whether the evaluation can be delivered on time and within its means: a work plan, a budget and a risk register. They apply to the evaluation itself the program work plan and budget outline described in Lesson 2, Section 4.6 (Needs Assessment and Planning Models).

The evaluation work plan

The work plan lists each evaluation task with the person or role responsible, its start and end dates and the output that marks its completion, and a Gantt chart shows the same tasks on a timeline. Its milestones, the dated checkpoints on which later work depends, include ethics approval, the data access agreement, the end of baseline data collection, the interim briefing and the final report. For Cedar Valley, the plan works backward from the second-wave budget decision: the interim briefing must reach the executive before that decision, so baseline data collection and the first analysis are scheduled to finish in time for it. Tasks that depend on outside bodies, such as research ethics review, review under the First Nations data governance agreement and requests for linked administrative data, often take longest and are started first.

The evaluation budget

The evaluation budget is separate from the program budget. Its largest line is usually staff time for the evaluation lead, analysts, research coordinators and interviewers, each costed as hours or full-time equivalents multiplied by a rate that includes benefits. Other common lines are incentives or honoraria for participants and for community members who advise the evaluation, transcription of interviews and focus groups, data access and linkage fees charged by data stewards, translation and printing of instruments, travel to clinics and community meetings, and the production of knowledge translation products. Each line shows its calculation, so that a reviewer can check it and the team can revise it when a quantity changes.

The risk register

A risk register lists what could stop the evaluation from answering its questions on time. For each risk it records the likelihood, the impact, the mitigation and the owner, the person responsible for watching the risk and acting on it. Likelihood and impact are often rated low, medium or high, so that the team attends first to risks that are both likely and serious. The table shows three illustrative entries for Cedar Valley.

4.7 Worked Example: The Cedar Valley Evaluation Proposal

An evaluation proposal brings the parts of an evaluation plan together in one document that a funder can review. The worked example is the 6,000-word proposal that the Cedar Valley evaluation team wrote at the start of the first wave, before any of the illustrative results in Sections 2 and 3 existed. A proposal describes a plan, so its executive summary states what the evaluation will do, why, and how its findings will be used. The table shows how the team allocated the 6,000 words and which lesson covers each part.

Bringing It All Together

This lesson completed the evaluation plan by asking what a program costs, whether it is good value, how evidence is combined into a judgement, and how that judgement reaches the people who will use it. Section 1 showed that the budget of the fictional Cedar Valley Connector program ($840,000) understates its economic cost, which rises to $972,000 from the health-system perspective and to $1,022,000 when volunteer time is valued, and that time-driven activity-based costing puts the connector time for each participant at $525. It also showed how spare capacity in a start-up year inflates the average cost per participant. Section 2 used these costs in an illustrative cost-utility analysis, which gave an incremental cost of $1,868, a gain of 0.015 QALYs and an ICER of $124,533 per QALY, with scenarios ranging from $41,617 to $131,200 and a probability of 0.282 of being cost-effective at $100,000 per QALY.

Section 3 placed the economic result beside evidence on reach, equity, cultural safety and effectiveness in a rubric agreed in advance, and reached separate judgements of merit (good) and worth at year-one caseloads (adequate), each with a stated level of confidence. Section 4 turned to reports, executive summaries, data visualization, knowledge translation and the conditions for use, and ended with the work plan, budget and risk register of an evaluation and a worked example of a complete evaluation proposal. The lesson's main argument is that economic and other evidence serve decisions best when they are combined openly, with the standards agreed before the results arrive and the findings delivered in time and in forms that each audience can use.

The final reflection asks you to bring the lesson together by turning the evidence from an evaluation into main messages, a recommendation and an explanation of an economic result for a decision-maker.

Final Knowledge Assessment

This glossary defines the terms, tools and people introduced in Lesson 10, grouped by the lesson's main themes.