Introduction

The first three lessons described a program, its need and its theory. None of those products says what the evaluation should ask, and an evaluation that tried to answer every question about a program would answer few of them well. The evaluator's next task is to decide whose questions the evaluation will answer and how those people will take part. The 2024 CDC Program Evaluation Framework places this work in its first step, Assess context, which asks who the interest holders are, and in its third step, Focus the evaluation questions and design, which asks who the intended users are and what they need to know (Kidder et al., 2024). Its cross-cutting action, Engage collaboratively, applies at every step.

This section identifies and maps interest holders, Section 2 sets out how they can be engaged, Section 3 addresses equity and Indigenous evaluation, and Section 4 turns interests into prioritized evaluation questions. Lesson 1 explained why this course uses the term interest holder. Several frameworks below were published under the older term, and this lesson describes them in the current vocabulary.

1.1 Who Counts as an Interest Holder

An interest holder is a person, group or organization with an interest in a program or its evaluation. The term covers those who decide about, deliver, take part in or are affected by the program, and those who will use or be affected by the evaluation's findings. The definition is deliberately broad, because a list built from an organizational chart tends to omit the people a program serves, the people it fails to reach, and the communities whose resources it draws on.

Guba and Lincoln (1989), in their account of fourth generation evaluation, sorted interest holders into three classes. Agents are the people who develop, fund, deliver and use the program, such as the Cedar Valley executive, the coordinator, the connectors, the referring clinicians and the community organizations that receive partner grants. Beneficiaries are the people who gain from it, including referred older adults, their families and the community groups that gain members. Guba and Lincoln called their third class victims: people disadvantaged by the program or its evaluation because they are excluded, bear its costs, or lose an opportunity. At Cedar Valley this class could include lonely older adults with no regular clinician, older adults in second-wave clinics who wait a year, and volunteer-run groups that receive more referrals than they can absorb. Guba and Lincoln argued that an evaluation which hears only from agents will reflect only the agents' view of the program.

Rights holders

In Canada, First Nations, Inuit and Métis governments and organizations are often described as rights holders. The term recognizes the inherent right of self-determination and the Aboriginal and Treaty rights affirmed in section 35 of the Constitution Act, 1982. A First Nation that partners with a health authority takes part as a government with authority over matters that affect its members, and its role in an evaluation is set by agreement between the Nation and the health authority. An evaluator who treats a First Nation as one more interest group, to be consulted in the same way as a seniors' club, misreads the relationship. Section 3 develops the principles that follow.

Methods for identifying interest holders

Evaluators usually combine several methods. A brainstorm with the core team produces a first list, which a review of program documents extends. Each column of the logic model from Lesson 3 implies a set of people: inputs point to funders and partners, activities to staff, and outcomes to the people served and those around them. A snowball step asks each person on the list who else has an interest. Bryson (2004) describes a sequence of such techniques, which Bryson, Patton and Bowman (2011) adapted for evaluation. A final check asks who is missing, such as people who were referred and never attended, people never referred, and people with limited English or no transportation. Section 3 introduces PROGRESS-Plus, a checklist that supports this check.

1.2 Primary Intended Users

An evaluation cannot give every interest holder an equal say in every decision. Michael Quinn Patton's utilization-focused evaluation addresses the question of whose needs come first with the idea of primary intended users: specific, identifiable people who have a stake in the findings and the capacity to act on them (Patton, 2008). Patton's research on how federal health evaluations in the United States were used found that use depended most on what he called the personal factor, which is the presence of an identifiable individual or small group who personally cares about the evaluation and the findings it produces.

Primary intended users therefore differ from audiences. An audience is any group that will receive findings, such as the health authority board, clinicians across the region, other health authorities and the public. Primary intended users are a small group, named in the evaluation plan, who work with the evaluator to set the purpose, choose the questions, review the design, interpret the findings and decide what to do with them. They also differ from participants, who provide data, although one older adult on the evaluation working group may be all three.

Patton describes the qualities of a good primary intended user. The person cares about the findings, has time to take part, holds authority or influence over a decision the evaluation can inform, is open to unwelcome evidence, and is credible with a group whose interests matter. Because people change jobs, Patton also advises working with several users so that the evaluation survives turnover.

1.3 Mapping Interest Holders

Mapping interest holders helps the evaluator decide how much attention each needs and in what form. Two tools from strategic management are widely used.

The power-interest grid

Read as a management tool, the grid suggests that the people with most power deserve most attention and that subjects need only be kept informed. An evaluator may instead decide to strengthen the voice of subjects by giving them a formal role and the support to use it, as the ethics of evaluation often require. The Program Evaluation Standards discussed in Lesson 1 include propriety standards on responsive and inclusive orientation and on human rights and respect (Yarbrough et al., 2011). In the Cedar Valley case, referred older adults have the highest interest of any group and little formal power, so the evaluation plan gives two of them seats on the evaluation working group and pays them for their time. The grid also captures a single moment, and a group's position can shift quickly if media attention or a budget decision raises the stakes.

The salience model

The salience model extends the grid recalled in the Background box above: it keeps power and replaces interest with two further attributes, legitimacy and urgency. The model of Mitchell, Agle and Wood (1997) asks how much attention managers give to each interest holder and explains that attention by three attributes. Power is the ability to impose one's will on the program through authority, money or public influence. Legitimacy is a socially accepted claim to have a say, grounded in a contract, a right, a moral claim or a risk borne. Urgency is the degree to which a claim calls for immediate attention, because it is time-sensitive or matters greatly to the claimant. Combining the three attributes gives seven types. Groups with one attribute are latent, groups with two are expectant, and groups with all three are definitive.

Mitchell, Agle and Wood stressed that salience is dynamic. A dependent group can become definitive by forming an alliance with a powerful one. The model is also descriptive: it explains how much attention groups tend to receive from managers. An evaluator still has to decide, on ethical and methodological grounds, how much attention each group should receive, and the two answers often differ.

1.4 How Roles Differ Across Interest Holders

Interest holders want different things from an evaluation, contribute different kinds of knowledge, and face different risks from taking part. Planning their roles well means stating those differences openly. The tabs summarize the patterns an evaluator should expect, using the Cedar Valley case.

One person can hold several roles. The connector hosted by the First Nations health centre is a member of staff and may also belong to the community the partnership serves. A good map records such overlaps, because they affect who can speak for whom and where conflicts of interest may arise.

1.5 Building an Interest Holder Map

An interest holder map brings identification and analysis together in a single table. It lists each interest holder, states their main interest in the evaluation, describes their influence over the program or the evaluation, and proposes a role for them. Section 2 adds the level of engagement for each group. The map is a working document that belongs in the evaluation plan, where it shows readers whose perspectives shaped the evaluation. The table is the Cedar Valley team's first map.

A map leaves open how each group will take part. Section 2 turns to that question.

Introduction

Section 1 produced a map of who has an interest in the evaluation of the fictional Cedar Valley Connector program and how much influence each group holds. A map leaves open the question of how each group takes part. Some groups need only to be told what the evaluation found, some should be asked for their views, and some should share decisions about the evaluation itself. This section offers a vocabulary for those choices, describes the family of evaluation approaches built around participation, sets out the Canadian framework for engaging patients as partners, and considers how evaluators handle the differences in power and the conflicts that engagement brings to the surface.

2.1 Why Engage Interest Holders

Evaluators give four main reasons for engaging interest holders. The first is use. Patton's work on the personal factor, introduced in Section 1, suggests that people who help shape an evaluation are more likely to understand, trust and act on its findings. The second is quality. People who deliver or receive a program know things the evaluator does not, such as which survey items will confuse participants, which records are unreliable, and which explanations for a finding fit local conditions. The third is ethics and rights. People whose lives a program affects have a claim to a say in how it is judged, and for rights holders that claim rests on self-determination. The fourth is capacity, since involvement can leave behind skills that outlast the project.

Engagement also has costs. It takes time and money, adds delays, and can raise expectations that the evaluation cannot meet, and weaken the perceived independence of findings if decision-makers suspect that the program's supporters shaped them. A review of empirical research on involvement in evaluation by Brandon and Fukunaga (2014) found that most studies were descriptive and that evidence on the effects of involvement was limited. Evaluators should therefore state why they are engaging each group, choose the form of engagement that serves that reason, and check during the evaluation whether it does.

Arnstein's ladder

For an evaluator, the ladder offers a test of an evaluation plan: if participants' views cannot change any decision, the plan should say so and avoid calling the arrangement a partnership.

2.2 The IAP2 Spectrum of Public Participation

The spectrum is most useful when it is applied separately to each group and each decision. Engagement in an evaluation is rarely uniform: the same group may collaborate on the choice of questions, be consulted on the survey, and be informed about the analysis plan. The promise at each level matters as much as the label, because offering more influence than decision-makers will honour damages trust. The table applies the levels to the Cedar Valley evaluation.

The last row needs a comment. The IAP2 spectrum was written for public participation in decisions that a government or organization owns. The authority of a First Nation over its own pathway and its members' data rests on self-determination, so the arrangement is better understood as a decision that belongs to the Nation, which the health authority supports, than as the top rung of a participation scale that the health authority controls.

Engagement at each stage of an evaluation

The accordion lists the main opportunities for engagement, with the Cedar Valley plan for each.

2.3 Participatory, Collaborative and Empowerment Evaluation

A family of evaluation approaches places the involvement of interest holders at the centre of the evaluation itself. Lesson 1 placed these approaches on the use and valuing branches of the evaluation theory tree. Fetterman, Rodríguez-Campos and Zukoski (2018) distinguish three members of the family mainly by who controls the evaluation.

In collaborative evaluation, the evaluator remains in charge of the evaluation and works in an ongoing partnership with interest holders, who contribute to decisions at points the evaluator and the partners agree on. Rodríguez-Campos and Rincones-Gómez (2013) set out a Model for Collaborative Evaluations that structures this partnership. A Canadian-led team, Shulha and colleagues (2016), studied the practice of evaluators who used collaborative approaches and proposed eight evidence-based principles for collaborative approaches to evaluation. In summary, the principles ask evaluators to clarify why collaboration is wanted, to build meaningful relationships, to develop a shared understanding of the program, to promote participatory processes suited to the setting, to monitor and respond to the resources available, to monitor the progress and quality of the evaluation, to promote evaluative thinking, and to follow through so that the evaluation is used.

In participatory evaluation, the evaluator and interest holders share control. Cousins and Whitmore (1998) distinguished two streams. Practical participatory evaluation aims to support program decision-making and use, and it usually involves primary users such as managers and staff. Transformative participatory evaluation aims to empower people who have been marginalized and to support social change, and it draws on traditions of participatory action research. Cousins and Whitmore described any participatory evaluation along three dimensions: who controls technical decisions about the evaluation, how widely participants are drawn from different interest holder groups, and how deeply participants are involved across the stages of the evaluation.

In empowerment evaluation, program staff and community members conduct the evaluation themselves, and the evaluator acts as a coach or critical friend. David Fetterman introduced the approach in the 1990s (Fetterman, 1994), and Fetterman and Wandersman (2005) set out ten principles: improvement, community ownership, inclusion, democratic participation, social justice, community knowledge, evidence-based strategies, capacity building, organizational learning and accountability. A common three-step form asks a group to agree on its mission, take stock of its current activities by rating them, and plan for the future with goals, strategies and evidence. Getting To Outcomes (Chinman, Imm & Wandersman, 2004) is a structured version for community prevention programs. Critics have questioned whether self-evaluation can produce credible judgements of merit, and a review by Miller and Campbell (2006) found that many projects described as empowerment evaluations followed its principles only loosely.

A single evaluation can combine these approaches across its strands, as the Cedar Valley plan does, provided the plan states which decisions are shared and which are reserved.

2.4 Patient Partners and the SPOR Patient Engagement Framework

The framework was written for research, and evaluation teams in Canadian health authorities often adopt it as a standard for partnership with patients. The Cedar Valley evaluation working group has done so, and each principle shapes its practice. The older adult partners include a man who uses a wheelchair and a woman whose first language is Punjabi, and the team asks them to help reach others like them, while recognizing that no individual can speak for a whole group. Partners receive an orientation to evaluation, meeting papers a week in advance, and a debrief call after each meeting. The working group’s terms of reference name lived experience as one of the forms of expertise the group relies on, and the older adult partners joined before the evaluation questions were drafted, so that the questions reflect their priorities from the start.

Several practices make partnership work. At least two partners from a constituency should sit on any group, so that no one speaks alone. Written terms of reference state what partners will do, what decisions they share, and how they can step back, and meetings are scheduled with partners' health, caregiving and transportation in mind. Engagement should itself be evaluated: the Public and Patient Engagement Evaluation Tool developed at McMaster University (Abelson et al., 2016) asks partners and organizers to rate the quality of engagement, and the GRIPP2 checklist (Staniszewska et al., 2017) guides reporting of patient and public involvement in publications.

Compensation

Partners give time and expertise, and paying them recognizes that contribution. CIHR has issued guidance on paying patient partners, and many health authorities and research units publish rates. The usual principles are to pay for preparation as well as meeting time at a stated rate, to reimburse expenses such as transportation and caregiving promptly or in advance, to offer a choice of payment method and the option to decline payment, and to budget for compensation from the start. Payments can affect income-tested benefits and may be taxable, so partners should be told this and offered choices that suit their circumstances. Honoraria and gifts for Elders and knowledge keepers follow the protocols of the Nation or community involved, which differ among Nations and are agreed with the partners.

Support and fair pay make partnership possible, and they leave open how power is shared once partners are in the room. Section 2.5 takes up that question.

2.5 Managing Power and Conflict

Bringing interest holders together brings their differences in power into the room. Power in an evaluation comes from formal authority, control of money and data, professional expertise and its vocabulary, fluency in the dominant language, health and energy, and history, including the colonial history that shapes relationships between health systems and Indigenous Peoples. When these differences go unaddressed, engagement slides toward the tokenism that Arnstein described: partners attend meetings, the people with power make the decisions, and the evaluation reports that all voices were heard.

House and Howe (1999) proposed deliberative democratic evaluation as one response. It rests on three principles: inclusion of all relevant interests, dialogue among them, and deliberation that reaches defensible conclusions through reasoned discussion. Several practical strategies follow. Terms of reference should state decision rules in advance, such as consensus with a stated fallback, so that decisions do not default to the most senior person present. Partners with less power can meet before main meetings to prepare, and the evaluator can brief them on technical points in plain language. Facilitation methods such as round-robin turns and anonymous ranking reduce the advantage of people who speak most. Co-chairing by a partner and a manager shares control of the agenda. When disagreement remains, the report can record it, so that minority views reach decision-makers.

Conflicts among interest holders are normal in evaluation, and naming their source helps resolve them. Moore (2014), writing on mediation, distinguishes conflicts over data, relationships, values, structures and interests. In evaluation, value conflicts concern what counts as success, and interest conflicts concern jobs, budgets and reputations. The evaluator also has interests. An internal evaluator employed by the health authority should disclose that position and agree how independence will be protected, consistent with the propriety standard on conflicts of interest discussed in Lesson 1.

Engagement decides who is in the room. Equity-focused evaluation asks whether the evaluation, and the program it judges, serve the people with the greatest need, and Indigenous evaluation asks whose knowledge and authority shape the work. Section 3 takes up both questions.

Introduction

Sections 1 and 2 asked who has an interest in an evaluation and how they take part. This section asks two further questions. The first is whether the program and its evaluation serve the people with the greatest need, which is the concern of equity-focused evaluation. The second is whose knowledge, values and authority shape the evaluation, which is central to Indigenous and culturally responsive evaluation. Lesson 1 introduced culturally responsive and Indigenous evaluation as approaches on the valuing branch of the evaluation theory tree, and its Section 3 described Chapter 9 of TCPS 2 on research involving First Nations, Inuit and Métis Peoples. This section develops the principles that guide practice. In the fictional Cedar Valley Connector program these questions are concrete: some older adults face more barriers to taking part than others, the program has a partnership with First Nations, and one First Nation is co-designing a land-based pathway.

3.1 Equity-Focused Evaluation

Whitehead (1992) defined health inequities as differences in health that are unnecessary and avoidable and that are also considered unfair and unjust. Equity-focused evaluation judges whether a program contributes to more equitable outcomes, with particular attention to the groups who are worst off, and it does so through processes that are themselves equitable (Bamberger & Segone, 2011). It adds four questions to an evaluation: who the program reaches and who it misses, whether its benefits are distributed fairly, whether it narrows or widens existing gaps, and whether the evaluation's own methods exclude some groups. The 2024 CDC framework, described in Lesson 1, makes advancing equity a cross-cutting action at every step of an evaluation.

The third question deserves emphasis. Lorenc and colleagues (2013) reviewed systematic reviews and found that some kinds of interventions, particularly those that depend on people choosing to take part and having the resources to do so, can widen inequalities even when they improve average outcomes. Programs of this kind generate inequalities because more advantaged people take them up more often or benefit more. The Cedar Valley program depends on a referral from a primary care clinician and on travel to community activities, so older adults without a regular clinician, without transportation or with limited English may benefit less. An evaluation that reports only average change among participants would miss this.

Checking for the groups that matter

PROGRESS-Plus is a checklist of social factors that stratify health opportunities and outcomes (O'Neill et al., 2014). The acronym stands for place of residence, race, ethnicity, culture and language, occupation, gender and sex, religion, education, socioeconomic status and social capital, and the Plus adds factors such as age, disability and sexual orientation, together with features of relationships and time-dependent circumstances. Evaluators use the checklist at three points: when identifying interest holders, to see who is missing from the map; when choosing data to collect, so that results can be disaggregated by the factors most likely to matter; and when analyzing, to compare reach and outcomes across groups.

Equity also concerns measurement. The three-item UCLA Loneliness Scale was developed in English in the United States, and translated versions need testing to confirm that the items carry the same meaning for older adults who speak other languages. Survey modes matter as well: an online-only follow-up would under-represent older adults without internet access. Disaggregation brings its own constraints. Subgroups can be small, which makes estimates imprecise and raises the risk of identifying individuals, so evaluators plan sample sizes with subgroup comparisons in mind, suppress small cells in reports, and combine quantitative comparisons with interviews that explain the differences found.

3.2 GBA Plus and Intersectionality

Gender-based Analysis Plus (GBA Plus) is the Government of Canada's analytical process for assessing how diverse groups of women, men and gender diverse people may experience policies, programs and initiatives. Women and Gender Equality Canada supports its use across federal departments. The Plus signals that the analysis goes beyond sex and gender to consider other identity factors that intersect with them, such as age, disability, education, ethnicity, Indigeneity, income, language, geography and sexual orientation. GBA Plus distinguishes sex, a set of biological attributes, from gender, which refers to socially constructed roles, behaviours, expressions and identities. Statistics Canada's 2021 Census was the first to ask separately about sex at birth and gender, and evaluators who collect these data should follow similar practice.

GBA Plus draws on the idea of intersectionality, a term introduced by the legal scholar Kimberlé Crenshaw (1989) to describe how race and gender together shaped the discrimination that Black women faced in ways that neither category alone could explain. Hankivsky and colleagues at Simon Fraser University developed an Intersectionality-Based Policy Analysis framework (Hankivsky, 2012) with guiding principles and a set of descriptive and transformative questions that can be adapted to evaluation. An intersectional evaluation asks how combinations of social positions shape access to and benefit from a program, and it treats those positions as products of power relations, which differ from fixed traits of individuals.

GBA Plus and intersectional analysis complement the Health Equity Impact Assessment introduced in Lesson 2. The assessment is mainly a planning tool that anticipates a program's effects on equity, and the analysis described here carries the same concern into the evaluation, where it tests whether those effects occurred.

3.3 Indigenous Evaluation

Indigenous evaluation in Canada takes place in a specific context. The Truth and Reconciliation Commission's Calls to Action (2015) include calls to identify and close gaps in health outcomes between Indigenous and non-Indigenous people and to recognize the value of Indigenous healing practices. In British Columbia, the In Plain Sight report (Turpel-Lafond, 2020) documented widespread Indigenous-specific racism in the provincial health care system. The First Nations Health Authority has held responsibility for federal First Nations health programs in the province since 2013, and the Declaration on the Rights of Indigenous Peoples Act (2019) commits the province to aligning its laws with the United Nations Declaration on the Rights of Indigenous Peoples. An evaluator working with First Nations, Inuit or Métis communities works within this history and these commitments.

Two further points guide the rest of this section. First, Canadian practice is distinctions-based: First Nations, Inuit and Métis Peoples are distinct, and there are more than 200 First Nations in British Columbia alone, each with its own governance, language, laws and protocols. Second, the principles below come from published Indigenous scholarship and from national First Nations organizations. They are starting points for an evaluator. The protocols for any particular evaluation are learned from, and set by, the Nation or community involved, and practices described by one Nation should not be treated as rules for another.

Relational accountability

Shawn Wilson, an Opaskwayak Cree scholar, set out an Indigenous research paradigm in Research Is Ceremony (Wilson, 2008). In this paradigm, knowledge is relational: it exists in relationships with people, with the land and environment, with ideas and with the cosmos. Relational accountability follows from this view. A researcher or evaluator is accountable to all of these relationships, and the quality of the work is judged partly by whether it honours them through respect, reciprocity and responsibility. These values echo the four Rs of respect, relevance, reciprocity and responsibility (Kirkness & Barnhardt, 1991) introduced in Lesson 1.

For an evaluator, relational accountability changes the order and pace of work. Relationships come before data, and building them takes time that a project plan must allow for. The evaluator asks what obligations the relationship creates, including obligations to share findings in forms the community can use, to bring benefit to the community, and to maintain contact after the report is delivered. The questions worth asking are those the community recognizes as its own. At Cedar Valley the evaluation team meets the First Nations partners several times before any evaluation questions are drafted, and the plan budgets for return visits to share findings.

Two-Eyed Seeing

In evaluation, Two-Eyed Seeing asks that Indigenous and Western ways of knowing each contribute with their own strengths, and that neither be reduced to an input for the other. For the Cedar Valley land-based pathway, Western methods might contribute loneliness scores and records of participation, while the Nation's own understanding of wellness and connection, carried in stories, teachings and the guidance of Elders, defines what success looks like and how it is recognized. Writers on Two-Eyed Seeing caution that the phrase is sometimes used as a label for adding Indigenous content to a design that remains entirely Western. Practising it requires ongoing co-learning, Indigenous knowledge holders who lead the Indigenous side of the work, and decisions about how the two perspectives come together that are made jointly.

OCAP® and First Nations data governance

OCAP® is a registered trademark of the First Nations Information Governance Centre (FNIGC). The acronym stands for ownership, control, access and possession, and the principles assert that First Nations have jurisdiction over information about their communities and members. FNIGC is the steward of the principles and offers training on them, and evaluators who work with First Nations data should complete that training and follow FNIGC's guidance. OCAP® applies to First Nations. Inuit and Métis organizations have set out their own approaches to research and data, such as the National Inuit Strategy on Research of Inuit Tapiriit Kanatami (2018), and an evaluation involving those Peoples follows their guidance. Internationally, the CARE Principles for Indigenous Data Governance (Carroll et al., 2020) state related commitments to collective benefit, authority to control, responsibility and ethics.

An evaluation that falls outside research ethics board review under TCPS 2 Article 2.5, as Lesson 1 described, still carries these obligations, and many Nations have their own research and data review processes. The Cedar Valley team and the First Nations partners therefore agree in writing, before data collection begins, which data about the Nation's members will be collected, where they will be stored, who may analyze them, how findings will be reviewed before release, and what happens to the data when the evaluation ends. Health authority administrative data, such as emergency department visits, also become data about the Nation's members once they are linked to identify First Nations clients, so any such linkage is covered by the same agreement.

Indigenous evaluation in practice

The accordion gathers general principles from the frameworks above and from Indigenous evaluation scholarship, including LaFrance and Nichols (2009) and Kovach (2009). Each principle is applied through the protocols of the specific Nation or community.

3.4 Culturally Responsive Evaluation

Culturally responsive evaluation places the culture and context of the community served at the centre of every stage of an evaluation (Hood, Hopson & Kirkhart, 2015). Frierson and colleagues (2010) set out a framework that considers culture at each stage, from preparing for the evaluation and engaging interest holders, through framing questions, designing the evaluation and adapting instruments, to collecting and analyzing data and disseminating and using the results. Kirkhart (1995) argued that culture is a matter of validity: conclusions reached through measures, methods or relationships that participants do not recognize may be wrong, and she called the relevant standard multicultural validity.

Culturally responsive practice begins with the evaluator. Tervalon and Murray-García (1998) proposed cultural humility as a lifelong commitment to self-reflection and critique, to addressing power imbalances, and to building partnerships with communities, and they contrasted it with the idea of cultural competence as a body of knowledge that can be mastered. An evaluator practising cultural humility examines how their own background and position shape what they notice and what they take for granted, and they build teams that include people who share the lived experience of the communities served.

At Cedar Valley, culturally responsive practice affects instruments, data collection and interpretation. The region includes older adults who speak Punjabi, Cantonese or Tagalog at home, so the team arranges translation of the follow-up survey with cognitive testing in each language, offers interviews in participants' preferred languages with trained interviewers from the same communities, and asks community members to help interpret findings for their groups. Loneliness and belonging may be expressed differently across cultures, and older adults may judge the community activities on offer by different standards, so interviews ask participants what connection means to them as well as whether they feel lonely.

The first three sections have produced a map of interest holders, a plan for engaging them, and commitments to equity and Indigenous data governance. Section 4 uses these to draft and prioritize the evaluation questions, test whether the program is ready to be evaluated, and set out the structure of the evaluation matrix.

Introduction

The first three sections of this lesson identified the interest holders in the fictional Cedar Valley Connector program, planned how they will take part, and set out commitments to equity and to First Nations governance of data. This section turns those interests into the questions the evaluation will answer. The questions are the centre of an evaluation plan, because they determine the design, the data, the analysis and the report.

4.1 What Makes a Good Evaluation Question

Evaluation questions are the small number of high-level questions that the evaluation as a whole will answer. They differ from research questions, which seek knowledge that generalizes beyond the setting, and from survey items, which are the questions put to respondents. Davidson (2005) recommends that evaluations be organized around a few big-picture questions that are explicitly evaluative, asking how good, how well or how valuable something is, so that the evaluation ends in a judgement and goes beyond a description of what happened.

Morra Imas and Rist (2009) classify evaluation questions into three types. Descriptive questions ask what is happening, such as who was referred and what services they received. Normative questions compare what is happening with what should happen, such as a target, a standard or a rubric level of the kind introduced in Lesson 1. Cause-and-effect questions ask whether the program caused a change, which requires a credible comparison of the kind that Lessons 6 to 8 develop. Rossi, Lipsey and Henry (2019) classify questions by the aspect of the program they address: need, design and theory, implementation, outcomes and impact, and efficiency.

Good evaluation questions are linked to a decision and a named primary intended user, answerable with evidence obtainable in time, specific about population, outcome and time period, traceable to a link in the logic model, neutrally worded, and alert to differences between groups. The table shows weak questions from an early Cedar Valley brainstorm and the revisions the working group agreed.

4.2 Drafting Questions with Primary Intended Users

Cronbach (1982) proposed that evaluators identify questions in two phases. In the divergent phase, the evaluator gathers as many potential questions as possible from many sources. In the convergent phase, the evaluator and the users select the questions the evaluation will answer. Fitzpatrick, Sanders and Worthen (2011) list sources for the divergent phase, including interest holders' concerns, the logic model, evaluation approaches, research literature and standards, expert consultants, and the evaluator's own judgement. For the convergent phase they suggest asking, of each candidate question, who would use the answer and how, whether the answer would reduce uncertainty or add information not already available, whether it bears on the program's future, and whether it can be answered with the resources and time available.

Patton (2008) adds a test of intended use. Before a question is adopted, the evaluator shows the primary intended users simulated findings, invented results laid out as they would appear in the report, and asks what they would do if the findings came out that way and what they would do if the findings came out the other way. A question whose possible answers would lead to the same action, or to no action, is a candidate for removal or rewording. Preskill and Jones (2009) describe a practical sequence for engaging interest holders in developing questions, and the Cedar Valley working group followed a similar sequence in two workshops.

4.3 Prioritizing Questions

Most evaluations generate more good questions than they can answer, and explicit criteria make the choice transparent. The Cedar Valley working group agreed four criteria before scoring, each scored from 1 (low) to 3 (high). Decision relevance asks whether the answer would change the second-wave decision or a delivery decision, and the group gave it double weight because informing that decision is the evaluation's main purpose. Importance to interest holders asks how much the question matters to the groups on the map, with deliberate attention to groups with less power. Answerability asks whether credible evidence can be obtained with the available resources before the decision. Equity relevance asks whether the answer would show whether the program serves the people with the greatest need.

The scores structure the discussion, and the final choice remains a judgement made by the primary intended users. Three candidates tied at 12, and the group resolved the tie by merging C2 and C3 into a single effectiveness question, since the pre-post change in C2 is one of the sub-questions needed to answer C3. C5 and C7 became sub-questions of the reach question, since fidelity and community capacity both help explain who benefits. C4 was deferred to a second phase: the executive values it, but a year of follow-up and a data linkage agreement mean that no answer could reach the second-wave decision. The plan records the deferral and the reason, so that the question is not lost.

Weights are value judgements, so they should be agreed before scoring and reported with the results. Scoring by a group dominated by one interest can also give that interest's priorities an appearance of objectivity, which is why the Cedar Valley group included older adult partners. The First Nations partners' question was handled differently. Consistent with the partnership agreement described in Section 3, the partners proposed and worded their own question, and the working group included it in the set by agreement without scoring it against the health authority's criteria.

4.4 Evaluability Assessment

Evaluability assessment asks whether a program is ready to be evaluated in a way that will be useful. Joseph Wholey and his colleagues developed it at the Urban Institute in the 1970s after observing that many federal program evaluations in the United States produced findings that no one used, often because the programs had unclear goals, implausible theories or no usable data (Wholey, 1979). Wholey proposed that a program is ready for a useful evaluation when four conditions hold: its goals and the information its managers need are well defined and agreed; its objectives are plausible given its resources and activities; relevant data can be obtained at reasonable cost; and the intended users agree on how they will use the findings.

Leviton and colleagues (2010) reviewed the use of evaluability assessment in public health and described it as a low-cost pre-evaluation step that can also improve a program's design. Its usual steps are to involve the intended users, clarify the intended program through the logic model, explore how it actually operates, judge the plausibility of its theory, check the data, and agree on scope and use. Davies (2013) distinguished three aspects of evaluability: evaluability in principle, which concerns the program's design and theory; evaluability in practice, which concerns the availability of data; and the institutional context, which concerns whether an evaluation would be useful and practical. An evaluability assessment can recommend evaluating the whole program now, evaluating some components now and others later, improving the program's design or data before evaluating, or postponing evaluation until a decision is in view.

The assessment changed the Cedar Valley plan in two ways. The impact question can be answered only if baseline data are collected in second-wave clinics before they start the program, so screening and measurement begin there at once, and the emergency department question moves to a later phase. Lesson 6 returns to this assessment when it asks whether randomization is feasible for the program.

4.5 The Evaluation Matrix

The evaluation matrix is the main planning table of an evaluation. It has one row for each evaluation question or sub-question, and its columns state how each will be answered: the indicators that will be measured, with the criteria or standards that define good performance; the data sources; the methods of data collection and analysis; the timing of collection and reporting; and the person or team responsible. Some evaluators add columns for the design and for the groups by which results are disaggregated. A completed matrix shows whether every question has evidence and whether any data lack a question. This lesson introduces the structure, and Lesson 5 develops indicators, data sources and data systems, culminating in a completed matrix for Cedar Valley.

4.6 Worked Example: The Cedar Valley Interest Holder Map and Evaluation Questions

This worked example brings the lesson together for Cedar Valley in an interest holder map and four prioritized evaluation questions with a justification. The map extends the table in Section 1.5 with the engagement level from Section 2.

Bringing It All Together

This lesson moved the evaluation of the fictional Cedar Valley Connector program from a described program to a focused evaluation. It began by identifying interest holders, including rights holders and the people a program does not reach, and by distinguishing the small group of primary intended users from the wider audiences who receive findings. The power-interest grid and the salience model helped map interest holders, and the lesson noted that both describe how attention tends to be distributed, which leaves the evaluator to decide on ethical grounds how it should be distributed.

The lesson then placed engagement on the IAP2 Spectrum of Public Participation, distinguished collaborative, participatory and empowerment evaluation by who controls the evaluation, and set out the SPOR Patient Engagement Framework, compensation and strategies for managing power and conflict. Equity-focused evaluation, PROGRESS-Plus and GBA Plus direct an evaluation to the people with the greatest need. Indigenous evaluation rests on relational accountability (Wilson, 2008), Two-Eyed Seeing as brought forward by Mi'kmaw Elder Albert Marshall, and the First Nations principles of OCAP®, a registered trademark of the First Nations Information Governance Centre, applied through the protocols of each Nation.

Finally, the lesson turned interests into evaluation questions. Good questions are evaluative, answerable, specific and tied to decisions. Questions are drafted in divergent and convergent phases, tested with simulated findings, prioritized against criteria agreed in advance, and checked by an evaluability assessment, and the evaluation matrix links each question to its evidence.

The final reflection asks you to apply the lesson's main tools to a new program, from choosing primary intended users to prioritizing evaluation questions.

Final Knowledge Assessment

This glossary defines the terms, frameworks and people introduced in Lesson 4, and you can search it at any point in the lesson.