This glossary collects the key concepts, people, and ideas you will meet in this lesson. Use it as a reference while you work through the material, or as a review before assessments. Type in the search box to filter entries.

Evaluation: what it is and why it matters

Every lesson in this course has followed the same path: assess a situation, identify the factors that matter, choose a theory that explains them, and design a program. This lesson asks the question that path has been leading toward. Did it work? This is not a course on evaluation, and the lesson will not go into technical detail. Its aim is familiarity, because evaluation is so much a part of the program environment that anyone who plans programs will eventually be doing it. This section covers what evaluation is, why the funding environment now demands it, and how it connects to the theories you have studied.

Evaluation without the jargon

Evaluation sounds formal and technical. Stripped down, it is a short list of questions. Click each card for the question and an example.

The point can be summarized in one sentence: evaluation is relevant to the entire scope of this course, because it is the way you carry through your use of social and behavioural theory to the point where you can determine whether the whole process, of assessing a situation, identifying factors to address, and using the assessment with relevant theory to plan and implement an intervention, made a difference.

The current program environment

There is also a pragmatic reason to evaluate. In the current and future environment of program funding there is a strong emphasis on evidence and accountability. To keep funding, a program needs evidence that it is doing something about the problem it was meant to address. This imperative is relatively recent. Serious evaluation of health promotion and behavioural interventions did not become the norm until a consensus formed among researchers and practitioners in the 1980s that better evaluation was needed to improve program planning and performance.

In the United States the turning point was legislative. The Government Performance and Results Act (GPRA) of 1993, followed by the Program Assessment Rating Tool (PART) of the White House Office of Management and Budget, required agencies to develop performance monitoring and accountability procedures to ensure that public funds were well spent. The influence of this accountability movement has since been integrated into standard practice at every level, including private and global health promotion efforts. One consequence is the spread of strategic planning documents that set goals as a guide to action and as a method for evaluating progress. If the plan says "we will reduce the number of youth who smoke by 25%," then documented change in youth smoking, compared with that goal, becomes the measure of success. The Healthy People documents of the United States Department of Health and Human Services (Healthy People 2010, Healthy People 2020, and their successors) are large, comprehensive planning documents of this kind.

Evidence-based practice and model programs

An even more profound change than accountability, arguably, is the move within public health to develop a body of evidence-based practice, in the same way that medicine developed evidence-based standards of care. For health promotion and prevention this means three things. First, evaluating the program becomes very important, because evaluation data are the evidence. Second, many agencies have developed directories of interventions that have evidence of effectiveness, and these directories often rate programs by the quality of their evidence. A program implemented in several places with different populations and found effective in all of them is rated more highly than a program tried once, even if it worked that one time. Third, programs with good evidence are called model programs or something similar, and public and even private funders increasingly require grant recipients to use a model program, or to justify why no appropriate program of that type exists for the community, population, or situation to be addressed.

The consequence for research is that a great deal of effort now goes into identifying model programs, which means building or locating the evidence base, and along with that, compiling evidence for the usefulness or inapplicability of the theoretical approaches used to structure those programs. Every well-evaluated Health Belief Model program, whether it succeeds or fails, is also a data point about the Health Belief Model.

Why evaluate? Four reasons

Four reasons to evaluate cover most real situations. Accountability responds to the public need to show that the cost and resources invested in a program had some effect, and, since GPRA and PART, to legal requirements for monitoring systems. Learning and improvement means that as a program runs, evaluation data show what is working and what is not, feedback that lets you make changes as you go. Theory means testing the validity of a theoretical linkage, or testing a model program or approach that has been shown to work somewhere else. Efficiency and other issues covers what it costs for the program to achieve its goals, and other kinds of assessment. Use the sorter to practise telling them apart.

How evaluation relates to theory

How does evaluation relate to theory? The answer runs through the whole course. A theory-based program is a set of predictions. The Health Belief Model predicts that if people come to see themselves as susceptible to a serious condition, and see a course of action whose benefits outweigh its barriers, they will act. Social Cognitive Theory predicts that observing a credible model and building self-efficacy will change behaviour. Each prediction names something that ought to change, and each of those things is something an evaluation can measure. Theory, in other words, tells the evaluator what to look for. The diagram shows the loop.

The loop also explains why evaluation can be called the way to test theory. When a program falls short, evaluation can separate two very different explanations: the theory did not fit this population, or the program was not delivered as designed. Section 2 returns to this distinction under the heading of process evaluation and fidelity. For now, hold on to the idea that a program without a theory has nothing specific to evaluate except whether it happened, and a theory without an evaluation is a prediction nobody checked.

The director's problem is common, and it points to the rest of the lesson. Attendance sheets answer the first of the evaluation questions (did we do what we proposed?) but not the second or third. To answer those, she would have needed to decide at the start what the theory said should change, measure it before the program, and measure it again afterward. The next section names those different kinds of evaluation and the questions each one answers.

Process, outcome, and impact evaluation

Once you look closely at evaluation it becomes more complex, because there are many things one could evaluate in any program and several goals an evaluation can serve. The way to make this manageable is to start with three basic categories: process, outcome, and impact. A project may use one, two, or all three at once, because each has a different purpose. Other types exist that this lesson does not cover in detail: cost-effectiveness evaluation, formative evaluation used during program development, and quality assurance evaluation. This section works through the three basic types, the vocabulary that comes with them, and a Canadian program that needs all three.

Three questions

The simplest way to hold the three types apart is by the question each one asks. The tabs give the question, the way it is answered, and where the answer comes from.

Process evaluation: did we do what we planned?

Process evaluation is more than bookkeeping. It matters for testing theory. Suppose an intervention is based on the stages of change model from the Transtheoretical Model. Its effectiveness may depend in part on periodically assessing each participant to identify their stage in changing, say, their diet, because the program's components are tied to a stage for each individual. If the intervention turns out less effective than expected, it would be very helpful to know, through process evaluation, whether the stage assessments were done properly. If they were not, then at least you know that the problem may lie in the details of implementation rather than in the theory or the intervention design. Two terms follow from this. Click each card.

Outcome evaluation: short-term effects

With outcome evaluation you start looking at the effects of a program. How you answer the question depends on your program goals. What did you expect to happen in the short term, and was that expectation guided by theory and by your assessment? The expected outcome has everything to do with theory, assessment, and how these were incorporated into the program design, for example through the PRECEDE-PROCEED process. The outcome data might be pre- and posttest scores on knowledge, records of health service use, evidence of a policy change, or other educational, ecological, or policy data. The diagram shows the timeline that outcome and impact evaluation share, and where the two part company.

Impact evaluation: the health problem itself

Impact evaluation reaches the real health impact of a program, and with it the theoretical links you may have drawn between short-term outcome and long-term impact. Consider a smoking intervention with several activities designed to raise awareness of risks and of prevention options. The hypothesis, drawn from the Health Belief Model, is that if a group of people come to see their own susceptibility to cancer from smoking, and see quitting options that do not present an insurmountable barrier, they are likely to quit. The program is therefore linking awareness change to behaviour change, and behaviour change to a reduction in cancer. Awareness is the short-term outcome. Cancer is the long-term impact. Measuring the impact requires a program that stays in place for an extended period, or the ability to follow participants as a cohort for years, and it means collecting follow-up data on cancer morbidity and mortality rather than on awareness.

Practise: process, outcome, or impact?

Each type reduces to one question, and for outcome and impact the answer is what your logic model and theory say should have happened. Use the classifier below to test yourself on evaluation questions drawn from Canadian programs.

The second case is the stages-of-change example from earlier in this section in different clothes. The process evaluation showed that the replication lacked fidelity and delivered a low dose, so the disappointing outcome says little about the curriculum and nothing about the theory. It says a great deal about implementation. The next section puts all three types of evaluation into a single structure, the logic model, and introduces the methods and pitfalls of making a case that a program caused a change.

Logic models, methods, and confounds

The last section separated three types of evaluation. This one puts them into a single structure, the logic model, which we reach by way of PRECEDE-PROCEED. It then turns to method: how do you make the case that a program caused a change? The answer is a ladder of designs, a list of confounds, and a reminder that the right evaluation is the one that fits the program.

From PRECEDE to a logical chain

Recall the PRECEDE assessments: social (the community context and quality-of-life issues), epidemiological (the health problems and affected group, from morbidity and mortality data), behavioural and environmental (the risk factors behind those problems), educational and ecological (norms, attitudes, awareness, and policies understood as predisposing, enabling, and reinforcing factors), and administrative and policy (resources, community politics, and structures that help or hinder implementation). Can you see the logical chain? Health problems result from risks, which result from predisposing, enabling, and reinforcing factors, within a community and policy context. Real situations rarely line up so neatly, but for designing an intervention and its evaluation the chain is useful, because the intervention goes back up it: you target several factors, which should affect risks, which should affect the health problem. The diagram shows the PROCEED side of that logic.

The parts of a logic model

A logic model is a diagram or structure that links what you plan to do with its expected outcomes and impacts. It has five linked components. The health problem comes from the epidemiological assessment (HIV, cancer, malaria, diabetes, and who it affects) and may name the contributing factors you intend to address. Outputs are the activities you plan, such as materials, events, screenings, or trainings, and the factors each is meant to change; the choice is guided by the factors identified and the theory you believe appropriate. Inputs are the resources, staff, components, and funds invested, some identified in the administrative and policy assessment. Projected outcomes and impacts are the short-term effects hypothesized for the outputs, typically enabling and predisposing factors such as knowledge, and the longer-term effects on risk behaviour and health status. Indicators and measures are the criteria and tools for measuring them: health data, survey or qualitative data, measures of skill.

Take a worked example: Harfield County, a fictional rural county where type 2 diabetes is rising among adults and young adults. Large supermarkets are scarce, the newspaper rarely covers health, and the county's strength is its churches, service organizations, and a veterans group, full of people who are leaders or at least "movers." From that assessment the planners choose a social-cognitive approach with social network components, and the logic model follows.

Read the table against the theory. Modelling by leaders is Social Cognitive Theory's observational learning; delivery through churches and veterans centres is a social network strategy; knowledge and skills are the outcomes the theory predicts; diabetes incidence is the impact. Each column is also a type of evaluation: process for outputs, outcome for the short-term column, impact for the long-term column. Now build one.

Evaluation methods: making a case

This lesson does not present methods in detail, but their shared rationale is worth stating. Evaluation methods are tools for making a case for the effectiveness, or lack of effectiveness, of an intervention; it is a little like being a lawyer. The more rigorous the method, the more you can argue that the intervention, and nothing else, was responsible for the change. As designs get more rigorous they add control groups (people who do not get the intervention) and, at the top, random assignment. The accordion runs from simple to rigorous.

Confounds: what weakens the case

When you argue that a program caused a change, you must also address what could weaken the case. Some weaknesses are methodological. Others are confounds: unplanned occurrences or other issues that complicate the claim that only the intervention caused the change. Seven are listed here; designs often build in ways to offset them.

What kinds of outcome or impact?

Different evaluations give different results, and results short of a classic experiment are not without merit. Programs run in the real world, and success has many levels. When getting a program in place and operating is itself an achievement, a process evaluation may be all you need. When the time period is short, or barriers are many and only a limited effect can be expected, an outcome evaluation may be the best you can do. When longer-term impacts are realistic, because the program runs for ten years or includes a long-term follow-up, look for them. A clinical intervention calls for clinical outcomes.

The kind of data matters too. Quantitative data can be expressed numerically and analyzed statistically: fixed-choice survey answers, official demographic or epidemiological records. Collected the same way from many people, they support comparison and generalization. Qualitative data are narrative, descriptive, and subjective: extended interviews, focus groups, and observation, where people have time to explain their actions, beliefs, and interpretations. Gathered from fewer people, they reveal meaning, context, and the dynamics of interactions such as those between patient and doctor. The conclusion: there is no cookie-cutter standard. Evaluation is tailored to the situation, the program, and the evidence needed.

A realistic answer is a periodic inventory with baseline and follow-up surveys, benchmarked where possible against provincial data, plus interviews with patients and link workers. It could claim that participants improved, not that the program alone caused it. Regression to the mean threatens the claim, since providers refer the most isolated patients; so does attrition, since those who did not benefit stop attending. Interviews would reveal what "connection" meant to participants and why some referrals never became attendance. The final section asks where, in a career, all of this gets used.

Careers: putting theory to work

This section is short and practical. Its argument is that a good working knowledge of social and behavioural theory and its real-world application has become an increasingly important qualification for public health work, because so much weight now falls on programs that make sense, are effective, and are evidence-based. Theory, properly connected to real circumstances and people, is what lends coherence to an intervention. Theory is also what allows a meaningful evaluation, because the theoretical base tells you what ought to change and therefore what to evaluate. Theory plus evaluation produces the evidence that improves the knowledge base about what works. That, to put it bluntly, is where the money is.

Six career paths

Public health careers where theory matters can be grouped into six settings, each with a characteristic role. The table adds the Canadian organizations where you would find them.

Two ends of the funding process

The first two settings are two ends of one process. A government agency, federal, provincial or state, local, or international, disseminates and manages funds allocated to a health problem. Its grant announcements describe the purpose, the activities requested, the funds available, and the criteria for success, and are increasingly based on a program framework or theoretical approach the agency has adopted. Program heads and program officers make those decisions but do not carry out the program; the organizations selected do, overseen by a program officer. A panel of experts and agency staff rates applications on the best plan at the best price with adequate organizational capability. Agencies also do policy work, issuing guidelines and regulations (air and water pollution, secondhand smoke) and building consensus on recommended policies.

Community-based organizations are at the other end, in direct contact with people and with intimate knowledge of the situations surrounding health behaviour. Here theory is used in program design and evaluation, in response to a solicitation or when the organization has developed a program it believes meets a need and must seek funding. The exception is the private foundation, a nonprofit that acts like a public agency by disseminating funds; the Ford Foundation and the Kaiser Family Foundation are United States examples, and Canada's community foundations, such as the Vancouver Foundation, play a similar role. A task common to all these organizations is writing grant applications, which typically contain six parts.

Consulting, industry, and health care

More and more of the work of government agencies is carried out by private consulting organizations with specific expertise: evaluation, program design, organizational development, cultural competency, communications campaigns, materials, information technology. You work in a project team, and the firm competes through proposals much like grant applications. Consultants usually work under contracts rather than grants, which means more hands-on interaction with the funding agency's project officer. Knowledge of theory and its relationship to design, implementation, and evaluation is very important here.

Private businesses that provide health insurance may run workplace health promotion to manage costs: wellness screening and education, exercise programs, health facility memberships, sometimes in-house staff. Industry associations advocate workplace health policy, and labour unions often handle wellness programming and lobby as well. Healthcare providers, public or private, large or small, have three key roles beyond care: disseminating health information, providing interpersonal communication and role modelling, and giving real-world feedback on which public health approaches work. They are also a setting for research on health beliefs and barriers to treatment.

Academic settings and knowledge transfer

Universities generate theory through research, evaluate theory-driven interventions, and identify useful evaluation methods, all of which means competing for research grants through peer-reviewed applications, often with community partners. Results are shared in journals and at conferences, and also in research briefs, policy recommendations, and with the partnering communities, which is how research gets transferred to practice. Canadian funders call it knowledge translation. Academics also teach students and practitioners and advise agencies. The possibilities are many and can only increase.

Four people, four settings

Four hypothetical profiles, none of them real cases, illustrate the settings. Three are set in the United States, where this typology of settings originated; the fourth is set in a Canadian setting the typology does not cover. Click each card.

One closing observation applies to all four profiles and both cases. Every setting rewards the same combination: theory connected to real circumstances, a program built from it, and an evaluation that checks whether the predicted changes occurred. That combination is what this course has been assembling since its first lesson, and it is the qualification public health increasingly asks for.

Bringing It All Together

This lesson closed the loop the course has been drawing since its first pages. Evaluation, in plain terms, asks whether you did what you proposed, whether the program had an effect on what it set out to achieve, and whether the program model and theory were useful, with cost and community goals sometimes added. The funding environment now demands it: the accountability movement that followed the Government Performance and Results Act and the Program Assessment Rating Tool, and its Canadian counterparts in Treasury Board results policies and provincial standards, made stated goals the measure of success, and the move to evidence-based practice made evaluation data the evidence that registries compile and funders require. The four reasons to evaluate, accountability, learning and improvement, theory, and efficiency, each shape what gets collected. And theory is what makes an evaluation specific, because a theory-based program predicts particular changes that an evaluation can check.

The three basic types of evaluation ask three questions. Process evaluation asks whether the components were implemented as planned, and supplies fidelity and dosage data that separate a failed theory from a failed delivery. Outcome evaluation asks what short-term effect the program had, measured from baseline to follow-up on the factors theory targeted. Impact evaluation asks whether the health problem itself changed, which needs an extended program or a followed cohort. The logic model gives each its column: inputs, the problem, outputs, short-term outcomes, long-term impacts, and the indicators for each, all descending from the logical chain of PRECEDE assessments. Making a case that the program caused the change is like being a lawyer: designs climb from record keeping through qualitative approaches, periodic inventories, and benchmarking to quasi-experimental and classic experimental designs, and each must anticipate confounds such as history, maturation, testing, regression to the mean, selection bias, attrition, and diffusion of treatment. There is no cookie-cutter standard; success has many levels, quantitative and qualitative data answer different questions, and empowerment evaluation lets a community define success on its own terms.

The final section asked where all of this is practised. The answer was six settings, government and public agencies, nonprofit and community-based organizations, private consulting, private industry, healthcare providers, and academic settings, each using theory in program design, implementation, or evaluation, and each mapped in this lesson onto Canadian institutions from the Public Health Agency of Canada to regional health authorities, community health centres, unions, universities, and Indigenous health organizations such as the First Nations Health Authority. The closing argument of that section is the course's argument: theory connected to real circumstances gives a program coherence and tells the evaluator what to measure, and theory plus evaluation produces the evidence public health now runs on.

Final Knowledge Assessment

This assessment covers all material from this lesson. You must score 100% to complete the lesson. Review the feedback for any incorrect answers and try again.