Program Evaluation: Types, Designs and Methods Explained

Key Takeaways

• Program evaluation is a systematic judgment of whether a program works, how it works and for whom.
• Five types answer five different questions: formative, process, outcome, impact and economic.
• Only impact evaluation can say the program caused a result, because only impact evaluation estimates what would have happened without it.
• Design follows the question and the budget.
• Routine monitoring data sets the ceiling on what any evaluation can conclude.
• CommCare captures the person-level, longitudinal records that outcome and impact evaluations depend on.

What is program evaluation?

Program evaluation is the systematic collection and analysis of evidence to judge the effectiveness of a program for the people it serves. The CDC describes it as a way to determine how effective and efficient programs are in reaching program outcomes.

Monitoring and evaluation answer different questions. Monitoring tracks program activities and outputs as they happen, such as home visits completed or clients registered each month. Good monitoring and evaluation tools let the same records serve both jobs, while evaluation steps back at set points and asks what those counts add up to.

Different program stakeholders commission evaluations for different reasons:

• Funders want to know whether their money produced results worth renewing.
• Ministries of health need evidence before scaling a pilot nationally.
• Program managers use findings to fix delivery problems mid-stream.
• Organizations build institutional learning so the next program starts smarter.

Several reference models structure the evaluation process, and the CDC Program Evaluation Framework, updated in 2024, is one widely used option. It lays out six steps:

1. Assess the context.
2. Describe the program.
3. Focus the evaluation questions and design.
4. Gather credible evidence.
5. Generate and support conclusions.
6. Act on findings.

The framework pairs those steps with five evaluation standards: relevance and utility, rigor, independence and objectivity, transparency, and ethics. Most other models follow a similar logic.

The five types of program evaluation and the question each answers

The five types of program evaluation are formative, process, outcome, impact and economic. They work best as a sequence that follows a program as it matures, with each type of evaluation picking up where the last one leaves off.

Teams mix up outcome and impact evaluation more than any other pair. According to the CDC, outcome evaluation measures whether intended outcomes were achieved but cannot determine what caused them, while impact evaluation compares results against an estimate of what would have happened if the program never ran.

A maternal health program shows how the sequence plays out. Formative work tests if frontline workers can use a new counseling script, and process evaluation checks that they deliver it at each home visit. Outcome evaluation tracks antenatal care attendance, while impact evaluation tests how much of any rise the program caused. Economic evaluation then prices each additional appointment.

Formative vs summative evaluation

Formative evaluation happens during delivery to improve a program, and summative evaluation happens at the end to judge whether it worked. What separates them is the purpose of the evaluation.

Process evaluation usually sits on the formative side, since its findings feed straight back into delivery. Funders lean on outcome and impact evaluation for summative calls, using the evaluation results to decide what continues, and economic evaluation can serve either purpose.

Most programs end up doing both. A mid-course review might show half of households missing their second appointment, prompting a scheduling fix. Two years later, a summative outcome evaluation feeds the renewal decision.

Developmental evaluation for programs still taking shape

Developmental evaluation supports program development before a stable model exists. BetterEvaluation, drawing on Michael Quinn Patton's work, describes it as an approach for developing social change initiatives in complex or uncertain environments, with close to real-time feedback to program staff.

Formative evaluation refines a model that already exists. Developmental evaluation helps the model emerge in the first place. The evaluator works as part of the team and feeds back evidence as decisions get made, where traditional evaluation keeps the evaluator outside to protect independence.

Patton names innovation, radical program redesign, replication, complex issues and crises as the settings where it fits best. Once the model settles, teams usually shift to formative and then summative work.

Program evaluation designs: Experimental, quasi-experimental and non-experimental

Program evaluation designs fall into experimental, quasi-experimental and non-experimental families, and each one builds a different comparison. That comparison sets the limits on what an evaluation can claim.

1. Experimental: random assignment to program and control groups.
2. Quasi-experimental: a comparison group without randomization, built through methods such as difference-in-differences, matching or regression discontinuity.
3. Non-experimental: pre-post comparisons and case studies with no comparison group.

Design decides whether a team can claim attribution or only contribution. An attribution claim says the program caused the change, and it needs an experimental or strong quasi-experimental design. Non-experimental designs can still support a contribution claim, meaning the program plausibly played a part alongside other factors.

Randomization is off the table more often than textbooks suggest. A universal service cannot withhold access from a control group, and a government rollout follows a political timeline rather than a study protocol. Emergency response is another common exception. In these cases, quasi-experimental methods or staggered rollouts usually offer the best available comparison.

When randomization does work, the results carry weight. Mathematica Policy Research ran a randomized evaluation of CARE's CommCare-supported maternal and child health program in Saharsa district, Bihar, across 70 subcenters split evenly between intervention and control. The Bihar RCT results, drawn from a survey of 1,500 women, showed those in intervention areas were 73% more likely to have three or more antenatal care visits. That study is one of 8 randomized controlled trials in CommCare's evidence base, which spans 130+ studies and independent evaluations over 16 years.

For teams planning an experimental or quasi-experimental design, the World Bank's Impact Evaluation in Practice, second edition, is a standard reference. It walks through randomized assignment, instrumental variables, regression discontinuity, difference-in-differences and matching.

Program evaluation methods: Matching evaluation questions to evidence

Program evaluation methods are the tools used to collect and analyze evidence, and they should follow the evaluation questions. Start with a short list of key questions, three to five at most, and trace every method choice back to one of them.

Methods fall into three groups:

• Quantitative methods: household surveys, service delivery records from frontline workers, administrative data and skills assessments. These show how much changed and for how many people.
• Qualitative methods: interviews, focus groups, direct observation and document review. These explain why something happened and how participants experienced it.
• Mixed methods: the combination most effective evaluations rely on, with numbers for what changed and qualitative evidence for why.

If clinic attendance climbs across a program but stalls in two districts, the survey shows where the gap is. Interviews with frontline workers then explain it, perhaps a washed-out road or a referral clinic that moved. For a deeper look at each approach, see this guide to data collection methods.

Settle sampling and disaggregation before collection starts. Splitting results by district or age group after the fact rarely works if collection did not capture those fields.

What program data needs before it can be evaluated

Program data needs four conditions in place before it can be evaluated. Evaluators call this evaluation readiness, and plenty of evaluations stall here, long before analysis, because routine records were never built to answer the questions being asked. Each condition has to be designed in from the start:

1. Baselines captured before services start. Change can only be measured against a starting point, so baseline and endline surveys need a place in the timeline from day one.
2. Unique IDs that follow one person or household across visits. Outcome evaluation compares the same people over time, and loose forms with no link between them cannot do that.
3. Disaggregation by sex, age, location and service type. Breaking figures out this way shows which groups a program reaches and which it misses.
4. Validation at the point of capture. Fixing errors during analysis costs more and recovers less than catching them while the frontline worker is still with the client, which is the job skip logic and range checks do.

CommCare builds these conditions in from the first form. Longitudinal case records keep one persistent record per client or household, with full history visible at every visit. Offline mobile collection works where services happen, with or without a signal, and syncs once connectivity returns.

Holding these conditions gets harder at scale. The IRC works across more than 40 countries and 20 US cities, and consolidated a patchwork of disconnected data tools onto CommCare so frontline numbers compare across programs.

Build evaluation into the program with CommCare

Building evaluation into the program means writing the evaluation plan before the first form goes live. Indicators added after launch leave gaps no analysis can fill, so set evaluation questions and indicators before app design and make sure every form collects what the evaluation needs.

Case management carries the follow-up. When every appointment updates the same client record, outcome evaluation gets its before-and-after picture with no separate tracing exercise. Research and evaluation teams running structured studies can use CommCare for research to capture consent and assign study arms, with longitudinal follow-up in the same platform frontline workers use.

If your program is heading toward an outcome or impact evaluation, talk to the CommCare team about building the data it will need from day one.

FAQ

What is a counterfactual in program evaluation?

A counterfactual is the estimate of what outcomes would have been for the same population without the program. Since no one can observe the same people both with and without a program, evaluators use control groups or comparison groups as stand-ins. Selection bias weakens those stand-ins when the groups differ in ways that also affect outcomes, which is why randomization produces the most credible counterfactual.

Do you need an IRB for program evaluation?

Sometimes. Under the US Common Rule, an activity needs IRB review when it is research, meaning a systematic investigation designed to contribute to generalizable knowledge. Evaluations done only for internal program improvement often fall outside that definition, while plans to publish often bring an evaluation in. Outside the US, national ethics committees apply their own rules. In every case, the determination belongs to the IRB or ethics committee.

Should a program evaluation be internal or external?

Choose external when funders need independent findings and internal when the main goal is program improvement. Internal evaluators know the program and cost less, though their independence is weaker. An external team gives funders the independence they often require, at higher cost and with less context. Many programs use a hybrid, with an external lead setting design and analysis while program staff manage collection.

What does fidelity mean in a process evaluation?

Fidelity is the degree to which program implementation matches the program design, across content, dose, quality and reach. A program with low fidelity cannot tell whether its model failed or simply was not delivered. Fidelity data usually comes from routine service delivery records, which is why capturing each service interaction matters well before any evaluation starts.