Program Evaluation for the Homelessness Sector

Types of Program Evaluation

In this lesson, you will learn about three types of program evaluation. Different types of evaluation are used depending on what you are trying to accomplish.

Man and woman talk about program evaluation

 

 

 

Definitions 

Evaluation is a process by which a program is examined to determine whether it is meeting its goals and objectives through the activities taking place and in the manner expected. Essentially you are trying to decide if you have accomplished the things you intended in the way you intended.

There are different types of evaluations and these sometimes have different names. In the video above, they have outlined three types: Formative, Process, and Summative evaluations.

Formative evaluation asks the question “How can we make this program better?Process evaluation asks “Are we doing this program the way we said we would do it?Summative (or outcomes) evaluation asks “Did our program meet the objectives that we set out?

Formative evaluation

Formative evaluation is normally done at the beginning of your program. You are collecting information to see if your program is working or could be improved. 

What are some of the things you could do?

Very early on in your community kitchen you may ask participants if they understand the information that is being presented. You may want to know if participants are able to follow the recipes you are using in class. You might simply want to track the number of people using your program or whether they are leaving part way through the class.This is information you can collect early on to see if you need to make changes to the program. 

Typically, formative evaluation is done internally. Changes can provide feedback to make improvements to the program. 

Process evaluation

A process evaluation is normally done throughout the program to monitor that the program is being done the way you set out to do it or to get an accurate description.

These are the sorts of things you will be tracking in your process evaluation.

  • How many people are accessing your program? 
  • Are they the people that you thought would be accessing your program? 
  • Are the clients using the program, the clients you intended it for?

Maybe it’s something more specific that you want to know. Maybe you intended that the program be run by a peer. It turns out that the person you thought would lead the group wasn’t interested or available and the program ends up being led by a staff member. This is an important change in the way the program was intended that you will want to track in order to deliver if your program is having the impact that you intended.

Process evaluations can be done by people running the program (internal evaluation), but it could also be somebody that is external.  

Summative (Outcomes) evaluation

As you have seen, summative evaluations are also called outcome evaluations. Another term for them is impact evaluations.

When you do an outcome evaluation, you are trying to provide information about the worth of your program, the effectiveness of your program or the efficacy of your program. 

Worth means are the changes that the program created worthwhile? Worth is measured in relation to the resources required to run the program. Effectiveness means “Did the program bring about the changes that you had hoped?” Efficacy is an ideal world situation. Under the best circumstances can this program create the change it was intended to create. Most program evaluations will be concerned with effectiveness.

Summative evaluations may be done at the end of a program. But for some programs, they will never be “complete.” In this case, the evaluation can happen at a predetermined time while the program is operating. You can use outcome evaluation to find out whether participants learned what you had hoped they would learn. You can evaluate whether their nutrition has changed, whether they have stronger relationships - whether they see each other more frequently than when the program started.

Notice that some of these questions require collecting information at the beginning of the program so you can document change over time. 

Who should conduct the evaluation? 

There are benefits to using internal and external evaluators. Some people feel that outcome evaluations should be done by external evaluators. With internal evaluators, there can be more engagement of staff and clients. Who you choose to do the evaluation will depend on your individual situation.

FORMATIVE PROCESS SUMMATIVE (OUTCOME)
How can we make this program better?  Are we doing this program the way we said we would do it? Did our program meet the objectives that we set out to meet?
Provide feedback to make improvements to the program Monitor the program to ensure it is being delivered the way it was intended Provide information about the worth of program, its effectiveness or efficacy
Near the beginning Throughout the program At a set time usually when the program has been running long enough to show results.
Internal evaluator Internal or external evaluators External or internal evaluators
Talk to participants. Track how many people attend the program. Are people engaging in the program? Talk to participants and program staff. Track how many people are using the program. Are they who you expected? Collect pre- / post-assessments. Interview participants about their experiences in the program. Did they achieve the changes you hoped they would?

Table 1. Comparison of three types of evaluation.