by Darryl Upfold and Nigel Turner 

Funders increasingly require programs to engage in evaluation so they can demonstrate accountability and improve outcomes. The purpose of this chapter is to help service providers who treat problem gamblers to design and implement an evaluation of their program, or to assist service providers in communicating effectively with external consultants (or colleagues) who provide program evaluation services. The chapter will present a framework for developing and conducting a program evaluation, using the concept of the program logic model and the evaluation framework. Then the different kinds of program evaluations will be described, followed by data collection methods, and a brief review of statistical analysis. Program evaluation is a very large topic that can only be dealt with in a limited manner in this manual. For more information on program evaluation, consult Graham et al. (1994) or one of the other references listed at the end of this section. It should be noted that throughout the text references will be made to the weaknesses of specific methods. No method of research is without flaw. The purpose of noting the weakness is to make readers aware of the limitations so that they can choose the best approach for their needs.

A Framework for Program Evaluation: The Logic Model and Evaluation Framework

Program Logic Model

Before any systematic approach to program evaluation can be conducted, it is advisable to develop a program logic model and an evaluation framework. Other program evaluation frameworks are available, but the logic model and evaluation framework have been adopted by most substance abuse agencies in Ontario.

A logic model is a diagrammatic representation of what an agency provides and what it hopes to help clients achieve, and comprises:

In a workshop format, facilitated by an external consultant or internal staff, it typically takes about one day to develop a program logic model. Whether a program develops a logic model on its own, or with assistance from an external consultant, all levels of staff should be represented (e.g., management, supervisory, front line, support staff; Board members and consumers can also be invited to participate) in the development of the model. The program logic model has typically been used by established programs to plan for program evaluation. However, it can also be used as a planning tool by agencies that are planning for new services.

Adapted from: Porteous, Sheldrick & Stewart, 1997

The comprehensive evaluation framework

An evaluation framework is developed from, and is an extension of, the agency’s logic model. To develop it, specific evaluation questions are derived from each implementation objective and outcome objective. Next, outcome indicators (information that will be collected for analysis to answer the evaluation question) for each evaluation question are identified. Then, consideration is given to the data source for the outcome indicators: are the outcome indicators already being routinely collected as part of the agency’s assessment protocol, or do new questions, forms or questionnaires need to be introduced into the agency? It can take several days in a workshop format to develop an evaluation framework, and again, staff from all levels of the program should be involved.

Example:

Focusing the evaluation

The process described above will result in the development of a comprehensive evaluation framework, with evaluation questions developed for each implementation and outcome objective. However, it is highly unlikely that any agency would have the necessary resources to conduct a comprehensive program evaluation (i.e., to evaluate each implementation and outcome objective). Therefore, program staff also need to focus their evaluation on the objectives that are most important for them to evaluate, whether it is implementation objectives for a newly established program, or outcome objectives for a well-established program. It is easy to make the mistake of planning to do an evaluation that later proves to exceed the agency’s resources and capability to complete. It is important to remember that every objective included in the evaluation plan will require data collection, data entry, data management and analysis. For agencies that have little experience with program evaluation it is recommended that initial attempts at evaluation be started realistically and modestly, perhaps focusing on one implementation or outcome objective. This will provide agency staff with the experience and confidence to continue to evaluate other aspects (objectives) of their program.

Program evaluation is intended to be an ongoing activity. After completing one evaluation, the agency would then re-visit the evaluation framework, select another objective to be the focus and plan for another evaluation.

The evaluation workplan

Following the development of the comprehensive evaluation framework, and the focused framework, an evaluation workplan should be developed. The workplan will lay out the responsibilities of everyone who is involved in the evaluation and will cover questions such as:

• Which staff will be responsible for modifications to forms and questionnaires?
• What data collection tools and methods will be used?
• Who is responsible for data collection, data entry and data analysis?
• What is the timeline for the evaluation, when will data collection begin, and how many clients need to be included in the evaluation to make it useful?

Using existing data sources

Although program evaluation does require human and fiscal resources to be successfully completed, agencies should also be aware that they are already collecting a certain amount of data (e.g., on intake forms, assessment forms, clinical notes, termination summaries, workload measurement systems, etc.), and that data can be used in program evaluation. Agencies in Ontario also routinely collect information for their funding source using computerized software. This is another data source that can be used in evaluation, and the software can be customized by agencies to accommodate collecting additional variables for evaluation.

In many cases existing forms that are used by agencies can be codified to make information collected by assessment and counselling staff quantifiable (i.e., by turning words into numbers), and therefore more useful for program evaluation. As much as possible, program evaluation should be integrated into the routine information-gathering practices of an agency, so that forms and questionnaires fulfill clinical and evaluation purposes. This optimizes the usefulness of the information, and minimizes the workload to collect it.

The preceding section describes a framework for conducting program evaluation, beginning with developing a logic model, the comprehensive and focused evaluation frameworks, and an evaluation workplan. The process described above can be applied to most of the following types of evaluations. The next section reviews the different kinds of program evaluations that can be conducted.

A Note on Human and Fiscal Resources

Many program evaluations grind to a halt because the resources required to successfully complete the evaluation were underestimated, or not considered at all. It is important to acknowledge that program evaluation is an activity that requires fiscal resources and human resources (both time and skills) to carry out the workplan. Fiscal resources might be required to:

• print new forms
• mail out surveys
• rent a room for a focus group
• purchase consultation (e.g., for questionnaire design, developing a computerized database, data management and statistical analysis).

Some agencies can use program evaluation expertise within their own staff, perhaps even through rewriting job descriptions to allow that person more time to be involved in program evaluation. In some communities, partnerships can be developed with colleges and universities, where graduate students might assist in some aspects of evaluation that require a particular level of expertise that may not be available in the agency (e.g., statistical analysis).

Back to Program Evaluation for Problem Gambling Services