Documents

Information
Exchange
Services

Special
Topics

Resources

State
Information

Online
Resources

This page contains links to external Web sites.
The Treatment Improvement Exchange has no control over their content or availability.

Chapter 10 of TAP 11: Treatment for Alcohol and Other Drug Abuse: Opportunities for Coordination

Chapter 10–Evaluation

Evaluation is a word that has considerable variations in meaning depending upon the context in which it is applied. In its most general use, it includes gathering and analyzing information concerning an individual, program, group of programs, or other entities. There is usually a standard, whether explicit or implicit, against which the evaluation data are compared and judgments are made (Weiss, 1972).

Evaluation is important for a variety of reasons. Some of these include (Schinke, Botvin & Orlandi, 1991):

determining whether or not program objectives or individual treatment goals have been met;
planning and making decisions about individuals or program elements based on appraisals of achievements compared to goals and objectives;
monitoring standards of performance;
generalizing the effectiveness of a program or program component to other populations;
fostering program and individual accountability; and
promoting positive awareness of treatment effectiveness.

Substance abuse professionals frequently evaluate (formally or informally) the progress their patients are making in the treatment process. Based on these assessments, they may continue the treatment as planned, modify the treatment plan and services, or terminate the treatment because goals have been achieved or there is no progress being made.

Evaluation information about one or more programs is often helpful to program administrators, referral sources, funding agencies, policymakers, and advocates. Evaluation may focus on the design of programs, the way in which they are conducted, and both short– and long–term outcomes. It also may examine the cost–efficiency of programs compared to their effectiveness (Schinke, Botvin & Orlandi, 1991).

It is clear that substance abuse treatment does work for many individuals. When treatment objectives are achieved, chemical dependency treatment is cost–effective compared with the frequently incurred alternatives of lost productivity, increased health care costs, and criminal justice services. However, there are variations in the effectiveness of different programs. Thus, to make informed decisions about policies and funding at local and State levels, decision makers must take a careful look at the evaluation of programs.

In this chapter, three levels of substance abuse evaluation, as depicted in Figure 10–A, will be summarized. Treatment outcome evaluations look at information from many programs to determine the effectiveness of various treatment modalities. The findings from several treatment outcome studies will be reviewed. Program–level evaluation is essential for accountability, making informed decisions, and modifying program elements. This is crucial for ensuring that programs are both effective and proficient in meeting program objectives. Finally, evaluation of an individual's progress during treatment provides similar advantages. It assesses individual accountability and allows the patient, direct treatment providers, and others with an appropriate concern to make necessary decisions about the continuation of the treatment. The benefits of performing evaluations at each of these levels, and the possible applications of results will be highlighted.

Figure 10 - Levels of Substance Abuse Evaluations

Treatment Outcome Evaluations

Treatment outcome evaluations are conducted to inform practitioners and decision makers about the efficacy of various treatment modalities and program components. The general findings from such evaluations indicate that substance abuse treatment does work for significant numbers of patients. However, conclusions cannot be made that all treatment approaches work equally well for all individuals; nor can it be stated that every alcohol– or drug–involved person will derive any benefit from treatment. Many of the treatment effectiveness studies to date have focused on narrow population groups–usually males. It cannot necessarily be generalized that similar programs would be equally effective for women, adolescents, or other special population groups. Many studies also have been limited to one type of substance abuse, such as heroin or alcohol. Again, whether or not a particular modality would produce similar results for persons abusing different substances or those with polysubstance abuse problems cannot be determined without additional research.

Two large studies have focused on populations of narcotic–involved offenders. The Drug Abuse Reporting Program (DARP) measured treatment outcomes on 44,000 patients admitted to 52 treatment programs from 1969 through 1973. The types of programs included in the study were outpatient detoxification, methadone maintenance, therapeutic communities, and drug–free outpatient. A comparison group consisted of persons interviewed and scheduled for treatment who did not show up at the program. Treatment outcome measures included drug use, productive activity, alcohol use, and criminality. Some general findings from this study include the following (Hubbard, 1992; Institute of Medicine, 1990; Tims, Fletcher & Hubbard, 1991):

Drug use declined dramatically between pre–treatment and post–treatment measurements and continued to diminish during the three years following treatment. Post–treatment measures, compared with pre–treatment, indicated substantially less use of opiate drugs and nonopioid drugs, including cocaine. However, there was some increase noted in the use of alcohol and marijuana.
The most favorable outcomes for male opiate addicts were associated with methadone maintenance, therapeutic communities, and outpatient drug–free treatment. Detoxification alone was found to be considerably less effective.
Criminal behavior resulting in arrests or incarceration declined following treatment.
Employment levels six months after treatment were substantially higher than pre–treatment levels.
Patients remaining in treatment at least three months showed better outcomes. The longer they remained in treatment, the better the outcome on average.

The Treatment Outcome Prospective Study (TOPS) collected data on 10,000 patients in 40 methadone, residential, and outpatient drug–free treatment programs between 1979 and 1981. The sample population for this study was predominately young adult males. However, women made up 30 percent of the sample, youth under age 21 comprised 25 percent of the study group from residential and outpatient drug–free programs, and racial/ethnic minority group members were included. The study measured drug use, alcohol consumption, mental health, criminal behavior, and economic productivity (Hubbard, 1992).

A composite portrait of those included in the study suggests that on average, they began regular drug use at age 16 but did not enter treatment for the first time until age 24. There was an average of five treatment admissions among the sample. Most had been treated in more than one type of treatment program. About 20 percent had also been treated for alcohol problems, and approximately 25 percent had received previous mental health treatment (Hubbard, 1992).

Some findings from this study include the following (Hubbard, 1992; Institute of Medicine, 1990):

Patients remaining in treatment for at least three months exhibited more positive treatment outcomes. However, the major changes in behavior were seen only among those who stayed in treatment for more than a year. Those who remained in methadone or residential treatment for one year or more showed significant decreases in heroin use following treatment.
Although decline in heroin, cocaine, and psychotherapeutic drug use was noted, especially for those remaining in treatment longer than three months, marijuana and heavy alcohol use tended to continue after treatment.
After treatment, persons in the TOPS sample indicated substantial decreases in indicators of depression.
Individuals from the criminal justice system under legal pressure to participate in treatment did as well or better than those who voluntarily took part.
Involvement in the criminal justice system also helped retain persons in treatment, and more substantial changes in behavior during treatment were noted for individuals referred from criminal justice agencies.
The criminal justice system tended to refer fewer persons to methadone programs, and it was found that individuals coming from the criminal justice system to drug–free programs received fewer services than other persons in the same programs.
Outpatient programs had the poorest retention rates. Forty–one percent of patients dropped out within the first four weeks and only 18 percent eventually completed treatment.
Contrary to the positive findings about employment rates by the DARP study, TOPS researchers found that the level of employment six months after treatment was slightly lower for all program types. This may, in part, reflect economic conditions during the respective periods in which the studies were conducted.
Reports of illegal activities decreased after treatment in all modalities. The most significant change occurred with those in residential programs.

Another major study of treatment effectiveness is currently in progress. The Drug Abuse Treatment Outcome Study (DATOS) is collecting data between 1991 and 1993. Fifty programs, both publicly and privately funded, including detoxification, methadone maintenance, therapeutic communities, drug–free outpatient, and chemical dependency units are being studied. Approximately 20,000 persons are included in the study sample. Emphasis is being placed on the process of treatment and client change measures during treatment (Tims, Fletcher & Hubbard, 1991).

One national study of alcohol treatment also was conducted in the 1970s. A sample of 593 patients were followed at 18 and 48 months after treatment. At four years after treatment, 21 percent of treatment participants had been abstinent for at least one year before the study was conducted. Both outpatient and inpatient alcohol treatment showed similar results (Hubbard, 1992).

While these studies provide significant information about treatment outcomes, they have some limitations. More information is needed about the comparative effects of different treatment approaches and the benefits of particular treatment components. Both treatment services and the types and needs of patient populations have changed since these earlier studies were conducted. Much additional research is needed on patient differences and how treatment variations respond to diverse needs. The complex process of individual change and the treatment factors that foster this require additional study, as well (Hubbard, 1992).

Despite the need for further evaluation, several points about treatment effectiveness can be made in summary. Overall, treatment is effective, and its benefits outweigh the costs of providing treatment. Generally, the more time spent in treatment, the better the treatment outcome. Individuals who are legally mandated to participate in treatment do as well or better than those who seek treatment on their own. Frequency of drug use and criminal behavior have shown decreases during treatment. Persons whose values and behaviors are more consistent with the majority of society have more favorable treatment outcomes. Persons with severe psychopathology and persons with histories of extensive criminal activity tend to have poorer treatment outcomes. Treatment effectiveness varies within modalities and among programs because of differences in staff, clinical competence, and experience (Hubbard, 1992; Singer, 1992).

Program Evaluation and Accountability

Program evaluation is vital for a variety of reasons. Accountability is one of the five critical areas of substance abuse treatment. Both programs and patients must be held accountable for how they conduct themselves and the results of their efforts. Program evaluation helps determine whether or not a particular agency is performing the intended services and how effective they are in achieving treatment goals. This information is essential for judges and other agencies who need to refer persons to treatment.

Another important reason for programs to be evaluated is to provide information to the administrators and staff about the effectiveness of the program. This information can be supportive of program elements that are working effectively, or it can provide the data needed to make informed decisions about program change. Positive evaluation results can be used to bolster community support and elicit funding for a program. Both program procedures and outcomes are monitored by decision makers and funding agencies. Evaluation information documents the effectiveness of programs.

Needs Assessments

Needs assessment is an important prelude to program evaluation and accountability. Needs assessment activities should be undertaken before programs start and periodically after they are operating to ensure that they are appropriately oriented to the specific needs manifested in the community. Once treatment programs are started, inertia tends to keep them moving in the same direction if new information is not provided. For example, drug use trends change over time, but a program that has been addressing the problem of heroin use may not adapt to the problem of cocaine dependency or polysubstance abuse unless this need is clearly documented. Needs assessments at the community or State level help determine how resources should best be allocated. Other reasons for conducting needs assessments include (Kimmel, 1993):

generating information for advocacy purposes;
responding to external mandates, such as government agencies and other funding sources;
justifying decisions that have already been made; and
verifying information received through other sources.

A primary purpose of needs assessment is to determine the size and nature of the substance abuse problem in a given area (e.g., community, State). This will include collecting data such as (Kimmel, 1993):

the total population of the area;
the number of persons who use alcohol and other drugs;
of those, the group at risk of substance abuse or addiction;
those exhibiting serious problems of substance abuse and chemical dependency;
those currently receiving treatment;
those requiring publicly funded treatment services; and
those who may not be expected to benefit from treatment.

It also may be important to estimate the impact of substance abuse on the community or State. For example, a needs assessment might involve gathering factual information about the number of alcohol– or drug–involved persons and the estimated costs of lost productivity, accidents, health care, and criminal justice services. Another aspect of needs assessment is development of an inventory of available services and funding sources for treatment.

Developing information about needs can be costly and time consuming, but so can funding of services that do not meet needs effectively. There are a variety of methods for conducting needs assessments including both quantitative and qualitative data collection.

Assessment of needs and resources is important to both the development of new programs and the continuation of existing ones. To adequately evaluate programs, information about needs and resources is important for comparison.

Formative Evaluation

Formative (sometimes called process) evaluation reviews program procedures. This type of evaluation measures the integrity of a program and is used to modify program practices. It provides documentation that the program is being operated as planned. Formative evaluation results are helpful to a variety of persons:

Program administrators and staff can use the data to make decisions about continuing or changing certain aspects of services.

Outside monitors can document that appropriate services meeting acceptable standards are being provided.

Funding sources can be shown that money is being spent appropriately.

Those referring patients to the program can consider its ability to consistently deliver appropriate services.

Summative Evaluation

Summative (sometimes called outcome) evaluation documents a program's effectiveness or ineffectiveness in reaching its intended goals. Summative evaluations will measure such areas as changes in participants' attitudes and behaviors regarding substance abuse and changes in areas such as academic or work performance and attendance (Schinke, Botvin & Orlandi, 1991).

The goal of treatment is that chemically addicted persons stop using alcohol or other drugs and continue their abstinence after completing treatment. Treatment is "deemed successful when, three to five years after treatment, a former addict is no longer using drugs" (Office of National Drug Control Policy [ONDCP], 1990, p. 22). Concomitant goals may include improved health, employment, relationships, and family functioning.

Outcome evaluation should not be viewed as an either/or alternative–either a program is totally successful or a complete failure. Most programs will have degrees of success, and it is extremely unlikely that a program will be able to accomplish all the treatment goals of every patient. Rather, program evaluation should examine programs along a continuum from high to low success rates. Often, a great deal can be learned by further exploration of the types of patients who are succeeding or failing in the program or particular service elements that appear to be more or less effective. For example, does a particular counseling approach or group technique result in more frequent successes? Is the program likely to be more successful with alcohol–rather than drug–dependent persons? How many persons are admitted to the program and what are the dropout and completion rates? Such indicators can be used to modify programs to increase effectiveness.

The results of summative evaluations are also useful to program personnel in making decisions about continuing or modifying services. Both funding sources and policymakers need information about the outcome of programs to make informed decisions.

The Evaluation Process

There are five basic components to an evaluation design. Agencies that are able to demonstrate development of these elements will be more likely to collect useful evaluation data.

Program Objectives

Objectives should be clear, specific, measurable, and practical in order to guide the evaluation effort. A time frame for achieving each objective is also important. It is vital that the agency's program mission and the objectives be in agreement, so that the program is not working at cross purposes with the overall agency's intent. Agencies should provide clearly written program objectives that address both program procedures and intended outcomes.

Management Information System

A management information system allows for the collection and retrieval of information as efficiently as possible. Computerized systems are capable of producing these results with increased ease, speed, organizational efficiency, and convenience. Computers also reduce the need for filing space and excessive paperwork. However, some agencies may not be able to use computer systems because of funding, lack of trained personnel, or other constraints. Manual systems can produce the same results for management information, but may be more labor intensive. Agencies should be able to describe how they will collect, store, retrieve, and compile data for the evaluation process.

Evaluation Method(s)

Various methods of evaluation are appropriate for different purposes. Three general evaluation designs are summarized below. These do not represent all that can be developed.

Descriptive Studies. Descriptive evaluations do not provide explanations of results, explore causal factors, or make predictions. They merely describe a particular process or finding. Both quantitative and qualitative data may be used in descriptive studies. Quantitative data are obtained by counting categories, such as the number of persons entering treatment, the number of staff in the program, the number of hours of services provided, and the number of lapses to drinking or drug use reported by patients. These data may be arranged to show certain patterns, such as rank order, intervals between certain items within a category, or the ratio between specific measures. Qualitative data can be collected through reviews of patient records, interviews with staff or patients, open ended questions on surveys, and similar means. Typical ways of collecting data for descriptive studies include survey questionnaires, records reviews, meetings, observations, and structured interviews (Schinke, Botvin & Orlandi, 1991).

Before and After Studies. Sometimes called pretest/posttest, these studies attempt to show changes that have occurred during the course of treatment. Data are collected before the program or particular intervention is begun and at other intervals throughout the process and/or at its conclusion. For example, assessment information may be collected on a person entering treatment, including the frequency with which alcohol or other drugs are used and various problems that result from substance abuse, such as family arguments, days of work missed, vehicle accidents, arrests, and medical problems. This same information may be collected periodically during the course of treatment and at the conclusion of treatment. This information can be aggregated for all patients and used to indicate that services were delivered as planned and that changes occurred with patients. It is not, however, possible to state conclusively from these studies that all changes were the result of the treatment program, because other factors also can intervene with patients. For example, family counseling, loss or change of jobs, suspension of driving privileges, changes in the way alcohol and drug cases are handled by law enforcement, and other such factors could affect the data that are collected during or after treatment. To provide a better picture of the effects of treatment, data should be collected at intervals following discharge from the program. Many individuals make significant changes during treatment but relapse quickly upon release.

Experimental Studies. Experimental studies compare a group of persons receiving treatment to a control group that is similar in size and characteristics but does not receive the same treatment. They compare the effects that occur both with and without the program in order to examine possible causal relationships. These studies are much more difficult and expensive to conduct, and they are not practical for all agencies. However, they are likely to produce the most convincing evidence of the effectiveness of a particular treatment program.

Evaluation Procedures

Agencies should have standard operating policies and procedures for collecting, recording, organizing, and processing the data. The methods of collecting information should be specified, such as interviews, surveys, self–reports, observations, and records reviews. Staff must understand what information is to be collected, when it is to be collected, and from whom. Data are often collected on paper by designated staff and then recorded in the management information system by different staff. Data files (whether computerized or manual) should be organized to facilitate reference to and retrieval of the data when needed. Data processing involves compiling, analyzing and interpreting the data to provide useful information to others in the most comprehensible manner possible. It is important that those processing the data remain objective and explore the range of possible conclusions that can be drawn from a particular set of findings. Often uncomplicated procedures, such as finding frequencies, ranges, percentages, and averages is sufficient for program evaluation. However, more complex statistical procedures, such as regression, multiple analysis of covariance, and discriminant analysis may provide a more definitive explanation of the relationship between treatment services and outcomes.

Reporting and Using Results

Programs should develop reports of the data produced through evaluation processes. These will be most useful if they are prepared in an understandable way without using professional jargon. Reports may be written or verbal and should be shared both within and outside the agency. It is also vitally important that there be evidence that program personnel attempt to use evaluation findings to make appropriate program changes. Policymakers and funding sources may wish to inquire about previous evaluation findings and program modifications that have resulted from these.

Confidentiality

While all aspects of the evaluation process are important, agencies also need to safeguard the confidentiality of patients. Often, identifying codes are used, rather than patients' names. All aspects of the process–collecting, recording, organizing, processing, and reporting data–should ensure the privacy of patients.

Patient Evaluation and Accountability

Closely linked to program evaluation is patient evaluation. The criterion for successful substance abuse treatment is continuing abstinence from alcohol or other drugs three to five years after treatment (ONDCP, 1990). Additional indicators of successful completion of treatment include the alleviation of related problems such as health, employment, financial status, relationships, and illegal behaviors.

However, as with program evaluation, it may be helpful to view treatment outcomes for substance abuse along a continuum. Between the extremes of treatment success and treatment failure are a range of possible outcomes. Those who decrease the use of substances but do not stop using them altogether cannot be considered as total treatment successes. Neither can an occasional lapse of drinking or drug use be viewed as a treatment failure. In addition to changes in consumption of alcohol or drugs, other outcome dimensions should be considered, including improvements in physical and emotional health, interpersonal relationships, vocational functioning, and criminal behavior (Hoffman, Harrison & Streed, 1991). Any change that diminishes the negative effects of alcohol and other drug use on the individual and society is at least a partial success.

Accountability is an important aspect of patient treatment success. Accountability involves delineating clear expectations for the behavior of patients in treatment. When these are met the individual should be rewarded. Rewards may include praise, privileges, and material items. However, if expectations are not met, consequences are warranted. Patients should be held accountable for showing up and being on time for treatment sessions. In residential settings, patients may be held responsible for performing daily chores and other duties. Urine testing is another form of accountability. Regular, random urine tests to determine whether or not drugs are being used, and appropriate sanctions for positive tests, will help patients acquire the self–control needed to succeed in treatment. Accountability measures in treatment are vital in helping individuals make responsible choices, including decisions about their alcohol or other drug use (ONDCP, 1990).

Evaluation of patients can be accomplished through a variety of means. A thorough assessment as described in Chapter 3 is important in developing the treatment plan. During the course of treatment, assessment should be ongoing in order to determine if additional problems exist or there is a change in the status of areas assessed earlier. Both should be documented. Positive changes, such as decreasing or stopping the use of substances, improved health, employment or academic stability, improved family relations, and the like, can indicate treatment progress. Concomitantly, the lack of improvement in some areas may indicate that the treatment plan needs to be modified to more nearly meet the needs of the patient.

Formal and informal evaluation procedures should be used intermittently during the course of treatment and following discharge. Informal procedures might include conversations between patients and program staff, observations of patient interactions and behaviors, and self–reports by patients. Assessment forms, questionnaires, structured interviews, and reviews of various records (e.g., treatment program, medical, legal) would be more formal evaluation procedures. Ongoing documentation should be made of individual patient success or problems in treatment.

Conclusion

Accountability is one of the five critical areas of substance abuse treatment. Program and patient evaluation is important for documenting program accountability. Programs need to furnish the services they say they will provide and in a manner that is consistent with currently acceptable treatment standards. They also should demonstrate that the services are effective in helping patients stop abusing alcohol and other drugs. Further, they must be able to accomplish these tasks in a manner that is cost–effective. Program costs should be within a reasonable proximity of similar programs providing corresponding services and achieving comparable outcomes.

The information gained from evaluations is valuable to persons making referrals for treatment. It is also vital for decision makers and funding agencies. Program personnel must use evaluation results to make appropriate modifications in treatment programs.

Systems coordination is essential in the area of program evaluation and accountability, just as in other areas. Treatment providers, policymakers, and funding sources must work collaboratively toward improving evaluation processes and treatment outcomes. Suggestions for coordination will be provided in Chapter 12.

References

Hoffman, N.G., Harrison, P.A., & Streed, S.G. (1991). Outcome evaluation. In J. Westermeyer & R.S. Krug (Eds.), Substance abuse services: A guide to planning and management. Chicago: American Hospital Publishing, Inc.

Hubbard, R.L. (1992). Evaluation and treatment outcome. In J.H. Lowinson, P. Ruiz, R.B. Millman & J.G. Langrod (Eds.), Substance abuse: A comprehensive textbook. Baltimore: Williams & Wilkins.

Institute of Medicine. (1990). Treating drug problems. Washington, DC: National Academy Press.

Kimmel, W. A. (1993). Need, demand, and problem assessment for substance abuse services (Technical Assistance Publication Series 3). Rockville, MD: Center for Substance Abuse Treatment.

Office of National Drug Control Policy (1990, June). Understanding drug treatment (White Paper). Washington, DC: Author.

Schinke, S.P., Botvin, G.J., & Orlandi, M.A. (1991). Substance abuse in children and adolescents: Evaluation and intervention. Newbury Park, CA: Sage Publications.

Singer, A. (1992). Effective treatment for drug–involved offenders. Newton, MA: Education Development Center, Inc.

Tims, F.M., Fletcher, B.W., & Hubbard, R.L. (1991). Treatment outcomes for drug abuse clients. In R.W. Pickens, C.G. Leukefeld, & C.R. Schuster (Eds.), Improving drug abuse treatment (Research Monograph 106). Rockville, MD: National Institute on Drug Abuse.

Weiss, C.H. (1972). Evaluation research: Methods of assessing program effectiveness. Englewood Cliffs, NJ: Prentice–Hall, Inc.

Previous | Table of Contents | Next
Top of Page

Last Updated 11-7-02