TIP 14 — TIPs — Documents — Home

Chapter 1—Introduction

Outcomes monitoring systems (OMSs) are broad-based efforts that aggregate data from many alcohol and other drug (AOD) abuse treatment programs. Their goal is to assess and ultimately improve patient outcomes, including the individual's functioning in society following treatment. An OMS can help to establish accountability for the expenditure of public funds. To allow the State to address the relevant questions, the OMS must, at the minimum, include sufficient information on patients, their treatment and their posttreatment functioning.

In each of the United States and territories, a State agency has responsibility for ensuring that AOD abuse treatment programs receiving Federal dollars provide effective treatment at the lowest possible cost. This assurance is difficult to give because many different approaches to providing treatment are employed today across the country, and patients have many different types of substance use disorders. The burden on the State agency to ensure accountability has intensified in recent years as a result of the rising costs for provision of healthcare services and the healthcare reform movement.

The healthcare reform movement has brought about changes in the management of healthcare in numerous States, leading many single State agencies (SSAs) to recognize the need for initiating outcomes monitoring or augmenting existing efforts. SSA directors and staff, as well as involved constituencies, need assistance in developing and improving OMSs. This Treatment Improvement Protocol (TIP) provides needed information and guidelines.

As part of its ongoing commitment to identify and respond to critical issues in the AOD field, the Center for Substance Abuse Treatment (CSAT) convened a panel of professionals in AOD abuse treatment and allied fields. The panel was charged with examining outcomes-based accountability as it applies to AOD treatment and with conceptualizing and developing a TIP on the subject. In preparing this TIP, the panel was asked to do the following:

• Consider what monitoring activities currently exist
• Identify supportive resources and obstacles to improving the information base
• Make recommendations to SSAs about the creation and operation of outcomes monitoring systems.

Thus, the purpose of this TIP is to assist single State agencies in the development, implementation, and management of an OMS to improve treatment outcomes and to increase accountability for AOD treatment expenditures.

Because the success of a State OMS will depend on collaboration among many stakeholders, there are a number of agencies and professionals in the audience for this TIP:

• AOD abuse treatment planners, administrators, and research and evaluation personnel at State AOD agencies who are responsible for the design, implementation, management, or oversight of treatment programs
• Utilization reviewers, quality assurance personnel, and staff of managed care programs who evaluate the outcomes of AOD treatment programs
• AOD direct service providers including physicians, psychologists, counselors, social workers, nurses, and therapists who provide the full range of AOD services
• Other interested allied health professionals
• Professionals from other fields who encounter the consequences of AOD abuse, such as criminal justice, public health, and social services
• Third-party payers who have a stake in ensuring not only the effectiveness of AOD abuse treatment but also its cost-effectiveness.

The field of AOD abuse treatment has not been exempt from the influence of forces that have been acting on the Nation's healthcare delivery system. The Nation has moved into an era of increasing emphasis on outcomes-based accountability (Relman, 1988). While calls for healthcare reform emphasize universal coverage and cost containment, access to care and low-cost care alone will not guarantee improved healthcare. Reform efforts must require the use of effective forms of treatment that have demonstrated their value in curing or arresting disease or relieving patients' distress. Decisions about the best care for individual patients must be based on outcomes (Allo et al., 1988; Longabaugh, 1991).

Alcohol and other drug abuse treatment has always received intensive scrutiny—far more than have other healthcare services. This scrutiny is probably because of the nature of addiction and the visibility of its effects. Although attitudes have changed greatly over recent decades, persons with AOD problems are still viewed with a great deal of ambivalence. For example, many people who say that alcoholism is an illness also say that alcoholics drink because they want to and that alcoholics are morally weak individuals (Caetano, 1989; Gallup, 1987). Abuse of illegal drugs elicits even harsher judgments. Media attention to so-called treatment failures that involve repeated offenses of driving while intoxicated or other more highly publicized crimes increases public skepticism about the benefits of treatment.

Despite lingering reservations about treatment effectiveness, a considerable body of research documents the fact that receiving AOD treatment is better than not receiving it and that treatment costs are offset by savings in other areas (Luckey, 1987). Treatment leads to substantial reductions in alcohol and drug problems; improvements in virtually all other areas of patient functioning, including physical health, psychological and social functioning, and employment; and reduced incidence of criminal behavior (Hubbard et al., 1989; McLellan et al., 1992a; Pickens and Fletcher, 1991; Tims et al., 1991). Cost-offset studies consistently document reduced healthcare costs following treatment for AOD abuse (Holder, 1987; Holder and Blose, 1992, 1986; Holder and Hallan, 1986, 1981; Holder and Shachtman, 1987; Holder et al., 1985; Jones and Vischi, 1979), along with reduced criminality and increased employee productivity (Alander and Campbell, 1975; Alfano et al., 1987; Harrison and Hoffmann, 1989; Hoffmann et al., 1984; Hubbard et al., 1989). Findings from State studies are remarkably consistent in this regard (Young, 1994). In the most rigorous and comprehensive study of cost offsets conducted to date, a return on taxpayer investment of $7.14 for every $1 spent on AOD treatment was recently documented in California (California Department of Alcohol and Drug Programs, 1994).

However, while it is known that treatment is effective, it is also known that no single treatment approach is effective for all persons with AOD problems. Outcomes are determined by a number of factors: the characteristics of individuals seeking treatment, the nature and severity of their problems, the treatment process and the services provided, posttreatment environmental conditions, and the interactions among these factors (Ball and Ross, 1991; Bromet and Moos, 1977; Budde et al., 1992; Berglund et al., 1991; French et al., 1993; Gerstein and Harwood, 1990; Harrison et al., 1988; Institute of Medicine, 1990; Joe et al., 1992; McLellan et al., 1993, Vanicelli et al., 1983).

Much more about treatment could be learned from well-planned outcomes monitoring systems. Such systems could provide answers to many questions: What kind of treatment is best for different populations of patients? What specific components of treatment are essential to recovery? What are the relationships between the specific treatment services patients receive and the outcomes of their treatment? Do different kinds of treatment settings provide benefits to different kinds of patients? Do particular combinations of services improve outcomes? Does increasing or decreasing the length, the frequency, or the intensity of specific services improve prognosis? To what extent does the context of treatment influence outcomes? Although some programs offer gender-specific services or services tailored to cultural or ethnic populations, selected age groups, or other special populations, we do not yet know whether these specialized services have affected treatment outcomes.

It is widely believed that matching patients to programs or services that best meet their needs will improve treatment outcomes (Institute of Medicine, 1990; Smart et al., 1990-1991). Although results of fairly small-scale studies conducted to date are promising (McLellan et al., 1983a, 1983b; Schottenfeld et al., 1992), much larger efforts involving more diverse populations and services will be required to determine if patient-treatment matching attains the goal of improving treatment outcomes and to calculate the cost-effectiveness of different treatment approaches (McLellan and Alterman, 1991).

Across the country, wide variations exist in the types of care given to patients with AOD disorders. Differences exist within and among States in the use of various therapeutic modalities, treatment settings, treatment goals, and variety of services offered, as well as in the quantity of specific services available.

Treatment in the United States typically consists of some combination of services such as detoxification, medical care, psychological assessment, educational lectures and films, group therapy, individual counseling, family counseling, recreational and occupational therapy, medication, use of community-based peer support groups such as Alcoholics Anonymous, and continuing care or relapse prevention sessions. Depending on the modality, treatment may last anywhere from a few sessions to a year or longer. Much information about the benefits of these different combinations of services could be learned from outcomes monitoring systems.

The single State agency has a key role with respect to AOD treatment. Its job is to ensure the provision of high-quality care for patients, based on their individual needs, in order to achieve the best outcomes for the lowest possible costs. State agencies typically play a role in the provision of program funding or provider reimbursements, program development, program licensing, and/or patient assessment and placement standards. An outcomes monitoring system, as described in this TIP, is a type of management information system (MIS) designed to collect sufficient information to allow the State to fulfill its role and responsibilities in these areas. To address the relevant questions, the OMS must, at a minimum, include sufficient information on patients, the treatment they receive, and how they are doing after treatment.

Outcomes monitoring can be defined as the assessment, at some point following treatment, of patient status in key life areas related to substance use disorders. The purposes of outcomes monitoring are the following:

• Help States, treatment professionals, and policymakers determine the effectiveness of certain types of AOD abuse treatment for different types of patients
• Improve program performance by using outcomes data to identify weaknesses or gaps in services, and provide feedback to enhance system performance
• Improve the patient assessment process, using empirical outcomes data to develop and refine treatment placement criteria that optimize the chance of successful outcomes
• Document cost offsets and minimize inefficiencies and unnecessary expenditures in treatment programs
• Help improve resource allocations
• Provide justification to support funding requests, especially the use of public dollars, and the necessity of including treatment in a basic healthcare benefits package
• Improve managed care by providing data on treatment outcomes to guide decisions about the wise use of the healthcare dollar.

Outcomes monitoring systems share characteristics in common with experimental research, quality assurance and improvement efforts, and program evaluation. All include data gathering and, to varying degrees, require some standardization of instruments and procedures and training of data-gathering personnel. However, there are differences in outcomes monitoring systems, particularly in their purposes and scopes.

Experimental Research

Experimental research has as its primary goal the accumulation of knowledge and thus emphasizes theory development and hypothesis testing. Experimental designs involve the randomization of subjects and the use of control groups. Only experimental designs can establish a causal link between a specific intervention and a treatment outcome. Most experimental research is relatively expensive because it involves comprehensive data collection by extensively trained staff and the greatest degree of technical sophistication in the outcome evaluation approaches reviewed here.

As an example, consider a project designed to test a new medication to determine whether it helps to alleviate the subjective experience of craving among persons addicted to cocaine. The project design might include six outpatient treatment sites, each recruiting 60 volunteers. Patients would be told that the research involves the use of an experimental medication and random assignment to one of three groups: a group receiving the experimental drug, a group receiving an antidepressant medication proven to help reduce craving, and a group receiving a placebo. The pills each group received would look alike; neither the patients nor the doctors and staff involved in the study would know which patient was receiving which medication (this is called a double-blind study).

Patients in all three groups would also receive the standard treatment provided by the outpatient program to which they were admitted. For 12 weeks, the patients would meet with staff once a week to complete questionnaires on the frequency and intensity of craving. Staff would also record their observations of patient behavior. At the end of the study, the pharmaceutical company testing the medication would analyze the data on the three groups of patients.

Even if the findings showed an effect of the experimental drug on alleviating a craving for cocaine, changes in clinical practice are dependent on a variety of outside factors. These include approval by the U.S. Food and Drug Administration, the price of the drug, the availability of other medications, loyalty of clinical practitioners to other drugs, or even philosophical objections to the use for this purpose of any medication for people with addictions. Thus, the knowledge gained as a result of experimental research may be far removed from changes in clinical practice or general healthcare policy.

Quality Assurance and Improvement

Historically, quality assurance efforts have been focused on the delivery of services, not outcomes. There is now greater attention on outcomes as well. Two well-known efforts in this area are conducted by the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) and the Commission on Accreditation of Rehabilitation Facilities (CARF). Licensure agencies provide other examples of quality assurance.

Since 1992, JCAHO has been revising its standards on quality assessment and improvement to reflect increasing emphasis on effective and efficient use of healthcare resources (JCAHO, 1992). JCAHO and similar organizations, in collaboration with healthcare professionals, develop quality standards and provide assistance to healthcare providers in meeting or exceeding those standards. Such efforts typically examine the full spectrum of governance, management practices, and clinical processes. They focus on structures and activities and the integration and coordination of these related aspects of program performance. The goal is continual quality improvement.

What distinguishes quality assurance efforts of this type from individual program evaluation is their broad-based nature. Through consensus, uniform standards are adopted and individual program performance is assessed against these practice parameters. Accreditation or licensure by an external reviewer attests to the achievement of established standards.

Program Evaluation

As with quality assurance, the purpose of program evaluation is improved service delivery. Program evaluation designs, however, are typically individualized to meet the unique needs of a program or group of programs (see accompanying box). The evaluation may be initiated by the program itself or it may be mandated by an external funding or review agency. The audience includes program administrators and clinical staff as well as funders. Like quality assurance, program evaluation may involve direct observations of service delivery, review of records, and interviews with patients, staff, and other stakeholders. Data collected may be qualitative as well as quantitative. The emphasis is on program performance, not individual patient outcomes. Program evaluation has recently evolved to include more of an emphasis on patient outcomes, but the collection of outcomes data is often not systematic.

Outcomes monitoring systems are broad-based efforts that aggregate data from many programs. The primary goal of outcomes monitoring systems is to assess and ultimately improve patient outcomes. An OMS can help to establish accountability for the expenditure of public funds. Examples of existing State outcomes monitoring systems are provided in Appendix B.

While there may be many ways to use an OMS, the primary audience is broader than the participating programs and may include State or Federal regulatory agencies or other groups with a stake in the treatment's being monitored. Findings from outcomes monitoring systems may lead to policy changes at the State or Federal levels or by other healthcare payers.

Outcomes monitoring systems use standardized data elements and data collection procedures at different sites. OMSs include some contact with the patient or the use of collateral records following treatment. OMSs have some components in common both with experimental research and with program evaluation, but there are also major differences. The patient—not the program—is the unit of data analysis. Monitoring is ongoing rather than time limited. The number of patients involved is usually quite large. The system may include a variety of treatments in a variety of settings at multiple sites.

The outcomes monitoring system assesses the treatment delivery system "as is"; patients are not involved in experimental research. Because an outcomes monitoring system does not involve the use of experimental and control groups, the data collected cannot causally link a specific outcome to a specific intervention. However, meaningful treatment service comparisons can be achieved because collecting data on the same patients before and after treatment allows each patient to serve as his or her own control (California Department of Alcohol and Other Drug Programs, 1994).

One fundamental question must be resolved when planning an OMS: Is the major objective to address questions related to the treatment service delivery system as a whole or to compare individual treatment program results? Consensus on this issue may not be easy to achieve. Strong arguments can be made for both approaches.

Advocates who favor designing the OMS to measure individual program performance point to the benefits for consumers and payers of identifying superior and inferior programs. Opponents of this approach express serious doubts about its feasibility—considering sample sizes necessary for valid comparisons and the enormous amount of heterogeneity among treatment populations.

Measuring treatment system outcomes versus measuring individual treatment program outcomes is not necessarily an either/or situation. In any OMS designed to address program results, data could also be aggregated to address questions pertaining to the broader treatment system. The converse is not necessarily true, however. An OMS could be designed to address questions about the treatment system but be insufficient to provide individual program comparisons.

Members of the national consensus panel for this TIP did not reach consensus on this issue. Therefore, the recommendations which follow take into account both approaches. While planners of future outcomes monitoring systems may find debate on this issue to be contentious, they will also find it worthwhile. The ensuing discussions will bring to the fore many related issues: the role of the State in monitoring and improving treatment outcomes; factors involved at the program level that may relate to outcomes; the degree of scientific rigor necessary to ensure confidence in results; the importance of study design, sample size, and technical expertise in making sure the monitoring system can do what it is expected to do; and the benefits and drawbacks of linking funding to performance.

While an OMS can be designed to measure the performance of an individual program, there are considerable obstacles to this use of the system. To conduct valid comparisons of treatment programs in terms of treatment outcomes, program samples must be sufficiently large to ensure that observed differences are statistically significant. Even when outcome differences are significant, the poorer outcomes may be attributable to patient characteristics at admission. Many patient characteristics are associated with poorer outcomes. These characteristics include severity of AOD use, early age of onset, a history of antisocial behavior, vocational and social instability, and the severity of psychiatric disorders (Berglund et al., 1991; Budde et al., 1992; Gottheil et al., 1992; Harrison and Hoffmann, 1987; Harrison et al., 1988, 1991; Longabaugh, 1991; McLellan et al., 1983a).

Two programs may offer comparable quality services by equally skilled staff, but the one that serves higher risk patients will probably have poorer outcomes. Unless differences in patient characteristics can be controlled in statistical analyses, the outcomes comparisons will not be meaningful. Sufficiently large sample sizes can make valid program comparisons possible; however, sample size is a key factor in driving up the costs of an OMS. Finally, other factors such as sample bias threaten the validity of program comparisons (Gerson et al., 1985; Harrison and Hoffmann, 1989; Stinchfield et al., 1994a). An example of the effects of sample bias introduced by disparity in consent and posttreatment followup rates is described on the next page.

Focusing on the aggregated data from all programs may produce even more important benefits than individual program outcome comparisons. An aggregated data set of sufficient size can address a variety of broad questions about the correlates of successful outcomes and the best use of available resources. In contrast to the report-card approach of measuring individual program performance, the systems approach facilitates a self-correcting treatment service delivery system in which continuous feedback is used to improve services, make management decisions, and refine policy.

Because an OMS will necessarily include data on patient characteristics, severity of functioning, and services provided, along with patient outcomes data, groups of patients can be compared with other groups of patients with similar characteristics who receive different services, or services that are different in intensity or duration. Pooling data from a diversity of patients and treatment models and services will ultimately enhance the ability to determine what kinds of service and levels of service are most likely to benefit specific populations of patients. The emphasis here is on treatment components (or service packages) rather than on treatment programs. Analysis targets the sets of services that comprise treatment rather than the individual treatment program.

Example of Sample Bias Suppose you were to compare the abstinence rates 6 months posttreatment for Program A and Program B. (Set aside for now whether or not this measure is a good one for program outcomes.) Let us even assume that characteristics of the patient populations at both treatment sites were virtually identical on all measures that might predict outcomes (again, set aside skepticism). Data analysis reveals that Program A has a 50 percent abstinence rate and Program B, a 65 percent abstinence rate. Some might want to promote Program B as superior, but closer scrutiny is required. Both programs had 110 admissions during the study period. At Program A, 100 patients consented to followup and 80 of these were contacted for the 6-month followup interview. The Program A followup sample consisted of 73 percent (80 /110) of the original admissions; 40 of these (50 percent) reported no alcohol or other drug use after treatment. At Program B, 95 patients consented to followup and 62 of these were contacted for the 6-month followup interview. The followup sample consists of 56 percent (62 / 110) of the original admissions; 40 of these reported no alcohol or drug use after treatment. The first problem with this finding is that the observed difference in outcomes (40 of 80 interviewees versus 40 of 62 interviewees reporting abstinence) is not statistically significant. That means that there is a greater than 5 percent probability that the difference could have occurred by chance with samples of this size. Second, there is a much greater potential for sample bias at Program B than at Program A because fewer patients consented to followup and fewer of those for whom followup attempts were made were successfully contacted. Actually, the number of patients from both programs who reported abstinence is identical—40. It cannot be determined from the data available which program is superior. It depends on how the nonlocated patients are doing.

System questions are generally those for which answers can be enacted into statewide policy, such as standardized patient placement, continued-stay criteria, and resource allocation. Examples of such questions are:

• Can the higher costs associated with inpatient or residential treatment be justified in terms of superior outcomes? For which kinds of patients? Under which conditions?
• What is the optimal intensity of outpatient treatment for different patients at different stages in the recovery process? How many days a week? How many hours a day?
• Can an optimal length of stay be determined based on patient characteristics or treatment progress? Is there a minimum length of stay, short of which treatment is of no value? Is there a maximum beyond which no further benefits accrue? Should treatment last much longer than is currently the case but perhaps have greater flexibility for crisis intervention and decreasing levels of involvement?
• What ancillary or adjunct services are essential to improved treatment outcomes for which groups of patients? Housing assistance? Education or job training? Childcare? Parenting skills training? Transportation?
• Are specialized programs for special populations associated with better outcomes for certain groups of patients (women, adolescents, racial/ethnic or other cultural groups, cocaine abusers, alcohol abusers, persons with psychiatric disorders or histories of crime, for example. If so, are the outcomes so superior as to justify a greater investment of resources in such programs?
• Are certain therapeutic modalities associated with better outcomes for some kinds of patients rather than others?

Two examples of the way statewide aggregated data can be used to address broader questions illustrate the benefits of this approach to system design. (See the Hypothetical Example boxes on the pages that follow.) Each example, while hypothetical, illustrates a practical and feasible accomplishment of a State OMS. However, this is not to suggest that getting to this point of OMS capability is a simple task.

The next section includes some fundamental issues that need to be addressed early in planning an OMS.

Once a commitment is made to developing a State outcomes monitoring system—once the idea is sold—there is often a temptation to move immediately to designing data collection forms. But the process demands a great deal of thoughtful reflection before any concrete design issues can be addressed. Some basic realities that must be considered are briefly discussed here as guiding principles.

The nature of addiction must be considered in defining beneficial outcomes. Persons with AOD problems are diverse in their personal characteristics and life histories, their substance use patterns, the adverse consequences they have experienced, and the resources they bring to bear on their attempts to recover from addiction. In recognition of this diversity, it is essential to measure treatment outcomes in terms of each individual patient's improvement along a continuum rather than in comparison with some predetermined arbitrary success measure. A dichotomous measure of success versus failure is too simplistic to characterize treatment outcomes (Longabaugh, 1991; Stinchfield et al., 1994b; Wells et al., 1988).

Many patients will improve sufficiently to justify the investment in treatment even when they do not attain ideal treatment goals (McLellan et al., 1993; Walsh et al., 1991). Because AOD abuse leads to both health and social problems, treatment outcomes measurement must include many dimensions. Measures must address AOD use as well as the areas in which AOD use has had an impact on patients' lives, such as general health, social and occupational functioning, and legal involvements.

The specific outcomes measures selected should be derived from the definition of treatment benefits and related to the goals of treatment. At the same time, it is imperative to foster a climate of realistic expectations regarding what treatment can and should be expected to do. For example, to the extent that AOD use impairs job performance and social functioning and leads to crime, beneficial treatment can be expected to show concrete effects in these areas. But treatment in its limited timeframe cannot be expected to compensate for economic deprivation, educational deficits, and other longstanding difficulties.

The OMS must be feasible. Designers of a successful outcomes monitoring system will have to balance practicality against the ideal design. Availability of resources will dictate the scope and capabilities of any system. AOD resources are shrinking in many areas, even while demands placed upon them are increasing. As time goes on, more is expected of State agencies and treatment providers. In this climate, outcomes monitoring may be seen as just one more added burden. Without a distinct and visible payoff, enthusiasm for the undertaking is likely to be lacking.

An outcomes monitoring system must be acceptable to a broad spectrum of stakeholders—especially treatment program administrators and staff who will be asked to collect much of the required data (Camp et al., 1992). The final design must minimize treatment program disruption and demands on staff. At the same time, its findings must be credible and useful to its audience. The design must ensure valid data and generalizable results.

Costs of an ideal outcomes monitoring system would make it prohibitive. Therefore, planners will have to agree to accept the limitations of their OMS. No set of instruments can measure everything everyone wants measured. Nor will any set of procedures live up to the demands of all researchers. More data collection and more stringent methods translate into more time and money expended (Longabaugh, 1991).

Tough decisions will have to be made at the outset regarding the essential components of the system. Which information is most useful? What methods are most practical? What followup intervals and procedures are feasible? What can be done well within a limited budget? What can be done first? Would it make sense to implement a skeletal system and supplement and improve it later?

Not all treatment program goals are suitable for use as outcomes-based accountability measures. While certain personal changes have profound meaning for individual patients, they may be difficult and time consuming to measure. The most useful outcomes measures will have broad public and political appeal, such as evidence of reduced harm to society and cost savings in other areas that result from successful treatment (Allo et al., 1988).

Hypothetical Example: Poorer Outcomes Found for Rural Patients Suppose an analysis of aggregated statewide program data revealed that patients admitted to treatment programs in rural areas had poorer outcomes than those admitted to programs in urban areas, even when differences in patient characteristics and treatment services were taken into account. Further analysis revealed that the differences in outcomes were largely attributable to the use of community support services after treatment: on average, urban patients reported much greater involvement with community resources than rural patients. In discussions to interpret this finding, treatment staff consistently emphasized the relative lack of recovery support services in remote areas. This lack led to a recommendation to allocate more State resources toward increasing the availability of community supports in rural areas. Without this analysis of aggregate statewide data, an erroneous conclusion might have been reached and resources wasted. If the focus had been on individual program comparisons, the evaluation might have concluded that rural treatment programs were of poorer quality than urban AOD abuse treatment programs. A great deal of time and money may have been spent to improve rural programs when this improvement was not what was needed.

A successful OMS requires some degree of standardization across participating treatment programs. In order to compare patients, treatment services, or outcomes, some degree of standardization will be required. Some consensus must be achieved with respect to baseline and outcomes measures and the use of data collection instruments and procedures. While planners may agree in principle, the difficulty arises when specific decisions must be made. Currently, there are no instruments that have universal acceptance for purposes of patient assessment or treatment service description (Longabaugh, 1991). Without a directive for certain standardized data elements or procedures superimposed by the Federal Government, States will be on their own to reach internal consensus. While there may be benefits to having all States adopt at least a subset of identical data elements to measure outcomes, no mechanism exists to ensure this standardization.

General recommendations regarding OMSs must take into account the diversity of States. A great deal of diversity exists in the States and territories that review the recommendations of the consensus panel presented in this TIP. These units vary in terms of the populations they serve; the prevalence of alcohol and other specific drug use disorders; the rates of injection drug use, HIV/AIDS, and tuberculosis among the AOD-using populations; the mix of rural and urban communities; poverty rates; language diversity; the racial or ethnic composition of the populace; and the extent to which cultural factors influence substance use patterns and attitudes toward treatment.

Differences also exist among States in terms of their financial resources and the burdens placed upon these resources. States also vary in terms of how programs currently operate and how they are funded, as well as in the variety of treatment philosophies, programs, and services they have in operation.

There are vast differences among States in terms of their current MIS capabilities and how much must be accomplished before an outcomes monitoring system can be implemented. While some States have already designed and put into practice statewide data collection systems that include a followup component, others have yet to begin the planning process.

Finally, it is important to acknowledge the ever-changing landscape of the healthcare delivery system. Any general recommendations with regard to AOD abuse treatment outcomes monitoring systems must be considered in the context of other State or Federal requirements that may be imposed.

An OMS is not successful unless its findings are put to practical use. The most frequent and justifiable complaint about large-scale data collection requirements is that the information disappears into a black hole. What too often happens is that the bulk of planning effort and financial resources are committed to data collection. Data analysis and the dissemination of results receive short shrift. Unless everyone whose efforts are needed to ensure success is convinced that something useful will emerge from all their hard work, the process will fizzle out. Initial planners must consider in concrete ways how the resulting information will be used and what benefits it will have. They must determine who will use the results.

Hypothetical Example: Recovery Maintenance Services Found To Be a Strong Predictor of Successful Outcomes Suppose an analysis of aggregated statewide program data found that one of the strongest correlates of successful outcomes is length of involvement in program-directed continuing care or recovery maintenance services. In this hypothetical analysis, continued contact with professional services increased the likelihood of abstinence even when patients were attending AA or similar groups. In fact, participation in continuing care after primary treatment was completed was found to be a stronger predictor of successful outcomes than any specific component of primary treatment itself. A recommendation emerged to prioritize the use of limited available resources to promote the development and expansion of continuing care rather than to improve or add intreatment service components.

Treatment program administrators and staff? Referral agents? Legislators? Consumers? Will findings be used to reward high performers and penalize poor performers? Will findings be used to shut down programs? Will the results be the basis of performance-based contracting? Will results be used as the basis for recommendations to match patients to programs or services? Achieving the buy-in and turning empirically derived information into policy are addressed in detail in Chapters 2 and 8.

Progress can be incremental. The perfect outcomes monitoring system cannot be designed and implemented in the abstract. The process may have to begin on a small scale with a limited number of data elements and/or a limited number of participating programs. Unanticipated difficulties will occur. Revisions will be required. Some questions will not be answered. Improvements can be made as time goes on. The scope can be increased. States can learn from their own successes and shortcomings—and they can learn from the efforts of other States.

Chapter 2 examines the political considerations necessary to plan a successful outcomes monitoring system. This chapter addresses the involvement of a variety of stakeholders, the formation of steering committees, and the obstacles and anxieties that will be encountered. Chapter 3 addresses ethical issues related to outcomes monitoring systems.

Chapter 4 examines a variety of methods appropriate to statewide outcomes monitoring systems and discusses the benefits and limitations of various strategies. This chapter includes some specific recommendations as to patient samples, program samples, patient followup data collection methods, followup intervals and procedures, and program staff training. Chapter 5 focuses on the content of an OMS: the information to be collected to describe patients at baseline and followup and to describe treatment. Chapter 6 addresses legal issues relevant to outcomes monitoring systems.

Chapter 7 addresses some technical considerations with respect to management information systems and appropriate hardware, software, and data transfer along with staffing and budgetary concerns. Chapter 8 is devoted to the dissemination of findings and turning empirical results into policy.

Appendix A is the bibliography, containing all references cited in the text. Appendix B describes current State outcomes monitoring systems, and Appendix C provides a list of representatives of Federal agencies and national professional associations who participated in the Federal resource panel. The latter group provided advice in the early stages of planning the CSAT consensus panel and the TIP. Appendix D provides the names and affiliations of all those who reviewed and provided comments on the first draft of the TIP; their advice contributed greatly to the final shaping of the document.

Last Updated 11-7-02