Meaningful Outcome Evaluation

For Native American Treatment Programs

  Matthew Kelley Ph.D.

  “Cross-site” and “Localize” Techniques

Remarkably Complex:  

To better prepare for today’s cost-accountable and 

resource- limited world, modern outcome design must 

illustrate at least three things:  the client’s experience 

and change over time; the cost-effectiveness of a program, 

and the worthiness of various protocols for various populations. 

The bottom line, of course, is that rigorous evaluation must 

make treatment better.

Sanchez Craig, (1986), Thombs (1994), and the Institute of Medicine (1990) point out that the relapse rates of treatment facilities are often higher than what is publicly presented. This public over-statement is often due to the lack of research resources, the inevitable variation of treatment quality from group to group, weak research methods, and the facility's both unconscious and conscious vested-interest in presenting positive results.

Usefulness: To correctly understand outcome measures, it is always essential to first define data validity (appropriateness of measurement), data reliability (confidence in data accuracy), and the etiological context in which the drinking problems occur. The “context” of a relapse rate must take into account specific client characteristics such as individual pathology, the amount of aftercare support, the client’s motivation, and the client’s original drinking characteristics. The National Institute of Alcohol Abuse and Alcoholism state that outcome data cannot be functionally understood unless there is a full understanding of the client's original predictive variables. Others suggested that the analysis of specific relapse rates alone is too simplistic and does not place the problem in an appropriate perspective (Moos, Finney, & Gamble, 1982). Furthermore, these authors suggest that "treatment" is only one of the many factors that contribute to successful outcome. For example, it is possible to identify environments so suppressive that they would eventually encourage even a normally happy, highly productive, and neurologically "resilient" person into a pattern of excessive drinking. In other words, success criteria are always relative to the individual's problems and conflicts (Institute of Medicine, 1990).

Reliability of assessment data.    Two main challenges to accurate data collection are both the desire to respect the client's private world and the frequent, non-reliability of the client's self-report. One study concluded, after attempting to verify self-reports with collateral interviews and blood and urine testing, that only 65% of those people reporting total abstinence were truthful about their drinking habits (Fuller, Lee, & Gordis, 1988).  In another study using collateral information to crosscheck the client's self-report, about 50% of the cross-verifications did not correspond to self-report (Watson, Tilleskjor, Hoodecheck-Schow, Pucel, & Jacobs, 1984). The Institute of Medicine's (1990), along with other researchers, believe that self-reports are neither inherently valid nor invalid and that the circumstances where such reports are given can either increase or decrease their validity (Lettieri, 1992; Skinner, 1984; Sobell, Sobell, & Nirenberg, 1987). In his report to the National Institute on Alcohol Abuse and Alcoholism, Lettieri also stated that reliability actually depends on the methodological sophistication of the person gathering the data, the personal characteristics of the respondent, and the quality of rapport between the interviewer and respondent.

In order to increase the validity of a verbal self-report these authors recommended that: (a) the client is free of alcohol at assessment; (b) the client is medically stable with no major health symptoms; (c) the interview is structured and carefully developed; (d) the client suspects that his or her statements will be cross verified; (e) there is good rapport between client and interviewer; (f) the client has followed his or her aftercare suggestions; (g) the client has no obvious motivation in distorting facts; (h) the client is assured that all comments will be confidential; (i) the interviewer, and related staff, appear neutral and non-punitive; (j) two or more assessment instruments are used.

Relapse statistics can also be skewed by other variables. Many

studies wrongly eliminated clients who were difficult to contact, or

clients who have what they called "unstable" situations such as

being unmarried, or those who are non-compliant (Wallace, McNeill,

& Gilfillan, 1990).

Variations of defining success. The wide variation and constant confusion of reported outcome results also reflect the complex range of assessment standards, assessment protocols, treatment quality, population differences, etc.. 

Analysis becomes even more complex when success is defined in different ways by various facilities. Some treatment facilities describe "abstinence" as "no drinking at all" while other facilities expand the definition of abstinence to include clients who might have had major slips but who stayed relatively healthy and out of trouble (Institute of Medicine, 1990).  Most of these studies make little mention of either their assessment protocol or their relapse criteria. For example, even the DSM-IV stated that 65% of all "highly functioning" treatment participants will become abstinent for at least one year (American Psychiatric Association, 1994, p.202).  Definitions of "highly functioning" and "abstinent" were not offered. Of course, for the DSM-IV to select-out the highly functioning people from the group of “real-life” people (as they did here) makes a mockery of such information.  

In another important point relative to measuring success, as mentioned above, the DSM-IV concluded that an estimated 20% or more people with alcohol dependence will eventually establish their own long-term sobriety even without treatment. The self-treatment rate (spontaneous abstinence or spontaneous controlled drinking) also seems to vary according to the age of the person (Fillmore et al., 1988).

Drinking‘Results’ as the bottom-line misses the mark.    As an additional problem, mainstream outcome evaluations often focus on a program’s “bottom-line” drinking results rather than the challenge, drama, and dynamic process of the client’s adventure through the intermittent “process” of regaining control over their life.  Unlike plumbers and surgeons who cause something to happen by direct mechanical intervention, treatment providers (through caring and intelligent action) can only make critical contributions in an attempt to push or influence a client towards either self-healing or into a self-transformative process.  Kibel (1996)  pointed out that much of what happens to clients is actually beyond the therapist’s control.  What comes before or after treatment is largely outside of the therapist’s influence. Any contribution that the therapist makes usually occurs in small increments, in varying degrees, and often with time lags. Great therapists and good programs will (hopefully) only increase the probability that positive change will occur in the client. Good therapists and their practices may only catalyze (or synthesize) a range of short-term (often small and even nebulous) achievements that reinforce and enhance each other.  Ideally, the therapist’s effort builds both a potential and momentum towards the realization of the client’s long-term success. These contributions may even be indefinable and may be different for each person. Insisting on the exclusive use of outcome data for “bottom-line” program assessment often overlooks both the important contributions of a quality program and the real world challenges of environ/psycho/social/bio context.

Ethical Considerations for Native American Evaluations

Very importantly, there are also special ethical considerations when evaluating behavior within any minority culture.  Native American communities are justifiably cautious about outside evaluators and research projects that they often perceive as self-serving to either the dominant culture as a whole, or self-serving to the individual researcher (Norton, 1996, Oct.).  Relatively few benefits return to the local community. This perceived one-way street often leaves the bitter feeling of abuse.

Among the Navajo Nation, for example, terms such as “research” and “study” are justifiably taboo and imply (to many) a self-serving motivation from the dominant culture.  A more acceptable approach is to develop a design structure, which honestly conveys the mission of “needs assessment,” “program evaluation,” or “analysis.” These terms tend to imply an “action-oriented” approach; an evaluation approach that has immediate and internal benefits to the people (“stakeholders”)  involved.

Evaluation projects among Native American communities must also guard against cultural exploitation or social damage to the participants. This is especially important when helping a high-risk population such as Native American substance abusers;  a population already victimized by persistent cultural-laden prejudices, misinformation, and various forms of economic exploitation. 

       Ethical cross-cultural evaluation must always reflect a minimum of the following principles: (a) the design must be action-oriented so it directly addresses immediate local problems; (b) local leaders and local specialists must be intimately involved; (c) ethnic experts must be fully involved in all implementation and assessment; (d) there must be tangible benefits to the community (Corey, 1994).  Special accommodations must also be made to insure accurate cross-cultural communication (i.e. language translation, culturally sensitive form-design, avoiding overly academic and legalistic language, maintaining culturally comfortable assessment and observational environments) (Attneave, 1989; Norton, 1996). Of course, all generally accepted and standard ethical research  guidelines must also be met.

Self-appraisal “localized” evaluation . The first goal of outcome evaluation is to immediately feedback useful information to all “local” program stakeholders (categories of participants from funders to clients) in an attempt to both improve and understand the program.  This “action-based” orientation emphasizes both deep self-appraisal and program analysis rather than judgment and comparison with other programs. Because each program largely remains unique in it’s mix of resources, context, and client etiology, this evaluation design must always be specifically tailored for and then interpreted within it’s own context. Whatever the chosen design, to be useful, these protocols should still provide meaningful measurement (validity), be reliably gathered (data collected with a high level of confidence).

In addition to both their awkwardness and incompleteness in real real-life situations, comparative control studies are certainly premature in Native American treatment setting.  In such situations, outcome success is already accepted as low, is often confusing and incomplete, and has generally been presented out of relevant context.  The critical variables of traditional Native American therapeutic techniques are also currently far too ill defined to make control groups a present possibility. In many cases, because participants have already established a significant history of treatment failures, a “within-group” pre and post treatment comparison is better suited.  [If a client participated in several mainstream treatment regimes without much change (as most of the Native American clients have) and then, after completing a new program, this same participant suddenly experiences positive movement in behavioral indices associated with his or her destructive drinking pattern, the new intervention will likely prove significant.]  The specific active and critical variables of any new program, of course, are what remains undetermined in such a “internally” or non-controlled outcome study.    

“Cross-site” (Comparative) vs. “Local” (Self-appraisal) Evaluation.   It is also useful to distinguish between the two basic types of evaluation; comparative (cross-site) and self-appraisal (local) evaluation. Comparative evaluation involves comparing similar data between similar or non-similar programs while self-appraisal evaluation involves gathering valid, reliable, and penetrating data for internal use only. A comparative evaluation (cross-site) is a “quick-indicator”  thermometer involving only a few (hopefully) predictive change-over-time indicators. A rigorous self-appraisal (local) evaluation can then tell the deeper story of taste, color, texture, and consumer satisfaction (detailed information need by the program itself).

Self-appraisal “localized” evaluation . The first goal of outcome evaluation is to immediately feedback useful information to all “local” program stakeholders (categories of participants from funders to clients) in an attempt to both improve and understand the program.  This “action-based” orientation emphasizes both deep self-appraisal and program analysis rather than judgment and comparison with other programs. Because each program largely remains unique in it’s mix of resources, context, and client etiology, this evaluation design must always be specifically tailored for and then interpreted within it’s own context. Whatever the chosen design, to be useful, these protocols should still provide meaningful measurement (validity), be reliably gathered (data collected with a high level of confidence). 

The more rigorous local evaluation designs might include a combination of “change-over-time” methods such as: fourth generation evaluation (Guba & Lincoln, 1988)

Send mail to brboyd80@yahoo.com with questions or comments about this web site.
Copyright © 2005 Na' Nizhoozhi Center Inc.
Last modified: May 24, 2005