Prove It!

Prove It! Guidelines, Measures and Cautions
about Practice Outcome Evaluation

James W. Drisko, PhD, LICSW
Smith College School for Social Work
Northampton, Massachusetts

from the Massachusetts NASW Symposium '98

What You Need to "Prove It!

A comprehensive, relevant and useful understanding of what you wish to change
Measures to operationalize your understanding fully:

valid to what and whom and where you wish to make changes (sensitivity to issues of race, ethnicity, gender, class, literacy, etc.)

reliable for all the intended population and over time

sensitive enough to capture the intended changes in less than the planned time frame

allowing for progress, no change and regression

reflecting the different ratings of client, family, significant others, the larger social world (employers, teachers, support systems) and clinical and non-clinical staff

ratings by 1) self report, 2) by others in a position to know, 3) direct observation, 4) indirect naturalistic or documentary sources (arrest records, child abuse reports) -- as is feasible

at as many points in time as necessary to allow an appraisal of the extent and duration of change (and allowing a sense of the course of change as a bonus!)

3. A representative sample of the population you wish to
       study --

        including those "outliers" who fit the criteria but add complexity
        (and make the evaluation more useful)

4. Ethical access to the sample -- with fully voluntary,
       informed consent:

       which often alters the sample

       which should include information about what's being done for
       evaluation purposes which might not be needed if there was not
       evaluation

5. A clear, detailed understanding of the "treatment(s)" or
      "intervention(s)"

      probably the single most common problem with clinical and
      program evaluations!

      which ideally grapples with how your treatment includes factors
      made more central by other theories (the relationship, reinforcement
      contingencies, use of environmental supports)

6. A study design which allows inference about causation

      a true experimental design {ideally a Solomon four group or
      factorial design} OR a planned series of single case designs involving
      alteration of key criteria to allow inference of causation via replication
      logic

7. Ongoing openness to changes in the client, intervention(s),
       and context which undermine inferences about causation

      threats to the validity of the evaluation as a whole such as alterations
      in the treatment over time, reactivity to measures, differences in drop
      out rates, changes in environmental supports which enhance or
      impede the treatment)

Issues:

Changes in the funding of social work services increasingly require research-based evidence for the effectiveness of services (Corcoran & Vandiver, 1996). Some Massachusetts state agencies now require formal evaluation in contract requirements. While practice evaluation is hardly new-and appears acceptable to many clients (Campbell, 1988)--both its form and importance require attention to this issue in its contemporary form. Sound evaluation is needed to ensure social workers are able to met current economic and political challenges, maintain practice and program excellence, strengthen services to both current and new
populations and understand the assets and liabilities of other's views of sound, ethical evaluation (Jordan & Franklin, 1995; Roth & Fonagy, 1996).

READING LIST

Anastas, J. W. (2000). Research design for social work and the human services. (2nd ed.). New York: Columbia University Press. (Great introduction!)

Anastas, J. W., & MacDonald, M. L. (1994). Research design for social work and the human services. New York, NY: Lexington Books.

Anderson, M. L. (1993). Studying across difference: Race, class, and gender in qualitative research. In J. H. Stanfield & R. M. Dennis (Eds.), Race and ethnicity in research methods (pp. 39-52). Newbury Park, CA: Sage.

Bergin, A., & Garfield, S. (1994) (4th ed.) Handbook of psychotherapy and behavior change. New York: Wiley. (The best information there is in summary form -- includes a wide range of treatments but almost not social work studies)

Campbell, J. (1988). Client acceptance of single-system evaluation procedures. Social Work Research and Abstracts, 24(2), 21-22. (Pretty well accepted by this sample)

Corcoran, K., & Fischer, J. (1994). Measures for clinical practice: A sourcebook (2nd ed.). New York: The Free Press. [Vol. 1: Couples, families and children; Vol. 2: Adults.] (Includes full versions of many instruments -- but hardly all in common use. Remember copyright laws!)

Corcoran, K., & Vandiver V. (1996). Maneuvering the maze of managed care: Skills for mental health practitioners. New York: Free Press. (A great overview and dictionary, too)

Hudson, W. W. (1982). The clinical measurement package: A field manual. Homewood, IL: The Dorsey Press. (Not widely used, but developed by a social worker)

Jones, Reginald L. (1996). Handbook of test and measurements for black populations. (2 vol.) Hampton, VA: Cobb & Henry. (The only handbook for this population I am aware of)

Jordan, C., & Franklin, C. (1995). Clinical assessment for social workers:
Quantitative and qualitative methods. Chicago: Lyceum. (Good balanced view of merits and limits of both forms of research)

Lambert, M. J. (Ed.). (1982). The effects of psychotherapy (Vol. 2). New York: The Human Sciences Press. (A classic)

Lyerly, S. B. (1978). Handbook of psychiatric rating scales (2d ed.). Rockvile, MD: National Institute of Mental Health. Washington, DC: U. S. Govt. Printing Office. DHEW Publication No. (ADM)78-775.

Levitt, J., & Reid, W. (1981). Rapid assessment instruments for practice. Social Work Research and Abstracts, 17(1), 13-19. (dated, but good overview)

NASW Code of Ethics. (1996). Washington, DC: Author.

Nelson, J. (1985). Verifying the independent variable in single-subject research. Social Work Research and Abstracts, 10, 3-8.

Mullen, E., & Magnabosco, J. (Eds.) (1997). Outcomes measures in the human services. Washington, DC: NASW Press. (Conceptual, not a measures source)

Roth, A., & Fonagy, P. (1996). What works for whom? A critical review of psychotherapy research. New York: Guilford. (A good summary of the
literature, though, as always, lots omitted)

Sue, S., Zane, N., & Young, K. (1994). Research on psychotherapy with
culturally diverse populations. In A. Bergin & S. Garfield, Handbook of
psychotherapy and behavior change, pp. 783-817. New York: Wiley.

Try getting any of these through your local library by interlibrary loan. A few are references books everywhere, but others may be available to you locally.

to Social Work Resources Home Page

3/98, last update 1/20/00