What Makes a Publishable Qualitative Report?

James W. Drisko, PhD, Smith College School for Social Work 
      jdrisko@smith.edu or www.drisko.net

SSWR, Atlanta, GA, January 20, 2001

Standards for Evaluating Qualitative Research?

>        Qualitative research is not a single, unified tradition (Riessman, 1994).

>    It includes a wide range of philosophies, research purposes, intended audiences, methods and reporting styles (Denzin & Lincoln, 1994).

>        The range of legitimate forms is both a strength and a source of confusion.

>        Which makes shaping standards difficult (Drisko, 1997; Padgett, 1998).

>        Some authors believe standards are inimical to the discovery focus of qualitative research;  confining where innovation and creativity is needed (Conboy, 1998).

>        Others promote standards for pragmatic reasons, to enhance status, to guide  researchers, to orient teachers and their students, and to promote qualitative research as a whole.

Standards are important, as they:

>        Orient and guide editors and reviewers of manuscripts submitted for publication.

>        Help inform and direct potential funders, consumers and other interested parties as they assess the merit and worth of a qualitative report.

>        Orient students and the lay public about sound qualitative research, and

>        Guide educational efforts.

Standards are only useful, if they are:

>        Widely understood,

>        In considerable depth,

>        By reviewers, editors and funders.

To date, a base of widespread, thoroughly informed and experienced evaluators of qualitative research is only now developing in social work (Padgett, 1998). 

The Near Future - Building Infrastructure

>        Growing awareness of the merits of qualitative research may expand knowledgeable editorial board members (Rubin, 2000).

>        Better trained reviewers should allow qualitative research to be reviewed within its own terms, honoring diverse epistemologies, purposes and methods.

>        But avoiding “ghettoization” remains an issue.

>        Better educated researchers will provide more complete “maps” of what they are have done.

>        Better trained educators should lead to more knowledgeable students (tomorrow’s reviewers, teachers and leaders),

>        But providing opportunities for training in more than one qualitative research method will likely remain a challenge due to time and experience requirements.   

Criteria for Evaluating Qualitative Research   (Drisko, 1997)

 1) Identification of the chosen philosophy/epistemology

–  A clear statement allows the author to orient readers and reviewers and is the basis for establishing internal consistency.

–  However, not all researchers imagine epistemology as a “choice,”

–  But all potential reviewers should be sufficiently familiar with the identified epistemology to be credible reviewers.

 2) Identification of audience and research objectives

–  State for whom the research is intended.

–  Qualitative research may be used to explore, discover, share perspective(s), raise consciousness, evaluate and even test theory.  State the objectives.

–  Some authors suggest qualitative research can be a vehicle for effective social change.

 3) Specification of the study methods

       Sample.   Researchers should clearly specify the nature of the study sample and the rationale for its selection.

–  The obligation to seek out, weigh and report contradictory evidence is often related to sampling strategy.

       Transferability.  (or Generalization) describes the applicability of findings and conclusions derived from one context to a second context (Leininger, 1994).

–  Transferability is often sought, but not always.

–  Overgeneralization from a specific, located sample is frequent in reports.

       Data collection.   Researchers must specify the nature of data collection employed in the study in detail, as fits the study philosophy and purposes.

–  One challenge is to illustrate in detail how codes/concepts were formulated.

–  Another key challenge is to provide sufficient raw data to allow readers to form their own interpretations or to challenge the author’s views.

        Analytic methods.  Researchers should clearly identify their chosen methods of data analysis, consistent with the study philosophy and objectives.

–  These named methods must also be evident in the report, with detail on specific strategies (member checks, triangulation, etc) clearly specified.

–  Inconsistencies should be stated and discussed.

       Interpretive criteria.

    a) Credibility/Believability.  Data and analysis must fully convey what local participants' know or experience within their local context (Leininger, 1994).

     b) Placing meanings in context. (Lincoln & Guba, 1985).

       c) Confirmability.  Reported efforts to corroborate data and to challenge and/or affirm interpretation or theory (Patton, 1989; Reid, 1994; Drisko, 1997).

       d) Completeness or Saturation.   Showing exhaustively knowledge of the experiences or events under study (Patton, 1987); credibly conveying a full view of the experience or events under study (Padgett, 1998), and contradictions.

 4) Identification of biases

–  Identification funding sources, institutional and organizational connections which might influence the study and interpretation, including initial expectations of study results;

–  Demonstrating substantial self-reflection and self-analysis.

–  Member checks, peer reviews helpful.

 5) Explicit maintenance of social work ethics

–  Social work researchers must gain prior informed consent from research participants, and

–  Explicitly report how they have followed, social work ethics in their research activities. 

 In discussion and conclusions:

6) assurance of consistency between conclusions and study philosophy, objectives, and presented data.

–  Conclusions from limited samples or specific settings should not be over-generalized.

–  Available raw data should “make the case” so conclusions are credible, compelling.

Additionally, most qualitative researchers should report and examine potentially disconfirming data in any report.

–   Such efforts build credibility,

–  Often open valuable new avenues for discovery and clarification,

–  Underscore limits to transferability (generalization) and/or applicability of newly developed theory.

• These criteria seek to provide flexibility for qualitative researchers across epistemologies, research purposes and methodological approaches, while

• Requiring demonstration of internal coherence among question, purpose, method and conclusions; and

•  Provision of a clear “map” for the reader to follow the researcher’s decision making and rationale(s).

Carl Brun (1997) and Gerald Mallon (1998) note use of these standards has been helpful.

back to top

back to Drisko's Criteria

Some Published Examples -  
SSWR Best Research of 1998 Nominees

˜     LaSala (1998) studied the effects of parental and in-law disapproval on gay men's relationships.      LaSala, M.  (1998).  Coupled gay men, parents, and in-laws:  Intergenerational disapproval and the need for a thick skin. Families in Society, 79(6), 585-595.

˜     Drisko (1998) evaluated and compared two Family Preservation programs with different models serving similar clients.      Drisko, J. W.  (1998).  Utilization-focused evaluation of two intensive family preservation programs.  Families in Society, 79(1), 62-74.

˜     Lindsley (1998) studied the impact of homelessness and living in shleters on family relationships.      Lindsley, E.  (1998). The impact of homelessness and shelter life on family relationships. Family Relations, 47, 243-252.

 An Analysis of How the Study Met (or did not meet) these Criteria

I.     Study Objective - LaSala’s study

>        LaSala examined “the impacts of parental and in-law disapproval on gay men's relationships.”

>        A primary objective was to “develop new understandings that would begin to describe how gay men's partnerships are impacted by these intergenerational relationships.”   and

Stated Objective and Rationale

>        “Qualitative research methods were chosen for two reasons. As this area was relatively unexplored, quantifiable variables could not yet be known...

>        “qualitative investigation is considered ideal for generating thorough and holistic descriptions of complex processes (Reid, 1993) or patterns, themes, and organizing constructs (Fortune & Reid, 1998). ”

 Study Epistemology

>        Unstated

>        Appears post-positivist or perhaps realist.

>        Methods employed are consistent with this inference.

>        Includes descriptive statistics in considerable detail.

Target Audience

>        Clinical social workers who work with gay men.  (Inferred from Discussion)

>        Implicitly - Academics who will train and sensitize professionals who will work with gay males and their families,

>        their Students, and

>        Gay men, their families and those who work to support them.

Sample  (and Transferability)

>        “Data were collected from an accidental sample of 40 men.”  (Abstract)

>        “A nonprobability convenience sample was gathered. The techniques of snowball sampling (Fortune & Reid, 1998) were used. The data analyzed herein were part of a larger study that examined the interactions between coupled gay men and their parents and inlaws (LaSala, 1997).”

>        Sample “N=20 couples from the Albany, NY area.  37 were white, 2 Latino, and 1  Chinese-American. Most were middle-class. Ages ranged from 23-48 years

>        Couples were together from 10 months to 27 years, mean of 6.89 yrs (sd=4.91 years).

>        All but five men had come out to their parents. The length of time from disclosure to interview ranged from 3 months to 20 years, mean of 9.78 years.”

Data Collection

>        “For the purpose of collecting data, a standardized open-coded interview (Patton 1980) was developed. The data collection plan could also be considered in-depth qualitative interviewing described by Taylor and Bogdan (1984) as in-person encounters between researchers and informants directed toward understanding the respondents' perspectives in their own words.”

>        “The interview included open-ended questions eliciting the men's perceptions of their parents' initial feelings about their homosexuality, their current opinions, their beliefs about how their parents and in-laws felt about their partners and relationships, and how their parents' and in-laws' opinions impacted their relationships.”

>        “Multiple items were used to address the targeted areas of study (Kahn & Cannell, 1957). Indirect questions about sleeping arrangements during parental visits along with more direct queries about the specific impacts of parental opinions were designed to be multiple ways of getting at the same information. Thorough notes were taken...interviews were audiotaped.”

>        “It was anticipated that members of a couple could hold different perceptions of intergenerational impacts on their relationship and that a participant could potentially censor responses in the presence of his partner. Therefore, each respondent was interviewed privately and simultaneously but in separate rooms of each couple's home.”

Bias Recognition and Control

>        “Interviews were conducted by the author and a [graduate] research assistant... and both are openly gay men. Several authors cite the advantages of the researcher and the respondents sharing demographic and social similarities (Kahn & Cannell, 1957; Lofland & Lofland, 1984; Mann, 1970).”

>        “Since we shared the experiences and could speak the language of gay male culture, it was relatively easy for us to gain entry into gay men's lives. Being partnered gay men enabled us to establish the trust and rapport necessary to get respondents to feel comfortable in discussing personal and potentially painful topics.”  (continues)

>         “On the other hand, personal experience can also contribute to bias. We ran the risk of seeing our respondents' experiences exclusively through our own lenses. In order to minimize the potential for subjective bias, the data was reviewed with heterosexual clinical and research colleagues regularly throughout the data collection and analysis processes.”

Data Analysis - Coding

>        “...After reviewing the answers of eight to ten of the interviewees, lower level codes were established. As coding of responses within and across targeted areas continued, it became apparent that several codes could be combined to form axial codes (Glaser, 1978). Toward the end of the coding process, preliminary themes and patterns were identified and categories established.”

Data Analysis - Memos

>        “Several authors emphasize the importance of [memos with citations.]

>        “Memos were written throughout the data collection and analysis process... memos were used to integrate the results of the primary coding in order to develop secondary codes and categories. Memo writing also served to identify findings that emerged out of the axial codes and categories. ”

Data Analysis - Reliability

>        “To check for coding reliability, the research assistant independently coded written and tape recorded data into key categories that emerged from the data analysis. The overall agreement between his codes and the author's was 89%.”

Reporting Results    (3 issues as examples)

>        “Gay men are almost certain to face parental and in-law antipathy regarding their homosexuality and their relationships with their partners…(cites) Virtually all of the respondents in this research experienced a great deal of parental hostility upon coming out as indicated by their statements:

Reporting Results  1 -  The Raw Data

>        “When I was a teenager, I brought a newspaper home that had two men embracing on the cover. My father let me know that I had to move out by the end of the week.”

>        “They were both very upset. She [mother] was pregnant and [resumed] smoking. He was angry and raised his voice, “If this is true we're going to change our name!”

Reporting Results -  Issue 2

>        Independence.  Many of the men were adamant about not letting their parents' feelings affect their partnerships. They made statements that indicated emotionally independent mind-sets which precluded being influenced by parental disapproval:

Reporting Results 2 - The Raw Data

>        “I don't take her input. She knows not to give it.”

>        “I respect her right to feel that way [negatively about interviewee's homosexuality] and to hove her opinion. I recognize that her experiences are different than mine. I can validate her feelings and not buy into them.”     (plus 2 more)

Reporting Results - Issue 3

>        Parents positive. ...While most of the men reported disapproval from parents and in-laws for their sexual orientation, several indicated that they received some support for their relationships. Most of the respondents who perceived their parents to be in some way supportive, readily admitted to the beneficial influences on their unions.

Reporting Results 3 - The Raw Data

>        “It’s just one less thing to worry about.”

>        “[Our relationship] demonstrates commitment and permanence which she [mother] understands. With her support, domes In the relationship are less difficult [He said the same regarding the effect of his partner's parents positive feelings.]”

Discussion/Conclusions

>        “Analysis of the data from this study suggests that coupled gay men use a variety of boundary setting mind-sets and behaviors to cope (lists types).”

>        “Some couples had members who stated they were detrimentally impacted by intergenerational, anti-homosexual sentiment. Often, men in such couples were unable to put their partnerships' needs above those of families of origin.”

>        “While the men were for the most part able to protect their relationships from parental hostility, they were also able to benefit from the support of parents and in-laws when it was offered. This has been found for lesbian couples (Meyer, 1989). Couples were likewise able to turn their parents' negativity into a galvanizing force for their relationships.”

Discussion - Cautions

>        “This sample was male, mostly white, and middle class. It is reasonable to assume cultural and socioeconomic factors play a role in couple and intergenerational dynamics. African Americans and Latinos are underrepresented in the literature describing homosexual relationships. The study of these groups... would be particularly enlightening.”

Discussion - Clinical Implications

>        “A consistent and repetitive finding is that gay men face almost certain hostility when they reveal their sexual orientation to their parents. The men in this study emphasized the importance of maintaining independence or a thick skin as a way to protect themselves from parental hostility.”

>        “In assisting a gay man who is contemplating disclosure of his sexual orientation to his parents, a clinician should assess his ability to cope with a painful family crisis. Therapists should not encourage gay men to come out to their parents unless they are fully prepared for adverse reactions.”

>        “Gay male couples are not necessarily adversely affected by parental and in-law disapproval.”

>        “In this study, the small number of couples who were negatively affected by their parents and partner's parents' attitudes suggests that these men may be less likely to prioritize the needs of their partnerships over those of their families of origin.”

A Review Applying Drisko’s (1997) Standards:

>        A strong study - reliability discussed,

>        “Traditional” epistemologically, which makes evaluation easier,

>        Researcher “located,” biases stated,

>        Methods thoroughly documented, though use of several different named methods (with different details) may create the appearance of some internal inconsistency,

>        Readers are very well “led” through the research methods.

>        Internally consistent,

>        Impact of diversity explicitly noted.

>        Findings not overgeneralized (though one statement might be close),

>        No mention of informed consent,

>        Too little raw data offered to allow a reader to draw a different conclusion(s)

          (a common problem of article length qualitative reports),

>        Lacks disconfirming data analysis (but positives of a bad situation are noted),

Applying Padgett’s (1998) Methodological Approaches to Rigor:

>        Single interviews (limited engagement),

>        No triangulation of data (but first person reports are pivotal to study objectives).

>        No formal peer debriefing (though discussion between the researcher and interviewer will provide some debriefing),

>        No member checks,

>        No formal audit trail process noted, but the report does map out how the study was undertaken in considerable detail.

(end of assessment of LaSala study)

back to top

back to Drisko's Criteria

back to LaSala study assessment

An Analysis of How the Study Met (or did not meet) these Criteria  -- Exemplar Two
 

II.    Study Objective - Drisko’s study

>        Drisko undertook a “comparative evaluation of The Brightside's Intensive Family Intervention [IFI] Program and the Massachusetts Department of Social Services' Family Life Center [FLC], which both draw from the same referral pool.  Qualitative interviews with IFI and FLC client families provide an understanding of the programs and their effectiveness from the consumers' point of view.”

Stated Objective

>        Primary objective:  “The views of these consumer families  [will] provide qualitative, ‘utilization-focused’ information to program staff and administrators.”

>        “Clients' views  were [sought and] coded [involving and empowering child-abusing parents - one of the remaining villified populations].”

>        “In addition, quantitative outcome data document the effectiveness of each program at service termination, after six-months and after one-year. Findings demonstrate excellent success overall by both programs [which may make the qualitative findings more valuable and worth considering to some of the intended audiences].”

Rationale

>        “while programs espouse orientation by particular theoretical models,       there is little research to document these theories actually guide program implementation.  Littell (1994) states different approaches need exploration       and comparison.”

>        “Further, it is unclear if the program results are consistent with theoretical predications of expected change.  Several authors (Bath & Haapala, 1994; Littell, 1994; Pecora, 1993; Wells & Biegel, 1990) identify a need for investigations examining similar programs offered in different contexts, such as public versus private agency contexts.  This evaluation addresses these ends:  It compares two programs with well defined and distinct models. “

>        “How these questions are asked and analyzed will shape the nature and utility of the answers obtained.  Most evaluations of intensive family preservation programs focus on before and after measures of outcome, ...targeting behavioral goals in summary fashion.  Such evaluations pay little attention to the details of client's perceptions of the program process and how it fit, or did not fit, with their needs, personal or ethnic styles, and interests (Wells & Freer, 1994).  For example, an evaluation of a Homebuilders' program by Kinney and colleagues (1990) included questions such as ‘Was Homebuilders helpful or not helpful to your family?’ (p. 57) and ‘Do you think your therapist really cared about your family?’ (p. 58).  The first question was answered on a 1 to 5 point Likert-type scale.  The second question was answered categorically:  "Yes", "No", and "Not Sure.”    This qualitative, utilization-focused, evaluation starts with clients' views in their own voices, rather than forcing clients to choose among someone else's categories.” 

>        “To whom we ask these questions further shapes the nature of the answers we obtain. Rapp, Kisthardt, Gowdy, and Hanson (1994) note that "current research not only does not support client empowerment but actually does injustice to it" (p. 381)…However, as most intensive family preservation programs seek to empower clients to be more active agents in their own lives, it is consistent to seek and use their views in that they are consumers of services. Optimizing programs to consider clients' worries, needs, and viewpoints can enhance program processes and effectiveness. It can simultaneously be empowering for the participating clients.”

Study Epistemology

>        “This project employs a utilization-focused approach  (Patton, 1986, 1987).  This

epistemologically pragmatic method aims at maximizing the utility of findings to an audience of program designers, staff and administrators.  Quantitative and/or 

qualitative components may be included in a utilization-focused evaluation so as  

to best obtain information needed by the targets audiences.”

Target Audience

>        “The views of these consumer families  [will] provide qualitative, "utilization-focused" information to program staff and administrators.

>        Funders.

>        Implicitly - Academics who will train and sensitize professionals and their students.

Sample  (and Transferability)

>        “The study draws from a theoretical sample (Strauss, 1987, Strauss & Corbin, 1990). Cases were selected to illustrate the diverse service needs, referral issues, and secondary issues.

>        “All 47 families interviewed for this study were referred by the Massachusetts Department of Social Services (DSS) area offices serving the Greater Springfield/ Holyoke, Massachusetts, metropolitan area…All 47 families in the study had been reported for substantiated child abuse or neglect. Presenting problems leading to referral included child abuse or neglect, child behavior problems, parental substance abuse, domestic violence, family transition problems, and mental or physical illness. More than 50% of referred families had a substance abuse history, in most cases with cocaine. Overall, approximately 38% of the families were headed by single parents (32% for IFI and 52% for FLC). Staff impressions and psychological testing of some FLC parents indicated that up to one third of the parents appeared to be intellectually challenged, mainly with borderline intellectual functioning (WAIS IQ 70 - 85) that could be either organic or secondary to environmental limitations. A few parents were possibly mildly retarded (IQ 55 - 70).”

>        “Although all parents had to give informed consent to participate, those who had doubts and concerns about the programs were actively sought out to ensure a broad base for sound evaluation.”

Data Collection

>        “In this project, the voices of consumers were emphasized, which was consistent with program goals and evaluation objectives.”

>        “Staff initially asked the parents (or parent) of each client family if they would consider fully voluntary participation in the study with assurances that participation would be confidential and have no impact on future services. If permission was given; a date was set as close to the sixth week of FLC participation or eighth week of IFI participation as possible. This allowed ‑time for families to get to know the program, staff, and services, yet avoided evaluation while termination was under way, potentially evoking feelings of loss. Written consent was obtained by the interviewer.  Interviews were audiotaped.”

>        “The interview was semistructured in order to help the participant examine his or her experiences in the program. Broad open‑ended questions (Patton, 1987, 1996; Strauss, 1987) generated a wide range of responses on these topics.”

Bias Recognition and Control

>        “[Clients] who had doubts and concerns about the programs were actively sought out to ensure a broad base for sound evaluation.”

Data Analysis

>        “From the interview material, the evaluator and a second social worker with child welfare expertise coded comments reflecting themes of importance to the client. These themes were repeated, although in somewhat different form, by other clients.  Initially, the transcribed material was heavily coded in an effort to find the structure of meaning and importance placed on the material by the client (Miles & Huberman, 1984; Strauss & Corbin, 1990).” 

>        “The coding procedure was then refined and reconfigured to highlight topics particularly important to the clients. In this way, the clients' views of each program were pieced together to form a working view of each program that is close to the clients' experiences and yet focused enough to be of use in reviewing the program and suggesting modifications to it (Patton, 1980, 1987, 1996)...The relevance, urgency, and practical utility of the consumers' comments, together with ensuring a high level of credibility and authenticity, were of key interest in the coding process (Drisko, 1997).”

Reporting Results    (3 issues as examples)

>        Referral.  “Neither IFI nor FLC families experienced a crisis prior to referral to either program. Instead, their family circumstances appeared to be chronic: "Things weren't good, but they weren't much different than usual either." "No, it was things as usual when we began." Some FLC families appeared to be in "casework crises," whereby changes in worker's views or actions evoked a crisis. Unlike models that use the disequilibrating impact of crises to facilitate change, these programs engaged families that had not experienced crisis. Few families expressed a clear interest in working with either program as a sign of motivation prior to referral, as is typical of Homebuilders models (Kinney et al., 1990).

Reporting Results -  Issue 2

>        Staff.  “Parents were initially suspicious of staff from either program. They feared these newcomers would be inconsistent and unreliable and that their experiences would be similar to those they have had in the past with other child protective staff. One mother noted that workers “ knocked on my door every day­ even weekends ‑ for three weeks ....They left notes and called ... saying just when they were coming. When they did just what they said ... I said, 'What the heck' and let them in. They treated me with respect.

Families quickly determined that IFI and FLC staff were more reliable and willing to understand their views and needs than were workers with whom they had had earlier experiences. Most parents noted that a positive aspect of IFI and FLC staff was that they "did not tell [us] what to do" but worked conjointly with the parents.”

Reporting Results - Issue 3

>        Single worker vs. Two person team.  “In general, the IFI families state that the team model

was very useful and ensured consistent staff availability. The two person team also allowed parents and family members to make the most use of the IFI staff person with whom they felt most comfortable. "I kinda warmed up to her because she talked my language better. The two person model also allowed one staff member to baby‑sit while freeing up the other staff member to accompany parents at appointments or interviews: "One of them held the fort while one went out with me." "It was great one could stay with the babies while we went out to the school meeting." Participants did not indicate problems in communication and consensus between team members, though most parents preferred the more senior MSW workers... Issues of gender or racial differences or preferences were not noted by IFI families.  Still,

>        “Most FLC families felt that the single‑worker model was fine, given the qualities demonstrated by their worker: "He's great. I couldn't ask for better." Some families, however, noted that the single‑worker model offered no room for choice if "personal chemistry" was not optimal or for input from workers of different sexes: "I would have liked to talk to a woman about some of this."

Discussion/Conclusions

>        “Both the FLC and the IFI are highly effective programs. Program staffing, contact with clients, and concrete services were portrayed by parents in the same way as they were described by program designers. At the same time, both programs are highly individualized and tailor service delivery to the needs of a given family. However, the consistent use of specific family‑systems theories to shape interventions was not evident from the statements of these client parents, nor did evidence indicate that crises "opened" families to change.”

>        “Agency auspices. Parents' comments indicated that agency auspices were typically not well understood and were secondary to staff actions. Agency auspices mattered less than did staff caring and reliability as relationships developed.”

>        “Program philosophies and theory base.  Neither program appeared to be tightly driven or constrained by a single philosophy of intervention or theory. ”

>        “Team versus single-worker approach. The IFI families indicated that the team model increased staff availability and built a sense of support. Teams also allowed families to choose to work with workers they felt more in tune with. This included gender or racial match preferences: "He knows this community, even my neighborhood."

>        “Longer interventions. Most participants stated that the 10 to 12 week IFI model was "just right," whereas half the families viewed the 6 to 8 week FLC model as "too short." Given the lack of a crisis to open families to interventions and the common detrimental effects of child-protective processes, clients need a longer duration model to prove staff members' caring and credibility.”

>        “Staff from both programs reported that many (perhaps 25 %) of these parents had, or appeared to have, intellectual deficits, which may hamper their efforts to organize family life and appointments independently. Family preservation programs should have access to a skilled dual diagnosis (mental retardation/mental health) clinician for specific compensatory planning around such client needs.”

Discussion - Cautions/Limitations

>        “Although the result of this utilization‑focused comparative evaluation may not be readily generalizable to different settings and program types, the concerns mentioned by these families suggest important content for inclusion in future program planning as well as future studies of family preservation services. This is a key strength of qualitative evaluation methods.”

>        “Including the voices and views of consumers in program evaluations is also consistent with the empowerment goals of the family preservation movement.”

Feedback to the Author from Manuscript Reviewer

>        An important question, very well conceptualized

>        Very solid on qualitative research: “The author really  knows qualitative research.”

>        “Traditional” epistemologically, which makes evaluation easier,

>        Methods thoroughly documented, though sample selection very brief (and left out efforts to find “unhappy families”).

>        Raw data offered only as illustrations (an artifact of page length - this was not so in the 115 page document to the programs!!)

>        Sample and its diversity on several dimensions clearly stated.

>        Readers are very well “led” through the different program types, research methods and key findings.

>        Findings limited to these programs, with enough information to draw parallels to populations served by other programs.

Drisko’s additional comments:

>        Researcher minimally “located,”  (research was funded by Brightside who ran IFI.)

>        No mention of informed consent (though it was very carefully done).

>        Lacks of discussion of efforts to seek out unhappy families undermines credibility of a theoretciual sampling plan and minimizes disconfirming data analysis (also page length artifacts).

Applying Padgett’s Methodological Approaches to Rigor:

>        Single interviews (limited engagement), though lengthy involvement with data and analysis,

>        Triangulation of data (qualitative with quantitative outcome data, client parents and staff), views of parents pivotal to the evaluation.

>    No mention of peer debriefing (though discussion among the researcher, the interviews, program administration and line staff all offered perspective, serving as an peer debriefers/bias identifiers.)

>    Coding was checked for reliability by external raters (= a check on reliability).

>    Few member checks,

>    No formal audit trail process noted (though it existed!), but the report does map out how the study was undertaken in considerable detail, which is evident as well in the data analysis sections.

(While no formal audit trail was discussed in the longer evaluation report, more detail on links among raw data, open, axial and some selective codes, and researcher memos was offered and discussed).

(end evaluation of Drisko study)

 

back to top

back to Drisko's Criteria

back to LaSala study assessment

back to Drisko study assessment

REFERENCES

            Altheide, D., & Johnson, J. (1994). Criteria for assessing the interpretive validity in qualitative research.  In N. Denzin & Y. Lincoln (Eds.), Handbook of qualitative research (pp. 485-499). Thousand Oaks, CA:  Sage.

            Anastas, J. (2000). Research design for social work and the human services.  (2^nd ed.). New York, NY: Columbia University Press.

            Beamon, S.  (1995).  Maximizing credibility and accountability in qualitative data collection and data analysis:  A social work research case example.  Journal of Sociology and Social Welfare, 22(4), 99-114.

            Bernstein, S., & Epstein, I.  (1994).  Grounded theory meets the reflective practitioner:  Integrating qualitative and quantitative methods in administrative practice.  In E. Sherman & W. Reid. (Eds.), Qualitative research in social work (pp. 423-434). New York, NY:  Columbia.

            Brun, C. (1997). The process and implications of doing qualitative research: An analysis of 54 doctoral dissertations. Journal of Sociology and Social Welfare, 24(4), 95-112.

            Burnette, D.  (Ed.). (1998).  Teaching qualitative research:  A compendium of model syllabi.  Alexandria, VA:  Council on Social Work Education.

             Conboy, A.  (1998).  Personal communications and a series of email communications regarding the use of evaluative criteria by a class of doctoral social work students at Yeshiva University’s Wurzweiler School of Social Work.

             Creswell, J.  (1998).  Qualitative inquiry and research design:  Choosing among five traditions.  Thousand Oaks:  Sage.

            Denzin, N., & Lincoln, Y. (1994). Introduction: Entering the field of qualitative research. In N. Denzin & Y. Lincoln (Eds.) Handbook of qualitative research (pp. 1-17). Thousand Oaks, CA: Sage.

            Drisko, J.  (1997).  Strengthening qualitative studies and reports: Standards to enhance academic integrity.  Journal of Social Work Education, 33, 185-197.

            Drisko, J.  (1998).  Clients strengthen programs:  A utilization-focused evaluation of two intensive family preservation programs.  Families in Society:  The Journal of Contemporary Human Services, 79(1), 62-74.

             Drisko, J.  (2000).  Qualitative data analysis:  It’s not just anything goes.  Juried workshop presented at the Society for Social Work and Research, Fourth Annual Conference in Charleston, SC (January 31, 2000).   (www.drisko.net)

            Glaser, B. (1992). Basics of grounded theory analysis.  Mill Valley, CA: Sociology Press.

            Glaser, B., & Strauss, A.  The discovery of grounded theory.  Chicago:  Aldine.

            Leininger, M. (1994). Evaluation criteria and critique of qualitative studies. In J. Morse (Ed.), Critical issues in qualitative research methods (pp. 95-115).  Newbury Park, CA: Sage.

            Lincoln, Y., & Guba, E. (1985). Naturalistic inquiry.  Beverly Hills, CA: Sage.

            Mallon, G.  (1998).  We don't exactly get the welcome wagon: The experiences of gay and lesbian adolescents  in child welfare systems.  New York:  Columbia

            Morse, J. (1991). On the evaluation of qualitative proposals. Qualitative Health Research, 1(2), 147-151.

            Padgett, D.  (1999).  Rigor in qualitative research.  Juried paper in symposium with J. Drisko, J. Anastas and D. Shelby, workshop presented at the Society for Social Work and Research, Third Annual Conference in Austin, TX (January, 1999).  

            Padgett, D.  (1998).  Qualitative methods ins social work research.  Thousand Oaks, CA:  Sage. 

            Patton, M. Q. (1986). Utilization-focused evaluation. Newbury Park, CA: Sage.

            Patton, M. Q. (1987). Using qualitative methods in evaluation. Newbury Park, CA: Sage.

            Reid, W. (1994). Reframing the epistemological debate. In E. Sherman & W. Reid. (Eds.), Qualitative research in social work (pp. 464-481). New York, NY: Columbia.

            Rubin, A.  (2000).  Standards for rigor in qualitative inquiry. Research on Social Work Practice, 10, 173-178.

            Rodwell, M.  (1998).  Social work constructivist research.  New York:  Garland Publishing. 

 WEB RESOURCES

             Chenial, R.  The qualitative inquiry project.  Retrieved January 14, 2001 from the World Wide Web:  http://www.nova.edu/ssss/qualinq.html

            Drisko, J.  (1997-2000).  Qualitative pages - social work resources.  Retrieved January 17, 2001 from the World Wide Web: http://sophia.smith.edu/~jdrisko/qualres.htm    or    www.drisko.net

            Duffy, M.  Critique of Qualitative Research Studies  (actually a short bib).  Retrieved January 17, 2001 from the World Wide Web: http://members.tripod.com/Mary_Duffy/BIBcritique.html

            Fine Foundation.  Analysis Criteria for Qualitative Studies.  Retrieved January 17, 2001 from the World Wide Web:     http://www.iptv.org/FINELINK/publications/criteria.html#qual    

            Heath, A.  (undated).  The proposal in qualitative research. Retrieved January 17, 2001 from the World Wide Web:    http://www.nova.edu/~ron/heath.html 

            Rice-Lively, M. L.  (1995).  Research Proposal Evaluation Form.   Retrieved January 16, 2001 from the World Wide Web:     http://www.gslis.utexas.edu/~marylynn/qreval.html
 

back to top

back to Drisko's Criteria

back to LaSala study assessment

back to Drisko study assessment

     back to Social Work Resources Home

presented 1/20/01, web version 1/23/01