Cultural Reading

An Introduction to the Cultural Reading of The Program Evaluation Standards (2nd edition)

Welcome to the American Evaluation Association (AEA) site of the Final Report of A Cultural Reading of The Program Evaluation Standards (2nd edition). This is a report of the Diversity Committee of the American Evaluation Association that was approved by the AEA Board, November 2004. This site contains the primary source materials, intermediate working documents and final reports of the Diversity Committee’s Cultural Reading Task Force as well as an opening orientation to our work and its many participants.

Overview The Cultural Reading began as text conversations among a group of professional evaluators who share expertise in and concern for issues of cultural diversity and cultural context in evaluation. The purpose of the cultural reading was to review The Program Evaluation Standards (2nd edition) with respect to coverage of cultural diversity, treatment of cultural concerns, and attention to cultural competence. This conversation was intended to build knowledge among Diversity Committee members and other relevant stakeholders within the AEA; to identify specific passages that are in need of revision; and to gather ideas on relevant resources and suggested changes for input to The Joint Committee on Standards for Educational Evaluation. Throughout this reflection, culture is broadly defined, inclusive of race, ethnicity, gender, age, sexual orientation, social class, disability, language, and educational level or disciplinary background. It includes both individual characteristics and those of a group or collective (e.g., community or organizational culture).

Background At a meeting of the Diversity Committee convened by Chair Edith Thomas at the 2002 Annual Meeting of the AEA, discussion turned to agenda items for 2003. It was noted that The Program Evaluation Standards (2nd edition) would be up for consideration by the Joint Committee in fall 2003, and Karen Kirkhart expressed the strong opinion that these standards needed to be revised to update and improve their attention to dimensions of cultural diversity. This stimulated interest among members present. Kirkhart proposed that the Diversity Committee do a group reading of the Standards and record their critical reflections, exchange ideas, etc. so that the Committee would be in a more informed position to offer comment, reflect interpretations from different perspectives, and impact the process of Standards revision through established channels, via the AEA representative to the Joint Committee, Dianna Newman. The cultural reading became a 2003 action item for the Diversity Committee, chaired by Satish Verma. Karen Kirkhart volunteered to chair the work group. The Board approved this action item at its February 2003 as an element of the Diversity Committee’s 2003 Action Plan.

Participants Initial readers were members of the Diversity Committee in 2002 (when the idea was initiated) and 2003 (when Draft 1 was completed). Charles Thomas joined the reading group in Spring 2003 to provide continuity with the Building Diversity Initiative’s Task Force on Guiding Principles for Evaluators Working Across Cultures, which he chaired. In alphabetical order, the initial readers were:

Claude F. Bennett, U.S. Department of Agriculture

Denice A. Cassaro, Cornell University

Melvin E. Hall, Northern Arizona University

Stafford Hood, Arizona State University

Lester Horvath, Evaluation Consultant, Connecticut

Elmima C. Johnson, National Science Foundation

Karen E. Kirkhart, Syracuse University

Donna M. Mertens, Gallaudet University

Sharon Rallis, University of Connecticut

Edith P. Thomas, U.S. Department of Agriculture

Charles L. Thomas, George Mason University

Satish Verma, Louisiana State University

Elizabeth Whitmore, Carleton University 

 

The work of the 2003 Task Force was supported by:

Crystal Collette, Syracuse University

David Schlesselman, Syracuse University 

 

In 2004, the Diversity Committee, chaired by Melvin Hall, appointed a Task Force to synthesize the text generated by the initial readers and to prepare comments in formats suitable for dissemination to the Joint Committee and other relevant audiences. The members of the 2004 Task Force were (in alphabetical order):

Denice A. Cassaro, Cornell University

Cindy A. Crusto, Yale University

Melvin E. Hall, Northern Arizona University

Elmima C. Johnson, National Science Foundation

Karen E. Kirkhart, Syracuse University

Joan LaFrance, Mekinak Consulting

Donna M. Mertens, Gallaudet University

Craig W. Russon, Kellogg Foundation

 

Procedure The first round of the Cultural Reading began in January 2003 and concluded in August 2003, yielding roughly 100 single-spaced pages of text material, summarizing and critiquing the Standards. Members read the standards together, posting their comments to the entire group via email, then conversing and exchanging ideas around the postings. Readers created their own “archives” by saving the emails to a folder. Each standard was read in order of presentation in the book. Not everyone logged in comments on every standard, and some readers were more active than others in the process. Karen’s Spring 2003 Graduate Assistant, David Schlesselman pulled together the comments at the end of each of the four sections—Utility, Feasibility, Accuracy and Propriety—as well as an integrated document at the end. This became Draft 1. In Fall 2003, Crystal Collette, then a Graduate Assistant at Syracuse University, searched the literature and compiled a bibliography of culturally relevant references by standard. This bibliography was provided to the Joint Committee as a freestanding document in 2004 and was subsequently incorporated in the second draft of the Cultural Reading.  

In 2004, moving the cultural reading from a private to a public document remained an action item of the Diversity Committee. The second round of the Cultural Reading began in January 2004 and concluded in August 2004, yielding a set of prioritized action items and summary matrixes organized by individual standard as well as by category (Utility, Feasibility, Propriety, Accuracy), a narrative summary, and an edited version of the full text. The content and format of these products are summarized below. 

Cultural Reading Documents Click on the links below to view the documents in PDF format.  

To download a free PDF reader go to: http://www.adobe.com/products/acrobat/readstep2.html

  • Executive Summary: Narrative summary, including overview of recommended corrective actions.
  • Priority Recommendations Important corrective actions summarized for all standards, matrix format.
  • Standards Overview by Categories Summarizes the relevance to cultural competence, current strengths, concerns or limitations, and recommendations for standards by major category—Utility, Feasibility, Propriety and Accuracy, matrix format.
  • Summary of Utility Standards: Summarizes the relevance to cultural competence, current strengths, concerns or limitations, and recommendations for each of the Utility standards, matrix format.
  • Summary of Feasibility Standards: Summarizes the relevance to cultural competence, current strengths, concerns or limitations, and recommendations for each of the Feasibility standards, matrix format.
  • Summary of Accuracy Standards: Summarizes the relevance to cultural competence, current strengths, concerns or limitations, and recommendations for each of the Accuracy standards, matrix format.
  • A Cultural Reading of the Program Evaluation Standards  (2nd ed.): Narrative discussion among readers of each of the thirty standards, inclusive of Overview, Guidelines and Common Errors, Case Illustrations and Analyses, and suggestions for Supporting Documentation. (This is a large document which could take a while to download depending on your connection.)

Current Status These documents were approved by the Diversity Committee on September 28, 2004 and sent to the AEA Board of Directors for review and approval. The Board made minor revisions to the Executive Summary to improve clarity and approved the revised documents as the Final Report of the Cultural Reading Task Force on November 3, 2004. These documents are available to support the work of AEA members, other AEA Committees, and the Task Force of the Joint Committee on Standards for Educational Evaluation charged with revising the second edition of The Program Evaluation Standards (Donald Yarbrough, Chair). The documents may also be useful to other persons or organizations that have an interest in educational evaluation. Elmima Johnson, a participant in both phases of the cultural reading, assumed the position of AEA representative to the Joint Committee in January 2005. The Diversity Committee will seek continued opportunity for input as the Joint Committee Task Force moves through the revision process.

We welcome dialogue and discussion. Comments may be addressed to:

Search