Date: Friday, June 29, 2018
"It takes a lot of people to make a winning team. Everybody’s contribution is important” – Gary David Goldberg
Earlier this month, I had the opportunity to attend my first official AEA event as Executive Director, the 2018 Summer Institute! Over 600 attendees convened in Atlanta, GA, to learn about social impact measurement, fear factors in evaluation, appreciative inquiry and many other topics that support the professional development of evaluators.
In addition to learning more about how evaluators think, I took exceptional note of what is important to AEA members. I had many enlightening conversations with members that gave me a good sense of why people join AEA, what they expect from the association and how they want an organization that stays true to its (and their) values. I love that members are very vocal about what they want from the organization, which tells me they are vested in the ownership of AEA.
Professional development and networking are a key reason many people join associations. AEA members seem especially focused on adherence to the organization’s values and guiding principles. This is what makes me proud to be the Executive Director of AEA. Focus on values and principles are key for membership retention, which in turn, supports organizational growth.
AEA’s staff puts the organization’s values at the forefront of everything we do. Whether it is considering the right type of keynote speaker for an event, soliciting exhibitors for the conference or accepting corporate sponsorship, we always determine the fit for the organization’s values.
Over the next few months, the AEA staff will be busy preparing for Evaluation 2018, as well as implementing programs and initiatives. We are forming some new working groups (Guiding Principles and Membership), evaluating current programs (GEDI and MSI) and increasing our professional development offerings (Coffee Breaks and eStudies). We are starting the initial planning of a new website (redesign, content and navigation) to begin implementation in the near future, but in the meantime, members can look forward to the reorganization of the current AEA website content to ease navigation.
Earlier this year I talked about “casting a wider net” to ensure organizational growth. When speaking with members at the Summer Institute, I was most impressed with how many members approached me with ideas for potential partnerships with like-minded organizations and issues within the field of evaluation that they would like to see AEA address. Many of these suggestions resonated with topics shared on EvalTalk, so I want members to know that I value all voices, and am focused on initiatives and outreach that will continue to grow the organization, support the inclusiveness of diverse opinions and support diverse populations, all within the field of evaluation.
Walking the Talk features reflections from AEA members on what the association’s values mean to them and how these values have guided and impacted their work. This month, hear from Melvin Hall, PhD, Professor in the Department of Educational Psychology at Northern Arizona University.
Throughout 2017, AEA, with support from the Kellogg Foundation, hosted a series of regional Dialogues on Race and Class in America. The series of three events culminated in a plenary session at the 2017 Annual Meeting. I was pleased to serve as the moderator for this series, during which evaluators and professionals from other fields discussed the significance of race and class as contextual variables in the U.S. The goal of the dialogues was to illustrate that race and class need not be a part of the evaluation’s scope of work, to become significant factors in how the evaluation unfolds.
Two very significant comments stood out for me among the many valuable things the dialogues revealed. The first was an urgent plea from outside the evaluation community, for evaluators to “take up more space” in the corridors of power, where important policy decisions are too frequently made without the benefit of evidence collected in a systematic and purposeful way.
The second comment was a concern that the training evaluators receive may not provide the skillset necessary to efficaciously handle the thorny issues of race and class when they emerge through an evaluation process. Both comments supported the overall question posed by the dialogues: What must an evaluator know, to effectively discharge their duties in a society where race and class continue to be powerful forces in the lived experiences of the entire population?
Evaluators are best known for their techniques of gathering, analyzing and reporting evidence. But what are we to do when our techniques cannot screen out critical issues embedded in the context of our work? I believe that following the values espoused by AEA requires that we be vigilant in reflecting upon the critical contextual factors that we may overlook without purposeful attention to the assumptions underlying the programs we evaluate, and the techniques we employ in our evaluations.
In my view, this is particularly true when programs under review claim to produce societal benefits. Many programs that purport to provide enhancements to opportunity and quality of life do so with differential impacts along gender, ethnic, class, ability, or race demographics. Without evaluations that are sensitive to these issues, evaluators may inadvertently support or commend such programs for continuation or even expansion.
Like the field of measurement, in evaluation, the term validity refers to the appropriateness of the inferences drawn from the work involved. Therefore, valid evaluation inferences are not the result of the techniques utilized, but rather the appropriateness of the application and inferences coming out of the evaluation. In both the AEA Cultural Competence Statement and the Guiding Principles for Evaluators, attaining this validity for inferences requires evaluators to actively take a stance toward their work that enhances the attention given to context.
As evaluators, we must not fall into the habit of taking validity for granted, or assuming a method or technique, when well applied, will inoculate our findings from personal or institutionalized bias. Ethical service requires that the evaluator remain personally alert and vigilant, and this is only possible through self-education and reflection. With the development of more tools and heuristics to assist with grounding evaluations in cultural context, all evaluators must seek out the best means of ensuring that their work is of the highest quality and meet ethical standards of the profession.
Melvin Hall, PhD, is a Professor in the Department of Educational Psychology at Northern Arizona University, Affiliated Researcher of the Center for Culturally Responsive Evaluation and Assessment at the University of Illinois, Senior Scholar at the Association of American Colleges and Universities, and distinguished scholar of the Marie Fielder Institute at Fielding Graduate University. He is a recent member of the AEA Board of Directors and Co-Chair of the AEA Task Force on Member Engagement, Diversity, and Leadership Development.
From Beth Michel, MPH, and Nicole Bowman, PhD
The Indigenous Peoples in Evaluation Topical Interest Group (IPE TIG) was formed in the early 2000’s by Dr. Fiona Cram, Dr. Joan LaFrance, Dr. Katherine Tibbetts and 10 others. Our initiative is an essential component of the AEA, and our group continues to welcome and support Indigenous evaluation professionals and professionals working collaboratively with Indigenous Nations and communities, globally. The IPE TIG actively contributes to domestic and international efforts that are enhancing the overall improvement of the evaluation discipline, including regular collaboration with other TIGs focused on supporting minority and underrepresented groups in evaluation, Voluntary Organizations for Professional Evaluation (VOPEs), and Indigenous partners.
The IPE TIG members contribute to international efforts that are enhancing the overall growth of evaluation practices through the EvalIndigenous Task Force. The task force is developing the Native Voices project, led by Dr. Fiona Cram, a collection of community-based Indigenous evaluation stories and case studies from the global community. This exciting project will provide valuable lessons and knowledge for the evaluation community, including AEA members. The EvalIndigenous 2020 agenda is prioritizing the strategic plan and preparing for an update at the 2019 EvalPartner Global Evaluation Forum.
Our IPE TIG Co-Chair, Nicole Bowman, PhD, and other AEA members attended the 2018 Canadian Evaluation Society Conference (see photo, left). The conference plenary panel included Larry Bremner, Nan Wehipeihana, Kate McKegg, and Nicole Bowman. Their well-received panel discussion offered a strong reminder about evaluation practice and its ability to empower communities when it is properly used. The plenary panel will be posted on the Canadian Evaluation Society website soon.
The Evaluation 2018 theme is Speaking Truth to Power and the IPE TIG leadership considers it to be an important and challenging discussion about how we, as evaluators, are approaching evaluation and supporting progress in the field. The evaluation community has an opportunity to nurture awareness and understanding in our profession. It is important for AEA to continue welcoming and purposefully supporting all people involved in the evaluation field. The IPE TIG leadership looks forward to supporting efforts that fill the gaps to engage evaluators at all levels in a strategic and resourceful manner.
The IPE TIG leadership would like to acknowledge the active voices and contributions of Nicole Bowman and many other AEA members throughout the AEA EvalTalk discussion thread in May 2018. You can access the May 2018, Week 1 AEA EvalTalk discussion thread here.
AEA President Leslie Goodyear was recently interviewed by the Greater Boston Evaluation Network (GBEN). She answers six questions from GBEN, reflecting on the current vision and future of AEA, thoughts on the next era of innovation for evaluation, what makes for a strong local evaluation association and what she enjoys most about being an evaluator.
Hear more from Leslie in GBEN’s interview here.
From Natalie DeHart, AEA Membership Programs Coordinator
As you may know, the Guiding Principles Task Force (GPTF) has completed its work to update AEA’s Guiding Principles. In May, the AEA Board unanimously approved the submission of the updated Guiding Principles to the members for their vote during the Board Elections process in July 2018. All members are encouraged to familiarize themselves with the updates prior to voting on July 5, 2018.
Click here to view the updated Evaluators' Ethical Guiding Principles.
Members of the Guiding Principles Task Force included:
During the course of their work, the GPTF gathered member input along the way including:
Assuming the AEA members vote to adopt the Guiding Principles, members are encouraged to volunteer for the Guiding Principles Implementation Working Group if they wish to get involved in engaging with the Guiding Principles. For a more detailed timeline of the work of the GPTF, click here.
The GPTF also had a virtual town hall on June 15 to share the process of the GPTF and to offer an opportunity for members to engage with the Task Force. In this town hall, GPTF members Beverly Parsons, Mike Morris, Lisa Aponte-Soto and President Leslie Goodyear spoke to the updated version of the Guiding Principles. If you missed the opportunity to discuss this work with GPTF members, you can view the recording here. You can also click here to view some FAQs from the town hall.
All of this information is available on the Guiding Principles page of the AEA website. For questions about the Guiding Principles or the member vote, please contact Natalie DeHart, Membership Programs Coordinator, at info@eval.org or 202-367-1166.
You will be asked to vote for the future leadership of AEA and adaptation of the Guiding Principles when elections open on July 5, 2018. We will be collecting your votes through an anonymous ballot, as done in the past.
Please take a moment to carefully consider each candidate as thoughtful selection of the leadership ensures the vitality and longevity of the association. The ballot includes extensive personal statements developed by each nominee to help you make informed selections.
You will be voting for the following board positions: one (1) President-elect and three (3) Members-at-Large. Here are a few things to note when you go to cast your ballot.
The AEA Board of Directors appointed a task force in March 2017 to update (with member input) the AEA Guiding Principles for Evaluators to address developments in the evaluation field. In May 2018, the Board of Directors unanimously approved the submission of the updated Guiding Principles to AEA members for their vote.
As a member of AEA, please take this opportunity to vote on whether the updated Guiding Principles should be adopted by AEA.
Click here to review the updated Guiding Principles.
To vote, please select one of the options below:
Please be certain to check your spam inbox to ensure you do not miss the opportunity to cast your vote. To troubleshoot your access to the ballot, contact Zachary Grays in the AEA offices at info@eval.org. For technical issues, contact support@vote-now.com or to vote by telephone, call (844) 797-4575 (M-F, 9am-6pm, EDT), to speak with a Vote-now specialist. Please enter your voter passcode at the prompt.
The ballot will be available on July 5, 2018 at 12:00 AM ET and will close August 4, 2018 at 11:59 PM ET. Reminder notices will be sent to those who have not cast their ballot by July 14, 2018.
Mark your calendar for receipt of the ballot and stay tuned for additional elections details.
From the AEA Education Team
2017 Conference Recordings Available Here
Select Presidential Plenary Evaluation 2017 recordings are now available on the AEA Digital Knowledge Hub. Sessions include: From Learning to Action: Employing Evaluation to Advance the Public Good; Dialogues on Race & Class in America; Change Agents: Advancing the Global SDGs by Ensuring the Right to Evaluation by All; and The Role of Evaluation in Improving Healthcare: A Call for Embedded Evaluation.
Whether you missed last year's sessions or are just looking to re-watch presentations, you can now access this educational content — as well as past Coffee Breaks and live eStudies — on-demand and at your fingertips. It's never been easier to immerse yourself in peer-driven content and rich, in-depth presentations.
eStudy 091: Designing Useful Surveys | July 17 and 19, 2-3:30 p.m. ET Presented by Michelle Kobayashi,Vice President, National Research Center, Inc.
Surveys for program evaluation, performance measurement, or needs assessment can provide excellent information for evaluators. However, designing effective surveys requires an eye both to unbiased question design as well as the best methods for data administration. Neglecting these two aspects impacts the success of the survey.
eStudy 092: Consulting and Collaborating with Communities and Nonprofits | August 7 and 9, 12-1:30 p.m. ET Presented by Susan M. Wolfe, Community Consultant, Susan Wolfe and Associates; Ann Webb, President, Price Community Evaluation Solutions
This eStudy will provide evaluators who work with community members and community-based organizations with specific skills, strategies, and insight needed to effectively engage in communities. Community-based organizations include government, education, health care, faith-based, and nonprofit organizations.
eStudy 093: Introduction to Usability/UX Testing for Evaluators | September 18 and 20, 12-1:30 p.m. ET Presented by Kay Corry Aubrey, Usabiility Consultant, Usability Resources Inc.
In this session you will gain an understanding of what usability testing is, when to use it, the language of usability/User Experience and techniques for blending it with other qualitative methods. You will be introduced to best practices for planning, moderating, analyzing and reporting on a usability study. You will also learn approaches to take when running a usability study at different points in a product’s development (e.g., concept, early prototype, released product). You will gain hands-on experience with heuristic reviews and be exposed to other methods for evaluating the usability of interactive products.
Looking for a short break to boost your learning? Take advantage of the Coffee Break session AEA is offering in August. Details below, with more information and links to register available on the AEA Coffee Break page.
Evaluation Capacity Building (ECB) Framework & A Nonprofit's Story | Tuesday, August 7, 2-2:20 p.m. ET Presented by Beverly Peters
This coffee break will discuss how motivation and collaboration influence evaluation capacity building, as well as how nonprofit organization In4All (formerly BEC) increased their capacity to do program evaluation using this ECB framework.
From the AEA Board of Directors
The AEA Board of Directors would like the opportunity to engage more with AEA members and discuss a variety of strategic and visionary topics with the membership. The virtual Town Hall approach allows a regular opportunity to pose strategic questions and topics to the membership for input.
The Board will use the GoTo Meeting platform with a Q&A feature for receiving and addressing questions. President Leslie Goodyear will host 30-60 minute sessions. The following session topics and guest facilitators have been confirmed.
Join AEA President Leslie Goodyear and Board member Tessie Catsambas as they discuss the AEA Ends Goals statement and highlight the work the Board is currently performing on Policy Governance (PG). Register here.
In this AEA Town Hall, join AEA President Leslie Goodyear and members of the Evaluation 2018 Local Arrangements Working Group (LAWG), based in Cleveland, OH, to discuss the culture, sights and attractions in Cleveland, as well as what attendees can look forward to during this year's program. The discussion will be held on Friday, August 17, at 2 p.m. ET. Register here.
From Sheila B. Robinson, Potent Presentations Initiative Coordinator
Over the last few months, I shared results from a Potent Presentations Survey called “You, the Presenter: What Would Help You Up Your Game?” I shared who respondents were, types of presentations they most often give, types of presentations they anticipate giving in the future, along with what respondents identified as priority areas for improving their presentation practice. Last month’s article looked at what formats respondents prefer for new learning and featured a list of respondents’ most admired presenters.
This month, I’ll share who was named most often along with the characteristics respondents said admire. These characteristics easily fell across our p2i categories.
Delivery style appeared to be very important to respondents, with a large variety of words used to describe it. The most commonly used terms were confidence (also fearless and comfortable) and funny (also wit and humor). Warmth, realness, authenticity, frank, conversational, and accessible were also mentioned, as were energy, focus, and passionate.
Related to Delivery, storytelling came up several times as something respondents admire. For example, respondents mentioned:
Also related to Delivery was presenters having insight or expertise. For example, respondents mentioned:
Messaging was also important, with respondents identifying the following:
Closely related to Messaging were mentions of content and how it was presented. For example:
Engagement, or audience engagement was mentioned multiple times, with one presenter described as always pushing … to come up with unique audience engagement strategies, and another as engages people in innovative ways. Another was described as responsive to audience comments.
Design was mentioned only a few times with respondents identifying for example, good slide decks, simple slides and slides are visual.
Two additional quotes lend insight into how some of these presenter characteristics play out during presentations:
Finally, you’ll remember that 69 presenters were named in our list. The top 12 presenters are those who received more than one mention. Here they are (names next to each other indicate the same number of mentions):
How will you use the insights from this survey to inform your presentations? Lastly, I’m certain some of our readers were respondents, and to you I offer many thanks!
p2i Needs Your Help!
Please contact me at p2i@eval.org and let’s talk! I’m happy to help, offer guidance, or collaborate on any of these.