Date: Friday, September 28, 2018
With Fall officially upon us, we’re in high gear getting everything ready for the annual conference, Evaluation 2018, in Cleveland, Ohio. It’s going to be a great gathering of evaluators from around the U.S. and the world, with plenty of time to share and learn from each other. We’re about a month out from the conference itself, and that means everyone who’s involved in conference planning is crossing items off their to-do lists, answering members’ questions, double-checking registration, confirming speakers, ordering supplies, finalizing rosters and getting excited for Cleveland. Phew!
So much goes into making a great AEA conference. I want to highlight a couple “behind the scenes” activities we have been engaged in, and that we hope will help make a great conference.
We know that not all members can make it to the annual conference, but there are still so many ways to get involved and share what’s happening in your evaluation world! For example, to move forward on next steps for the updated Guiding Principles and the newly approved Evaluator Competencies, we’re putting together working groups that will think about how to disseminate these important documents, develop trainings and workshops, and ensure that AEA members are aware of their importance. We’ll still have a few more virtual Town Halls this year. In fact, we’re already planning one for November that will focus on learnings and vision from the editors of the American Journal of Evaluation. And as I’ve mentioned many times this year, there’s the Issues and Ideas Portal on the AEA website – login and check it out! It’s a space where you can share those brilliant ideas and concerns you have for the evaluation field and the association. We track what’s submitted so that we can see what’s trending and what’s urgent.
I’ve been making my way through the conference program and I’m so excited about the sessions, posters, ignite talks, workshops, receptions and more. It will be hard to choose what to attend, but it’s also an amazing opportunity to exchange ideas with colleagues and friends from around the world. I hope to see you there. Hello, Cleveland!
The Face of AEA spotlights our members and their backstories - why they joined the profession, what drives them and memorable lessons they've learned along the way. Know someone who should be featured? Email the AEA editor, Kristin Fields, at kfields@eval.org.
Affiliation: Claremont Graduate University & Ersoylu Consulting Degrees: Claremont Graduate University – Ph.D. Evaluation and Applied Research Methods, (In Progress) Claremont Graduate University – M.A. Psychology California State University, San Bernardino – M.S. Industrial/Organizational Psychology San Francisco State University – B.A. Psychology Years in the Evaluation Field: 10 years Joined AEA: 2014
AEA provides an opportunity to connect with other professionals, share my research and learn more about the current research and state of the field. Being a member also enables me to welcome those that are new to evaluation and provide mentorship.
An evaluation project that will always be memorable for me was my work with a community-based participatory partnership. I designed and facilitated a qualitative evaluation intervention focused on repairing the relationship between two culturally different partners (i.e., a Tongan community organization and a local university). They were partnered on a cancer navigation intervention aimed at improving the cancer screening rates among Tongan women in Southern California. Working collaboratively with the partners on the evaluation resulted in the strengthening of individual empowerment and community-academic relationship.
This experience provided me with an understanding of how to design and conduct a culturally sensitive evaluation. It also culminated in my first peer-reviewed article, and being a first-time author, in the fall 2015 issue of Progress in Community Health Partnerships: Research, Education, and Action. Our article is entitled, “Enhancing community-based participatory research partnerships through appreciative inquiry.”
I would recommend that they begin with leveraging the dearth of opportunities that AEA provides them to make connections with others in the field. This includes participating in discussion boards, training opportunities, conferences and topical interest groups (TIGs). Engaging with one or more of these opportunities will help them build quality relationships with other professionals and access resources beyond AEA.
Additionally, everything an evaluator does is an opportunity to create a positive impact upon programs, communities, individuals, etc. To ensure this, it is crucial that they conduct themselves with integrity and adhere to AEA’s Guiding Principles for Evaluators, and AEA’s Statement on Cultural Competence in Evaluation. I also recommend they strive to be situationally aware in their work by being attentive to their own bias and responsive to a variety of contextual dynamics. This can include culture and political situations in programs. Ultimately, connecting with other professionals and being in engaged with AEA’s resources are invaluable.
From Ewa Sobczynska, Environmental Topical Interest Group (TIG)
Are you planning to come to Evaluation 2018 in Cleveland, Ohio? If so, consider offsetting the carbon emissions associated with your conference travel. In case you didn’t know, close to 95 percent of AEA’s carbon footprint is due to travel — air travel, in particular — associated with our annual conference, according to AEA’s 2015 Green Audit.
Why participate in voluntary carbon offsetting?
Carbon offsetting offers an opportunity for direct consumer action on the carbon emissions you simply cannot delete from your daily life. It is a practical solution to counter your personal carbon emissions and give you an opportunity to invest in the growth of renewable energy.
In addition, investing in carbon offset projects creates an alternative source of financing for greenhouse gas emissions mitigation projects and is an incubator for future carbon market innovation. AEA encourages its members and all conference participants to take this action, along with any other actions you might be taking to fight climate change and to encourage long-term environmental sustainability. By engaging in more sustainable lifestyle habits, or more efficient ways to live (such as increased recycling, driving a hybrid car, eating less meat or choosing more sustainably produced food, etc.), the sum total of our collective actions makes a difference.
What can you do?
Continuing the efforts started in 2015 by the AEA’s Environmental Sustainability Working Group, we encourage you to make a voluntary contribution through Native Energy, or a similar organization, to a project that will reduce or capture carbon, thus offsetting the carbon resulting from your travel. Calculate the emissions associated with your trip here and make a contribution. Important! If you choose Native Energy, please enter “AEA” as your “Company” so we can track our impact.
From Lana Rucks, AEA Cleveland Local Affiliate Working Group Chair
The Cleveland LAWG committee and Ohio Program Evaluators’ Group (OPEG) is looking forward to welcoming you to Cleveland in the near future at Evaluation 2018! While at the conference, make sure to visit our table to ask questions and meet the team.
If you have not already done so, download the local resource guide. This guide has a host of information about the region and can be a key resource as you plan your stay. If you missed the AEA Town Hall meeting with members from the LAWG, watch it here for a quick introduction to a region that we all love!
We are continuing the “Evaluation Without Borders” program piloted by the Washington D.C. evaluators at last year’s conference. As context, Evaluation Without Borders is a volunteer consulting event that allows for evaluation professionals to share their knowledge and expertise with community-based non-profit organizations. We thank those individuals who have already signed-up and encourage others to participate. To learn more, contact Seema Mahato at sm618312@ohio.edu.
If you have more questions about where to go or what to do in Cleveland, contact Clara Pelfrey at clara.pelfrey@case.edu. If you are with a Topical Interest Group (TIG) or a large group looking for a meeting location, reach out to Jan Noga at jan.noga@pathfinderevaluation.com.
If there is any other way that the we can support your visit to Cleveland, do not hesitate to reach out to me, Lana Rucks, at lrucks@therucksgroup.com. Remember: Cleveland Rocks!
At this year’s Evaluation 2018, plenary sessions will feature in-depth conversations around what it means to speak truth to power, the theme of this year’s conference. On Thursday, Nov. 1, Brooke Haycock will present “American Grit.” During this hour-long session, Brooke will give a unique performance that weaves quantitative data with narrative storytelling, drawn from more than 300 interviews with youth, educators and leaders from higher education, business and industry.
We spoke with Brooke to understand why speaking truth to power is more important than ever, what inspired her to develop this presentation, what she hopes people take away from the experience and more.
The theme of Evaluation 2018 is “Speaking Truth to Power.” In your opinion, why, in today’s culture, is it important for evaluators to take this theme to heart?
At a time when there is an all-out assault on facts and truth, I think the work of evaluators is more critical than ever before. One of my favorite Gloria Steinem quotes reads (pardon the crassness): “The truth will set you free, but first it will piss you off.” Truth and data, the very ore evaluators excavate, have the power to educate and empower, to disrupt faulty narratives and expose injustices, and to amplify the perspectives and experiences of those who too often go unheard in assessments of agency and programmatic quality. As ambassadors of truth, evaluators have a critical role to play in speaking truth—no matter how uncomfortable—to real power.
What inspired you to create this type of presentation? What is it about the format that engages people?
I’ve spent the past 17 years, up until very recently, as an artist embedded in a data-driven education advocacy organization called The Education Trust. There, I witnessed time and time again, the power of honest quantitative data with audiences of education practitioners and policymakers. I also witnessed the pushback, often driven by deficit assumptions about the capacity of all students to achieve and the capacity of schools and educators to serve all students well. Sometimes, the numbers alone are not enough to convince the skeptics. You need to share the stories behind those numbers. A playwright-actor by training with a background in student organizing, I set out to create a body of theatrical work aligned with our data work, based entirely on interviews with students, families and educators. Among the most recent products of that work, American Grit seeks to marry the qualitative with the quantitative to explore the gaps in real college and career readiness through the stories of two cousins. A departure from traditional modes of presentation, at least in my field of education, it is my hope that theater and storytelling—rooted in data—can engage both hearts and minds and can amplify the stories and voices of students in ways that compel adults to act.
How have you seen the landscape of education shift, whether positive or negative? What can evaluators do to help move American education forward in a positive manner?
As accountability shifts from the federal government to the states and localities, and we see the rollback of many of the federal protections for students—particularly those historically underserved by our institutions—evaluators play an evermore critical role in identifying the gaps and elevating the bright spots in policy and practice at every level.
What are some takeaways you hope attendees get out of your presentation?
I am humbled to be invited to share my work with experienced evaluators at this conference and think I have more lessons to learn from them than they from me. What I hope to contribute is a reminder of the power in marrying quantitative and qualitative data and the importance of amplifying the experiences, voices and perspectives of those served by the agencies and programs they evaluate—those who have most at stake in how effective and responsive those institutions and efforts are.
At Evaluation 2018, plenary speakers Aurora Martin and Miguel Willis will present their session, “Equity in Action: Innovative Strategies to Build Culturally Responsive Evaluation Capacity Among Access to Justice Interventions.” We spoke with them about why the theme of this year’s conference is important to evaluators, their perspective on what can be learned from the next generation and lessons they hope to impart during their session.
Why, in today’s culture, is it important for evaluators to take the theme of Evaluation 2018, Speaking Truth to Power, to heart?
Now more than ever, speaking truth to power has a sense of urgency in the digital context. Here, new tools of communication for community engagement and empowerment are challenged by the efforts of those intent on disinformation; where identities are defined by data beyond our individual control; and where equitable and meaningful economic opportunity and political participation are driven by innovation and division. Evaluation in this complicated context raises the need to both pause and keep pace with the rapidity of such dynamic developments. There is so much at stake, that pausing for the design of the ethical parameters and right set of questions is necessary. Yet there is so much to catch up with that real-time evaluation methods must also be determined to integrate into programs and prototypes, from design to deployment and from policy to practice.
Your talk focuses on engaging the next generation to create social change – what trends or habits have you seen among younger generations that the evaluation community should pay attention to?
It is hard to not generalize, especially with the many differences between regions and the broad urban-rural divide. But, at a basic level, younger generations are born into a startup culture and one that is so digital that the method of learning and processing relationships and information is different. There are opportunities to create meaningful learning communities with digital tools and remote connections, which are valuable when addressing issues of distance or isolation. At the same time, the challenges of being born into such a fast-paced time with tools that allow for more public scrutiny and manipulation can make it hard to define what are successful outcomes. The next generation is going to be defined by an integrated way of life that pushes the boundaries of categorization for better or worse. Customer Relations Management systems are an example of the algorithm of integrated and predictive profiles. As a community practitioner, it seems that evaluation as a methodology can be of great assistance because it provides a powerful framework and tools to make bite-sized sense of a tremendous amount of information that is more readily available.
With change comes disruption, and some discomfort, too. What is your advice to those who might be hesitant to engage with change?
It is OK to be cautious, but to engage; in fact, it is probably best to cautiously engage as a self-awareness and reflection exercise.
What are some takeaways you hope attendees get from your presentation?
We hope this presentation challenges notions of inclusion and equity, and also of how we evaluate success – when it starts and ends, and from whose vantage points – given that our attempt to develop programs were intentionally ambitious about creating a set of tipping point programs, with goals of inclusion, equity, diversity, and community building. We have certainly had our proof of concept discussions at different stages of the programs we have developed, but we are committed to creating a pathway of continuous improvement and opportunity for underrepresented next-gen innovators, whatever the sector they enter and lead.
From Zachary Grays, AEA Headquarters
The American Evaluation Association (AEA) is pleased to announce the continued partnership with the Center for Culturally Responsive Evaluation and Assessment (CREA) to offer a unique thread of professional development training options as part of the pre- and post-conference offerings during Evaluation 2018 (Oct. 29 – Nov. 3) in Cleveland, Ohio. CREA was established in 2011 in the College of Education at the University of Illinois at Urbana-Champaign, with Stafford Hood, Ph.D., Sheila M. Miller Professor, serving as its founding director.
CREA is a culturally diverse and interdisciplinary global community of researchers and practitioners in the areas of evaluation and assessment. CREA’s primary focus is to address the growing need for policy-relevant efforts that take seriously the influences of cultural norms, practices, and expectations in the design, implementation, and evaluation of social and educational interventions. To learn more about CREA, click here.
What can attendees expect from this AEA-CREA partnership? Take a look at the courses being offered this year:
Full Day Session:
Half Day Sessions:
Save the date! The Fifth International CREA Conference
March 27-29, 2019
Pre-Conference Workshops: March 26, 2019
Theme: Intersectionality as Critical Inquiry, Method and Practice: Moving Beyond Nominal Categories and False Dichotomies in Culturally Responsive Evaluation and Assessment
From Natalie DeHart, AEA Headquarters
We are excited to announce that in July 2018, the AEA membership voted to ratify the newly updated Guiding Principles. The vote was overwhelmingly positive in favor of the newly updated Guiding Principles. A huge thank you to the entire Task Force, named below, for all of their hard work over the past two years.
The 2017-2018 Guiding Principles Task Force (GPTF) has been working hard over the past two years, collecting member input, wordsmithing and updating the language, as per their charge to:
While the original Principles were written in 1994, it is the policy of AEA to review the Guiding Principles every five years to make sure they remain current as the field of evaluation develops over time. During the course of their work, the GPTF gathered valuable member input along the way, including:
Next Steps:
Session at Evaluation 2018: Join the GPTF for their session on Friday, November 2, from 8 a.m. – 9 a.m. to discuss next steps for implementing the Guiding Principles into your work as an evaluator. Here is some information about the specifics of the session:
This session (#2854) will briefly present the AEA board-commissioned revision process over the past two years, including the criteria used to update the principles. Special attention will be given to the principle formerly titled General and Public Welfare and now titled Common Good and Equity. The session will engage evaluators in a discussion of the principles and how they are used in their practice and also strategize how the revised principles can be disseminated and understood across the evaluation community.
Brochures: Newly printed brochures will be available at Evaluation 2018 for all attendees. If you are not attending the conference and would like a new brochure, you may request one to be sent to you from the AEA office starting in November.
Guiding Principles Working Group: Be on the lookout for a call for volunteers for the Guiding Principles Working Group. If you would like to participate, please contact Natalie DeHart, AEA Staff, at info@eval.org or call 202-367-1166.
Do you lead or participate in one of AEA's Topical Interest Groups (TIGs)? We want to hear from you and spotlight your work. Send an email to the AEA editor, Kristin Fields (kfields@eval.org) to share news, updates and articles for consideration in an upcoming AEA newsletter.
From Sheila B. Robinson, Potent Presentations Initiative Coordinator
I’ve seen tons of advice on overcoming public speaking fear — everything from relaxation exercises, to power poses, deep breathing and “just love your audience and you won’t be nervous.” Really? I had stage fright because I loved my audiences. I wanted so much to provide value for them and be worth their precious time and attention. While these strategies are sound and may provide relief for some, they didn't work for me.
Last month, I shared my secret — I just did it. I just put myself in positions where I’d have to speak, and over time I became more comfortable. But I also learned something else. I once asked a psychologist for advice on stage fright. Not for public speaking – I figured I’d just continue gaining experience to overcome that fear. I asked because I was about to do something unusual — perform a piano recital. I had learned a special piece for my mother, one of her favorites, but I hadn’t performed since my teens. It wasn’t so much about anxiety, but what my nerves did to my hands. I asked the psychologist what I could do to ensure my hands wouldn’t shake.
“How can I get rid of nerves at the performance?” His answer blew me away. “You can’t.” He told me the anxiety would be there, and the best way to deal with it is to simply let it be there. If I tried to fight it, control it, or attack it, it would just grow stronger and would steal my attention from the performance. After all, anxiety is fueled by adrenaline, and what happens when we ready ourselves to fight something? More adrenaline!
Once I understood this, I tried his suggestion of self-talk. Before getting on a stage I would acknowledge my rapid heartbeat and shaky hands instead of tensing up trying to fight them off. I learned to say to myself, “Here’s some anxiety. A little stage fright. It’s OK. It’s not going to be a problem. I’ll just let it be there.”
Calming self-talk doesn’t produce adrenaline. And guess what? Adrenaline has a pretty short half-life, and if my body wasn’t continuing to produce it, I felt calmer faster. And the performance? Nailed it. Years later, I stumbled across this quote which hangs in my office and is attributed to Samuel Pamer, a 113-year-old asked about the secret of his longevity. He said, "When it rains, I let it."
Here’s another perspective that may help: People don’t perceive you being as nervous as you think they do. Check out The Illusion of Transparency and Public Speaking Fear on Six Minutes, a great blog on public speaking and presentations skills.
Have you successfully used p2i tools or p2i principles in your presentations? Do you have “before” and “after” slide examples you would be willing to share? Do you have ideas for, or are you interested in writing a blog article on Potent Presentations? Do you have an interest in sharing your tips for Potent Presentations through a brief video or webinar?
Please contact me at p2i@eval.org and let’s talk! I’m happy to help, offer guidance, or collaborate on any of these.
From the AEA Board of Directors
Did you miss the AEA Town Hall meetings? You can now access them on AEA’s website here. Be sure to watch the most recent conversation featuring Jean King and AEA Board member Susan Tucker discussing AEA’s Evaluator Competencies.
From the AEA Education Team
The Digital Knowledge Hub is an online platform featuring professional development opportunities for evaluators, by evaluators. View past eStudies, like this one:
eStudy 091: Designing Useful Surveys
Surveys for program evaluation, performance measurement, or needs assessment can provide excellent information for evaluators. However, designing effective surveys requires an eye for both unbiased question design and the best methods for data administration. Michelle Kobayashi shares guidelines and methods for survey development that will increase response rates and create reliable and valid questionnaires.
Presented by Kanti Gopal Kovvali, Author, Organizational Unlearning Specialist, Visiting Faculty TISS and NMIMS
The eStudy will expose participants to a radically different evaluation process that is rapid, transformational and sustainable. The eStudy will help participants to change their paradigms about evaluation, introduce new methodologies and tools and transfrom their orientation from doing good research to facilitating transformational research.
eStudy 095: Using the Cultural Consensus Method to Evaluate Program Impacts | November 27, 29, 12-1:30 p.m. EDT
Presented by Peggy Ochandarena, Chief of Party- Enhancing Palestinian Justice Program, Chemonics International Inc; Co-Director, Global Impact Collaboratory and Roseanne Schuster, Assistant Research Scientist, School of Human Evolution and Social Change, Arizona State University; Director of MEL, Global Impact Collaboratory.
Culture is the shared beliefs of a particular group of people, and strongly shapes what is socially acceptable and thus shapes action. Understanding the culture of an intervention's beneficiaries is critically important in designing interventions for effectiveness. In this webinar, leaders of the Global Impact Collaboratory, a partnership between Arizona State University and Chemonics International, give learners hands-on interaction with the theory, instrument design, and analysis for the CCM, with demonstration of its use in international development projects and application to a case study.
Cost-Inclusive Evaluation: Language Power Understands | Tuesday, Oct. 2, 2018, 2-2:20 p.m. ET Presented by Brian Yates
Including costs in evaluations changes how program providers, participants, policy-makers, and funders perceive programs. That can be good. When we report how well programs implement plans, we evaluate fidelity. When we report changes in outcomes targeted by programs such as behaviors, thoughts, feelings of individuals or communities, we valuate effectiveness. If we report the value of resources consumed by programs, we evaluate costs. Assessing the value of resources generated by programs evaluates benefits. Measuring costs of programs relative to benefits evaluates cost-benefit. Considering costs of programs relative to effectiveness evaluates cost-effectiveness. Comparing cost-benefit or cost-effectiveness of competing programs can be exciting, difficult, and helpful in getting the attention of the powerful. Including costs in evaluations can help evaluations get used.
¡Feliz Ano Año Nuevo! – Culturally and Linguistically Appropriate Support in Community Engaged Work | Tuesday, Oct. 9, 2018, 2-2:20 p.m. ET Presented by Grisel M. Robles-Schrader & Josefina Serrato
The Spanish language is made up of many different dialects. To provide appropriate Spanish translation and interpretation for a CE service, our team assesses several factors including translation and interpretation needs of the project; team capacity; staff translation/interpretation experiences; and willingness of the research teams to accept coaching and collaborate. Presenters will share “effective practices” and lessons learned based on their experiences ensuring the availability of translation and interpretation of CE services to effectively engage Spanish-speaking communities.
Creating Great Focus Groups | Thursday, Oct. 11, 2018, 2-2:20 p.m. ET Presented by Rhonda Williams, Ph.D.
This webinar will present ideas for preparing, organizing and structuring focus groups to create sessions that are beneficial to both the evaluator and the participants. The handling of focus group logistics helps to create a welcoming environment, where participants are more likely to provide information needed for evaluations. Additional attention should be given to the development of questions that will allow the needed data to surface in focus group conversations. Focus groups are an efficient way to gain information from stakeholders; they provide more personal vehicle to obtain important feedback. You will learn some extra tips to make these sessions great.