Date: Friday, December 21, 2018
“There is no progress or accomplishment without Sacrifice” —Unknown
Thank you for making 2018 a remarkable year for the American Evaluation Association! Our dedicated leadership, members and staff have done an incredible job at expanding our outreach while strengthening our programs and services.
We had an incredible annual conference, managed a new Executive Director transition, and celebrated other highlights, including:
This was also a great year for feedback from our membership. In this issue of the newsletter, we share highlights from the report of the bi-annual Membership Survey, which provided great data to guide our leaders and staff in making decisions that benefit our members.
Another stride is the recent launch of our new online Career Center. This resource is free to AEA members to upload resumes, search exclusive job postings from relevant companies, and subscribe to tailored job alerts.
Our forward focus will include increasing the value of member services and further expanding AEA’s role in the evaluation industry. Other initiatives will include a strategic communications plan, an enhanced AEA website, stronger partnerships with local affiliates and increasing our revenue to ensure sustainability of the association.
AEA is stronger than ever due to our continually expanding community that comes together to donate time, resources and talent in support of our mission. Members of our National Board, Task Forces, Working Groups, Topical Interest Groups, Local Affiliates and others all contributed to achieving our goals this year. Our outstanding staff continues their steadfast work to sustain AEA’s position as a pillar of the evaluation community.
Thank you for being a part AEA, and we are looking forward to continued growth in 2019!
The Member Survey Working Group has completed its work in launching and analyzing the survey results for the 2018 AEA Member Survey. The membership survey is conducted every two years and is designed and analyzed by a working group of members. The most recent survey, released in the spring/summer of 2018 and analyzed this fall, uncovered a variety of perspectives that are now driving strategic and operational conversations and decisions.
The 2016 AEA Member Survey had results that were very similar to this year’s survey. Comparison data of the two surveys is available in the report that was presented to the AEA Board of Directors during Evaluation 2018. Topical Interest Groups (TIGs) are still the most valued reason members remain engaged with AEA, while AEA’s annual conference is our most valued member benefit. Satisfaction ratings have remained consistent over the course of the past few years, as well. The AEA staff team and newly formed Membership Advisory Working Group will continue to promote recruitment and engagement of AEA members.
A big thank you to our Member Survey Working Group Members for all their work over the past year and a half:
From Krista Collins, Director of Evaluation and Insights at Boys & Girls Clubs of America
This year, the federal budget again reduced funding for 21st Century Community Learning Centers and summer learning programs, claiming that limited evidence exists to support the positive impact of after-school programs. While research and evaluation studies continue to show that youth who participate in high-quality after-school programs are more likely to demonstrate better academic engagement, develop healthy social relationships, and avoid risky behaviors, these positive outcomes are often overshadowed by an outdated focus on randomized control trials and standardized test scores.
In the past two decades, research on positive youth development has proliferated the after-school field, offering new standards for trained youth development professionals to help structure and implement high-quality programs, respond to the needs and interests of young people, and empower them with the motivation, knowledge and skills for lifelong success. These learnings highlight the critical importance of staff training, program quality and tailored implementation as facilitators of positive outcomes, as well as the value of promoting social emotional resilience, workforce readiness and 21st century leadership skills as outcomes that predict overall achievement and well-being. At the same time, these findings call us to question the current standards of evidence that are used to decide the availability of programs and practices that can change the odds for youth.
As director of evaluation and insights at Boys & Girls Clubs of America, and a strong advocate of identifying what works and getting that information directly into the hands of practitioners, aligning to AEA’s guiding principles for evaluation means pushing the boundaries of what is expected and exploring new methodologies, measures and analyses that better capture what success looks like. For example, a common metric used to judge the value of any program is attendance. Recently, after-school practitioners have expressed concerns about declining teen participation, especially at a time when after-school programs are most needed to support on-time high school graduation. However, this absolute focus on high participation rates undermines the value of partnering with community organizations to track other after-school activities such as employment, community service or committing to a leadership position in school clubs and sports teams that are also indicators of holistic positive development.
Similarly, the most successful after-school programs and educational interventions recognize that a one-size-fits all approach is an inappropriate response to the different backgrounds, needs and aspirations that make up today’s families and communities. To promote equity, practitioners and evaluators alike must acknowledge that youth development is rarely linear and interventions must be responsive at the onset of services, as well as flexible over time, to develop a more sensitive understanding of how young people grow within and across different contexts. While evaluators must attend to project requirements and expectations, adhering to the guiding principles means having the competence to advocate for the most systematic and contextually relevant approach and the integrity to adjust methods in respect of the shared desire to promote equitable positive outcomes for all.
One thing I learned early in my career and continue to value daily is the symbiotic relationship between evidence-based practice and practice-based evidence. While insights generated through rigorous scientific methods are vital to the design and implementation of effective youth programs, it is equally essential to establish communication channels with frontline practitioners, parents, school partners and other key stakeholders to continuously expand our definition of after-school success. Practicing with the guiding principles in mind means the evaluator is responsible for taking on additional efforts to reduce the time between data collection and reporting. Feedback should be shared regularly, and in ways that different types of learners can digest and internalize (e.g. dashboards, presentations, infographics, interactive activities). This allows for constant reflection and iteration to ensure that practitioners have the very best tools, resources and strategies available to create transformative opportunities for all youth.
Walking the Talk is a column authored by AEA members, sharing what the association's values mean to them and how these values guide and impact their work. Interested in contributing? Email the AEA editor, Kristin Fields, at kfields@eval.org.
Do you lead or participate in one of AEA's Topical Interest Groups (TIGs)? We want to hear from you and spotlight your work. Send an email to the AEA editor, Kristin Fields (kfields@eval.org) to share news, updates and articles for consideration in an upcoming AEA newsletter.
From Sheila B. Robinson, Potent Presentations Initiative Coordinator
According to a survey from Statista, 18 percent of people polled indicated that reading more was one of their New Year’s resolutions for 2018. I am one of the 18 percent. Alas, I kept part of my resolution… I bought more books!
While I didn’t get though every book I’ve purchased, I am doing better in one particular genre: books on presentations. In 2018, I read three books on presentation design with a little nudge from my friends at the Presentation Guild. They run a wonderful online book club from time to time and it is open to the public. Only the last session – a capstone chat with the author – is reserved for members only.
Here is a little about each book:
Resonate: Present Visual Stories that Transform Audiences by Nancy Duarte Duarte generously gives readers an opportunity to read and experience this book with a free multi-media version. I read the multi-media version on a tablet and later purchased the physical book to reread, and found both well worth my time. The multi-media version includes extra content in the form of embedded links and videos, and one feature I especially enjoyed is the Director’s Cut. Readers click on an icon to reveal a bit of the inside scoop from Nancy herself on the making of this book (for example, why she actually writes books using PowerPoint).
Beyond Bullet Points (4th Ed.): Using PowerPoint to tell a Persuasive Story That Gets Results by Cliff Atkinson While this book is focused on PowerPoint, it walks the reader through a storyboarding process to create a powerful message using the author’s story template and formatter – also offered as free tools on his website. The examples he uses are for a corporate context communication strategy, but can readily be translated for other contexts (e.g., evaluation, presentations to non-profits, government, education).
Impossible to Ignore: Creating Memorable Content to Influence Decisions by Carmen Simon, PhD Simon is a cognitive scientist who walks the reader through why certain strategies work with audiences when you want them to remember and recall your messages. This book is full of research conclusions and science – a good, heady read, but not too challenging. I like that she goes deeper than simply offering a surface level list of “to-dos” and helps the reader understand why we remember certain messages and not others.
I enjoyed each of these books and found them terrific complements to our Potent Presentations Tools and Guidelines. They primarily focus on message, with Resonate and Beyond Bullet Points also focusing on visual communication, and thereby addressing aspects of design. At the heart of each author’s message is the understanding that our audiences need to be central to our presentation planning and design.
Please contact me at p2i@eval.org and let’s talk! I’m happy to help, offer guidance, or collaborate on any of these.
Did you miss any AEA Town Hall meetings? You can now access them on AEA’s website here. This year's topics include a review of the AEA Guiding Principles, issues and opportunities confronting the field of evaluation and new AEA evaluator competencies.
Access AEA Town Hall recordings here.
From the AEA Education Team
The Digital Knowledge Hub is an online platform featuring professional development opportunities for evaluators, by evaluators. Check out the latest prerecorded eStudy now available for purchase:
eStudy 087: Nonparametric Statistics So many of us have encountered situations where we did not end up with the robust, bell-shaped data set we expected to analyze. In these cases, traditional statistical methods lose their power and are no longer appropriate. This workshop provides a brief overview of parametric statistics in order to contrast them with non-parametric statistics. Data situations requiring non-parametric statistics will be reviewed and appropriate techniques will be demonstrated using screenshots of SPSS analysis software. The instructor will demonstrate how to run the non-parametric statistics in SPSS.
Save the date for this upcoming live eStudy course.
eStudy 098: Working with Assumptions to Unravel the Tangle of Complexity, Values, Cultural Responsiveness
January 15, 29, February 12, at 12-1:30 p.m. ET | Presented by Jonathan Morell, Ph.D. Principal, 4.669 Evaluation and Planning/Editor, Evaluation and Program Planning; Apollo M. Nkwake, CE. Ph.D. International Technical Advisor, Monitoring and Evaluation, Education Development Center; Katrina L. Bledsoe, Ph.D. Research Scientist, Education Development Center/Principal Consultant Katrina Bledsoe Consulting
It is impossible for evaluators not to make assumptions that simplify the world in which programs, initiatives and “wicked problems” exist. Simplification—and parsing out to key values—is necessary because without it, no evaluation can reveal relationships that matter. We always need a model that provides a simple and straightforward guide for the construction of evaluation designs and data interpretation. The model may be formal or informal, elaborate or sparse, formally constructed or implicit. But always, there is a model, and always, to be useful, the model must provide a parsimonious explanation of the phenomena—and world—at hand.