Date: Monday, January 6, 2025
Hello, AEA365 community! Liz DiLuzio here, Lead Curator of the blog. This week is Individuals Week, which means we take a break from our themed weeks and spotlight the Hot Tips, Cool Tricks, Rad Resources and Lessons Learned from any evaluator interested in sharing. Would you like to contribute to future individuals weeks? Email me at AEA365@eval.org with an idea or a draft and we will make it happen.
We are Ashley Bishop (MEL Technical Lead) and Safal Khatri (Evaluation & Learning Technical Lead) and we both work at Winrock International.
In 2023, we commissioned the final evaluation of the USAID/Girls Leadership and Empowerment through Education (GLEE) Project in Mali’s Kayes and Mopti regions. This 6-year project focused on decreasing barriers to quality education in safe and supportive environments for adolescent girls ages 10-18 years old while increasing the adoption of positive, healthy behaviors.
As part of the final evaluation, we integrated a method that would allow us to engage with girls as key informants on USAID/GLEE’s outcomes. The final evaluation was commissioned to EdIntersect, School-to-School International, and Centre d’Etude et de Recherche sur l’Information en Population et Sante (CERIPS). Their team advised the use of Most Significant Change Technique, which was adapted to integrate with and complement the other quantitative and qualitative methods being used.
Most Significant Change Technique, as described by creator Rick Davies and Jess Dart, can be used using a full 10-step methodology or modified to fit the evaluation research needs. The evaluation team decided to focus on the use of Step 4: Collecting significant change stories and Step 5: Selecting the most significant of the stories, and integrated them both into focus group discussions with adolescent girls.
The girls shared their stories orally and then together decided on which of their group’s stories was the most significant. The focus group facilitation team documented the stories and the selection process.
The MSC Technique guide includes Step 7: Verification of stories. Because of how MSC was adapted in our case, we didn’t verify the stories, because the data collection from the focus groups immediately integrated into the final evaluation findings. However, we triangulated the MSC data with other evaluation findings to build stronger cases. In the future, we would collect stories earlier in the data collection phase so there is more time for triangulation of data.
After having presented on this case study to various groups, one of the most common questions we receive is “did you receive any negative stories?” For the most part, the stories collected do focus on positive significant changes. The focus group discussion guide instructed the groups of girls that their choice of “significant” could be something positive or negative. However, our team feels that Most Significant Change as a technique may not be well positioned to collect negative feedback simply by the way people understand the terms “significant change.” The team would have needed to follow up purposefully to drill into potential negative changes, which may undermine the openness of the technique. MSC did allow us to collect unintended outcomes, but most were positive unintended outcomes. We think that other methods may be better for being more intentional collection of negative outcomes or feedback.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.