Date: Monday, February 3, 2025
Hello, AEA365 community! Liz DiLuzio here, Lead Curator of the blog. This week is Individuals Week, which means we take a break from our themed weeks and spotlight the Hot Tips, Cool Tricks, Rad Resources and Lessons Learned from any evaluator interested in sharing. Would you like to contribute to future individuals weeks? Email me at AEA365@eval.org with an idea or a draft and we will make it happen.
Hello! I’m Nael Jean-Baptiste, a Senior Monitoring and Evaluation Advisor at Save the Children. During the 2024 annual conference, I had the opportunity to share how my colleague (Meg Pollak) and I have used Natural Language Processing (NLP) and Generative AI (Gen-AI) to support the development of a Social Behavior Change (SBC) strategies for a Resilience Food Security Activity funded by the US Bureau for Humanitarian Assistance in Mali. In this blog, I’m sharing the main take aways resulting from the discussion I had with the audience.
NLP and Generative AI were used to analyze Focus Group Discussions (FGDs) to gather women nutrition and Infant Young Child Feeding practices as well as to get community perspective on action to put in place to promote the recommended best practices. By extracting key sentences using a frequency-based approach, the FGDs were transformed the into summaries which then went through a validation process with the program staff. This process helped highlight the most Smal Doable Action and ensured that the SBC activities were shaped by community voices. When using NLP algorithm and Gen-AI in qualitative data analysis, always remember that the goal is not just to summarize but to faithfully reflect community perspectives while taking into consideration the context, which is crucial for effective social behavior change interventions.
Developing in-house algorithms for extractive text summarization can be challenging, especially when it requires coding expertise. Our trick was to combine in-house methods with Generative AI tools. We generated summaries using both approaches and compared them to identify overlaps, selecting the most representative content. This hybrid method-maintained quality, coherence, and reliability while speeding up the analysis process.
Human Expertise and AI Must Go Hand in Hand. Gen-AI and NLP algorithm can only take you so far without human expertise. In Mali, our technical staff’s deep knowledge of local customs and nuances was essential in ensuring the summaries made sense in the context. The takeaway here is simple— NLP algorithm and Gen-AI tools can be incredibly powerful, but local context and human interpretation are non-negotiable. While they are powerful, they do not replace the need for human domain of knowledge and expertise.
Complement, not a Replacement. Gen-AI and NLP algorithms should be seen as an assistive tool, not a replacement. They just make you more efficient and productive. While it ensures consistency and transparency in the method of analysis employed for qualitative data, you must use them wisely with the inclusion a human process validation of their output.
Need for Caution. AI produces inaccurate information (hallucination) and bias due to the content of the training set. Therefore, human knowledge is vital for reducing bias and expert input is a necessary part of the qualitative analysis process. It is a good practice to always ensure a layer of human oversight to guarantee the output make sense to the context.
Importance of Crafting Tailored Prompts. Each NLP algorithm serves a specific purpose. In our case, extractive summarization was ideal for highlighting the most repeated Small Doable Actions. It’s crucial to understand the analysis goal before choosing the method and tool. When using Gen-AI, take the time to craft well-structured prompts to ensure relevant, high-quality output. consistent prompts also make comparisons across tools more reliable.
Thanks for reading! I hope these insights spark some ideas on how to integrate AI into your own work. Feel free to reach out or comment with your experiences using AI in evaluation.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.