Scaling with Data Pipelines by Jahannie Torres-Rodríguez and Harrison Sharitt

Date: Monday, January 20, 2025

Hello! We are Jahannie Torres-Rodríguez, Senior Data Intelligence Analyst, and Harrison Sharitt, Data Engineer from VentureWell’s Data Intelligence (DI) team. VentureWell is a Massachusetts-based nonprofit that specializes in funding, training, and cultivating a pipeline of science and technology inventors, innovators, and entrepreneurs. Together with our stakeholders, we are driven to solve the world’s biggest challenges and create positive social and environmental impact.  

Driving organizational learning often requires integrating data from various sources. This approach enhances context and generates valuable insights. One beneficial method for evaluators and data professionals is the use of data pipelines. In this post, we explore what data pipelines are, how they function, and why they are essential for fostering continuous improvement and learning within organizations.

What are data pipelines?

Data pipeline is a term frequently used when talking about integrating external data sources into internal tools such as Customer Relationship Management (CRM) systems or analytics platforms. Think of it as the organizational equivalent of a water pipeline! 

Before the water reaches our homes, buildings, or land, it goes through several steps. It starts in water sources such as lakes, rivers, or the ocean. When extracted, it goes through treatments to improve its quality, remove components, and guarantee it is suitable for specific uses. Finally, it is distributed for drinking, cleaning, irrigation, and other purposes.

Data pipelines follow a similar process:

1. Source identification: Identify reliable sources with relevant data.
2. Transformation: Clean the data to filter or remove irrelevant or inaccurate elements.
3. Usage: Incorporate the processed data into the organization’s system and use it to support learning.

Hot Tip #1: Identifying trustworthy data sources starts with evaluating these key aspects:

• Permissions: Confirm that you have the rights and necessary permissions to use the data for your intended purposes.
• Quality: Assess the data’s accuracy, completeness, and reliability. Ensure it meets your organization’s standards. 
• Frequency: Determine how often the data source is updated and ensure it aligns with your organization’s needs.
• Format: Understand the format in which the data are available (e.g., CSV files, Application Programming Interface (API), or specialized formats like .dta) and the limitations. Choose formats that are compatible with your systems and workflows.
• Methodology: Verify if the source provides sufficient documentation on how data are collected and organized.

Hot Tip #2: External data sources rarely come in the perfect format. When building a pipeline, transform the data to align with your tools and infrastructure. Common transformations include:

• Filtering: Remove irrelevant or unnecessary data.
• Reformatting: Adjust formats to meet system requirements, such as converting date formats, addresses, or numerical precision.
• Matching: Link incoming data with existing records, such as connecting external IDs with internal records.
• Deduplication: Identify and remove duplicate records from the incoming data.
• Existing data: Identify and exclude data for records that are already present in your systems, or use the new data to update existing records 
• Validation: Ensure data is accurate and complete by cross-checking against internal standards.

Why are data pipelines essential for driving organizational learning?

Whether you’re building the pipelines or utilizing the data they provide, it is important to understand why they matter.

• Saves time: Data collection and processing can be labor-intensive; automating these tasks through data pipelines significantly reduces the manual workload.
• Ensures consistency and quality: Automated pipelines enforce standardization across datasets, ensuring consistency in formatting, validation, and integration. 
• Enables long-term insights: For organizations aiming to track outcomes over time, pipelines provide a sustainable method of gathering and integrating data without completely relying on methods like survey administration.

Check out our post tomorrow on reducing manual work with automations!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.