Building a Strong Foundation: Understanding Data Architectures by Jahannie Torres-Rodríguez and Joey Kleiner

Date: Sunday, January 19, 2025

Hello! We are Jahannie Torres-Rodríguez, Senior Data Intelligence Analyst and Joey Kleiner, Senior Manager of Data Engineering from VentureWell’s Data Intelligence (DI) team. VentureWell is a Massachusetts-based nonprofit that specializes in funding, training, and cultivating a pipeline of science and technology inventors, innovators, and entrepreneurs. Together with our stakeholders, we are driven to solve the world’s biggest challenges and create positive social and environmental impact.  

At VentureWell,  data is essential to our internal learning and decision-making processes. To transform data into valuable insights, it is crucial to have a well-designed data architecture in place. In this post, we explore the fundamentals of data architecture and share best practices for building systems that are scalable and adaptable.

What is data architecture?

Think of data architecture like a cooperative board game! When you open the box, it usually contains instructions, tokens, and a board that defines where things are placed. Each player has a specific role and everyone plays together to contribute towards a common goal.

Like a board game, an organization’s data architecture defines the rules, policies, standards, and models that determine how data are collected, stored, integrated, and utilized. It is a high-level framework that provides the foundation for effective data management. 

Hot Tip #1: When we’re designing a data architecture—whether starting from scratch or scaling an existing one—we start by understanding our organization’s unique needs and goals. Here’s what we focus on:

• How are the different areas of the organization related?
• What variety of data sources will we need to integrate?
• How do the data points relate to each other?
• What’s the current and projected volume of data?
• What analytical capabilities do we need?
• Who owns the data and what are the roles?
• Are there any legal or regulatory requirements we must comply with?
• What budget and resources are available?

Hot Tip #2: A robust architecture is more than just a way to store information!

In addition to describing how data are stored, data architectures also include processes and policies. Data architectures and their elements tend to vary across organizations depending on their data maturity phase. The following diagram presents a general data architecture model that illustrates how data flows from producers to consumers and highlights the elements that sustain the flow, such as metadata, master/reference data, quality, governance, and privacy.

Source: Adapted from Airbyte

Lessons Learned

• Data Governance: Consistency and quality don’t happen by chance! We’ve learned that developing clear policies, standards, and processes is essential. For instance, developing a Data Retention Standard has helped us define when data should be archived or retained, ensuring efficient and consistent management across the organization.
• Discovery Process: Spending time with data end-user stakeholders at the beginning of a data project is critical for implementing new data architectures. We’ve learned the importance of gathering information, clarifying purpose and identifying requirements in order to effectively design data architectures that are scalable and align with business goals. 
• Data Security and Privacy: Designing permission sets tailored to different user groups has been critical in protecting sensitive information. We’ve also learned the importance of documenting de-identification processes and establishing protocols for sharing data with broader audiences. 
• Automation and Data Pipelines: Manual processes are prone to inefficiencies and errors. Automating data collection and integration has allowed us to streamline workflows and enhance insights. A great example is our integration of data from the United States Patent and Trademark Office, which enables us to learn how entrepreneurial teams transfer technologies to market after participating in our programs.
• Metadata Management: Traceability and usability depend on clear documentation of stored data and its lineage. To address this, we’ve implemented field naming conventions that make it easy to identify versions and track the origins of data fields.

By focusing on these areas, we’ve built an architecture that is adaptable, secure, and aligned with organizational goals. Check out our post tomorrow on scaling with data pipelines!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.