Third week notes on introduction to DE
In this week we learned how to build data systems that follows the principles of a good architecture, we went from following good practices to evaluating the different types of architecture while we consider building systems that answers our client needs.
The fundamentals of this week were the principles of good architecture. I liked how Joe grouped the principles in these 3 blocks:
How data architect impacts other teams and individuals?
Choose common components: common components allow teams to work closely and in a collaborative environment, as well as simplifying the complexity of the architecture.
Architecture is leadership: as a Data Engineer you advocate for the implementation of guiding principles and good architecture, like training teams in common components and guiding them towards the adoption of best practices.
Data architecture is an ongoing process
Always be architecting: A continuos improvement process to meet the current and future needs.
Making reversible decisions: So, to keep moving forward, is always good thinking about the kind of decisions you can make: one-way or two-way decisions. In the first one there is not turning back for you, while in the two-way decisions you can try it, and if wasn’t what you expected, you can come back and move forward in a new direction.
Design loosely-coupled systems: This allows you to gain the flexibility to change the components broken or non-updated when is needed without changing the whole system. This facilitate the maintenance processes and enhance collaboration through API interactions between components.
The unspoken topics, but understood priority
Plan for failure: Everything fails all the time, so it’s crucial to be prepared to handle them. This involves focusing on concepts like high availability, reliability, durability. Additionally in terms of recovery plans, it’s essential to understand the business’s tolerance level by identifying the RTP and RPO.
Prioritize security: Embrace a zero-trust approach. This means no one and nothing is trusted by default.
Embrace FinOps: Optimize your resources by tracking your costs to make better decisions.
Architect for scalability: The market is dynamic, so be prepared by making your architectures scalable to meet the demand needs when is required, and reducing the capacity when the peaks are over. In this way you only pay for what you need, and learn how to handle the capacity behavior of your applications and systems.
Those principles and the considerations to choose the right tools are the foundation to build good architectures.
Regarding the considerations for choosing the right technologies, trade-offs, opportunity costs and current solutions/capabilities are a good mindset to choose between servers, containers or serverless options, build vs buy, or keep monolithic systems or evolve toward modular systems. All of these decisions supported in their respective cost analysis (TCO).
Amazing insight @irinavelez thanks for sharing!
Now that you got the principles, how would you approach a project by an individual that have an idea and need you to help building the infrastructure for the database? would you discuss this principles with him even if he doesn’t have knowledge about data engineering? or would you build this database using the principles even if it wasn’t a requirement?
Hi, I think it’s relevant to revisit the data engineering framework “Think like a Data Engineer”. First, it’s important to understand the purpose behind this database, and the specific needs of the stakeholders. Since, it’s a database, the requirement might come from an upstream stakeholder (such as a source system owner), or functional stakeholders.
Understanding the purpose behind, the principles can serve as guiding lights for building the appropriate architecture. These principles can even be part of the conversation without needing to explicitly refer to them as principles.
This is an amazing insight! I think using the “Think like a Data Engineer” path helps a lot to understand what business owner need and how we should provide the information to get the best result.
I like how you get into the discussion on the things we should incorporate without making a reference as principles, I think this can help to have a friendly conversation and still addressing the things that are important for us as data engineer!
Thank you for sharing!