In this final week we acquire the building blocks of implementing a system dedicated to solve a problem and provide a solution to stakeholders
The first course is over and even if you don’t fully understand the implementation part it is ok, just keep on the next courses as this was more theoretical to help you understand the terminology and behind the scenes on what an actual data engineering project looks like.
Link to my notes: Data-Engineering/Introduction-to-data-engineering/week4 at main · sotoblanco/Data-Engineering
1 Like
Hello @pastorsoto
Thank you for the summary of the last week of the first course. One of the key highlights for me was the framework to gather requirements. This methodology is really useful to meet the stakeholders’ needs aligned to business goals.
Developing the ability to differentiate between functional and non-functional requirements is key to building high quality data products. This has been one of the most difficult parts for me to understand.
Functional Requirements are about what the system is able to do. I like to think of them as features that allow the system to perform its main purpose. In our example these would be: the ability to recommend new products to clients based on their behavior in less than 1 second, and present organized hourly data to marketing team. Another functional requirement is ensuring the format needed by the downstream stakeholders.
Non-functional Requirements are related to the characteristics and attributes of the system to support the functional requirements. These kinds of requirements are typically related to latency, reliability, high availability, cost, scalability. And I think this is where tradeoffs in selecting the right tools are evaluated, since there are multiple paths to solve the same problem. The underlying technology should be be able to meet the functional requirements and stakeholders needs.
Great insight @irinavelez I think having those thoughts in mind when learning help to extrapolate to the real-world and be able to use in different situations.
Which one do you think is the most difficult to build, functional requirements or non-functional requirements? is there situations when security doesn’t play a huge role in building your system?
