Choice of using Lambda function in the Streaming pipeline

[damolavictor]

Hello Team,

I am about starting the final lab for week 4 and when the instructor shared the architecture diagram for the streaming pipeline, I noticed the use of lambda functions for the model inference and stream transformation. I understand this is a lab but I am concerned if this is the most cost effective/efficient way to achieve this same task since lambda functions rack up costs on the number of times they are invoked and the duration they run. And we are looking at around 10,000 concurrent users(which can be higher) as the organization expands reach.

Please can you give more clarity on this maybe I am assuming wrongly and let me know why this approach suits our use case and also what possible cost-effective alternatives can be implemented to effectively achieve the same purpose.

I will share the architecture diagram for reference.

[Georgios]

Hello @damolavictor,
Yes, you are correct and thinking about cost is a task as an Engineer to be considered. That is why Joe explains about serverless and containers advantages in week 3 of the course. Later in the specialization you will see different architectures (mostly medallion) which are more cost-effective than this one. I guess lambda functions are used for simplicity just to demonstrate how to implement this type of architecture. Hope it helps

[damolavictor]

Thank you for this. Will be looking forward tot the upcoming sections to learn more about this