- Module # Module-1
- Link to the classroom item you are referring to: https://www.coursera.org/learn/data-storage-and-queries/lecture/I3YUr/amazon-redshift-cloud-data-warehouse
- Description
After completing the query course, I can say I appreciate all the lab setup grunt work done by the team with the given complexities that AWS requires to get things going, I wouldn’t really had the time or the will to do it myself even if I wanted. Even then it makes the labs challenging sometimes to complete fully in allocated time.
Having said that I was curious to see how much AWS DWH ecosystem has evolved over the past years (I used Redshift in 2015 last time). To my surprise, most of the complexities of managing Redshift that were required back then are still applicable today. Also adding Athena and all other extras makes things even more complicated.
Compared to something like BigQuery (probably similar Snowflake but I don’t have experience with it) I don’t see really a point of having all the moving parts that AWS DLH architecture benefits. The only argument I heard in the lessons was “having to sync data back” or “storage cost reduction”, those points are not valid as long-term storage cost on BigQuery are same as for S3 buckets and loading doesn’t cost anything. Don’t get me wrong I’m not arguing for using BigQuery, but I’m just trying to make sense who wins if choosing AWS DLH or Redshift, I only see it as a necessity if someone is locked in AWS, but even then the extra complexities on managing it seems to be too much to just sync data to other clouds DWH with less overhead.
Would be great to hear from someone having experience in several cloud/open source options.
Also would be interesting to see similar courses that are cloud agnostic or rather using Open source tools, but I guess that wouldn’t happen as there wouldn’t be any sponsors.