Last year, in one use case, I was asked to implement a near real-time pipeline in AWS. I attempted to build a landing area in Redshift using materialized views with auto-updates triggered by changes, reading directly from the Kinesis data stream.
From my experience, this approach resulted in significant processing overhead, and the clients immediately shut it down due to the high costs. Personally, I didn’t like the lack of control over the quality of ingestion. I definitely plan to learn more about this topic.
For this specific use case, in the end, we opted for a refined ELT approach, ensuring dashboard updates within five minutes.
Hi Mattia, thanks for sharing this great experience. I can imagine that auto-updates can be indeed very costly. Instead of reading directly from the Kinesis data stream, configuring the Kinesis Firehose to micro-batch may also be an option. Have you considered Firehose at that time? Would love to learn from your experience ☺️
Last year, in one use case, I was asked to implement a near real-time pipeline in AWS. I attempted to build a landing area in Redshift using materialized views with auto-updates triggered by changes, reading directly from the Kinesis data stream.
From my experience, this approach resulted in significant processing overhead, and the clients immediately shut it down due to the high costs. Personally, I didn’t like the lack of control over the quality of ingestion. I definitely plan to learn more about this topic.
For this specific use case, in the end, we opted for a refined ELT approach, ensuring dashboard updates within five minutes.
Hi Mattia, thanks for sharing this great experience. I can imagine that auto-updates can be indeed very costly. Instead of reading directly from the Kinesis data stream, configuring the Kinesis Firehose to micro-batch may also be an option. Have you considered Firehose at that time? Would love to learn from your experience ☺️
Great write, I like the practical aspects, I recently explored the same idea in my newsletter “The Data Modernisation Playbook”, can be read here: https://open.substack.com/pub/thedatamodernisationplaybook/p/003-zero-etl-the-future-of-data-integration?r=1tj5ll&utm_medium=ios
Hi Muhammad ,
Thank you so much for you comment ,
And thanks for sharing your article
🙏💐