Summary
Batch vs. streaming is a long running debate in the world of data integration and transformation. Proponents of the streaming paradigm argue that stream processing engines can easily handle batched workloads, but the reverse isn't true. The batch world has been the default for years because of the complexities of running a reliable streaming system at scale. In order to remove that barrier, the team at Estuary have built the Gazette and Flow systems from the ground up to resolve the pain points of other streaming engines, while providing an intuitive interface for data and application engineers to build their streaming workflows. In this episode David Yaffe and Johnny Graettinger share the story behind the business and technology and how you can start using it today to build a real-time data lake without all of the headache.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. Sign up free at dataengineeringpodcast.com/rudderstack (https://www.dataengineeringpodcast.com/rudderstack)
Your host is Tobias Macey and today I'm interviewing David Yaffe and Johnny Graettinger about using streaming data to build a real-time data lake and how Estuary gives you a single path to integrating and transforming your various sources
Interview
Introduction
How did you get involved in the area of data management?
Can you describe what Estuary is and the story behind it?
Stream processing technologies have been around for around a decade. How would you characterize the current state of the ecosystem?
What was missing in the ecosystem of streaming engines that motivated you to create a new one from scratch?
With the growth in tools that are focused on batch-oriented data integration and transformation, what are the reasons that an organization should still invest in streaming?
What is the comparative level of difficulty and support for these disparate paradigms?
What is the impact of continuous data flows on dags/orchestration of transforms?
What role do modern table formats have on the viability of real-time data lakes?
Can you describe the architecture of your Flow platform?
What are the core capabilities that you are optimizing for in its design?
What is involved in getting Flow/Estuary deployed and integrated with an organization's data systems?
What does the workflow look like for a team using Estuary?
How does it impact the overall system architecture for a data platform as compared to other prevalent paradigms?
How do you manage the translation of poll vs. push availability and best practices for API and other non-CDC sources?
What are the most interesting, innovative, or unexpected ways that you have seen Estuary used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Estuary?
When is Estuary the wrong choice?
What do you have planned for the future of Estuary?
Contact Info
Dave Y (mailto:dave@estuary.dev)
Johnny G (mailto:johnny@estuary.dev)
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ (https://www.pythonpodcast.com) covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast (https://www.themachinelearningpodcast.com) helps you go from idea to production with machine learning.
Visit the site (https://www.dataengineeringpodcast.com) to subscrib