Discover this podcast and so much more

Podcasts are free to enjoy without a subscription. We also offer ebooks, audiobooks, and so much more for just $11.99/month.

Unlocking The Power of Data Lineage In Your Platform with OpenLineage

Unlocking The Power of Data Lineage In Your Platform with OpenLineage

FromData Engineering Podcast


Unlocking The Power of Data Lineage In Your Platform with OpenLineage

FromData Engineering Podcast

ratings:
Length:
58 minutes
Released:
May 18, 2021
Format:
Podcast episode

Description

Data lineage is the common thread that ties together all of your data pipelines, workflows, and systems. In order to get a holistic understanding of your data quality, where errors are occurring, or how a report was constructed you need to track the lineage of the data from beginning to end. The complicating factor is that every framework, platform, and product has its own concepts of how to store, represent, and expose that information. In order to eliminate the wasted effort of building custom integrations every time you want to combine lineage information across systems Julien Le Dem introduced the OpenLineage specification. In this episode he explains his motivations for starting the effort, the far-reaching benefits that it can provide to the industry, and how you can start integrating it into your data platform today. This is an excellent conversation about how competing companies can still find mutual benefit in co-operating on open standards.
Released:
May 18, 2021
Format:
Podcast episode

Titles in the series (100)

Weekly deep dives on data management with the engineers and entrepreneurs who are shaping the industry