OpenLineage

What OpenLineage does

OpenLineage is an open-source framework and standard API for collecting and analyzing data lineage metadata across heterogeneous data pipeline environments. The project provides a vendor-neutral specification, defined using OpenAPI, that models three core entities — datasets, jobs, and runs — and supports extensible custom metadata through a facet system. It enables organizations to track how data flows through orchestrators, processing engines, warehouses, and data quality tools, supporting use cases including root cause analysis, impact analysis, regulatory compliance documentation, and data observability. Native integrations span Apache Airflow (as a built-in Provider from Airflow 2.7.0 onward), Apache Spark, Apache Flink, dbt, Snowflake, BigQuery, Redshift, Trino, Postgres, Great Expectations, Microsoft Purview, and others, with version 1.50.0 representing the current major release line.

The technology stack consists of the OpenLineage Specification (the standard itself, licensed under Apache 2.0), Java and Python client libraries for emitting lineage events, transport backends supporting HTTP, Kafka, and filesystem targets, and the Marquez project as the reference implementation for lineage collection, aggregation, and visualization. The system supports both runtime lineage (emitted from executing pipelines) and static lineage (introduced in v1.0, modeled from code analysis for rarely executed or design-time lineage). The project is governed under the LF AI & Data Foundation as a Graduate project, ensuring structural vendor neutrality.

OpenLineage operates as a non-commercial, community-driven open-source initiative with no direct revenue model. Development is sustained by contributions from founding and active member organizations including Astronomer, Datakin, Snowflake, Microsoft, and GetInData/Xebia, with the project hosted under The Linux Foundation. Distribution occurs entirely through self-serve channels — GitHub, pip, Maven Central, and the bundled Apache Airflow Provider — with adoption driven by community engagement (Slack, GitHub, monthly TSC meetings, global meetup groups) and by data platform vendors integrating OpenLineage into their products. The primary customer base comprises data engineering and platform teams, data governance and compliance professionals, and data tool vendors who adopt the standard to provide lineage capabilities without building proprietary alternatives.

Name

OpenLineage

Legal name

OpenLineage

Website

https://openlineage.io

Company type

Private

Founded year

2021

Operating status

Operating

Headcount range

1–10 employees

Short description

OpenLineage is an open-source framework and standard API for data lineage metadata collection, governed by the LF AI & Data Foundation. It enables consistent lineage tracking across heterogeneous data tools via Java and Python clients and integrations with Airflow, Spark, dbt, and Snowflake.

Ownership category

akta.pro rank

Product category

Data Lineage and Metadata Management

NAICS

Computing Infrastructure Providers, Data Processing, Web Hosting, and Related Services (518)

SIC

Services-Prepackaged Software (7372)

akta.pro primary industry

GraphQL, gRPC & Modern API Protocols (BPAMAOAK)

Keywords

• Data lineage metadata
• Open source standard
• Metadata collection API
• Pipeline lineage tracking
• Data observability framework

Headquarters

HQ city

San Francisco

HQ country

United States

HQ region

North America

Offices1 record

Markets served

GTM type

B2B

Offering type

Software

Cost components

Technology or R&D, Personnel, Operations, Marketing or Sales

Revenue model

1. Open Source / No Direct Revenue: OpenLineage is a fully open-source project under the Apache 2.0 license, governed by the LF AI & Data Foundation. There is no direct revenue model. The project is sustained by contributions from member companies (Astronomer, Datakin, Snowflake, Microsoft, and others) and the Linux Foundation.

Go-to-market motion2 records

Distribution channels5 records

Marketing channels11 records

Core offering

OpenLineage is an open framework for data lineage collection and analysis. It provides an open standard API (OpenAPI/JSON Schema) for capturing lineage metadata events about datasets, jobs, and runs, supported by Java and Python client libraries and native integrations with data pipeline tools including Apache Airflow, Spark, Flink, dbt, Trino, Great Expectations, Snowflake, BigQuery, and Redshift. The framework enables unified cross-tool lineage graphs through an extensible facet system and supports both runtime and static lineage.

Product overview

OpenLineage is an open framework for data lineage collection and analysis, consisting of an open specification (OpenLineage Specification), client libraries in Java and Python, and integrations with data pipeline tools. The OpenLineage Specification defines a standard API for capturing lineage events from datasets, jobs, and runs. Client libraries (Java and Python) enable programmatic emission of lineage metadata. Marquez serves as the reference backend implementation for collecting, storing, and visualizing lineage data. The ecosystem includes native integrations with Apache Airflow, Spark, Flink, dbt, Great Expectations, Trino, and various data warehouses (Snowflake, BigQuery, Redshift).

Differentiator

Problem solved

Functional benefit

Products and services

• OpenLineage Specification Open standard API specification for capturing lineage metadata events. Defines a generic model of Dataset, Job, and Run entities with extensible facets for metadata enrichment, defined using OpenAPI and JSON Schema. Used by data engineers, data tool vendors, and data governance teams to unify lineage across heterogeneous data stacks.
• OpenLineage Java Client Java SDK (io.openlineage.client) for emitting lineage events from Java-based systems and Spark applications. Includes packages for circuitBreaker, dataset, metrics, run, job, and transport support, plus a server package with model classes. Distributed via Maven Central.
• OpenLineage Python Client Python SDK for creating and emitting lineage metadata events. Forms the basis for Apache Airflow and dbt integrations. Provides core data structures (RunEvent, RunState, Run, Job, Dataset, Transport) and optional dependencies for Kafka, S3/GCS/Azure, MSK IAM, and AWS DataZone. Installed via pip.
• Marquez (Reference Implementation)

Quantifiable outcome

• Northwestern Mutual deployed OpenLineage/Marquez and activated new production use cases in approximately one month
• +1 more outcomes

Named customers6 records

Segments3 records

Ideal customer profiles3 records

Technology focussed Yes

API detail

Has API

Yes

API docs

API detail

Core technology

AI maturity

App detail

Integration17 records

AI capability2 records

Feature6 records

Partnerships

Ten partnerships are on record, tiered core and minor.

• Marquez (LF AI & Data Foundation)coreStrategic or Co-development Partner · 3 September 2021Marquez is the reference implementation of the OpenLineage API, an LF AI & Data Foundation project that collects, aggregates, and visualizes a data ecosystem's metadata. OpenLineage includes Marquez as its reference backend and the OpenLineage community collaborates directly with Marquez on API compatibility and joint releases. Marquez ships an OpenLineageResource that implements the OpenLineage API.
• Apache AirflowcoreTechnology or Integration · 3 September 2021Apache Airflow is the most prominent integration for OpenLineage. Starting with Airflow 2.7.0, the OpenLineage integration became a native Airflow Provider (apache-airflow-providers-openlineage) bundled directly in Airflow. Supports BigQuery, Great Expectations, Postgres, Redshift, Snowflake, and custom extractors.
• Apache SparkcoreTechnology or Integration · 3 September 2021Apache Spark is a core integration target for OpenLineage. The openlineage-spark integration (io.openlineage:openlineage-spark) captures lineage from Spark applications via a SparkListener interface, supporting Spark 2.4 through 3.x. Supports BigQuery, Iceberg, Delta Lake, and various file systems (S3, GCS, Azure Blob).
• dbt (dbt Labs)coreTechnology or Integration · 3 September 2021dbt integration is a core part of OpenLineage's initial release. The dbt-ol wrapper analyzes dbt's catalog.json, run_results.json, and manifest.json files to emit OpenLineage events to a backend. Supports extracting test severity information from dbt tests.
• SnowflakecoreTechnology or IntegrationSnowflake integrated OpenLineage via the Access History View, enabling enterprise users to consume table lineage and process lineage in consolidated OpenLineage format. The integration was developed by Snowflake with contributions from Datakin. Snowflake's Data Cloud serves 5,900+ companies including 241 Fortune 500 companies.
• Microsoft (Azure Databricks, Microsoft Purview)coreTechnology or IntegrationMicrosoft collaborated with OpenLineage to enable lineage extraction from Azure Databricks Spark workloads to Microsoft Purview. The Azure Databricks to Microsoft Purview Solution Accelerator uses the OpenLineage Spark integration and translates events into Purview native assets. Microsoft contributed eight pull requests to OpenLineage for Azure Blob File System support, Azure Function endpoints, and Azure Synapse lineage extraction.
• AstronomercoreStrategic or Co-development PartnerAstronomer is a founding contributor and key sponsor of OpenLineage. The company hosts OpenLineage community meetups at their offices in San Francisco and sponsored conference talks. Astronomer's Astro platform integrates OpenLineage and the company employs the OpenLineage Community Manager.
• DatakincoreStrategic or Co-development PartnerDatakin is a founding contributor and co-designer of OpenLineage. Senior engineers from Datakin (including Julien Le Dem as OpenLineage Project Lead, Minkyu Park) authored key integrations including the initial specification and Snowflake integration. Datakin built the OpenLineage standard into their data observability platform.
• GetInData/XebiaminorStrategic or Co-development PartnerGetInData/Xebia is an active contributor to OpenLineage, sponsoring meetups and contributing to integrations. Committers from GetInData/Xebia (Maciej Obuchowski, Jakub Dardziński, Kacper Muda) have contributed to Apache Flink, Spark, Airflow, and Python client development.
• Apache FlinkcoreTechnology or IntegrationApache Flink is a major integration target. OpenLineage events from Flink jobs have been supported for two years. Recent joint efforts between Flink and OpenLineage communities brought native integration, including Flink 2 job status changed listener and seamless OpenLineage & Apache Flink integration.

Scale indicators6 records

Recent moves8 records

Expansion highlights5 records

Direct peers

• DataHub: DataHub is an open-source metadata platform originated at LinkedIn that supports lineage, discovery, and governance. It has adopted OpenLineage for runtime lineage, making it both a complementary consumer and a competing open metadata platform.
• OpenMetadata: OpenMetadata is an open-source unified metadata, lineage, and data quality platform. It competes with OpenLineage for the open-source metadata standard mindshare while also integrating with similar data stack tools.
• Apache Atlas: Apache Atlas is an open-source metadata governance and lineage framework under the Apache Foundation. It addresses the same data lineage and metadata management problem as OpenLineage but with a heavier governance-focused model rather than an API-first standard.
• Datakin: Datakin is a founding contributor and co-designer of OpenLineage that builds a commercial data observability platform on top of the OpenLineage specification. It is the closest commercial implementation directly extending OpenLineage.
• Marquez: Marquez is the reference backend implementation of the OpenLineage API, also hosted under LF AI & Data. It is jointly developed with OpenLineage and represents the canonical open-source visualization and metadata store for OpenLineage events.

Emerging players

• Amundsen: Amundsen is an open-source data discovery and metadata platform originated at Lyft, focused on data catalog and search. It addresses adjacent metadata needs with weaker lineage depth compared to OpenLineage.
• Great Expectations: Great Expectations is an open-source data quality framework that integrates with OpenLineage via the OpenLineageValidationAction. It is a complementary ecosystem partner that emits data quality validation results as OpenLineage facets.
• Monte Carlo Data: Monte Carlo is a commercial data observability platform that provides lineage, freshness, and data quality monitoring. It is a competing commercial implementation targeting the same use cases OpenLineage enables (root cause, impact analysis).

Broad incumbents

• Collibra: Collibra is a leading enterprise data catalog and governance platform that integrates with OpenLineage for static lineage. It represents the incumbent commercial data governance stack that OpenLineage partly complements and partly competes with.
• Microsoft Purview: Microsoft Purview is Microsoft's enterprise data governance and catalog platform. It consumes OpenLineage events from Azure Databricks via a Solution Accelerator, making it both a major consumer and a competing lineage ecosystem.

Market position

Strengths4 records

Weaknesses4 records

Competitive moat5 records

Key risks5 records

Key highlights7 records

Customer concentration

• Xx.com/OpenLineage
• GitHubgithub.com/OpenLineage/OpenLineage
• YouTubeyoutube.com/@openlineageproject6897/videos
• LinkedInlinkedin.com/company/openlineage

Compliance1 record

Revenue estimate

Valuation estimate

Number of profiles

Profiles1 record

Funding overview

Funding rounds

Investors

Funding detail is available on the Subscription and Enterprise plan.Contact sales →

M&A

Investments

M&A and investment is available on the Subscription and Enterprise plan.Contact sales →

Frequently asked questions about OpenLineage