Data Lineage

​

What is Data Lineage?

​

Why Data Lineage?

• “What required ancillary data must flow for this dataset to be fresh today?”
• “If I deprecate this dataset, what will it break downstream?”
• “Which workflow produces this dataset and where did it read from?”
• “If my dataset is ‘late’, where exactly is the blockage upstream?”

​

Prerequisites

• A Portal instance URL — e.g. https://<YOUR INSTANCE>.spotifyportal.com
• An External Access token — Follow the Create an API key guide to generate a static token under Admin → App Settings → External access. There is no need to provide access to any specific plugin or RBAC permission.
• Datasets ingested into the catalog (optional but recommended) — If you want lineage events to link to catalog entities, ensure your datasets have been ingested via one of the Data Experience warehouse integrations. Lineage events will still be accepted without this, but dataset nodes will appear as “unlinked” in the graph.

​

Quick Setup

1. Using the External Access token from Prerequisites, set the following environment variables:
   export OPENLINEAGE_URL="https://<YOUR INSTANCE>.spotifyportal.com"
   export OPENLINEAGE_ENDPOINT="/api/data-registry/lineage"
   export OPENLINEAGE_API_KEY="<YOUR EXTERNAL ACCESS TOKEN>"
   
2. Try a simple curl command:
   curl -v $OPENLINEAGE_URL$OPENLINEAGE_ENDPOINT \
     -X POST \
     -H "Authorization: Bearer $OPENLINEAGE_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{
             "eventType": "START",
             "eventTime": "2026-08-03T19:40:00.000Z",
             "run": { "runId": "a1b2c3d4-e5f6-7890-abcd-ef1234567421" },
             "job": {
                 "namespace": "airflow://my-airflow",
                 "name": "ATL_PIPELINE.UPDATE_AGENTS"
             },
             "inputs": [{
                 "namespace": "redshift://data-experience.us-east-1:5439",
                 "name": "dev.private.profits"
             }],
             "outputs": [{
                 "namespace": "redshift://data-experience.us-east-1:5439",
                 "name": "dev.public.agents"
             }],
             "producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
             "schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json"
         }'
   
3. If the curl response status code is 201, validate the event was received by visiting: https://<YOUR INSTANCE>.spotifyportal.com/data-overview/lineage

​

How does it work?

​

OpenLineage Run Events

​

Event type

​

Event Time

​

Namespace

​

Mapping lineage to catalog entities

​

Dataset and job naming conventions

​

Integration with your orchestrator

​

dbt

​

Apache Airflow

​

Viewing Data Lineage in Portal

​

Troubleshooting

1. The HTTP response code from POST /api/data-registry/lineage — a non-201 response indicates the event was rejected.
2. Your token configuration — ensure the External Access token is valid and included in the Authorization: Bearer <token> header.
3. Namespace and name matching — verify that the namespace and name values in your RunEvent match exactly what is registered in the catalog.

​

Authentication errors (401/403)

• Missing or invalid Bearer token: Ensure the Authorization: Bearer <token> header is present and the token value is correct.
• Token not configured: Verify that the token has been saved under Admin → App Settings → External access.
• Expired or rotated token: If the token was recently rotated, ensure you are using the new value.

​

Malformed or rejected events

• Invalid JSON: The request body is not valid JSON.
• Missing required fields: The following fields are required: eventType, eventTime, run (with runId), job (with namespace and name), producer, and schemaURL.
• Invalid eventType: Must be one of START, RUNNING, COMPLETE, ABORT, FAIL, or OTHER.
• Invalid eventTime: Must be an ISO-8601 timestamp including timezone offset, e.g. "2026-08-03T19:40:00.000Z".
• Wrong schemaURL: Should be https://openlineage.io/spec/2-0-2/OpenLineage.json.

{
  "status": "ok",
  "data": {
    "nodesUpserted": 4,
    "edgesUpserted": 3
  }
}

​

Lineage not appearing in the graph

• Namespace/name mismatch: The most common reason for missing lineage is a mismatch between the namespace or name in your RunEvent and what is registered in the catalog. Your workflow might publish bq://project/dataset/table but your catalog dataset is registered with a different URI. Verify the values match exactly against the Dataset entity page in Portal.
• Datasets not registered in the catalog: Unregistered datasets still appear in the graph but as dimmed, unlinked nodes with no entity page link. Ingest them via a Data Experience warehouse integration to get full lineage on the entity page.
• Upstream datasets not registered: Unregistered upstreams also render as unlinked nodes. Check with the owners of your upstream data to verify that their data has been ingested.
• Only START event sent: If you only send a START event without a terminal event (COMPLETE, FAIL, or ABORT), the lineage graph may not update fully. Ensure each run emits both a START and a terminal event.
• Wrong Portal instance: Verify that OPENLINEAGE_URL points to the correct Portal instance where your datasets are cataloged.
• Unlinked nodes: When Portal cannot reconcile a dataset node against the Data Registry, the node is still created but with dataset_id: null. These unlinked nodes are kept in the graph to avoid creating holes when traversing, but appear with dimmed styling in the UI. A notification is sent to the Portal instance admin when unlinked events are received.

​

Next Steps

• Data Experience Overview — Get started with the Data Experience plugin
• Create an API key — Generate an External Access token for authenticating lineage events
• OpenLineage Specification — Learn about the OpenLineage object model, run cycle, and facets
• OpenLineage Integrations — Out-of-the-box integrations for Airflow, Spark, dbt, Dagster, Flink, and more
• Ingest warehouse datasets — Connect your data warehouses via the BigQuery, Snowflake, Redshift, or Databricks integrations so that lineage events link to catalog entities