> ## Documentation Index
> Fetch the complete documentation index at: https://backstage.spotify.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Integration: Databricks

> Ingest dataset metadata from Databricks Unity Catalog into Portal's software catalog.

The Databricks integration enables you to ingest dataset metadata from your Databricks Unity Catalog into the Backstage Software Catalog. Databricks' legacy Hive metastore is not supported.

### Configuration & Authentication

1. Create a [Service Principal](https://docs.databricks.com/aws/en/admin/users-groups/service-principals) in Databricks. This can be at either the account or workspace level.
2. This service principal must be granted the `USE CATALOG`, `USE SCHEMA`, and `SELECT` privileges on each catalog, schema, and table (respectively) you wish to ingest datasets from. See the [Databricks API documentation](https://docs.databricks.com/api/workspace/tables/list) for more details.
3. **For tags support**: The service principal must also have access to query the `system.information_schema.table_tags` system table to retrieve tag information. This requires the `USE SCHEMA` privilege on the `system.information_schema` schema.
4. **SQL Warehouse requirement**: To retrieve table tags, you must configure a SQL Warehouse ID in your Databricks source configuration. The service principal must have `CAN_USE` permission on this warehouse.
5. Generate OAuth credentials for the service principal, and take note of the client ID and secret. Be mindful of the lifetime set for these credentials and remember to rotate the secrets before they expire to prevent ingestion failures.
6. Back in Portal, visit the Databricks configuration page in **Plugins**.
7. For each workspace you'd like to ingest datasets from, add a new source with the workspace URL, service principal credentials, and optionally a warehouse ID for tags support. The integration will discover all catalogs, schemas, and tables which the service principal has access to.

### Naming

The naming structure for Datasets created from Databricks is as follows: `[metastore_id].[catalog].[schema].[table_name]`. The metastore ID is used to ensure the uniqueness of names in the Software Catalog, but typically you will not need to be aware of it. Learn more about the [metastore ID in the Databricks documentation](https://kb.databricks.com/unity-catalog/find-your-metastore-id).

### Ingestible Assets

The Databricks integration currently ingests the following asset types:

* Tables
* Views
* Materialized Views

### Tags and Labels

The Databricks integration supports ingesting [table tags](https://docs.databricks.com/aws/en/sql/language-manual/information-schema/table_tags) from the Unity Catalog `system.information_schema.table_tags` system table. Tags are converted to labels in the Backstage Software Catalog.

#### Requirements for Tags Support

To enable tag ingestion, you must:

1. **Configure a SQL Warehouse**: Add a `warehouseId` to your Databricks source configuration in **Plugins**. This warehouse is used to execute SQL queries against the system tables.

2. **Grant warehouse permissions**: The service principal must have `CAN_USE` permission on the specified SQL Warehouse.

3. **Grant system schema access**: The service principal must have `USE SCHEMA` privilege on the `system.information_schema` schema to query the `table_tags` system table.

#### How Tags Work

* Table tags defined in Databricks Unity Catalog are retrieved using SQL queries against `system.information_schema.table_tags`
* Each tag consists of a `tag_name` and optional `tag_value`
* Tags are converted to labels in Backstage with the tag name as the key and tag value as the value
* If a tag has no value, the label value will be an empty string

If `warehouseId` is not configured, the integration will still work but will skip tag ingestion and log a warning message.

### Troubleshooting

If you are experiencing difficulties ingesting Databricks resources, verify your service principal has the necessary privileges to access the following APIs:

* [List catalogs](https://docs.databricks.com/api/workspace/catalogs/list)
* [List schemas](https://docs.databricks.com/api/workspace/schemas/list)
* [List tables](https://docs.databricks.com/api/workspace/tables/list)