Data Experience Quick Start - Spotify for Backstage

This guide contains the minimal steps you’ll need to take to get connected with the Data Experience. The plugins needed are all running by default so they just have to be configured. More detailed information can be found for the Data Experience and each of the integrations in the sidebar.

What you’ll achieve

Ingest data warehouse tables as APIs into Portal’s Software Catalog
Make datasets searchable alongside your software components
Provide ownership and lifecycle information for data governance

Prerequisites

Admin access to Portal
Access to create credentials in your preferred integration(s)

Step 1: Configure Authentication

BigQuery
Redshift
Snowflake
Databricks

Create a GCP service account with the following roles:
- roles/bigquery.dataViewer
- roles/bigquery.jobUser
Download the JSON credentials file
Navigate to Plugins and open the Data Experience plugin
Go to Modules at the top and select Manage module on Bigquery
Add an item to the Sources list:
- Enter your GCP project ID
- Paste the service account JSON into the Credentials field
Click the Save button on both the drawer and at the top right of the config page

Create an IAM User with one of the following policies depending on your needs.

Allow access to all Redshift clusters

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "redshift:DescribeClusters",
        "redshift-data:ListSchemas",
        "redshift-data:ListDatabases",
        "redshift:GetClusterCredentialsWithIAM",
        "redshift-data:ExecuteStatement",
        "redshift-data:DescribeStatement",
        "redshift-data:GetStatementResult"
      ],
      "Resource": "*"
    }
  ]
}

Allow access to limited Redshift clusters

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "redshift:DescribeClusters",
        "redshift-data:ListSchemas",
        "redshift-data:ListDatabases",
        "redshift-data:ExecuteStatement"
      ],
      "Resource": "arn:aws:redshift:YOUR_REGION:YOUR_AWS_ACCOUNT_ID:cluster:YOUR_CLUSTER_NAME"
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": [
        "redshift:GetClusterCredentialsWithIAM"
      ],
      "Resource": "arn:aws:redshift:YOUR_REGION:YOUR_AWS_ACCOUNT_ID:dbname:YOUR_CLUSTER_NAME/*"
    },
    {
      "Sid": "VisualEditor2",
      "Effect": "Allow",
      "Action": [
        "redshift-data:DescribeStatement",
        "redshift-data:GetStatementResult"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "redshift-data:statement-owner-iam-userid": "${aws:userid}"
        }
      }
    }
  ]
}

Create an access key for this newly created IAM User.
Navigate to App Settings → Integrations and hit view on the AWS section.
Add a new item under the Account Override list
- Enter the AWS Account ID and select Custom
- Enter the IAM User’s Access key ID and Secret access key
Click the Save button on the drawer and config page
Navigate to Plugins and open the Data Experience plugin
Go to Modules at the top and select Manage module on Redshift
Add an item to the sources list:
- Enter your AWS accountId
- Expand the section below which applies to your situation and follow the instructions

I've enabled access to all Redshift clusters

Enter the AWS Account ID
Ensure Regions is selected
Add an item to the Regions list
Add the regions in which your clusters reside to the Regions list

I've enabled access to limited Redshift clusters

Enter the AWS Account ID
Ensure Clusters is selected
Add an item to the Clusters list
Add the Identifier and Region to in the drawer that pops out and save the item
Save the top level drawer with all new items

Click the Save button on both the drawer and at the top right of the config page

Follow Snowflake’s guide to generate a key-pair for authentication. Be sure to follow the instructions to assign the public key to a Snowflake user.
Navigate to Plugins and open the Data Experience plugin
Go to Modules at the top and select Manage module on Snowflake.
Under the Sources key, add a new item and fill out the former
- Enter the Warehouse that should be used for executing queries. If omitted, the user’s default warehouse will be used.
- Enter the Role that should be used for executing queries. If omitted, the user’s default role will be used.
Click the Save button on both the drawer and at the top right of the config page

Create a Service Principal in Databricks. This can be at either the account or workspace level.
This service principal must be granted the USE CATALOG and USE SCHEMA privileges on each catalog and schema you wish to ingest datasets from. See the Databricks API documentation for more details.
Generate OAuth credentials for the service principal, and take note of the client ID and secret. Be mindful of the lifetime set for these credentials and remember to rotate the secrets before they expire to prevent ingestion failures.
Navigate to Plugins and open the Data Experience plugin
Go to Modules at the top and select Manage module on Databricks
For each workspace you’d like to ingest datasets from, add a new source with the workspace URL and service principal credentials. The integration will discover all catalogs, schemas, and tables which the service principal has access to.
Click the Save button on both the drawer and at the top right of the config page

(Optional) Step 2: Configure Registry Ingest Schedule

Configure how often datasets are ingested from your sources to the data registry.

From Plugins > Data Experience, go to Frequency
Enter a set number of days, hours or minutes
Click the Save button at the top right

Step 3: Test & Verify

Wait for the first sync to complete - the first sync should happen within minutes of set up, but you can monitor progress and manually run an ingest by visiting the Data Overview page, accessible from Portal’s navigation.
Search for your datasets in Portal’s search

Next Steps

Add filters to exclude test/staging tables
Configure integrations with other data warehouses
Integrate with dbt to bring your dbt projects into the Software Catalog
Integrate with TechDocs to add documentation for your datasets
Create your first check for datasets in Soundcheck
Add news tags or labels to help discovery of your datasets with Entity Overlays

Troubleshooting

No datasets appearing? Verify service account permissions and project visibility. Consider extending the maximum entity name length in Portal
Missing owner/lifecycle? Verify your entity defaults are set to valid catalog entities

​What you’ll achieve

​Prerequisites

​Step 1: Configure Authentication

​(Optional) Step 2: Configure Registry Ingest Schedule

​Step 3: Test & Verify

​Next Steps

​Troubleshooting