> ## Documentation Index > Fetch the complete documentation index at: https://backstage.spotify.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Data Experience Quick Start > Minimal steps to connect a data warehouse and start ingesting dataset metadata into Portal. This guide contains the minimal steps you'll need to take to get connected with the Data Experience. The plugins needed are all running by default so they just have to be configured. More detailed information can be found for the Data Experience and each of the integrations in the sidebar. ## What you'll achieve * Ingest data warehouse tables as APIs into Portal's Software Catalog * Make datasets searchable alongside your software components * Provide ownership and lifecycle information for data governance ## Prerequisites * Admin access to Portal * Access to create credentials in your preferred integration(s) ## Step 1: Configure Authentication 1. Create a GCP service account with the following roles: * `roles/bigquery.dataViewer` * `roles/bigquery.jobUser` 2. Download the JSON credentials file 3. Navigate to **Plugins** and open the **Data Experience** plugin 4. Expand the keys on the sidebar `dataExperience` > `registry` > `integrations` > `bigquery` 5. Add an item to the `sources` list: * Enter your GCP project ID * Paste the service account JSON into the `credentials` field 6. Scroll to the bottom of the page and click the **Save changes** button 1. Create an IAM User with one of the following policies depending on your needs. ```json theme={"theme":{"light":"github-light","dark":"dracula"}} { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "redshift:DescribeClusters", "redshift-data:ListSchemas", "redshift-data:ListDatabases", "redshift:GetClusterCredentialsWithIAM", "redshift-data:ExecuteStatement", "redshift-data:DescribeStatement", "redshift-data:GetStatementResult" ], "Resource": "*" } ] } ``` ```json theme={"theme":{"light":"github-light","dark":"dracula"}} { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "redshift:DescribeClusters", "redshift-data:ListSchemas", "redshift-data:ListDatabases", "redshift-data:ExecuteStatement" ], "Resource": "arn:aws:redshift:YOUR_REGION:YOUR_AWS_ACCOUNT_ID:cluster:YOUR_CLUSTER_NAME" }, { "Sid": "VisualEditor1", "Effect": "Allow", "Action": [ "redshift:GetClusterCredentialsWithIAM" ], "Resource": "arn:aws:redshift:YOUR_REGION:YOUR_AWS_ACCOUNT_ID:dbname:YOUR_CLUSTER_NAME/*" }, { "Sid": "VisualEditor2", "Effect": "Allow", "Action": [ "redshift-data:DescribeStatement", "redshift-data:GetStatementResult" ], "Resource": "*", "Condition": { "StringEquals": { "redshift-data:statement-owner-iam-userid": "${aws:userid}" } } } ] } ``` 2. Create an access key for this newly created IAM User. 3. Navigate to **App Settings** → **Integrations** and expand the **AWS** section. 4. Add a new item under the `accounts` list * Enter the AWS `accountId` * Enter the IAM User's `accessKeyId` and `secretAccessKey` * The remaining fields may remain blank. 5. Scroll to the bottom of the page and click the **Save changes** button 6. Navigate to **Plugins** and open the **Data Experience** plugin 7. Expand the keys on the sidebar `dataExperience` > `registry` > `integrations` > `redshift` 8. Add an item to the `sources` list: * Enter your AWS `accountId` * Expand the section below which applies to your situation and follow the instructions * Ensure the **Option 1** tab is selected * Add an item to the `sources` list * Enter the AWS `accountId` * Add the regions in which your clusters reside to the `regions` list * Ensure the **Option 2** tab is selected * Add an item to the `sources` list * Enter the AWS `accountId` * Add the clusters you've granted access to in the `clusters` list * Enter the cluster `identifier` and `region` for each cluster 9. Scroll to the bottom of the page and click the **Save changes** button 1. Follow [Snowflake's guide](https://docs.snowflake.com/en/user-guide/key-pair-auth) to generate a key-pair for authentication. Be sure to follow the instructions to [assign the public key to a Snowflake user](https://docs.snowflake.com/en/user-guide/key-pair-auth#assign-the-public-key-to-a-snowflake-user). 2. Navigate to **Plugins** and open the **Data Experience** plugin 1. Expand the keys on the sidebar `dataExperience` > `registry` > `integrations` > `snowflake` 1. Under the `sources` key, select the **Option 2** tab and add a new item to the list - Enter `SNOWFLAKE_JWT` in the `authenticator` field - Enter the `username` of the user the public key was assigned to - Enter the `privateKey` - Enter the `warehouse` that should be used for executing queries. If omitted, the user's default warehouse will be used. - Enter the `role` that should be used for executing queries. If omitted, the user's default role will be used. 1. Scroll to the bottom of the page and click the **Save changes** button 1. Create a [Service Principal](https://docs.databricks.com/aws/en/admin/users-groups/service-principals) in Databricks. This can be at either the account or workspace level. 2. This service principal must be granted the `USE CATALOG` and `USE SCHEMA` privileges on each catalog and schema you wish to ingest datasets from. See the [Databricks API documentation](https://docs.databricks.com/api/workspace/tables/list) for more details. 3. Generate OAuth credentials for the service principal, and take note of the client ID and secret. Be mindful of the lifetime set for these credentials and remember to rotate the secrets before they expire to prevent ingestion failures. 4. Back in Portal, navigate to **Plugins** > **Data Experience** > **Databricks** 5. For each workspace you'd like to ingest datasets from, add a new source with the workspace URL and service principal credentials. The integration will discover all catalogs, schemas, and tables which the service principal has access to. ## (Optional) Step 2: Configure Registry Ingest Schedule Configure how often datasets are ingested from your sources to the data registry. 1. From **Plugins** > **Data Experience**, expand the keys on the sidebar `dataExperience` > `registry` > `integrations` > `defaults` > `schedule` > `frequency` > `cron` 2. Enter a valid crontab string. This is `0 */6 * * *` (every 6 hours) by default 3. Scroll to the bottom of the page and click the **Save changes** button ## Step 3: Test & Verify 1. Wait for the first sync to complete - when this happens will depend on how you've configured your schedules in steps 3 and 4. You can monitor progress by visiting the **Data Overview** page, accessibile from Portal's navigation. 2. Search for your datasets in Portal's search ## Next Steps * [Add filters to exclude test/staging tables](./configuration#filters) * Configure integrations with other data warehouses * [Integrate with dbt to bring your dbt projects into the Software Catalog](./integration-dbt) * [Integrate with TechDocs to add documentation for your datasets](./techdocs) * [Create your first check for datasets in Soundcheck](../../core-features-and-plugins/soundcheck) * [Add news tags or labels to help discovery of your datasets with Entity Overlays](../../core-features-and-plugins/entity-overlays) ## Troubleshooting * **No datasets appearing?** Verify service account permissions and project visibility. Consider extending the maximum entity name length in Portal * **Missing owner/lifecycle?** Verify your [entity defaults](./configuration#defaults) are set to valid catalog entities