Skip to main content

Bulk catalog ingestion

Bulk catalog ingestion is a guided workflow for onboarding multiple repositories into your Portal catalog at once. It supports multiple source control providers (GitHub, GitLab, Azure DevOps) and offers two management modes: Portal-managed or YAML-managed components. You can choose the management mode for each ingestion batch.

Catalog Builder Introduction

Prerequisites​

Before using bulk catalog ingestion, ensure at least one source control provider is configured in your Portal instance:

  • GitHub - via GitHub App
  • GitLab - via Personal Access Token
  • Azure DevOps - via Personal Access Token

If you need help configuring providers, refer to your Portal instance's integration documentation.

Accessing Bulk Catalog Ingestion​

You can open bulk catalog ingestion from two locations:

  1. Sidebar - Click "Create" in the left sidebar, then select "Catalog ingestion"
  2. Catalog Page - Click the "Import" button at the top of the catalog page

Catalog Builder Sidebar

Step 1: Select Your Provider​

If you have multiple providers configured, choose which source control platform you want to pull repositories from. This determines where the workflow will fetch your organizations and repositories.

Note: If you only have one provider configured, this step is automatically skipped.

Step 2: Choose Management Mode​

Choose how component metadata will be managed for this ingestion. You'll make this choice each time you run the workflow. You have two options:

Managed by Portal​

Portal creates and manages component metadata without requiring catalog-info.yaml files. Best for quick onboarding without coordinating Pull Request approvals across teams.

Managed by YAML​

Component metadata is stored in source-controlled catalog-info.yaml files within your repositories. Portal reads these files and synchronizes changes automatically.

For a detailed comparison and help choosing the right mode, see Understanding Management Modes.

Step 3: Select Organizations​

Choose which organization(s), groups, or projects you want to pull repositories from. This step filters the list of repositories you'll see in the next step.

Note: For GitLab, this includes both groups and subgroups. For simplicity, we refer to all of these as "organizations" in the workflow.

What's Next?​

The remaining steps depend on which management mode you selected:

Managed by Portal​

Continue with Portal Mode Walkthrough to complete steps 4-6:

  • Select repositories (converted 1:1 to components)
  • Optionally fill in missing details
  • Review and complete ingestion

Managed by YAML​

Continue with YAML Mode Walkthrough to complete steps 4-6:

  • Select repositories (must have or will get catalog-info.yaml)
  • Fill in missing details for repos without YAML
  • Review and complete Pull Request creation