Skip to main content

Event-Driven Catalog Component Updates

Improve the user experience and reduce rate-limiting issues by enabling event-driven updates for catalog components in Portal.

Why Enable Event-Driven Updates?​

  • Reduced rate limiting: Fewer scheduled updates means fewer API calls, which lowers the chance of hitting GitHub's rate limits.
  • More frequent updates: Real-time notifications ensure your catalog components stay up to date.
  • Better user experience: Catalog information reflects repository changes quickly, reducing wait times for new or updated component data.

Prerequisites​

  1. Access to Portal and the Config Manager: Make sure you have permission to update configurations in Portal.
  2. Administrative access to your GitHub organization: You'll need the ability to create or manage GitHub Apps and configure webhooks for your organization.

Steps to Enable Event-Driven Updates​

1. GitHub Integration​

Set up a GitHub App (if not already done)

Create a Webhook

Webhooks allow GitHub to notify Portal when specific events happen (e.g., push, repository changes).

  • Follow GitHub's official documentation to subscribe the GitHub App to webhook events.
  • Subscribe the webhook to push and repository events.
  • It is required to secure your webhook requests by adding a webhook secret.
  • Set the Webhook URL to:
    https://<your-organization>.spotifyportal.com/api/events/http/github. Replace <your-organization> with the subdomain used by your Portal instance.

2. Configure the GitHub Events Module​

Add the Webhook Secret

  1. In Portal's Config Manager, go to the GitHub events module configuration at /config-manager/events/modules/%40backstage%2Fplugin-events-backend-module-github.
  2. Add the webhookSecret with your chosen secret (the same one you set in GitHub).
  3. Click Save changes.
  4. Start the GitHub Events Module if it's not already running by clicking the Start Module button.

Config Manager showing @backstage/plugin-events-backend-module-github with webhookSecret field

  1. Go to the GitHub Events plugin page at /config-manager/events and click the Start Plugin button.

Config Manager Events plugin page with Start Plugin button highlighted

3. (Optional) Lower the Catalog Provider's Scan Frequency​

The schedule configuration controls periodic scanning, which essentially serves as a fallback for event-driven updates. Although reducing schedule.frequency is optional, using a high scan frequency alongside webhooks can lead to unnecessary API requests and negate the many benefits of real-time, event-driven updates.

  1. In the Config Manager, locate your GitHub catalog provider settings at /config-manager/catalog#root~1.catalog~1.providers~1.github__title.
  2. Update the following configuration:
  • schedule.frequency: It's recommended to lower this (for example, to 1 day), since webhook notifications handle immediate changes.
  • schedule.timeout: There is no need to change this value but if you haven't previously set it, we recommend setting this to 1 hour.

Config Manager Catalog plugin page showing schedule.frequency and schedule.timeout fields

  • Add host and organization if you haven't previously done so:
    • host: Set this to github.com or your GitHub Enterprise hostname if you're using GitHub Enterprise.
    • organization: Your GitHub organization.
    • Click Save changes.

Config Manager Catalog plugin page showing host and organization fields

4. Verify Event-Driven Updates Are Working​

  1. Locate a Catalog Component
    • In Portal's Catalog, pick a component that is linked to a GitHub repository.
  2. Make a Change
    • In GitHub, edit the repository's catalog-info.yaml file (e.g., update metadata.description) and commit the change.
  3. Check the Catalog
    • Return to the Portal Catalog and refresh the page to see if the change is reflected.
    • You should see the updated description or other metadata changes almost immediately.
  4. Revert Changes
    • If you're only testing, remember to revert the catalog-info.yaml file to its original content.

Webhook Limitations and Fallback Behavior​

While webhooks provide real-time updates for most repository changes, there are some specific scenarios where webhook notifications may not be received:

  • Repository Transfers: When a repository is transferred between organizations or users, the webhook may not receive the transfer event. This is a limitation of GitHub's webhook system.

In these cases, the periodic scheduled catalog updates (configured in step 3) serve as a fallback mechanism. The catalog will be updated during the next scheduled scan, ensuring that all changes are eventually reflected in your software catalog, even if they weren't caught by the webhook system.

This dual approach (webhooks + scheduled updates) provides both immediate updates for most changes and eventual consistency for edge cases.

Wrapping Up​

You've successfully set up event-driven updates for your Portal catalog. This configuration provides real-time updates for your catalog components while reducing API calls and the risk of rate-limit issues.

If you encounter any issues, check your webhook configuration in GitHub and verify your webhook secret in Portal's Config Manager.