HTTP - Spotify for Backstage

The HTTP integration plugin for Soundcheck supports the collection of the facts from HTTP endpoints. The HTTP fact collector is intentionally simple and should be used for simple use cases. It doesn’t support pagination and any custom logic besides a simple transformation of the HTTP response body. In case you have a more complex use-case, consider implementing a custom integration.

Prerequisites

Configure HTTP secrets in Backstage

If your endpoints use a token or an API key authentication (or any other secret data), you may configure the secrets at the root level of app-config.yaml. Here’s an example configuration:

soundcheck:
  collectors:
    http:
      secrets:
        - key: grafanaApiToken
          value: ${API_TOKEN}

Referencing a secret in an HTTP header template (e.g. Authorization: Bearer {{ secrets.grafanaApiToken }}) will send the secret value to the target server. Make sure the request URL points to a trusted endpoint before including any secrets in the request.

Configure reusable config values in Backstage

If your endpoints need non-secret configurable values you can reuse across templates, you may configure them at the root level of app-config.yaml. Here’s an example configuration:

soundcheck:
  collectors:
    http:
      config:
        - key: title
          value: Soundcheck

The config template variable previously exposed the entire Backstage app-config.yaml (e.g. {{ config.app.title }}). It now only exposes the key-value pairs defined under soundcheck.collectors.http.config. Existing templates that reference other app-config.yaml fields will need to be migrated to use explicitly declared config entries.

Add the HttpFactCollector to Soundcheck

First, add the @spotify/backstage-plugin-soundcheck-backend-module-http package:

yarn workspace backend add @spotify/backstage-plugin-soundcheck-backend-module-http

Then add the following to your packages/backend/src/index.ts file:

packages/backend/src/index.ts

const backend = createBackend();

backend.add(import('@spotify/backstage-plugin-soundcheck-backend'));
backend.add(import('@spotify/backstage-plugin-soundcheck-backend-module-http'));
// ...

backend.start();

Consult the Soundcheck Backend documentation for additional details on setting up the Soundcheck backend.

Plugin Configuration

The collection of HTTP facts is driven by configuration. To learn more about the configuration, consult the Defining HTTP Fact Collectors section. HTTP Fact Collector can be configured via YAML or No-Code UI. If you configure it via both YAML and No-Code UI, the configurations will be merged. It’s preferable to choose a single source for the Fact Collectors configuration (either No-Code UI or YAML) to avoid confusing merge results.

No-Code UI Configuration Option

Make sure the prerequisite Configure HTTP secrets in Backstage is completed if your endpoints require any secret data.
Make sure the prerequisite Configure reusable config values in Backstage is completed if your endpoints use reusable config values in templates.
To enable the HTTP Integration, go to Soundcheck > Integrations > HTTP and click the Configure button. To learn more about the No-Code UI config, see the Configuring a fact collector (integration) via the no-code UI.

YAML Configuration Option

Create a http-facts-collectors.yaml file in the root of your Backstage repository and fill in all your HTTP Fact Collectors. A simple example HTTP fact collector is listed below.

- factName: grafana_dashboard
  request:
    method: GET
    url: http://grafana.com/api/dashboards/uid/{{ entity.metadata.annotations['grafana.com/dashboard-uid'] }}
    headers:
      Accept: application/json
      Content-Type: application/json
      Authorization: Bearer {{ secrets.grafanaApiToken }}
  response:
    transform: dashboard
    emptyOnStatuses: [404]
    schema: |
      {
        "title": "Grafana Dashboard",
        "description": "Grafana Dashboard",
        "type": "object",
        "properties": {
          "id": {
            "type": "string"
          },
          "uid": {
            "type": "string"
          },
          "title": {
            "type": "string"
          },
          "tags": {
            "type": "array",
            "items": {
              "type": "string"
            }
          }
        }
      }
  retries:
    maxRetries: 3
    retryDelay: 5000
    retryOnStatuses: [408, 502, 503, 504]
- factName: owners_failed_check_results
  filter:
    kind: 'Component'
    spec.type: 'service'
  request:
    targetPluginId: soundcheck
    method: GET
    url: '{{ targetPluginBaseUrl }}/results?entityRef={{ entity.relations | where: "type", "ownedBy" | first | map: "targetRef" }}'
    headers:
      Authorization: Bearer {{ targetPluginAuthToken }}
  response:
    transform: results[state='failed']
    emptyOnStatuses: [404]
    schema: |
      {
        "title": "Check Results",
        "description": "Soundcheck Check Results",
        "type": "object",
        "properties": {
          "results": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "entityRef": {
                  "type": "string"
                },
                "checkId": {
                  "type": "string"
                },
                "scope": {
                  "type": "string"
                },
                "state": {
                  "type": "string"
                },
                "timestamp": {
                  "type": "string"
                }
              }
            }
          }
        }
      }

Note: this file will be loaded at runtime along with the rest of your Backstage configuration files, so make sure it’s available in deployed environments in the same way as your app-config.yaml files.

Add a Soundcheck collects field to the app-config.yaml and reference the newly created http-facts-collectors.yaml file.

# app-config.yaml
soundcheck:
  collectors:
    http:
      collects:
        $include: ./http-facts-collectors.yaml

Defining HTTP Fact Collectors

This section describes the data shape and semantics of HTTP Fact Collectors.

Shape Of HTTP Fact Collector

The following is an example of HTTP Fact Collector YAML configuration:

soundcheck:
  collectors:
    http:
      collects:
        - factName: grafana_dashboard
          request:
            method: GET
            url: http://grafana.com/api/dashboards/uid/{{ entity.metadata.annotations['grafana.com/dashboard-uid'] }}
            headers:
              Accept: application/json
              Content-Type: application/json
              Authorization: Bearer {{ secrets.grafanaApiToken }}
          response:
            transform: dashboard
            emptyOnStatuses: [404]
            schema: |
              {
                "title": "Grafana Dashboard",
                "description": "Grafana Dashboard",
                "type": "object",
                "properties": {
                  "id": {
                    "type": "string"
                  },
                  "uid": {
                    "type": "string"
                  },
                  "title": {
                    "type": "string"
                  },
                  "tags": {
                    "type": "array",
                    "items": {
                      "type": "string"
                    }
                  }
                }
              }
          retries:
            maxRetries: 3
            retryDelay: 5000
            retryOnStatuses: [408, 502, 503, 504]
          rateLimit:
            duration: 60000
            max: 30
          filter:
            kind: 'Component'
            spec.type: 'service'
          cache: false
          cacheFullResultsForEtags: false

Below are the details for each field.

`factName` [required]

The name of the fact to be collected.

Minimum length of 1
Maximum length of 100
Alphanumeric with single separator instances of periods, dashes, underscores, or forward slashes

`request` [required]

HTTP request configuration:

targetPluginId [optional] - Optional, provide if calling a plugin’s HTTP endpoint. If provided, liquid template tags {{ targetPluginBaseUrl }} and {{ targetPluginAuthToken }} will be populated with the plugin’s data.
method [optional] - HTTP method (GET or POST). Optional, if not provided the method will be defaulted to GET.
url [required] - HTTP request URL. Supports liquid templating. The following variables are available for use in templates:
- entity - The entity the fact is collected for. Usage example: {{ entity.metadata.name }}
- config - The reusable config values. Usage example: {{ config.title }}
- targetPluginBaseUrl - Base URL of the plugin with ID provided in targetPluginId. Usage example: {{ targetPluginBaseUrl }}
headers [optional] - HTTP request headers. Header keys and values support liquid templating. The following variables are available for use in templates:
- entity - The entity the fact is collected for. Usage example: {{ entity.metadata.name }}
- config - The reusable config values. Usage example: {{ config.title }}
- secrets - The HTTP secrets configuration. Usage example: {{ secrets.grafanaApiToken }}
- targetPluginAuthToken - Auth token that can be used for authenticating calls towards plugin with ID provided in targetPluginId. Usage example: {{ targetPluginAuthToken }}
body [optional] - HTTP request body. Supports liquid templating. The following variables are available for use in templates:
- entity - The entity the fact is collected for. Usage example: {{ entity.metadata.name }}
- config - The reusable config values. Usage example: {{ config.title }}

`response` [optional]

HTTP response configuration:

transform [optional] - JSONata expression to transform the HTTP response body. Optional, if not provided an unmodified response body will be used as the fact data.
emptyOnStatuses [optional] - A list of HTTP status codes to return an empty response on (when received, an empty fact will be collected instead of throwing a collection error). Optional, if not provided all non-OK responses will result in collection errors.
schema [optional] - A JSON schema describing the HTTP response body. Optional, primarily used for Check Creation No-Code UI autocomplete. Not used for any validations.

`retries` [optional]

HTTP request retries configuration:

maxRetries [optional] - Max number of retries. Optional, if not provided failed requests won’t be retried.
retryDelay [optional] - Retry delay in milliseconds. Optional, if not provided failed requests will be retried immediately.
retryOnStatuses [optional] - A list of HTTP status codes to retry the request on. Optional, if not provided all failed requests will be retried.

`rateLimit` [optional]

Per-collection rate limiting configuration. When set, outbound requests for this collection are throttled to stay within the specified window. See Rate Limiting for full details and 429 handling behavior.

duration [required] - The length of the rate limit window in milliseconds. Minimum value: 1.
max [required] - The maximum number of requests allowed within each duration window. Minimum value: 1.

Example:

rateLimit:
  duration: 60000
  max: 30

`frequency` [optional]

The frequency at which the fact collection should be executed. Possible values are either a cron expression { cron: ... } or HumanDuration. If provided, it overrides the default frequency provided at the top level. If not provided, it defaults to the frequency provided at the top level. If neither collector’s frequency nor default frequency is provided, the fact will only be collected on demand. Example:

frequency:
  minutes: 10

`initialDelay` [optional]

The amount of time that should pass before the first invocation happens. Possible values are either a cron expression { cron: ... } or HumanDuration. Example:

initialDelay:
  seconds: 30

`batchSize` [optional]

The number of entities to collect facts for at once. Optional, the default value is 1. Example:

batchSize: 100

`filter` [optional]

A filter specifying which entities to collect the specified facts for. Matches the filter format used by the Catalog API. If provided, it overrides the default filter provided at the top level. If not provided, it defaults to the filter provided at the top level. If neither collector’s filter nor default filter is provided, the fact will be collected for all entities. See filters for more details.

`exclude` [optional]

Entities matching this filter will be skipped during the fact collection process. Can be used in combination with filter. Matches the filter format used by the Catalog API.

filter:
  - kind: component
exclude:
  - spec.type: documentation

`cache` [optional]

If the collected facts should be cached, and if so for how long. Possible values are either true or false or a nested { duration: HumanDuration } field. If provided it, overrides the default cache config provided at the top level. If not provided, it defaults to the cache config provided at the top level. If neither collector’s cache nor default cache config is provided, the fact will not be cached. Example:

cache:
  duration:
    hours: 24

`cacheFullResultsForEtags` [optional]

When collecting entity-independent facts (the same URL for every entity), enable ETag-based caching to avoid redundant API calls. When enabled, the collector caches the response body alongside the ETag returned by the upstream endpoint. On subsequent requests to the same resolved URL, it sends an If-None-Match header. If the server responds with 304 Not Modified, the cached body is reused without re-fetching or re-storing the data.

soundcheck:
  collectors:
    http:
      collects:
        - factName: slack-channels
          cacheFullResultsForEtags: true
          request:
            url: https://api.example.com/channels
            headers:
              Authorization: 'Bearer {{ secrets.API_KEY }}'

The upstream HTTP endpoint must return ETag response headers. When the endpoint does not support ETags, the collector falls back to default behavior. Enabling this option increases memory usage since the full response body is held in the cache.

This follows a similar pattern to the SCM collector’s cacheFullResultsForEtags option. Unlike the SCM collector where this is a top-level flag applied across all extractions, the HTTP collector supports enabling this per fact entry inside collects.

Rate Limiting

The HTTP fact collector supports optional per-collection rate limiting. Each collection entry can include a rateLimit field that throttles outbound requests to the target endpoint using a sliding window.

Configuration

Field	Type	Required	Description
`duration`	`number`	Yes	The length of the rate limit window in milliseconds. Minimum value: `1`.
`max`	`number`	Yes	The maximum number of requests allowed within each `duration` window. Minimum value: `1`.

Example:

soundcheck:
  collectors:
    http:
      collects:
        - factName: grafana_dashboard
          rateLimit:
            duration: 60000
            max: 30
          request:
            method: GET
            url: http://grafana.com/api/dashboards/uid/{{ entity.metadata.annotations['grafana.com/dashboard-uid'] }}
            headers:
              Authorization: Bearer {{ secrets.grafanaApiToken }}

The example above limits requests for the grafana_dashboard collection to 30 requests per 60-second window.

Automatic 429 handling

When a collection has rateLimit configured and the target endpoint returns an HTTP 429 Too Many Requests response, the collector automatically:

Pauses the rate limiter queue for that collection.
Waits for the duration specified by the Retry-After response header (interpreted as seconds). If the header is missing or cannot be parsed, the collector waits 60 seconds.
Resumes the queue and retries the request.

This pause-and-retry behavior is scoped to the individual collection — other collections continue unaffected.

Collections without rateLimit configured treat HTTP 429 responses as a normal collection error, preserving existing behavior.

Relationship to the global worker rate limit

Per-collection rate limiting is independent of the global Soundcheck worker concurrency settings. The global rate limit controls how many collection jobs Soundcheck processes at once across all collectors, while per-collection rate limiting controls how fast a single HTTP collection sends requests to its target endpoint. Check out some additional recommendations on how to avoid exceeding API rate limits.

Shape of HTTP Fact

The shape of HTTP Fact is based on the Fact Schema. The following is an example of the collected HTTP fact:

factRef: http:default/grafana_dashboard
entityRef: component:default/queue-proxy
scope: default
timestamp: 2025-01-09T15:50+00Z
data:
  id: null
  uid: null
  title: Production Overview
  tags:
    - production
  timezone: browser
  schemaVersion: 16
  refresh: 25s

The resulting fact data will contain the HTTP response.

Shape of HTTP Fact Check

The shape of HTTP Fact Check matches the Shape of a Fact Check. The following is an example of the HTTP fact check:

soundcheck:
  checks:
    - id: has-production-dashboard-tag
      name: Grafana dashboard has production tag
      ownerEntityRef: group:default/toast-infra
      description: Grafana dashboard has production tag
      filter:
        kind: Component
        spec.type: service
      rule:
        factRef: http:default/grafana_dashboard
        path: $.tags
        operator: any:equal
        value: production
      passedMessage: Success! Grafana dashboard has production tag
      failedMessage: Failure! Grafana dashboard doesn't have production tag

​Prerequisites

​Configure HTTP secrets in Backstage

​Configure reusable config values in Backstage

​Add the HttpFactCollector to Soundcheck

​Plugin Configuration

​No-Code UI Configuration Option

​YAML Configuration Option

​Defining HTTP Fact Collectors

​Shape Of HTTP Fact Collector

​factName [required]

​request [required]

​response [optional]

​retries [optional]

​rateLimit [optional]

​frequency [optional]

​initialDelay [optional]

​batchSize [optional]

​filter [optional]

​exclude [optional]

​cache [optional]

​cacheFullResultsForEtags [optional]

​Rate Limiting

​Configuration

​Automatic 429 handling

​Relationship to the global worker rate limit

​Shape of HTTP Fact

​Shape of HTTP Fact Check

Prerequisites

Configure HTTP secrets in Backstage

Configure reusable config values in Backstage

Add the HttpFactCollector to Soundcheck

Plugin Configuration

No-Code UI Configuration Option

YAML Configuration Option

Defining HTTP Fact Collectors

Shape Of HTTP Fact Collector

`factName` [required]

`request` [required]

`response` [optional]

`retries` [optional]

`rateLimit` [optional]

`frequency` [optional]

`initialDelay` [optional]

`batchSize` [optional]

`filter` [optional]

`exclude` [optional]

`cache` [optional]

`cacheFullResultsForEtags` [optional]

Rate Limiting

Configuration

Automatic 429 handling

Relationship to the global worker rate limit

Shape of HTTP Fact

Shape of HTTP Fact Check