> ## Documentation Index
> Fetch the complete documentation index at: https://backstage.spotify.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# HTTP
> Collect facts from arbitrary HTTP endpoints with token or API key authentication and simple response body transformations.
The HTTP integration plugin for [Soundcheck](https://backstage.spotify.com/plugins/soundcheck/) supports the collection of the facts from HTTP endpoints.
The HTTP fact collector is intentionally simple and should be used for simple use cases.
It doesn't support rate limiting, pagination and any custom logic besides a simple transformation of the HTTP response body.
In case you have a more complex use-case, consider implementing a [custom integration](https://backstage.spotify.com/docs/plugins/soundcheck/tutorials/custom-fact-collector).
## Prerequisites
### Configure HTTP secrets in Backstage
If your endpoints use a token or an API key authentication (or any other secret data),
you may configure the secrets at the root level of `app-config.yaml`. Here's an example configuration:
```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
soundcheck:
collectors:
http:
secrets:
- key: grafanaApiToken
value: ${API_TOKEN}
```
Referencing a secret in an HTTP header template (e.g. `Authorization: Bearer {{ secrets.grafanaApiToken }}`)
will send the secret value to the target server. Make sure the request URL points to a trusted endpoint before
including any secrets in the request.
### Configure reusable config values in Backstage
If your endpoints need non-secret configurable values you can reuse across templates,
you may configure them at the root level of `app-config.yaml`. Here's an example configuration:
```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
soundcheck:
collectors:
http:
config:
- key: title
value: Soundcheck
```
The `config` template variable previously exposed the entire Backstage
`app-config.yaml` (e.g. `{{ config.app.title }}`). It now only exposes the
key-value pairs defined under `soundcheck.collectors.http.config`. Existing
templates that reference other `app-config.yaml` fields will need to be
migrated to use explicitly declared `config` entries.
### Add the HttpFactCollector to Soundcheck
First, add the `@spotify/backstage-plugin-soundcheck-backend-module-http` package:
```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
yarn workspace backend add @spotify/backstage-plugin-soundcheck-backend-module-http
```
Then add the following to your `packages/backend/src/index.ts` file:
```ts packages/backend/src/index.ts highlight={4} theme={"theme":{"light":"github-light","dark":"dracula"}}
const backend = createBackend();
backend.add(import('@spotify/backstage-plugin-soundcheck-backend'));
backend.add(import('@spotify/backstage-plugin-soundcheck-backend-module-http'));
// ...
backend.start();
```
Consult the [Soundcheck Backend documentation](../../../setup-and-installation#backend-setup) for additional details on setting up the Soundcheck backend.
## Plugin Configuration
The collection of HTTP facts is driven by configuration. To learn more about the configuration, consult the [Defining HTTP Fact Collectors](#defining-http-fact-collectors) section.
HTTP Fact Collector can be configured via YAML or No-Code UI. If you configure it via both YAML and No-Code UI, the configurations will be merged.
It's preferable to choose a single source for the Fact Collectors configuration (either No-Code UI or YAML) to avoid confusing merge results.
### No-Code UI Configuration Option
1. Make sure the prerequisite [Configure HTTP secrets in Backstage](#configure-http-secrets-in-backstage) is completed if your endpoints require any secret data.
2. Make sure the prerequisite [Configure reusable config values in Backstage](#configure-reusable-config-values-in-backstage) is completed if your endpoints use reusable config values in templates.
3. To enable the HTTP Integration, go to `Soundcheck > Integrations > HTTP` and click the `Configure` button. To learn more about the No-Code UI config, see the [Configuring a fact collector (integration) via the no-code UI](../index#configuring-a-fact-collector-integration-via-the-no-code-ui).
### YAML Configuration Option
1. Create a `http-facts-collectors.yaml` file in the root of your Backstage repository and fill in all your HTTP Fact Collectors.
A simple example HTTP fact collector is listed below.
```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
- factName: grafana_dashboard
request:
method: GET
url: http://grafana.com/api/dashboards/uid/{{ entity.metadata.annotations['grafana.com/dashboard-uid'] }}
headers:
Accept: application/json
Content-Type: application/json
Authorization: Bearer {{ secrets.grafanaApiToken }}
response:
transform: dashboard
emptyOnStatuses: [404]
schema: |
{
"title": "Grafana Dashboard",
"description": "Grafana Dashboard",
"type": "object",
"properties": {
"id": {
"type": "string"
},
"uid": {
"type": "string"
},
"title": {
"type": "string"
},
"tags": {
"type": "array",
"items": {
"type": "string"
}
}
}
}
retries:
maxRetries: 3
retryDelay: 5000
retryOnStatuses: [408, 502, 503, 504]
- factName: owners_failed_check_results
filter:
kind: 'Component'
spec.type: 'service'
request:
targetPluginId: soundcheck
method: GET
url: '{{ targetPluginBaseUrl }}/results?entityRef={{ entity.relations | where: "type", "ownedBy" | first | map: "targetRef" }}'
headers:
Authorization: Bearer {{ targetPluginAuthToken }}
response:
transform: results[state='failed']
emptyOnStatuses: [404]
schema: |
{
"title": "Check Results",
"description": "Soundcheck Check Results",
"type": "object",
"properties": {
"results": {
"type": "array",
"items": {
"type": "object",
"properties": {
"entityRef": {
"type": "string"
},
"checkId": {
"type": "string"
},
"scope": {
"type": "string"
},
"state": {
"type": "string"
},
"timestamp": {
"type": "string"
}
}
}
}
}
}
```
**Note:** this file will be loaded at runtime along with the rest of your Backstage configuration files, so make sure it's available in deployed environments in the same way as your `app-config.yaml` files.
2. Add a Soundcheck collects field to the `app-config.yaml` and reference the newly created `http-facts-collectors.yaml` file.
```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
# app-config.yaml
soundcheck:
collectors:
http:
collects:
$include: ./http-facts-collectors.yaml
```
## Defining HTTP Fact Collectors
This section describes the data shape and semantics of HTTP Fact Collectors.
### Shape Of HTTP Fact Collector
The following is an example of HTTP Fact Collector YAML configuration:
```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
soundcheck:
collectors:
http:
collects:
- factName: grafana_dashboard
request:
method: GET
url: http://grafana.com/api/dashboards/uid/{{ entity.metadata.annotations['grafana.com/dashboard-uid'] }}
headers:
Accept: application/json
Content-Type: application/json
Authorization: Bearer {{ secrets.grafanaApiToken }}
response:
transform: dashboard
emptyOnStatuses: [404]
schema: |
{
"title": "Grafana Dashboard",
"description": "Grafana Dashboard",
"type": "object",
"properties": {
"id": {
"type": "string"
},
"uid": {
"type": "string"
},
"title": {
"type": "string"
},
"tags": {
"type": "array",
"items": {
"type": "string"
}
}
}
}
retries:
maxRetries: 3
retryDelay: 5000
retryOnStatuses: [408, 502, 503, 504]
filter:
kind: 'Component'
spec.type: 'service'
cache: false
cacheFullResultsForEtags: false
```
Below are the details for each field.
#### `factName` \[required]
The name of the fact to be collected.
* Minimum length of 1
* Maximum length of 100
* Alphanumeric with single separator instances of periods, dashes, underscores, or forward slashes
#### `request` \[required]
HTTP request configuration:
* `targetPluginId` \[optional] - Optional, provide if calling a plugin's HTTP endpoint.
If provided, liquid template tags `{{ targetPluginBaseUrl }}` and `{{ targetPluginAuthToken }}` will be populated with the plugin's data.
* `method` \[optional] - HTTP method (`GET` or `POST`). Optional, if not provided the method will be defaulted to `GET`.
* `url` \[required] - HTTP request URL. Supports [liquid templating](https://liquidjs.com/tutorials/intro-to-liquid.html).
The following variables are available for use in templates:
* `entity` - The entity the fact is collected for. Usage example: `{{ entity.metadata.name }}`
* `config` - The reusable config values. Usage example: `{{ config.title }}`
* `targetPluginBaseUrl` - Base URL of the plugin with ID provided in `targetPluginId`. Usage example: `{{ targetPluginBaseUrl }}`
* `headers` \[optional] - HTTP request headers. Header keys and values support [liquid templating](https://liquidjs.com/tutorials/intro-to-liquid.html).
The following variables are available for use in templates:
* `entity` - The entity the fact is collected for. Usage example: `{{ entity.metadata.name }}`
* `config` - The reusable config values. Usage example: `{{ config.title }}`
* `secrets` - The HTTP secrets configuration. Usage example: `{{ secrets.grafanaApiToken }}`
* `targetPluginAuthToken` - Auth token that can be used for authenticating calls towards plugin with ID provided in `targetPluginId`. Usage example: `{{ targetPluginAuthToken }}`
* `body` \[optional] - HTTP request body. Supports [liquid templating](https://liquidjs.com/tutorials/intro-to-liquid.html).
The following variables are available for use in templates:
* `entity` - The entity the fact is collected for. Usage example: `{{ entity.metadata.name }}`
* `config` - The reusable config values. Usage example: `{{ config.title }}`
#### `response` \[optional]
HTTP response configuration:
* `transform` \[optional] - [JSONata](https://docs.jsonata.org/overview.html) expression to transform the HTTP response body.
Optional, if not provided an unmodified response body will be used as the fact data.
* `emptyOnStatuses` \[optional] - A list of HTTP status codes to return an empty response on
(when received, an empty fact will be collected instead of throwing a collection error).
Optional, if not provided all non-OK responses will result in collection errors.
* `schema` \[optional] - A JSON schema describing the HTTP response body.
Optional, primarily used for [Check Creation No-Code UI](../../checks#managing-checks-via-the-no-code-ui) autocomplete.
Not used for any validations.
#### `retries` \[optional]
HTTP request retries configuration:
* `maxRetries` \[optional] - Max number of retries. Optional, if not provided failed requests won't be retried.
* `retryDelay` \[optional] - Retry delay in milliseconds. Optional, if not provided failed requests will be retried immediately.
* `retryOnStatuses` \[optional] - A list of HTTP status codes to retry the request on. Optional, if not provided all failed requests will be retried.
#### `frequency` \[optional]
The frequency at which the fact collection should be executed. Possible values are either a cron expression `{ cron: ... }` or [HumanDuration](https://backstage.io/docs/reference/types.humanduration).
If provided, it overrides the default frequency provided at the top level. If not provided, it defaults to the frequency provided at the top level. If neither collector's frequency nor default frequency is provided, the fact will only be collected on demand.
Example:
```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
frequency:
minutes: 10
```
#### `initialDelay` \[optional]
The amount of time that should pass before the first invocation happens. Possible values are either a cron expression `{ cron: ... }` or [HumanDuration](https://backstage.io/docs/reference/types.humanduration).
Example:
```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
initialDelay:
seconds: 30
```
#### `batchSize` \[optional]
The number of entities to collect facts for at once. Optional, the default value is 1.
Example:
```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
batchSize: 100
```
#### `filter` \[optional]
A filter specifying which entities to collect the specified facts for. Matches the [filter format](https://backstage.io/docs/reference/catalog-client.entityfilterquery) used by the Catalog API.
If provided, it overrides the default filter provided at the top level. If not provided, it defaults to the filter provided at the top level. If neither collector's filter nor default filter is provided, the fact will be collected for all entities.
See [filters](/plugins/soundcheck/core-concepts/filters) for more details.
#### `exclude` \[optional]
Entities matching this filter will be skipped during the fact collection process. Can be used in combination with filter. Matches the [filter format](https://backstage.io/docs/reference/catalog-client.entityfilterquery) used by the Catalog API.
```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
filter:
- kind: component
exclude:
- spec.type: documentation
```
#### `cache` \[optional]
If the collected facts should be cached, and if so for how long. Possible values are either `true` or `false` or a nested `{ duration:` [HumanDuration](https://backstage.io/docs/reference/types.humanduration) `}` field.
If provided it, overrides the default cache config provided at the top level. If not provided, it defaults to the cache config provided at the top level. If neither collector's cache nor default cache config is provided, the fact will not be cached.
Example:
```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
cache:
duration:
hours: 24
```
#### `cacheFullResultsForEtags` \[optional]
When collecting entity-independent facts (the same URL for every entity), enable
ETag-based caching to avoid redundant API calls. When enabled, the collector
caches the response body alongside the ETag returned by the upstream endpoint. On
subsequent requests to the same resolved URL, it sends an `If-None-Match` header.
If the server responds with `304 Not Modified`, the cached body is reused without
re-fetching or re-storing the data.
```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
soundcheck:
collectors:
http:
collects:
- factName: slack-channels
cacheFullResultsForEtags: true
request:
url: https://api.example.com/channels
headers:
Authorization: 'Bearer {{ secrets.API_KEY }}'
```
The upstream HTTP endpoint must return `ETag` response headers. When the
endpoint does not support ETags, the collector falls back to default behavior.
Enabling this option increases memory usage since the full response body is
held in the cache.
This follows a similar pattern to the SCM collector's [`cacheFullResultsForEtags`](/plugins/soundcheck/core-concepts/fact-collectors/3p-integrations/scm#caching-full-etag-responses-optional) option. Unlike the SCM collector where this is a top-level flag applied across all extractions, the HTTP collector supports enabling this per fact entry inside `collects`.
### Rate Limiting
This fact collector doesn't support rate limiting. If your target HTTP endpoint has rate limits
it's recommended to implement a [custom fact collector](https://backstage.spotify.com/docs/plugins/soundcheck/tutorials/custom-fact-collector) that will respect the configurable limits.
Check out some [recommendations](https://backstage.spotify.com/docs/plugins/soundcheck/troubleshooting/#i-am-seeing-rate-limit-issues-what-can-i-do)
on how to avoid exceeding API rate limits.
### Shape of HTTP Fact
The shape of HTTP Fact is based on the [Fact Schema](/plugins/soundcheck/api-reference/facts/submit-facts#body-facts).
The following is an example of the collected HTTP fact:
```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
factRef: http:default/grafana_dashboard
entityRef: component:default/queue-proxy
scope: default
timestamp: 2025-01-09T15:50+00Z
data:
id: null
uid: null
title: Production Overview
tags:
- production
timezone: browser
schemaVersion: 16
refresh: 25s
```
The resulting fact data will contain the HTTP response.
### Shape of HTTP Fact Check
The shape of HTTP Fact Check matches the [Shape of a Fact Check](https://backstage.spotify.com/docs/plugins/soundcheck/core-concepts/checks#overall-shape-of-a-check).
The following is an example of the HTTP fact check:
```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
soundcheck:
checks:
- id: has-production-dashboard-tag
name: Grafana dashboard has production tag
ownerEntityRef: group:default/toast-infra
description: Grafana dashboard has production tag
filter:
kind: Component
spec.type: service
rule:
factRef: http:default/grafana_dashboard
path: $.tags
operator: any:equal
value: production
passedMessage: Success! Grafana dashboard has production tag
failedMessage: Failure! Grafana dashboard doesn't have production tag
```