Datadog - Spotify for Backstage

Datadog integration plugin for Soundcheck. Datadog integration plugin supports the extraction of the following facts:

Service Definition
Service Level Objective

Prerequisites

Configure Datadog integration in Backstage

Integrations are configured at the root level of app-config.yaml, here’s an example configuration for Datadog:

soundcheck:
  collectors:
    datadog:
      appKey: appKey
      apiKey: apiKey

      # If you wish to override the api URL
      # baseUrl: https://us5.datadoghq.com

Add the DataDogFactCollector to Soundcheck

First add the package:

yarn workspace backend add @spotify/backstage-plugin-soundcheck-backend-module-datadog

Then add the following to your packages/backend/src/index.ts file:

packages/backend/src/index.ts

const backend = createBackend();

// ...
backend.add(import('@spotify/backstage-plugin-soundcheck-backend'));
backend.add(
  import('@spotify/backstage-plugin-soundcheck-backend-module-datadog'),
);
// ...

backend.start();

Consult the Soundcheck Backend documentation for additional details on setting up the Soundcheck backend.

Plugin Configuration

Collection of facts is driven by config. To learn more about the config, see the Defining Datadog Fact Collectors. Datadog Fact Collector can be configured via YAML or No-Code UI. If you configure it via both YAML and No-Code UI, the configurations will be merged. It’s preferable to choose a single source for the Fact Collectors configuration (either No-Code UI or YAML) to avoid confusing merge results.

No-Code UI Configuration Option

Make sure the prerequisite Configure Datadog integration in Backstage is completed and Datadog instance details are configured.
To enable the Datadog Integration, go to Soundcheck > Integrations > Datadog and click the Configure button. To learn more about the No-Code UI config, see the Configuring a fact collector (integration) via the no-code UI.

YAML Configuration Option

Create datadog-facts-collectors.yaml in the root of your Backstage repository and fill in all your Datadog fact collectors. A simple example Datadog fact collector is listed below.

Note: this file will be loaded at runtime along with the rest of your Backstage configuration files, so make sure it’s available in deployed environments in the same way as your app-config.yaml files.

---
appKey: dummy
apiKey: dummy
collects:
  - type: service-definition
    filter:
      - spec.lifecycle: 'production'
        spec.type: 'website'
    cache: false

Add a soundcheck collectors field to app-config.yaml and reference the newly created datadog-facts-collectors.yaml

# app-config.yaml
soundcheck:
  collectors:
    datadog:
      $include: ./datadog-facts-collectors.yaml

Rate Limiting (Optional)

This fact collector can be rate limited in Soundcheck using the following configuration:

soundcheck:
  job:
    workers:
      datadog:
        limiter:
          max: 1000
          duration: 60000

In this example the fact collector is limited to 1000 executions per minute. This fact collector handles 429 rate limit errors from Datadog. Soundcheck will automatically wait and retry requests that are rate limited.

Defining Datadog Fact Collectors

This section describes the data shape and semantics of Datadog Fact Collectors.

Overall Shape Of A Datadog Fact Collector

The following is an example of a descriptor file for a Datadog Fact Collector:

---
baseUrl: https://datadog.com
appKey: dummy
apiKey: dummy
collects:
  - type: service-definition
    filter:
      - spec.lifecycle: 'production'
        spec.type: 'website'
    exclude:
      - kind: 'template'
    cache: false

See below for details about these fields.

`baseUrl` [optional]

The base URL of the Datadog instance to use. If not provided, the plugin will attempt to use the default URL https://api.datadoghq.com.

`appKey` [optional]

The Datadog appKey to use for authentication.

`apiKey` [optional]

The Datadog apiKey to use for authentication.

`collects` [required]

An array describing which facts to collect and how to extract them. See below for details about the overall shape of a fact extractor.

Overall Shape Of A Fact Extractor

Each extractor supports the fields described below.

`type` [required]

The type of the extractor (e.g. service-definition, service-level-objective).

`frequency` [optional]

The frequency at which the fact extraction should be executed. Possible values are either a cron expression { cron: ... } or HumanDuration. If provided it overrides the default frequency provided at the top level. If not provided it defaults to the frequency provided at the top level. If neither extractor’s frequency nor default frequency is provided the fact will only be collected on demand. Example:

frequency:
  minutes: 10

`initialDelay` [optional]

The amount of time that should pass before the first invocation happens. Possible values are either a cron expression { cron: ... } or HumanDuration. Example:

initialDelay:
  seconds: 30

`batchSize` [optional]

The number of entities to collect facts for at once. Optional, the default value is 1. Note: Fact collection for a batch of entities is still considered as one hit towards the rate limits by the Soundcheck Rate Limiting engine, while the actual number of hits will be equal to the batchSize. Example:

batchSize: 100

`filter` [optional]

A filter specifying which entities to collect the specified facts for. Matches the filter format used by the Catalog API. If provided it overrides the default filter provided at the top level. If not provided it defaults to the filter provided at the top level. If neither extractor’s filter nor default filter is provided the fact will be collected for all entities. See filters for more details.

`exclude` [optional]

Entities matching this filter will be skipped during the fact collection process. Can be used in combination with filter. Matches the filter format used by the Catalog API.

filter:
  - kind: component
exclude:
  - spec.type: documentation

`cache` [optional]

If the collected facts should be cached, and if so for how long. Possible values are either true or false or a nested { duration: HumanDuration } field. If provided it overrides the default cache config provided at the top level. If not provided it defaults to the cache config provided at the top level. If neither extractor’s cache nor default cache config is provided the fact will not be cached. Example:

cache:
  duration:
    hours: 24

Entity configuration

In your catalog-info.yaml add the following metadata annotation to allow the plugin to map an entity to a service in Datadog. Make sure your SLO is tagged service:service-id in Datadog so that it can be mapped too.

metadata:
  annotations:
    datadoghq.com/service-id: test-service-id

NoteThe datadoghq.com/service-id annotation is required for the integration to work, if this is missing then we will not be able to collect the facts for the entity

Collecting Service Definition Fact

A service definition fact contains information about service definition from Datadog Service Definition API.

Shape of A Service Definition Fact Collector

The shape of a Service Definition Fact Collector matches the Overall Shape Of A Datadog Fact Collector (restriction: type: service-definition). The following is an example of the Service Definition Fact Collector config:

- type: service-definition
  cache: true
  frequency:
    cron: '0 * * * *'

Shape of A Service Definition Fact Check

The shape of a service-definition Fact Check matches the Shape of a Fact Check. The following is an example of the service-definition fact check:

soundcheck:
  checks:
    - id: requires_service_definition_description
      description: Requires service definition to have a description that is not empty
      passedMessage: The check has passed!
      failedMessage: The check has failed!
      rule:
        factRef: datadog:default/service-definition
        path: $.data.attributes.schema.description
        operator: notEqual
        value: ''

The following is an example response from the Service Definition Fact collector. See Datadog’s documentation for more detail.

{
  "data": {
    "attributes": {
      "meta": {
        "githubHtmlUrl": "",
        "ingestedSchemaVersion": "v2.2",
        "ingestionSource": "api",
        "lastModifiedTime": "2024-03-14T15:40:25.517342435Z",
        "origin": "dd_ui",
        "originDetail": "editor",
        "warnings": []
      },
      "schema": {
        "ddService": "backstage_plugins_repo",
        "description": "This is a test for backstage integration with DD",
        "integrations": {},
        "languages": ["javascript"],
        "schemaVersion": "v2.2",
        "type": "web"
      }
    },
    "id": "1eda3490-8d1a-4686-8e88-83f54fba1231",
    "type": "service-definition"
  }
}

Collecting Service Level Objective Fact

An service-level-objective fact contains information about service-level-objective from Datadog Service Level Objective API.

Shape of A Service Level Objective Fact Collector

The shape of an Service Level Objective Fact Collector matches the Overall Shape Of A Datadog Fact Collector (restriction: type: service-level-objective). The following is an example of the Service Level Objective Fact Collector config:

- type: service-level-objective
  cache: true
  frequency:
    cron: '0 * * * *'

Shape of A Service Level Objective Fact Check

The shape of a service-level-objective Fact Check matches the Shape of a Fact Check. The following is an example of the service-level-objective fact check:

soundcheck:
  checks:
    - id: requires_service_level_objective_to_be_defined
      description: Requires at least one SLO to be defined
      passedMessage: The check has passed!
      failedMessage: The check has failed!
      rule:
        factRef: datadog:default/service-level-objective
        path: $.meta.pagination.total
        operator: greaterThan
        value: 0

The following is an example response from the Service Level Objective Fact collector. See Datadog’s documentation for more detail.

{
  "data": {
    "attributes": {
      "slos": [
        {
          "data": {
            "attributes": {
              "allTags": ["service:backstage_plugins_repo"],
              "createdAt": 1710768504,
              "creator": {
                "email": "jdoe@gmail.com",
                "id": 1001417642,
                "name": "John doe"
              },
              "description": "This SLO tracks the availability of the backstage_plugins_repo service. Availability is measured as the number of successful requests divided by the number of total requests for the service",
              "envTags": ["none"],
              "groups": null,
              "modifiedAt": 1710768546,
              "monitorIds": null,
              "name": "Availability SLO for backstage_plugins_repo service",
              "overallStatus": [
                {
                  "error": null,
                  "errorBudgetRemaining": null,
                  "indexedAt": 1710768510,
                  "rawErrorBudgetRemaining": null,
                  "spanPrecision": 1,
                  "state": "breached",
                  "status": 0,
                  "target": 90,
                  "timeframe": "7d"
                }
              ],
              "query": {
                "denominator": "sum:datadog.agent.started{host:local}.as_count()",
                "metrics": null,
                "numerator": "sum:datadog.dogstatsd.client.service_checks{host:local} by {client}.as_count()"
              },
              "serviceTags": ["backstage_plugins_repo"],
              "sloType": "metric",
              "status": {
                "calculationError": null,
                "errorBudgetRemaining": null,
                "indexedAt": 1710768510,
                "rawErrorBudgetRemaining": null,
                "sli": 0,
                "spanPrecision": 1,
                "state": "breached"
              },
              "teamTags": [],
              "thresholds": [
                {
                  "target": 90,
                  "targetDisplay": "90",
                  "timeframe": "7d",
                  "warning": 92,
                  "warningDisplay": "90"
                }
              ]
            },
            "id": "bf9ab3930e5c5220a3a53155072ab453",
            "type": "slo"
          }
        }
      ]
    },
    "type": "service_level_objective_search_results"
  },
  "links": {
    "first": "https://datadoghq.eu/api/v1/slo/search?query=service%3Abackstage_plugins_repo&page[offset]=0&page[limit]=10",
    "last": null,
    "next": "https://datadoghq.eu/api/v1/slo/search?query=service%3Abackstage_plugins_repo&page[offset]=10&page[limit]=10",
    "prev": null,
    "self": "https://datadoghq.eu/api/v1/slo/search?query=service%3Abackstage_plugins_repo"
  },
  "meta": { "pagination": { "total": 1, "type": "offset_limit" } }
}

Shape of A Datadog Track

The following is an example of the Soundcheck track that utilizes these checks

soundcheck:
  tracks:
    - id: demo
      name: Demo
      ownerEntityRef: group:default/owning_group
      description: Demonstration of Soundcheck Datadog Fact Collector
      levels:
        - ordinal: 1
          name: First level
          description: Checks leveraging SoundCheck's Datadog Fact Collector
          checks:
            - id: requires_service_definition_to_have_description
              name: Requires service definition to have description
              description: Requires service definition to have description that is not empty
            - id: requires_service_level_objective_to_be_defined
              name: Requires SLOs to be defined
              description: Requires at least one SLO to be defined
      filter:
        catalog:
          metadata.tags: some-tag

​Prerequisites

​Configure Datadog integration in Backstage

​Add the DataDogFactCollector to Soundcheck

​Plugin Configuration

​No-Code UI Configuration Option

​YAML Configuration Option

​Rate Limiting (Optional)

​Defining Datadog Fact Collectors

​Overall Shape Of A Datadog Fact Collector

​baseUrl [optional]

​appKey [optional]

​apiKey [optional]

​collects [required]

​Overall Shape Of A Fact Extractor

​type [required]

​frequency [optional]

​initialDelay [optional]

​batchSize [optional]

​filter [optional]

​exclude [optional]

​cache [optional]

​Entity configuration

​Collecting Service Definition Fact

​Shape of A Service Definition Fact Collector

​Shape of A Service Definition Fact Check

​Collecting Service Level Objective Fact

​Shape of A Service Level Objective Fact Collector

​Shape of A Service Level Objective Fact Check

​Shape of A Datadog Track

Prerequisites

Configure Datadog integration in Backstage

Add the DataDogFactCollector to Soundcheck

Plugin Configuration

No-Code UI Configuration Option

YAML Configuration Option

Rate Limiting (Optional)

Defining Datadog Fact Collectors

Overall Shape Of A Datadog Fact Collector

`baseUrl` [optional]

`appKey` [optional]

`apiKey` [optional]

`collects` [required]

Overall Shape Of A Fact Extractor

`type` [required]

`frequency` [optional]

`initialDelay` [optional]

`batchSize` [optional]

`filter` [optional]

`exclude` [optional]

`cache` [optional]

Entity configuration

Collecting Service Definition Fact

Shape of A Service Definition Fact Collector

Shape of A Service Definition Fact Check

Collecting Service Level Objective Fact

Shape of A Service Level Objective Fact Collector

Shape of A Service Level Objective Fact Check

Shape of A Datadog Track