Skip to main content

Datadog

Datadog integration plugin for Soundcheck.

Datadog integration plugin supports the extraction of the following facts:

Prerequisites​

Configure Datadog integration in Backstage​

Integrations are configured at the root level of app-config.yaml, here's an example configuration for Datadog:

soundcheck:
  collectors:
    datadog:
      appKey: appKey
      apiKey: apiKey

      # If you wish to override the api URL
      # baseUrl: https://us5.datadoghq.com

Add the DataDogFactCollector to Soundcheck​

First add the package:

yarn workspace backend add @spotify/backstage-plugin-soundcheck-backend-module-datadog

Legacy Backend​

If you are still using the legacy backend, in packages/backend/src/plugins/soundcheck.ts, add the DataDogFactCollector:

import { SoundcheckBuilder } from '@spotify/backstage-plugin-soundcheck-backend';
import { Router } from 'express';
import { PluginEnvironment } from '../types';
+ import { DataDogFactCollector } from '@spotify/backstage-plugin-soundcheck-backend-module-datadog';

export default async function createPlugin(
  env: PluginEnvironment,
): Promise<Router> {
  return SoundcheckBuilder.create({ ...env })
    .addFactCollectors(
+      DataDogFactCollector.create(env.logger)
    )
    .build();
}

New Backend System​

If you are using the New Backend System, you can just add the following:

packages/backend/src/index.ts
const backend = createBackend();

backend.add(import('@spotify/backstage-plugin-soundcheck-backend'));
backend.add(
  import('@spotify/backstage-plugin-soundcheck-backend-module-datadog'),
);
// ...

backend.start();

See the soundcheck-backend documentation for additional details on creating the Soundcheck backend.

Plugin Configuration​

Collection of facts is driven by config. To learn more about the config, see the Defining Datadog Fact Collectors.

  1. Create datadog-facts-collectors.yaml in the root of your Backstage repository and fill in all your Datadog fact collectors. A simple example Datadog fact collector is listed below.

Note: this file will be loaded at runtime along with the rest of your Backstage configuration files, so make sure it's available in deployed environments in the same way as your app-config.yaml files.

---
appKey: dummy
apiKey: dummy
collects:
  - type: service-definition
    filter:
      - spec.lifecycle: 'production'
        spec.type: 'website'
    cache: false
  1. Add a soundcheck collectors field to app-config.yaml and reference the newly created datadog-facts-collectors.yaml
# app-config.yaml
soundcheck:
  collectors:
    datadog:
      $include: ./datadog-facts-collectors.yaml

Rate Limiting (Optional)​

This fact collector can be rate limited in Soundcheck using the following configuration:

soundcheck:
  job:
    workers:
      datadog:
        limiter:
          max: 1000
          duration: 60000

In this example the fact collector is limited to 1000 executions per minute.

This fact collector handles 429 rate limit errors from Datadog. Soundcheck will automatically wait and retry requests that are rate limited.

Defining Datadog Fact Collectors​

This section describes the data shape and semantics of Datadog Fact Collectors.

Overall Shape Of A Datadog Fact Collector​

The following is an example of a descriptor file for a Datadog Fact Collector:

---
baseUrl: https://datadog.com
appKey: dummy
apiKey: dummy
collects:
  - type: service-definition
    filter:
      - spec.lifecycle: 'production'
        spec.type: 'website'
    cache: false

See below for details about these fields.

baseUrl [optional]​

The base URL of the Datadog instance to use. If not provided, the plugin will attempt to use the default URL https://api.datadoghq.com.

appKey [optional]​

The Datadog appKey to use for authentication.

apiKey [optional]​

The Datadog apiKey to use for authentication.

collects [required]​

An array describing which facts to collect and how to extract them. See below for details about the overall shape of a fact extractor.

Overall Shape Of A Fact Extractor​

Each extractor supports the fields described below.

type [required]​

The type of the extractor (e.g. service-definition, service-level-objective).

frequency [optional]​

The frequency at which the fact extraction should be executed. Possible values are either a cron expression { cron: ... } or HumanDuration. If provided it overrides the default frequency provided at the top level. If not provided it defaults to the frequency provided at the top level. If neither extractor's frequency nor default frequency is provided the fact will only be collected on demand. Example:

frequency:
  minutes: 10

initialDelay [optional]​

The amount of time that should pass before the first invocation happens. Possible values are either a cron expression { cron: ... } or HumanDuration.

Example:

initialDelay:
  seconds: 30

filter [optional]​

A filter specifying which entities to collect the specified facts for. Matches the filter format used by the Catalog API. If provided it overrides the default filter provided at the top level. If not provided it defaults to the filter provided at the top level. If neither extractor's filter nor default filter is provided the fact will be collected for all entities.

cache [optional]​

If the collected facts should be cached, and if so for how long. Possible values are either true or false or a nested { duration: HumanDuration } field. If provided it overrides the default cache config provided at the top level. If not provided it defaults to the cache config provided at the top level. If neither extractor's cache nor default cache config is provided the fact will not be cached. Example:

cache:
  duration:
    hours: 24

Entity configuration​

In your catalog-info.yaml add the following metadata annotation to allow the plugin to map an entity to a service in Datadog. Make sure your SLO is tagged service:service-id in Datadog so that it can be mapped too.

metadata:
  annotations:
    datadoghq.com/service-id: test-service-id
Note

The datadoghq.com/service-id annotation is required for the integration to work, if this is missing then we will not be able to collect the facts for the entity

Collecting Service Definition Fact​

A service definition fact contains information about service definition from Datadog Service Definition API.

Shape of A Service Definition Fact Collector​

The shape of a Service Definition Fact Collector matches the Overall Shape Of A Datadog Fact Collector (restriction: type: service-definition).

The following is an example of the Service Definition Fact Collector config:

- type: service-definition
  cache: true
  frequency:
    cron: '0 * * * *'

Shape of A Service Definition Fact Check​

The shape of a service-definition Fact Check matches the Shape of a Fact Check.

The following is an example of the service-definition fact check:

soundcheck:
  checks:
    - id: requires_service_definition_description
      description: Requires service definition to have a description that is not empty
      passedMessage: The check has passed!
      failedMessage: The check has failed!
      rule:
        factRef: datadog:default/service-definition
        path: $.data.attributes.schema.description
        operator: notEqual
        value: ''
      schedule:
        frequency:
          minutes: 1
        filter:
          metadata.tags: some-tag

The following is an example response from the Service Definition Fact collector. See Datadog's documentation for more detail.

{
  "data": {
    "attributes": {
      "meta": {
        "githubHtmlUrl": "",
        "ingestedSchemaVersion": "v2.2",
        "ingestionSource": "api",
        "lastModifiedTime": "2024-03-14T15:40:25.517342435Z",
        "origin": "dd_ui",
        "originDetail": "editor",
        "warnings": []
      },
      "schema": {
        "ddService": "backstage_plugins_repo",
        "description": "This is a test for backstage integration with DD",
        "integrations": {},
        "languages": ["javascript"],
        "schemaVersion": "v2.2",
        "type": "web"
      }
    },
    "id": "1eda3490-8d1a-4686-8e88-83f54fba1231",
    "type": "service-definition"
  }
}

Collecting Service Level Objective Fact​

An service-level-objective fact contains information about service-level-objective from Datadog Service Level Objective API.

Shape of A Service Level Objective Fact Collector​

The shape of an Service Level Objective Fact Collector matches the Overall Shape Of A Datadog Fact Collector (restriction: type: service-level-objective).

The following is an example of the Service Level Objective Fact Collector config:

- type: service-level-objective
  cache: true
  frequency:
    cron: '0 * * * *'

Shape of A Service Level Objective Fact Check​

The shape of a service-level-objective Fact Check matches the Shape of a Fact Check.

The following is an example of the service-level-objective fact check:

soundcheck:
  checks:
    - id: requires_service_level_objective_to_be_defined
      description: Requires at least one SLO to be defined
      passedMessage: The check has passed!
      failedMessage: The check has failed!
      rule:
        factRef: datadog:default/service-level-objective
        path: $.meta.pagination.total
        operator: greaterThan
        value: 0
      schedule:
        frequency:
          minutes: 1
        filter:
          metadata.tags: some-tag

The following is an example response from the Service Level Objective Fact collector. See Datadog's documentation for more detail.

{
  "data": {
    "attributes": {
      "slos": [
        {
          "data": {
            "attributes": {
              "allTags": ["service:backstage_plugins_repo"],
              "createdAt": 1710768504,
              "creator": {
                "email": "jdoe@gmail.com",
                "id": 1001417642,
                "name": "John doe"
              },
              "description": "This SLO tracks the availability of the backstage_plugins_repo service. Availability is measured as the number of successful requests divided by the number of total requests for the service",
              "envTags": ["none"],
              "groups": null,
              "modifiedAt": 1710768546,
              "monitorIds": null,
              "name": "Availability SLO for backstage_plugins_repo service",
              "overallStatus": [
                {
                  "error": null,
                  "errorBudgetRemaining": null,
                  "indexedAt": 1710768510,
                  "rawErrorBudgetRemaining": null,
                  "spanPrecision": 1,
                  "state": "breached",
                  "status": 0,
                  "target": 90,
                  "timeframe": "7d"
                }
              ],
              "query": {
                "denominator": "sum:datadog.agent.started{host:local}.as_count()",
                "metrics": null,
                "numerator": "sum:datadog.dogstatsd.client.service_checks{host:local} by {client}.as_count()"
              },
              "serviceTags": ["backstage_plugins_repo"],
              "sloType": "metric",
              "status": {
                "calculationError": null,
                "errorBudgetRemaining": null,
                "indexedAt": 1710768510,
                "rawErrorBudgetRemaining": null,
                "sli": 0,
                "spanPrecision": 1,
                "state": "breached"
              },
              "teamTags": [],
              "thresholds": [
                {
                  "target": 90,
                  "targetDisplay": "90",
                  "timeframe": "7d",
                  "warning": 92,
                  "warningDisplay": "90"
                }
              ]
            },
            "id": "bf9ab3930e5c5220a3a53155072ab453",
            "type": "slo"
          }
        }
      ]
    },
    "type": "service_level_objective_search_results"
  },
  "links": {
    "first": "https://datadoghq.eu/api/v1/slo/search?query=service%3Abackstage_plugins_repo&page[offset]=0&page[limit]=10",
    "last": null,
    "next": "https://datadoghq.eu/api/v1/slo/search?query=service%3Abackstage_plugins_repo&page[offset]=10&page[limit]=10",
    "prev": null,
    "self": "https://datadoghq.eu/api/v1/slo/search?query=service%3Abackstage_plugins_repo"
  },
  "meta": { "pagination": { "total": 1, "type": "offset_limit" } }
}

Shape of A Datadog Track​

The following is an example of the Soundcheck track that utilizes these checks

soundcheck:
  programs:
    - id: demo
      name: Demo
      ownerEntityRef: group:default/owning_group
      description: Demonstration of Soundcheck Datadog Fact Collector
      levels:
        - ordinal: 1
          name: First level
          description: Checks leveraging SoundCheck's Datadog Fact Collector
          checks:
            - id: requires_service_definition_to_have_description
              name: Requires service definition to have description
              description: Requires service definition to have description that is not empty
            - id: requires_service_level_objective_to_be_defined
              name: Requires SLOs to be defined
              description: Requires at least one SLO to be defined
      filter:
        catalog:
          metadata.tags: some-tag