Skip to main content

PagerDuty

The PagerDuty integration plugin for Soundcheck supports the extraction of the following facts:

Prerequisites​

Configure PagerDuty integration in Backstage​

Integrations are configured at the root level of the app-config.yaml file. Here's an example configuration for PagerDuty:

soundcheck:
  collectors:
    pagerduty:
      token: ${PAGERDUTY_TOKEN}
      server: api.pagerduty.com

Add the PagerDutyFactCollector to Soundcheck​

First, add the @spotify/backstage-plugin-soundcheck-backend-module-pagerduty package:

yarn workspace backend add @spotify/backstage-plugin-soundcheck-backend-module-pagerduty

Legacy Backend​

If you are still using the legacy backend, in packages/backend/src/plugins/soundcheck.ts, add the PagerDutyFactCollector:

  import { SoundcheckBuilder } from '@spotify/backstage-plugin-soundcheck-backend';
  import { Router } from 'express';
  import { PluginEnvironment } from '../types';
+ import { PagerDutyFactCollector } from '@spotify/backstage-plugin-soundcheck-backend-module-pagerduty';

  export default async function createPlugin(
    env: PluginEnvironment,
  ): Promise<Router> {
    return SoundcheckBuilder.create({ ...env })
      .addFactCollectors(
+        PagerDutyFactCollector.create({
+          config: env.config,
+          cache: env.cache,
+          logger: env.logger,
+        }),
    )
    .build();
}

Consult the the Soundcheck Backend documentation for additional details on setting up the Soundcheck backend.

New Backend System​

If you are using the New Backend System, you can just add the following:

packages/backend/src/index.ts
const backend = createBackend();

backend.add(import('@spotify/backstage-plugin-soundcheck-backend'));
backend.add(import('@spotify/backstage-plugin-soundcheck-backend-module-pagerduty'));
// ...

backend.start();

Plugin Configuration​

The collection of PagerDuty facts is driven by the configuration. To learn more about the configuration, consult the Defining PagerDuty Fact Collectors section.

  1. Create a pagerduty-facts-collectors.yaml file in the root of your Backstage repository and fill in all your PagerDuty Fact Collectors. A simple example PagerDuty fact collector is listed below.

    ---
    token: dummy
    server: api.pagerduty.com
    collects:
      - type: Standards
        filter:
          - spec.lifecycle: 'production'
            spec.type: 'website'
        cache: false

    Note: this file will be loaded at runtime along with the rest of your Backstage configuration files, so make sure it's available in deployed environments in the same way as your app-config.yaml files.

  2. Add a Soundcheck collectors field to the app-config.yaml file and reference the newly created pagerduty-facts-collectors.yaml file.

    # app-config.yaml
    soundcheck:
      collectors:
        pagerduty:
          $include: ./pagerduty-facts-collectors.yaml

Rate Limiting (Optional)​

This fact collector can be rate limited in Soundcheck using the following configuration:

soundcheck:
  job:
    workers:
      pagerduty:
        limiter:
          max: 400
          duration: 60000

PagerDuty API has a limit of 960 requests per minute. We recommend setting your rate limit to something below this, i.e. in the example above, we set the rate limit to 400 executions every minute.

This fact collector handles 429 rate limit errors from PagerDuty. Soundcheck will automatically wait and retry requests that are rate limited.

Defining PagerDuty Fact Collectors​

This section describes the data shape and semantics of PagerDuty Fact Collectors.

Overall Shape Of A PagerDuty Fact Collector​

The following is an example of a descriptor file for a PagerDuty Fact Collector:

---
frequency:
  cron: '0 * * * *'
filter:
  kind: 'Component'
cache:
  duration:
    hours: 2
collects:
  - type: Service
    filter:
      - spec.lifecycle: 'production'
        spec.type: 'website'
    cache: false

Below are the details for each field.

frequency [optional]​

The frequency at which the collector should be executed. Possible values are either a cron expression { cron: ... } or HumanDuration. This is the default frequency for each extractor.

filter [optional]​

A filter specifying which entities to collect the specified facts for. Matches the filter format used by the Catalog API. This is the default filter for each extractor.

cache [optional]​

If the collected facts should be cached, and if so, for how long. Possible values are either true or false or a nested { duration: HumanDuration } field. This is the default cache config for each extractor.

collects [required]​

An array describing which facts to collect and how to extract them. See below for details on the overall shape of a fact extractor.

Overall Shape Of A Fact Extractor​

Each extractor supports the fields described below.

type [required]​

The type of the extractor (e.g. Service, Standards).

frequency [optional]​

The frequency at which the fact extraction should be executed. Possible values are either a cron expression { cron: ... } or HumanDuration. If provided, it overrides the default frequency provided at the top level. If not provided, it defaults to the frequency provided at the top level. If neither extractor's frequency, nor default frequency is provided, the fact will only be collected on demand. Example:

frequency:
  minutes: 10

initialDelay [optional]​

The amount of time that should pass before the first invocation happens. Possible values are either a cron expression { cron: ... } or HumanDuration.

Example:

initialDelay:
  seconds: 30

filter [optional]​

A filter specifying which entities to collect the specified facts for. Matches the filter format used by the Catalog API. If provided, it overrides the default filter provided at the top level. If not provided, it defaults to the filter provided at the top level. If neither extractor's filter, nor default filter is provided, the fact will be collected for all entities.

cache [optional]​

If the collected facts should be cached, and if so, for how long. Possible values are either true or false or a nested { duration: HumanDuration } field. If provided, it overrides the default cache config provided at the top level. If not provided, it defaults to the cache config provided at the top level. If neither extractor's cache, nor default cache config is provided, the fact will not be cached. Example:

cache:
  duration:
    hours: 24

Entity configuration​

In your catalog-info.yaml file, add following metadata annotation to allow the plugin to map an entity to a service in PagerDuty.

metadata:
  annotations:
    pagerduty.com/service-id: pd-test-service-id

Alternatively, you can also configure an integration key.

metadata:
  annotations:
    pagerduty.com/integration-key: pd-test-integration-key

Collecting Service Facts​

A Service Fact contains information about a service from the PagerDuty Service API.

Shape of A Service Fact Collector​

The shape of a Service Fact Collector matches the Overall Shape Of A PagerDuty Fact Collector (restriction: type: Service).

Here's an example of a Service Fact Collector configuration:

- type: Service
  cache: true
  frequency:
    cron: '0 * * * *'

Shape of A Service Fact​

The shape of a Service Fact is based on the Soundcheck Fact Schema.

Shape of A Service Fact Check​

The shape of a Service Fact Check matches the Shape of a Fact Check.

Here's an example of a Service Fact Check configuration:

soundcheck:
  checks:
    - id: requires_type_to_be_service
      description: Requires type to be service
      passedMessage: The check has passed!
      failedMessage: The check has failed!
      rule:
        factRef: pagerduty:default/service
        path: $.type
        operator: equal
        value: service
      schedule:
        frequency:
          minutes: 1
        filter:
          metadata.tags: some-tag

The following is an example response from the Service Fact collector. See PagerDuty's documentation for more detail.

{
  "service": {
    "id": "PIJ90N7",
    "type": "service",
    "summary": "My Application Service",
    "self": "https://api.pagerduty.com/services/PIJ90N7",
    "html_url": "https://subdomain.pagerduty.com/service-directory/PIJ90N7",
    "name": "My Application Service",
    "auto_resolve_timeout": 14400,
    "acknowledgement_timeout": 600,
    "created_at": "2015-11-06T11:12:51-05:00",
    "status": "active",
    "alert_creation": "create_alerts_and_incidents",
    "integrations": [
      {
        "id": "PQ12345",
        "type": "generic_email_inbound_integration_reference",
        "summary": "Email Integration",
        "self": "https://api.pagerduty.com/services/PIJ90N7/integrations/PQ12345",
        "html_url": "https://subdomain.pagerduty.com/services/PIJ90N7/integrations/PQ12345"
      }
    ],
    "escalation_policy": {
      "id": "PT20YPA",
      "type": "escalation_policy_reference",
      "summary": "Another Escalation Policy",
      "self": "https://api.pagerduty.com/escalation_policies/PT20YPA",
      "html_url": "https://subdomain.pagerduty.com/escalation_policies/PT20YPA"
    },
    "teams": [
      {
        "id": "PQ9K7I8",
        "type": "team_reference",
        "summary": "Engineering",
        "self": "https://api.pagerduty.com/teams/PQ9K7I8",
        "html_url": "https://subdomain.pagerduty.com/teams/PQ9K7I8"
      }
    ],
    "incident_urgency_rule": {
      "type": "use_support_hours",
      "during_support_hours": {
        "type": "constant",
        "urgency": "high"
      },
      "outside_support_hours": {
        "type": "constant",
        "urgency": "low"
      }
    },
    "support_hours": {
      "type": "fixed_time_per_day",
      "time_zone": "America/Lima",
      "start_time": "09:00:00",
      "end_time": "17:00:00",
      "days_of_week": [
        1,
        2,
        3,
        4,
        5
      ]
    },
    "scheduled_actions": [
      {
        "type": "urgency_change",
        "at": {
          "type": "named_time",
          "name": "support_hours_start"
        },
        "to_urgency": "high"
      }
    ],
    "auto_pause_notifications_parameters": {
      "enabled": true,
      "timeout": 300
    }
  }
}

Collecting Standards Fact​

The Standards Fact contains information about standards from PagerDuty Standards API.

Shape of A Standards Fact Collector​

The shape of a Standards Fact Collector matches the Overall Shape Of A PagerDuty Fact Collector (restriction: type: Standards).

Here's an example of a Standards Fact Collector configuration:

- type: Standards
  cache: true
  frequency:
    cron: '0 * * * *'

Shape of A Standards Fact​

The shape of a Standards Fact is based on the Soundcheck Fact Schema.

Shape of A Standards Fact Check​

The shape of a Standards Fact Check matches the Shape of a Fact Check.

Here's an example of a Standards Fact Check:

soundcheck:
  checks:
    - id: requires_resource_type_to_be_technical_service
      description: Requires resource type to be technical_service
      passedMessage: The check has passed!
      failedMessage: The check has failed!
      rule:
        factRef: pagerduty:default/standards
        path: $.resource_type
        operator: equal
        value: technical_service
      schedule:
        frequency:
          minutes: 1
        filter:
          metadata.tags: some-tag

The following is an example response from the Standards Fact collector. See PagerDuty's documentation for more detail.

{
  "standards": [
    {
      "active": true,
      "description": "A description provides critical context about what a service represents or is used for to inform team members and responders. The description should be kept concise and understandable by those without deep knowledge of the service.",
      "id": "01CXX38Q0U8XKHO4LNKXUJTBFG",
      "pass": true,
      "name": "Service has a name",
      "type": "has_technical_service"
    },
    {
      "active": true,
      "description": "Ensure that no incident goes unaddressed, even if the on-call responder on the first level of the escalation policy is unavailable.",
      "exclusions": [],
      "id": "01CXX38Q0Y8D9IYFAEDCH5F53L",
      "inclusions": [],
      "name": "Service has an escalation policy with 2 or more unique levels",
      "resource_type": "technical_service",
      "type": "minimum_escalation_policy_rule_depth"
    },
  ]
}

Collecting Incidents Fact​

The Incidents Fact contains information about incidents from PagerDuty Incidents API.

Shape of A Incidents Fact Collector​

The shape of an Incidents Fact Collector matches the Overall Shape Of A PagerDuty Fact Collector (restriction: type: Incidents).

Here's an example of an Incidents Fact Collector configuration:

- type: Incidents
  statuses:
    - triggered
    - acknowledged
    - resolved
  cache: true
  frequency:
    cron: '0 * * * *'

Provide a factName to define mutliple configurations of the Incidents Fact Collector:

- factName: triggered_incidents
  type: Incidents
  statuses:
    - triggered
  cache: true
  frequency:
    cron: '0 * * * *'
- factName: acknowledged_and_recieved_incidents
  type: Incidents
  statuses:
    - acknowledged
    - resolved
  cache: true
  frequency:
    cron: '0 * * * *'

Shape of An Incidents Fact​

The shape of an Incidents Fact is based on the Soundcheck Fact Schema.

Shape of An Incidents Fact Check​

The shape of an Incidents Fact Check matches the Shape of a Fact Check.

Here's an example of an Incidents Fact Check:

soundcheck:
  checks:
    - id: has_less_than_5_incidents
      rule:
        factRef: pagerduty:default/Incidents # default fact name ref
        path: $.total
        operator: lessThan
        value: 5
    - id: has_less_than_5_triggered_incidents
      rule:
        factRef: pagerduty:default/triggered_incidents # custom factName ref
        path: $.total
        operator: lessThan
        value: 5

The following is an example response from the Incidents Fact Collector. See PagerDuty's documentation for more detail.

"total": 1,
"incidents": [
  {
    "id": "PT4KHLK",
    "type": "incident",
    "summary": "[#1234] The server is on fire.",
    "self": "https://api.pagerduty.com/incidents/PT4KHLK",
    "html_url": "https://subdomain.pagerduty.com/incidents/PT4KHLK",
    "incident_number": 1234,
    "title": "The server is on fire.",
    "created_at": "2015-10-06T21:30:42Z",
    "updated_at": "2015-10-06T21:40:23Z",
    "status": "triggered",
    "incident_key": "baf7cf21b1da41b4b0221008339ff357",
    "service": {
      "id": "pd-test-service-id",
      "type": "service_reference",
      "summary": "My Mail Service",
      "self": "https://api.pagerduty.com/services/PIJ90N7",
      "html_url": "https://subdomain.pagerduty.com/service-directory/PIJ90N7",
    },
    "assignments": [],
    "assigned_via": "escalation_policy",
    "last_status_change_at": "2015-10-06T21:38:23Z",
    "resolved_at": "2015-10-06T21:38:23Z",
    "first_trigger_log_entry": {
      "id": "Q02JTSNZWHSEKV",
      "type": "trigger_log_entry_reference",
      "summary": "Triggered through the API",
      "self": "https://api.pagerduty.com/log_entries/Q02JTSNZWHSEKV?incident_id=PT4KHLK",
      "html_url":
        "https://subdomain.pagerduty.com/incidents/PT4KHLK/log_entries/Q02JTSNZWHSEKV",
    },
    "alert_counts": {
      "all": 1,
      "triggered": 1,
      "resolved": 0,
    },
    "is_mergeable": true,
    "escalation_policy": {
      "id": "PT20YPA",
      "type": "escalation_policy_reference",
      "summary": "Another Escalation Policy",
      "self": "https://api.pagerduty.com/escalation_policies/PT20YPA",
      "html_url": "https://subdomain.pagerduty.com/escalation_policies/PT20YPA",
    },
    "teams": [
      {
        "id": "PQ9K7I8",
        "type": "team_reference",
        "summary": "Engineering",
        "self": "https://api.pagerduty.com/teams/PQ9K7I8",
        "html_url": "https://subdomain.pagerduty.com/teams/PQ9K7I8",
      },
    ],
    "pending_actions": [],
    "acknowledgements": [],
    "alert_grouping": {
      "grouping_type": "advanced",
      "started_at": "2015-10-06T21:30:42Z",
      "ended_at": null,
      "alert_grouping_active": true,
    },
    "last_status_change_by": {
      "id": "PXPGF42",
      "type": "user_reference",
      "summary": "Earline Greenholt",
      "self": "https://api.pagerduty.com/users/PXPGF42",
      "html_url": "https://subdomain.pagerduty.com/users/PXPGF42",
    },
    "priority": {
      "id": "P53ZZH5",
      "type": "priority_reference",
      "summary": "P2",
      "self": "https://api.pagerduty.com/priorities/P53ZZH5",
    },
    "resolve_reason": null,
    "conference_bridge": {
      "conference_number": "+1-415-555-1212,,,,1234#",
      "conference_url": "https://example.com/acb-123",
    },
    "incidents_responders": [],
    "responder_requests": [],
    "urgency": "high",
  },
]

Shape of A PagerDuty Track​

The following is an example of the Soundcheck track that utilizes these checks

soundcheck:
  programs:
    - id: demo
      name: Demo
      ownerEntityRef: group:default/owning_group
      description: Demonstration of Soundcheck PagerDuty Fact Collector
      levels:
        - ordinal: 1
          name: First level
          description: Checks leveraging SoundCheck's PagerDuty Fact Collector
          checks:
            - id: requires_resource_type_to_be_technical_service
              name: Requires resource type to be technical service
              description: Requires resource type to be technical service
            - id: requires_type_to_be_service
              name: Requires type to be service
              description: Requires type to be service
      filter:
        catalog:
          metadata.tags: some-tag

For a description of the data collected about a service, refer to the PagerDuty Service API documentation.