Soundcheck Backend

Spotify Plugins for Backstage: Soundcheck - Backend

Soundcheck incentivizes quality, reliability, and alignment of your software ecosystem. With Soundcheck, engineering organizations define development and operational standards, and measure the health of software components. Soundcheck provides teams with strong guidance to cultivate behavior and improve an organization's DevOps practices. Prioritize and visualize tech health and alignment to organizational best practices within Backstage.

There are 6 fundamental elements that make up Soundcheck:

Fact: Un-opinionated information about a component.
Check: A standard or best practice a component is graded against.
Check Result: The result of running a check against a component. Possible results are passed, failed, warning or not applicable.
Program: A tech health initiative, organized into levels.
Level: A group of checks that represent a milestone within a program.
Certification: The outcome of passing all applicable checks within a level.

Together, they show you how any given software component is performing against your organization's tech health initiatives.

Prerequisites

1. Complete Spotify plugin bundle setup

This package is a Backstage plugin and is part of the “Spotify Plugins for Backstage” bundle. In order for you to use this plugin, you must purchase a license key at https://backstage.spotify.com/. After purchasing a license key, follow the instructions on https://backstage.spotify.com/account before continuing.

2. Check your Backstage version

Please ensure your Backstage version matches the supported version shown under the installation instructions on https://backstage.spotify.com/account, and always check compatibility before updating Backstage.

3. Set Node options

If you're running Backstage with Node 20 or later, you'll need to pass the flag --no-node-snapshot to Node in order to use Soundcheck's message templating feature.

One way to do this is to specify the NODE_OPTIONS environment variable before starting Backstage: export NODE_OPTIONS=--no-node-snapshot

Installation

The next steps describe how to install the Soundcheck backend plugin.

1. Install the plugins

Add the Soundcheck packages as dependencies to your Backstage instance

yarn workspace backend add @spotify/backstage-plugin-soundcheck-backend

2. Install Soundcheck backend

The Soundcheck backend handles the ingestion of check results in the database, and serves data to the Soundcheck UI.

Create a new file, packages/backend/src/plugins/soundcheck.ts with the following content:

import { SoundcheckBuilder } from '@spotify/backstage-plugin-soundcheck-backend';
import { Router } from 'express';
import { PluginEnvironment } from '../types';

export default async function createPlugin(
  env: PluginEnvironment,
): Promise<Router> {
  return SoundcheckBuilder.create({ ...env }).build();
}

Set up the Soundcheck backend router in /packages/backend/src/index.ts. You will need to import the module from the previous step, create a new plugin environment for Soundcheck and add the router to the express app.

  // packages/backend/src/index.ts

  /* ... */

  import proxy from './plugins/proxy';
  import techdocs from './plugins/techdocs';
  import search from './plugins/search';
+ import soundcheck from './plugins/soundcheck';

  /* ... */

  const techdocsEnv = createEnv('techdocs');
  const searchEnv = createEnv('search');
  const appEnv = createEnv('app');
+ const soundcheckEnv = createEnv('soundcheck');

  /* ... */

  apiRouter.use('/techdocs', await techdocs(techdocsEnv));
  apiRouter.use('/proxy', await proxy(proxyEnv));
  apiRouter.use('/search', await search(searchEnv));
+ apiRouter.use('/soundcheck', await soundcheck(soundcheckEnv));

If you are using the New Backend System, instead of the above, you can just add the following to your index.ts file:

const backend = createBackend();

backend.add(import('@backstage/plugin-app-backend/alpha'));
+ backend.add(import('@spotify/backstage-plugin-soundcheck-backend'));
// ...

backend.start();

3. Add program config

Programs, levels and checks are driven by config. To learn more about the config, see the Defining Programs, Levels and Checks section.

Create soundcheck-programs.yaml in the root of your Backstage repository and fill in all your checks and programs. A simple example program with a single level and check is listed below.

Note: this file will be loaded at runtime along with the rest of your Backstage configuration files, so make sure it's available in deployed environments in the same way as your app-config.yaml files.

---
- id: test-certified
  name: Test Certified
  ownerEntityRef: group:default/example-owner
  description: >
    Improve quality and reliability of your software component
    by measuring the use of testing best practices.
  documentationURL: https://www.backstage.io
  levels:
    - ordinal: 1
      checks:
        - id: tests-run
          name: Tests run on CI
          description: >
            Indicates whether your system is set up correctly to run tests and
            report the results. You must have at least one test.

Add a soundcheck field to app-config.yaml and reference the newly created soundcheck-programs.yaml
```
# app-config.yaml
soundcheck:
  programs:
    $include: ./soundcheck-programs.yaml
```

4. Next Steps

Install the Soundcheck frontend plugin. Installation instructions can be found here.

5. Scaling & Rate Limiting (Optional)

To optimize the scalability of Soundcheck, it is highly recommended to configure a Redis connection. This integration enables Soundcheck to efficiently distribute tasks such as fact collection and check executions across multiple instances of the soundcheck-backend.

To configure Soundcheck to use Redis add the following to your app-config.yaml file:

soundcheck:
  job:
    queues:
      type: redis
      host: <redis-host>
      port: <redis-port>
      username: <redis-username>
      password: <redis-password>

Rate Limiting

Soundcheck offers configurable rate limiting to control the frequency of fact collections on a per-fact collector basis. This feature is useful when a fact collector leverages a third-party API that imposes rate limits (e.g., GitHub API).

Configure rate limits for specific fact collectors by enhancing your app-config.yaml file with the following details:

soundcheck:
  job:
    workers:
      <fact_collector_id>:
        limiter:
          max: <max_number_of_jobs>
          duration: <duration_>

The max parameter establishes an upper threshold for the number of fact collection jobs Soundcheck will perform from with the specified collector within the given duration. Should the job count exceed this limit, additional jobs are queued.

The duration parameter determines the time window during which the specified maximum number of collection jobs (set by max) is enforced. After this duration, the count resets, allowing Soundcheck to process a new set of collection jobs up to the specified maximum.

For example we can limit Soundcheck to only requesting 5000 fact collections per hour from the github fact using the following configuration:

soundcheck:
  job:
    workers:
      github:
        limiter:
          max: 5000
          duration: 3600000

By default each worker has a rate limit of max: 100 and duration: 1000, aka 100 executions per second.

Automatic Rate Limiting

Many of Soundcheck's fact collectors will automatically handle rate limit errors that call third party APIs (provided the third party integration implements this feature) by pausing collection temporarily and retrying after some time. This pause time is determined by the third party integration, but has a default of 60 seconds.

6. Slack Notifications (Optional)

To enable Slack notifications, Slack needs to be integrated within Soundcheck.

Note: This is an optional Soundcheck feature. If you choose not to enable this feature, you can skip the instructions below.

To integrate Slack within Soundcheck for the first time, follow the steps below.

Manifest file

To create a custom Slack app, use the manifest file provided below. The provided file lists the scopes required for Soundcheck notifications.

Refer to Slack's Create apps using manifests documentation to learn more.

The first step in the Slack documentation linked above contains a Click to create Slack app button that will allow you to begin the Slack app creation process. Clicking on the button will open a Create an app prompt with two options: from scratch or from an app manifest. Select from an app manifest in order to use the manifest provided below.

Step 1 of 3 in the Slack app creation process consists of picking a workspace to develop your Slack app in. Step 2 of 3 will prompt you to enter the app manifest. Click on the YAML tab and copy/paste the manifest below.

# This file is an example Slack app manifest used for installing a custom app
# in your workspace to deliver Soundcheck notifications. For more information,
# see https://api.slack.com/reference/manifests
display_information:
  name: Soundcheck for Backstage
  description: Enables notifications from Soundcheck for Backstage in Slack.
  background_color: '#9BF0E1'
features:
  bot_user:
    display_name: Soundcheck for Backstage
    always_online: true
oauth_config:
  scopes:
    bot:
      # scopes needed to read user information
      - users:read
      - users:read.email
      # scopes needed to send messages
      - chat:write
      - chat:write.public
settings:
  interactivity:
    is_enabled: true
  org_deploy_enabled: false
  socket_mode_enabled: true
  token_rotation_enabled: false

Step 3 of 3 displays an app creation review summary. Click on the Create button to complete the process.

Tokens

A custom Slack app, an app token, an oauth token, and a signing secret must be created to set up Slack integration.

app token: The app token can be found, or generated, in the App-Level Token box. If you are generating an app token for the first time, be sure to give it the connections:write scope.
oauth token: The oauth token can be found in the Oauth & Permissions section of the Oauth tokens for Your Workspace box. It should look something like xoxb-....
signing secret: Go to the Basic Information section of your Slack app. The signing secret can be found in the App Credentials box.

Next, the Slack app token, oauth token, and signing secret must be added to your app-config.yaml (or equivalent) file.

soundcheck:
  # ...
  slack:
    token: xoxb-*************-*************-************************ #aka "oauth token", or "bot token"
    appToken: xapp-*-***********-*************-****************************************************************
    signingSecret: ********************************
  # ...

Socket Mode

Socket mode must be enabled for your Slack app. Under Settings, click on Socket Mode. Then, click on the Enable Socket Mode switch to toggle it on. Finally, make sure that the Interactivity & Shortcuts feature is enabled.

Adding the Slack app to a Slack channel

In order to get Slack notifications from your Slack app, it must be added to a Slack channel. The Slack channel can be either a public or a private channel that lives in the same workspace that you selected when creating the Slack app.

Right-click on the desired channel in Slack, select View Channel Details, click on the Integrations tab, and then click on Add Apps. Search for your Slack app by name, find it in the search results, and then click on the Add button.

7. Email Notifications (Optional)

To enable email notifications, SMTP server details must be configured within Soundcheck.

Note: This is an optional Soundcheck feature. If you choose not to enable this feature, you can skip the instructions below.

To integrate email notifications within Soundcheck for the first time, an email configuration must be added to your app-config.yaml (or equivalent) file. Examples are provided below.

SMTP Connection Authentication

Here's an example of the email configuration for username and password authentication.

soundcheck:
  # ...
  email:
    # hostname or IP address of the SMTP server
    host: sandbox.smtp.mailtrap.io
    # port to connect to
    port: 2525
    # sender email address
    from: info@soundcheck.com
    # authentication details
    auth:
      # username
      user: d92xxxxxxxxxxx
      # password
      pass: 3f4xxxxxxxxxxx
  # ...

Here's an example of the email configuration for OAuth2 authentication.

soundcheck:
  # ...
  email:
    # hostname or IP address of the SMTP server
    host: smtp.gmail.com
    # port to connect to
    port: 465
    # sender email address
    from: info@soundcheck.com
    # authentication details
    auth:
      # username
      user: user@example.com
      # client id of the application
      clientId: 000000000000-xxx0.apps.googleusercontent.com
      # client secret of the application
      clientSecret: XxxxxXXxX0xxxxxxxx0XXxX0
      # optional refresh token (used to generate a new access token if the existing one expires/fails)
      refreshToken: 1/XXxXxsss-xxxXXXXXxXxx0XXXxxXXx0x00xxx
      # optional access token
      accessToken: ya29.Xx_XX0xxxxx-xX0X0XxXXxXxXXXxX0x
  # ...

Node Options

If you're running Backstage with Node 20 or later, you'll need to pass the flag --no-node-snapshot to Node in order to use the email notifications feature. One way to do this is to specify the NODE_OPTIONS environment variable before starting Backstage.

export NODE_OPTIONS=--no-node-snapshot

Defining Programs, Levels and Checks

This section describes the data shape and semantics of Soundcheck programs.

Overall Shape Of A Program

The following is an example of a descriptor file for a Soundcheck Program:

id: gs-sherlock-holmes
name: Golden State for Sherlock Holmes
ownerEntityRef: group:default/ArthurConanDoyle
description: >
  Score the quality of your Sherlock Holmes adaptation against literary standards.
documentationUrl: https://en.wikipedia.org/wiki/Sherlock_Holmes
group: gs-public-domain-lit
filter:
  catalog:
    kind: story
    spec.type: sherlock
levels:
  - ordinal: 1
    name: Looks like Sherlock
    description: Character is physically recognizable as "Sherlock".
    checks:
      - id: wears-hat
        name: Wears a Hat
        description: >
          Sherlock is known for his deerstalker hat, a standard Sherlock character should
          incorporate a distinctive hat accessory.
      - id: home-address
        name: Lives at 221B
        description: >
          Sherlock's address is as famous as his name, standard adaptations should
          attempt to house the detective at 221 Baker St apartment B.
      - id: smokes-tobacco
        name: Smokes Tobacco
        description: >
          Sherlock traditionally smokes tobacco from cigars or a pipe but the exact form
          is left to the individual interpretation.  This check only applies to adaptations
          targeting adults.
        filter:
          catalog:
            metadata.annotations['target-audience']: 'over-18'
      - id: uses-narcotics
        name: Abuses Illicit Substances
        description: >
          One of Sherlock's defining flaws is that he abuses narcotics.  Standard Holmesian
          characters should incorporate this flaw.  This check only applies to adaptations
          targetting adults.
        filter:
          catalog:
            metadata.annotations['target-audience']: 'over-18'
  - ordinal: 2
    name: Embodies Sherlock
    description: Character adaptation goes deeper than physical resemblance.
    checks:
      - id: hyper-observant
        name: Character Notices Everything
        description: >
          Sherlock is marked by his ability to deduce significantly more information
          from his surroundings than any person should be able to.
      - id: condescendingly-smart
        name: Condescendingly Smart
        description: >
          Sherlock is generally depicted as intelligent to the point of arrogance,
          demonstrated by condescending to other people without regard for their
          own specific competencies.
      - id: musical-talent
        name: Musical Virtuosity
        description: >
          Traditionally Sherlock is portrayed as a violinist but a standardized
          Sherlock should use some form of musical talent as a meditative practice.
  - ordinal: 3
    name: The Rest
    description: >
      Sherlock the person is only one part of Sherlock the character, just as important
      is the host of supporting characters built around him.
    checks:
      - id: watson
        name: The Sidekick
        description: >
          Sherlock is almost always accompanied by Dr. Watson, generally as his roommate,
          only friend and crime solving partner.
      - id: moriarty
        name: The Rival
        description: >
          Sherlock's abilities demand a suitably advanced rival.  Moriarty generally
          demonstrates a similar intelligence and and applies it in an equal, but
          opposed fashion, often turning Sherlock's most profound strengths into
          debilitating weakenesses.
      - id: mrs-hudson
        name: The Housekeeper
        description: >
          While on the surface she plays Sherlock's housekeeper and landlady, Mrs. Hudson
          is regularly utilized as a serendipitous font of inspiritation for the detective.

See below for details about these fields.

Program Fields

The Program object is composed of some top level summary fields as well as more complex nested fields for filters and levels.

`id`, `name` and `description` [required]

The id field is the unique identifier for this particular Soundcheck Program. All programs defined in your Backstage instance must have unique ids.

The name field is the human readable identifier for this particular Soundcheck Program. There is no uniqueness constraint, but collisions can lead to confusing behavior in the UI.

The description field provides additional details about the motivation behind this particular set of checks.

`ownerEntityRef` [required]

The ownerEntityRef field relates this particular Soundcheck Program to the person or team responsible for maintaining it. This string follows the structure outlined at: https://backstage.io/docs/features/software-catalog/references.

`documentationURL` and `group` [optional]

The documentationURL field specifies a link users can follow to find out more details about this particular program.

The group field is used to connect logically similar but functionally disjoint programs into families or 'groups'. An example of this is the "test-certified" group, which would reference programs such as "test-certified-for-services" and "test-certified-for-websites". These may have entirely different sets of checks in order to reflect the different type of entities they apply to, but fill the same need.

`filter` [optional]

The filter field allows program owners to specify the type of entities that a particular program applies to. For more information about the shape of the filter object see Entity Filter below.

`levels` [required]

The levels field offers an additional layer of specificity on top of the program definition, allowing program owners to provide a progressive certification for entity owners to track their progress against custom milestones. For more information about the shape of the Level object see Level Fields below.

Level Fields

The level object is composed of several top level summary fields and a repeated check definition field.

`ordinal` [required]

The ordinal field is used to uniquely identify a particular level within a program. Level ordinals are designed to be monotonically increasing, starting from 0 or 1, and are used to determine where this level fits within certification progression. An entity is only considered to be certified for a particular level "X" of a program if it passes all of the checks within level X and is certified for every level with an ordinal less than X.

`name` and `description` [optional]

The name field of a level provides a human readable title for levels for cases where the ordinal alone does not provide the whole story.

The description field allows for further expression regarding the purpose of a particular level. This field supports GitHub-flavored markdown.

`checks` [required]

The checks field of a level specifies all of the checks that an entity must pass in order to be certified against this particular level within this program. For more information about the shape of the Check object see Check Fields below.

Check Fields

The check object defines the atomic unit of work within the Soundcheck certification process. These fields are used to define what a particular program is certifying an entity against.

`id` [required]

The id field links this definition to the actual check results (regardless of origin) that have been consumed by Soundcheck.

`name` and `description` [required]

The name field of a check provides the human readable title for this check.

The description field is used to provide entity owners with additional context on the motivation and implementation of a particular check. This field supports strings or markdown.

`filter` [optional]

In addition to filtering programs against entities, program owners have the option to further filter the checks within a program definition. The filter field allows program owners to exempt entities from specific checks within a program when they do not make sense for entities of that type. For example the program "Test Certified For Services" may apply to services written in both NodeJs and Java, but the check "Uses Recent Java Version" would want to filter for only services that are tagged as "Java'. For more information about the shape of the filter object, see Entity Filter below.

`Entity` Filter

filter objects are used to narrow down both program and check applicability. Currently, Soundcheck only supports filtering based on the shape of a Software Catalog Entity. For example if the filter below was attached to the program "Java Production Services", only entities marked with the type "service", lifecycle "production", and tagged as "Java" would be eligible for certification against it.

filter:
  - catalog:
      spec.lifecycle: 'production'
      spec.type: 'service'
      spec.metadata.tags: 'java'

`catalog` [optional]

The catalog field is the container for filter objects related to the Software Catalog. This field can be populated by either a single CatalogFilter object (see below) or an array of CatalogFilters. When an array of filters is provided an Entity will match as long as ANY of the filters in the array are a match.

`CatalogFilter` Object

CatalogFilters are defined by a map of FilterKey to FilterValue (see below), and when a particular CatalogFilter contains multiple entries it is considered to match an Entity only if every entry matches that Entity.

`FilterKey` Object

FilterKeys represent a specific subset of fields within the Software Catalog Entity descriptor. FilterKeys fall into three categories:

String `FilterKey`s

String FilterKeys relate to a set of Software Catalog Entity properties that are represented by one String. The properties of this type that Soundcheck can filter against are:

'kind'
'spec.type'
'spec.lifecycle'
'spec.owner'
'spec.system'

Array `FilterKey`s

Array FilterKeys relate to Software Catalog Entity properties that are represented by repeated Strings. When a FilterKey is related to an Array field on an Entity, Soundcheck considers the Entity to be a match if any of those repeated Strings matches against the FilterValue object. The only property of this type that Soundcheck can filter against is 'metadata.tags'.

Record `FilterKey`s

Record FilterKeys relate to Software Catalog Entity properties that are represented by Maps. When a Record FilterKey is provided the specific key in the Software Catalog Entity property must also be provided, using square bracket ([...]) notation. The properties of this type that Soundcheck can filter against are 'metadata.annotations' and 'metadata.labels'.

Example specification: 'metadata.annotations["backstage.io/techdocs-ref"]'

`FilterValue` Object

FilterValues represent the pattern this particular filter is looking for in entities it is attempting to match against. There are three different types of FilterValue objects.

String FilterValue

When a String is provided as the FilterValue Soundcheck is checking whether the Entity property specified by the corresponding FilterKey matches (case INsensitive) the provided value.

filter:
  catalog:
    spec.lifecycle: 'production'

Array FilterValue

When an Array of Strings is provided as the FilterValue Soundcheck is checking whether the Entity property specified by the corresponding FilterKey matches (case INsensitive) any of the strings in the provided value.

filter:
  catalog:
    spec.type: ['service', 'website']

Presence FilterValue

ValueMatcher FilterValues are used to test for the presence or absence of a particular FilterKey in an Entity. Providing a ValueMatcher with * will consider any value (except undefined) to be a match, and a value of !* will only match for values that are missing or undefined.

filter:
  catalog:
    metadata.annotations["backstage.io/techdocs-ref"]: { match: '*' }
    metadata.annotations["wikipedia.org"]: { match: '!*' }

Check Result History

By default, Soundcheck retains only the latest result for each check, so that the size of the database is a function of the number of entities and the number of checks, rather than the number of check results. However, it's possible to enable tracking of check result history in order to track the change in check results over time.

Soundcheck does not currently provide an API for interacting with check result history. However, the check result history data is used by Soundcheck Tech Health page to visualize snapshots and trends of the check pass rates for every entity and their applicable checks. It's also possible for other systems to integrate with check result history by reading from the Soundcheck database directly.

Enabling Check Result History

To enable check result history, set soundcheck.results.history.enable to true in config:

# app-config.yaml
soundcheck:
  results:
    history:
      enable: true

Reading Check Result History

Check result history is stored in the check_result_history table in the Soundcheck database. Unless overridden in Backstage configuration, the Soundcheck database will be part of the main Backstage database, and named backstage_plugin_soundcheck.

SELECT *
FROM check_result_history
WHERE entity_ref = 'component:default/example-component';

The table contains the same fields as the object accepted by the check result submission endpoint, along with a unique check_result_history_id field and a timestamp corresponding to the time the result was submitted.

Certification History

By default, Soundcheck retains only the latest certifications for each entity, so that the size of the database is a function of the number of entities and the number of track levels, rather than the number of level certifications. However, it's possible to enable tracking of certification history in order to track the change in certifications over time.

Soundcheck does not currently provide an API for interacting with certification history. However, the certification history data is used by Soundcheck Tech Health page to visualize snapshots and trends of the level certification pass rates for every entity and their applicable tracks. It's also possible for other systems to integrate with certification history by reading from the Soundcheck database directly.

Enabling Certification History

To enable check result history, set soundcheck.certifications.history.enable to true in config:

# app-config.yaml
soundcheck:
  certifications:
    history:
      enable: true

Reading Certification History

Certification history is stored in the certification_history table in the Soundcheck database. Unless overridden in Backstage configuration, the Soundcheck database will be part of the main Backstage database, and named backstage_plugin_soundcheck.

SELECT *
FROM certification_history
WHERE entity_ref = 'component:default/example-component';

The table contains the same fields as the certification object, along with a unique certification_history_id field and atimestamp corresponding to the time of the certification.

Backfilling Certification History

If check result history was enabled prior to enabling certification history, certification history will be automatically backfilled based on the data stored in check_result_history table. Soundcheck will detect this situation upon startup and initiate a one-time "certify world" job that will certify all entities that have applicable tracks.

Additionally to the automatic backfill, certification backfill can also be initiated via REST API:

POST localhost:7007/api/soundcheck/certify_world

Request Body:

See Certification Request Schema.

Example:

{
  "scope": "default",
  "numberOfDays": 90,
  "batchSize": 100,
  "trackId": "test-certified"
}

Advanced Applicability Filters

In case if a standard Entity Filter is not enough for your use-case to define check's or track's applicability to an entity, you can set up advanced check and track applicability filters.

Configuring Check Applicability Filter

In packages/backend/src/plugins/soundcheck.ts, add the ApplicabilityFilter:

  import { SoundcheckBuilder } from '@spotify/backstage-plugin-soundcheck-backend';
  import { Router } from 'express';
  import { PluginEnvironment } from '../types';
+ import { ApplicabilityFilter } from "@spotify/backstage-plugin-soundcheck-node";

  export default async function createPlugin(
    env: PluginEnvironment,
  ): Promise<Router> {
+   const checkApplicabilityFilter: ApplicabilityFilter = async (
+     id: string, // check id
+     entity: Entity,
+   ): Promise<boolean> => {
+     // your implementation:
+     // return true if a check with the given id should apply to the given entity and false if not
+   };
    return SoundcheckBuilder.create({ ...env })
+     .addCheckApplicabilityFilters(
+       checkApplicabilityFilter,
+     )
      .build();
  }

Configuring Track Applicability Filter

In packages/backend/src/plugins/soundcheck.ts, add the ApplicabilityFilter:

  import { SoundcheckBuilder } from '@spotify/backstage-plugin-soundcheck-backend';
  import { Router } from 'express';
  import { PluginEnvironment } from '../types';
+ import { ApplicabilityFilter } from "@spotify/backstage-plugin-soundcheck-node";

  export default async function createPlugin(
    env: PluginEnvironment,
  ): Promise<Router> {
+   const trackApplicabilityFilter: ApplicabilityFilter = async (
+     id: string, // track id
+     entity: Entity,
+   ): Promise<boolean> => {
+     // your implementation:
+     // return true if a track with the given id should apply to the given entity and false if not
+   };
    return SoundcheckBuilder.create({ ...env })
+     .addTrackApplicabilityFilters(
+       trackApplicabilityFilter,
+     )
      .build();
  }

Integrating with Soundcheck

Soundcheck enables developers and other stakeholders to evaluate how closely their software components align with their organization's development and operational standards. These standards are established through the creation of programs which Soundcheck uses to assess and visualize a software component's alignment with these standards through check results.

Organizations can run their own checks and submit the results to Soundcheck. This allows organizations to incorporate checks into their build pipelines, or push results to Soundcheck from other first- or third-party tools they may be using for automated checks. Check results are submitted to Soundcheck via the Check Results API.

Alternatively, organizations may leverage the Soundcheck Fact Framework, which provides a mechanism by which Soundcheck itself can collect facts about an entity in Backstage, and execute checks based on those facts.

Fact Framework

Soundcheck's fact framework enables Soundcheck to collect facts (un-opinionated information) about an entity. Soundcheck can collect facts about an entity using Fact Collectors, or receive facts on entities through the submission of facts via the Facts API.

Fact Collectors

Soundcheck can be extended with additional fact collectors. A fact collector can collect one or more facts on a given entity.

Built-in Fact Collectors

Soundcheck comes with two built-in fact collectors: catalog and soundcheck. These fact collectors are provided by default with additional configuration required. A check using facts from these collectors can be defined normally. However, if you'd like the check to execute periodically the check must have a schedule with filter because facts from these collectors are not collected on a schedule. Catalog and soundcheck collectors do not require a collector.yaml file to be present, just the checks.

Catalog Fact Collector

The catalog fact collector exposes information from Backstage's Software Catalog as facts to Soundcheck. It provides a single fact on entities: catalog:default/entity_descriptor which provides the entity's descriptor as fact data.

This enables the creation of checks against an entity's metadata, to ensure that it is in compliance with your organizations standards and best practices.

An example fact collected by the catalog fact collector:

factRef: catalog:default/entity_descriptor
entityRef: component:default/artist-web
data:
  apiVersion: backstage.io/v1alpha1
  kind: Component
  metadata:
    name: artist-web
    description: The place to be, for great artists
    labels:
      example.com/custom: custom_label_value
    annotations:
      example.com/service-discovery: artistweb
      circleci.com/project-slug: github/example-org/artist-website
    tags:
      - java
    links:
      - url: https://admin.example-org.com
        title: Admin Dashboard
        icon: dashboard
        type: admin-dashboard
  spec:
    type: website
    lifecycle: production
    owner: artist-relations-team
    system: public-websites
timestamp: 2023-02-20T13:50:35Z

An example catalog check:

- id: has_required_tags
  rule:
    any:
      - factRef: catalog:default/entity_descriptor
        path: $.metadata.tags
        operator: contains
        value: java
      - factRef: catalog:default/entity_descriptor
        path: $.metadata.tags
        operator: contains
        value: data
  schedule:
    frequency:
      cron: '* * * * *'
    filter:
      kind: 'Component'
  passedMessage: |
    Tag found, check passed.
  failedMessage: |
    No `java` or `data` tag found, check failed.

Soundcheck Fact Collector

The Soundcheck fact collector exposes Soundcheck program certifications as facts. It provides facts for each Soundcheck program using the fact reference: soundcheck:default/program/:programId where :programId is the identifier for the program whose certification is contained in the fact.

This enables the creation of checks against an entity's certification level in other programs.

Third Party Integrations

Soundcheck currently supports two of out-of-the-box third party integrations (TPI):

Source Control Management
GitHub

Source Code Management (SCM) Fact Collector

The SCM fact collector provides integration with source control management providers (GitHub, bitbucket, etc.). The SCM module allows for fact extraction from the files hosted by these providers. The types of fact extractions, the list of providers, and full configuration and specification details regarding the SCM module can be found in the SCM module's README.md file.

GitHub Fact Collector

The GitHub fact collector enables Soundcheck to gather facts regarding GitHub repositories. For full details, see the GitHub module's README.md file.

Fact Checks

A fact check defines a rule, comprised of one or more conditions: a combination of facts, operators, and values that determine whether the check emits a passed or failed check result to Soundcheck.

Shape of a Fact Check

id [required] - The unique identifier for this fact check. Check results emitted as a result of this check use this id.
name [optional] - A name for the fact check suitable for display on a user interface.
description [optional] - A description of the fact check suitable for display on a user interface.
rule [required] - One or more conditions that determine whether the check emits a passed or failed check result to Soundcheck.
passedMessage [optional] - A message to be included in the check result if the check passes.
failedMessage [optional] - A message to be included in the check result if the check fails.
warning [optional] - A boolean flag that indicates whether the check should emit a warning check result instead of failed if the check fails. Defaults to false. Warning check results don't impact the certification progress of an entity.
schedule [optional] - How often the fact check should be executed, and on what entities. This field is necessary for catalog and soundcheck checks if you would like those checks executed periodically.

Below is an example fact check which checks if a PagerDuty integration key is present in an entity's metadata.

id: has_pagerduty_integration_key
rule:
  factRef: catalog:default/entity_descriptor
  path: $.metadata.annotations[pagerduty.com/integration-key]
  operator: matches
  value: .+
failedMessage: |
  No PagerDuty Integration Key defined for this entity.

Rules

A rule contains one or more conditions.

The simplest form of a condition consists of a fact reference, an operator, and a value. When the check is executed, the operator is used to compare the fact against the value.

Example of a simple condition:

rule:
  factRef: catalog:default/entity_descriptor
  path: $.metadata.annotations[pagerduty.com/integration-key]
  operator: matches
  value: .+
  failedMessage: |
    No PagerDuty Integration Key defined for this entity.

More complex conditions can by created through the use of boolean expressions. A rule may optionally have either an all or an any expression at its root, containing an array of conditions. The all operator specifies that all conditions contained within must be truthy for the check to pass. The any operator only requires one condition to be truthy for the check to pass.

Example of a complex condition:

rule:
  any:
    - factRef: catalog:default/entity_descriptor
      path: $.metadata.labels['language']
      operator: equal
      value: java
    - factRef: catalog:default/entity_descriptor
      path: $.metadata.labels['language']
      operator: equal
      value: kotlin

Operators

Soundcheck includes the following operators by default:

String, Numeric, and Boolean
- equal - fact must equal value
- notEqual - fact must not equal value
String
- matches - fact string matches the given value regex
String (SemVer) - fact strings which can be loosely coerced to a SemVer. value strings which can be coerced to a SemVer or a Range depending on the operator.
- semverGt - fact version must be greater than value version
- semverGte - fact version must be greater than or equal to value version
- semverLt - fact version must be less than value version
- semverLte - fact version must be less than or equal to value version
- semverEq - fact version must equal value version
- semverNeq - fact version must not equal value version
- semverSatisfies - fact version satisfies the value range
- semverGtr - fact version is greater than all the versions possible in the value range
- semverLtr - fact version is less than all the versions possible in the value range
Numeric
- lessThan - fact must be less than value
- lessThanInclusive- fact must be less than or equal to value
- greaterThan - fact must be greater than value
- greaterThanInclusive - fact must be greater than or equal to value
Array
- in - fact must be included in value (an array)
- notIn - fact must not be included in value (an array)
- contains - fact (an array) must include value
- doesNotContain - fact (an array) must not include value
- hasLengthOf - fact (an array) must have length of value
DateTime
- after - fact DateTime must be after value DateTime. fact DateTime should be given as an ISO 8601 DateTime. value should be given as:
  - ISO 8601 DateTime. Example:
    - 2023-09-07T21:41:49+0000 - Is the fact after 21:41:49 September 7th 2023
  - now - If now is given, the fact DateTime will be measured against the time when the check occurs in UTC.
  - ISO 8601 Duration. Examples:
    - P6M - Is the fact after 6 months from now.
    - -P6D - Is the fact after 6 days ago.
- before - fact DateTime must be before value DateTime. fact DateTime should be given as an ISO 8601 DateTime. value DateTime should be given as:
  - ISO 8601 DateTime. Example:
    - 2023-09-07T21:41:49+0000 - Is the fact before 21:41:49 September 7th 2023
  - now - If now is given, the fact DateTime will be measured against the time when the check occurs in UTC.
  - ISO 8601 Duration. Examples:
    - P1Y - Is the fact before 1 year from now.
    - -P6M - Is the fact before 6 months ago.

Additionally, you can extend Soundcheck's functionality by adding your own operators. To do so implement an Operator and pass it to the SoundcheckBuilder when creating Soundcheck.

For example, to add an operator that checks if a string is an anagram of another string:

import { Operator } from '@spotify/backstage-plugin-soundcheck-node';

const isAnagramOf = new Operator<string, string>({
  name: 'isAnagramOf',
  evaluator: (fact: string, value: string) => {
    // O(n log n) implementation for brevity
    return fact.split('').sort().join('') === value.split('').sort().join('');
  },
});

Then when creating Soundcheck, pass the operator to the SoundcheckBuilder:

return SoundcheckBuilder.create({ ...env })
  .addOperators([isAnagramOf])
  .build();

Now the isAnagramOf operator can be used in your rules:

rule:
  factRef: catalog:default/entity_descriptor
  path: $.metadata.labels['anagram']
  operator: isAnagramOf
  value: listen

This rule will pass if the entity's metadata.labels['anagram'] is an anagram of 'listen' (e.g., 'silent') otherwise it will fail.

Messages

Pass messages are displayed to the user when a check passes. Fail messages are displayed to the user when a check fails. Messages can be provided as a string or as a markdown string. Pass and fail messages also support Liquid templating. The following variables are available for use in templates:

entity - The entity the check was executed on.
facts - A map of facts used by the check. Individual facts can be looked up in this map by fact reference. For example, facts['catalog:default/entity_descriptor'].
- If all the facts used by the check have unique fact names they can be accessed in facts by fact name rather than having to use the entire fact reference. For example, if check uses the catalog:default/entity_descriptor it can be accessed by facts['catalog:default/entity_descriptor'] or facts['entity_descriptor'].
- If only a single fact is used by the check, the fact can be accessed directly. For example, if catalog:default/entity_descriptor was the only fact used by the check it can be accessed as fact rather than facts['catalog:default/entity_descriptor'].

Here is an example of a pass message using templates:

{{ entity.metadata.name }} has a PagerDuty Integration Key defined.

in this example, Soundcheck would replace {{ entity.metadata.name }} with the name of the entity the check was executed on.

Schedule

A fact check may optionally be scheduled by providing how often the fact check should be executed and which entities it should be executed on.

frequency [required] - A cron expression { cron: ... } or HumanDuration, describing the frequency on which Soundcheck should execute the check.
entityRefs [optional] - The entities to run the check on. If this is provided, filter is ignored. If neither this nor filter are provided the check will be executed on all entities.
filter - A filter specifying which entities to run the check on. If entityRefs is provided the filter is ignored. If neither this nor filter are provided the check will be executed on all entities. See the filter format used by the Catalog API.

Executing Checks

Fact checks are executed, or triggered, by Soundcheck in the following ways:

A dependent fact is updated, either through fact collectors or, or via the Facts API.
A fact check is scheduled, in which case Soundcheck will automatically execute the check on the specified frequency.
A fact check is manually triggered via the Checks API.

Checks API

Use the Checks API to:

List all checks
Get a check by id
Execute a check against one more entities.

Each API call is discussed in detail below.

NOTE: Your environment may not have checks specified in the app-config.yaml file. Here is a sample configuration with two checks defined. The first is checking a fact collected by the SCM plugin for the existence of a README.md file at the root level of a repository. The second analyzes the metadata returned by the catalog fact collector for the existence of either an internal or external metadata tag.

soundcheck:
  checks:
    - id: has_required_tags
      rule:
        any:
          - factRef: catalog:default/entity_descriptor
            path: $.metadata.tags
            operator: contains
            value: internal
          - factRef: catalog:default/entity_descriptor
            path: $.metadata.tags
            operator: contains
            value: external
      passedMessage: |
        Tag found, check passed.
      failedMessage: |
        No `internal` or `external` tag found, check failed.
    - id: has_readme
      rule:
        factRef: scm:default/readme
        path: $.exists
        operator: equal
        value: true
      passedMessage: |
        README.md was found at repository root, check passed.
      failedMessage: |
        README.md was not found at repository root, check failed.

The existence of these checks is assumed when executing API calls in the rest of this Checks API section.

Get a List of Checks

The endpoint to get a list of checks is at /api/soundcheck/checks. For a local deployment, the call would look like this:

GET localhost:7007/api/soundcheck/checks

With the above check defined, hitting the /checks endpoint will yield the following output:

{
  "checks": [
    {
      "id": "has_required_tags",
      "rule": {
        "any": [
          {
            "factRef": "catalog:default/entity_descriptor",
            "operator": "contains",
            "value": "internal",
            "path": "$.metadata.tags"
          },
          {
            "factRef": "catalog:default/entity_descriptor",
            "operator": "contains",
            "value": "external",
            "path": "$.metadata.tags"
          }
        ]
      },
      "passedMessage": "Tag found, check passed.\n",
      "failedMessage": "No `internal` or `external` tag found, check failed.\n"
    },
    {
      "id": "has_readme",
      "rule": {
        "factRef": "scm:default/readme",
        "operator": "equal",
        "value": true,
        "path": "$.exists"
      },
      "passedMessage": "README.md was found at repository root, check passed.\n",
      "failedMessage": "README.md was not found at repository root, check failed.\n"
    }
  ]
}

Get a Check by ID

The endpoint to get a single check by ID is at /api/soundcheck/checks/:checkId. For a local deployment, the call would look like this:

GET localhost:7007/api/soundcheck/checks/has_readme

where 'has_readme' above is the check id, which was defined at the beginning of this section.

Executing the call above results in the following output:

{
  "check": {
    "id": "has_readme",
    "rule": {
      "factRef": "scm:default/readme",
      "operator": "equal",
      "value": true,
      "path": "$.exists"
    },
    "passedMessage": "README.md was found at repository root, check passed.\n",
    "failedMessage": "README.md was not found at repository root, check failed.\n"
  }
}

Trigger a Set of Checks to Execute

This endpoint allows for a set of checks to be executed in soundcheck. To execute a set of checks, POST to:

POST localhost:7007/api/soundcheck/checks/execute

The POST request takes a json request body composed of one of four sets of parameters as follows:

1.) An array of checkIds and an array of entityRefs:

{
  "entityRefs": ["component:default/queue-proxy", "component:default/searcher"],
  "checkIds": ["has_required_tags", "has_readme"]
}

POSTing the above request results in output like the following:

{
  "results": [
    {
      "entityRef": "component:default/queue-proxy",
      "timestamp": "2023-02-15T19:33:15.945Z",
      "checkId": "has_required_tags",
      "state": "failed",
      "details": {
        "notes": {
          "type": "notes",
          "data": "No `internal` or `external` tag found, check failed.\n"
        }
      }
    },
    {
      "entityRef": "component:default/searcher",
      "timestamp": "2023-02-15T19:33:15.945Z",
      "checkId": "has_required_tags",
      "state": "failed",
      "details": {
        "notes": {
          "type": "notes",
          "data": "No `internal` or `external` tag found, check failed.\n"
        }
      }
    }
  ]
}

Note that the response above has no results for the 'has_readme' check, this is because that check does not apply to the given components and thus is not executed.

2.) An array of checkIds and a filter:

{
  "checkIds": ["has_required_tags", "has_readme"],
  "filter": { "metadata.tags": "java" }
}

This will execute the given list of checks against all entities that match the filter, and return an output similar to the following, which has been truncated for brevity:

{
    "results": [
        {
            "entityRef": "component:default/artist-lookup",
            "timestamp": "2023-02-15T20:24:40.473Z",
            "checkId": "has_required_tags",
            "state": "failed",
            "details": {
                "notes": {
                    "type": "notes",
                    "data": "No `internal` or `external` tag found, check failed.\n"
                }
            }
        },
        ...
      ]
}

3.) A single check which is a FactCheckerSchema and an array of entityRefs:

{
  "check": {
    "id": "has_required_tags",
    "rule": {
      "factRef": "catalog:default/entity_descriptor",
      "path": "$.metadata.tags",
      "operator": "contains",
      "value": "internal"
    }
  },
  "entityRefs": ["component:default/queue-proxy", "component:default/searcher"]
}

This will execute the supplied check against the given list of entities, and return a results output like the following, where each entry describes the entity and the result of the check:

{
  "results": [
    {
      "entityRef": "component:default/queue-proxy",
      "timestamp": "2023-02-15T20:43:44.990Z",
      "checkId": "has_required_tags",
      "state": "failed"
    },
    {
      "entityRef": "component:default/searcher",
      "timestamp": "2023-02-15T20:43:44.990Z",
      "checkId": "has_required_tags",
      "state": "failed"
    }
  ]
}

4.) A single check which is a FactCheckerSchema and filter:

{
  "check": {
    "id": "has_required_tags",
    "rule": {
      "factRef": "catalog:default/entity_descriptor",
      "path": "$.metadata.tags",
      "operator": "contains",
      "value": "internal"
    }
  },
  "filter": { "metadata.tags": "java" }
}

This will execute the given check against all entities matching the filter, and return an array of results describing the result of the check against all entities which passed the filter:

{
  "results": [
    {
      "entityRef": "component:default/artist-lookup",
      "timestamp": "2023-02-15T20:49:11.601Z",
      "checkId": "has_required_tags",
      "state": "failed"
    },
    {
      "entityRef": "component:default/playback-order",
      "timestamp": "2023-02-15T20:49:11.601Z",
      "checkId": "has_required_tags",
      "state": "failed"
    },
    {
      "entityRef": "component:default/podcast-api",
      "timestamp": "2023-02-15T20:49:11.601Z",
      "checkId": "has_required_tags",
      "state": "failed"
    },
    {
      "entityRef": "template:default/springboot-template",
      "timestamp": "2023-02-15T20:49:11.601Z",
      "checkId": "has_required_tags",
      "state": "failed"
    }
  ]
}

Finally, this request also allows an optional dry-run query parameter which can be set to empty, true, or false.

true (or empty, ie: '?dryRun' with no value set) indicates that the request is indeed a dry-run, and will execute the checks but will not persist the results.
false indicates a standard request to perform the checks and record the results. This is also the default behavior if the dryRun parameter is not specified.

Check Results API

Use the Check Results API to submit and retrieve check results to/from Soundcheck.

Submitting Results

Submit check results to Soundcheck.

POST localhost:7007/api/soundcheck/results

Optionally, users can provide a returnUpdatedResults=true parameter to indicate that all provided results that were either new or updated an existing result should be returned in the response:

POST localhost:7007/api/soundcheck/results?returnUpdatedResults=true

Request Body

See Check Results Schema.

Example Request Body

Passed

{
  "results": [
    {
      "entityRef": "component:default/petstore",
      "checkId": "tests-run",
      "state": "passed"
    }
  ]
}

Failed

{
  "results": [
    {
      "entityRef": "component:default/petstore",
      "checkId": "tests-run",
      "state": "failed",
      "details": {
        "notes": {
          "data": "Tests were not executed."
        }
      }
    }
  ]
}

Warning

Warning check results don't impact the certification progress of an entity and can be used as a temporary indication that the check is not passing and will be failing soon.

{
  "results": [
    {
      "entityRef": "component:default/petstore",
      "checkId": "tests-run",
      "state": "warning",
      "details": {
        "notes": {
          "data": "The check produced warnings."
        }
      }
    }
  ]
}

Not Applicable

{
  "results": [
    {
      "entityRef": "component:default/petstore",
      "checkId": "tests-run",
      "state": "not-applicable",
      "details": {
        "notes": {
          "data": "This check is not applicable to this component."
        }
      }
    }
  ]
}

Example Requests

BACKSTAGE_BACKEND=localhost:7007 && \
curl \
  -H 'Content-Type: application/json' \
  "${BACKSTAGE_BACKEND}/api/soundcheck/results" \
  --data @- << EOF
{
  "results": [
    {
      "entityRef": "component:default/petstore",
      "checkId": "tests-run",
      "state": "passed"
    }
  ]
}
EOF

BACKSTAGE_BACKEND=localhost:7007 && \
curl \
  -H 'Content-Type: application/json' \
  "${BACKSTAGE_BACKEND}/api/soundcheck/results?returnUpdatedResults=true" \
  --data @- << EOF
{
  "results": [
    {
      "entityRef": "component:default/petstore",
      "checkId": "tests-run",
      "state": "passed"
    }
  ]
}
EOF

POST /api/soundcheck/results HTTP/1.1
Host: localhost:7007
Content-Type: application/json
Content-Length: 129

{
  "results": [
    {
      "entityRef": "component:default/petstore",
      "checkId": "tests-run",
      "state": "passed"
    }
  ]
}

POST /api/soundcheck/results?returnUpdatedResults=true HTTP/1.1
Host: localhost:7007
Content-Type: application/json
Content-Length: 129

{
  "results": [
    {
      "entityRef": "component:default/petstore",
      "checkId": "tests-run",
      "state": "passed"
    }
  ]
}

Responses

200 Response

Returns a status message by default:

{
  "status": "Result ingestion complete."
}

If returnUpdatedResults=true was specified, then this endpoint returns those given check results which were either new or which resulted in a change of state.

See Check Results Schema.

Example Response Body for `returnUpdatedResults=true`

{
  "results": [
    {
      "entityRef": "string",
      "checkId": "string",
      "scope": "string",
      "state": "passed",
      "details": {
        "notes": {
          "type": "notes",
          "version": 1,
          "data": "string"
        }
      }
    }
  ]
}

400 Response

Bad Request

Body did not match expected schema.

See Error Schema.

Example Response Body

{
  "error": {
    "name": "string",
    "message": "string",
    "stack": "string"
  },
  "request": {
    "method": "string",
    "url": "string"
  },
  "response": {
    "statusCode": 0
  }
}

500 Response

Internal Server Error

Soundcheck encountered an unexpected condition that prevented it from fulfilling the request.

See Error Schema.

Example Response Body

{
  "error": {
    "name": "string",
    "message": "string",
    "stack": "string"
  },
  "request": {
    "method": "string",
    "url": "string"
  },
  "response": {
    "statusCode": 0
  }
}

Retrieving Results

GET localhost:7007/api/soundcheck/results

Returns check results for a given entity.

Query Parameters

entityRef Required. A reference to the entity to retrieve check results for.
checks Optional. Filters check results to only those with the provided check ID(s). Accepts multiple values: checks=A,checks=B

Example Requests

BACKSTAGE_BACKEND=localhost:7007 && \
curl \
  -H 'Accept: application/json' \
  "${BACKSTAGE_BACKEND}/api/soundcheck/results?entityRef=component:default/petstore&checks=tests-run"

GET /api/soundcheck/results?entityRef=component:default/petstore&checks=tests-run HTTP/1.1
Host: localhost:7007
Accept: application/json

Responses

200 Response

See Check Results Schema.

Example Response Body

{
  "results": [
    {
      "entityRef": "component:default/petstore",
      "checkId": "tests-run",
      "scope": "default",
      "state": "failed",
      "details": {
        "notes": {
          "type": "notes",
          "version": 1,
          "data": "Tests were not executed."
        }
      }
    }
  ]
}

400 Response

Bad Request

Invalid or missing query parameter(s).

500 Response

Internal Server Error

Soundcheck encountered an unexpected condition that prevented it from fulfilling the request.

Example Response Body

{
  "error": {
    "name": "string",
    "message": "string",
    "stack": "string"
  },
  "request": {
    "method": "string",
    "url": "string"
  },
  "response": {
    "statusCode": 0
  }
}

Facts API

Use the Facts API to submit facts to Soundcheck.

Submitting Facts

Submit Facts to Soundcheck.

POST localhost:7007/api/soundcheck/facts

Request Body

See Facts Schema.

Example Request Body

{
  "facts": [
    {
      "factRef": "catalog:default/petstore_metadata",
      "entityRef": "component:default/petstore",
      "data": {
        "example": "petstore data"
      },
      "timestamp": "2023-02-06T19:33:48.590+00:00"
    }
  ],
  "cache": {
    "duration": {
      "hours": 24
    }
  }
}

Example Requests

BACKSTAGE_BACKEND=localhost:7007 && \
curl \
  -H 'Content-Type: application/json' \
  "${BACKSTAGE_BACKEND}/api/soundcheck/facts" \
  --data @- << EOF
{
  "facts": [
    {
      "factRef": "catalog:default/petstore_metadata",
      "entityRef": "component:default/petstore",
      "data": {
        "example": "petstore data"
      },
      "timestamp": "2023-02-06T19:33:48.590+00:00"
    }
  ],
  "cache": {
    "duration": {
      "hours": 24
    }
  }
}
EOF

POST /api/soundcheck/results HTTP/1.1
Host: localhost:7007
Content-Type: application/json
Content-Length: 129

{
  "facts": [
    {
      "factRef": "catalog:default/petstore_metadata",
      "entityRef": "component:default/petstore",
      "data": {
        "example": "petstore data"
      },
      "timestamp": "2023-02-06T19:33:48.590+00:00"
    }
  ],
  "cache": {
    "duration": {
      "hours": 24
    }
  }
}

Responses

200 Response

Returns the factsRefs which were submitted.

Example Response Body

{
  "factRefs": ["catalog:default/petstore_metadata"]
}

400 Response

Bad Request

Body did not match expected schema.

See Error Schema.

500 Response

Internal Server Error

Soundcheck encountered an unexpected condition that prevented it from fulfilling the request.q

Track API

Use the Track API to:

List all tracks
Get a track by ID

Each API call is discussed in detail below.

Get a List of Tracks

The endpoint to get a list of tracks is at /api/soundcheck/tracks. For a local deployment, the call would look like this:

GET localhost:7007/api/soundcheck/tracks

Query Parameters

tracks Optional. Filters tracks to only those with the provided track ID(s). Accepts multiple values: tracks=A,tracks=B.
entityRef Optional. A reference to the entity to retrieve applicable tracks for. When this query param is provided, only tracks (and checks) which are applicable to this entity are returned.
onlyApplicableChecks Optional. Used in conjunction with the entityRef query params, determined whether all checks in a track are included in the response or only checks which are applicable to the given entity.

The response body has tracks containing all tracks.

{
  "tracks": [
    ...
  ]
}

Get a Track

The endpoint to get a track is at /api/soundcheck/track/:trackId. Where :trackId is the ID of the track you would like to retrieve.

For a local deployment, the call would look like this: GET localhost:7007/api/soundcheck/track/:trackId

Query Parameters

entityRef Optional. A reference to the entity to validate applicability for. When this query param is provided, the requested track is only returned if it is applicable to the provided entity, otherwise {} is returned.
onlyApplicableChecks Optional. Used in conjunction with the entityRef query params, determined whether all checks in the track are included in the response or only checks which are applicable to the given entity.

The response body has track containing the requested track.

{
  "track": {
    ...
  }
}

Aggregations API

Use the Aggregations API to get aggregations from Soundcheck.

Requesting Aggregations

Get an aggregation from Soundcheck

POST localhost:7007/api/soundcheck/aggregations

Request Body

See Facts Schema.

Example Aggregation Request Body

{
  "type": "individualCheckPassRates",
  "filter": {
    "numberOfDays": 3,
    "entityKinds": {
      "included": ["Component"]
    },
    "entityTypes": {
      "included": ["service"]
    }
  }
}

Example Aggregation Requests

BACKSTAGE_BACKEND=localhost:7007 && \
curl \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer {your authorization token}'\
  "${BACKSTAGE_BACKEND}/api/soundcheck/aggregations" \
  --data @- << EOF
{
    "type": "individualCheckPassRates",
    "filter": {
        "numberOfDays": 3,
        "entityKinds": {
            "included": ["Component"]
        },
        "entityTypes": {
            "included": ["service"]
        }
    }
}
EOF

Aggregation Responses

200 Response

Returns the requested Aggregation.

Example Aggregation Response Body

"individualCheckPassRates": [
        {
            "id": "alerting-configured",
            "snapshotPassRate": 100,
            "trendPassRates": [
                0,
                100,
                100
            ],
            "checkName": "Alerting is configured",
            "checkDescription": "Components must have an on-call rotation identifier configured in component-info.yaml. This enables consumers of your component to ping an on-call goalie from Backstage in the event of an incident.\nThis check will NOT look to see what level of coverage your rotation provides; you may have a round-the-clock rotation, or a 10am-5pm local time rotation, whatever is most appropriate for your component.\n"
        },
        ...
]

400 Response

Bad Request

If the input did not match the schema.

500 Response

Internal Server Error

Soundcheck encountered an unexpected condition that prevented it from fulfilling the request.q

Schemas

Check Results Schema

results An array of check results.

Check Result Schema

entityRef A reference to the entity that the check result is for.
checkId The unique identifier of the check that the result is for.
state The check results state. One of passed, failed, warning or not-applicable.
scope Optional. The scope within which this check was performed. Defaults to default.
details Optional. Additional details on the check result. Currently only supports notes.
- notes
  - type: Optional. Type of the check result details, currently only supports notes. Defaults to notes.
  - version: Optional. Version of the check result details, can be any number. Defaults to 1.
  - data: Markdown providing additional context on the check result (e.g., why it failed).

Certification Schema

entityRef A reference to the entity that the certification is for.
trackId The unique identifier of the track that the certification is for.
levelOrdinal The ordinal of the track's level that the certification is for.
state The certifications state. One of passed or failed.
scope Optional. The scope within which this certification was performed. Defaults to default.

Certification Request Schema

scope Optional. The scope within which entities will be certified. Defaults to default.
numberOfDays Optional. Number of days the certifications should be created for. Defaults to 1.
batchSize Optional. A number of entities to be fetched and certified at once. Defaults to 100.
trackId Optional. A track ID to certify entities against. If not provided the entities will be certified against all applicable tracks.

Facts Schema

facts An array of facts.
cache A cache config.

Fact Schema

factRef A unique reference to the fact.
entityRef A reference to the entity this fact is collected against.
data The data collected of the fact's data schema.
timestamp The date/time at which this fact was collected.

Cache Config Schema

A boolean
- true - Indicates that the fact should be cached forever.
- false - Indicates that the fact should not be cached.
Or an object containing a duration field
- duration Specifies the cache duration. An object with one or more of the following:
  - years: Optional. The number of years.
  - months: Optional. The number of months.
  - weeks: Optional. The number of weeks.
  - days: Optional. The number of days.
  - hours: Optional. The number of hours.
  - minutes: Optional. The number of minutes;
  - seconds: Optional. The number of seconds.
  - milliseconds: Optional. The number of milliseconds.

Error Schema

error
- name Name of the error.
- message Error message.
- stack Stack trace.
- request
  - method HTTP request method.
  - url URL
- response
  - statusCode HTTP status code.

Aggregation Schema

type: The type of aggregation to perform, one of:
- individualCheckPassRates
- overallCheckPassRates
- individualEntitiesPassRates
- overallEntityPassRates
- individualTracksPassRates
- overallTrackPassRates
- groupsPassRates
filter: An Aggregation_Filter to limit how the aggregation is calculated.

Aggregation Filter Schema

All fields are optional.

entityRefs: an InclusionFilter containing those entity references to include/exclude from the aggregation.
checkIds: an InclusionFilter containing those identifiers of the checks to include/exclude from the aggregation.
tracks: an array of Aggregation Track Filters. Only check results from the specified tracks and levels will be included in the aggregation.
scope: Only results with the specified scope will be included in the aggregation.
numberOfDays: A numeric value indicating how many days of history to include in the aggregation.
entityKinds: an InclusionFilter containing the kinds of entities to include/exclude from the aggregation.
entityLifecycles: an InclusionFilter containing the lifecycles of entities to include/exclude from the aggregation.
entityTypes: an InclusionFilter containing the types of entities to include/exclude from the aggregation.
checkOwnerFilters: an InclusionFilter containing group references. Only the results of checks owned/not owned by the given owners will be included in the aggregation.
entityOwnerFilters: an InclusionFilter containing group references. Only entities owned/not owned by the given owners will be included in the aggregation.

Aggregation InclusionFilter Schema

included: An optional string or array of strings.
excluded: An optional string or array of strings.

Aggregation Track Filter

trackId: The name of the track. Required.
levels: An array of level ordinals. Optional.

Spotify Plugins for Backstage: Soundcheck - Backend​

Prerequisites​

1. Complete Spotify plugin bundle setup​

2. Check your Backstage version​

3. Set Node options​

Installation​

1. Install the plugins​

2. Install Soundcheck backend​

3. Add program config​

4. Next Steps​

5. Scaling & Rate Limiting (Optional)​

Rate Limiting​

Automatic Rate Limiting​

6. Slack Notifications (Optional)​

Manifest file​

Tokens​

Socket Mode​

Adding the Slack app to a Slack channel​

7. Email Notifications (Optional)​

SMTP Connection Authentication​

Node Options​

Defining Programs, Levels and Checks​

Contents​

Overall Shape Of A Program​

Program Fields​

id, name and description [required]​

ownerEntityRef [required]​

documentationURL and group [optional]​

filter [optional]​

levels [required]​

Level Fields​

ordinal [required]​

name and description [optional]​

checks [required]​

Check Fields​

id [required]​

name and description [required]​

filter [optional]​

Entity Filter​

catalog [optional]​

CatalogFilter Object​

FilterKey Object​

String FilterKeys​

Array FilterKeys​

Record FilterKeys​

FilterValue Object​

String FilterValue​

Array FilterValue​

Presence FilterValue​

Check Result History​

Enabling Check Result History​

Reading Check Result History​

Certification History​

Enabling Certification History​

Reading Certification History​

Backfilling Certification History​

Advanced Applicability Filters​

Configuring Check Applicability Filter​

Configuring Track Applicability Filter​

Integrating with Soundcheck​

Fact Framework​

Fact Collectors​

Built-in Fact Collectors​

Catalog Fact Collector​

Soundcheck Fact Collector​

Third Party Integrations​

Source Code Management (SCM) Fact Collector​

GitHub Fact Collector​

Fact Checks​

Shape of a Fact Check​

Rules​

Operators​

Messages​

Schedule​

Executing Checks​

Checks API​

Get a List of Checks​

Get a Check by ID​

Trigger a Set of Checks to Execute​

Check Results API​

Spotify Plugins for Backstage: Soundcheck - Backend

Prerequisites

1. Complete Spotify plugin bundle setup

2. Check your Backstage version

3. Set Node options

Installation

1. Install the plugins

2. Install Soundcheck backend

3. Add program config

4. Next Steps

5. Scaling & Rate Limiting (Optional)

Rate Limiting

Automatic Rate Limiting

6. Slack Notifications (Optional)

Manifest file

Tokens

Socket Mode

Adding the Slack app to a Slack channel

7. Email Notifications (Optional)

SMTP Connection Authentication

Node Options

Defining Programs, Levels and Checks

Contents

Overall Shape Of A Program

Program Fields

`id`, `name` and `description` [required]

`ownerEntityRef` [required]

`documentationURL` and `group` [optional]

`filter` [optional]

`levels` [required]

Level Fields

`ordinal` [required]

`name` and `description` [optional]

`checks` [required]

Check Fields

`id` [required]

`name` and `description` [required]

`filter` [optional]

`Entity` Filter

`catalog` [optional]

`CatalogFilter` Object

`FilterKey` Object

String `FilterKey`s

Array `FilterKey`s

Record `FilterKey`s

`FilterValue` Object

String FilterValue

Array FilterValue

Presence FilterValue

Check Result History

Enabling Check Result History

Reading Check Result History

Certification History

Enabling Certification History

Reading Certification History

Backfilling Certification History

Advanced Applicability Filters

Configuring Check Applicability Filter

Configuring Track Applicability Filter

Integrating with Soundcheck

Fact Framework

Fact Collectors

Built-in Fact Collectors

Catalog Fact Collector

Soundcheck Fact Collector

Third Party Integrations

Source Code Management (SCM) Fact Collector

GitHub Fact Collector

Fact Checks

Shape of a Fact Check

Rules

Operators

Messages

Schedule

Executing Checks

Checks API

Get a List of Checks

Get a Check by ID

Trigger a Set of Checks to Execute

Check Results API