Checks - Spotify for Backstage

A check is a comparison between a defined acceptable outcome and the actual outcome of a given process. Checks indicate where the software aligns to or deviates from organizational standards and best practices.

Managing checks via the No-Code UI

Check Creation

Creating a custom check

To create a check, select the facts that you want to use, organize them into rules and give your check a meaningful name and a description.

The rules are what defines a check and what a track will use in determining if an entity passes or fails. Simple expressions can be combined using Any Of / All Of to create more complex rules. You also have the option of adding Pass/Fail messages for additional details of why something passed/failed.

Creating a check from a template

Several check templates are included with Soundcheck depending on which integrations you have installed. These check templates prepopulate the create check form with a check name, description, rules, messages, and filters to help expedite the creation of common checks.

Editing a check

Once a check is created, you will be able to manage and edit your check on its detail page. You’ll find no differences in how the pages look when creating/editing a check. From the checks listing page, you will see an option to edit you check, which will bring you to the details page shown below.

You can make as many changes as desired, but just keep in mind that changes are reflected in tracks that may already have said check assigned. Checks can also be a part of multiple tracks, which could have the unintentional effect of causing some tracks to start failing. While the track you want the check updated in may pass, a track setup somewhere else may fail.

Dry Run and Test a Check

Checks can be dry ran and tested in the Review and Test of the check form. This step can be viewed when both editing an existing check and creating a new check.

Here you can view the final configuration options for your check, as well as dry run against applicable entities. If you selected filters for your check, the entities table should already be populated. Otherwise you can type a name in the search bar to get a list of entities. Click the button in the dry run section to trigger a dry run. When complete, the dry run will show the check result as well as a details button.

Dry runs executes the check as configured in the form against a fresh set of facts. Dry run results are NOT persisted. This will trigger additional fact collection which will count towards any API limits. Clicking the details button will open a dialog with additional information, including the full fact, rule by rule breakdowns, and resolved values.

Fact Explorer

In the Check Form, facts can be explored via the Explore Facts button.

After entering a fact, both the fact and path can be tested by entering an entity in the Sample and Test Section. You do not need to enter a path to sample the fact.

This will fetch a fact from the cache if available. If not, this will trigger a fact collection and could count towards your third party API rate limits.

Export to YAML

Individual checks can be saved as YAML via the dropdown menu.

All checks from all sources can be saved as YAML via the Export all checks button on the top toolbar.

Import from YAML

Checks can be created from YAML via the Import YAML check(s) button on the top toolbar. The checks will be created in the Soundcheck database and can be modified later just like other checks created via No-Code UI. If using RBAC, make sure that your user has soundcheck.check.create permission.

Notes:

YAML file may contain either a single check or an array of checks.
If the YAML file contains checks with IDs that already exist in Soundcheck, such checks will be skipped and the import will continue as normal.
If one or more checks fail to import, their IDs will be listed in the warning banner and the import of the remaining checks will continue as normal.

Managing checks via yaml configuration

Checks can be created in code via yaml configuration files.

Overall shape of a Check

A fact check (or more commonly, just a ‘check’) defines a rule comprised of one or more conditions: a combination of facts, operators, and values that determine whether the check emits a passed or failed check result to Soundcheck.

Note: the id field is a unique identifier for the check. It is used to reference the check in tracks and within the Soundcheck UI. It must be unique across all checks in your library.

Soundcheck supports both checks defined in yaml files and via a no-code UI. Checks defined in yaml files can be sourced locally or remotely, and require only two fields: id and rule. Here is an example of a basic check defined in yaml (more examples):

id: has_pagerduty_integration_key
rule:
  factRef: catalog:default/entity_descriptor
  path: $.metadata.annotations['pagerduty.com/integration-key']
  operator: matches
  value: .+
failedMessage: |
  No PagerDuty Integration Key defined for this entity.
passedMessage: |
  PagerDuty Integration Key defined for this entity.

Soundcheck version 1.32.0+ introduced a new JSON path library (our default path parser). This library is stricter when validating the path field and requires certain characters to be escaped. In the example above $.metadata.annotations['pagerduty.com/integration-key'] pagerduty.com/integration-key is escaped with ''.

Adding yaml checks to your check library

To add checks defined in a yaml file to your check library, you need to source them in you app-config.yaml. You have the option of storing them locally or remotely. Here is an example configuration should your check yaml files be local to your Backstage project:

app-config.yaml

soundcheck:
  checks:
    - $include: ./path-to-local-folder/checks-file-1.yaml
    - $include: ./path-to-local-folder/checks-file-2.yaml
    - $include: ./path-to-local-folder/checks-file-3.yaml

To maintain your checks files in a remote repo, you can source them in app-config.yaml in a similar manner. Here is an example configuration:

app-config.yaml

soundcheck:
  checks:
    - https://github.com/user/repo/blob/main/checks-file-1.yaml
    - https://github.com/user/repo/blob/main/checks-file-2.yaml
    - https://github.com/user/repo/blob/main/checks-file-3.yaml
  remote_file_updates:
    disabled: false
    frequency:
      minutes: 5

Note: you cannot combine local and remote sourcing for checks files. This includes the usage of $include within a remotely sourced checks file. The below option will NOT work:

app-config.yaml

soundcheck:
  checks:
    - https://github.com/user/repo/blob/main/checks-file-1.yaml
    - https://github.com/user/repo/blob/main/checks-file-2.yaml
    - $include: ./path-to-local-folder/checks-file-3.yaml

Remote file update options

The remote_file_updates object is optional configuration allowing you to control if and when Soundcheck looks for updates within your remote files. This configuration is global for Soundcheck, so will apply to both track and check files. If not explicitly set, see the below for default values.

Key	Default	Description
`disabled`	false	Enables Soundcheck to look for updates in remote files
`frequency`	minutes: 5	The frequency at Soundcheck will look for updates in remote files. Possible values are either a cron expression `{ cron: ... }` or `HumanDuration`.

Migrating checks from yaml to no-code UI

To migrate the checks from yaml to no-code UI follow these steps:

Remove references to checks from soundcheck.checks configuration.
Import yaml checks previously referenced in soundcheck.checks configuration using the “Import from YAML” button located on the Checks tab in the UI.

Your check history will be preserved, and you can continue to use the checks as usual.

Check Fields

Field	Required	Description
`id`	Yes	The unique identifier for this fact check.
`name`	No	A name for the fact check suitable for display on a UI.
`description`	No	A description of the fact check suitable for display on a UI.
`rule`	Yes	One or more conditions that determine check result.
`passedMessage`	No	Message if the check passes.
`failedMessage`	No	Message if the check fails.
`schedule`	No	How often and on what entities the fact check should run.
`ownerEntityRef`	No	Reference to the owner of the check.
`filter`	No	Default filter specifying what entities the check applies to.
`pathResolver`	No	The path resolver the rules will use. Defaults to `jsonpath`.
`warning`	No	Whether the check generates a `failed` or `warning` result.
`applicable`	No	One or more conditions that determine if the check is applicable to an entity.
`notApplicableMessage`	No	Message if the check is determined to be not applicable based on `applicable` conditions.

The name and description fields can be used to provide more context to the check. If not provided, the id will be used as the name when viewing the check in Soundcheck. When creating checks via the no-code UI, the name, description, and owner, fields become required.

Rules

A rule contains one or more conditions. The simplest form of a condition consists of a fact reference, an operator, and a value. When the check is executed, the operator is used to compare the fact against the value. Example of a simple condition:

rule:
  factRef: catalog:default/entity_descriptor
  path: $.metadata.annotations['pagerduty.com/integration-key']
  operator: matches
  value: .+
  failedMessage: |
    No PagerDuty Integration Key defined for this entity.

More complex conditions can by created through the use of boolean expressions. A rule may optionally have either an all, any, or ‘not’ expression at its root, containing an array of conditions. The all operator specifies that all conditions contained within must be truthy for the check to pass. The any operator only requires one condition to be truthy for the check to pass, while the not operator negates the result of the condition it contains. Example of a complex condition:

rule:
  any:
    - factRef: catalog:default/entity_descriptor
      path: $.metadata.labels['language']
      operator: equal
      value: java
    - factRef: catalog:default/entity_descriptor
      path: $.metadata.labels['language']
      operator: equal
      value: kotlin

Example of a check with a negated condition:

- id: no_r_rated_language
  description: >
    Check that the language is not Rust or Ruby.
  failedMessage: |
    Language is Rust or Ruby.
  passedMessage: |
    Language is not Rust or Ruby.
  rule:
    not:
      - any:
          - factRef: catalog:default/entity_descriptor
            path: $.metadata.labels['language']
            operator: equal
            value: rust
          - factRef: catalog:default/entity_descriptor
            path: $.metadata.labels['language']
            operator: equal
            value: ruby

Value Input

The rule value can be a singular or a list, or an object with another fact. Example using a list of values:

rule:
  factRef: catalog:default/entity_descriptor
  path: $.kind
  operator: in
  value:
    - Group
    - Component

Example using a fact:

rule:
  factRef: catalog:default/entity_descriptor
  path: $.spec.type
  operator: equal
  value:
    factRef: custom:default/custom_fact
    path: $.service_type

Path Resolvers

The path field in a rule is used to resolve a path to a value in a fact, and it is this resolved value that is used in the comparison with the value field. This allows you to reference specific parts of a fact to compare against a value. Soundcheck supports multiple path resolvers, which can be specified in the pathResolver field of a check. If no path resolver is specified, Soundcheck will use its default path resolver (JSON Path). The path resolver option applies to all rules for a check. Example:

id: annotation-defined
ownerEntityRef: group:default/check-owner
name: Annotation Defined
description: Verifies that annotations are defined on the entity.
pathResolver: jsonata
rule:
  factRef: catalog:default/entity_descriptor
  path: $keys($.metadata.annotations)
  operator: contains
  value: 0

Available options of the box are:

jsonpath - default
lodash
jmespath
jsonata

JSON Path

Soundcheck’s default path resolver. JSONPath is a query language for JSON, similar to XPath for XML. It allows you to navigate and filter JSON data to extract specific values or sets of values. JSONPath expressions are used to traverse the JSON data structure and extract the data that matches the expression.

Key Features of JSONPath:

Dot Notation and Bracket Notation: You can use dot notation ($.store.book[0].title) or bracket notation ($['store']['book'][0]['title']) to navigate the JSON structure.
Wildcard (*): Selects all elements in an array or all properties in an object.
Recursive Descent (..): Searches recursively through the JSON structure.
Filters: Allows you to filter JSON data based on expressions. For example, $.store.book[?(@.price < 10)] selects books with a price less than 10.
Slicing: Similar to array slicing in Python, e.g., $.store.book[:2] selects the first two books.

Example Given the following JSON data:

{
  "store": {
    "book": [
      {
        "category": "reference",
        "author": "Nigel Rees",
        "title": "Sayings of the Century",
        "price": 8.95
      },
      {
        "category": "fiction",
        "author": "Evelyn Waugh",
        "title": "Sword of Honour",
        "price": 12.99
      }
    ],
    "bicycle": {
      "color": "red",
      "price": 19.95
    }
  }
}

JSONPath Expressions:

$.store.book[*].author returns ["Nigel Rees", "Evelyn Waugh"]
$.store.bicycle.color returns "red"
$..price returns [8.95, 12.99, 19.95]
$.store.book[?(@.price < 10)].title returns ["Sayings of the Century"]

For more information on the JSON Path syntax, see the JSON Path documentation.

Lodash Get

lodash.get is a utility function from the Lodash library, which is a popular JavaScript utility library. The get function is used to access deeply nested properties in objects.

Key Features of lodash.get

Path Notation: You can specify the path to the property using a dot notation string or an array of keys.
Simplicity: Easy to use syntax.

Example Given the following JSON object:

{
  "store": {
    "book": [
      {
        "category": "reference",
        "author": "Nigel Rees",
        "title": "Sayings of the Century",
        "price": 8.95
      },
      {
        "category": "fiction",
        "author": "Evelyn Waugh",
        "title": "Sword of Honour",
        "price": 12.99
      }
    ],
    "bicycle": {
      "color": "red",
      "price": 19.95
    }
  }
}

lodash.get Expressions:

'store.book[0].author' returns "Nigel Rees"

For more information on the Lodash Get syntax, see the Lodash Get documentation.

JMES Path

JMESPath is a query language for JSON, designed to allow you to extract and transform elements from JSON data structures. Similar to JSONPath, it enables you to navigate and filter JSON data, but it has its own syntax and features.

Key Features of JMESPath:

Dot Notation: Used to navigate through JSON objects. For example, foo.bar will navigate to the bar property of the foo object.
Filters: Allows filtering of arrays. For example, foo[?bar > 10] will filter the foo array to include only elements where bar is greater than 10.
Projections: Allows transformation of arrays and objects. For example, foo[*].bar will create a list of the bar properties from each object in the foo array.
Functions: JMESPath includes built-in functions such as length, max, min, sum, and contains.

Example Given the following JSON data:

{
  "store": {
    "book": [
      {
        "category": "reference",
        "author": "Nigel Rees",
        "title": "Sayings of the Century",
        "price": 8.95
      },
      {
        "category": "fiction",
        "author": "Evelyn Waugh",
        "title": "Sword of Honour",
        "price": 12.99
      }
    ],
    "bicycle": {
      "color": "red",
      "price": 19.95
    }
  }
}

JMESPath Expressions:

store.book[*].author returns ["Nigel Rees", "Evelyn Waugh"]
store.bicycle.color returns "red"
store.book[?price > 10].title returns ["Sword of Honour"]
store.book[?category == 'fiction'].price returns [12.99]

For more information on the JMES Path syntax, see the JMES Path documentation.

JSONata

JSONata is a lightweight query and transformation language for JSON data. It allows you to query, filter, and transform JSON data in a concise and expressive way. JSONata includes features for selecting data, performing calculations, and creating new JSON structures based on existing data.

Key Features of JSONata:

Expression Language: JSONata expressions enable complex querying and transformations of JSON data.
Path Notation: Similar to XPath for XML, JSONata uses a path notation to navigate through JSON structures.
Functions: JSONata includes built-in functions for manipulating data, such as mathematical operations, string operations, and more.
Conditionals: You can use conditional expressions to filter and transform data based on specific criteria.
Mapping and Reducing: JSONata supports mapping and reducing operations to transform arrays and objects.

Example Given the following JSON data:

{
  "store": {
    "book": [
      {
        "category": "reference",
        "author": "Nigel Rees",
        "title": "Sayings of the Century",
        "price": 8.95
      },
      {
        "category": "fiction",
        "author": "Evelyn Waugh",
        "title": "Sword of Honour",
        "price": 12.99
      }
    ],
    "bicycle": {
      "color": "red",
      "price": 19.95
    }
  }
}

JSONata Expressions:

store.book.author returns ["Nigel Rees", "Evelyn Waugh"]
store.bicycle.color returns "red"
store.book[price < 10].title returns ["Sayings of the Century"]
store.book.{ "title": title, "cost": price } returns:

[
  { "title": "Sayings of the Century", "cost": 8.95 },
  { "title": "Sword of Honour", "cost": 12.99 }
]

For more information on the JSONata syntax, see the JSONata documentation.

Operators

Boolean, Numeric, & String Operators

These operators support boolean, numeric, and string values.

Operator	Description
`equal`	Fact must equal value.
`notEqual`	Fact must not equal value.

When dealing with arrays of booleans, numbers, or strings, these operators can be used in combination with the all:, any:, and none: prefixes to validate the elements of the array. See the Array Operators section for additional details.

Numeric Operators

These operators support only numeric values.

Operator	Description
`lessThan`	Fact must be less than value.
`lessThanInclusive`	Fact must be less than or equal to value.
`greaterThan`	Fact must be greater than value.
`greaterThanInclusive`	Fact must be greater than or equal to value.

When dealing with arrays of numbers these operators can be used in combination with the all:, any:, and none: prefixes to validate the elements of the array. See the Array Operators section for additional details.

String Operators

These operators support only string values.

Operator	Description
`matches`	Fact string matches given value regex.

When dealing with arrays of strings these operators can be used in combination with the all:, any:, and none: prefixes to validate the elements of the array. See the Array Operatorssection for additional details.

SemVer Operators

These operators support only string values that represent semantic versions or semantic version ranges.

Operator	Description
`semverGt`	Fact version must be greater than value version.
`semverGte`	Fact version must be greater than or equal to value version.
`semverLt`	Fact version must be less than value version.
`semverLte`	Fact version must be less than or equal to value version.
`semverEq`	Fact version must equal value version.
`semverNeq`	Fact version must not equal value version.
`semverSatisfies`	Fact version satisfies value range.
`semverGtr`	Fact version is greater than all versions in value range.
`semverLtr`	Fact version is less than all versions in value range.

When dealing with arrays of string representing semantic versions or semantic version ranges these operators can be used in combination with the all:, any:, and none: prefixes to validate the elements of the array. See the Array Operators section for additional details.

Date/time Operators

These operators support string values in ISO 8601 date/time or duration format. Negative durations are also supported: -P1Y will be interpreted as 1 year before check execution. Additionally, Soundcheck supports the use of the now keyword to represent the current date/time.

Operator	Description
`after`	Fact value is a date/time occurring after the value.
`before`	Fact value is a date/time occurring before the value.

Exists Operator

A value of “true” or “false” must be provided for this operator. Using “true” will expect the Fact value to exist, whereas using “false” it will expect the Fact value to NOT exist in the Fact.

Operator	Description
`exists`	Fact value must exist/be defined.

When dealing with arrays this operator can be used in combination with the all:, any:, and none: prefixes to validate the elements of the array. See the Array Operators section for additional details.

Array Operators

These operators support array values.

Operator	Description
`in`	Fact must be included in array value.
`notIn`	Fact must not be included in array value.
`contains`	Fact array must include value.
`doesNotContain`	Fact array must not include value.
`hasLengthOf`	fact array must have length equal to value.

Additionally, you can use the following prefixes in combination with any of the above operators to validate the elements of an array:

all: - Validates that all elements in the array satisfy the condition.
any: - Validates that at least one element in the array satisfies the condition.
none: - Validates that none of elements in the array satisfies the condition.

For example, to validate that all elements in an array are greater than 10 we can use the greaterThan operator with the all: prefix:

factRef: kubernetes:default/k8s_pods
path: $.items[*].spec.terminationGracePeriodSeconds
operator: all:greaterThan
value: 10

Schedule

NOTE: Check scheduling will be deprecated soon, make sure your fact collectors are scheduled instead. When Soundcheck collects a fact it will automatically execute all checks that depend on that fact.

Scheduling options for fact checks:

Field	Required	Description
`frequency`	Yes	Cron expression or HumanDuration for check execution frequency.
`entityRefs`	No	Entities to run check on (overrides filter).
`filter`	No	Filter specifying which entities to execute the check on.
`exclude`	No	Filter specifying which entities to exclude.

Messages

Pass messages - passedMessage - are displayed to the user when a check passes.
Fail messages - failedMessage - are displayed to the user when a check fails.
Not applicable messages - notApplicableMessage - are displayed to the user when a check is determined to be not applicable based on applicable conditions.

Messages can be provided as a string or as a markdown string. Pass, fail and not applicable messages also support Liquid templating. The following variables are available for use in templates:

entity - The entity the check was executed on.
facts - A map of facts used by the check. Individual facts can be looked up in this map by fact reference. For example, facts['catalog:default/entity_descriptor'].
- If all the facts used by the check have unique fact names they can be accessed in facts by fact name rather than having to use the entire fact reference. For example, if check uses the catalog:default/entity_descriptor it can be accessed by facts['catalog:default/entity_descriptor'] or facts['entity_descriptor'].
- If only a single fact is used by the check, the fact can be accessed directly. For example, if catalog:default/entity_descriptor was the only fact used by the check it can be accessed as fact rather than facts['catalog:default/entity_descriptor'].

Example - Using the Entity Object

Here is an example of a pass message using templates:

{{ entity.metadata.name }} has a PagerDuty Integration Key defined.

In this example, Soundcheck would replace {{ entity.metadata.name }} with the name of the entity the check was executed on. This would resolve to:

My Service has a PagerDuty Integration Key defined.

Example - Check Fail Message With a Single Fact

Given this check with the following rule:

rule:
  all:
    - factRef: catalog:default/entity_descriptor
      path: $.kind
      operator: equal
      value: Component

Since this check only has a single fact, we can use the fact property in our failed message.

Entity kind {{ fact.data.kind }} is a not Component.

This would resolve to:

Entity kind System is not Component.

Example - Check Fail Message with Multiple Facts

Given this check with the following rule:

rule:
  all:
    - factRef: kubernetes:default/k8s_pods
      path: $.items[*].spec.terminationGracePeriodSeconds
      operator: all:greaterThan
      value: 10
    - factRef: catalog:default/entity_descriptor
      path: $.spec.type
      operator: all:equal
      value: service

Since this check uses multiple facts, you will need to access the facts property. To create a message with the kubernetes fact:

Pods in cluster {{ facts['kubernetes:default/k8s_pods'].data.clusters }} do not have termination grace periods set above 10 seconds.

This would resolve to:

Pods in cluster gke-services-us-1 do not have termination grace periods set above 10 seconds.

Example - Testing JSON Output

To see the facts available it is recommend to use the fact explorer or the check dry run in order to get the fact information. These features are available in the check form. Check dry runs will also fully resolve the pass/fail message. You can also render the output as JSON. This is particularly useful for displaying facts for testing and debugging. Wrap in a code block to preserve formatting. For example:

```json
  {{ fact | json: 4 }}
```

Would give you an output like:

{
  "factRef": {
    "source": "github",
    "scope": "default",
    "name": "repository_details"
  },
  "data": {
    "entityRef": "component:default/example",
    "html_url": "https://github.com/backstage/backstage",
    "description": "Backstage is an open platform for building developer portals",
    "fork": false,
    "url": "https://api.github.com/repos/backstage/backstage"
  }
}

If the check has multiple facts, you will need to use the facts object:

```json
  {{ facts['github:default/repository_details'] | json: 4 }}
```

Markdown Support

Pass/fail messages support nested markdown within a <details> block. For example:

<details>
  <summary>Details</summary>
  ### Debug
  - Check your imports.
  - Use the non-deprecated fields.
</details>

Sensitive Facts

Some facts are considered sensitive and cannot be used for templating without explicitly enabling this feature. See docs for details.

Filters

Checks can be filtered to the desired set of entities using catalog filters, with the exception of the CATALOG_FILTER_EXISTS symbol.

NOTE: Specifying a filter on a check sets the default filter for the check. This filter is used when the check is added to a Track using Soundcheck’s UI. If using YAML to define Tracks, the filter must be set on the check within the Track itself.

Note that check schedules also allow a filter to be set. You may configure your check to use either filter, or both. The schedule’s filter is used to determine which entities the check should run on when the schedule is triggered. Very commonly, the schedule’s filter is the same as the check’s filter, implying that the scheduled check should run on the same set of entities to which the check applies, but this need not be the case. For example, say you have a set of components to which a check applies, but only expect a subset of those entities to change with any regularity. You could set the check’s filter to the set of all components to which the check applies, and the schedule’s filter to the subset of components that are expected to change more frequently. This setup would allow you to save on compute and rate limits (in the event the check relies on a fact from an external source, such as SCM or Github), while still ensuring that the check is applicable to all entities to which it should apply. You can also set an exclusion filter using the exclude property for both applicability and scheduling. You must use filter if you are using the exclude property. See filters for more details.

Check Filter Examples

Here is an example of a check with a filter set to limit the check to only apply to components of type service, and exclude any entities with a tag of skip:

id: example_check_with_filter
rule:
  all:
    - factRef: catalog:default/entity_descriptor
      path: $.metadata.description
      operator: exists
      value: true
ownerEntityRef: group:default/backstage
name: Example Check With Filter
description: This is a sample check with a check-level filter set.
filter:
  kind:
    - component
  spec.type:
    - service
exclude:
  metadata.tags:
    - skip
passedMessage: 'Pass: The entity has a description set in its metadata.'
failedMessage: 'Failed: The entity does not have a description set in its metadata.'

Below is the same check, but now with a schedule and a more restrictive filter set on the schedule:

id: example_check_with_dual_filter
rule:
  all:
    - factRef: catalog:default/entity_descriptor
      path: $.metadata.description
      operator: exists
      value: true
ownerEntityRef: group:default/backstage
name: Example Check With Filter
description: This is a sample check with a check-level filter set.
filter:
  kind:
    - component
  spec.type:
    - service
exclude:
  spec.lifecycle:
    - deprecated
passedMessage: 'Pass: The entity has a description set in its metadata.'
failedMessage: 'Failed: The entity does not have a description set in its metadata.'
schedule:
  frequency:
    hours: 1
  filter:
    kind:
      - component
    spec.type:
      - service
    spec.lifecycle:
      - production
  exclude:
    spec.system:
      - staging-server

Notes:

The scheduling of checks is not recommended. Instead, schedule the collection of facts that the check depends on through the Fact Collectors’ configurations. Collection of facts will trigger the execution of all checks that depend on those facts. The exception to this recommendation is when you have checks that rely on facts from the Software Catalog or TechInsights, both of which must be scheduled checks.
Scheduling a check without a filter will run the check on all entities in the catalog, regardless of the check’s top-level filter. This is not recommended.

Applicability Rule

Check applicability rules can be used when standard catalog filters are not enough to determine applicability. These determine applicability at the same time check rules are, and can utilize collected facts.

Note: Prefer using catalog filters as these prevent non-applicable checks from executing. Applicability rules can be computationally expensive.

When both catalog filters and applicability rule are defined, the check will be applicable to the entities that both match the catalog filters and pass the applicability rule. Some pages will show the check until the applicability is evaluated. Once evaluated, not applicable checks will not be shown on the Soundcheck for Entity page by default. To view the not applicable checks, click on the small gear icon beside the list of Tracks for the component, and select Show Not Applicable Checks:

Applicability rule has the same structure as the check rule. Below is the example of a check with the applicability rule (applicabile condition):

- id: no_blocker_issues
  name: No Blocker Issues
  description: >
    Check that the component doesn't have blocker issues in SonarQube.
  passedMessage: |
    The component doesn't have blocker issues.
  failedMessage: |
    Blocker issues found in SonarQube, please fix them ASAP.
  rule:
    factRef: sonarqube:default/issues
    path: $.issues[?(@.severity === 'BLOCKER')]
    operator: hasLengthOf
    value: 0
  filter:
    kind: Component
  applicable:
    not:
      - factRef: sonarqube:default/project-tags
        path: $.tags
        operator: includes
        value: do-not-check
  notApplicableMessage: |
    SonarQube project has the 'do-not-check' tag.

Executing Checks

Fact checks are executed by Soundcheck:

When a dependent fact is updated.
Via scheduled execution based on frequency and entities.
Manually triggered via the Checks API.
Manually triggered by clicking on Execute Check icon (⟳) on Soundcheck Entity Tab.

Examples

---
- id: has_less_than_ten_open_issues
  rule:
    factRef: github:default/repo_details
    path: $.open_issues
    operator: lessThan
    value: 10
  passedMessage: |
    Less than 10 open issues
  failedMessage: |
    Ten or more open issue(s)
- id: is_repo_private
  rule:
    factRef: github:default/repo_details
    path: $.private
    operator: equal
    value: true
  passedMessage: |
    Repo is private
  failedMessage: |
    Repo is not private, change repo to private
- id: default_branch_is_main
  rule:
    factRef: github:default/repo_details
    path: $.default_branch
    operator: equal
    value: main
  passedMessage: |
    Default banch is main
  failedMessage: |
    Change default branch to main
- id: has_readme_check # The name of the check
  rule: # How to evaluate this check
    factRef: scm:default/readme_and_catalog_info_files_exist_fact # The fact data to reference
    path: $.readme_exists # The path to the field to analyze
    operator: equal # Indicates the operation to apply
    value: true # The desired value of the field indicated in path, above.
- id: has_catalog_info_file_check
  rule:
    factRef: scm:default/readme_and_catalog_info_files_exist_fact
    path: $.catalog_info_exists
    operator: equal
    value: true
- id: has_pagerduty_integration_key
  rule:
    factRef: catalog:default/entity_descriptor
    path: $.metadata.annotations['pagerduty.com/integration-key']
    operator: matches
    value: .+
  schedule:
    frequency:
      cron: '*/5 * * * 1-5'
    filter:
      kind: 'Component'

- id: has_description
  rule:
    factRef: github:default/repo_details
    path: $.description
    operator: matches
    value: .+
  passedMessage: |
    Repo has a description
  failedMessage: |
    Repo does not have a description

Push Workflow

Soundcheck supports a basic push workflow via our Facts API instead of using a fact collector. Facts sent to this API will automatically run checks associated with the fact’s factRef and entityRef.

Check Creation - UI

In the UI, instead of selecting a fact from the drop down, you will type in the full factRef as seen below in the screen shot. This factRef must be the same as the one you plan to call the Fact API with.

Because Soundcheck is unable to fetch these facts, the following UI features are impacted for these types of checks:

Dry runs will not work.
Manually running the check will not work.
The fact will not show up in the dropdown lists.

To test these checks, see testing your check

Check Creation - YAML

Check creation in YAML is no different from a standard check. Make sure to utilize the same factRef you will be calling the Fact API with.

- id: test-check
  rule:
    factRef: api:default/apiFact
    path: $.value
    operator: equal
    value: test
  passedMessage: |
    Check Passed
  failedMessage: |
    Check Failed

Testing Your Check

You will not be able to manually run a check from the UI like with a standard check. To test your check, make an API call to the Facts API after check creation with the factRef for the check and the entityRef of the entity you want to test. Soundcheck will enqueue and execute checks relevant to this fact and entity after receiving this API call.

REST API

We include a REST API for Checks. See API Reference for details.

Check Result History

By default, Soundcheck retains historical results of checks for 120 days. Other systems can integrate with check result history by reading from the Soundcheck database directly. Check histories can be seen in the Soundcheck UI on the Tech-Health pages, and on the check insights pages (the default view when clicking on a check in the Soundcheck UI).

Modifying Check Result History Settings

To disable check result history, set soundcheck.results.history.enable to false in the config (no value or a value of true will leave it enabled). For example:

app-config.yaml

soundcheck:
  results:
    history:
      enable: false

Soundcheck only updates history when the state of a check result changes. History is retained for a default time of 120 days. You can modify this setting using retentionTimeInDays. For example, this config will instruct Soundcheck to retain history for the past year:

app-config.yaml

soundcheck:
  results:
    history:
      enable: true
      retentionTimeInDays: 365

Soundcheck will automatically clean up old check results when the retention time is reached. The cadence of this cleanup is configurable, as is the time it’s allowed to run before being terminated as a hung process. The default values are as follows, and can be changed according to the needs of your organization:

app-config.yaml

  results:
    history:
      enable: true
      retentionTimeInDays: 120
      cleanupFrequencyCron: '0 0 0 * * *' // Daily at midnight
      cleanupTimeoutInMinutes: 5

Appendix

Recommended GitHub Settings Checks

Pre-built Recommended GitHub Settings checks are based on the GitHub collector configuration and defined as follows:

- id: 21ed0e9c-7fd9-4e59-89dd-33902b1e6bcb
  ownerEntityRef: group:default/backstage
  name: Branch Is Not Locked
  description: This check ensures that the default branch is not locked.
  rule:
    all:
      - factRef: github:default/branch_protections
        path: $.lock_branch.enabled
        operator: equal
        value: false
  filter:
    - kind:
        - component
  passedMessage: Default branch is not locked.
  failedMessage: The default branch is locked, but shouldn't be.

- id: ca05baf0-ff68-4ebd-9559-58a94de84ac3
  ownerEntityRef: group:default/backstage
  name: Pull Request Requires Review
  description: Pull requests should require at least one reviewer.
  rule:
    all:
      - factRef: github:default/branch_protections
        path: $.required_pull_request_reviews.required_approving_review_count
        operator: greaterThan
        value: 0
  filter:
    - kind:
        - component
  passedMessage: Passed because pull requests require at least one reviewer.
  failedMessage: Pull requests do not require any reviewers.

- id: 013091a6-2bc8-46dc-8d34-58207c5b7370
  ownerEntityRef: group:default/backstage
  name: Force Pushes Disabled
  description: >-
    Check to ensure that the repository does not allow force pushes to the main
    branch.
  rule:
    all:
      - factRef: github:default/branch_protections
        path: $.allow_force_pushes.enabled
        operator: equal
        value: false
  filter:
    - kind:
        - component
  passedMessage: Passed, force pushes are not allowed.
  failedMessage: Force pushes are not disabled, but should be.

- id: b812cac0-8f80-4fa0-83cd-6a725c16ac2a
  ownerEntityRef: group:default/backstage
  name: Repository Has Owner
  description: Verifies that the repository has an owner set.
  rule:
    all:
      - factRef: github:default/repository_details
        path: $.owner.login
        operator: matches
        value: .+
  filter:
    - kind:
        - component
  passedMessage: Repository has owner.
  failedMessage: Repository has no owner set.

- id: 0d602bee-f46b-4ff9-8dad-198d82308fa4
  ownerEntityRef: group:default/backstage
  name: Admins Must Follow Rules
  description: This check that ensures that even administrators cannot bypass branch rules.
  rule:
    all:
      - factRef: github:default/branch_protections
        path: $.enforce_admins.enabled
        operator: equal
        value: true
  filter:
    - kind:
        - component
  passedMessage: Admins must follow branch rules, passed.
  failedMessage: Admin rule enforcement is not enabled, but should be.

- id: b113ad76-bb17-41af-980e-1f515f061004
  ownerEntityRef: group:default/backstage
  name: Deletions Are Forbidden
  description: Check to ensure that the repository does not allow deletions.
  rule:
    all:
      - factRef: github:default/branch_protections
        path: $.allow_deletions.enabled
        operator: equal
        value: false
  filter:
    - kind:
        - component
  passedMessage: Repository does not allow deletions.
  failedMessage: Repository allows deletions but shouldn't.

Basic SCM Compliance Checks

Pre-built Basic SCM Compliance checks are based on the SCM collector configuration and defined as follows:

- id: fe5dbf8f-a7dc-43a0-bdc1-4423c0726649
  ownerEntityRef: group:default/backstage
  name: Has Readme File
  description: Check to ensure the component has a readme file.
  rule:
    all:
      - factRef: scm:default/required_files_exist
        path: $.readme
        operator: equal
        value: true
  filter:
    - kind:
        - component
  passedMessage: Component has a readme file.
  failedMessage: Component is missing a readme file.

- id: pre-canned-license-file-check
  ownerEntityRef: group:default/backstage
  name: Has License File
  description: Check to ensure the component has a license file.
  rule:
    all:
      - factRef: scm:default/required_files_exist
        path: $.license
        operator: equal
        value: true
  filter:
    - kind:
        - component
  passedMessage: File found.
  failedMessage: File not found.

- id: pre-canned-conduct-file-check
  ownerEntityRef: group:default/backstage
  name: Defines Code of Conduct
  description: Check to ensure the component has a code of conduct file.
  rule:
    all:
      - factRef: scm:default/required_files_exist
        path: $.code_of_conduct
        operator: equal
        value: true
  filter:
    - kind:
        - component
  passedMessage: File found.
  failedMessage: File not found.

- id: 81c5e85f-cd52-4a0d-a025-2577f4a30db0
  ownerEntityRef: group:default/backstage
  name: API Report File Compliance
  description: Ensures that the api-report file contains the text "Do not edit this file".
  rule:
    all:
      - factRef: scm:default/api-report-has-no-edit-warning
        path: $.matches
        operator: equal
        value: true
  filter:
    - kind:
        - component
  passedMessage: API Report contains "Do not edit this file" warning.
  failedMessage: API Report is missing a "Do not edit this file" warning.

- id: 8ab4de4f-01f3-46b0-8994-14178614543e
  ownerEntityRef: group:default/backstage
  name: Has API Report
  description: Check to ensure the component has an api-report.md file.
  rule:
    all:
      - factRef: scm:default/required_files_exist
        path: $.api_report
        operator: equal
        value: true
  filter:
    - kind:
        - component
  passedMessage: Passed, 'api-report.md' file found.
  failedMessage: Failed, 'api-report.md' file not found.

- id: pre-canned-contributions-file-check
  ownerEntityRef: group:default/backstage
  name: Has Contributions File
  description: Check to ensure the component has a CONTRIBUTING.md file.
  rule:
    all:
      - factRef: scm:default/required_files_exist
        path: $.contributing
        operator: equal
        value: true
  filter:
    - kind:
        - component
  passedMessage: File found.
  failedMessage: File not found.

​Managing checks via the No-Code UI

​Check Creation

​Creating a custom check

​Creating a check from a template

​Editing a check

​Dry Run and Test a Check

​Fact Explorer

​Export to YAML

​Import from YAML

​Managing checks via yaml configuration

​Overall shape of a Check

​Adding yaml checks to your check library

​Remote file update options

​Migrating checks from yaml to no-code UI

​Check Fields

​Rules

​Value Input

​Path Resolvers

​JSON Path

Key Features of JSONPath:

​Lodash Get

Key Features of lodash.get

​JMES Path

Key Features of JMESPath:

​JSONata

Key Features of JSONata:

​Operators

​Boolean, Numeric, & String Operators

​Numeric Operators

​String Operators

SemVer Operators

​Date/time Operators

​Exists Operator

​Array Operators

​Schedule

​Messages

​Example - Using the Entity Object

​Example - Check Fail Message With a Single Fact

​Example - Check Fail Message with Multiple Facts

​Example - Testing JSON Output

​Markdown Support

​Sensitive Facts

​Filters

​Check Filter Examples

​Applicability Rule

​Executing Checks

​Examples

​Push Workflow

​Check Creation - UI

​Check Creation - YAML

​Testing Your Check

​REST API

​Check Result History

​Modifying Check Result History Settings

​Appendix

​Recommended GitHub Settings Checks

​Basic SCM Compliance Checks

Managing checks via the No-Code UI

Check Creation

Creating a custom check

Creating a check from a template

Editing a check

Dry Run and Test a Check

Fact Explorer

Export to YAML

Import from YAML

Managing checks via yaml configuration

Overall shape of a Check

Adding yaml checks to your check library

Remote file update options

Migrating checks from yaml to no-code UI

Check Fields

Rules

Value Input

Path Resolvers

JSON Path

Lodash Get

JMES Path

JSONata

Operators

Boolean, Numeric, & String Operators

Numeric Operators

String Operators

Date/time Operators

Exists Operator

Array Operators

Schedule

Messages

Example - Using the Entity Object

Example - Check Fail Message With a Single Fact

Example - Check Fail Message with Multiple Facts

Example - Testing JSON Output

Markdown Support

Sensitive Facts

Filters

Check Filter Examples

Applicability Rule

Executing Checks

Examples

Push Workflow

Check Creation - UI

Check Creation - YAML

Testing Your Check

REST API

Check Result History

Modifying Check Result History Settings

Appendix

Recommended GitHub Settings Checks

Basic SCM Compliance Checks