Skip to main content

Checks

A check is a comparison between a defined acceptable outcome and the actual outcome of a given process. Checks indicate where the software aligns to or deviates from organizational standards and best practices.

Soundcheck Checks

Check logic consists of four main components: facts, paths, operators, and values. These components are then organized via a Boolean calculator in the UI. You can see an example of the rule layout a user will see when creating a check in the image below.

Soundcheck Boolean Calculator

Managing checks via the No-Code UI

Check Creation

Creating a custom check

To create a check, select the facts that you want to use, organize them into rules and give your check a meaningful name and a description. The rules are what defines a check and what a track will use in determining if an entity passes or fails. Simple expressions can be combined using Any Of / All Of to create more complex rules. You also have the option of adding Pass/Fail messages for additional details of why something passed/failed.

Soundcheck Create Check

Creating a check from a template

Several check templates are included with Soundcheck depending on which integrations you have installed. These check templates prepopulate the create check form with a check name, description, rules, messages, and filters to help expedite the creation of common checks.

Editing a check

Once a check is created, you will be able to manage and edit your check on its detail page. You’ll find no differences in how the pages look when creating/editing a check. From the checks listing page, you will see an option to edit you check, which will bring you to the details page shown below.

Soundcheck Edit Check

You can make as many changes as desired, but just keep in mind that changes are reflected in tracks that may already have said check assigned. Checks can also be a part of multiple tracks, which could have the unintentional effect of causing some tracks to start failing. While the track you want the check updated in may pass, a track setup somewhere else may fail.

Dry Run and Test a Check

Checks can be dry ran and tested in the Review and Test of the check form. This step can be viewed when both editing an existing check and creating a new check.

Check Review and Test

Here you can view the final configuration options for your check, as well as dry run against applicable entities. If you selected filters for your check, the entities table should already be populated. Otherwise you can type a name in the search bar to get a list of enitites.

Click the button in the dry run section to trigger a dry run. When complete, the dry run will show the check result as well as a details button.

Check Dry Run

Dry runs executes the check as configured in the form against a fresh set of facts. Dry run results are NOT persisted. This will trigger additional fact collection which will count towards any API limits.

Clicking the details button will open a dialog with additional information, including the full fact, rule by rule breakdowns, and resolved values.

Check Review Details

Fact Explorer

In the Check Form, facts can be explored via the Explore Facts button.

Check Fact Explorer

After entering a fact, both the fact and path can be tested by entering an entity in the Sample and Test Section. You do not need to enter a path to sample the fact.

Check Fact Explorer Details

This will fetch a fact from the cache if available. If not, this will trigger a fact collection and could count towards your third party API rate limits.

Export to YAML

Checks can be saved as YAML via the dropdown menu on the check cards.

Check-Export

Managing checks via yaml configuration

Checks can be created in code via yaml configuration files.

Overall shape of a Check

A fact check (or more commonly, just a 'check') defines a rule comprised of one or more conditions: a combination of facts, operators, and values that determine whether the check emits a passed or failed check result to Soundcheck.

Note: the id field is a unique identifier for the check. It is used to reference the check in tracks and within the Soundcheck UI. It must be unique across all checks in your library.

Soundcheck supports both checks defined in yaml files and via a no-code UI. Checks defined in yaml files can be sourced locally or remotely, and require only two fields: id and rule.

Here is an example of a basic check defined in yaml (more examples):

id: has_pagerduty_integration_key
rule:
factRef: catalog:default/entity_descriptor
path: $.metadata.annotations['pagerduty.com/integration-key']
operator: matches
value: .+
failedMessage: |
No PagerDuty Integration Key defined for this entity.
passedMessage: |
PagerDuty Integration Key defined for this entity.

Soundcheck version 1.32.0+ introduced a new JSON path library (our default path parser).

This library is stricter when validating the path field and requires certain characters to be escaped. In the example above $.metadata.annotations['pagerduty.com/integration-key'] pagerduty.com/integration-key is escaped with ''.

Adding yaml checks to your check library

To add checks defined in a yaml file to your check library, you need to source them in you app-config.yaml. You have the option of storing them locally or remotely.

Here is an example configuration should your check yaml files be local to your Backstage project:

app-config.yaml
soundcheck:
checks:
- $include: ./path-to-local-folder/checks-file-1.yaml
- $include: ./path-to-local-folder/checks-file-2.yaml
- $include: ./path-to-local-folder/checks-file-3.yaml

To maintain your checks files in a remote repo, you can source them in app-config.yaml in a similar manner. Here is an example configuration:

app-config.yaml
soundcheck:
checks:
- https://github.com/user/repo/blob/main/checks-file-1.yaml
- https://github.com/user/repo/blob/main/checks-file-2.yaml
- https://github.com/user/repo/blob/main/checks-file-3.yaml
remote_file_updates:
disabled: false
frequency:
minutes: 5

Note: you cannot combine local and remote sourcing for checks files. This includes the usage of $include within a remotely sourced checks file. The below option will NOT work:

app-config.yaml
soundcheck:
checks:
- https://github.com/user/repo/blob/main/checks-file-1.yaml
- https://github.com/user/repo/blob/main/checks-file-2.yaml
- $include: ./path-to-local-folder/checks-file-3.yaml

Remote file update options

The remote_file_updates object is optional configuration allowing you to control if and when Soundcheck looks for updates within your remote files. This configuration is global for Soundcheck, so will apply to both track and check files. If not explicitly set, see the below for default values.

KeyDefaultDescription
disabledfalseEnables Soundcheck to look for updates in remote files
frequencyminutes: 5The frequency at Soundcheck will look for updates in remote files. Possible values are either a cron expression { cron: ... } or HumanDuration.

Check Fields

FieldRequiredDescription
idYesThe unique identifier for this fact check.
nameNoA name for the fact check suitable for display on a UI.
descriptionNoA description of the fact check suitable for display on a UI.
ruleYesOne or more conditions that determine check result.
passedMessageNoMessage if the check passes.
failedMessageNoMessage if the check fails.
scheduleNoHow often and on what entities the fact check should run.
ownerEntityRefNoReference to the owner of the check.
filterNoDefault filter specifying what entities the check applies to.

The name and description fields can be used to provide more context to the check. If not provided, the id will be used as the name when viewing the check in Soundcheck.

When creating checks via the no-code UI, the name, description, and owner, fields become required.

Rules

A rule contains one or more conditions.

The simplest form of a condition consists of a fact reference, an operator, and a value. When the check is executed, the operator is used to compare the fact against the value.

Example of a simple condition:

rule:
factRef: catalog:default/entity_descriptor
path: $.metadata.annotations['pagerduty.com/integration-key']
operator: matches
value: .+
failedMessage: |
No PagerDuty Integration Key defined for this entity.

More complex conditions can by created through the use of boolean expressions. A rule may optionally have either an all, any, or 'not' expression at its root, containing an array of conditions. The all operator specifies that all conditions contained within must be truthy for the check to pass. The any operator only requires one condition to be truthy for the check to pass, while the not operator negates the result of the condition it contains.

Example of a complex condition:

rule:
any:
- factRef: catalog:default/entity_descriptor
path: $.metadata.labels['language']
operator: equal
value: java
- factRef: catalog:default/entity_descriptor
path: $.metadata.labels['language']
operator: equal
value: kotlin

Example of a check with a negated condition:

- id: no_r_rated_language
description: >
Check that the language is not Rust or Ruby.
failedMessage: |
Language is Rust or Ruby.
passedMessage: |
Language is not Rust or Ruby.
rule:
not:
- any:
- factRef: catalog:default/entity_descriptor
path: $.metadata.labels['language']
operator: equal
value: rust
- factRef: catalog:default/entity_descriptor
path: $.metadata.labels['language']
operator: equal
value: ruby

Value Input

The rule value can be a singular or a list, or an object with another fact.

Example using a list of values:

rule:
factRef: catalog:default/entity_descriptor
path: $.kind
operator: in
value:
- Group
- Component

Example using a fact:

rule:
factRef: catalog:default/entity_descriptor
path: $.spec.type
operator: equal
value:
factRef: custom:default/custom_fact
path: $.service_type

Path Resolvers

The path field in a rule is used to resolve a path to a value in a fact, and it is this resolved value that is used in the comparison with the value field. This allows you to reference specific parts of a fact to compare against a value.

Soundcheck supports multiple path resolvers, which can be specified in the pathResolver field of a check. If no path resolver is specified, Soundcheck will use its default path resolver (JSON Path).

JSON Path

Soundcheck's default path resolver.

JSONPath is a query language for JSON, similar to XPath for XML. It allows you to navigate and filter JSON data to extract specific values or sets of values. JSONPath expressions are used to traverse the JSON data structure and extract the data that matches the expression.

Key Features of JSONPath:
  • Dot Notation and Bracket Notation: You can use dot notation ($.store.book[0].title) or bracket notation ($['store']['book'][0]['title']) to navigate the JSON structure.
  • Wildcard (*): Selects all elements in an array or all properties in an object.
  • Recursive Descent (..): Searches recursively through the JSON structure.
  • Filters: Allows you to filter JSON data based on expressions. For example, $.store.book[?(@.price < 10)] selects books with a price less than 10.
  • Slicing: Similar to array slicing in Python, e.g., $.store.book[:2] selects the first two books.

Example

Given the following JSON data:

{
"store": {
"book": [
{
"category": "reference",
"author": "Nigel Rees",
"title": "Sayings of the Century",
"price": 8.95
},
{
"category": "fiction",
"author": "Evelyn Waugh",
"title": "Sword of Honour",
"price": 12.99
}
],
"bicycle": {
"color": "red",
"price": 19.95
}
}
}

JSONPath Expressions:

  • $.store.book[*].author returns ["Nigel Rees", "Evelyn Waugh"]
  • $.store.bicycle.color returns "red"
  • $..price returns [8.95, 12.99, 19.95]
  • $.store.book[?(@.price < 10)].title returns ["Sayings of the Century"]

For more information on the JSON Path syntax, see the JSON Path documentation.

Lodash Get

lodash.get is a utility function from the Lodash library, which is a popular JavaScript utility library. The get function is used to access deeply nested properties in objects.

Key Features of lodash.get
  • Path Notation: You can specify the path to the property using a dot notation string or an array of keys.
  • Simplicity: Easy to use syntax.

Example

Given the following JSON object:

{
"store": {
"book": [
{
"category": "reference",
"author": "Nigel Rees",
"title": "Sayings of the Century",
"price": 8.95
},
{
"category": "fiction",
"author": "Evelyn Waugh",
"title": "Sword of Honour",
"price": 12.99
}
],
"bicycle": {
"color": "red",
"price": 19.95
}
}
}

lodash.get Expressions:

  • 'store.book[0].author' returns "Nigel Rees"

For more information on the Lodash Get syntax, see the Lodash Get documentation.

JMES Path

JMESPath is a query language for JSON, designed to allow you to extract and transform elements from JSON data structures. Similar to JSONPath, it enables you to navigate and filter JSON data, but it has its own syntax and features.

Key Features of JMESPath:
  • Dot Notation: Used to navigate through JSON objects. For example, foo.bar will navigate to the bar property of the foo object.
  • Filters: Allows filtering of arrays. For example, foo[?bar > 10] will filter the foo array to include only elements where bar is greater than 10.
  • Projections: Allows transformation of arrays and objects. For example, foo[*].bar will create a list of the bar properties from each object in the foo array.
  • Functions: JMESPath includes built-in functions such as length, max, min, sum, and contains.

Example

Given the following JSON data:

{
"store": {
"book": [
{
"category": "reference",
"author": "Nigel Rees",
"title": "Sayings of the Century",
"price": 8.95
},
{
"category": "fiction",
"author": "Evelyn Waugh",
"title": "Sword of Honour",
"price": 12.99
}
],
"bicycle": {
"color": "red",
"price": 19.95
}
}
}

JMESPath Expressions:

  • store.book[*].author returns ["Nigel Rees", "Evelyn Waugh"]
  • store.bicycle.color returns "red"
  • store.book[?price > 10].title returns ["Sword of Honour"]
  • store.book[?category == 'fiction'].price returns [12.99]

For more information on the JMES Path syntax, see the JMES Path documentation.

JSONata

JSONata is a lightweight query and transformation language for JSON data. It allows you to query, filter, and transform JSON data in a concise and expressive way. JSONata includes features for selecting data, performing calculations, and creating new JSON structures based on existing data.

Key Features of JSONata:
  • Expression Language: JSONata expressions enable complex querying and transformations of JSON data.
  • Path Notation: Similar to XPath for XML, JSONata uses a path notation to navigate through JSON structures.
  • Functions: JSONata includes built-in functions for manipulating data, such as mathematical operations, string operations, and more.
  • Conditionals: You can use conditional expressions to filter and transform data based on specific criteria.
  • Mapping and Reducing: JSONata supports mapping and reducing operations to transform arrays and objects.

Example

Given the following JSON data:

{
"store": {
"book": [
{
"category": "reference",
"author": "Nigel Rees",
"title": "Sayings of the Century",
"price": 8.95
},
{
"category": "fiction",
"author": "Evelyn Waugh",
"title": "Sword of Honour",
"price": 12.99
}
],
"bicycle": {
"color": "red",
"price": 19.95
}
}
}

JSONata Expressions:

  • store.book.author returns ["Nigel Rees", "Evelyn Waugh"]
  • store.bicycle.color returns "red"
  • store.book[price < 10].title returns ["Sayings of the Century"]
  • store.book.{ "title": title, "cost": price } returns:
  [
{ "title": "Sayings of the Century", "cost": 8.95 },
{ "title": "Sword of Honour", "cost": 12.99 }
]

For more information on the JSONata syntax, see the JSONata documentation.

Operators

Boolean, Numeric, & String Operators

These operators support boolean, numeric, and string values.

OperatorDescription
equalFact must equal value.
notEqualFact must not equal value.

When dealing with arrays of booleans, numbers, or strings, these operators can be used in combination with the all:, any:, and none: prefixes to validate the elements of the array. See the Array Operators section for additional details.

Numeric Operators

These operators support only numeric values.

OperatorDescription
lessThanFact must be less than value.
lessThanInclusiveFact must be less than or equal to value.
greaterThanFact must be greater than value.
greaterThanInclusiveFact must be greater than or equal to value.

When dealing with arrays of numbers these operators can be used in combination with the all:, any:, and none: prefixes to validate the elements of the array. See the Array Operators section for additional details.

String Operators

These operators support only string values.

OperatorDescription
matchesFact string matches given value regex.

When dealing with arrays of strings these operators can be used in combination with the all:, any:, and none: prefixes to validate the elements of the array. See the Array Operatorssection for additional details.

SemVer Operators

These operators support only string values that represent semantic versions or semantic version ranges.

OperatorDescription
semverGtFact version must be greater than value version.
semverGteFact version must be greater than or equal to value version.
semverLtFact version must be less than value version.
semverLteFact version must be less than or equal to value version.
semverEqFact version must equal value version.
semverNeqFact version must not equal value version.
semverSatisfiesFact version satisfies value range.
semverGtrFact version is greater than all versions in value range.
semverLtrFact version is less than all versions in value range.

When dealing with arrays of string representing semantic versions or semantic version ranges these operators can be used in combination with the all:, any:, and none: prefixes to validate the elements of the array. See the Array Operators section for additional details.

Misc. Operators

These operators support values of any type.

OperatorDescription
existsFact value must exist/be defined.

When dealing with arrays these operators can be used in combination with the all:, any:, and none: prefixes to validate the elements of the array. See the Array Operators section for additional details.

Array Operators

These operators support array values.

OperatorDescription
inFact must be included in array value.
notInFact must not be included in array value.
containsFact array must include value.
doesNotContainFact array must not include value.
hasLengthOffact array must have length equal to value.

Additionally, you can use the following prefixes in combination with any of the above operators to validate the elements of an array:

  • all: - Validates that all elements in the array satisfy the condition.
  • any: - Validates that at least one element in the array satisfies the condition.
  • none: - Validates that none of elements in the array satisfies the condition.

For example, to validate that all elements in an array are greater than 10 we can use the greaterThan operator with the all: prefix:

factRef: kubernetes:default/k8s_pods
path: $.items[*].spec.terminationGracePeriodSeconds
operator: all:greaterThan
value: 10

Schedule

NOTE: In general checks should not be scheduled; prefer scheduling fact collections. When Soundcheck collects a fact it will automatically execute all checks that depend on that fact.

Scheduling options for fact checks:

FieldRequiredDescription
frequencyYesCron expression or HumanDuration for check execution frequency.
entityRefsNoEntities to run check on (overrides filter).
filterNoFilter specifying which entities to execute the check on.
excludeNoFilter specifying which entities to exclude.

Messages

Pass messages - passedMessage - are displayed to the user when a check passes. Fail messages - failedMessage - are displayed to the user when a check fails. Messages can be provided as a string or as a markdown string. Pass and fail messages also support Liquid templating. The following variables are available for use in templates:

  • entity - The entity the check was executed on.
  • facts - A map of facts used by the check. Individual facts can be looked up in this map by fact reference. For example, facts['catalog:default/entity_descriptor'].
    • If all the facts used by the check have unique fact names they can be accessed in facts by fact name rather than having to use the entire fact reference. For example, if check uses the catalog:default/entity_descriptor it can be accessed by facts['catalog:default/entity_descriptor'] or facts['entity_descriptor'].
    • If only a single fact is used by the check, the fact can be accessed directly. For example, if catalog:default/entity_descriptor was the only fact used by the check it can be accessed as fact rather than facts['catalog:default/entity_descriptor'].

Here is an example of a pass message using templates:

{{ entity.metadata.name }} has a PagerDuty Integration Key defined.

in this example, Soundcheck would replace {{ entity.metadata.name }} with the name of the entity the check was executed on.

You can also render the output as JSON. This is particularly useful for displaying facts for testing and debugging. Wrap in a code block to preserve formatting. For example:

```json
{{ fact | json: 4 }}
```

Would give you an output like:

{
"factRef": {
"source": "github",
"scope": "default",
"name": "repository_details"
},
"data": {
"entityRef": "component:default/example",
"html_url": "https://github.com/backstage/backstage",
"description": "Backstage is an open platform for building developer portals",
"fork": false,
"url": "https://api.github.com/repos/backstage/backstage"
}
}

Filters

Checks can be filtered to the desired set of entities using the same Entity Filter syntax as Tracks.

NOTE: Specifying a filter on a check sets the default filter for the check. This filter is used when the check is added to a Track using Soundcheck's UI. If using YAML to define Tracks, the filter must be set on the check within the Track itself.

Note that check schedules also allow a filter to be set. You may configure your check to use either filter, or both. The schedule's filter is used to determine which entities the check should run on when the schedule is triggered. Very commonly, the schedule's filter is the same as the check's filter, implying that the scheduled check should run on the same set of entities to which the check applies, but this need not be the case.

For example, say you have a set of components to which a check applies, but only expect a subset of those entities to change with any regularity. You could set the check's filter to the set of all components to which the check applies, and the schedule's filter to the subset of components that are expected to change more frequently. This setup would allow you to save on compute and rate limits (in the event the check relies on a fact from an external source, such as SCM or Github), while still ensuring that the check is applicable to all entities to which it should apply.

Check Filter Examples

Here is an example of a check with a filter set to limit the check to only apply to components of type service:

id: example_check_with_filter
rule:
all:
- factRef: catalog:default/entity_descriptor
path: $.metadata.description
operator: exists
value: true
ownerEntityRef: group:default/backstage
name: Example Check With Filter
description: This is a sample check with a check-level filter set.
filter:
catalog:
kind:
- component
spec.type:
- service
passedMessage: 'Pass: The entity has a description set in its metadata.'
failedMessage: 'Failed: The entity does not have a description set in its metadata.'

Below is the same check, but now with a schedule and a more restrictive filter set on the schedule:

id: example_check_with_dual_filter
rule:
all:
- factRef: catalog:default/entity_descriptor
path: $.metadata.description
operator: exists
value: true
ownerEntityRef: group:default/backstage
name: Example Check With Filter
description: This is a sample check with a check-level filter set.
filter:
catalog:
kind:
- component
spec.type:
- service
passedMessage: 'Pass: The entity has a description set in its metadata.'
failedMessage: 'Failed: The entity does not have a description set in its metadata.'
schedule:
frequency:
hours: 1
filter:
kind:
- component
spec.type:
- service
spec.lifecycle:
- production
exclude:
spec.system:
- staging-server

Notes:

  • The scheduling of checks is not recommended. Instead, schedule the collection of facts that the check depends on through the Fact Collectors' configurations. Collection of facts will trigger the execution of all checks that depend on those facts. The exception to this recommendation is when you have checks that rely on facts from the Software Catalog or TechInsights, both of which must be scheduled checks.
  • Scheduling a check without a filter will run the check on all entities in the catalog, regardless of the check's top-level filter. This is not recommended.
  • The careful observer will notice that the two filters above are slightly different. The check filter (not the schedule filter) contains the necessary catalog: key. This is the filter type used in Soundcheck for checks and tracks and corresponds to the options available in the Soundcheck No-Code UI. The schedule filter on the other hand does not contain the catalog: key, and is the more powerful filter type used by catalog.
  • Exclusion filters using the exclude setting are only supported with the schedule filter.

Executing Checks

Fact checks are executed by Soundcheck:

  1. When a dependent fact is updated.
  2. Via scheduled execution based on frequency and entities.
  3. Manually triggered via the Checks API.
  4. Manually triggered by clicking on Execute Check icon (⟳) on Soundcheck Entity Tab.

Examples

---
- id: has_less_than_ten_open_issues
rule:
factRef: github:default/repo_details
path: $.open_issues
operator: lessThan
value: 10
passedMessage: |
Less than 10 open issues
failedMessage: |
Ten or more open issue(s)
- id: is_repo_private
rule:
factRef: github:default/repo_details
path: $.private
operator: equal
value: true
passedMessage: |
Repo is private
failedMessage: |
Repo is not private, change repo to private
- id: default_branch_is_main
rule:
factRef: github:default/repo_details
path: $.default_branch
operator: equal
value: main
passedMessage: |
Default banch is main
failedMessage: |
Change default branch to main
- id: has_readme_check # The name of the check
rule: # How to evaluate this check
factRef: scm:default/readme_and_catalog_info_files_exist_fact # The fact data to reference
path: $.readme_exists # The path to the field to analyze
operator: equal # Indicates the operation to apply
value: true # The desired value of the field indicated in path, above.
- id: has_catalog_info_file_check
rule:
factRef: scm:default/readme_and_catalog_info_files_exist_fact
path: $.catalog_info_exists
operator: equal
value: true
- id: has_pagerduty_integration_key
rule:
factRef: catalog:default/entity_descriptor
path: $.metadata.annotations['pagerduty.com/integration-key']
operator: matches
value: .+
schedule:
frequency:
cron: '*/5 * * * 1-5'
filter:
kind: 'Component'

- id: has_description
rule:
factRef: github:default/repo_details
path: $.description
operator: matches
value: .+
passedMessage: |
Repo has a description
failedMessage: |
Repo does not have a description

REST API

We include a REST API for Checks. See API Reference for details.

Check Result History

By default, Soundcheck retains historical results of checks for 120 days.

Other systems can integrate with check result history by reading from the Soundcheck database directly. Check histories can be seen in the Soundcheck UI on the Tech-Health pages, and on the check insights pages (the default view when clicking on a check in the Soundcheck UI).

Modifying Check Result History Settings

To disable check result history, set soundcheck.results.history.enable to false in the config (no value or a value of true will leave it enabled). For example:

app-config.yaml
soundcheck:
results:
history:
enable: false

Soundcheck only updates history when the state of a check result changes.

History is retained for a default time of 120 days. You can modify this setting using retentionTimeInDays. For example, this config will instruct Soundcheck to retain history for the past year:

app-config.yaml
soundcheck:
results:
history:
enable: true
retentionTimeInDays: 365

Soundcheck will automatically clean up old check results when the retention time is reached. The cadence of this cleanup is configurable, as is the time it's allowed to run before being terminated as a hung process. The default values are as follows, and can be changed according to the needs of your organization:

app-config.yaml
  results:
history:
enable: true
retentionTimeInDays: 120
cleanupFrequencyCron: '0 0 0 * * *' // Daily at midnight
cleanupTimeoutInMinutes: 5