Checks
A check is a comparison between a defined acceptable outcome and the actual outcome of a given process. Checks indicate where the software aligns to or deviates from organizational standards and best practices.
Check logic consists of four main components: facts, paths, operators, and values. These components are then organized via a Boolean calculator in the UI. You can see an example of the rule layout a user will see when creating a check in the image below.
Managing checks via the No-Code UI
Check Creation
Creating a custom check
To create a check, select the facts that you want to use, organize them into rules and give your check a meaningful name and a description. The rules are what defines a check and what a track will use in determining if an entity passes or fails. Simple expressions can be combined using Any Of / All Of to create more complex rules. You also have the option of adding Pass/Fail messages for additional details of why something passed/failed.
Creating a check from a template
Several check templates are included with Soundcheck depending on which integrations you have installed. These check templates prepopulate the create check form with a check name, description, rules, messages, and filters to help expedite the creation of common checks.
Editing a check
Once a check is created, you will be able to manage and edit your check on its detail page. You’ll find no differences in how the pages look when creating/editing a check. From the checks listing page, you will see an option to edit you check, which will bring you to the details page shown below.
You can make as many changes as desired, but just keep in mind that changes are reflected in tracks that may already have said check assigned. Checks can also be a part of multiple tracks, which could have the unintentional effect of causing some tracks to start failing. While the track you want the check updated in may pass, a track setup somewhere else may fail.
Dry Run and Test a Check
Checks can be dry ran and tested in the Review and Test of the check form. This step can be viewed when both editing an existing check and creating a new check.
Here you can view the final configuration options for your check, as well as dry run against applicable entities. If you selected filters for your check, the entities table should already be populated. Otherwise you can type a name in the search bar to get a list of enitites.
Click the button in the dry run section to trigger a dry run. When complete, the dry run will show the check result as well as a details button.
Dry runs executes the check as configured in the form against a fresh set of facts. Dry run results are NOT persisted. This will trigger additional fact collection which will count towards any API limits.
Clicking the details button will open a dialog with additional information, including the full fact, rule by rule breakdowns, and resolved values.
Fact Explorer
In the Check Form, facts can be explored via the Explore Facts button.
After entering a fact, both the fact and path can be tested by entering an entity in the Sample and Test Section. You do not need to enter a path to sample the fact.
This will fetch a fact from the cache if available. If not, this will trigger a fact collection and could count towards your third party API rate limits.
Export to YAML
Checks can be saved as YAML via the dropdown menu on the check cards.
Managing checks via yaml configuration
Checks can be created in code via yaml configuration files.
Overall shape of a Check
A fact check (or more commonly, just a 'check') defines a rule comprised of one or more conditions: a combination of facts,
operators, and values that determine whether the check emits a passed or failed check result to Soundcheck.
Note: the
idfield is a unique identifier for the check. It is used to reference the check in tracks and within the Soundcheck UI. It must be unique across all checks in your library.
Soundcheck supports both checks defined in yaml files and via a no-code UI.
Checks defined in yaml files can be sourced locally or remotely, and require only
two fields: id and rule.
Here is an example of a basic check defined in yaml (more examples):
id: has_pagerduty_integration_key
rule:
factRef: catalog:default/entity_descriptor
path: $.metadata.annotations['pagerduty.com/integration-key']
operator: matches
value: .+
failedMessage: |
No PagerDuty Integration Key defined for this entity.
passedMessage: |
PagerDuty Integration Key defined for this entity.
Soundcheck version 1.32.0+ introduced a new JSON path library (our default path parser).
This library is stricter when validating the path field and requires certain characters to be escaped.
In the example above $.metadata.annotations['pagerduty.com/integration-key'] pagerduty.com/integration-key is escaped with ''.
Adding yaml checks to your check library
To add checks defined in a yaml file to your check library, you need to source them in you app-config.yaml. You have the option of storing them locally or remotely.
Here is an example configuration should your check yaml files be local to your Backstage project:
soundcheck:
checks:
- $include: ./path-to-local-folder/checks-file-1.yaml
- $include: ./path-to-local-folder/checks-file-2.yaml
- $include: ./path-to-local-folder/checks-file-3.yaml
To maintain your checks files in a remote repo, you can source them in app-config.yaml in a similar manner. Here is an example configuration:
soundcheck:
checks:
- https://github.com/user/repo/blob/main/checks-file-1.yaml
- https://github.com/user/repo/blob/main/checks-file-2.yaml
- https://github.com/user/repo/blob/main/checks-file-3.yaml
remote_file_updates:
disabled: false
frequency:
minutes: 5
Note: you cannot combine local and remote sourcing for checks files. This includes the usage of
$includewithin a remotely sourced checks file. The below option will NOT work:
soundcheck:
checks:
- https://github.com/user/repo/blob/main/checks-file-1.yaml
- https://github.com/user/repo/blob/main/checks-file-2.yaml
- $include: ./path-to-local-folder/checks-file-3.yaml
Remote file update options
The remote_file_updates object is optional configuration allowing you to control if and when Soundcheck looks for updates within your remote files. This configuration is global for Soundcheck, so will apply to both track and check files. If not explicitly set, see the below for default values.
| Key | Default | Description |
|---|---|---|
disabled | false | Enables Soundcheck to look for updates in remote files |
frequency | minutes: 5 | The frequency at Soundcheck will look for updates in remote files. Possible values are either a cron expression { cron: ... } or HumanDuration. |
Check Fields
| Field | Required | Description |
|---|---|---|
id | Yes | The unique identifier for this fact check. |
name | No | A name for the fact check suitable for display on a UI. |
description | No | A description of the fact check suitable for display on a UI. |
rule | Yes | One or more conditions that determine check result. |
passedMessage | No | Message if the check passes. |
failedMessage | No | Message if the check fails. |
schedule | No | How often and on what entities the fact check should run. |
ownerEntityRef | No | Reference to the owner of the check. |
filter | No | Default filter specifying what entities the check applies to. |
pathResolver | No | The path resolver the rules will use. Defaults to jsonpath. |
The name and description fields can be used to provide more context to the check.
If not provided, the id will be used as the name when viewing the check in Soundcheck.
When creating checks via the no-code UI, the name, description, and owner, fields become required.
Rules
A rule contains one or more conditions.
The simplest form of a condition consists of a fact reference, an operator, and a value. When the check is executed, the operator is used to compare the fact against the value.
Example of a simple condition:
rule:
factRef: catalog:default/entity_descriptor
path: $.metadata.annotations['pagerduty.com/integration-key']
operator: matches
value: .+
failedMessage: |
No PagerDuty Integration Key defined for this entity.
More complex conditions can by created through the use of boolean expressions. A rule may
optionally have either an all, any, or 'not' expression at its root, containing an array of
conditions. The all operator specifies that all conditions
contained within must be truthy for the check to pass. The any operator only requires one
condition to be truthy for the check to pass, while the not operator negates the result of the
condition it contains.
Example of a complex condition:
rule:
any:
- factRef: catalog:default/entity_descriptor
path: $.metadata.labels['language']
operator: equal
value: java
- factRef: catalog:default/entity_descriptor
path: $.metadata.labels['language']
operator: equal
value: kotlin
Example of a check with a negated condition:
- id: no_r_rated_language
description: >
Check that the language is not Rust or Ruby.
failedMessage: |
Language is Rust or Ruby.
passedMessage: |
Language is not Rust or Ruby.
rule:
not:
- any:
- factRef: catalog:default/entity_descriptor
path: $.metadata.labels['language']
operator: equal
value: rust
- factRef: catalog:default/entity_descriptor
path: $.metadata.labels['language']
operator: equal
value: ruby
Value Input
The rule value can be a singular or a list, or an object with another fact.
Example using a list of values:
rule:
factRef: catalog:default/entity_descriptor
path: $.kind
operator: in
value:
- Group
- Component
Example using a fact:
rule:
factRef: catalog:default/entity_descriptor
path: $.spec.type
operator: equal
value:
factRef: custom:default/custom_fact
path: $.service_type
Path Resolvers
The path field in a rule is used to resolve a path to a value in a fact, and it is this resolved value that is used in
the comparison with the value field. This allows you to reference specific parts of a fact to compare against a value.
Soundcheck supports multiple path resolvers, which can be specified in the pathResolver field of a check. If no path
resolver is specified, Soundcheck will use its default path resolver (JSON Path).
The path resolver option applies to all rules for a check.
Example:
id: annotation-defined
ownerEntityRef: group:default/check-owner
name: Annotation Defined
description: Verifies that annotations are defined on the entity.
pathResolver: jsonata
rule:
factRef: catalog:default/entity_descriptor
path: $keys($.metadata.annotations)
operator: contains
value: 0
Available options of the box are:
jsonpath- defaultlodashjmespathjsonata
JSON Path
Soundcheck's default path resolver.
JSONPath is a query language for JSON, similar to XPath for XML. It allows you to navigate and filter JSON data to extract specific values or sets of values. JSONPath expressions are used to traverse the JSON data structure and extract the data that matches the expression.
Key Features of JSONPath:
- Dot Notation and Bracket Notation: You can use dot notation (
$.store.book[0].title) or bracket notation ($['store']['book'][0]['title']) to navigate the JSON structure. - Wildcard (
*): Selects all elements in an array or all properties in an object. - Recursive Descent (
..): Searches recursively through the JSON structure. - Filters: Allows you to filter JSON data based on expressions. For example,
$.store.book[?(@.price < 10)]selects books with a price less than 10. - Slicing: Similar to array slicing in Python, e.g.,
$.store.book[:2]selects the first two books.
Example
Given the following JSON data:
{
"store": {
"book": [
{
"category": "reference",
"author": "Nigel Rees",
"title": "Sayings of the Century",
"price": 8.95
},
{
"category": "fiction",
"author": "Evelyn Waugh",
"title": "Sword of Honour",
"price": 12.99
}
],
"bicycle": {
"color": "red",
"price": 19.95
}
}
}
JSONPath Expressions:
$.store.book[*].authorreturns["Nigel Rees", "Evelyn Waugh"]$.store.bicycle.colorreturns"red"$..pricereturns[8.95, 12.99, 19.95]$.store.book[?(@.price < 10)].titlereturns["Sayings of the Century"]
For more information on the JSON Path syntax, see the JSON Path documentation.
Lodash Get
lodash.get is a utility function from the Lodash library, which is a popular JavaScript utility library. The get
function is used to access deeply nested properties in objects.
Key Features of lodash.get
- Path Notation: You can specify the path to the property using a dot notation string or an array of keys.
- Simplicity: Easy to use syntax.
Example
Given the following JSON object:
{
"store": {
"book": [
{
"category": "reference",
"author": "Nigel Rees",
"title": "Sayings of the Century",
"price": 8.95
},
{
"category": "fiction",
"author": "Evelyn Waugh",
"title": "Sword of Honour",
"price": 12.99
}
],
"bicycle": {
"color": "red",
"price": 19.95
}
}
}
lodash.get Expressions:
'store.book[0].author'returns"Nigel Rees"
For more information on the Lodash Get syntax, see the Lodash Get documentation.
JMES Path
JMESPath is a query language for JSON, designed to allow you to extract and transform elements from JSON data structures. Similar to JSONPath, it enables you to navigate and filter JSON data, but it has its own syntax and features.
Key Features of JMESPath:
- Dot Notation: Used to navigate through JSON objects. For example,
foo.barwill navigate to thebarproperty of thefooobject. - Filters: Allows filtering of arrays. For example,
foo[?bar > 10]will filter thefooarray to include only elements wherebaris greater than10. - Projections: Allows transformation of arrays and objects. For example,
foo[*].barwill create a list of thebarproperties from each object in thefooarray. - Functions: JMESPath includes built-in functions such as
length,max,min,sum, andcontains.
Example
Given the following JSON data:
{
"store": {
"book": [
{
"category": "reference",
"author": "Nigel Rees",
"title": "Sayings of the Century",
"price": 8.95
},
{
"category": "fiction",
"author": "Evelyn Waugh",
"title": "Sword of Honour",
"price": 12.99
}
],
"bicycle": {
"color": "red",
"price": 19.95
}
}
}
JMESPath Expressions:
store.book[*].authorreturns["Nigel Rees", "Evelyn Waugh"]store.bicycle.colorreturns"red"store.book[?price > 10].titlereturns["Sword of Honour"]store.book[?category == 'fiction'].pricereturns[12.99]
For more information on the JMES Path syntax, see the JMES Path documentation.
JSONata
JSONata is a lightweight query and transformation language for JSON data. It allows you to query, filter, and transform JSON data in a concise and expressive way. JSONata includes features for selecting data, performing calculations, and creating new JSON structures based on existing data.
Key Features of JSONata:
- Expression Language: JSONata expressions enable complex querying and transformations of JSON data.
- Path Notation: Similar to XPath for XML, JSONata uses a path notation to navigate through JSON structures.
- Functions: JSONata includes built-in functions for manipulating data, such as mathematical operations, string operations, and more.
- Conditionals: You can use conditional expressions to filter and transform data based on specific criteria.
- Mapping and Reducing: JSONata supports mapping and reducing operations to transform arrays and objects.
Example
Given the following JSON data:
{
"store": {
"book": [
{
"category": "reference",
"author": "Nigel Rees",
"title": "Sayings of the Century",
"price": 8.95
},
{
"category": "fiction",
"author": "Evelyn Waugh",
"title": "Sword of Honour",
"price": 12.99
}
],
"bicycle": {
"color": "red",
"price": 19.95
}
}
}
JSONata Expressions:
store.book.authorreturns["Nigel Rees", "Evelyn Waugh"]store.bicycle.colorreturns"red"store.book[price < 10].titlereturns["Sayings of the Century"]store.book.{ "title": title, "cost": price }returns:
[
{ "title": "Sayings of the Century", "cost": 8.95 },
{ "title": "Sword of Honour", "cost": 12.99 }
]
For more information on the JSONata syntax, see the JSONata documentation.
Operators
Boolean, Numeric, & String Operators
These operators support boolean, numeric, and string values.
| Operator | Description |
|---|---|
equal | Fact must equal value. |
notEqual | Fact must not equal value. |
When dealing with arrays of booleans, numbers, or strings, these operators can be used in combination with the all:,
any:, and none: prefixes to validate the elements of the array. See the Array Operators section
for additional details.
Numeric Operators
These operators support only numeric values.
| Operator | Description |
|---|---|
lessThan | Fact must be less than value. |
lessThanInclusive | Fact must be less than or equal to value. |
greaterThan | Fact must be greater than value. |
greaterThanInclusive | Fact must be greater than or equal to value. |
When dealing with arrays of numbers these operators can be used in combination with the all:, any:, and none:
prefixes to validate the elements of the array. See the Array Operators section for additional details.
String Operators
These operators support only string values.
| Operator | Description |
|---|---|
matches | Fact string matches given value regex. |
When dealing with arrays of strings these operators can be used in combination with the all:, any:, and none:
prefixes to validate the elements of the array. See the Array Operatorssection for additional details.
SemVer Operators
These operators support only string values that represent semantic versions or semantic version ranges.
| Operator | Description |
|---|---|
semverGt | Fact version must be greater than value version. |
semverGte | Fact version must be greater than or equal to value version. |
semverLt | Fact version must be less than value version. |
semverLte | Fact version must be less than or equal to value version. |
semverEq | Fact version must equal value version. |
semverNeq | Fact version must not equal value version. |
semverSatisfies | Fact version satisfies value range. |
semverGtr | Fact version is greater than all versions in value range. |
semverLtr | Fact version is less than all versions in value range. |
When dealing with arrays of string representing semantic versions or semantic version ranges these operators can be used
in combination with the all:, any:, and none: prefixes to validate the elements of the array. See the
Array Operators section for additional details.
Misc. Operators
These operators support values of any type.
| Operator | Description |
|---|---|
exists | Fact value must exist/be defined. |
When dealing with arrays these operators can be used in combination with the all:, any:, and none: prefixes to
validate the elements of the array. See the Array Operators section for additional details.
Array Operators
These operators support array values.
| Operator | Description |
|---|---|
in | Fact must be included in array value. |
notIn | Fact must not be included in array value. |
contains | Fact array must include value. |
doesNotContain | Fact array must not include value. |
hasLengthOf | fact array must have length equal to value. |
Additionally, you can use the following prefixes in combination with any of the above operators to validate the elements of an array:
all:- Validates that all elements in the array satisfy the condition.any:- Validates that at least one element in the array satisfies the condition.none:- Validates that none of elements in the array satisfies the condition.
For example, to validate that all elements in an array are greater than 10 we can use the greaterThan operator with
the all: prefix:
factRef: kubernetes:default/k8s_pods
path: $.items[*].spec.terminationGracePeriodSeconds
operator: all:greaterThan
value: 10
Schedule
NOTE: In general checks should not be scheduled; prefer scheduling fact collections. When Soundcheck collects a fact it will automatically execute all checks that depend on that fact.
Scheduling options for fact checks:
| Field | Required | Description |
|---|---|---|
frequency | Yes | Cron expression or HumanDuration for check execution frequency. |
entityRefs | No | Entities to run check on (overrides filter). |
filter | No | Filter specifying which entities to execute the check on. |
exclude | No | Filter specifying which entities to exclude. |
Messages
Pass messages - passedMessage - are displayed to the user when a check passes. Fail messages - failedMessage - are displayed to the user
when a check fails. Messages can be provided as a string or as a markdown string. Pass and fail messages
also support Liquid templating. The following variables are available for use in templates:
entity- The entity the check was executed on.facts- A map of facts used by the check. Individual facts can be looked up in this map by fact reference. For example,facts['catalog:default/entity_descriptor'].- If all the facts used by the check have unique fact names they can be accessed in
factsby fact name rather than having to use the entire fact reference. For example, if check uses thecatalog:default/entity_descriptorit can be accessed byfacts['catalog:default/entity_descriptor']orfacts['entity_descriptor']. - If only a single fact is used by the check, the fact can be accessed directly. For example, if
catalog:default/entity_descriptorwas the only fact used by the check it can be accessed asfactrather thanfacts['catalog:default/entity_descriptor'].
- If all the facts used by the check have unique fact names they can be accessed in
Here is an example of a pass message using templates:
{{ entity.metadata.name }} has a PagerDuty Integration Key defined.
in this example, Soundcheck would replace {{ entity.metadata.name }} with the name of the entity the check was
executed on.
You can also render the output as JSON. This is particularly useful for displaying facts for testing and debugging. Wrap in a code block to preserve formatting. For example:
```json
{{ fact | json: 4 }}
```
Would give you an output like:
{
"factRef": {
"source": "github",
"scope": "default",
"name": "repository_details"
},
"data": {
"entityRef": "component:default/example",
"html_url": "https://github.com/backstage/backstage",
"description": "Backstage is an open platform for building developer portals",
"fork": false,
"url": "https://api.github.com/repos/backstage/backstage"
}
}
Filters
Checks can be filtered to the desired set of entities using the same Entity Filter syntax as Tracks.
NOTE: Specifying a filter on a check sets the default filter for the check. This filter is used when the check is added to a Track using Soundcheck's UI. If using YAML to define Tracks, the filter must be set on the check within the Track itself.
Note that check schedules also allow a filter to be set. You may configure your check to use either filter, or both. The schedule's filter is used to determine which entities the check should run on when the schedule is triggered. Very commonly, the schedule's filter is the same as the check's filter, implying that the scheduled check should run on the same set of entities to which the check applies, but this need not be the case.
For example, say you have a set of components to which a check applies, but only expect a subset of those entities to change with any regularity. You could set the check's filter to the set of all components to which the check applies, and the schedule's filter to the subset of components that are expected to change more frequently. This setup would allow you to save on compute and rate limits (in the event the check relies on a fact from an external source, such as SCM or Github), while still ensuring that the check is applicable to all entities to which it should apply.
Check Filter Examples
Here is an example of a check with a filter set to limit the check to only apply to components of
type service:
id: example_check_with_filter
rule:
all:
- factRef: catalog:default/entity_descriptor
path: $.metadata.description
operator: exists
value: true
ownerEntityRef: group:default/backstage
name: Example Check With Filter
description: This is a sample check with a check-level filter set.
filter:
catalog:
kind:
- component
spec.type:
- service
passedMessage: 'Pass: The entity has a description set in its metadata.'
failedMessage: 'Failed: The entity does not have a description set in its metadata.'
Below is the same check, but now with a schedule and a more restrictive filter set on the schedule:
id: example_check_with_dual_filter
rule:
all:
- factRef: catalog:default/entity_descriptor
path: $.metadata.description
operator: exists
value: true
ownerEntityRef: group:default/backstage
name: Example Check With Filter
description: This is a sample check with a check-level filter set.
filter:
catalog:
kind:
- component
spec.type:
- service
passedMessage: 'Pass: The entity has a description set in its metadata.'
failedMessage: 'Failed: The entity does not have a description set in its metadata.'
schedule:
frequency:
hours: 1
filter:
kind:
- component
spec.type:
- service
spec.lifecycle:
- production
exclude:
spec.system:
- staging-server
Notes:
- The scheduling of checks is not recommended. Instead, schedule the collection of facts that the check depends on through the Fact Collectors' configurations. Collection of facts will trigger the execution of all checks that depend on those facts. The exception to this recommendation is when you have checks that rely on facts from the Software Catalog or TechInsights, both of which must be scheduled checks.
- Scheduling a check without a filter will run the check on all entities in the catalog, regardless of the check's top-level filter. This is not recommended.
- The careful observer will notice that the two filters above are slightly different. The check
filter (not the schedule filter) contains the necessary
catalog:key. This is the filter type used in Soundcheck for checks and tracks and corresponds to the options available in the Soundcheck No-Code UI. The schedule filter on the other hand does not contain thecatalog:key, and is the more powerful filter type used by catalog. - Exclusion filters using the
excludesetting are only supported with the schedule filter.
Executing Checks
Fact checks are executed by Soundcheck:
- When a dependent fact is updated.
- Via scheduled execution based on frequency and entities.
- Manually triggered via the Checks API.
- Manually triggered by clicking on Execute Check icon (⟳) on Soundcheck Entity Tab.
Examples
---
- id: has_less_than_ten_open_issues
rule:
factRef: github:default/repo_details
path: $.open_issues
operator: lessThan
value: 10
passedMessage: |
Less than 10 open issues
failedMessage: |
Ten or more open issue(s)
- id: is_repo_private
rule:
factRef: github:default/repo_details
path: $.private
operator: equal
value: true
passedMessage: |
Repo is private
failedMessage: |
Repo is not private, change repo to private
- id: default_branch_is_main
rule:
factRef: github:default/repo_details
path: $.default_branch
operator: equal
value: main
passedMessage: |
Default banch is main
failedMessage: |
Change default branch to main
- id: has_readme_check # The name of the check
rule: # How to evaluate this check
factRef: scm:default/readme_and_catalog_info_files_exist_fact # The fact data to reference
path: $.readme_exists # The path to the field to analyze
operator: equal # Indicates the operation to apply
value: true # The desired value of the field indicated in path, above.
- id: has_catalog_info_file_check
rule:
factRef: scm:default/readme_and_catalog_info_files_exist_fact
path: $.catalog_info_exists
operator: equal
value: true
- id: has_pagerduty_integration_key
rule:
factRef: catalog:default/entity_descriptor
path: $.metadata.annotations['pagerduty.com/integration-key']
operator: matches
value: .+
schedule:
frequency:
cron: '*/5 * * * 1-5'
filter:
kind: 'Component'
- id: has_description
rule:
factRef: github:default/repo_details
path: $.description
operator: matches
value: .+
passedMessage: |
Repo has a description
failedMessage: |
Repo does not have a description
REST API
We include a REST API for Checks. See API Reference for details.
Check Result History
By default, Soundcheck retains historical results of checks for 120 days.
Other systems can integrate with check result history by reading from the Soundcheck database directly. Check histories can be seen in the Soundcheck UI on the Tech-Health pages, and on the check insights pages (the default view when clicking on a check in the Soundcheck UI).
Modifying Check Result History Settings
To disable check result history, set soundcheck.results.history.enable to false
in the config (no value or a value of true will leave it enabled). For example:
soundcheck:
results:
history:
enable: false
Soundcheck only updates history when the state of a check result changes.
History is retained for a default time of 120 days.
You can modify this setting using retentionTimeInDays. For example, this config
will instruct Soundcheck to retain history for the past year:
soundcheck:
results:
history:
enable: true
retentionTimeInDays: 365
Soundcheck will automatically clean up old check results when the retention time is reached. The cadence of this cleanup is configurable, as is the time it's allowed to run before being terminated as a hung process. The default values are as follows, and can be changed according to the needs of your organization:
results:
history:
enable: true
retentionTimeInDays: 120
cleanupFrequencyCron: '0 0 0 * * *' // Daily at midnight
cleanupTimeoutInMinutes: 5