> ## Documentation Index
> Fetch the complete documentation index at: https://backstage.spotify.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Diagnose and resolve common Soundcheck issues including checks not running, missing facts, fact collector errors, and incorrect check results.

This section covers the most common problems our first time adopters face and how to troubleshoot them. This guide covers the `Open Source Backstage + Soundcheck` and is **not** intended for `Portal + Soundcheck` (although some answers may be the same).

### Why isn't my check running?

<Accordion title="See the answer">
  There are 3 ways to make a check run. For testing your check you can use option 3, for a sustainable check execution
  use option 1.

  > NOTE: Executing a check is NOT the same as collecting a fact. Fact collection is what fetches data from an API and
  > is subject to caching. Check execution is simply looking at that data and verifying if it satisfies the check
  > criteria. However, after a new fact is collected the checks will execute.

  #### Option 1 - Schedule fact collection (Recommended)

  Each integration has its own documentation page - [Github example
  here](/plugins/soundcheck/core-concepts/fact-collectors/3p-integrations/github#frequency-optional). In most cases
  to schedule a fact collection you just need to add following section to your YAML but please refer to the
  [specific](/plugins/soundcheck/core-concepts/fact-collectors/) fact collector documentation. Once a new fact is
  collected, soundcheck will trigger all reliant checks automatically, so you don't need to schedule them separately.

  ```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
  ---
  frequency:
  cron: '0 * * * *'
  ```

  If you are using the No-Code UI, some integration can be configured to collect facts. Go to `Soundcheck` ->
  `Integrations` and then click on `Configure` next to the collector you want to configure. From there you can schedule
  fact collection like on the screenshot below.

  > NOTE: If the Configure button is disabled or not present that means you either configured the integration from the
  > YAML or that this integration does not support No Code UI yet.

  <Frame>
    <img
      src="https://mintcdn.com/spotify-89f50c35/ulmNVnZzSFL3BsWR/plugins/soundcheck/images/schedule-fact-collector.png?fit=max&auto=format&n=ulmNVnZzSFL3BsWR&q=85&s=02da981d30344c903f530dc5576c3674"
      alt="Schedule Fact
Collector"
      width="1344"
      height="1312"
      data-path="plugins/soundcheck/images/schedule-fact-collector.png"
    />
  </Frame>

  #### Option 2 - Schedule check (Not recommended)

  We recommend you use the first option when possible. However, for some integrations, for example [soundcheck](/plugins/soundcheck/core-concepts/fact-collectors/3p-integrations/soundcheck),
  you will need to schedule the check. This is described in checks section [here](/plugins/soundcheck/core-concepts/checks#schedule).

  #### Option 3 - Manually run check (Recommended for testing only)

  To manually trigger a check to run, navigate to a component page, next click on `Soundcheck` tab and then click the
  `Execute Check` button.

  <Frame>
    <img src="https://mintcdn.com/spotify-89f50c35/ulmNVnZzSFL3BsWR/plugins/soundcheck/images/execute-check.png?fit=max&auto=format&n=ulmNVnZzSFL3BsWR&q=85&s=8241d4b462a63ebaa973e5a8c8c3579a" alt="Execute Check" width="900" height="766" data-path="plugins/soundcheck/images/execute-check.png" />
  </Frame>
</Accordion>

### I'm seeing Track pass percentages that don't align with the Check pass percentages in the Track

<Accordion title="See the answer">
  Pass percentages are based on the number of entities that are applicable to the Check or Track in
  question.

  As an example: say you have a Check that is applicable to 10 entities, and all 10 of them pass.
  Your pass percentage will be 100% for the Check. Say we then make a new Track and add only this
  Check to the Track, but we also set the Track's filters to make it apply to all 10 entities of the
  Check as well as an additional 40 entities. In this case the Track pass percentage will show as
  10/50 = 20%, even though the Check's status on the Track insights page will show as 100%.
</Accordion>

### Why don't I see any history for my checks or certifications?

<Accordion title="See the answer">
  In new versions, the history is enabled by default however you need to make sure you [haven't disabled
  history](/plugins/soundcheck/core-concepts/checks#check-result-history).
</Accordion>

### I am seeing rate limit issues, what can I do?

<Accordion title="See the answer">
  There is a couple of things to try when faced with rate limits.

  #### Check your frequency settings

  Depending on your needs you may tweak the frequency of the fact collection to reduce the number of requests being
  made. The trade off here is you will have data that is potentially stale. Refer to the
  [specific](/plugins/soundcheck/core-concepts/fact-collectors/) collector on how to change the frequency. For example this
  is how you do it in [Github
  Collector](/plugins/soundcheck/core-concepts/fact-collectors/3p-integrations/github#frequency-optional)

  #### Enable cache for the fact collector

  Refer to the [specific](/plugins/soundcheck/core-concepts/fact-collectors/) collector on how to do that. For example this
  is how you do it for [Github
  Collector](/plugins/soundcheck/core-concepts/fact-collectors/3p-integrations/github#cache-optional)

  #### Enable Rate limiting

  You can [enable](/plugins/soundcheck/setup-and-installation#rate-limiting) limiting the amount of requests that
  soundcheck will make from a specific collector. It is highly recommended to also [enable redis
  queues](/plugins/soundcheck/setup-and-installation#scaling-and-rate-limiting) so that the
  limits apply across all pods.

  #### Use API to submit check results

  Normally soundcheck works in pull model, where it fetches facts on a schedule and then executes checks. Soundcheck
  also has an API that enables it to work in push model where other systems such as CI pipelines can push data to
  soundcheck.

  If you [push new facts](/plugins/soundcheck/api-reference/facts/submit-facts) via the API, any
  checks which depend on these facts will automatically be executed.
</Accordion>

### I have a lot of catalog entities and my Backstage instance is now slowing down, what can I do?

<Accordion title="See the answer">
  Refer to the [Scaling Backstage](https://backstage.io/docs/deployment/scaling/) documentation.
</Accordion>

### I'm seeing high system resource usage, what can I do?

<Accordion title="See the answer">
  1. Ensure caching is enabled

  Verify that caching is configured and active. Without persistent cache store, each Soundcheck instance
  will independently perform cache warming jobs and maintain its own in-memory cache. This can significantly
  increase the load on the database and consume additional system resources. Enabling caching allows
  all instances to share cache data efficiently and reduces redundant work.
  More details are available [here](https://backstage.io/docs/overview/architecture-overview/#cache).

  2. Allocate sufficient memory for Node.js old space

  Soundcheck runs cache warming jobs that can be memory-intensive. These jobs may temporarily increase
  both memory and CPU usage, especially during initialization or when preloading large caches.
  To ensure stable operation, users should monitor CPU and memory utilization and adjust the
  [Node.js memory limit](https://nodejs.org/api/cli.html#--max-old-space-sizesize-in-megabytes) as needed.
  For example:

  ```sh theme={"theme":{"light":"github-light","dark":"dracula"}}
  export NODE_OPTIONS="--max-old-space-size=4096"
  ```

  This allocates up to 4 GB of memory for the old space. Adjust the value based on available system
  resources and observed workload behavior. Regular monitoring and gradual tuning of this setting can
  help maintain optimal performance and prevent crashes caused by insufficient heap memory.

  3. Instrument system and application metrics

  To gain deeper visibility into performance, instrument standard system, Node.js, and
  [Soundcheck-specific metrics](../setup-and-installation#open-telemetry-and-instrumentation).
  Monitoring CPU, memory, garbage collection activity, cache hit rates, and job durations can help
  identify bottlenecks early and guide targeted optimizations.
</Accordion>

### Why can't I use all the filters available in YAML in the No Code UI?

<Accordion title="See the answer">
  We are working on bringing the functionality parity to the No code UI. Check our [release
  notes](https://backstage.spotify.com/release-notes/) to see latest improvements.
</Accordion>

### Why am I getting errors with Isolated VM?

<Accordion title="See the answer">
  Isolated VM version 5.0.1 works with node version 22.
  Isolated VM version 4.7.2 works with node version 20.
  Isolated VM version 4.6.0 works with node version 18.

  If you're having issues with Isolated VM errors, check the version of Node you're using, and
  manually pin the version of Isolated VM in your `package.json` to the appropriate version.
</Accordion>

### I'm seeing 'Cannot return null for non-nullable field' type errors on Soundcheck pages

<Accordion title="See the answer">
  Soundcheck makes use of Redis caching. If you encounter an issue with stale or incompatible data,
  clearing the cache may help. You can restart your entire Redis instance, or if you'd like to clear
  only the Soundcheck cache entries, you can use the `redis-cli` command line tool:

  ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
  redis-cli KEYS "soundcheck:*" | xargs redis-cli DEL
  ```

  This will delete all keys that start with `soundcheck:` in your Redis instance (which is to say,
  *all* Soundcheck cache entries), causing Soundcheck to
  (passively) repopulate the cache with the correct types.
</Accordion>

### I'm seeing styling issues in the UI

<Accordion title="See the answer">
  The new Soundcheck UI now has a hard dependency on [Backstage UI](https://ui.backstage.io/). Ensure you are using the latest version and are importing the proper CSS files.
</Accordion>

### My custom theme is no longer applying correctly to Soundcheck

<Accordion title="See the answer">
  The new Soundcheck UI uses [Backstage UI](https://ui.backstage.io/), which has a different [approach to theming](https://backstage.io/docs/conf/user-interface/).
</Accordion>

### My SCM JSON/YAML or Regex retriever returns empty data when Custom Branch is set

<Accordion title="See the answer">
  Branch names containing `/` aren't supported and silently return empty fact data. Use a single-segment name (e.g. replace `/` with `-`). See the [`branch` field](/plugins/soundcheck/core-concepts/fact-collectors/3p-integrations/scm#branch-optional) and upstream [#2815](https://github.com/backstage/backstage/issues/2815) / [#32727](https://github.com/backstage/backstage/issues/32727).
</Accordion>