> ## Documentation Index
> Fetch the complete documentation index at: https://backstage.spotify.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Soundcheck - Advanced Performance Tuning

> Optimize Soundcheck performance with the in-memory catalog and cache warming service to reduce database load and speed up fact collection at scale.

This guide introduces a performance-related optimization in Soundcheck comprised of two new configuration options:
the **In-Memory Catalog** and the **Cache Warming Service**.

These optimizations are designed to work together, and while the in-memory catalog can be used by itself to provide
marginal performance increases, the cache warming service will not operate without the in-memory catalog.

## Quick Start

Don't care about the details? Just want to turn it on and go? No problem! Add the following to your
configuration:

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
soundcheck:
  catalogOptions:
    type: memory # Use in-memory catalog
    refreshInterval: 4 # Rebuild index every X hours (4 by default)
  cacheWarmer:
    frequency: 3 # Run every 3 hours (default)
    timeout: 1 # Max 1 hour per run (default)
    warmGroupsAboveSize: 50 # Only warm groups with 50+ members (default)
```

## In-Memory Catalog for Soundcheck

### Overview

The In-Memory Catalog for Soundcheck maintains a limited copy of your Backstage Catalog entities in memory to enable high-performance querying.
This impacts Soundcheck **only**. The rest of the Backstage plugins as well as Backstage itself is unaffected by this capability.

### When to Use

* **Large Catalogs**: If your catalog has tens of thousands of entities or more
* **Performance Criticality**: If performance of Tech-Insights is slow and/or speed is critical to your use case

### How to Use

Enable the in-memory catalog for Soundcheck via the configuration settings as shown below:

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
soundcheck:
  catalogOptions:
    type: memory # Use in-memory catalog (recommended for large catalogs)
    refreshInterval: 4 # Rebuild index every X hours (4 by default)
```

#### Configuration Parameters

| Parameter         | Type   | Default | Description                                 |
| ----------------- | ------ | ------- | ------------------------------------------- |
| `type`            | string | -       | Set to `memory` to enable in-memory catalog |
| `refreshInterval` | number | 4       | Hours between index rebuilds                |

The `type` parameter can also be set to `default`, but all `catalogOptions` can simply be omitted entirely in this case.
Setting `type` to default or omitting `catalogOptions` will result in Soundcheck running in default mode and using the
standard Catalog.

### Benefits and Tradeoffs

**Benefits:**

* **Speed**: Soundcheck as a whole performs more quickly, as the system has near instant access to
  the Catalog entities.
* **Catalog Load**: Alleviates load on the Catalog plugin.

**Considerations:**

* **Memory usage**: Stores the entire catalog in memory (\~1G for a catalog of 250k entities, allocate ram accordingly)
* **Eventual consistency**: Index refreshes periodically, so recent catalog changes will not be immediately visible
* **Startup time**: Initial index build occurs on startup and will take some time. For a catalog of roughly 250k entities,
  the startup cost is about 5 minutes, after which the refresh will happen transparently in the background without
  user-facing impacts.

### Monitoring

The service logs memory usage and build duration:

```
"[CatalogIndexer] Index built successfully: 240346 entities, 8739 owners in ownedBy index, 287644ms build time, heap 1.57 GB → 2.21 GB (Δ 651.9 MB), entity map size: 711.75 MB, ownedBy index size: 17.86 MB, total: 729.61 MB"
```

## Cache Warming Service

### Overview

The Cache Warming Service proactively pre-populates a cache used by Soundcheck's Tech Insights pages, making them
significantly faster.

**NOTE:** Due to the number of requests made to Catalog by this service, this service **only** operates when the in-memory
Catalog is active (see above).

### Configuration

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
soundcheck:
  cacheWarmer:
    frequency: 3 # Run every 3 hours (default)
    timeout: 1 # Max 1 hour per run (default)
    warmGroupsAboveSize: 50 # Only warm groups with 50+ members (default)
```

#### Configuration Parameters

| Parameter             | Type   | Default | Description                                                                                                                                                      |
| --------------------- | ------ | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `frequency`           | number | 3       | Hours between cache warming runs. The TTL of this cache is 6 hours, do not set this above that value.                                                            |
| `timeout`             | number | 1       | Maximum hours allowed per warming run                                                                                                                            |
| `warmGroupsAboveSize` | number | 50      | Minimum group size to include in warming. Setting this value lower will result in better performance at the tradeoff of increased startup-time and memory usage. |

### Benefits and Tradeoffs

**Benefits:**

* **Fast page loads**: Tech Insights pages will load significantly faster.
* **Predictable performance**: Tech Insights pages will load predictably for all users as the cache will always be warm.

**Considerations:**

* **Configuration tuning**: May need to adjust `warmGroupsAboveSize` based on your org structure
* **Initial delay**: First cache warm happens on startup. This delay is highly dependent on the number and complexity of
  your organization's tracks and how many large groups are to be warmed. Startup times are logged and may be monitored, and
  we recommend tuning to your tolerance of startup times versus your need for speed in the Tech Insights pages.

### Tuning Recommendations

**Small Gains, Low Cost:**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
cacheWarmer:
  frequency: 5 # Less frequent updates
  warmGroupsAboveSize: 100 # Only warm very large groups
```

**Spotify-Recommended:**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
cacheWarmer:
  frequency: 3 # More frequent updates
  warmGroupsAboveSize: 50 # Warm more groups
```

**Performance is Paramount:**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
cacheWarmer:
  frequency: 1 # Warm every hour
  warmGroupsAboveSize: 10 # Warm most groups
```

### Monitoring

The service logs detailed warming activity (below, the latest messages are first):

```
[CacheWarming] Scheduled cache warming completed successfully in 394170ms
...
[CacheWarming] Completed warming track Combat Climate Change (18 groups) in 37062ms
[CacheWarming] Warming cache for track: Combat Climate Change
[CacheWarming] Groups to warm: group:default/example-1, group:default/example-2, ... group:default/example-18
[CacheWarming] Found 18 top-level groups to warm
[CacheWarming] Found 18 groups with size >= 50 to warm
[CacheWarming] Discovering top-level groups...
[CacheWarming] Found 40 tracks to warm cache for
[CacheWarming] Starting scheduled cache warming...
```

## Troubleshooting

### High Memory Usage

* Increase `cacheWarmer.warmGroupsAboveSize` to warm fewer groups

### Slow Page Loads

* Decrease `cacheWarmer.warmGroupsAboveSize` to warm more groups
* Verify `catalogOptions.type: memory` is enabled

### Cache Warming Timeouts (Rare)

* Increase `cacheWarmer.timeout`
* Increase `cacheWarmer.warmGroupsAboveSize` to reduce workload
* Increase `cacheWarmer.frequency` to distribute work over time

### Catalog Data Seems Stale

* Decrease `catalogOptions.refreshInterval` for more frequent updates.