Skip to main content

Soundcheck - Advanced Performance Tuning

This guide introduces a performance-related optimization in Soundcheck comprised of two new configuration options: the In-Memory Catalog and the Cache Warming Service.

These optimizations are designed to work together, and while the in-memory catalog can be used by itself to provide marginal performance increases, the cache warming service will not operate without the in-memory catalog.

Quick Start​

Don't care about the details? Just want to turn it on and go? No problem! Add the following to your configuration:

soundcheck:
  catalogOptions:
    type: memory # Use in-memory catalog
    refreshInterval: 4 # Rebuild index every X hours (4 by default)
  cacheWarmer:
    frequency: 3 # Run every 3 hours (default)
    timeout: 1 # Max 1 hour per run (default)
    warmGroupsAboveSize: 50 # Only warm groups with 50+ members (default)

In-Memory Catalog for Soundcheck​

Overview​

The In-Memory Catalog for Soundcheck maintains a limited copy of your Backstage Catalog entities in memory to enable high-performance querying. This impacts Soundcheck only. The rest of the Backstage plugins as well as Backstage itself is unaffected by this capability.

When to Use​

  • Large Catalogs: If your catalog has tens of thousands of entities or more
  • Performance Criticality: If performance of Tech-Insights is slow and/or speed is critical to your use case

How to Use​

Enable the in-memory catalog for Soundcheck via the configuration settings as shown below:

soundcheck:
  catalogOptions:
    type: memory # Use in-memory catalog (recommended for large catalogs)
    refreshInterval: 4 # Rebuild index every X hours (4 by default)

Configuration Parameters​

ParameterTypeDefaultDescription
typestring-Set to memory to enable in-memory catalog
refreshIntervalnumber4Hours between index rebuilds

The type parameter can also be set to default, but all catalogOptions can simply be omitted entirely in this case. Setting type to default or omitting catalogOptions will result in Soundcheck running in default mode and using the standard Catalog.

Benefits and Tradeoffs​

Benefits:

  • Speed: Soundcheck as a whole performs more quickly, as the system has near instant access to the Catalog entities.
  • Catalog Load: Alleviates load on the Catalog plugin.

Considerations:

  • Memory usage: Stores the entire catalog in memory (~1G for a catalog of 250k entities, allocate ram accordingly)
  • Eventual consistency: Index refreshes periodically, so recent catalog changes will not be immediately visible
  • Startup time: Initial index build occurs on startup and will take some time. For a catalog of roughly 250k entities, the startup cost is about 5 minutes, after which the refresh will happen transparently in the background without user-facing impacts.

Monitoring​

The service logs memory usage and build duration:

"[CatalogIndexer] Index built successfully: 240346 entities, 8739 owners in ownedBy index, 287644ms build time, heap 1.57 GB → 2.21 GB (Δ 651.9 MB), entity map size: 711.75 MB, ownedBy index size: 17.86 MB, total: 729.61 MB"

Cache Warming Service​

Overview​

The Cache Warming Service proactively pre-populates a cache used by Soundcheck's Tech Insights pages, making them significantly faster.

NOTE: Due to the number of requests made to Catalog by this service, this service only operates when the in-memory Catalog is active (see above).

Configuration​

soundcheck:
  cacheWarmer:
    frequency: 3 # Run every 3 hours (default)
    timeout: 1 # Max 1 hour per run (default)
    warmGroupsAboveSize: 50 # Only warm groups with 50+ members (default)

Configuration Parameters​

ParameterTypeDefaultDescription
frequencynumber3Hours between cache warming runs. The TTL of this cache is 6 hours, do not set this above that value.
timeoutnumber1Maximum hours allowed per warming run
warmGroupsAboveSizenumber50Minimum group size to include in warming. Setting this value lower will result in better performance at the tradeoff of increased startup-time and memory usage.

Benefits and Tradeoffs​

Benefits:

  • Fast page loads: Tech Insights pages will load significantly faster.
  • Predictable performance: Tech Insights pages will load predictably for all users as the cache will always be warm.

Considerations:

  • Configuration tuning: May need to adjust warmGroupsAboveSize based on your org structure
  • Initial delay: First cache warm happens on startup. This delay is highly dependent on the number and complexity of your organization's tracks and how many large groups are to be warmed. Startup times are logged and may be monitored, and we recommend tuning to your tolerance of startup times versus your need for speed in the Tech Insights pages.

Tuning Recommendations​

Small Gains, Low Cost:

cacheWarmer:
  frequency: 5 # Less frequent updates
  warmGroupsAboveSize: 100 # Only warm very large groups

Spotify-Recommended:

cacheWarmer:
  frequency: 3 # More frequent updates
  warmGroupsAboveSize: 50 # Warm more groups

Performance is Paramount:

cacheWarmer:
  frequency: 1 # Warm every hour
  warmGroupsAboveSize: 10 # Warm most groups

Monitoring​

The service logs detailed warming activity (below, the latest messages are first):

[CacheWarming] Scheduled cache warming completed successfully in 394170ms
...
[CacheWarming] Completed warming track Combat Climate Change (18 groups) in 37062ms
[CacheWarming] Warming cache for track: Combat Climate Change
[CacheWarming] Groups to warm: group:default/example-1, group:default/example-2, ... group:default/example-18
[CacheWarming] Found 18 top-level groups to warm
[CacheWarming] Found 18 groups with size >= 50 to warm
[CacheWarming] Discovering top-level groups...
[CacheWarming] Found 40 tracks to warm cache for
[CacheWarming] Starting scheduled cache warming...

Troubleshooting​

High Memory Usage​

  • Increase cacheWarmer.warmGroupsAboveSize to warm fewer groups

Slow Page Loads​

  • Decrease cacheWarmer.warmGroupsAboveSize to warm more groups
  • Verify catalogOptions.type: memory is enabled

Cache Warming Timeouts (Rare)​

  • Increase cacheWarmer.timeout
  • Increase cacheWarmer.warmGroupsAboveSize to reduce workload
  • Increase cacheWarmer.frequency to distribute work over time

Catalog Data Seems Stale​

  • Decrease catalogOptions.refreshInterval for more frequent updates.