How Soundcheck improves tech health and developer productivity at Spotify

Data analysis and research was conducted by Christian Farfan Centeno, Senior Data Scientist at Spotify, and Karl Erliksson, Senior Data Scientist at Spotify.

tl;dr: Spotify has thousands of developers managing tens of thousands of software components. We cut through the complexity that comes with this kind of scale through Backstage. And the real magic happens through plugins like Soundcheck, which codifies standards and incentivizes tech health improvements to drive developer productivity and higher performing teams.

This is not exactly a hot take 🌶️, but the entire process of creating software rewards faster speed and lower quality. Cutting corners, breaking things, and taking on tech debt are all in service of getting good ideas built and products into the market as fast as possible. The goal of most projects is to meet a deadline to ship, not to create quality code. Think hare 🐰 not tortoise 🐢.

If developers aren’t incentivized to care about tech health, a vicious cycle of short-term gain/long-term pain can set in around software creation and maintenance. Ultimately, this cycle slows teams down and leads to a wealth of challenges, from lack of standardized code to engineer attrition.

At Spotify, we created the Soundcheck plugin to combat some of these same challenges as our code base and engineering teams grew. What’s Soundcheck? Well, I’m so glad you asked!

Soundcheck ensures that all of your software is built to the engineering standards that you expect at your organization, while making it easy for all of your engineers to know about and follow those standards. At Spotify, Soundcheck started as an effort to draw a straight line between an individual developer’s decision-making to our topline goals for engineering excellence and quality.

And the proof is in the pudding! Just by showing our developers opportunities to improve our tech health, we’ve driven a significant decrease in technology fragmentation and seen higher developer productivity rates (skip to the proof section for some tl;dr stats, we’ll be following up with a Part 2 blog post detailing the methodology behind the studies we’ve ran).

In essence, using Soundcheck at Spotify has greatly reduced the speed vs. quality paradox. We get to move quickly without compromising tech health because even when we build fast and acquire tech debt, there’s a clear path to address it.

So let’s dive in on how Soundcheck works at Spotify!

Testing 1…2…3

In a company of Spotify’s scale, teams work with around tens of thousands of software components on a daily basis. Pre-Soundcheck, there were around 30K+ software components in Spotify’s Backstage and that number was growing fast.

Manual methods to track and maintain these components did not scale. For example, we had this Service Operational Quality Checklist floating around that tracked the quality of services owned by squads across standards like Security, Recoverability, and Maintainability, scoring them between Poor-Excellent using color codes (see a screenshot of the old days below).

Imagine maintaining such checklists for 30K+ components! 😱 This complexity led our internal teams that focused on improving developer experience to build Soundcheck, which made tracking development and operational standards and measuring the tech health of software components much easier.

Having software components that look and work in similar ways lowers the threshold and cost of collaboration, greatly simplifies changes in the ownership of components at an individual or at an organization level, and enables us to reduce toil through automation.

But Soundcheck doesn’t exist in a vacuum — it works in harmony with a broader tech health strategy at Spotify. You’ve probably heard us talk about Golden Technologies or the Golden Path 🌟 before, but let’s do a quick primer on everything Golden within Spotify.

Golden Paths are our onboarding tutorials, both for new joiners as well as existing engineers picking up new disciplines. They are guides that introduce the engineer to how we develop in a specific discipline or a specific area at Spotify.

Golden Technologies are a set of standard languages, frameworks, and infrastructure systems which are expected (sometimes mandatory) to be used for all production components in Spotify. Using them allows teams to focus on their features, since maintenance is reduced with Platform teams taking care of upgrades.

Golden State is the desired state of our software components. It's the target state in which we want Spotify's tech stack to be, so we can keep our fleet of software up-to-date easily and bring everyone across the company into new technological paradigms, enabling Spotify to keep up with the ever-evolving technology landscape in the world.

So you’ve got to follow the Golden Path (the recommended set of technologies and patterns) using Golden Technologies (standardized/blessed/supported stuff) to get to the Golden State. Got it? Awesome. 👏

At its core, Soundcheck applies checks and guidance against components to ensure the components are aligned to an organization’s tech health initiatives. Some of its features include checks, tracks, levels and certifications, and the fact framework.

So let’s start with tracks: tracks encourage alignment to organizational best practices and standards. Think of a track as a long-term health initiative that your organization wants to adopt. Some examples of tracks within Spotify’s Soundcheck include “The Golden State for Web”, “Test Certified for Backend”, and “Fleet Management.”

We’ve leveraged Soundcheck to measure adoption of these initiatives by rolling out the Golden State track. The Golden State track is a set of checks that recommends actions for engineers to evolve their software components towards the desired Golden State.

Advisory Boards and TAG (Technology Architecture Group) are responsible for the definition of the Golden Technologies for their discipline and govern the checks associated with each level in the Golden State track.

If a component passes all the checks for Golden State Level 1 (also known as the standardization level), then that component is verified as using the expected Golden Technologies.

Getting to Golden

Let’s get into some Psych 101 🧠 for a minute. In Knowledge Workers' Deeper Needs: Why Economic Incentives and Technology Are Not Enough , the author mentions that Victor Frankl’s book A Man’s Search for Meaning helped him better understand intrinsic motivators — or what he has come to consider our deeper needs. In this book, Frankl argues that a human’s deepest need is purpose.

Engineers are creative people and they have tons of amazing ideas for solving complex problems. However, they need to find their sense of purpose and be challenged on a daily basis to be able to be productive — which is, in part, why we’ve used gamification within Soundcheck levels. Gamification has enabled squads to understand and achieve their goals while giving a sense of satisfaction and accomplishment which has helped cultivate a stronger sense of purpose.

And let’s face it, engineers (and all humans) can also be competitive. At Spotify, squads view customized dashboards that define broader org targets and notice each other progressing towards their goals and earning badges. This encourages them to engage in healthy competition and work towards their targets while having fun. Believe me when I say there is a lot of Slack chatter about Squad A creeping past Squad B on the number of levels their components are clearing!

While we track the above with customized dashboards today, the Soundcheck team is working hard to add the ability to view and filter aggregated dashboards within Soundcheck itself — for both our internal users and plugin customers. For Backstage adopters using the Soundcheck plugin, this means they’ll be able to view and compare tech health of their teams without context switching away from Backstage to a different reporting tool.

Let us dive deeper into Soundcheck’s incentive structure through Test Certified for X (TC4X), one of Spotify’s gamified, multi-step tracks for getting squads within Spotify to improve testing practices and test coverage of our backend, web, client, or data pipeline (it shamelessly borrows from Google’s TC initiative).

Within each of these tracks, there are levels and certifications. For example in TC4B (Test Certified for Backend), Level 1 is all about measuring and raising awareness of a squad’s current test and code quality situation. Level 2 targets getting into a steady state with basic quality in place ensuring health of automatic checks, and Level 3 focuses on how the service behaves as a part of the ecosystem of services. Think of Level 3 as how this service can be a good citizen. 🏅 As engineers make checks within a level flip from red to green, they receive badges for completion of a level: bronze for Level 1 certification, silver for achieving Level 2 and — of course — a gold badge for hitting Level 3 (see an example below).

For failing checks, Soundcheck allows internal users to set a resolution plan for themselves letting the owner initiate when they can make a check go from red to green (see the screenshot below). This is not currently available for external Soundcheck customers, but it is on our roadmap!

By applying the TC4X track, squads:

• Get guidance on how to write correct tests using the best tools and tech available
• Have increased confidence in the quality of their features
• Get support from test mentors on test infrastructure
• Learn from squads already following the Test Certified Track
• Collaborate better with other squads by reaching out to check and track owners to solve failing checks together
• Have fun and get bragging rights! 🎉

The proof…or is it the pudding?

We ran a recent analysis on Soundcheck’s impact at Spotify on a squad’s engineering metrics and found some really exciting results. We’ll be sharing much more on the impact study methodology in Part 2 but here’s the tl;dr on what we found.

Soundcheck had measurable impact across the following dimensions:

• Productivity boosts: The more you pass, the more you deploy. As squads increase their Golden State pass rate, deployments increase as well; with up to 7 more deployments per percentage point gained at component level.
• Efficiency gains: Good as gold; squads experience a ~20% decrease in master build failures, and a ~25% decrease in high-urgency incidents when maintaining a Golden State pass rate of 85%.
• Virtuous cycle: Progress, not perfection; when high-urgency incidents do occur, squads release fixes more quickly. Squads have a ~1.5 hour faster resolution of high-urgency incidents per 10 percentage points in a squad’s Golden State pass rate. Squads average over 3 high urgency incidents a month, which translates to ~5 hours saved monthly per squad.

Additionally both Golden State and TC4X squad scores were found to correlate with decreases to the average merge time of a squad’s PRs.

In another study, Our Client Platform Studio — the team enabling our developers to build and release features for the Spotify mobile and web clients — also analyzed the impact of driving large tech improvements for our iOS and Android developers, which included the rollout of the Mobile Golden State track in Soundcheck.

Throughout 2022, the Client Platform team found the following results:

• Mobile developers developed code faster. The equivalent of 35 full time employee (FTEs) hours were saved by an up to 46% decrease in time waiting for PR builds and tests.
• Mobile developers iterated faster on their code and 8 FTEs were saved since time to run local build invocations dropped by up to 56%.
• Mobile developer satisfaction increased from 50% to 75% between Q4’21 and Q4’22. They were able to iterate faster on their features, and they reported feeling more productive and creative.

Check out a demo

So there you have it! The speed vs. quality tradeoff doesn’t have to be such a tradeoff. At Spotify, Backstage and Soundcheck have allowed us to tackle ambitious tech health goals without compromising developer productivity and speed — in fact, we see Soundcheck creating net-positive effects on our developer experience.

We’re continuing to iterate on the Soundcheck plugin to help other companies realize similar value. You can check out the full Release Notes on what we’ve been up to over the past few months. If you’re interested in learning more, please join one of our weekly demos. We can’t wait to hear what you think!