How Spotify measures the value of Backstage

All data analysis and research was conducted by Michael Bellato, Senior Data Scientist at Spotify. Last updated: April 20, 2023

tl;dr We measured Backstage usage with developer activity, development lifecycle, and developer retention. Spoiler alert: It makes all three better!

How Backstage translates to quantifiable impact

At Spotify, we get a lot of questions around Backstage and the value it can give to organizations, and rightly so. We talk about how Backstage makes developers happier and more productive, but what does that actually mean in practice? A couple years ago, we published a blog detailing how we measure Backstage success at Spotify. This blog highlighted proxy metrics to gauge things we care about moving the needle on like time-to-tenth-PR, number of plugins contributed, satisfaction rates, etc.

But this time around, we wanted to share metrics that we believe affect a business’ bottom line through a clear return on investment: developer activity, development lifecycle, and developer retention. And we found some pretty cool stuff…

At Spotify, we see frequent Backstage users:

That’s a lot! So let’s unpack all of these stats a bit.

Spotify devs do better with Backstage

Before we get too deep into the findings, let’s talk about our methodology for this analysis. Michael Bellato, Senior Data Scientist at Spotify, observed Backstage activity for the first half of 2022 for developers. After observing that group, he measured developer activities, retention, and satisfaction for the same group for the next 6 months to a year. For simplicity, this pre-post approach in our analysis does not account for potential seasonality effects. As well, given the level of adoption for Backstage within Spotify with our internal developer community, there was no possibility to carry out our analysis by using a control group of non Backstage users. We therefore used observational methods to determine Backstage’s value and grouped users into 2 categories: frequent and less frequent users. In order to ensure both groups of users were as comparable as possible, we then controlled and matched factors such as their tenure (in number of weeks) and work mode (i.e. office or remote).

Now onto the findings! At Spotify, we believe that Backstage improves developer effectiveness, which is a function of satisfaction and productivity. So we set out to quantify what these terms mean in practice for us. For satisfaction, we felt that retention was a good indicator of how a developer’s happiness or satisfaction at a company affects the bottom line. For productivity, we looked at developer output and speed. Lastly, we found what Backstage means for the longevity of a developer’s output.

Productivity

There are so many ways to quantify productivity. Through user research we have discovered that Backstage increases productivity through the ability to find, understand, and take action on information; in general, organizations find they are able to do more with less. With consideration to these findings, we have focused this research on productivity as developer output and speed.

First, we found that developers that use Backstage more often have 2.3x more developer activity in GitHub in the following six months than the bottom Backstage users. From our internal developer survey that we send out every quarter we also noticed that developers that are more active have higher ratings of their perceived productivity (Psst! If you’re interested in setting up your own survey in Backstage, check out Pulse).

We also saw that frequent Backstage users create over 2x more code changes than infrequent Backstage users. On top of that, these code changes take 17% less cycle time for more frequent Backstage users.

When we combined the reduced cycle time with increased PR activity, we found that this had an effect of saving 3 full time employees (FTEs) for a team of 10 developers. Developers that are more productive in less time, save time and resources for an engineering organization.

Development lifecycle and speed

But what does speed matter if you’re not able to ensure improvements in the overall development lifecycle? When your developers deploy new software, you want to ensure it meets standards and you don’t have to change or deprecate it in short order. From user research we found that Backstage’s flexibility enables organizations to address a wide range of needs in a wide range of contexts, thus adding efficiencies to existing capabilities. These efficiencies are what we have been able to confirm at Spotify with the adoption of Backstage throughout the different stages of the development lifecycle; evidenced with the productivity improvements measured in our most recent analysis. At Spotify, we found that high frequency Backstage users deploy software to production 2x as often and their software stays deployed 3x as long. We believe that higher rates of deployments to production and their increased longevity are indications of higher code quality, as well as increased developer speed and productivity.

Satisfaction

In our analysis, we found that employees that use Backstage more frequently are 5% more likely to be at Spotify 12 months later. For a team of 100 developers, this is an increase in five FTEs. Keeping your developers happy matters.

Conclusion

At Spotify, high frequency Backstage users deploy software more often in less time. These users are more active, report higher perceived productivity ratings, and stay at Spotify longer.

We plan to continue this type of research and share our results with you because we believe in the power of Backstage and how it affects a company’s bottom line.

But don’t just take our word for it! Adopters like Toyota North American Motors have run their own analysis and found huge savings (to the tune of $10M) with Backstage.

Interested in sharing your Backstage story? Contact us.

Want to hear more news like this? Sign up for the Backstage community newsletter.