Beyond the classroom: The impact of learning with Skill Exchange

Research and analysis by Matt Klubeck, Data Scientist at Spotify.

Review and validation by Vani D’ Angelo, Senior Data Scientist at Spotify, and Munji Choi, Senior Data Scientist at Spotify.

tl;dr We conducted an observational study and found that employees who participated in applied learning activities on Skill Exchange displayed a 19% increase in productivity rates, a 17% increase in frequency of achieving flow, a 9% increase in velocity satisfaction, a 21% increase in promotion rates, and a 15% decrease in attrition rates.

At Spotify, we believe learning is key in driving innovation, agility, and of course, happiness. For us, learning comes in a variety of forms; there’s formal learning, like courses or training modules, and there’s informal learning, which comes from peer-to-peer collaboration and other on-the-job experiences. We call these informal learning opportunities applied learning.

Measuring the impact of learning is a historically challenging area for many organizations. We can ask for feedback on whether participants attended and liked a training, but how do we know it actually had an impact on their work? And how does that tie back to an organization’s goals?

Consider the Kirkpatrick Model, a schema for evaluating the effectiveness of training at four levels:

1. Reaction: Did the employee enjoy the training?
2. Learning: Did the employee learn something?
3. Behavior: Did the employee practice the behavior as a result of the training?
4. Results: The business impact of the behavior

Level Four is the holy grail, but this is where organizations often struggle: showing the real business impact of learning. When budgets are cut, learning is among the first to go. Which is why it’s so important to show our leaders that learning is worth investing in! And this is doubly challenging for informal learning, where data is rarely collected.

Enter Skill Exchange!

We built Skill Exchange to connect all types of learning, whether that’s hacking on a project, finding a mentor, or embedding with a different team. By providing a way for Spotify employees to focus on applied learning, we believe we are creating a more robust learning landscape and culture that leads to measurable benefits.

But that’s not all! Because Skill Exchange data allows us to evaluate the learning journeys of our users across various types of learning, we can get a picture of which activities and behaviors are more or less likely to result in long-term positive outcomes. We can even begin to assess the business impact of learning.

We recently conducted an observational study to measure those Level Four holy grail metrics for Skill Exchange. So how did we do it and what did we find? Let’s dive into the details!

Study methodology

Thanks to Skill Exchange and a strong data-informed culture at Spotify, we were able to coalesce learning activity data with business-critical metrics: developer productivity, attrition, and career growth. We observed these metrics for the Research and Development (R&D) job family over a three-year period after the internal launch of Skill Exchange in July 2020. As a result, we were able to find significant correlational relationships between Skill Exchange participation and our business-critical employee metrics.

Metric definitions

• Developer productivity: On a quarterly basis using our internal version of Pulse, we ask Spotify developers to self-report aspects of their developer experience. We ask them on a scale of one to five (1) how productive they felt, (2) how often they are able to achieve “flow”, and (3) how satisfied they are with their engineering velocity. We use the percentage of positive responses (a rating of four or five) on these questions to build out three different productivity metrics. We call these “Overall Productivity”, “Achieving Flow”, and “Velocity”.
• Attrition: The percentage of employees that left the company within one year from the observation period.
• Career growth: The percentage of employees that earned a promotion within one year from the observation period. Note: We measure career growth in many ways at Spotify beyond promotions. For this study however, we only looked at promotions.

Study setup

Our study included ~6,000 Spotify employees in R&D, with a total observation timeframe from July 2020 to July 2022.

We captured learning participation every quarter during this timeframe and then assessed our metrics in the 12 months following (the performance period). Our goal was to compare Spotifiers who did participate in Skill Exchange learning opportunities to those who did not. So we segmented our R&D Spotifiers into Learners (the treatment group) and Non-Learners (the control group).

We ran these groups through a Chi-Squared test and a Logit Model to find out if there was a statistically significant difference between them.

Caveats

Before we get to the results of our study, it’s important to flag our caveats. This study is observational, so the findings we present are strictly correlational. We cannot make any statements of causality between our variables and metrics. Additionally, since participation in Skill Exchange is on a signup basis, the findings are also subject to volunteer bias and reverse causality.

With that laid out, let’s dig into our results!

The holy grail data

Through observing employee metrics post-participation in Skill Exchange learning opportunities, we found that our Learners were way more likely to have favorable outcomes compared to the Non-Learners. We found that Learners were…

• Note: these comparisons are relative, not absolute differences.
• Note: for Attrition Rate, the favorable outcome is a decrease, whereas for all the other metrics the favorable outcome is an increase.

Using the same methodology as our Learner vs Non-Learner test, we were also able to drill into the specific types of opportunities available on Skill Exchange and assess their individual outcomes.

Embedding

Embedding on different teams is a unique learning opportunity that pushes developers to work in new contexts, teams, and tech. It helps them expand their knowledge of the company beyond their usual scope. We found that Embedders were 62% less likely to leave the company within one year than Non-Embedders.

Embedding can help developers solidify their relationship with the company and expand their network, lowering risk of attrition. Before Skill Exchange, Embedding was only possible if you had connections on other teams. But with Skill Exchange, Embedding culture at Spotify has blossomed, allowing any employee the chance to dive into a new, temporary experience.

Mentorships

Participating in mentorships is another powerful way for Spotifiers to grow and learn from other people in the company. We found that Mentors and Mentees were 52% more likely to earn a promotion than Non-Mentors and Mentees.

Mentorships are a common way for people to propel their careers and receive career advice. With Skill Exchange acting as an opportunity hub, Spotifiers looking to take that next step in their career can more easily find each other.

Hacking

Lots of companies host hackathons — but few are able to build a true hacking culture. Skill Exchange provides a unique hack experience by allowing developers to browse through potential projects and find something interesting to work on. We’ve found that Hackers display more positive outcomes across all our metrics compared to Non-Hackers. They were 14% less likely to leave the company within one year, 22% more likely to earn a promotion within one year, 17% more likely to report high productivity, 17% more likely to report high frequency of achieving flow, and 8% more likely to report high satisfaction with their velocity.

By working closely with others on projects outside of their normal purview, developers gain the opportunity to connect and build community, learn new skills, learn how others do their work, and enjoy Spotify’s culture.

Calculating business impact

It’s important to emphasize that due to the caveats we shared in the methodology section, the results of this study are strictly correlational. However, if we assume a 100% direct causal relationship between the informal learning opportunities (embeds, mentorship, hacking) and the metrics defined in the methodology section, we can calculate the direct business impact of the availability of these learning opportunities.

Under these assumptions, given the overall 15% reduction in attrition in the Learners group (sample size = 1,000 per year), we can estimate that Spotify saved 20 employees per year over the course of the observed two years. Replacing lost developers is no small feat. US industry reports estimate that average recruitment costs, excluding the new hire’s salary and compensation, can amount to $50,000 USD per developer. For our example, this translates to up to $1M in savings for every 20 employees staying on board, or $2M in savings across our total observation period. These estimates can vary on factors such as country of employment, level of seniority and expertise, and on the current recruitment practices of an organization.

Using similar calcuations, we can estimate that Skill Exchange also helped approximately:

• 60 employees earn a promotion per year (120 total)
• 90 employees feel more productive per year (180 total)
• 60 people achieve higher levels of flow (120 total)
• 50 people feel more satisfied with their engineering velocity (100 total)

This goes to show how much there is to gain by investing in employee growth and development through opportunities that can be offered via solutions like Skill Exchange.

Why this matters

Sure, it’s easy to keep track of how many employees are watching training videos or attending a “formal” class. But when it comes to how employees network and work together, the process of collecting that data isn’t always so clear. Being able to quantify the business impact of learning and development goes a long way in helping understand why investing in it is so important. As Spotify’s learning culture continues to grow, with it comes happy and thriving employees who are more productive and more likely to stay at the company.

Skill Exchange is the foundation of Spotify’s learning culture. Not only does it help accelerate business-critical employee metrics, but it also empowers us with the data we need to measure those outcomes. Skill Exchange’s backend database, which tracks employee engagement with these opportunities, helps fill in the gap that many companies are missing - data on the impact of informal learning. How employees develop their skills and careers is complicated, but Skill Exchange helps make the learning journey less murky.

Learn more or find out how it can help your organization.