How we measure Backstage proof of value at Expedia Group

Sneha Kumar, Expedia Group
Proof-of-value metrics used at Expedia Group: ship on day one, reduced context switching, search result relevance, and technical documentation reach

Expedia Group’s (EG) vision is to power global travel for everyone, everywhere. In the quest to do so, we are focusing on simplification and enhancing foundational platform capabilities to empower our developers, suppliers, partners, and travelers.

EG Developer Experience proudly adopted Backstage as our developer front door in August 2020 after a successful proof of concept to evaluate search, technical documentation, and plugin contribution capabilities. An amazing partnership blossomed between Spotify, EG, and the broader adopter community as we evolved Backstage for our internal implementation and contributions to Backstage open source.

An important question emerged as Backstage continues to gain momentum with our 6K+ developer community at EG — how to show the value of Backstage to them in a meaningful way?

Let’s first examine the EG internal developer journey

The EG Developer Experience Team is part of a broader organization called Development Platform & Infrastructure that was chartered to simplify the developer journey.

We kicked off a journey map workshop to refine our developer journey map representing the next 6–12 month horizon. We focused on getting these three concepts right:

  • Align our journey map phases to the DevOps process and software development lifecycle
  • Identify developer jobs to be done by phase
  • Identify metrics by phase
Expedia Group Internal Developer Journey Phases, from top to bottom: onboard, build, validate & test, deploy & release, run monitor & run; developer jobs to be done; journey metrics by phase (inputs & outputs); Backstage proof of value metrics

Developer jobs to be done by phase

Our team identified developer jobs to be done (JTBD) by phase to define critical workflows. These jobs to be done follow a user story format. An example JTBD looks like “When onboarding as a developer, I want intuitive search so that I can discover EG Platform content quickly.”

This JTBD exercise enables a common steel thread exercise, which means testing developer JTBD on the critical path with an aim to reduce friction. Our goal is to enable as many jobs to be done as possible across the journey within Backstage.

Journey metrics by phase

We then envisioned a set of overall metrics by journey phase. Example metrics include increasing the percentage of people shipping on day one and reducing the lead time for change during the build, validate, and test phases.

And now where things get exciting — measuring Backstage proof of value

Proof of value fundamentally means answering the “why” for Backstage in a way that resonates with our customers. These metrics are a subset of our overall journey metrics. Our working hypothesis is that increased metrics transparency will be valuable for existing customers and increase willingness for new customers to adopt Backstage and contribute plugins/template actions.

We have oriented our Backstage product roadmap in Q4 (and beyond) to enabling value creation for our customers and simplifying the developer journey by aligning to these metrics.

The proof of value metrics we are starting with:

MetricDescriptionHigh Level Work Planned in Q4
Ship on day oneTime from first commit to deploying app code in productionAdd curriculum content to developer bootcamp for new hires to leverage Backstage to create a new “hello world” app in production on day one

Partner with EG IT to automate access to common developer-facing tools when developers receive laptop in hand
Reduced context switching by increasing number of integrated toolsMeasure number of developer-facing tools integrated with Backstage to reduce context switching for developers since they do not need to toggle between multiple tools for developer jobs to be doneContinue plans to integrate with internal developer-facing tools with Backstage in Q4 (e.g., Runtime Compute Platform)
Search result relevanceRanking of relevant content developer is searching for out of all search results

Monitored by clickthroughs of search results
Refine proof of concept on search modal to easily view quick results across all content types in Backstage and make contribution back to OS community

Enhance user analytics instrumentation for search
Technical documentation reachNumber of developers who have viewed a documentN/A – initial reporting provided by current user analytics instrumentation

We are at the initial phase of establishing a baseline for our Backstage proof-of-value metrics. We look forward to sharing an update with the adopter community in the coming months! In the meantime, if you missed it, you can read about the metrics that Spotify uses to measure Backstage success.