Exploring DORA, SPACE and Core 4 – Without the Jargon
What do they measure? Who is affected? Why are they important?
I’ve noticed that software engineering productivity frameworks contain a lot of technical jargon - “change failure rate”, “change lead time”, “regrettable attrition”, etc. And I asked myself, how can we make them more accessible?
So I’ve spent some time translating DORA, SPACE and Core 4 into “plain language”. You might find this useful if you want to share the approaches and their benefits with non-technical stakeholders.
I also wanted to understand how each metric relates to my productivity model. Seeing them in context helped me to understand their impact and reflect on areas that are well covered or could be developed in future.
My Team Productivity Model
I find it useful to separate team productivity into four main areas with each of them driving business impact:
People Interactions: The foundation of our productivity is based on the quality of our interactions. If we work well together, we are more likely to create valuable solutions.
Task Execution: The efficiency of our task execution will affect how quickly we can do our work. The less distraction and delay, the more productive we are.
Output Quantity: The quantity of outputs is an indicator of how much stuff we can produce. If we struggle to deliver things, we are more likely to be unproductive.
Customer Impact: The impact we have on our customer is a very clear measure of our productivity. When we create value for them, we are most likely creating value for our business.
l have referenced these model areas below (in square brackets).
Read this story if you want to learn more about my model:
DORA In Plain Language
The DORA four keys are labelled as “software delivery metrics” for measuring performance. It doesn’t mention “productivity” being a focus, but many engineering teams consider it in that context.
“Technology-driven teams need ways to measure performance so that they can assess how they’re doing today, prioritize improvements, and validate their progress. DORA has identified four software delivery metrics—the four keys—that provide an effective way of measuring the outcomes of the software delivery process. DORA’s research shows that these performance metrics predict better organizational performance and well-being for team members.” [Source]
Throughput
Change Lead Time
How long does completed work take to reach the customer? [Customer Impact]
Deployment Frequency
How often do customers receive new value? [Customer Impact]
Stability
Change Fail Percentage
How often is new customer value unusable? [Customer Impact]
Failed Deployment Recovery Time
How long is new customer value unusable? [Customer Impact]
SPACE In Plain Language
SPACE does not define metrics, but suggest potential metrics. Below I’ve only included a selection of them.
“Productivity is about more than the individual or the engineering systems; it cannot be measured by a single metric or activity data alone; and it isn't something that only managers care about. The SPACE framework was developed to capture different dimensions of productivity because without it, the myths just presented will persist. The framework provides a way to think rationally about productivity in a much bigger space and to choose metrics carefully in a way that reveals not only what those metrics mean, but also what their limitations are if used alone or in the wrong context.” [Source]
Satisfaction and Wellbeing
Employee Satisfaction
How engaged are team members? [People Interactions]
Developer Efficacy
How successful are teams delivering customer value? [Customer Impact]
Burnout
How healthy is the working environment? [People Interactions]
Performance
Customer Impact
What impact does a team have on its customer? [Customer Impact]
Quality
How good is the quality of a solution? [Customer Impact]
Reliability
How reliable is the service provision? [Customer Impact]
Activity
Design and Coding Quantity
How much code and documentation do teams create? [Output Quantity]
CI/CD Utilisation
How well is delivery automated? [Task Execution]
Operational Activity (Incidents/Issues/Mitigation)
How well does the team deal with incidents? [People Interactions]
Communication and Collaboration
Discoverability of Documentation and Expertise
How easy is it for team members to find and get what they need? [Task Execution]
Speed of Work Integration
How quickly does work get accepted? [People Interactions]
Quality of Code Review
How well do teams support each other with coding? [People Interactions]
Level of Connection
How well are team members connected? [People Interactions]
Team Member Onboarding Time
How long does it take for new team members to become effective? [People Interactions]
Efficiency and Flow
Complexity of Processes
How complex are team processes to navigate? [Task Execution]
Perceived Level of Flow
How much do team members experience flow state? [Task Execution]
Level of Interruption
How do interruptions affect team members? [Task Execution]
Quality of Time Spent Across the System
How much time do team members spent navigating the system? [Task Execution]
Core 4 In Plain Language
Core 4 is an attempt to accurately measure software engineering productivity by unifying DORA metrics with DevEx and SPACE insights.
“To help simplify the landscape, we’ve developed a unified approach to measuring developer productivity called the DX Core 4. The DX Core 4 encapsulates DORA, SPACE, and DevEx and includes four dimensions: speed, effectiveness, quality, and business impact. The DX Core 4 provides a focused set of metrics that work effectively at any sized organization, and can be augmented with additional metrics for specific goals.” [Source]
Speed
PR Throughput
How frequently do teams make changes to the code? [Output Quantity]
Lead Time
How long does completed work take to reach the customer? [Customer Impact]
Deployment Frequency
How often do customers receive new value? [Customer Impact]
Perceived Rate Of Delivery
How do teams rate their speed to deliver for the customer? [Task Execution]
Effectiveness
Developer Experience Index (DXI)
How do teams rate the work environment? [People Interactions]
Time to 10th PR
How long does it take a new team member to regularly deliver code changes? [People Interactions]
Ease of Delivery
How do teams rate the tech environment? [Task Execution]
Regrettable Attrition
How often do team members leave the business? [People Interactions]
Quality
Change Failure Rate
How often is new customer value unusable? [Customer Impact]
Failed Deployment Recovery Time
How long is new customer value unusable? [Customer Impact]
Perceived Software Quality
How do teams rate the quality of their solutions? [Customer Impact]
Operational Health And Security Metrics
How healthy and secure is the delivered solution? [Customer Impact]
Impact
Percentage of Time Spent On New Capabilities
How much time do teams spent on creating new customer value? [Customer Impact]
Initiative Progress and ROI
What is the return of investment for software development projects? [Business Impact]
Revenue Per Engineer
How much revenue does the business generate for each engineer? [Business Impact]
Research and Development as Percentage of Revenue
What percentage of revenue is re-invested into research and development? [Business Impact]
What Are The Insights?
Customer Impact
I was surprised about how many of these metrics are optimising for customer experience. I hadn’t expected this - the metric descriptions all sound very technical.
An example is DORA’s and Core 4’s “Change Lead Time”: It doesn’t mention the customer at all, but we can reframe it as “How long does completed work take to reach the customer?”.
I am finding that many software engineering teams don’t directly connect their work to a customer. As a result they may end up working in a tech bubble, not maximising the value they can create.
I’m a big believer that that customer value trumps most other metrics. Without it, there would be no business revenue and therefore no money to pay for an engineering team. A notable exception are team health metrics which are critical for guarding against burnout.
Output Quantity
I noticed that there are only a few metrics that measure “how much stuff” teams create.
I very much welcome this and imagine this is due to the realisation that the amount doesn’t really matter, but its the value a solution creates.
I’m concerned that many teams are still looking exclusively at how many story points or features they might deliver. The range of metrics above highlights that other areas are significant and very important.
Task Execution
It’s clear that the speed at which teams can turn their ideas into reality is important.
A team’s work environment has to be set up in a way that it nurtures engineering creativity and reduces bottlenecks as much as possible.
I’m finding that many teams focus almost all their productivity improvement efforts on optimising task execution. This is usually an easy sell and there are proven ways to achieve it.
But it’s clear from the above that there are other important areas that need to be taken care off as well, notably customer impact.
People Interactions
There is a good number of metrics that address this area, but I feel like this hasn’t been fully developed yet.
This is surprising as there is very significant evidence about how team norms drive team effectiveness. You can read more about it here:
SPACE is the exception here as it has a strong focus on “Communication and Collaboration” and particularly the “Level of Connection” metric stands out.
My hunch is that measuring quality of interactions and connectedness between people is a relatively new area and it’s not (yet) clear how to best measure it.
Business Impact
Core 4 has introduced metrics which directly reference business impact, for example the amount of revenue generated per engineer. These are useful for understanding how well the investment into technology affects the finances of the business.
I’m usually a fan of looking at financials, but I wonder if there are too many variables at play here that can affect the outcome.
I believe we need to ensure that team productivity metrics are measuring what’s in the circles of control and influence of software engineering teams. Only those metrics can lead to learning and improvement.
Closing Thoughts
There has been significant progress in developing productivity frameworks and relevant metrics over the last years. Core 4 does a good job of covering a wide range of areas that should help with finding a good balance.
What’s missing for me is more clarity around measuring the quality of interactions within a team.
Its my experience that tells me that the level of connection between team members directly affects their productivity. Once this is paired with a clear understanding of how to best impact the customer, productivity naturally emerges.