Part 3 of a three: On metrics, instrumentation, and the discipline of reading patterns instead of chasing numbers.

If you’re rethinking how engineering teams operate in an AI-accelerated world (smaller teams, real-time inception, plan review replacing standup, paired code review, AI as collaborator inside human accountability) then the metrics need to evolve with the operating model. Otherwise you’ll either claim victory while shipping confidently wrong code at machine speed, or conclude the model isn’t working when it actually is.

This piece is about how to know. Which metrics still tell the truth, which ones lie under AI, what to add, and the principle that holds the whole framework together.

Start with the principle, not the metrics

No single metric tells the story. The pattern across multiple metrics tells the story. When one metric strays from the rest, that’s a question to investigate, not a verdict to react to.

This is the most important rule in the entire framework, and it’s the one most engineering organizations get wrong. Velocity gets treated as the productivity number. Deployment frequency becomes the modernization number. NPS becomes the quality number. Each of these, in isolation, can be hit while the underlying system is degrading and the more weight a single metric carries, the more incentive there is to optimize for the metric rather than the thing it was supposed to measure. The whole field knows this and the whole field keeps doing it anyway, because executives want a single number to point at and engineering leaders cave to the request.

The discipline that breaks the cycle is reading the pattern. Velocity, outcome hit rate, change failure rate, and token cost per outcome all have to move together to claim improvement. Gaming any one of them gets exposed by the others. A team that inflates velocity by shipping smaller items won’t see outcome hit rate improve. A team that softens success criteria will see token cost per outcome rise without value showing up. The metrics check each other. That’s the structural defense against the single-metric distortion that has plagued engineering organizations for decades.

It also shifts the role of metrics from judgment to diagnosis. A divergent metric doesn’t mean the team is failing, it means there’s something to investigate. Maybe the metric is wrong for that team’s context. Maybe the team is going through a transition. Maybe the underlying system has a real problem the other metrics haven’t surfaced yet. The leader’s job isn’t to read the metric and react. It’s to read the divergence and ask the right question.

Keep DORA, but use it differently

DORA isn’t just a metrics framework, it’s a legitimacy framework. It has years of research behind it, a published methodology, and broad adoption across the industry. When you say you’re tracking DORA, you’re speaking a language every senior engineering leader and most CTOs already understand. Walking it back entirely would cost credibility and make the new model harder to evaluate.

The four classic metrics survive, but two of them need to be used differently in the AI-accelerated world.

Deployment frequency becomes delivery of value

Deployment frequency counts events. Value delivery counts outcomes. The two diverge sharply when AI is involved, because deployment events become cheap and abundant while validated value remains scarce. A team can deploy fifty times in a month and deliver three actual pieces of value. The old metric would call that excellent. The new metric would call it noise.

Track this two ways. Frequency of value delivery answers how often the team shipped work that achieved its stated outcome. Average value per delivery answers how much each delivery moved the needle. Frequency alone can be gamed by shipping small things. Average alone can be gamed by shipping rarely. Together they tell you whether the team is delivering meaningfully and at meaningful weight.

Lead time for changes gets layered

DORA’s commit-to-production lead time stays as a CI/CD health metric. But layer a longer measurement on top: idea-to-validated-outcome cycle time. From when a problem was raised at inception to when the deployed work was confirmed to have moved the needle. Both are observable. Both are accountable. Together they tell you whether the pipeline is healthy and whether the operating model is delivering.

Change failure rate and MTTR survive intact

Change failure rate may be the single most important guard rail in the entire framework. If AI is generating significant code volume, change failure rate is the direct measurement of whether your accountability mechanisms are working. If failure rate stays stable or improves while you’re shipping more through AI pairing, the paired review is doing its job. If it spikes, your senior judgment layer is missing things which is exactly the failure mode the new operating model is designed to prevent.

MTTR also survives intact, and connects to the architectural prerequisites that make this whole model possible. The new SDLC, properly implemented, should produce systems that are easier to recover because they’re better instrumented, more clearly bounded, and more honestly owned. If MTTR doesn’t improve over time, that’s a signal the architectural work hasn’t kept up with the delivery model. The team is running the new operating model on top of an old, tangled system, and the speed gains are masking accumulating fragility.

From the DORA+ extensions, keep what measures quality. Drop what measures activity.

The expanded DORA+ set has been getting cluttered over the years. The principled way to curate it is to keep what measures quality, detection, and learning. Discard what measures volume of activity because AI makes activity metrics actively dangerous, since they can be inflated without any real work being done.

Keep

Mean time to detect, rework rate, and defect escape rate stay as quality and detection signals. These are observable from system behavior, not from human reporting, which makes them hard to game.

Test coverage stays, and gets a higher bar. AI is going to write a significant portion of the tests in the new model, that’s actually one of the better uses of AI. Tracking a minimum threshold becomes important precisely because AI makes it cheap to hit. There’s no longer an excuse for low coverage. But coverage alone has always been a weak metric: AI is good at generating tests that look comprehensive but actually just exercise the happy path. So pair coverage with test effectiveness: mutation testing or similar techniques that validate the tests are catching real problems. Coverage as the volume floor, effectiveness as the truth check.

Code review depth stays, with a reframe. The traditional metric measured review speed, how fast PRs get merged. The new model needs reviews that are slower but more substantive: catching architectural and domain issues, not just style and syntax. The metric category survives. What good looks like inverts.

Modify

Velocity is the most contested decision, and it’s worth being precise about why it stays. The traditional use of velocity is predictive, it tells the business how much work the team can complete in a future sprint. That use is broken in the new model and not worth recovering. AI introduces too much variance into per-item effort, the smaller team has different throughput characteristics, and the cadence isn’t sprint-based anymore.

But there’s a different use of velocity that survives: long-term trending. Not “how many points will we complete next sprint” but “is the team’s overall throughput moving in the right direction over quarters.” Even if the unit of measurement is fuzzy, the trend tells you whether the operating model changes are paying off. Track it per quarter or per month, not per sprint. And always pair it with delivery of value. If velocity is rising but outcome hit rate is falling, the team is getting fast at shipping the wrong things, which is the exact failure mode the new operating model is supposed to prevent.

Estimation accuracy gets elevated from a soft metric to a first-class diagnostic instrument. In the traditional sprint model, the gap between estimate and reality was treated as normal noise. In the new model, that gap is information. If estimates are consistently too low for boundary-crossing work, your bounded contexts are leakier than the team thinks. If estimates are consistently too high for greenfield features, AI pairing is paying off and the team hasn’t internalized the new baseline. Track it at the team and work-type level, never at the individual level. Pulling it into individual performance reviews destroys the diagnostic value, because engineers will inflate estimates to protect themselves.

Discard

Sprint commitment accuracy, sprint goal achievement rate, and burndown rate go away. They measure throughput of estimated work in fixed time boxes, which is the exact frame the new operating model is moving away from. Keeping them creates pressure to keep working in sprints, which keeps the team measuring the wrong thing.

Lines of code, commits per day, pull requests per engineer, and code churn also go away. Anything that measures volume of output rather than quality or outcome. AI makes these metrics actively dangerous because they can be inflated without any real work being done.

Add what the old framework couldn’t see

DORA evolved gives you the foundation. Layered on top are four new measurements that the old framework couldn’t produce, because the old SDLC didn’t generate them.

Outcome hit rate

The percentage of approved plans that achieved their stated outcome. This is the most important metric in the whole framework. Every plan review establishes success criteria: what business or product metric this work is intended to move, what threshold counts as having worked. The close-out loop with stakeholders measures whether it actually did.

A team with 70% outcome hit rate is genuinely outperforming a team with 95% velocity attainment. Most engineering organizations don’t track this today because it’s harder than counting commits. The new SDLC makes it measurable for the first time, because the structure forces success criteria to be defined at inception.

This is also the most gameable metric in the framework, and it’s worth being honest about that. The attack is in how success criteria get defined. If the team or leader sets soft, vague, or already-likely criteria, hit rate looks great while nothing is actually moving. “Improve user experience” hits 100%. “Lift checkout conversion by 3%” hits when it hits. The defense is plan review run by engineering leaders who see the criteria across many plans and recognize the pattern of soft framing. The metric requires cultural integrity to keep honest, which means it’s also a leadership integrity check.

Phase duration distribution

The new SDLC is gated. Each work item moves through inception, plan review, execution, paired review, deployment, and outcome validation. Capture timestamps at each gate and you get something the old framework couldn’t produce: where in the lifecycle is the time actually going.

If inception is long, the room isn’t converging fast enough, or you don’t have the right people in it, or vibe coding isn’t being used to compress abstract debate. If plan review is queued, you don’t have enough review capacity. If execution is long, your AI workflow isn’t producing the gain it should. The diagnostic question is whether the agent has the right tooling, the right context, or the right guardrails. If outcome validation is long, your stakeholder loop is broken or your observability isn’t surfacing the signal fast enough.

Phase-level instrumentation also tells you something important about your tooling. Jira, VersionOne, and the other major project management platforms aren’t the constraint, they’re flexible enough to capture whatever lifecycle you want them to capture. They always have been. What they currently capture in most organizations is sprint throughput, because that’s what the prevailing methodology asked them to capture. When you change the operating model, the configuration changes with it. You can tell whether an organization has actually adopted a new SDLC by looking at what their project management tool is capturing. If they’ve added “AI” labels to their existing story workflow, they haven’t changed anything. If they’ve reconfigured the workflow to capture inception, plan review, paired code review, and outcome validation as first-class states with timestamps and artifacts, they’ve actually done the work.

Active investment, cost to completion, and realized loss

Most engineering organizations don’t measure abandonment, the work that gets started and dropped without delivering value. It disappears into “we deprioritized it” or “we pivoted.” That hidden flow is enormous, and AI compounds it because the cost of starting work has dropped, which makes it easier to start things and abandon them.

Treat engineering work as a portfolio rather than a stream. Active investment is the total cost, engineer time plus token cost, of work currently in flight, before completion. That’s your exposure. Cost to completion is the estimated remaining cost to bring active work across the line. Together with active investment, it tells you the total expected cost of the current portfolio. At-risk investment is what’s already been spent on work that’s struggling, where the team is uncertain whether to continue. Realized loss is the cost of work that was started and stopped without delivering value, the actual dollar figure, quarter over quarter.

This framing speaks the language executives and finance already use. Portfolios have positions, exposure, and realized losses. They get reviewed and rebalanced. Decisions about cutting losses are normal portfolio management, not failures. That’s the right mental model for engineering leadership in the AI era, capacity is finite, cost is now visible per item, and the leader’s job is to manage the portfolio rather than just push throughput.

Token cost per outcome

Most organizations adopting AI tooling are flying blind on cost. They have a corporate vendor bill that arrives monthly, and they have engineering productivity claims, and there’s no connective tissue between the two. They can’t tell you what a feature cost in tokens. They can’t tell you which teams are consuming the most. They can’t tell you whether the cost per delivered outcome is going up or down. They’re paying the bill without understanding it.

Track token cost at the work-item level, attributed to the lifecycle phase it served. Total tokens for a piece of work that successfully delivered its stated value gives you cost per outcome, the metric that lets you reason about ROI. Over time, you should expect this number to fall if the operating model is healthy: better prompts, better tooling, better engineer judgment about when to use AI heavily versus lightly, better domain clarity that reduces wasted tokens on confused exploration. If it’s flat or rising, something is broken.

Track token cost at the work-item level, never at the individual engineer level. A senior engineer doing high-leverage architectural thinking with AI may consume significantly more tokens than a junior engineer doing well-scoped implementation work, because architectural exploration is more open-ended. Using token consumption as an individual performance metric will distort behavior the same way velocity per engineer did. Track at the work item and team level. That keeps the metric diagnostic instead of weaponized.

The deeper structural point is about the cost model itself. Traditional cost-of-engineering math is salary plus overhead, divided across delivered features. That math is becoming insufficient because a meaningful chunk of cost is now variable token consumption. An engineer’s salary is predictable. Their token usage isn’t. So cost-per-feature now has two components: human cost (fixed and well-understood) and AI cost (variable and previously invisible). The companies running into runaway AI bills are the ones who never built unit economics for AI. They knew their engineering headcount cost. They didn’t know their per-feature AI cost. Without that visibility, they couldn’t tell when the trade was becoming unprofitable.

Reading the pattern

With this framework in place, the work shifts from collecting metrics to reading them. A few principles for doing that well.

Metric movements are read as patterns, not as individual events. A single metric moving 10% in either direction is rarely meaningful in isolation. The same metric moving 10% while three others move with it tells a story.

Divergent metrics get investigated, not reacted to. The investigation might confirm the metric is right and action is needed. It might reveal the metric is being affected by something outside the operating model and no action is needed. It might surface that the metric definition itself needs to evolve. All three are legitimate outcomes. The investigation is the work.

Consider what it looks like when outcome hit rate drops while every other metric stays stable or improves. Velocity is up, deployment frequency is steady, change failure rate is fine, token cost per outcome is reasonable. The single-metric reaction would be “outcome hit rate is dropping, we need to focus on outcomes.” That reaction is wrong. The pattern says the team is shipping efficiently, with quality, at reasonable cost but the work isn’t moving the needle. That’s not an engineering execution problem. It’s an inception problem. The team is being pointed at the wrong work. The fix isn’t in execution, it’s in product alignment, stakeholder engagement, or how problems are being raised at the front of the funnel.

The team itself should be reading the metrics. This isn’t a leadership-only instrument. The engineers running plan reviews, the leads running code reviews, the engineers doing pair-planning all benefit from seeing the patterns and asking the same questions the leader is asking. Metrics that only the leader sees become political instruments. Metrics that the team sees become operational ones.

Metrics evolve. The set you start with is the best framework you have today. As you operate and learn, some metrics will prove more diagnostic than expected, others will prove less useful, and new ones will emerge from patterns the team notices. The discipline isn’t to commit to a fixed set forever, it’s to commit to the practice of reading patterns rigorously, regardless of which specific metrics you’re reading.

The argument is falsifiable

The point of this framework isn’t to claim the new operating model works. It’s to give you the instruments to find out. These metrics are observable. They’re accountable. They can be checked against the numbers. If the new SDLC isn’t actually working, these metrics will say so.

That’s the discipline that separates a serious operating model proposal from a thought-leadership pitch. You don’t have to trust the model on faith. You instrument it, you read the patterns, and you let the numbers speak.

The biggest risk to all of this isn’t the team gaming the metrics. A determined team can game any individual metric in any framework. But gaming the pattern requires real work, and if a team has done real work, they’ve also moved the underlying outcomes. The structural defense holds.

The biggest risk is the leader gaming them. A leader invested in showing the operating model is working can soften criteria, reframe abandonment, and pressure-test results to look favorable. That failure mode is structural. it’s the paper leader pattern applied to the metrics themselves. The defense is the same defense that runs through everything: leaders who’ve earned their judgment through experience, surrounded by people who can tell them when the numbers are lying. No metric framework can substitute for that.

Build the instruments. Read the patterns. Let the truth surface. That’s the work.