peterpascale

Our Cup Runneth Over (But Not in a Good Way)

In the software release process we're continually looking for ways to improve our ability to deliver. From process to craftsman techniques there are a large number of levers to apply to this problem. These levers need to shift the Volume to verify and/or the Rate at which verification occurs. If they don't, they aren't contributing to the solution. 

The QA Verification process is governed by the laws of physics, and as such, it can be described mathematically. Each release contains a volume of functionality (V). The developer and tester's ability to verify occurs at some rate (r) - for the purpose of discussion think of it as 'how much software can be verified in a day'. A given release has a capacity (C) – the volume we can support – which is simply the rate times the number of days in a release cycle (D):

C = r * D

As such, a release volume can be verified if it is equal to or less than our capacity:

V <= r * D

If the volume of software released is greater than the capacity to verify it, quality and ability to deliver new features is at risk. Now if all this math stuff is too much, consider this visual metaphor:

This is a common experience in our industry. This unverified software pouring over the funnel is the risk of regressions and new defects making their way to production. While the quality implication is clear – consider also that critical late-found defects also have the ability to delay or cancel a release, and that even if we tolerate some quality issues (which we shouldn’t) the funnel is only so large. 

The math and the visual metaphor highlight the two factors that should receive our improvement energy – Volume (the size that’s being poured), and Rate (the size of the funnel).

[What about Days? - Volume is being produced at a somewhat regular rate. Adding days simply adds Volume. It doesn’t improve the situation and in fact makes it worse as the business waits longer between releases.]

Volume

Volume can be misleading at first – I’m not advocating slower release of new features. I’m advocating better focus on regression exposure and improving our ability to reason about what has changed.

Large monolithic codebases make it extremely difficult to reason about what has changed (and therefore what needs to be regression tested) in a given release. If one line of code in this monolith has changed, the whole thing has changed and must be built, deployed and verified.

A large portion of the volume isn't exposed to regression, but reasoning about what part(s) changed is impossible. We need the apparent volume we introduce in the release cycle to be closer to the actual volume we need to verify.

Rate

Rate in our industry has largely been a function of manual testing and headcount. Manual testing does not impact efficiency, and we’ve outstripped the ability of manual testing to keep up. Rate improvements must focus on improving the efficiency of the Dev-Release - QA process. Things like automation and practices that promote early detection of defects (CI, unit tests, automated functional tests, etc).  

In as much as a practice, tool, investment, or automation reduces Volume to actual Volume or improves Rate efficiency, it improves delivery efficiency. Options that don’t impact Volume or Rate in this way don’t help us and are a waste of time. We need to be asking ourselves how our efforts improve our collective ability to reason about change (Volume) and make verification more efficient (Rate) and focus our investment on these things.