Flows vs Tests: Per-Test Billing Broke QA | by Keith Haag | May, 2026 | MediumSitemapOpen in appSign up<br>Sign in

Medium Logo

Get app<br>Write

Search

Sign up<br>Sign in

Flows vs Tests: Per-Test Billing Broke QA

Per-test billing rewards volume, not value. Here’s the structural fix that realigns incentives.

Keith Haag

4 min read·<br>Just now

Listen

Share

Press enter or click to view image in full size

Image by AuthorYou pay your QA vendor for 100 tests per month. Next month, you have 120 tests. The month after, 145.<br>Here’s what nobody is asking: do the tests actually prove your application works?<br>Probably not. And the reason isn’t incompetence — it’s that the pricing model itself rewards the wrong behavior.

The Per-Test Trap<br>Suppose your checkout workflow has four steps: browse products, add to cart, enter payment details, confirm the order.<br>A thoughtful engineer writes one test: navigate to the product page, click through the full flow, assert the confirmation page renders with the correct order number. One script. Complete coverage of the most important thing your application does.<br>But the vendor bills per test. And somehow, “checkout coverage” from that vendor arrives looking like this:<br>test_01: verify the product page loads<br>test_02: verify the “Add to Cart” button exists<br>test_03: verify the cart count increments<br>test_04: verify the cart renders the correct product name<br>test_05: verify the cart renders the correct price<br>test_06: verify the checkout page loads from the cart<br>test_07: verify the address form renders<br>test_08: verify the address form validates required fields<br>test_09: verify shipping options appear<br>test_10: verify the credit card form renders<br>test_11: verify the “Place Order” button is clickable<br>test_12: verify the success page renders<br>Twelve tests. Billed twelve times. Twelve green checkmarks every CI run.<br>But do you know whether checkout actually works?<br>You don’t. Because not a single one of those tests proves that a customer can complete a purchase. They all test rendering states. The only test that proves checkout works is one that actually performs checkout from start to finish — and that’s not what per-test pricing incentivizes anyone to write.<br>This isn’t malice. The vendor isn’t trying to scam you. They’re responding rationally to the incentive structure you gave them. If you pay per tomato seed, you’ll get a lot of seeds and no tomatoes.<br>The Invoice Is the Smallest Problem<br>The billing problem is irritating but manageable. You can negotiate. You can audit. You can cap test counts.<br>The real damage is what you’re left maintaining.<br>Twelve separate test files. Twelve points of failure for every UI change. When your designer tweaks the checkout layout, how many of those tests break? Probably most of them. You’ve multiplied your maintenance surface by twelve to achieve worse coverage than one well-written flow.<br>And those twelve tests aren’t just a maintenance burden — they’re twelve browser context initializations, twelve teardowns, and twelve times the CI minutes. You’re paying in dollars, compute, and developer patience. The invoice is the smallest line item on the receipt.<br>CI run times inflate. Developers start ignoring failures because “it’s probably just the button test again.” The test suite that was supposed to build confidence has trained your team that test failures are noise.<br>Per-test pricing doesn’t just produce bloated suites — it devalues the entire concept of automated testing in your organization.<br>The Structural Fix: Pay for Journeys, Not Clicks<br>There is an alternative, and it’s simpler than you’d think.<br>One billable unit = one user journey from entry to measurable outcome. A checkout flow that covers product selection through order confirmation is one unit. The vendor can’t atomize it into twelve tests because twelve fragments don’t add up to the journey. The pricing model makes fragmentation impossible.<br>Incentives realign completely.<br>When you charge per journey, the vendor’s goal is to cover each journey as efficiently as possible — one script, complete coverage, minimal maintenance footprint. Writing twelve rendering checks for checkout doesn’t move the needle on revenue. The only rational move is to write the one test that actually matters.<br>This isn’t about trusting your vendor to “do the right thing.” It’s about designing the contract so that doing the right thing is the easiest way to make money.<br>Why Nobody Talks About This<br>Per-test pricing has been the industry default for decades. It’s how tools were priced. It’s how agencies billed. Nobody stopped to ask if it made sense — it was just how QA billing worked.<br>But per-test pricing exists because it’s easy to measure, not because it’s the right unit of value. Nobody wakes up in the morning hoping their test count went up. They wake up hoping their checkout flow didn’t break. The unit of measurement should match the unit of value.<br>The Three-Question Audit<br>You can evaluate whether your QA vendor’s pricing is aligned with...