Do the Tests Actually Test?

Your suite is green, so you feel safe. But a test can pass while checking nothing at all, and the agent that wrote your code also wrote your tests, with the same blind spots baked into both. A green suite you cannot trust is worse than no suite, because it tells you to stop looking. This chapter gets you a way to tell a real safety net from a green light that means nothing.

8.7.1A passing test can check nothingAdd note

A test passes when nothing in it fails. That sounds obvious until you see the trap: a test that checks nothing also has nothing to fail, so it passes forever.

Green is only as good as the check inside the test. A test that runs your code but never asserts the result is a green light wired to nothing. It will stay green while the feature underneath rots.

8.7.2Coverage counts lines, not correctnessAdd note

Coverage is the percent of your code that your tests actually run. Tools like Vitest coverage report it, and it is genuinely useful for one thing: finding code no test touches at all.

But high coverage is not proof of correctness. Running a line is not the same as checking what it produced. This test hits every line of applyDiscount and asserts nothing that matters:

That is 100 percent coverage on a test that would stay green if applyDiscount returned garbage.

8.7.3Watch for tests that can't failAdd note

A worthless test is one that cannot turn red no matter how broken the code gets. Three smells give them away:

1. No real assertion: it runs your code but never checks the output, or only asserts something trivially true like expect(true).toBe(true).
2. Over-mocked: so much of the real code is faked that the test only checks the mock you wrote, not your actual logic.
3. Mirror test: it restates the implementation instead of the expected result, so it agrees with any bug the code already has.

8.7.4Test an AI feature on shape, not on wordsAdd note

The AI features you built earlier fail this chapter from both ends at once. Assert the exact sentence a model returned and the test is flaky by design; assert nothing and it is a green light wired to nothing.

So assert on shape and constraints instead: valid JSON, a category that is one of the values you allow, a score between zero and one, an answer that cites a document which exists. In your unit and end-to-end tests, mock the model call the way you mock any paid outside service, or your suite is slow, expensive, and different every run.

The eval set you built for that feature is a different animal, and it does not belong in the suite that gates your merges. Run it on a schedule and read the results, because a real model call costs money and can fail for reasons that have nothing to do with your change.

8.7.5Break the code on purposeAdd note

The fastest way to trust a test is to make it fail on purpose. Change the code it guards to something wrong, rerun, and watch for red.

If the test turns red, it was really checking that behavior. If everything stays green after you broke the code, those tests were never testing anything.

You do not do this by hand. Hand the check to your agent, which can break each piece, rerun, and report which tests actually caught it:

Do this now: paste the prompt, then pick your most important test and break the code under it by hand to confirm it turns red.