Portfolio → negative-control

Systems & tools · Testing toolnegative-control

negative-control

A green test is not evidence until you have seen it turn red. This tool breaks your code on purpose, runs the test that should catch it, and puts the code back. If the test turns red, it was guarding something. If it stays green, it wasn't, and now you know.

Method schematic. The working file is restored from a copy before the verdict is reported.

Swipe sideways to see the whole diagram.

Method schematic. The working file is restored from a copy before the verdict is reported.

Challenge

A passing test may never reach the behavior it claims to protect. A failed test run can also be mistaken for a meaningful result.

My contribution

I defined the method, verdicts, and safety rules: distinguish a missed bug from a test that never ran, and restore the working file from a copy. AI coding agents implemented the tool.

Evidence

Follow the weak-test example, the three verdicts, and the real failures behind the safety rules.

Related capabilities Verification & quality judgment Requirements & systems design

3verdicts, not two: caught · no teeth · did not run

6real failures of earlier versions, each now a rule

12ways the tool breaks itself, all caught by its own tests · 2026-09-29

30 → 0false greens this method found in AICC, then cleared · Aug 2026

Source on GitHub

Why green isn't evidence

A test can pass for the wrong reason

tests/test_pricing.py

# passes today. Also passes with the fee set to 0,
# because 0 >= 0 is true. It never checked the fee.
def test_small_order_has_a_fee():
    assert shipping_fee(50) >= 0

# fixed: now breaking the fee turns it red
def test_small_order_has_a_fee():
    assert shipping_fee(50) == 8.0

A test can check the wrong value, check something that is always true, or never reach the code it is named after. All of these look exactly like a test that works: green.

The only way to tell them apart is to break the code the test claims to guard, and watch. People do this by hand: edit a line, run the test, edit it back. The last step is where it goes wrong.

How it works

You write the break. The tool does the rest, carefully.

write a recipe→break one line→run the named test→restore from a copy→verdict

{ "label": "small orders ship free by mistake", "file": "pricing.py", "replace": " return SHIPPING_FEE", "with": " return 0.0", "test": "tests/test_pricing.py::test_small_order_has_a_fee" }

Break the decision, not the wording. Change which message is chosen, not the text of the message
Delete, don't comment out. A commented-out copy still matches a test that searches the source
Point at one test. A green verdict then points straight at the weak one
Recipes rot. --check confirms in milliseconds that every recipe still matches its code, and runs inside the test suite so a rotted recipe turns red on its own

Three verdicts

“Did not run” is not the same as “found nothing”

red ✓

Caught

The test passed on the untouched code and failed with the break. Nothing to do: this is the goal.

green ✗

No teeth

Every test still passed. Fix the test or the case it uses. Never change the recipe just to make it red.

broken ?

Did not run

The text wasn't found, the target leads outside the project, the test already failed before the break, or it timed out. This is no result, and it is never read as green.

The third verdict matters most. In a long list, a run that measured nothing looks like a run that found nothing, and the next thing people do is delete the test.

Built from failures

Every safety rule is something that actually went wrong

Restore from a copy

An early version restored with git checkout. That brings back the last commit, and it threw away an uncommitted refactor in the same file.

Read the exit code

Searching the output for “failed” read a collection error as green: “no teeth”, about a run in which nothing was checked.

Clear the bytecode cache

Two same-size breaks within one second ran as the first break, so the second was wrongly reported as having no teeth.

One run at a time

Two runs on one project each saved the other's broken file as “the original”, and a break was left in the code. Now a lock on the project itself refuses the second run.

Stay inside the project

A path like ../x or a folder shortcut inside the project let a break reach files outside it. Targets are now judged by where they actually lead.

The test must pass first

A test that already fails turns red whatever the break does, and was counted as “caught”. Each test now runs once on the untouched code first.

Status & scope

What exists, what's proven, where it stops

Built

One Python file, no dependencies beyond pytest
Recipe format, --check for rotted recipes, a lock for one run at a time
A worked example with a weak test to find

Verified · 2026-09-29

Its own 27 tests pass
It breaks itself 12 ways; every one turns its tests red, and the files come back clean
The example's output matches its documentation
Reviewed before release; the fixes and the remaining limits are in the README

Scope

It covers the break someone thought of; it shows a test can fail, not what was never tested
It runs your tests, so it is not a sandbox: use it on code you trust
A skipped test still reads as green, and a hard kill can't restore the file; the README says what to do