Passed Every Test — 22 defects that survived a green test suite
Twenty-two confirmed defects from building and shipping one working system, grouped by the eight ways a check can run, report success and prove nothing. Symptom, cause and fix for each — two of them still only partly fixed.
Twenty-two defects that survived a green test suite, grouped by the eight ways a check can pass without proving anything.
What is in it
- README.md — the taxonomy, a 7-question checklist, and all 20 entries with symptom, cause and fix
- defects.csv — the register as data, sortable by blind spot or area
- blindspots.json — counts per blind spot and which fixes are still partial
Claims, and how to check them
- every entry was observed, not constructed as an example
check: each has a specific symptom, a confirmed cause and the fix that was applied - the seven blind spots are stack-independent
check: read the blind spot column alone; none of the seven names a language or library - two fixes are still incomplete and are marked as such
check: blindspots.json lists partial_fixes; the entries carry the note in place
What it does not do
- no code library or test framework — this is a register, not a tool
- not a study: 20 defects from one system, not a random sample, so no percentage here is a base rate
- does not cover concurrency, distributed systems or performance regressions
- two of the twenty fixes are only partial and say so
You need already
- nothing — plain markdown, CSV and JSON
blind_spots: 8defects: 22examples_from: Python, SQLite, FastAPI, CSSformat: markdown, csv, jsonlanguage: English
Twenty-two defects. Every one of them survived a check that should have caught it.
The bugs are ordinary — a missing migration, a shadowed variable, an empty column. The interesting part is the second column: for each one, the specific mechanism by which a test ran, reported success, and proved nothing. That list is short, it repeats, and it is the part that transfers to a codebase in another language doing another job.
THE EIGHT WAYS A CHECK LIES
• The check could not fail — an assertion that passes for a reason unrelated to correctness. Worse than no check, because it reports a colour.
• The environment was too clean — the test builds its world from the current definitions, so a mismatch between definitions and the world that already exists cannot occur there.
• The API returns a value instead of failing — a missing key gives you a blank, an unknown CSS class styles nothing, an append is not a replace. Nothing raises, so nothing is logged.
• The fix never reached the artefact — the source says one thing and the file on disk says another.
• Only the successes were counted — a report of what worked is not a report.
• The comment contradicted the code — a confident wrong sentence stops the next reader from checking.
• Two concepts shared one column — a disclosure that confidently answers the question next to the one being asked.
• A substring was treated as an identity — a cleanup matching on a name, deleting something real.
WHAT IS IN IT
• README.md — the taxonomy, an eight-question checklist to run against your own suite, and all twenty-two entries with symptom, confirmed cause and applied fix
• defects.csv — the register as data, so you can sort it by blind spot or by area
• blindspots.json — the counts, and which fixes are still partial
ENTRIES LOOK LIKE THIS
A fresh install died at import while the development machine stayed fine indefinitely — because create_all skips tables that already exist and therefore never resolves their foreign keys. A suite reported 15/15 while three of its most important assertions never executed, guarded behind an if that an empty result set made false. A cleanup query written to delete test fixtures matched a finished production item, because the filter was a substring and a real title contained the word "fastest".
NO CLEAN ENDINGS
Two of the twenty-two carry fixes marked partial, in place, because the underlying blind spot is still open and the next occurrence will also be silent. A register of tidy resolutions would be a marketing document.
WHAT THIS IS NOT
Not a survey, not a study, not best practice. Twenty-two defects from one system built by one operator over one stretch of work. The sample is small and it is not random — it is what happened. No percentage here is a base rate. What is offered is the taxonomy and the checklist, both of which cost real time to learn.
WHO THIS IS FOR
Anyone whose test suite is green and who has started to wonder what that actually proves. The examples are Python, SQLite and a web app; the eight blind spots are not.
English. Markdown, CSV and JSON — no software to install.