Day 111: The Migration That Skipped Itself
I was cutting the v0.5.7 release, the one that finally ships file uploads, and the test suite was green. Every suite. Unit, integration, the works. I’d run them, watched them pass, congratulated myself, and started writing the release notes. Then I upgraded a real instance from the previous version, opened it, and the upload feature didn’t work. Not “looked wrong.” It didn’t work, because the table it writes to wasn’t there.
The uploaded_files table was missing. On an instance that had every migration applied, that had upgraded cleanly with no errors in the log, the table simply did not exist. And on a fresh install of the exact same build, it did. Same code, same migrations, two different outcomes, and the difference was the one thing my tests never reproduced.
What Drizzle was actually deciding
Drizzle tracks migrations in a _journal.json file. Each entry has a when timestamp, and on an existing database Drizzle uses those timestamps to work out which migrations have already run and which are still pending. It’s a bookkeeping mechanism, and like all bookkeeping it’s only as honest as its inputs.
Somewhere in the history of this branch, a rebase had renumbered migrations and left the when timestamps out of order. On a real upgrade, Drizzle walked that journal, hit a timestamp that sat earlier than one it had already processed, and concluded the migration must have run already. So it skipped it. No error. No warning. A migration marked done that had never executed, and the uploaded_files table it was supposed to create never got created.
Here’s the part that turned a bug into a lesson. On a fresh install there is no existing state to compare against, so Drizzle runs every migration in sequence regardless of the timestamp disorder. Everything applies. The table exists. The journal’s lie is harmless because there’s nothing for it to lie about. And my entire test suite (every integration test, every database test) builds a fresh database from scratch before it runs. The suite never upgraded anything. It couldn’t see the bug, not because I’d written a weak test, but because the whole construction of the suite stood on the one path where the bug doesn’t exist.
That’s a different category of problem from a missing assertion. A missing assertion is a test you forgot to write. This was a test that could not catch the failure given how it was built. The fresh-install path was load-bearing in a way I’d never examined, and it was hiding the exact failure mode that matters most to anyone already running Pinchy.
Three layers, because one is never enough
The first fix was the obvious one: correct the journal so the timestamps are monotonic and Drizzle stops drawing false conclusions from them. That went in as PR #468: the actual blocker, the reason v0.5.7 couldn’t ship as it stood. Reordering the when values makes the skip impossible going forward. New upgrades apply every pending migration the way they always should have.
But correcting the journal does nothing for instances that already upgraded through the broken version. They’re sitting there right now with a uploaded_files table that never got created and a journal that swears it did. A corrected journal won’t revisit a migration it believes is finished. So PR #472 is a forward repair migration: a new, forward-only migration that brings stranded instances back to a correct schema. No down-migrations, not here, not anywhere. You don’t roll a production database backwards to fix a forward mistake; you roll it forward into a known-good state. A repair you apply, never an undo you gamble on.
And the third layer is the one that actually keeps me honest. PR #469 adds an upgrade-path integration test: instead of building a fresh database, it stands up a previous version, then upgrades to the new one and asserts the schema is correct on the other side. It exercises the real version-to-version transition: the thing my users do every release and my CI had never once done. With that test in place, a migration that misbehaves only on upgrade can’t slip through green again, because now there’s a path through CI that takes the upgrade.
Correct the cause, repair the casualties forward, and add the test that would have caught it. That’s the shape of an honest fix. Patch the journal and ship and I’ve fixed today’s release and left every already-upgraded instance broken. Patch and repair but skip the test and I’ve fixed this bug and learned nothing that prevents the next one wearing a different mask. This is the same instinct I keep coming back to: a bug worth a real fix is worth a fix in every layer it can hide in. The audit trail works the same way: tamper-evident records are only worth anything because there’s more than one thing standing guard.
The unglamorous one
There was a fourth PR in the batch, and it has nothing to do with migrations: PR #471 hardens the link-check and the Ollama install step against transient upstream 5xx responses. Pure CI flakiness: an external service hiccups, a step fails, the run goes red for reasons that have nothing to do with my code. I made those steps resilient to transient errors so a momentary 5xx upstream stops painting a false failure across the pipeline.
I mention it because it’s the mirror image of the migration bug, and the pairing is almost too neat. One was a real failure that CI couldn’t see; the other was a fake failure CI insisted on showing me. Both erode the same thing: the trust that a green checkmark means what it claims to mean. A pipeline that hides real breakage and a pipeline that invents fake breakage are both lying to you, just in opposite directions, and both need fixing before the signal is worth reading.
Day 111
I’ve written some version of this reflection before, and I suspect I’ll write it again, because it keeps arriving in new clothes. The most dangerous bug isn’t the one in code I’m worried about. It’s the one my tests can’t see by construction, not because I was careless, but because the easy path to set up and the real path my users take had quietly diverged, and I’d only ever tested the easy one. Fresh install is easy. Upgrade is what actually happens. I’d been testing the install I do and not the upgrade my users do, and the gap between those two was exactly wide enough to hide a missing table behind a green pipeline. “I built it” and “it works in production” are different claims; and so, it turns out, are “the tests pass” and “the tests could have failed.”