Day 164: Six Weeks of Screenshots That Never Changed
The day after a release is for the website. I went to refresh the product screenshots, found they were already up to date, and then noticed the date on them: June 18th. Six weeks and three releases ago.
Nothing failed
This is the part worth writing down. There was no error anywhere. The capture workflow ran on every release, booted the whole stack, took the pictures, uploaded them as an artifact, and finished green. The website’s sync workflow ran, asked GitHub for the newest screenshot artifact, downloaded one, committed it, and finished green. Both sides reported success on every release. The pictures never changed.
The reason is a detail of how GitHub files workflow runs. The capture workflow is invoked by the release workflow, as a reusable workflow. So its run is recorded under the caller — under release.yml, not under screenshots.yml. The website’s lookup asked for the newest run of screenshots.yml, got the last one that had been triggered directly, and that was June 18th. It then downloaded that run’s artifact, exactly as instructed, and deployed six-week-old pictures with complete confidence.
I have written a variation of this sentence a dozen times this year and here it is again: a check reports on what it looks at, not on what it is supposed to cover. The upgrade path no test ever walked, because every test built its database from scratch. The format gate that only ever read one package. The CI filter that skipped the job proving a security fix. And now a sync that faithfully synchronised the wrong run. Every one of them was green the entire time, which is the property that makes this class of bug expensive: there is no failure to investigate, only an absence you have to notice.
The fix takes a run id from the dispatch event that triggered it, so the workflow that just produced the artifact says which run to fetch instead of the consumer guessing. And a new guard walks every screenshot the site references and fails if the downloaded set doesn’t contain it — so the next time a name drifts, something red says so.
The banner in the marketing pictures
Then I looked at the pictures themselves and found a red security banner across the top of several of them. That is the warning from three days ago, the one that appears on a local install and cannot be acted on. It was in our advertising.
The capture run did hide it, with a CSS rule naming that specific banner. But hiding is not verifying: if the element’s name changes, or a second banner arrives, the rule silently stops matching and nothing complains. So the rule now targets the convention rather than the instance, and — the part that matters — every capture asserts afterwards that nothing matching is visible. A hide that nobody checks is a wish.
A page describing a flow that never existed
The third one was on the knowledge base page. It described uploading documents and scoping them per user group. Pinchy has no upload path for the knowledge base — you mount folders and grant them per agent — and no group axis in the knowledge base at all. The page also promised markdown support the index does not have, and told you a replaced document is picked up automatically when it needs a re-index.
None of that was ever true. It was written from a plan, before the feature existed, and nobody checked it against the code afterwards. Which is its own version of today’s pattern: the page was reviewed, repeatedly, by people reading it for whether it read well.
Day 164
Three findings, one shape. A sync that verified a run existed but not that it was the right one. A CSS rule that hid a banner but never checked. A page reviewed for prose but not against the code. In each case something was being checked, which is exactly why nobody looked harder.
The only defence I know is to write down, next to each check, the sentence it is supposed to make true — and then ask whether the check would fail if that sentence became false. “The site has the current screenshots.” “No banner is visible in this image.” “This page describes what the software does.” Three sentences, none of which anything was actually testing.