The Deploy Gate That Cried Wolf

A good build failed to deploy, and the reason made me rethink my test gate

Today I tagged a release of my trading bot, watched the pipeline march through lint, test, and build, all green, and then watched the deploy stage fail.

The code was fine. The tests were fine. CI had passed the identical commit twice. What failed was my deploy gate's philosophy, and the fix taught me a cleaner way to think about what each stage of a pipeline is actually FOR.

---

The setup: a deploy script with ten gates

My deploys are a tagged release plus a script that walks ten gates in order: preflight, secrets check, database backup, ship the artifact, build, migrate, run the full test suite, switch symlinks, health checks, notify. Any gate failing aborts BEFORE the running stack is touched. It has saved me before, and it saved me today too, just in an embarrassing way.

Gate 7 was the culprit: "run the full test suite inside the freshly built image". The intent was sound: the artifact that ships is the artifact that passed. But there was a hidden assumption in HOW it ran the suite.

It ran the tests against the production database.

Each test wraps itself in a transaction and rolls back, so nothing is ever left behind. That part worked perfectly. But inside a test's transaction, the production data is VISIBLE. And production data grows.

What actually broke

Weeks earlier, I had created a real trading bot in the app, an auto bot that owns a particular instrument. My test suite, written long before, creates its own test bot on that same instrument and asserts things about it.

The application has an integrity rule: one owning bot per instrument. A correct, important rule. So when the test tried to create its bot, production said no:

BotError: US.SPUS is already managed by the auto bot
'US Stock Full Automation Bot Trade'

Thirty tests failed. None of them were wrong about the code. They were wrong about the world they assumed they were running in. My own production data had drifted underneath assumptions my tests made years... fine, weeks earlier. The gate blocked a good build because the DATABASE had changed, not the code.

That is a false positive, and false positives are how gates lose their credibility. A gate that cries wolf trains you to bypass it, and a bypassed gate is worse than no gate at all.

The fix: separate the three questions

The insight that untangled it: my deploy was smearing three different questions across one stage.

1. Is the code correct? That question wants a clean, migrated, empty schema. It must be immune to whatever production data has grown into.
2. Does the migration work on real data? That is a different gate, and I already had it: the migration runs against the production database earlier in the sequence.
3. Is the running system healthy on real data? Also already covered: health checks after the switch.

So gate 7 needed to stop borrowing production's data. Now it creates a throwaway database, migrates it from zero, runs the full suite against that clean schema inside the shipping image, and drops it. Exactly what CI does, but proving the shipped artifact rather than the repository.

One tag later, the pipeline went green through all four stages and the release deployed itself. Same code as the failed attempt. The only thing that changed was the gate answering the question it was actually asked.

The residue: honest debt

There is a second, quieter lesson. Those thirty tests were never hermetic. They passed for months because CI always handed them an empty database, and the assumption only surfaced when a gate finally ran them somewhere with history.

The proper fix, test by test, is to make each one own its world: create what it needs, wipe what could interfere, assume nothing about existing rows. I am doing that gradually. The gate fix removed the urgency; the hygiene remains on the list, honestly labeled as debt instead of pretended away.

Takeaways

• A deploy test gate answers ONE question: is the code correct. Give it a clean schema.
• Real-data concerns deserve their own gates (migration, health), not a shared one.
• When a gate fails a good build, do not bypass it. Fix the gate's philosophy, or it will train you to ignore it on the day it is right.
• Tests that pass only on empty databases are loans. The interest arrives later.

---

Pictures to add later

1. GitLab pipeline: the failed run (lint/test/build green, deploy red)
2. GitLab pipeline: the passing re-run (all four green)