There is a moment during every deploy when two versions of your application are running at once. The new containers are up and taking traffic; the old ones are still draining. It lasts seconds, and most of the time nobody thinks about it.
It is also the moment that turns an ordinary column rename into an outage.
The failure
You rename entry.paid_at to entry.settled_at. One migration, one code change,
one release. Locally it is fine. In CI it is fine, because CI starts from an
empty database and runs one version of the code.
In production the migration runs, and for the next forty seconds the old
containers — which are still serving requests — issue SELECT ... paid_at against
a table where that column no longer exists. Every one of those requests is a 500.
The deploy “succeeded”. The dashboard went red anyway.
The reflex fix is a maintenance window: stop traffic, migrate, start traffic. That works, and it means you can now only ship at night, which means you ship less often, which means each release carries more change and more risk. The cure compounds the disease.
Expand, migrate, contract
The alternative is to never have a schema that only one version of the code can use. You split the change across releases so that at every point, both the version before and the version after can run against whatever the database currently looks like.
Release one — expand. Add the new column. Do not remove the old one.
ALTER TABLE entry ADD COLUMN settled_at TIMESTAMPTZ NULL;
Additive, nullable, and invisible to the running code. Nothing can break, because nothing reads it yet.
Ship application code that writes both columns and reads the old one. Now every new row is correct in both places, and old containers are still served by the column they know about.
Between releases — backfill. Copy the history across, in batches, outside the deploy:
UPDATE entry
SET settled_at = paid_at
WHERE settled_at IS NULL
AND paid_at IS NOT NULL;
Do this in chunks with a bounded WHERE, not as one statement across ten million
rows — a single long transaction holds locks and bloats the write-ahead log for
the entire time it runs.
Release two — switch the read. Now that both columns are fully populated,
ship code that reads settled_at and still writes both. If this release has to be
rolled back, the old code still works, because paid_at is still being written.
Release three — contract. Once release two is stable and you are past the point of rolling back, stop writing the old column and drop it.
ALTER TABLE entry DROP COLUMN paid_at;
The same rule, elsewhere
Once you see it, the pattern covers most schema changes that would otherwise need downtime.
Adding a NOT NULL column: add it nullable, backfill, then add the
constraint. Adding a NOT NULL column with a default to a large table rewrites
it in older PostgreSQL versions; check what your version actually does before
assuming it is instant.
Adding an index: CREATE INDEX takes a lock that blocks writes for the
duration. CREATE INDEX CONCURRENTLY does not — at the cost of running outside a
transaction and needing a cleanup step if it fails, which means your migration
tooling has to know it is a special case.
Changing a column’s type: the same three steps, with a new column of the new type rather than a new name.
Removing a field from an API response: identical logic, one layer up. Stop sending it only after every client has stopped reading it — and in a public API, “every client” includes the ones you cannot see.
What it costs
Three releases instead of one, and a period where the schema carries a column that exists only for the transition. That is real overhead, and on a small internal tool with one instance it may genuinely not be worth it.
What you buy is that deploys stop being events. No window, no coordination, no waiting until traffic is low — which means releases get smaller, more frequent, and individually less frightening.
That is the actual goal. The migration technique is just how you get there.