A Rollback Should Not Rebuild

From the team behind aidrop.it — one workspace to build, host, and keep changing your code.

A deploy goes wrong and the instinct is to check out the previous commit and deploy again. That is not a rollback. It is a new build of old code, and a build is not a pure function of a commit: the base image tag has moved, an unpinned dependency resolves to a newer version, the build tool itself has changed. The image that ran is the only artefact proven to work, so a rollback is starting that image again, by name, without rebuilding it.

The second half is less obvious and matters more: a rollback restores code, not the schema, and not the data the new version already wrote. Whether the old image can run against the database as it is now was decided when the migration was written.

A rebuild of an old commit is a new artefact

Two builds of the same commit, a week apart, are not the same image. The Dockerfile says node:20 and that tag now points at a different patch release. The lockfile was not committed, so a minor version moved. The builder upgraded. Each is small, and any one is enough for the "known good" revision to fail in a way it never did before.

The fix is to keep what the build produced. Every successful build keeps its image, tagged by the commit it was built from, and a rollback is "start that tag". Nothing is rebuilt and nothing is promoted through a second environment: the image that ran is the image that runs. A registry that lets a tag be overwritten, or that prunes old images to save space, has quietly deleted your rollbacks.

A rollback restores code, not schema

When the bad version deployed, it probably migrated the database forward. Starting the old image against that schema works only if the migration was additive. A renamed or dropped column means the old code queries something that is no longer there, and now two versions are broken instead of one.

The rule that makes rollback survivable is written into the migration, not the deploy tool: expand first, contract later. Add the new column and write both; ship; read the new one; and only remove the old shape in a later release, when no deployable version still depends on it. It is the same transition two applications sharing a database need, applied to one application across time.

Rows written in the new shape, emails sent, webhooks fired, payments taken: none of these roll back with the image. A rollback is not a restore, and a restore is a rehearsal you run before you need it. List what the bad version did to the outside world before deciding whether going back even helps.

Roll back the settings with the image

An image ran under settings: a port, a health path, a size, the names of the values it read. A rollback that starts the old image under the current settings is half a rollback, and the half that fails is the one you cannot see in the code. Bind the settings to the build they ran with, and start both together.

Then check from outside. A rollback that "completed" is a process statement; answering on the public address is the only evidence that counts, and it takes one request from a machine that is not the server.

Before you roll back: two commands and one record

Run the two commands before starting anything. The first answers whether the old image is safe to start against the schema as it is now; the second answers whether that image still exists under its commit. Then fill in the record and keep it with the incident, so the fix-forward has something to start from.

# Did the schema move between the revision that worked and the one that did not?
git diff --stat <good-sha> <bad-sha> -- migrations/
# any output: read the migration; a drop or a rename means the old image cannot run as-is

# Is the image that ran still there, by commit rather than by "latest"?
docker image ls --format '{{.Repository}}:{{.Tag}}  {{.CreatedAt}}' | grep <good-sha-12>

ROLLBACK RECORD
Date / operator:
Good revision (image tag = commit):
Bad revision rolled back from:
Image started without rebuilding:        [ ] confirmed
Settings restored with it:               [ ] port  [ ] health path  [ ] size  [ ] value names
Migrations between the two:              none / additive only / destructive → stop, fix forward
What the bad version wrote or sent, and who handles it:
Outside check on the public address passed at:
Fix-forward owner / next step:

FAQ

What is the difference between a rollback and a redeploy? A redeploy builds again and may produce a different image from the same commit. A rollback starts an image that already ran, unchanged. Only the second returns you to a state that is known to work.

Can I roll back a database migration by redeploying old code? No. Redeploying old code restores the code; the schema stays where the migration left it, and dropped data does not come back. Write migrations to expand first and contract later, so an older version can run against a newer schema, and decide before deploying which release removes the old shape.

How does a rollback work on a host that keeps every image? On aidrop.it every successful build keeps its image under a tag, the first twelve characters of its commit, and the registry keeps every image a build pushed. Starting an older tag runs that image without rebuilding it, under the settings that build ran with; returning to the tip is the same call with the newest tag. It restores code, not schema, so migrations have to be forward-compatible.

Should I roll back or fix forward? Roll back when the old image plus the current schema is known to work and the bad version wrote nothing you have to unwind. Fix forward when a migration was destructive or data has already moved, because going back would then break a second version rather than repair the first.