Recover a stalled deploy: three recovery paths

🧪

Hands-on lab

Stall a deploy halfway on all four engines, then recover it two ways — resume from the checkpoint, and bypass a script you fixed by hand.

Demos/Learn/course8-module-05 →

A deploy stopped halfway. The tables changed, a couple of scripts ran, and then it hit a bad one and quit. Now you’ve got a half-forged deploy on your hands — and the question isn’t what broke this time. You already know how to read that. The question is how you pick it back up: without redoing all the work that already landed, and without breaking what’s already in.

Hey folks. I’m Forge Barrett, master of the Content Forge here at SchemaSmith.

Modules 1 through 4 were diagnosis — find the phase, read the error, name it. This one’s recovery. And there’s a myth to clear up first: people think recovery means --ResumeQuench. It doesn’t. There are three tools here, and reaching for the right one is the whole skill.

Before you pick a recovery move, confirm which work units failed and why — Failures.log gives you that answer in one read, so you’re choosing a tool with the full picture rather than guessing from a partial trace.

Three tools, three situations

• Fix the source, re-run plain. The default. A plain re-run throws away any leftover checkpoint and re-converges from the top — safe, because every mechanical phase is idempotent and a script that already succeeded stays skipped. Most failures need nothing more than this.
• --ResumeQuench. Keeps the checkpoint and resumes at the failure, skipping the phases that already finished. For when redoing the completed work is expensive.
• Mark-done. You fixed the problem by hand, outside the package, and you want SchemaSmith to treat a run-once script as complete and never run it again.

Let’s earn each one.

Beat 1 — the checkpoint is your map

The lab stalls a deploy on purpose: beat1-broken adds an index (IX_Customer_FullName) and two run-once After-scripts — 01_backfill_ok.sql, which succeeds, and 02_backfill_broken.sql, which inserts a customer with a NULL email into a NOT NULL column. Deploy it and it stops at the After slot:

Unable to quench '.\beat1-broken\...\02_backfill_broken.sql': Cannot insert the value NULL
  into column 'Email', table 'diag_recovery.dbo.Customer'; column does not allow nulls.

(515 on SQL Server, 23502 on PostgreSQL, 1048 on MySQL.) The index built, 01 ran, 02 died. Half-finished. Now — SchemaSmith left you a map. Because the lab points CheckpointDirectory at a local ./checkpoints, you can open it:

[Completed Steps]
ModifiedTables
IndexesAndConstraints
TableDataDelivery
ForeignKeys
...
[After Scripts]
./beat1-broken/Templates/Main/After Scripts/01_backfill_ok.sql

Four phases done. 01 done. 02 is the frontier. That file is preserved on failure and deleted on success — a green run leaves nothing behind. Now watch what the two re-runs do with it.

A plain re-run throws the map away

Re-run without the flag — even before fixing anything — and the log re-runs every phase: Quenching modified tables, Quenching indexes and constraints, Quenching foreign keys, Quenching after database scripts. That’s CleanupCheckpoints() doing its job: no flag means discard the checkpoint and start clean. (01 still skips — you’ll see Skipping (previously quenched) — but that’s the run-once tracking table, consulted on every run, not the checkpoint.) Safe, idempotent, and it redoes all the finished work.

--ResumeQuench picks up where it stopped

Fix 02 — give it a real email — and add the flag:

schemaquench --ConfigFile:quench.settings.beat1-broken.json --LogPath:"$PWD/logs" --ResumeQuench

Now the log opens differently:

Resuming from checkpoint (Completed Steps: 4, Completed Scripts: 0)
  Kindling the forge
  Quenching after database scripts

Straight from Kindling to the After slot. The four completed phases — modified tables, indexes, foreign keys, data — skipped. Exit 0, checkpoint gone.

One thing to notice: Kindling the forge ran anyway. Kindling and the missing-tables step are never checkpointed — they rebuild session-scoped temp state the later phases depend on, and that state doesn’t survive between runs. So resume skips the completed phases, but it always re-lights the forge first. “Resume” isn’t “skip literally everything before the failure.” Good deal — that’s the map, and both ways to read it.

Beat 2 — bypass a script you fixed by hand

Sometimes you don’t want to re-run the script at all. beat2-broken adds 03_seed_catalog_broken.sql — a Product insert with a NULL Sku. Deploy it: 01, 02, and the seed all skip (previously quenched), and 03 fails on the NOT NULL. But say the real fix isn’t in the script — you’re going to seed the catalog correctly by hand, and you never want that broken script to run.

Two steps. First, the real remediation, outside the package:

INSERT INTO Product (ProductId, Name, Sku, UnitPrice) VALUES (1, 'Anvil', 'ANV-001', 199.99);

Then tell SchemaSmith the script is done — insert a row into the run-once tracking table. The key is five columns: the script’s template-root-relative path, the product, the slot, the template, the schema.

-- SQL Server
INSERT SchemaSmith.CompletedMigrationScripts ([ScriptPath],[ProductName],[QuenchSlot],[template_name],[schema_name])
VALUES ('After Scripts/03_seed_catalog_broken.sql','Shop','After','Main','');

-- PostgreSQL — double-quote the schema and table; they're case-sensitive
INSERT INTO "SchemaSmith"."CompletedMigrationScripts" ("ScriptPath","ProductName","QuenchSlot",template_name,schema_name)
VALUES ('After Scripts/03_seed_catalog_broken.sql','Shop','After','Main','');

-- MySQL
INSERT INTO SchemaSmith_CompletedMigrationScripts (ScriptPath,ProductName,QuenchSlot,template_name,schema_name)
VALUES ('After Scripts/03_seed_catalog_broken.sql','Shop','After','Main','');

Re-run plain. 03 logs Skipping (previously quenched) and the deploy goes green. You bypassed the broken script for good — the catalog’s seeded your way, and the row you inserted is the record that says so. Note the two paths that matter here: run-once tracking uses the slot-relative path (After Scripts/03_seed_catalog_broken.sql), not the longer form the checkpoint file uses. Match it exactly or the skip won’t take.

Which tool, when

• Plain re-run — a script bug you fixed. Re-run, done. The default, and usually the answer.
• --ResumeQuench — same fix, but you don’t want to redo expensive completed phases.
• Mark-done — you handled it another way and want the run-once script skipped permanently.

And one lever underneath all of it: a filename ending [ALWAYS] runs every deploy and is never tracked (for idempotent maintenance), while TrackRunOnceMigrations: false turns run-once tracking off entirely and runs everything, every time.

Check yourself: A deploy fails at an After-script, and you fix the script. Your teammate insists you must use `--ResumeQuench` to recover. Are they right — and what does the flag actually change?

No — they’re not right that you must. After a source fix, a plain re-run recovers fine: it discards the leftover checkpoint and re-converges from the top, safely, because the mechanical phases are idempotent and any script that already succeeded stays skipped via the CompletedMigrationScripts table. What --ResumeQuench changes is how much gets redone: it keeps the checkpoint and skips the phases that already completed, resuming right at the failure. So it’s an optimization for not repeating expensive completed work — not a requirement for recovery. (And even it re-runs Kindling and missing-tables every time, since those rebuild session state the later phases need.)

What’s next

One module left, and it ties the whole course together:

• Module 6 · Per-engine dialects & your runbook — how the same failure reaches you differently on SQL Server, PostgreSQL, and MySQL (the error channels, the codes, the quirks you’ve been collecting), and assembling everything from Course 8 into a team diagnostic runbook you’ll actually reach for at 2am.

---

A stalled quench isn’t a ruined one. The metal’s still good — it’s just gone cold mid-shaping, and you’ve got three ways to bring the heat back: start the piece over, pick up from the last clean strike, or set one bad weld aside because you’ve already fixed it by hand. Knowing which one the moment calls for is the difference between an hour lost and a minute.

Got a deploy that stopped halfway and you’re not sure how to pick it up? Email me at forgebarrett@schemasmith.com — tell me where it stalled and which tool you reached for, and we’ll walk it through.

Next up: Course 8 · Module 6 — Per-engine dialects & your runbook, the finale.

Until then, may every stalled quench find its second heat.

— Forge

Mark this lesson complete