Preview, throttle, and resume fleet deploys

🧪

Hands-on lab

Roll a schema change across the whole fleet — preview it first, tune the throttle, survive one drifted tenant without aborting the rest, and resume only the tail you fixed. SQL Server, PostgreSQL, MySQL, and MariaDB.

Demos/Learn/course7-module-04 →

You kick off a deploy across a thousand tenant databases. Four hundred in, one of them fails. Does the whole run die on the spot? Do the other five hundred and ninety-nine get touched twice when you retry? Do you even know which one broke?

Hey folks. I’m Forge Barrett, master of the Content Forge here at SchemaSmith.

Modules 1 through 3 built your fleet — you fanned out across it, steered it from config, and grew it at onboarding. Today’s the capstone: the day you run a change across all of it at once, one tenant goes wrong, and you come out the other side with the fleet whole and your nerves intact. Four moves — preview, tune, survive, resume. Let’s fire it up.

Look before you leap

Before you loose a fleet-wide run, confirm the fleet you think you’re about to strike. --PreviewTargets resolves the whole roster and lays it out — and quenches nothing:

cd sqlserver
schemaquench --ConfigFile:quench.settings.json --PreviewTargets

Pre-flight diagnostics for Shop (--PreviewTargets)
Template: Main [required]
  db: fleet_tenant_001
  db: fleet_tenant_002
  db: fleet_tenant_003
  db: fleet_tenant_004
  db: fleet_tenant_005
RESULT: PASS

Five tenants, discovered live from the catalog — the same DatabaseIdentificationScript fan-out from Module 1, no config roster in sight. Exit code 0, not a single object touched. This is your dry-run over the targets: the databases the run would fan across, listed before you commit. If a required template resolved to zero tenants, you’d get RESULT: FAIL and exit 2 instead — a run that would’ve quietly done nothing, caught before it started.

Tune the throttle

SchemaSmith deploys the whole fleet through one bounded pool of workers — not one thread per server, one pool over everything. MaxThreads is the ceiling: default 10, valid range 1 through 20. Turn it down to go easy on a shared box; turn it up for headroom on a big fleet. Here it is pinned to three in quench.settings.tuned.json:

{
  "MaxThreads": 3,
  ...
}

schemaquench --ConfigFile:quench.settings.tuned.json

  MaxThreads: 3
...
Completed quench of Shop

All five tenants quenched, three at a time at most. One knob, one number — the difference between a polite rollout and one that pins your I/O to the wall.

One tenant drifts — and the fleet keeps going

Real fleets drift. Somebody touches one tenant out-of-band and it stops matching the package. Simulate it — the lab’s drift-tenant-003.sql relaxes fleet_tenant_003’s Product.Sku to allow NULLs and parks a row with a NULL Sku in it. Now run the fleet, and wire in a checkpoint directory — you’ll want it in a minute:

schemaquench --ConfigFile:quench.settings.json --CheckpointDirectory:./checkpoints

fleet_tenant_003 can’t reconcile Sku back to NOT NULL — it’s holding a NULL — so that one work unit fails. But ContinueOnDatabaseFailure defaults to true, so the rest of the fleet quenches anyway:

[localhost,11433].[fleet_tenant_003] FAILED to quench:
Cannot insert the value NULL into column 'Sku', table 'fleet_tenant_003.dbo.Product'; column does not allow nulls. UPDATE fails.
[localhost,11433].[fleet_tenant_001] Successfully Quenched
[localhost,11433].[fleet_tenant_002] Successfully Quenched
[localhost,11433].[fleet_tenant_004] Successfully Quenched
[localhost,11433].[fleet_tenant_005] Successfully Quenched
Template 'Main' had 1 failed work unit(s)
One or more database quenches FAILED

Four of five hardened; one failed; the run exits 2. That’s the whole point of the default — one bad tenant shouldn’t hold the other nine hundred and ninety-nine hostage. The exact reconcile error is engine-specific — same failure, different words:

PostgreSQL:  23502: column "sku" of relation "product" contains null values
MySQL:       Invalid use of NULL value

But the fleet-level story is identical on all four: one fails, the rest finish, and Template 'Main' had 1 failed work unit(s) names the count.

Want the opposite posture? Set "ContinueOnDatabaseFailure": false on the Main template and the run aborts at the first failure — tenants after the one that broke never get attempted. One thing that trips people up: One or more database quenches FAILED prints in both modes. That line’s the run-level verdict, not the tell. The difference is what didn’t run — continue mode finishes the rest of the fleet, abort mode stops cold.

Resume the tail

Here’s where the checkpoint directory earns its keep. You’ve triaged: it was one drifted tenant. Fix it — the lab’s reset-tenant-003.sql clears the offending row so Sku can go back to NOT NULL. Now, instead of running the whole fleet again, you resume:

schemaquench --ConfigFile:quench.settings.json --CheckpointDirectory:./checkpoints --ResumeQuench

[localhost,11433].[fleet_tenant_001]   [fleet_tenant_001] Resuming from checkpoint (Completed Steps: 5, Completed Scripts: 0)
[localhost,11433].[fleet_tenant_002]   [fleet_tenant_002] Resuming from checkpoint (Completed Steps: 5, Completed Scripts: 0)
[localhost,11433].[fleet_tenant_003]   [fleet_tenant_003] Resuming from checkpoint (Completed Steps: 1, Completed Scripts: 0)
[localhost,11433].[fleet_tenant_004]   [fleet_tenant_004] Resuming from checkpoint (Completed Steps: 5, Completed Scripts: 0)
[localhost,11433].[fleet_tenant_005]   [fleet_tenant_005] Resuming from checkpoint (Completed Steps: 5, Completed Scripts: 0)
...
[localhost,11433].[fleet_tenant_003] Successfully Quenched

Read the checkpoint counts. The four that finished are at Completed Steps: 5 — every step already struck, so they do no real work. Only fleet_tenant_003, stopped at Completed Steps: 1, picks its remaining steps back up and finishes. The run exits 0, and on that clean success the checkpoint files are cleared out.

That’s the move that scales. A failed run keeps its checkpoints; a clean one clears them. So on a thousand-tenant fleet where nine hundred and ninety-nine hardened and one drifted, resume redeploys one — not a thousand. Fix the tenant, re-quench, done.

When you reach for each

Preview before every big run — it costs nothing and tells you the blast radius. Tune MaxThreads to your infrastructure, not a guess. Leave ContinueOnDatabaseFailure on so a single sick tenant doesn’t stall the fleet, and lean on --ResumeQuench to make a partial failure a one-tenant fix-up instead of a full redo. That’s the difference between deploying a fleet and operating one.

Check yourself: A fleet-wide run fails on 1 of 500 tenants and exits 2. You fix that one tenant and re-run with --ResumeQuench pointed at the same checkpoint directory. What happens to the 499 that already succeeded?

Nothing — they do no real work. The failed run preserved its checkpoints, so on resume every tenant is re-dispatched but each picks up from its own checkpoint. The 499 that finished are at their final completed step, so they’re instant no-ops; only the one tenant you fixed — stopped partway through — runs its remaining steps and finishes. That’s why resume scales: a partial failure becomes a one-tenant fix-up, not a 500-tenant redo. (And on the clean success, the checkpoint files are cleared.)

---

A smith running a long line of work doesn’t scrap the whole batch when one piece cracks in the quench. He sets the good pieces aside, reheats the one that failed, and strikes it true — the rest never see the fire twice. That’s operating a fleet here: preview the batch before you commit, hold the heat steady across every piece, let the one bad casting fail without dragging the rest down, and pick that single piece back up right where it fell.

Running fleet-wide deploys and holding your breath every time? Email me at forgebarrett@schemasmith.com — tell me how big your fleet is and how a bad tenant plays out today, and I’ll show you where preview and resume take the fear out of it.

That’s operating the fleet safely — but surviving a bad tenant isn’t the same as understanding it. Next up, Module 5: when a fleet-wide run comes back with failures, we diagnose why each tenant broke — read the trail it left behind, name the phase, and fix with intent instead of a blanket retry.

Until then, may your previews always run clean, your one bad tenant never sink the fleet, and every resume pick up right where the last strike fell.

— Forge

Mark this lesson complete