Read why tenants failed in different ways

🧪

Hands-on lab

Roll one change across the fleet and watch two tenants fail in two different phases. Read Failures.log to name both breaks at a glance, then drill into each tenant's own evidence, fix each root cause, and resume only the tail you fixed. SQL Server, PostgreSQL, MySQL, and MariaDB.

Demos/Learn/course7-module-05 →

You push one schema change to the whole fleet. It comes back Template 'Main' had 2 failed work unit(s) and exits with an error. Two tenants broke — but they didn’t break for the same reason, and one blanket retry won’t fix either. Which two? Which phase? And why?

Hey folks. I’m Forge Barrett, master of the Content Forge here at SchemaSmith.

Module 4 taught you to keep the fleet moving when a tenant goes down — ContinueOnDatabaseFailure lets the healthy tenants finish, and --ResumeQuench redeploys only the one you fixed. But it treated the failure like a black box: step around it, patch it, move on. Today we crack the box open. When a fleet-wide run comes back with failures, you’ll find which tenants broke, read what each one left behind, name the phase it died in, and fix each with intent. Let’s fire it up.

The change looks harmless

You’re hardening the fleet: one new rule, that every customer email must be unique. In the package that’s a single line — a UQ_Customer_Email unique index on Customer. You roll it across all five tenants at once:

cd sqlserver
schemaquench --ConfigFile:quench.settings.after.json

[localhost,11433].[fleet_tenant_001] Successfully Quenched
[localhost,11433].[fleet_tenant_002] FAILED to quench:
[localhost,11433].[fleet_tenant_003] Successfully Quenched
[localhost,11433].[fleet_tenant_004] FAILED to quench:
[localhost,11433].[fleet_tenant_005] Successfully Quenched
Template 'Main' had 2 failed work unit(s)
One or more database quenches FAILED

Three green, two down, exit 2. The fleet did its job — ContinueOnDatabaseFailure held the line and the healthy tenants hardened. But that 2 failed work unit(s) line is a body count, not a diagnosis. Time to read the metal.

The fast fleet-triage read: Failures.log

Failures.log names every failed tenant the moment a fleet run comes back red — the fast triage read before you walk the deeper Progress.log trail in the next section.

SchemaSmith writes a SchemaQuench - Failures.log alongside the Progress.log in your log directory. Where the Progress.log is every tenant’s story braided together, the Failures.log is a roll-up scoped to the run’s failures only — and it opens with a header that names the count right away:

2 failure(s): 2 Template:Main

Then, one block per failed tenant:

─── FAILED  [Template:Main]  [localhost,11433].[fleet_tenant_004] ───
Error: The ALTER TABLE statement conflicted with the FOREIGN KEY constraint "FK_OrderItem_Product". The conflict occurred in database "fleet_tenant_004", table "dbo.Product", column 'ProductId'.
Debug SQL: ./artifacts\SchemaQuench - Quench Foreign Keys localhost,11433.fleet_tenant_004.sql
Context (last 25 lines):
        Enable/Disable CDC
    Quenching object scripts
    Quenching between table and keys scripts
    Quenching indexes and constraints
        Collect index level extended properties
        Add New Computed Columns
        Add Missing Indexes
          Creating index [dbo].[Customer].[UQ_Customer_Email]
        Add Missing Xml Indexes
        Turn on Temporal Tracking for tables defined as temporal
        Add missing ProductName extended property to indexes
        Add missing ProductName extended property to xml indexes
        Add Missing Statistics
        Add Missing Defaults
        Add Missing Check Constraints
        Add Missing FullText Indexes
    Quenching after table scripts
    Quenching foreign keys
        Add Missing Foreign Keys
          Adding foreign key [dbo].[OrderItem].[FK_OrderItem_Product]
  The ALTER TABLE statement conflicted with the FOREIGN KEY constraint "FK_OrderItem_Product". The conflict occurred in database "fleet_tenant_004", table "dbo.Product", column 'ProductId'.

  Resolved SQL written to: ./artifacts\SchemaQuench - Quench Foreign Keys localhost,11433.fleet_tenant_004.sql
  FAILED to quench:
The ALTER TABLE statement conflicted with the FOREIGN KEY constraint "FK_OrderItem_Product". The conflict occurred in database "fleet_tenant_004", table "dbo.Product", column 'ProductId'.
  Resolved SQL written to: ./artifacts\SchemaQuench - Quench Foreign Keys localhost,11433.fleet_tenant_004.sql

─── FAILED  [Template:Main]  [localhost,11433].[fleet_tenant_002] ───
Error: The CREATE UNIQUE INDEX statement terminated because a duplicate key was found for the object name 'dbo.Customer' and the index name 'UQ_Customer_Email'. The duplicate key value is (dupe@shop.example).
Debug SQL: ./artifacts\SchemaQuench - Quench Indexes localhost,11433.fleet_tenant_002.sql
Context (last 25 lines):
    ...
        Add Missing Indexes
          Creating index [dbo].[Customer].[UQ_Customer_Email]
  The CREATE UNIQUE INDEX statement terminated because a duplicate key was found for the object name 'dbo.Customer' and the index name 'UQ_Customer_Email'. The duplicate key value is (dupe@shop.example).

  Resolved SQL written to: ./artifacts\SchemaQuench - Quench Indexes localhost,11433.fleet_tenant_002.sql
  FAILED to quench:
The CREATE UNIQUE INDEX statement terminated because a duplicate key was found for the object name 'dbo.Customer' and the index name 'UQ_Customer_Email'. The duplicate key value is (dupe@shop.example).
  Resolved SQL written to: ./artifacts\SchemaQuench - Quench Indexes localhost,11433.fleet_tenant_002.sql

One file. Two tenants named, two errors quoted in full, two Debug SQL: artifact paths, and a phase trail in the Context block that shows exactly where in the convergence pipeline each tenant fell. You know who failed, why, where, and where to look next — before you’ve opened anything else.

The Context (last 25 lines) is the phase trail from that tenant’s Progress.log up to the point of failure. For fleet_tenant_004 you can read right down the trail: the run cleared Quenching indexes and constraints, then entered Quenching foreign keys, fired Adding foreign key [dbo].[OrderItem].[FK_OrderItem_Product], and hit the wall. Phase named without scrolling the interleaved log.

During a live run, the *** FAILED [Template:Main] *** banner also fires inline in the terminal as each tenant falls — one line per failure, right where it happens, so you see it without waiting for the run to finish.

The fleet hands you a scene, not a confession (the deeper read)

The Failures.log roll-up names the tenants and the phases. For a single incident — cracking open the artifact, tracing the orphan row, verifying the fix before resuming — you go deeper. SchemaSmith leaves you three per-tenant trails through the interleaved Progress.log.

The FAILED to quench: line names the tenant and carries the engine’s own words. In the Progress.log, scroll to each failure block and the tenant is right there in the tag, with the raw engine error underneath it — no decoder ring needed.

Each tenant’s checkpoint tells you how far it got. One thing that trips people up: after a partial-failure run, every tenant keeps a checkpoint — the healthy ones too. It’s not a list of who broke. What it is is a depth gauge. Open one and read [Completed Steps]:

# Database: fleet_tenant_002
[Completed Steps]
ModifiedTables

# Database: fleet_tenant_001
[Completed Steps]
ModifiedTables
IndexesAndConstraints
TableDataDelivery
ForeignKeys

A healthy tenant lists all four mechanical steps. fleet_tenant_002 lists exactly one — it died right after ModifiedTables, at the very next phase. The checkpoint doesn’t just say that it broke; it says where.

The resolved-SQL artifacts are named per server and database. In one artifact folder you’ll find SchemaQuench - Quench Indexes localhost,11433.fleet_tenant_002.sql sitting right next to the one for tenant 004 — each tenant’s evidence tagged with its own name, and the log points you straight at the one that failed with a Resolved SQL written to: line. For the deep read of any single incident — cracking open the artifact, walking the phase map end to end — that’s Course 8, Reading the black box. Here we stay up at fleet altitude: find the tenant, read its trail, name the phase.

Tenant 002 — the index that wouldn’t take

Read 002’s failure block:

[localhost,11433].[fleet_tenant_002] FAILED to quench:
The CREATE UNIQUE INDEX statement terminated because a duplicate key was found
for the object name 'dbo.Customer' and the index name 'UQ_Customer_Email'.
The duplicate key value is (dupe@shop.example).

The message names the table, the index, and the exact colliding value. This tenant already had two customers sharing dupe@shop.example — so the moment the run tried to build the new unique index over that data, it couldn’t. The underlying error is 1505 on SQL Server (the number isn’t printed, but the message is unmistakable). Its checkpoint stopped at ModifiedTables, which puts the death one phase later, at indexes and constraints — exactly where a unique index gets built. Phase named. The rollout met dirty data.

Same failure, different words on each engine:

PostgreSQL:  23505: could not create unique index "uq_customer_email"
MySQL:       Duplicate entry 'dupe@shop.example' for key 'Customer.UQ_Customer_Email'

Tenant 004 — the key that wouldn’t hold

Now read 004. Different block, and — here’s the twist — a completely different failure:

[localhost,11433].[fleet_tenant_004] FAILED to quench:
The ALTER TABLE statement conflicted with the FOREIGN KEY constraint "FK_OrderItem_Product".
The conflict occurred in database "fleet_tenant_004", table "dbo.Product", column 'ProductId'.

That’s a foreign key, not an index. This tenant got touched out-of-band — somebody dropped FK_OrderItem_Product and left an orphaned OrderItem pointing at a product that doesn’t exist. And here’s the part worth burning in: your rollout never touched that foreign key. So why did it fail?

Because SchemaSmith isn’t a script runner marching through your one change — it’s a convergence engine that re-checks the whole declared model on every run. It saw the FK was missing against the package, recreated it, and validated it — and the orphan row failed the check. Its checkpoint got three steps deep (ModifiedTables, IndexesAndConstraints, TableDataDelivery) before dying at ForeignKeys. The error is 547 on SQL Server, 23503 on PostgreSQL, 1452 on MySQL. Drift anywhere in a tenant can surface during a change that has nothing to do with it. For the deep method on this phase, that’s Course 8, Index, constraint & FK failures.

Two failures hid behind one count

Step back and look at what just happened. The fleet-level summary said 2 failed work unit(s) — one number. But those two tenants broke for two unrelated reasons, in two different phases, needing two different fixes:

Tenant	Phase	Cause	Fix
fleet_tenant_002	indexes & constraints	duplicate emails	dedupe the data
fleet_tenant_004	foreign keys	dropped FK + orphan row	remove the orphan

That’s the whole lesson. You can’t blanket-retry a fleet. Re-running the failed tenants without touching a thing gets you the same two failures — because nothing fixed the duplicate emails and nothing reparented the orphan. On a real fleet, “some tenants failed, let’s just run it again” is how you burn an afternoon watching the same body count roll by. Read each tenant, fix each cause.

Fix with intent, then resume the tail

You’ve diagnosed both. Fix each at its root — collapse the duplicate email on 002, delete the orphan row on 004 — then resume. Not a full fleet redeploy: a resume, pointed at the checkpoints the failed run left behind.

schemaquench --ConfigFile:quench.settings.after.json --ResumeQuench

[fleet_tenant_001] Resuming from checkpoint (Completed Steps: 4, Completed Scripts: 0)
[fleet_tenant_002] Resuming from checkpoint (Completed Steps: 1, Completed Scripts: 0)
[fleet_tenant_003] Resuming from checkpoint (Completed Steps: 4, Completed Scripts: 0)
[fleet_tenant_004] Resuming from checkpoint (Completed Steps: 3, Completed Scripts: 0)
[fleet_tenant_005] Resuming from checkpoint (Completed Steps: 4, Completed Scripts: 0)
...
Completed quench of Shop

Read the depth gauges one more time. The three healthy tenants come in at Completed Steps: 4 — fully done, so they do no real work and land as instant no-ops. Only the two you fixed pick up where they fell: 002 from step 1, 004 from step 3, each finishing the phases it never reached. Exit 0, and on that clean success the whole set of checkpoints clears out. Diagnose, fix at the root, resume the tail — and the fleet comes back whole.

Reading a fleet

A fleet failure isn’t harder to diagnose than a single one — it’s the same read, done per tenant, out of one shared log. When you have the Deployment Failure Triage roll-up, open Failures.log first: it names every failed tenant, its error, the phase context, and the artifact pointer in one file. Then use the interleaved Progress.log, per-tenant checkpoints, and artifacts as the deeper per-incident read. Find who failed. Gauge how far each got. Read the engine’s own words to name the cause. Then fix each root and resume — never a blanket retry, because one count can hide two very different breaks. Locate, read, resume.

Check yourself: A fleet-wide run comes back with `2 failed work unit(s)`. You open the checkpoint directory and find a checkpoint file for all five tenants, not just the two that failed. What does that tell you, and how do you find the two that broke?

The presence of a checkpoint doesn’t mark a tenant as failed — after a partial-failure run, every tenant that ran keeps its checkpoint, healthy ones included; the whole set only clears when a later run fully succeeds. So the checkpoint directory is a depth gauge, not a failure list. To find the two that broke, open Failures.log first: it names every failed tenant, its error, the phase context, and the artifact pointer in one place. The interleaved Progress.log is the deeper trace: the FAILED to quench: lines name the failing tenants and carry the engine’s error. Then use each failed tenant’s checkpoint [Completed Steps] to see how far it got — a tenant stopped at one completed step died at the very next phase, which names where to look. (The healthy tenants sit at all four completed steps; on the resume they re-dispatch as instant no-ops.)

---

A smith running a long line of work through the quench doesn’t scrap the batch when two pieces come out cracked. He picks each one up and looks close — and he can tell at a glance these two didn’t fail the same way: one starved of heat, one struck wrong. Same batch, different flaws, different fixes. That’s diagnosing a fleet. The run gives you a count; the metal gives you the reason. Read each piece, fix its own flaw, and put it back to the fire right where it fell.

Staring at a fleet run that came back with failures you couldn’t explain? Email me at forgebarrett@schemasmith.com — tell me what the count said and what the log showed, and I’ll help you read the two breaks apart.

And that’s Course 7. You can stand up a database-per-tenant fleet, steer it from config, grow it at onboarding, operate it safely when one tenant drifts — and now, when a fleet-wide change comes back with failures, read the trail each tenant left, name the phase, and bring the whole fleet home. One toolset, one workflow, from the first tenant to the thousandth.

Until then, may your fleet runs come back clean — and when two pieces crack, may the metal tell you true which was heat and which was the hammer.

— Forge

Mark this lesson complete