When your deploy fails: locate, read, recover

It’s 2am and your phone won’t stop. The nightly deploy failed. Someone flipped the customer email index to UNIQUE — except the data had duplicates sitting in it, and the whole quench stopped cold. Now you’re staring at a terminal trying to work out what happened, how far it got, and what it changed before it quit.

Hey folks. I’m Forge Barrett, master of the Content Forge here at SchemaSmith.

Courses 1 through 7 got your schema forged, versioned, deployed, and running across a whole fleet. This course is about the moment it doesn’t go clean — and how you stay calm anyway, because SchemaSmith doesn’t fail silently. Every run leaves a trail. Learn to read it and a 2am page stops being a mystery and starts being a checklist.

You don’t memorize failures — you read a phase map

Diagnostics feels infinite. Every engine error code, every column change that won’t take, every constraint that won’t hold. Nobody memorizes that, and you don’t have to. Every deploy runs the same ordered set of phases, and every failure happens inside one of them. Find the phase, know its handful of failure modes, pick the recovery. Locate, read, recover. That’s the whole method — and it reads the same on SQL Server, PostgreSQL, MySQL, and MariaDB.

The engine converges — it doesn’t just run your scripts

Here’s the mental model everything else hangs on. SchemaSmith’s table engine isn’t a script runner marching through a stack of migrations. Each mechanical phase looks at the live database, compares it to the model you declared in JSON, works out the difference, and generates DDL for exactly that difference — nothing more. Then it hardens it in.

That’s why every mechanical phase is idempotent: run it again on a converged database and it does nothing, because there’s no difference left to close. And it’s why diagnosing a failure comes down to one question:

What difference did this phase try to make — and why did the DDL for it fail?

Answer that and you’ve solved the incident. The rest of this course is teaching you where to find the answer.

The failure surface — your first look

When a deploy fails, SchemaSmith signals it immediately and rolls up every failure into one file.

The *** FAILED banner. As each work unit fails, the engine prints a line to Progress.log that you can’t miss:

[localhost,11433].[diag_blackbox] *** FAILED [Template:Main] ***

That’s your real-time signal — the exact database and template that stopped.

SchemaQuench - Failures.log. After the run, open this file first. For every failure you get three things in one block: the error message, a Debug SQL: pointer to the artifact that reproduces it, and a Context (last 25 lines) phase trail that names exactly where execution stopped. For a single-database incident you can read the whole picture without touching Progress.log at all.

SchemaSmith leaves you a trail

Three kinds of evidence, every run:

The logs. SchemaQuench - Failures.log is the fast first read — one block per failure, error + artifact pointer + phase trail. SchemaQuench - Progress.log is the full run narrative: every phase as it ran, the FAILED to quench: block with the engine’s own error, and the Resolved SQL written to: line pointing at the artifact. SchemaQuench - Errors.log mirrors the error on SQL Server; on PostgreSQL and MySQL the detail stays in the Progress log, so that’s the file you read on every engine. Every run also copies both into a numbered backup — SchemaQuench.0001, .0002, and so on — so a later run never overwrites the evidence from the one that broke.
The resolved-SQL artifacts. One .sql file per mechanical phase, written whether the run passes or fails, with every token expanded to its real value. They’re copy-runnable: paste one into your client and you reproduce exactly what that phase did — which is how you watch a failure happen by hand. The log points you straight at the file: Resolved SQL written to: …. (A failed script — one of your own Before/After files — gets its own artifact with the offending batch flagged -- >>> FAILING BATCH <<<; we meet those in Module 4.)
The checkpoints. A checkpoint lets an interrupted run pick up mid-stride. Here’s the tell: a run that succeeds deletes its checkpoint. A green run leaves logs and artifacts but no checkpoint. So a checkpoint still sitting there is your first signal — the last run stopped partway, and its filename names the database and template that broke.

The phase map, in a healthy run

Before you can read a broken run, read a clean one. Deploy the lab’s baseline and open Progress.log — the phases march by in order:

Begin Quench of Shop
[localhost,11433].[diag_baseline] Begin Quench
[localhost,11433].[diag_baseline]   Quenching missing tables and columns
[localhost,11433].[diag_baseline]   Quenching modified tables
[localhost,11433].[diag_baseline]   Quenching indexes and constraints
[localhost,11433].[diag_baseline]   Quenching foreign keys
[localhost,11433].[diag_baseline] Successfully Quenched

That ordered list is the map. Structure changes happen up top (missing tables, then modified tables); indexes and constraints come next; foreign keys are the last mechanical phase, closing the envelope. If a run seeds data — a script slot Module 4 covers — that data lands before the FK phase, which is exactly why an orphaned-row failure surfaces at Quenching foreign keys rather than earlier. Every module in this course zooms into one stretch of that map.

And the artifact for a phase is copy-runnable — here’s the head of the missing-tables one, the declared model handed straight to the engine:

DECLARE @TableDefinitions VARCHAR(MAX)= '[
  {
    "Name": "[Customer]",
    "Columns": [
      { "Name": "[CustomerId]", "DataType": "INT", ... },
      ...

That’s the SQL Server form — PostgreSQL and MySQL write equivalent artifacts for the same phase, and Module 6 maps the per-engine differences. When Module 1’s deploy fails, you’ll read the engine’s error straight out of the FAILED to quench: block, open the same kind of artifact, and run the resolved SQL by hand to watch it break.

Read the exit code first

Before you open a single file, the exit code triages it for you:

0 — success. Nothing to do.
2 — a quench failed. This is your diagnostic case; go read the trail.
3 — an unhandled exception. Something broke outside the normal phase flow.
4 — the log backup itself failed.

The track ahead

Six modules, each one a stretch of the phase map:

Module 1 · Reading the black box — trigger a real failure and follow it from FAILED to quench to the exact flagged batch. Name the phase.
Module 2 · Structure-change failures — the missing-tables and modified-tables phases: not-null adds, type conversions, truncation.
Module 3 · Index, constraint & FK failures — including our 2am incident: the unique index that won’t take on dirty data.
Module 4 · Script-slot & data-delivery failures — your own Before/After scripts and the data seed.
Module 5 · The recovery toolkit — fix-and-resume versus fix-by-hand-and-mark-done.
Module 6 · Per-engine dialects & your runbook — how the same failure reports differently on each engine, and you assemble your team’s diagnostic runbook.

New to SchemaSmith? Start at Course 1, Module 1 and come back when schemaquench --version answers.

Check yourself: You finish a run and the checkpoint directory is empty. What does that tell you — and what would a leftover checkpoint file mean instead?

An empty (or absent) checkpoint directory means the last run succeeded — a checkpoint exists only to let an interrupted run resume, and SchemaSmith deletes it the moment a run completes clean. A checkpoint still sitting there means the opposite: the last run stopped partway, and the checkpoint’s filename names the exact database and template that broke. The presence or absence of that file is your first read before you open a single log.

A good smith doesn’t panic when a piece cracks in the quench. He reads the metal — where it stressed, how far the crack ran, what the fire was doing when it went. The break tells you what happened if you know how to look. That’s all a failed deploy is: a piece that cracked, and a trail that tells you exactly where.

Got a deploy that failed in a way you couldn’t explain? Email me at forgebarrett@schemasmith.com — tell me what stopped and what the log said, and I’ll help you read the trail.

Next up: Course 8 · Module 1 — Reading the black box, where a deploy fails on purpose and we follow it from FAILED to quench all the way to the phase that broke — and the copy-runnable artifact that reproduces it.

Until then, may your runs finish green — and when they don’t, may the trail be short.

— Forge

Mark this lesson complete