When your deploy fails: locate, read, recover

It’s 2am and your phone won’t stop. The nightly deploy failed. Someone flipped the customer email index to UNIQUE — except the data had duplicates sitting in it, and the whole quench stopped cold. Now you’re staring at a terminal trying to work out what happened, how far it got, and what it changed before it quit.

Hey folks. I’m Forge Barrett, master of the Content Forge here at SchemaSmith.

Courses 1 through 7 got your schema forged, versioned, deployed, and running across a whole fleet. This course is about the moment it doesn’t go clean — and how you stay calm anyway, because SchemaSmith doesn’t fail silently. Every run leaves a trail. Learn to read it and a 2am page stops being a mystery and starts being a checklist.

You don’t memorize failures — you read a phase map

Diagnostics feels infinite. Every engine error code, every column change that won’t take, every constraint that won’t hold. Nobody memorizes that, and you don’t have to. Every deploy runs the same ordered set of phases, and every failure happens inside one of them. Find the phase, know its handful of failure modes, pick the recovery. Locate, read, recover. That’s the whole method — and it reads the same on SQL Server, PostgreSQL, MySQL, and MariaDB.

The engine converges — it doesn’t just run your scripts

Here’s the mental model everything else hangs on. SchemaSmith’s table engine isn’t a script runner marching through a stack of migrations. Each mechanical phase looks at the live database, compares it to the model you declared in JSON, works out the difference, and generates DDL for exactly that difference — nothing more. Then it hardens it in.

That’s why every mechanical phase is idempotent: run it again on a converged database and it does nothing, because there’s no difference left to close. And it’s why diagnosing a failure comes down to one question:

What difference did this phase try to make — and why did the DDL for it fail?

Answer that and you’ve solved the incident. The rest of this course is teaching you where to find the answer.

The failure surface — your first look

When a deploy fails, SchemaSmith signals it immediately and rolls up every failure into one file.

The *** FAILED banner. As each work unit fails, the engine prints a line to Progress.log that you can’t miss:

[localhost,11433].[diag_blackbox] *** FAILED [Template:Main] ***

That’s your real-time signal — the exact database and template that stopped.

SchemaQuench - Failures.log. After the run, open this file first. For every failure you get three things in one block: the error message, a Debug SQL: line — the artifact that reproduces it, or a reminder to rerun with --WriteDebugSql — and a Context (last 25 lines) phase trail that names exactly where execution stopped. For a single-database incident you can read the whole picture without touching Progress.log at all.

SchemaSmith leaves you a trail

Three kinds of evidence — two on every run, one when you ask:

The logs. SchemaQuench - Failures.log is the fast first read — one block per failure, error + artifact pointer + phase trail. SchemaQuench - Progress.log is the full run narrative: every phase as it ran, the FAILED to quench: block with the engine’s own error, and, under the error, the line that points at the artifact. SchemaQuench - Errors.log mirrors the error for some failures and not others: a failed script of your own lands there on every engine, a phase the engine computes only on SQL Server, and a failed data delivery on none. The Progress log has the detail every time, so that’s the file you read on every engine. Every run also copies both into a numbered backup — SchemaQuench.0001, .0002, and so on — so a later run never overwrites the evidence from the one that broke.
The resolved-SQL artifacts — when you ask. Run with --WriteDebugSql (or "WriteDebugSql": true in your settings) and SchemaQuench writes one .sql file per mechanical phase it reaches, with every token expanded to its real value. Without it nothing is written, and the line under an error says so: Rerun with --WriteDebugSql to write the resolved SQL of this step to ./artifacts. They’re opt-in because they hold your real values — passwords and keys included. Most are copy-runnable: paste one into your client and you reproduce exactly what that phase did — which is how you watch a failure happen by hand. The table phases are the exception: they work on temp tables the Parse Table Json … artifact builds earlier in the same session, so read those for what the phase did rather than pasting them alone. With the switch on, the log points you straight at the file: Resolved SQL written to: ….
The checkpoints. A checkpoint lets an interrupted run pick up mid-stride. Here’s the tell: a run that succeeds deletes its checkpoint. A green run leaves logs (and artifacts, if you asked) but no checkpoint. So a checkpoint still sitting there is your first signal — the last run stopped partway, and its filename names the database and template that broke.

The phase map, in a healthy run

Before you can read a broken run, read a clean one. Deploy the lab’s baseline and open Progress.log — the phases march by in order:

Begin Quench of Shop
[localhost,11433].[diag_baseline] Begin Quench
[localhost,11433].[diag_baseline]   Quenching missing tables and columns
[localhost,11433].[diag_baseline]   Quenching modified tables
[localhost,11433].[diag_baseline]   Quenching indexes and constraints
[localhost,11433].[diag_baseline]   Quenching foreign keys
[localhost,11433].[diag_baseline] Successfully Quenched

That ordered list is the map. Structure changes happen up top (missing tables, then modified tables); indexes and constraints come next; foreign keys are the last mechanical phase, closing the envelope. If a run seeds data — a script slot Module 4 covers — that data lands before the FK phase, which is exactly why an orphaned-row failure surfaces at Quenching foreign keys rather than earlier. Every module in this course zooms into one stretch of that map.

And every phase leaves its resolved SQL behind. The declared model gets its own artifact, SchemaQuench - Parse Table Json ….sql — here’s the head of SQL Server’s, the JSON handed straight to the engine:

DECLARE @TableDefinitions VARCHAR(MAX)= '[
  {
    "$schema": "../../../.json-schemas/tables.sqlserver.schema",
    "Name": "[Customer]",
    "Columns": [
      {
        "Name": "[CustomerId]",
        "DataType": "INT",
        ...

The missing-tables artifact, SchemaQuench - Quench Missing Tables And Columns ….sql, starts at DECLARE @v_SQL NVARCHAR(MAX) = '' and works on temp tables (#Tables, #Indexes, …) that earlier steps of the same session created — so it won’t run pasted cold into a fresh client window. That’s the SQL Server form — PostgreSQL and MySQL write equivalent artifacts for the same phase, and Module 6 maps the per-engine differences. When Module 1’s deploy fails, you’ll read the engine’s error straight out of the FAILED to quench: block, open the same kind of artifact, and run the resolved SQL by hand to watch it break.

Read the exit code first

Before you open a single file, the exit code triages it for you:

0 — success. Nothing to do.
1 — finished, but incomplete. Something was skipped or downgraded; the progress log names each one.
2 — the run failed for a reason you can fix: a failed quench, a refused validation, a missing setting. The progress log starts the reason with FAILED. This is your diagnostic case; go read the trail.
3 — an unhandled exception. A defect in the tool, not in your package or your settings.
4 — the log backup itself failed.

The track ahead

Seven modules: six along the phase map, and one for the log you hand to someone else:

Module 1 · Reading the black box — trigger a real failure and follow it from FAILED to quench to the exact flagged batch. Name the phase.
Module 2 · Structure-change failures — the missing-tables and modified-tables phases: not-null adds, type conversions, truncation.
Module 3 · Index, constraint & FK failures — including our 2am incident: the unique index that won’t take on dirty data.
Module 4 · Script-slot & data-delivery failures — your own Before/After scripts and the data seed.
Module 5 · The recovery toolkit — fix-and-resume versus fix-by-hand-and-mark-done.
Module 6 · Per-engine dialects & your runbook — how the same failure reports differently on each engine, and you assemble your team’s diagnostic runbook.
Module 7 · Before you share the log — what SchemaSmith masks in a log, the secret that slips through, and why the artifact never leaves your machine.

New to SchemaSmith? Start at Course 1, Module 1 and come back when schemaquench --version answers.

Check yourself: You finish a run and the checkpoint directory is empty. What does that tell you — and what would a leftover checkpoint file mean instead?

An empty (or absent) checkpoint directory means the last run succeeded — a checkpoint exists only to let an interrupted run resume, and SchemaSmith deletes it the moment a run completes clean. A checkpoint still sitting there means the opposite: the last run stopped partway, and the checkpoint’s filename names the exact database and template that broke. The presence or absence of that file is your first read before you open a single log.

A good smith doesn’t panic when a piece cracks in the quench. He reads the metal — where it stressed, how far the crack ran, what the fire was doing when it went. The break tells you what happened if you know how to look. That’s all a failed deploy is: a piece that cracked, and a trail that tells you exactly where.

Got a deploy that failed in a way you couldn’t explain? Email me at forgebarrett@schemasmith.com — tell me what stopped and what the log said, and I’ll help you read the trail.

Next up: Course 8 · Module 1 — Reading the black box, where a deploy fails on purpose and we follow it from FAILED to quench all the way to the phase that broke — and the copy-runnable artifact that reproduces it.

Until then, may your runs finish green — and when they don’t, may the trail be short.

— Forge

Mark this lesson complete