lifecycle

The commands an operator uses between installing a server and retiring one: backing it up, restoring it, checking its media, and moving its store forward when the on-disk format changes. Part of #20.

Every one of these opens the store directly, and fjall holds an exclusive lock, so they run with the server stopped. For a restore that is not just a limitation: it is the only way to be sure nothing is writing behind you.

Backup

spindle backup <config> <file>

Writes a consistent snapshot of the store. It refuses to overwrite an existing file — a backup that silently replaces the good copy with a bad one is discovered at restore time, which is the worst possible moment.

The archive carries rows, not media blobs. Media lives outside the store (on disk or in S3) and is copied separately. This is deliberate — a backup file that inlined a terabyte of media would be a file nobody takes — but it means a restore is only half done when the rows are in.

Restore

spindle restore <config> <file>

Refuses a store that already holds rows. A restore into a populated store is a merge: rows the target holds and the backup does not survive it, so the result matches neither the backup nor the original, and it looks like a success. Restore into an empty store, or clear the old one first.

The restore reports which media the rows it wrote still need. A row-only restore leaves a server that is indistinguishable from a healthy one until somebody asks for a file, so the report is how you find out before your users do.

Verifying media

spindle verify-media <config>

The same audit a restore prints, on its own. Blobs go missing without a restore in sight — a bucket lifecycle rule, a half-copied directory, a disk that came back smaller. Exits non-zero when anything is absent, and names the media IDs rather than only the content hashes: the hash is what the backend calls it, the ID is what you have to look for in your other copy.

Migration

spindle migrate <config> [--dry-run]

Moves a store forward to the schema the current binary speaks.

The server never migrates on its own. A binary that rewrites the store the moment it starts is a change you cannot back out of: by the time you know it happened, the old bytes are gone. So a store at another schema makes the server refuse to start, naming this command, and the rewrite waits until you ask for it — with the chance to take a backup first.

--dry-run prints the plan and writes nothing. Use it. It is where you find out whether the plan contains a step you cannot undo.

Take a backup first

Not a formality. migrate is the one lifecycle command that rewrites data in place, and a backup taken beforehand is the only route back from a step marked irreversible.

Rollback limits

Reversibility is declared per step and reported before anything runs.

step is	means
reversible	the older binary can still read the store afterwards, and re-stamps the marker on its own terms. Downgrading works.
irreversible	it cannot. Going back means restoring the backup taken beforehand, and losing everything written since.

A plan containing one irreversible step is irreversible as a whole, and migrate says so on its first line, before the steps.

Two limits worth stating plainly, because neither is obvious:

• There is no migrate --down. A reversible step is reversible because the older binary can still read what the newer one wrote, not because Spindle can walk the step backwards. Nothing undoes a migration in place.
• A binary older than its store cannot be helped. migrate reports no path and touches nothing. The steps that would be needed have not been written, because the version they lead to did not exist when that binary was built. Upgrade the binary, or restore a backup from before the migration.

What a failure leaves behind

The schema marker is written last, and only after every step succeeded.

A store whose marker claims to be current while its data is half-rewritten would be refused by nothing and misread by everything — exactly the silent failure the marker exists to prevent, reintroduced by the tool meant to move past it. So a failed step leaves the marker where it was: the store still describes itself as the version it can still be read as, and you can fix the cause and run migrate again rather than reaching for the backup.

Format versions

"The storage format" is four formats stacked, each with its own number, and a bug report or a migration step has to say which one it means:

layer	named by	now	what a bump means
key schema	KEY_SCHEMA_VERSION in spindle-core	1	the byte layout of keys: a keyspace discriminant reused or reordered, or a key's fields moved. Adding a keyspace is not a bump: every existing key keeps its bytes and its meaning.
record schema	RECORD_VERSION in spindle-store	1	the encoding of the values under those keys (log entries and bodies), which carry their version as their first byte.
content digest	CONTENT_DIGEST_VERSION in spindle-core	2	how content addresses are derived: a domain tag, a length width or a field order in the hashing. Keys and records are untouched by such a change, which is why it has its own number (#78): a store under the old derivation would open and misread every node address.
engine format	the major version of fjall in Cargo.toml	3	the on-disk layout of the store itself: journal, segments, the engine's own version file. Spindle does not choose it and cannot rewrite it.

The first three are what the schema marker records and what migrate moves between; all three are pinned by crates/spindle-store/tests/format_compatibility.rs, whose fixtures are bytes written under the versions they name, so a bump fails a test until fixtures for the new version and a decision about data written under the old one exist. The fourth is what the marker cannot describe, because the marker is stored inside it.

The transitions so far, numbered so they can be referred to:

#	layer	from	to	where	how a store crosses it
1	engine format	fjall 2	fjall 3	#193	it does not. fjall 3 refuses a v2 directory with InvalidVersion(Some(V2)) and has no in-process upgrade. Restore from a backup taken under fjall 3, or start a fresh store.
2	content digest	1	2	#330	it does not, yet. The store is refused as content_digest 1 (a store from before the marker existed counts as 1 if it holds any row), and no migration step re-derives the addresses. Start a fresh store.

Both were accepted deliberately, because no Spindle had data to lose. They are kept on the books rather than forgotten because each is an example of what the machinery below does not do: transition 1 is the case it cannot help with, and crates/spindle-store/tests/backend_compatibility.rs holds fixtures written under fjall 2 to prove the refusal is clean and explains itself; transition 2 is a step the table could hold and does not, because nobody has needed it. Key and record schemas have never moved.

The current state of the table

No schema change has yet been given a data rewrite (transition 2 above was refused rather than migrated), so the migration table is empty and spindle migrate reports "already at this binary's schema" on every store in existence.

That is worth saying rather than implying otherwise. What exists today is the machinery and its guarantees, proven by crates/spindle-store/tests/schema_migration.rs against synthetic tables — chaining, the no-path refusal, dry runs writing nothing, and the marker never landing ahead of the data. Those are properties of the machinery rather than of any particular step, which is why they can be established before the first real step is written, under whatever pressure produced the schema change that needed it.