Migrating to v0.11.0

v0.11.0 adds node artifacts --- content-addressed file edges between pipeline nodes. Two changes need attention: the runs-store schema moves from 4 to 5 (auto-applied on open), and authors of a custom pkg/storage.StateStore backend must add one method. If you only run the sparkwing CLI or controller, the schema migration runs itself and there is nothing to do.

Runs-store schema 4 to 5

The nodes table gains an artifact_manifest column that records a producer node's published-artifact manifest digest. The column shipped with node artifacts, but the schema-version constant was left at 4, so a database already at schema 4 --- anyone upgrading from v0.9.2, v0.9.3, or v0.10.0 --- never ran the additive migration. The column stayed absent and every node read (ListNodes, GetNode) failed with "no such column", breaking all run reads, not just artifacts. v0.11.0 bumps the schema to 5 with a migration that adds the column, so the upgrade repairs an affected database on open.

Before --- a schema-4 database opened by a v0.11.0 binary that did not bump the version: the column is missing and node reads error.

After --- opening the same database with v0.11.0 runs the v5 migration, which adds nodes.artifact_manifest (default empty) and records schema version 5. Node reads and artifact writes work again.

Upgrade steps: none for a plain CLI or controller --- the store auto-migrates on open. A module that pins an older (schema-4) sparkwing and shares the same state database has its pre-commit / pre-push gate refuse the migrated database until the pin is bumped; bump the pin to v0.11.0 in the same stroke as the release, exactly as the schema 3-to-4 migration documented.

Why a minor bump: a runs-store schema change is a breaking change under the release standard, and the version constant is what the release schema gate keys off. Leaving it at 4 both skipped the repair and let the gate pass despite a real break.

StateStore implementers add SetNodeArtifactManifest

A producer node records the digest of its published-artifact manifest so a downstream consumer can stage the same files. The orchestrator writes that digest through StateStore.SetNodeArtifactManifest and reads it back from GetNode (the store.Node.ArtifactManifest field). Any type that implements StateStore must provide the method:

SetNodeArtifactManifest(ctx context.Context, runID, nodeID, manifestDigest string) error

Before --- a custom backend that satisfied StateStore before v0.11.0:

type myBackend struct{ /* ... */ }

func (b *myBackend) SetNodeStatus(ctx context.Context, runID, nodeID, status string) error {
    return b.mutateNode(runID, nodeID, func(n *store.Node) { n.Status = status })
}

// ... the rest of the StateStore methods ...

After --- persist the digest on the node so GetNode returns it:

func (b *myBackend) SetNodeArtifactManifest(ctx context.Context, runID, nodeID, manifestDigest string) error {
    return b.mutateNode(runID, nodeID, func(n *store.Node) { n.ArtifactManifest = manifestDigest })
}

Why: artifacts are immutable, content-addressed edges. The producer publishes files keyed by content digest and records one manifest digest on its node; the consumer reads that digest and stages the exact bytes. Storing the digest on the node is what lets a consumer dispatched the moment the producer finishes always see the reference, and what lets a cache hit carry the manifest forward unchanged.

Edge cases / gotchas:

• An empty manifestDigest clears the reference. Treat it as a no-op-equivalent write, not an error.
• The digest must survive a later FinishNode on the same node. Store it as its own column or field rather than packing it into the node's output blob.
• A backend that does not support artifact edges may implement the method as a no-op returning nil and still compile, but consumers on that backend then stage nothing. A functional backend persists the digest and returns it from GetNode.