Migrating to v0.36.0

Local runs now execute each node as its own process instead of on a goroutine inside the dispatcher, which matches how nodes have always run in a Kubernetes pod. Most pipelines need no changes: everything that already worked on the cluster keeps working locally. What breaks is anything that relied on all nodes sharing one process -- a package-level variable one job wrote and another read, a job output that was never JSON, or a step that expected a terminal on stdin. The fixes are mechanical and each is described below.

Three other things shift meaning without breaking any API: Inline() now says which host a job runs on rather than which memory it shares, a plan-level node's stored CPU and memory figures are now that node's own usage rather than a share of the dispatcher's, and an OnFailure recovery node now gets the same dispatch envelope as every other node. Each is covered below. The release also advances the runs-store schema twice, to version 15 and then to 16; each has its own section.

Process-per-node

Every job in a local run is executed by a separate invocation of the pipeline binary. The dispatcher still owns the plan, the cache lookup, the concurrency slot, and the SkipIf decision; only the job body moves.

Why: local and cluster execution had different failure modes for the same pipeline. A node that leaked memory, corrupted global state, or crashed the process took the whole local run with it, while the same node in a pod took only itself. Sharing one address space also made a pipeline's cross-job data flow work locally in ways it could never work on the cluster, so bugs surfaced on the first deploy rather than on the laptop.

Cross-job state must travel through outputs

A package-level variable set by one job and read by another worked in a local run and silently read its zero value in a pod. Now it reads the zero value in both.

Symptom: a downstream job sees an empty string, a nil map, or a zero struct where the upstream job set a value. No error is reported -- the value simply was never there.

Before:

var buildDigest string // package-level: shared only within one process

type Build struct{ sparkwing.Base }

func (j *Build) Work(w *sparkwing.Work) (*sparkwing.WorkStep, error) {
    return sparkwing.Step(w, "build", func(ctx context.Context) error {
        buildDigest = computeDigest(ctx)
        return nil
    }), nil
}

type Deploy struct{ sparkwing.Base }

func (j *Deploy) Work(w *sparkwing.Work) (*sparkwing.WorkStep, error) {
    return sparkwing.Step(w, "deploy", func(ctx context.Context) error {
        return deploy(ctx, buildDigest) // empty in another process
    }), nil
}

After: declare the output with Produces[T], return it from the step, and hand the consumer a RefTo[T].

type BuildOut struct {
    Digest string `json:"digest"`
}

type Build struct {
    sparkwing.Base
    sparkwing.Produces[BuildOut]
}

func (j *Build) Work(w *sparkwing.Work) (*sparkwing.WorkStep, error) {
    return sparkwing.Step(w, "build", func(ctx context.Context) (BuildOut, error) {
        return BuildOut{Digest: computeDigest(ctx)}, nil
    }), nil
}

type Deploy struct {
    sparkwing.Base
    Build sparkwing.Ref[BuildOut]
}

func (j *Deploy) Work(w *sparkwing.Work) (*sparkwing.WorkStep, error) {
    return sparkwing.Step(w, "deploy", func(ctx context.Context) error {
        return deploy(ctx, j.Build.Get(ctx).Digest)
    }), nil
}

func (p *Pipeline) Plan(_ context.Context, plan *sparkwing.Plan, _ sparkwing.NoInputs, _ sparkwing.RunContext) error {
    build := sparkwing.Job(plan, "build", &Build{})
    sparkwing.Job(plan, "deploy", &Deploy{Build: sparkwing.RefTo[BuildOut](build)}).Needs(build)
    return nil
}

RefTo[T] is checked at plan time against the producer's Produces[T], so a type mismatch panics with the node id before any step runs rather than yielding a zero value at execution time.

The same applies to any other shared-process channel: an in-memory cache, a sync.Map, an open file handle, a database pool held in a package variable. Each node process builds its own.

Job outputs must be JSON-serializable

A node's output reaches downstream nodes only as JSON in the run store. Two checks enforce this.

At plan time, a job whose declared output type can never be encoded is rejected before the run starts:

pipeline declares job outputs that cannot cross a process boundary:
  job "build" returns pkg.BuildOut: field Done: chan bool is a channel
  a node's output reaches downstream nodes only by JSON round-trip through the run store
  see docs/migrations/v0.36.0.md#process-per-node

Only shapes encoding/json refuses for every value are rejected here: channels, funcs, complex numbers, and unsafe.Pointer. A struct whose fields are all unexported is fine -- it encodes as {} -- so an output embedding a sync.Mutex or an *os.File still plans.

At execution time, a node whose output value fails to encode fails, rather than reporting success and handing its consumers nothing:

build: output of type pkg.BuildOut could not be encoded as JSON: json: unsupported type: chan int
  a node's output reaches downstream nodes only by JSON round-trip through the run store
  see docs/migrations/v0.36.0.md#process-per-node

This catches what the plan-time check cannot: an any field holding an unencodable value, an error field, a cyclic pointer graph, a NaN float, or a json.Marshaler that returns an error.

Fix: give the output type a JSON representation. Replace the unencodable field with data (a path instead of an *os.File, a string instead of an error), tag it json:"-" if downstream nodes do not need it, or implement json.Marshaler on the type.

stdin is /dev/null and stdout is a pipe

A node process does not inherit your terminal. Its stdin is closed, and its stdout and stderr are pipes the dispatcher reads and forwards into the run log. This has always been true in a pod; it is now true locally too.

Symptom: a step that prompted for input hangs or reads EOF immediately; a library that probes isatty disables color, progress bars, or interactive mode; a tool that requires a TTY refuses to run.

Fix: take the value as a pipeline argument or a secret rather than prompting for it. For tools that insist on a terminal, pass their non-interactive flag (--yes, --no-input, --batch) or set the environment variable they honor (CI=true, DEBIAN_FRONTEND=noninteractive). Approvals are the supported way to pause a run for a human: use an approval node, which the dispatcher resolves without the node process needing a terminal.

Node output is still forwarded to your terminal as it happens, so a local run reads the same as before. A single line longer than 1MiB is truncated in the forwarded copy, with the truncation marked; the node's own log file keeps what the node wrote.

Inline() means the dispatcher's host, not its memory

Inline() used to mean two things at once: the job skips the configured Runner, and the job runs inside the dispatcher's own process. Only the first is still true everywhere.

What it means now: the job runs on the dispatcher's host rather than being handed to a cluster runner. In a local run it is its own process, exactly like every other job -- it starts faster because nothing spins up a pod, not because it shares anything. Dispatched to a cluster runner, an inline job still runs inside the dispatcher itself, where the old advice holds.

What to do: nothing, if you used Inline() for what it is for -- skipping runner boot on cheap glue. Two habits need revisiting:

• An inline job that read a package variable another job wrote. That worked locally and never worked on the cluster; now it works nowhere. Fix it the way cross-job state describes.
• The "keep it under a second" rule. That is cluster-only advice. It exists because an inline job on the cluster path occupies the dispatcher's goroutine pool and delays scheduling for every other node. A local inline job costs only its own process, so a slow one is just a slow job.

.Inline() on an approval gate still panics, and Requires labels are still ignored for inline nodes.

Spawned children run inside their parent

sw.JobSpawn and sw.JobSpawnEach now work from inside a job wherever it runs -- a pod, a local node process -- where before any spawn outside the dispatcher hard-failed. The child is a real node in the run record: a <parent>/<id> row, its own logs, its own outcome.

What to know about cost: the child runs inside its parent's process, because it is that job's sub-work, and the parent stays blocked while it runs. Each is still measured and priced as its own node -- the child has its own sampler attachment and its own learned profile under <parent>/<id>. Because they share a process, the sampler divides each interval's reading among everyone attached, so a parent and three children take a quarter each rather than the process's whole draw four times over. The one figure that spans everything is the parent's exit accounting (cpu_nanos, max_rss_bytes, process_wall_nanos): the kernel charges the whole process at reap, so the parent's row carries the CPU its children burned as well as its own. A child's row has no exit accounting of its own and is priced from its samples.

Promoting spawned work to plan-level jobs (JobFanOut, or Needs edges) buys scheduling, not pricing: each unit then gets its own process, its own concurrency slot, and its own admission charge.

A child spawned inside a node process has its WhenRunner terms checked against what that process advertises, matching the gate the dispatcher has always applied on its own path: a child naming a label the host does not advertise is skipped, with that reason on its row, rather than running wherever the parent happened to land.

Per-node CPU and memory now measure only that node

A node's stored samples used to be a share of the dispatcher's process-wide reading, divided among however many nodes were running at that moment. A plan-level node now has a process to itself, so its samples are its own usage measured directly, and its exit accounting (see the version 15 section) records what the kernel charged that process.

The even split has not gone away; it just divides by one in the common case. It still applies wherever several nodes really do share a process: a job and the children it spawns (above), a worker started with --runner inprocess, and a test binary or a program embedding the SDK.

Symptom: a pipeline's learned capacity charge moves after the upgrade. Charges rise where the old split under-counted -- a node that did most of the work in a fan-out was charged a fraction of a reading it earned all of -- and where nodes shorter than one two-second sampling tick used to record nothing at all. Rollup peaks for fan-out-heavy pipelines rise for the same reason.

What you do: normally nothing. Profiles carry forward at their current price and converge on the measured one within about twenty runs, and sparkwing runs stats --capacity shows the figure admission is reserving.

If a profile is stuck on a measurement you no longer trust -- one freak run that recorded an absurd peak, or a charge that visibly disagrees with what the box is doing -- clear it and let it re-learn:

sparkwing runs stats --reset --pipeline myrepo/build

That drops the learned samples and the demand floor while preserving an explicit .Resources() pin, so admission keeps charging the pin during the re-learn. --all --yes resets every pipeline.

Cluster pricing is unchanged: a pod is already one node per process, and a controller-backed run is folded by the controller.

OnFailure recovery nodes get the full dispatch envelope

A recovery node is dispatched like any other node now, so it goes through the cache lookup, the concurrency slot, and SkipIf on the way in. The local dispatcher used to run a recovery body directly and skip all three; a pod never did.

Symptom: a rollback that used to run now fails or skips.

• Enrolled in a Concurrency group that is full under OnLimit: Fail, it fails with concurrency key "..." slot full under OnLimit:Fail instead of running beside the work it was declared exclusive with. Under OnLimit: Queue it waits for room.
• A SkipIf predicate on it is evaluated, and a true one skips it.
• A Memoize declaration on it takes effect, so a repeated identical failure replays the stored result rather than re-running the body.

What you do: if a recovery node should always run, do not declare Memoize, SkipIf, or Concurrency on it. Those declarations were never honored locally, so a pipeline carrying one has been running two different ways depending on where it was dispatched.

The typed in-run ref resolver is gone

Ref[T].Get had two paths: a live-Go-value resolver that only a shared process could satisfy, and the JSON resolver everything else used. Only the JSON one is left, and RuntimePlumbing.Keys.RefResolver went with it. Nothing a pipeline author writes changes -- Ref[T].Get reads the same -- but a program driving the orchestrator's plumbing directly has one key fewer to install.

Before:

ctx = sparkwingruntime.WithResolver(ctx, func(nodeID string) (any, bool) {
    v, ok := liveOutputs[nodeID] // the producer's own Go value
    return v, ok
})

After:

ctx = sparkwingruntime.WithJSONResolver(ctx, func(nodeID string) ([]byte, bool) {
    b, ok := storedOutputs[nodeID] // the JSON on the producer's node row
    return b, ok
})

Variables sparkwing sets on a node process

The dispatcher stamps a node process's environment with the run and node it is executing, the loopback controller to reach, and the run-level selections (--dry-run, --start-at, --stop-at) the node cannot otherwise see. Two are worth naming because they are visible in a process listing and belong to sparkwing rather than to you:

• SPARKWING_PARENT_LIVENESS_FD names the descriptor a node reads to learn that its dispatcher died, so an abandoned node stops instead of running on against a run nobody owns. Do not set it yourself: it grants the node the right to read and close that descriptor, and pointing it at a descriptor sparkwing did not open means reading something else's.
• SPARKWING_RUNNER_TYPE is local for a node process, the same value Runtime().Runner reported when nodes ran inside the dispatcher. The work runs on this machine either way, so a WhenRunner("local") job dispatches exactly as before.

Object-store local runs execute process-per-node

Who is affected: local sparkwing run invocations whose active profile uses object-store NDJSON state, including state: { type: s3 } in the CI-embedded deployment mode.

What changes: this was the last local shape that ran node bodies inside the dispatcher. Every node now re-enters the pipeline binary in its own process. The dispatcher mounts an authenticated, run-scoped API on an ephemeral 127.0.0.1 port so those child processes can reach the selected bucket. This does not require a deployed controller or Kubernetes.

What you do: no profile or pipeline YAML migration is required. Apply the process-per-node checklist: move cross-job values through typed outputs or artifacts, keep outputs JSON-serializable, and make steps non-interactive. Environments that restrict child processes or loopback listeners must permit both for local Sparkwing runs.

The selected state bucket and mirror_local behavior do not change. Object-store state also still does not fold measured local capacity profiles; use SQLite state or a controller when the executing machine needs learned admission pricing.

Local node logs use the declared logs surface

Who is affected: local runs whose active profile declares a logs surface.

What changes: each spawned node resolves the run profile and opens its declared log destination. A profile with logs: { type: s3 } writes the node body to that bucket, while controller-backed logs go to the logs service. A profile with no logs surface keeps the run's normal local files.

What you do: no change is required when the declared surface is valid and reachable. The node inherits the dispatcher's environment, including backend credentials, so make those credentials available to child processes and fix invalid backend configuration before upgrading. Sparkwing no longer falls back silently: a declared surface that cannot open fails the node and names the profile and surface type. To keep laptop logs local, use a profile with no logs surface, or declare a filesystem logs surface where project-profile validation requires one.

Verify a remote surface by running a small pipeline under the profile, then use sparkwing runs logs --run <id> --profile <name> and confirm a known node line is present.

Runs-store schema advances to version 15

What changes: three additive columns on nodes -- cpu_nanos, max_rss_bytes, and process_wall_nanos -- holding the kernel's exit accounting for the process that executed the node and the span it was drawn over, plus one on node_metrics, cpu_time_nanos, marking a sample as a per-command report and carrying the CPU it measured. Nothing is dropped or rewritten, and every row carried across the upgrade keeps the zero default.

What you do: nothing. The first v0.36.0 binary to open the store runs the migration on open in milliseconds, and concurrent opens coordinate at the database level. The step is safe to replay: a process killed mid-migration re-applies it harmlessly on the next open.

Why: now that each local node is its own process, the kernel can say exactly what that process cost. Sampling every two seconds cannot: it sees nothing between ticks, and nothing at all in a node that lives under one tick, which is why short nodes taught their pipeline nothing and kept being priced at a cold start. Capacity pricing reads these columns as a floor under the sampled figures, so expect charges to rise for pipelines whose work lives between ticks. A zero in any of these columns means nothing measured it -- a Kubernetes pod, a node executed inside the dispatcher, a sampler tick rather than a command report -- and is read as absent, never as work that cost nothing.

The exit figures cover the whole process, runtime startup and plan rebuild and teardown included, which is wider than the node's own started_at..finished_at window. That is why the span is stored beside the CPU rather than derived from those timestamps: pricing a process's whole CPU against the narrower window would report a rate the machine never gave, and a node whose own work took two milliseconds would be charged for most of a core. It also means a node's recorded duration is now the process's whole life, so ETAs count the startup the box paid for.

Rollback: as with every schema advance, a binary older than this release refuses to open a migrated database, naming both versions and the remedy. Upgrade every sparkwing that shares the store in one sitting -- including the admission daemon, which opens the same store to check whether a run is still live. To roll back the binary, also restore the store from before the upgrade ($SPARKWING_HOME/state.db, default ~/.sparkwing/state.db), or delete it and let it be recreated at the cost of learned profiles and run history.

Runs-store schema advances to version 16

What changes: one new table, node_bounces, holding each request to restart a running job's process and what became of it: who asked, when, and whether the stop landed on live work or arrived after the job had already finished. No existing table is touched.

What you do: nothing. The first binary to open the store creates the table, in the same millisecond-scale step every other advance runs on open. The step is safe to replay: the table and its index are both created only if absent, so a process killed between the step and its version stamp re-applies it harmlessly.

Why: sparkwing runs bounce has to survive the gap between the operator asking and the runner acting. The two are different processes -- the CLI has no handle on the job's child -- so the request is a row the runner polls on the loop that already heartbeats the job. The row is also what makes an operator's kill distinguishable from a crash: a node process that dies without writing a terminal outcome is a failure, and a bounce must not be one. Requests accumulate per job rather than overwriting, so a job bounced three times has three rows and the run's history says what happened to each.

Rollback: the same rule as any schema advance -- an older binary refuses a migrated database. Restore the store from before the upgrade or delete it, at the cost of learned profiles and run history.