Authoring idiomatic pipelines

A pipeline's Plan method and each job's Work method build the DAG. Plan inspection and execution must produce the same structure. Put I/O and host-dependent decisions in registered job or step callbacks, which execute after dispatch and may repeat on retry.

sparkwing pipeline lint checks each Plan body and the guards: blocks in .sparkwing/sparkwing.yaml, reports each violation by rule name, and exits non-zero so it can gate a push or a CI job. sparkwing pipeline lint --rules prints the live rule set.

Adding a pipeline

Start with the scaffold so the Go registration and YAML catalog entry are created together:

sparkwing pipeline new --pipeline deploy --template minimal

sparkwing.Register connects a name to its Go implementation in the pipeline binary. .sparkwing/sparkwing.yaml defines the repository catalog and its triggers, defaults, and guards. sparkwing pipeline list reads that catalog without compiling Go, so a Go registration alone will not appear.

When adding a pipeline by hand, pair its registration in .sparkwing/jobs/:

func init() {
    sparkwing.Register("Deploy", func() sparkwing.Pipeline[sparkwing.NoInputs] {
        return &Deploy{}
    })
}

with an entry under pipelines: in .sparkwing/sparkwing.yaml:

pipelines:
  - name: deploy
    entrypoint: Deploy

entrypoint names the registration key, and name is what sparkwing run invokes. The two may differ, and several names may share one entrypoint. The pipeline reads the invoked name from RunContext.Pipeline. Run sparkwing pipeline list to confirm the catalog entry, then sparkwing pipeline lint to check the pipeline source and guards.

Sequencing jobs with Needs

A multi-job pipeline dispatches in the order its edges require, not the order Plan calls Job. Needs declares that ordering: a job never dispatches until every job it needs has succeeded.

type Deploy struct{ sparkwing.Base }

func (p *Deploy) Plan(ctx context.Context, plan *sparkwing.Plan, in sparkwing.NoInputs, rc sparkwing.RunContext) error {
    build := sparkwing.Job(plan, "build", p.build)
    test := sparkwing.Job(plan, "test", p.test).Needs(build)
    sparkwing.Job(plan, "deploy", p.deploy).Needs(test)
    return nil
}

func (p *Deploy) build(ctx context.Context) error {
    _, err := sparkwing.Bash(ctx, "go build ./...").Run()
    return err
}

func (p *Deploy) test(ctx context.Context) error {
    _, err := sparkwing.Bash(ctx, "go test ./...").Run()
    return err
}

func (p *Deploy) deploy(ctx context.Context) error {
    return sparkwing.Bash(ctx, "./deploy.sh").MustBeEmpty("deploy failed")
}

test will not dispatch until build succeeds, and deploy waits on test in turn. A job can chain any number of upstream Needs; a job with none dispatches as soon as the runner has a slot. When a downstream job needs an upstream job's typed output rather than just its completion, wire a Ref and still add the Needs edge explicitly (see "Discarded Ref results" below) -- RefTo does not add the edge for you.

The Work return contract

A job with more than one step implements Workable instead of passing a plain func to Job: it declares a Work(w *sparkwing.Work) (*sparkwing.WorkStep, error) method, registers its steps onto w via Step, and returns.

type ExampleDeploy struct{ sparkwing.Base }

func (j *ExampleDeploy) Work(w *sparkwing.Work) (*sparkwing.WorkStep, error) {
    sparkwing.Step(w, "apply", j.apply)
    return nil, nil
}

func (j *ExampleDeploy) apply(ctx context.Context) error { return nil }

The two return values are the job's typed output step and a Plan-time materialization error. An untyped job -- one that does not embed Produces[T] -- has no output to designate, so it returns nil, nil once its steps are registered. A typed job returns the step whose value becomes the Produces[T] output that RefTo exposes downstream:

func (j *ExampleBuild) Work(w *sparkwing.Work) (*sparkwing.WorkStep, error) {
    compile := sparkwing.Step(w, "compile", j.compile)
    publish := sparkwing.Step(w, "publish", j.publish)
    publish.Needs(compile)
    return publish, nil
}

I/O in Plan (plan-io)

A Plan body that shells out, touches the filesystem, or makes an HTTP call runs that I/O every time the plan is read. The SDK's side-effect helpers refuse outright: sparkwing.Bash / Exec / Shell and anything in sparkwing/docker, sparkwing/git, or sparkwing/services panic through the runtime plan-guard, naming the call. Plain os, os/exec, and net/http calls have no such guard -- they run silently on every read, and this lint rule catches them. Move I/O, including configuration reads, into a registered job or step callback. Work also constructs the graph before dispatch and must remain pure.

Don't shell out while the DAG is built:

type Release struct{ sparkwing.Base }

func (p *Release) Plan(ctx context.Context, plan *sparkwing.Plan, in sparkwing.NoInputs, rc sparkwing.RunContext) error {
    sha, _ := sparkwing.Bash(ctx, "git rev-parse HEAD").Lines() // refused during plan construction
    sparkwing.Job(plan, "publish-"+sha[0], p.publish)
    return nil
}

func (p *Release) publish(ctx context.Context) error { return nil }

Do the I/O inside the registered job callback:

type Release struct{ sparkwing.Base }

func (p *Release) Plan(ctx context.Context, plan *sparkwing.Plan, in sparkwing.NoInputs, rc sparkwing.RunContext) error {
    sparkwing.Job(plan, "publish", p.publish)
    return nil
}

func (p *Release) publish(ctx context.Context) error {
    sha, err := sparkwing.Bash(ctx, "git rev-parse HEAD").Lines()
    if err != nil {
        return err
    }
    return sparkwing.Exec(ctx, "publish", sha[0]).MustBeEmpty("publish failed")
}

Choosing Bash versus Exec to run a shell command

Choose by where the values in the command come from.

Use Exec whenever an argument is dynamic -- a branch name, an image tag, anything built from a variable. Exec runs the argv directly with no shell, so there is no quoting to get wrong and no way for a value containing $, backticks, or ; to be read as shell syntax:

tag := "app:" + sha
_, err := sparkwing.Exec(ctx, "docker", "push", tag).Run()

Reserve Bash for a command line that itself needs shell features -- a pipe, a redirect, a glob, a conditional. Pass any dynamic value in through .Env() instead of interpolating it into the line, so it never reaches the shell parser:

sparkwing.Bash(ctx, `git -C "$R" status --porcelain`).Env("R", repo).MustBeEmpty("dirty tree")

Interpolating an untrusted value straight into a Bash line is a shell-injection risk; Exec, or Bash with .Env(), avoids it.

Branching on the runtime environment (plan-runtime-branch)

Plan renders the same DAG wherever it runs, so plan inspection and dispatch agree on the shape. Reading os.Getenv, switching on runtime.GOOS, or calling IsLocal() in the body branches the structure on the host that happens to read it. Express the condition where it belongs: a job-level SkipIf, evaluated on the executing host, or a pipeline guard that gates the whole run.

Don't branch the DAG on the host environment:

func (p *Deploy) Plan(ctx context.Context, plan *sparkwing.Plan, in sparkwing.NoInputs, rc sparkwing.RunContext) error {
    if os.Getenv("ENV") == "prod" { // a different DAG depending on where Plan runs
        sparkwing.Job(plan, "deploy-prod", p.deployProd)
    }
    return nil
}

Declare the job unconditionally. After dependencies complete, its SkipIf callback runs on the executing host under the job's award, inside its concurrency slot. An environment read there sees that host's environment:

func (p *Deploy) Plan(ctx context.Context, plan *sparkwing.Plan, in sparkwing.NoInputs, rc sparkwing.RunContext) error {
    sparkwing.Job(plan, "deploy-prod", p.deployProd).
        SkipIf(func(ctx context.Context) bool { return os.Getenv("ENV") != "prod" })
    return nil
}

A skipped job satisfies Needs, so dependent jobs can still run. If skipping preparation must also skip its dependents, apply the predicate to those jobs or to a JobGroup containing them.

An approval's SkipIf runs in a separate evaluation job before requesting approval, holding no capacity. A true predicate marks the approval skipped and satisfies its dependencies. Approvals cannot declare concurrency groups. Put the group on the job after the approval so that job takes its slot on its executing host. The predicate can read its declared Needs, secrets and pipeline references on both run-host and controller-owned runs.

To gate the whole pipeline instead of one job, use a guards: block (see below).

Runner labels (runner-label)

The linter flags blank runner labels on Requires and Prefers. The host trims outer whitespace, drops empty labels and keeps the first copy of each label in declaration order. Both SDKs preserve the declarations for host processing. Alternatives within a selector term keep their existing OR semantics. An Inline() job executes on the dispatcher's host, where Requires and Prefers select no runner.

Avoid blank labels and labels on inline jobs:

sparkwing.Job(plan, "build", func(ctx context.Context) error { return nil }).Requires("")

sparkwing.Job(plan, "setup", func(ctx context.Context) error { return nil }).Inline().Requires("linux")

Do label the job that needs a runner, and leave the inline job to the dispatcher:

sparkwing.Job(plan, "build", func(ctx context.Context) error { return nil }).Requires("linux")
sparkwing.Job(plan, "setup", func(ctx context.Context) error { return nil }).Inline()

Discarded Ref results (unused-ref)

A Ref is the typed handle a downstream job reads an upstream job's output through. Creating one with RefTo and discarding it -- into _ or as a bare statement -- is dead code: either wire it into a job or drop the producing edge.

Don't throw the Ref away:

build := sparkwing.Job(plan, "build", &Build{})
_ = sparkwing.RefTo[BuildOut](build) // nothing reads this Ref

Do wire it into the job that consumes the output:

build := sparkwing.Job(plan, "build", &Build{})
out := sparkwing.RefTo[BuildOut](build)
sparkwing.Job(plan, "deploy", &Deploy{Build: out}).Needs(build)

Shared cache across a group (group-cache-shared)

JobGroup.Memo applies one key function to every member. A constant key makes every member share one result. Give members doing different work distinct keys.

Don't cache the group:

sparkwing.JobFanOut(plan, "matrix", goVersions, func(v string) (string, any) {
    return v, &Test{GoVersion: v}
}).Memo(func(ctx context.Context) (sparkwing.MemoValue, error) {
    return sparkwing.MemoKey("tests"), nil // one key for every Go version
})

Do key each member:

matrix := sparkwing.JobFanOut(plan, "matrix", goVersions, func(v string) (string, any) {
    return v, &Test{GoVersion: v}
})
for _, member := range matrix.Members() {
    version := member.ID()
    member.Memo(func(ctx context.Context) (sparkwing.MemoValue, error) {
        return sparkwing.MemoKey("tests", version), nil
    })
}

A key callback returns (MemoValue, error). Return errors when inputs cannot be read; return sparkwing.NoMemo, nil to bypass memoization. The executing host hashes the returned JSON value. Errors, panics, empty callback values, and resolution deadlines fail before the body runs. Use .Cache() to restore dependency directories before executing a job.

Configuring generated jobs

source.Generate receives a plan for jobs created after the source completes. Apply placement, retry and memo declarations to each generated job inside that callback. NeedsGenerated waits for the generator's settlement barrier, including an empty expansion.

source := sparkwing.Job(plan, "discover", &Discover{})
shards := sparkwing.RefTo[[]Shard](source)
generated := source.Generate(func(ctx context.Context, expansion *sparkwing.Plan) error {
    for _, shard := range shards.Get(ctx) {
        sparkwing.Job(expansion, shard.Name, &Bench{Shard: shard}).
            Requires("gpu").Retry(2)
    }
    return nil
})
sparkwing.Job(plan, "finish", func(context.Context) error { return nil }).
    NeedsGenerated(generated)

The host validates the complete expansion before accepting it. Each generator can declare at most 256 jobs. A failed source settles the barrier without executing the callback. See Generated jobs.

Declarative trigger filters and run guards

The branches, paths, and actions fields under on.push and on.pull_request record intent. The controller dispatches the pipeline named by the webhook URL without reading those fields.

Use a pipeline guard when the policy depends on the branch Sparkwing checks out. when: [git:branch=main] blocks a run from any other checked-out branch before a step starts. The literal name matches the head branch; it does not match a pull request's base branch. git:branch=default matches the repository's default branch: a GitHub App trigger records it, and any other run reads origin's HEAD from its checkout, asking origin when the clone did not record it. A run whose default branch cannot be resolved is refused rather than judged without it. These guards do not implement path filters, custom pull-request actions, or pull-request base-branch matching; Triggers describes that boundary.

Unsatisfiable guards (guard-misuse)

A pipeline's guards: block gates dispatch on where the run is dispatched, args, and git branch -- where:local / where:controller, arg:FLAG=VALUE, and git:branch=NAME / git:branch=default. when blocks the run when not every token matches; unless blocks it when any token matches. A token in both lists or a when naming two mutually exclusive execution locations prevents dispatch. A duplicate token is redundant. The linter reports both kinds of defect. The default token matches only when the dispatch supplies default-branch metadata; use a literal branch for controller webhook and local trigger claims.

Don't write guards that can never all hold:

# .sparkwing/sparkwing.yaml
pipelines:
  - name: deploy
    entrypoint: Deploy
    guards:
      when: [where:local, where:controller] # mutually exclusive
      unless:  [where:controller]                # also rejected -> contradiction

Do pick tokens that can be satisfied together:

# .sparkwing/sparkwing.yaml
pipelines:
  - name: deploy
    entrypoint: Deploy
    guards:
      when: [where:controller]  # run only on a controller
      unless:  [git:branch=main]     # never from the checked-out main branch

Running the linter

sparkwing pipeline lint            # every pipeline in the repo
sparkwing pipeline lint --pipeline deploy    # one pipeline by name
sparkwing pipeline lint --rules          # print the rule charters

Add -o json for machine-readable findings. Point --dir at a source tree other than the convention (.sparkwing/jobs). A non-zero exit on any finding makes the command a drop-in gate for a pre-push hook or a CI job.