Migrating to v0.6.0

v0.6.0 is the biggest shape change since v0.5. It reaches into pipeline YAML, profile YAML, the SDK, the orchestrator's resolution model, and the dispatch surface. Three forces drive the redesign:

1. One pipeline = one deployment shape. The v0.5 multi-target pattern (one Go entrypoint with N internal target shapes) is gone. Each pipeline binds an entrypoint to a single shape; multi-environment work becomes multiple pipelines that share an entrypoint.
2. Backends are uniform. Secrets, state, cache, and logs are all surfaces of the same backends.Surfaces bundle. The old separate pkg/sources package is gone; secrets are a fourth surface.
3. Profiles are named bundles, both in the project and per-user. The project YAML holds project-level profiles; ~/.config/sparkwing/profiles.yaml holds user-level overrides. --profile NAME replaces the active bundle wholesale.

Plan on touching every .sparkwing/ repo and ~/.config/sparkwing/profiles.yaml once. Hard cut: there is no v0.5 compatibility runway.

What disappears

What's new

• defaults: block at the project YAML top level: profile, args, guards, requires. Pipelines inherit; declare your own block to opt out.
• profiles: map at the project YAML top level: named backend bundles. Same shape as ~/.config/sparkwing/profiles.yaml's profiles:.
• backends.Surfaces with four fields: Secrets, State, Cache, Logs. Every profile must declare all four.
• type: none secrets backend for pipelines that never call Secret(ctx, ...).
• type: env and type: filesystem secrets backends (env vars / dotenv).
• type: controller universal: secrets/state/cache/logs all express via {type: controller, url, token | token_env}.
• token_env: VAR on any controller-typed spec to source the bearer token from an env var (intended for checked-in project YAML).
• Surface inheritance: a controller-typed surface with empty url/token inherits from the profile's top-level controller: block at load time.
• pipelines[].profile: NAME: per-pipeline profile selector that overrides defaults.profile. --profile from CLI overrides everything.
• pipelines[].requires: [labels]: pipeline-level runner-label requirements unioned with each job's Job.Requires(...). The reserved label local pins to in-process.
• defaults.guards and defaults.requires: replace wholesale by pipeline-level block when non-empty.
• Guard token grammar namespace:rest:
  • profile:local, profile:controller, profile:name=NAME
  • git:branch=NAME, git:branch=default
  • arg:FLAG=VALUE
• Git.DefaultBranch populated from origin's HEAD symref; feeds git:branch=default guard evaluation.
• RegisterEntrypoint[T](name, factory) + BindPipelinesFromYAML(cfg): one Go entrypoint can back many pipelines; YAML maps name to entrypoint.
• sparkwing.NewSecretResolverFromSpec(ctx, spec): SDK factory taking backends.Spec directly. Replaces NewSecretResolverFromSource.

Resolution chains (recap)

Profile (wholesale at every layer):

1. --profile NAME           (~/.config/sparkwing/profiles.yaml)
2. pipeline.profile NAME    (project profiles map)
3. defaults.profile NAME    (project profiles map)
4. nil                      (no profile; orchestrator falls back to sqlite-only test/dev shape)

Args (per-key merge):

schema.Default < schema.Computed < defaults.args < pipeline.args < CLI --flag

Guards / Requires (replace at pipeline level):

pipeline.guards if non-empty, else defaults.guards
pipeline.requires if non-empty, else defaults.requires

YAML before / after

Project YAML

Before (v0.5):

profile: prod                     # repo hint (top-level)

pipelines:
  - name: deploy
    entrypoint: Deploy
    tags: [deploy, prod]
    hidden: false
    description: "Push to prod"
    on:
      push: { branches: [main] }
    secrets:
      - { name: DEPLOY_TOKEN, required: true }
    defaults:
      replicas: "10"
    dispatch:
      source: prod-secrets
      runners: [prod-pool]
      protected: true
      requires_approval: true
    targets:
      prod: { runners: [prod-pool] }
      staging: { runners: [staging-pool] }

runners:
  prod-pool:
    type: kubernetes
    labels: [linux, gpu]

sources:
  default: prod-secrets
  entries:
    prod-secrets:
      type: profile
      profile: prod

After (v0.6):

defaults:
  profile: prod                   # project-default profile selector
  args:
    region: us-east
  guards:
    require: [profile:controller]
  requires: [linux]

profiles:
  prod:
    controller:
      url: https://controller.prod.example.com
      token_env: PROD_CONTROLLER_TOKEN
    secrets: { type: controller }
    state:   { type: controller }
    cache:   { type: controller }
    logs:    { type: controller }
  staging:
    controller:
      url: https://controller.staging.example.com
      token_env: STAGING_CONTROLLER_TOKEN
    secrets: { type: controller }
    state:   { type: controller }
    cache:   { type: controller }
    logs:    { type: controller }

pipelines:
  - name: deploy-prod
    entrypoint: Deploy
    on:
      push: { branches: [main] }
    args:
      replicas: "10"
  - name: deploy-staging
    entrypoint: Deploy
    profile: staging
    args:
      replicas: "3"

User profiles.yaml

Before:

default: prod
profiles:
  prod:
    controller: https://api.example.dev
    token: swu_xxx
    state: { type: s3, bucket: prod-state }
    cache: { type: s3, bucket: prod-cache }
    logs:  { type: controller }
    gitcache: { type: s3, bucket: prod-gitcache }

After:

profiles:
  prod:
    controller: { url: https://api.example.dev, token: swu_xxx }
    secrets: { type: controller }
    state:   { type: controller }
    cache:   { type: controller }
    logs:    { type: controller }

Every profile must declare all four surfaces. Use secrets: { type: none } if you have no secrets backend.

Migration recipe

Per repo:

1. Bump the SDK pin

go get github.com/sparkwing-dev/sparkwing@v0.6.0
go mod tidy

2. Move to sparkwing.yaml if you haven't already

If your repo still has .sparkwing/pipelines.yaml, do v0.5's flatten first:

mv .sparkwing/pipelines.yaml .sparkwing/sparkwing.yaml

Then merge any runners.yaml, sources.yaml, backends.yaml, sparks.yaml content as top-level sections. v0.5's migration guide has details.

3. Trim removed pipeline fields

Remove these keys from every pipelines[] entry: tags, hidden, description (if cosmetic), on.manual, on.deploy, dispatch, secrets. Rename defaults to args. The parser hard-rejects each removed field with a clear "unknown field X" error.

4. Split targets: blocks into N pipelines

For each targets: block, write one pipeline per target. Move per-target runners to requires; source is now via the active profile's secrets: surface; protected is a guards.require: [git:branch=default] token.

5. Move project hints into defaults:

# Was:
profile: prod
# Becomes:
defaults:
  profile: prod

6. Add a profiles: map to project YAML (recommended)

profiles:
  prod:
    controller: { url: https://controller.prod.example.com, token_env: PROD_TOKEN }
    secrets: { type: controller }
    state:   { type: controller }
    cache:   { type: controller }
    logs:    { type: controller }

This lets pipelines reference profile: prod without users needing the same name in their personal profiles.yaml.

7. Reshape ~/.config/sparkwing/profiles.yaml

• Drop default: field.
• Wrap each profile's controller + token into a nested controller: { url, token }.
• Remove gitcache, cost_per_runner_hour, auto_allow, default_runner, log_store, artifact_store, detect:.
• Add secrets: surface (use type: none if the profile has no secrets backend).
• Make sure every profile declares all four surfaces.

8. Update Go code

• Drop sparkwing.Target(ctx) calls.
• Drop Job.OnTarget("X") / WorkStep.OnTarget / JobGroup.OnTarget calls.
• Drop PipelineConfig[T] usages. Use WithArgs[T] with YAML args: defaults instead.
• Replace sparkwing.NewSecretResolverFromSource(ctx, src, profileLookup) with sparkwing.NewSecretResolverFromSpec(ctx, spec).
• Replace imports of pkg/sources with pkg/backends. The Spec type covers everything Source did, plus type: env and type: none.
• The Secrets() any provider interface still works the same way; the YAML secrets: block is gone but the Go-side declaration via struct tags + reflection persists.

9. Update guard tokens

profile-local        -> profile:local
profile-controller   -> profile:controller
profile-name:NAME    -> profile:name=NAME
git-branch:NAME      -> git:branch=NAME
git-branch:default   -> git:branch=default
arg:FLAG=VALUE       -> (unchanged)

10. Run pre-commit and pre-push gates

sparkwing run pre-commit
sparkwing run pre-push

Fix any errors that surface.

What didn't change

• Plan / Job / Work / Step / Spawn primitives.
• Job.Requires(...), Job.Prefers(...), Job.WhenRunner(...) runner labels.
• Job.Inline, Job.Optional, Job.ContinueOnError, retry / timeout modifiers.
• RunAndAwait cross-pipeline awaits.
• Triggers: push, webhook, schedule, pre_commit, pre_push.
• The Secrets() any provider interface.
• PipelineSecrets[T](ctx) typed accessor.
• sparkwing.Secret(ctx, "NAME") runtime lookup.
• Plan.Spawn(...) dynamic child-job creation.

Common gotchas

• "profile NAME: secrets surface is required" -- every profile must declare all four surfaces. Add secrets: { type: none } for profiles that don't need them.
• "defaults.profile NAME is not declared in profiles:" -- the named profile must exist in the project's profiles: map.
• "pipeline NAME: profile NAME is not declared in project profiles:" -- same, but for per-pipeline overrides.
• secrets: block at pipeline level errors with "unknown field" -- gone in v0.6. Move declarations to the Go-side Secrets() any provider.
• --target flag rejected -- gone. Pipeline name is the selector.

Worked example: split a multi-target deploy

Before (one pipeline, multiple targets):

pipelines:
  - name: deploy
    entrypoint: Deploy
    defaults: { replicas: "5" }
    dispatch: { source: prod-secrets }
    targets:
      prod:
        defaults: { replicas: "10" }
        dispatch: { runners: [prod-pool], protected: true }
      staging:
        defaults: { replicas: "3" }
        dispatch: { runners: [staging-pool] }

After (two pipelines, one entrypoint):

defaults:
  profile: prod

pipelines:
  - name: deploy-prod
    entrypoint: Deploy
    args:
      replicas: "10"
    requires: [prod-pool]
    guards:
      require: [profile:controller, git:branch=default]

  - name: deploy-staging
    entrypoint: Deploy
    profile: staging
    args:
      replicas: "3"
    requires: [staging-pool]

Operators run sparkwing run deploy-prod or sparkwing run deploy-staging. Each pipeline gets its own --help page, its own flag set, its own policy.