ci-embedded mode

Run sparkwing pipelines inside an existing CI job (GitHub Actions, Buildkite, GitLab CI, CircleCI, ...) without standing up a sparkwing cluster. State, logs, and artifacts go to S3-compatible storage so a remote dashboard can follow the run live and replay it after the CI VM exits.

When to use

ci-embedded is the migration wedge: keep your CI vendor's job orchestration, let sparkwing handle the pipeline DSL + caching + dashboard.

Quick start (GitHub Actions)

.github/workflows/ci.yaml:

name: ci
on: [push]

jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with: { go-version: '1.26' }
      - run: curl -fsSL https://sparkwing.dev/install.sh | bash

      - name: Run sparkwing release pipeline
        env:
          AWS_ACCESS_KEY_ID:     ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          AWS_REGION:            us-west-2
        run: sparkwing run release-prod --sw-mode=ci-embedded --sw-workers=4

State, cache, and logs destinations come from the resolved profile in ~/.config/sparkwing/profiles.yaml. Select one with --profile NAME (or set defaults.profile in .sparkwing/sparkwing.yaml); there is no environment-based auto-selection. See storage backends for the configuration shape.

A pipeline node that fails fails the GHA job (exit code propagates).

How it works

1. --sw-mode=ci-embedded plumbs through sparkwing -> the pipeline binary via env vars (SPARKWING_MODE, SPARKWING_WORKERS).
2. The orchestrator resolves state, cache, and logs from the active profile (selected by --profile or defaults.profile) -- e.g. an S3-backed profile.
3. Run and node state goes to the profile's state: surface. A sqlite state surface keeps records locally and uploads one runs/<runID>/state.ndjson dump to the cache surface when the pipeline exits; an S3 state surface writes that same NDJSON continuously over the life of the run.
4. Per-node log lines route to the resolved Logs backend instead of ~/.sparkwing/runs/<id>/.
5. A dashboard configured with the matching backends reads everything back.

Flags

State, cache, and logs come from the resolved profile; see storage backends for the configuration shape.

Recommended: SPARKWING_NO_SPARKS_RESOLVE=1 in CI

If your .sparkwing/sparkwing.yaml declares a sparks: block, sparkwing auto-refreshes the resolved overlay at run time by default. That shells out to go env / go list, which means CI runners would need a Go toolchain even on a cache hit. Set SPARKWING_NO_SPARKS_RESOLVE=1 in the CI step's env so the runner trusts the committed .resolved.mod overlay and never resolves on its own.

Workflow then becomes:

# locally, when you want a fresh resolve
sparkwing pipeline sparks update
git diff .sparkwing/.resolved.mod
git commit -am "bump sparks-core"
git push                          # triggers publish + run with frozen overlay

CI never re-resolves; the publish step on your laptop (or in a publish-on-merge workflow) is the deliberate "go fresh" surface. Repos without a sparks: block ignore this var -- it's a no-op.

Profile-based config (laptop)

~/.config/sparkwing/profiles.yaml:

profiles:
  ci-team:
    state: { type: s3, bucket: my-team-sparkwing, prefix: state }
    cache: { type: s3, bucket: my-team-sparkwing, prefix: cache }
    logs:  { type: s3, bucket: my-team-sparkwing, prefix: logs }

Then:

sparkwing run release-prod --sw-mode=ci-embedded --profile ci-team

Watching from a laptop dashboard

After (or during) a ci-embedded run, point your local dashboard at the same bucket:

sparkwing serve start \
    --profile ci-team \
    --read-only

The dashboard reads state.ndjson for run metadata, the LogStore for per-node lines, and the ArtifactStore for any blobs the pipeline saved. A profile whose state: surface is S3 streams run and node records to runs/<runID>/state.ndjson continuously (flushed on a sub-second interval, or earlier once enough envelopes buffer), so a dashboard pointed at the same bucket follows the run live. A sqlite state: surface has no live view: its NDJSON dump is written to the cache surface when the pipeline exits.

Fresh laptop, no SQLite (--no-local-store)

The default invocation above still opens ~/.sparkwing/state.db so locally-triggered runs can coexist with the remote ones. On a clean machine that has only the bucket -- new hire, ephemeral container, etc. -- pass --no-local-store to skip SQLite entirely and have the dashboard list runs directly from <artifact-store>/runs/*/state.ndjson:

sparkwing serve start \
    --profile ci-team \
    --no-local-store \
    --read-only

This mode is read-only by construction: the orchestrator's write endpoints (cancel, retry, approvals) are not mounted, since there's no local SQLite to persist to. Passing --no-local-store without both --log-store and --artifact-store (directly or via --profile) errors out -- the dashboard would have nowhere to read from.

S3 layout

<bucket>/<prefix>/
    cache/                                  # ArtifactStore
        runs/<runID>/state.ndjson           # final run + node dump
        <user-keys...>                      # pipeline-saved blobs
    logs/                                   # LogStore
        <runID>/<nodeID>/<seq>.ndjson       # one object per flush

S3 has no native append, so every write is its own object. The logs surface buffers each node's lines and writes one object per flush, which keeps a chatty node at a handful of requests rather than one per line. A flush lands when the buffer reaches batch_bytes, when batch_interval elapses, when a reader asks for the node's log, and when the node finishes. max_log_objects and max_log_bytes bound what one node can write; past either the surface drops further lines and ends the node's log with one marker line counting them. See storage backends for the keys and their defaults. Reads list+concat by prefix.

Exit codes

• 0 if every pipeline node succeeds.
• 1 if any node fails or the orchestrator errors.

The exit code is what the wrapping CI job sees, so a failed sparkwing node fails the CI step.

Caveats

• No webhooks. ci-embedded mode is invoked by the host CI; let GitHub Actions / Buildkite handle the trigger.
• Caching across runs depends on stable bincache.PipelineCacheKey output (sha256 over source + go toolchain). Same source tree on the same Go version = warm cache.
• Worker count vs CPU. GHA hosted runners default to 2 CPUs. --sw-workers=NumCPU (the default) usually fits fine; larger numbers trade memory pressure for less queueing.

Buildkite

steps:
  - label: "release"
    command: |
      sparkwing run release-prod --sw-mode=ci-embedded --sw-workers=4
    plugins:
      - aws-credentials#v1.0:
          role: arn:aws:iam::1234:role/buildkite-sparkwing

State, cache, and logs come from the resolved profile. Declare a profile for the run and pass it with --profile:

# ~/.config/sparkwing/profiles.yaml
profiles:
  buildkite:
    state: { type: s3, bucket: my-team-sparkwing, prefix: state/ }
    cache: { type: s3, bucket: my-team-sparkwing, prefix: cache/ }
    logs:  { type: s3, bucket: my-team-sparkwing, prefix: logs/  }

sparkwing run release-prod --sw-mode=ci-embedded --profile buildkite

GitLab CI

release:
  image: alpine:latest
  before_script:
    - apk add --no-cache curl
    - curl -fsSL https://sparkwing.dev/install.sh | sh
  script:
    - sparkwing run release-prod --sw-mode=ci-embedded --sw-workers=4

Declare a gitlab profile in ~/.config/sparkwing/profiles.yaml and select it with --profile gitlab (same shape as the Buildkite example above).

Related

• The storage interface + filesystem / S3 backends.
• The dashboard's storage-aware reads.