Sparks Libraries

Reference for the sparks library ecosystem: the spark.json manifest, the consumer sparks: block in .sparkwing/sparkwing.yaml, version resolution, and the sparkwing pipeline sparks CLI.

What a sparks library is

A sparks library is a normal Go module that declares itself as a sparks library by placing a spark.json manifest at its module root. It exposes opinionated helpers (ArgoCD sync, ECR auth, Kustomize deploy, language-specific go vet / goimports style checks, etc.) that do not belong in the unopinionated sparkwing SDK. Consumers import its Go packages directly; the manifest and the resolver only layer discoverability, version pinning, and update ergonomics on top of standard Go module mechanics.

There is no plugin binary, no dynamic loader, no runtime injection. A sparks library is code that consumers import and link at pipeline compile time.

Relationship to the SDK

The split is deliberate and stable.

In the SDK (sparkwing/ package): unopinionated, language-and-tooling-agnostic primitives.

• Docker: Build, BuildAndPush, Push, Login, ComputeTags
• Git: ShortCommit, IsDirty, FilesetHash, CurrentBranch, Tags, PushTag
• Services: WithServices (docker-run backed sidecars)
• Approval gates: JobApproval with ApprovalConfig (ApprovalApprove / ApprovalDeny / ApprovalFail expiry policies)
• Plan / modifiers: ExpandFrom, CacheKey, Requires, AwaitPipelineJob, typed Ref[T] outputs

In a sparks library: anything with deep opinions on specific tooling.

• ArgoCD sync, Kustomize patch, deploy.Run composite
• ECR detection, AWS profile discovery, netrc seeding
• Go-specific checks (GoFmt, GoVet, GoTest) - future sparkwing-go library
• Ruby, Python, Java toolchain helpers - future per-language libraries
• Anything that ties a pipeline to a specific registry, control plane, or cloud provider

The rule of thumb: if the helper would make zero sense outside one opinionated stack, it belongs in a sparks library, not the SDK.

spark.json schema

Every sparks library places a spark.json file at its module root. It is valid JSON with the following fields.

packages[] entry schema

Example spark.json

Current-truth reference is sparks-core/spark.json. Abbreviated:

{
  "name": "sparks-core",
  "description": "Core pipeline library for sparkwing - Docker builds, GitOps deploys, AWS helpers, and pre-commit checks",
  "author": "your-github-handle",
  "version": "v0.10.0",
  "sdk_min_version": "v0.9.0",
  "stability": "beta",
  "packages": [
    {
      "path": "docker",
      "description": "Docker build, push, multi-registry tagging with deterministic content hashing"
    },
    {
      "path": "gitops",
      "description": "GitOps deployment with kustomize patching, retry, and ArgoCD sync"
    },
    {
      "path": "aws",
      "description": "AWS profile detection and ECR authentication"
    }
  ]
}

Consumer manifest: the sparks: block

A consumer repo declares the sparks libraries it wants live-tracked under the sparks: key in .sparkwing/sparkwing.yaml. The block is optional - if absent, the pipeline compiles using the exact versions pinned in the consumer's go.mod and no overlay is created.

Schema

# .sparkwing/sparkwing.yaml
sparks:
  - name: <short name>            # must match the library's spark.json "name"
    source: <go module path>      # e.g. github.com/sparkwing-dev/sparks-core
    version: <constraint>         # exact tag, range, or "latest"

Per-entry fields:

Example: exact pins

sparks:
  - name: sparks-core
    source: github.com/sparkwing-dev/sparks-core
    version: v0.10.3
  - name: sparkwing-go
    source: github.com/example/my-sparks
    version: v0.2.1

Deterministic: every build uses exactly these tags. No network call to the module proxy on the hot path.

Example: latest

sparks:
  - name: sparks-core
    source: github.com/sparkwing-dev/sparks-core
    version: latest

Opt-in live tracking: every sparkwing run <pipeline> hits the module proxy to discover the newest non-prerelease tag. Acceptable cost (~100ms per run) given the user opted in. Use --sw-no-update to bypass when offline.

Example: semver ranges

sparks:
  - name: sparks-core
    source: github.com/sparkwing-dev/sparks-core
    version: ^v0.10.0       # any v0.10.x or higher minor within v0.x
  - name: sparkwing-go
    source: github.com/example/my-sparks
    version: ~v0.2.1        # any v0.2.x >= v0.2.1
  - name: sparks-ruby
    source: github.com/sparkwing-dev/sparks-ruby
    version: v0.3.0         # exact

Ranges trade off determinism for ergonomic updates. Resolution picks the highest tag satisfying the constraint at build time.

Resolution and the overlay-modfile pattern

The consumer's go.mod and go.sum are never modified by sparkwing tooling. This is a hard rule. go mod tidy remains the user's authority over what is in their go.mod.

Flow

On every sparkwing run <pipeline> run (and on explicit sparkwing pipeline sparks resolve):

1. If the sparks: block is absent, no-op. Compile with plain go build against the user's go.mod.
2. Otherwise resolve each entry to a concrete version:
   • exact tag: no network call, used as-is
   • range: module-proxy call to list tags, pick highest that matches
   • latest: module-proxy call for newest non-prerelease tag
3. Compare resolution against the current go.mod require lines. If every resolved version already matches what is in go.mod, take the fast path - no overlay is written, compile as normal.
4. Otherwise materialize an overlay modfile at .sparkwing/.resolved.mod (gitignored). It is a copy of the user's go.mod with require lines for drifted sparks libraries rewritten to the resolved versions.
5. Run go mod download -modfile=.sparkwing/.resolved.mod to populate .sparkwing/.resolved.sum.
6. Compile the pipeline with go build -modfile=.sparkwing/.resolved.mod ....

The git-tracked go.mod and go.sum remain pristine; git status after a sparkwing run shows no changes. Consumers who never declare a sparks: block see behavior identical to plain Go builds.

Fast-path skip

When the resolved versions match go.mod exactly (including for latest entries where the module proxy returns the same tag already pinned), no overlay is generated and no overlay-driven go build indirection happens. This keeps the common case - exact pins, or a latest that has not moved - zero-cost beyond the proxy lookup itself.

latest resolution

latest hits the Go module proxy on every run (proxy.golang.org/<module>/@latest). For modules covered by GOPRIVATE, sparkwing falls back to go list -m -json <module>@<query>, which walks the git remote directly and picks the highest semver tag that is not a prerelease. Authentication reuses the same mechanisms as go get: ~/.netrc for HTTPS, SSH keys for ssh://git@....

Cost: ~100ms per latest entry per run. Users who pinned exact tags pay nothing.

GOPROXY and GOPRIVATE

Both latest and semver-range resolution go through proxy.golang.org by default. Modules whose path matches GOPRIVATE (or GONOPROXY) bypass the proxy; for those, sparkwing resolves tags by invoking go list -m -json <module>@<query>, which walks the git remote directly using the user's configured auth. Set GOPROXY=direct to force direct resolution for everything. No separate sparkwing auth flow exists - if go get works, sparks resolution works.

Offline work: --sw-no-update

sparkwing run <pipeline> --sw-no-update skips the resolution step entirely. If a previous overlay exists at .sparkwing/.resolved.mod, it is reused; otherwise compile uses the git-tracked go.mod. Useful on flights, in offline CI, or while debugging a stale pin without touching the network.

Ghost pin guidance

A sparks: overlay MASKS a stale or ghost version in go.mod at build time - the overlay's rewritten require lines take precedence during compile. It does NOT replace normal go.mod hygiene.

go mod tidy remains the authority for what is in go.mod. If the checked-in go.mod pins a tag that does not exist (a "ghost pin"), that is still a repo-level bug - fix it with a real pin. The overlay is a build-time convenience, not a substitute for a correct go.mod.

Interaction with go.work

The Go toolchain refuses -modfile whenever a go.work is in scope ("go: -modfile cannot be used in workspace mode"). The overlay mechanism uses -modfile=.resolved.mod, so the two cannot coexist.

When sparkwing detects a workspace, it skips the overlay and prints a warning to stderr:

warning: /path/to/go.work in effect; skipping sparks resolution.
         Modules resolve from go.mod + workspace, not .resolved.mod.
         To use local copies of sparks libs too, add them to go.work.

Builds resolve sparks libs from go.mod (+ any workspace use directives) instead of .resolved.mod. This is the right call when you are deliberately iterating on multiple modules together: list every repo you are editing in .sparkwing/go.work, e.g.

// .sparkwing/go.work
go 1.26.3
use (
    .
    ../../sparkwing
    ../../sparks-core
)

The workspace then resolves everything from local checkouts, and sparks pinning is suspended for the duration. The pre-push gate refuses to push a committed go.work or go.work.sum, so this stays a local-only convenience -- shipped builds always go through the overlay.

sparkwing pipeline sparks resolve itself refuses to run while a workspace is in scope (the same toolchain limitation applies to go mod download -modfile=X). Remove the workspace file or set GOWORK=off in your shell to refresh pins.

Cache tiers

Compiled pipeline binaries are cached under a PipelineCacheKey that hashes the pipeline source, local replace targets, the resolved sparks versions, and the overlay modfile contents. The same key is used locally (~/.sparkwing/cache/pipelines/<key>/) and in gitcache (/bin/<key>).

Three tiers of cache behavior fall out of that key, each with a rough latency cost. Actual numbers vary by machine, network, and repo size; the values below are order-of-magnitude on a developer laptop.

In-cluster, the Go build cache is persisted in gitcache so that a new sparks version incurs the middle tier rather than the cold tier across worker pods. The build-cache key is derived from Go version, architecture, and sparks versions; it is stored via the existing /cache/<key> gitcache endpoint.

sparkwing pipeline sparks CLI

The sparkwing pipeline sparks command group manages the sparks: block and the overlay. What each subcommand does:

• list -- show the declared sparks libraries and their resolved versions.
• lint -- validate a library's spark.json (schema, required fields, package-path existence).
• resolve -- resolve versions per the sparks: block and materialize the overlay modfile at .sparkwing/.resolved.mod + .resolved.sum. Idempotent, cheap when nothing has drifted, and never touches git-tracked go.mod.
• update -- bump one or all libraries to the latest version within their declared range. Edits the sparks: block only.
• add / remove -- add or remove a library entry in the sparks: block.
• warmup -- pre-compile pipeline binaries across consumer repos after a release (see Warmup below).

For the exact flags each subcommand takes, see cli-reference.md.

Warmup

sparkwing pipeline sparks warmup pre-compiles pipeline binaries across consumer repos after a sparks library release. It resolves the latest versions, compiles each pipeline in the repo, and uploads the binaries to gitcache (pass --clear-cache to discard the existing local binary cache first). The next sparkwing run <pipeline> run - locally or in-cluster - gets a binary-cache hit instead of paying the full compile cost.

Warmup uses the exact same build path as sparkwing, so cache keys match. It is an optimization, not a requirement: pipelines always resolve versions on build, warmup just removes the first-run compile cost after a release.

Most useful as a post-release step in a sparks library's own release pipeline. After tagging and pushing a new version, iterate over consumer repos and warm each:

for repo in repo-a repo-b repo-c; do
    cd ~/code/$repo && sparkwing pipeline sparks warmup
done

Authoring a sparks library

A sparks library is a Go module. Author steps:

1. Create a Go module (normal go mod init <module>).
2. Add spark.json at the module root, filling in the schema above. packages[] must list every user-importable sub-package.
3. Pick a version. Stay on v0.x until the public surface is stable; push through v1.0.0 only when you are ready to commit to the surface under semver-major stability.
4. Tag releases normally (git tag v0.1.0 && git push --tags). The Go module proxy will pick up the tag; sparkwing's resolver reads from there.
5. Never force-push a tag. Force-pushing a module tag breaks the Go module proxy checksum database and cascades into every consumer's go.sum mismatch. Always increment (e.g. v0.1.0 -> v0.1.1), never overwrite.
6. For private repos, ensure GOPRIVATE covers the module path and that consumers can fetch via ~/.netrc (HTTPS) or SSH. Sparkwing does not invent a separate auth flow; it reuses go get's.

Depending on another sparks library

A sparks library can list others in its manifest dependencies. This is informational - actual Go module resolution happens through the library's own go.mod. Declaring a dependency in spark.json lets tooling show the relationship in sparkwing pipeline sparks list and lets future resolver work transitively check compatibility.

Non-goals

Explicit scope limits, baked in to avoid drift:

• No binary plugins. Sparks libraries are Go modules, linked at pipeline compile time. No .so loading, no RPC plugin model, no Wasm runtime.
• No forced updates. Consumers who pin exact versions stay on those versions forever. latest is opt-in per library entry in the sparks: block. Sparkwing never silently bumps a library the consumer did not ask to track.
• No auto-discovery. Consumers explicitly list every sparks library they use in the sparks: block. There is no classpath scan, no go.mod walk to detect libraries by manifest presence, no implicit enrollment.
• No modification of git-tracked files. go.mod, go.sum, and the rest of the repo stay pristine after any sparkwing pipeline sparks * or sparkwing run. Generated files live under .sparkwing/ with names starting .resolved. and are gitignored.
• No cross-module locking. Each consumer resolves independently. There is no workspace-level lock that spans multiple consumer repos.

Cross-references

• sdk.md - the unopinionated SDK that sparks libraries layer helpers on top of.
• pipelines.md - how pipelines are authored against the SDK.
• sparks-core.md - an example sparks library.