# sparkwing -- full documentation (v0.56.0, latest) > Concatenated corpus of every topic at https://sparkwing.dev/docs/. Topics are separated by `========` rules so an agent can grep slug headers. Index: https://sparkwing.dev/llms.txt ======================================== # DOC: admission (v0.56.0) ======================================== # Local admission policy Sparkwing's admission daemon decides when work may consume CPU and memory on one machine. It does not assess job risk. Risk declarations and approvals remain deterministic pipeline contracts and are unaffected by admission mode. The default `classic` mode preserves Sparkwing's admission behavior from before selectable modes. To change the machine-wide policy, create `~/.config/sparkwing/admission.yaml` as an owner-only file and restart the daemon. `sparkwing queue` reports the active mode. ```yaml mode: classic # classic, off, auto, jev, or custom ``` The modes are: - `classic` keeps strict priority and FIFO ordering, permits one opportunistic backfill past a blocked head, then protects that head. It is the default so upgrading Sparkwing does not change admission behavior. - `off` does not gate CPU or memory. The daemon still owns run lifecycle, measurement, queue visibility, and deterministic concurrency groups. - `auto` uses measured duration and resource profiles, weighted workload classes, bounded short-job backfill, and aging. - `jev` starts from `auto` and asks TypeSafe Jev whether a well-measured short candidate should spend additional bounded backfill time. Every hard resource and semaphore check remains in code. Missing credentials, timeouts, malformed responses, and low-confidence answers fall back to `auto`. - `custom` runs the deterministic scheduler with operator-selected weights, aging, and backfill-delay budgets. ## Workload classes Classes express latency sensitivity without becoming absolute priority: `critical`, `interactive`, `normal`, and `batch`. Managed pre-commit and pre-push hooks are interactive; post-commit and scheduled work are batch; other work is normal. Sparkwing never infers critical work. A pipeline can override the inference when it has a stable operational reason: ```go plan.AdmissionClass(sparkwing.AdmissionInteractive) ``` Higher-weight classes are considered more frequently, while aging raises older work until it overtakes newer arrivals. `Plan.Priority` and `--sw-priority` remain explicit strict ordering overrides and are separate from workload class. The orchestrator uses `SPARKWING_ADMISSION_CLASS` to carry its trigger-derived class across the CLI-to-pipeline process boundary. Valid values are `critical`, `interactive`, `normal`, and `batch`; an invalid or empty value is ignored. Pipeline code normally uses `Plan.AdmissionClass` instead, and that explicit choice overrides the inherited environment value. Auto backfill uses measured p99 duration. Unknown-duration jobs receive the single opportunistic backfill retained for first-run liveness, but cannot keep passing a protected waiter. Measured short jobs may continue using spare capacity until the older waiter's delay budget is spent. ## Custom policy Custom fields also tune the deterministic baseline and fallback used by `jev`: ```yaml mode: custom custom: class_weight: critical: 12 interactive: 8 normal: 4 batch: 0 aging_every: 4 backfill_delay: critical: 0s interactive: 2s normal: 10s batch: 30s interactive_burst: cores: 1 max_p99: 2s min_samples: 3 ``` `aging_every` is the number of newer admissions that add one point to an older waiter's class score. Zero disables class-weighted ordering. Omitted values keep the `auto` defaults. The interactive burst is a single CPU-only lane: eligible work must have at least `min_samples`, fit within both `cores` and `max_p99`, and be interactive or explicitly critical. It may exceed CPU reservations by at most `cores`; memory and semaphores remain hard limits, and a second burst waits. ## Jev policy Set `TYPESAFE_API_KEY` in the daemon's server-side environment, then select `jev`: ```yaml mode: jev jev: model: jev-latest timeout: 300ms min_confidence: 0.5 min_probability: 0.7 max_backfill: 5s ``` Jev receives queue resource and duration summaries, pipeline/repository labels, and the candidate's workload class. It does not receive pipeline arguments, logs, source, or secrets. Its answer is a typed choice between waiting and a short backfill. Sparkwing validates confidence and duration bounds, then rechecks the live deterministic ledger before any grant. Queue JSON reports Jev attempt, admit, and fallback counters for comparison with `auto`. ## Stress-testing admission The manual `admission-stress` pipeline supplies repeatable synthetic workloads; it is not part of `gate` or release checks. Its profiles pin the resource dimension under test so each admission mode sees the same charge while the work itself performs real sleeps, hashing, allocation, and page touching. The CPU-heavy and light-sequential profiles intentionally leave memory unpinned so host memory pressure cannot obscure a CPU admission experiment. `light-sequential` pins its complete run so its roughly one-second end-to-end duration becomes the learned admission unit: ```bash sparkwing run admission-stress-light-sequential --class interactive sparkwing run admission-stress-light-parallel --class normal sparkwing run admission-stress-medium-fan-in --class batch sparkwing run admission-stress-cpu-heavy --class batch sparkwing run admission-stress-heavy --class batch sparkwing run admission-stress # the complete phased matrix, class=batch ``` Each workload has a stable pipeline name so its measured history cannot be mixed with a different shape. To exercise duration-aware contention, start one or more `admission-stress-cpu-heavy` runs as batch work, then submit `admission-stress-light-sequential` as interactive work. Use `admission-stress-heavy` to verify that hard memory limits remain intact. Repeat the same sequence after each admission-mode change and compare queue decisions and elapsed time. The profiles are intentionally short and bounded: each heavy node runs for four seconds with two CPU workers; the combined-resource profile has three such nodes using 512 MiB each. ======================================== # DOC: api-reference (v0.56.0) ======================================== # HTTP API reference Every route the controller and logs service register, with the scope each requires, generated from the routing code. All paths are under the `/api/v1` base (webhook and `/metrics` excepted). Scope enforcement and the token model are in [auth.md](auth.md); `admin` is the superset that satisfies any scope check. `public` routes run with no bearer check (the GitHub webhook is HMAC-verified instead); `authenticated` routes take any valid bearer and check no further scope. ## Controller | Method | Path | Scope | |---|---|---| | `GET` | `/api/v1/agents` | `runs.read` | | `PUT` | `/api/v1/agents/{name}` | `admin` | | `POST` | `/api/v1/agents/{name}/heartbeat` | `nodes.claim` | | `GET` | `/api/v1/approvals/pending` | `runs.read` | | `GET` | `/api/v1/artifacts/{key}` | `runs.read` | | `GET` | `/api/v1/auth/bootstrap-needed` | `public` | | `POST` | `/api/v1/auth/login` | `public` | | `POST` | `/api/v1/auth/logout` | `public` | | `GET` | `/api/v1/auth/session` | `public` | | `GET` | `/api/v1/auth/whoami` | `authenticated` | | `GET` | `/api/v1/compute-limits` | `runs.read` | | `PUT` | `/api/v1/compute-limits` | `admin` | | `POST` | `/api/v1/concurrency/{key}/acquire` | `runs.state` | | `POST` | `/api/v1/concurrency/{key}/cancel-waiter` | `admin` | | `POST` | `/api/v1/concurrency/{key}/force-release` | `admin` | | `POST` | `/api/v1/concurrency/{key}/heartbeat` | `runs.state` | | `GET` | `/api/v1/concurrency/{key}/holder` | `runs.state` | | `GET` | `/api/v1/concurrency/{key}/notify` | `runs.read` | | `POST` | `/api/v1/concurrency/{key}/release` | `runs.state` | | `GET` | `/api/v1/concurrency/{key}/resolve` | `runs.state` | | `GET` | `/api/v1/concurrency/{key}/state` | `runs.read` | | `GET` | `/api/v1/credits` | `runs.read` | | `POST` | `/api/v1/credits/grants` | `admin` | | `GET` | `/api/v1/credits/history` | `runs.read` | | `GET` | `/api/v1/credits/settings` | `runs.read` | | `PUT` | `/api/v1/credits/settings` | `admin` | | `GET` | `/api/v1/crons` | `runs.read` | | `DELETE` | `/api/v1/crons/repos` | `runs.write` | | `PUT` | `/api/v1/crons/repos` | `runs.write` | | `GET` | `/api/v1/crons/{id}` | `runs.read` | | `POST` | `/api/v1/crons/{id}/disarm` | `runs.write` | | `DELETE` | `/api/v1/crons/{id}/override` | `runs.write` | | `PUT` | `/api/v1/crons/{id}/override` | `runs.write` | | `POST` | `/api/v1/crons/{id}/pause` | `runs.write` | | `POST` | `/api/v1/crons/{id}/resume` | `runs.write` | | `POST` | `/api/v1/crons/{id}/run` | `runs.write` | | `GET` | `/api/v1/egress` | `admin` | | `POST` | `/api/v1/gitcache/git/register` | `admin` | | `GET` | `/api/v1/gitcache/git/{path...}` | `admin` | | `POST` | `/api/v1/gitcache/git/{path...}` | `admin` | | `POST` | `/api/v1/gitcache/refresh` | `runs.write` | | `POST` | `/api/v1/gitcache/seed` | `admin` | | `GET` | `/api/v1/health` | `public` | | `POST` | `/api/v1/maintenance/reconcile-orphans` | `admin` | | `POST` | `/api/v1/nodes/claim` | `nodes.claim` | | `POST` | `/api/v1/nodes/claim/prepare` | `nodes.claim` | | `GET` | `/api/v1/object-store/breaker` | `admin` | | `POST` | `/api/v1/object-store/reset-breaker` | `admin` | | `GET` | `/api/v1/pipelines/{name}/latest` | `runs.read` | | `GET` | `/api/v1/pipelines/{name}/profile` | `nodes.claim` | | `POST` | `/api/v1/pipelines/{name}/profile/contention` | `runs.state` | | `POST` | `/api/v1/pipelines/{name}/profile/observations` | `runs.state` | | `PUT` | `/api/v1/pipelines/{name}/profile/pin` | `runs.state` | | `POST` | `/api/v1/pipelines/{name}/profile/waits` | `runs.state` | | `GET` | `/api/v1/pool` | `runs.read` | | `POST` | `/api/v1/pool/checkout` | `admin` | | `POST` | `/api/v1/pool/heartbeat` | `admin` | | `POST` | `/api/v1/pool/return` | `admin` | | `GET` | `/api/v1/queue/state` | `runs.read` | | `GET` | `/api/v1/runs` | `runs.read` | | `POST` | `/api/v1/runs` | `runs.state` | | `DELETE` | `/api/v1/runs/{id}` | `admin` | | `GET` | `/api/v1/runs/{id}` | `runs.read` or `nodes.claim` or `triggers.claim` | | `GET` | `/api/v1/runs/{id}/approvals` | `runs.read` | | `GET` | `/api/v1/runs/{id}/approvals/{nodeID}` | `runs.read` | | `POST` | `/api/v1/runs/{id}/approvals/{nodeID}` | `approvals.write` | | `POST` | `/api/v1/runs/{id}/approvals/{nodeID}/request` | `admin` | | `GET` | `/api/v1/runs/{id}/attempts` | `runs.read` | | `POST` | `/api/v1/runs/{id}/cancel` | `runs.write` | | `GET` | `/api/v1/runs/{id}/debug-pauses` | `runs.read` | | `POST` | `/api/v1/runs/{id}/debug-pauses` | `admin` | | `GET` | `/api/v1/runs/{id}/events` | `runs.read` | | `POST` | `/api/v1/runs/{id}/events` | `runs.state` | | `POST` | `/api/v1/runs/{id}/finish` | `runs.state` | | `POST` | `/api/v1/runs/{id}/gitcache/git/register` | `nodes.claim` | | `GET` | `/api/v1/runs/{id}/gitcache/git/{path...}` | `nodes.claim` | | `POST` | `/api/v1/runs/{id}/gitcache/git/{path...}` | `nodes.claim` | | `POST` | `/api/v1/runs/{id}/heartbeat` | `nodes.claim` | | `GET` | `/api/v1/runs/{id}/nodes` | `runs.read` or `nodes.claim` or `triggers.claim` | | `POST` | `/api/v1/runs/{id}/nodes` | `runs.state` | | `GET` | `/api/v1/runs/{id}/nodes/{nodeID}` | `nodes.claim` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/activity` | `nodes.claim` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/annotations` | `nodes.claim` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/artifact-manifest` | `nodes.claim` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/auto-retry/reset` | `runs.state` | | `GET` | `/api/v1/runs/{id}/nodes/{nodeID}/bounce` | `nodes.claim` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/bounce` | `runs.write` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/bounce/consume` | `nodes.claim` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/claim` | `nodes.claim` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/claim/validate` | `logs.write` | | `GET` | `/api/v1/runs/{id}/nodes/{nodeID}/debug-pause` | `runs.read` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/deps` | `runs.state` | | `GET` | `/api/v1/runs/{id}/nodes/{nodeID}/dispatch` | `runs.read` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/dispatch` | `nodes.claim` | | `GET` | `/api/v1/runs/{id}/nodes/{nodeID}/dispatches` | `runs.read` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/execution-finish` | `nodes.claim` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/execution-start` | `nodes.claim` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/finalize-ready` | `runs.state` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/finish` | `runs.state` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/heartbeat` | `nodes.claim` | | `GET` | `/api/v1/runs/{id}/nodes/{nodeID}/logs` | `runs.read` or `logs.read` or `nodes.claim` or `triggers.claim` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/logs` | `runs.state` | | `GET` | `/api/v1/runs/{id}/nodes/{nodeID}/logs/stream` | `runs.read` or `logs.read` or `nodes.claim` or `triggers.claim` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/mark-ready` | `runs.state` | | `GET` | `/api/v1/runs/{id}/nodes/{nodeID}/metrics` | `runs.read` or `nodes.claim` or `triggers.claim` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/metrics` | `nodes.claim` | | `GET` | `/api/v1/runs/{id}/nodes/{nodeID}/output` | `nodes.claim` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/release` | `runs.write` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/revoke-ready` | `runs.state` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/start` | `runs.state` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/status` | `runs.state` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/steps/annotations` | `nodes.claim` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/steps/finish` | `nodes.claim` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/steps/skip` | `nodes.claim` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/steps/start` | `nodes.claim` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/steps/summary` | `nodes.claim` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/summary` | `nodes.claim` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/touch` | `nodes.claim` | | `POST` | `/api/v1/runs/{id}/nodes/{nodeID}/usage` | `nodes.claim` | | `GET` | `/api/v1/runs/{id}/paused` | `runs.read` | | `GET` | `/api/v1/runs/{id}/pending-triggers` | `triggers.read` or `nodes.claim` or `triggers.claim` | | `POST` | `/api/v1/runs/{id}/plan` | `runs.state` | | `GET` | `/api/v1/runs/{id}/receipt` | `runs.read` | | `POST` | `/api/v1/runs/{id}/retry` | `runs.write` | | `GET` | `/api/v1/runs/{id}/steps` | `runs.read` | | `GET` | `/api/v1/secrets` | `admin` | | `POST` | `/api/v1/secrets` | `admin` | | `POST` | `/api/v1/secrets/rotate` | `admin` | | `DELETE` | `/api/v1/secrets/{name}` | `admin` | | `GET` | `/api/v1/secrets/{name}` | `secrets.read` | | `GET` | `/api/v1/services` | `authenticated` | | `GET` | `/api/v1/storage` | `runs.read` | | `PUT` | `/api/v1/storage/quotas/{principal}` | `admin` | | `PUT` | `/api/v1/storage/quotas/{principal}/allowance` | `admin` | | `PUT` | `/api/v1/storage/settings` | `admin` | | `GET` | `/api/v1/tokens` | `admin` | | `POST` | `/api/v1/tokens` | `admin` | | `DELETE` | `/api/v1/tokens/{prefix}` | `admin` | | `GET` | `/api/v1/tokens/{prefix}` | `admin` | | `POST` | `/api/v1/tokens/{prefix}/metered` | `admin` | | `POST` | `/api/v1/tokens/{prefix}/rotate` | `admin` | | `GET` | `/api/v1/trends` | `runs.read` | | `GET` | `/api/v1/triggers` | `triggers.read` | | `POST` | `/api/v1/triggers` | `runs.write` | | `POST` | `/api/v1/triggers/claim` | `triggers.claim` | | `GET` | `/api/v1/triggers/spawned-child` | `triggers.read` | | `GET` | `/api/v1/triggers/{id}` | `triggers.read` or `nodes.claim` or `triggers.claim` | | `POST` | `/api/v1/triggers/{id}/claim` | `triggers.claim` | | `POST` | `/api/v1/triggers/{id}/done` | `triggers.claim` | | `POST` | `/api/v1/triggers/{id}/heartbeat` | `triggers.claim` | | `GET` | `/api/v1/users` | `admin` | | `POST` | `/api/v1/users` | `admin` | | `DELETE` | `/api/v1/users/{name}` | `admin` | | `DELETE` | `/api/v1/webhooks/github/bindings` | `admin` | | `POST` | `/api/v1/webhooks/github/bindings` | `admin` | | `GET` | `/metrics` | `public` | | `POST` | `/webhooks/github/{pipeline}` | `public` | ## Logs service | Method | Path | Scope | |---|---|---| | `GET` | `/api/v1/health` | `public` | | `GET` | `/api/v1/logs/search` | `logs.read` | | `DELETE` | `/api/v1/logs/{runID}` | `logs.write` | | `GET` | `/api/v1/logs/{runID}` | `logs.read` | | `GET` | `/api/v1/logs/{runID}/{nodeID}` | `logs.read` | | `POST` | `/api/v1/logs/{runID}/{nodeID}` | `logs.write` | | `GET` | `/api/v1/logs/{runID}/{nodeID}/stream` | `logs.read` | | `GET` | `/metrics` | `public` | ======================================== # DOC: api (v0.56.0) ======================================== # Controller HTTP API The controller and the logs service expose HTTP APIs under the `/api/v1` base path. The CLI, runners, the dashboard, and pipelines' cross-run refs are all clients. Responses are JSON. The complete route surface -- every method, path, and the scope each requires, for both services -- is generated from the routing code in [api-reference.md](api-reference.md). This page covers the cross-cutting behavior that table doesn't. ## Authentication Requests carry a bearer token, and each route declares the scope it needs; `admin` satisfies any check. Token kinds, the scope set, the unauthenticated endpoints, and first-visit admin bootstrap are in [auth.md](auth.md). ## Webhooks `POST /webhooks/github/{pipeline}` ingests GitHub deliveries. It is verified by HMAC (`X-Hub-Signature-256`) rather than a bearer token, since GitHub can't carry one; the handler acts on `push` and `pull_request` (opened/synchronize/reopened) and answers `ping`. A delivery naming a repository the pipeline is not bound to answers `404`, re-sending a body the controller already accepted answers `409` with the run the first delivery produced, and a delivery with no `X-GitHub-Delivery` header answers `400`. See [security.md](security.md). `POST /api/v1/webhooks/github/bindings` (scope `admin`) stores the secret one repository's deliveries to one pipeline are signed with and allows that repository for the pipeline; `DELETE` on the same path removes it. Stored bindings add to the `GITHUB_WEBHOOK_BINDINGS` document rather than replacing it. `sparkwing cluster webhooks connect` drives both sides; see [hooks.md](hooks.md). ## Logs service Logs live in a separate service keyed by run and node (`/api/v1/logs/{runID}/{nodeID}`), with a whole-run read and an SSE stream for live tail. The routes and their scopes are in [api-reference.md](api-reference.md). ## Run coordination A pipeline binary needs more than node state from whatever holds its runs: it dispatches its own child triggers, and it measures what the run cost so the next run of the same pipeline is priced from evidence. Those reach the controller as routes too -- `/api/v1/runs/{id}/pending-triggers` and `/api/v1/triggers/{id}/claim` for the child-trigger loop, `/api/v1/pipelines/{name}/profile/observations`, `/contention`, and `/waits` for the capacity profile, `/api/v1/runs/{id}/nodes/{nodeID}/usage` for a reaped process's accounting, and `/api/v1/maintenance/reconcile-orphans` for the sweep that closes runs whose orchestrator died. A capacity write names a pipeline rather than a run, so it is bound to a live claim on a run of that pipeline: a node claim for a runner executing one node, or the run's trigger claim for the orchestrator, which records the queue wait before the first node exists and the run's measurement after the last one is gone. ## Concurrency The `.Memoize()` and `.Concurrency()` coordination primitives are backed by the `/api/v1/concurrency/{key}/*` routes (acquire, heartbeat, release, state, resolve). See [caching.md](caching.md) for the model. ======================================== # DOC: architecture (v0.56.0) ======================================== # Architecture **This page describes the production deployment** - the sparkwing stack running in a shared Kubernetes cluster, where webhooks arrive from GitHub, a team looks at a central dashboard, and runners are pooled for work. **For local dev, almost none of this applies.** On a laptop, `sparkwing` compiles and runs your pipeline as a host subprocess and records each run under `~/.sparkwing/`. `sparkwing serve start` spawns a detached local web server (`pkg/localws`, embedded in the CLI); it owns the SQLite store, the log files, and the dashboard on one port (default `http://127.0.0.1:4343`) - no controller pod, no cache, no runner pods, no separate logs service. See [native-mode.md](native-mode.md). The rest of this page is about the in-cluster shape you deploy once per team, not once per developer. --- Sparkwing (prod deployment) is a self-hosted CI/CD platform that runs on Kubernetes. The stack is five pods: a controller, cache, web, runner, and logs. Building container images (Docker-in-Docker) and hosting an image registry, when a pipeline needs them, are external infrastructure the chart does not deploy. ## Components ``` ┌──────────────────────────────────────────────────────────────────┐ │ Kubernetes Cluster │ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ Controller │ │ Cache │ │ Web │ │ │ │ (API + queue │ │ (git HTTP + │ │ (dashboard) │ │ │ │ + webhooks │ │ blob store │ │ │ │ │ │ + pool mgmt)│ │ + pkg proxy│ │ │ │ │ └──────┬───────┘ └──────────────┘ └──────────────┘ │ │ │ │ │ ┌──────┴───────┐ ┌──────────────┐ │ │ │ Runner │ │ Logs │ │ │ │ (warm pool, │ │ (log store) │ │ │ │ polls + │ │ │ │ │ │ claims) │ │ │ │ │ └──────────────┘ └──────────────┘ │ └──────────────────────────────────────────────────────────────────┘ ▲ ▲ │ │ ┌────┴────┐ ┌────┴────┐ │ sparkwing │ │ git │ │ (CLI) │ │ (push) │ └─────────┘ └─────────┘ ``` Five pods: sparkwing-controller, sparkwing-cache, sparkwing-web, sparkwing-runner, and sparkwing-logs. ### Controller The central coordinator. Receives job triggers, queues work, and serves it to runners that poll and claim. - **API server** (port 4344): HTTP endpoints for triggers, run status, agent polling, secrets, and authorization - **Job queue**: in-memory queue with SQLite persistence (`/data/state.db`) for run state, metadata, secrets, and tokens - **Webhooks**: receives GitHub webhook payloads, verifies HMAC signatures, and triggers matching pipelines - **Pool management**: maintains a pool of PVCs pre-loaded with Docker build cache; handles checkout and return for runner jobs - **Run backend**: holds pending runs for runners to poll and claim; the controller does not push work to runners - **Heartbeat monitor**: reclaims a node whose runner stops renewing its lease (default 3-minute lease) - **Queue timeout**: fails pending nodes that exceed their `queue_timeout` (default 15 minutes) - **Metrics collector**: stores the per-node CPU/memory samples runners push as they execute (no cluster metrics-server involved) ### Runner Executes pipeline binaries. A standing warm-pool Deployment runs the unified `sparkwing-runner` binary, which polls the controller and claims pending nodes. For per-node isolation it launches a Kubernetes Job that runs `sparkwing run-node`. The runner downloads code from the cache, compiles and runs the pipeline, and reports results. Off-cluster runners (developer machines, workstations, and servers) connect to the controller and claim nodes through its claim API; the route set and scopes are in [api-reference.md](api-reference.md). ### Cache Git HTTP server, blob store, and package proxy. Mirrors bare repositories from GitHub, refreshing a mirror when a clone reads its refs and at most once per freshness window. Serves git clones over HTTP so runners do not need SSH keys. Also stores: - **Source snapshots**: SHA-scoped Git bundles for unpublished commits and opt-in working-tree triggers - **Artifacts**: job output files - **Binary cache**: compiled pipeline binaries - **Dependency cache**: saved / restored by pipelines (gems, node_modules, etc.) - **Package proxy**: caching reverse proxy for npm, PyPI, Go modules, RubyGems, and Alpine packages See [Cache](gitcache.md) for endpoints and configuration. ### Dashboard Next.js web app showing pipeline runs, logs, node status, and documentation. Its services panel (`GET /api/v1/health/services`) probes the health endpoint of each service it has been given a URL for: the controller and logs service from `--controller` / `--logs`, and the cache from `--cache` (probe-only; omit it and the cache is left off the panel). The `sparkwing-full` chart fills all three in: `web.cache.url` defaults to the runner-bundle's cache Service the same way `web.logs.url` defaults to its logs Service, and a release that deploys no cache starts the web pod without the flag. Services report partial failure in the body while still answering HTTP 200 -- a filling disk, a stalled fetch loop, an unwritable cache directory -- and reserve a 5xx for a total outage. The panel decodes that body, so a service reporting `{"status":"degraded","problems": [...]}` shows amber with its problems listed, not green. Slowness is measured here rather than reported by the service, so a service that is both slow and degraded lists both. `sparkwing configure profiles test` applies the same rule from the CLI, and additionally fails when a health body cannot be read at all: it answers an operator once, where the panel repaints on a cycle. ### DinD (Docker-in-Docker) Optional, external infrastructure the chart does not deploy. When a pipeline builds container images, point runners at a shared Docker daemon; runner jobs connect to it, optionally with a warm PVC mounted for Docker cache. ### Registry Optional, external infrastructure the chart does not deploy. Pipelines that build images push them to a registry you provide - an in-cluster one you run yourself, or an external service (ECR, GCR, Docker Hub, etc.). That is up to the pipeline author. ### Logs Dedicated log storage and streaming service. Runners send step output via HTTP; the dashboard reads live logs via SSE. ## Component Communication All in-cluster communication uses Kubernetes service DNS names. Every component talks over HTTP - there are no custom protocols. ### Who talks to whom ``` sparkwing CLI ──────► Controller trigger a run; poll until terminal GitHub ────────► Controller push webhook (HMAC verified) Controller ────► k8s API warm PVC pool (PVCs, warmer pods) Runner ────────► k8s API create / watch per-node Jobs Runner ────────► Controller claim node; heartbeat; report finish; fetch details Runner ────────► Cache clone repo, download code + packages Runner ────────► Logs stream step output Runner ────────► DinD Docker builds (tcp://localhost:2375) Runner ────────► Registry docker push (localhost:30500) Dashboard ─────► Controller read runs / agents / pipelines Dashboard ─────► Logs live log stream (SSE) Cache ─────────► GitHub git fetch (background, every 30s) sparkwing CLI ──────► Cache refresh or seed an exact Git commit sparkwing CLI ──────► Controller seed/query source through authenticated proxy ``` ### Network policies The charts deploy no NetworkPolicy. On a cluster that runs default-deny ingress, each component needs these allow rules: | Component | Accepts traffic from | |-----------|---------------------| | Controller | External (webhooks), Dashboard, Runners | | Cache | Controller, Runners | | DinD | Runners, Controller (cache warmers) | | Dashboard | External (port 4343) | | Logs | Runners, Dashboard | | Registry | Runners, Nodes (image pulls) | ### Internal service addresses All components discover each other via k8s DNS. No hardcoded IPs. | Service | Internal address | Port | |---------|-----------------|------| | Controller | `sparkwing-controller.sparkwing.svc.cluster.local` | 80 -> 4344 | | Cache | `sparkwing-cache.sparkwing.svc.cluster.local` | 80 -> 8090 | | Logs | `sparkwing-logs.sparkwing.svc.cluster.local` | 80 -> 4345 | | DinD | `dind.sparkwing.svc.cluster.local` | 2375 | | Dashboard | `sparkwing-web.sparkwing.svc.cluster.local` | 80 -> 4343 | | Registry | `registry.registry.svc.cluster.local` | 5000 (NodePort 30500) | ### Environment variables set on runners These are set on every runner pod: | Variable | Purpose | |----------|---------| | `SPARKWING_CONTROLLER_URL` | Controller base URL | | `SPARKWING_LOGS_URL` | Logs service URL | | `SPARKWING_RUN_ID` | The run this node belongs to | | `SPARKWING_NODE_ID` | The node being executed | | `SPARKWING_HOME` | State / cache / logs root | | `SPARKWING_AGENT_TOKEN` | Supervisor bearer for controller + logs calls; assisted job-body children receive a scoped loopback capability instead | ### Environment variables set on a local node process A local run executes each node as its own process, so the same variables above are set on it. A run whose state lives behind the admission daemon also gets `SPARKWING_API_SOCKET` and no `SPARKWING_AGENT_TOKEN`: the node sends its state and concurrency calls down that unix socket and the daemon takes its peer uid as the principal. `SPARKWING_CONTROLLER_URL` is set either way, but on that path it is a placeholder host the socket transport ignores, so a step that dials it reaches nothing. A run that opens the store itself points that variable at a loopback controller the dispatcher mounts for the run and passes that controller's token. More variables describe the process boundary itself. Sparkwing sets all of them; they are not knobs. | Variable | Purpose | |----------|---------| | `SPARKWING_API_SOCKET` | The admission daemon's controller API socket, when the run reaches its state through the daemon | | `SPARKWING_PARENT_LIVENESS_FD` | Descriptor the node reads to notice its dispatcher died, so an abandoned node stops rather than running on against a run nobody owns | | `SPARKWING_RUNNER_NAME` | `local` -- the runner name `Runtime().Runner` reports | | `SPARKWING_RUNNER_TYPE` | `local` -- the runner type `Runtime().Runner` reports | | `SPARKWING_RUNNER_LABELS` | The labels the local runner advertises, comma-separated; what `WhenRunner` matches against | `SPARKWING_PARENT_LIVENESS_FD` in particular should never be set by hand: it is the node's authority to read and close that descriptor, and naming one sparkwing did not open points the node at another subsystem's file. ### Turning off the dev.env fallback A process that resolves a service URL reads it from its own environment first and falls back to the assignment of the same name in `$SPARKWING_HOME/dev.env`, which the local dashboard writes for a development stack. The fallback covers every key it is asked for: `SPARKWING_CONTROLLER_URL` and `SPARKWING_LOGS_URL` for the run-node and trigger paths, and `SPARKWING_CACHE_URL` for the artifact backend. `SPARKWING_DEV_ENV_DISABLE`, holding any value, closes the fallback for all of them, so the process resolves a service URL from its own environment and nowhere else. A test suite is the case that needs it. A suite running inside a node inherits the operator's home, so an unset URL would otherwise resolve to whatever development service that dev.env names and the suite would talk to it. Before every step that starts a product suite, the repository gate exports the variable, pins `SPARKWING_HOME` to a directory of its own, and clears the bindings the node injects: the admission socket, the controller, logs and cache URLs, the agent and lease tokens, the run and node ids, and the parent liveness descriptor. ### Controller API endpoints The controller's full route set, methods, and required scopes are in [api-reference.md](api-reference.md). ## Data Flow ### Local Development ``` sparkwing run build-deploy → compiles .sparkwing/ into a binary → runs the binary locally → pipeline does whatever its code says (build, test, deploy, etc.) ``` ### Remote Execution (pipeline trigger) ``` sparkwing pipeline trigger build-deploy --profile 1. sparkwing resolves the profile -> controller URL 2. sparkwing refreshes or seeds the exact Git commit in the cache 3. sparkwing POSTs the trigger with that commit SHA 4. controller enqueues run 5. a runner polls the controller and claims the run 6. runner clones the exact SHA from cache 7. runner compiles and runs the pipeline binary 8. runner streams logs to logs service 9. runner sends periodic heartbeats to controller to hold its claim 10. runner reports completion to controller 11. sparkwing pipeline trigger follows controller state and displays result ``` `--working-tree` replaces step 2 with a mandatory synthetic-commit bundle seed. The trigger is not admitted if that upload fails. Off-cluster runners can read source through the controller's authenticated Git proxy, so they need only outbound HTTPS; a private direct cache remains an alternative, uses only `SPARKWING_CACHE_TOKEN` for writes, and never receives the controller bearer. Login-enabled dashboard ingress exposes the same machine-bearer proxy path without browser-session authentication. ### Git Push Trigger ``` git push origin main 1. GitHub sends webhook to sparkwing-controller (external) 2. Controller verifies HMAC signature 3. Controller matches push against sparkwing.yaml triggers 4. Controller enqueues matching runs 5. Same execution flow as steps 5-11 above ``` ## Storage | Component | Storage | Contents | |-----------|---------|----------| | Controller | SQLite at `/data/state.db` | Run state, metadata, secrets, tokens, audit log | | Cache | PVC at `/data/` | Bare repos, uploads, artifacts, binary cache, dependency cache, package proxy | | DinD | PVC | Docker layers and build cache | | Logs | PVC at `/data/` | Append-only log files per run | | Registry | PVC | Container images | ## Cluster Setup The Helm chart for the cluster topology lives in this repo under `charts/sparkwing-full`: ```bash helm install sparkwing ./charts/sparkwing-full -n sparkwing --create-namespace ``` Then add a profile pointing at the controller's URL: ```yaml # ~/.config/sparkwing/profiles.yaml profiles: prod: controller: url: https://sparkwing.example.com token: ``` Select it per run with `--profile prod`, or make it the project default by setting `defaults.profile: prod` in `.sparkwing/sparkwing.yaml`. The same pipelines run against any sparkwing controller without changes; only the profile and registries differ. ======================================== # DOC: artifacts (v0.56.0) ======================================== # Node artifacts Artifacts move **files** between nodes. A producer node declares the files it emits; a consumer node declares which producers it draws from, and the orchestrator stages those files into the consumer's workspace before it runs. The transfer is explicit and content-addressed: a consumer never reaches into another node's working directory, and the files it receives are an immutable snapshot of what the producer published. Use artifacts for files. Use [`Ref[T]`](sdk.md) for data values --- a number, a struct, a computed string passed as a typed output. The two are complementary: a node can return a typed `Ref[T]` *and* publish artifacts in the same run. ## The model A producer declares its output files by glob with `Outputs`, relative to its working directory: ```go build := sparkwing.Job(plan, "build", func(ctx context.Context) error { return nil }).Outputs("dist/**") ``` A consumer declares `Consumes(producer)`. That stages the producer's published files into the consumer's workspace before it runs, and implies `Needs(producer)` so the producer is ordered first: ```go sparkwing.Job(plan, "deploy", func(ctx context.Context) error { return nil }).Consumes(build) ``` By default staged files land at the paths the producer declared them under (`dist/...`). Pass `Into` to relocate the whole set under a prefix with the producer's internal structure preserved: ```go sparkwing.Job(plan, "archive", func(ctx context.Context) error { return nil }).Consumes(build, sparkwing.Into("artifacts/build")) ``` `Into` applies to the whole producer; per-file remapping is intentionally absent, since it would couple the consumer to the producer's filenames. Consuming a producer that declared no `Outputs` is a plan-time error --- there is nothing to stage, so the edge is a mistake the plan rejects up front rather than a silent no-op. When one consumer draws two producers whose staged paths overlap, the plan emits a lint warning and the last-staged producer wins at that path. Both modifiers exist at group scope too: `JobGroup.Outputs` declares the same globs on every member, and `JobGroup.Consumes` stages a producer into every member's workspace. See the [SDK reference](sdk-reference.md) for the full signatures. ## Always publish, always stage Publishing and staging are independent of memoization. A node that declares `Outputs` publishes its files every run, whether or not it also declares [`.Memoize()`](caching.md). A consumer stages its declared artifacts every time it runs. A producer that promises outputs it cannot deliver fails: if a declared file is unreadable when the orchestrator captures it, the node fails rather than publishing a partial set. A glob that legitimately matches nothing records an empty set instead --- some outputs are optional, and absence is not failure. Capture stays inside the workspace by path: it resolves each match's symlinks and compares the target against the workspace root. A path a glob names literally fails the node when it resolves outside; a path a wildcard swept up, and any target that resolves to a directory outside the workspace, is skipped with a warning instead, since the pipeline never promised that file. The check reads paths, so it sees symlinks only: a hard link planted in the workspace still publishes the file it points at if the runner can read it. ## Immutable, content-addressed edges Each published file is stored under the digest of its own bytes, so identical files across runs and producers store once. A producer's published set is described by a manifest --- the list of relative paths and their content digests --- stored under the manifest's own digest. The producer node records that one digest. A consumer stages by reading the producer's recorded manifest digest, fetching the manifest, and writing each file's bytes at the recorded path with the recorded permissions. Because the edge is the digest, the files a consumer receives are exactly the bytes the producer published; they cannot drift between publish and stage. This is what lets caching and artifacts compose. A [cache hit](caching.md) replays a producer's typed output without re-running it, and carries the producer's artifact manifest forward unchanged --- so a downstream `Consumes` stages the same files whether the producer ran or hit. Caching a file-producing node is supported: pair `.Memoize()` with `Outputs`. ## Both execution modes Artifacts flow the same way wherever a node runs. In a local run each node process captures and stages against its working directory, which is the machine's own filesystem, so a producer and a consumer on the same machine still exchange files through the store rather than by leaving them where the next node happens to look. In distributed execution each node runs in its own worker pod with a fresh workspace; the worker resolves the shared artifact store and stages the producer's files into the pod before the body runs, then publishes the node's outputs back to the store on success. The producer and consumer never share a filesystem --- the store is the only channel --- which is why the edge has to be content-addressed rather than a path handoff. One local asymmetry survives and is worth naming: a run's node processes do share a working directory, so a producer that writes a file and a consumer that reads it by path still work locally without declaring anything, and still break the first time that pipeline runs in a pod. Declare `Outputs` and `Consumes` for anything that has to travel between jobs. ## Non-goals - **Passing data, not files.** A node's computed value --- a version string, a count, a struct --- travels as a typed [`Ref[T]`](sdk.md), not as an artifact. Artifacts carry files; `Ref[T]` carries data. - **A shared mutable scratch tree.** Artifacts are immutable snapshots handed from one node to another, not a working directory several nodes read and write in place. When steps need a shared mutable tree, keep them as steps within one job --- a job's steps share one workspace. - **Reaching into another node's directory.** A consumer receives only what a producer declared with `Outputs`, staged through the store. There is no path by which a node reads another running node's workspace. ======================================== # DOC: auth (v0.56.0) ======================================== # Authentication + authorization Sparkwing uses a shared-secret bearer token model with typed principals and per-endpoint scope annotations. ## Token format Raw tokens are `_`: - `swu_...` -- user. Created for humans (`sparkwing cluster tokens create --type user`, or `sparkwing cloud connect --admin-token-stdin`, which mints one and writes the profile that holds it). - `swr_...` -- runner. Created for remote machine agents or pool replicas. - `sws_...` -- service. Created for in-cluster back-channel callers. The **prefix segment** is the first 12 characters of a raw token. It's a non-secret identifier used in `sparkwing cluster tokens list`, `revoke`, and audit logs. The remaining ~35 characters carry the secret entropy. ## Metered runners A token carries a `metered` marker the operator sets, either at mint (`sparkwing cluster tokens create --metered`) or afterwards (`sparkwing cluster tokens set-metered --prefix P --metered true`). That marker is the only thing that decides whether the work a runner does costs credits. A claim-mode runner chooses its own labels, so a label saying "cloud" proves nothing and metering never reads one. A claim by a metered token reserves its first minute inside the claim's own transaction. The reservation is what makes the check safe when several runners poll at once: each one's spend is visible to the next before either claim commits, so a balance that covers one minute hands out one node, not one per runner. When the balance cannot cover the reservation, `POST /api/v1/nodes/claim` answers `402` with `"code": "insufficient_credits"`, the node stays ready, the run records a `credits_blocked` event, and the runner keeps polling. The live claim's fenced execution acknowledgement starts billing immediately before the node body runs. Claiming, queueing, provisioning, image pulls and runner startup do not consume the reservation. A heartbeat after execution starts charges the seconds since the previous charge, and the finish charges the tail the last heartbeat missed and refunds whatever is left. A finish or expired claim before execution refunds the complete reservation, so a node that runs for four seconds pays for four seconds. Two bounds apply. No single charge bills more than the charge cap (30 seconds by default), so a controller outage or a stalled heartbeat loop does not bill the gap it left behind. A node that is requeued -- its lease reaped, its runner lost, or its attempt reset for a retry -- releases its charge window, so the next attempt starts a fresh reservation and the idle time between attempts is never billed. Once the balance reaches zero the node keeps running for the grace period. The first heartbeat after that window fails the node with the failure reason `credits_exhausted`, releases its claim, and answers `409`, which is how the runner learns to stop. The run records a `credits_exhausted` event naming the balance and how long it had been spent. A token with no marker is neither checked nor charged, so a deployment that marks none bills nothing. ## Credits Cloud runner time is prepaid. Amounts are stored in micro-credits: a million micro-credits is one credit, and a hundred credits is one dollar, so a ten dollar top-up is a thousand credits. At the default rate a cloud runner second costs 0.02 credits, which is 1.2 credits a minute and 72 credits ($0.72) an hour, so ten dollars buys just under fourteen hours. A second is priced by the node's cpu class. The rate table prices one class per whole-core size, and a node is billed at the class it pinned, which is the class the pod is given. Nothing a claimant says about itself reaches the price, because a runner that priced its own work would bill a 64-core node at the smallest class. A request above the largest class the table prices fails the node with `unpriced_cpu_class` and a `credits_unpriced_class` event naming both sizes, rather than leaving a node no claim can pay for. An installation that never set a table bills the default ladder, which carries GitHub Actions' Linux x64 rates to the second: 2-core 10,000 micro-credits, 4-core 20,000, 8-core 36,667, 16-core 70,000, 32-core 136,667, 64-core 270,000. `credit_rate_micro_per_second` is the four-core entry of that ladder under another name. Once a table exists that setting is derived: a `PUT` that names it, alone or beside `rate_table`, answers `400` and says to write the table. `sparkwing cluster credits settings --rate-table 2=10000,4=20000,8=36667` sets the ladder and needs `admin`. A stored table this build cannot read is an error on every credit read rather than a silent return to the flat rate. The balance is the sum of grants less the sum of charges, computed in SQL over the `credit_grants` and `credit_charges` tables. A grant is `free` or `paid` and records who added it and the payment it came from. A charge is a `reservation` a claim took, the `usage` an interval billed, or the `refund` of a reservation a node did not use; each names the run, node, token prefix, and seconds it covered, and the class and rate it was billed at, so a later change to the table never reprices a charge already written. `sparkwing cluster credits show` prints the balance, the rate table, the charge cap and the last day's burn. `sparkwing cluster credits grant --kind free|paid --amount N` adds credits and needs `admin`. `sparkwing cluster credits history` lists every movement newest first. ## Retained storage Runner time stops costing when a node ends; retained bytes keep costing while they are kept, so they are billed from the same ledger. An installation bills storage only once an operator prices it: `storage_rate_micro_per_gb_day` is what one gibibyte kept for one day costs and `storage_free_allowance_bytes` is what every team keeps unbilled, and both are zero until written, so an installation that sets neither writes no storage charge. What the storage charge bills is run-event payload bytes, which is the one thing the controller durably stores and already measures per team. Artifact content, cache entries and hosted logs are not metered on this release, because the controller never sees their sizes: a runner writes artifact blobs straight to the object store and the controller holds only the manifest digest. A published manifest counts one object against the team's quota and carries no bytes, so it costs nothing here. Each storage pass bills every team holding bytes for the interval since it was last billed. The pass runs on the controller's hourly storage timer, so the meter's error is one pass interval of bytes held and released between two passes, not a whole day of them. A team the ledger has never billed is stamped with the current instant and billed from the next pass, so pricing storage never bills for the past, and so a team's first bytes cost one pass before the meter reaches them. A team that drops to nothing keeps no watermark, and an interval is never billed for longer than the bytes in it have been held, so an idle stretch is not charged against whatever a team stores next. A team whose retained runs all carry no creation date bills nothing for that interval. Each team's watermark moves by compare-and-set, so two controllers on one database bill an interval once whatever either clock says, and a clock that steps backwards bills nothing rather than billing twice. The amount is `bytes x rate x seconds` divided by a gibibyte-day, truncated toward zero, so a fraction of a micro-credit is never billed and truncation forgives at most one micro-credit per team per pass. Three gibibytes retained against a one-gibibyte free allowance for one day is two gibibyte-days, which at 833,333 micro-credits a gibibyte-day is 1,666,666 micro-credits, or 25 credits a gibibyte-month, GitHub's $0.25. The charge is a `storage` row naming the team, the bytes it billed and the interval it covered, so `sparkwing cluster credits show` and `credits history` separate retained bytes from runner time. It carries no cpu class and no per-second rate, because neither priced it. Each team's allowance is how many retained bytes it asked to keep. The pass expires its oldest finished runs above the allowance before it bills, and it never bills for more than the allowance, so the allowance is both what a team keeps and the most it pays for; an allowance of zero keeps everything and caps nothing. Read and write one with `sparkwing cluster credits allowance --principal NAME --gb N`, which needs `admin`. Only that verb writes it: rewriting a team's quota leaves the allowance where it stands. An empty balance is a hard cut, the same as it is for runner time: a write that would grow a team's retained bytes is refused with `402` and a reason naming the team and the balance, and the pass drains retained bytes down to the free allowance. That drain takes only runs whose retention window has already elapsed, so non-payment never removes anything inside the window, and an installation with no retention window drains nothing. The cut is late by up to one storage pass, at most an hour: a balance that reaches zero between passes keeps accepting writes until the next one, and the team's storage quota is what bounds how much can land in the meantime. For a team that held nothing before, the stamp pass comes first, so the cut can take two passes to engage. ## Runner classes A class is a whole number of cores with the memory that comes with it, and it is the unit a pipeline buys. The ladder is 2, 4, 8, 16, 32, and 64 cores, and each class carries 4 GiB of memory for each of its cores: 8 GiB at two cores, 32 GiB at eight, 256 GiB at sixty-four. A node takes the smallest class that covers both halves of what it pinned, so a pin of three cores and 20 GB takes the 8-core class because the 4-core class carries only 16 GiB. The pod is created with cpu and memory requests and limits equal to its class, so a node never outgrows the class it is billed at. The 2-core class runs on the warm pool and starts in seconds. A larger class starts a Kubernetes node of its own, which takes one to two minutes during the preview, and the controller refuses every warm claim and offer for it so it cannot land on a machine it shares. `warm_cpu_class_cores` is the largest class the warm pool serves, 2 by default; zero starts a node of its own for every class. Local claim-mode agents are unmetered and claim by their labels as they always have. Each class above the warm one names the band of machines it runs on. The 4-core and 8-core classes select nodes labeled `sparkwing.dev/cpu-band: small` and tolerate the `sparkwing.dev/cpu-band=small:NoSchedule` taint. The 16-core class and every class above it select and tolerate `large` on the same key, and their pods carry a required anti-affinity on that label across `kubernetes.io/hostname`, so one of them holds a machine alone: the taint by itself does not give it the machine, because two 16-core pods fit one 48-vCPU node. The operator's own node selector and tolerations are merged in and win on this key, so a cluster that pins Jobs its own way keeps doing so. A cluster that serves classes above the warm one needs node pools carrying that label and that taint; without them the pod is unschedulable and the node fails as below, with the scheduler's own message. The whole machine is bought with throughput. Concurrency in the large band is the number of machines the pool's own limit allows, one Job to each, and nothing queues behind it: a pool bounded at three 32-vCPU machines runs three 16-core Jobs at once and fails the fourth after the five-minute wait below, and a class that fills the pool's limit on its own runs one at a time. A claim answers with the class it billed, as `credit_cpu_class_cores` and `credit_cpu_class_memory_bytes`, and the Job is created from those two figures, so the pod shape and the bill agree whichever ladder the operator priced. A claim that names a node carries `sizes_to_class` to say it creates the node's executor at that class; a metered claim without it is held to the warm class, and an unmetered one may not set it at all, so a customer's local agent can never route a class node to itself. The metered token is the operator's own pool, and class routing trusts it: the warm loop and the Job dispatcher share one process and one token, so the controller takes the flag at its word until the Job builder moves server-side and the pod shape is the controller's own. A runner cpu or memory ceiling below the billed class fails the node naming both, because a customer must never be billed for a class the pod cannot get. A pod no node accepts within five minutes fails the node with the scheduler's own message, which is what a class larger than the cluster provisions looks like. A node whose `.Requires()` labels no runner advertises, and which no fallback may take, fails after five minutes with a `node_unmatchable` event naming the labels, the labels the fallback does advertise, and the class, so work the fleet cannot serve ends where an operator can see it. The class is stamped on the node when it becomes ready, so the queue read leaves the classes a warm runner may not take out of the scan entirely and a 2-core node behind thousands of larger ones is still claimed at once. ## Compute guards The guards bound what the controller starts before the ledger bills it. Each is one non-negative integer, and zero is unlimited, so a controller that sets none behaves as it did before the guards existed. | Guard | Bounds | Measured against | |-------|--------|------------------| | `max_concurrent_runners` | cloud runners held at once | one principal | | `max_global_runners` | cloud runners the controller holds | every principal | | `runner_alarm` | cloud runner count that logs a warning, set below the ceiling | every principal | | `max_run_seconds` | wall-clock seconds a run may hold cloud runners for | one run | | `max_nodes_per_run` | nodes a run may carry, which is what bounds a dynamic fan-out | one metered principal's runs | | `max_runs_per_hour` | runs created in the last hour | one metered principal | | `max_global_nodes_per_run` | nodes a run may carry | every run | | `max_global_runs_per_hour` | runs created in the last hour | every run | | `min_cron_interval_seconds` | shortest interval a controller schedule may declare | every controller schedule | | `runner_scale_base` | runners one step of paid credit buys, at most a million; zero uses `max_concurrent_runners` | one principal | | `runner_scale_step_credits` | paid credit that earns one more base, at most a billion; zero turns scaling off | the controller's ledger | | `runner_scale_ceiling` | most a scaled cap may reach, at most a million; zero uses `max_global_runners` | one principal | A cloud runner is a claim a metered token holds, so the runner guards count exactly the work credits pay for. `max_nodes_per_run` and `max_runs_per_hour` are a team's own budget: they measure the principal whose token created the run and apply only while that principal holds a metered token, so local work and unmetered runners pass them untouched. The `max_global_*` pair is the operator's own ceiling and counts every run whichever principal created it. The hourly window counts runs by their creation stamp, so a run that has already finished still occupies the budget until it ages out of the hour. ### Scaling the per-principal runner cap `max_concurrent_runners` scales with what the controller loaded recently, so a customer that has paid for capacity gets it and one that has paid nothing cannot spawn a thousand pods. The cap is the base plus one more base for every `runner_scale_step_credits` of `paid` credit granted in the last 30 days, held under `runner_scale_ceiling`. The base is `runner_scale_base`, or `max_concurrent_runners` when that is zero; the ceiling is `runner_scale_ceiling`, or `max_global_runners` when that is zero. With a base of 100, a step of 5000 credits and 15000 credits loaded, a principal is held to 400 runners. Every scaling setting is zero by default, which holds each principal to the static `max_concurrent_runners`, and the rule applies only while that guard is set. Scaling only ever raises that guard: a ceiling below it is ignored. `runner_scale_base` and `runner_scale_ceiling` are capped at a million runners and `runner_scale_step_credits` at a billion credits, so a typo cannot mint a cap. `free` credit earns nothing and a payment ages out after 30 days. A refund is a `reversal` grant naming the payment's reference, and it is matched to that payment rather than to its own date: a refund settled after the window still takes back the payment that bought the cap, and refunding a payment that has already aged out leaves this month's payments alone. The ledger belongs to the controller and a controller serves one team, so every metered principal on it derives the same cap. The derivation is held for a minute so a claim costs no ledger query, and any grant or reversal retires it at once. A ledger the derivation cannot read holds the principal to the static `max_concurrent_runners` and names the failure in the controller log. `max_global_runners` is checked first, so the controller's own ceiling still refuses a claim a scaled cap would have allowed. `GET /api/v1/compute-limits` reports the result as `usage.derived_runner_cap` with the `usage.recent_paid_micro` it was read from, which is the window's paid grants less the reversals of them, and `sparkwing cluster limits show` prints it as `DERIVED RUNNER CAP`. Work a guard refuses answers `429` with `"code": "compute_limit"` naming the guard, its ceiling and what was measured, and a `Retry-After` saying how soon to ask again. The run records a `compute_limit_blocked` event that `sparkwing runs status` prints on its `guard:` line; a refusal is recorded against a run the refused principal owns, and a guard that names no principal records nothing and reaches the operator through the log. A claim refused this way leaves the node ready and the runner keeps polling. A run that passes `max_run_seconds` loses its node on the next heartbeat: the node fails with the reason `compute_limit`, its claim is released, and the heartbeat answers `409`. `min_cron_interval_seconds` is measured over a schedule's next fires rather than its text, so `*/5 * * * *` and `0,5,10,...` both measure five minutes. It is checked when a repository arms its schedules and again when the tick is about to fire one, so a guard set after arming still binds. `sparkwing cluster limits show` prints every guard with the cloud runners in use, per principal and in total. `sparkwing cluster limits set --name G --value N` sets one guard and needs `admin`; a value of zero removes it. ## Scopes The scope constants live in `pkg/controller/auth.go`; the full route-to-scope mapping is in the generated [api-reference.md](api-reference.md): | Scope | Unlocks | |-------------------|---------------------------------------------------------------------------------------------------| | `runs.read` | GET `/api/v1/runs`, `/runs/{id}`, `/runs/{id}/nodes`, `/runs/{id}/events`, `/trends`, `/agents`, `/queue/state`, `/credits`, `/credits/history`, `/compute-limits`, per-node metrics GETs, and similar deployment-wide reads. `/runs/{id}` alone also admits a `nodes.claim` or `triggers.claim` token holding a live claim on that run | | `runs.write` | POST `/api/v1/triggers`, `/runs/{id}/cancel`, `/runs/{id}/retry`, `/runs/{id}/nodes/{id}/bounce`, `/runs/{id}/nodes/{id}/release`, `/gitcache/refresh` | | `nodes.claim` | POST `/nodes/claim`, `heartbeat`, the per-node write routes, GET claimed node data, GET the claimed run and trigger, and read-only Git proxy routes scoped to a live claimed run | | `logs.read` | GET on logs-service (`/api/v1/logs/*`, `/api/v1/logs/search`) | | `logs.write` | POST + DELETE on logs-service (`/api/v1/logs/{runID}/{nodeID}`, `/api/v1/logs/{runID}`) | | `triggers.read` | GET `/api/v1/triggers`, `/triggers/{id}`, `/triggers/spawned-child`. `/triggers/{id}` alone also admits a `nodes.claim` or `triggers.claim` token holding a live claim on that run | | `triggers.claim` | POST `/api/v1/triggers/claim`, `/triggers/{id}/heartbeat`, `/triggers/{id}/done`, and GET the live claimed trigger and its run. The heartbeat and the done name a trigger, and each is bound to the claimant that trigger's row records | | `runs.state` | POST `/api/v1/runs`, `/runs/{id}/finish`, `/runs/{id}/plan`, `/runs/{id}/nodes`, `/runs/{id}/events`, per-node `start`, `finish`, `deps`, `status`, the offer-round routes `mark-ready`, `revoke-ready`, `finalize-ready`, `auto-retry/reset`, the slot routes `/concurrency/{key}/acquire`, `heartbeat`, `release`, `holder`, `resolve`, and PUT `/pipelines/{name}/profile/pin`. Every write naming a run is bound to a run the caller owns; the pin names a pipeline and is bound to a live claim on a run of it | | `secrets.read` | GET `/api/v1/secrets/{name}`, resolved against the repository of the run the caller holds a claim in | | `approvals.write` | POST `/api/v1/runs/{id}/approvals/{nodeID}` (approve / deny a gate) | | `admin` | tokens / users / secrets CRUD, the token metering marker, credit grants, the compute guards, run delete, gitcache seed, warm-pool checkout / return / heartbeat, and the two cross-run concurrency routes `force-release` and `cancel-waiter` -- see [api-reference.md](api-reference.md) for the per-route mapping | Scope checks are set membership. `admin` is a superset -- any handler's scope check passes if the principal carries `admin`. A runner needs `nodes.claim`, `triggers.claim`, `runs.state`, `secrets.read`, and `logs.write`. That set claims work, drives the run it claimed from plan to finish, reads the secrets its repository owns, and ships logs. It mints no token, reads no user, lists no secret, and cannot read cached source for an unclaimed run. A pool replica that executes already-created nodes still needs `runs.state`, because `start`, `finish`, and event append are its own writes; it can drop `triggers.claim` when a separate dispatcher claims triggers. The set carries neither `runs.read` nor `triggers.read`, and a node process opens with exactly those two reads: `GET /api/v1/runs/{id}` and `GET /api/v1/triggers/{id}`. Both admit a caller holding a live claim on that run in place of the scope, so the claim the runner already took is what opens them. Add `runs.read` only to give a token the deployment-wide view. A warm-pool dispatcher hands nodes to a pool on that same set. It opens and closes each offer round while holding the run's trigger claim, so the readiness routes admit it without an `admin` token. A route can narrow a field below its route scope. The node dispatch reads (`GET /api/v1/runs/{id}/nodes/{nodeID}/dispatch` and `/dispatches`) admit `runs.read`, but fill `env_json` only for an `admin` principal. Every reader still gets `redacted_keys`, the names the snapshot dropped as credentials. ## Claim ownership Scope decides which routes a token may call; the claim decides which node it may write. `POST /api/v1/nodes/claim` binds the claim to the **claiming token**: the controller records that token's prefix segment alongside the principal name and the client-supplied `holder_id`. The prefix is what the gate matches on, because it is unique per token while a principal name is a free-form label two tokens may share; the name stays for display. Afterwards the per-node write routes require that token plus the exact holder, membership, reservation, and claim generation while the lease is unexpired. A missing fence gets `403` with `"error": "claim_required"`; an expired or stale fence gets `409 Conflict`. Another runner token is refused, and `POST /runs/{id}/nodes/{nodeID}/heartbeat` answers `409` unless the token, the principal, and the holder id all match. `admin` bypasses the check, which is what lets a dispatcher mark a node ready, start it, and finish it. The lease is an authorization window, so the claimant does not choose how long it lasts. `lease_secs` above the server cap of 10 minutes is clamped, on the claim and on every heartbeat; a runner renews well inside that. `POST /api/v1/triggers/claim` binds the same way and increments a claim generation. A trigger's id is the id of the run it creates, so trigger-driven node mutations carry that exact generation and are accepted only while the same token holds the live claim for that run. A stale generation gets `409`. `POST /api/v1/triggers` requires more than `runs.write` when it names `parent_run_id`. A node-spawned child carries the exact live claim for `parent_run_id` and `parent_node_id`; a trigger-spawned child carries the exact live trigger generation for `parent_run_id`. Another principal, another token with the same principal name, a stale generation, a different parent node, or no claim cannot attach lineage or inherit repository provenance. `admin` retains its operator override. `GET /api/v1/runs/{id}` accepts `runs.read` or a live `nodes.claim` or `triggers.claim` owner of that run. `GET /api/v1/triggers/{id}` likewise accepts `triggers.read` or either live claim. Claim-scoped access expires with the lease and never widens list routes. Run-definition writes -- run create and finish, plan snapshot, node create, and run-level events -- require the source trigger's exact live generation. Per-node writes accept either the node's exact live claim or that source trigger generation. A run heartbeat likewise accepts one exact live node claim from the run or the source trigger generation. A runner with `runs.state` but without the applicable fence gets `403 claim_required`; a stale generation gets `409 Conflict`. An assisted executor acknowledges its claim generation and next monotonic attempt ordinal immediately before each job-body invocation. Node log appends must carry that started ordinal in addition to the exact claim fence. The logs service validates it against the controller and stores it in an immutable attempt substream. Trigger-owned node logs carry the trigger generation and started attempt ordinal; trigger-generation-only logs are reserved for the coordinator's `_compile` output. Ordinary node reads return executor and attempt attribution but remove holder and reservation values; claim responses still return the fence the winner must present. `PUT /pipelines/{name}/profile/pin` is the one `runs.state` write that names a pipeline instead of a run, and a pin becomes a hard Kubernetes limit for every later run of it. The caller must hold a live claim on some run of that pipeline, so a token executing one pipeline cannot pin another's. `admin` bypasses, which is what lets a dispatcher pin a pipeline it is not running. The two reads a node process opens with, `GET /api/v1/runs/{id}` and `GET /api/v1/triggers/{id}`, run the ownership check the other way: a caller without `runs.read` or `triggers.read` is admitted when it holds a live claim on that run, and refused with `403 missing_scope` otherwise. `mark-ready`, `revoke-ready`, and `finalize-ready` take `runs.state` plus the live claim on the run's trigger, the same gate `auto-retry/reset` carries. A node claim never satisfies them: readiness is a dispatcher decision, and the dispatcher is whoever claimed the trigger. A caller that does not hold that claim gets `403 claim_required`, and a run with no trigger row answers `404`. `admin` bypasses. The slot routes under `/api/v1/concurrency/{key}/` -- `acquire`, `heartbeat`, `release`, `holder` and `resolve` -- take `runs.state` plus a live claim on the run the request names, so a pipeline that declares a concurrency group or a memoized node runs on the runner scope set. `acquire` and `resolve` name their run outright; `heartbeat`, `release` and `holder` name a holder, and the controller reads the run off that holder's row. A caller holding no live claim on that run gets `403 claim_required`, and so does a holder whose lease has already lapsed, because a lapsed row proves nothing about who is calling. The two routes that act on rows another run owns, `force-release` and `cancel-waiter`, stay `admin`. `admin` bypasses all of it. A `nodes.claim` token also reaches only the runs it is working on. The node read routes (`GET nodes/{id}`, `nodes/{id}/output`, `nodes/{id}/bounce`) and `POST /runs/{id}/heartbeat` answer `403 claim_required` unless the caller holds an unexpired claim on some node of that run. `admin` bypasses; so does `runs.read` on the reads, which already grants the wider view through `GET /runs/{id}/nodes`. Node mutations validate the exact fence in the same transaction as the write. If the lease expires or another generation wins while an old executor is paused, the old write cannot land. An append already accepted by the log service remains confined to the old attempt substream rather than entering the replacement's log. The execution view (`GET /api/v1/runs/{id}?include=secret_values`) follows the same rule: it returns plaintext argument values to an `admin` principal, or to a `nodes.claim` principal holding an unexpired claim on one of the run's nodes. A trigger claimant may read the run record but receives redacted secret values until it holds a node claim. A controller serving unauthenticated returns **plaintext**, because the whole API is open in that mode and handing a runner `***` as a real argument value would corrupt the run rather than protect it. Once authentication is on, a request that carries no principal is refused. ## Secret ownership A secret carries an owning repository slug, or none. Store one with `sparkwing secrets set --name DEPLOY_KEY --file ./key --repo acme/web --profile prod`. A secret stored with neither `--repo` nor `--shared` answers `admin` callers only; `--shared` opens an unscoped secret to **every run in the cluster**, so reserve it for values that are genuinely shared, such as a registry pull token. `GET /api/v1/secrets/{name}` resolves differently per principal: - An `admin` principal reads any row. `?repo=` selects a repository's row, `?run=` selects the repository of that run, and without either the unscoped row answers. - A `secrets.read` principal without `admin` cannot name a repository. It names the run it is executing with `?run=`, and the controller answers only when the caller holds that run's claim; the name then resolves against that run's repository, falling back to an unscoped row only when that row is shared. A caller holding no claim reads nothing. A caller holding claims in one repository may omit `?run`; holding claims in two, it must name the run. So one runner token cannot lift another repository's deploy credential by asking for it by name, and a token working two runs cannot read the wrong one's credential by accident. `GET /api/v1/secrets` (the list) and the secret writes stay `admin`; the list carries each row's repository and shared flag. Token creation validates scopes against that same set: a scope the controller does not honor is rejected with a `400` naming the offending scope and the valid set, so a typo fails at mint time instead of producing a token that authenticates and then fails every scope check. A token with no scopes is still legal; it just unlocks nothing. Per-endpoint scope annotations live in `pkg/controller/server.go`. If you add a new route, annotate it with `requireScope`. `GET /api/v1/auth/whoami` is authenticated by the middleware like any other route but carries no scope check, so any valid token can read back its own principal, kind, scopes, and prefix. The logs service uses it to resolve tokens against the controller. It shows as `public` in [api-reference.md](api-reference.md) because that table is generated from `requireScope` wrappers -- there, `public` means no scope check, not no authentication. ## Unauthenticated endpoints Routes registered on the controller's outer router are matched before the auth middleware runs, so they are open regardless of auth config: the health and metrics probes (k8s httpGet probes and Prometheus scrapes can't carry `Authorization`), the service-discovery endpoint the runner uses to find the cache pod, the browser session endpoints the dashboard uses to establish, validate, and end a session (login, logout, session), the bootstrap probe, and the GitHub webhook, which is HMAC-verified instead of bearer-authenticated. The logs service opens its health and metrics probes the same way. Every registered route is listed in [api-reference.md](api-reference.md). With controller-backed dashboard login enabled, the browser authenticates same-origin dashboard requests with its `HttpOnly` session cookie. The dashboard validates that session before its server-side proxy adds the service bearer to an upstream controller request; the service credential never enters browser HTML or JavaScript. CLI and automation clients should authenticate directly to the controller through a profile rather than send a bearer to the browser-facing dashboard proxy. ## Dashboard authorization The dashboard proxies a fixed list of controller routes: the run, node, approval, agent, and trend reads the SPA renders, plus the trigger, cancel, retry, debug-release, approval-resolve, and run-delete writes its buttons issue. A second list covers the logs service and carries reads only, so the browser cannot delete a run's logs or append a forged line through the web pod. Every other path under `/api/v1/` answers `404` at the web pod and never reaches the upstream, so a signed-in tab cannot mint a token, read a secret, or create a user through the proxy. Both lists live in `internal/web/proxy_routes.go`, and a test holds each entry to the scope `pkg/controller/server.go` and `pkg/logs/server.go` register for that route. A browser session carries the scopes of the user who signed in. The proxy checks them against the target route before forwarding, so an account holding only `runs.read` reads runs and gets `403` on cancel. Create narrower accounts with `sparkwing cluster users add --scope runs.read,logs.read`; omitting `--scope` grants `admin`. The first-visit bootstrap account defaults to `admin` and may carry more scopes beside it, but a scope set that omits `admin` is rejected with `400`, and `sparkwing cluster users list` prints the scope set of every account. The web pod's own service token needs `runs.read` plus `logs.read`. Add `runs.write` where the UI cancels, retries, or releases a debug pause, and `approvals.write` where it resolves approval gates. That token bounds what the proxy can reach at all; the session's scopes bound what one signed-in user reaches through it. Deleting a run from the dashboard needs `admin` on both sides, because the controller registers `DELETE /api/v1/runs/{id}` at `admin`: the web pod's token must carry `admin` and so must the signed-in account. Leave `admin` off that token where operators should delete runs with the CLI instead; the dashboard button then reports `delete needs the admin scope` and nothing is removed. `sparkwing-web --require-login` needs a controller session backend. Pass `--controller URL`, or select a `--profile` whose `controller.url` is set. A state-only configuration such as `--state-spec=postgres://... --require-login` now fails at startup instead of silently serving an unauthenticated dashboard. The controller URL must be an absolute `http` or `https` URL without embedded credentials, a query, or a fragment. Every dashboard response carries `Content-Security-Policy` (`default-src 'self'` plus a per-response nonce for the bundle's inline scripts), `X-Frame-Options: DENY`, `X-Content-Type-Options: nosniff`, and `Referrer-Policy: same-origin`, and adds `Strict-Transport-Security` when the request carries evidence of TLS: the listener terminates TLS itself, a peer inside `--trusted-proxy-cidrs` forwarded `X-Forwarded-Proto: https`, or the operator passed `--hsts` because TLS terminates somewhere that forwards no trusted header. That same evidence decides the scheme the CSRF origin check expects, so a dashboard behind an HTTPS proxy keeps `Secure` cookies without the insecure-cookie override. The page reads its configuration from `/sparkwing-runtime.js`, which carries the dashboard version and the login mode. The service bearer stays in the web process and rides only its server-side proxy, so the browser talks to one origin and `connect-src 'self'` holds. A dashboard that carries `--token`, runs without `--require-login`, and binds a non-loopback address refuses to start, because every caller that reaches the listener would drive the controller with that token. Pass `--require-login`, bind a loopback address (chart: `web.addr`), or accept the exposure with `--allow-unauthenticated-remote` (chart: `web.allowUnauthenticatedRemote`). `--token` with no controller, logs, or profile backend is a startup error too: nothing would authenticate with it, so the dashboard would serve unauthenticated while the flag suggested otherwise. Login throttling uses the TCP peer address and ignores forwarded headers by default. When a reverse proxy fronts `sparkwing-web`, pass its egress networks as `--trusted-proxy-cidrs=` or set the chart's `web.trustedProxyCIDRs`. Sparkwing accepts `X-Forwarded-For` only from a trusted peer and walks append-style chains from right to left until it reaches the nearest untrusted address. Values to its left are ignored. A malformed entry in the trusted suffix or an untrusted immediate peer falls back to the TCP peer. IPv4-mapped CIDRs with prefix lengths `/96` through `/128` normalize to IPv4; broader mapped prefixes fail startup. List proxy networks, not client networks. The controller throttles `POST /api/v1/auth/login` the same way and takes the same `--trusted-proxy-cidrs` flag, because it is reachable without going through the dashboard. `sparkwing-web` forwards each browser's resolved address to the controller as `X-Forwarded-For`, so the controller's list must include the web pod's source; otherwise the controller ignores the header and keys every dashboard login on the web pod's own address. See [security.md](security.md#login-and-hashing-budgets) for its budgets, the per-prefix bearer budget, and the argon2 memory bound. The login, first-admin, and logout forms carry a CSRF token in both a `SameSite=Strict` cookie and a hidden field. Sparkwing rejects a missing, cross-origin, or mismatched token with `403` before it calls the controller. Unsafe browser API requests (`POST`, `PUT`, `PATCH`, and `DELETE` under `/api/v1/`) also require a same-origin request whose `X-CSRF-Token` header matches both the browser's `sw_csrf` cookie and the live controller session. The dashboard proxy removes browser cookies and the CSRF header before adding its server-side bearer to controller or logs-service requests. Logout also verifies the token against the live controller session. It clears the browser session only after the controller confirms revocation; a controller failure returns `502` and leaves the cookies in place so the browser does not claim a session was revoked when it was not. The dashboard resolves the controller session on every HTML, data, and API request. Hashed files under `/_next/static/` contain no tenant data and do not touch the session backend. Deleting a session on another web replica or at the controller therefore takes effect on the next protected data request rather than after a local cache expires. A controller `401` authoritatively clears the browser session; a controller outage, `5xx`, or malformed response returns `502` and preserves the cookies so a transient failure cannot log out every user. The controller answers `5xx` when the state store or the session signing key is unreadable, so only an unknown or expired session reaches the browser as `401`. Browser redirects preserve the original path and query as one encoded `next` value and accept only same-origin absolute paths. Login cookies are `Secure` by default, so a login-required dashboard must be served over HTTPS. A plain `http://localhost` port-forward can reach health endpoints but cannot retain those cookies. For a loopback-only development process, `SPARKWING_WEB_INSECURE_COOKIES=1` permits HTTP cookies. The dashboard reads that variable once at startup and refuses a non-loopback bind with it set. That check reads the bind address only: a proxy or sidecar in front of a loopback bind still carries the cookie unencrypted to everything it publishes. An operator who publishes the dashboard over plain HTTP through a proxy or ingress adds `--allow-insecure-cookies-remote` to accept cookies that travel without TLS; the chart renders that flag with the variable whenever `ingress.allowInsecure` opts a TLS-less ingress in. ## First-visit signup Controller authentication is enabled at startup when the tokens table contains an active token. `--require-auth` makes startup fail when it does not, and `--bootstrap-admin-token-file` (`SPARKWING_BOOTSTRAP_ADMIN_TOKEN`) puts the first admin token in that table before the listener binds, so a provisioned controller starts with both satisfied; see the [security operator checklist](security.md#operator-checklist). A freshly-installed sparkwing cluster has no users, so there is nothing to log in *as*. While controller authentication is disabled, browsing to `/login` on an empty cluster renders a "Create first admin" form. Submitting it creates the first admin user via `POST /api/v1/users`, then signs the new admin in automatically. The bootstrap path is one-shot and latched: once any user exists, the controller serves `{"needed": false}` to the probe, the login page reverts to the standard sign-in form, and `POST /api/v1/users` goes back to requiring an admin token. There is no way to reopen the bootstrap path short of restarting the controller against a freshly emptied database. When controller authentication is enabled, the bootstrap probe reports `{"needed": false}` and `POST /api/v1/users` requires an admin token even if the users table is empty. An operator can use that token with `sparkwing cluster users add` to create the first dashboard user. That first account has to be an admin, so leave `--scope` off, or name a list that contains `admin`; a narrower list is refused with `400` while the users table is empty. After the first admin is created, additional users are added via `sparkwing cluster users add`. Pass `--scope` to bound what that account's dashboard sessions reach; omitting it grants `admin`. ## CLI Every `sparkwing` command that talks to a remote controller reads connection info from a profile. Register one first: ```sh # Register a prod profile (controller URL + admin bearer). # --token-stdin prompts without echo on a terminal and reads a pipe otherwise. sparkwing configure profiles add --name prod \ --controller https://sparkwing.example.com \ --token-stdin ``` `--token` accepts the bearer on the command line instead, but every process on the machine can read it from the process list and the shell records it in history. Use it only where a prompt or a pipe is impossible. Then the tokens commands are terse: ```sh # Mint a user admin token. Emits the raw token ONCE. Stash it. sparkwing cluster tokens create --type user --principal alice --scope admin --profile prod # List all active tokens. sparkwing cluster tokens list --profile prod # List including revoked, for audit. sparkwing cluster tokens list --include-revoked --profile prod # Revoke a token by its non-secret prefix. sparkwing cluster tokens revoke --prefix swu_6cF9r2Kp --profile prod # Look up metadata for a prefix. sparkwing cluster tokens lookup --prefix swu_6cF9r2Kp --profile prod # Rotate: mint a replacement, with a grace window before the old one 401s. sparkwing cluster tokens rotate --prefix swu_6cF9r2Kp --grace 48h --profile prod ``` `--grace` is capped at 7 days; a larger value is rejected with `400`. Revoking the old prefix cuts an open grace window short, so a rotation you started before learning the old token leaked can still be stopped. Deleting a user removes the user row, deletes every session that user holds, and revokes every token whose principal is that name, in one transaction. The token the delete request authenticates with is left alone, so an operator whose admin token shares a name with the account being deleted keeps working. Principals are free-form labels, so any other token minted under the same name is revoked too, including one minted for an unrelated caller; keep human account names and service principal names distinct. Profiles are the only path for targeting a remote cluster, which keeps it hard to accidentally point at the wrong one. The `SPARKWING_CONTROLLER_URL` environment variable is a fallback only for the local dashboard dev flow, not for remote-cluster targeting. ## Argon2 parameters Hash parameters (`pkg/store/tokens.go`): - `time = 1` - `memory = 64 MiB` - `threads = 4` - key length = 32 bytes Measured on an arm64 laptop: ~8-15ms per `argon2.IDKey`. Token lookup on the hot path is prefix-indexed + cached in-process for 60s, so argon2 only runs on cold lookups. Concurrent hashing is capped by a memory budget, and a hash that waits more than 250ms for a slot is shed with `503` and a `Retry-After` instead of queueing. Requests that arrive while one token is being verified wait on that verification rather than starting one of their own, so a fleet polling the claim routes costs one hash per token per cache window however many runners poll and however often. `sparkwing_auth_token_cache_total` counts verifications by how the cache answered them (`hit`, `miss`, `coalesced`) and `sparkwing_auth_hashing_rejected_total` counts the hashes the memory budget shed; a climbing rejection count on ordinary polling means the window is too short or the budget too small. A claim or heartbeat that is answered `503` with a `Retry-After` is a load signal, so the runner waits the header out, capped at 30 seconds, and logs it at debug with at most one warning a minute. It does not fail the poll or the node. A claim that finds no work can also name the interval the runner should wait before polling again, in the `X-Sparkwing-Poll-After` header. The controller widens what it suggests with how long it has had no work, up to `--idle-claim-poll` (default 5s), and stops suggesting anything the moment work arrives or is handed out, so a queue that fills returns its fleet to full cadence on the next poll. A runner caps what it accepts at 8 seconds whatever the header says, and the controller refuses to start unless two of the longest wait its suggestion permits, spread included, still fit inside `--placement-hold` and `--placement-liveness`, because a runner silent past those windows stops counting as live for local-first placement. The header is advice a runner may only widen its own cadence to: it never polls faster than it was configured to, it spreads its return with jitter so a fleet advised together does not come back together, and a runner that ignores the header polls exactly as often as it always did. A controller running a limits profile enforces the suggestion instead, answering an early claim `429` with the rest of the wait; see [security.md](security.md#idle-poll-enforcement). A host's own admission daemon and the loopback controller suggest nothing: they serve one machine's runs, where a widened idle poll costs pickup latency and protects no fleet. ## How long revocation takes to bite The verified-token cache holds an answer for 60 seconds, keyed by the token's public prefix and a SHA-256 of the whole credential, so the raw token is never held in controller memory between requests. That window is the outer bound on how long a revocation the replica did not serve takes to bite. Revoking a token, rotating one, and deleting a user all drop the affected prefixes from the controller replica that served the request, so the next request on that replica re-reads the row and gets `401`. A cached entry also carries the row's `expires_at` and `revoked_at`, which are rechecked on every hit, so a token that expires or whose rotation grace closes mid-cache stops authenticating on time rather than at the end of the cache window. An authentication that was already reading the row when the revoke landed does not install its entry, so it cannot put the revoked row back into the cache. Three windows remain: - **Other controller replicas.** Invalidation is in-process. A replica that did not serve the revoke keeps its cached entry for up to 60 seconds. Restart or scale the controller to zero to close it now. - **The loopback controller each run starts.** A local run serves the admin API from the orchestrator process over the same tokens table, behind its own 60-second cache that a controller restart does not reach. It is bound to loopback and exits with the run. - **The logs service.** `sparkwing-logs` resolves callers through the controller's `whoami` and caches the answer for its own TTL (60s by default), on top of whatever the controller replica held. Its worst case is the sum of the two. Sessions carry no cache: the controller reads the `sessions` row on every request and the dashboard resolves the session on every protected request, so deleting a session or a user logs that browser out on its next request. A session expires 12 hours after its last use and the controller renews it when under an hour remains, but never past seven days from the moment it was created. Reaching that age deletes the row and answers `401`, so the browser signs in again. An embedder changes the cap with `controller.Server.WithSessionMaxLifetime`. ## Extension points - **OIDC / SSO**: not implemented. The `users` + `sessions` tables are shape-compatible; an OIDC callback can populate sessions directly by writing `sha256(session id)` into `sessions.hash` and keeping the raw id only in the browser cookie. There is no `csrf_token` column: Sparkwing derives that token per request as an HMAC of the session id under a key in `sparkwing_meta`. - **Audit trail**: the principal name is stamped onto the OTel trace span. There is no dedicated audit database. - **Per-user multi-tenancy**: principals are a free-form label. Adding a roles model is orthogonal and doesn't require a wire-shape change. - **Fine-grained `admin` split**: `triggers.claim`, `runs.state`, and `secrets.read` carved the runner's work out of `admin`. What remains can be split further into `cache.write`, `locks.admin`, and similar when a real caller needs that narrower trust. - **Execution capabilities beyond assisted nodes**: workstation and gateway agents keep their enrollment bearer in the supervisor and give each job-body child a process-lifetime loopback capability for its exact run, node, and acknowledged attempt log/lifecycle. Schema 30 is the internal current-node dependency; schema 31 adds the current-attempt mutation fence and durable grants for `Memoize`, `Concurrency`, `ToolSlot`, `RunAndAwait`, cross-pipeline references, and dynamic `SpawnNode` before this path can ship. Other execution modes retain their documented credential boundary. A future capability service could make the same split portable across container and process boundaries that do not share one supervisor. ======================================== # DOC: authoring-pipelines (v0.56.0) ======================================== # Authoring idiomatic pipelines A pipeline's `Plan` method and each job's `Work` method build the DAG. Plan inspection and execution must produce the same structure. Put I/O and host-dependent decisions in registered job or step callbacks, which execute after dispatch and may repeat on retry. `sparkwing pipeline lint` checks each `Plan` body and the `guards:` blocks in `.sparkwing/sparkwing.yaml`, reports each violation by rule name, and exits non-zero so it can gate a push or a CI job. `sparkwing pipeline lint --rules` prints the live rule set. ## Adding a pipeline Start with the scaffold so the Go registration and YAML catalog entry are created together: ```sh sparkwing pipeline new --name deploy --template minimal ``` `sparkwing.Register` connects a name to its Go implementation in the pipeline binary. `.sparkwing/sparkwing.yaml` defines the repository catalog and its triggers, defaults, and guards. `sparkwing pipeline list` reads that catalog without compiling Go, so a Go registration alone will not appear. When adding a pipeline by hand, pair its registration in `.sparkwing/jobs/`: ```go func init() { sparkwing.Register("deploy", func() sparkwing.Pipeline[sparkwing.NoInputs] { return &Deploy{} }) } ``` with an entry under `pipelines:` in `.sparkwing/sparkwing.yaml`: ```yaml pipelines: - name: deploy entrypoint: Deploy ``` The `name` matches the registration, and `entrypoint` names the Go type. Run `sparkwing pipeline list` to confirm the catalog entry, then `sparkwing pipeline lint` to check the pipeline source and guards. ## Sequencing jobs with `Needs` A multi-job pipeline dispatches in the order its edges require, not the order `Plan` calls `Job`. `Needs` declares that ordering: a job never dispatches until every job it needs has succeeded. ```go type Deploy struct{ sparkwing.Base } func (p *Deploy) Plan(ctx context.Context, plan *sparkwing.Plan, in sparkwing.NoInputs, rc sparkwing.RunContext) error { build := sparkwing.Job(plan, "build", p.build) test := sparkwing.Job(plan, "test", p.test).Needs(build) sparkwing.Job(plan, "deploy", p.deploy).Needs(test) return nil } func (p *Deploy) build(ctx context.Context) error { _, err := sparkwing.Bash(ctx, "go build ./...").Run() return err } func (p *Deploy) test(ctx context.Context) error { _, err := sparkwing.Bash(ctx, "go test ./...").Run() return err } func (p *Deploy) deploy(ctx context.Context) error { return sparkwing.Bash(ctx, "./deploy.sh").MustBeEmpty("deploy failed") } ``` `test` will not dispatch until `build` succeeds, and `deploy` waits on `test` in turn. A job can chain any number of upstream `Needs`; a job with none dispatches as soon as the runner has a slot. When a downstream job needs an upstream job's typed output rather than just its completion, wire a `Ref` and still add the `Needs` edge explicitly (see "Discarded `Ref` results" below) -- `RefTo` does not add the edge for you. ## The `Work` return contract A job with more than one step implements `Workable` instead of passing a plain func to `Job`: it declares a `Work(w *sparkwing.Work) (*sparkwing.WorkStep, error)` method, registers its steps onto `w` via `Step`, and returns. ```go type ExampleDeploy struct{ sparkwing.Base } func (j *ExampleDeploy) Work(w *sparkwing.Work) (*sparkwing.WorkStep, error) { sparkwing.Step(w, "apply", j.apply) return nil, nil } func (j *ExampleDeploy) apply(ctx context.Context) error { return nil } ``` The two return values are the job's typed output step and a Plan-time materialization error. An untyped job -- one that does not embed `Produces[T]` -- has no output to designate, so it returns `nil, nil` once its steps are registered. A typed job returns the step whose value becomes the `Produces[T]` output that `RefTo` exposes downstream: ```go func (j *ExampleBuild) Work(w *sparkwing.Work) (*sparkwing.WorkStep, error) { compile := sparkwing.Step(w, "compile", j.compile) publish := sparkwing.Step(w, "publish", j.publish) publish.Needs(compile) return publish, nil } ``` ## I/O in `Plan` (`plan-io`) A `Plan` body that shells out, touches the filesystem, or makes an HTTP call runs that I/O every time the plan is read. The SDK's side-effect helpers refuse outright: `sparkwing.Bash` / `Exec` / `Shell` and anything in `sparkwing/docker`, `sparkwing/git`, or `sparkwing/services` panic through the runtime plan-guard, naming the call. Plain `os`, `os/exec`, and `net/http` calls have no such guard -- they run silently on every read, and this lint rule catches them. Move I/O, including configuration reads, into a registered job or step callback. `Work` also constructs the graph before dispatch and must remain pure. Don't shell out while the DAG is built: ```go type Release struct{ sparkwing.Base } func (p *Release) Plan(ctx context.Context, plan *sparkwing.Plan, in sparkwing.NoInputs, rc sparkwing.RunContext) error { sha, _ := sparkwing.Bash(ctx, "git rev-parse HEAD").Lines() // runs on every plan read sparkwing.Job(plan, "publish-"+sha[0], p.publish) return nil } func (p *Release) publish(ctx context.Context) error { return nil } ``` Do the I/O inside the registered job callback: ```go type Release struct{ sparkwing.Base } func (p *Release) Plan(ctx context.Context, plan *sparkwing.Plan, in sparkwing.NoInputs, rc sparkwing.RunContext) error { sparkwing.Job(plan, "publish", p.publish) return nil } func (p *Release) publish(ctx context.Context) error { sha, err := sparkwing.Bash(ctx, "git rev-parse HEAD").Lines() if err != nil { return err } return sparkwing.Exec(ctx, "publish", sha[0]).MustBeEmpty("publish failed") } ``` ## Choosing `Bash` versus `Exec` to run a shell command Choose by where the values in the command come from. Use `Exec` whenever an argument is dynamic -- a branch name, an image tag, anything built from a variable. `Exec` runs the argv directly with no shell, so there is no quoting to get wrong and no way for a value containing `$`, backticks, or `;` to be read as shell syntax: ```go tag := "app:" + sha _, err := sparkwing.Exec(ctx, "docker", "push", tag).Run() ``` Reserve `Bash` for a command line that itself needs shell features -- a pipe, a redirect, a glob, a conditional. Pass any dynamic value in through `.Env()` instead of interpolating it into the line, so it never reaches the shell parser: ```go sparkwing.Bash(ctx, `git -C "$R" status --porcelain`).Env("R", repo).MustBeEmpty("dirty tree") ``` Interpolating an untrusted value straight into a `Bash` line is a shell-injection risk; `Exec`, or `Bash` with `.Env()`, avoids it. ## Branching on the runtime environment (`plan-runtime-branch`) `Plan` renders the same DAG wherever it runs, so `explain` and dispatch agree on the shape. Reading `os.Getenv`, switching on `runtime.GOOS`, or calling `IsLocal()` in the body branches the structure on the host that happens to read it. Express the condition where it belongs: a job-level `SkipIf`, evaluated at dispatch, or a pipeline guard that gates the whole run. Don't branch the DAG on the host environment: ```go func (p *Deploy) Plan(ctx context.Context, plan *sparkwing.Plan, in sparkwing.NoInputs, rc sparkwing.RunContext) error { if os.Getenv("ENV") == "prod" { // a different DAG depending on where Plan runs sparkwing.Job(plan, "deploy-prod", p.deployProd) } return nil } ``` Declare the job unconditionally. Its `SkipIf` callback runs on the coordinator after dependencies complete; an environment read there sees the coordinator's environment: ```go func (p *Deploy) Plan(ctx context.Context, plan *sparkwing.Plan, in sparkwing.NoInputs, rc sparkwing.RunContext) error { sparkwing.Job(plan, "deploy-prod", p.deployProd). SkipIf(func(ctx context.Context) bool { return os.Getenv("ENV") != "prod" }) return nil } ``` To gate the whole pipeline instead of one job, use a `guards:` block (see below). ## Runner labels (`runner-label`) The linter rejects blank runner labels on `Requires`, `Prefers`, and `WhenRunner`. An empty string is dropped when labels are normalized, so the term vanishes; a whitespace label survives and matches no runner, so the term can never be satisfied. An `Inline()` job executes on the dispatcher's host, where `Requires` and `Prefers` select no runner; `WhenRunner` still applies there, matched against the inline runner. Avoid blank labels and labels on inline jobs: ```go sparkwing.Job(plan, "build", func(ctx context.Context) error { return nil }).Requires("") sparkwing.Job(plan, "setup", func(ctx context.Context) error { return nil }).Inline().Requires("linux") ``` Do label the job that needs a runner, and leave the inline job to the dispatcher: ```go sparkwing.Job(plan, "build", func(ctx context.Context) error { return nil }).Requires("linux") sparkwing.Job(plan, "setup", func(ctx context.Context) error { return nil }).Inline() ``` ## Discarded `Ref` results (`unused-ref`) A `Ref` is the typed handle a downstream job reads an upstream job's output through. Creating one with `RefTo` and discarding it -- into `_` or as a bare statement -- is dead code: either wire it into a job or drop the producing edge. Don't throw the `Ref` away: ```go build := sparkwing.Job(plan, "build", &Build{}) _ = sparkwing.RefTo[BuildOut](build) // nothing reads this Ref ``` Do wire it into the job that consumes the output: ```go build := sparkwing.Job(plan, "build", &Build{}) out := sparkwing.RefTo[BuildOut](build) sparkwing.Job(plan, "deploy", &Deploy{Build: out}).Needs(build) ``` ## Shared cache across a group (`group-cache-shared`) `JobGroup.Memoize` applies one key function to every member. A constant key makes every member share one result. Give members doing different work distinct keys. Don't cache the group: ```go sparkwing.JobFanOut(plan, "matrix", goVersions, func(v string) (string, any) { return v, &Test{GoVersion: v} }).Memoize(func(ctx context.Context) (sparkwing.CacheKey, error) { return sparkwing.Key("tests"), nil // one key for every Go version }) ``` Do key each member: ```go matrix := sparkwing.JobFanOut(plan, "matrix", goVersions, func(v string) (string, any) { return v, &Test{GoVersion: v} }) for _, member := range matrix.Members() { version := member.ID() member.Memoize(func(ctx context.Context) (sparkwing.CacheKey, error) { return sparkwing.Key("tests", version), nil }) } ``` A key callback returns `(CacheKey, error)`. Return errors when inputs cannot be read; return `sparkwing.NoCache, nil` to bypass memoization. Errors, panics, empty keys, and resolution deadlines fail before dispatch. Use `.CacheDir()` to restore dependency directories before executing a job. ## Configuring a dynamic fan-out (`dynamic-group-inert`) `JobFanOutDynamic` builds its members from an upstream job's output, so the group is empty while `Plan()` runs. Every `JobGroup` setter applies to the members present when it is called, which on a dynamic group is none of them: the call compiles, reads as configuration, and is dropped. Don't configure the group: ```go shards := sparkwing.Job(plan, "discover", &Discover{}) sparkwing.JobFanOutDynamic(plan, "bench", shards, func(s Shard) (string, any) { return s.Name, &Bench{Shard: s} }).Requires("gpu").Retry(2) // neither reaches a generated job ``` Do configure the jobs the callback returns. `Requires`, `Prefers`, and `WhenRunner` have provider interfaces a `Workable` implements, and each generated instance can answer from its own data: ```go type Bench struct { sparkwing.Base Shard Shard } func (b Bench) Requires() []string { return []string{b.Shard.Runner} } ``` The group itself stays useful as a dependency target: `Needs(group)` on a downstream job waits for every generated member. ## Declarative trigger filters and run guards The `branches`, `paths`, and `actions` fields under `on.push` and `on.pull_request` record intent. The controller dispatches the pipeline named by the webhook URL without reading those fields. Use a pipeline guard when the policy depends on the branch Sparkwing checks out. `require: [git:branch=main]` blocks a run from any other checked-out branch before a step starts. The literal name matches the head branch; it does not match a pull request's base branch. `git:branch=default` matches only when the dispatch supplies default-branch metadata, which controller webhook and local trigger claims do not. These guards do not implement path filters, custom pull-request actions, or pull-request base-branch matching; [Triggers](hooks.md) describes that boundary. ## Unsatisfiable guards (`guard-misuse`) A pipeline's `guards:` block gates dispatch on the resolved profile, args, and git branch -- `profile:local` / `profile:controller` / `profile:name=NAME`, `arg:FLAG=VALUE`, and `git:branch=NAME` / `git:branch=default`. `require` blocks the run when not every token matches; `reject` blocks it when any token matches. A token in both lists or a `require` naming two mutually exclusive profiles prevents dispatch. A duplicate token is redundant. The linter reports both kinds of defect. The `default` token matches only when the dispatch supplies default-branch metadata; use a literal branch for controller webhook and local trigger claims. Don't write guards that can never all hold: ```yaml # .sparkwing/sparkwing.yaml pipelines: - name: deploy entrypoint: Deploy guards: require: [profile:local, profile:controller] # mutually exclusive reject: [profile:controller] # also rejected -> contradiction ``` Do pick tokens that can be satisfied together: ```yaml # .sparkwing/sparkwing.yaml pipelines: - name: deploy entrypoint: Deploy guards: require: [profile:controller] # run only against a controller profile reject: [git:branch=main] # never from the checked-out main branch ``` ## Running the linter ``` sparkwing pipeline lint --all # every pipeline in the repo sparkwing pipeline lint --name deploy # one pipeline by name sparkwing pipeline lint --rules # print the rule charters ``` Add `-o json` for machine-readable findings. Point `--dir` at a source tree other than the convention (`.sparkwing/jobs`). A non-zero exit on any finding makes the command a drop-in gate for a pre-push hook or a CI job. ======================================== # DOC: backends (v0.56.0) ======================================== # Storage backends Backends are configured per profile, not in a separate file. A profile declares four persistence surfaces plus how to reach a controller: - **state** -- run records, plan snapshots, status - **cache** -- content-addressed artifacts and compiled pipeline binaries - **logs** -- per-job log streams - **secrets** -- where `sparkwing.Secret` values resolve from A profile fully describes "where do my runs go and what auth do I need to get there." The same pipeline source runs on a laptop with the filesystem, in CI with S3, or against a self-hosted controller -- you switch by selecting a profile, not by editing a backends file. Laptop profiles live in `~/.config/sparkwing/profiles.yaml`; project profiles in `.sparkwing/sparkwing.yaml` (see [config-reference.md](config-reference.md)). ```yaml # ~/.config/sparkwing/profiles.yaml profiles: laptop: state: { type: sqlite } cache: { type: filesystem, path: ~/.cache/sparkwing } logs: { type: filesystem, path: ~/.cache/sparkwing/logs } shared-team: state: { type: s3, bucket: team, prefix: state } cache: { type: s3, bucket: team, prefix: cache } logs: { type: s3, bucket: team, prefix: logs } prod: controller: { url: https://api.example.dev, token: swu_xxx } # state/cache/logs are implied by the controller; reads/writes go through it. ``` Select a profile with `--profile NAME`; it applies wholesale. Without `--profile`, the project's `defaults.profile` in `.sparkwing/sparkwing.yaml` applies, falling back to the built-in local (sqlite + filesystem) defaults. `sparkwing profile` prints which profile resolved and why. ## Backend types | Surface | Types | Use | | --- | --- | --- | | `state` | `sqlite`, `postgres`, `s3`, `gcs`, `azure-blob`, `controller` | Run records, plan snapshots, status | | `cache` | `filesystem`, `s3`, `gcs`, `azure-blob`, `controller` | Content-addressed artifact and compiled-binary store | | `logs` | `filesystem`, `s3`, `gcs`, `azure-blob`, `controller`, `stdout` | Per-job log stream persistence | ### Object-store log batching An object store charges per request, so the `s3` logs surface buffers each node's lines and writes one object per flush rather than one per line. Four optional keys tune it; every one defaults to a value that suits a chatty CI node, and a profile that names none behaves the same as one that names the defaults. | Key | Default | Meaning | | --- | --- | --- | | `batch_interval` | `2s` | Longest a buffered line waits before its object is written | | `batch_bytes` | `262144` | Buffer size that triggers an early flush | | `max_log_objects` | `2000` | Objects one node's log may cost | | `max_log_bytes` | `67108864` | Bytes one node's log may hold | ```yaml profiles: team: logs: type: s3 bucket: my-team-sparkwing prefix: logs/ batch_interval: 5s batch_bytes: 524288 ``` A flush also lands when a reader asks for the node's log, and when the node finishes. The finish flush runs before the node's status is written, on the failing and cancelled paths as well as the succeeding one, so a node whose status reads terminal has a complete log. Past `max_log_objects` or `max_log_bytes` the surface drops further lines and ends that node's log with one marker line counting them. A flush the object store refuses loses its whole batch; those lines are counted into the node's dropped-line total and reported as a `logs_drop` event. The keys are valid only on an object-store logs surface. A `filesystem` surface appends to an open file and a `controller` surface takes a streaming append, so neither reads them, and a profile that sets one anywhere else is refused at load with the key named. A negative value is refused the same way. Each state backend is one deployment shape. See [Deployment modes](deployment-modes.md) for when to pick each: - `sqlite` -- the local path; the default when no profile is selected. - `s3`, `gcs`, `azure-blob` -- per-run NDJSON state on a shared bucket. Cache reservation, approvals, and debug pauses coordinate over object-store CAS where the bucket enforces write preconditions (S3 today; `gcs`/`azure-blob` recognized but not yet implemented). Where it does not, cache reservation degrades to last-write-wins, while approvals and debug pauses report not-supported and need Postgres. Pipeline triggers report not-supported here whatever the bucket does: the backend enqueues a trigger and has no path that claims one, so `sparkwing.RunAndAwait` refuses instead of waiting. - `postgres` -- shared database for cross-runner coordination. Triggers, approvals, debug pauses all work. - `controller` -- runners talk to a hosted controller over HTTP, Sparkwing Cloud included. The controller owns the underlying database. `mysql` is reserved in the schema but not implemented; declaring it fails at run start with a clear error. Local execution is process-per-node under every state backend. A node body runs in its own process of the pipeline binary and reaches run state through a controller the dispatcher mounts on loopback for the run: the full controller when state is a local SQLite database, and the node-facing subset of the same API over whatever else the profile named -- object-store state included. Nothing local executes inside the dispatcher's own process, so a bucket-backed CI run and a laptop run behave the same way. One measurement does not follow. The measured pipeline profiles that size admission are folded from the local run store, so only `sqlite` state feeds them; a bucket-backed run records its per-node metric samples in the bucket but folds no profile and stores no exit accounting, exactly as it did before. Point `state` at `sqlite` (or a controller) on the machine whose capacity you want learned. Required fields per type: - `filesystem` -- `path` - `s3`, `gcs`, `azure-blob` -- `bucket` (plus optional `prefix`) - `postgres`, `mysql` -- exactly one of `url` or `url_source` (the latter names a secret in the resolved source) - `controller` requires `controller: ` or `url:`. A profile with a sibling `controller:` block inherits that profile name. - `stdout`, `sqlite` -- no required fields Recognized backend types that aren't implemented in the current build surface a clear error at run start ("type X is recognized but not implemented in this build") instead of silently falling back. The fourth surface, `secrets`, names where `sparkwing.Secret` values resolve from (laptop dotenv or controller-stored); see [security.md](security.md). ## Per-pipeline backend selection A pipeline pins its backends by pointing at a profile that declares them. Put the profile in a `profiles:` entry and set `profile:` on the pipeline; that profile then applies to its runs (typically for an audit requirement). Project profiles in `.sparkwing/sparkwing.yaml` are validated on load and must declare all four surfaces -- secrets, state, cache, and logs -- even when only one differs from the shared backends (laptop `profiles.yaml` entries are not validated this way): ```yaml # .sparkwing/sparkwing.yaml profiles: prod-audit: secrets: { type: env } state: { type: s3, bucket: prod, prefix: state } cache: { type: s3, bucket: prod, prefix: cache } logs: { type: s3, bucket: prod-audit-logs, prefix: "${RUN_ID}/" } pipelines: - name: release-prod entrypoint: Release profile: prod-audit ``` The selected profile applies wholesale -- the pipeline's `profile:` when set, otherwise the project's `defaults.profile`. Project defaults are not layered in per surface; the chosen profile's own surfaces are what apply. Any surface the chosen profile leaves unset falls back to the built-in local default (sqlite state, no shared cache or logs), not to another profile. ## Pipeline binary distribution Compiled pipeline binaries live in the cache surface under `bin/`. On a cache hit, the orchestrator fetches and execs without recompiling. An optional `cache.binaries` sub-spec isolates binaries to a separate destination: ```yaml profiles: shared-team: cache: type: filesystem path: ~/.cache/sparkwing binaries: type: s3 bucket: sparkwing-binaries prefix: "${PIPELINE_NAME}/" ``` A pipeline compile reads `bin/` from the sub-spec when one is declared and from the cache surface otherwise. The publish command writes that same destination when `--profile` names this profile, so what it uploads is what a later run finds; its `--artifact-store` URL names a destination outside any profile. Only one level is read: a `binaries` block inside a `binaries` block is ignored. ## Migrating from `backends.yaml` For the before/after of moving `backends.yaml` `defaults:` and `environments:` into per-profile specs, see the [v0.5.0 migration guide](migrations/v0.5.0.md#profiles-absorb-all-backend-specs). ======================================== # DOC: build-caching (v0.56.0) ======================================== # Build Caching How sparkwing makes Docker builds fast, and where the time actually goes. ## Where build time goes Benchmark: full-stack Dockerfile installing 225 npm packages + 100 Ruby gems (including native extensions like pg, nio4r, bootsnap). Tested on Apple Silicon Mac and EKS (Graviton arm64). ### Local Mac results | Scenario | Time | What's cached | |---|---|---| | True cold (system prune) | 94s | nothing | | Base images cached | 92s | container images | | Images + warm cache mounts | 61s | images + compiled deps | | Full layer cache (unchanged) | 1s | everything | ### Derived cost breakdown | Component | Cost | Notes | |---|---|---| | Base image pulls | ~2s | Docker Hub CDN is fast | | Package downloads (npm + gems) | ~10–15s | registry CDNs are fast on good networks | | Native extension compilation | ~40–50s | gcc/make for pg, nio4r, bootsnap, sassc... | | npm resolution + linking | ~10–15s | CPU-bound, not network | | Docker layer export | ~5–7s | writing to image store | **The bottleneck is CPU, not network.** Compiling native extensions accounts for ~50% of a cold build. Package downloads are only ~15% of the total. ### EKS results (Graviton arm64, 2 vCPU) | Scenario | Time | Savings | |---|---|---| | Cold build | 105s | -- | | Layer cache (nothing changed) | 0s | 105s (100%) | | --no-cache, cache mounts warm | 101s | 4s (4%) | | Dep change + warm cache mounts | 101s | 2s vs cold dep change | | Cold build + proxy (cached) | 98s | 7s (7%) | The EKS build is 1.7x slower than the Mac, primarily from CPU limits (2 vCPU pod) and EBS disk I/O (network-attached storage). ### EKS time breakdown (~105s) | Component | Time | % of build | |---|---|---| | `bundle install` (download + compile) | ~76s | 72% | | `npm install` (resolve + link) | ~15s | 14% | | Docker layer export | ~13s | 12% | | Base image + setup | ~2s | 2% | Native extension compilation (pg, nio4r, bootsnap, sassc) dominates. No caching strategy can skip compilation -- only layer cache (unchanged rebuild) or pre-compiled base images avoid it. ## Caching layers -- what each one does Sparkwing has four caching layers. Each addresses a different failure mode: ### 1. Docker layer cache (biggest win: ~99% speedup) When nothing in the Dockerfile changes, every layer is cached and the build completes in ~1s. This is Docker's default behavior -- no sparkwing configuration needed. **Breaks when:** any file referenced by `COPY` changes (code, package.json, Gemfile), Dockerfile changes, or build args change. ### 2. BuildKit cache mounts (second biggest: ~34% speedup) ```dockerfile RUN --mount=type=cache,target=/root/.npm npm install RUN --mount=type=cache,target=/usr/local/bundle bundle install ``` Cache mounts persist compiled artifacts and downloaded packages across builds even when the layer cache is busted. The mount directory survives `--no-cache` and Dockerfile changes -- only `docker builder prune -af` clears it. **Benchmark proof:** - Images cached, cold mounts: 92s - Images cached, warm mounts: 61s - **Savings: 31s (34%)** The savings come from skipping native extension recompilation (pg, nio4r, etc.), not from skipping downloads. **Breaks when:** the base image's runtime version changes (Ruby 3.3 → 3.2 compiled extensions are incompatible), or build cache is pruned. ### 3. Warm PVC pool (multiplier for cache mounts) The controller pre-warms PVCs with Docker image layers. The DinD sidecar on each runner pod mounts a warm PVC at `/var/lib/docker`. Since the warmer is additive (never wipes the PVC), BuildKit cache mounts from previous job runs persist on the PVC. This means cache mounts survive across pipeline runs -- not just within a single build session. The first build on a PVC is cold; every subsequent build benefits from warm mounts. **Breaks when:** the PVC is recycled (new PVC from the pool), or the warmer is run with a destructive reset (it currently doesn't -- see `warmer.go`). ### 4. Dependency proxy (reliability + bandwidth, modest speed) sparkwing-cache includes a package proxy that caches npm, pip, gem, Go module, and Alpine package downloads in-cluster. Runners fetch packages from the cache proxy instead of the public internet when the pipeline points the package manager at it -- there is no automatic interception; see "For using the proxy in Dockerfiles" below for the build arg that wires it up. **Speed impact:** ~4s savings on a 104s EKS build. The proxy eliminates network egress for cached packages, but since package downloads are only ~15% of total build time, the absolute savings are small on fast networks. **Where the proxy matters:** - **Reliability:** builds succeed when npmjs.org or rubygems.org have outages (stale-on-error fallback serves cached responses) - **Bandwidth:** 290MB of cached packages not re-downloaded from the internet on every cold build, across every node - **Constrained networks:** air-gapped clusters, metered egress, cross-region builds where registry latency is high - **Concurrent builds:** 10 runners building simultaneously don't each fetch the same 200MB from upstream **Does NOT help when:** the bottleneck is compilation (most builds), the network is fast (AWS to npm CDN), or the packages aren't in the cache yet (first build). ## Recommendations ### For pipeline authors (Dockerfile best practices) Always use BuildKit cache mounts for package managers: ```dockerfile # Node RUN --mount=type=cache,target=/root/.npm npm ci # Ruby RUN --mount=type=cache,target=/usr/local/bundle bundle install # Python RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt # Go RUN --mount=type=cache,target=/go/pkg/mod \ --mount=type=cache,target=/root/.cache/go-build \ go build ./... # Rust RUN --mount=type=cache,target=/usr/local/cargo/registry \ --mount=type=cache,target=/app/target \ cargo build --release ``` ### For using the proxy in Dockerfiles ```dockerfile ARG PROXY_URL="" # npm RUN --mount=type=cache,target=/root/.npm \ if [ -n "$PROXY_URL" ]; then npm config set registry ${PROXY_URL}/proxy/npm/; fi && \ npm ci # pip RUN --mount=type=cache,target=/root/.cache/pip \ pip install --index-url ${PROXY_URL:-https://pypi.org}/proxy/pypi/simple/ \ --trusted-host sparkwing-cache.sparkwing.svc.cluster.local \ -r requirements.txt # apk (Alpine) RUN if [ -n "$PROXY_URL" ]; then \ sed -i "s|https://dl-cdn.alpinelinux.org|${PROXY_URL}/proxy/alpine|g" /etc/apk/repositories; \ fi && apk add --no-cache git ``` `PROXY_URL` is a build arg your pipeline sets -- it is not injected automatically. Default it to empty (no proxy) and, for in-cluster builds, pass the cache's service URL yourself, e.g. `--build-arg PROXY_URL=http://sparkwing-cache.sparkwing.svc.cluster.local` when invoking the Docker build from your pipeline. ### What NOT to optimize (and why) These were all investigated and benchmarked. The savings are real but small because **builds are CPU-bound, not network-bound**. - **Base image pull time** -- only ~2s on fast networks. The warm PVC pool already pre-pulls common images. - **Package download caching (proxy/mounts)** -- saves 2–7s on a 105s build. Downloads are ~15% of total time; compilation is ~72%. The proxy's value is reliability (builds work when registries are down) and bandwidth savings, not speed. - **P2P artifact distribution** -- overkill at current scale. The dependency access pattern (many small files, different per-repo) doesn't benefit from peer-to-peer the way large uniform blobs do. ### What WOULD help - **More CPU for runner pods** -- compilation is the bottleneck. Bumping from 2 to 4 vCPU would let `bundle install --jobs=4` actually parallelize and could cut ~30% off build time. - **Faster disk** -- EBS adds ~6s to Docker layer exports vs local SSD. Local NVMe instances (c5d, m5d) or higher IOPS gp3 would help. - **Pre-compiled base images** -- a custom base image with common gems pre-installed eliminates compilation entirely for those deps. This is the nuclear option: build time drops to seconds, but you own the base image. ## Proxy service The package proxy is part of sparkwing-cache. It fronts these upstream registries: | Registry | Upstream | URL rewriting | |---|---|---| | npm | registry.npmjs.org | Yes (tarball URLs in metadata) | | pypi | pypi.org | Yes (file URLs in simple index) | | pythonhosted | files.pythonhosted.org | No | | rubygems | rubygems.org | No | | golang | proxy.golang.org | No | | alpine | dl-cdn.alpinelinux.org | No | **URL rewriting:** npm packuments and PyPI simple pages carry absolute upstream URLs, so the proxy rewrites them onto itself. It rewrites against `SPARKWING_CACHE_PUBLIC_URL` when that is set, and caches the rewritten body. Unset, the cached copy stays exactly as upstream sent it and each response is rewritten from its own request's `Host` header, so a caller who sends a forged `Host` only ever changes its own response. A per-request response carries `Cache-Control: private, max-age=0` and `Vary: Host`, which keeps a shared intermediary from handing one client's body to another; a response rewritten against the public URL is host-independent and stays `Cache-Control: public` for the entry's remaining TTL. A fixed base is correct only when every client dials the cache at the same address, so set it exactly then. The runner-bundle chart does it for you through `cache.publicUrl` when `cache.service.type` is `ClusterIP`. On a `LoadBalancer` or `NodePort` Service -- or through `kubectl port-forward` -- clients reach the cache at more than one address, so the chart leaves the value empty and the proxy rewrites per request instead. An off-cluster runner pool that does share one address can set `cache.publicUrl` itself. Changing the public URL (setting it, clearing it, or pointing it elsewhere) leaves already-cached mutable entries written against the old value. They keep being served that way until their TTL expires -- `PROXY_CACHE_TTL`, 10 minutes by default -- so plan for that window, or wipe `PROXY_CACHE_DIR` to end it at once. Immutable entries carry no upstream URLs and are unaffected. **Cache policy:** - Immutable content (.tgz, .whl, .gem, .zip, .jar, .apk): cached until the max-age cleanup threshold (default 168h / 7 days) - Metadata (JSON, HTML): 10-minute TTL, stale-on-error fallback - Background cleanup: removes expired entries hourly **Endpoints:** - `GET /proxy/{registry}/{path}` -- cached reverse proxy - `GET /stats` -- cache size per registry - `GET /health` -- liveness/readiness **Configuration (env vars):** - `PROXY_CACHE_DIR` -- cache directory (default: `/data/proxy`) - `PROXY_CACHE_TTL` -- metadata TTL (default: `10m`) - `PROXY_MAX_AGE` -- cleanup threshold for immutable entries (default: `168h`) - `SPARKWING_CACHE_PUBLIC_URL` (`--public-url`) -- base URL clients use to reach the proxy, e.g. `http://sparkwing-cache.sparkwing.svc.cluster.local`. A scheme and host, optionally with a `/proxy` path; any other path, a query, or a fragment fails startup with the offending value named. Empty rewrites per request from the `Host` header (default) - `SPARKWING_CACHE_TRUST_FORWARDED_HOST` (`--trust-forwarded-host`) -- honor `X-Forwarded-Host` and `X-Forwarded-Proto` when rewriting per request. Only set it when a reverse proxy is the only route to the port (default: off). Proxies append, so the right-most element wins and it has to parse as a host with an optional port; anything else is refused with a 400. The flag is inert when `SPARKWING_CACHE_PUBLIC_URL` is set, because that base ignores the request entirely ## Native frontend exports The repository's candidate installer requests `bin/install.sh --reuse-web`. This calls `bin/build-web.sh --reuse`, which can reuse the existing static export without npm installation, Next compilation, or recopying unchanged assets. Ordinary calls to either script still build fresh. Both build modes use production settings and explicitly install development dependencies needed by TypeScript and PostCSS. The Next compiler cache remains available for rebuilds. Reuse is local to the checkout. A private receipt under `internal/web/.build-state/` records hashes of frontend source and configuration, including untracked files and ignored dotenv files, builder scripts, Node/npm identity, effective npm configuration, and relevant build environment values. It also records a hash of every published export file. A changed input or a missing, edited, or extra output file triggers a fresh build. Environment values are hashed, never written into the receipt. Before recording proof, inputs are checked again; a mismatch fails the build. Generated frontend directories are excluded only at the web root. Changes to frontend build inputs outside this boundary must extend the proof before reuse can cover them. Go source is outside the frontend proof and retains its native compiler cache. The normal Go build and candidate provenance checks still run. The native installer and direct frontend builder coordinate with a file lock through Go compilation. Hosts without `flock`, unsupported input trees, and custom Node preload or npm script-shell configuration build fresh without recording reusable proof. The receipt lives outside the embedded export. This is local iteration reuse; it makes no reproducibility claim about remote font downloads or other external build services. ======================================== # DOC: caching (v0.56.0) ======================================== # Caching Sparkwing caches at four levels: 1. **Job results.** `.Memoize(key, opts...)` replays a recorded result for the same content key, skipping execution. 2. **Dependency directories.** `.CacheDir(...)` restores dependency stores before execution and saves them after success. See [Dependency caches](#dependency-caches). 3. **Build layers.** Docker layers, BuildKit cache mounts, warm PVCs, and dependency proxies reuse build inputs. See [Build caching](build-caching.md). 4. **Pipeline binaries.** Sparkwing reuses a compiled pipeline until its source inputs change. See [Pipeline binary cache](#pipeline-binary-cache). Result keys identify work across groups and runs. [`Concurrency`](sdk.md#concurrency) independently bounds how many nodes run. ## The model ```go shard := sparkwing.Job(plan, "coverage-shard-1", func(ctx context.Context) error { return nil }) shard.Memoize(func(ctx context.Context) (sparkwing.CacheKey, error) { return sparkwing.Key("coverage", "shard-1", "v1"), nil }, sparkwing.TTL(7*24*time.Hour)) ``` When the orchestrator evaluates `shard`, it: 1. Runs upstream dependencies so `Ref[T]` values are resolved. 2. Resolves the `CacheKeyFn` after dependencies complete. A returned error, panic, empty key, or expired resolution deadline fails before dispatch. 3. Runs uncached for `NoCache, nil`; otherwise looks up the content key. A live entry replays its output and records a cache-hit event. 4. Otherwise it runs the node and persists the output under the hash. `.Memoize()` resolves its key once per node before dispatch. `TTL(d)` bounds how long a stored result stays reusable. Omit it for the default (`sparkwing.DefaultCacheTTL`, 7 days); values above `sparkwing.MaxCacheTTL` (35 days) are clamped with a plan-time warning. ## Building keys ```go sparkwing.Key("deploy", "prod", "v1.2.3") build := sparkwing.Job(plan, "build", func(ctx context.Context) (string, error) { return "example-digest", nil }) buildOut := sparkwing.RefTo[string](build) deploy := sparkwing.Job(plan, "deploy", func(ctx context.Context) error { return nil }).Needs(build) deploy.Memoize(func(ctx context.Context) (sparkwing.CacheKey, error) { return sparkwing.Key("deploy", "prod", buildOut.Get(ctx)), nil }) ``` Choose parts with stable, distinct representations. `Key` formats each part with `%v`, which omits type information, and separates parts with byte `0x1e`. A part containing that separator can alias multiple parts. Resolve a `Ref` inside the callback to key on its output; passing the `Ref` itself keys on its node ID. ## What a cache hit skips A hit restores the recorded typed output into the current node's row so that downstream `Ref[T]` values resolve it. The node's action, steps, and `Verify` check are skipped. Declare file outputs with [`Outputs`](artifacts.md). A cache hit carries that artifact manifest forward, and downstream [`Consumes`](artifacts.md) stages its files. Files omitted from the manifest are not restored. ## In-flight dedupe Nodes resolving the same key share an in-flight execution across groups and runs against a shared controller. The first arrival computes the result; followers wait and replay a reusable successful result. A follower executes its own work when the leader ends without a reusable result. ## Opting out per invocation Return `sparkwing.NoCache, nil` to run uncached for one invocation. Return an error when the key cannot be computed; an empty key fails resolution, even when returned with a nil error: ```go skipCache := false sparkwing.Job(plan, "maybe", func(ctx context.Context) error { return nil }). Memoize(func(ctx context.Context) (sparkwing.CacheKey, error) { if skipCache { return sparkwing.NoCache, nil } return sparkwing.Key("maybe", "v1"), nil }) ``` `sparkwing run --sw-no-cache` disables cache *reads* for a whole run while still writing results on success, so the next run hits a freshly populated cache. ## Limitations - **No partial-node caching.** Caching is per node; you cannot skip one step inside a job. Split the cachable work into its own node. - **Bounded retention.** Cache entries expire after their `TTL`. The controller sweeps expired entries automatically on a schedule, and the cache is additionally capped at roughly ten thousand rows, evicting the least recently used entries past that cap. - **Build-layer caching is separate.** See [build-caching.md](build-caching.md). ## Dependency caches `.CacheDir()` declares dependency directories to restore before execution and save after the node's first successful run under that key. ```go sparkwing.Job(plan, "test", runTests). CacheDir(sparkwing.GoModules()) sparkwing.Job(plan, "web-test", runWebTests). CacheDir(sparkwing.NpmCache()) sparkwing.Job(plan, "gems", runSpecs). CacheDir(sparkwing.Dir("vendor/bundle", sparkwing.KeyFromFile("Gemfile.lock"))) ``` Groups take the same declaration and apply it to every member: `group.CacheDir(sparkwing.GoModules())`. ### Directory helpers `GoModules()` targets GOMODCACHE. `NpmCache()` targets the directory reported by `npm config get cache`. `Dir()` names a directory explicitly; choose a key that invalidates every stale input stored there. ### Keys The key is `dep----`, where the hash is the content of the ecosystem's lockfile (`go.sum`, `package-lock.json`, or the `KeyFromFile` target). Editing the lockfile changes the key; restoring its previous bytes restores the previous key. Platform is part of the key because compiled dependency content is not portable across it. Restores require an exact key match. ### Storage - **Laptop:** archives under `$SPARKWING_HOME/depcache/`. - **Cluster:** the sparkwing-cache service's `/cache/` blob store, reached through `SPARKWING_CACHE_URL` (node pods) or `SPARKWING_GITCACHE_URL` (warm runners), authenticated with the runner's agent token. Every pod in the cluster shares one cache. Both use tar.gz archives. The cache service bounds one archive at `sparkwing-cache --max-cache-archive-bytes` (`SPARKWING_CACHE_MAX_ARCHIVE_BYTES`), 500 MB by default and unbounded at `0`. A larger archive is refused with `413` naming the cap, and the node logs a warning and proceeds without the dependency cache. `--max-artifact-bytes` (`SPARKWING_CACHE_MAX_ARTIFACT_BYTES`) is the same cap for one uploaded artifact, also 500 MB. Both caps are applied before the first byte reaches the volume, so one pipeline cannot spend a team's quota, or the store ceiling, on a single object. The SDK skips an upload over 500 MB client-side before it asks; that constant is the client's own, and the service's cap is what actually holds. `--max-store-bytes` and `--max-store-objects` (`SPARKWING_CACHE_MAX_STORE_BYTES`, `SPARKWING_CACHE_MAX_STORE_OBJECTS`) bound the artifact, dependency-archive and upload trees together rather than one object. At or above either one the service refuses every upload with `507` naming the ceiling, while reads and deletes keep working, and a later measurement that finds the store back under the ceiling thaws it. `--warn-store-bytes` and `--warn-store-objects` (`SPARKWING_CACHE_WARN_STORE_BYTES`, `SPARKWING_CACHE_WARN_STORE_OBJECTS`) mark the store as warning without refusing anything, and `--store-reconcile` (`SPARKWING_CACHE_STORE_RECONCILE`, hourly by default, `0` measures once at startup) is how often the service walks its trees and replaces its running count with the measurement. All of them are off until set, and the chart carries them as `cache.limits.*`. `GET /health` reports the store under `store_ceiling` (frozen, warning, counted bytes and objects, and when it was last measured), and the same state is exported as `sparkwing.cache.store_*`. The cache serves no delete, so recovery runs through two bearer-gated admin routes. `POST /admin/store-ceiling/measure` walks the trees immediately, which is how freeing space on the volume turns into uploads flowing again rather than a wait for the interval, and `POST /admin/store-ceiling/thaw` accepts uploads until the next measurement (a `409` when none is scheduled). Both answer with the ceiling state. ### Guarantees A missing lockfile, an unreachable cache service, an oversized archive, or a failed extract logs a warning and execution proceeds without the dependency cache. A failed node never saves. A restore is also skipped when the target directory already has content -- a warm runner's existing cache is left alone. For scoping a tool's cache directory to the current worktree rather than persisting it across runs, see `sparkwing.ToolCacheDir` in [sdk.md](sdk.md); the two compose -- `ToolCacheDir` names a directory, `Dir()` can persist one. See `examples/dep-cache/` for a runnable cold/warm demo. ## Pipeline binary cache Sparkwing compiles the pipeline module and stores the binary under `$SPARKWING_HOME/cache/pipelines/v1/entries//` for reuse until its source inputs change. ### The key The key fingerprints the inputs sparkwing controls: the Go major/minor version, `GOOS`/`GOARCH`, the `go build` flags sparkwing passes, the contents of `.sparkwing/`, the contents of every local `replace` target, the directives of a covering `go.work`, and the resolved module overlays. Build environment the calling shell exports -- `GOFLAGS`, `GOEXPERIMENT`, `CGO_ENABLED` -- is not among them, so two shells that disagree on one compute the same key. Contents are hashed, not timestamps -- editing a file back to its previous bytes restores the previous key. Paths are recorded relative to the module, and local `replace` targets are recorded by module path rather than by where they sit on disk. Two checkouts of the same commit therefore compute the same key from different directories and share one compiled binary, instead of each building their own. **Files Git ignores are excluded from the key.** Directories outside a Git repository hash every file. Set `SPARKWING_HASH_ALL_FILES=1` when a build depends on ignored files, including generated assets embedded by Go. Builds pass `-trimpath`, which keeps the build directory out of the binary. That is what lets two checkouts produce byte-identical output; the cost is that panics report module-relative paths rather than paths on your machine. Builds also pass `-ldflags "-s -w"`, which drops the symbol table and DWARF and takes roughly 30% off the binary and a third off its link time. Panic tracebacks and `runtime/debug.ReadBuildInfo` survive. A debugger still attaches, without variable names or line numbers, and a core dump cannot be symbolised. Set `SPARKWING_NO_BINCACHE=1` to run the pipeline through `go run .` when you need those. ### Bounding the cache After each new entry, Sparkwing reclaims inactive entries to fit a byte ceiling and an entry count. | Variable | Default | Meaning | | --- | --- | --- | | `SPARKWING_CACHE_MAX_BYTES` | `2GiB` | Total size ceiling. Accepts a suffix (`512MiB`, `4GB`). `0` disables. | | `SPARKWING_CACHE_MAX_ENTRIES` | `20` | Entry count ceiling. `0` disables. | Pruning advances through a bounded second-chance queue. An entry used since it entered the queue moves behind the other candidates, so use rather than build age drives retention without an unbounded directory scan. A kernel-backed lease spans lookup through process exit; prune skips active executions and writers rather than relying on a timing window. Prune bounds entry discovery and deletion. It reports logical cache bytes removed separately from observed filesystem capacity gained. The latter is evidence, not an admission decision: callers remeasure the filesystem after pruning because concurrent activity can change free space. Inspect and reclaim on demand: ```bash sparkwing cache info # size, ceilings, recent entries sparkwing cache info --all -o json # every entry, machine-readable sparkwing cache prune # trim to the configured ceilings sparkwing cache prune --max-bytes 512MiB # trim to a smaller budget sparkwing cache prune --all # reclaim everything ``` ### Seeing what an entry is Sparkwing records which checkouts have used each entry, and how often: ``` MOST RECENTLY USED (2 of 2) c1df5cd6-4789f450 71.1 MiB just now x7 ~/code/sparkwing/.sparkwing +1 more checkout(s) 322ecb34-31432125 71.2 MiB 2d ago x1 ~/worktrees/feature-branch/.sparkwing ``` `cache info` counts entries used by several checkouts on the `shared:` line. ### Why did it recompile? `sparkwing cache explain` prints the key, whether it is cached, and every input behind it with its own digest: ``` INPUTS go toolchain 669365bbd24f go1.26 platform 8828cb814901 darwin/arm64 build flags 60dbf03edb4e -trimpath -ldflags -s -w module tree 035b55fe2c64 36 files, 346.1 KiB replace example.com/sample/module e68a991b153a 1439 files, 10.0 MiB (19 gitignored, excluded) ``` Comparing two checkouts input by input shows exactly what differs -- if `module tree` matches and a replace target does not, the pipeline source is identical and a dependency is not. The ignored-file count helps identify edits excluded from the key. When other cached entries came from the same checkout, `explain` lists them with the inputs that changed since, which is the direct answer to why the last run recompiled. To skip the binary cache entirely for one invocation, set `SPARKWING_NO_BINCACHE=1`; sparkwing falls back to `go run .`. ### The shared artifact store A filesystem, bucket, or controller `cache:` backend shares pipeline binaries across machines. The publisher writes a `.sha256` sidecar, and a fetch discards a binary whose digest differs. Store write permissions determine who can publish binaries. A missing sidecar causes the run to compile from source. Set `SPARKWING_ARTIFACT_DIGEST_BACKFILL=1` only for a trusted store to accept blobs without sidecars and write digests from the downloaded bytes. ======================================== # DOC: ci-embedded (v0.56.0) ======================================== # ci-embedded mode Run sparkwing pipelines **inside** an existing CI job (GitHub Actions, Buildkite, GitLab CI, CircleCI, ...) without standing up a sparkwing cluster. State, logs, and artifacts go to S3-compatible storage so a remote dashboard can follow the run live and replay it after the CI VM exits. ## When to use | Scenario | Mode | | -------- | ---- | | Laptop dev loop, fast feedback | `local` (default) | | Migrating from GHA / Buildkite, want better DX without changing CI vendor | **`ci-embedded`** | | Self-hosted cluster with runners, fan-out | `distributed` | ci-embedded is the migration wedge: keep your CI vendor's job orchestration, let sparkwing handle the pipeline DSL + caching + dashboard. ## Quick start (GitHub Actions) `.github/workflows/ci.yaml`: ```yaml name: ci on: [push] jobs: release: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-go@v5 with: { go-version: '1.26' } - run: curl -fsSL https://sparkwing.dev/install.sh | bash - name: Run sparkwing release pipeline env: AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} AWS_REGION: us-west-2 run: sparkwing run release-prod --sw-mode=ci-embedded --sw-workers=4 ``` State, cache, and logs destinations come from the resolved profile in `~/.config/sparkwing/profiles.yaml`. Select one with `--profile NAME` (or set `defaults.profile` in `.sparkwing/sparkwing.yaml`); there is no environment-based auto-selection. See [storage backends](backends.md) for the configuration shape. A pipeline node that fails fails the GHA job (exit code propagates). ## How it works 1. `--sw-mode=ci-embedded` plumbs through `sparkwing` -> the pipeline binary via env vars (`SPARKWING_MODE`, `SPARKWING_WORKERS`). 2. The orchestrator resolves state, cache, and logs from the active profile (selected by `--profile` or `defaults.profile`) -- e.g. an S3-backed profile. 3. Run and node state goes to the profile's `state:` surface. A sqlite state surface keeps records locally and uploads one `runs//state.ndjson` dump to the cache surface when the pipeline exits; an S3 state surface writes that same NDJSON continuously over the life of the run. 4. Per-node log lines route to the resolved `Logs` backend instead of `~/.sparkwing/runs//`. 5. A dashboard configured with the matching backends reads everything back. ## Flags | Flag | Default | Description | | ---- | ------- | ----------- | | `--sw-mode=ci-embedded` | (off) | Enables this mode. | | `--sw-workers=N` | `runtime.NumCPU()` | Caps the local dispatcher. Takes effect only alongside `--sw-mode`; passed on its own the CLI consumes it and forwards nothing, so the cap silently stays at `runtime.NumCPU()`. GHA hosted runners are 2-CPU, so `--sw-workers=4` over-subscribes a small VM -- pick deliberately. | | `--profile NAME` | (none) | Selects a profile from `~/.config/sparkwing/profiles.yaml` (override the path with `SPARKWING_PROFILES`). Absent, the pipeline's own `profile:` field applies, then the project's `defaults.profile` in `.sparkwing/sparkwing.yaml`. With nothing selected the run falls back to local SQLite plus filesystem and never reaches the bucket. | State, cache, and logs come from the resolved profile; see [storage backends](backends.md) for the configuration shape. ### Recommended: `SPARKWING_NO_SPARKS_RESOLVE=1` in CI If your `.sparkwing/sparkwing.yaml` declares a `sparks:` block, sparkwing auto-refreshes the resolved overlay at run time by default. That shells out to `go env` / `go list`, which means CI runners would need a Go toolchain even on a cache hit. **Set `SPARKWING_NO_SPARKS_RESOLVE=1` in the CI step's env** so the runner trusts the committed `.resolved.mod` overlay and never resolves on its own. Workflow then becomes: ```sh # locally, when you want a fresh resolve sparkwing pipeline sparks update git diff .sparkwing/.resolved.mod git commit -am "bump sparks-core" git push # triggers publish + run with frozen overlay ``` CI never re-resolves; the publish step on your laptop (or in a publish-on-merge workflow) is the deliberate "go fresh" surface. Repos without a `sparks:` block ignore this var -- it's a no-op. ## Profile-based config (laptop) `~/.config/sparkwing/profiles.yaml`: ```yaml profiles: ci-team: state: { type: s3, bucket: my-team-sparkwing, prefix: state } cache: { type: s3, bucket: my-team-sparkwing, prefix: cache } logs: { type: s3, bucket: my-team-sparkwing, prefix: logs } ``` Then: ```sh sparkwing run release-prod --sw-mode=ci-embedded --profile ci-team ``` ## Watching from a laptop dashboard After (or during) a ci-embedded run, point your local dashboard at the same bucket: ```sh sparkwing serve start \ --profile ci-team \ --read-only ``` The dashboard reads `state.ndjson` for run metadata, the LogStore for per-node lines, and the ArtifactStore for any blobs the pipeline saved. A profile whose `state:` surface is S3 streams run and node records to `runs//state.ndjson` continuously (flushed on a sub-second interval, or earlier once enough envelopes buffer), so a dashboard pointed at the same bucket follows the run live. A sqlite `state:` surface has no live view: its NDJSON dump is written to the cache surface when the pipeline exits. ### Fresh laptop, no SQLite (`--no-local-store`) The default invocation above still opens `~/.sparkwing/state.db` so locally-triggered runs can coexist with the remote ones. On a clean machine that has *only* the bucket -- new hire, ephemeral container, etc. -- pass `--no-local-store` to skip SQLite entirely and have the dashboard list runs directly from `/runs/*/state.ndjson`: ```sh sparkwing serve start \ --profile ci-team \ --no-local-store \ --read-only ``` This mode is read-only by construction: the orchestrator's write endpoints (cancel, retry, approvals) are not mounted, since there's no local SQLite to persist to. Passing `--no-local-store` without both `--log-store` and `--artifact-store` (directly or via `--profile`) errors out -- the dashboard would have nowhere to read from. ## S3 layout ``` // cache/ # ArtifactStore runs//state.ndjson # final run + node dump # pipeline-saved blobs logs/ # LogStore //.ndjson # one object per flush ``` S3 has no native append, so every write is its own object. The logs surface buffers each node's lines and writes one object per flush, which keeps a chatty node at a handful of requests rather than one per line. A flush lands when the buffer reaches `batch_bytes`, when `batch_interval` elapses, when a reader asks for the node's log, and when the node finishes. `max_log_objects` and `max_log_bytes` bound what one node can write; past either the surface drops further lines and ends the node's log with one marker line counting them. See [storage backends](backends.md#object-store-log-batching) for the keys and their defaults. Reads list+concat by prefix. ## Exit codes - `0` if every pipeline node succeeds. - `1` if any node fails or the orchestrator errors. The exit code is what the wrapping CI job sees, so a failed sparkwing node fails the CI step. ## Caveats - **No webhooks**. ci-embedded mode is invoked by the host CI; let GitHub Actions / Buildkite handle the trigger. - **Caching across runs** depends on stable `bincache.PipelineCacheKey` output (sha256 over source + go toolchain). Same source tree on the same Go version = warm cache. - **Worker count vs CPU**. GHA hosted runners default to 2 CPUs. `--sw-workers=NumCPU` (the default) usually fits fine; larger numbers trade memory pressure for less queueing. ## Buildkite ```yaml steps: - label: "release" command: | sparkwing run release-prod --sw-mode=ci-embedded --sw-workers=4 plugins: - aws-credentials#v1.0: role: arn:aws:iam::1234:role/buildkite-sparkwing ``` State, cache, and logs come from the resolved profile. Declare a profile for the run and pass it with `--profile`: ```yaml # ~/.config/sparkwing/profiles.yaml profiles: buildkite: state: { type: s3, bucket: my-team-sparkwing, prefix: state/ } cache: { type: s3, bucket: my-team-sparkwing, prefix: cache/ } logs: { type: s3, bucket: my-team-sparkwing, prefix: logs/ } ``` ```sh sparkwing run release-prod --sw-mode=ci-embedded --profile buildkite ``` ## GitLab CI ```yaml release: image: alpine:latest before_script: - apk add --no-cache curl - curl -fsSL https://sparkwing.dev/install.sh | sh script: - sparkwing run release-prod --sw-mode=ci-embedded --sw-workers=4 ``` Declare a `gitlab` profile in `~/.config/sparkwing/profiles.yaml` and select it with `--profile gitlab` (same shape as the Buildkite example above). ## Related - The storage interface + filesystem / S3 backends. - The dashboard's storage-aware reads. ======================================== # DOC: cli-cache (v0.56.0) ======================================== # CLI reference: sparkwing cache Every `sparkwing cache` command, flag, and argument, generated from the CLI's own command registry. All command groups are indexed in [cli-reference.md](cli-reference.md). ## `sparkwing cache` Inspect or trim the compiled pipeline binary cache Compiled pipeline binaries are keyed by their source fingerprint and stored under $SPARKWING_HOME/cache/pipelines. Automatic pruning after compilation keeps recently used entries within the configured byte and entry limits. Use these commands to inspect entries or reclaim space. ### Subcommands - `info` -- Print cache dir, size, ceilings, and recent entries - `prune` -- Evict least recently used binaries down to the ceilings - `explain` -- Show a pipeline's cache key and the inputs behind it ### Examples ```sh # See what is cached sparkwing cache info # Reclaim space now sparkwing cache prune ``` ## `sparkwing cache explain` Show a pipeline's cache key and the inputs behind it Prints the cache key for a pipeline module, whether that key is already cached, and every input that produced it -- the Go toolchain, the platform, the module tree, each local replace target, a covering go.work, and the resolved module pins -- each with its own digest and how much it covered. File counts show how many files were excluded because Git ignores them. Edits to excluded files leave the cache key unchanged. When other cached entries came from the same checkout, each is listed with the inputs that differ from the current key. That is the direct answer to why a rebuild happened. ### Flags | Flag | Description | |---|---| | `--dir PATH` | Pipeline module directory (default: ./.sparkwing) | | `-o, --output FORMAT` | Output format: pretty \| json \| plain (default: pretty on TTY, json when piped) | ### Examples ```sh # Why did this rebuild? sparkwing cache explain # Agent-readable sparkwing cache explain -o json ``` ## `sparkwing cache info` Print cache dir, size, ceilings, and recent entries Lists the cache directory, its total size, the configured ceilings, and the most recently used entries with their sizes and last-use times. Entries are ordered by last use, which is what pruning evicts on -- not by when they were built. ### Flags | Flag | Description | |---|---| | `-o, --output FORMAT` | Output format: pretty \| json \| plain (default: pretty on TTY, json when piped) | | `--all` | List every entry instead of the ten most recent | ### Examples ```sh # Human-readable sparkwing cache info # Agent-readable sparkwing cache info -o json # Every entry sparkwing cache info --all ``` ## `sparkwing cache prune` Evict least recently used binaries down to the ceilings Removes the least recently used cached binaries until the cache fits both the byte ceiling and the entry ceiling. Defaults come from $SPARKWING_CACHE_MAX_BYTES and $SPARKWING_CACHE_MAX_ENTRIES; either accepts 0 to disable that dimension. An execution lease protects each running binary. Prune skips active and busy entries, bounds the number examined, and reports observed capacity separately from removed entries. Callers making admission decisions remeasure filesystem capacity after pruning. ### Flags | Flag | Description | |---|---| | `--max-bytes SIZE` | Byte ceiling (512MiB and similar sizes) | | `--max-entries N` | Entry ceiling | | `--all` | Remove every entry, ignoring both ceilings | | `-o, --output FORMAT` | Output format: pretty \| json \| plain (default: pretty on TTY, json when piped) | ### Examples ```sh # Trim to the configured ceilings sparkwing cache prune # Trim to a smaller budget sparkwing cache prune --max-bytes 512MiB # Reclaim everything sparkwing cache prune --all ``` ======================================== # DOC: cli-cloud (v0.56.0) ======================================== # CLI reference: sparkwing cloud Every `sparkwing cloud` command, flag, and argument, generated from the CLI's own command registry. All command groups are indexed in [cli-reference.md](cli-reference.md). ## `sparkwing cloud` Connect this machine to a sparkwing controller One command joins a controller: 'connect' verifies the controller answers, mints a user token when you hand it an admin credential, and writes the profile that every other command selects with --profile. 'status' reports what that connection authenticates as. 'disconnect' removes the profile and revokes its token. Nothing here edits profiles.yaml by hand. Enroll this machine as a runner with 'sparkwing cluster runners add'. ### Subcommands - `connect` -- Write the profile that reaches a controller - `status` -- Report the connection, its principal, and the probes - `disconnect` -- Revoke the connection's token and drop the profile ### Examples ```sh # Connect with a one-time admin token sparkwing cloud connect --controller https://api.sparkwing.example --admin-token-stdin # Report the connection sparkwing cloud status --profile api-sparkwing-example ``` ## `sparkwing cloud connect` Write the profile that reaches a controller Verifies the controller answers its health route, then writes a profile carrying the controller URL and a token. --admin-token-stdin reads an admin credential from stdin and mints a user token with it, carrying runs.read, runs.write, triggers.read, logs.read and approvals.write. The admin credential is never stored; only the minted token reaches profiles.yaml. --token-stdin stores a token you already hold. Neither flag connects to a controller serving unauthenticated. --name defaults to the controller host with every character outside a-z0-9 turned into a dash, so https://api.sparkwing.example becomes api-sparkwing-example. An existing profile of that name is never replaced without --force, because the token it holds stays live until it is revoked. --set-default writes defaults.profile into this repository's .sparkwing/sparkwing.yaml, so runs in this checkout select the connection with no flag. The name resolves against the project's own profiles: block first and profiles.yaml second, so the token stays out of the checkout. The command closes with the dashboard URL the controller announces and the probes 'sparkwing configure profiles test' runs. ### Flags | Flag | Description | |---|---| | `--controller URL` | Controller base URL (required) | | `--name NAME` | Profile name (default: derived from the controller host) | | `--admin-token-stdin` | Read an admin token from stdin and mint a user token with it | | `--token-stdin` | Read an already-minted user token from stdin | | `--scope CSV` | Comma-separated scopes for the minted token (default: runs.read,runs.write,triggers.read,logs.read,approvals.write) | | `--set-default` | Set defaults.profile in this project's .sparkwing/sparkwing.yaml | | `--force` | Replace an existing profile of that name | ### Examples ```sh # Connect with a one-time admin token sparkwing cloud connect --controller https://api.sparkwing.example --admin-token-stdin # Connect and make it this repository's default sparkwing cloud connect --controller https://api.sparkwing.example --name prod --admin-token-stdin --set-default # Store a token someone minted for you sparkwing cloud connect --controller https://api.sparkwing.example --token-stdin ``` ## `sparkwing cloud disconnect` Revoke the connection's token and drop the profile Revokes the profile's token on its controller, then removes the profile. A revoke this credential is not allowed to make leaves the token live and names the prefix and the command that finishes the job, so a connection is never dropped silently. The profile's own token revokes only when it carries admin; --admin-token-stdin supplies one that does. A prefix the controller reports as anything but a user token is refused, naming what it found. --keep-token drops the profile and touches no credential. ### Flags | Flag | Description | |---|---| | `--name NAME` | Profile name to disconnect (required) | | `--admin-token-stdin` | Read an admin token from stdin and revoke with it | | `--keep-token` | Remove the profile without revoking its token | ### Examples ```sh # Disconnect and revoke with an admin token sparkwing cloud disconnect --name prod --admin-token-stdin # Drop the profile and leave the token alone sparkwing cloud disconnect --name prod --keep-token ``` ## `sparkwing cloud status` Report the connection, its principal, and the probes Prints the selected profile, its controller, the principal and scopes the controller reports for its token, the announced dashboard URL, and the controller, auth, logs and gitcache probes. Exits non-zero when a probe fails. ### Flags | Flag | Description | |---|---| | `--profile NAME` | Profile naming the connection to report | | `-o, --output FORMAT` | Output format: json\|table | ### Examples ```sh # Report the connection sparkwing cloud status --profile prod # Machine-readable status sparkwing cloud status --profile prod -o json ``` ======================================== # DOC: cli-cluster (v0.56.0) ======================================== # CLI reference: sparkwing cluster Every `sparkwing cluster` command, flag, and argument, generated from the CLI's own command registry. All command groups are indexed in [cli-reference.md](cli-reference.md). ## `sparkwing cluster` Operate and inspect the sparkwing cluster Inspect controller health, executors, admission, users, tokens, images, and webhooks. Select the controller with --profile NAME. Configure profiles with 'sparkwing configure profiles'. 'worker' executes queued triggers on this machine. 'gc' removes stale warm-runner storage. Manage secrets with 'sparkwing secrets' and the local dashboard with 'sparkwing serve'. ### Subcommands - `status` -- Connectivity + fleet + queue health check against a remote cluster - `agents` -- Inspect the controller's fleet view - `runners` -- Enroll and retire this machine as a runner - `worker` -- Claim triggers from a profile's controller and run them in-process - `gc` -- Sweep stale warm-PVC state - `users` -- Manage dashboard login users - `tokens` -- Manage controller API tokens - `credits` -- Inspect and top up the prepaid credit balance - `limits` -- Read and set the compute guards - `image` -- Rollout helpers for images referenced by a gitops repo - `webhooks` -- Connect, inspect, and replay GitHub webhooks - `concurrency` -- Inspect a single concurrency namespace: holders + queue - `object-store` -- Operate the controller's object-store request budget ### Examples ```sh # Cluster health summary sparkwing cluster status --profile prod # List fleet agents sparkwing cluster agents list --profile prod ``` ## `sparkwing cluster agents` Inspect the controller's fleet view Hits GET /api/v1/agents on the selected profile's controller. Prints persisted executor registrations, including idle and offline agents and gateways, plus recent legacy claim-only runners. ### Subcommands - `list` -- Print the controller's known agents - `enroll` -- Enroll or update a trusted executor ### Examples ```sh # List prod agents sparkwing cluster agents list --profile prod ``` ## `sparkwing cluster agents enroll` Enroll or update a trusted executor Binds one exact runner or service token prefix to an operator-owned executor envelope. The token must be live and carry nodes.claim; its stored principal becomes audit metadata. Re-enrollment with the same credential updates trusted scheduling fields without changing live headroom. Changing the prefix requires a new heartbeat. Use a distinct revocable token for every coordinator membership. The prefix is accepted as input but is never returned by the agents API. A controller accepts at most 256 enrolled executors. Adding another returns `executor enrollment limit reached: maximum 256 per controller`. ### Flags | Flag | Description | |---|---| | `--name NAME` | Executor name (required) | | `--token-prefix PREFIX` | Exact runner or service token prefix (required) | | `--kind KIND` | Executor kind (agent\|gateway) (default: agent) | | `--location WHERE` | Trusted placement location (local\|cloud\|unknown) (default: unknown) | | `--capability LABEL` | Trusted capability (repeatable) | | `--base-priority N` | Base scheduling priority (0-100) (default: 0) | | `--priority-ceiling N` | Highest effective priority (0-100) (default: 100) | | `--max-concurrent N` | Trusted concurrent slot ceiling (default: 1) | | `--budget-cores N` | CPU contribution ceiling (0 = uncapped) (default: 0) | | `--budget-memory-bytes N` | Memory contribution ceiling in bytes (0 = uncapped) (default: 0) | | `--profile NAME` | Admin controller profile (required) | ### Examples ```sh # Enroll a workstation agent sparkwing cluster agents enroll --profile prod --name desk --token-prefix swr_01234567 --kind agent --location local --capability linux --max-concurrent 2 --budget-cores 4 --budget-memory-bytes 8589934592 # Enroll a capacity gateway sparkwing cluster agents enroll --profile prod --name build-gateway --token-prefix sws_01234567 --kind gateway --location cloud --capability linux-amd64 --max-concurrent 8 ``` ## `sparkwing cluster agents list` Print the controller's known agents Fetches /api/v1/agents and renders a table of fleet members. Registered executors report their operator-assigned identity, kind, trusted placement location, capabilities, concurrency limit, and measured resource headroom. A stale registration remains visible as offline; recent legacy claim-only runners remain visible too. Use -q to print names, one per line, for shell piping (xargs and similar commands). ### Flags | Flag | Description | |---|---| | `--profile NAME` | Profile name (required) | | `-o, --output FMT` | Output format (json\|table) | | `-q, --quiet` | Print agent names, one per line | ### Examples ```sh # List agents on prod sparkwing cluster agents list --profile prod # Just agent names for piping sparkwing cluster agents list --profile prod -q ``` ## `sparkwing cluster concurrency` Inspect a single concurrency namespace: holders + queue Shows who holds a concurrency namespace's slots and the queue of waiters behind it, each with its admission-rank position. Weighted admission can run a later fitting waiter before an earlier non-fitting waiter, so position is not always run order. Use it to tell whether a node is wedged or waiting for budget. Hits GET /api/v1/concurrency/{namespace}/state on the selected profile's controller. For a controller's whole admission state -- every key, its holders and waiters, and each registered runner's free capacity -- through the same view as the local queue, use 'sparkwing queue --profile NAME'. This command narrows to one namespace. ### Flags | Flag | Description | |---|---| | `--namespace NAME` | Concurrency namespace to inspect (required) | | `--profile NAME` | Profile selecting the controller (required) | | `-o, --output FORMAT` | Output format (json\|table) | ### Examples ```sh # Who holds and who's queued sparkwing cluster concurrency --namespace deploy-prod --profile prod ``` ## `sparkwing cluster credits` Inspect and top up the prepaid credit balance Cloud runner time is prepaid. One hundred credits is one dollar, so a ten dollar top-up is a thousand credits. The balance is grants minus charges: a claim reserves a minute of cloud runner time before it is granted, heartbeats charge the seconds they cover, and the finish refunds whatever of the reservation the node did not use. Runners the operator did not mark metered are never charged. ### Subcommands - `show` -- Print the balance, the rate table, and the recent burn - `grant` -- Add free or paid credits to the ledger, or reverse a paid grant - `history` -- List grants and charges, newest first - `settings` -- Read or set the credit rate table, the grace period, and the charge cap - `allowance` -- Read or set how many retained bytes a team keeps ### Examples ```sh # Read the balance and the burn sparkwing cluster credits show --profile prod # Load ten dollars sparkwing cluster credits grant --kind paid --amount 1000 --reference pay_12345 --profile prod ``` ## `sparkwing cluster credits allowance` Read or set how many retained bytes a team keeps Retained bytes are what a team still has stored after its runs end, and the storage pass bills them at the storage rate. The allowance is how many of them the team asked to keep: the pass expires its oldest finished runs above the allowance before it bills and never bills above it, so the allowance is both what the team keeps and the most it pays for. An allowance of zero keeps everything and caps nothing. This verb is the only writer; rewriting a team's quota leaves the allowance alone. Reading with no flag reports the calling token's own allowance and what it currently retains. An admin token reads another team by naming it. Setting one needs the admin scope and names the team. The free allowance every team keeps unbilled and the price of a gibibyte-day are controller-wide settings on `sparkwing cluster credits settings`. ### Flags | Flag | Description | |---|---| | `--principal NAME` | Team whose allowance to read or set; required to set one | | `--gb N` | Gibibytes of retained storage to keep; 0 keeps everything | | `--bytes N` | Bytes of retained storage to keep; 0 keeps everything | | `-o, --output FORMAT` | Output format: pretty \| json \| plain (default: pretty on TTY, json when piped) | | `--profile NAME` | Profile name (required) | ### Examples ```sh # Read what this token's team keeps sparkwing cluster credits allowance --profile prod # Keep fifty gibibytes for a team sparkwing cluster credits allowance --principal acme --gb 50 --profile prod ``` ## `sparkwing cluster credits grant` Add free or paid credits to the ledger, or reverse a paid grant Adds credits and records who added them, which kind they are, and the payment they came from. One hundred credits is one dollar. A grant that lifts the balance above zero lets metered runners claim again and stops the cancellation of nodes running on an empty balance. A reference is the payment id: granting it twice returns the first grant rather than adding the credits again. A reversal takes a refunded payment back out with a negative amount, its own reference (the refund id) and --reverses naming the paid grant's reference. Requires the admin scope. ### Flags | Flag | Description | |---|---| | `--kind KIND` | Grant kind: free \| paid \| reversal (required) | | `--amount N` | Credits to add, negative on a reversal; 100 credits is one dollar (required) | | `--reference REF` | Payment id or operator note recorded with the grant; granting the same one twice returns the first grant | | `--reverses REF` | Reference of the paid grant a reversal takes back | | `--profile NAME` | Profile name (required) | ### Examples ```sh # Load ten dollars against a payment sparkwing cluster credits grant --kind paid --amount 1000 --reference pay_12345 --profile prod # Hand out trial credits sparkwing cluster credits grant --kind free --amount 500 --profile prod # Take a refunded payment back out sparkwing cluster credits grant --kind reversal --amount -1000 --reference re_9 --reverses pay_12345 --profile prod ``` ## `sparkwing cluster credits history` List grants and charges, newest first Lists every movement of the ledger newest first: grants with their kind and reference, and the reservation a claim took, the usage an interval billed, and the refund of a reservation a node did not use, each with the run, node, token prefix and seconds it covered, and the cpu class and rate it was billed at. Charges render negative because they take credits out and a refund renders positive. -o json emits one JSON record per line. ### Flags | Flag | Description | |---|---| | `--limit N` | Maximum rows of each kind, up to 1000 (0 = the controller's default) | | `-o, --output FORMAT` | Output format: pretty \| json \| plain (default: pretty on TTY, json when piped) | | `--profile NAME` | Profile name (required) | ### Examples ```sh # Read the ledger sparkwing cluster credits history --profile prod # Sum today's charges sparkwing cluster credits history --profile prod -o json | jq 'select(.type=="charge") | .amount_micro' ``` ## `sparkwing cluster credits settings` Read or set the credit rate table, the grace period, and the charge cap Prints the runtime settings the ledger prices work with, and sets the ones named by a flag. The rate table prices one cloud runner second at every cpu class, and a node is billed at the smallest class covering the cpu and memory it pinned; a request above the largest class fails the node. The rate is what a four-core second costs, which is the four-core entry of the table under another name, so a body may name one or the other, never both. The warm cpu class is the largest class the warm runner pool serves: a node above it starts a Kubernetes node sized to its class instead, and zero starts a node of its own for every class. The grace period is how long a node keeps running after it has consumed the reservation its claim paid for with the balance at zero: a node inside that reservation is never cancelled, because the ledger already took payment for it. The charge cap is the most seconds any one charge may bill, which forgives a controller outage or a stalled heartbeat loop rather than billing the gap. A flag left off leaves that setting alone, and a refused value moves nothing. Grace zero cancels a metered node at the first heartbeat past its reservation, which bounds the unpaid overrun to one heartbeat interval per node. An installation that never set a table bills the default ladder. Reading needs the runs.read scope and setting needs admin. ### Flags | Flag | Description | |---|---| | `--rate-table PAIRS` | Price every cpu class, as CORES=MICRO pairs: 2=10000,4=20000,8=36667 | | `--warm-cpu-class-cores N` | Largest cpu class the warm runner pool serves; a larger class starts a node of its own | | `--rate-micro N` | Micro-credits one four-core cloud runner second costs, 1 to 1000000000000; refused once a rate table exists | | `--grace-seconds N` | Seconds a node runs past its reservation on an empty balance; 0 cancels at the next heartbeat | | `--max-charge-seconds N` | The most seconds any one charge may bill, 6 to 86400 | | `-o, --output FORMAT` | Output format: pretty \| json \| plain (default: pretty on TTY, json when piped) | | `--profile NAME` | Profile name (required) | ### Examples ```sh # Read the settings sparkwing cluster credits settings --profile prod # Cut a node off at the first heartbeat past its reservation sparkwing cluster credits settings --grace-seconds 0 --profile prod # Reprice a cloud runner second at 0.03 credits sparkwing cluster credits settings --rate-micro 30000 --profile prod # Price the six sizes at the GitHub Actions rates sparkwing cluster credits settings --rate-table 2=10000,4=20000,8=36667,16=70000,32=136667,64=270000 --profile prod ``` ## `sparkwing cluster credits show` Print the balance, the rate table, and the recent burn Prints the balance in credits, what was granted and charged, the price of a cloud runner second at every cpu class, the credits burned over the last day, the grace period a node gets past the reservation its claim paid for, and the cap on what any one charge may bill. A controller that was never granted anything reads a zero balance and charges nothing, because nothing is metered until an operator marks a token. ### Flags | Flag | Description | |---|---| | `-o, --output FORMAT` | Output format: pretty \| json \| plain (default: pretty on TTY, json when piped) | | `--profile NAME` | Profile name (required) | ### Examples ```sh # Read the balance sparkwing cluster credits show --profile prod # Read the balance as JSON sparkwing cluster credits show --profile prod -o json ``` ## `sparkwing cluster gc` Sweep stale warm-PVC state Operator-facing manual invocation of the warm-PVC sweep. Normally fires at 'sparkwing cluster worker' startup; exposed as a subcommand so operators can trigger it against a running pod via kubectl exec during incident response. When --profile is omitted, the run-directory sweep is skipped; the mtime-based git/ and tmp/ sweeps still run and free disk. Supply --profile to enable the full sweep. ### Flags | Flag | Description | |---|---| | `--root DIR` | Warm-PVC root (default: $SPARKWING_HOME resolution) | | `--profile NAME` | Profile name; without it run-dir sweep is skipped | ### Examples ```sh # mtime-only sweep in-pod (no controller) sparkwing cluster gc # Full sweep against prod controller sparkwing cluster gc --profile prod # Target a specific warm root sparkwing cluster gc --root /var/lib/sparkwing --profile prod ``` ## `sparkwing cluster image` Rollout helpers for images referenced by a gitops repo Update an image tag in a GitOps repository, commit and push the change, sync ArgoCD, and wait for rollout. Publish the image before using these commands. ### Subcommands - `rollout` -- Bump a kustomization image tag, commit+push, sync ArgoCD, optionally wait ### Examples ```sh # Update the example runner image sparkwing cluster image rollout --image fictional-runner --tag commit-abc123 --wait ``` ## `sparkwing cluster image rollout` Bump a kustomization image tag, commit+push, sync ArgoCD, optionally wait Rewrites the newTag: field for the image whose entry in the gitops repo's kustomization.yaml matches --image (suffix match against the ECR / registry URL), commits + pushes the change, optionally triggers an ArgoCD sync, and optionally blocks on kubectl rollout status. Gitops repo resolution order: 1. --gitops-repo PATH explicit flag 2. SPARKWING_GITOPS_REPO explicit environment configuration If neither is set, rollout exits before reading or changing a repository. Sparkwing never guesses a path from the user's home-directory layout. The command is idempotent: if the newTag already matches --tag there is nothing to commit, and the pipeline continues to sync + wait without error. Use --dry-run to preview the plan without writing, committing, pushing, syncing, or waiting. Tool requirements: - argocd missing -> sync is skipped with a one-line notice - kubectl missing -> --wait / --tail-logs error before side effects This verb does not build or push the image itself. The consumer pipeline that produced --tag is responsible for publishing the image to the registry before calling rollout. ### Flags | Flag | Description | |---|---| | `--image NAME` | Short image name (matches the suffix of the ECR URL) (required) | | `--tag TAG` | New tag to write in kustomization.yaml (required) | | `--gitops-repo PATH` | Gitops repo path (or SPARKWING_GITOPS_REPO) | | `--namespace NS` | Kubernetes namespace for rollout status + logs (default: sparkwing) | | `--argocd-app NAME` | ArgoCD app name (default: derived from --image) | | `--message MSG` | Commit message (default: 'chore: bump to ') | | `--wait` | Block until 'kubectl rollout status deployment/' returns | | `--tail-logs` | After rollout, 'kubectl logs -f -l app=' until ctrl-c | | `--dry-run` | Print what would happen without writing, committing, pushing, or syncing | ### Examples ```sh # Preview the example runner image update sparkwing cluster image rollout --image fictional-runner --tag commit-abc123 --dry-run # Bump and wait for the rollout sparkwing cluster image rollout --image fictional-runner --tag commit-abc123 --wait # Bump, sync, wait, then tail pod logs sparkwing cluster image rollout --image fictional-service --tag commit-abc123 --wait --tail-logs ``` ## `sparkwing cluster limits` Read and set the compute guards Compute guards bound what the controller starts before the credit ledger bills it: the cloud runners one principal holds at once, the cloud runners the whole controller holds, the wall-clock seconds a run may hold them for, the nodes one run may carry, the runs created per hour, and the shortest interval a cloud schedule may declare. Every guard is zero by default, which is unlimited, so a controller that sets none behaves as it did before the guards existed. ### Subcommands - `show` -- Print every compute guard and the cloud runners in use - `set` -- Set one compute guard ### Examples ```sh # Read the guards and what they measure sparkwing cluster limits show --profile prod # Cap the cloud runners one principal holds sparkwing cluster limits set --name max_concurrent_runners --value 20 --profile prod ``` ## `sparkwing cluster limits set` Set one compute guard Sets one guard to a ceiling, or to zero to remove it. The guards are max_concurrent_runners, max_global_runners, runner_alarm, max_run_seconds, max_nodes_per_run, max_runs_per_hour, max_global_nodes_per_run, max_global_runs_per_hour, min_cron_interval_seconds, runner_scale_base, runner_scale_step_credits and runner_scale_ceiling. The per-principal guards bind a principal holding a metered token; the two max_global settings bind every run. The three runner_scale settings raise max_concurrent_runners by one runner_scale_base for every runner_scale_step_credits of paid credit granted in the last 30 days, held under runner_scale_ceiling. Work past a guard answers 429 with a Retry-After and the run records a compute_limit_blocked event. Requires the admin scope. ### Flags | Flag | Description | |---|---| | `--name GUARD` | Guard to set (required) | | `--value N` | Ceiling; 0 removes it (required) | | `--profile NAME` | Profile name (required) | ### Examples ```sh # Hold the fleet under fifty cloud runners sparkwing cluster limits set --name max_global_runners --value 50 --profile prod # Warn at forty sparkwing cluster limits set --name runner_alarm --value 40 --profile prod # Remove the per-run node cap sparkwing cluster limits set --name max_nodes_per_run --value 0 --profile prod # Add a hundred runners per 5000 credits loaded sparkwing cluster limits set --name runner_scale_step_credits --value 5000 --profile prod ``` ## `sparkwing cluster limits show` Print every compute guard and the cloud runners in use Prints each guard with its ceiling, or "unlimited" when nothing set one, then the cloud runners claimed now in total and per principal. A cloud runner is a claim a metered token holds, so a controller that marks no token metered reads zero. ### Flags | Flag | Description | |---|---| | `-o, --output FORMAT` | Output format: pretty \| json \| plain (default: pretty on TTY, json when piped) | | `--profile NAME` | Profile name (required) | ### Examples ```sh # Read the guards sparkwing cluster limits show --profile prod # Read the guards as JSON sparkwing cluster limits show --profile prod -o json ``` ## `sparkwing cluster object-store` Operate the controller's object-store request budget The controller counts every object-store request it makes, by class (put, get, list, delete), against a per-minute rate and a per-day budget. A class that spends either budget trips: writes of that class fail closed and reads keep serving until their own budget trips. The state appears on 'sparkwing cluster status' and on the controller's Prometheus metrics as sparkwing_object_store_requests_total, sparkwing_object_store_trips_total, and sparkwing_object_store_tripped. Budgets come from SPARKWING_OBJECT_STORE__PER_MINUTE and SPARKWING_OBJECT_STORE__PER_DAY on the controller process. SPARKWING_OBJECT_STORE_TRIP_RESET chooses whether a tripped class clears when its day window rolls (day, the default) or waits for an operator (manual). A local process that must finish past a tripped budget sets SPARKWING_OBJECT_STORE_BREAKER=off. The same breaker carries the bucket ceiling. A controller started with --max-bucket-bytes or --max-bucket-objects measures the bucket on an interval, freezes object writes once it holds more than the ceiling, and reports the freeze on health and as sparkwing_object_store_bucket_ceiling_frozen. Buckets are unlimited by default. ### Subcommands - `status` -- Show the controller's object-store request budget - `reset-breaker` -- Clear a tripped object-store budget or ceiling freeze ### Examples ```sh # Clear a tripped budget sparkwing cluster object-store reset-breaker --profile prod ``` ## `sparkwing cluster object-store reset-breaker` Clear a tripped object-store budget or ceiling freeze Clears every tripped request class on the selected controller, resets its per-minute and per-day window counters, and thaws a frozen bucket ceiling, then prints the budget as it stands. Lifetime request and trip totals survive, so the metrics keep their history. A thawed bucket that is still over its ceiling freezes again at the next measurement, so a thaw buys the window to delete objects or raise the ceiling. Reach for this after fixing what caused the trip. A budget that keeps tripping wants a larger limit or a caller that stops retrying, not a repeated reset. Hits POST /api/v1/object-store/reset-breaker on the selected profile's controller, which needs an admin-scoped token. ### Flags | Flag | Description | |---|---| | `--profile NAME` | Profile selecting the controller (required) | | `-o, --output FORMAT` | Output format (json\|table) | ### Examples ```sh # Clear a tripped budget sparkwing cluster object-store reset-breaker --profile prod ``` ## `sparkwing cluster object-store status` Show the controller's object-store request budget Prints each request class with its per-minute rate, its per-day budget, how much of each window the controller has spent, how many times the class has tripped, and whether it is refusing requests now, followed by the bucket ceiling: what the bucket holds, the ceilings it is held to, and whether object writes are frozen. Changes nothing. Hits GET /api/v1/object-store/breaker on the selected profile's controller, which needs an admin-scoped token. ### Flags | Flag | Description | |---|---| | `--profile NAME` | Profile selecting the controller (required) | | `-o, --output FORMAT` | Output format (json\|table) | ### Examples ```sh # Read the budget sparkwing cluster object-store status --profile prod ``` ## `sparkwing cluster runners` Enroll and retire this machine as a runner Turns one machine into a runner for the selected profile's controller in a single command. 'add' mints a scoped runner token, writes the owner-only agent config, and installs the user service. 'remove' stops that service and revokes the token. Use 'sparkwing cluster agents list' to see the runners a controller knows about. ### Subcommands - `add` -- Mint a runner token, write the config, start the service - `remove` -- Stop the runner service and revoke its token ### Examples ```sh # Enroll this machine sparkwing cluster runners add --profile prod --name dev-laptop # Retire this machine sparkwing cluster runners remove --profile prod ``` ## `sparkwing cluster runners add` Mint a runner token, write the config, start the service Mints a runner token carrying nodes.claim, triggers.claim, runs.state, secrets.read and logs.write against the profile's controller, writes ~/.config/sparkwing/agent.yaml at mode 0600, then installs and starts the user service: a systemd user unit on Linux, a LaunchAgent on macOS. On Windows it prints the manual supervision steps instead. The config is written in claim mode, which is the mode that executes work. An existing config is never replaced without --force, because the token it holds stays live until it is revoked. Nothing is minted until the config validates and the machine answers: a missing sparkwing-runner, an unreachable service manager, or an unusable setting fails first. If a step after the mint fails, the output names the live token and the command that revokes it. The command prints the token prefix and the revoke command. The raw token reaches only the config file. ### Flags | Flag | Description | |---|---| | `--name NAME` | Runner name, shown in the dashboard (required) | | `--labels CSV` | Comma-separated self-asserted placement labels | | `--max-concurrent N` | Concurrent jobs this machine accepts (default: 2) | | `--contribution SPEC` | CPU and memory this machine contributes (4,8gb or 50%,50%) (default: 50%,50%) | | `--logs URL` | Logs service URL (default: the profile's logs surface) | | `--config PATH` | Agent config to write (default: ~/.config/sparkwing/agent.yaml) | | `--force` | Replace an existing agent config | | `--no-service` | Write the config without installing or starting the service | | `--profile NAME` | Profile naming the controller to enroll against (required) | ### Examples ```sh # Enroll this machine sparkwing cluster runners add --profile prod --name dev-laptop # Enroll with a capacity ceiling and labels sparkwing cluster runners add --profile prod --name build-box --max-concurrent 4 --contribution 4,8gb --labels linux,arch=amd64 # Write the config and supervise the agent yourself sparkwing cluster runners add --profile prod --name dev-laptop --no-service ``` ## `sparkwing cluster runners remove` Stop the runner service and revoke its token Reads the token out of the agent config, stops and removes the user service, then revokes that token on the profile's controller. The service stops first, so a claim in flight finishes against a credential that still authenticates. A prefix the controller reports as anything but a runner token is refused, naming what it found. A service file that runs a different agent config is left alone. The config file stays on disk holding the revoked token; 'runners add --force' replaces it. ### Flags | Flag | Description | |---|---| | `--config PATH` | Agent config to read the token from (default: ~/.config/sparkwing/agent.yaml) | | `--no-service` | Revoke the token without touching the service | | `--profile NAME` | Profile naming the controller that issued the token (required) | ### Examples ```sh # Retire this machine sparkwing cluster runners remove --profile prod ``` ## `sparkwing cluster status` Connectivity + fleet + queue health check against a remote cluster Answers "is this cluster alive?" in one command. Runs the connectivity / auth probes from 'profiles test' plus cluster- state probes that hit /api/v1/agents, /api/v1/pool, /api/v1/triggers (status=claimed), and /api/v1/runs?since=24h. Sections: CONNECTIVITY controller / auth / logs / gitcache FLEET agents (connected vs stale) + warm-runner pool QUEUE stuck triggers + recent-run success rate Exit 0 when every probe is ok or warn; exit 1 when any probe fails (auth reject, controller down, HTTP 5xx). Warnings are informational -- low success rate, empty pool, stale agents -- and don't change the exit code so scripts can still condition on "is the cluster reachable at all?". ### Flags | Flag | Description | |---|---| | `--profile NAME` | Profile name (required) | | `-o, --output FMT` | Output format: pretty\|json | ### Examples ```sh # Quick-check prod sparkwing cluster status --profile prod # Structured output for a status dashboard sparkwing cluster status --profile prod -o json ``` ## `sparkwing cluster tokens` Manage controller API tokens All subcommands resolve controller URL + admin bearer from the profile named by --profile. Token creation prints the raw value to stdout once -- save it before leaving this command. ### Subcommands - `create` -- Mint a new API token - `list` -- List token prefixes + metadata - `revoke` -- Mark a token revoked - `lookup` -- Print metadata for a single token - `rotate` -- Mint a replacement token with a grace window - `set-metered` -- Mark an existing token as one credits pay for ## `sparkwing cluster tokens create` Mint a new API token Creates a token of the given --type scoped to --principal. Comma-separated --scope lists which API surfaces the token may call. The raw token is printed to stdout exactly once; after this command exits it cannot be recovered. ### Flags | Flag | Description | |---|---| | `--type KIND` | Token type: user \| runner \| service (required) | | `--principal NAME` | Name identifying the token holder (required) | | `--scope CSV` | Comma-separated scopes; use sparkwing docs read --topic auth for the supported set | | `--ttl DURATION` | Token lifetime (30d, 720h, and similar durations). 0 = never expires | | `--metered` | Mark the token as one whose node claims cost credits | | `--profile NAME` | Profile name (required) | ### Examples ```sh # Mint a service token with write scopes sparkwing cluster tokens create --type service --principal deploy-bot --scope runs.read,runs.write --profile prod # Mint a user token that expires in 30 days sparkwing cluster tokens create --type user --principal fictional-user --scope admin --ttl 720h --profile prod # Mint a metered cloud runner token sparkwing cluster tokens create --type runner --principal agent:cloud-pool --scope nodes.claim --metered --profile prod ``` ## `sparkwing cluster tokens list` List token prefixes + metadata Prints the non-secret prefix + metadata (type, principal, scopes, last-used) for every token. The raw token value is never printed by this command. The SCOPES column shows the comma-separated scope set granted to each token. Tokens carrying the controller's "admin" superset render as "*" since admin short-circuits every other scope check. An empty scope set renders as "-". Use -o json to get a structured array with explicit scope arrays, suitable for piping into jq. ### Flags | Flag | Description | |---|---| | `--type KIND` | Filter by token type | | `--include-revoked` | Include revoked tokens in the output | | `-o, --output FORMAT` | Output format: pretty \| json \| plain (default: pretty on TTY, json when piped) | | `--profile NAME` | Profile name (required) | ### Examples ```sh # List all active tokens sparkwing cluster tokens list --profile prod # Audit every revoked service token sparkwing cluster tokens list --type service --include-revoked --profile prod # Inspect the warm-runner pool token's scopes as JSON sparkwing cluster tokens list --profile prod -o json | jq 'select(.principal=="agent:fictional-runner") | .scopes' ``` ## `sparkwing cluster tokens lookup` Print metadata for a single token Prints the JSON metadata for a token given its non-secret prefix. Useful for confirming principal + scopes before revoking or rotating. ### Flags | Flag | Description | |---|---| | `--prefix PREFIX` | Non-secret token prefix (required) | | `--profile NAME` | Profile name (required) | ### Examples ```sh # Inspect a token before revoking sparkwing cluster tokens lookup --prefix a1b2c3d4 --profile prod ``` ## `sparkwing cluster tokens revoke` Mark a token revoked Subsequent requests using the token receive HTTP 401. Revocation is immediate and irreversible. ### Flags | Flag | Description | |---|---| | `--prefix PREFIX` | Non-secret token prefix (from 'tokens list') (required) | | `--profile NAME` | Profile name (required) | ### Examples ```sh # Revoke a leaked token sparkwing cluster tokens revoke --prefix a1b2c3d4 --profile prod ``` ## `sparkwing cluster tokens rotate` Mint a replacement token with a grace window Creates a new token and schedules the old token for revocation after --grace. During the grace window, both tokens work, which lets callers cycle credentials without downtime. The controller caps --grace at 7 days, and revoking the old prefix cuts a grace window short. ### Flags | Flag | Description | |---|---| | `--prefix PREFIX` | Non-secret prefix of the token to rotate (required) | | `--grace DURATION` | Window during which the old token still authenticates (maximum 168h) (default: 24h) | | `--ttl DURATION` | TTL of the new token (0 = preserve the old token's remaining TTL) | | `--profile NAME` | Profile name (required) | ### Examples ```sh # Rotate a token with a 48h grace window sparkwing cluster tokens rotate --prefix a1b2c3d4 --grace 48h --profile prod ``` ## `sparkwing cluster tokens set-metered` Mark an existing token as one credits pay for Sets or clears the metering marker on a token that is already minted, which is how a warm pool already running starts costing credits without a new credential. Metering is an operator decision: a runner's own labels never make its work billable. A claim by a metered token reserves a minute of cloud runner time and is refused when the balance cannot cover it. ### Flags | Flag | Description | |---|---| | `--prefix PREFIX` | Non-secret token prefix (from 'tokens list') (required) | | `--metered BOOL` | true to charge this token's claims, false to stop (required) | | `--profile NAME` | Profile name (required) | ### Examples ```sh # Start charging the warm pool sparkwing cluster tokens set-metered --prefix swr_a1b2c3d4 --metered true --profile prod # Stop charging a token sparkwing cluster tokens set-metered --prefix swr_a1b2c3d4 --metered false --profile prod ``` ## `sparkwing cluster users` Manage dashboard login users Seeds admin credentials in the controller's users table, used by the web pod's login flow. Connection info comes from the profile named by --profile. ### Subcommands - `add` -- Create a dashboard user - `list` -- Print every user - `delete` -- Remove a dashboard user ## `sparkwing cluster users add` Create a dashboard user Prompts for a password on stdin with echo disabled when stdin is a TTY (the password is not shown on-screen or recorded in shell history). Passing --password skips the prompt -- useful for CI seed flows but leaks via shell history if used interactively. --scope sets what the account's dashboard sessions may reach; omitting it grants admin. The first account on a controller must be an admin, so a --scope list that omits admin is refused until one exists. ### Flags | Flag | Description | |---|---| | `--name NAME` | Dashboard username (required) | | `--password PASSWORD` | Password (omit to prompt interactively) | | `--scope LIST` | Comma-separated scopes (omit to grant admin; the first account must include admin) | | `--profile NAME` | Profile name (required) | ### Examples ```sh # Interactive add of the first admin sparkwing cluster users add --name fictional-user --profile prod # Read-only account, once an admin exists sparkwing cluster users add --name viewer --scope runs.read,logs.read --profile prod # Non-interactive add for CI sparkwing cluster users add --name ci-bot --password "$CI_BOT_PW" --profile prod ``` ## `sparkwing cluster users delete` Remove a dashboard user Deletes the user row, every session that user holds, and revokes every token minted under that principal name except the token this request authenticates with, in one transaction. The sessions and tokens are revoked and the auth cache on the serving replica is cleared; auth.md describes the windows that remain elsewhere. ### Flags | Flag | Description | |---|---| | `--name NAME` | Dashboard username to remove (required) | | `--profile NAME` | Profile name (required) | ### Examples ```sh # Delete a user sparkwing cluster users delete --name fictional-user --profile prod ``` ## `sparkwing cluster users list` Print every user Prints name, scopes, created_at, and last_login_at for every user in the controller's users table. ### Flags | Flag | Description | |---|---| | `--profile NAME` | Profile name (required) | ### Examples ```sh # List users sparkwing cluster users list --profile prod ``` ## `sparkwing cluster webhooks` Connect, inspect, and replay GitHub webhooks Manage GitHub webhooks through the installed 'gh' command and its credentials. 'connect' registers a repository against a pipeline on both sides and 'disconnect' removes it; the deliveries view joins delivery records with Sparkwing triggers and run outcomes. ### Subcommands - `connect` -- Connect a GitHub repository to a pipeline - `disconnect` -- Remove a repository's webhook and its controller binding - `list` -- List GitHub hooks configured on a repo - `deliveries` -- List recent deliveries for a hook, joined with trigger state - `replay` -- Queue a redelivery of a specific delivery UUID ### Examples ```sh # Connect a repository to a pipeline sparkwing cluster webhooks connect --profile prod --repo your-org/my-app --pipeline build # List hooks on a repo sparkwing cluster webhooks list --repo your-org/my-app # Recent deliveries for a hook sparkwing cluster webhooks deliveries --repo your-org/my-app --hook 123456789 --since 1h --profile prod ``` ## `sparkwing cluster webhooks connect` Connect a GitHub repository to a pipeline Registers both sides of a webhook in one command. It generates a signing secret, stores the binding on the controller, creates or updates the repository's webhook through 'gh' so it posts to the controller's delivery URL for this pipeline, and asks GitHub for a ping so the answer the controller gave is part of the output. The secret is never printed and never passed in a command line; the controller stores it and verifies every delivery's HMAC against it. Re-running the command rotates the secret on both sides. The delivery URL comes from the controller: its --external-url when it announces one, and otherwise the URL this command reached it at. ### Flags | Flag | Description | |---|---| | `--repo OWNER/NAME` | GitHub repo (owner can be omitted if gh has a default) (required) | | `--pipeline NAME` | Pipeline the deliveries fire (required) | | `--events LIST` | Comma-separated GitHub events (default: push,pull_request) | | `--profile NAME` | Profile name (the controller that stores the binding) (required) | ### Examples ```sh # Connect push and pull-request triggers sparkwing cluster webhooks connect --profile prod --repo your-org/my-app --pipeline build # Connect pushes only sparkwing cluster webhooks connect --profile prod --repo your-org/my-app --pipeline build --events push ``` ## `sparkwing cluster webhooks deliveries` List recent deliveries for a hook, joined with trigger state Fetches recent deliveries via 'gh api' and, for each one, looks up the matching sparkwing trigger by GITHUB_DELIVERY env stamp. Surfaces TRIGGER_ID + RUN_STATUS columns so operators see GitHub-side status alongside the run it produced. --since filters deliveries client-side (GitHub's API does not take a time filter). Default: 24h. ### Flags | Flag | Description | |---|---| | `--repo OWNER/NAME` | GitHub repo (required) | | `--hook N` | GitHub hook id from 'webhooks list' (required) | | `--since DURATION` | Only deliveries newer than this (default: 24h) | | `-o, --output FMT` | Output format (json\|table) | | `--profile NAME` | Profile name (used for trigger/run lookups) (required) | ### Examples ```sh # Recent deliveries for a hook sparkwing cluster webhooks deliveries --repo your-org/my-app --hook 123456789 --since 1h --profile prod ``` ## `sparkwing cluster webhooks disconnect` Remove a repository's webhook and its controller binding Removes the binding the controller verifies deliveries against, then deletes the webhook on GitHub through 'gh'. The controller answers with the webhook it was bound to, so a repository connected to two controllers under the same pipeline name loses only this one. A webhook written by hand is matched by its pipeline path instead, and every deleted hook is printed with its URL. Either side already being absent is reported rather than failing, so a half-finished connect is cleaned up by running this once. ### Flags | Flag | Description | |---|---| | `--repo OWNER/NAME` | GitHub repo (required) | | `--pipeline NAME` | Pipeline the webhook fires (required) | | `--profile NAME` | Profile name (the controller holding the binding) (required) | ### Examples ```sh # Disconnect a repository sparkwing cluster webhooks disconnect --profile prod --repo your-org/my-app --pipeline build ``` ## `sparkwing cluster webhooks list` List GitHub hooks configured on a repo Calls 'gh api /repos/OWNER/NAME/hooks' and prints id, derived pipeline, active flag, last-delivery status, and URL. The PIPELINE column is parsed from the hook URL path (/webhooks/github/). Hooks posting to the older unscoped /webhooks/github endpoint render as "(unscoped)" so operators can spot them for cleanup. Non-sparkwing hooks render as "(non-sparkwing)". ### Flags | Flag | Description | |---|---| | `--repo OWNER/NAME` | GitHub repo (owner can be omitted if gh has a default) (required) | | `-o, --output FMT` | Output format (json\|table) | ### Examples ```sh # List hooks on a repo sparkwing cluster webhooks list --repo your-org/my-app ``` ## `sparkwing cluster webhooks replay` Queue a redelivery of a specific delivery UUID Requests another attempt for the selected GitHub webhook delivery. Read the hook's deliveries to inspect the resulting attempt. ### Flags | Flag | Description | |---|---| | `--repo OWNER/NAME` | GitHub repo (required) | | `--hook N` | GitHub hook id (required) | | `--delivery UUID` | Delivery GUID to redeliver (required) | ### Examples ```sh # Redeliver a webhook attempt sparkwing cluster webhooks replay --repo your-org/my-app --hook 123456789 --delivery 00000000-0000-4000-8000-000000000001 ``` ## `sparkwing cluster worker` Claim triggers from a profile's controller and run them in-process Polls the trigger queue at the selected profile's controller and executes each claimed trigger in-process on this host. Use sparkwing-runner for --runner k8s|warm and image or service-account flags. Run against a remote controller via --profile prod (or whichever profile), or against a local 'sparkwing serve start' via --profile local. ### Flags | Flag | Description | |---|---| | `--profile PROFILE` | Profile name from profiles.yaml (required) | | `--poll DUR` | Claim poll interval when the queue is empty (default: 1s) | | `--heartbeat DUR` | Claim-lease heartbeat cadence (default: 5s) | ### Examples ```sh # Run against a named profile sparkwing cluster worker --profile local # Faster polling for tight dev loops sparkwing cluster worker --profile local --poll 250ms ``` ======================================== # DOC: cli-commands (v0.56.0) ======================================== # CLI reference: sparkwing commands Every `sparkwing commands` command, flag, and argument, generated from the CLI's own command registry. All command groups are indexed in [cli-reference.md](cli-reference.md). ## `sparkwing commands` Index of every command: one path and synopsis per line Search command paths and synopses with --query; every word must match. --path narrows to a subtree, with or without the leading sparkwing. Results are lexical, at most 40 by default. JSON ends with a kind:page record reporting total, returned, truncated and next_cursor. Continue with --cursor and the same filters, or --limit 0 for every match. Rows carry path, synopsis and full-tree subcommand_count. Read a selected command with --help. Hidden commands require --include-hidden. Plain prints paths only, with continuation on stderr. --format markdown exports the full reference and rejects query/pagination flags. --split-dir writes generated files. ### Flags | Flag | Description | |---|---| | `-q, --query TEXT` | Match every word against paths and synopses | | `--limit N` | Maximum records; 0 returns every remaining match (default: 40) | | `--cursor CURSOR` | Continue after next_cursor with the same filters and binary version | | `--format markdown` | Export the full command reference as Markdown | | `-o, --output FORMAT` | Output format: pretty \| json \| plain (default: pretty on TTY, json when piped) | | `--split-dir DIR` | With --format markdown: write one page per top-level command group into DIR (plus a cli-reference.md index), pruning stale generated pages | | `--path PREFIX` | Only emit commands at or under PREFIX, matched by whole path components, with or without the leading 'sparkwing' (runs, sparkwing runs, runs list, and similar paths); a prefix matching nothing is an error | | `--include-hidden` | Also emit Hidden:true commands (default: skip) | ### Examples ```sh # Find status commands sparkwing commands --query status # Just the pipelines subtree sparkwing commands --path pipeline # The same subtree, fully qualified sparkwing commands --path "sparkwing pipeline" # All paths, one per line sparkwing commands --limit 0 -o plain ``` ======================================== # DOC: cli-completion (v0.56.0) ======================================== # CLI reference: sparkwing completion Every `sparkwing completion` command, flag, and argument, generated from the CLI's own command registry. All command groups are indexed in [cli-reference.md](cli-reference.md). ## `sparkwing completion` Emit a shell completion script (bash|zsh|fish) Prints a completion script for the selected shell. Source it from your shell rc: \# bash source <(sparkwing completion --shell bash --output plain) \# zsh (add 'autoload -U compinit; compinit' once above) source <(sparkwing completion --shell zsh --output plain) \# fish sparkwing completion --shell fish --output plain | source zsh and fish get per-item descriptions; bash is name-only because compgen lacks the facility. ### Flags | Flag | Description | |---|---| | `-o, --output FORMAT` | Output format: pretty \| json \| plain (pretty on a terminal, json when piped) | | `--shell NAME` | bash \| zsh \| fish (required) | ### Examples ```sh # Wire completion for the current zsh session source <(sparkwing completion --shell zsh --output plain) # Install persistent completion for fish sparkwing completion --shell fish --output plain > ~/.config/fish/completions/sparkwing.fish ``` ======================================== # DOC: cli-configure (v0.56.0) ======================================== # CLI reference: sparkwing configure Every `sparkwing configure` command, flag, and argument, generated from the CLI's own command registry. All command groups are indexed in [cli-reference.md](cli-reference.md). ## `sparkwing configure` Configure laptop-local settings Configure this machine. 'init' prepares the configuration directory and reports its contents. 'profiles' manages controller connections. 'xrepo' registers local repositories. Manage controller users and tokens with 'sparkwing cluster'. Manage secrets with 'sparkwing secrets'. ### Subcommands - `init` -- Set up ~/.config/sparkwing/ and report laptop-level config status - `profiles` -- Manage connection profiles for remote controllers - `xrepo` -- Manage the laptop-local repo registry ### Examples ```sh # First-time laptop setup sparkwing configure init # Status of laptop config sparkwing configure init -o json # List profiles sparkwing configure profiles list # Add a new profile sparkwing configure profiles add --name prod --controller https://api.sparkwing.example --token $TOKEN # Register the current repo with the cross-repo registry sparkwing configure xrepo add ``` ## `sparkwing configure init` Set up ~/.config/sparkwing/ and report laptop-level config status Idempotent setup + status command for laptop-level sparkwing config. Creates ~/.config/sparkwing/ if it doesn't exist, then reports which config files are present (profiles.yaml, repos.yaml, secrets.env), the running CLI + Go toolchain version, and a curated list of next-step commands. Pairs with the per-project flow: use this one on a fresh laptop after install, then run 'sparkwing pipeline new --name ' inside each project to scaffold .sparkwing/ + your first pipeline in one step (no separate init needed). Re-running on an already-set-up laptop re-applies 0700 to ~/.config/sparkwing/ and reports each config file's mode, naming any that group or other users can read. --dry-run skips both the mkdir and the permission fix so the command reports existing state. Run inside a sparkwing project, it also reports whether this checkout's declared git hooks fire, and names the command that arms them. It installs nothing and changes no git configuration. ### Flags | Flag | Description | |---|---| | `-o, --output FORMAT` | Output format: pretty \| json \| plain (default: pretty on TTY, json when piped) | | `--dry-run` | Probe + report without creating or tightening ~/.config/sparkwing/ | ### Examples ```sh # First-time laptop setup sparkwing configure init # Status of laptop config (agent-readable) sparkwing configure init -o json # Probe without writing anything sparkwing configure init --dry-run ``` ## `sparkwing configure profiles` Manage connection profiles for remote controllers Profile config lives at $SPARKWING_PROFILES (if set), else $XDG_CONFIG_HOME/sparkwing/profiles.yaml, else ~/.config/sparkwing/profiles.yaml. Permissions on save are 0600. Every human-driven client command (tokens, users, runs retry/cancel/prune/logs, gc) reads connection info from the selected profile via --profile NAME. No --controller/--token flags exist on other commands; profiles are the only config surface. ### Subcommands - `add` -- Register a new connection profile - `list` -- Print every registered profile - `show` -- Print one profile's full config - `remove` -- Delete a profile - `duplicate` -- Copy one profile's config into another - `set` -- Update fields on an existing profile - `test` -- Probe controller/auth/logs/gitcache for one profile ## `sparkwing configure profiles add` Register a new connection profile Creates a new entry in profiles.yaml. --name and --controller are required; the token is optional. --token-stdin reads the token from stdin and prompts without echo when stdin is a terminal; prefer it over --token, which is visible to other processes in the process list and recorded in shell history. Configure storage and service backends by editing profiles.yaml. ### Flags | Flag | Description | |---|---| | `--name NAME` | Profile name (unique per profiles.yaml) (required) | | `--controller URL` | Controller base URL (required) | | `--token TOKEN` | Bearer token, visible to other processes and shell history (omit for local/unauthed stacks) | | `--token-stdin` | Read the bearer token from stdin, prompting without echo on a terminal | ### Examples ```sh # Add a prod profile, prompting for the token sparkwing configure profiles add --name prod --controller https://api.sparkwing.example --token-stdin # Add a prod profile from a piped token printf %s "$TOKEN" | sparkwing configure profiles add --name prod --controller https://api.sparkwing.example --token-stdin # Add a local profile without auth sparkwing configure profiles add --name local --controller http://127.0.0.1:4344 ``` ## `sparkwing configure profiles duplicate` Copy one profile's config into another Copies the source profile into a new destination profile. The destination name must be unused. ### Flags | Flag | Description | |---|---| | `--src NAME` | Source profile name (required) | | `--dst NAME` | Destination profile name (must not exist yet) (required) | ### Examples ```sh # Branch prod into a staging-prod profile sparkwing configure profiles duplicate --src prod --dst staging-prod ``` ## `sparkwing configure profiles list` Print every registered profile Prints a table of profile name, controller URL, logs URL, and token. JSON is one profile per line; the token is redacted in every mode. ### Flags | Flag | Description | |---|---| | `-o, --output FORMAT` | Output format: pretty \| json \| plain (default: pretty on TTY, json when piped) | ### Examples ```sh # List profiles sparkwing configure profiles list # Agent-readable record sparkwing configure profiles list -o json ``` ## `sparkwing configure profiles remove` Delete a profile Removes the named entry from profiles.yaml. ### Flags | Flag | Description | |---|---| | `--name NAME` | Profile name to remove (required) | ### Examples ```sh # Remove a stale profile sparkwing configure profiles remove --name old-stage ``` ## `sparkwing configure profiles set` Update fields on an existing profile Only flags you pass are overwritten. --token="" explicitly clears the token (empty value, not an omitted flag), and --token-stdin with empty input clears it too. --token-stdin reads the token from stdin and prompts without echo when stdin is a terminal; prefer it over --token, which is visible to other processes in the process list and recorded in shell history. Use --show-token on 'profiles show' afterward to confirm. ### Flags | Flag | Description | |---|---| | `--name NAME` | Profile name to mutate (required) | | `--controller URL` | New controller URL | | `--token TOKEN` | New bearer token, visible to other processes and shell history (empty string clears) | | `--token-stdin` | Read the new bearer token from stdin, prompting without echo on a terminal | ### Examples ```sh # Rotate a profile's token sparkwing configure profiles set --name prod --token-stdin # Change a profile's controller sparkwing configure profiles set --name prod --controller https://api.sparkwing.example ``` ## `sparkwing configure profiles show` Print one profile's full config Prints all fields of the profile named by --name. Token is redacted unless --show-token is passed. ### Flags | Flag | Description | |---|---| | `--name NAME` | Profile name (required) | | `--show-token` | Print the raw token (redacted by default) | ### Examples ```sh # Show a named profile sparkwing configure profiles show --name prod # Show a named profile with the raw token sparkwing configure profiles show --name prod --show-token ``` ## `sparkwing configure profiles test` Probe controller/auth/logs/gitcache for one profile Sequentially checks the profile's controller (/api/v1/health), auth (/api/v1/runs?limit=1 + /api/v1/auth/whoami), logs service (if configured), and gitcache (if configured). Each probe prints ok / warn / fail along with latency and any error detail. Exit code is non-zero when any probe fails. Missing optional services (logs, gitcache) count as warn, not fail, so a minimally-configured laptop profile can still exit 0. ### Flags | Flag | Description | |---|---| | `--profile NAME` | Profile name (required) | | `-o, --output FMT` | Output format (json\|table) | ### Examples ```sh # Probe a named profile sparkwing configure profiles test --profile prod # JSON for scripting sparkwing configure profiles test --profile prod -o json ``` ## `sparkwing configure xrepo` Manage the laptop-local repo registry The registry maps pipeline names to local checkouts so cross-repo RunAndAwait calls resolve without hardcoded WithFreshRepo annotations. Auto-populated when you run 'sparkwing run ' in a .sparkwing/-bearing repo (set SPARKWING_NO_AUTO_REGISTER=1 to disable). ### Subcommands - `list` -- List registered checkouts and their pipelines - `add` -- Register a checkout - `remove` -- Remove a registered checkout - `prune` -- Remove checkouts whose pipeline directory is gone ### Examples ```sh # Register the current checkout sparkwing configure xrepo add # Show the fleet the registry reaches sparkwing configure xrepo list # Drop entries whose checkout is gone sparkwing configure xrepo prune ``` ## `sparkwing configure xrepo add` Register a checkout Registers a checkout explicitly. The path defaults to the current directory. ### Arguments - `[path]` (optional) -- Checkout path; defaults to the current directory ### Examples ```sh # Register the current checkout sparkwing configure xrepo add # Register another checkout sparkwing configure xrepo add ../service ``` ## `sparkwing configure xrepo list` List registered checkouts and their pipelines Shows each registered checkout, its status, and the pipelines it provides. ### Flags | Flag | Description | |---|---| | `-o, --output FORMAT` | Output format: json \| table | | `--pipelines` | Include pipeline names (default: true) | ### Examples ```sh # List registered checkouts sparkwing configure xrepo list # Emit one JSON record per checkout sparkwing configure xrepo list -o json # Skip pipeline discovery sparkwing configure xrepo list --pipelines=false ``` ## `sparkwing configure xrepo prune` Remove checkouts whose pipeline directory is gone Removes registered checkouts that no longer contain a .sparkwing directory. ### Examples ```sh # Remove stale registry entries sparkwing configure xrepo prune ``` ## `sparkwing configure xrepo remove` Remove a registered checkout Removes every registry entry matching a path or basename. ### Arguments - `` (required) -- Registered path or basename to remove ### Examples ```sh # Remove a checkout by basename sparkwing configure xrepo remove service ``` ======================================== # DOC: cli-crons (v0.56.0) ======================================== # CLI reference: sparkwing crons Every `sparkwing crons` command, flag, and argument, generated from the CLI's own command registry. All command groups are indexed in [cli-reference.md](cli-reference.md). ## `sparkwing crons` Arm, inspect and drive this host's local pipeline schedules Runs the pipelines that declare an on.schedule cadence in their .sparkwing/sparkwing.yaml, on this machine, from this home's runs store. Declaring a cadence does not arm it. `sparkwing crons install` arms a repo's schedules on the host it is run from, and installs one OS timer -- a systemd user timer on Linux, a launchd agent on macOS -- that calls `sparkwing crons tick` every minute. Sparkwing evaluates every cron expression itself inside that tick, so the machine holds one timer however many schedules are armed. Each tick resolves every due instant exactly once: it launches the run, skips it when the previous scheduled run is still going and the policy is skip, or records it missed when it fell outside the catch-up window. A scheduled run carries the trigger source "schedule" and executes through the same detached path as `sparkwing run --sw-detached`. Arming pins by default: install compiles the pipeline and keeps that binary, so a checkout updated afterwards does not change what runs unattended. Re-run install to move the pin, `crons unlock` to follow the checkout again, and `crons set` to override a declared cadence on this host alone. --profile NAME points every verb but tick, lock and unlock at a controller instead of this host. `crons install --profile` pushes the repo's `where: controller` entries to it, pinned at HEAD unless --follow; the controller evaluates them from a loop of its own, one evaluator per store, and each fire becomes a trigger the cluster clones and runs. ### Subcommands - `install` -- Arm a repo's declared schedules on this host and install the OS timer - `uninstall` -- Disarm a repo's schedules, and remove the timer when nothing is left - `disarm` -- Remove one schedule from this host - `lock` -- Pin one schedule to the checkout as it stands - `unlock` -- Let one schedule follow the checkout again - `set` -- Override a declared cadence on this host - `reset` -- Drop this host's override of a schedule - `status` -- Report the OS timer, the last tick, and what is armed here - `list` -- List the schedules armed on this host - `show` -- Show one schedule's full record and its recent fires - `next` -- Show the instants a schedule fires next - `pause` -- Stop a schedule firing, keeping it armed - `resume` -- Let a paused schedule fire again - `run` -- Launch a schedule's pipeline now - `tick` -- Evaluate every armed schedule once (the OS timer's entry point) ### Examples ```sh # Arm this repo's schedules on this host sparkwing crons install # See what is armed and when it next fires sparkwing crons list # Check the timer and the last tick sparkwing crons status # Push this repo's controller schedules sparkwing crons install --profile prod ``` ## `sparkwing crons disarm` Remove one schedule from this host Deletes one schedule, its fire history and its pinned pipeline binary. Every other schedule of the same pipeline and repo stays armed. To stop a schedule without losing its history, pause it instead. ### Arguments - `NAME` (required) -- Schedule id, repo/pipeline[/name], pipeline/name, or a unique pipeline name ### Flags | Flag | Description | |---|---| | `--profile NAME` | Profile name; omit for this host | | `-o, --output FMT` | Output format: pretty\|json\|plain | ### Examples ```sh # Remove one named entry sparkwing crons disarm sweep/quick ``` ## `sparkwing crons install` Arm a repo's declared schedules on this host and install the OS timer Reads .sparkwing/sparkwing.yaml, records every on.schedule entry that declares "where: local" against this home, and ensures the OS timer that runs the tick. An entry declaring "where: controller" is reported and left alone: this host does not fire it. Each pipeline is compiled first and has to appear in the binary's own description, because a schedule fires unattended: a pipeline that will not build is refused here before unattended execution. That compile is also the pin: the binary is copied under the sparkwing home and recorded with the checkout's HEAD, so every fire runs what was armed however the checkout moves afterwards. --follow arms without a pin, and each fire compiles the checkout. --no-prove skips the compile, and so pins nothing. --only arms a subset, naming pipelines or pipeline/name entries; a name the repo does not declare is refused before anything is written. A repo that declares no schedule is reported as nothing to arm and installs no timer. Re-running install is the explicit update: it re-pins at the current checkout, republishes what the repo declares, marks a pipeline that stopped declaring a cadence undeclared, and re-bases this host's overrides onto the new declaration. Pause state, cursor, fire history and the override values survive. Arming is per host. Another machine runs the same schedule only when the schedule is also armed on that machine. --profile NAME pushes the repo's "where: controller" entries to that controller instead, and reports the "where: local" ones as this host's. The push needs a git origin, because the cluster clones the source at each fire; it pins every fire to the checkout's HEAD unless --follow, which clones the branch tip. A HEAD no remote branch carries is refused, because every fire would fail at the clone; uncommitted edits are a warning, since the pushed commit is what runs. Re-running the push is the explicit update, and it moves the pin. ### Flags | Flag | Description | |---|---| | `--profile NAME` | Profile name; omit for this host | | `--repo DIR` | Repo directory (default: discovered via nearest .sparkwing/) | | `--fleet` | Arm every registered repo instead of one | | `--only NAMES` | Arm only these pipelines or pipeline/name entries (comma-separated or repeatable) | | `--follow` | Arm without pinning, so every fire compiles the checkout | | `--no-prove` | Arm without compiling the pipelines first, which pins nothing | | `-o, --output FMT` | Output format: pretty\|json\|plain | ### Examples ```sh # Arm the current repo sparkwing crons install # Arm a different repo sparkwing crons install --repo /path/to/repo # Arm two entries only sparkwing crons install --only nightly,sweep/quick # Arm without pinning sparkwing crons install --follow # Arm every registered repo sparkwing crons install --fleet # Push the controller entries to a cluster sparkwing crons install --profile prod # Push them following the branch tip sparkwing crons install --profile prod --follow ``` ## `sparkwing crons list` List the schedules armed on this host One row per schedule: its id, its repo/pipeline name, the cron expression and zone it is read in, when it next fires, when it last fired, that fire's outcome, and whether it is armed, paused, or undeclared. Schedules the repo no longer declares are hidden behind a count; --all shows them. They keep their history and never fire. ### Flags | Flag | Description | |---|---| | `--profile NAME` | Profile name; omit for this host | | `--all` | Include schedules the repo no longer declares | | `-o, --output FMT` | Output format: pretty\|json\|plain | ### Examples ```sh # What is armed here sparkwing crons list # Include withdrawn schedules sparkwing crons list --all # What a controller evaluates sparkwing crons list --profile prod # Machine-readable (NDJSON) sparkwing crons list -o json ``` ## `sparkwing crons lock` Pin one schedule to the checkout as it stands Compiles the pipeline, keeps that binary under the sparkwing home, and records the checkout's HEAD against the schedule. Every later fire runs that binary, so editing or updating the checkout does not change what an unattended run executes. The pin covers the pipeline the repo declares. Scripts and binaries the pipeline runs from the checkout or from PATH are outside it. ### Arguments - `NAME` (required) -- Schedule id, repo/pipeline[/name], pipeline/name, or a unique pipeline name ### Flags | Flag | Description | |---|---| | `-o, --output FMT` | Output format: pretty\|json\|plain | ### Examples ```sh # Pin a schedule at HEAD sparkwing crons lock nightly ``` ## `sparkwing crons next` Show the instants a schedule fires next Shows upcoming times in each schedule's configured time zone. Supply a schedule name to inspect one expression; omit it to merge upcoming times from every armed schedule. ### Arguments - `NAME` (optional) -- Schedule id, repo/pipeline[/name], pipeline/name, or a unique pipeline name; omit for every armed schedule ### Flags | Flag | Description | |---|---| | `--profile NAME` | Profile name; omit for this host | | `--count N` | How many instants to show (default: 5) | | `-o, --output FMT` | Output format: pretty\|json\|plain | ### Examples ```sh # What fires next on this host sparkwing crons next # Check one expression sparkwing crons next fictional-nightly --count 10 ``` ## `sparkwing crons pause` Stop a schedule firing, keeping it armed A paused schedule still advances its cursor on every tick, so resuming it fires the next due instant instead of replaying the ones that passed while it was paused. ### Arguments - `NAME` (required) -- Schedule id, repo/pipeline[/name], pipeline/name, or a unique pipeline name ### Flags | Flag | Description | |---|---| | `--profile NAME` | Profile name; omit for this host | | `-o, --output FMT` | Output format: pretty\|json\|plain | ### Examples ```sh # Pause a schedule sparkwing crons pause fictional-nightly ``` ## `sparkwing crons reset` Drop this host's override of a schedule Returns the schedule to what the repo declares. The pin, the pause state, the cursor and the fire history are untouched. ### Arguments - `NAME` (required) -- Schedule id, repo/pipeline[/name], pipeline/name, or a unique pipeline name ### Flags | Flag | Description | |---|---| | `--profile NAME` | Profile name; omit for this host | | `-o, --output FMT` | Output format: pretty\|json\|plain | ### Examples ```sh # Run what the repo declares sparkwing crons reset nightly ``` ## `sparkwing crons resume` Let a paused schedule fire again Resumes at the next due instant. The instants that passed while the schedule was paused are behind its cursor and do not run. ### Arguments - `NAME` (required) -- Schedule id, repo/pipeline[/name], pipeline/name, or a unique pipeline name ### Flags | Flag | Description | |---|---| | `--profile NAME` | Profile name; omit for this host | | `-o, --output FMT` | Output format: pretty\|json\|plain | ### Examples ```sh # Resume a schedule sparkwing crons resume fictional-nightly ``` ## `sparkwing crons run` Launch a schedule's pipeline now Runs the pipeline immediately, whatever the cadence says and whether or not the schedule is paused, and records the launch in the schedule's history as a manual fire. The cursor does not move: a manual run is not one of the cadence's due instants, so the next one still fires on time. ### Arguments - `NAME` (required) -- Schedule id, repo/pipeline[/name], pipeline/name, or a unique pipeline name ### Flags | Flag | Description | |---|---| | `--profile NAME` | Profile name; omit for this host | | `-o, --output FMT` | Output format: pretty\|json\|plain | ### Examples ```sh # Run a schedule's pipeline now sparkwing crons run fictional-nightly ``` ## `sparkwing crons set` Override a declared cadence on this host Lays this host's own value over what the repo declares, for the cron expression, the zone, the overlap policy, the catch-up window and the launch's arguments. Everything left unnamed keeps the declared value, and a field named again replaces the previous override. --arg replaces the declared argument set whole, so name every argument the schedule should launch with. The override survives re-arming; `sparkwing crons reset` drops it. `sparkwing crons list` marks an overridden expression with *, and `sparkwing crons show` prints the declared, override and effective value side by side. ### Arguments - `NAME` (required) -- Schedule id, repo/pipeline[/name], pipeline/name, or a unique pipeline name ### Flags | Flag | Description | |---|---| | `--profile NAME` | Profile name; omit for this host | | `--cron EXPR` | Cron expression to run instead of the declared one | | `--tz ZONE` | Zone the expression is read in, such as America/Denver or local | | `--overlap POLICY` | What a due instant does while the previous run is going: skip\|queue | | `--catch-up DUR` | How late a due instant may still fire, such as 6h | | `--arg K=V` | Argument the launch passes (repeatable; replaces the declared set) | | `-o, --output FMT` | Output format: pretty\|json\|plain | ### Examples ```sh # Run it later on this host sparkwing crons set nightly --cron '0 5 * * *' # Read the expression locally sparkwing crons set nightly --tz local # Launch with arguments sparkwing crons set sweep/quick --arg depth=shallow --arg dry-run=true ``` ## `sparkwing crons show` Show one schedule's full record and its recent fires Prints every stored field with absolute times, then the instants that have resolved, newest first: when each was due, when the tick decided it, what it decided, the run it launched and that run's current status, and the reason for any outcome that is not a launch. NAME is a schedule id, a repo/pipeline name, or a bare pipeline name that is unique across this host's schedules. ### Arguments - `NAME` (required) -- Schedule id, repo/pipeline[/name], pipeline/name, or a unique pipeline name ### Flags | Flag | Description | |---|---| | `--profile NAME` | Profile name; omit for this host | | `--fires N` | How many recent fires to show (default: 10) | | `-o, --output FMT` | Output format: pretty\|json\|plain | ### Examples ```sh # Inspect one schedule sparkwing crons show fictional-nightly # Read further back sparkwing crons show fictional-nightly --fires 50 ``` ## `sparkwing crons status` Report the OS timer, the last tick, and what is armed here Answers whether this host is actually evaluating what it armed: whether the timer is installed and running, whether it runs this sparkwing or one that has since moved, when the tick last landed and what it reported, and how many schedules are armed, paused, and undeclared. Exits non-zero when schedules are armed and the timer is not running, runs another binary, or has not ticked in the last few minutes, so a check script can read the exit code. A host with nothing armed is healthy. --profile NAME reads a controller's scheduler instead: its counts, when its loop last ticked, and what that tick reported. ### Flags | Flag | Description | |---|---| | `--profile NAME` | Profile name; omit for this host | | `-o, --output FMT` | Output format: pretty\|json\|plain | ### Examples ```sh # Read the host's scheduler health sparkwing crons status # Machine-readable sparkwing crons status -o json # Read a controller's scheduler sparkwing crons status --profile prod ``` ## `sparkwing crons tick` Evaluate every armed schedule once (the OS timer's entry point) What the systemd timer or launchd agent runs every minute. It takes an exclusive lock so two ticks never resolve the same instant, re-reads the declaration of every schedule that follows its checkout -- a pinned schedule keeps the declaration it was armed with -- evaluates every declared unpaused schedule against its cursor, launches what is due, and records each outcome. Quiet on success: one summary line and the id of each run it launched. It exits non-zero only when the tick itself could not run, so a schedule that fails to launch is recorded against that schedule and the timer stays green. --dry-run prints what this minute would resolve and writes nothing. Run it by hand on a host whose platform has no sparkwing timer, from that machine's own scheduler, once a minute. ### Flags | Flag | Description | |---|---| | `--dry-run` | Evaluate and report without launching or recording anything | | `-o, --output FMT` | Output format: pretty\|json\|plain | ### Examples ```sh # Evaluate every armed schedule once sparkwing crons tick # See what this minute would do sparkwing crons tick --dry-run ``` ## `sparkwing crons uninstall` Disarm a repo's schedules, and remove the timer when nothing is left Deletes every schedule of one checkout, and its fire history, from this home. When no schedule remains armed anywhere, the OS timer goes too: the timer exists to serve armed schedules and nothing else. --fleet disarms every schedule this home holds. --profile NAME deletes the repo's schedules from that controller instead, naming the repo by its git origin. ### Flags | Flag | Description | |---|---| | `--profile NAME` | Profile name; omit for this host | | `--repo DIR` | Repo directory (default: discovered via nearest .sparkwing/) | | `--fleet` | Disarm every schedule this home holds | | `-o, --output FMT` | Output format: pretty\|json\|plain | ### Examples ```sh # Disarm the current repo sparkwing crons uninstall # Disarm everything on this host sparkwing crons uninstall --fleet # Remove this repo from a controller sparkwing crons uninstall --profile prod ``` ## `sparkwing crons unlock` Let one schedule follow the checkout again Drops the pin and the pinned binary, so every later fire compiles the checkout as it stands at that minute, and the tick's refresh reads the repo's declaration again. ### Arguments - `NAME` (required) -- Schedule id, repo/pipeline[/name], pipeline/name, or a unique pipeline name ### Flags | Flag | Description | |---|---| | `-o, --output FMT` | Output format: pretty\|json\|plain | ### Examples ```sh # Follow the checkout again sparkwing crons unlock nightly ``` ======================================== # DOC: cli-daemon (v0.56.0) ======================================== # CLI reference: sparkwing daemon Every `sparkwing daemon` command, flag, and argument, generated from the CLI's own command registry. All command groups are indexed in [cli-reference.md](cli-reference.md). ## `sparkwing daemon` Inspect or refresh the local admission daemon The admission daemon starts on demand when a pipeline needs it. Status never starts one. Restart replaces only an answering daemon with this installed build, using the same drain, durable lease, and reattachment path as automatic version takeover; a stopped daemon stays stopped. Stop drains an answering daemon and launches no successor. ### Subcommands - `status` -- Report whether wingd is running and which build it serves - `restart` -- Refresh an answering wingd to this installed build - `stop` -- Drain an answering wingd and leave it stopped - `recover-state` -- Preserve unreadable daemon state after its holders stop ### Examples ```sh # Machine-readable status sparkwing daemon status -o json # Refresh only if already running sparkwing daemon restart # Stop it and leave it stopped sparkwing daemon stop ``` ## `sparkwing daemon recover-state` Preserve unreadable daemon state after its holders stop Fail-closed recovery for a daemon that cannot parse its durable state. The unreadable bytes may describe leases whose runs still hold host capacity, so first stop or verify those runs, then pass --yes. Recovery holds the daemon election lock, moves state.json to a state.json.corrupt-