Storage

Sparkwing stores immutable bytes in S3 or the filesystem backend. The controller authorizes uploads, downloads and records. Working-tree source bundles use the same signed upload path as other run inputs.

Sparkwing's storage contract separates immutable bytes from the records that authorize a job to read them. The controller signs uploads and downloads. The object backend is S3 or the filesystem signer.

All physical keys are relative to the configured storage root. For --cache-blob-store=s3://bucket/controller, CAS blobs, artifact members and manifests, source bundles, pipeline binaries, staged uploads and logs live under controller/. Records remain database rows referring to those bytes. S3 copy sources, signed downloads (including CloudFront), log listings and deletion use that same prefix. Set bucket measurement to the same URL and scope IAM object permissions to controller/*. Log lifecycle rules cover controller/logs/<class>/. CloudFront uses an empty origin path because its signed paths include the prefix. The filesystem backend anchors every key to its configured data directory through one directory handle.

Bytes and recordsSection anchor link

Each object is named by its SHA-256 digest. Uploads verify the expected length and digest before commit. Downloads verify both at EOF. A missing object or a checksum mismatch cannot produce a successful output read or artifact stage. Memo uses a miss and dependency caches start cold when verification fails. Artifact staging and dependency-cache restore close the verified download before publishing files. A close failure preserves the destination for that download and returns an error.

A record names an output, artifact manifest, memo result, dependency cache, pipeline binary or source bundle. Each reference requires either the exact uploader's receipt or an authorized read edge. Knowing a digest does not grant access. Presence queries reveal only objects the caller may read.

The filesystem and S3 manifest readers derive file metadata from the tenant's immutable object. They verify its full length and digest, then close it before returning metadata. The record transaction checks that metadata against the submitted members and verifies each member's existence and authority.

Each upload ticket must stage and verify its own bytes, including when identical content already exists. Missing staging fails the commit. Before settlement, a retry keeps the staged object; after settlement, it replays the stored receipt under current authority. Staging cleanup follows settlement.

Blob collection uses the store's live reference graph and a 24-hour tombstone grace period. References added after marking protect their bytes at collection. Object age alone cannot delete a live blob. Both physical backends implement the collector's deletion interface. Replaying a deletion of absent bytes succeeds. The controller supplies its configured backend to the retained-data pass. Standalone controllers install S3 or, without a bucket, filesystem storage under their home directory's data directory. Filesystem transfers use signed data upload, object download and log upload routes. Deployment bucket measurement stays separate from tenant accounting and deletes nothing.

Memo and dependency-cache records carry repository and ref scope. Manual writes cannot overwrite verified branch entries. Memo reads its own scope before a verified default-branch fallback. Dependency caches try their own ref, the pull-request base, then the verified default branch.

Outputs use the terminal result's committed record selector and verified blob bytes. Native hosts consume StorageBinding and OutputRecords.ReadResult.

A memo hit copies output and manifest records in one transaction under the executing job's own claim, which then settles the job as memoized. The coordinator never settles a memo.

Dependency-cache keys use the declaration, platform and the first present lockfile inside the workspace. The SDK's DirCache.Key and host share that calculation. Moving the checkout preserves its keys. An escaping path or symlink runs cold. Hosts with a signed-data binding restore through it and refresh the binding after successful execution, so retries save under the final attempt. A refused binding runs cold or skips the save. Local runs keep local archives.

Source and package downloadsSection anchor link

Runners clone the source repository directly. The controller releases the run's permitted Git credential. A machine owner can use --repos during enrollment or runner.repos in the machine file to restrict builds and permit fallback to that machine's Git credentials. Working-tree snapshots travel as signed source bundles.

Package managers use their upstream registries. To use a separate registry mirror, set its tool configuration and list the required environment names in the operator's run.submit_env_allow; Go toolchain and network base names cannot be submitted, so set them in the runner's own environment. Network proxy and CA settings remain available to pipeline processes. Sparkwing provides no Git mirror service or package proxy.

Direct source fetches create a separate repository per checkout, with its own Git objects and no persistent mirror. Failed fetches remove their checkout. Existing directories are refused and preserved. The runner's cleanup covers temporary files and finished runs.

Local pipeline binaries stay in the local binary cache. It stores entries under cache/pipelines/v1/entries in the Sparkwing home. Cache status and pruning operate on that layout. Older layouts have no migration or retirement path. cache info -o json omits legacy_entries and legacy_bytes.

Remote binaries use signed transfers and require authorized build provenance before execution. The binary client implements that check. Host binding is pending, so remote hosts build into their machine's binary cache without shared binary transfers. See Caching for authoring cache declarations.

LogsSection anchor link

Logs use immutable segments with their own signed upload path. Log reads pass through the controller for exact egress accounting. They do not use the blob download path and remain readable when a team reaches its download cap.

A log belongs to one job attempt or one trigger, plan or eval unit. Its generation fences both signing and segment commits. Eval logs retain a separate source-job generation. The uploader verifies bytes, uses batches of sixteen segment URLs and seals the committed segment list with a manifest.

The batch signer derives every key from that identity and uses the run's stored retention class. It checks the backend lifecycle before returning sixteen PUT URLs and one manifest URL with a fifteen-minute expiry. The filesystem signer accepts an identical retry and refuses different bytes under an existing key. These URLs grant no log download access.

S3 lifecycle readiness requires an enabled prefix rule expiring the stored class after at least its days. The controller's maintenance pass deletes each log object once its class has passed since its upload was signed, so set the rule longer than the class (for example the class plus seven days) as a backstop. Tag and size conditions cannot establish expiry for all objects. The backend reads the rules and leaves their configuration to the operator. The filesystem backend provides an expiry pass for the controller's hourly maintenance. It expires objects by their stored class, including retired classes. An identical upload retry preserves the original age. The pass leaves CAS data to its separate reachability rules. The controller runs this pass in its retained-data pass when its physical backend is configured. Standalone controllers construct this backend at startup. Host log bindings remain pending.

The reader uses authorized segment records, verifies each segment before sending its bytes and resumes at an uncompressed byte offset. It reports how many bytes it sent even when a later segment fails. Objects without a persisted log record do not establish a readable log. The tagged components still need controller and host bindings. Both backends open exact log keys for this reader. The S3 backend checks response lengths against the segment or manifest bound before returning a stream.

Run deletion and team purges use the configured physical data backend to remove logs across stored retention classes, including retired classes. The team recheck also removes logs uploaded through a still-valid URL after the first purge. Scoped deletion leaves other tenants, other runs and CAS objects untouched. Object listings select physical cleanup only and grant no read authority. The old sharded filesystem object reader and writer, their locks and the raw artifact plug-in interface are removed. Filesystem deployment measurement uses fs.NewUsageReporter and counts physical files without a logical object layout. It grants no byte-read or record authority.

Server.WithDataObjects installs the same physical backend's manifest reader through Store.SetResultBlobRecords before serving. Terminal result attachment uses the live tenant transaction and verifies physical manifest bytes. A corrupt manifest refuses completion without changing job state. Restoring the bytes permits completion and identical replay. Authenticated signing and record bindings remain separate.

Physical log keys have one owner in internal/blob. Signed slots use LogObjectKey: zero names the final manifest and positive indexes name immutable segments. The physical uploader-to-store proofs use real tenant segment commits and reads, reopen SQLite and verify byte offsets with filesystem and S3 backends. Authentication and award issuance remain fixtures.

Retained log reads select tagged subjects from the tenant's committed run view. The controller rechecks read authority before each persisted segment and each live-ring continuation. It verifies each segment through the configured blob backend before sending its bytes. Filesystem and S3 dashboards use this same reader. Local run-directory logs and the separate raw S3 log reader are removed. The separate live-ring backend and web fallback are removed too. The committed reader owns ring continuation. Deployment bucket measurements use the read-only MeasurementStore and cannot publish or read object payloads.