v0.34.0 Migration Guide

Lost log lines fail the run

A run whose log lines were dropped -- the log store stayed unreachable past the append retry budget -- now fails. The node carries the logs_dropped failure reason, and the logs_drop event carries the lost-line count and the first error.

It used to print status: success with rc 0. The run had done its work; what was missing was any record of it, which is the same false all-clear logs_auth already fails loud for.

Before

WARN logs append dropped after retries ... dropped_total=5
{"event":"run_summary","status":"success"}
$ echo $?
0

After

{"event":"run_summary","status":"failed", ...
  "error":"4 log line(s) lost: the log store stayed unreachable past
  the append retry budget: ..."}
$ echo $?
1

Two things to check when this fires:

1. The backend really is unreachable. The run record's invocation.backends names it. For s3, check the bucket, AWS_REGION, credentials, and SPARKWING_S3_ENDPOINT.
2. A controller-state deployment serves log appends. A controller with no logs service mounted answers every append with 404, which exhausts the budget on every line.

To keep the old behavior, set SPARKWING_LOGS_DROP_POLICY=warn. Only that exact value opts out, so a misspelling cannot silently restore the behavior the variable exists to opt out of.

Such runs also finish at their real speed now. Every line used to pay the full retry budget against a store that was down rather than flaky, which took one 84ms pipeline to 43.9s; a short breaker window after the first exhausted budget brings the same run to 7.1s. Lines dropped inside that window are still counted.

Read commands follow the project's default profile

runs list, runs status, and runs logs resolve the project's defaults.profile when no --profile is passed, so they read the store sparkwing run writes to. They used to read local SQLite regardless of what the project declared.

If you relied on the old behavior to inspect the local mirror, name it: declare a local profile and pass --profile <local>.

Two consequences worth knowing:

• On a machine that shares a bucket but never ran the pipeline, runs list returned empty and runs status printed the right status and then exited 1 -- the exit code came from local SQLite whichever store answered. Both are fixed by the same change.
• --profile NAME now resolves against the project's own profiles: block as well as ~/.config/sparkwing/profiles.yaml. A name declared in both resolves to the user's file, which is the operator's own override of what the repository ships.

Backend specs declare their required fields

A backend spec is now rejected at config load when its type is missing a field it cannot work without:

sqlite with no path is still valid: the profile resolver fills in the host's own state database.

Before

logs:
  type: s3      # no bucket

Loaded clean, rendered as s3:// in sparkwing profile -o json, and failed only when a run tried to write to a bucket named "".

After

sparkwing error: profile "bucket": logs backend type=s3 requires bucket

Add the missing field. The check runs over the user profiles file and the project's profiles: block alike -- neither validated required fields before.