Node protocol

Protocol 2 connects a run host to an SDK process containing pipeline code. The host owns scheduling, placement, admission, retries, deadlines, storage, masking and state transitions. The SDK declares jobs and runs step bodies and closures when the host requests them. The Go and TypeScript SDKs use the same wire types.

The declarations in internal/nodeproto define the wire. The schemas and corpus in internal/nodeproto/conformance/testdata exercise requests, replies, strict plans, errors and generated declarations across SDKs.

Process startup

The host starts a Go pipeline binary as <entry> --sw-node-protocol. It starts a TypeScript entry as node <main> --sw-node-protocol. The working directory is the acquired repository checkout.

TypeScript package imports use Node's native CommonJS and ESM loading. This includes symlinked dependencies installed by pnpm and imports inside job callbacks. Default and named exports follow Node's interop rules.

On Windows the host passes inherited decimal handles through --sw-protocol-handles <request>,<frame> in argv. SDK transport initialization marks the pipes close-on-exec or clears handle inheritance before starting tools. Go package initialization orders this before packages that import the SDK. Independent packages can initialize earlier, so pipeline packages should start commands inside execution callbacks.

Owner, evaluation and attempt processes use the same platform launcher and process-tree cleanup. The Windows host clears inheritance on its protocol-handle copies after a successful or failed launch.

Every process receives hello first. Its runner is the identity granted by the host, including the actual runner name, type and labels. Labels are a required array, including [] when empty; null is rejected. Execution processes also receive a binding to one run and job, with a loopback route URL and a delegated bearer. Discovery and initial planning omit that binding. The SDK keeps the binding in memory. Credentials and protocol handles travel through the private pipe or argv handle argument, respectively.

-> {"id":"1","op":"hello","protocol":2,"engine":{"version":"v0.71.0"},"runner":{"name":"local","type":"local","labels":["os=linux"]},"binding":{"run_id":"r1","job_id":"build","routes":{"url":"http://127.0.0.1:41873","bearer":"example-bearer"}}}
<- {"reply":"1","ok":true,"result":{"protocol":2,"sdk":{"language":"go","version":"v0.71.0"},"features":[]}}

Hello occurs exactly once. A major mismatch returns protocol_mismatch with supported_majors in error details. The selected major applies to the process. Documents and HTTP requests carry their own data without another version stamp. The SDK's optional operations are advertised in hello's features array.

Framing

Each request or frame is one JSON object followed by a newline. Correlation IDs are nonempty and unique within a process. Replies can arrive out of order.

A successful reply contains reply, ok: true and an object result. A failed reply contains reply, ok: false and an inner error object. Exactly one of result and error is present. The common error shape is:

{"error":{"code":"forbidden","message":"the operation is not authorized","hint":"check the declared dependency","details":{"reason":"reference_undeclared"}}}

HTTP errors use that envelope. A protocol reply places the inner object directly in its error field:

-> {"id":"2","op":"describe"}
<- {"reply":"2","ok":false,"error":{"code":"plan_failed","message":"pipeline discovery failed"}}

code and message are required strings. hint and details are optional. Clients branch on codes. Decoders accept additive error fields and future codes. Requests, replies and log records accept additive fields while checking the types and required fields of known members. Plans have a stricter boundary described below. Duplicate or unsolicited replies, malformed frames and oversized frames fail the protocol.

A log frame has ts and optional level, step, event, msg and object attrs. The SDK leaves job attribution to the host. The host assigns the bound job, filters debug records and masks text before storage or truncation. Planning and closure-evaluation logs belong to the run log. Unknown events remain ordinary records.

{"ts":"2026-10-15T12:00:00Z","level":"info","step":"compile","msg":"compiling","attrs":{"packages":3}}

stdout and stderr remain separate from frames. Customer text cannot supply protocol replies. Closing the request pipe cancels active operations and ends the SDK process. Cancellation acknowledgements and the original operation replies allow the host to wait for cleanup before closing the process.

Operations

Describe and plan

describe lists registered pipelines and their inputs, help, risks and secret names. Secrets are declared here once. A pipeline description always includes name and args. Each argument includes name, type and required, with optional description, default, enum, secret classification and owning job_id. A failed pipeline has an error with the shared code, message and optional hint and details, an empty args array and no secret declarations. Other pipelines retain their complete descriptions. The host refuses the failed pipeline with its sanitized cause before planning. It withholds describe output when any pipeline fails because that output may contain unclassified secrets.

isolated_home_required is a fixed refusal for pipelines that must avoid an operational store. Go pipelines return or wrap sparkwing.ErrIsolatedHomeRequired from Plan. The host supplies a temporary HOME and SPARKWING_HOME remedy before intake, even when secret discovery fails. It discards the SDK's message, hint and details for this code and keeps other unclassified output withheld.

-> {"id":"3","op":"describe"}
<- {"reply":"3","ok":true,"result":{"pipelines":[{"name":"ship","args":[{"name":"target","type":"string","required":false,"default":"staging"}],"secrets":[{"name":"DEPLOY_TOKEN"}]}]}}

plan.pipeline names the SDK registration key, which is the YAML entrypoint. args is an explicit object, including {} when empty. run requires run_id and pipeline; run.pipeline is the invoked YAML pipeline name. Optional context includes git, trigger, started_at, dry_run and no_memo.

The host selects the argument schema by registration key. For YAML name: deploy-prod, entrypoint: Deploy, register Deploy. Initial planning and execution replan send Deploy as plan.pipeline and deploy-prod as run.pipeline.

Planning occurs at most once per process. Execution processes reconstruct the plan from the accepted inputs and compare job declarations with the accepted hash. A changed declaration fails with plan_drift before execution. Generated jobs reconstruct their ancestor generator chain from recorded outputs under the same rule. Reconstruction cannot change persisted topology.

-> {"id":"4","op":"plan","pipeline":"ship","args":{"target":"staging"},"run":{"run_id":"r1","pipeline":"ship","dry_run":false,"no_memo":false}}
<- {"reply":"4","ok":true,"result":{"jobs":[{"id":"build","steps":[{"id":"compile"},{"id":"test","needs":["compile"]}],"result_step":"test"},{"id":"deploy","deps":["build"],"retry":{"attempts":1,"backoff_ms":1000},"timeout_ms":600000,"steps":[{"id":"apply"}]}]}}

Plan document

A plan has a required jobs array. Its optional top-level declarations are concurrency, priority, claim_wait_ms, admission_class and source. The host resolves these settings and checks operator ceilings when it accepts the plan. Run identity, accepted hashes, authorization and computed concurrency keys remain host state.

Each job is flat:

| env, requires, prefers, inline, concurrency | Execution declarations |

A pipeline_refs entry reads successful verified default-branch output by default. Its optional from_any_run: true permits other successful runs. The latest-output request must opt out with from_any_run=true, and the controller refuses that request unless the declared reference permits it. Direct job-output reads enforce the same default.

Executable jobs declare one or more steps. An approval has no step body. Steps have id, optional needs, groups, finally, risks, optional, continue_on_error, dry-run declarations and closures. result_step selects the job output. Failure policy is fail_fast or collect_all.

closures may contain skip_if, before_run, after_run and generate arrays, a verify ID and a memo_key ID. These identify code without serializing it. Approvals support skip conditions and separate recovery jobs.

memo.input carries a typed recipe instead of closures.memo_key. Exactly one is required when memo is present. Recipe kinds are files and repo_files (with patterns), tree (with root), env (with names), parts (with ordered JSON parts), and compose (with nested inputs). The executing host evaluates recipes against its checkout and effective job environment under admission. Callbacks return values for the same host canonical JSON hash. Both paths use a versioned memo namespace, independent of SDK language. See Caching for path matching and filesystem behavior.

env maps names to {value, startup?}. Body values take effect after planning, before execution. startup: true and the engine's fixed interpreter variable list apply before process startup. Startup values must leave declarations unchanged. Commands and cleanups inherit the attempt environment.

Runner planning and SDK installation use the same remote base and run.submit_env_allow policy. Install processes receive no job runtime or job-declared environment. Proxy credentials are masked before either process starts.

Concurrency entries contain a group with optional scope, capacity, cost, on_limit and queue_timeout_ms. Omitted values resolve to global, 1, 1 and queue. Explicit zero capacity or cost is invalid. Acceptance checks consistent group definitions. The store derives scoped keys from accepted declarations and the authenticated execution award. Scopes are global, run and box. Policies are queue, fail, skip and cancel_others.

Runner size is a requires label, such as sparkwing-4cpu. source carries optional checkout declarations full_history, depth, tags, submodules and lfs. Presence is significant for all optional numeric and boolean settings.

nodeproto.ValidatePlan validates initial plans and expansions for every placement. It rejects duplicate and unknown keys at every nesting level, null required collections, invalid enums, cycles, unresolved required dependencies and contradictory declarations. Omitted optional fields remain unset. It rejects registered secret values contained in any job, step, group, approval or identifier-bearing reference. Errors name the field path without echoing the secret. Host acceptance separately checks authorization and resolved settings.

Steps and settlement

The host schedules step dependencies. Every run_step carries the positive, node-wide attempt ordinal and an explicit array of terminal step IDs for that job and attempt. settled: [] declares an empty snapshot. Step-condition evals carry the same attempt and settlement state.

-> {"id":"5","op":"run_step","job":"build","step":"compile","attempt":1,"settled":[]}
<- {"reply":"5","ok":true,"result":{"outcome":"success","output":{"packages":3}}}
-> {"id":"6","op":"run_step","job":"build","step":"test","attempt":1,"settled":["compile"]}
<- {"reply":"6","ok":true,"result":{"outcome":"failed","error":{"code":"sdk_error","message":"tests failed"}}}

The SDK checks known, unique settled IDs and all transitive prerequisites before invoking a predicate or body. A terminal step cannot execute again in that attempt. Parallel steps share completed local outputs. Settled predecessors without local output read as their output type's zero value. A new, non-overlapping attempt resets outputs before applying its snapshot. Stale, incomplete or overlapping state fails with protocol_error before mutation.

Body outcomes are success or failed. A body error, panic or output encoding failure returns ok: true with a failed result. Failure to invoke the body returns ok: false. Persisted job outcomes also include cancelled, skipped, superseded and memoized, with a separate reason.

Application retries advance the attempt ordinal under the same claim. Lease-loss recovery uses a separate disruption allowance.

Closures and generated jobs

eval.closure names a declaration's opaque ID. Result shapes depend on its kind:

Executable-job skip_if and memo_key closures run on the executing host under the actual job award after concurrency and machine admission. The host checks SkipIf first, then memo reuse, before invoking the body. A skip or memo hit settles the job and releases its groups. Generator evaluations retain their separate expansion authority.

Failed evals return value: null and a canonical inner error. after_run receives the final step-result shape in input, preserving the final failure. Successful completion passes no error to the Go hook.

Generators run after successful or memoized source outputs. Each generated job names exactly one of the source's generator IDs in its groups. The host wraps the returned jobs in a plan and validates them against existing topology and concurrency declarations. It accepts at most 256 jobs per generator, preserving empty generator groups. Acceptance, replay, source generation and cancellation checks occur at the store boundary. Failed, skipped, cancelled and superseded sources settle their barriers without invoking generators.

Cancellation

cancel selects either a job with an optional step or an active correlation ID in target. Job-only cancellation includes active evals for that job. Target cancellation can select an eval without a job or step. Selectors cannot be mixed or cross the process binding. The SDK acknowledges after the selected operation has stopped. That operation also receives its own reply.

-> {"id":"7","op":"eval","closure":"build/verify/0"}
-> {"id":"8","op":"cancel","target":"7"}
<- {"reply":"7","ok":false,"error":{"code":"cancelled","message":"evaluation cancelled"}}
<- {"reply":"8","ok":true,"result":{}}

Host service routes

Bound SDK calls use the URL and bearer supplied by hello. Every request carries Authorization: Bearer <bearer>. Identifiers are opaque path segments, escaped before transport. The host derives run, job and claim authority from the bearer. Hello supplies the protocol negotiation. HTTP failures use the canonical error envelope, including its code and masked details. A failed child embeds the inner error object in its error member.

Service request fields cannot be null. SDK encoders send empty objects and arrays for required collections, including child args: {} and cleanup argv: []. Optional fields may be omitted. The host rejects null fields.

Secret responses require both a string value and a boolean masking classification, including empty values and false. Output responses require an output member, which can be JSON null. OIDC tokens and cleanup IDs must be nonempty. A successful child result requires output. Missing result fields fail the call instead of becoming an absent optional reference.

Mask registration waits for acknowledgement before returning. It also updates the matcher used to validate later generated identifiers. Annotations and summaries wait for acknowledgement, and write failures fail the operation. Annotation levels are debug, info, warn and error. Positions are one-based when supplied. The host masks text before applying record bounds.

Limits

internal/limits owns the limits used by both SDKs and the host.

SDK cancellation and registered cleanup commands share the time remaining under the funded lease and drain deadline. At that bound the host kills the process tree, including cleanup descendants. Cloud Jobs retain the same bound in their Kubernetes deadline; a controller outage cannot extend it.

Operator ceilings and inherited deadlines also apply. Invalid declarations fail when accepted. Dependency-cache failures allow cold execution. Artifact staging failures and missing required output reads fail the job.

Version selection

The pinned SDK version selects a matching engine. Each engine negotiates its node protocol once through hello. Optional wire fields, routes, error codes and log events are additive. Optional operations are advertised in features. Breaking wire changes require a new major. Strict plan fields must be understood by the accepting engine.