Migrating to v0.65.0

Before: historical release instructions

Historical release snapshot. For the current command surface, follow the latest migration and unreleased changes.

Metric sample kindsSection anchor link

Metric producers must identify interval readings and completed-command reports explicitly. MetricSample.OneShot() uses Kind; it no longer infers a command from positive CPU time.

Before:

sample := store.MetricSample{TS: at, CPUMillicores: 200}

After:

sample := store.MetricSample{Kind: store.MetricInterval, TS: at, CPUMillicores: 200}

Set store.MetricCommand for completed commands, even when CPUTime is zero. Interval readings must leave CPUTime zero. HTTP metric requests use "kind": "interval" or "kind": "command". Missing kinds and unavailable CPU or RSS readings remain unknown and do not qualify for learning. A later successful reading does not erase an earlier unavailable reading.

Stop producers and consumers before upgrading controllers, workers and local executors together. Older HTTP servers reject the new request field; older readers can ignore it and retain the previous incorrect inference. Rebuild pinned SDK consumers that produce metrics. A database requirement protects direct database access, not mixed-version HTTP readers.

The database migration labels historical positive-CPU-time points as command reports. Historical zero-CPU-time points remain unknown because their kind cannot be recovered. Learned profile format 11 discards incompatible learned estimates on access or update while preserving explicit pins and wait statistics. New observations rebuild estimates; leave resource pins in place until their replacement estimates have been verified for the workload.

Schema 51 marks pre-upgrade nodes with an unknown metric so replay cannot rebuild estimates from their old samples or exit totals. Numeric diagnostics remain; the earliest sample can lose its command classification. Nodes with an existing unknown sample are unchanged. New nodes created after migration remain eligible. The process-tree-accounting database requirement rejects older read-write store openers. Drain or stop runs and close all writers before upgrading.

Local process measurementsSection anchor link

Dedicated node processes sample their observed process tree on Linux and macOS. CPU rates use changes in live and reaped-child counters. RSS sums resident bytes; shared pages can be counted more than once, and sampling can miss short peaks. The first interval retains the initial RSS observation. Run totals approximate overlap using each node's maximum reading within a sampling bucket.

Embedded execution, observed process identity loss, unfinished nodes and multiple attempts within the source run do not qualify for learning. Identity loss includes ordinary child reaping when no identity-specific receipt is available. Unavailable readings remain unknown. Metric delivery errors fail the affected attempt; an automatic retry can still succeed.

Profile observationsSection anchor link

Profile observation callers must set CPUMeasured: true (HTTP "cpu_measured": true) only when CPU was measured. An omitted or false flag leaves existing profiles unchanged after input validation, including memory and duration values. Measured zero CPU remains valid.

Host and worker admission require that flag before using learned CPU or memory values, including demand floors and previous-version costs. Explicit pins retain precedence. Duration estimates displayed for existing profiles are separate from resource admission.

Controller run profilesSection anchor link

The controller records a run resource observation only when one uncached node contributes measurements. For multiple uncached nodes it retains valid node profiles but withholds the run observation, including its duration contribution. Independent worker samples do not establish a simultaneous run total. Explicit pins remain unchanged; node profiles still supply worker and Kubernetes costs.