Fast builds: best practices

A living checklist of things that make sparkwing pipelines iterate fast. These emerged from measuring real iterations (touch a file, rebuild, redeploy, observe running) and watching where the time went.

Target for a single-app Go service, laptop -> cluster via remote trigger: < 15 seconds from edit to running and healthy.

1. Mount Go build + module cache in your DockerfileSection anchor link

Impact: 13s -> 0.7s on the Go compile step.

Without cache mounts, every docker build starts from scratch when the COPY . . layer invalidates - which happens on any source change. With them, Go's incremental compiler only rebuilds what actually changed.

# syntax=docker/dockerfile:1
FROM golang:1.22-alpine AS build
WORKDIR /app
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod \
    go mod download
COPY . .
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 go build -o /out/server ./cmd/server

The cache mounts persist across builds on the same BuildKit builder. On Cloud, each job pod gets a separate Docker-in-Docker (DinD) daemon and fresh storage. Builds within that job can reuse its cache. A later job starts with an empty cache, even on the same node or in the same run. The timings above require a warm builder.

Cache mounts survive image-layer invalidation, so a changed source file can still reuse Go's compiler cache within the same job.

Same pattern works for Rust (/usr/local/cargo/registry, /app/target), Node (/app/node_modules), Maven (~/.m2), etc.

Reuse build layers across Cloud jobsSection anchor link

Export build layers to a registry and import them in the next job. The runner image includes buildx. After authenticating to your registry with docker login, create a builder with the docker-container driver:

docker buildx create --name cloud-build --driver docker-container --use
docker buildx build --builder cloud-build \
  --cache-from type=registry,ref=registry.example/team/app:buildcache \
  --cache-to type=registry,ref=registry.example/team/app:buildcache,mode=max \
  --tag registry.example/team/app:latest --push .

Use a cache reference separate from the output image tag. Each job must import and export it explicitly. This reuses build layers, while the Dockerfile's RUN --mount=type=cache directories remain local to the builder. See Docker's registry cache documentation for supported options and driver requirements.

2. Don't push to registries you don't needSection anchor link

Impact: ~5-7 seconds when iterating against a local cluster.

If your pipeline pushes to multiple registries every build, you are paying the round-trip cost on every iteration. Gate the registry list on the deploy target:

// Example: only push to a remote registry when running in production
var registries []string
if os.Getenv("KUBERNETES_SERVICE_HOST") != "" {
    registries = []string{prodRegistry}
} else {
    registries = []string{"localhost:30500"} // local registry
}

3. Use remote triggers for iterationSection anchor link

sparkwing pipeline trigger build-deploy --profile prod runs the pipeline in-cluster against your current commit and follows it to a terminal state, streaming the run's node logs through the controller. It sends the commit SHA to the controller and eagerly refreshes the cache so the runner sees your just-pushed commit without the cache having to fetch it mid-clone -- no waiting on a CI queue between edits.

A git-push-driven webhook is the audited production path, but for "change a log line and re-run" the direct trigger is the faster gear.

4. Register your repo with the cache on startupSection anchor link

If you delete the cache PVC (or stand up a new cluster), repos have to be re-registered before the runner can clone them. The failure mode is cryptic:

fatal: repository '.../<repo>/' not found

Register up-front - script it as part of your cluster bootstrap:

POST /git/register?name=<repo>&repo=<ssh-url>

5. Let sparkwing resolve spark libraries automaticallySection anchor link

Spark libraries are resolved at build time from the sparks: block in .sparkwing/sparkwing.yaml - you do not need to bump go.mod across repos when a new version ships. See sparks.md for details.

# .sparkwing/sparkwing.yaml
sparks:
  - name: my-spark-lib
    source: github.com/example/my-spark-lib
    version: latest

When a new version is tagged, every pipeline picks it up on its next build automatically. The binary cache key includes the resolved version, so a new release triggers a recompile only once - then it is cached.

To force an immediate update: sparkwing pipeline sparks update.

6. Keep your Dockerfile's external dependencies cachedSection anchor link

Per-build apk add, curl kubectl, npm install without cache mounts blow a few seconds every build. Either:

  • move them into an earlier, rarely-changing layer (before COPY . .)
  • mount the tool cache (/var/cache/apk, ~/.npm, ~/.m2)
  • bake them into the base image

7. Consolidate redundant image tagsSection anchor link

Every extra docker push is a separate manifest API call (~200ms apiece on a warm remote registry, ~1-2s apiece on cold local registries). If you are pushing :latest, :commit-xxx, :files-yyy, and a long deploy tag, drop the ones nothing consumes. The ones that matter:

  • :commit-xxx-files-yyy-prod - what gitops pins in kustomization
  • :latest - nice for humans doing kubectl set image manually

Anything else is likely debugging residue.

8. Match your local cluster to productionSection anchor link

When your local and prod clusters have different deployment layouts (one has the app, the other does not), iteration in local mode fails on deploy - the image pushes fine but kubectl rollout restart hits deployment not found. Apply the same gitops layout locally that ArgoCD applies to prod.