Context and constraints
The supplied brief describes a fleet of more than 200 edge nodes in parcel sorting centers. The architectural problem is maintaining deployment consistency without making local operations depend on a continuously available central network.
This dossier describes the proposed system and its operating assumptions. It does not claim that the rollout strategy or the performance outcomes have been validated against production telemetry.
Deployment decisions
Each site is an independent failure domain. A lightweight k3s cluster runs local services and keeps a durable event queue. A versioned desired-state repository provides an auditable release history. Container images are immutable and pinned by digest.
Rollouts begin with a canary site. Readiness checks gate traffic, while a disruption budget and sufficient spare capacity make rolling replacement possible. A failed readiness check stops the promotion instead of propagating a broken release to the fleet.
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
Zero downtime is a release objective, conditional on healthy spare capacity, correct readiness checks, and backward-compatible schema changes. It is not a guarantee for every failure mode.
Offline behavior
Events carry stable identifiers, sequence metadata, and a site identifier. The edge writes the event before acknowledging it. Synchronization retries with bounded exponential backoff; ingestion deduplicates by event ID. Delivery is at least once, with idempotent processing rather than an unqualified exactly-once promise.
Failure analysis and runbook
- Registry unavailable: keep the last known healthy digest and pause promotion.
- Cloud link lost: retain events locally and alert on disk capacity and queue age.
- Readiness failure: restore the previous desired state and investigate application health.
- Incompatible migration: use an expand-and-contract migration before removing old fields.
Validation plan
Measure per-site queue age, deployment convergence, error rate, and recovery time under injected disconnections. Before publishing a production post-mortem, attach the actual incident timeline, impact window, and evidence for each claimed improvement.