The symptom: a cheap node runs out of memory

In early June 2026 a single‑node DigitalOcean DOKS cluster running Kubernetes 1.36 began restarting pods despite normal kubectl top pods metrics. A quick htop showed the kubelet process growing from ~110 MiB to nearly 1 GiB within hours, a clear sign of a leak on a 2 GiB worker.

Heap profiling reveals a million contexts

Using Go’s built‑in pprof endpoint (/debug/pprof/heap) and go tool pprof -top, the author captured a heap snapshot. The top contributors by object count were context.(*cancelCtx).propagateCancel and context.withCancel, together accounting for ~45 % of allocated objects—almost a million context structs [1]. By memory size the same calls represented >60 % of the heap (≈115 MiB) [1]. Those numbers are absurd for a component whose typical workload is managing container lifecycles.

Root cause: a regression in startPodSync

The leak traced to a code change introduced on 2026‑02‑19. The original logic created a cancellable context only when none existed:

if status.ctx == nil || status.ctx.Err() == context.Canceled {
    status.ctx, status.cancelFn = context.WithCancel(context.Background())
}

The new version replaced it with a single line executed on every startPodSync iteration:

ctx, status.cancelFn = context.WithCancel(parentCtx)

When status.cancelFn already referenced a previous cancel function, the assignment overwrote it without invoking the old CancelFunc. Go’s context documentation states that failing to call the cancel function leaves the child context attached to its parent indefinitely, leaking memory until the parent itself is cancelled [2]. Because startPodSync runs for each pod on each reconciliation loop, the leak compounded quickly, eventually inflating kubelet’s resident memory to ~974 MiB.

Patch workflow and operational impact

The issue was filed as GH‑139823 on 2026‑06‑17 [3]. A minimal fix reverted the regression to the original guard‑check, avoiding the wholesale context replacement. Initial PR tests passed locally but broke Kubernetes CI’s end‑to‑end suite due to unrelated probe‑worker context handling. Maintainers therefore merged a simplified revert on 2026‑06‑25, with a back‑port to the release-1.36 branch merged on 2026‑07‑03 [3]. The patch will ship in the upcoming v1.36.3 patch release (target 2026‑07‑14).

Business implications

  • Cost avoidance – On a low‑memory node the leak raised kubelet’s RSS by ~864 MiB, forcing premature pod restarts and higher CPU cycles for the OOM killer. For larger clusters the cumulative effect could drive additional node provisioning, inflating cloud spend.
  • Risk reduction – Unchecked memory growth can trigger node‑level outages, breaking SLAs for latency‑sensitive workloads. The quick rollback restored node stability within minutes, illustrating the value of runtime profiling.
  • Organizational change – The incident highlighted the need for production‑grade observability on control‑plane components, not just application pods. Adding regular process_resident_memory_bytes alerts for kubelet is now a standard rule in many SRE playbooks.

Takeaways for architects

  1. Profile in production‑like environments – Small, cheap clusters surface resource bugs that remain hidden on beefy nodes.
  2. Never store cancel functions without guaranteeing call‑through – Go’s go vet has a lostcancel check, but storing the function in a struct can evade it, as noted on Hacker News [4].
  3. Fast upstream collaboration pays off – The Kubernetes community’s rapid triage and back‑port process turned a week‑long outage risk into a patch ready for the next release.