The Images service, built in Rust on Workers, runs on every machine in Cloudflare’s edge network. To handle client connections, Cloudflare uses hyper, an open-source HTTP library for Rust [1]. Recently, the Images team discovered a race condition bug in the hyper library that impacted how the Images binding returned processed image data back to the client [2]. The bug caused truncated large image responses over sockets, but only intermittently and only for larger images.
The team spent six weeks chasing the nearly invisible bug, which was finally fixed with a simple flush before shutdown [3]. The bug was not introduced by the December rearchitecture, but the new intermediary changed who was reading on the response side of the socket, occasionally filling the buffer and triggering the bug. The fix was merged into hyperium/hyper via PR #4018 and will be available in a future hyper release [4].
