The demand for inference is skyrocketing, and the token craze is only getting crazier [1]. With frontier models being released almost every other week, the need for cheap inference is becoming increasingly important. Wafer AI has announced that they have served GLM-5.2 on AMD Instinct MI355X at 2626 tok/s/node aggregate and 213 tok/s single-stream, achieving over 2x lower cost than Blackwell [2]. This milestone was reached through optimizing speculative decoding on ROCm and tuning the MoE kernels.
The team at Wafer AI chose to quantize the base bf16 GLM-5.2 to MXFP4 with AMD Quark, which was found to be lossless compared to z-ai's official FP8 quantization [3]. They also selected sglang as the inference engine, which was able to take advantage of the quantization while remaining coherent. To improve throughput, the team enabled speculative decode on sglang, which required two fixes: modifying the MTP head to match the main decoder stack and adding a ROCm guard to the fused multi-step metadata kernel [4]. With these changes, the team was able to reach a single-stream decode number of 213 tok/s.
For aggregate throughput, the team optimized the MoE kernels, which allowed them to reach 2626 tok/s/node at 2.4 RPS [5]. This achievement demonstrates that AMD is becoming a viable option for inference, with performance per dollar ratios that are comparable to NVIDIA's Blackwell. As the demand for inference continues to grow, the importance of cheap and efficient solutions will only continue to increase.
