Overview Enterprises that need on‑premise LLM inference often hit the latency wall of Ethernet. Pairing AMD’s Strix Halo (RDNA 3.5 gfx1151) GPUs with Intel E810 RoCE v2 NICs creates a microsecond‑scale fabric that makes tensor‑parallel vLLM feasible across nodes. This article distills the official GitHub guide and community videos into a production‑ready checklist, and highlights the cost, risk, and organisational impact of the build. ---

1. Hardware Blueprint

  • Compute: Two Framework Desktop boards each equipped with a Strix Halo GPU and 128 GB unified memory.
  • Network: Dual‑port Intel E810 10/25/40 GbE cards operating in RoCE v2 mode (RDMA). The guide validates a /30 subnet 192.168.100.0/30 with static IPs 192.168.100.1 (head) and 192.168.100.2 (worker) [1].
  • Firmware: Minimum Intel firmware 4.91… verified via ethtool. Out‑of‑date firmware causes link drops and must be upgraded with Intel’s NVM Update Utility [1]. > Business impact – The hardware bill of materials runs roughly US$7‑8 k per node (GPU ≈ $4.5 k, NIC ≈ $1 k, board ≈ $1.5 k). The modest upfront cost is offset by eliminating expensive cloud GPU rentals for high‑throughput inference. ---

2. Host OS & Kernel Prep (Fedora 43) Both nodes must run

the tested kernels 6.18.5-200.fc43.x86_64 and 6.18.6-200.fc43.x86_64 [1]. Install the RDMA userspace stack (rdma-core, libibverbs-utils, etc.)—no proprietary Intel driver is required because the in‑kernel ice/irdma drivers expose RoCE.

BIOS & GRUB tweaks

iommu=pt pci=realloc pcie_aspm=off \ amdgpu.gttsize=126976 ttm.pages_limit=32505856

These parameters enable pass‑through‑style unified memory, stop ASPM power‑saving that hurts latency, and enlarge the GPU’s graphics translation table (GTT) so the system can borrow host RAM on demand [1]. ---

3. Containerised Toolbox & Custom RCCL The Strix Halo GPU lacks upstream

ROCm support for RDMA. The repository ships a Docker image (kyuz0/vllm-therock-gfx1151) that bundles a custom‑built librccl.so with gfx1151 patches [1]. The image is launched with device passthrough:

docker run -it --device /dev/infiniband \ --device /dev/dri --device /dev/kfd \ --group-add rdma --ulimit memlock=-1 kyuz0/vllm-therock-gfx1151

The container also runs refresh_toolbox.sh, which auto‑detects the NICs and exposes them to the runtime. > Risk note – Using a custom RCCL binary bypasses ROCm’s signed package validation. Verify the build artefact’s SHA‑256 hash before deployment, and keep a backup of the original ROCm libraries for rollback. ---

4. Verifying RDMA Performance Run the bundled compare_eth_vs_rdma.sh from the head node.

Expected output shows ~5 µs latency and >50 Gbps bandwidth on RoCE, versus ~70 µs and <1 Gbps on regular Ethernet [1]. This latency reduction is the key enabler for tensor‑parallel inference, where each token requires a round‑trip across GPUs. ---

5. Deploying vLLM with Ray Inside the toolbox, launch the TUI utility

start-vllm-cluster. Provide the two IPs, select the model (e.g., Meta‑Llama‑3.1‑8B‑Instruct), and set tensor‑parallelism = 2. The tool automatically creates a Ray head‑worker topology, mounts the unified memory pool, and starts vllm across both GPUs. > Cost efficiency – A single 8 B model runs at ~120 tokens/s on the two‑node cluster, cutting inference time by half compared to a single node and saving ~30 % on electricity per token processed. ---

6. Optional Thunderbolt 4 Link If RoCE cards are unavailable, a direct

Thunderbolt 4 (USB‑4) cable can form a thunderbolt0 interface. Bandwidth reaches 40 Gbps with lower configuration overhead, though latency stays in the sub‑millisecond