Jamesob’s public repo details a pragmatic, cost‑focused path to run SOTA large language models (LLMs) on‑premises without relying on cloud APIs [1]. The design targets two budget tiers: a $2K entry point using 2× RTX 3090 for 48 GB VRAM, and a high‑end $40K‑$46K configuration that rivals Claude Opus‑level performance. This article concentrates on the latter, unpacking the hardware decisions, software stack, and the business implications for enterprises that need data‑private inference.

Hardware Choices and Cost Breakdown

The core of the system is a last‑gen AMD EPYC Milan 7313P (16‑core, $504) paired with an ASRock Rack ROMED8‑2T motherboard that offers seven PCIe 4.0 x16 slots ($715). Four NVIDIA RTX Pro 6000 Blackwell GPUs—each providing 96 GB of HBM2 VRAM—are the primary compute units, costing roughly $46,000 total. By sourcing DDR4 ECC RAM (128 GB) and SSD storage from eBay, Jamesob keeps the base platform under $5.6 K while allocating the bulk of the budget to VRAM, the true bottleneck for LLM inference.

PCIe Gen4 Switching for Low‑Latency P2P

A standout feature is the use of a c‑payne Microchip Switchtec PM40100 PCIe Gen4 switch. This fabric‑level interconnect allows the GPUs to exchange tensor‑parallel data at wire speed (27.5 GB/s uni‑directional, 50.4 GB/s bi‑directional, ~0.4 µs latency) instead of routing through the motherboard’s CPU root complex, which would add tens of microseconds of overhead [1]. The switch requires custom BIOS tweaks—forcing the slot to x16, disabling ASPM, and enabling Re‑Size BAR—to ensure stable Gen4 training.

Software Stack and Model Serving

Each model lives on a replicated ZFS pool and is accessed via Docker‑compose runners. The repo provides ready‑to‑deploy configurations for the 594 B parameter GLM‑5.2‑Int8Mix model, exposing ~80 token/s at 240k context length through vLLM. Speech‑to‑text workloads are also supported via Whisper‑large‑v3, demonstrating the rig’s versatility across generative and perceptual AI tasks. Docker isolation simplifies multi‑tenant use cases, a key concern for C‑suite governance.

Power, Cooling, and Operational Risks

Running four 96 GB cards on a single 110 V circuit forces a power‑capping strategy: each GPU is limited to 350 W, keeping peak draw under 1.5 kW and fitting within two 1.7 kW PSUs. However, this compromises peak throughput and may require a future 220 V upgrade for sustained workloads. The custom wood enclosure mitigates acoustics but lacks redundant cooling—enterprises should budget for liquid‑cool loops or rack‑mount solutions to meet SLA requirements.

Business Impact

  • Data Privacy – Local inference eliminates outbound API calls, satisfying regulations such as HIPAA or GDPR, a point echoed in broader LLM‑localisation guides [2][3].
  • CapEx vs. OpEx – A one‑time $46 K investment replaces recurring cloud inference spend, which can exceed $10 K/month for comparable throughput.
  • Scalability – The PCIe switch architecture scales to additional GPUs without re‑architecting the motherboard, future‑proofing the platform for 2027 model releases.

In summary, Jamesob’s blueprint shows that with disciplined component sourcing and careful firmware tuning, an enterprise can field a private, high‑performance LLM server that rivals commercial SaaS offerings while retaining full control over data and costs.

Sources

  1. jamesob/local-llm – GitHub repository – https://github.com/jamesob/local-llm
  2. Run LLMs Locally: Hardware Tiers, Tools Compared & Setup Guide – Big Data Boutique – https://bigdataboutique.com/blog/how-to-run-llms-locally-a-practical-guide-for-developers
  3. Guide to Local LLMs in 2026: Privacy, Tools & Hardware – SitePoint – https://www.sitepoint.com/definitive-guide-local-llms-2026-privacy-tools-hardware