Building a Dual V100 AI Workstation for Local LLMs

In today's software development landscape, local Large Language Models (LLMs) offer unprecedented privacy, zero latency, and freedom from subscription fees. While I generally avoid purely "vibe coding" for my projects, having robust inline code completion and chat directly integrated into IDEs speeds up development significantly. However, consumer GPUs with high VRAM often cost enough that paying for cloud inference ends up being the more economical option for most developers.

To overcome this, I recently completed constructing a custom, dual-rack compute node specifically designed to host LLMs and compile complex codebases without breaking the bank. Here is the full engineering breakdown of how I designed, assembled, and configured this dual Tesla V100 machine from scratch.

Fully assembled rig.

The core philosophy behind this build was maximizing compute performance and Tensor VRAM density while strictly controlling costs. Consumer-grade GPUs with large VRAM, such as the RTX 3090 or RTX 4090, carry massive market premiums. On the secondary enterprise market, Nvidia Tesla V100 32GB GPUs offer excellent compute capabilities and high-bandwidth memory at a fraction of the cost.

NVIDIA GV100 Chip. 

Since enterprise components demand specific architecture that consumer motherboards rarely accommodate, I carefully selected components based on cost, local market availability, and technical compatibility.

• CPUs - 2 × Intel Xeon E5-2695 V3: Offering a total of 28 cores and 56 threads, these processors provide substantial multi-threaded horsepower for heavy compilation tasks at nominal cost.
• Motherboard - Machinist D8 MAX X99 Gaming: A dual-socket LGA2011-3 board that delivers sufficient PCIe lane bifurcation and multi-CPU memory access required for heavy server workloads.
• Memory - 128GB Kingston 3200MHz DDR4 RAM: Provides ample headroom for context caching, system-level operations.
• GPUs - 2 × Nvidia Tesla V100 32GB: Delivering 64GB of combined VRAM, this configuration easily hosts 14B to 32B parameter models fully offloaded in memory.
• PCIe Switching and Interconnect - PLX8749 Expansion Card and Dual GPU NVLink Board: The PLX switch manages dedicated PCIe lane allocation, while the NVLink board provides high-speed inter-GPU communication.
• High-Speed Cabling - 4 × SFF-8654 Cables: Links the PLX8749 expansion adapter directly to the GPU NVLink carrier board. Standard riser cables cannot handle high-frequency PCIe signal integrity across physical distances; SFF-8654 (SlimSAS) high-density shielded cables are mandatory to prevent packet loss, crosstalk, and signal degradation across the 32 high-speed PCIe differential lanes. 
• Display Output - Nvidia GT210 1GB: Since Tesla V100 cards lack display outputs, this low-power passive card handles basic host display output.
• Networking - Intel i210 SFP PCIe Card: Offers fiber connectivity to my existing network.
• Storage Distribution - 256GB KingSpec NVMe SSD and 3TB Western Digital HDD: The NVMe drive hosts the core OS for rapid boot times, while the high-capacity HDD stores massive LLM files.
• Power Supply - 2 × 750W ASUS PSUs with 230V Y-Splitter: Enterprise GPUs impose sharp transient power spikes. Isolating the system components on one 750W unit and the GPU acceleration board on the second 750W unit ensures total system stability under heavy loads.

To keep the budget tight, key enterprise compute hardware -the CPUs, GPUs, mainboard, PLX8749 card, and specialty interface cables - were imported directly. Everything else, including memory, PSUs, coolers, aluminum extrusions, fasteners, and storage drives, was sourced directly from the local computer and electronics hardware market.

Standard ATX cases cannot accommodate passive enterprise GPUs alongside liquid coolers and dual power supplies. To solve this, I designed and built a open-frame enclosure using 440mm × 440mm × 220 mm dimensions made from V-Slot 2020 Aluminum Extrusion bars.

3D rendered view of the mounting structure.

 

Frame assembly in progress.

The structure uses 4 V-Slot 2020 aluminum extrusions linked using 12 standard 90° corner brackets alongside 12 hidden 90° corner brackets to ensure maximum rigidity. The structure features a two-tier layout:

1. Bottom Rack: Houses the X99 motherboard, primary 750W system PSU, NVMe SSD, and 3TB mechanical hard drive mounted via custom brackets.
2. Top Rack: Secures the dual Tesla V100 GPUs, the secondary 750W GPU PSU, and the liquid cooling arrays.

SXM2 socket in V100 16GB version.

Enterprise Tesla cards are passively cooled and rely on server chassis fans running at extreme static pressures. Since server fans create unbearable noise in a personal working environment, I engineered a modified liquid cooling setup:

• I repurposed two Gamdias Aura GL360 360mm Liquid Coolers to draw heat away from the V100 dies.
• The stock CPU cooler fans were replaced with high static pressure Foxconn PVA080G12Q 80x80mm 4-pin industrial fans mounted via custom fabricated mounts.
• Additional 80x80mm 3-pin cooling fans provide direct active airflow across the V100 voltage regulator modules.
• All cooler fans and liquid cooling pumps receive power directly through the Dual GPU NVLink interface board.

GPU Cooler setup.

When mounting liquid cooling blocks onto bare-die enterprise GPUs like the V100, uneven pressure can crack the silicon or cause improper thermal contact with the HBM2 stacks. Using a torque screwdriver, I precisely torqued all GPU retention screws to 0.25 N·m in a cross-pattern sequence, ensuring perfect thermal paste distribution without stressing the package.

Tesla V100 GPUs mounted on the NVLink carrier board.

To force the PLX8749 expansion card to split available PCIe bandwidth cleanly into two x16 channels over the four SFF-8654 links, set the onboard jumper switches as follows:

• SW1-1: MID
• SW1-2: DOWN
• SW2-1: MID
• SW2-2: DOWN

Connection diagram from PLX8749 to NVLink carrier board.

During the initial system startup, the motherboard repeatedly hung and reset immediately after displaying BIOS POST code 97. This occurred because the host motherboard could not address the large memory-mapped I/O spaces requested by dual 32GB GPUs during initial boot.

To fix this hardware initialization issue, follow this mandatory initialization sequence:

1. On the NVLink GPU carrier board, temporarily set the power toggle switch to OFF and set the operational mode switch to MT (Manual Mode).
2. Power on the system, enter the BIOS setup menu, and navigate to PCI Subsystem Settings.
3. Set Above 4G Decoding to Enabled. Save settings and shut down the machine.
4. Flip the GPU carrier board switches to ON and AT (Automatic Mode) positions.
5. Power on the machine again. The system will now correctly pass POST code 97 and enumerate both GPUs smoothly.

With the physical build complete and stable at POST, I installed Ubuntu Server 26.04 LTS for a lean system footprint.

During OS setup, the drive paths were mapped strictly to segregate OS operations from large model storage:

• / (Root directory) : Installed on the NVMe SSD for fast system booting and package management.
• /home : Mounted directly to the 3TB HDD to store large model weights.

Once booted into Ubuntu Server, update the local repository index and install the Nvidia driver and utilities stack:

sudo apt update
sudo apt install -y nvidia-driver-535 nvidia-utils-535

After installation completes, reboot the machine:

sudo reboot

Upon rebooting, run the management utility to verify that both Tesla V100 cards and their bus interfaces are correctly recognized:

nvidia-smi

You should see both 32GB Tesla V100 graphics cards listed alongside their current thermal and power profiles.

Before settling on my final setup, I conducted extensive evaluation runs using vLLM and standalone llama.cpp binaries. While vLLM offers high throughput for concurrent API batching and llama.cpp provides excellent low-level parameter tuning, Ollama proved to be the clear winner for my specific development workflow. Considering feature completeness, model management flexibility, native service integration, and effortless multi-model context switching, Ollama offered the most seamless experience.

To set the Ollama execute the official automated installation script:

curl -fsSL https://ollama.com/install.sh | sh

By default, Ollama saves all downloaded model weights inside ~/.ollama/models, which sits on the root partition. Because our root partition is a small 256GB SSD, hosting multiple 14B+ models will quickly exhaust disk space. We must redirect Ollama's storage path to our 3TB HDD located under /home.

Create a dedicated directory on the mechanical drive and transfer ownership to the system user created by Ollama:

sudo mkdir -p /home/ollama_models
sudo chown -R ollama:ollama /home/ollama_models

Next, edit the ollama.service systemd unit file to inject custom environment variables:

sudo systemctl edit ollama.service

In the editor window that opens, append the following environment configuration under the [Service] block:

[Service]
Environment="OLLAMA_MODELS=/home/ollama_models"
Environment="OLLAMA_HOST=0.0.0.0:11434"

Save and close the file, then apply the changes and restart the systemd service:

sudo systemctl daemon-reload
sudo systemctl restart ollama

With storage successfully redirected to the mechanical drive, we can pull code completion and chat models. For my coding workflow, the Qwen 2.5 Coder series offers an exceptional balance between context comprehension and execution speed.

Testing Visual Studio Code with Continue plugin.

The final piece of the project is connecting our local AI backend to the development workspace. In VS Code I did linkage through "Continue" plugin:

1. Launch Visual Studio Code on development machine.
2. Navigate to the Extensions Marketplace, search for Continue, and click Install.
3. Open the config.json settings file inside Continue and configure it to point to new local compute server using the native Ollama provider:

name: Main Config
version: 1.0.0
schema: v1
models:
  - name: Qwen2.5 Coder 14B (Chat)
    provider: ollama
    model: qwen2.5-coder:14b
    apiBase: http://<IP-ADDRESS>:11434
    roles:
      - chat
      - edit
      - apply
    capabilities:
      - tool_use
  - name: Qwen2.5 Coder 1.5B (Autocomplete)
    provider: ollama
    model: qwen2.5-coder:1.5b
    apiBase: http://<IP-ADDRESS>:11434
    roles:
      - autocomplete
      - embed
  - name: Qwen2.5 Coder 32B (Chat)
    provider: ollama
    model: qwen2.5-coder:32b
    apiBase: http://<IP-ADDRESS>:11434
    roles:
      - chat
      - edit
      - apply
    capabilities:
      - tool_use
  - name: Qwen2.5 Coder 7B (Autocomplete)
    provider: ollama
    model: qwen2.5-coder:7b
    apiBase: http://<IP-ADDRESS>:11434
    roles:
      - autocomplete
      - embed

Operating in a room with an ambient temperature of 31°C, the custom liquid cooling array equipped with high-RPM Foxconn fans delivers remarkable thermal stability:

• Idle System State: Both Tesla V100 GPUs rest comfortably at 42°C.
• Maximum GPU Stress: Even under sustained maximum VRAM allocation and heavy model switching between deepseek-r1:32b, qwen2.5-coder:32b, and qwen2.5-coder:14b, GPU temperatures peak at just 68°C.
• CPU Thermal Load: During heavy multi-threaded CPU tasks, such as compiling llama.cpp across all 56 execution threads, the dual Xeon processors reach a maximum temperature of 69°C.

With 64GB of aggregate HBM2 running across a dedicated NVLink interconnect, this dual-V100 server handles heavy parameter models effortlessly. In daily development usage, the server runs deepseek-r1:32b, qwen2.5-coder:14b and qwen2.5-coder:32b without any noticeable latency. Switching between models during active workflows - such as coding in VS Code with qwen2.5-coder:32b and qwen2.5-coder:14b; while using deepseek-r1:32b in Open WebUI for high-level reasoning - is super fast, completely error-free, and handles VRAM reallocation flawlessly.