The 8GB VRAM Sweet Spot: How We Benchmarked 19 Local LLMs to Build the Ultimate Local AI Workflow

Running a good and usable coding model on 8GB VRAM was practically impossible just a few months ago, but now with MoE (Mixture of Experts) models, it is very possible.

To understand why this is such a game-changer, we have to look at the difference between traditional dense models and modern MoEs:

• Dense Models: In a standard dense model, every single parameter is active for every token generated. If you run an 8B or 14B dense model, all 8 or 14 billion parameters must be computed and streamed from memory on every single token. On an 8GB VRAM budget, dense models hit a wall quickly. Anything large enough to write complex code reliably was simply too big to fit.
• MoE Models (Mixture of Experts): MoEs split the model into multiple specialized sub-networks called 'experts.' While the total model footprint might be 35 billion parameters on disk, an internal router only activates a small slice (for example, 3 billion active parameters, often labeled as 35B-A3B) for any given token. You get the reasoning depth and knowledge breadth of a massive 35B model, but only compute a fraction of it per token.

How a 20GB Model Loads onto an 8GB GPU

So how does a 20GB quantized model actually run on a machine with only 8GB of VRAM?

The answer is layer offloading (supported by runtimes like llama.cpp). The runtime fills your 8GB of GPU VRAM with as many model layers as it can hold, and pushes the remaining layers into your system RAM (we have 32GB DDR4). When you prompt the model, the computation flows seamlessly between your GPU and CPU.

The Trade-off in Speed

There is, of course, a real catch: memory bandwidth physics.

Your GPU's dedicated GDDR6 VRAM moves data at blazing speeds (roughly 380 to 440 GB/s). Your system RAM, traveling across the motherboard's PCIe bus, only moves at around 30 to 50 GB/s. Because the model has to pass through both, the entire generation speed is handcuffed by the slower system RAM.

On our RTX 2080 Max-Q, a 20GB offloaded model runs at around 18.5 tokens per second. That is totally acceptable when you need deep architectural refactoring or tough math, but for quick back-and-forth chat or snappy multi-turn agent loops, 18 tok/s starts to feel like watching paint dry.

Overcoming the Speed Barrier with Small Dense Models

This is where modern small dense models (between 1.5B and 9B parameters) come to the rescue.

Because these compact dense models weigh only 1 to 5 GB, they fit 100% inside your 8GB VRAM. Without the PCIe system RAM bottleneck, they run at full GPU memory bandwidth, pumping out 38 to 75+ tokens per second with near-instant responsiveness.

The breakthrough in our workflow was realizing we don't have to choose one over the other. By teaming up small, blazing-fast dense models for everyday triage, rapid tool calling, and vision, and reserving the heavy 20GB model for tough reasoning passes, we get an uncompromising local setup that feels as snappy as a cloud API.

To find the exact dream team for this setup, we put 19 local models through their paces using BenchLoop. Here is the raw data, the benchmark receipts, and how we tied them together.

Step 2: The Benchmark Odyssey (31 Runs Across 19 Models)

We didn't just eyeball chat responses; we used BenchLoop to generate receipts.

BenchLoop scores models across deterministic task suites:

• Speed: Latency, TTFT, and generation throughput (tok/s).
• Coding: Executable Python tasks evaluated in a sandboxed interpreter.
• Tool Calling: Strict JSON / function-call schema compliance.
• Agent: Multi-turn agentic loops (where the model emits a tool call, BenchLoop executes it, feeds results back, and checks for hallucination).
• Data Extraction & Instruct Following: Constraint adherence and structured output.

Across several weeks, we tested 19 unique builds, including members of the Qwen 3.5/3.6 family, Tiel-Coder-35B, Ornith-1.0 and 1.5, Nex-N2.5, MiniCPM5, Ling-3.0, and LFM2.5.

Here is how the top contenders stacked up in our local leaderboard:

Full Local Benchmark Results (RTX 2080 Max-Q)

Step 3: The Secret Sauce - Why the Pi Harness Changes Everything

Notice how raw coding numbers for smaller models like MiniCPM5-2B were sitting at ~43%?

Small models usually stumble when asked to output strict JSON tool calls or code directly from the cold start. They lack the parameter capacity to "think" and "format" at the very same instant.

BenchLoop features an evaluation mode called the pi harness. Instead of forcing raw tool calls, it instructs the model to:

1. First reason silently inside <think>...</think> tags.
2. Emit structured actions using Hermes tags: <tool_call>{"name": "...", "arguments": {...}}</tool_call>.
3. The parser strips the thought trace so callers only receive clean execution.

The Impact on Scores

[MiniCPM5-2B Coding Score]
Raw Harness: ████░░░░░░ 43.8%
Pi Harness:  ██████░░░░ 60.4% (+16.6% boost from scratchpad reasoning!)

[Ornith-1.5-9B Coding Score]
Raw Harness: ████████░░ 85.4%
Pi Harness:  █████████░ 87.5% (with 96.9% agent pass rate)

[Nex-N2.5-mini Coding Score]
Raw Harness: █████████░ 93.8%
Pi Harness:  █████████░ 91.7% (with 91.7% reliability)

The <think> buffer acts like an inline scratchpad. For local setups, this means smaller, faster models suddenly become viable for tool-assisted workflows without breaking downstream parsers.

Step 4: The Shortlisted Lineup

Out of the 19 candidates, we pruned down our shortlist to 5 essential models that cover every operational base:

Active Model Roster:
├── Nex-N2.5-mini-Q4_K_M.gguf       [19.71 GB] -> The Heavy Brain
├── Ornith-1.5-9B-Q4_K_M_2.gguf     [ 5.38 GB] -> The Daily Driver
├── MiniCPM5-2B-Q8_0.gguf           [ 2.50 GB] -> The Fast Router
├── LFM2.5-VL-3B-Q4_K_M.gguf        [ 1.56 GB] -> The Multimodal Eye
└── LFM2.5-VL-1.6B-UD-Q6_K_XL.gguf  [ 0.93 GB] -> The Rapid Assistant

Visual Benchmark Graphs

Step 5: The Perfect Role for Each Model

Here is how each piece fits into our daily ensemble:

1. MiniCPM5-2B-Q8_0 - The Fast Traffic Cop

• Generation Speed: 74.5 tok/s
• VRAM: 2.5 GB
• Perfect Role: Triage & Router. It inspects incoming prompts, answers quick lookups, extracts basic parameters, and routes heavy programming problems or visual tasks to the right specialist without making you wait.

2. Ornith-1.5-9B-Q4_K_M_2 - The Daily Workhorse

• Generation Speed: 37.8 tok/s
• VRAM: 5.38 GB
• Perfect Role: Terminal agent, git assistant, and primary coding partner. With a 96.9% agent pass rate and 90% tool-call score, it stays loaded in your 8GB VRAM all day long without exhausting memory.

3. Nex-N2.5-mini-Q4_K_M - The Deep Thinker

• Generation Speed: 18.5 tok/s
• Memory: 19.7 GB (Offloaded across VRAM & 32GB RAM)
• Perfect Role: Heavy architectural design, intricate debugging, and complex math. You don't want to chat casually at 18 tok/s, but when you need a 93.8% accurate solution to a tricky refactor, you summon Nex.

4. LFM2.5-VL-3B-Q4_K_M (+ mmproj) - The Vision Inspector

• Generation Speed: 77.8 tok/s
• VRAM: 1.56 GB
• Perfect Role: UI screenshot review, architecture diagram parsing, and image extraction. Small enough to load in seconds, fast enough to keep your loop fluid.

How We Budget 8GB VRAM for Zero Friction

If you don't want to wait 10 seconds for model swapping every time you click a button, here is the golden pairing:

┌──────────────────────────────────────────────────────────────┐
│                  8 GB VRAM Allocation Budget                 │
├───────────────────────────────┬──────────────────────────────┤
│ Ornith-1.5-9B (Workhorse)     │ 5.38 GB                      │
│ LFM2.5-VL-1.6B (Vision/Router)│ 0.93 GB                      │
│ 8k Context Buffer + Overhead  │ ~1.40 GB                     │
├───────────────────────────────┴──────────────────────────────┤
│ Total: ~7.71 GB / 8.0 GB (Zero swapping, instant response!)   │
└──────────────────────────────────────────────────────────────┘

You get a full-time multimodal assistant and an elite agent running simultaneously inside your GPU memory.

Try Benchmarking Your Own Stack

If you want to run these exact tests on your own machine, grab BenchLoop via pip:

pip install benchloop-cli

# Run a quick smoke test on your running endpoint
benchloop run --model "qwen3:8b" --harness pi --provider openai_compat --endpoint http://localhost:11434

# Check the local dashboard
benchloop dashboard

Stop guessing which quantized weights are best from Discord hype. Test them, check your hardware receipts, and build an ensemble that actually flies.