What GPU do I need to run a 70B LLM locally?

At Q4_K_M, a 70B model needs ~40GB VRAM — single-card on a Mac M4 Max 64GB unified memory, or two RTX 4090s with tensor parallelism. An RTX 5090 32GB still falls short by ~8GB and forces partial CPU offload, which is 10-50× slower.

Can an RTX 4090 run Llama 4 Maverick?

No — Llama 4 Maverick has 400B total parameters loaded into memory at inference (only 17B are active per token, but all weights occupy VRAM). At Q4 you need ~230GB VRAM, which requires multi-GPU (4× A100 80GB) or a Mac Ultra 512GB. RTX 4090's 24GB can run Llama 4 Scout (109B) only with partial CPU offload.

Is a Mac M4 Max better than an RTX 5090 for local LLMs?

For memory-bound workloads (large models, long context), the M4 Max 64GB or 128GB wins because unified memory beats 32GB of RTX 5090 VRAM. For compute-bound workloads (short context, batched inference), the RTX 5090's CUDA throughput is 2-3× faster. Use the Apple Silicon toggle on this estimator to compare your specific model.

How much VRAM does Kimi K2.7-Code need?

Kimi K2.7-Code is a 1T-parameter MoE model (32B active per token), and every expert must be loaded. At Q4_K_M with 32K context it needs about 572 GB (Q3_K_M: about 463 GB) — multi-GPU server territory (e.g. 8× H100 80GB). Use the model selector to see the estimate for your settings.

Do MoE models need more VRAM than dense models of the same active size?

Yes — for inference VRAM, ALL expert weights are loaded into memory, not just the active ones. Llama 4 Maverick has 17B active per token but 400B total in VRAM. Gemma 4 26B MoE has ~4B active but all 26B loaded. The per-token compute is cheaper, but the memory footprint matches the total parameter count.

Does context length affect VRAM?

Yes — KV cache memory scales linearly with context. A 70B model at 4K context needs ~1GB extra for KV cache; at 128K context that grows to ~32GB. Models with grouped-query attention (GQA, kv_head_fraction < 1) use proportionally less. The estimator's context slider shows live VRAM impact.

What GPU for Local LLMs? RTX 5090/4090 Estimator

Configuration

Model

Quantization

Use Case

Context Window

4K tokens

1K 128K

Batch Size

1 32

Total VRAM Required

-- GB

Weights: 0.0 GB

KV Cache: 0.0 GB

Overhead: 0.0 GB

Recommended Hardware

Consumer

Apple Silicon

Server / Datacenter

System RAM Needed

0 GB

What GPU Do I Need for Local LLMs? A Hardware Planning Guide

Understanding LLM Hardware Requirements

When planning to run local Large Language Models (LLMs), the primary bottleneck is almost always memory (VRAM), not purely compute speed. If a model fits entirely within your GPU's VRAM, it will run incredibly fast. If it spills over into system RAM (often called "offloading" using tools like Llama.cpp), inference slows down drastically. This tool calculates the necessary VRAM layout to help you choose the right graphics card for your specific AI workload, preventing costly hardware mistakes.

The total formula for LLM VRAM consumption involves three components: the base model weights (scaled by quantization), the context window memory (KV Cache), and baseline operational overhead. By tweaking these fields in the estimator above, you can accurately map your requirements to consumer GPUs (like the RTX 4090 or RX 7900 XTX) or workstation silicon (like Apple's M-series Unified Memory).

Does Quantization Matter for GPU Choices?

Yes, immensely. Running an uncompressed 8-bit or 16-bit model requires massive datacenter infrastructure. Consumer setups thrive on quantization—techniques like GGUF (Q4_K_M) or AWQ/ExLlama formats compress models down to ~0.5 to 0.7 GB per billion parameters. This means a 7B to 9B model can comfortably fit on an inexpensive 8GB GPU. When sizing your graphics card, always estimate based on the quantization level you actually intend to run.

Apple Silicon vs Nvidia GeForce

Traditional discrete GPUs (Nvidia, AMD) have a hard wall: they top out at 24GB on consumer tiers (like the RTX 3090/4090). To run extremely massive models like Mixtral 8x22B or Llama 3 70B, you either need multiple graphics cards or workstation-class hardware. Apple Silicon (M1/M2/M3/M4 Max and Ultra) changes this dynamic by sharing memory between the CPU and GPU. A Mac Studio with 128GB of Unified Memory can run massive parameter sizes fluidly, effectively functioning as a high-VRAM GPU powerhouse without the associated PCI-e complexities.

Training vs. Inference on your GPU

Inference (generation) is highly predictable in memory usage. But if you plan to fine-tune an LLM using LoRA or QLoRA, memory depends as much on sequence length as on model size: every token of context needs activation memory (and the full-vocabulary logits), on top of the weights and the adapters' optimizer states. The training overlay uses the same estimate as the Can I Fine-Tune LLM calculator (Unsloth, gradient checkpointing). If you toggle the Training overlay in this estimator, you will immediately see why tuning typically pushes local developers out of the 8GB tier and into 16GB, 24GB, or multi-card setups.

Sources & Methodology

GPU recommendations and VRAM estimates are based on empirical testing, manufacturer specifications, and established performance tiers. Key references:

llama.cpp GitHub — GPU backend compatibility and VRAM allocation
Tim Dettmers: Which GPU for Deep Learning — consumer GPU performance tier analysis
Ollama Hardware Requirements — recommended hardware per model
Apple M-series Unified Memory performance — Metal GPU compute specs

Recommendations represent general guidance. Actual performance depends on driver updates, cooling, thermal throttling, and workload specifics. Always benchmark your exact model + hardware combo before committing to a purchase.

What GPU Do I Need for Local LLMs? RTX 5090 / RTX 4090 / M4 Max Estimator