Run AI at home

A model file is just its weights — billions of numbers — plus a little metadata. Running it locally means three steps. A runtime such as llama.cpp, Ollama, LM Studio, MLX or vLLM loads those numbers into memory, ideally your graphics card's memory (VRAM). Your prompt is turned into tokens and pushed through every layer of the network at once; this is prefill, and it's compute-heavy but parallel. Then the model writes its answer one token at a time; this is decode, and for each new token the hardware has to read essentially all the active weights from memory again.

Nothing leaves your computer: no API key, no per-token bill, no rate limits, and it works offline. The trade-offs are that the best open models are big, your hardware sets the ceiling on size and speed, and you're the one keeping it updated.

Three things share the memory: the weights, the KV cache and some working space.

Weights = parameters × bits per weight ÷ 8. A 32-billion-parameter model at 16-bit is 64 GB; at a 4-bit quantization (Q4_K_M, about 4.85 bits once its scale factors are counted) it's about 19 GB. That's why quantization is what makes local AI practical.

The KV cache stores, for every token in the context, the attention keys and values of every layer, so they don't have to be recomputed. Its size per token is 2 (keys and values) × layers × KV heads × head size × bytes per number. For a Llama-3-8B-style model that's 2 × 32 × 8 × 128 × 2 bytes = 128 KB per token, so a 32K-token context costs about 4 GB on top of the weights. Newer designs shrink this: grouped-query attention shares keys across heads, DeepSeek-style latent attention compresses them, and hybrid models (Qwen3.5/3.6, Gemma, Nemotron) use full attention on only a fraction of their layers. We read each model's real config, so the calculator uses its true figure.

Working space — activation buffers, the runtime and, on a desktop, whatever your screen and browser already use — is typically 0.5 to 1.5 GB.

memory ≈ params × bpw / 8                         # weights
       + 2 × layers × kv_heads × head_dim × 2B × ctx  # KV cache (16-bit)
       + ~0.8 GB                                     # buffersCopy

If it doesn't fit in VRAM, runtimes can offload: some layers stay on the GPU and the rest run from system RAM on the CPU. It works, but every token now waits on the slower memory. Mixture-of-experts models offload unusually well — keep the attention and shared layers on the GPU and push only the big, rarely-used expert weights to RAM (--n-cpu-moe in llama.cpp).

Generating one token means reading every active weight once. So the speed limit is simply memory bandwidth divided by the bytes of active weights (plus the part of the KV cache being read). An RTX 4090 moves about 1,000 GB/s; a 19 GB model can therefore manage at most about 50 tokens a second, and real runtimes reach roughly 60–75% of that limit.

tokens/s ≈ efficiency × bandwidth / (active_params × bpw / 8 + kv_read)
# RTX 4090, 32B dense @ Q4_K_M:  0.65 × 1008 / 19.4 ≈ 34 tok/s
# same model on dual-channel DDR5 (CPU): 0.6 × 80 / 19.4 ≈ 2.5 tok/s
# 30B-A3B MoE @ Q4_K_M on that CPU:     0.6 × 80 / 1.8 ≈ 26 tok/sCopy

That last line is why mixture-of-experts models changed local AI: a 30-billion-parameter MoE with 3 billion active parameters needs the memory of a 30B model but runs at the speed of a 3B one. It's also why Apple Silicon Macs, AMD's Ryzen AI Max and NVIDIA's DGX Spark are popular: their large, fast unified memory holds models no consumer graphics card can.

Prefill — reading your prompt — is different: it's compute-bound and processes thousands of tokens in parallel. GPUs do hundreds to thousands of tokens a second there; CPUs often only tens, which is why a CPU-only coding agent feels slow even when its typing speed looks fine. Tricks that help: flash attention (less memory traffic, needed for a quantized KV cache), prompt caching (llama.cpp and Ollama keep the KV cache of your last prompt), and speculative decoding (a tiny draft model or the model's own multi-token-prediction head guesses several tokens and the big model checks them in one pass).

GGUF is llama.cpp's single-file format, used by Ollama, LM Studio, Jan and most desktop apps. It runs on NVIDIA, AMD, Intel and Apple GPUs and on plain CPUs. Names describe the quantization: Q4_K_M is a 4-bit “k-quant”, medium mix; Q5_K_M and Q6_K are closer to the original; Q8_0 is nearly lossless; IQ types (“i-quants”) squeeze harder using an importance matrix measured on sample text; Unsloth's UD- “dynamic” files keep sensitive layers at higher precision. MXFP4 is the 4-bit float format some models (gpt-oss) are trained in natively.

MLX is Apple's framework for Apple Silicon; MLX builds are often 10–30% faster than GGUF on a Mac. Safetensors in BF16 or FP8 are the original weights, used by vLLM and SGLang on servers with big NVIDIA or AMD GPUs.

Rules of thumb: Q4_K_M is the sweet spot for most people; prefer a bigger model at Q4 over a smaller one at Q8; below Q3, quality falls off quickly — except for very large MoE models, which tolerate 2–3 bits surprisingly well. The calculator picks the highest quality that still runs smoothly on your machine.

Easiest: Ollama. Install it from ollama.com (Windows, macOS, Linux). It runs in the background and serves an API on port 11434. Any GGUF on Hugging Face can be pulled directly:

ollama run hf.co/unsloth/Qwen3.5-9B-GGUF:Q4_K_M
# more context, smaller KV cache (set before starting the server)
OLLAMA_CONTEXT_LENGTH=32768 OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serveCopy

Most control: llama.cpp. Download a release for your GPU from GitHub (CUDA for NVIDIA, Vulkan or ROCm/HIP for AMD and Intel, Metal is built in on Mac), or brew install llama.cpp / winget install llama.cpp. llama-server downloads the model, serves an OpenAI-compatible API on port 8080 and a chat page in your browser. It handles split files for very large models, which Ollama doesn't always.

llama-server -hf unsloth/Qwen3.5-35B-A3B-GGUF:Q4_K_M -c 32768 -ngl 99 -fa on --jinja
# too big for VRAM? keep the experts of the first N layers in system RAM
llama-server -hf unsloth/Qwen3.5-122B-A10B-GGUF:Q4_K_M -c 32768 -ngl 99 --n-cpu-moe 30 -fa on --jinjaCopy

Point and click: LM Studio. Install from lmstudio.ai, search a model in Discover, and it shows which quantizations fit your GPU. Turn on the local server (port 1234) to use it from other apps. On a Mac it offers both GGUF and MLX builds.

Apple Silicon: MLX. pip install mlx-lm, then mlx_lm.server --model mlx-community/… serves an OpenAI-compatible API. By default macOS lets the GPU use about two-thirds to three-quarters of unified memory; sudo sysctl iogpu.wired_limit_mb=… raises that until the next reboot.

Servers and multi-GPU: vLLM or SGLang on Linux with NVIDIA or AMD cards. They batch many requests at once and split a model across GPUs (--tensor-parallel-size 2), at the cost of needing the whole model in VRAM.

Drivers matter: on NVIDIA keep the driver current; on Windows consider setting CUDA – Sysmem Fallback Policy to “Prefer no sysmem fallback” in the NVIDIA Control Panel, so a model that doesn't fit fails loudly instead of silently crawling. On AMD, check your card first: ROCm (which Ollama's default AMD build and vLLM need) covers the RX 9000 and 7000 series and the RX 6800 and up; the RX 6600/6700 family works on Linux with HSA_OVERRIDE_GFX_VERSION=10.3.0; the RX 5000 series, Vega and the RX 400/500 cards have no ROCm at all and run only through Vulkan — LM Studio's Vulkan runtime, llama.cpp's Vulkan build, or Ollama's Vulkan backend — somewhat slower and with no vLLM. Radeons older than the HD 7000 series can't run models on the GPU.

All of these runtimes speak the OpenAI chat-completions API, so most coding tools can use them by changing the base URL: http://localhost:11434/v1 for Ollama, http://localhost:8080/v1 for llama-server, http://localhost:1234/v1 for LM Studio. The API key can be any text.

# Aider
aider --model openai/local --openai-api-base http://localhost:8080/v1 --openai-api-key x
# OpenCode (opencode.json)
{ "provider": { "local": { "npm": "@ai-sdk/openai-compatible",
  "options": { "baseURL": "http://localhost:11434/v1" },
  "models": { "qwen3.5:35b-a3b": {} } } } }Copy

For agents, pick a model with tool calling (filter “Coding agents” above), give it at least 32K of context — agent system prompts and file reads fill it fast — and start llama.cpp with --jinja so the model's own chat template formats tool calls correctly. Browse the agents themselves on our Tools page.

Memory is for the weights plus an 8K-token context and buffers. The last column shows how each runs on the computer set in the calculator. Labs with the most open models: Qwen (28), DeepSeek (13), Mistral (12), Z.ai (12), Meta (10), Moonshot AI (8).

Not sized yet (no public parameter count): Hy4, DeepSeek V4.1 Flash 1017.

Sizes and attention shapes from each model's Hugging Face repository; GGUF and MLX builds are the most widely used community conversions. Refreshed with the model catalog.