Runlocal

How Runlocal works

Runlocal turns your hardware and a target model into a command you can run. Pick your GPU (or type in custom VRAM and memory bandwidth), set your system RAM, choose a model and a quantization level, and it computes how much memory the weights and KV cache need, how many transformer layers fit in VRAM, and how many spill to the CPU. The result is a copy-paste llama.cpp or Ollama command with the right --n-gpu-layers and context size, plus a tokens/sec estimate derived from memory bandwidth.

Reading the offload split

Partial GPU offload is normal, not a failure. When a model does not fit entirely in VRAM, Runlocal fills the GPU with as many layers as the KV cache and runtime overhead allow, then runs the rest on the CPU. The offload bar shows that GPU/CPU split at a glance, and the tokens/sec figure already accounts for the slower CPU path. If a model will not fit even with system RAM added, Runlocal shows a warning instead of a command that would only run out of memory.

Frequently asked questions

How accurate is the tokens/sec estimate?

It is a bandwidth-bound estimate, not a benchmark. Local decoding is dominated by memory bandwidth, so Runlocal divides the bytes moved per token by your GPU and CPU bandwidth, scaled by a real-world efficiency factor. Open the "why this estimate" panel to see the exact formula and numbers. Real speed varies with your driver, batch size, and CPU.

Why does the model size differ from the file size on Hugging Face?

Runlocal estimates weight size as parameters times bytes-per-parameter for the chosen quant. GGUF files vary slightly because some tensors are stored at higher precision. The estimate is close enough to decide fit and offload, not to predict the byte-exact download.

What does --n-gpu-layers do?

It tells llama.cpp how many of the model's transformer layers to load onto the GPU; the rest run on the CPU. Runlocal solves for the largest number that fits your VRAM after reserving space for the KV cache and runtime overhead.

Does it work for Apple Silicon?

Yes. M1 Pro, M2 Max, and M4 Max are included with their unified memory and bandwidth. On Apple Silicon the "VRAM" is shared system memory, so the fit math uses the unified memory figure.

Do you store my hardware or send it anywhere?

No. Runlocal is a static page with no backend. Every calculation runs in your browser and nothing is sent to a server.