Best local LLM by memory: what fits in 16GB to 512GB

The strongest open model that fits at 16, 24, 32, 64, 128, 256 and 512GB of VRAM or unified memory, with real download sizes and sourced speeds.

· 12 min read

The best local LLM for your machine is decided by memory before anything else: the model's weights, plus room for its context, have to fit in your graphics card's memory (VRAM) or your Mac's unified memory. As of 10 November 2026, the strongest open models that fit are Gemma 4 at 16GB, Qwen3.8 27B at 24GB and 32GB, the same model at higher precision at 64GB, Qwen3.8 Flash Next at 128GB, and at 512GB either a mid-sized model at near-full precision or a larger one squeezed to 2 bits. Kimi K3.1 Pro and full-precision DeepSeek V4.1 Flash need more than one machine.

This guide goes size by size: 16, 24, 32, 64, 128, 256 and 512GB. Every size below is the actual download as listed on Hugging Face by the lab or by ggml-org, the llama.cpp project's own account, checked on 10 November 2026. Speeds come only from published, reproducible measurements, and where none exist for a model we say so rather than guess.

VRAM and unified memory are not the same budget

A graphics card's VRAM belongs to the GPU alone, so a 24GB card can give almost all of its 24GB to the model. On a Mac, the GPU shares unified memory with macOS and every app you have open, so a 24GB Mac has noticeably less than 24GB for a model. NVIDIA's DGX Spark works the same way as a Mac, with 64GB or 128GB of unified memory shared by its CPU and GPU.

In our view, the practical rule is simple: on a Mac or other unified-memory machine, read the table one size down from your total, unless you are willing to close everything else while the model runs.

Two other things take memory on top of the weights:

  • The context: the conversation, files and tool output the model is holding. A coding agent sends long prompts, and Qwen's model card advises keeping at least 128K tokens of context to preserve the model's thinking ability. How much memory that takes depends on the model's architecture; your runtime reports it when it loads the model.
  • Optional extras: the image encoder (the mmproj file) for models that read images, and draft files for speculative decoding. You can leave them out if you only need text.

The best local LLM at each memory size

Memory Our pick Quantisation Download size Source
16GB Gemma 4 26B A4B (or 12B for long context) Google's QAT Q4_0 14.44GB (12B: 6.98GB) 26B A4B, 12B
24GB Qwen3.8 27B Q4_K_M 19.40GB ggml-org
32GB Qwen3.8 27B, with a long context Q4_K_M 19.40GB ggml-org
64GB Qwen3.8 27B Q8_0 28.60GB ggml-org
128GB Qwen3.8 Flash Next IQ4_NL 102.04GB ggml-org
256GB Qwen3.8 Flash Next Q8_0 162.61GB ggml-org
512GB DeepSeek V4.1 Flash 1017 (2-bit), or a smaller model at high precision Q2_K 334.82GB ggml-org

The picks are our judgement, based on the labs' own published benchmarks (linked in each section below) and on what actually fits. Vendors' benchmarks are run by the vendor; treat them as a guide to which model to try first, not as a verdict.

What is the best LLM for 16GB?

With a 16GB card, such as the RTX 5060 Ti with 16GB listed in llama.cpp's CUDA benchmark thread, the strongest model that fits entirely on the card is Gemma 4 26B A4B in Google's own quantisation-aware 4-bit build, a 14.44GB file (Google on Hugging Face). It is a mixture-of-experts model with 25.2B parameters, of which 3.8B are active for each token (model card), so it generates quickly for its size. The catch is room: on a 16GB card it leaves little memory for context, which is fine for chat and too little for a coding agent.

For long prompts, Gemma 4 12B is the better fit at 16GB: Google's 4-bit build is 6.98GB (Hugging Face), leaving most of the card for context. Google's model card puts it at 72.0% on LiveCodeBench v6 against 77.1% for the 26B A4B, and gives both a 256K-token context window.

On a 16GB Mac, our advice is the 12B: the 26B A4B file plus macOS will not leave enough working room.

Can a 16GB card run a bigger mixture-of-experts model?

Yes, partly. llama.cpp can keep the experts of a mixture-of-experts model in system memory and the rest on the GPU, with the --n-cpu-moe and --cpu-moe options (llama-server documentation). That lets a card run Qwen3.6 35B A3B, whose Q4_K_M file is 20.54GB (ggml-org), with the overflow in the computer's RAM. It works, but anything that crosses from system memory to the card travels over PCIe, so it is slower than a model that fits on the card, especially when processing long prompts.

What is the best LLM for 24GB VRAM?

At 24GB (an RTX 4090 or RTX 3090, both listed with 24GB in llama.cpp's CUDA thread), the pick is Qwen3.8 27B at Q4_K_M, a 19.40GB file (ggml-org). It is a dense 27B model released under Apache 2.0, with a native context of 262,144 tokens (model card).

On Qwen's own benchmarks it is a clear step up for coding. Its model card gives 61.7 on SWE-bench Pro against 53.5 for its predecessor Qwen3.6 27B (Qwen ran SWE-bench Pro with the Claude Code harness), and 73.0 on Terminal Bench 2.1 against 63.4. If you drive it with the Pi coding agent and thinking off, see our brief on the Qwen3.8-27B pi adapter.

The alternative is Qwen3.6 35B A3B, a 20.54GB file at Q4_K_M (ggml-org). With only 3B of its 35B parameters active per token (model card), it generates much faster than a dense 27B model on the same card, at a lower score: 49.5 on SWE-bench Pro on Qwen's card. Choose it when speed matters more than the hardest tasks.

Qwen3.8 27B at Q4_K_M leaves only a few gigabytes of a 24GB card for context. If you need more, llama.cpp can store the context at 8-bit with --cache-type-k q8_0 and --cache-type-v q8_0 (llama-server documentation), which roughly halves its footprint compared with the default 16-bit. Since llama.cpp 1.0 there is also an FP8 option, f8_e4m3, still without published quality figures.

What fits in 32GB?

At 32GB (an RTX 5090, listed with 32GB in llama.cpp's CUDA thread, or a 32GB Mac), the best use of the extra memory is not a bigger model but more context for the same one. Qwen3.8 27B at Q4_K_M (19.40GB, ggml-org) leaves a large share of a 32GB card for the conversation and files, which is what a coding agent working on a real repository needs.

The 8-bit version of the same model is 28.60GB (ggml-org): it fits on a 32GB card, but with too little left for a useful context.

Gemma 4 31B, Google's dense model, is another option here at 17.65GB in Google's 4-bit build (Hugging Face), with 80.0% on LiveCodeBench v6 on Google's card. For agentic coding, Qwen's own comparison on the Qwen3.6 35B A3B card puts Gemma 4 31B well behind the Qwen models on SWE-bench Pro (35.7).

64GB: the same model, at higher precision

In our view, 64GB is best spent on precision and context for a current model rather than on an older, larger one. Qwen3.8 27B at Q8_0 is 28.60GB (ggml-org), close to the full-precision model's quality, with room left for a long context.

Full precision does not fit comfortably. The BF16 file is 53.81GB (ggml-org), and Apple's machine learning team measured the BF16 model taking up to 68GB of memory with a 65,536-token context on a Mac Studio (15 October 2026).

128GB: Qwen3.8 Flash Next

At 128GB, a Mac Studio or MacBook Pro with 128GB, or a 128GB DGX Spark, the step up is Qwen3.8 Flash Next. Qwen's model card describes it as 125B parameters with 6B active per token, plus 51B of n-gram embeddings and 4B for multi-token prediction. The IQ4_NL build is 102.04GB (ggml-org). On a Mac, Ollama's MLX build of the model now needs about 27 GiB less memory than before; our brief on Ollama 0.40.16 explains the change and its caveats.

On the same card, Qwen reports 62.5 on SWE-bench Pro, against 61.7 for Qwen3.8 27B and 56.0 for DeepSeek V4 Flash 0731. Note its licence: it is published under the Qwen Community licence, not Apache 2.0 like the 27B, so read the terms before using it in a product.

The other candidates at 128GB, with sizes from each publisher's file list on Hugging Face:

  • DeepSeek V4 Flash (284B parameters, 13B active, per DeepSeek's model card): ggml-org's 2-bit Q2_K_S build is 98.59GB and Q2_K is 117.34GB (ggml-org). A bigger model at a much harsher quantisation; the 117.34GB file leaves little room on a 128GB machine.
  • Mistral Small 4 (119B parameters, 6.5B active, per Mistral's model card), under Apache 2.0: Mistral's own build in NVFP4, NVIDIA's 4-bit format, is 70.81GB (Hugging Face).

256GB and 512GB: precision or size

At 256GB, Qwen3.8 Flash Next fits at Q8_0, a 162.61GB file (ggml-org), close to full quality with plenty of room for context. DeepSeek V4 Flash also fits in its MXFP4 build, 154.99GB (ggml-org), which is close to how DeepSeek publishes it: the model card says its expert weights are already stored in FP4.

At 512GB, the question becomes whether to run a model at high precision or a bigger one at 2 bits (sizes from the Hugging Face file lists):

Model Parameters Build Size Source
DeepSeek V4.1 Flash 1017 552B backbone Q2_K 334.82GB ggml-org
DeepSeek V4.1 Flash 1017 552B backbone DeepSeek's own weights 510.30GB DeepSeek
Mistral Large 4 about 1T, 52B active Q2_K 380.71GB ggml-org
Mistral Large 4 about 1T, 52B active Q4_K_M 634.80GB ggml-org
Qwen3.5 397B A17B 397B, 17B active GPTQ Int4 235.75GB Qwen

DeepSeek V4.1 Flash 1017, DeepSeek's October update published under the MIT licence, is the largest current DeepSeek model that fits in a single 512GB machine, but only at 2 bits: ggml-org publishes Q2_K and Q2_K_S builds of it (334.82GB and 298.01GB) and nothing larger (ggml-org). DeepSeek's own weights total 510.30GB (Hugging Face), which leaves nothing for context or the operating system. DeepSeek's technical summary gives 552B backbone parameters, with 8B active per token when reading the prompt and 16B when generating.

Mistral Large 4 fits at 2 bits too, at 380.71GB for Q2_K, but its 4-bit build is 634.80GB (ggml-org), more than one 512GB machine holds.

In our view, 2-bit quantisation costs a model noticeably more quality than 4- or 8-bit, so a 512GB machine is not automatically better served by the biggest model it can hold. Try the 2-bit DeepSeek against Qwen3.8 Flash Next at Q8_0 on your own work before settling on one.

Which models need more than one machine?

Kimi K3.1 Pro, Moonshot's 2.8-trillion-parameter model (104B active per token, per its model card), has its weights on Hugging Face. They were trained to run at 4 bits (MXFP4), and the published files still total 1,561.35GB (file list). To run Kimi locally, you need several machines sharing the model, such as a cluster of 512GB Mac Studios; our Mac Studio guide covers how Apple's clustering works.

DeepSeek V4.1 Flash at full quality (510.30GB of weights, DeepSeek) and Mistral Large 4 at 4 bits or more (634.80GB at Q4_K_M, ggml-org) are in the same position: more than one 512GB machine, or a server with several large GPUs.

How fast will it run?

Generation speed is set mostly by memory bandwidth: for every token, the hardware reads the model's active weights from memory. That is why a mixture-of-experts model with 3B active parameters generates far faster than a dense 27B model of a similar file size, and why the same model runs faster on a card or chip with more bandwidth.

Measured speeds for these exact models on each card are scarce. As of 10 November 2026 we found no reproducible, published benchmark from a lab, a runtime project or a hardware maker for most of the picks above, and we do not estimate them. What does exist is llama.cpp's own benchmark threads, which run one small reference model (Llama 2 7B at Q4_0) with the same command on every machine. They show how hardware compares, not how fast any model above will be:

Hardware Memory @ bandwidth Tokens/s generated (Llama 2 7B Q4_0) Sources
RTX 5090 32GB @ 1.79TB/s 290.02 llama.cpp CUDA thread, NVIDIA specs
RTX 4090 24GB @ 1TB/s 186.21 llama.cpp CUDA thread, NVIDIA Ada whitepaper
M5 Ultra (80-core GPU) up to 512GB @ 1.2TB/s 179.10 llama.cpp Apple thread, Apple specs
RTX 3090 24GB @ 936GB/s 158.16 llama.cpp CUDA thread, NVIDIA Ampere whitepaper
M5 Max (40-core GPU) 48 to 128GB @ 614GB/s 119.92 llama.cpp Apple thread, Apple specs
RTX 5060 Ti 16GB @ 448GB/s 90.94 llama.cpp CUDA thread, NVIDIA specs

The one current model with a published, reproducible measurement is Qwen3.8 27B: Apple's machine learning team measured it at 19.69 tokens per second on an M5 Ultra Mac Studio and 12.46 on an M3 Ultra, at full BF16 precision with an 8,192-token context, using MLX LM (15 October 2026). A 4-bit build reads less than half the bytes per token, so it should be faster on the same machine.

We will add measurements to this guide as they are published.

How to run these models

Every GGUF build above runs in llama.cpp, and the -hf option downloads a build straight from Hugging Face (llama-server documentation):

# Qwen3.8 27B at Q4_K_M, the 24GB pick, with a 64K-token context
llama-server -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M -c 65536

# Gemma 4 12B in Google's own 4-bit build, the long-context 16GB pick
llama-server -hf google/gemma-4-12B-it-qat-q4_0-gguf -c 65536

LM Studio and Ollama run GGUF models too; check each model's page in those apps for the builds they offer, since names and quantisations differ from ggml-org's.

Use the big machine from your laptop

The memory that decides which model you can run is usually in one machine: a desktop with a large graphics card, or a Mac Studio at home or in the office. The laptop you carry to a café or onto a train cannot hold the same model.

Corewire connects the runtimes already on that machine (Ollama, LM Studio, MLX LM and others) to your other computers over your own private, encrypted network, and gives each model one name, such as mac-studio/qwen3-coder:30b, that a coding agent on your laptop can use. The Claude Code guide shows the one command that starts Claude Code on a model on another of your machines. You can also run the agent itself on the big machine and drive it from your laptop, as the sessions guide explains.

Questions people ask

What is the best local LLM for 24GB of VRAM?

Qwen3.8 27B at Q4_K_M, a 19.40GB download (ggml-org), as of 10 November 2026. Qwen3.6 35B A3B (20.54GB at Q4_K_M) is the faster alternative, with lower scores on Qwen's own coding benchmarks.

Can I run DeepSeek locally?

Yes, with enough memory. DeepSeek V4 Flash fits a 128GB machine at 2 bits (98.59GB, Q2_K_S) and a 256GB one in its 4-bit MXFP4 build (154.99GB), per ggml-org's builds. The newer V4.1 Flash 1017 needs 512GB at 2 bits (334.82GB, ggml-org).

Can I run Kimi locally?

Not on one machine. Kimi K3.1 Pro's weights total 1,561.35GB even at their native 4-bit precision (Hugging Face), so it needs a cluster of large-memory machines or a multi-GPU server.

Is a 16GB Mac enough for a local LLM?

For a small model, yes: Gemma 4 12B in Google's 4-bit build is 6.98GB (Hugging Face). macOS and your apps share the same 16GB, so the larger models in this guide that suit a 16GB graphics card are too tight on a 16GB Mac.

More in New models

Corewire shares the models on your computers with every device you own, and with the coding agents you already use. Download it free or read the guides.