Running a model on your own machine means no API bills, no data leaving your computer, and no rate limits — but it also means living inside your hardware's limits. The good news is that what you can run is mostly determined by one number you can look up in a minute. This guide explains what actually constrains local inference, what quantization really trades away, and what to realistically expect from different classes of machine.

Memory is the gate

The first question is not how fast your chip is; it is whether the model fits in memory at all. A model that does not fit either will not load or spills onto disk and slows to a crawl. So before anything else, compare the model's size on disk to the memory available to your accelerator. A useful floor: you need roughly the size of the model file, plus headroom for the context — the conversation and any documents — and the runtime itself, often a few extra gigabytes. If a quantized model file is 20 GB, plan for meaningfully more than 20 GB of usable memory.

Speed matters too, but it is the second question. Memory bandwidth — how fast the chip can stream the weights — largely sets how many tokens per second you get once a model fits. Raw compute matters most for chewing through a long prompt. But none of that is relevant if the model never fits in the first place, which is why memory capacity is the gate you check first.

Unified memory versus VRAM

There are two common memory setups, and the distinction drives what you can run.

On a traditional PC with a discrete GPU, the model must fit in the graphics card's dedicated VRAM to run fast. VRAM is fast but limited and expensive; consumer cards top out well below what the largest models need. If a model overflows VRAM into system RAM, throughput collapses. So on this kind of machine, your GPU's VRAM is the hard ceiling.

Apple Silicon Macs and some newer systems use unified memory: the CPU and GPU share one large pool. A Mac with a lot of unified memory can therefore hold models that would never fit on a consumer graphics card, because most of that shared memory is available to the GPU. The tradeoff is that unified-memory bandwidth, while high, is generally lower than a top discrete GPU's, so a model that fits comfortably may still run slower than it would on a smaller card that can barely hold it. For local LLM work the ability to fit large models often matters more than peak bandwidth, which is why high-memory Macs have become popular for running big open-weight models at home.

Quantization, concretely

Models are typically trained in 16-bit precision, meaning each weight is stored in 16 bits. Quantization compresses those weights to fewer bits — most commonly 8, 5, or 4, sometimes lower — which shrinks the file and the memory needed to run it, at some cost to quality. The notation you will see, like Q4 or Q8, refers to the bits used per weight.

The rough tradeoff: 8-bit is nearly indistinguishable from the full model at about half the size. 4-bit is roughly a quarter of the size and, for most uses, keeps the large majority of the original quality — which is why it is the community default. Below 4-bit, quality degrades faster and more visibly, though it can be worth it to squeeze a bigger model onto a smaller machine. The counterintuitive but well-supported rule of thumb: a larger model at 4-bit usually beats a smaller model at full precision that takes the same memory. Given a fixed memory budget, prefer a bigger model quantized over a smaller one unquantized.

  • Q8 — about 8 bits per weight, roughly half the full size, quality essentially unchanged. Use it when it fits and you want maximum fidelity.
  • Q4 (especially the K-quant variants like Q4_K_M) — about 4 bits per weight, roughly a quarter of full size, keeps most of the quality. The default most people should start with.
  • Q5 / Q6 — a middle ground when you have spare memory above Q4 and want a little more headroom on quality.
  • Q2 / Q3 — aggressive compression for fitting large models on constrained hardware; expect visible quality loss and test before relying on it.

GGUF and friends

You will see a few file formats. GGUF is the dominant single-file format for local inference: it bundles the weights, tokenizer, and metadata into one portable file and runs across Metal, CUDA, CPU, and more. It is what the popular tools download and what most quantized community models ship as. On Apple Silicon you will also see MLX, Apple's framework, which can run faster on Macs. The original full-precision weights usually come as safetensors, which you convert and quantize to GGUF (or MLX) for local use. For most people the practical takeaway is simple: look for a GGUF build at Q4 or Q8 and you are ready to go.

The tooling sits in layers worth knowing. llama.cpp is the open-source engine that quietly powers much of the ecosystem. Ollama wraps it with a one-command experience and sensible 4-bit defaults. LM Studio adds a graphical model browser that shows whether a given quantization will fit before you download it. They are different front doors to largely the same underlying inference.

What to expect by hardware class

Exact speeds depend on the specific model, quantization, and settings, so treat these as capability tiers, not benchmarks.

  • A typical laptop (integrated graphics, 8 to 16 GB RAM). You can run small models — think a few billion parameters at 4-bit — for chat, summarizing, and light coding help. They are genuinely useful and private, but not frontier-quality. Expect modest speed and keep the context small.
  • A gaming PC with a discrete GPU (roughly 12 to 24 GB VRAM). You can comfortably run mid-sized models at 4-bit and get fast, responsive output. This is a sweet spot for quality per dollar, as long as you stay within the card's VRAM.
  • A Mac with a lot of unified memory (64 GB and up). The fit-large-models machine. You can load big open-weight models that a consumer GPU cannot hold, trading some raw speed for the ability to run models much closer to hosted quality. Popular for serious local work.
  • A workstation or multi-GPU rig. With enough combined VRAM you can run the largest open-weight models and serve them to others, approaching what you would otherwise pay an API for — at a steep hardware cost and real setup effort.

The whole decision collapses to a short checklist: find the model's quantized file size, confirm it fits your accelerator's memory with headroom, prefer a bigger model at 4-bit over a smaller one at full precision, and grab the GGUF build. Get those right and local inference is far less mysterious than it looks.