Running Local LLMs by Hardware Tier in 2026: What Actually Fits on What You Own

Most guidance on running models locally starts with a leaderboard. That ordering fails, because the highest-ranked model you cannot load helps you less than the mid-tier model that runs at usable speed on hardware you already own.

Start with memory. Everything else follows from it.

The Arithmetic That Decides Everything

A model’s memory requirement tracks its parameter count multiplied by the bytes per parameter, plus overhead for the context window and the runtime itself.

At full 16-bit precision, budget roughly two gigabytes of memory per billion parameters. A 7-billion-parameter model needs about 14GB before context. A 70-billion-parameter model needs about 140GB, which puts it outside any single consumer card.

Quantization changes the picture more than any other variable. Reducing weights to 8-bit roughly halves the requirement. Four-bit roughly quarters it. That 70B model at 4-bit lands near 35 to 40GB, which moves it from impossible to merely demanding.

The cost of quantization arrives as quality degradation, and it lands unevenly. Reasoning and long-context tasks degrade noticeably at 4-bit. Classification, extraction, and summarization often survive it with no difference a user would notice. Test on your actual workload rather than trusting a general claim in either direction.

Tier One: 8 to 12GB of VRAM

A mainstream gaming card or a base-configuration laptop with unified memory.

What runs well: 7B and 8B parameter models at 4-bit quantization, with room for a reasonable context window. Response speed lands comfortably above reading pace.

What these models do well: summarization, classification, extraction from documents, simple code completion, and drafting that a human will edit. Anything where the task shape stays narrow and the output gets reviewed.

Where they fail: multi-step reasoning, long-context synthesis, and code generation complex enough that reviewing the output costs more than writing it. Expect confident errors on anything requiring the model to hold several constraints at once.

Honest positioning: this tier handles a meaningful share of production text processing at zero marginal cost per call, and it disappoints anyone expecting a frontier model on a budget.

Tier Two: 16 to 24GB of VRAM

A high-end consumer card or a well-specified workstation.

What runs well: 13B and 14B models at 8-bit, or 30B-class models at 4-bit with a modest context window.

What changes: instruction following tightens noticeably. The model holds multi-part instructions without dropping one. Code generation reaches the point where review costs less than authorship for routine work.

The tradeoff to watch: a 30B model at aggressive quantization on 24GB leaves little headroom for context. Long documents force a choice between the larger model with a short window and the smaller model with room to work. For document-heavy workloads the smaller model with the longer window usually wins.

Tier Three: 48GB and Above

Dual cards, a workstation GPU, or a high-memory unified architecture.

What runs well: 70B-class models at 4-bit with usable context, or 30B-class at 8-bit with generous context.

What changes: the gap against hosted frontier models narrows to something a business can reason about. Not closed, and narrow enough that data residency or unit economics can justify the difference.

What it costs: hardware in the several-thousand-dollar range, plus the operational responsibility that follows. Somebody patches it, monitors it, and answers when it stops.

Tier Four: Server-Class Deployment

Multiple accelerators in a rack, whether owned or rented.

At this point the question moves past hardware selection and becomes an infrastructure program. You need model serving, batching, autoscaling, observability, and a person who owns all four. Our self-hosted platforms guide covers the platform layer, and our open-source stack comparison covers what runs on it.

The honest threshold: below sustained, predictable volume, renting inference beats owning it. Buy the hardware when utilization stays high enough that the fixed cost divides into a per-call number that beats the API, or when regulation removes the API from consideration entirely.

What Local Actually Buys You

Three benefits justify the operational burden, and only three.

Data never leaves your boundary. No prompt logs on somebody else’s infrastructure, no subprocessor list to audit, no jurisdictional question about who can compel access. For regulated workloads this alone decides the matter, which our data residency framework covers in depth.

Marginal cost per call approaches zero. After the hardware, an additional million tokens costs electricity. High-volume, repetitive workloads reach a crossover point where owning beats renting by a wide margin.

Nobody deprecates your model. A hosted provider retires a version and your prompts behave differently overnight. A local model sits exactly where you left it, which matters most for anything you validated once and want to keep stable.

What local does not buy you: better quality, less work, or freedom from evaluation. The model still drifts against your data even when its weights never change, because the inputs change around it.

Quantization Formats Worth Knowing

The format decides both what fits and how fast it runs.

GGUF dominates CPU and mixed CPU-GPU inference, and it handles partial offload gracefully when a model exceeds available VRAM. Slower than pure GPU execution, and forgiving when memory runs short.

GPTQ and AWQ target GPU execution and generally run faster than GGUF at comparable size, with less flexibility about spilling to system memory.

The practical rule: if the model fits entirely in VRAM, prefer a GPU-native format. If it does not fit and you would rather accept slower generation than choose a smaller model, GGUF with partial offload keeps you running.

Measure tokens per second on your own hardware rather than trusting a published figure. Memory bandwidth, not raw compute, usually sets the ceiling, and two cards with similar specifications can differ substantially.

The Variable Everyone Underestimates

Context window consumes memory that scales with sequence length, and long contexts consume a great deal of it. A configuration that loads cleanly at 4,000 tokens can exhaust memory at 32,000.

Plan the context you actually need before selecting the model. Teams routinely choose a larger model, discover it cannot hold their documents, and end up with worse results than a smaller model would have delivered with room to read.

Where Local Stops Making Sense

Three conditions send you back to hosted inference regardless of hardware.

Frontier reasoning requirements. The gap at the top remains real. Work demanding the strongest available reasoning belongs on a hosted frontier model, and no local configuration closes that distance today.

Bursty, unpredictable volume. Idle hardware costs the same as busy hardware. Spiky workloads favor per-call pricing.

No operational owner. A local deployment without somebody accountable for patching, monitoring, and capacity becomes an outage waiting for a quiet week. Our inference cost playbook covers the economics of the choice in more depth.

A Selection Method That Works

Write down the workload shape. Task type, typical input length, required output quality, and volume per day.
Set the context requirement from real documents rather than from an estimate.
Compute the memory budget at the quantization level you will accept.
Pick the largest model that fits with headroom, then test at the quantization you planned rather than at full precision.
Measure against your own labeled examples. A benchmark rank tells you nothing about your extraction task.
Re-run the comparison quarterly. Both the models and the quantization methods keep improving, and last quarter’s answer expires.

The Takeaway

Memory determines the shortlist. Quantization determines whether the shortlist includes anything interesting. Your workload determines whether the tradeoff costs you anything that matters.

Teams that start from a leaderboard buy hardware for a model they will not run. Teams that start from their own documents and their own memory budget usually discover that a smaller model, chosen deliberately, does the job.