How Much VRAM Do Local AI Models Need? A Planning Guide for Local LLMs and Image Generation

2026-10-23 ｜ GPU Repair ｜ About 20 minutes ｜ HK GPU Fix Editorial Team

Summary

Running a language model or an image generator on your own graphics card sounds simple until it runs out of memory halfway through a job. This guide explains how model weights, context, and image settings consume VRAM, how quantization changes the maths, what happens when work spills into system RAM, and how to tell a memory bottleneck from a fault.

Why "How Much VRAM?" Has No Single Number

Ask how much VRAM a local AI model needs and you will get a different answer from every forum thread, because the question hides several smaller questions. Which model, at which quantization level, with how long a context window, generating how many images at once, at which resolution, and with what else already using the card? Change any one of those and the memory budget changes with it.

A useful way to think about it is to split the budget into parts. Most of the budget is the model itself: the weights, the numbers the model learned during training. On top of that sits working memory that grows while the model runs: the key-value cache (KV cache) for a language model, which stores information about the conversation so far, and the intermediate images, or latents, for an image generator. Then there is the memory the rest of the system needs: the display output, the desktop, and any game, browser, or second application sharing the same card. A plan that only counts the weights will always underestimate the real requirement, and the shortfall usually appears at the worst moment, partway through a long generation.

This guide is a planning guide, not a shopping list for a particular card. It explains the maths, the trade-offs, and the diagnostic signs, so you can judge whether your current card can run the workload you actually want, what to turn down first when it cannot, and when a capacity problem is really a different fault wearing a memory problem's clothes.

What Actually Lives in VRAM When a Local Model Runs

Five things compete for VRAM during local AI work, and they behave differently.

First, the model weights. These are loaded once and stay resident for the whole session. Their size is fixed for a given model and quantization level, which makes them the easiest part to plan for and the part most guides quote on their own.

Second, the KV cache for language models. As a conversation grows, the model stores a running summary of the tokens it has already processed so it does not have to recompute them for every new word. This cache grows with context length: a short chat costs little, while a very long document or a long multi-turn conversation costs much more. It also grows with batch size when several sequences run at once. This is the part that surprises people most, because a model that loads happily can still run out of memory halfway through a long session.

Third, activations and temporary workspace. During each step of generation, the model creates intermediate results that exist only briefly. These are smaller than the weights for a single short sequence, but they scale with sequence length, batch size, and, for image models, resolution.

Fourth, the image pipeline for image generation. A diffusion model holds the main model weights, plus a text encoder, a variational autoencoder (VAE) that converts between pixel images and the compressed latent space, and the latent images themselves as they are refined step by step. Optional add-ons such as ControlNet units or style adapters add their own weights on top.

Fifth, everything else on the card. The operating system reserves memory for the display, the desktop compositor keeps its share, and any other application using the GPU takes its share too. On a machine that also drives a high-resolution monitor or a second screen, this background share is not negligible. Planning against the card's total advertised VRAM, rather than the VRAM actually free after the system takes its cut, is one of the most common planning errors.

Memory also fragments. A card can report some free VRAM yet fail to load a model, because the free space is split into pieces too small for one large allocation. Closing other GPU applications before loading a model, and loading the largest model first, reduces this problem.

The Weight Maths: Quantization Levels Explained

Quantization is the main lever for fitting a model onto a card. In simple terms, it stores each weight with fewer bits, so the model takes less memory at some cost to numerical precision. The original training precision for many models is 16-bit floating point (FP16) or brain floating point (BF16), both using two bytes per parameter. Halving the precision roughly halves the weight memory.

The common levels, from largest to smallest, are:

• FP32 (32-bit floating point): four bytes per parameter. Rarely used for local inference, because it doubles memory for no practical quality gain on consumer hardware.
• FP16 / BF16 (16-bit floating point): two bytes per parameter. The reference quality level for most models.
• FP8 (8-bit floating point): one byte per parameter. A newer option on hardware that supports it.
• INT8 (8-bit integer): one byte per parameter. Roughly half the memory of FP16.
• INT4 / Q4 (4-bit integer): half a byte per parameter. Roughly a quarter of the memory of FP16. The most common way to fit large models onto consumer cards.
• Lower levels (Q3, Q2, and similar): less than half a byte per parameter. These fit even larger models, but quality degradation becomes much more noticeable, and they suit experimentation more than everyday use.

The arithmetic is straightforward: multiply the parameter count by the bytes per parameter. The table below shows weight memory only, before KV cache, activations, and system reserve. Treat it as a floor, not a total.

Two cautions belong with this table. First, real files carry overhead: quantization metadata, embeddings, and the runtime's own structures add something on top of the bare multiplication, so a Q4 model in the 8-billion class typically needs more than the bare 4 GB once loaded. Second, dropping to a lower quantization level is not free quality. For short, simple prompts the difference can be hard to notice; for long reasoning, precise instruction following, code, or fine detail, lower precision shows up as vaguer answers, more errors, and less consistent formatting. The right level is the highest one that fits comfortably with room left for the KV cache, not the lowest one that barely loads.

Mixture-of-experts models deserve a separate note, because their headline parameter count counts every expert, while only a subset of experts is active for any given token. Their total weight memory follows the full parameter count if the whole model is loaded, but their per-token compute follows the active subset. Do not plan an expert model from its active parameter count alone; plan from the loaded weight size the runtime reports.

Planning for Local LLMs

For a language model, the planning sequence runs in this order: weights first, then context, then everything else.

Start with the weights at the quantization level you intend to use, using the table above as a floor and adding a margin for runtime overhead and system reserve. Then ask how long your context needs to be. Short chats, single questions, and brief summaries keep the KV cache small. Long documents, large code files, and long multi-turn sessions grow it steadily, and at some point the cache, not the weights, becomes the binding constraint. If you regularly work with long inputs, a smaller model at higher precision with a long context can be a better experience than a larger model at low precision that runs out of memory halfway through the document.

Batch size matters the same way. Running several sequences at once multiplies the KV cache and activation memory. A card that handles one conversation comfortably may struggle with four parallel generations, and the failure mode is the same: slowdown first as memory pressure grows, then an out-of-memory error when the budget is finally exceeded.

The table below gives planning bands rather than promises. "Comfortable" here means weights plus a reasonable context and system reserve fit without offloading; your own mix of context length and background applications moves the boundary in either direction.

Speed is a separate question from fitting. A model that fits is not automatically fast: generation speed depends on the card's compute capability and memory bandwidth as well as capacity, and a heavily offloaded model can be dramatically slower than a smaller model that fits entirely in VRAM. If your priority is responsive conversation rather than the largest possible model, a medium model that fits fully in VRAM usually gives a better experience than a very large model that mostly lives in system RAM.

Context Length: The Hidden Second Budget

Context length deserves its own section because it is the budget most people forget. The KV cache stores, for every layer of the model, the keys and values of every token processed so far. Double the context and you roughly double the cache. Raise the batch size and the cache multiplies again. This is why a model can load, answer short questions all day, and then fail on the one long document that matters.

The practical levers, in order of least damage, are: shorten the input (summarize or split long documents before feeding them in), reduce the batch size to one sequence, start a fresh conversation instead of carrying an ever-growing history, and only then reduce the quantization level or switch to a smaller model. If your runtime lets you cap the context length, setting a cap that matches your real use prevents a slow memory creep across a long session from turning into a sudden failure.

Watch for the warning pattern: generation starts fast, then slows progressively as the conversation grows, then fails with an out-of-memory error on a long input. That progressive slowdown is the signature of a growing KV cache approaching the card's limit, and it points at context management rather than at a faulty card.

Image Generation: A Different Kind of Memory Load

Image generation loads memory differently from chat. The model weights are again the base, but the working set scales with image size, batch size, and optional add-ons rather than with conversation length. The number of sampling steps mainly affects time, not peak memory: twenty steps and fifty steps use roughly the same peak memory, but fifty steps take longer.

Resolution is the big lever. Doubling the width and height of an image quadruples the pixel count, and the latent representation grows with it. A resolution that works comfortably can fail when pushed one step higher, and the failure often arrives together with upscaling or a second refinement pass that adds its own memory on top. Batch size multiplies the cost the same way: generating four images at once needs roughly four times the working memory of generating one, even though the weights are loaded only once.

Some image tools offer memory-saving modes such as attention slicing, CPU offload of the VAE, or tiled processing. These trade speed for memory and are worth trying before concluding that a card cannot do the job at all. They slow generation, sometimes substantially, but they change a hard failure into a slow success, which is the right trade when the alternative is no image at all.

When VRAM Runs Out: Offloading to System RAM

When a model does not fit in VRAM, most runtimes do not simply refuse. They offload: part of the model stays in VRAM and the rest lives in system RAM, with layers shuttled back and forth over the PCIe bus as needed. Some tools offload whole layers to the CPU for computation as well. This keeps the job running, but it changes the performance character completely.

The cost comes from bandwidth. VRAM is designed to feed a graphics processor at very high bandwidth; system RAM reaches the processor over a much narrower path, and CPU computation is slower for this kind of work than GPU computation. The result is generation that is far slower than a fully VRAM-resident model: text appears word by word with long pauses, and image generation stretches from seconds to minutes. The slowdown grows with the share of the model that is offloaded: offloading a small tail of layers is a modest penalty, while offloading most of the model turns the graphics card into a bystander.

System RAM itself then becomes a second budget. The offloaded weights, plus the operating system and other applications, must fit in physical RAM. If physical RAM also runs out, the operating system starts paging to disk, and performance collapses from slow to effectively unusable, often accompanied by a system that feels frozen for other tasks too. A machine with generous, fast system RAM copes with partial offloading far better than a machine with barely enough RAM to hold the model.

Use offloading as a deliberate choice, not a default you never notice. If a model only runs with most of its layers offloaded, ask whether a smaller model or a lower quantization level that fits in VRAM would serve the same purpose faster. Offloading is at its best as a bridge: it lets a slightly-too-large model run at all, and it lets you confirm that a workflow is worth pursuing before deciding on a hardware change.

VRAM Bottleneck or Something Else?

A memory error is not always a memory problem, and slow generation is not always a capacity problem. The table below separates the common lookalikes. The key habit is to change one thing at a time and watch which symptom moves.

Two diagnostic habits save the most time. First, watch the numbers while the job runs: VRAM used, system RAM used, and GPU utilization together tell you whether the card is working hard, waiting on data, or idle while the CPU does the work. A card at low utilization during slow generation is waiting, and the wait is usually memory movement or CPU fallback. Second, test with a smaller model. If the small model runs briskly and the large one crawls, you have a capacity and offloading story. If even the small model crawls or crashes, the problem sits elsewhere: drivers, cooling, power, or a fault.

A Practical Planning Checklist

Work through this list before deciding anything about hardware:

1. Write down the workload: which model, which quantization level, how long the context or how large the image, and how many at once.
2. Compute the weight floor from the quantization table, then add room for KV cache or image working memory and for the system reserve.
3. Close other GPU applications, load the model, and watch free VRAM before and during a realistic session, not just at load.
4. If it fits but slowly, check how much is offloaded before assuming the card is too slow.
5. If it fails only on long inputs, manage context first: shorter inputs, smaller batches, fresh sessions.
6. If it fails only at high resolution or large batches, reduce those before reducing model quality.
7. Only after the settings levers are exhausted should capacity itself be judged too small.

When an Upgrade Makes Sense, and When It Does Not

An upgrade makes sense when the workload is settled, the settings levers have been tried, and the card still cannot hold the model plus a realistic context or image size without heavy offloading. It does not make sense as a first response to a single out-of-memory error, which is more often a quantization level, a context length, or a background application away from being solved.

It also does not make sense as a fix for a fault. A card that crashes, shows artefacts, or resets the display driver across different workloads is not short of capacity; it has a problem that more VRAM will not solve. That distinction matters because the remedies are different: settings and capacity planning for a capacity problem, and inspection for a fault. Repair pricing on this site follows the published three-tier range of HK$400–2,800, with the actual quote given after inspection.

Finally, remember that local AI is only one tenant on the card. A card chosen purely for the largest model may be a poor fit for the games, creative work, or display setup it also serves. Plan for the whole machine, keep a margin for the system reserve, and prefer the configuration that runs your everyday workload well over the one that barely loads the largest model you might one day try.

FAQ

How much VRAM do I need for a 7–8 billion parameter chat model?

At Q4, the weights alone are around 4 GB by the arithmetic above, and a comfortable plan allows room on top for the KV cache, activations, and the system reserve. A card with 8 GB of VRAM generally handles this class with sensible context lengths, while 12 GB or more leaves comfortable headroom for longer conversations and other applications sharing the card.

Is Q4 quality good enough for everyday use?

For short chats, drafting, and general questions, most people find Q4 perfectly usable. For long reasoning, code, and tasks where small errors compound, higher precision is noticeably more reliable. Treat Q4 as the practical default for fitting larger models, and step up in precision when the work demands accuracy rather than size.

Why does my model load but fail on long documents?

Because the KV cache grows with context length. The weights fit, but the weights plus the growing cache do not. Shorten the input, split the document, reduce the batch size, or start a fresh session, and the same model will usually complete the same work.

Does offloading damage the card?

Offloading itself is a normal runtime feature, not damage. It moves data between VRAM and system RAM and may use the CPU for some layers. The cost is speed, not harm. The thing to watch is system RAM: if physical RAM is also exhausted, disk paging makes the whole machine slow, which feels like a card problem but is not one.

Should I buy more VRAM or more system RAM first?

It depends on which budget is binding. If models fail to load or need heavy offloading, VRAM is the constraint. If the machine slows everywhere once a model is offloaded, or other applications crawl, system RAM is the constraint. Watch both readings during a realistic session before deciding, because upgrading the wrong one leaves the original bottleneck untouched.

My card crashes during AI loads but not games. Is that a VRAM fault?

Not necessarily. AI loads stress memory capacity and sustained compute in ways games may not, so a capacity problem shows up there first. But crashes, artefacts, or driver resets that appear across different workloads, or that worsen over time, point away from capacity and toward drivers, cooling, power, or hardware. Note exactly when the failure happens and whether it follows the card across workloads; that pattern is what an inspection starts from.

Where to Go from Here

Start from the workload, do the weight arithmetic, leave room for context and the system reserve, and let the settings levers do their work before judging capacity. Most local AI memory problems are planning problems, and planning problems have free answers.

If the pattern points to a fault rather than capacity — crashes or artefacts across workloads, or a card that behaves the same no matter the settings — bring it in for inspection rather than guessing further. HK GPU Fix is at Room 1209, Dali Plaza, Shek Mun, Sha Tin, telephone 2186 8010, WhatsApp 6558 6806. Repair pricing follows the published three-tier range of HK$400–2,800, with the actual quote given after inspection.