A deep dive into model size, memory math, quantization internals, and what fits on your GPU

When you load a model from Hugging Face, it doesn't execute in your system RAM — it executes on your GPU, which means every weight, every activation, and every cached attention state has to live inside your VRAM. Unlike system RAM, VRAM is a finite, non-expandable resource (at least per-card), and it has to hold everything at once: the model weights, the key-value cache for your context window, the CUDA runtime overhead, and any activation buffers needed during generation.

The short version of the problem: you can have a beautiful 13B-parameter model that benchmarks extremely well — but if it requires 28 GB at full precision and your GPU has 16 GB, it simply won't load.

This guide breaks down exactly how to calculate what a model costs in VRAM, what all those cryptic quantisation suffixes mean at a bit-level, and which models fit on two specific real-world GPUs: a desktop RTX 5070 Ti (16 GB GDDR7) and a laptop RTX 3060 (6 GB GDDR6).

A model's "size" — the 7B, 13B, 70B you see in names — is its parameter count: the number of individual floating-point numbers that make up the learned weights. Every one of those numbers has to be stored somewhere.

How much space each parameter takes depends on the numerical precision (data type) used to store it:

So the baseline formula for just loading the weights is:

For a 7B model at FP16:
7 × 2 = 14 GB — just for the weights.

For the same model at INT4:
7 × 0.5 = 3.5 GB — an enormous difference.

But weights alone don't tell the whole story. Before you start generating a single token, you need to account for three more things.

Already covered above. This is your baseline floor — the model can't exist in memory at less than this.

Loading the CUDA runtime, driver context, and inference framework (llama.cpp, PyTorch, vLLM, etc.) consumes VRAM before a single weight is touched. This overhead is typically 500 MB to 2 GB depending on the framework. Ollama and llama.cpp sit at the lower end (~0.5–1 GB); vLLM can be higher due to its PagedAttention allocator.

Always reserve at least 1 GB as a safety buffer when planning.

This is the component most people underestimate, and it's the one that will burn you when working with long context.

When an autoregressive transformer generates tokens, it needs to compute attention over all previously generated tokens. Rather than recomputing the key and value vectors for every past token on every new forward pass, transformers cache them. This is the KV (Key-Value) cache, and it grows linearly with every token generated.

The exact formula (per inference call, batch size = 1) is:

The 2 accounts for the key tensor and the value tensor — one of each, per layer.
n_kv_heads is specifically the number of KV heads, which in modern models using Grouped Query Attention (GQA) is smaller than the total attention head count. Llama 3 8B for example has 32 attention heads but only 8 KV heads — a 4× reduction in cache size compared to vanilla multi-head attention.

Worked example — Llama 3.1 8B at 8K context, FP16 KV cache:

That's manageable. But at 32K context:

And at 128K context (the model's native max):

With a 5 GB model loaded at Q4 and 8 GB of KV cache, you've already hit 13 GB on a 16 GB card — before counting framework overhead. Context length isn't free.

During inference, the model needs scratch space to compute the output of each layer before passing it to the next. This activation memory is small at batch size 1 (a few hundred MB) and grows with batch size. For single-user local inference you can treat it as included in the ~20% overhead factor.

A practical total estimate for single-user inference:

Or equivalently, the rule of thumb with a ~20% overhead factor baked in:

You've seen filenames like Llama-3.1-8B-Q4_K_M.gguf or qwen3-14b-q5_k_m.gguf. These aren't arbitrary — the suffix encodes exactly how the model weights were compressed.

GGUF (GPT-Generated Unified Format) is the standard container for quantised local models, created by Georgi Gerganov for the llama.cpp project. A single .gguf file contains the architecture metadata, tokeniser, and compressed weights — everything needed to run inference, no separate config files required.

A suffix like Q4_K_M has three parts:

The original llama.cpp quantisation types like Q4_0, Q5_0, Q8_0 use simple block quantisation: a block of 32 weights shares one scale factor (and optionally one zero-point for _1 variants). Dequantisation is a single affine transform per block — fast, but limited in quality.

Q8_0 is the standout exception: at 8 bits with only a scale per block, it is essentially lossless (perplexity increase of ~0.01 points). It's still widely used and a safe fallback.

The 4-bit and 5-bit legacy types (Q4_0, Q5_0) have been largely superseded by K-quants for quality, though they decode slightly faster on some older hardware.

K-quants (introduced in llama.cpp) use a two-level block hierarchy:

This "double quantisation" gives K-quants a piece-wise affine approximation of the weight distribution, capturing both local variation (within a 32-weight block) and global variation (across the super-block). The result is dramatically better quality per bit than legacy formats.

Critically, K-quants are also mixed-precision: the _M (medium) variant, for example, quantises roughly half of the attention and feed-forward weights at 6-bit and the rest at 4-bit, resulting in an effective bit-width slightly above 4 despite the Q4 prefix.

Perplexity impact at 7B (lower Δppl = less quality loss):

Source: llama.cpp official quantisation documentation and community benchmarks.

The newest family is I-quants (IQ3_M, IQ4_XS, etc.). These use an importance matrix — a calibration step that analyses which weights matter most for a representative sample of inputs — and allocates bits non-uniformly. The most critical weights get higher precision; less important weights are compressed harder.

The result is better quality at the same average bit-width versus K-quants, at the cost of slower dequantisation and dependence on calibration quality. IQ4_XS vs Q4_K_M is a common comparison: IQ4_XS is slightly smaller (4.46 bits/weight vs 4.89) and often a bit faster for token generation, though slightly slower on prompt processing.

For NVIDIA GPU inference (not CPU/hybrid), two other formats are common:

If you're using Hugging Face transformers directly with bitsandbytes, you get:

A word on MoE models, because they're increasingly common and they behave differently.

In a dense model, every token passes through every parameter. In a Mixture-of-Experts model, the network contains many "expert" sub-networks, but each token only activates a small subset. For example, Qwen3 30B-A3B has 30B total parameters but only ~3B are active per token.

The VRAM implication: all experts must still reside in memory for fast switching, even if only a few are active at once. MoE doesn't save VRAM — it saves compute. So when planning VRAM, treat a MoE model by its total parameter count, not its active count.

The benefit shows up in inference speed: a 30B MoE that activates 3B per token generates tokens at roughly the speed of a 3B dense model, despite having quality closer to a 14B+ dense model.

The 5070 Ti is a meaningfully better AI inference card than its naming might suggest. GDDR7 memory bandwidth (~896 GB/s) is roughly 2.5× the bandwidth of the RTX 3060 (~360 GB/s), and memory bandwidth is what determines token generation speed for autoregressive LLMs — these are memory-bound workloads, not compute-bound.

6 GB is a tight constraint. At this tier, quantisation is not optional — it's survival. Q4_K_M is the default starting point, and you'll need to be careful with context length.

At 6 GB, 7–8B models at Q4_K_M are your ceiling. Context window must be kept short (4K–8K tokens) or KV cache will overflow into system RAM, causing a 5–30× performance penalty. The Qwen3 4B at Q4_K_M is arguably the best overall bet at this tier: it has both a "thinking" mode (extended chain-of-thought reasoning) and a fast "non-thinking" mode, and at ~2.8 GB it leaves ample headroom for context.

This is where things get interesting. With 16 GB you have genuine flexibility.

The sweet spot on the 5070 Ti is 14B models at Q4_K_M or Q5_K_M. At ~9–10.5 GB they leave 5–7 GB of headroom for your KV cache, which at a practical 8K–16K context for most workflows is perfectly manageable. The DeepSeek-R1 14B distill deserves special mention — it's a distilled version of the full R1 reasoning model and delivers competition-level maths and logic performance for its size.

The Qwen3 30B-A3B MoE model is borderline: at Q4_K_M it needs approximately 17–18 GB, which technically exceeds your 16 GB. However, some GGUF tools can partially offload to system RAM with minimal penalty (since only 3B parameters are active per token, the hot path is narrow). In practice, many users report it running with short-to-medium contexts on 16 GB with llama.cpp's split offloading. If you try it, limit your context to 4K–8K.

Code generation benefits more from quantisation headroom than from raw precision — it's better to run a 7B code model at Q4 than a 3B at Q8. Syntax sensitivity means Q4_K_M is the floor recommended for code; Q5_K_M is preferable when VRAM allows.

Qwen2.5-Coder 14B at Q5_K_M is the recommended primary coding model for the 5070 Ti. It consistently outperforms DeepSeek-Coder V2 on real-world benchmarks and fits with 5+ GB headroom for context. The specialised Qwen3-Coder-Next (80B MoE, 3B active) scores remarkably on SWE-Bench, but at ~49 GB at Q4 it requires a system with 64 GB unified memory or dual enterprise GPUs — well beyond either of your cards.

TTS models are an order of magnitude smaller than LLMs and behave quite differently. Instead of billions of transformer parameters, they're typically sequence-to-sequence or vocoder architectures measured in tens or hundreds of millions of parameters.

You're already familiar with this one from your SullySoftware work. Some hard numbers:

Kokoro runs comfortably on either card with trivial VRAM impact. On the 5070 Ti you could run Kokoro and a 14B LLM simultaneously if you structured it as a TTS pipeline.

For your specific setup: Kokoro, Piper, StyleTTS2, and F5-TTS all run trivially on either GPU. Fish Audio S1 requires 12 GB, making it comfortable on the 5070 Ti but not the laptop 3060. The largest TTS models (Fish S1, Bark) are the only ones that would challenge either card, and they're mostly relevant for voice cloning use cases.

Every model on Hugging Face has its parameter count either in the name or on the model card. If it's not explicit, open the config.json file and multiply num_hidden_layers × hidden_size × 4 (a rough approximation for transformer FFN layers) — or just check the model card's "Files and versions" section for the file sizes.

If you don't know the architecture specifics, use this rule of thumb from community measurements:

Add 1 GB for CUDA/framework overhead as a floor.

If the model fits with 1–2 GB to spare, you're fine. If it fits with less than 1 GB spare, you're gambling with the KV cache — reduce context length or go one quantisation level lower.

For exact numbers without manual arithmetic, the community has built excellent tools:

Can run:

Cannot run:

Best all-rounder: Qwen3 4B at Q4_K_M
Best 7B option: Qwen3 8B at Q4_K_M, context ≤ 8K
Best TTS: Kokoro-82M (trivial VRAM, near-real-time)

Can run:

Cannot run:

Best general LLM: Qwen3 14B at Q5_K_M
Best coding model: Qwen2.5-Coder 14B at Q5_K_M
Best reasoning: DeepSeek-R1 14B distill at Q4_K_M
Best TTS: Kokoro-82M for latency; Fish Audio S1 for quality/multilingual

Last updated: April 2026. Model ecosystem moves fast — use the calculators linked above for the most current estimates on newly released models.