Running LLMs on a Laptop: GGUF and llama.cpp

Quantization from Scratch built quantization from its parts, and Small and Specialized Models showed why it matters most at batch 1: each generated token reads every weight once, so speed is bandwidth divided by bytes. On a laptop, batch 1 is the normal case. llama.cpp (Gerganov and contributors) is the inference engine many local tools are built on, and GGUF is its file format. This post opens both and checks the bandwidth arithmetic on a real machine.

Table of contents

What is in a GGUF file

A GGUF file is everything needed to run a model, in one file: a short header, a list of key-value metadata (the architecture, its hyperparameters, and the tokenizer’s whole vocabulary and merges), a table describing each tensor (name, shape, type, offset), then the tensor data, aligned (to 32 bytes unless the metadata says otherwise, with each tensor’s offset counted from the start of the data section) so the file can be memory-mapped: the operating system maps the file into memory and loads pages as they are first touched, so a model “loads” in a moment and a second process shares the same pages. That holds on the CPU and on Apple’s unified memory, where the GPU can read the mapped pages; with a separate GPU (CUDA, Vulkan), offloaded weights are still copied into its memory. general.file_type 15 in the output is the code for Q4_K_M.

r = GGUFReader(D + "qwen2.5-1.5b-instruct-q4_k_m.gguf")
fields = r.fields
def value(key):
    f = fields[key]
    return f.parts[f.data[0]].tobytes().decode() if f.types and f.types[0].name == "STRING" else f.parts[f.data[0]][0]
kv = [k for k in fields if not k.startswith("GGUF.")]      # the reader adds GGUF.version and the counts itself
print(f"qwen2.5-1.5b-instruct-q4_k_m.gguf: {len(kv)} metadata keys, {len(r.tensors)} tensors")
for key in ("general.architecture", "qwen2.block_count", "qwen2.context_length", "qwen2.embedding_length",
            "qwen2.attention.head_count", "qwen2.attention.head_count_kv", "general.file_type"):
    print(f"  {key:32s} {value(key)}")
print(f"  {'tokenizer.ggml.tokens':32s} {len(fields['tokenizer.ggml.tokens'].data):,} entries (the vocabulary, in the file)")
by_type = collections.defaultdict(lambda: [0, 0, 0])
for t in r.tensors:
    b = by_type[t.tensor_type.name]
    b[0] += 1; b[1] += int(t.n_elements); b[2] += int(t.n_bytes)
print("tensors by type:")
for name, (n, elems, nbytes) in sorted(by_type.items(), key=lambda kv: -kv[1][2]):
    print(f"  {name:6s} {n:4d} tensors {elems / 1e6:8.1f}M weights {nbytes / 2**20:8.1f} MiB  {8 * nbytes / elems:5.2f} bits per weight")
print("examples:", ", ".join(f"{t.name} {t.tensor_type.name}" for t in r.tensors[:6]))

Full script: gguf_contents.py

The script first downloads three of the official GGUF files at a pinned revision, then lists the tensors kept in Q6_K and prints each file’s size and its average bits per weight.

qwen2.5-1.5b-instruct-q4_k_m.gguf: 26 metadata keys, 339 tensors
  general.architecture             qwen2
  qwen2.block_count                28
  qwen2.context_length             32768
  qwen2.embedding_length           1536
  qwen2.attention.head_count       12
  qwen2.attention.head_count_kv    2
  general.file_type                15
  tokenizer.ggml.tokens            151,936 entries (the vocabulary, in the file)
tensors by type:
  Q4_K    169 tensors   1345.4M weights    721.7 MiB   4.50 bits per weight
  Q6_K     29 tensors    431.6M weights    337.6 MiB   6.56 bits per weight
  F32     141 tensors      0.1M weights      0.6 MiB  32.00 bits per weight
examples: output.weight Q6_K, token_embd.weight Q4_K, blk.0.attn_norm.weight F32, blk.0.ffn_down.weight Q6_K, blk.0.ffn_gate.weight Q4_K, blk.0.ffn_up.weight Q4_K
the 29 Q6_K tensors: output.weight, 14 attn_v and 14 ffn_down (layers [0, 1, 5, 6, 7, 8, 9, 10, 13, 16, 19, 21, 24, 27])
qwen2.5-1.5b-instruct-q3_k_m.gguf     0.92 GB,  4.13 bits per weight on average
qwen2.5-1.5b-instruct-q4_k_m.gguf     1.12 GB,  5.00 bits per weight on average
qwen2.5-1.5b-instruct-q8_0.gguf       1.89 GB,  8.50 bits per weight on average

What Q4_K_M means

The type names describe how weights are packed:

• Q8_0, Q4_0: blocks of 32 weights with one 16-bit scale, at 8 or 4 bits: 8.5 and 4.5 bits per weight. Quantization from Scratch’s group scales, with a block of 32.
• K-quants (Q2_K to Q6_K): super-blocks of 256 weights divided into sub-blocks (8 of 32 for Q4_K, 16 of 16 for Q3_K and Q6_K), each sub-block with its own scale, and in some types a minimum, stored in 4 to 8 bits, and the super-block with one or two 16-bit values (a scale, and a minimum where the type has them) for those: two levels of scales, similar in spirit to QLoRA’s double quantization. Q4_K comes to 4.5 bits per weight, Q6_K to about 6.6.
• The suffix (S, M, L) is a recipe for mixing types across tensors. Q4_K_M keeps most tensors in Q4_K but puts the output layer, and the attention value and MLP down-projections of half the layers, in Q6_K, because errors there hurt most. The file above shows it: 169 tensors in Q4_K, 29 in Q6_K, and the norms and attention biases, small vectors, left in 32 bits. “4-bit” Q4_K_M averages 5.0 bits per weight, the extra coming almost all from the tensors kept in Q6_K.

Speed: bandwidth divided by bytes

Decoding one token reads every tensor once, except the token-embedding table, of which it reads one row (plus the KV cache, negligible at the short contexts used here). So tokens per second can’t exceed the memory bandwidth divided by those bytes. Measured on this laptop, an Apple M4, with the model on its GPU (Metal) and on its CPU; the speed and quality scripts together take about a minute:

BW = 93e9
prompt = "<|im_start|>user\nExplain what a KV cache is, in detail.<|im_end|>\n<|im_start|>assistant\n"
def bytes_per_token(path):
    tensors = GGUFReader(path).tensors
    emb = next(t for t in tensors if t.name == "token_embd.weight")
    return sum(int(t.n_bytes) for t in tensors) - int(emb.n_bytes) * (1 - 1 / emb.shape[-1])

print(f"{'file':36s} {'GB read/token':>13s} {'bound':>9s} {'GPU (Metal)':>12s} {'of bound':>9s} {'CPU, 8 threads':>15s}")
for f in ("qwen2.5-1.5b-instruct-q8_0.gguf", "qwen2.5-1.5b-instruct-q4_k_m.gguf", "qwen2.5-1.5b-instruct-q3_k_m.gguf"):
    size = bytes_per_token(D + f)
    speeds = {}
    for name, layers in (("gpu", -1), ("cpu", 0)):
        llm = Llama(D + f, n_gpu_layers=layers, n_ctx=1024, n_threads=8, verbose=False, seed=0, use_mlock=True)
        runs = []
        for _ in range(3):
            stamps = []
            for _ in llm(prompt, max_tokens=128, temperature=0.0, top_k=1, repeat_penalty=1.0, stream=True):
                stamps.append(time.perf_counter())
            runs.append((len(stamps) - 1) / (stamps[-1] - stamps[0]))
        speeds[name] = sorted(runs)[1]
        del llm
    print(f"{f:36s} {size / 1e9:11.2f}   {BW / size:6.0f}/s {speeds['gpu']:9.0f}/s {speeds['gpu'] / (BW / size):8.0%} "
          f"{speeds['cpu']:12.0f}/s")

Full script: speed.py

file                                 GB read/token     bound  GPU (Metal)  of bound  CPU, 8 threads
qwen2.5-1.5b-instruct-q8_0.gguf             1.64       57/s        57/s     101%           62/s
qwen2.5-1.5b-instruct-q4_k_m.gguf           0.98       95/s        86/s      90%           90/s
qwen2.5-1.5b-instruct-q3_k_m.gguf           0.82      114/s        73/s      64%           85/s

The bound uses How a GPU Runs Deep Learning’s measured 93 GB/s, from copy kernels that read and write; decoding mostly reads, and Apple specifies this chip’s memory at 120 GB/s (as How a GPU Runs Deep Learning notes). So the bound here is the measured copy bandwidth, not the hardware’s ceiling, and the fastest runs cross it: Q8_0 on the GPU decoded at 101% of it, about 94 GB/s, and on the CPU at 62 tokens a second, about 102 GB/s. For Q8_0, speed is bandwidth divided by bytes.

Smaller files didn’t speed up in proportion. Q4_K_M reads 40% fewer bytes than Q8_0 and ran 51% faster on the GPU, 90% of its bound. Q3_K_M reads 16% fewer bytes again and ran slower, 73 tokens a second against 86, 64% of its bound; the CPU fell short on it too, 85 against the 124 its bytes would allow at 102 GB/s. The most likely cause is the cost of the Q3_K kernels themselves: 3-bit weights split across bytes, with packed 6-bit scales, take more work to unpack than Q4_K or Q8_0, and Q3_K_M also mixes four quantization types across its tensors (Q3_K, Q4_K, Q5_K and Q6_K). This run didn’t isolate the cause; llama.cpp’s test-backend-ops perf -o MUL_MAT times each matrix type separately and would. The bandwidth bound is a ceiling, reached only when dequantizing is cheap.

The CPU, with 8 threads, matched or beat the GPU for all three files (62, 90 and 85 tokens a second against 57, 86 and 73), and the 4-bit gap is within noise for a median of three runs. Both read the same unified memory; that the CPU got more out of it here, about 102 GB/s against 94, likely reflects the GPU’s fixed cost per token for launching and synchronizing its kernels, which matters at batch 1 for a small model. Treat these numbers as one laptop’s: they’re timed through the Python bindings, which add a little per-token overhead, with three runs each and no attempt to control temperature.

Quality: what each level costs

The quality script, quality.py, imports clean from corpus.py, the corpus code of Sequences Before Attention, which strips a post’s frontmatter and components. It reads two posts from content/posts/, this blog’s own source tree at commit ed51ca8, which the folder holds as a snapshot:

POSTS = "content/posts/"
texts = [clean(open(POSTS + f + ".mdx").read()) for f in ("kv-cache-explained", "bpe-tokenizer-explained")]
emb = next(t for t in GGUFReader(D + "qwen2.5-1.5b-instruct-q8_0.gguf").tensors if t.name == "token_embd.weight")
print("token_embd shape in the file:", list(emb.shape))
for f in ("qwen2.5-1.5b-instruct-q8_0.gguf", "qwen2.5-1.5b-instruct-q4_k_m.gguf", "qwen2.5-1.5b-instruct-q3_k_m.gguf"):
    llm = Llama(D + f, n_gpu_layers=-1, n_ctx=1024, logits_all=True, verbose=False)
    total, n = 0.0, 0
    for text in texts:
        seq = llm.tokenize(text.encode(), add_bos=False)[:1024]
        llm.reset(); llm.eval(seq)
        logits = np.array(llm.scores[:len(seq)])
        logp = logits - np.log(np.exp(logits - logits.max(-1, keepdims=True)).sum(-1, keepdims=True)) - logits.max(-1, keepdims=True)
        for i in range(512, len(seq)):
            total -= logp[i - 1, seq[i]]; n += 1
    print(f"{f:36s} {total / n / math.log(2):.3f} bits per token")
    del llm

Full script: quality.py

token_embd shape in the file: [np.uint64(1536), np.uint64(151936)]
qwen2.5-1.5b-instruct-q8_0.gguf      3.061 bits per token
qwen2.5-1.5b-instruct-q4_k_m.gguf    3.114 bits per token
qwen2.5-1.5b-instruct-q3_k_m.gguf    3.185 bits per token

Bits per token on two of this blog’s posts, written after the model was trained (the opening 1,024 tokens of each, the second half scored), computed from llama.cpp’s own logits: lower is better, and each extra bit per token means the model finds the text twice as surprising. Going from 8.5 to 5.0 bits per weight cost 0.053 bits per token, about 1.7% more; going to 4.1 bits per weight cost 0.124, about 4%. In perplexity terms, 8.3 becomes 8.7 and then 9.1.

That’s 1,024 scored tokens, enough to rank the files but not to say how a task’s accuracy would change. llama.cpp’s own measure for this, the KL divergence of each file’s next-token distribution from the full-precision model’s over a long text, is the more sensitive test; Quantization from Scratch measured quantization error layer by layer, and a task needs its own evaluation (Evaluating LLMs). Read with the speed table, Q4_K_M is the usual choice for good reason: most of Q8_0’s quality at little more than half its size, and on this machine the fastest of the three on the GPU.

Where the work runs

llama.cpp runs on almost anything because its kernels are written for each backend: NEON and AVX vector instructions on CPUs, Metal on Apple GPUs, CUDA and Vulkan elsewhere. n_gpu_layers sets how many layers run on the GPU (-1 in the Python bindings, or any number at least the layer count, for all of them, with the output layer counted as one more); the rest stay on the CPU, and the token-embedding table stays there too, since a token reads only one row of it. That’s how a model larger than GPU memory still runs, at the CPU’s speed for its share. On a laptop with unified memory, CPU and GPU read the same memory, so the CPU is a real alternative for a small model rather than a fallback, as the speed table showed.

Where this leads

• Evaluating LLMs turns to evaluation: measuring whether a model, local or not, is good enough for a task.
• Where Models Run comes back to choosing model sizes and placements.

Takeaways

• A GGUF file holds everything needed to run a model: metadata, the tokenizer’s vocabulary and merges, and aligned tensors that can be memory-mapped, so on the CPU and on unified memory, loading is mapping, not copying.
• Q8_0 and Q4_0 are blocks of 32 weights with one scale; K-quants add a second level of scales over super-blocks of 256; the S/M/L suffix mixes types across tensors. Q4_K_M averages 5.0 bits per weight here.
• At batch 1, decoding speed is bounded by memory bandwidth divided by the bytes read per token. Q8_0 matched the measured copy bandwidth; Q4_K_M came close; Q3_K_M fell well short, and was slower than Q4_K_M, most likely because its kernels cost more to unpack.
• On a unified-memory laptop the CPU matched or beat the GPU for a 1.5B model at batch 1.
• Quality fell gently: 3.06, 3.11 and 3.19 bits per token for Q8_0, Q4_K_M and Q3_K_M on two of this blog’s posts.

References

Start with the GGUF specification for the file format, then llama.cpp’s repository for the quantization types and its perplexity and KL-divergence tools.

• Gerganov and contributors, llama.cpp: the engine, its quantization types, and its perplexity and KL-divergence tools.
• GGUF: the file format specification in the ggml repository, for the header, metadata and alignment.
• Qwen Team, Qwen2.5-1.5B-Instruct-GGUF: the files measured here.