Tag: local-inference

All the articles with the tag "local-inference".

• Running LLMs on a Laptop: GGUF and llama.cpp

  Updated:16 Oct, 2026

  Local inference is the quantization and memory-bandwidth story at consumer scale: GGUF packages a quantized model in one memory-mappable file, and llama.cpp turns bytes per second into tokens per second. Inside a real GGUF file, what K-quant names like Q4_K_M mean, and tokens per second measured on this laptop's GPU and CPU against the bandwidth prediction, with what each quantization level costs in quality.