META LLAMA 3.2LEADING

LLAMA 3.2 1B INSTRUCT

513 tok/s single-stream decode on one consumer GPU — ahead of llama.cpp with tighter variance.

PARAMETERS

ARCHITECTURE

Dense

QUANT

Q4_K_M

VRAM

2 GB+

OVERVIEW

Llama 3.2 1B is the fastest path to local text generation: instruction-tuned, 128K context, and small enough that decode runs at interactive-agent speeds on any modern GPU.

IronWorks runs the standard GGUF Q4_K_M export fully GPU-resident with whole-decode-loop CUDA graph capture, so per-token launch overhead is amortized to near zero.

PREREQUISITES

Any NVIDIA GPU with 2 GB+ VRAM (measured: RTX 3090 Ti)
~0.8 GB model file (Q4_K_M)
CPU-only mode also supported

DEPLOY

Install IronWorks

terminal

# CLI (CUDA build)
cargo install ironworks-cli --features cuda

# or build from source
git clone https://github.com/nervosys/IronWorks && cd IronWorks
cargo build --release -p ironworks-cli --features cuda

Run

terminal

# Single completion
iwx complete -m Llama-3.2-1B-Instruct-Q4_K_M.gguf --gpu \
  -p "Explain KV caching in two sentences." -n 256

# Interactive chat
iwx chat -m Llama-3.2-1B-Instruct-Q4_K_M.gguf --gpu

# OpenAI-compatible server
iwx serve -m Llama-3.2-1B-Instruct-Q4_K_M.gguf --port 8080

TUNING

Persistent KV cache (instant re-prompting)

Cache prefill KV to disk in 512-token chunks; reruns and shared prefixes skip prefill entirely — measured 7.3× faster time-to-first-token on repeated prompts.

terminal

iwx chat -m Llama-3.2-1B-Instruct-Q4_K_M.gguf --gpu \
  --kv-cache-dir ~/.cache/ironworks/kv

MEASURED PERFORMANCE

METRIC	MEASURED	NOTES
Decode (tg128, greedy)	513.6 ± 6.3 tok/s	llama.cpp same-session: 503.6 ± 10.6 (+2.0%)
Variance	±1.2%	tighter than llama.cpp ±2.1%

RTX 3090 Ti, single GPU, greedy tg128, same-session A/B vs llama.cpp CUDA build, 2026-05-28.

REFERENCES

Llama 3.2 1B Instruct GGUF (bartowski) ↗