META LLAMA 3.2AT PARITY

LLAMA 3.2 3B INSTRUCT

262 tok/s marginal decode via a parallel-KV flash-decode kernel — the workhorse small model.

PARAMETERS

ARCHITECTURE

Dense

QUANT

Q4_K_M

VRAM

4 GB+

OVERVIEW

Llama 3.2 3B is the quality step up from 1B that still fits comfortably on 4 GB cards. IronWorks ships a dedicated f16 flash-decode kernel with a parallel KV walk for this shape (hidden=3072, head_dim=128).

Pure decode rate reaches 262 tok/s, at the edge of llama.cpp's 259; end-to-end short runs land slightly behind while per-kernel work continues at this hidden size.

PREREQUISITES

NVIDIA GPU with 4 GB+ VRAM (measured: RTX 3090 Ti)
~2 GB model file (Q4_K_M)

DEPLOY

Install IronWorks

terminal

# CLI (CUDA build)
cargo install ironworks-cli --features cuda

# or build from source
git clone https://github.com/nervosys/IronWorks && cd IronWorks
cargo build --release -p ironworks-cli --features cuda

Run

terminal

iwx complete -m Llama-3.2-3B-Instruct-Q4_K_M.gguf --gpu \
  -p "Write a haiku about memory bandwidth." -n 256

iwx serve -m Llama-3.2-3B-Instruct-Q4_K_M.gguf --port 8080

TUNING

Persistent KV cache

terminal

iwx chat -m Llama-3.2-3B-Instruct-Q4_K_M.gguf --gpu \
  --kv-cache-dir ~/.cache/ironworks/kv

MEASURED PERFORMANCE

METRIC	MEASURED	NOTES
Marginal decode rate	262 tok/s	llama.cpp same-session: 259 (2026-06-09)
End-to-end (tg128)	~230 tok/s	llama.cpp 268 — per-kernel gap at hidden=3072 being closed

RTX 3090 Ti, single GPU, greedy, same-session A/B vs llama.cpp CUDA build, clock-locked, 2026-06-09.

REFERENCES

Llama 3.2 3B Instruct GGUF (bartowski) ↗