← ALL RECIPES
GOOGLE GEMMA 4OPTIMIZING

GEMMA 4 31B

Dense 31B on one 24 GB card — with lossless multi-token-prediction self-speculation (+18%).

PARAMETERS

31B

ARCHITECTURE

Dense

QUANT

Q4_K_M / Q6_K mix

VRAM

24 GB (or any budget via unified memory)

01

OVERVIEW

Gemma 4 31B is a frontier-quality dense model that just fits a 24 GB consumer card at 4-bit. IronWorks runs it graph-captured at 26.7 tok/s and adds two capabilities llama.cpp does not have in this form:

MTP self-speculation: the Gemma 4 assistant (MTP) head drafts multiple tokens per verify pass. Greedy output is byte-identical to plain decode and measured +18% faster; with temperature sampling the verifier uses exact speculative-sampling acceptance (Leviathan/Chen) for +8% on general content — losslessly, in both regimes.

Unified memory: --vram-budget runs the model token-exact at ANY VRAM budget by streaming cold layers from pinned host memory — placement changes speed, never output.

02

PREREQUISITES

  • 24 GB GPU for full residency (RTX 3090/4090 class; measured: RTX 3090 Ti)
  • Smaller cards work via --vram-budget (token-exact, bandwidth-limited)
  • ~19 GB model file (Q4_K_M)
03

DEPLOY

Install IronWorks

terminal
# CLI (CUDA build)
cargo install ironworks-cli --features cuda

# or build from source
git clone https://github.com/nervosys/IronWorks && cd IronWorks
cargo build --release -p ironworks-cli --features cuda

Run (full residency, 24 GB)

terminal
iwx complete -m gemma-4-31b-it-Q4_K_M.gguf --gpu \
  -p "Prove there are infinitely many primes." -n 512

MTP self-speculative decode (+18%, lossless)

Point --mtp-model at the Gemma 4 assistant-head GGUF. temperature 0 → greedy verify (output == plain greedy); temperature >0 → sampling verify (output ~ target sampling).

terminal
iwx complete -m gemma-4-31b-it-Q4_K_M.gguf --gpu \
  --mtp-model gemma-4-31b-it-assistant-Q4_0.gguf \
  --speculative-tokens 4 \
  -p "Write an essay on GPU memory hierarchies." -n 1024
04

TUNING

Run on a 12–16 GB card (unified memory)

Caps VRAM for the large per-layer matrices; over-budget layers stream from pinned host memory each token. Output is byte-identical at any budget.

terminal
iwx complete -m gemma-4-31b-it-Q4_K_M.gguf --gpu \
  --vram-budget 12G \
  -p "Hello" -n 256

Persistent KV cache for long system prompts

terminal
iwx serve -m gemma-4-31b-it-Q4_K_M.gguf --port 8080 \
  --kv-cache-dir ~/.cache/ironworks/kv
05

MEASURED PERFORMANCE

METRICMEASUREDNOTES
Decode (graph, tg128)26.66 tok/seager 24.51; llama.cpp reference ~39 — structural fusion work in progress
MTP self-spec (greedy)28.43 tok/s (+18%)byte-identical output, accept 0.986; IronWorks-only mode
MTP self-spec (sampling)+8% on general contentexact speculative sampling, accept 0.547
12 GB budget1.86 tok/s, deterministic39 layers resident + 21 streamed; token-exact

RTX 3090 Ti, single GPU, greedy, clock-locked, same-session A/B/A for MTP, 2026-06-13.

Keep benchmark contexts ≤512 on 24 GB cards at full residency — KV growth past ~24 GB triggers Windows WDDM demotion on any engine.

REFERENCES