MISTRALAT PARITY

MISTRAL 7B V0.3

143 tok/s decode at parity with llama.cpp — the stable mid-size baseline.

PARAMETERS

ARCHITECTURE

Dense

QUANT

Q4_K_M

VRAM

6 GB+

OVERVIEW

Mistral 7B remains a widely deployed dense baseline. IronWorks runs it fully GPU-resident with graph-captured decode at practical parity with llama.cpp (−3%, within run variance on consumer hardware).

PREREQUISITES

NVIDIA GPU with 6 GB+ VRAM (measured: RTX 3090 Ti)
~4.4 GB model file (Q4_K_M)

DEPLOY

Install IronWorks

terminal

# CLI (CUDA build)
cargo install ironworks-cli --features cuda

# or build from source
git clone https://github.com/nervosys/IronWorks && cd IronWorks
cargo build --release -p ironworks-cli --features cuda

Run

terminal

iwx complete -m Mistral-7B-Instruct-v0.3-Q4_K_M.gguf --gpu \
  -p "What is Rust's ownership model?" -n 256

iwx serve -m Mistral-7B-Instruct-v0.3-Q4_K_M.gguf --port 8080

TUNING

Persistent KV cache

terminal

iwx serve -m Mistral-7B-Instruct-v0.3-Q4_K_M.gguf --port 8080 \
  --kv-cache-dir ~/.cache/ironworks/kv

MEASURED PERFORMANCE

METRIC	MEASURED	NOTES
Decode (tg128)	143.6 ± 0.4 tok/s	llama.cpp same-session: 148.2 ± 2.3 (−3.1%)

RTX 3090 Ti, single GPU, greedy tg128, same-session A/B vs llama.cpp CUDA build, 2026-05-28.

REFERENCES

Mistral 7B Instruct v0.3 GGUF ↗