NVIDIA NEMOTRONLEADING

NEMOTRON-H 8B

85 tok/s native GPU Mamba-2 decode — hybrid SSM without CPU fallbacks.

PARAMETERS

ARCHITECTURE

Hybrid SSM

QUANT

Q4_K_M

VRAM

8 GB+

OVERVIEW

Nemotron-H interleaves Mamba-2 state-space layers with attention, cutting KV-cache size and keeping decode cost flat in context length. IronWorks executes the full Mamba-2 recurrence on GPU (warp-level scan, stateful across tokens) inside the captured decode graph.

PREREQUISITES

8 GB+ NVIDIA GPU (measured: RTX 3090 Ti)
~5 GB model file (Q4_K_M)

DEPLOY

Install IronWorks

terminal

# CLI (CUDA build)
cargo install ironworks-cli --features cuda

# or build from source
git clone https://github.com/nervosys/IronWorks && cd IronWorks
cargo build --release -p ironworks-cli --features cuda

Run

terminal

iwx complete -m Nemotron-H-8B-Q4_K_M.gguf --gpu \
  -p "Explain state-space models simply." -n 512

MEASURED PERFORMANCE

METRIC	MEASURED	NOTES
Decode	85 tok/s	native GPU Mamba-2 path; no same-session llama.cpp head-to-head published yet

RTX 3090 Ti, single GPU, greedy, 2026-05-28.