Mellum2-12B-A2.5B-Instruct-Q4_K_M
The practical deployment quant of JetBrains' Mellum-2 coding MoE, on one RDNA4 card — execution-verified coding quality, tool-calling, per-task long-context retrieval, and the energy/thermal cost of running it.
Execution-verified HumanEval+ pass@1 0.8841 at ~11 GB VRAM, with 2.7x Granite-4.0-H-Small's prefill on a discrete card and 3.7x on Strix Halo. Not a quality upgrade — W8 type_accuracy regresses -0.067 against Granite on both platforms tested, under matched greedy sampling. Adopt where footprint and per-request latency are the binding constraints. Usable multi-key retrieval context is ≈32K, not the published 131K.
Model card
- Architecture
- MellumForCausalLM — 64-expert MoE, top-8 routing, SWA-1024 on 21/28 layers + layer-selective YaRN x16 on the remaining 7 full-attention layers
- Parameters
- 12.15B total / ~2.4B active
- Base model
- Mellum-2 (JetBrains)
- Quant(s)
- Q4_K_M (this page) · Q8_0 (near-lossless anchor, separate arm)
- Native context
- 131,072
- License
- Apache-2.0
- Provenance
- Official JetBrains GGUF release
Configuration
exact serving config — reproducible- Engine
- llama.cpp b10143 (rocm-stable)
- Placement
- single pinned card, --llamacpp rocm --llamacpp-device ROCmN
- Context
- 32,768 (quality/energy runs) · 131,072 (RULER-only alias)
- KV cache
- f16 (not quantized, to avoid confounding the weight-quant comparison)
- Parallel
- --parallel 1
- Sampling
- greedy / temperature 0.0 for all scored metrics
- Flash attention
- --flash-attn on
- Rope / YaRN
- left at GGUF defaults — never overridden (SWA layers hardcode freq_scale=1.0; overriding corrupts long-context behavior)
Hardware / environment
public transparency block- GPU
- Radeon AI PRO R9700
- Arch
- gfx1201 / RDNA4
- Cards
- 4x single-GPU (eval pinned to 1 at a time)
- Topology
- PCIe Gen4, no XGMI
- ROCm
- ROCm (rocm-stable build)
- OS
- Fedora 44
- Kernel
- 7.1.5-201.fc44
- Engine
- llama.cpp
- Build
- b10143
- Quant
- Q4_K_M
Evaluation results
measured on-card · single arm: Mellum2-12B-A2.5B-Instruct-Q4_K_MHeadline
HumanEval+ in context — not a ranking, budgets differ
Three separate arms from the same campaign, shown side by side for reference only. The Thinking arm ran at a 6.7x larger generation budget with reasoning engaged on every problem — this is not a matched-compute comparison, and a higher score there does not mean “better”, only “different budget, different construct”.
| Arm | pass@1 | passed | gen budget (tok) | finish_reason |
|---|---|---|---|---|
| Instruct Q4_K_M (this page) | 0.8841 | 145/164 | 12288 | stop 164/164 |
| Instruct Q8_0 (reference) | 0.872 | 143/164 | 12288 | stop 164/164 |
| Thinking Q4_K_M (reference, 6.7x budget) | 0.9024 | 148/164 | 81920 | stop 164/164 |
Where this arm sits against the fleet
Same execution-verified HumanEval+ construct (n=164, temp 0.0) across every row that has a score; prefill/RULER/VRAM columns are populated only where measured on this cluster. Mellum2-Thinking-Q4's 0.9024 ran at a 6.7x generation budget (81,920 vs 12,288 tokens) relative to every other row here — a different construct, not a matched-compute ranking.
| Model | HumanEval+ pass@1 | Prefill tok/s @28K | RULER niah_multivalue@32K | VRAM |
|---|---|---|---|---|
| Gemma-4-26B-A4B-qat | 0.933 | 1135 | 1 | ~14 GB |
| gpt-oss-20b | 0.915 | — | — | — |
| Qwen3.6-35B-A3B | 0.909 | — | — | — |
| Mellum2-Thinking-Q4 (6.7x budget) | 0.9024 | — | — | ~11 GB |
| Ornith-1.0-9B-MTP | 0.896 | — | — | ~5.4 GB |
| Mellum2-Instruct-Q4 (this page) | 0.8841 | 4096 | 1 | ~11 GB |
| Mellum2-Instruct-Q8 | 0.872 | — | — | ~15 GB |
| Granite-4.0-H-Small | 0.86 | 1505 | 0.85 | 25.5 GB |
RULER long-context — per task, never the aggregate
The 4K column is a control, not a low-context throwaway: variable_track scores 0.225 at 4K vs 0.175 at 32K vs 0.25 at 128K — flat across a 32x context range. That is a capability the model lacks at any length, so it is excluded from the long-context reading below rather than misread as a 32K collapse. The RULER aggregate is deliberately not shown anywhere on this page — it is dragged down by that flat task.
| Task | 4K (control) | 32K | 128K | reading |
|---|---|---|---|---|
| niah_single | — | 1 | 1 | reliable to 128K |
| niah_multikey | 1 | 0.9 | 0.5 | reliable to 32K; coin-flip at 128K |
| niah_multivalue | — | 1 | 0.775 | reliable to 32K; degrades at 128K |
| variable_track | 0.225 | 0.175 | 0.25 | flat at every length — capability limit, not a context effect |
| common_words | — | 1 | 0.95 | reliable to 128K |
Reading the retrieval curve
Excluding variable_track, retrieval degrades gradually with context length and shows no cliff — consistent with this project's prior long-context findings on other architectures. Retrieval ceiling across the four retrieval-shaped tasks is ≈0.975 at 32K. Usable multi-key retrieval context is ≈32K; single-needle retrieval (niah_single) holds to 128K. JetBrains' published 131K context figure is a spec/training number, not a measured per-task reliability ceiling — the two are different claims, and RULER was run on this Q4_K_M Instruct arm only (neither the Q8_0 nor Thinking arm has long-context data).
RAG head-to-head vs Granite-4.0-H-Small — quality, matched sampling, two platforms
Both models greedy (temp 0.0 / top_p 1.0 / top_k 0), ctx 32768, --parallel 1, identical corpus, repeats=3, identical task sample verified by task_id. Measured on a discrete R9700 card and a memory-constrained Strix Halo (gfx1151) host.
| Workload | discrete Mellum2-Instruct-Q4 | discrete Granite | Δ | Strix Mellum2-Instruct-Q4 | Strix Granite | Δ |
|---|---|---|---|---|---|---|
| W8 type_accuracy (n=60) | 0.7 | 0.767 | -0.067 | 0.678 | 0.767 | -0.089 |
| W6 gate_accuracy (n=30) | 0.667 | 0.667 | 0.000 | 0.711 | 0.589 | +0.122 |
| W7 triage_accuracy (n=20) | 0.25 | 0.25 | 0.000 | 0.25 | 0.25 | 0.000 |
Reading: Granite leads type classification; gating is a wash or favors Mellum-2 depending on hardware
Granite holds a consistent edge on W8 type_accuracy (the larger-n metric, n=180) on both platforms tested. W6 gate_accuracy is a dead tie on the discrete card and a Mellum-2 win (+0.122) on Strix Halo. W7 ties everywhere and does not discriminate at this corpus's ~130-token mean prompt length. Anomaly, not explained: Granite is greedy and deterministic, yet its W6 moved 0.667 (discrete) to 0.589 (Strix) while its W8 stayed pinned at 0.767 on both platforms — a hardware-dependent difference on one workload but not another.
Same head-to-head — latency and footprint, both platforms
Sampling-independent metrics. The ingestion-speed ratio grows on the more constrained platform: 2.7x on discrete, 3.7x on Strix Halo.
| Metric | discrete Mellum2-Instruct-Q4 | discrete Granite | Strix Mellum2-Instruct-Q4 | Strix Granite |
|---|---|---|---|---|
| Prefill @ 28K | 4,096 tok/s | 1,505 tok/s | 1,414 tok/s | 387 tok/s |
| Prefill ratio vs Granite | 2.7x | — | 3.7x | — |
| W8 wall p50 | 0.90 s | 1.85 s | 1.69 s | 4.22 s |
| W6 wall p50 | 1.32 s | 5.53 s | 5.51 s | 12.44 s |
| VRAM in-lane | ~11 GB | 25.5 GB | ~11 GB | 25.5 GB |
Strix Halo build caveat
Strix figures above are on llama.cpp build b10143 (ROCm backend). Under identical conditions on the same host, build b9752 (ROCm) measured only 316 tok/s prefill, and build b9747 (Vulkan) measured 1,187 tok/s — a 4.5x swing from the build alone. Any Strix throughput figure should be read together with its build; figures here are consistently b10143. Cause of the older build's slowness is not established.
The deployment fit: memory pressure as the binding constraint
For a RAG lane where VRAM (or a smaller host) is the binding constraint, this arm trades -0.067 W8 type_accuracy and 0.000 W6 gate_accuracy (discrete, matched sampling) for roughly 14.5 GB of VRAM (25.5 GB to ~11 GB), 2.7x prefill throughput, and 2-3x lower per-request latency — a trade that widens further on constrained hardware (3.7x prefill on Strix Halo, gate_accuracy flips in Mellum-2's favor). This is not a quality upgrade: the type_accuracy regression is real and holds on both platforms tested. It is a fit for lanes where footprint and latency are the deciding constraints.
Energy efficiency in context — tokens per watt
The Thinking arm's number is shown for reference only (different arm, not this page's own result): thinking costs roughly 20% more energy per token than either Instruct quant, and spent 27.5% of its busy time at/near the power cap vs 0-4% for the Instruct arms.
What this page does not establish
No LiveCodeBench was run for this arm — HumanEval+ is the only execution-verified coding benchmark here, and it measures isolated-function correctness, not agentic or multi-file coding. No FIM (fill-in-the-middle) completion quality was evaluated; the FIM sentinel tokens were confirmed present in the tokenizer, which is a tokenizer fact, not a quality measurement. BFCL and structured-output are both ceiling-saturated across this project's fleet and are reported as a pass/fail gate, not a quality ranking. A streaming-mode pass@1 was also measured (0.878) purely as a token-cost characterization exercise and diverged from the non-streaming score on 6/20 problems at temperature 0 — it is not treated as an official score anywhere; the 0.8841 above (non-streaming) is the score of record.
Energy & Thermal
windowed to the valid HumanEval+ run · busy-only (gpu_use_pct >= 10) · power cap 300WTime spent at/above 98% of the 300W power cap (busy samples only)
Each arm ran on a different physical card this session — a partial confound, noted rather than eliminated. Shown for context; not this page's own result except the middle row.
Decode tok/s (145.74, not shown as a tile here) is a session-cumulative counter, an approximation; power and junction ARE windowed to the scored run.
Provenance & caveats
empirical — tagged, n-counted, honestProvenance tags
Caveats
- One HumanEval+ item (HumanEval_32) is unscoreable in this evaluation setup for every arm — not an EvalPlus dataset defect — capping the achievable pass@1 ceiling at 163/164 rather than 164/164, symmetrically across arms.
- RULER long-context data exists for this arm only — the Q8_0 and Thinking arms of this campaign were not run through RULER, so no quant-vs-long-context or thinking-vs-long-context comparison is possible from this data.
- The RULER aggregate is never quoted — it is dragged down by
variable_track, a task the model fails at every length tested including the 4K control. - Perplexity was used only as a load/logits-path tripwire (it passed) and is explicitly not used to rank quants in this campaign — the PPL ladder across Mellum-2 quants is non-monotonic (BF16 scored worse than Q8_0 with fully-verified GPU residency on both).
- JetBrains' own published KLD-based quant-fidelity table predicts Q4_K_M should degrade markedly vs higher-precision quants. This campaign's execution-verified pass@1 A/B does not show that gap — a different construct (argmax task outcome vs full-distribution divergence), not a refutation of JetBrains' own
primary-source, self-reported figures. - The ROCm-ordinal-to-physical-card mapping on this host is a per-session permutation, not a fixed identity — verified independently each session this campaign, not assumed stable across runs.
- The RAG head-to-head against Granite-4.0-H-Small used greedy decoding (temp 0.0 / top_p 1.0 / top_k 0) for both models on both platforms — that condition applies to every number in that section and its Strix Halo counterpart.
- Strix Halo (gfx1151) throughput figures are specific to llama.cpp build b10143 on ROCm; a different build measured on the same host varied by up to 4.5x on prefill under identical conditions, so any Strix number must be read together with its build.