A personal memory system has two very different appetites. Most of the time it is reading: notes, clippings, transcripts arrive continuously in the background and have to be classified, triaged, and folded into an index before anyone asks it anything. Occasionally it has to answer a question, which means pulling the right passages back out of everything stored so far and saying something true about them, grounded in what was actually retrieved rather than what sounds plausible. Those are two different jobs with two different bottlenecks, and this post is about picking one model to sit on the answering side of that split.
Nowledge Mem is the system in question: a personal knowledge and memory service that ingests continuously and answers questions over what it has ingested. The retrieval half of that pipeline, the embedding model that turns notes and queries into vectors, was covered in an earlier post: Too Long, Didn't Embed. This one is about the other half. Once the right passages come back from retrieval, something has to read them and produce an answer that doesn't drift from what was actually retrieved.
Mellum-2, JetBrains' second-generation model family, was built primarily for agentic software development workflows. That is where evaluation started: coding benchmarks, tool use, and agentic gating decisions. But when those early evaluation runs revealed its prefill throughput, it demanded a look as the answer-generator in Nowledge Mem. While interactive question-answering is the end goal, task classification and agentic gating serve as our discriminating quality proxies: they test strict adherence to retrieved context without hallucinating tags or making ungrounded leaps.
If you just want the numbers, you can view them here.
Mellum-2 did not win the primary generator job outright. It is, on the numbers below, a genuinely strong candidate for a narrower one.
Why Granite-4.0-H-Small holds the generator role
Granite-4.0-H-Small is a hybrid architecture: most of its layers are Mamba-2 state-space layers, with a small number of standard attention layers mixed in. That distinction matters more than it sounds like it should, because it changes what happens to memory as a transcript grows. In a normal transformer, every attention layer's KV cache grows with the sequence: twice as much context, twice as much cache. In Granite-4.0-H-Small, only the attention layers grow. The Mamba-2 layers carry a fixed-size recurrent state instead of an accumulating cache, so the bulk of the model's cost per token stays flat no matter how long the input gets. For a memory system whose whole premise is reading long, messy, real-world material, that is close to the ideal cost shape: the part of the model that scales with length is a small minority of it.
That architecture earned Granite the generator role on quality, not just on cost, though only on one of the two workloads that actually test grounded generation. Measured head-to-head against Mellum-2 with both models running at 4-bit quantization and decoding greedy at temperature 0. Granite holds a real lead on note classification: 0.767 against Mellum-2's 0.700 (correctly typing an incoming item, 60 distinct tasks scored over 180 observations, the larger of the two samples). On the second discriminating workload, an agentic gating decision (30 tasks, 90 observations), the two models tie at 0.667 on a single 32 GB card. The architecture keeps paying off as inputs grow, too: because only a handful of its layers carry a growing cache, long inputs stay cheap for it in a way they aren't for a normal transformer, and memory is rarely the thing that ends up capping how much context it can be given.
None of that makes Granite the cheap option. It draws roughly 25.5 GB of VRAM in this lane and its prefill throughput is comparatively modest. Because both models were benchmarked at Q4_K_M, that VRAM gap (25.5 GB vs ~11 GB) is a reflection of pure architecture rather than quantization tiers. Granite earns its VRAM footprint with a consistent edge on classifying what's grounded, and on a single card that is not a small thing to be good at. It remains a solid, defensible pick on its own merits.
In this post the Granite model we refer to and compare against is Granite 4.0. IBM has since released the Granite 4.1 family of models with a more flexible dense desig. While a direct replacement for Granite-4.0-H-Small does not exist, Granite-4.1-8B is our pick to look at later.
Mellum-2 is not Mellum-1
JetBrains publishes the Mellum family under Apache-2.0, and it comes in two generations that do not do the same job. Mellum-1 is a 4B dense model, Llama-style architecture, 8,192-token context, and it is genuinely what its name suggests: a fill-in-the-middle code-completion model built for IDE-inline suggestions. It was not evaluated on-card for this post; it is included here only to draw the line against its successor, because the two are easy to conflate and the conflation matters.
Mellum-2 is a different animal. It's a mixture-of-experts model, 12.15B total parameters with roughly 2.4B active per token, 64 experts with 8 active on any given token, and a native context window of 131,072 tokens. It ships as instruct/chat and reasoning ("Thinking") variants. It is not a fill-in-the-middle model. The FIM sentinel tokens do still exist in its tokenizer, inherited vocabulary, but no completion behavior was measured or claimed here. Anyone reaching for Mellum-2 expecting Mellum-1's job is reaching for the wrong generation.
The part of Mellum-2's architecture worth understanding in detail is its attention layout, because it's the direct cause of both the best and the weakest numbers below. Of its 28 layers, 21 use sliding-window attention (SWA) with a fixed 1,024-token window, and only 7 do full, unrestricted attention across the whole sequence. Context extension (a 16x YaRN scaling factor) is applied only to those 7 full-attention layers; the sliding-window layers keep their native scale regardless of how long the served context is.
Why this layout produces both the fastest prefill and the earliest retrieval decay
Sliding-window attention is fixed-cost regardless of sequence length: a 1,024-token window costs the same whether the input is 4K or 128K tokens long. With 21 of 28 layers running that way, prefill on a long input is mostly linear-cost work with a handful of quadratic-cost layers mixed in, which is why prefill throughput on this model is a multiple of a comparably-sized dense or lightly-SWA model. The same layout is why retrieval quality degrades with length in a specific pattern: a single fact anywhere in a long input can still be found by the 7 full-attention layers, so single-needle retrieval holds up well past 32K. But combining several facts scattered through a long input, tracking which key goes with which value, needs cross-referencing that the sliding-window layers structurally cannot do beyond their own 1,024-token neighborhood. The full-attention layers still have to route all of that on their own, and their capacity to do it degrades as the effective sequence they're reasoning over grows. Fast prefill and early multi-fact decay come from the same design choice.
The measurements
Everything below was measured on a single 32 GB card, ROCm backend, llama.cpp (build b10143), with the Granite comparison repeated on a Strix Halo APU on the same build. Quality, speed, and footprint numbers for the Granite comparison come from the two models loaded and scored in the same session under identical settings (both at Q4_K_M, single card each, one concurrent request slot, flash attention on, continuous batching, the same corpus, three repeats per workload, both models decoding greedy at temperature 0), not from separately-run historical figures. Every retrieval and coding number is on Mellum-2's Instruct build unless stated otherwise.
Quality vs. speed vs. footprint
| Workload | Granite-4.0-H-Small Q4_K_M | Mellum-2 Instruct Q4_K_M | Delta |
|---|---|---|---|
| Note-type classification accuracy (60 tasks, 180 observations) | 0.767 | 0.700 | -0.067 |
| Agentic gate-decision accuracy (30 tasks, 90 observations) | 0.667 | 0.667 | tied |
| Triage/coverage accuracy (20 tasks, 60 observations) | 0.250 | 0.250 | tied |
The third workload's tie is not a meaningful result: on this corpus its prompts average only about 130 tokens, so it never exercises the long-transcript case it's meant to test. The second tie is real: both models decoding under identical settings land on the exact same gate-decision accuracy. Read the first row as the one workload with a genuine quality gap on discrete hardware, and the second as a wash.
| Metric | Granite-4.0-H-Small Q4_K_M | Mellum-2 Instruct Q4_K_M |
|---|---|---|
| Prefill, 28K-token prompt | 1,505 tok/s (18.8 s) | 4,096 tok/s (7.0 s), 2.7x |
| Classification workload, median wall time | 1.85 s | 0.90 s |
| Gate-decision workload, median wall time | 5.53 s | 1.32 s |
| VRAM resident, in-lane (64K ctx) | 25.5 GB | ~11 GB |
Every speed and footprint number favors Mellum-2, by a wide margin: 2.7x the prefill throughput on an identical prompt, roughly 2x lower median latency on classification and over 4x lower on gate-decisions, and less than half the VRAM. Put the two tables together and the shape of the trade-off is narrower than it looks at first glance: Mellum-2 is the faster, cheaper option at every operational metric, giving up ground on exactly one of the two workloads that test grounded generation.
Same models, a Strix Halo APU
The same two models, same corpus, both still decoding greedy at temperature 0, same llama.cpp build, run again on a Strix Halo APU: a unified-memory chip with no discrete VRAM pool, a very different memory subsystem than the 32 GB card above.
Quality shifts a little by hardware:
| Workload | discrete: Mellum-2 | discrete: Granite | Strix: Mellum-2 | Strix: Granite |
|---|---|---|---|---|
| Type classification | 0.700 | 0.767 | 0.678 | 0.767 |
| Gate decision | 0.667 | 0.667 | 0.711 | 0.589 |
| Triage (non-discriminating) | 0.250 | 0.250 | 0.250 | 0.250 |
Granite's type-classification score is identical on both platforms, 0.767, greedy and deterministic both times. Mellum-2 moves slightly on Strix (0.700 down to 0.678), widening the note-classification accuracy gap from 0.067 to 0.089 depending on hardware.
Granite's gate-decision score exhibits an unexpected shift: 0.667 on the discrete card down to 0.589 on the APU, a move on a workload that both models run at temperature 0. Mellum-2 moves in the opposite direction (0.667 to 0.711). Whatever is behind Granite's gate-decision swing is hardware-dependent rather than a sampling artifact, since both models are greedy on both platforms. What the swing means for deployment: gating on the APU favors Mellum-2 by +0.122, flipping the read from the discrete card.
The larger, more legible effect is speed:
| Metric | discrete Mellum-2 | discrete Granite | Strix Mellum-2 | Strix Granite |
|---|---|---|---|---|
| Prefill @28K | 4,096 tok/s | 1,505 tok/s | 1,414 tok/s | 387 tok/s |
| Ratio, Mellum-2 : Granite | 2.7x | — | 3.7x | — |
| Classification wall p50 | 0.90 s | 1.85 s | 1.69 s | 4.22 s |
| Gate-decision wall p50 | 1.32 s | 5.53 s | 5.51 s | 12.44 s |
| VRAM/memory, in-lane | ~11 GB | 25.5 GB | ~11 GB | 25.5 GB |
Moving from the discrete card to the APU costs Granite 3.9x on prefill (1,505 down to 387 tok/s) and costs Mellum-2 only 2.9x (4,096 down to 1,414 tok/s). The prefill ratio between the two models widens from 2.7x to 3.7x. That runs against the usual intuition that a big-GPU model's advantage shrinks once the big GPU is taken away; here it grows. The reason is architectural: a fixed 1,024-token sliding window touches a fixed slice of the KV cache no matter how long the sequence is, which saves compute on a card where FLOPs are the constraint and saves memory traffic on unified memory where bandwidth is the constraint. Granite pays a different bill either way: its Mamba-2 layers carry a flat recurrent state, but the handful of attention layers mixed in still accumulate a growing cache, and that's the part of the model a weaker memory subsystem punishes hardest. Constrained hardware exposes exactly the axis where Mellum-2's layout was already ahead.
On this chip, the llama.cpp build matters more than the hardware. Prefill on the same Strix Halo APU, same model, prompt, and context, measured 316 tok/s on one llama.cpp build against 1,414 tok/s on another, a 4.5x swing from the build alone. On the faster build, ROCm on this APU outperforms Vulkan (1,414 vs. 1,187 tok/s). Every Strix number in this section is from the faster build (b10143); a throughput figure from an unspecified or older build on this hardware is not comparable.
Coding, and the quantization call
Because Mellum-2 was designed for agentic development, coding performance got a closer look. Mellum-2 Instruct at Q4_K_M scores 0.884 pass@1 on HumanEval+ (EvalPlus, n=164, greedy decoding, execution-verified), a competitive coding result for a ~2.4B-active model. It clears function-calling and structured-output checks cleanly at both quant levels tested.
The quantization call is straightforward: the Q8_0 build scores 0.872 on the same coding benchmark; the Q4_K_M build scores 0.884. That is noise rather than a true quality gain for the smaller quant, but it confirms there is no measured quality reason to run the larger build. Q4_K_M holds a substantial footprint advantage: roughly 9.0 GB resident against roughly 13.75 GB for Q8_0 (at 32K context). With quality a wash and VRAM meaningfully different, the 4-bit build is the sensible default.
Energy
Mellum-2 Instruct Q4_K_M measured at roughly 0.58 tokens per watt under sustained decode load, sampled per-card alongside power draw and junction temperature through the same runs. Combined with its ~11 GB VRAM footprint at 64K context, that makes it cheap to run continuously in background ingestion pipelines.
Long context: a gradual ceiling, not the spec number
Mellum-2's advertised context is 131,072 tokens. What the RULER-style needle-in-haystack sweep actually shows, run at 4K, 32K, and 128K:
| Task | 32K | 128K | Reading |
|---|---|---|---|
Single-needle retrieval (niah_single) |
1.0 | 1.0 | Holds to 128K |
Common-word retrieval (common_words) |
1.0 | 0.95 | Holds to 128K |
Multi-value retrieval (niah_multivalue) |
1.000 | 0.775 | Reliable to 32K; degrades by 128K |
Multi-key retrieval (niah_multikey) |
0.900 | 0.500 | Reliable to 32K; coin-flip at 128K |
Retrieval quality degrades gradually with length: single-fact lookup stays essentially perfect all the way to 128K, while combining and cross-referencing several facts drops from 90% reliable at 32K to a coin flip at 128K. The honest ceiling for multi-fact retrieval work on this model is roughly 32K, not the 131K figure on the spec sheet.
One further task in the same sweep, a multi-step variable-tracking exercise (variable_track), scored 0.225 at a 4K control length, 0.175 at 32K, and 0.25 at 128K: flat and low across a 32x range of input length. That is not a context-length effect. A task that fails identically whether the input is short or long measures a reasoning floor rather than context degradation.
Where this leaves Mellum-2
Mellum-2 was trained primarily for agentic software development workflows, and its execution-verified coding and tool-use capabilities are solid for its ~2.4B active parameter count. But its sliding-window attention layout makes it a surprisingly effective candidate for RAG pipelines where prefill throughput and memory footprint are the binding operational constraints.
It did not take the generator role outright from Granite-4.0-H-Small Q4_K_M. It gives up 0.067 to 0.089 on note classification depending on hardware. That is a real quality cost, and it leans in Granite's favor on both discrete GPUs and APUs. On agentic gate decisions, however, there is no cost to buy back: a dead tie on discrete hardware, and a +0.122 lead for Mellum-2 on the APU.
What that trade buys: roughly 14.5 GB less VRAM (25.5 GB down to ~11 GB), 2.7x prefill throughput on discrete GPUs widening to 3.7x on APUs, and 2x to over 4x lower per-request latency. The operational advantage doesn't shrink on constrained hardware—it grows.
Granite-4.0-H-Small Q4_K_M remains the safer default generator when pure classification quality is the primary variable and VRAM is abundant. Mellum-2 Q4_K_M is the model to reach for when memory footprint is tight, prefill latency dominates, or the hardware footprint is constrained.
