Too Long, Didn't Embed.

I fed a 13,000-token document to a local embedding server (llama.cpp's llama-server) and got back nothing. No vector. An HTTP 200 from the server, an empty data field, and a cheerful suggestion to "increase the physical batch size." So I raised the batch size. Same result. Raised it again. Same result. The model was not slow, not broken, not out of memory. It simply could not do the thing I was asking, and nothing in the error told me that raising the batch size would never, ever help.

That was the first surprise of the weekend. It was not the last. Before it was over I had watched a different model, EmbeddingGemma, quietly hand back garbage past a limit it never announced. Both failures looked like success right up until the search results were wrong.

The short version: for most home-lab retrieval, reach for Qwen3-Embedding-0.6B, and step up to 4B only if your retrieval is genuinely hard. If you read nothing else, skip to the checklist at the end. The rest is the why.

What an embedding actually is (the 30-second version)

An embedding model turns text into a fixed-length list of numbers (a vector), positioned so similar meanings land near each other: "how do I reset my password" and "forgot my login" end up close together. That is the whole trick behind semantic search and RAG (embed your documents once, embed the query, return the nearest neighbors). The catch nobody prints on the box: how the model reads your text before it produces that vector decides how it fails on long inputs, and that splits embedding models into two families that behave nothing alike.

The split that reframes everything: encoders vs decoders

Here is the consequence first, because it is the part to remember: feed a long document to an encoder and you hit a hard wall; feed it to a long-context decoder and it keeps going. Same "embedding model" label, opposite behavior.

Non-causal encoders (BGE-M3, EmbeddingGemma, nomic-embed, anything ModernBERT-based) read the whole text at once, every token looking at every other token, so the sequence has to fit whole inside one fixed window. Causal decoders (Qwen3-Embedding, gte-Qwen2) read left to right like a chat model, so a long input splits cleanly across batches, and they inherit the long context of the base LLM they were built from.

In our testing we pushed single inputs of increasing length at each model until it refused. "Wall" below is the model's trained context ceiling; "first failure" is the test length where it actually rejected the input.

The encoders hit their wall and stop. The long-context decoders ran until we ran out of test document. The two decoders at the bottom, e5-mistral and SFR, walled almost as low as an encoder despite being decoders, which is the trap the next section unpacks.

The instinct when an encoder rejects a long input is to raise the batch size, because the error tells you to. It cannot work, and the reason is the encoder's bidirectional attention: it computes every token's meaning from every other token in the sequence at once. Token 5,000's vector depends on token 1 and token 8,000 at the same time. Split the input and you have changed what each token means, because half its context is gone. The sequence has to fit the trained window as a single unit.

One trap: being a decoder is necessary but not sufficient. e5-mistral-7b-instruct and SFR-Embedding-Mistral are decoders, but their position encoding (the mechanism a model uses to track where each token sits, more on that below) was only trained to about 4,096 tokens. Our coarse test could not even separate their wall from BGE-M3's (both were fine at 3,601 and failed at 13,820), and their published 4,096 ceiling is actually lower than some encoders. Check the real max sequence length, not just the architecture.

The EmbeddingGemma silent ceiling

EmbeddingGemma is a genuinely nice little model. It is 0.71 GB and scored a perfect 1.00 recall@1 on GPU on a small sanity check (recall@1 = was the correct document ranked first). I liked it immediately. Then I put real documents through it.

Its context clamps at 2,048 tokens, which is simply the largest position it was ever trained to represent. Past 2,048 the server returns a clean HTTP 400:

input (N tokens) is larger than the max context size (2048)

An honest error, at least. So I reached for the usual trick: RoPE scaling. RoPE is the scheme most models use to encode token position, and "scaling" tries to stretch it to a longer window than it was trained on. I launched with --rope-scaling linear --rope-scale 4 --ctx-size 8192, confirmed the flags reached the process, and the runtime still hard-capped at 2,048. As a control, short documents at ctx 8192 with the same scaling scored a clean 1.0 nDCG, so the flags were live and simply had zero effect on the served window. You cannot stretch a fixed positional table after the fact.

That is the good outcome. The bad one is what started the weekend: on a different runtime (an NPU starter path), the same model did not error past its limit. It returned vectors. They were gibberish, finite-looking numbers that encoded nothing, sailing into the index looking exactly like real data. We later confirmed on GPU that the model itself is fine, so this was a backend defect, not the model. But if you had trusted the output, you would never have known.

The general rule: a model's native context ceiling is the truth. If it is 2,048, treat 2,048 as the truth and chunk to fit. Long-context embedders earn their long context by being trained long or by inheriting a long base LLM, never by post-hoc scaling of a short encoder.

The silent failures worth memorizing

The empty-200 batch-size trap from the intro is one member of a family: failures that either throw nothing or throw something that sends you the wrong way. The two that will actually blindside you:

• Baked-in wrong pooling. Pooling is how a model's per-token vectors get combined into the single document vector, and the wrong setting silently produces wrong numbers with no error. Community causal GGUFs (gte-Qwen2-7B-instruct, e5-mistral, SFR) often ship pooling_type=none and refuse to serve until you load with --pooling last. Official Qwen3-Embedding bakes last; EmbeddingGemma bakes mean. Worse, a load-on-request path can re-derive the launch command and quietly drop a per-load --pooling, so the flag has to live in the registered recipe, not just the load command.
• Wrong-backend fallback. The model "loads" and answers, but it is quietly running on CPU (slow) or misconfigured (garbage), looking healthy the entire time.

Both share the intro's signature: a plausible-looking result that is wrong.

Bigger is not better (we measured)

I assumed more parameters meant better retrieval. Across a five-task retrieval panel, scored with nDCG@10 (did the right documents land near the top; higher is better), it is more complicated:

The 4B genuinely beats the 0.6B, separated with confidence on four of five tasks (SciFact +0.088, FiQA +0.124, ArguAna +0.033, SciDocs +0.067). But 8B over 4B was +0.009 on average with every confidence interval overlapping: 1.6x the VRAM for no reliable gain. And on an easy "short title to matching content" shape, 0.6B, 4B, and 8B were statistically indistinguishable (0.772 / 0.801 / 0.808, all overlapping). Pick by the shape of your queries, not the parameter count. If your retrieval is easy, the small model ties the big one.

A few smaller findings while you are tuning:

• Quantization barely matters for embedders. 4-bit vs 8-bit EmbeddingGemma on SciFact was a statistical tie (0.754 vs 0.765), and at short context 4-bit saved only ~0.05 GB, because the compute and KV buffers dominate the footprint, not the tiny weights.
• -ub 8192 on a decoder is wasted memory. It inflated Qwen3-Embedding-0.6B from 2.21 to 7.02 GB for zero quality gain. Decoders already split long sequences at the default micro-batch; only encoders benefit from a bigger physical batch, and even then only up to their trained ceiling.
• Matryoshka is a free index shrink. A Matryoshka model lets you chop the vector shorter to shrink your index. EmbeddingGemma truncates cleanly: 768 native scored 0.765, 512 was lossless at 0.766, and 256 held at 0.750 for one-third the index size. At 128 it broke (0.684, recall@1 fell from 0.64 to 0.55). Usable down to 256; do not go below it. Handily, the model this post recommends carries the same lever: Qwen3-Embedding natively supports Matryoshka output dimensions from 32 up to its native size, so you can shrink its index the same way (validate your own floor, as we did for EmbeddingGemma).

A short checklist to not get burned

• Before you pick an embedder, check two things: architecture (encoder or decoder) and real max sequence length. That pair predicts how it fails on long inputs.
• Run a tiny retrieval sanity check on your data: score recall@1 (is the right document ranked first?) and nDCG (are the right documents near the top?) over a ~5-doc sample with known answers. A dozen-line script is enough, and a perfect recall@1 on five hand-picked docs would have caught my gibberish vectors in thirty seconds.
• Confirm it is on the GPU, not a silent CPU fallback.
• Confirm pooling is set correctly, in the registered recipe, not just the ad-hoc load command.
• For long documents, chunk them or pick a decoder with real long context. Do not crank the batch size and hope.
• Do not reflexively grab the biggest model. Match it to your query shape, then check whether the small one already ties.

The reason these bugs eat a weekend is that embeddings fail quietly. A model that returns a plausible-looking vector for garbage input is indistinguishable from one that works, right up until your search results are subtly, unaccountably wrong. Test retrieval, not liveness. The server being up tells you nothing about whether the numbers mean anything.