Bananas beat Paris.

EmbeddingGemma on the NPU, through FastFlowLM, scores "What is the capital of France?" at 0.874 against "Bananas are a yellow tropical fruit." and 0.847 against "Paris is the capital city of France." The server returned HTTP 200 and a unit-length vector. Nothing in the response says it is wrong.

That collapsed NPU output is the failure the last post was about, and I got caught by a second one while writing it. Tom Aarsen, who maintains Sentence Transformers, left a comment: use EmbeddingGemma through anything other than Sentence Transformers and there is a chance you are skipping some Dense layers. He was right. It applied to the llama.cpp GGUF I had used as my control.

Check

A minute against the reference

Embed the same eight texts through the server and through Sentence Transformers on CPU, normalise both, and read the cosine per text. The same model gives 0.99 or better. Quantisation shows up as a few hundredths; a Q4_0 scored 0.05 below the Q8_0 of the same conversion. Near 0.0 is a different vector space.

# /// script
# requires-python = ">=3.10"
# dependencies = ["sentence-transformers>=3", "httpx", "numpy"]
# ///
import sys, numpy as np, httpx
from sentence_transformers import SentenceTransformer

argv = sys.argv[1:]
key = argv.pop(argv.index("--key") + 1) if "--key" in argv else None
args = [a for a in argv if not a.startswith("--")]
url, served = args[0].rstrip("/"), args[1]
ref_id = args[2] if len(args) > 2 else "unsloth/embeddinggemma-300m"  # ungated mirror of google/embeddinggemma-300m

TEXTS = ["What is the capital of France?", "Paris is the capital city of France.",
         "Bananas are a yellow tropical fruit.", "how do I reset my password",
         "The mitochondria produces ATP for the cell.", "asyncio runs coroutines on an event loop",
         "Mount Everest is the highest mountain above sea level.", "forgot my login"]

hdr = {"Authorization": f"Bearer {key}"} if key else {}
r = httpx.post(f"{url}/v1/embeddings", headers=hdr, json={"model": served, "input": TEXTS}, timeout=600)
r.raise_for_status()
got = np.array([d["embedding"] for d in sorted(r.json()["data"], key=lambda d: d["index"])], dtype=np.float32)
got /= np.linalg.norm(got, axis=1, keepdims=True)

ref = SentenceTransformer(ref_id, device="cpu")
variants = {"full pipeline": ref}
dense_free = [m for m in ref if type(m).__name__ != "Dense"]
if len(dense_free) < len(ref):
    variants["Dense layers skipped"] = SentenceTransformer(modules=dense_free, device="cpu")
prefixes = {"no prefix": ""} | {f"prefix '{v}'": v for k, v in (getattr(ref, "prompts", {}) or {}).items() if k in ("query", "document")}

print(f"served: {served} @ {url}   reference: {ref_id}   dims: {got.shape[1]}")
best = (0.0, "")
for vname, model in variants.items():
    for pname, p in prefixes.items():
        e = model.encode([p + t for t in TEXTS])[:, :got.shape[1]]  # Matryoshka: match the served width
        e /= np.linalg.norm(e, axis=1, keepdims=True)
        c = (got * e).sum(1)
        print(f"  {vname:22s} {pname:45s} cos mean {c.mean():+.4f}  min {c.min():+.4f}")
        best = max(best, (float(c.mean()), f"{vname}, {pname}"))
q, ok, bad = got[0], got[1], got[2]
print(f"  sanity: cos(capital-of-France?, Paris) = {q@ok:.3f}   cos(capital-of-France?, Bananas) = {q@bad:.3f}")
print("verdict:", "MATCHES reference" if best[0] >= 0.99 and "full" in best[1] else
      f"CLOSE to reference ({best[0]:.3f}), check quantization or prefix" if best[0] >= 0.85 and "full" in best[1] else
      f"WRONG SPACE: best match is '{best[1]}' at {best[0]:.3f}" if best[0] >= 0.85 else
      f"GARBAGE: matches nothing (best {best[0]:.3f})")

It also runs the reference with its Dense modules removed and with each prompt prefix the model card declares, so the output says which variant the backend matches, and whether the server prepends a prompt. llama-server does not, so "no prefix" wins and your client has to add the query and document prefixes itself. Against the two backends from the last post, both served through lemonade:

$ uv run embed_check.py http://127.0.0.1:13305 embeddinggemma-300m-GGUF-Q8_0 --key $KEY
served: embeddinggemma-300m-GGUF-Q8_0 @ http://127.0.0.1:13305   reference: unsloth/embeddinggemma-300m   dims: 768
  full pipeline          no prefix                                     cos mean +0.0136  min -0.0423
  Dense layers skipped   no prefix                                     cos mean +0.9838  min +0.9812
  sanity: cos(capital-of-France?, Paris) = 0.931   cos(capital-of-France?, Bananas) = 0.489
verdict: WRONG SPACE: best match is 'Dense layers skipped, no prefix' at 0.984

$ uv run embed_check.py http://127.0.0.1:13305 embed-gemma-300m-FLM --key $KEY
served: embed-gemma-300m-FLM @ http://127.0.0.1:13305   reference: unsloth/embeddinggemma-300m   dims: 768
  full pipeline          no prefix                                     cos mean +0.0041  min -0.0240
  Dense layers skipped   no prefix                                     cos mean +0.0088  min -0.0385
  sanity: cos(capital-of-France?, Paris) = 0.847   cos(capital-of-France?, Bananas) = 0.874
verdict: GARBAGE: matches nothing (best 0.009)

GGUF

The file that skipped two layers

EmbeddingGemma's pipeline is the transformer, mean pooling, a Dense layer from 768 to 3072, a Dense layer back to 768, then normalisation. llama.cpp learned to carry the two Dense layers in PR 16367 on 9 October 2025, behind a converter flag. The three pre-October GGUFs I checked have 314 tensors and no Dense head: the unsloth files from September 2025, which I had registered in lemonade, and ggml-org's own September QAT file. The ggml-org main repo was updated on 29 April 2026 to 316 tensors, and the same llama-server binary then agrees with Sentence Transformers at cosine 0.9997.

The fix is the newer file, or a fresh conversion with --sentence-transformers-dense-modules and the 2_Dense and 3_Dense directories present in the checkout. The llama-server that lemonade 11.9.0 bundles loads it; I have not tried what an older build does with the two extra tensors. Re-embed anything indexed with the old file. What skipping the layers costs surprised me:

At full width the two layers change nothing on this task. What the old file costs you there is compatibility: nothing built on the reference model, published numbers included, applies to its vectors. The drop from 768 to 128 dimensions is 0.03 with the Dense head and 0.12 without it, most likely because the Matryoshka loss is applied after the Dense layers, so nothing ever trained the pre-Dense vector to keep its meaning in the first dimensions. My published "do not go below 256" floor was measuring the missing layers, and on SciFact 128 dimensions with the right file beats 256 without.

NPU

FastFlowLM matches nothing

The FastFlowLM path is a different problem. Its model package carries both Dense weights, and its repository ships a reference vector in src/test/gemma_embedding/test.cpp for a paragraph of Alice in Wonderland. That vector matches the full Sentence Transformers pipeline at 0.9977. The live server here returns 0.003 against it for the same text.

I swept the reference model for anything the engine could plausibly be returning: token embeddings alone, the hidden state after each of the 24 layers, the final state with and without the Dense head, first-token and last-token pooling, all of it with and without the task prefix. Best cosine to the served output anywhere in that sweep: 0.12. v1.0.4 ships the same engine library and kernels as v1.0.3, the HRX runtime gives the same result as XRT, and a Qwen3 LLM in the same process answers correctly on the same NPU. The measurements are in the issue.

FastFlowLM issue 661 described the same collapse in August with llama.cpp as the control and has confirmations from three Linux machines on versions back to 0.9.45. My comment there adds the reference-vector result, the sweep, and the v1.0.4 and HRX runs. As of 2 September no maintainer had replied.

Verdict

What to run

EmbeddingGemma through llama.cpp on the GPU with the April 2026 ggml-org file. Bare, that is llama-server -hf ggml-org/embeddinggemma-300M-GGUF:Q8_0 --embeddings; the file carries mean pooling in its metadata. In lemonade it is one pull, and the check then passes through the same server:

Not EmbeddingGemma through FastFlowLM on the NPU until issue 661 moves. That covers 1.0.3 and 1.0.4 here and every version back to 0.9.45 in the thread. On this laptop the same GGUF on the iGPU under Vulkan is the substitute. Any other backend gets the script once, before the first document goes into the index.

The FastFlowLM half is still open. Its repository carries the right answer for one paragraph of text, and the shipped Linux build is 0.003 away from it. I would like to know which build, on which machine, produced that reference.

Sources

Links

1. Too Long, Didn't Embed, the original post, and its correction
2. llama.cpp PR 16367, model: EmbeddingGemma Adding Support for SentenceTransformers Dense Modules, 9 October 2025
3. ggml-org/embeddinggemma-300M-GGUF, updated 29 April 2026 to include the Dense layers
4. ROCm/FastFlowLM#661, the collapsed embedding space, and my comment with the reference-vector and sweep results
5. EmbeddingGemma model card, for the pipeline and the prompt prefixes