The hardware: 4x AMD Radeon AI PRO R9700 (RDNA4, gfx1201, 32GB each), PCIe only, no XGMI between cards. gfx1201 gets no mention anywhere in ZML's docs, site, or the zmlai/llmd image, so this was unverified territory going in.
ZML builds its own inference stack from the ground up instead of building on PyTorch or JAX: a Zig-based tensor runtime with its own compiler backend targeting NVIDIA CUDA, AMD ROCm, Google TPU, Intel oneAPI, and Apple Metal from one codebase. llmd is their inference server built on that runtime, an alpha release (shipped July 2026) that serves LLaMA, Gemma, Qwen, and Mistral models over an OpenAI-compatible API, with continuous batching, paged attention, prefix caching, and tensor-parallel sharding out of the box.
The part worth testing on AMD hardware specifically: llmd ships native support for DFlash, a speculative decoding method from the z-lab team that drafts blocks of tokens with a small diffusion model and verifies them against the target model in one pass. That native support currently targets the Gemma-4 series, and Gemma-4's MoE variant (26B-A4B) specifically is where I'd previously seen DFlash pay off with a different (quantized, single-card) implementation, so testing ZML's own version of it was worth an afternoon.
What DFlash actually does at inference time
The draft model is a separate, small checkpoint, not a subset of the target model's own weights: llmd requires it as an explicit second path (--dflash-model=), and fails to start rather than silently falling back to plain decoding if it's missing.
Standard autoregressive decode at batch size 1 is strictly memory-bandwidth bound, leaving GPU ALUs largely idle while continually moving model weights from VRAM. DFlash changes execution mechanics by generating a candidate block of draft tokens and processing them through the target model in a single batched verification pass. This shifts execution toward higher arithmetic intensity and ALU utilization, trading raw compute density for lower end-to-end decode latency.
Running it
llmd ships as a per-backend container. I used docker.io/zmlai/llmd:rocm, but that tag is a moving alias; the same image is also published dated (20260731.0-rocm as of this writing), and that's the one worth pinning to if you want to reproduce this later against the same build. Podman needs explicit device passthrough for /dev/kfd and /dev/dri:
podman run -d --name llmd-test \
--device /dev/kfd --device /dev/dri --group-add keep-groups \
--security-opt seccomp=unconfined --security-opt label=disable \
-e HIP_VISIBLE_DEVICES=0,1 \
-v /path/to/models:/models:ro \
-p 8000:8000 \
docker.io/zmlai/llmd:rocm \
--model=/models/gemma-4-26B-A4B-it \
--dflash-model=/models/gemma-4-26B-A4B-it-DFlash
A few things that will trip up a first attempt:
This image's own /etc/group defines neither video nor render, so --group-add video and --group-add render both fail identically with "no matching entries in group file", regardless of which host group you actually belong to. --group-add keep-groups skips that lookup and just carries over the host process's groups, which is what you want here.
Always pin HIP_VISIBLE_DEVICES explicitly. Leaving it unset doesn't mean "pick one GPU," it means "shard across every GPU the container can see." I hit this directly: loading a small Qwen model with all 4 cards visible and no pin produced a hard thread N panic: attempt to unwrap error: IncompatibleSharding at load, because its attention-head count doesn't divide evenly by 4. Pin the exact device count you intend to use, even for a model that would comfortably fit on one card.
The bigger issue: llmd's hf:// model loader has no persistent weight cache. Every restart re-downloads the full model from Hugging Face straight into GPU memory; nothing gets written to disk, mounted volume or not (confirmed with podman diff against a full successful load: zero bytes under the mounted cache path). This behavior reflects a cloud-native design pattern—streaming weights directly into VRAM for ephemeral container deployments—but creates a major bottleneck for local self-hosting workflows. The effect on reload time is severe: llmd's own load-time log line went from Loaded weights [47.00GiB, 15m31.188s] over hf:// to Loaded weights [47.00GiB, 11.565s] off local disk, same checkpoint, same hardware.
The workaround is to sidestep hf:// entirely. llmd also accepts a plain local directory in HF repo layout (config.json + *.safetensors + tokenizer files), so download once with the standard tool and mount that instead:
hf download google/gemma-4-26B-A4B-it --local-dir /path/to/models/gemma-4-26B-A4B-it
hf download z-lab/gemma-4-26B-A4B-it-DFlash --local-dir /path/to/models/gemma-4-26B-A4B-it-DFlash
Point --model= and --dflash-model= at those directories (as in the command above) and every subsequent container start reads straight off local disk.
One more real constraint: llmd's current CLI has no quantization flag at all, BF16 only. Gemma-4-26B-A4B-it in BF16 is ~47GB, which does not fit one 32GB R9700. llmd reported needing 47.0GB against 27.2GB "available" per device (its --gpu-memory-fraction default of 0.9 applied to the ~30.3GB this card exposes to the driver) and failed with a clean OutOfMemory before touching the network. There's no explicit tensor-parallel flag either; llmd auto-shards across however many devices HIP_VISIBLE_DEVICES exposes, so two cards (HIP_VISIBLE_DEVICES=0,1) split it to ~23.5GB each and it loaded and served.
The numbers
These are smoke-test numbers, not a benchmark, and output equivalence was not rigorously verified. Single box, one fixed prompt, five requests per arm, one variable changed between arms (--dflash-model present or absent; same two cards, same weights, same context otherwise). Decode tok/s is client-side wall clock over a streamed response. Power is a single rocm-smi snapshot mid-generation per arm (two snapshots for the DFlash arm, one for the DFlash-off arm), not a sustained average. No batching, no concurrency sweep, no draft-acceptance-rate instrumentation, and output streams were not diffed token-for-token at temperature=0 to guarantee exact distribution equivalence. Treat all of this as a preliminary directional evaluation, not a production baseline.
Gemma-4-26B-A4B-it, 2-card tensor-parallel split (no XGMI, PCIe Gen 4 only), BF16, single request stream, 300 output tokens, greedy decoding:
| DFlash on | DFlash off | Δ | |
|---|---|---|---|
| Decode tok/s (avg of 5) | ~26 | ~15.6 | 1.66x |
| Time to first token | 0.11-0.24s (n=5) | 0.12-0.19s (n=5) | roughly flat, wider spread on the DFlash side is likely n=5 noise |
| GPU power, both cards combined | ~481W (two snapshots: 469W, 493W) | 284W (one snapshot) | 1.7x |
| Tokens per watt | 26 / 481 ≈ 0.054 | 15.6 / 284 ≈ 0.055 | roughly a wash |
DFlash yields a real 1.66x decode speedup here. I'd previously measured a larger 2.87-3.59x decode win from a different, quantized, single-card DFlash implementation of this same model, so 1.66x is a smaller number—consistent in direction but not directly comparable given differing precision and card count.
What it isn't is a free lunch: the verification step is a batched forward pass per round, driving higher ALU compute utilization and drawing proportionally more power (matching the ~1.7x speedup almost exactly). Consequently, both arms land at essentially the same tokens-per-joule. If your primary optimization target is wall-clock latency, DFlash wins outright. If your constraint is energy per token, this test found parity, not an efficiency gain.
A quick side trip: Strix Halo, and how this stacks up against the incumbents
I also tried the same test on a second box: a Strix Halo APU (gfx1151, 94GB unified memory, single GPU device). gfx1151 is equally undocumented in ZML's materials, and it also just works: same podman flags, same DFlash setup, no code changes. One extra gotcha showed up here that discrete cards never hit: llmd's default KV-cache sizing tries to fill however much memory it thinks is "available," and on unified memory that figure isn't capped by a fixed VRAM size the way a discrete card is. A completely unrelated tiny 0.5B model asked for a 47GB KV cache by default, sizing itself against the system's shared memory allocation. On a box running concurrent workloads, that is a severe hazard. Setting --max-context-len explicitly mitigates this and should be treated as mandatory on unified-memory targets.
With context length capped, Gemma-4-26B-A4B-it plus DFlash loaded and served fine on the single iGPU without multi-device sharding. Decode throughput averaged roughly 5-7 tok/s across five runs, showing higher variance than the R9700 setup. Monitoring GPU clocks and thermals during repeat passes showed no signs of throttling: memory clock remained pinned, core clock fluctuated within normal operating parameters, yet individual runs varied up to 2.7x in speed without visible clock degradation. This behavior remains unresolved.
None of the numbers below come from a tuned configuration on any side. They reflect default flags, first-pass runs, differing quantizations, and varying interconnect setups. A key driver of the performance difference between setups is interconnect topology: ZML's 2-card tensor-parallel setup running over raw PCIe Gen 4 incurs cross-card all-reduce synchronization latency at every layer during decode, whereas single-card setups bypass inter-GPU transfers entirely:
| Engine / hardware | Gemma-4-26B-A4B config | Topology | Decode tok/s |
|---|---|---|---|
| llama.cpp, 1x R9700, no draft | Q4_K_M | Single Card | 26.9 |
| llama.cpp + Lucebox DFlash, 1x R9700 | Q4_K_M + published 0.4B draft | Single Card | 77.2 (2.87x) |
| llama.cpp, native ROCm, 1x Strix Halo | Q4_K_XL | Single iGPU | 37-43 |
| llama.cpp, Vulkan, 1x Strix Halo | Q4_K_XL | Single iGPU | 52-64 |
| ZML/llmd + DFlash, 2x R9700 | BF16, no quant available | TP2 over PCIe | ~26 |
| ZML/llmd + DFlash, 1x Strix Halo | BF16, no quant available | Single iGPU | ~5-7 |
The quantized incumbents sit 3-13x ahead of ZML's unquantized runs (Lucebox's 77.2 vs ZML's ~26 on the R9700; Strix Halo Vulkan's 64 vs ZML's ~5 on APU hardware). A substantial portion of this gap stems from precision differences (BF16 vs Q4) and interconnect overhead (PCIe TP2 communication penalty), alongside early runtime optimization maturity in ZML's alpha release.
Where this leaves things
gfx1201 and gfx1151 runtime execution in ZML's ROCm backend is functional: the engine initializes and serves inference on hardware targets unlisted in official documentation, spanning both discrete GPUs and APUs. DFlash speculative decoding for Gemma-4 functions as claimed regarding speedup ratios, though token-for-token output equivalence requires formal verification.
ZML's unified compiler architecture, targeting five hardware platforms from a single Zig codebase, presents a compelling engineering approach. Getting two undocumented AMD targets to run and serve without backend crashes shows notable runtime resilience out of the box. However, in its current alpha state, llmd is not yet a drop-in replacement for llama.cpp or vLLM on consumer AMD hardware: it lacks quantization support, GGUF loading, and persistent local caching without explicit workarounds, while default KV-cache allocation rules present risks on unified-memory platforms. It remains a project to track closely as feature coverage and performance tuning progress.
