d1-Omni-2B on four GPUs

Model

d1-Omni-2B

The engine requirement and the unsupported question types are vendor-reported and were not re-tested. Everything else in this report was measured.

Method

The set, the repeats and the wire

The set is 37 labelled records and 126 questions: 12 text controls, 12 synthetic voice notes (8 English, 4 German), 9 silent screen recordings and 4 screen recordings with a spoken soundtrack. Voice notes come from macOS text-to-speech at 16 kHz mono. Recordings are rendered frames, 1280x720 H.264 at 2 fps. Labels are known by construction. The last frame of each recording is also sent as a JPEG still, which adds 13 records and 26 questions, so d1 answers 152 questions per pass. The earlier "126 questions" figure counts only the original set.

urgency is a 4-level score (can wait, this week, today, right now). production asks whether the speaker says they are working on the production system.

Repeats. Each request was sent 3 times. d1 returned the same answers and nearly the same probabilities each time (the 8060S miss reads 0.182 on every repeat), so the repeats confirm determinism and give latency spread, not accuracy variance. The effective sample is 152 questions, not the 456 rows.

Wire. One System One request per record, with a task string in state and the questions in questions. Audio and video travel as base64 data URIs in files (WAV, MP4). A recording with a soundtrack sends the MP4 and its WAV as two files. Stills travel in images. Text controls carry the voice note's transcript in state.

Scoring. A yes/no answer is yes at probability 0.5 or above. A choice or score answer is the highest-probability option. p(correct) is the probability the model gave the labelled answer. Latency is wall-clock time per request, so p50 and p95 are over requests, with nearest-rank percentiles. Brier is the mean of (1 - p(correct))^2.

Devices

Results by device

Four devices ran the set. The Mac was shared with other work, so its latency is indicative only.

Rows differ by device because video did not run everywhere (section on memory), so this accuracy column is not like-for-like. "p95 all" mixes modalities. The like-for-like table keeps only the text, audio and image rows every device ran:

All four devices give 360/366, and the same six rows are wrong on each. The 0.984 to 0.987 spread in the first table comes from which modalities ran, not from the device.

RX 7900 XTX video is 15 requests (five clips) and video+audio is 3 (one clip), so read those rows as single points. The Mac has no video rows: video failed to decode there.

Modality and question type

Results by modality and question type

Cells are correct/rows and mean p(correct). Rows are questions times 3 repeats, so the 3 text misses and 3 audio misses are the same question repeated.

Only the 8060S run covers all five modalities:

Audio has the lowest mean p of the single-input modalities (0.864) and carries the miss. Stills are the cleanest (78/78, 0.918). video+audio is the slowest because the clip and its soundtrack are both encoded.

By question type

Every miss is a choice question. Score questions (urgency) are all correct and carry the lowest mean p, 0.761 on every device.

By question and by modality, Radeon 8060S

video+audio yes/no questions have the lowest mean p of the yes/no cells (0.842), driven by production (0.764), the one question that depends on the spoken soundtrack.

Calibration

Underconfident, and right in every bin

For a yes/no question the probability of the answer given is stored exactly, so a true reliability table can be built. Pooled over the four device runs (516 rows, same model and quant), every confidence bin is 100% right. d1 reports 0.625 mean confidence on 15 rows that were all right. Clef-Flash, 75 rows, shows the same pattern with three rows per bin below 0.9, so its bins say little. Expected calibration error is 0.0806 for d1 and 0.0632 for Flash, both driven by underconfidence.

Across all question types the picture is the same. Bins are by the probability on the correct answer, pooled over 1,593 d1 rows:

The 24 wrong answers all sit below 0.31 on the correct option. The lowest p(correct) among right answers is 0.495, a score question. Per-option probabilities were not stored, so the probability of the wrong answer actually given is unknown for choice and score misses, and a confidence-in-answer table cannot be built for those types.

Brier score against Clef-Flash

d1 is the Radeon 8060S run restricted to the 222 text and still rows Flash also answered. Brier here is on the correct-option probability only: exact for yes/no, a lower bound on the full multiclass score for choice and score. Lower is better.

Pooled over all modalities and four devices, d1 is 0.0103 on yes/no, 0.0277 on choice and 0.0678 on score. The gap to Flash is mostly the score questions (0.0633 against 0.0106) and the single wrong choice question.

Comparison

Against Clef-Flash 9B Q8_0 and Clef-Omni

Clef-Flash, same set, same run day

d1 is faster on both modalities and less accurate by 3 rows (the German note, repeated 3 times) and less confident. Both models ran on the same Radeon RX 7900 XTX.

Earlier Clef-Omni runs

Clef-Omni was run on the same 37 records and 126 questions on an earlier harness: MLX on an Apple M4 Pro on 2026-10-10, and PyTorch with int4 experts on the GPUs on 2026-10-11. Those runs got one pass, no stills and a different engine, and were read from their stored results, not re-run. p(correct) and latency are therefore not controlled comparisons. d1 below is restricted to its 126 non-still questions.

The earlier summary lists Clef-Omni's mean p as not recorded. The stored rows carry it (field p) and it is computed here: 0.824 to 0.828. The one Clef-Omni miss, on the M4 Pro, is a different question from d1's: voice07 language, an English note read as German at p 0.578. d1 got that question right.

Median latency in seconds. On the Radeon 8060S d1 is lower by 43.6x (text), 33.6x (audio), 3.2x (video) and 1.9x (video+audio). The RX 7900 XTX pair is the same card under the earlier PyTorch harness and under llama.cpp. The Apple Silicon pair compares an M4 Pro with an unspecified Mac that was shared with other work; d1 is slower on audio there, 1.42 s against 0.63 s.

Memory: earlier Clef-Omni peaks were 21.0 GB (M4 Pro, MLX), 21.1 GiB (RX 7900 XTX) and 23.5 GiB (8060S and R9700), as reported by those runs. d1 uses 2.44 to 4.56 GiB. The measures differ (peak during a run against server memory), so a ratio of roughly 5 to 10 is indicative.

The miss

One German voice note, wrong as audio and as text

The note is "Bitte ruf heute noch den Kunden wegen des Angebots zurück" (call the customer back today about the offer), labelled work. d1 answers travel in every repeat on every device, and also when the same sentence is sent as text, so the cause is the model's reading of the sentence, not audio decoding and not a device. p(work) is 0.18 to 0.19 for audio and 0.31 for text.

d1 got the other three questions on the same note right, and the other three German notes (voice09 to voice11) were 12/12 each, at mean p 0.888, 0.902 and 0.844. Clef-Flash (text twin only) and the earlier Clef-Omni run (voice and text) both labelled this note work. Why d1 reads the sentence as travel was not investigated.

Memory

Memory against ubatch, and which media fit

Within one device the growth is 0.22 GiB per 1024 of ubatch from 1024 to 2048 (R9700) and 0.245 GiB per 1024 from 4096 to 8192 (8060S). Across the mixed devices, end to end, it is 0.30 GiB per 1024 (2.12 GiB over 7168 tokens). An earlier summary rounded the figure to 0.25 GiB.

llama.cpp refuses a request whose input exceeds the physical batch size ("input (4484 tokens) is too large to process, increase the physical batch size"). Text, audio and still images fit at every ubatch tested, including 1024. Video does not:

Rejection-message counts run from 3,596 to 4,540 tokens (3.6K to 4.5K). At 4096 the 3.6K clips fit and the 4.5K ones do not; at 1024 and 2048 no clip fits; at 8192 all 13 do. The usage field of the accepted runs reports about twice the rejection count for video (7,213 against 3,596) and about three times for video+audio (10,979 against 3,652). Both are shown and the reason was not established. Stills are 1,069 tokens in the usage field at ubatch 1024 against 1,869 at 2048 and above, so the image encoder produced fewer tokens at the smaller batch.

Practical limits

What stops a deployment

• No ffmpeg, no video. The Metal build on the Mac had no ffmpeg. Every video and video+audio request failed with "Failed to load image or audio file", while audio and stills worked. The Linux runs used a build with ffmpeg. The cause comes from the run notes.
• No audio input on Clef-Flash. It was run on text and stills only.
• Video needs a large ubatch. Clips of 3.6K to 4.5K tokens need 4096 or 8192, which is 3.13 to 4.56 GiB.
• Concurrency. Servers ran one slot (--parallel 1). Throughput under load was not measured.

Sizing media by base64 length overestimates audio

A proxy in front of the server admitted requests by estimating input tokens from base64 length, against a 2048-token physical batch limit. It rejected all 36 audio requests with estimates of about 29,800 to 56,800 tokens, when the server used 366 to 494.

The estimate is 81x to 115x too high for audio (median 98x), roughly right for video and 3.8x to 6.1x high for video+audio. A proxy for decision wires that carry media needs a per-modality estimate. The same proxy rejected all 39 stills as unsupported images. Text through it: d1 0.979 accuracy at 0.11 s p50, Clef-Flash 1.000 at 0.16 s.

Limitations

What this does not show

• Small set. 37 records, 126 questions (152 with stills). With 150 of 152 distinct questions right, the Wilson 95% interval on accuracy is 0.953 to 0.996. Per-modality cells hold 9 to 13 records.
• Synthetic voice. One text-to-speech system at 16 kHz, no noise, accents or overlap. Four of twelve notes are German; no other language was tested.
• Rendered video. Screen recordings are drawn frames of terminals, browsers and editors at 2 fps, not captured sessions.
• One run day. All d1 and Flash measurements come from a single day. Repeats return identical answers.
• One judgement label. The topic label of the German note is a judgement call, and the only miss depends on it.
• Unequal runs. ubatch differs by device (1024, 4096, 8192) and video ran on two devices, so per-device accuracy is not like-for-like outside the text, audio and still rows.
• Not measured: energy and power, long-context behaviour, noisy or real-world audio, throughput under concurrency, and the multiple, extraction and boxes question types.

Sources

Sources and caveats

• Computed here: every accuracy, mean p, latency, calibration and Brier figure, from the stored per-row results of each run.
• From the run reports, not the result files: the memory figures (2.44, 2.66, 3.13, 3.58 and 4.56 GiB) and the device each was measured on.
• From the model card, not re-tested: the llama.cpp requirement and the unsupported question types.
• Earlier run: Clef-Omni numbers, read from stored results of an earlier harness and not re-run.
• Second 8060S run: a ubatch 2048 run on the Radeon 8060S is used only in the video fit table and the miss table. The main 8060S results are the ubatch 8192 run.
• Two token counts: video token counts differ between the rejection message and the usage field. Both are reported.
• Brier: on the correct-option probability only, so exact for yes/no and a lower bound for choice and score.