On the smallest computer it supports, an 8 GB laptop, Nocturne reads your photos, voice notes and text notes and answers questions about what you kept. None of it leaves the computer. Lemonade's embeddable release makes that possible today.

This post is about what it took to run Lemonade, the local inference server orchestrator, inside an app whose users never know it is there. That one server also has to find the right chip on a Mac, a Windows laptop or a Linux desktop, AMD's NPU included. I wrote it mostly for other people building local-first apps with AI in them.

Context

Nocturne reads what you keep, on your own computer

Nocturne is a private desktop app for people who are not technical about AI. During the day you keep things, like a photo of a letter or a voice note. Nocturne reads them on your own computer whenever that computer is plugged in and idle, by night or by day. A laptop that sleeps catches up when it wakes. At the time you choose, you get one morning page with what needs you, what changed and what it let go of. You can ask it questions, and every answer cites the originals it came from.

The morning page in Nocturne, with sample data
The morning page in Nocturne, with sample data.

Nocturne is a Tauri app, a Rust core with a web view, and the source is at github.com/agync/nocturne. Version 0.6.0 shipped on 17 November for Linux on x86_64 and Arm, macOS on Apple silicon, and Windows.

How Nocturne talks to its private lemond, and what lemond runs
How Nocturne talks to its private lemond, and what lemond runs.

Why embed

The server lives inside the app

The people Nocturne is for should never have to install a server, choose a model or keep a service running. With Lemonade inside the app they install one thing, and on first run, setup downloads what their computer can run.

Lemonade fits because one OpenAI-style server covers chat, vision, embeddings, reranking, speech to text and text to speech. It runs them on Metal, ROCm, Vulkan or the CPU, and on an AMD NPU through FastFlowLM. The app asks for a model by name and mostly does not care which chip runs it. A private copy also means no other app can load or evict models under it. Nocturne pins one release (v2026.46.2 today), ships it among the app's resources and keeps everything the engine writes in the app's data folder.

Process

The app owns the process

Nocturne starts the engine itself. Each launch gets a random loopback port and fresh keys, with a separate admin key that only the supervisor holds. The environment is cleared of everything but a home folder, a path, a locale and the two keys. If the engine hangs, the app restarts it (supervisor.rs).

The engine's config.json keeps it offline outside a download window the person agreed to, and turns off discovery, updates and telemetry. Browser origins are locked by naming one under the reserved .invalid domain, because an empty list falls back to defaults that allow pages on the machine's own addresses.

Hardware

Pick a tier from what the computer reports

On first run, Nocturne calls GET /v1/system-info on the engine and picks the largest tier that memory and free disk allow:

Table
TierNeeds memoryChat modelSpeechDownload
Large60 GiBQwen3.6-35B-A3B-MTPWhisper Large v3 Turboabout 27 GB
Standard28 GiBQwen3.5-9BWhisper Smallabout 9 GB
Small14 GiBQwen3.5-4B-MTPWhisper Baseabout 4.8 GB
Tiny6 GiBQwen3.5-2BWhisper Baseabout 3.2 GB

Every tier uses Kokoro for the voice and Qwen3-Embedding-0.6B for recall.

The backend comes from the same report. A Mac gets Metal and a discrete AMD card gets ROCm. Any other graphics (AMD integrated, Intel and NVIDIA) gets Vulkan, and anything else runs on the CPU. The pinned engine reports no Intel graphics at all, so Nocturne finds them itself, from the PCI bus on Linux and pnputil on Windows. NVIDIA runs on Vulkan rather than CUDA, because the engine's CUDA backend serves llama.cpp only and wants a Turing or newer card.

On an 8 GB laptop, Nocturne's whole share (app, web view, engine and models) has a budget of 4.5 GB at its busiest. Measured on Tiny:

Table
MomentLinux, integrated Radeon (Vulkan)Mac, Apple M4 Max (Metal)
Idle, window open0.43 GB0.1 GB
Reading5.0 GB4.7 GB
Answering5.2 GB4.4 GB
Reading the page aloud2.5 GB3.8 GB
Memory of the app and every process under it, by phase, Tiny tier, from make e2e-real.

The Mac holds the budget when idle and answering. It goes about 0.2 GB over while reading a recording, when the speech model sits beside the mind and the recall model, and about 0.3 GB over on a page longer than the table's, when the voice alone grows to 3.7 GB. Linux is over while reading and answering, because the model servers there hold about 0.7 GB more graphics memory.

Residency

Save load options for every model

Staying inside that budget starts with load options. Load a model with no saved options and the engine sizes it to fill the free memory. For a small embedding model that means a 32768 context and about 3.7 GB of cache it never uses. Nocturne saves a context size for every model in the engine's recipe_options.json (8192 for recall), so a load the engine makes on its own stays small.

The engine evicts by count first. With max_loaded_models at its default of 1, a second chat model unloads the first even when both would fit. The engine's memory-pressure eviction watches graphics memory only and does not run on a Mac, so on a Mac the idle timers Nocturne saves per model hand memory back instead.

Constrained output

The schema's keys arrive sorted

Reading a note means asking the chat model for a list of memories (short facts) as JSON, each with the quote it came from, a sentence, a subject, dates and a confidence. I sent that as a JSON schema in response_format, with the quote first.

The model never saw that order. Nocturne's own serde_json builds a sorted map, and Lemonade parses the request body into nlohmann::json, which is sorted too. llama.cpp builds its grammar in the order the keys arrive, so the grammar had the model write each memory's confidence first and its quote seventh.

A JSON schema reaches llama.cpp with sorted keys, a GBNF grammar arrives as written
A JSON schema reaches llama.cpp with sorted keys, a GBNF grammar arrives as written.

The fix is a grammar in GBNF, llama.cpp's own format, sent in the request's grammar field. A grammar is a string, so it arrives as written. It carries the bounds too, at most 16 memories and 300 characters in each text:

TEXT
memory ::= "{" ws "\"span\"" ws ":" ws span "," ws "\"sentence\"" ws ":" ws text ...
text ::= "\"" char{1,300} "\"" ws

How much of the gain came from the order alone I can't separate from the changes below. If key order matters to you, send a grammar, check what every layer does to the JSON on its way through, and check that the backend you land on reads the grammar at all.

NPU

Put the mind on the NPU where there is one

That last check is about the NPU. Nocturne calls its chat model the mind, and since 0.4.0 the mind can run on an AMD XDNA 2 NPU where a computer has one. On the NPU it is Gemma 4 e4b through FastFlowLM, about 9.8 GB once loaded.

At each start npu.rs picks a placement. In Split the NPU reads notes and photos while the GPU mind answers. In NPU only, on shared memory where one mind fits, the NPU does both. Otherwise everything stays on the GPU. In practice a 48 GB laptop runs Split, a 32 GB one runs NPU only, and 8 and 16 GB computers keep the GPU.

Three things in the engine's source shaped this. Eviction counts models of a kind across every device, so an NPU and a GPU chat model share one pool, and Split needs max_loaded_models at 2. FastFlowLM ignores a grammar without saying so but honours a JSON schema, so NPU reading goes by schema and the sorted keys come back. On e4b that did not matter, since it kept every fact anyway. And a failed load makes the engine unload every model and retry, so after an NPU failure Nocturne loads the GPU mind afresh.

On a Ryzen AI Max, e4b on the NPU answered 12 of 12, kept all 42 key facts and cited with a precision of 1.0, against 11, 40 and 1.0 for Small's GPU mind on the same machine. Reading takes about 20 seconds per item, six times slower, which is fine for work that runs in the background. NPU only is the weaker case. On a 32 GB laptop it replaces Standard's Qwen3.5-9B with e4b.

Streaming

Stop a loop while it streams

A small model can fall into a loop, writing the same memory again and again or copying out its instructions, until it hits its token ceiling. So reading streams, and mind/watch.rs ends the request at the first repeated memory or copied instruction, keeping what came before. The note is read again in halves only when the kept memories miss a fact from it.

Evaluation

Measure each stage, or you tune the wrong one

Every model and memory result here comes from make e2e-real. It runs the whole product against the real engine and real models on one seeded day of 31 items, with twelve questions and ten held out. It is not a benchmark.

One aggregate score tells you answers are wrong but not where, so the run scores each stage on its own. On Tiny, 14 of 15 wrong answers across three runs came from a fact lost or mixed up while reading or storing the note. On this small set, retrieval ranked the right memory first whenever it existed, and the answer prompt was rarely the cause.

So most of the work went into reading and storing the note, plus one change to search:

  • in reading, the ordered grammar and its bounds, and a check that a reading kept every number, name and role, with one targeted re-read if not
  • in storing, no more merging memories only because they share a date
  • in search, an instruction prefix on queries for the embedding model

On Tiny, three runs per column:

Table
BeforeAfter the reading roundAfter the speed round
Correct answers (of 12)median 7median 10 (12, 10, 9)11 or 12 in each run
Citation precisionabout 0.78about 0.951.0 in later runs
Key facts kept in reading (of 42)not measured4141
Reading time per itemabout 4 sabout 12 sabout 10 s
Tiny tier before and after two tuning rounds, three runs each, from make e2e-real.

The 1.0 comes from the later runs for the NPU decision, on a Ryzen AI 9 HX 470. Those runs also gave Tiny 11 to 12 answers and 41 facts.

Online answers

Hand the key over at run time

Nocturne can also send the last step of a question to a hosted model the person chose. The app never talks to that provider. lemond does, through Lemonade's experimental cloud_providers feature, and only while the person has it turned on.

The key never goes in the engine's environment, argv or a file. Once the engine is up, the supervisor hands it over on loopback with POST /v1/cloud/auth. The order matters. Handing over a key makes Lemonade refresh the provider's model list, and that refresh drops a model row you registered yourself. So load the row's model first and hand over the key after, which works because a loaded model reads the key on every request. Anything holding the engine's per-launch key can still add a provider, so treat that key as an admin credential.

Windows

Windows wants a longer start

The first real Windows report came from a laptop with Intel graphics and an NVIDIA GPU. Nocturne never got past starting the engine on it. My first guess, filed upstream as lemonade-sdk/lemonade#4185, was a child process whose read ignores its timeout. That code path still looks wrong to me, but it was not the cause here.

On that laptop one WMI query for Win32_VideoController takes about 36 seconds, and the engine's hardware probe makes it twice before the listener binds. The result is kept in memory only, so the port stays closed for over 70 seconds at every start. My readiness limit was 60 seconds.

The fix, pull request #77, shipped in Nocturne 0.5.1. It raises the limit to five minutes and logs a line at the first minute saying the engine is still starting. If a start still fails, the log says whether the engine ever listened. If you embed lemond, size the readiness wait for hardware probing on the slowest machine you will meet, not on yours.

The rest of the Windows list is small and easy to miss. Every program Nocturne starts gets CREATE_NO_WINDOW (process.rs), because otherwise each console child opens a window of its own. lemond reads its home from USERPROFILE and stops at once without it. So the cleared environment sets it back, pointed at the app's folder, and points TMP and TEMP at a folder only the engine uses. And the engine's log passes through a pipe that masks both per-launch keys, because a person sending a log should not be sending a credential.

Ports

Keep the port out of the range the system lends

A slow start leaves another gap, in how the engine's port is claimed. The supervisor used to pick a free port for lemond from 20000 to 59999 by binding it as a test and letting it go. The engine binds only once it has finished starting, and a first start can take minutes, as on that Windows laptop.

About 68% of that range overlaps the ephemeral range, the ports the system lends to outgoing connections (32768 to 60999 on Linux, 49152 to 65535 on Windows and macOS). Any loopback connection made during the start could be given the engine's port. The engine's bind then failed with "Address already in use", the start counted as a retry, and a second engine launched. It showed up as one flaky CI test that expected one launch and saw two. Binding a loopback client to the engine's port during a slow start reproduced it every time.

The fix, pull request #92, shipped in 0.6.0. It draws engine ports from 20000 to 32767, below every system's default ephemeral range. The claim is held under the supervisor's own lock until the engine is running, so two supervisors in one process cannot pick the same port. If you pick the port for an engine that binds late, pick it outside the ephemeral range and hold it until the engine is up.

Sources

Sources

  1. Nocturne source, github.com/agync/nocturne, in particular crates/lemonade/src, docs/architecture/runtime.md and decision 0014 on the NPU mind
  2. Lemonade v2026.46.2, the pinned release
  3. The Windows start-up report, lemonade-sdk/lemonade#4185, and its follow-up comment
  4. Measurements from make e2e-real runs on the Tiny and Small tiers and the NPU models, recorded in the repository's docs and my working notes