# The Lemonade Backend Gateway *A First-Principles Design for Secure, Pluggable Inference Backends* **Status: R&D design proposal — not yet an RFC, not an implementation.** The design laid out here is grounded in how Lemonade's backends *actually* behave, in the capabilities already present in the codebase, and in the positions of the maintainers who reviewed the Working Group charter. It is intended to be read as one coherent, self-contained proposal. --- ## 1. Why this exists Lemonade runs inference by spawning backends as subprocesses (`llama-server`, `whisper-server`, `sd-server`, `flm`, `vllm`, and others). Today, adding a backend means writing a new C++ `WrappedServer` class and recompiling `lemond`. That limits Lemonade to in-tree backends, slows adoption of new inference engines, and forces maintainers to build, pin, and ship every engine for every platform and GPU vendor. At the same time, letting users run third-party or community binaries raises a real security problem: an untrusted binary that runs as a child of `lemond` with the same user, full network access, and the server's environment can read model files, reach the network, and exfiltrate API keys that happen to be in the environment. The Backend Plugins & Sandboxing Working Group exists to resolve exactly this tension: **make Lemonade open enough to run any engine, and safe enough that doing so does not compromise the machine.** This document is a first-principles proposal for how to meet both goals. --- ## 2. The options considered There are three natural ways to make a backend pluggable. Each takes a different position on where procedural adaptation logic lives, and each has hard limits. They are evaluated here on their own terms. **Option 1 — Declarative descriptors (backend as data).** Express a backend as a JSON manifest: identity, capabilities, an argument template, a health probe. Simple to write and submit. The limit is fundamental: the interesting part of a backend is *procedural*. Hardware detection, model-metadata introspection, conditional argument construction, and payload rewriting cannot be expressed as data without the data becoming a scripting language. Whatever cannot be declared leaks back into lemond's C++, so the "escape hatch" reintroduces exactly the code the format was meant to avoid. **Option 2 — In-process binary plugins (`dlopen`).** Load the backend's integration as a shared library inside lemond. Fully expressive as real code. The limit is security: it runs *inside lemond's memory space* with lemond's full privileges, which defeats the sandboxing that running third-party binaries demands, and a fault in the plugin takes the server down with it. **Option 3 — Out-of-process adapter program (this proposal).** Keep the adaptation as real code, but run it as a separate, sandboxed peer process behind a narrow contract. It is expressive (code, like Option 2) and isolated (sandboxable and crash-proof, like Option 1 was intended to be), without either option's limitation. Options 1 and 2 both decide the question by *where the code goes* — data in lemond's interpreter, or code in lemond's process. Both, for opposite reasons, leave untrusted behavior inside lemond's trust domain. The fix is to change what they are optimizing: ### 2.1 The trust boundary is the process boundary > **Draw the trust boundary at the *process* boundary.** A backend is really three concerns glued together in one C++ class today: (1) a declarative **identity** (`BackendDescriptor`), (2) arbitrary procedural **adaptation** (argv construction, hardware detection, payload rewriting, protocol translation), and (3) shared **subprocess + HTTP orchestration** (spawn, port lease, health, watchdog, eviction) that `WrappedServer` already provides. Option 1 pushes concern (2) into data and fails; Option 2 pushes (2) into lemond's process and is unsafe. This proposal **keeps (2) as real code, but runs it in its own sandboxed process**, drawing the line at that process boundary: - **lemond** stays a trusted C++ orchestrator with a *closed, typed* set of capability contracts. It is a fact authority, a resource broker, and the owner of the process/API edge. - **The plugin (an "adapter program")** is a self-contained, sandboxed, out-of-process peer process. It owns its own engine argv and spawn, performs the adaptation, and talks to `lemond` over a narrow, versioned contract. A fault in an adapter cannot take `lemond` down. An adapter cannot reach beyond what its sandbox grant allows or read secrets `lemond` chooses not to hand it. And because the adapter runs behind a wire contract, **the language it is written in is irrelevant to `lemond`** — a static binary (C/Rust/Go) or a container, whichever fits the engine. The result satisfies expressiveness and isolation *at once*: the sandbox becomes the security boundary, and the adapter becomes whatever code an engine really needs. --- ## 3. Requirements, derived from first principles ### 3.1 Functional needs - **N1 — Arbitrary procedural adaptation.** Hardware detection, model introspection, conditional flags, payload rewriting, new multimedia modalities. These are real, present in every non-trivial backend, and inherently code. A system that cannot express them only supports "spawn + HTTP passthrough" engines. This rules out pure JSON. - **N2 — No mandatory runtime on the host.** An adapter must not assume Python (or any interpreter) is installed; it is self-contained or explicitly brings its own (container). - **N3 — Distribution with low friction.** A recipe author publishes without rebuilding/relinking `lemond` or maintaining in-tree C++ — but without forcing a complex plugin-authoring build system. - **N4 — Declarative identity.** `lemond` needs searchable, presentable metadata (name, capabilities, devices, slot policy, grants) before it runs anything, for the CLI/GUI, `/system-info`, and consent flows. - **N5 — Closed, bounded public surface.** Plugins must not open arbitrary new public endpoints. New capability kinds are additive, reviewed contract changes. ### 3.2 Security needs (from the actual threat surface) Auditing today's code shows there is currently **no OS sandboxing for backend processes** and they inherit `lemond`'s full environment. The real leaks to close: - **S1 — Least privilege at the adapter boundary.** Untrusted code runs with only the filesystem/device/network it declares, enforced by the OS, with default-deny network egress. - **S2 — Secret isolation.** Backend processes must not inherit the ambient environment (`LEMONADE_API_KEY`, `LEMONADE_ADMIN_API_KEY`, `LEMONADE__API_KEY`). Today they leak by inheritance. - **S3 — Crash / stability isolation.** A plugin crash must not take `lemond` down or disturb other slots. - **S4 — Consent before capability grant.** The user reviews and approves declared grants before enabling a third-party plugin. - **S5 — Supply-chain integrity.** Installed plugins are pinned and content- hashed; the sandbox limits blast radius regardless of binary honesty. ### 3.3 The synthesis Needs N1+N2+N3 resolve to "the plugin carries its own procedural logic as a self-contained, sandbox-capable subprocess." N4+N5 resolve to "lemond describes that program declaratively and exposes only a closed, typed contract to it." S1–S5 resolve to "the sandbox wraps the outermost untrusted process — the engine for built-in/passthrough, the adapter (which contains the engine) for external-adapter plugins — with env scrubbing and default-deny grants." --- ## 4. The shape: the "Capability Gateway" ```mermaid flowchart TD subgraph lemond["lemond (trusted core)"] direction TB A["Closed capability contracts (typed)"] B["Fact authority + resource broker"] C["Process / API edge custodian"] end D["Adapter program (the PLUGIN)
sandboxed subprocess
arbitrary adaptation, owns engine argv/spawn
Landlock/seccomp · no ambient env"] E["Engine
e.g. llama-server, a container"] A -- "1. launch: context + grants" --> D B -- "2. capability RPC over local socket" --> D C -- "3. health / liveness / teardown" --> D D -- "owned subprocess or embedded engine" --> E ``` ### 4.0 Sandboxing vs. the adapter tier (two independent decisions) Two things are easy to conflate and must be kept separate: - **No engine binary is trusted, in-tree or not.** Being built-in grants no trust: we do not re-audit `kokoro` on every release any more than we audit a third-party plugin. Zero-trust is the aim, and the sandbox is the baseline for *every* engine process — built-in and plugin alike. Today's `llama.cpp` subprocess runs unsandboxed; under this design its engine process itself is wrapped by the kernel sandbox, exactly like any plugin's. "Built-in" buys packaging and support, never a trust exemption. What is trusted is lemond's own coordinator code (secret custody, arbitration, the process/API edge), and only because it is lemond's own code — the smallest trusted core that cannot be avoided — not because the backends it manages are trusted. - **The adapter *process* (the middle tier) is conditional.** It exists only when lemond is *not* the one doing the adaptation — i.e. only for external-adapter plugins. Built-in backends keep lemond's adaptation in-process (it is the coordinator's own code, not a trust claim about the engine) and their engine is still sandboxed; passthrough plugins need no adapter either. So the process tree is: | Kind | Process tree | Sandboxed? | |---|---|---| | Built-in backend | lemond → engine | engine sandboxed | | Passthrough plugin | lemond → engine | engine sandboxed | | External-adapter plugin | lemond → adapter → engine | adapter **and** engine sandboxed | The sandbox boundary sits at the *engine* (the untrusted thing) in every case; the adapter tier is added only when an untrusted adapter owns the adaptation. This keeps "even built-in engines are sandboxed" as a stated guarantee while avoiding an unconditional 2→3 tier expansion. ### 4.1 The manifest Declarative and small — it describes *where* the adapter is and *what it may do*, never *how*. Procedural intelligence is not here. ```json { "api_contract_version": "1", "recipe": "my-engine-custom", "display_name": "My Engine (Custom)", "extends": "llamacpp", // optional: inherit base recipe's options/contracts "adapter": { "kind": "binary", // "binary" | "container" | "passthrough" "source": "https://cdn.example/adapter/linux-x64/my-engine-adapter", "sha256": "9f3a…c2", "version": "1.2.3" }, "engine": { "reserved_args": ["--port", "--host", "-m"] // reject user args colliding with these at merge }, "capabilities": ["chat_completion", "completion", "embeddings"], "capability_enable_args": { // "I support X; enable it with arg A" "embeddings": ["--embeddings"], // appended when model_info.type == embedding "completion": ["--jinja", "--metrics"] // unconditional runtime args (always) }, "capability_contracts": { "chat_completion": { "response_fields": { "model": {"mode":"echo_request_model","sensitive":false} } } }, "slot_policy": "standard", // standard | exclusive_npu | coexist_by_type | unmetered "model_management": "lemond_managed", // or "self_managed" "model_reads": { "main": { "form": "gguf" } }, "lifecycle": { "spawn_shape": "single", "requested_ports": 1, "readiness": {"type":"http","endpoint":"/health"}, "self_manages_downloads": false }, "sandbox": { "read_paths": ["{hf_cache_models}", "{binary_dir}"], // scoped to models subtree, NOT cache root (D1) "write_paths": ["{scratch_dir}"], "devices": ["/dev/dri"], "network": { "allow_egress": false, "allow_loopback": true }, "env_allowlist": ["GGML_VK_VISIBLE_DEVICES"] }, "facts_scope": ["devices","igpu","rocm_arch","resolve_checkpoint","model_info","vram_pressure"] } ``` The schema has two adapter flavors. **`passthrough`** (§4.2) reuses an existing binary/container with a manifest only; **`binary` / `container`** ships an adapter program for engines that exceed the passthrough envelope. The manifest is "data, not code," but with one **deliberate, bounded exception**: **launch-time token substitution.** The `argv` / `command` arrays may contain tokens (`{checkpoint:main}`, `{port}`, `{ctx_size}`, `{custom_args}`) that lemond substitutes *once, when constructing the spawn* — this is how lemond hands the engine its model path, port, and user-option values. That is the sanctioned, declarative subset: tokens exist to *launch* the engine, and they appear only in the manifest's `argv`/`engine` blocks, never in request-time logic. What is *barred* is **request-time adaptation in data**: no `request_transform` field maps, no per-request payload rewriting expressed in JSON, no scripting. Those are what the earlier proposals tried to force into data and what degenerated into a Turing-complete language. Request-time adaptation lives in the named-transform registry (§6, bounded, vetted) or — when genuinely open-ended — in an adapter program (§4.2, out-of-process sandboxed). So the statement "no command-template scripting" is accurate; the launch-time `{token}` substitution is the one declarative exception, and it is only ever about how the engine is started. **Token vocabulary (launch-time only, resolved by lemond):** | Token | Meaning | |---|---| | `{checkpoint:role}` / `{checkpoint_relative:role}` | resolved checkpoint path (absolute / lexical-relative) for a recipe role (`main`, `draft`, `mmproj`, ...) | | `{hf_cache_models}` | the cache **models** subtree (`hub/models--…`), the scope a model read grant should use — never the cache *root*, which also holds credential files (D1) | | `{port}` / `{host}` | leased port / host for the engine (FD/socket custody or fallback, §5.6) | | `{ctx_size}`, `{threads}`, `{gpu_layers}`, ... | declared option values (defaults live in `options`) | | `{socket_fd}` | the leased socket descriptor, when the engine accepts an inherited FD | | `{custom_args}` | validated user-supplied arguments, appended to the argv array | | `{env:VAR}` | **forbidden** — leaks lemond's environment into child argv and bypasses the scrubber | Tokens are substituted only in the `argv`/`engine` blocks at spawn; the `env` block values may also reference token values but never read lemond's ambient environment. See §6 for the separate request/response surface. The manifest only declares identity, capability claims, lifecycle/ownership (driven by the recipe junctions of §9.1), and the **sandbox grants plus fact scope** — precisely what a consent dialog, a kernel sandbox, and the FactService need. The schema above shows both adapter flavors; the `capability_contracts`, `model_management`, `model_reads`, `lifecycle`, and `facts_scope` fields are covered in §4.3, §5, and §9. The optional **`platforms`** block is the OS × accelerator matrix. It declares *which* host/device combinations the recipe runs on and *what differs* per combination — GPU device nodes, accelerator env vars, arch gates, and any security-posture override (e.g. a `wsl2` vs `windows_native` distinction). Platform gating reuses the same machinery lemond applies to built-ins (`BackendSupport`, `SystemInfo::check_recipe_supported`, `ModelManager::filter_models_by_backend`), so lemond enables/disables recipes against the detected OS and GPU at model-cache build time, exactly as it does today. The matrix is **DRY by construction**: the base recipe (options, lifecycle, capability contracts, core grants) is defined once, and each platform block supplies only its deltas — `argv_extra`, `env`, `devices`, `support`, and an `extends` pointer that inherits another platform's block and overrides a few fields. lemond flattens this to an explicit per-OS × accelerator view before validation; the terseness is author ergonomics, not a hiding of what the resolver sees. To make the matrix concrete, here is the `ds4` recipe (DwarfStar 4, serving DeepSeek V4 Flash via antirez/ds4 — a real backend, see [PR #3047](https://github.com/lemonade-sdk/lemonade/pull/3047)), condensed to the platform-matrix essentials. It is a `passthrough` native binary: ```json "platforms": { "base": { "argv": ["--model","{checkpoint:main}","--host","{host}","--port","{port}", "--ctx","{ctx_size}","{custom_args}"] }, "linux": { "rocm": { "extends":"base", "argv_extra":["--gpu-layers","{gpu_layers}"], "env": { "HIP_VISIBLE_DEVICES": "{hip_visible_devices}" }, "devices": ["/dev/kfd","/dev/dri"], "support": { "archs": ["gfx1151"] } }, "cuda": { "extends":"base", "argv_extra":["--gpu-layers","{gpu_layers}"], "env": { "CUDA_VISIBLE_DEVICES": "{cuda_visible_devices}" }, "devices": ["/dev/dri"] } }, "wsl2": { "extends":"linux" }, "windows_native": { "extends":"base", "sandbox": { "mode": "degraded" } } } ``` The `base` `argv` is written once; each platform supplies only its deltas (`argv_extra`, `env`, `devices`, `support`), and `extends` inherits another block. In practice the real `ds4` recipe publishes **only the `linux.rocm` row** (and, by `extends`, `wsl2`) and gates on `gfx1151`; the `cuda` / `windows_native` rows here are illustrative of how the matrix generalizes, and would be absent (or marked unsupported) for a backend that does not ship those targets. lemond flattens the matrix and reuses `check_recipe_supported` / `filter_models_by_backend` to enable or disable the recipe against the detected OS and GPU, exactly as it does for built-ins. ### 4.2 The zero-code fast path Many engines already speak a standard OpenAI-compatible HTTP surface and need no adapter at all. For those, `adapter.kind: "passthrough"` makes `lemond`'s own generic adapter do the work: launch the engine with a bounded argument template, FD-socket-custody + health-wait, then HTTP-passthrough standard endpoints. This was already demonstrated by the `ExternalBackendServer` prototype ([PR #2880](https://github.com/lemonade-sdk/lemonade/pull/2880)) running `dflash-rocm` (podman) and custom `llamacpp` recipes with **no adapter program** — just a manifest. Zero-code works when the only adaptation needed is a bounded, *named* set of known transforms (see §6). Engines that exceed that envelope — protocol translation (whisper), binary sniffing + multipart (trellis), out-of-band streaming (moonshine) — graduate to the external-adapter mode. Both modes are first-class and equally secure. **Containerized engines are the clearest zero-code case.** A container *is* the engine, so the manifest needs only to declare how to launch it, which capability endpoints to proxy, and where the model files are. The DFlash speculative- decoding server is a working example (containerized in the `ExternalBackendServer` POC); `zml/llmd` is analogous. A condensed example follows; the pattern is the same for any containerized OpenAI-compatible engine. Commands are typed argv arrays, never shell strings: ```json "adapter": { "kind": "passthrough" }, "engine": { "kind": "container", "runtime": "podman", "image": "ghcr.io/zml/llmd:latest", "devices": ["/dev/kfd", "/dev/dri"], "groups": ["render"], "mounts": [ { "host": "{hf_cache_models}", "container": "/models", "mode": "ro" } ], "network": "host", "argv": ["llmd", "--model", "/models/{checkpoint_relative:main}", "--fd", "{socket_fd}", "--ctx", "{ctx_size}", "{custom_args}"], "stop": { "command": "podman", "args": ["rm","-f","-t","0","lemonade-{recipe}-{port}"] } }, "capability_contracts": { "chat_completion": { "endpoint": "/v1/chat/completions" } }, "sandbox": { "note": "exempt_container_backend" } ``` `image`, `devices`, `groups`, `mounts`, `network`, and `argv` are structured fields lemond validates before invoking the runtime; `{custom_args}` are range-checked args appended to the array, not interpolated into a shell string. This is what prevents a recipe from escalating to `--privileged` or mounting the host root. Note that the example's `argv` uses **`--fd {socket_fd}`** — the ideal path where the engine can accept the leased socket descriptor passed by lemond (§5.6). Engines that cannot take an inherited FD instead receive `--port {port}` and use **loopback-port-with-peer-verification** (§5.6), where lemond verifies the connected peer is the spawned engine rather than trusting the numeric port. Because the container runtime provides the isolation boundary, the `sandbox` block is informational for containerized engines — lemond marks the recipe `exempt_container_backend` rather than double-wrapping `podman` in the OS sandbox. That means container backends are **consent-gated, not sandbox-contained**: the OS sandbox cannot confine a container (nono exempts container runtimes), so the security value sits entirely in the **admin-consent gate**. Each imported container recipe declares exactly what it wants — `network:host`, device nodes (`/dev/kfd`, `/dev/dri`), mounts, and any widening `--security-opt` (e.g. `seccomp=unconfined`) — and the admin approves that exact set before it can load. Consent must be **authenticated** (`LEMONADE_ADMIN_API_KEY`) and must show the *requested-and-enforced* capability set, not a vague "sandboxed" badge — especially on native Windows, where container backends ship unsandboxed. Container-exemption detection must use the structured `engine.kind: container` declaration, never a substring match on "podman"/"docker" in argv (which is spoofable and validates nothing). **When zero-code is not enough, the adapter path is a manifest change, not a new architecture.** An engine with a genuinely non-OpenAI protocol, request-time payload transformation beyond the named-transform registry, or an out-of-band streaming head selects `adapter.kind: "binary"` / `"container"` and ships an adapter program that owns the engine and does the translation behind the capability contract. Such an adapter honors a short boot contract: it is launched with the lease + launch context (model paths, ports, grants), pulls facts from lemond on demand, spawns or hosts the engine, translates each capability request into the engine's own protocol, and returns a typed result — all under the sandbox and lease, crash-isolated from lemond. Moving an engine between the zero-code and adapter paths is an `adapter` manifest field — the contract, sandbox, lease, and fact service are identical either way. ### 4.2.1 Binary-drop-in variants of a built-in backend (`variant_of`) A common and important plugin is a **fork / repackaged build of a backend that is already built into lemond** — a community llama.cpp with extra kernels (ROCmFPX), a vendor's nightly, or an "upstream vulkan" reference. These are *wire-compatible*: same `llama-server`, same OpenAI surface, same argv the built-in constructs. For these, re-declaring the full `argv` in the manifest is redundant and error-prone; the author only really needs to say *where to get the binary*. A dedicated flavor supports this: **`variant_of: ""`** — the plugin is a binary-drop-in for a named built-in backend, and lemond reuses that built-in's procedural handling (argv construction, reserved-arg validation, capability enable-args, platform env) against the **sandboxed external binary**. It inherits every declarative base too (options, lifecycle, capability contracts). The manifest reduces to binary provenance plus any argv deltas: ```json { "api_contract_version": "1", "recipe": "llamacpp-rocm-nightly", "adapter": { "kind": "passthrough" }, "variant_of": "llamacpp", "version_policy": "roll_forward", // follow github_latest, not pin (S5 opt-out) "engine": { "kind": "binary", "source": "https://github.com/lemonade-sdk/llamacpp-rocm/releases/download/{version}/llama-{version}-ubuntu-rocm-{arch}-x64.zip", "binary": "llama-server" }, "platforms": { "linux": { "rocm": { "support": { "archs": ["gfx1151","gfx1150","gfx1100"] } } } } } ``` - **The trust model is preserved.** The procedural logic stays in trusted, in-lemond C++ (`variant_of` points at a built-in whose argv builder runs unchanged); the fork binary remains an untrusted, sandboxed subprocess. lemond never trusts the binary — it constructs the argv it would have handed the built-in and confines the external process the same way. - **Distinct from the `_bin` config override.** `config set llamacpp._bin=` works today but is a global, mutable, single-slot config mutation with no sandbox, consent, reproducibility, or named coexistence. `variant_of` is a declarative, sandboxed, consent-able, version-pinned recipe; several forks can coexist as named recipes rather than overwriting one `_bin` slot. - **Distinct from declarative `extends`.** `extends` reuses the *declarative* base but still requires the author to write `argv`. `variant_of` additionally inherits the built-in's *procedural* argv handling, so `argv` can be omitted (or overridden per-platform via `argv_extra`). - **Version policy.** `variant_of` defaults to hash/version **pinning** (S5); a `roll_forward` (`github_latest`) policy is an explicit opt-out for authors who want to track a moving nightly. This is the one trade the "track the nightly" benchmark and marketplace workflows need. - **Honest caveat.** Full inheritance couples the fork to the built-in's current argv behavior — usually desired (the fork tracks upstream) but not always what the fork's binary expects. The manifest therefore allows per-platform `argv_extra` / `reserved_args` overrides on top of the inheritance. This is the mechanism that collapses the fork registry in the "Backend Battle Nightly" benchmark ([PR #3069](https://github.com/lemonade-sdk/lemonade/pull/3069)) — four llama.cpp forks, each a short `variant_of: "llamacpp"` manifest instead of an ad-hoc `config set` + retry dance — and it is exactly the "massively simplified by 2951" that the review there flagged. ### 4.3 The capability contract Closed, typed, versioned. The set of capability kinds maps to the existing `server_capabilities.h` interfaces (`chat_completion`, `completion`, `responses`, `embeddings`, `reranking`, `transcription`, `image_*`, `tts`, `audio_generation`, `model_3d_*`, classification). New kinds are added by a reviewed, **additive** contract change — a plugin cannot invent routes. **The transport is a first-class, essential spec (Phase-1b critical path), and binary payloads need a side channel.** Several things the capability contract needs but the design must not leave implicit: - **Transport and framing.** lemond↔adapter control/data rides a Unix-domain socket (or loopback on Windows), with explicit framing, message versioning, request-id multiplexing, cancel-vs-completion semantics, backpressure, and a max message size. RPC channel auth is peercred-PID-match plus a per-lease `0700` socket — peercred alone verifies the *uid*, not that the peer is *this* adapter. - **Binary payloads (audio, images) must not be base64-in-JSON.** Base64 inflates ~33% and forces full buffering, which kills streaming latency. Audio/image is carried over a shared-memory or FD-passing side channel alongside the RPC, not encoded in the JSON envelope. This is dictated by §13's streaming-latency requirement, and the transport RFC must cover it. The common control plane is UDS-everywhere where possible; the engines that cannot speak UDS (cpp-httplib llama.cpp/whisper) are exactly those forcing the loopback data plane (§5.6), so the honest posture is "UDS control plane + peercred-authenticated loopback data plane for legacy engines," not one assumed transport. --- ## 5. The five contract primitives The lemond↔adapter contract is exactly five primitives. Together they carry the whole of what a backend needs from the server and what the server needs from a backend: ### 5.1 Fact service — lemond as just-in-time fact authority lemond holds a trusted `SystemInfoCache` and `ModelInfo`. Rather than pre- compute and push every fact a plugin might need (which would force lemond to enumerate the universe of model/hardware state), lemond **answers on demand**: ``` PLUGIN -> { "op":"fact.get", "facts":["devices","igpu","system_info", "resolve_checkpoint"], "scope": {...} } LEMOND -> { "op":"fact.result", "facts":{ ... only the adapter's own slot ... } } ``` Read-only, **secret-free**, scoped to the adapter's own slot (no cross-model leak), authoritative. The plugin never peers raw `/sys`/`/proc`; it asks lemond. **Static vs. dynamic facts.** Two kinds of fact behave differently and are served differently: - **Static facts** (`igpu`, `rocm_arch`, `model_info`, `resolve_checkpoint`) are stable for a given model/device; the adapter may query them JIT at any time. - **Dynamic facts** (`vram_pressure`, memory pressure) are *not* safe to query in isolation — the world can change between the query and the adapter acting on it. These are returned **atomically inside the lease grant** (§5.2), so the adapter reasons over reserved state, not a stale snapshot. **Why this matters:** `supports_embeddings` / `supports_reranking` are just `model_info.type` (an enum lemond already holds); iGPU detection is `SystemInfo::get_has_igpu()`. These are facts lemond *already computes* — they live in C++ only because there was no channel to an out-of-process adapter. The fact service provides that channel. ### 5.2 Resource broker / lease protocol — single-writer coordination Admin approval is **consent to capability**; it is not **scheduling**. Even with perfect approval, two plugins cannot arbitrate against each other — only lemond sees all resident models, live device usage, and memory pressure. So arbitration is not a trust gate; it is a *coordination problem* no single plugin can solve locally. lemond becomes a **resource broker** that maps resources + config + preferences + realtime status into a **just-in-time entitlement** per plugin: ``` PLUGIN -> { "op":"lease.acquire", "want": {"slot_policy":"exclusive_npu", "devices":["npu0"], "mem_mb":0, "fd_sockets":1} } LEMOND -> { "op":"lease.grant", "lease_id":"…", "entitlement":{ "devices":["npu0"], "mem_mb":4096, "sockets": [ {"fd": 7, "uds":"/run/lemonade/rec-{lease_id}.sock"} ], "npu_exclusive":true, "ttl_sec":120, "dynamic_facts": {"vram_pressure":0.41, "reserved_mem_mb":4096} } } PLUGIN -> { "op":"lease.heartbeat" } / LEMOND -> { "op":"lease.revoke", "reason":"lru_evict|memory|admin", "grace_ms":5000 } ``` The grant carries the **leased socket FDs** (§5.6) and the **dynamic facts** (§5.1) atomically: the adapter acts on the exact reserved memory/device state, not on a value fetched earlier that may have changed. Reclamation is physical (revoke → grace → kill the PID namespace/cgroup/job (§7)), not negotiated. **Entitlements are cooperative over physical VRAM — state this honestly.** The lease is atomic over lemond's *own bookkeeping* (which adapter holds what), but lemond does not own the GPU: `mem_mb` has no kernel enforcement unless a delegated cgroup's `memory.max` exists, and an unrelated process can consume VRAM immediately after a grant. Where a delegated cgroup is available it is enforced; otherwise it is advisory. The lease protocol also needs **epochs / sequence numbers** (so heartbeat and revoke cannot cross on the wire), a rule for in-flight requests at revoke, and TTL auto-extension during a long generation. NPU exclusivity should reuse the router's existing path rather than being a second, parallel arbiter. ### 5.3 Model inventory RPC — "ask the backend for its models" For plugins that self-manage their model catalog, lemond must be able to relay user intent without downloading (this mirrors `BackendOps::discover_models`, which already exists): ``` LEMOND -> { "op":"model.inventory" } PLUGIN -> { "op":"model.inventory.result", "models":[{"id":"…","status":"ready"}] } LEMOND -> { "op":"model.pull", "model":"…" } // relay user intent ``` ### 5.4 Checkpoint-form declaration — imported-form + read grants "my model dir must contain X" is a *read-permission / declaration* problem, not a trust problem. The plugin declares its expected form (plain GGUF, a directory with `genai_config.json`, a `.onnx` file, an `index.json`); lemond grants a scoped read and verifies completeness; the plugin inspects itself within its grant. ### 5.5 Global response pipeline / safe-response-fields — the API edge The plugin already knows the local paths (it loaded the model); the question is only what the **client** sees. Today only `LlamaCppServer` normalizes the response `model` field (and only on the non-streaming path — the SSE stream passes the backend's own model string through), so the status quo is smaller and leakier than "one mechanical helper." This strengthens the case for a **global response pipeline** applied uniformly, driven by a per-capability "safe-response-fields" contract, and it must explicitly **cover the streaming path**, not just single JSON responses — otherwise streaming responses continue to leak local paths and secrets. This is a presentation concern at the API/telemetry edge, not a per-plugin trust gate. ### 5.6 And the two pieces that hold them together - **Socket custody via FD/UDS passing** — lemond owns the network edge and passes it to the adapter, rather than handing out a port number and checking reachability. It `bind()`s a Unix-domain socket (Linux/macOS) or a loopback listener (Windows) itself and passes the file descriptor to the adapter on spawn (the systemd `LISTEN_FDS` idiom). The adapter forwards that FD to its engine. This eliminates port collisions and TOCTOU races (lemond never trusts that "whoever bound 127.0.0.1:N" is the intended engine); custody is by FD, so there is no port to hijack. - **Honest primary path for the real fleet.** FD custody covers engines that accept an inherited listening socket (vLLM/uvicorn via `--uds`/`--fd`, podman socket-activation). Engines that expose only `--host`/`--port` (cpp-httplib llama.cpp, whisper.cpp — the majority) **cannot** take an inherited FD without patching upstream. For those, the shippable primary is **loopback-port-with-peer-verification**: lemond binds and forwards to `127.0.0.1:port`, and verifies the connected peer is the spawned engine via `SO_PEERCRED` / `LOCAL_PEERPID` / the Windows PID owner — not a bare reachability check. FD/UDS custody is then an *opportunistic* optimization, not the assumed default. - **No veth-in-host is needed (and is not safely buildable unprivileged).** Creating a veth pair into the host namespace needs `CAP_NET_ADMIN`, which an unprivileged `lemond` does not have; and unprivileged user namespaces are restricted on several distros. Where an isolated namespace is desired, use **pasta** (passt, the rootless-podman userspace networking default) or a lemond-proxied `socketpair` — neither requires `CAP_NET_ADMIN`. Evaluate user-namespace availability at runtime and degrade honestly. - **Platform mechanics:** FD/UDS passing is the Unix ideal; on Windows the equivalent is **handle inheritance** via `CreateProcess` with `PROC_THREAD_ATTRIBUTE_HANDLE_LIST` (or `WSADuplicateSocket`), not a raw FD. The architectural custody concept is identical; the OS API differs and is codified in the Phase 1b RFC. - **The capability contract** (§4.3) bounds public behavior. **No shell, ever.** Adapter and container commands are always launched from typed `argv` arrays (`command` + `args: string[]`) via `execvp`/`CreateProcess`, never from a shell-parsed string. Custom arguments are validated against a reserved-flag allowlist before reaching the engine, and container recipes separate the image, mounts, and devices as structured fields so a recipe cannot inject `--privileged`, host mounts, or shell operators through `{custom_args}`. ### 5.7 The capability RPC also carries request control and trace Two things the capability contract must carry that are easy to omit: - **Request cancellation (`cancel_request`).** LLM generation is expensive; on client disconnect the signal must propagate to the adapter immediately so it can drop the compute batch. The capability RPC includes a `cancel_request` message per in-flight request id, not just global teardown. - **Distributed trace propagation.** Because the adapter runs out-of-process, the RPC payloads carry trace context so lemond can stitch spans across the process boundary into OpenInference/OpenTelemetry. These follow the **W3C Trace Context** standard (`traceparent` / `tracestate`), so the Gateway's telemetry interoperates with standard collectors without custom mapping in the adapter. --- ## 6. The named-transform registry The zero-code adapter stays "data, not code" by exposing a small, **versioned, closed** set of named transforms lemond ships. A manifest *selects* a name (data); lemond *runs* the vetted implementation. Selecting a transform never supplies arbitrary code. **When is an adapter actually needed? Only when the adaptation is procedural.** The decision rule is: if a backend's needs are expressible as a *bounded set of known transformations* — field renames, prompt wrapping, image-defaults merges, size resolution, forced defaults — then the **named-transform registry covers it**, no adapter program is required, and the process tree stays at two tiers (lemond + sandboxed engine). A separate sandboxed **adapter program** is only introduced when the backend requires genuine *procedural computation* the registry deliberately does not express — bespoke protocol translation, binary MIME sniffing/multipart, out-of-band streaming, or request-time logic that can't be a named, bounded hook. So the registry is not a fallback set of happy-path transforms; it is the *default* answer, and the adapter is the exception reserved for open-ended procedural work. A healthy registry covers the common cases; the adapter exists only for the cases the registry is intentionally closed against. > This "covers most cases" is about the **population of out-of-tree engines a > user would plug in** — which are predominantly schema-conformant, > OpenAI-compatible wrappers — not about the 15 in-tree backends. Among those > 15, a large fraction are high-touch precisely *because* maintainers wrote > bespoke C++ for engines that don't fit a standard schema (§10); those stay > in-tree. The registry targets the easy majority of *new* engines; the > adapter targets the hard minority. The two are not in tension once "most > cases" is read as "most engines Lemonade is asked to add," not "most existing > built-ins." The named-transform registry is one of several **declarative surfaces** — it is not the whole of "no adapter." Many things that look like procedural logic are actually declarative, just not request/response rewriting. The other surfaces that keep a backend adapter-free: - **`capability_enable_args`** — a backend declares "I support *capability X*; enable it with *argument A*." lemond appends the arg when the model's own type needs it (matching `model_info.type`, which the recipe already knows). This is how `--embeddings` / `--reranking` resolve: they are a capability declaration plus an enable-arg, *not* runtime branching. The same field also carries unconditional runtime args (e.g. `--jinja --metrics`) — apply-always, a degenerate case. - **Recipe-declared backend args** — static per-recipe arguments the backend declares it wants. - **Advertised lifecycle capabilities** — e.g. `downsize: supported` plus a known backend operation to invoke; "supports downsize" is a boolean the backend advertises, and lemond orchestrates the fixed op. - **`reserved_args` merge validation** — because lemond is already the one merging `{custom_args}` into the argv array, the reserved set is a validation rule in the merger: lemond rejects a user arg that collides with `--port` / `-m` before the array is built. Declarative schema + a reject rule, not adapter logic. - **Per-platform env / install / support** — already covered by the platform matrix and the no-managed-install path (library paths become platform `env` entries; per-arch builds via `support`). The unifying line: **an adapter is needed only for open-ended procedural computation** — bespoke protocol translation, binary MIME sniffing + multipart, out-of-band streaming, or arbitrary per-request logic that no bounded declarative surface can express. If a backend's needs are expressible as a combination of these declarative fields (named transforms, `capability_enable_args`, recipe args, lifecycle capabilities, merge validation, platform env), no adapter program is required. This is the *correct* and complete statement of the rule; the named-transform registry is the request/response slice of it. Every entry below is grounded in a real C++ backend, not invented: | name | kind | purpose | source impl | |---|---|---|---| | `legacy_max_tokens` | request | alias `max_completion_tokens`→`max_tokens` | `json_utils.cpp:69` | | `sanitize_grammar_bounds` | request | strip oversized min/max Length/Items from tool schemas (llama #17473) | `llamacpp_request.cpp:98` | | `fit_max_tokens` | round-trip | pre-fit `max_tokens` via `/tokenize` | `vllm_server.cpp` | | `echo_request_model` | response | `response.model` = request model | `llamacpp_server.cpp:621` | | `image_defaults` | request | merge steps/guidance_scale/sampler/seed | `thenoise_server.cpp:207` | | `resolve_size` | request | `"WxH"`/`w`/`h` → width/height | TheNoise `:191` | | `image_prompt_xml` | request | wrap params into `` | `sdcpp_server.cpp:403` | | `field_map` | request | rename fields (`prompt`→`caption`) | ThinkSound | | `default_field` | request | force fields (kokoro `model`/`stream`) | Kokoro | | `model_field_rewrite` | request | set model to backend checkpoint name | FLM | Deferred to the **external-adapter** mode (not zero-code): whisper multipart translation, trellis MIME sniffing + multipart, moonshine 3-port streaming head, SD ESRGAN secondary-subprocess upscale, and any custom per-engine rewrite beyond the closed set. ### What keeps the registry non-Turing-complete - **Closed, versioned, additive** vocabulary. - **Intent-only, no control flow** — a transform rewrites a structured message; it cannot loop, branch arbitrarily, or open endpoints. - **Vetted implementation in lemond** — the manifest never supplies code. - **Ordered + scoped per capability**, individually consentable and auditable. The registry is deliberately non-scripting: there is **no** generalized scripting, only named, bounded hooks over real implementations drawn from today's backends. The "Turing-complete JSON" failure mode is avoided by construction. --- ## 7. Security model - **Sandbox wraps the outermost untrusted process** — the engine for built-in/passthrough, the adapter (which contains the engine) for external-adapter plugins (see §4.0 tier table). Filesystem + network confinement via kernel allow-lists, with **default-deny network egress**: Linux **Landlock** + seccomp-bpf (`no_new_privs`), macOS **Seatbelt** (`sandbox_init`), and Windows via **WSL2** (Landlock against the WSL2 Linux kernel). The sandbox engine is **nono**, which is the natural cross-platform home (its `--block-net` / network allow-list realizes the default-deny egress). Nono is invoked one of two ways (see §13): - **Embedded via C FFI (preferred).** Compile the nono Rust library into `lemond` and call `Sandbox::apply()` through its C FFI bindings, using the **fork + sandbox + exec** pattern: lemond `fork()`s, the child applies the capability set (generated in-process from the manifest) and `exec()`s the engine. Sandbox primitives apply to the calling process and its descendants, so the sandbox must be applied in the child, never to lemond itself. This collapses the tree from `lemond → nono → engine` to `lemond → engine` (no supervising wrapper process), removes the separate nono binary as a runtime dependency, and keeps lemond self-contained. - **CLI wrapper (fallback).** Spawn `nono` as a supervising wrapper around the engine, for platforms/arches without a vendored nono library. Native-Windows kernel sandboxing is not currently offered by nono (WSL2-only); that remains a follow-on platform decision. The irony worth stating: the `dlopen` proposal removes the exact boundary the WG exists to create. Out-of-process is not a nice-to-have — **it is the security model.** - **Teardown must reap the whole tree, not just the immediate child.** `PR_SET_PDEATHSIG` and process-group teardown alone will leak a grandchild (e.g. the adapter crashes and its `llama-server` gets reparented to init, silently holding VRAM). Teardown is an **orchestrator** responsibility layered on the sandbox: on Linux and WSL2 it uses **PID namespaces + cgroup v2** (or, in the container path, runc already provides them); a native-Windows path would use **Job Objects** with kill-on-close if/when that path exists. Reclamation kills the namespace/cgroup/job, so the kernel reaps every descendant — no VRAM zombies. - **Secret isolation as a prerequisite.** Scrub `LEMONADE_*` + `LEMONADE__API_KEY` from child env by default; pass only an explicit `env_allowlist`; keep runtime cloud keys in lemond's memory only. This applies to built-ins too, independent of plugin work. - **Secrets are a filesystem concern too, not just an env one.** A read grant on a cache *root* can hand a sandboxed backend credential files that live there (e.g. `~/.cache/huggingface/token`, `stored_tokens`; other model repos keep their credentials similarly). The model read grant must therefore be scoped to the specific checkpoint file or the snapshot subtree, never a parent directory that also holds credentials. Because grants are default-deny, correct scoping excludes credential files by construction — no per-repo deny-list is needed. Env scrubbing alone does not close this: an over-broad grant gives the secret back through the filesystem. - **Default-deny grants, honest declaration.** The manifest's `sandbox` block is what the kernel enforces; an over-privileged recipe is more visible and more easily rejected. - **Consent is an authenticated action, and re-consent on widening.** lemond is headless and multi-client, so "the user reviews and approves grants" needs a home: approval is a `LEMONADE_ADMIN_API_KEY`-authenticated action, which protects against remote and accidental installs (it is *not* protection against same-user local malware, which lies outside any sandbox's remit). Any update that **widens a grant** (`sandbox`, `facts_scope`, `network`) triggers **mandatory re-consent**. The consent UI shows the **enforced-not-declared** posture per host — because on some targets (native Windows, container backends) the OS sandbox is not the containment, and the user must see what is actually enforced, not what the manifest merely declares. Optional publisher signing layers over the sha256 TOFU pin. The FactService is itself a *security improvement*: because lemond answers facts from its trusted cache, the untrusted adapter never needs raw `/sys`/`/proc` access — a smaller sandbox grant, not a larger one. --- ## 8. Trust boundary: what stays in lemond, what moves to the adapter Every adaptation concern in a backend resolves into one of three classes, and that classification determines where the concern lives: - **Class A — Stays in lemond (architectural, not per-backend).** Secrets custody (**A1** provider/cloud keys held only by lemond, handed out as scoped handles; **A6** backend-needed credentials delivered via explicit allowlist, never ambient); global coordination (**A2** — resource arbitration/scheduling is the broker's single-writer job, §5.2); process/API-edge custody (**A5** — lemond owns the socket/process/response boundary and reclamation, §7) + the sandbox. - **Class B — Moves to the adapter once lemond exposes a fact channel.** Hardware and model metadata, answered by lemond's FactService (§5.1). This is how "hardware introspection" and "model capabilities" — things every backend needs to decide flags — become pluggable without exposing raw `/sys`/`/proc`. - **Class C — Moves to the adapter (mechanical).** Field maps, prompt wrap, multipart, image-defaults, companion-file discovery, job-poll loops, streaming shape. Two specific cases are worth stating plainly. **Resource arbitration is coordination, not a gate.** Admin approval is consent to *capability*; it is not *scheduling*. Even fully-approved plugins cannot arbitrate against each other — only lemond sees all resident models, live device use, and memory pressure. So arbitration is about scheduling, and it is structurally impossible for any single plugin to do locally. lemond remains the broker (entitlements, leases, eviction) whether a backend is in-tree or a plugin. **argv integrity is the plugin's job; output presentation is lemond's edge.** lemond does not build the adapter's engine argv — the plugin does; lemond holds socket custody (FD/UDS passed on spawn, §5.6) and the plugin validates its own engine's reserved flags. Output sanitization (hiding local paths from clients) is a global response-pipeline step, applied at the API/telemetry edge. The consequence: **the only things that keep a backend in lemond are secrets and process/API-edge custody + the sandbox — both orthogonal to whether a backend is in-tree or pluggable.** Architecture, not trust, stays in lemond. --- ## 9. Lifecycle and recipe intersection Mapping the 15 in-tree backends (plus the containerized prototype) against the ideal `WrappedServer` lifecycle (`LOADING → READY ⇄ IN_USE → DOWNSIZING/DOWNSIZED → EVICTING → UNLOADED`) reveals a small set of **orthogonal lifecycle axes** that actually differ: - **Spawn shape** — single-subprocess / multi-port (Moonshine: 3) / secondary-subprocess (SD: ESRGAN upscale) / none (Cloud). - **Readiness** — `/health` default vs. custom (FLM → `/api/tags`). - **Model ownership** — lemond-downloaded / self-managed (FLM) / no-local-model (Cloud). - **Hardware conditionality** — at load (llama iGPU mmap) vs. at request. - **Downsize/restore** — only llama.cpp has real KV-erase via `/slots`. - **Request adaptation** — field-remap / prompt-wrap / multipart / sniff / job-poll / model-normalize. - **Streaming** — SSE / byte-stream / non-SSE / TCP out-of-band. - **Imported-checkpoint form** — plain / `genai_config.json` / `.onnx` / `index.json` / companion `.gguf`. **The key finding:** most lifecycle is **generic and reusable** (port lease, spawn, health, LRU/eviction, watchdog, state machine, secret scrubbing, sandbox). The per-backend delta is a short, enumerable list, and that list is the plugin contract surface. The motivating question is therefore answerable in bounded terms: *which lifecycle axes must a plugin express?* — not an open-ended *can the hardware-conditional C++ of a given backend be turned into data at all?* (it cannot, which is why the adapter is code). ### 9.1 Where recipes constrain the lifecycle Recipes (`server_models.json`) and the backend descriptors jointly decide what a load can do, and a plugin contract must honor these junctions: - **Checkpoint types are the recipe↔lifecycle junction.** Checkpoints are not a single `main` file: recipes carry `main`, `draft` (spec-decode), `mmproj` (vision), `npu_cache` (whisper NPU device cache), and `text_encoder` + `vae` (SD/split models). A recipe's checkpoint set determines which artifacts lemond provisions *before* spawn and which argv shape results. - **`self_manages_downloads` flips provisioning direction.** For FLM, the backend owns its model catalog; lemond must not try to download, and must tolerate the model appearing only after the backend pulls it. The plugin contract needs an explicit "model readiness" handshake here, not just a health probe. This is distinct from **"no local model" (Cloud)**, which has no checkpoints to fetch at all — it does *not* set `self_manages_downloads`. - **Egress ⇄ download validation rule.** A recipe that declares `self_manages_downloads: true` *must* also declare outbound network rights, because it has to fetch its own model; lemond rejects a manifest as invalid otherwise. This rule applies to *self-managed-download* backends only — a no-local-model backend (Cloud) must **not** be granted host egress on the strength of being "self-managed," because it isn't. - **Egress must be a host allowlist, not a bare bool.** A self-managed backend that gets `allow_egress: true` holds *both* outbound network and read access to the weights — exfiltrating them is then within policy. Replace the coarse boolean with an **enforced egress host allowlist** (e.g. `huggingface.co`, `cdn-lfs.huggingface.co`), realized via a host-filtering proxy or pasta, and acknowledge the residual risk in the doc. - **`slot_policy`** (`standard` / `exclusive_npu` / `coexist_by_type` / `unmetered`) is the router-level resource contract. A plugin recipe maps to one of these so eviction and slot logic stay in the trusted router. - **`dynamic_models`** decouples "what models exist" from `server_models.json` (FLM provider catalog, Cloud discovery). A plugin declaring it supplies the model-inventory hook (§5.3). - **`model_reads` is a plugin-declared format contract, not just a `form` tag.** A recipe lists the checkpoint roles it reads and their required-vs-optional status, and — for GGUF — the quant families it can read (`supported_quants`). This is *declarative*, not procedural: it lets lemond filter or reject models by format at model-cache build / load time (reusing `filter_models_by_backend`), so a plugin is never asked to serve a checkpoint format it cannot read. A community llama.cpp fork that adds new GGUFs (e.g. ROCmFPX quants upstream llama.cpp cannot read) declares exactly which families it supports; the vanilla backend does not list them, so a ROCmFPX-quantized model is only offered to the fork. See `example-rocmfpx-passthrough.json` and `plugin-model-recipe-interaction.md` for the concrete shape and the related gaps (plugin-named roles, typed options, provisioning hooks). These junctions drive the manifest's `model_management`, `model_reads`, and `lifecycle` fields (§4.1). ### 9.2 Design consequences - **The launch bundle carries the full resolved checkpoint set, not just a single `main` path.** Multi-role artifacts and compound models are the norm. - **A plugin declares its lifecycle "personality" up front** — spawn shape, readiness type, ownership direction, downsize support, adaptation kind, streaming shape, checkpoint form — because that is the manifest's real content (it feeds the consent dialog and the sandbox grant). - **Provisioning is directional** — lemond-owned (download then hand paths) vs. self-managed (wait for the backend to provision) — handled by the model- inventory RPC and a model-readiness handshake, not a single download path. - **Multi-port and secondary-subprocess backends are declared properties**, not hidden assumptions (the base `WrappedServer` assumes a single port; Moonshine and SD's ESRGAN upscale prove that is a simplification). --- ## 10. Candidacy: which backends could be plugins ### 10.0 How existing backends map Assessed against the eight lifecycle axes (§9), the 15 in-tree backends (plus the containerized prototype) fall into three natural tiers. The distribution is the point: a solid minority are clean, mechanical plugins; the flagship workhorses are not, by design. | Candidacy | Backends | |---|---| | **High** | TheNoise, OpenMoss, ThinkSound, Kokoro, Moonshine, containerized (born-plugins) | | **Medium** | OnnxRuntime, Whisper, stable-diffusion, AceStep, Trellis, RyzenAI, FastFlowLM, vLLM | | **Low** | llama.cpp, Cloud (no subprocess, architecturally lemond's) | **Natively pluggable (High):** TheNoise (image-defaults merge, size resolve, `/text2image` body), OpenMoss (spawn + `.extras.gguf` companion discovery), ThinkSound (field-map + `dit/t5/vae` companions), Kokoro (`index.json` resolution + forced `model`/`stream`), Moonshine (multi-port + TCP streaming), and containerized recipes (born plugins). These are mechanical, self-contained adaptation — the natural prototype starting point. **Bounded adaptation (Medium):** OnnxRuntime (exact-one `.onnx` selection), Whisper (multipart protocol translation is pluggable, while the `.rai` NPU-cache fetch, its path-traversal guards, and NPU/CPU switching stay in the coordinator), stable-diffusion (prompt-XML wrap + split/single branching are pluggable; the secondary ESRGAN upscale subprocess is a two-spawn shape), AceStep (async `/synth`+`/lm` job poll), Trellis (binary MIME sniffing + multipart), RyzenAI (spawn + passthrough; the `ExclusiveNpu` policy is router enforcement, not a backend-body property), FastFlowLM (self-managed provision + NPU validate + custom readiness, driven by the model-inventory and fact primitives), vLLM (quant detection + ROCm shim + CWSR door + `max_tokens` pre-fit — expressible as facts + plugin-owned argv + mechanical transforms, kept in-tree mainly for risk). **Kept in-tree (Low):** llama.cpp (flagship; iGPU-mmap, capability flags, reserved-flag handling, real KV-erase downsize) — and Cloud (no subprocess; secrets custody makes it architecturally lemond's by design). "Kept in-tree" here means kept as a coordinator-managed `WrappedServer` for stability, not that it is trusted: its engine is sandboxed like every other engine (see §4.0), so it earns no trust exemption from being built in. Note on llama.cpp: each of its ostensibly "high-touch" pieces is actually a **declarative surface** (§6) — `--embeddings`/`--reranking` are capability enable-args matched to `model_info.type`, `--load-mode none` is a per-device default, reserved-flag handling is `reserved_args` merge validation, and KV-erase downsize is an advertised lifecycle capability. So llama.cpp is **expressible as a zero-code passthrough**; it sits in the Low row purely as a **flagship-stability choice**, not because the Gateway cannot express it. ### 10.1 Candidacy is a portfolio choice, not a capability limit The tier a backend lands in reflects *how much it is worth externalizing*, not *whether the Gateway can express it*. The five contract primitives (§5) — the fact service, the lease broker, the model-inventory RPC, the checkpoint-form declaration, and the response pipeline — cover the concerns that had kept the "heavier" backends from looking pluggable at all: - Self-managed model ownership (FLM) → a declared **model-inventory RPC**. - Imported-checkpoint form (`.onnx` dir / `genai_config.json` / `index.json`) → a **checkpoint-form declaration** with a scoped read grant. - llama reserved-flag collision → a **`reserved_args` merge-validation rule** (§6); lemond rejects colliding user args before building the argv array, holding **socket custody** (FD/UDS, §5.6). - iGPU mmap-disable → a **per-device default** (`--load-mode none` on iGPU), decided from a hardware fact or user override — not plugin logic. - Local-path hiding → a **global response pipeline** step. - vLLM quant / shim / CWSR / `max_tokens` → **facts** + plugin-owned argv + mechanical transforms. - NPU exclusivity and eviction → the **lease broker** (coordination, not trust). - Cloud secrets → **secret custody**, which is architecturally lemond's by design. The genuinely inexpressible set is therefore very small — secrets and process/API-edge custody + the sandbox, none of which is per-backend. Under this lens, RyzenAI, FastFlowLM, and vLLM resolve to Medium (their earlier "Low" was largely a lack of the right contract primitives, not a hard capability limit). The only backends that remain "Low" are llama.cpp (kept in-tree for flagship stability, not capability) and Cloud (no subprocess; architecturally lemond's by design). Every remaining choice about externalizing a backend is about how much it buys today (flagship stability, install burden, concentrated domain knowledge), not about whether the Gateway can express it. ### 10.2 Migration does not gate adoption This is a design rule, stated once and firmly: > **Adopting the Gateway and porting existing in-tree backends are independent > decisions. Neither is a prerequisite for the other.** - The Gateway is **purely additive** — one new `GatewayServer` class plus an out-of-tree contract. It removes, rewrites, or gates nothing. - In-tree backends remain coordinator-managed C++ **indefinitely by default**, with their engines sandboxed under the same zero-trust baseline as plugins (§4.0). "Stays in-tree" is a packaging choice, not a trust claim. - Migration is **opportunistic, cost-benefit-driven, and reversible** — never required for adoption. - Adoption is proven by **one non-trivial plugin working well**, not by migrating the fleet. **Coexistence is the intended end state.** The strongest objection to out-of- process plugins assumed that adopting them forces the whole fleet to be re-expressed. It does not. The Gateway is adopted on its own merits, alongside the existing in-tree backends (which keep their engine sandboxes either way), and each backend is moved to plugin form only if and when that pays off. --- ## 11. Mapping onto the existing codebase **What stays:** the capability interfaces (`server_capabilities.h`) as the capability vocabulary; subprocess orchestration (spawn, port, health, watchdog, eviction, `PR_SET_PDEATHSIG`) in `WrappedServer`; slot policy, auth/CORS/ quad-prefix, client API; install/version-pin flow. **What changes:** one generic `GatewayServer : WrappedServer` parses the manifest, launches the adapter under the sandbox, and maps capability RPC to the `I*Server` interfaces. `ExternalBackendServer` ([PR #2880](https://github.com/lemonade-sdk/lemonade/pull/2880)) is a degenerate case where the adapter is pure HTTP passthrough. Built-ins stay as coordinator-managed `WrappedServer` C++, upgraded to enforce secret scrubbing and engine sandboxing (zero-trust baseline, §4.0). The marketplace distributes manifest + pinned adapter artifact (binary/container + sha256). **The simplification dividend:** lemond stops owning every engine's adaptation logic. It owns one small, stable protocol and one sandbox/lifecycle engine. --- ## 12. Comparison of the options | Axis | JSON descriptors | In-process dlopen | Capability Gateway | |---|---|---|---| | Procedural adaptation | ✗ | ✓ | ✓ (in adapter) | | Trust/privilege boundary | ✓ out-of-process | ✗ same as lemond | ✓ sandboxed subprocess | | Crash isolation | ✓ | ✗ | ✓ | | Hardenable by kernel sandbox | ✓ | ✗ (bypasses it) | ✓ (sandbox is the adapter) | | No mandatory host runtime | ✓ | ✓ | ✓ (binary or container) | | Marketplace submission friction | low | ~ (ABI/build) | ✓ (pinned binary/image + tiny manifest) | | New capability kinds | needs lemond changes | can drift | additive reviewed change | | Secret isolation | needs work | needs work + unsafe | designed in | --- ## 13. What is genuinely open Honest risks and decisions remain, to be resolved in the RFC process: - **Adapter ecosystem bootstrap.** Zero-code handles the common path; the first real external adapter is a larger authoring step than a one-line descriptor. The named-transform registry + passthrough template mitigates this. - **Protocol stability/versioning.** The capability contract needs an ABI/transport freeze before marketplace adoption. - **Cross-platform sandbox posture and scope.** The *sandbox boundary* (filesystem + network confinement) is available on all three platforms via `nono`: Linux **Landlock**, macOS **Seatbelt** (a *deprecated* Apple API — works but unsupported upstream), and Windows via **WSL2** (Landlock; feature coverage depends on the WSL2 kernel — "~85%" holds only for the stock 6.6 kernel, and rises with a rolling kernel; native Windows unsupported). Linux is the mature first target. **Landlock's network limits must be stated honestly:** its net rules are port-scoped, not address-scoped ("allow the control port" allows it to any host), cover **TCP only** (UDP/ICMP exfiltration sails through), and the network hooks are absent entirely below kernel 6.7 / Landlock ABI v4 (so "default-deny egress via Landlock" is simply unavailable on Ubuntu 22.04's 5.15). Where possible, prefer **UDS-everywhere** plus a seccomp `AF_INET`/`AF_INET6` deny for clean, address/ABI-independent egress-default-deny, and reserve Landlock-net for the genuine outbound-TCP case. **Sandbox invocation** is itself a decision (§7): prefer **embedded via C FFI** (fork + sandbox + exec, no supervising wrapper, self-contained lemond), keeping the **CLI wrapper** only as a fallback for platforms without a vendored nono library. Two things are deliberately treated as **orchestrator extensions**, not sandbox primitives, and are outside the initial sandbox work: **process-tree teardown / resource reclamation** (the broker's job; see §7 — solvable per-platform, no off-the-shelf primitive on macOS) and **GPU custody / VRAM allocation** (the resource broker's job, §5.2 — the sandbox grants the device, the broker decides which device and how much). The Windows posture is WSL2-gated; native Windows sandboxing is a follow-on platform decision. The consent UI must reflect **enforced-not-declared** posture per host, since native Windows and container backends ship unsandboxed. - **Performance of the extra loopback hop.** Negligible for token streaming; benchmarks needed for streaming TTS/audio. - **Supply chain for binaries.** Pin + sha256 + consent baseline; optional code signing later; sandbox is the primary protection. - **Wrapping vs. owning the engine.** Recommendation: adapter owns/spawns the engine and presents one face to lemond. Open RFC questions (from the trust-boundary work + review): who is authoritative when a fact and a plugin observation disagree (recommend lemond); FD/UDS socket custody semantics and the fallback path for engines that cannot take an inherited FD; "safe-response-fields" granularity; lease/revocation protocol; whether the fact service needs a dry-run/preview mode; and how to propagate `cancel_request` and trace context across the RPC that the process boundary unavoidably introduces. --- ## 14. Suggested roadmap 1. **Adopt the process-boundary framing in the charter** — replace "declarative JSON descriptors as the plugin" with "manifest + sandboxed adapter program behind a versioned capability contract." 2. **Land the secret-scrubbing + default-deny baseline** for all backend subprocesses, independent of plugin work — the highest-value security fix. 3. **Prototype `GatewayServer`** + one real adapter (start with a mostly- mechanical backend: TheNoise → OpenMoss → a containerized recipe, each proving one primitive) to prove adaptation + sandbox together, and to exercise the fact service (a fact-driven `igpu`/embedding decision). 4. **RFCs:** Phase-1a manifest schema; Phase-1b capability contract + transport; Phase-1c fact service + lease protocol (the broker); Phase-2 model inventory + checkpoint-form + response pipeline. --- ## 15. Verification appendix Two things this proposal is often judged on: whether it holds up against the concerns raised during the charter review, and whether its claims about the existing prototype are accurate. Both are grounded in the primary sources (review threads and the actual branch code), not assumed. ### 15.1 Design constraints surfaced by the charter review The Working Group charter ([PR #2951](https://github.com/lemonade-sdk/lemonade/pull/2951)) review raised a small set of substantive concerns that this design must meet. They are stated here as design constraints: | Concern | What the design must satisfy | |---|---| | JSON can't express arbitrary backend behavior | A plugin must carry real procedural logic (the adapter program), not a JSON template that would degenerate into a scripting language. | | In-process code inherits lemond's privileges | Untrusted plugin code must run out-of-process, under the sandbox, never in lemond's memory space. | | New capability kinds / endpoints | The public surface stays closed; new capabilities are additive, reviewed contract changes, not plugin-opened routes. | | Marketplace / build friction | Distribution must not require a complex plugin-authoring build; pinned binaries/containers + a tiny manifest keep friction low. | | Model/preparation ownership | With self-managed backends, lemond relays user intent and lets the backend own its catalog (a declared inventory contract), rather than fighting over downloads. | | Secret handling for model pulls | Backends that legitimately need a token receive it via an explicit, scoped allowlist — never ambient environment. | This document's premise directly follows from these constraints: the trust boundary sits at the process boundary, lemond is a fact authority + resource broker + API presenter, and adaptation is code that runs sandboxed. ### 15.2 Prototype & sandbox reality check (read the code, not the prose) These findings come from reading the prototype branch ([`feat/custom_backends`](https://github.com/lemonade-sdk/lemonade/tree/feat/custom_backends) and [`feat/custom_backends_nono`](https://github.com/lemonade-sdk/lemonade/tree/feat/custom_backends_nono)) directly: - **`ExternalBackendServer` is a passthrough envelope, not an adapter.** Its capability methods are thin `forward_request(path, request)` calls; it spawns the external binary, expands a bounded token map at launch, health-probes, then dumbly proxies HTTP. No payload rewrite, no protocol translation, no request-time introspection. This confirms the key review concern and marks the exact missing layer the Gateway's adapter program supplies (out-of-process, under the sandbox). - **Env scrubbing is real in the POC.** `build_sanitized_env` (`process_linux.cpp`) filters `LEMONADE_*`, `*_API_KEY`, `*_TOKEN`, `*_SECRET`, `*_PASS`, `*_AUTH`, and undeclared GPU-variability vars from the child env. On the branch its reach is uneven by platform: Linux is always scrubbed; Windows scrubs only when extra env is passed (a bare spawn inherits the full environment); macOS skips it on the `posix_spawn` path used for backends. The filter is also substring-based (`API_KEY`, `TOKEN`, ...), which over-scrubs (e.g. `TOKENIZERS_PARALLELISM`) and can under-scrub creatively named secrets — arguing for the manifest's `env_allowlist` model. - **Sandboxing wires by default in the branch.** `ProcessManager::start_process` resolves a `SandboxPolicy` from config when none is passed; `nono` read/write grants, device nodes, `--allow-port`, `--block-net`, and NPU sysfs grants are built; container backends (podman/docker) are auto-exempted. `process_linux.cpp` adds `setpgid` + whole-process-group teardown. Taken together, the prototype proves out-of-process + sandbox + secret-scrubbing are viable; it just lacks the adapter-translation leg and the contextual pull protocol the Gateway adds. This supports (rather than undercuts) the direction. --- ## 17. Glossary Terms and codes used in this document, defined for a standalone reader. | Term | Definition | |---|---| | **lemond** | The Lemonade HTTP server process (the executable `lemond`); the trusted coordinator this design keeps under its control. "Trusted" refers to lemond's own code (secret custody, arbitration, the process/API edge) — the smallest unavoidable trusted core — never to the backends it manages, whose engines are all sandboxed regardless of origin (§4.0). | | **adapter** (plugin) | The sandboxed, out-of-process program that owns a backend engine's adaptation. `adapter.kind` is `binary`/`container` (a shipped program) or `passthrough` (no program — lemond's own generic/handling adapter). | | **`passthrough`** | The `adapter.kind` value meaning "no shipped adapter program is needed": lemond's own launcher + HTTP proxy using launch-time tokens and/or `variant_of` inherited handling — the zero-code path where lemond launches a standard-HTTP engine and proxies capability endpoints to it. Not a built-in `WrappedServer` — lemond's *generic* adapter. | | **`variant_of`** | A binary-drop-in plugin flavor (§4.2.1): the recipe inherits a named built-in backend's procedural argv/handling against a sandboxed external binary, so the manifest is mostly binary provenance. | | **`version_policy`** | How a recipe's binary is versioned: `pin` (hash/version, the secure default) or `roll_forward` (follow `github_latest`) — an explicit opt-out for "track the nightly" workflows. | | **named-transform registry** | The bounded, versioned set of vetted request/response rewrite hooks lemond ships (§6); one of the declarative surfaces, not the whole of "no adapter." | | **capability_enable_args** | Manifest field declaring "I support capability X; enable it with argument A" (model-conditioned via `model_info.type`, or unconditional). | | **fact service** | lemond's read-only, scoped, secret-free introspection channel (§5.1); static vs dynamic facts (§5.1). | | **lease / entitlement** | The broker's grant of a resource to an adapter (§5.2): device, memory, socket FDs, exclusivity, TTL, and dynamic facts. Cooperative over physical VRAM (see §13). | | **FD/UDS socket custody** | lemond binds the socket and passes the file descriptor/FD to the adapter (§5.6); the no-port-to-hijack mechanism. | | **nono** | The kernel-sandbox engine (nono.sh, from the nolabs project): Landlock on Linux, Seatbelt on macOS, Landlock-under-WSL2 on Windows. lemond delegates filesystem/network confinement to it. Invoked either embedded via C FFI (`Sandbox::apply`, fork + sandbox + exec, preferred) or as a CLI wrapper (fallback) — see §7/§13. | | **A1 / A6** | Class-A secret-custody codes: provider/cloud keys held only by lemond (A1); backend-needed credentials delivered via explicit allowlist (A6), never ambient. | | **A2** | Class-A resource-coordination code — scheduling/arbitration is the broker's single-writer job (§5.2), not a plugin gate. | | **A5** | Class-A process/API-edge custody — lemond owns the socket/process/response boundary and teardown (§7). | | **CWSR door** | A gfx1151 AMD workaround in the vLLM backend: `needs_gfx1151_cwsr_fix()` enriches a startup timeout error; *distinct* from `--enforce-eager` (keyed on non-discrete-HBM arch). | | **ROCm shim** | A generated `sitecustomize.py` that vLLM's ROCm path injects (via `PYTHONPATH`) to prevent its CUDA NVML probe from faulting on AMD-only systems. | | **cross-platform posture** | Linux = full sandbox (Landlock); macOS = Seatbelt (deprecated Apple API, works); Windows = WSL2-only (native Windows sandbox unsupported); §13.