The Lemonade Backend Gateway

A First-Principles Design for Secure, Pluggable Inference Backends

Status: R&D design proposal — not yet an RFC, not an implementation.

The design laid out here is grounded in how Lemonade's backends actually behave, in the capabilities already present in the codebase, and in the positions of the maintainers who reviewed the Working Group charter. It is intended to be read as one coherent, self-contained proposal.

1. Why this exists

Lemonade runs inference by spawning backends as subprocesses (llama-server, whisper-server, sd-server, flm, vllm, and others). Today, adding a backend means writing a new C++ WrappedServer class and recompiling lemond. That limits Lemonade to in-tree backends, slows adoption of new inference engines, and forces maintainers to build, pin, and ship every engine for every platform and GPU vendor.

At the same time, letting users run third-party or community binaries raises a real security problem: an untrusted binary that runs as a child of lemond with the same user, full network access, and the server's environment can read model files, reach the network, and exfiltrate API keys that happen to be in the environment.

The Backend Plugins & Sandboxing Working Group exists to resolve exactly this tension: make Lemonade open enough to run any engine, and safe enough that doing so does not compromise the machine. This document is a first-principles proposal for how to meet both goals.

2. The options considered

There are three natural ways to make a backend pluggable. Each takes a different position on where procedural adaptation logic lives, and each has hard limits. They are evaluated here on their own terms.

Option 1 — Declarative descriptors (backend as data). Express a backend as a JSON manifest: identity, capabilities, an argument template, a health probe. Simple to write and submit. The limit is fundamental: the interesting part of a backend is procedural. Hardware detection, model-metadata introspection, conditional argument construction, and payload rewriting cannot be expressed as data without the data becoming a scripting language. Whatever cannot be declared leaks back into lemond's C++, so the "escape hatch" reintroduces exactly the code the format was meant to avoid.

Option 2 — In-process binary plugins (dlopen). Load the backend's integration as a shared library inside lemond. Fully expressive as real code. The limit is security: it runs inside lemond's memory space with lemond's full privileges, which defeats the sandboxing that running third-party binaries demands, and a fault in the plugin takes the server down with it.

Option 3 — Out-of-process adapter program (this proposal). Keep the adaptation as real code, but run it as a separate, sandboxed peer process behind a narrow contract. It is expressive (code, like Option 2) and isolated (sandboxable and crash-proof, like Option 1 was intended to be), without either option's limitation.

Options 1 and 2 both decide the question by where the code goes — data in lemond's interpreter, or code in lemond's process. Both, for opposite reasons, leave untrusted behavior inside lemond's trust domain. The fix is to change what they are optimizing:

2.1 The trust boundary is the process boundary

Draw the trust boundary at the process boundary.

A backend is really three concerns glued together in one C++ class today: (1) a declarative identity (BackendDescriptor), (2) arbitrary procedural adaptation (argv construction, hardware detection, payload rewriting, protocol translation), and (3) shared subprocess + HTTP orchestration (spawn, port lease, health, watchdog, eviction) that WrappedServer already provides.

Option 1 pushes concern (2) into data and fails; Option 2 pushes (2) into lemond's process and is unsafe. This proposal keeps (2) as real code, but runs it in its own sandboxed process, drawing the line at that process boundary:

A fault in an adapter cannot take lemond down. An adapter cannot reach beyond what its sandbox grant allows or read secrets lemond chooses not to hand it. And because the adapter runs behind a wire contract, the language it is written in is irrelevant to lemond — a static binary (C/Rust/Go) or a container, whichever fits the engine.

The result satisfies expressiveness and isolation at once: the sandbox becomes the security boundary, and the adapter becomes whatever code an engine really needs.

3. Requirements, derived from first principles

3.1 Functional needs

3.2 Security needs (from the actual threat surface)

Auditing today's code shows there is currently no OS sandboxing for backend processes and they inherit lemond's full environment. The real leaks to close:

3.3 The synthesis

Needs N1+N2+N3 resolve to "the plugin carries its own procedural logic as a self-contained, sandbox-capable subprocess." N4+N5 resolve to "lemond describes that program declaratively and exposes only a closed, typed contract to it." S1–S5 resolve to "the sandbox wraps the outermost untrusted process — the engine for built-in/passthrough, the adapter (which contains the engine) for external-adapter plugins — with env scrubbing and default-deny grants."

4. The shape: the "Capability Gateway"

lemond (trusted core)

1. launch: context + grants

2. capability RPC over local socket

3. health / liveness / teardown

owned subprocess or embedded engine

Closed capability contracts (typed)

Fact authority + resource broker

Process / API edge custodian

Adapter program (the PLUGIN)
sandboxed subprocess
arbitrary adaptation, owns engine argv/spawn
Landlock/seccomp · no ambient env

Engine
e.g. llama-server, a container

4.0 Sandboxing vs. the adapter tier (two independent decisions)

Two things are easy to conflate and must be kept separate:

So the process tree is:

Kind Process tree Sandboxed?
Built-in backend lemond → engine engine sandboxed
Passthrough plugin lemond → engine engine sandboxed
External-adapter plugin lemond → adapter → engine adapter and engine sandboxed

The sandbox boundary sits at the engine (the untrusted thing) in every case; the adapter tier is added only when an untrusted adapter owns the adaptation. This keeps "even built-in engines are sandboxed" as a stated guarantee while avoiding an unconditional 2→3 tier expansion.

4.1 The manifest

Declarative and small — it describes where the adapter is and what it may do, never how. Procedural intelligence is not here.

{
  "api_contract_version": "1",
  "recipe": "my-engine-custom",
  "display_name": "My Engine (Custom)",
  "extends": "llamacpp",             // optional: inherit base recipe's options/contracts
  "adapter": {
    "kind": "binary",                 // "binary" | "container" | "passthrough"
    "source": "https://cdn.example/adapter/linux-x64/my-engine-adapter",
    "sha256": "9f3a…c2",
    "version": "1.2.3"
  },
  "engine": {
    "reserved_args": ["--port", "--host", "-m"]   // reject user args colliding with these at merge
  },
  "capabilities": ["chat_completion", "completion", "embeddings"],
  "capability_enable_args": {           // "I support X; enable it with arg A"
    "embeddings": ["--embeddings"],     // appended when model_info.type == embedding
    "completion":   ["--jinja", "--metrics"]   // unconditional runtime args (always)
  },
  "capability_contracts": {
    "chat_completion": { "response_fields": { "model": {"mode":"echo_request_model","sensitive":false} } }
  },
  "slot_policy": "standard",          // standard | exclusive_npu | coexist_by_type | unmetered
  "model_management": "lemond_managed", // or "self_managed"
  "model_reads": { "main": { "form": "gguf" } },
  "lifecycle": { "spawn_shape": "single", "requested_ports": 1,
                 "readiness": {"type":"http","endpoint":"/health"},
                 "self_manages_downloads": false },
  "sandbox": {
    "read_paths": ["{hf_cache_models}", "{binary_dir}"],  // scoped to models subtree, NOT cache root (D1)
    "write_paths": ["{scratch_dir}"],
    "devices": ["/dev/dri"],
    "network": { "allow_egress": false, "allow_loopback": true },
    "env_allowlist": ["GGML_VK_VISIBLE_DEVICES"]
  },
  "facts_scope": ["devices","igpu","rocm_arch","resolve_checkpoint","model_info","vram_pressure"]
}

The schema has two adapter flavors. passthrough (§4.2) reuses an existing binary/container with a manifest only; binary / container ships an adapter program for engines that exceed the passthrough envelope.

The manifest is "data, not code," but with one deliberate, bounded exception: launch-time token substitution. The argv / command arrays may contain tokens ({checkpoint:main}, {port}, {ctx_size}, {custom_args}) that lemond substitutes once, when constructing the spawn — this is how lemond hands the engine its model path, port, and user-option values. That is the sanctioned, declarative subset: tokens exist to launch the engine, and they appear only in the manifest's argv/engine blocks, never in request-time logic.

What is barred is request-time adaptation in data: no request_transform field maps, no per-request payload rewriting expressed in JSON, no scripting. Those are what the earlier proposals tried to force into data and what degenerated into a Turing-complete language. Request-time adaptation lives in the named-transform registry (§6, bounded, vetted) or — when genuinely open-ended — in an adapter program (§4.2, out-of-process sandboxed). So the statement "no command-template scripting" is accurate; the launch-time {token} substitution is the one declarative exception, and it is only ever about how the engine is started.

Token vocabulary (launch-time only, resolved by lemond):

Token Meaning
{checkpoint:role} / {checkpoint_relative:role} resolved checkpoint path (absolute / lexical-relative) for a recipe role (main, draft, mmproj, ...)
{hf_cache_models} the cache models subtree (hub/models--…), the scope a model read grant should use — never the cache root, which also holds credential files (D1)
{port} / {host} leased port / host for the engine (FD/socket custody or fallback, §5.6)
{ctx_size}, {threads}, {gpu_layers}, ... declared option values (defaults live in options)
{socket_fd} the leased socket descriptor, when the engine accepts an inherited FD
{custom_args} validated user-supplied arguments, appended to the argv array
{env:VAR} forbidden — leaks lemond's environment into child argv and bypasses the scrubber

Tokens are substituted only in the argv/engine blocks at spawn; the env block values may also reference token values but never read lemond's ambient environment. See §6 for the separate request/response surface.

The manifest only declares identity, capability claims, lifecycle/ownership (driven by the recipe junctions of §9.1), and the sandbox grants plus fact scope — precisely what a consent dialog, a kernel sandbox, and the FactService need. The schema above shows both adapter flavors; the capability_contracts, model_management, model_reads, lifecycle, and facts_scope fields are covered in §4.3, §5, and §9.

The optional platforms block is the OS × accelerator matrix. It declares which host/device combinations the recipe runs on and what differs per combination — GPU device nodes, accelerator env vars, arch gates, and any security-posture override (e.g. a wsl2 vs windows_native distinction). Platform gating reuses the same machinery lemond applies to built-ins (BackendSupport, SystemInfo::check_recipe_supported, ModelManager::filter_models_by_backend), so lemond enables/disables recipes against the detected OS and GPU at model-cache build time, exactly as it does today. The matrix is DRY by construction: the base recipe (options, lifecycle, capability contracts, core grants) is defined once, and each platform block supplies only its deltas — argv_extra, env, devices, support, and an extends pointer that inherits another platform's block and overrides a few fields. lemond flattens this to an explicit per-OS × accelerator view before validation; the terseness is author ergonomics, not a hiding of what the resolver sees.

To make the matrix concrete, here is the ds4 recipe (DwarfStar 4, serving DeepSeek V4 Flash via antirez/ds4 — a real backend, see PR #3047), condensed to the platform-matrix essentials. It is a passthrough native binary:

"platforms": {
  "base": {
    "argv": ["--model","{checkpoint:main}","--host","{host}","--port","{port}",
             "--ctx","{ctx_size}","{custom_args}"]
  },
  "linux": {
    "rocm": { "extends":"base",
              "argv_extra":["--gpu-layers","{gpu_layers}"],
              "env": { "HIP_VISIBLE_DEVICES": "{hip_visible_devices}" },
              "devices": ["/dev/kfd","/dev/dri"],
              "support": { "archs": ["gfx1151"] } },
    "cuda": { "extends":"base",
              "argv_extra":["--gpu-layers","{gpu_layers}"],
              "env": { "CUDA_VISIBLE_DEVICES": "{cuda_visible_devices}" },
              "devices": ["/dev/dri"] }
  },
  "wsl2": { "extends":"linux" },
  "windows_native": { "extends":"base", "sandbox": { "mode": "degraded" } }
}

The base argv is written once; each platform supplies only its deltas (argv_extra, env, devices, support), and extends inherits another block. In practice the real ds4 recipe publishes only the linux.rocm row (and, by extends, wsl2) and gates on gfx1151; the cuda / windows_native rows here are illustrative of how the matrix generalizes, and would be absent (or marked unsupported) for a backend that does not ship those targets. lemond flattens the matrix and reuses check_recipe_supported / filter_models_by_backend to enable or disable the recipe against the detected OS and GPU, exactly as it does for built-ins.

4.2 The zero-code fast path

Many engines already speak a standard OpenAI-compatible HTTP surface and need no adapter at all. For those, adapter.kind: "passthrough" makes lemond's own generic adapter do the work: launch the engine with a bounded argument template, FD-socket-custody + health-wait, then HTTP-passthrough standard endpoints. This was already demonstrated by the ExternalBackendServer prototype (PR #2880) running dflash-rocm (podman) and custom llamacpp recipes with no adapter program — just a manifest.

Zero-code works when the only adaptation needed is a bounded, named set of known transforms (see §6). Engines that exceed that envelope — protocol translation (whisper), binary sniffing + multipart (trellis), out-of-band streaming (moonshine) — graduate to the external-adapter mode. Both modes are first-class and equally secure.

Containerized engines are the clearest zero-code case. A container is the engine, so the manifest needs only to declare how to launch it, which capability endpoints to proxy, and where the model files are. The DFlash speculative- decoding server is a working example (containerized in the ExternalBackendServer POC); zml/llmd is analogous. A condensed example follows; the pattern is the same for any containerized OpenAI-compatible engine. Commands are typed argv arrays, never shell strings:

"adapter": { "kind": "passthrough" },
"engine": {
  "kind": "container",
  "runtime": "podman",
  "image": "ghcr.io/zml/llmd:latest",
  "devices": ["/dev/kfd", "/dev/dri"],
  "groups": ["render"],
  "mounts": [ { "host": "{hf_cache_models}", "container": "/models", "mode": "ro" } ],
  "network": "host",
  "argv": ["llmd", "--model", "/models/{checkpoint_relative:main}",
           "--fd", "{socket_fd}", "--ctx", "{ctx_size}", "{custom_args}"],
  "stop": { "command": "podman", "args": ["rm","-f","-t","0","lemonade-{recipe}-{port}"] }
},
"capability_contracts": { "chat_completion": { "endpoint": "/v1/chat/completions" } },
"sandbox": { "note": "exempt_container_backend" }

image, devices, groups, mounts, network, and argv are structured fields lemond validates before invoking the runtime; {custom_args} are range-checked args appended to the array, not interpolated into a shell string. This is what prevents a recipe from escalating to --privileged or mounting the host root.

Note that the example's argv uses --fd {socket_fd} — the ideal path where the engine can accept the leased socket descriptor passed by lemond (§5.6). Engines that cannot take an inherited FD instead receive --port {port} and use loopback-port-with-peer-verification (§5.6), where lemond verifies the connected peer is the spawned engine rather than trusting the numeric port.

Because the container runtime provides the isolation boundary, the sandbox block is informational for containerized engines — lemond marks the recipe exempt_container_backend rather than double-wrapping podman in the OS sandbox. That means container backends are consent-gated, not sandbox-contained: the OS sandbox cannot confine a container (nono exempts container runtimes), so the security value sits entirely in the admin-consent gate. Each imported container recipe declares exactly what it wants — network:host, device nodes (/dev/kfd, /dev/dri), mounts, and any widening --security-opt (e.g. seccomp=unconfined) — and the admin approves that exact set before it can load. Consent must be authenticated (LEMONADE_ADMIN_API_KEY) and must show the requested-and-enforced capability set, not a vague "sandboxed" badge — especially on native Windows, where container backends ship unsandboxed. Container-exemption detection must use the structured engine.kind: container declaration, never a substring match on "podman"/"docker" in argv (which is spoofable and validates nothing).

When zero-code is not enough, the adapter path is a manifest change, not a new architecture. An engine with a genuinely non-OpenAI protocol, request-time payload transformation beyond the named-transform registry, or an out-of-band streaming head selects adapter.kind: "binary" / "container" and ships an adapter program that owns the engine and does the translation behind the capability contract. Such an adapter honors a short boot contract: it is launched with the lease + launch context (model paths, ports, grants), pulls facts from lemond on demand, spawns or hosts the engine, translates each capability request into the engine's own protocol, and returns a typed result — all under the sandbox and lease, crash-isolated from lemond. Moving an engine between the zero-code and adapter paths is an adapter manifest field — the contract, sandbox, lease, and fact service are identical either way.

4.2.1 Binary-drop-in variants of a built-in backend (variant_of)

A common and important plugin is a fork / repackaged build of a backend that is already built into lemond — a community llama.cpp with extra kernels (ROCmFPX), a vendor's nightly, or an "upstream vulkan" reference. These are wire-compatible: same llama-server, same OpenAI surface, same argv the built-in constructs. For these, re-declaring the full argv in the manifest is redundant and error-prone; the author only really needs to say where to get the binary.

A dedicated flavor supports this: variant_of: "<recipe>" — the plugin is a binary-drop-in for a named built-in backend, and lemond reuses that built-in's procedural handling (argv construction, reserved-arg validation, capability enable-args, platform env) against the sandboxed external binary. It inherits every declarative base too (options, lifecycle, capability contracts). The manifest reduces to binary provenance plus any argv deltas:

{
  "api_contract_version": "1",
  "recipe": "llamacpp-rocm-nightly",
  "adapter": { "kind": "passthrough" },
  "variant_of": "llamacpp",
  "version_policy": "roll_forward",          // follow github_latest, not pin (S5 opt-out)
  "engine": {
    "kind": "binary",
    "source": "https://github.com/lemonade-sdk/llamacpp-rocm/releases/download/{version}/llama-{version}-ubuntu-rocm-{arch}-x64.zip",
    "binary": "llama-server"
  },
  "platforms": { "linux": { "rocm": { "support": { "archs": ["gfx1151","gfx1150","gfx1100"] } } } }
}

This is the mechanism that collapses the fork registry in the "Backend Battle Nightly" benchmark (PR #3069) — four llama.cpp forks, each a short variant_of: "llamacpp" manifest instead of an ad-hoc config set + retry dance — and it is exactly the "massively simplified by 2951" that the review there flagged.

4.3 The capability contract

Closed, typed, versioned. The set of capability kinds maps to the existing server_capabilities.h interfaces (chat_completion, completion, responses, embeddings, reranking, transcription, image_*, tts, audio_generation, model_3d_*, classification). New kinds are added by a reviewed, additive contract change — a plugin cannot invent routes.

The transport is a first-class, essential spec (Phase-1b critical path), and binary payloads need a side channel. Several things the capability contract needs but the design must not leave implicit:

The common control plane is UDS-everywhere where possible; the engines that cannot speak UDS (cpp-httplib llama.cpp/whisper) are exactly those forcing the loopback data plane (§5.6), so the honest posture is "UDS control plane + peercred-authenticated loopback data plane for legacy engines," not one assumed transport.

5. The five contract primitives

The lemond↔adapter contract is exactly five primitives. Together they carry the whole of what a backend needs from the server and what the server needs from a backend:

5.1 Fact service — lemond as just-in-time fact authority

lemond holds a trusted SystemInfoCache and ModelInfo. Rather than pre- compute and push every fact a plugin might need (which would force lemond to enumerate the universe of model/hardware state), lemond answers on demand:

PLUGIN ->  { "op":"fact.get", "facts":["devices","igpu","system_info",
                                        "resolve_checkpoint"], "scope": {...} }
LEMOND ->  { "op":"fact.result", "facts":{ ... only the adapter's own slot ... } }

Read-only, secret-free, scoped to the adapter's own slot (no cross-model leak), authoritative. The plugin never peers raw /sys//proc; it asks lemond.

Static vs. dynamic facts. Two kinds of fact behave differently and are served differently:

Why this matters: supports_embeddings / supports_reranking are just model_info.type (an enum lemond already holds); iGPU detection is SystemInfo::get_has_igpu(). These are facts lemond already computes — they live in C++ only because there was no channel to an out-of-process adapter. The fact service provides that channel.

5.2 Resource broker / lease protocol — single-writer coordination

Admin approval is consent to capability; it is not scheduling. Even with perfect approval, two plugins cannot arbitrate against each other — only lemond sees all resident models, live device usage, and memory pressure. So arbitration is not a trust gate; it is a coordination problem no single plugin can solve locally. lemond becomes a resource broker that maps resources + config + preferences + realtime status into a just-in-time entitlement per plugin:

PLUGIN ->  { "op":"lease.acquire", "want": {"slot_policy":"exclusive_npu",
             "devices":["npu0"], "mem_mb":0, "fd_sockets":1} }
LEMOND ->  { "op":"lease.grant", "lease_id":"…", "entitlement":{
             "devices":["npu0"], "mem_mb":4096,
             "sockets": [ {"fd": 7, "uds":"/run/lemonade/rec-{lease_id}.sock"} ],
             "npu_exclusive":true, "ttl_sec":120,
             "dynamic_facts": {"vram_pressure":0.41, "reserved_mem_mb":4096} } }
PLUGIN ->  { "op":"lease.heartbeat" }  /  LEMOND -> { "op":"lease.revoke",
             "reason":"lru_evict|memory|admin", "grace_ms":5000 }

The grant carries the leased socket FDs (§5.6) and the dynamic facts (§5.1) atomically: the adapter acts on the exact reserved memory/device state, not on a value fetched earlier that may have changed. Reclamation is physical (revoke → grace → kill the PID namespace/cgroup/job (§7)), not negotiated.

Entitlements are cooperative over physical VRAM — state this honestly. The lease is atomic over lemond's own bookkeeping (which adapter holds what), but lemond does not own the GPU: mem_mb has no kernel enforcement unless a delegated cgroup's memory.max exists, and an unrelated process can consume VRAM immediately after a grant. Where a delegated cgroup is available it is enforced; otherwise it is advisory. The lease protocol also needs epochs / sequence numbers (so heartbeat and revoke cannot cross on the wire), a rule for in-flight requests at revoke, and TTL auto-extension during a long generation. NPU exclusivity should reuse the router's existing path rather than being a second, parallel arbiter.

5.3 Model inventory RPC — "ask the backend for its models"

For plugins that self-manage their model catalog, lemond must be able to relay user intent without downloading (this mirrors BackendOps::discover_models, which already exists):

LEMOND -> { "op":"model.inventory" }
PLUGIN -> { "op":"model.inventory.result", "models":[{"id":"…","status":"ready"}] }
LEMOND -> { "op":"model.pull", "model":"…" }   // relay user intent

5.4 Checkpoint-form declaration — imported-form + read grants

"my model dir must contain X" is a read-permission / declaration problem, not a trust problem. The plugin declares its expected form (plain GGUF, a directory with genai_config.json, a .onnx file, an index.json); lemond grants a scoped read and verifies completeness; the plugin inspects itself within its grant.

5.5 Global response pipeline / safe-response-fields — the API edge

The plugin already knows the local paths (it loaded the model); the question is only what the client sees. Today only LlamaCppServer normalizes the response model field (and only on the non-streaming path — the SSE stream passes the backend's own model string through), so the status quo is smaller and leakier than "one mechanical helper." This strengthens the case for a global response pipeline applied uniformly, driven by a per-capability "safe-response-fields" contract, and it must explicitly cover the streaming path, not just single JSON responses — otherwise streaming responses continue to leak local paths and secrets. This is a presentation concern at the API/telemetry edge, not a per-plugin trust gate.

5.6 And the two pieces that hold them together

No shell, ever. Adapter and container commands are always launched from typed argv arrays (command + args: string[]) via execvp/CreateProcess, never from a shell-parsed string. Custom arguments are validated against a reserved-flag allowlist before reaching the engine, and container recipes separate the image, mounts, and devices as structured fields so a recipe cannot inject --privileged, host mounts, or shell operators through {custom_args}.

5.7 The capability RPC also carries request control and trace

Two things the capability contract must carry that are easy to omit:

6. The named-transform registry

The zero-code adapter stays "data, not code" by exposing a small, versioned, closed set of named transforms lemond ships. A manifest selects a name (data); lemond runs the vetted implementation. Selecting a transform never supplies arbitrary code.

When is an adapter actually needed? Only when the adaptation is procedural. The decision rule is: if a backend's needs are expressible as a bounded set of known transformations — field renames, prompt wrapping, image-defaults merges, size resolution, forced defaults — then the named-transform registry covers it, no adapter program is required, and the process tree stays at two tiers (lemond + sandboxed engine). A separate sandboxed adapter program is only introduced when the backend requires genuine procedural computation the registry deliberately does not express — bespoke protocol translation, binary MIME sniffing/multipart, out-of-band streaming, or request-time logic that can't be a named, bounded hook. So the registry is not a fallback set of happy-path transforms; it is the default answer, and the adapter is the exception reserved for open-ended procedural work. A healthy registry covers the common cases; the adapter exists only for the cases the registry is intentionally closed against.

This "covers most cases" is about the population of out-of-tree engines a user would plug in — which are predominantly schema-conformant, OpenAI-compatible wrappers — not about the 15 in-tree backends. Among those 15, a large fraction are high-touch precisely because maintainers wrote bespoke C++ for engines that don't fit a standard schema (§10); those stay in-tree. The registry targets the easy majority of new engines; the adapter targets the hard minority. The two are not in tension once "most cases" is read as "most engines Lemonade is asked to add," not "most existing built-ins."

The named-transform registry is one of several declarative surfaces — it is not the whole of "no adapter." Many things that look like procedural logic are actually declarative, just not request/response rewriting. The other surfaces that keep a backend adapter-free:

The unifying line: an adapter is needed only for open-ended procedural computation — bespoke protocol translation, binary MIME sniffing + multipart, out-of-band streaming, or arbitrary per-request logic that no bounded declarative surface can express. If a backend's needs are expressible as a combination of these declarative fields (named transforms, capability_enable_args, recipe args, lifecycle capabilities, merge validation, platform env), no adapter program is required. This is the correct and complete statement of the rule; the named-transform registry is the request/response slice of it.

Every entry below is grounded in a real C++ backend, not invented:

name kind purpose source impl
legacy_max_tokens request alias max_completion_tokens→max_tokens json_utils.cpp:69
sanitize_grammar_bounds request strip oversized min/max Length/Items from tool schemas (llama #17473) llamacpp_request.cpp:98
fit_max_tokens round-trip pre-fit max_tokens via /tokenize vllm_server.cpp
echo_request_model response response.model = request model llamacpp_server.cpp:621
image_defaults request merge steps/guidance_scale/sampler/seed thenoise_server.cpp:207
resolve_size request "WxH"/w/h → width/height TheNoise :191
image_prompt_xml request wrap params into <sd_cpp_extra_args> sdcpp_server.cpp:403
field_map request rename fields (prompt→caption) ThinkSound
default_field request force fields (kokoro model/stream) Kokoro
model_field_rewrite request set model to backend checkpoint name FLM

Deferred to the external-adapter mode (not zero-code): whisper multipart translation, trellis MIME sniffing + multipart, moonshine 3-port streaming head, SD ESRGAN secondary-subprocess upscale, and any custom per-engine rewrite beyond the closed set.

What keeps the registry non-Turing-complete

The registry is deliberately non-scripting: there is no generalized scripting, only named, bounded hooks over real implementations drawn from today's backends. The "Turing-complete JSON" failure mode is avoided by construction.

7. Security model

The FactService is itself a security improvement: because lemond answers facts from its trusted cache, the untrusted adapter never needs raw /sys//proc access — a smaller sandbox grant, not a larger one.

8. Trust boundary: what stays in lemond, what moves to the adapter

Every adaptation concern in a backend resolves into one of three classes, and that classification determines where the concern lives:

Two specific cases are worth stating plainly.

Resource arbitration is coordination, not a gate. Admin approval is consent to capability; it is not scheduling. Even fully-approved plugins cannot arbitrate against each other — only lemond sees all resident models, live device use, and memory pressure. So arbitration is about scheduling, and it is structurally impossible for any single plugin to do locally. lemond remains the broker (entitlements, leases, eviction) whether a backend is in-tree or a plugin.

argv integrity is the plugin's job; output presentation is lemond's edge. lemond does not build the adapter's engine argv — the plugin does; lemond holds socket custody (FD/UDS passed on spawn, §5.6) and the plugin validates its own engine's reserved flags. Output sanitization (hiding local paths from clients) is a global response-pipeline step, applied at the API/telemetry edge.

The consequence: the only things that keep a backend in lemond are secrets and process/API-edge custody + the sandbox — both orthogonal to whether a backend is in-tree or pluggable. Architecture, not trust, stays in lemond.

9. Lifecycle and recipe intersection

Mapping the 15 in-tree backends (plus the containerized prototype) against the ideal WrappedServer lifecycle (LOADING → READY ⇄ IN_USE → DOWNSIZING/DOWNSIZED → EVICTING → UNLOADED) reveals a small set of orthogonal lifecycle axes that actually differ:

The key finding: most lifecycle is generic and reusable (port lease, spawn, health, LRU/eviction, watchdog, state machine, secret scrubbing, sandbox). The per-backend delta is a short, enumerable list, and that list is the plugin contract surface. The motivating question is therefore answerable in bounded terms: which lifecycle axes must a plugin express? — not an open-ended can the hardware-conditional C++ of a given backend be turned into data at all? (it cannot, which is why the adapter is code).

9.1 Where recipes constrain the lifecycle

Recipes (server_models.json) and the backend descriptors jointly decide what a load can do, and a plugin contract must honor these junctions:

These junctions drive the manifest's model_management, model_reads, and lifecycle fields (§4.1).

9.2 Design consequences

10. Candidacy: which backends could be plugins

10.0 How existing backends map

Assessed against the eight lifecycle axes (§9), the 15 in-tree backends (plus the containerized prototype) fall into three natural tiers. The distribution is the point: a solid minority are clean, mechanical plugins; the flagship workhorses are not, by design.

Candidacy Backends
High TheNoise, OpenMoss, ThinkSound, Kokoro, Moonshine, containerized (born-plugins)
Medium OnnxRuntime, Whisper, stable-diffusion, AceStep, Trellis, RyzenAI, FastFlowLM, vLLM
Low llama.cpp, Cloud (no subprocess, architecturally lemond's)

Natively pluggable (High): TheNoise (image-defaults merge, size resolve, /text2image body), OpenMoss (spawn + .extras.gguf companion discovery), ThinkSound (field-map + dit/t5/vae companions), Kokoro (index.json resolution + forced model/stream), Moonshine (multi-port + TCP streaming), and containerized recipes (born plugins). These are mechanical, self-contained adaptation — the natural prototype starting point.

Bounded adaptation (Medium): OnnxRuntime (exact-one .onnx selection), Whisper (multipart protocol translation is pluggable, while the .rai NPU-cache fetch, its path-traversal guards, and NPU/CPU switching stay in the coordinator), stable-diffusion (prompt-XML wrap + split/single branching are pluggable; the secondary ESRGAN upscale subprocess is a two-spawn shape), AceStep (async /synth+/lm job poll), Trellis (binary MIME sniffing + multipart), RyzenAI (spawn + passthrough; the ExclusiveNpu policy is router enforcement, not a backend-body property), FastFlowLM (self-managed provision + NPU validate + custom readiness, driven by the model-inventory and fact primitives), vLLM (quant detection + ROCm shim + CWSR door + max_tokens pre-fit — expressible as facts + plugin-owned argv + mechanical transforms, kept in-tree mainly for risk).

Kept in-tree (Low): llama.cpp (flagship; iGPU-mmap, capability flags, reserved-flag handling, real KV-erase downsize) — and Cloud (no subprocess; secrets custody makes it architecturally lemond's by design). "Kept in-tree" here means kept as a coordinator-managed WrappedServer for stability, not that it is trusted: its engine is sandboxed like every other engine (see §4.0), so it earns no trust exemption from being built in.

Note on llama.cpp: each of its ostensibly "high-touch" pieces is actually a declarative surface (§6) — --embeddings/--reranking are capability enable-args matched to model_info.type, --load-mode none is a per-device default, reserved-flag handling is reserved_args merge validation, and KV-erase downsize is an advertised lifecycle capability. So llama.cpp is expressible as a zero-code passthrough; it sits in the Low row purely as a flagship-stability choice, not because the Gateway cannot express it.

10.1 Candidacy is a portfolio choice, not a capability limit

The tier a backend lands in reflects how much it is worth externalizing, not whether the Gateway can express it. The five contract primitives (§5) — the fact service, the lease broker, the model-inventory RPC, the checkpoint-form declaration, and the response pipeline — cover the concerns that had kept the "heavier" backends from looking pluggable at all:

The genuinely inexpressible set is therefore very small — secrets and process/API-edge custody + the sandbox, none of which is per-backend. Under this lens, RyzenAI, FastFlowLM, and vLLM resolve to Medium (their earlier "Low" was largely a lack of the right contract primitives, not a hard capability limit). The only backends that remain "Low" are llama.cpp (kept in-tree for flagship stability, not capability) and Cloud (no subprocess; architecturally lemond's by design). Every remaining choice about externalizing a backend is about how much it buys today (flagship stability, install burden, concentrated domain knowledge), not about whether the Gateway can express it.

10.2 Migration does not gate adoption

This is a design rule, stated once and firmly:

Adopting the Gateway and porting existing in-tree backends are independent decisions. Neither is a prerequisite for the other.

Coexistence is the intended end state. The strongest objection to out-of- process plugins assumed that adopting them forces the whole fleet to be re-expressed. It does not. The Gateway is adopted on its own merits, alongside the existing in-tree backends (which keep their engine sandboxes either way), and each backend is moved to plugin form only if and when that pays off.

11. Mapping onto the existing codebase

What stays: the capability interfaces (server_capabilities.h) as the capability vocabulary; subprocess orchestration (spawn, port, health, watchdog, eviction, PR_SET_PDEATHSIG) in WrappedServer; slot policy, auth/CORS/ quad-prefix, client API; install/version-pin flow.

What changes: one generic GatewayServer : WrappedServer parses the manifest, launches the adapter under the sandbox, and maps capability RPC to the I*Server interfaces. ExternalBackendServer (PR #2880) is a degenerate case where the adapter is pure HTTP passthrough. Built-ins stay as coordinator-managed WrappedServer C++, upgraded to enforce secret scrubbing and engine sandboxing (zero-trust baseline, §4.0). The marketplace distributes manifest + pinned adapter artifact (binary/container + sha256).

The simplification dividend: lemond stops owning every engine's adaptation logic. It owns one small, stable protocol and one sandbox/lifecycle engine.

12. Comparison of the options

Axis JSON descriptors In-process dlopen Capability Gateway
Procedural adaptation ✗ ✓ ✓ (in adapter)
Trust/privilege boundary ✓ out-of-process ✗ same as lemond ✓ sandboxed subprocess
Crash isolation ✓ ✗ ✓
Hardenable by kernel sandbox ✓ ✗ (bypasses it) ✓ (sandbox is the adapter)
No mandatory host runtime ✓ ✓ ✓ (binary or container)
Marketplace submission friction low ~ (ABI/build) ✓ (pinned binary/image + tiny manifest)
New capability kinds needs lemond changes can drift additive reviewed change
Secret isolation needs work needs work + unsafe designed in

13. What is genuinely open

Honest risks and decisions remain, to be resolved in the RFC process:

Open RFC questions (from the trust-boundary work + review): who is authoritative when a fact and a plugin observation disagree (recommend lemond); FD/UDS socket custody semantics and the fallback path for engines that cannot take an inherited FD; "safe-response-fields" granularity; lease/revocation protocol; whether the fact service needs a dry-run/preview mode; and how to propagate cancel_request and trace context across the RPC that the process boundary unavoidably introduces.

14. Suggested roadmap

  1. Adopt the process-boundary framing in the charter — replace "declarative JSON descriptors as the plugin" with "manifest + sandboxed adapter program behind a versioned capability contract."
  2. Land the secret-scrubbing + default-deny baseline for all backend subprocesses, independent of plugin work — the highest-value security fix.
  3. Prototype GatewayServer + one real adapter (start with a mostly- mechanical backend: TheNoise → OpenMoss → a containerized recipe, each proving one primitive) to prove adaptation + sandbox together, and to exercise the fact service (a fact-driven igpu/embedding decision).
  4. RFCs: Phase-1a manifest schema; Phase-1b capability contract + transport; Phase-1c fact service + lease protocol (the broker); Phase-2 model inventory
    • checkpoint-form + response pipeline.

15. Verification appendix

Two things this proposal is often judged on: whether it holds up against the concerns raised during the charter review, and whether its claims about the existing prototype are accurate. Both are grounded in the primary sources (review threads and the actual branch code), not assumed.

15.1 Design constraints surfaced by the charter review

The Working Group charter (PR #2951) review raised a small set of substantive concerns that this design must meet. They are stated here as design constraints:

Concern What the design must satisfy
JSON can't express arbitrary backend behavior A plugin must carry real procedural logic (the adapter program), not a JSON template that would degenerate into a scripting language.
In-process code inherits lemond's privileges Untrusted plugin code must run out-of-process, under the sandbox, never in lemond's memory space.
New capability kinds / endpoints The public surface stays closed; new capabilities are additive, reviewed contract changes, not plugin-opened routes.
Marketplace / build friction Distribution must not require a complex plugin-authoring build; pinned binaries/containers + a tiny manifest keep friction low.
Model/preparation ownership With self-managed backends, lemond relays user intent and lets the backend own its catalog (a declared inventory contract), rather than fighting over downloads.
Secret handling for model pulls Backends that legitimately need a token receive it via an explicit, scoped allowlist — never ambient environment.

This document's premise directly follows from these constraints: the trust boundary sits at the process boundary, lemond is a fact authority + resource broker + API presenter, and adaptation is code that runs sandboxed.

15.2 Prototype & sandbox reality check (read the code, not the prose)

These findings come from reading the prototype branch (feat/custom_backends and feat/custom_backends_nono) directly:

Taken together, the prototype proves out-of-process + sandbox + secret-scrubbing are viable; it just lacks the adapter-translation leg and the contextual pull protocol the Gateway adds. This supports (rather than undercuts) the direction.

17. Glossary

Terms and codes used in this document, defined for a standalone reader.

Term Definition
lemond The Lemonade HTTP server process (the executable lemond); the trusted coordinator this design keeps under its control. "Trusted" refers to lemond's own code (secret custody, arbitration, the process/API edge) — the smallest unavoidable trusted core — never to the backends it manages, whose engines are all sandboxed regardless of origin (§4.0).
adapter (plugin) The sandboxed, out-of-process program that owns a backend engine's adaptation. adapter.kind is binary/container (a shipped program) or passthrough (no program — lemond's own generic/handling adapter).
passthrough The adapter.kind value meaning "no shipped adapter program is needed": lemond's own launcher + HTTP proxy using launch-time tokens and/or variant_of inherited handling — the zero-code path where lemond launches a standard-HTTP engine and proxies capability endpoints to it. Not a built-in WrappedServer — lemond's generic adapter.
variant_of A binary-drop-in plugin flavor (§4.2.1): the recipe inherits a named built-in backend's procedural argv/handling against a sandboxed external binary, so the manifest is mostly binary provenance.
version_policy How a recipe's binary is versioned: pin (hash/version, the secure default) or roll_forward (follow github_latest) — an explicit opt-out for "track the nightly" workflows.
named-transform registry The bounded, versioned set of vetted request/response rewrite hooks lemond ships (§6); one of the declarative surfaces, not the whole of "no adapter."
capability_enable_args Manifest field declaring "I support capability X; enable it with argument A" (model-conditioned via model_info.type, or unconditional).
fact service lemond's read-only, scoped, secret-free introspection channel (§5.1); static vs dynamic facts (§5.1).
lease / entitlement The broker's grant of a resource to an adapter (§5.2): device, memory, socket FDs, exclusivity, TTL, and dynamic facts. Cooperative over physical VRAM (see §13).
FD/UDS socket custody lemond binds the socket and passes the file descriptor/FD to the adapter (§5.6); the no-port-to-hijack mechanism.
nono The kernel-sandbox engine (nono.sh, from the nolabs project): Landlock on Linux, Seatbelt on macOS, Landlock-under-WSL2 on Windows. lemond delegates filesystem/network confinement to it. Invoked either embedded via C FFI (Sandbox::apply, fork + sandbox + exec, preferred) or as a CLI wrapper (fallback) — see §7/§13.
A1 / A6 Class-A secret-custody codes: provider/cloud keys held only by lemond (A1); backend-needed credentials delivered via explicit allowlist (A6), never ambient.
A2 Class-A resource-coordination code — scheduling/arbitration is the broker's single-writer job (§5.2), not a plugin gate.
A5 Class-A process/API-edge custody — lemond owns the socket/process/response boundary and teardown (§7).
CWSR door A gfx1151 AMD workaround in the vLLM backend: needs_gfx1151_cwsr_fix() enriches a startup timeout error; distinct from --enforce-eager (keyed on non-discrete-HBM arch).
ROCm shim A generated sitecustomize.py that vLLM's ROCm path injects (via PYTHONPATH) to prevent its CUDA NVML probe from faulting on AMD-only systems.
cross-platform posture Linux = full sandbox (Landlock); macOS = Seatbelt (deprecated Apple API, works); Windows = WSL2-only (native Windows sandbox unsupported); §13.