The Lemonade Backend Gateway

01Why this exists

Lemonade runs inference by spawning backends as subprocesses (llama-server, whisper-server, sd-server, flm, vllm, and others). Today, adding a backend means writing a new C++ WrappedServer class and recompiling lemond. That limits Lemonade to in-tree backends, slows adoption of new inference engines, and forces maintainers to build, pin, and ship every engine for every platform and GPU vendor.

At the same time, letting users run third-party or community binaries raises a real security problem: an untrusted binary that runs as a child of lemond with the same user, full network access, and the server's environment can read model files, reach the network, and exfiltrate API keys that happen to be in the environment.

02The options considered

There are three natural ways to make a backend pluggable. Each takes a different position on where procedural adaptation logic lives, and each has hard limits. They are evaluated here on their own terms.

Options 1 and 2 both decide the question by where the code goes — data in lemond's interpreter, or code in lemond's process. Both, for opposite reasons, leave untrusted behavior inside lemond's trust domain. The fix is to change what they are optimizing:

2.1The trust boundary is the process boundary

A backend is really three concerns glued together in one C++ class today: (1) a declarative identity (BackendDescriptor), (2) arbitrary procedural adaptation (argv construction, hardware detection, payload rewriting, protocol translation), and (3) shared subprocess + HTTP orchestration (spawn, port lease, health, watchdog, eviction) that WrappedServer already provides.

Option 1 pushes concern (2) into data and fails; Option 2 pushes (2) into lemond's process and is unsafe. This proposal keeps (2) as real code, but runs it in its own sandboxed process, drawing the line at that process boundary:

A fault in an adapter cannot take lemond down. An adapter cannot reach beyond what its sandbox grant allows or read secrets lemond chooses not to hand it. And because the adapter runs behind a wire contract, the language it is written in is irrelevant to lemond — a static binary (C/Rust/Go) or a container, whichever fits the engine.

The result satisfies expressiveness and isolation at once: the sandbox becomes the security boundary, and the adapter becomes whatever code an engine really needs.

03Requirements, derived from first principles

3.1Functional needs

3.2Security needs, from the actual threat surface

Auditing today's code shows there is currently no OS sandboxing for backend processes and they inherit lemond's full environment. The real leaks to close:

3.3The synthesis

04The shape: the Capability Gateway

4.0Sandboxing vs. the adapter tier — two independent decisions

Two things are easy to conflate and must be kept separate:

So the process tree is:

The sandbox boundary sits at the engine (the untrusted thing) in every case; the adapter tier is added only when an untrusted adapter owns the adaptation. This keeps "even built-in engines are sandboxed" as a stated guarantee while avoiding an unconditional 2→3 tier expansion.

4.1The manifest

Declarative and small — it describes where the adapter is and what it may do, never how. Procedural intelligence is not here.

{
  "api_contract_version": "1",
  "recipe": "my-engine-custom",
  "display_name": "My Engine (Custom)",
  "extends": "llamacpp",             // optional: inherit base recipe's options/contracts
  "adapter": {
    "kind": "binary",                 // "binary" | "container" | "passthrough"
    "source": "https://cdn.example/adapter/linux-x64/my-engine-adapter",
    "sha256": "9f3a…c2",
    "version": "1.2.3"
  },
  "engine": {
    "reserved_args": ["--port", "--host", "-m"]   // reject user args colliding with these at merge
  },
  "capabilities": ["chat_completion", "completion", "embeddings"],
  "capability_enable_args": {           // "I support X; enable it with arg A"
    "embeddings": ["--embeddings"],     // appended when model_info.type == embedding
    "completion":   ["--jinja", "--metrics"]   // unconditional runtime args (always)
  },
  "capability_contracts": {
    "chat_completion": { "response_fields": { "model": {"mode":"echo_request_model","sensitive":false} } }
  },
  "slot_policy": "standard",          // standard | exclusive_npu | coexist_by_type | unmetered
  "model_management": "lemond_managed", // or "self_managed"
  "model_reads": { "main": { "form": "gguf" } },
  "lifecycle": { "spawn_shape": "single", "requested_ports": 1,
                 "readiness": {"type":"http","endpoint":"/health"},
                 "self_manages_downloads": false },
  "sandbox": {
    "read_paths": ["{hf_cache_models}", "{binary_dir}"],  // scoped to models subtree, NOT cache root (D1)
    "write_paths": ["{scratch_dir}"],
    "devices": ["/dev/dri"],
    "network": { "allow_egress": false, "allow_loopback": true },
    "env_allowlist": ["GGML_VK_VISIBLE_DEVICES"]
  },
  "facts_scope": ["devices","igpu","rocm_arch","resolve_checkpoint","model_info","vram_pressure"]
}

The schema has two adapter flavors. passthrough (§4.2) reuses an existing binary/container with a manifest only; binary / container ships an adapter program for engines that exceed the passthrough envelope.

The manifest is "data, not code," but with one deliberate, bounded exception: launch-time token substitution. The argv / command arrays may contain tokens ({checkpoint:main}, {port}, {ctx_size}, {custom_args}) that lemond substitutes once, when constructing the spawn — this is how lemond hands the engine its model path, port, and user-option values. That is the sanctioned, declarative subset: tokens exist to launch the engine, and they appear only in the manifest's argv/engine blocks, never in request-time logic.

What is barred is request-time adaptation in data: no request_transform field maps, no per-request payload rewriting expressed in JSON, no scripting. Those are what the earlier proposals tried to force into data and what degenerated into a Turing-complete language. Request-time adaptation lives in the named-transform registry (§6, bounded, vetted) or — when genuinely open-ended — in an adapter program (§4.2, out-of-process sandboxed). So the statement "no command-template scripting" is accurate; the launch-time {token} substitution is the one declarative exception, and it is only ever about how the engine is started.

Token vocabulary — launch-time only, resolved by lemond:

Tokens are substituted only in the argv/engine blocks at spawn; the env block values may also reference token values but never read lemond's ambient environment. See §6 for the separate request/response surface.

The manifest only declares identity, capability claims, lifecycle/ownership (driven by the recipe junctions of §9.1), and the sandbox grants plus fact scope — precisely what a consent dialog, a kernel sandbox, and the FactService need.

The optional platforms block is the OS × accelerator matrix. It declares which host/device combinations the recipe runs on and what differs per combination — GPU device nodes, accelerator env vars, arch gates, and any security-posture override (e.g. a wsl2 vs windows_native distinction). Platform gating reuses the same machinery lemond applies to built-ins (BackendSupport, SystemInfo::check_recipe_supported, ModelManager::filter_models_by_backend), so lemond enables/disables recipes against the detected OS and GPU at model-cache build time, exactly as it does today. The matrix is DRY by construction: the base recipe (options, lifecycle, capability contracts, core grants) is defined once, and each platform block supplies only its deltas — argv_extra, env, devices, support, and an extends pointer that inherits another platform's block and overrides a few fields. lemond flattens this to an explicit per-OS × accelerator view before validation; the terseness is author ergonomics, not a hiding of what the resolver sees.

To make the matrix concrete, here is the ds4 recipe (DwarfStar 4, serving DeepSeek V4 Flash via antirez/ds4 — a real backend, see PR #3047), condensed to the platform-matrix essentials. It is a passthrough native binary:

"platforms": {
  "base": {
    "argv": ["--model","{checkpoint:main}","--host","{host}","--port","{port}",
             "--ctx","{ctx_size}","{custom_args}"]
  },
  "linux": {
    "rocm": { "extends":"base",
              "argv_extra":["--gpu-layers","{gpu_layers}"],
              "env": { "HIP_VISIBLE_DEVICES": "{hip_visible_devices}" },
              "devices": ["/dev/kfd","/dev/dri"],
              "support": { "archs": ["gfx1151"] } },
    "cuda": { "extends":"base",
              "argv_extra":["--gpu-layers","{gpu_layers}"],
              "env": { "CUDA_VISIBLE_DEVICES": "{cuda_visible_devices}" },
              "devices": ["/dev/dri"] }
  },
  "wsl2": { "extends":"linux" },
  "windows_native": { "extends":"base", "sandbox": { "mode": "degraded" } }
}

The base argv is written once; each platform supplies only its deltas (argv_extra, env, devices, support), and extends inherits another block. In practice the real ds4 recipe publishes only the linux.rocm row (and, by extends, wsl2) and gates on gfx1151; the cuda / windows_native rows here are illustrative of how the matrix generalizes, and would be absent (or marked unsupported) for a backend that does not ship those targets. lemond flattens the matrix and reuses check_recipe_supported / filter_models_by_backend to enable or disable the recipe against the detected OS and GPU, exactly as it does for built-ins.

4.2The zero-code fast path

Many engines already speak a standard OpenAI-compatible HTTP surface and need no adapter at all. For those, adapter.kind: "passthrough" makes lemond's own generic adapter do the work: launch the engine with a bounded argument template, FD-socket-custody + health-wait, then HTTP-passthrough standard endpoints. This was already demonstrated by the ExternalBackendServer prototype (PR #2880) running dflash-rocm (podman) and custom llamacpp recipes with no adapter program — just a manifest.

Zero-code works when the only adaptation needed is a bounded, named set of known transforms (see §6). Engines that exceed that envelope — protocol translation (whisper), binary sniffing + multipart (trellis), out-of-band streaming (moonshine) — graduate to the external-adapter mode. Both modes are first-class and equally secure.

Containerized engines are the clearest zero-code case. A container is the engine, so the manifest needs only to declare how to launch it, which capability endpoints to proxy, and where the model files are. The DFlash speculative-decoding server is a working example (containerized in the ExternalBackendServer POC); zml/llmd is analogous. A condensed example follows; the pattern is the same for any containerized OpenAI-compatible engine. Commands are typed argv arrays, never shell strings:

"adapter": { "kind": "passthrough" },
"engine": {
  "kind": "container",
  "runtime": "podman",
  "image": "ghcr.io/zml/llmd:latest",
  "devices": ["/dev/kfd", "/dev/dri"],
  "groups": ["render"],
  "mounts": [ { "host": "{hf_cache_models}", "container": "/models", "mode": "ro" } ],
  "network": "host",
  "argv": ["llmd", "--model", "/models/{checkpoint_relative:main}",
           "--fd", "{socket_fd}", "--ctx", "{ctx_size}", "{custom_args}"],
  "stop": { "command": "podman", "args": ["rm","-f","-t","0","lemonade-{recipe}-{port}"] }
},
"capability_contracts": { "chat_completion": { "endpoint": "/v1/chat/completions" } },
"sandbox": { "note": "exempt_container_backend" }

Because the container runtime provides the isolation boundary, the sandbox block is informational for containerized engines — lemond marks the recipe exempt_container_backend rather than double-wrapping podman in the OS sandbox.

4.2.1Binary-drop-in variants of a built-in backend (variant_of)

A common and important plugin is a fork / repackaged build of a backend that is already built into lemond — a community llama.cpp with extra kernels (ROCmFPX), a vendor's nightly, or an "upstream vulkan" reference. These are wire-compatible: same llama-server, same OpenAI surface, same argv the built-in constructs. For these, re-declaring the full argv in the manifest is redundant and error-prone; the author only really needs to say where to get the binary.

A dedicated flavor supports this: variant_of: "<recipe>" — the plugin is a binary-drop-in for a named built-in backend, and lemond reuses that built-in's procedural handling (argv construction, reserved-arg validation, capability enable-args, platform env) against the sandboxed external binary. It inherits every declarative base too (options, lifecycle, capability contracts). The manifest reduces to binary provenance plus any argv deltas:

{
  "api_contract_version": "1",
  "recipe": "llamacpp-rocm-nightly",
  "adapter": { "kind": "passthrough" },
  "variant_of": "llamacpp",
  "version_policy": "roll_forward",          // follow github_latest, not pin (S5 opt-out)
  "engine": {
    "kind": "binary",
    "source": "https://github.com/lemonade-sdk/llamacpp-rocm/releases/download/{version}/llama-{version}-ubuntu-rocm-{arch}-x64.zip",
    "binary": "llama-server"
  },
  "platforms": { "linux": { "rocm": { "support": { "archs": ["gfx1151","gfx1150","gfx1100"] } } } }
}

• The trust model is preserved. The procedural logic stays in trusted, in-lemond C++ (variant_of points at a built-in whose argv builder runs unchanged); the fork binary remains an untrusted, sandboxed subprocess. lemond never trusts the binary — it constructs the argv it would have handed the built-in and confines the external process the same way.
• Distinct from the _bin config override. config set llamacpp.<key>_bin= works today but is a global, mutable, single-slot config mutation with no sandbox, consent, reproducibility, or named coexistence. variant_of is a declarative, sandboxed, consent-able, version-pinned recipe; several forks can coexist as named recipes rather than overwriting one _bin slot.
• Distinct from declarative extends. extends reuses the declarative base but still requires the author to write argv. variant_of additionally inherits the built-in's procedural argv handling, so argv can be omitted (or overridden per-platform via argv_extra).
• Version policy. variant_of defaults to hash/version pinning (S5); a roll_forward (github_latest) policy is an explicit opt-out for authors who want to track a moving nightly. This is the one trade the "track the nightly" benchmark and marketplace workflows need.
• Honest caveat. Full inheritance couples the fork to the built-in's current argv behavior — usually desired (the fork tracks upstream) but not always what the fork's binary expects. The manifest therefore allows per-platform argv_extra / reserved_args overrides on top of the inheritance.

This is the mechanism that collapses the fork registry in the "Backend Battle Nightly" benchmark (PR #3069) — four llama.cpp forks, each a short variant_of: "llamacpp" manifest instead of an ad-hoc config set + retry dance — and it is exactly the "massively simplified by 2951" that the review there flagged.

4.3The capability contract

Closed, typed, versioned. The set of capability kinds maps to the existing server_capabilities.h interfaces (chat_completion, completion, responses, embeddings, reranking, transcription, image_*, tts, audio_generation, model_3d_*, classification). New kinds are added by a reviewed, additive contract change — a plugin cannot invent routes.

The transport is a first-class, essential spec (Phase-1b critical path), and binary payloads need a side channel. Several things the capability contract needs but the design must not leave implicit:

• Transport and framing. lemond↔adapter control/data rides a Unix-domain socket (or loopback on Windows), with explicit framing, message versioning, request-id multiplexing, cancel-vs-completion semantics, backpressure, and a max message size. RPC channel auth is peercred-PID-match plus a per-lease 0700 socket — peercred alone verifies the uid, not that the peer is this adapter.
• Binary payloads (audio, images) must not be base64-in-JSON. Base64 inflates ~33% and forces full buffering, which kills streaming latency. Audio/image is carried over a shared-memory or FD-passing side channel alongside the RPC, not encoded in the JSON envelope. This is dictated by §13's streaming-latency requirement, and the transport RFC must cover it.

The common control plane is UDS-everywhere where possible; the engines that cannot speak UDS (cpp-httplib llama.cpp/whisper) are exactly those forcing the loopback data plane (§5.6), so the honest posture is "UDS control plane + peercred-authenticated loopback data plane for legacy engines," not one assumed transport.

05The five contract primitives

The lemond↔adapter contract is exactly five primitives. Together they carry the whole of what a backend needs from the server and what the server needs from a backend.

5.1Fact service — lemond as just-in-time fact authority

lemond holds a trusted SystemInfoCache and ModelInfo. Rather than pre-compute and push every fact a plugin might need (which would force lemond to enumerate the universe of model/hardware state), lemond answers on demand:

{ "op":"fact.get", "facts":["devices","igpu","system_info",
                            "resolve_checkpoint"], "scope": {...} }

{ "op":"fact.result", "facts":{ ... only the adapter's own slot ... } }

Read-only, secret-free, scoped to the adapter's own slot (no cross-model leak), authoritative. The plugin never peers raw /sys//proc; it asks lemond.

Static vs. dynamic facts. Two kinds of fact behave differently and are served differently:

5.2Resource broker / lease protocol — single-writer coordination

Admin approval is consent to capability; it is not scheduling. Even with perfect approval, two plugins cannot arbitrate against each other — only lemond sees all resident models, live device usage, and memory pressure. So arbitration is not a trust gate; it is a coordination problem no single plugin can solve locally. lemond becomes a resource broker that maps resources + config + preferences + realtime status into a just-in-time entitlement per plugin:

{ "op":"lease.acquire", "want": {"slot_policy":"exclusive_npu",
  "devices":["npu0"], "mem_mb":0, "fd_sockets":1} }

{ "op":"lease.grant", "lease_id":"…", "entitlement":{
  "devices":["npu0"], "mem_mb":4096,
  "sockets": [ {"fd": 7, "uds":"/run/lemonade/rec-{lease_id}.sock"} ],
  "npu_exclusive":true, "ttl_sec":120,
  "dynamic_facts": {"vram_pressure":0.41, "reserved_mem_mb":4096} } }

{ "op":"lease.heartbeat" }

{ "op":"lease.revoke", "reason":"lru_evict|memory|admin", "grace_ms":5000 }

The grant carries the leased socket FDs (§5.6) and the dynamic facts (§5.1) atomically: the adapter acts on the exact reserved memory/device state, not on a value fetched earlier that may have changed. Reclamation is physical (revoke → grace → kill the PID namespace/cgroup/job (§7)), not negotiated.

5.3Model inventory RPC — "ask the backend for its models"

For plugins that self-manage their model catalog, lemond must be able to relay user intent without downloading (this mirrors BackendOps::discover_models, which already exists):

{ "op":"model.inventory" }

{ "op":"model.inventory.result", "models":[{"id":"…","status":"ready"}] }

{ "op":"model.pull", "model":"…" }   // relay user intent

5.4Checkpoint-form declaration — imported-form + read grants

"My model dir must contain X" is a read-permission / declaration problem, not a trust problem. The plugin declares its expected form (plain GGUF, a directory with genai_config.json, a .onnx file, an index.json); lemond grants a scoped read and verifies completeness; the plugin inspects itself within its grant.

5.5Global response pipeline — the API edge

The plugin already knows the local paths (it loaded the model); the question is only what the client sees. Today only LlamaCppServer normalizes the response model field (and only on the non-streaming path — the SSE stream passes the backend's own model string through), so the status quo is smaller and leakier than "one mechanical helper." This strengthens the case for a global response pipeline applied uniformly, driven by a per-capability "safe-response-fields" contract, and it must explicitly cover the streaming path, not just single JSON responses — otherwise streaming responses continue to leak local paths and secrets. This is a presentation concern at the API/telemetry edge, not a per-plugin trust gate.

5.6And the two pieces that hold them together

• Socket custody via FD/UDS passing — lemond owns the network edge and passes it to the adapter, rather than handing out a port number and checking reachability. It bind()s a Unix-domain socket (Linux/macOS) or a loopback listener (Windows) itself and passes the file descriptor to the adapter on spawn (the systemd LISTEN_FDS idiom). The adapter forwards that FD to its engine. This eliminates port collisions and TOCTOU races (lemond never trusts that "whoever bound 127.0.0.1:N" is the intended engine); custody is by FD, so there is no port to hijack.
  • Honest primary path for the real fleet. FD custody covers engines that accept an inherited listening socket (vLLM/uvicorn via --uds/--fd, podman socket-activation). Engines that expose only --host/--port (cpp-httplib llama.cpp, whisper.cpp — the majority) cannot take an inherited FD without patching upstream. For those, the shippable primary is loopback-port-with-peer-verification: lemond binds and forwards to 127.0.0.1:port, and verifies the connected peer is the spawned engine via SO_PEERCRED / LOCAL_PEERPID / the Windows PID owner — not a bare reachability check. FD/UDS custody is then an opportunistic optimization, not the assumed default.
  • No veth-in-host is needed (and is not safely buildable unprivileged). Creating a veth pair into the host namespace needs CAP_NET_ADMIN, which an unprivileged lemond does not have; and unprivileged user namespaces are restricted on several distros. Where an isolated namespace is desired, use pasta (passt, the rootless-podman userspace networking default) or a lemond-proxied socketpair — neither requires CAP_NET_ADMIN. Evaluate user-namespace availability at runtime and degrade honestly.
  • Platform mechanics: FD/UDS passing is the Unix ideal; on Windows the equivalent is handle inheritance via CreateProcess with PROC_THREAD_ATTRIBUTE_HANDLE_LIST (or WSADuplicateSocket), not a raw FD. The architectural custody concept is identical; the OS API differs and is codified in the Phase 1b RFC.
• The capability contract (§4.3) bounds public behavior.

5.7The capability RPC also carries request control and trace

Two things the capability contract must carry that are easy to omit:

06The named-transform registry

The zero-code adapter stays "data, not code" by exposing a small, versioned, closed set of named transforms lemond ships. A manifest selects a name (data); lemond runs the vetted implementation. Selecting a transform never supplies arbitrary code.

When is an adapter actually needed? Only when the adaptation is procedural. The decision rule is: if a backend's needs are expressible as a bounded set of known transformations — field renames, prompt wrapping, image-defaults merges, size resolution, forced defaults — then the named-transform registry covers it, no adapter program is required, and the process tree stays at two tiers (lemond + sandboxed engine). A separate sandboxed adapter program is only introduced when the backend requires genuine procedural computation the registry deliberately does not express — bespoke protocol translation, binary MIME sniffing/multipart, out-of-band streaming, or request-time logic that can't be a named, bounded hook. So the registry is not a fallback set of happy-path transforms; it is the default answer, and the adapter is the exception reserved for open-ended procedural work. A healthy registry covers the common cases; the adapter exists only for the cases the registry is intentionally closed against.

The named-transform registry is one of several declarative surfaces — it is not the whole of "no adapter." Many things that look like procedural logic are actually declarative, just not request/response rewriting. The other surfaces that keep a backend adapter-free:

• capability_enable_args — a backend declares "I support capability X; enable it with argument A." lemond appends the arg when the model's own type needs it (matching model_info.type, which the recipe already knows). This is how --embeddings / --reranking resolve: they are a capability declaration plus an enable-arg, not runtime branching. The same field also carries unconditional runtime args (e.g. --jinja --metrics) — apply-always, a degenerate case.
• Recipe-declared backend args — static per-recipe arguments the backend declares it wants.
• Advertised lifecycle capabilities — e.g. downsize: supported plus a known backend operation to invoke; "supports downsize" is a boolean the backend advertises, and lemond orchestrates the fixed op.
• reserved_args / reserved-args merge validation — because lemond is already the one merging {custom_args} into the argv array, the reserved set is a validation rule in the merger: lemond rejects a user arg that collides with --port / -m before the array is built. Declarative schema + a reject rule, not adapter logic.
• Per-platform env / install / support — already covered by the platform matrix and the no-managed-install path (library paths become platform env entries; per-arch builds via support).

Every entry below is grounded in a real C++ backend, not invented:

Deferred to the external-adapter mode (not zero-code): whisper multipart translation, trellis MIME sniffing + multipart, moonshine 3-port streaming head, SD ESRGAN secondary-subprocess upscale, and any custom per-engine rewrite beyond the closed set.

What keeps the registry non-Turing-complete

• Closed, versioned, additive vocabulary.
• Intent-only, no control flow — a transform rewrites a structured message; it cannot loop, branch arbitrarily, or open endpoints.
• Vetted implementation in lemond — the manifest never supplies code.
• Ordered + scoped per capability, individually consentable and auditable.

The registry is deliberately non-scripting: there is no generalized scripting, only named, bounded hooks over real implementations drawn from today's backends. The "Turing-complete JSON" failure mode is avoided by construction.

07Security model

08Trust boundary: what stays, what moves

Every adaptation concern in a backend resolves into one of three classes, and that classification determines where the concern lives:

Two specific cases are worth stating plainly.

09Lifecycle and recipe intersection

Mapping the 15 in-tree backends (plus the containerized prototype) against the ideal WrappedServer lifecycle (LOADING → READY ⇄ IN_USE → DOWNSIZING/DOWNSIZED → EVICTING → UNLOADED) reveals a small set of orthogonal lifecycle axes that actually differ:

9.1Where recipes constrain the lifecycle

Recipes (server_models.json) and the backend descriptors jointly decide what a load can do, and a plugin contract must honor these junctions:

• Checkpoint types are the recipe↔lifecycle junction. Checkpoints are not a single main file: recipes carry main, draft (spec-decode), mmproj (vision), npu_cache (whisper NPU device cache), and text_encoder + vae (SD/split models). A recipe's checkpoint set determines which artifacts lemond provisions before spawn and which argv shape results.
• self_manages_downloads flips provisioning direction. For FLM, the backend owns its model catalog; lemond must not try to download, and must tolerate the model appearing only after the backend pulls it. The plugin contract needs an explicit "model readiness" handshake here, not just a health probe. This is distinct from "no local model" (Cloud), which has no checkpoints to fetch at all — it does not set self_manages_downloads.
  • Egress ⇄ download validation rule. A recipe that declares self_manages_downloads: true must also declare outbound network rights, because it has to fetch its own model; lemond rejects a manifest as invalid otherwise. This rule applies to self-managed-download backends only — a no-local-model backend (Cloud) must not be granted host egress on the strength of being "self-managed," because it isn't.
  • Egress must be a host allowlist, not a bare bool. A self-managed backend that gets allow_egress: true holds both outbound network and read access to the weights — exfiltrating them is then within policy. Replace the coarse boolean with an enforced egress host allowlist (e.g. huggingface.co, cdn-lfs.huggingface.co), realized via a host-filtering proxy or pasta, and acknowledge the residual risk in the doc.
• slot_policy (standard / exclusive_npu / coexist_by_type / unmetered) is the router-level resource contract. A plugin recipe maps to one of these so eviction and slot logic stay in the trusted router.
• dynamic_models decouples "what models exist" from server_models.json (FLM provider catalog, Cloud discovery). A plugin declaring it supplies the model-inventory hook (§5.3).
• model_reads is a plugin-declared format contract, not just a form tag. A recipe lists the checkpoint roles it reads and their required-vs-optional status, and — for GGUF — the quant families it can read (supported_quants). This is declarative, not procedural: it lets lemond filter or reject models by format at model-cache build / load time (reusing filter_models_by_backend), so a plugin is never asked to serve a checkpoint format it cannot read. A community llama.cpp fork that adds new GGUFs (e.g. ROCmFPX quants upstream llama.cpp cannot read) declares exactly which families it supports; the vanilla backend does not list them, so a ROCmFPX-quantized model is only offered to the fork. See example-rocmfpx-passthrough.json and plugin-model-recipe-interaction.md for the concrete shape and the related gaps (plugin-named roles, typed options, provisioning hooks).

These junctions drive the manifest's model_management, model_reads, and lifecycle fields (§4.1).

9.2Design consequences

• The launch bundle carries the full resolved checkpoint set, not just a single main path. Multi-role artifacts and compound models are the norm.
• A plugin declares its lifecycle "personality" up front — spawn shape, readiness type, ownership direction, downsize support, adaptation kind, streaming shape, checkpoint form — because that is the manifest's real content (it feeds the consent dialog and the sandbox grant).
• Provisioning is directional — lemond-owned (download then hand paths) vs. self-managed (wait for the backend to provision) — handled by the model-inventory RPC and a model-readiness handshake, not a single download path.
• Multi-port and secondary-subprocess backends are declared properties, not hidden assumptions (the base WrappedServer assumes a single port; Moonshine and SD's ESRGAN upscale prove that is a simplification).

10Candidacy: which backends could be plugins

10.0How existing backends map

Assessed against the eight lifecycle axes (§9), the 15 in-tree backends (plus the containerized prototype) fall into three natural tiers. The distribution is the point: a solid minority are clean, mechanical plugins; the flagship workhorses are not, by design.

10.1Candidacy is a portfolio choice, not a capability limit

The tier a backend lands in reflects how much it is worth externalizing, not whether the Gateway can express it. The five contract primitives (§5) — the fact service, the lease broker, the model-inventory RPC, the checkpoint-form declaration, and the response pipeline — cover the concerns that had kept the "heavier" backends from looking pluggable at all:

The genuinely inexpressible set is therefore very small — secrets and process/API-edge custody + the sandbox, none of which is per-backend. Under this lens, RyzenAI, FastFlowLM, and vLLM resolve to Medium (their earlier "Low" was largely a lack of the right contract primitives, not a hard capability limit). The only backends that remain "Low" are llama.cpp (kept in-tree for flagship stability, not capability) and Cloud (no subprocess; architecturally lemond's by design). Every remaining choice about externalizing a backend is about how much it buys today (flagship stability, install burden, concentrated domain knowledge), not about whether the Gateway can express it.

10.2Migration does not gate adoption

This is a design rule, stated once and firmly:

• The Gateway is purely additive — one new GatewayServer class plus an out-of-tree contract. It removes, rewrites, or gates nothing.
• In-tree backends remain coordinator-managed C++ indefinitely by default, with their engines sandboxed under the same zero-trust baseline as plugins (§4.0). "Stays in-tree" is a packaging choice, not a trust claim.
• Migration is opportunistic, cost-benefit-driven, and reversible — never required for adoption.
• Adoption is proven by one non-trivial plugin working well, not by migrating the fleet.

Coexistence is the intended end state. The strongest objection to out-of-process plugins assumed that adopting them forces the whole fleet to be re-expressed. It does not. The Gateway is adopted on its own merits, alongside the existing in-tree backends (which keep their engine sandboxes either way), and each backend is moved to plugin form only if and when that pays off.

11Mapping onto the existing codebase

12Comparison of the options

13What is genuinely open

Honest risks and decisions remain, to be resolved in the RFC process:

• Adapter ecosystem bootstrap. Zero-code handles the common path; the first real external adapter is a larger authoring step than a one-line descriptor. The named-transform registry + passthrough template mitigates this.
• Protocol stability/versioning. The capability contract needs an ABI/transport freeze before marketplace adoption.
• Cross-platform sandbox posture and scope. The sandbox boundary (filesystem + network confinement) is available on all three platforms via nono: Linux Landlock, macOS Seatbelt (a deprecated Apple API — works but unsupported upstream), and Windows via WSL2 (Landlock; feature coverage depends on the WSL2 kernel — "~85%" holds only for the stock 6.6 kernel, and rises with a rolling kernel; native Windows unsupported). Linux is the mature first target.
  • Landlock's network limits must be stated honestly: its net rules are port-scoped, not address-scoped ("allow the control port" allows it to any host), cover TCP only (UDP/ICMP exfiltration sails through), and the network hooks are absent entirely below kernel 6.7 / Landlock ABI v4 — so "default-deny egress via Landlock" is simply unavailable on Ubuntu 22.04's 5.15. Where possible, prefer UDS-everywhere plus a seccomp AF_INET/AF_INET6 deny for clean, address/ABI-independent egress-default-deny, and reserve Landlock-net for the genuine outbound-TCP case. Sandbox invocation is itself a decision (§7): prefer embedded via C FFI (fork + sandbox + exec, no supervising wrapper, self-contained lemond), keeping the CLI wrapper only as a fallback for platforms without a vendored nono library.
  Two things are deliberately treated as orchestrator extensions, not sandbox primitives, and are outside the initial sandbox work: process-tree teardown / resource reclamation (the broker's job; see §7 — solvable per-platform, no off-the-shelf primitive on macOS) and GPU custody / VRAM allocation (the resource broker's job, §5.2 — the sandbox grants the device, the broker decides which device and how much). The Windows posture is WSL2-gated; native Windows sandboxing is a follow-on platform decision. The consent UI must reflect enforced-not-declared posture per host, since native Windows and container backends ship unsandboxed.
• Performance of the extra loopback hop. Negligible for token streaming; benchmarks needed for streaming TTS/audio.
• Supply chain for binaries. Pin + sha256 + consent baseline; optional code signing later; sandbox is the primary protection.
• Wrapping vs. owning the engine. Recommendation: adapter owns/spawns the engine and presents one face to lemond.

14Suggested roadmap

1. 1Adopt the process-boundary framing in the charter — replace "declarative JSON descriptors as the plugin" with "manifest + sandboxed adapter program behind a versioned capability contract."
2. 2Land the secret-scrubbing + default-deny baseline for all backend subprocesses, independent of plugin work — the highest-value security fix.
3. 3Prototype GatewayServer + one real adapter (start with a mostly-mechanical backend: TheNoise → OpenMoss → a containerized recipe, each proving one primitive) to prove adaptation + sandbox together, and to exercise the fact service (a fact-driven igpu/embedding decision).
4. 4RFCs: Phase-1a manifest schema; Phase-1b capability contract + transport; Phase-1c fact service + lease protocol (the broker); Phase-2 model inventory + checkpoint-form + response pipeline.

15Verification appendix

Two things this proposal is often judged on: whether it holds up against the concerns raised during the charter review, and whether its claims about the existing prototype are accurate. Both are grounded in the primary sources (review threads and the actual branch code), not assumed.

15.1Design constraints surfaced by the charter review

The Working Group charter (PR #2951) review raised a small set of substantive concerns that this design must meet. They are stated here as design constraints:

This document's premise directly follows from these constraints: the trust boundary sits at the process boundary, lemond is a fact authority + resource broker + API presenter, and adaptation is code that runs sandboxed.

15.2Prototype & sandbox reality check

These findings come from reading the prototype branch (feat/custom_backends and feat/custom_backends_nono) directly:

17Glossary

Terms and codes used in this document, defined for a standalone reader.