Module sandbox
Every module process runs confined: the coordinator side, the runner, services, probes, workloads and doctor, on
every platform. This page is the contract: what a module process may and may not reach. Each platform enforces it with
its own backend:
| Platform | Backend | Details |
|---|---|---|
| macOS | Seatbelt | backends/macos.md |
| Linux | Landlock + seccomp, plus a cgroup v2 leaf | backends/linux.md |
| Windows | AppContainer + Job Object | backends/windows.md |
A module never sees the backend. It declares what it needs in [sandbox], and an operator approves it.
What every module process may touch
Section titled “What every module process may touch”| Area | Coordinator | Runner | Service, probe |
|---|---|---|---|
| its bundle, its runtime, the platform’s base system | read, execute | read, execute | read, execute |
| work directory | none | read, write | none |
| data directory (kept across jobs) | read, write | read, write | read, write |
| private temporary directory | read, write | read, write (inside work) | read, write (inside data) |
approved host tools ([sandbox].tools) |
none | read, execute | read, execute |
approved folders ([sandbox].folders) |
none | read (input folders); create and write only (outboxes) | none |
| everything else: homes, other modules, the agent’s and coordinator’s state, credential stores, devices other than null, zero, random and urandom | denied | denied | denied |
Workloads (workload-protocol.md) get what a service gets, with their run directory (inside the data directory) as cwd and home, plus their private directory read-only: the control document, the settings, tools and limits files and, for a workload that lists secrets, the secrets file. Only that workload’s profile may read its private directory, so the module’s other processes never see its secrets. A workload never gets the container broker.
Processes
- A module process may spawn children, and children inherit the confinement.
- It may signal only processes in its own confinement.
- It never creates persistent or detached jobs (launchd, systemd, cron, scheduled tasks, services) and never leaves its process container.
- It cannot rely on sandboxing itself again: tools such as Bazel or SwiftPM must skip their own sandbox.
Local IPC. The only channels are the job’s broker endpoint, when granted, and the endpoint handles the agent hands
out itself (service-protocol.md): a job’s connectors to its module’s endpoint
services and an endpoint service’s channel, inherited and already connected, and the listening socket of a
listen_fd inbound listener, inherited and already listening (Inbound listeners), so no rule has to
allow a name. There are no
other sockets, pipes, D-Bus, mach services or Docker sockets. Name resolution and logging go through backend-listed
system facilities.
Grants are whole directories or single files, resolved to canonical absolute paths at launch. There are no globs and no “deny inside allow” exceptions.
Hiding which files exist is not promised (Landlock cannot restrict stat). Keep secrets out of readable paths and
out of the environment.
Identity. A module must not depend on its user, uid or SID, or on privileges.
Resources. The agent may enforce mem_gb and CPU as hard limits where the OS does (Linux cgroups, Windows Job
Objects). Exceeding the memory limit is a job fault (oom).
Verification. The parent verifies confinement after every spawn and kills an unconfined process. No unconfined module process ever runs.
[sandbox]: grants
Section titled “[sandbox]: grants”[sandbox]contract = 1net = { mode = "egress-allowlist", allow = ["api.example.org", "*.files.example.org:8443"] }tools = [{ id = "jdk", version = ">=17, <22", arch = "native", trust = "code-exec" }]devices = { gpu = "compute" }containers = [{ image = "docker.io/org/tool:1.2@sha256:<64 hex>", platform = "linux/amd64" }]exec_writable = falsefolders = [{ id = "inputs", access = "read" }, { id = "outbox", access = "write" }]
[[sandbox.container_sets]] # images approved by signature (core 2.5): see "Image sets"name = "tasks"registry = "ghcr.io"repository = "example/swe-tasks/"platform = "linux/amd64"key = "keys/tasks.pub"| Grant | What the process gets |
|---|---|
net.mode = "none" (default) |
No network at all. |
net.mode = "egress-allowlist" |
Only the allow hosts, host[:port] with port 443 by default and *. for subdomains, never IP addresses. Traffic goes through the agent’s local proxy, which enforces the list and refuses a name that resolves to a non-public address (loopback, link-local, private, multicast); oarbank_sdk.egress_proxy is the reference. The agent sets HTTPS_PROXY/HTTP_PROXY/ALL_PROXY; the backend blocks every other route. |
net.mode = "egress-any" |
Outbound connections to any public address, plus DNS. A separately approved full-trust grant. |
net.inbound |
Connections from the internet on nodes the owner assigns: the agent listens, and hands each connection it accepts to the endpoint service that lists the listener. A top-risk grant, independent of net.mode (Inbound listeners). |
tools |
Read and execute one installation of each host tool, detected and version-checked on the node, never a path the module names: see Host tools. The process finds each in OARBANK_TOOLS_FILE. trust = "code-exec" flags tools that run arbitrary code (a JVM, an interpreter, a shell) in the approval UI. On Windows, any readable binary is executable. |
devices.gpu = "compute" |
GPU compute through the platform’s APIs, with no display server. Flagged at approval: it widens the kernel surface. |
containers |
The job’s broker endpoint, for these digest-pinned images only. A stage that runs containers reserves the agent-provided containers pool. |
container_sets |
The job’s broker endpoint, for digest-pinned images under a registry and repository prefix that carry a cosign signature by a pinned key (or that a signed index lists), when the job lists them (Image sets). |
exec_writable = true |
Runners may execute files they wrote into the data or work directory, such as downloaded tools. Windows cannot enforce false without application control, so nodes report it as unavailable there. |
folders |
Runners only. Each id is a folder the operator maps to a path per node in the folder registry (Folders); the runner finds the granted ones in OARBANK_FOLDERS_FILE. access = "read": read files and listings, never write or execute. access = "write": an outbox: create files and directories and write them, never read, list, rename or delete anything there. |
Network rules, whatever the mode:
- never loopback or link-local;
- never listening (an endpoint service, and a workload that serves listeners, is handed its connections; an inbound
listener is the agent’s, and a
listen_fdservice only accepts on the socket the agent made; the backend refusesbindandlistento a workload, so a workload can never raise a firewall prompt); - never another unix socket or pipe.
Approval
- A version whose
[sandbox]requests anything cannot be enabled, canaried, pinned or promoted until an operator approves its exact requests. The approval is recorded by digest and audited. - A new version needs its own approval. A version that requests nothing needs none.
Placement. Every node reports, per capability, whether its backend enforces it: enforced, cooperative or
unavailable (and endpoints: whether its agent hands out service endpoints; a module with an endpoint service needs
it). A module runs only where all of its grants, and the always-on rules above, are enforced. A node that
cannot enforce them advertises SANDBOX_BACKEND_MISSING or CAPABILITY_NOT_ENFORCED for that module.
Host tools
Section titled “Host tools”A module asks for a host tool by the fleet’s tool definition id and says which versions it accepts; the node finds the installations, and the agent grants one per tool. The module never names a path, and the release carries no path.
[sandbox]tools = [ { id = "jdk", version = ">=17, <22", arch = "native", trust = "code-exec" }, { id = "python", version = "~> 3.12" }, { id = "samtools", version = ">=1.19" }, # a tool the fleet's admin defined (tools.define)]| Key | Default | Meaning |
|---|---|---|
id |
required | The fleet’s tool definition: jdk and python are built into Oarbank; an admin defines others (a generic executable with its version command). |
version |
any | Comma-separated clauses, all of which hold: = (or ==), !=, >, >=, <, <=, and ~>, which lets the last given segment grow (~> 17.0.2 is >=17.0.2, <17.1; ~> 17 is >=17, <18). Versions are dot-separated numbers with an optional pre-release (22-ea, which sorts below 22); a node reports a JDK 8 (1.8.0_392 in its release file) as 8.0.392. Reference: oarbank_sdk.toolversion, vectors in vectors/tool-versions.json. |
arch |
any |
any (the node’s own architecture preferred), native (only the node’s own: no x86_64 JDK under Rosetta on Apple silicon), arm64 or amd64. |
trust |
read |
code-exec flags a tool that runs arbitrary code. |
A version, or an arch other than any, needs requires.core >= 2.9: older cores ignore both
(manifest.md, rule 12).
On the node. The agent detects each defined tool at startup, when its release changes, when asked (Re-detect)
and hourly: a JDK by reading its installation’s release file (JAVA_VERSION, OS_ARCH, IMPLEMENTOR) without
running it; another tool by running its version command inside the sandbox (read and execute on that file only, no
network). It reports every installation with its canonical path, version, architecture and status, and the
coordinator places a module’s jobs only where each tool it asks for resolves. For each module the agent picks one
installation per tool: a path the operator set for this module on this node, else one set for the node, else one set
for its platform group or the fleet, else the highest detected version that satisfies version (the node’s own
architecture preferred). A path the operator adds that the node did not find itself arrives in the node’s signed
statement and is checked like a folder (no roots, homes, data or system directories) and by the detector before it is
granted.
What the runner gets. OARBANK_TOOLS_FILE maps each resolved tool to [{"path", "version", "arch"}]: a JDK’s
home directory, an executable’s file (an interpreter’s prefix is granted with it). oarbank_sdk.tools.path("jdk")
and tools.version("jdk") read it. A tool with no satisfying installation on a node keeps the module’s jobs off that
node (TOOL_NOT_FOUND, TOOL_VERSION_UNMET or TOOL_REFUSED), except jobs of stages that need no certification,
which get no tools anyway.
Migrating from tool registry ids. Before core 2.9 a module named an operator-defined registry id (java17) that
the coordinator mapped to paths per OS. That registry is gone: ask for jdk with a constraint
({ id = "jdk", version = ">=17", trust = "code-exec" }), read tools.path("jdk") (each entry is now an object,
not a path string), and publish a new module version (its grant digest changes, so the owner approves it once). A
module still asking for an id the fleet does not define never runs; its readiness checklist says so.
Folders
Section titled “Folders”A runner that works on the user’s own files on a node, or leaves results there, asks for folders by id. Nothing about where they are is in the module: the operator maps each id to a path per node, and the node checks it.
- Approval. Folders are part of the requests an operator approves per version. The approval names each folder and its access; an outbox’s approval warns that files the module creates may replace files of the same name there (give it an empty, dedicated directory).
- Mapping. The operator’s folder registry maps an id to an absolute path on each node, with an access that must equal the module’s. In signing mode a node accepts its mapping only in a statement signed by the owner’s release key, so a compromised coordinator cannot point an approved folder at another directory.
- On the node. The agent grants the folder’s canonical path (symlinks resolved) and refuses a mapping that is a
filesystem root, a home directory itself, inside the agent’s or the coordinator’s data, a system directory, or that
overlaps another grant or another folder. It reports each folder
okor why not; a job of a module that needs a folder the node does not provide waits withFOLDER_UNAVAILABLE. - Links. Every backend checks a link’s target, so a symlink inside an input folder that points elsewhere leads nowhere, and a runner cannot create a symlink or a hard link in an outbox.
- Execution. Nothing in a folder may be executed, except on Windows, where any readable file can be (as for tools).
- Enforcement. Nodes report
folders.readandfolders.writeinsandbox.enforcement; a module that requests folders runs only where both of its kinds are enforced.
Inbound listeners
Section titled “Inbound listeners”Some workloads must accept connections from the internet: a peer of a network that publishes its address, a public endpoint, a relay. Module code still never listens, binds or holds an internet socket. The module declares listeners; the agent owns them, maps them on the node’s router when it can, proves them reachable from outside, and relays every connection’s bytes to the service.
[requires]core = ">=2.10,<3"
[[sandbox.net.inbound]]name = "peer"port_hint = 9000 # the external port wanted; the owner decides the real one per nodeport_policy = "stable" # published somewhere costly to change: a port someone else holds is refusedmax_conns = 256max_conns_per_ip = 16new_conns_per_ip_per_s = 4idle_timeout_s = 120
[[services]]name = "node"exec = ["python", "-I", "{bundle}/node_service.py"]lifecycle = "always"endpoint = truelisteners = ["peer"]Every key is in manifest-reference.md; the rules are manifest.md rule 22.
- A grant approved by digest.
inboundis part of the version’s requests, shown at approval as “accepts connections from the internet on nodes the owner assigns: listener peer (TCP, port 9000 wanted, stable; at most 256 connections, 16 per address, 4 new a second per address, idle 120 s)” (InboundListener.approval_text()). A new version needs its own approval, as for every grant. - Three gates on the node. A listener exists on a node only when the approved version requests it, the owner’s
signed node statement assigns
<module>/<listener>to that node (with its external port policy, lower limits and an allowlist), and the module’s service that lists it is running there. A kill switch and the machine’s managed policy can close every listener; a compromised coordinator cannot open one. - Limits. The manifest’s limits are ceilings: the owner may lower them per node, never raise them. The agent enforces them while it relays: connections at once, per address (IPv6: per /64), new connections a second per address, the idle timeout and the byte rate.
- Hand-over.
handover = "connections"(the default, every OS): each accepted connection arrives on the service’s endpoint channel with the listener’s name and the client’s address (service-protocol.md).handover = "listen_fd"(macOS and Linux): the agent creates a unix socket in a directory only its own account can enter, listens on it itself, and the service’sstartinherits the listening descriptor inOARBANK_LISTEN_FD_<NAME>; the service only callsaccepton it, which its sandbox allows, whilelistenandbindstay refused. This serves third-party servers that take an inherited socket. Windows nodes refuse such a listener (LISTENER_HANDOVER_UNSUPPORTED) andoarbank-sdk checkwarns about it. - The client’s address. With
proxy_protocol(the default withlisten_fd), the first bytes of each connection are a PROXY protocol v2 header the agent writes itself (nothing from the network is passed through as one). - Narrowing. The service may narrow who connects to the networks it names (
listener_allow); the agent intersects them with the owner’s allowlist, so a module can only ever narrow. - Network mode.
inboundneeds none: answering an accepted connection is not egress.egress-anyis needed only when the workload also dials out to arbitrary peers.
Bootstrap jobs
Section titled “Bootstrap jobs”A job of a bootstrap stage (manifest.md, stages[].bootstrap) may run on a node
before the module’s goldens pass there, so it gets less than any other job of its module:
| Area | A bootstrap job |
|---|---|
| its bundle and runtime | read, execute |
| work directory (and the private temporary directory inside it) | read, write |
| data directory | none: OARBANK_MODULE_DATA is not set, and nothing it fetched stays on the node |
| network | the module’s egress-allowlist, through the agent’s proxy; none when the module’s mode is none (a module with a bootstrap stage never requests egress-any) |
| host tools, folders, GPU, the container broker, executing written files | none (OARBANK_TOOLS_FILE and OARBANK_FOLDERS_FILE list none) |
| module settings | an empty object in OARBANK_SETTINGS_FILE |
| secrets | none (a bootstrap stage lists none; no OARBANK_SECRETS_FILE) |
SandboxSection.for_bootstrap() returns these grants. The agent takes the bootstrap flag from the module’s entry in
the signed release, never from the grant. A node whose agent applies them reports grants.bootstrap as enforced;
bootstrap jobs run only there (CAPABILITY_NOT_ENFORCED elsewhere).
Cleanup runs
Section titled “Cleanup runs”A module’s node-side processes write only to its data directory, their work directories, the agent’s dataset cache (through staging) and the agent’s container runtime (through the broker). So when the module leaves a node, the agent can delete everything it put there without its help: the data directory, the cached datasets no other module on the node uses, and the container images it pulled that no other module uses. Before that it runs the module’s cleanup stage, if it declares one (manifest.md), with these grants:
| Area | A cleanup run |
|---|---|
| its bundle and runtime | read, execute |
| work directory (and the private temporary directory inside it) | read, write |
| data directory | read, write (OARBANK_MODULE_DATA): what the module kept there, until the agent deletes it |
| network | the module’s approved mode, through the agent’s proxy for egress-allowlist |
| host tools, folders, GPU, the container broker, module services | none (OARBANK_TOOLS_FILE and OARBANK_FOLDERS_FILE list none) |
| executing files it wrote | as approved (exec_writable) |
| module settings | its node settings, as the agent last had them ({} when it has none) |
| secrets | none (a cleanup stage lists none; no OARBANK_SECRETS_FILE) |
The agent takes the cleanup flag from the module’s entry in the signed release, never from anything else.
Containers: the agent’s broker
Section titled “Containers: the agent’s broker”A runner never talks to Docker or Podman itself. The agent owns a container runtime that mounts only its own work and module-data directories:
| Platform | Runtime |
|---|---|
| macOS | Colima, or Apple container |
| Linux | rootless Podman, or Docker Engine |
| Windows | an agent-owned WSL containers (WSLc) session: a VM of its own, WSL 2.9.3 or later |
Each job whose module may run containers gets an endpoint, OARBANK_BROKER (unix:/path or npipe://./pipe/<name>),
the only IPC its confinement allows. Requests are one JSON object per line, one request per connection. Use
oarbank_sdk.broker.
| Request | Fields | Answer |
|---|---|---|
container.run |
image (an approved reference), platform, args[], entrypoint?, mounts[] {src, dst, ro}, env{}, workdir?, network (only with an egress grant), timeout_s, cpus?, mem_gb?, gpus? ("none" or "all"; core 2.5) |
{ok, exit_code, stdout_tail, stderr_tail, stdout_path, stderr_path, duration_s}; the full output is written into the work directory |
container.pull |
image, platform |
{ok} once the image is present |
status |
{ok, running, images[], gpus}; gpus is "all" when this job may give its containers the GPU (it reserved the gpu pool and the node passes GPUs through), else "none" |
|
| any refusal | {ok: false, error, detail}: image_not_approved, bad_mount, network_not_granted, gpu_not_granted, gpu_unavailable, registry_unavailable (retryable), runtime_unavailable, platform_unavailable |
Mounts
- Mount sources are PortablePaths inside the work directory, or
data:<path>inside the data directory. - Destinations are POSIX absolute paths, for
linux/*images.
Platforms. A runtime runs the platforms it reports (containers.platforms on Windows); others are
platform_unavailable. Linux adds a foreign platform where binfmt has a handler for it, macOS runs linux/amd64 under
Rosetta, Windows runs its own architecture only (WSLc has no emulation).
The agent’s container run
- Always:
run --rm, the platform, the CPU and memory limits within the job’s reservation, the validated mounts and the job’s ownership labels. - Never: privileged mode, host networking, or another socket.
Ownership and filesystem behaviour
- Files a container writes belong to the job’s identity: rootless mode, or a user namespace, on Linux.
- Exec bits and case sensitivity follow the host filesystem.
- Containers die with their job.
Image sets
Section titled “Image sets”Suites with one image per task (agent attempts, test suites, benchmark tasks) cannot list every digest. A
[[sandbox.container_sets]] entry approves images by who signed them instead:
| Key | Meaning |
|---|---|
name |
The set’s name, unique; shown at approval and in the audit. |
registry |
The registry, host[:port], lowercase (docker.io for Docker Hub). |
repository |
A repository path. Ending with / it is a prefix: every repository below it; otherwise exactly that repository. |
platform |
The images’ OCI platform. |
key |
Bundle path of the cosign public key: one ECDSA P-256 key, PEM PUBLIC KEY (cosign generate-key-pair writes cosign.pub). |
index |
Optional: a tagged reference of a signed image index; only the digests it lists are members. |
Membership. A reference <registry>/<repository>[:tag]@sha256:<hex> (normalized as Docker does: docker.io,
library/) belongs to a set when its registry and platform equal the set’s and its repository is under the prefix, and:
- without
index: its digest carries a cosign signature by the key, in either format cosign writes: a Sigstore bundle attached as an OCI referrer (cosign 3, and 2.4+ with--new-bundle-format: a DSSE envelope over an in-toto statement with predicatehttps://sigstore.dev/cosign/sign/v1naming the digest), or the simple-signing manifest at tagsha256-<hex>.sig(cosign 2). The key alone verifies it: no transparency log is consulted; - with
index: the set’s index lists the digest. The index is an OCI artifact at theindexreference (artifact typeapplication/vnd.oarbank.image-set.v1+json, one layer of that type holding{"type": "oarbank.image-set/v1", "registry", "repository", "seq", "images": ["sha256:…"]}) whose own digest carries a signature by the key. A node refuses an index whoseseqis lower than one it already accepted for the set.oarbank_sdk.images.index_document()writes the layer;oras pushandcosign sign --keypublish it.
Digest pinning stays mandatory. container.run, container.pull and jobs.enqueue name images by digest. A job
runs a set’s images only if it lists them (jobs.enqueue items’ images, core 2.5); the coordinator refuses an image
outside every approved list and set with 422 image_not_approved, and the broker refuses a set image the job did not
list. The agent verifies membership before pulling (the runtime then pulls by digest), audits each digest’s first
run, and refuses anything else with image_not_approved and the reason in detail.
Approval shows the prefix and the key’s SHA-256 fingerprint, never a list of digests: new images in an approved set need no new module version. A new key, prefix or index needs one.
oarbank_sdk.images is the reference policy (stdlib only, ECDSA included); the agent’s Rust implementation agrees with
it on spec/vectors/image-signatures.json. oarbank_sdk.imagetest builds signed test images in an OCI image layout,
without cosign or a registry.
GPU passthrough
Section titled “GPU passthrough”container.run gpus = "all" gives the container every GPU of the node; a count comes in a later minor (an integer is
bad_request today). A stage that runs GPU containers reserves the agent’s gpu pool besides containers
(pools = {containers = 1, gpu = 1}), and the runner declares gpu.in_container = true with gpu.use shared or
exclusive, so the job holds a container token and the GPU for its lifetime, and the host’s GPU admission (the owner’s
gpu_jobs policy, protected processes using the GPU) applies to it. The broker refuses gpus = "all" with
gpu_not_granted for a job that did not reserve the gpu pool. The runner’s gpu.apis_any then names the APIs its
containers need, checked against the APIs the node reports in containers (runner-protocol.md).
| Platform | Mechanism | The node |
|---|---|---|
| Linux | CDI: a CDI spec (/etc/cdi, /var/run/cdi) with an all device, such as nvidia-ctk cdi generate writes; the run gets --device <kind>=all (Podman 4.1+, Docker 25+ with CDI) |
offers the gpu pool; facts containers.gpu = "cdi:<kind>"; container APIs from the spec: cuda (it mounts libcuda.so), vulkan (the NVIDIA Vulkan ICD, or a DRM render node for other kinds), opencl (libnvidia-opencl), rocm (/dev/kfd) |
| Windows | GPU-PV in the agent’s WSLc session: its guest writes a CDI spec (microsoft.com/wslc=gpu: /dev/dxg and the host driver’s libraries under /usr/lib/wsl, the same edits nvidia-ctk cdi generate --mode=wsl makes, for every vendor); the run gets --gpus all |
offers the gpu pool when the session’s VM has a GPU; facts containers.gpu = "cdi:microsoft.com/wslc"; container APIs directml (WSL’s D3D12 and DXCore, with any hardware GPU; the image brings libdirectml.so) and cuda (where the host’s NVIDIA driver provides its WSL library). GPU containers need a glibc image: the guest’s hook runs the image’s ldconfig |
| macOS (Apple silicon, macOS 14+) | a second agent-owned VM on krunkit (libkrun) whose virtio-gpu device carries Vulkan to the host’s GPU: Mesa’s Venus driver in the container, MoltenVK on the host; the run gets --device /dev/dri. Only jobs that reserved the gpu pool run there; other containers keep the Virtualization.framework VM with Rosetta. Metal itself never reaches a Linux container |
with krunkit installed: offers the gpu pool; facts containers.gpu = "virtio-gpu:venus"; container APIs vulkan |
On every platform the image brings the API’s user space where the mechanism does not: Mesa and the Vulkan loader. On
macOS that Mesa is the libkrun build of the Venus driver (Fedora: dnf copr enable slp/mesa-libkrun-vulkan, then
mesa-vulkan-drivers, pinned; stock Mesa fails vkCreateInstance under krunkit); on Linux, Mesa’s AMD and Intel drivers. The broker’s status answers gpus: "all" to a job that reserved the gpu pool on a node that passes GPUs through.
Testing your module
Section titled “Testing your module”oarbank-sdk conform runs the runner suite under this host’s backend with the grants the manifest declares (a bootstrap
stage’s runner specs with the bootstrap grants), plus a portability lint for every declared platform. A module that writes outside its directories, reads homes, or expects a
tool it didn’t request fails there first.
Source: spec/sandbox.md in the oarbank-sdk repository