Skip to content

The conformance kit

oarbank-sdk conform <module-dir> runs the checks a host relies on (--no-runner skips the suites that run node processes, --only workload runs only the workload suite). Run it before publishing and in CI; it exits non-zero on any failure.

Suite Checks
manifest Strict schema, UI contract references, no unknown fields, and the portability lint for every declared platform (Windows execs are .exe or the python token, CRLF line endings). A resolved view per declared platform (the runner’s exec, runtime, env and file subset under [bundle.platform_files]) and per coordinator platform (the coordinator’s exec; this host’s when coordinator_platforms is absent): every bundle path an exec names must reach that platform. The manifest and bundle lint (an unknown runner capability, an unknown mix, a mix coarser than determinism_scope while results.value is set, a platform_files glob that matches nothing) are reported as warnings
bundle The module builds into a bundle that verifies (no host imports, no symlinks)
settings The [settings].schema file parses as the core reads it (manifest.md: supported types, valid JSON Schema per key, defaults that validate, known x-oarbank fields, no credential, no required key with a default); its lint (a key named like a core module key, a name that looks like a credential) is reported as warnings. The fixtures’ settings validate against the declared keys and give a value to every required key
protocol The coordinator starts as a host on this machine starts it (its variant for this platform, OARBANK_PLATFORM, its env) and completes initialize, which carries host.platform and every host capability. It advertises every capability the manifest declares and implements every required verb. params.check, golden.list and spec.build are pure (same input, same output). By default there is one node class per declared platform, with platform set. Every node class has goldens, each golden’s stage compares (never determinism none), each golden’s platforms and expected_by_platform keys are declared platforms or their OSes, spec.build turns each golden into exactly the stage it names, and each golden is comparable (expected.digest, also per platform, or golden.compare). Each fixture operation (ops) runs through op.apply twice: it must be pure, ask only for the effects its [[operations]] entry declares, and name every datasets.create file well (digest and size always; origins https URLs on public host names). With the workload.report capability, the fixtures’ workload_reports (report data objects) are delivered as one batch from a reporting workload, twice: the answer must be a valid result, the same both times, and ask only for the effects coordinator.workload_effects declares
images For each [[sandbox.container_sets]] entry: its key is one ECDSA P-256 public key; every member the fixtures name (images) verifies with the reference policy (oarbank_sdk.images, which the agent shares), against the fixture’s OCI image layout or the set’s registry; and both refusals hold: an image outside the set’s prefix and an unsigned image inside it (a digest not in the index, for a set with one) are refused with image_not_approved
secrets (in the protocol and runner suites) Each declared secret gets a random canary (or the fixtures’ secrets value). Runner runs of a stage that lists secrets get OARBANK_SECRETS_FILE (0600, in the work directory) with exactly that stage’s names; every other run gets none. The fake host answers host.secrets.get only with secrets:read:self (-32002 otherwise). Any canary in a run’s output, result, failure.json, events or files, or in any coordinator verb’s answer (a built spec, a golden), fails the check: other stages and the coordinator side never see a value unless the manifest says so
runner doctor --json is a DoctorOutput, and its attrs.platform, when present, equals OARBANK_PLATFORM. The runner and doctor run with this host’s runner variant, env included; goldens of this host’s node class come first, and the expected value is resolved for this host’s platform. For the first golden whose datasets are available, the real runner runs the golden’s SpecEnvelope in a fresh workdir with a clean environment and writes a valid ResultEnvelope. result.evaluate accepts it and it matches the golden. A second run with another locale gives the same digest (the golden’s stage has determinism exact).
runner: stopping For a cancellable runner, the golden runs once more and gets a stop request (control.json stop and the agent’s nudge on this OS: SIGUSR1 to a runner started with it ignored on POSIX, the inherited OARBANK_CONTROL_EVENT on Windows; nothing else) as soon as the runner has written its first <W>/phase, so its start-up is never timed. The runner must acknowledge it within 2 s of the nudge, timed to its own acknowledgement (failure.json with fault = "transient", or the result if it was finishing), not to the process exit, and must exit within its stop_grace_s (75 after failure.json). A runner that writes no phase, or a golden that finishes before the stop is sent, cannot show it and the check is skipped, so name a phase when work starts and make one golden run long enough. Every runner spec of the fixtures (runner_specs, below) runs the same way and is checked against its expected outcome.
runner: GPU duty A runner that declares cooperative_throttle runs every duty spec (duty_specs, below): once it has written its first phase, the kit writes gpu_duty into control.json and nudges it, as the agent does beside a protected GPU program, waits a second, then measures for window_s: the runner must report the duty it holds (gpu_duty metric events, oarbank_sdk.duty), its reports must stay within the duty plus 0.1, and so must what the kit measures itself, the runner’s GPU time where this host shows it (macOS: the AGX driver’s per-process GPU time, as the agent reads it) or, when its reports say its units run on the CPU (a stand-in backend), its CPU time per thread; a host that shows neither checks the reports alone. A cooperative_throttle GPU runner without duty specs is a warning. Any connection the egress proxy refused fails that run, goldens included, even when the runner coped with the refusal. The runner and doctor run under this host’s sandbox backend with the module’s grants, in the per-OS job environment. A golden or runner spec whose stage reserves an endpoint service’s pool runs with that service started as an agent starts it and with its connector (OARBANK_SERVICE_<NAME>). Thumbnails a result names must be allowed images within 1 MiB.
runner: checkpoints For a stage that keeps checkpoints (stages[].checkpoint), the kit replays an interrupted run on its golden (else its first runner spec): once the runner has announced its first checkpoint it sends checkpoint-then-stop (stop with checkpoint), expects the acknowledgement (exit 75 with failure.json fault = "transient") within checkpoint_grace_s and a valid checkpoint (regular files in the workdir, unique names, within max_mb), then runs the spec again in a fresh workdir with that checkpoint under <W>/checkpoint/ and resume in its envelope: the resumed result’s digest must equal the uninterrupted one, and every file it names must be there
gpu The runner’s gpu.apis_any for this host’s platform against the APIs this host provides (oarbank-sdk gpu-apis, the probes a node’s agent runs): passes when the host provides one; otherwise every golden run, runner spec and endpoint service whose GPU APIs this host lacks is skipped with the reason, never failed, as a node like this host would get none of that work. This host’s node class carries its gpu_apis. The kit has no container runtime, so container GPU APIs are checked only on an agent
service Each endpoint service on this host’s platform that lists no listeners (one that does runs in the listener suite): its start gets an endpoint channel the kit owns, under this host’s sandbox backend as for the runner; it says hello and ready answers true within start_timeout_s; every service_specs request reaches it through a connection the kit hands it and gets the expected status; it takes ended for that attempt; stop succeeds. The service never listens itself: no process of it holds a listening socket (a TCP listener, a bound UDP socket, a unix socket accepting connections) after ready and after its requests, and a service that fails to start because the sandbox refused it a listening socket fails the check too
listener Each endpoint service on this host’s platform that lists inbound listeners (service-protocol.md), played as the agent plays it: its start gets an endpoint channel the kit owns and OARBANK_LISTENERS_FILE (every listener unmapped), plus, for a listen_fd listener, a listening unix socket the kit made in a private directory (OARBANK_LISTEN_FD_<NAME>; skipped on Windows, whose nodes refuse such a listener), under this host’s sandbox backend; it says hello and ready answers true within start_timeout_s. For each listener the kit then announces external 203.0.113.7:<port_hint, else 41000>, external_v6 [2001:db8::7]:<port>, state reachable, and hands the service a connection from 198.51.100.23:51000 (accepted on 192.0.2.10:41000, the PROXY v2 header written first when the listener has proxy_protocol): the service must take it (accepted), and every listener_specs entry for that listener must get its expected answer. The service’s own service_specs run too (a service with pools). Any listener_allow it sends must name its own listeners and IP networks. It stops, and never listens itself: no process of it holds a listening socket of its own (the socket the kit passed for listen_fd is the agent’s)
workload Each workload on this host’s platform (workload-protocol.md), with the kit playing the agent: a fresh module data directory with the run directory inside it, the private directory (control.json {"seq": 0}, the settings, tools and limits files and, for a workload that lists secrets, secrets.json 0600 with the canaries), the protocol’s environment plus OARBANK_CONFORMANCE_TOKEN, a value unique to the run, and <exec> run as the leader of a new process group (on Windows, of a Job Object the kit creates, breakaway refused), started with SIGUSR1 ignored, under this host’s sandbox backend as for the runner. It must keep running 3 s after the start; with health declared, status answers a workload-status-1 object with healthy: true within health.timeout_s; with freeze_ok, it survives the kit freezing its whole group (every member stopped, then continued); with cooperative_pause, a nudged pause: true shows as paused within 3 s (its supervisor’s .supervisor.json, a paused progress event, or its status op) and clearing it shows as resumed within 3 s. It stays in its process container: every process whose environment carries the run’s token (/proc/<pid>/environ on Linux, ps -E on macOS) must be in the leader’s process group, so a daemon that calls setsid and forks twice fails (the kit kills it afterwards); on Windows the Job Object holds every process by construction. It never listens (no process of the run holds a listening socket, and one the sandbox refused a listening socket fails too); every line of its events file is a workload-event-1 event; no secret canary appears in its log, events or files; and a nudged stop: true ends the leader within stop_timeout_s. --only workload runs this suite alone; --no-workloads skips it

Workload reports (workload_reports) are data objects of report events, as a module’s workload writes them; the protocol suite delivers them to workload.report (module-protocol.md).

Service specs are requests to an endpoint service (service-protocol.md): name, service, method (default POST), path, an optional JSON body and expect.status (default 200). The kit sends each one, as HTTP/1.1, through a connection it hands the service on its endpoint channel.

Listener specs are clients of an inbound listener: name, listener, and either send (UTF-8 text the kit writes; it then reads until the answer contains expect.contains or the service closes, at most 10 s) or path (one HTTP/1.1 request: method, default GET, an optional JSON body, expect.status, default 200, and optionally expect.contains in the body). Each runs on its own connection the kit hands the service, after the kit’s announcement, so an answer may include the announced address. A listener without specs is checked only for taking a connection (the report says so).

The kit does not call op.plan (nor op.apply beyond the fixture ops), check that enqueued job_keys equal the keys of job.plan’s inputs, validate result payloads against results.schema, or run more than the first runnable golden (list other tasks as runner_specs). It runs the runner and the coordinator only on this host’s platform: run it on each platform you declare (a CI matrix). Placement is the host’s job; the kit only checks the declarations. Test those in your own suite.

An optional conformance.json next to the manifest feeds the kit’s fake host:

{"datasets": {"scene:demo": {"kind": "scene", "attrs": {"scene": "atrium"}, "dir": "fixtures/demo"},
"upload:shots": {"kind": "upload", "files": [{"path": "a.png", "digest": "<64 hex>", "size": 2048}]}},
"settings": {"tool_datasets": {"renderer": "tool:renderer"}},
"tools": {"renderer": "/opt/renderer/4.2", "jdk": {"path": "/opt/homebrew/opt/openjdk@17", "version": "17.0.12", "arch": "aarch64"}},
"node_classes": [{"platform": "darwin-arm64", "os_version": "26.1", "pools": {"containers": 1}, "capabilities": []},
{"pools": {"imagediff": 0}}],
"params": [{"n": 3}, {"n": -1}],
"store": {"study/c_demo": {"name": "demo"}},
"folders": {"inputs": "fixtures/inputs"},
"ops": [{"verb": "import_model", "params": {"url": "https://models.example.org/m.bin", "sha256": "<64 hex>", "size": 1024}},
{"verb": "adopt", "target": "upload:shots"}],
"runner_specs": [{"name": "fetch-tools", "stage": "fetch", "payload": {"task": "fetch", "dataset_id": "tool:renderer-4"},
"expect": {"exit": 0, "artifacts": ["files"]}},
{"name": "sync-bad-cursor", "stage": "sync", "payload": {"task": "sync", "cursor": -1},
"expect": {"exit": 2, "reason": "render/bad_cursor"}}],
"service_specs": [{"name": "generate", "service": "model", "method": "POST", "path": "/v1/generate",
"body": {"prompt": "hello"}, "expect": {"status": 200}}],
"duty_specs": [{"name": "batch-at-0.3", "stage": "batch", "duty": 0.3, "window_s": 4,
"payload": {"prompts": ["a long enough task"], "max_tokens": 4000, "backend": "stub"}}],
"workload_reports": [{"unit": "3f2a91c0-1", "amount": 137}],
"listener_specs": [{"name": "echo", "listener": "echo", "send": "hello\n",
"expect": {"contains": "echo hello 203.0.113.7:9000"}}]}

Without node_classes the kit asks for one class per declared platform ({"platform": "<token>"}); a node class without platform gets only goldens that are not limited to some platforms.

A dataset with a dir (relative to the module) is copied into the runner’s workdir under the mount name the golden’s spec gives it. A dataset’s files are what host.datasets.query returns with with_files (an importer operation’s input). A golden needing a dataset the fixtures lack skips the runner suite for that golden, and the report says so.

The runner suite gives the runner what an agent would:

  • duty_specs ({name, stage?, payload, datasets?, mounts?, duty (0 to 1, default 0.3), window_s (default 4)}) are tasks long enough to measure the duty on (the kit stops a cancellable runner after the window, else waits for it to finish).
  • tools maps each [sandbox].tools id to an installation on this machine: a path, or {path, version, arch}. The kit resolves the path (as the agent does), grants it in the sandbox and lists it in OARBANK_TOOLS_FILE; a version the request’s constraint does not accept fails the check host tool versions.
  • settings are the owner’s values ({key: value}), checked against the declared keys. As on a node, runs, doctor and services get the node-scoped keys in OARBANK_SETTINGS_FILE (a fixture value, else the key’s default; a key with neither is absent); the coordinator side gets every key the same way (initialize, host.settings.get).
  • secrets (optional {name: value}) replaces the random canaries; the kit delivers each to the stages that list it, as an agent does.
  • images ({"<set name>": {"layout": "fixtures/images", "members": ["ghcr.io/org/tasks/t1@sha256:…"]}}) names members to verify and, optionally, an OCI image layout to read them from (oarbank_sdk.imagetest writes one; without it the set’s registry is asked).
  • folders maps each read folder of [sandbox].folders to a directory (relative to the module); every write folder gets a fresh empty outbox. The kit grants them in the sandbox and lists them in OARBANK_FOLDERS_FILE.
  • net.mode = "egress-allowlist" starts the reference allowlist proxy (oarbank_sdk.egress_proxy) with the manifest’s allow list and sets HTTPS_PROXY/HTTP_PROXY/ALL_PROXY; the sandbox lets the runner reach only it.
  • A golden whose stage reserves the containers pool is skipped: the kit has no container broker.

Runner specs are the runner’s other tasks (an ingestion sync, a fetch that provisions tools): work no golden exercises, whose grants (egress above all, including redirects to allowed hosts such as a release page’s asset host) would otherwise first meet the sandbox on a real agent. Each entry has a name, an optional stage (default: the default stage; never a stage that runs after another, which would need inputs), the payload, optional datasets and mounts (fixture datasets, mounted as for goldens) and expect: exit (default 0), artifacts (the exact artifact names, for exit 0) and reason (the failure.json reason, for other exits). The kit builds its envelope as an agent on this host would (the stage’s resources and timeout), runs it sandboxed with the module’s grants and the egress proxy, and reports the exit code, the result’s artifact names (each must be a Name, which a real agent also requires) or the failure.json (required for a non-zero exit). A spec whose datasets the fixtures lack, or whose stage reserves the containers pool, is skipped with the reason.

A runner spec for a bootstrap stage runs with the bootstrap grants instead (sandbox.md): the allowlist proxy, no tools, {} as settings, no module data directory. For exit 0 the kit also checks what the host will: the payload is empty, and every artifact’s files, hashed from the workdir, are exactly one pinned dataset’s (“artifacts match the pinned datasets”, naming the first file that differs). So a fetch is proven against the real origins, through the real allowlist, before the module ships.

A runner spec for a cleanup stage runs as the agent runs it (runner-protocol.md): the cleanup grants (no tools, no folders; the module data directory and its settings), an empty job_key and OARBANK_ATTEMPT_ID=0. Only the exit code and, for another exit than 0, failure.json are checked: the agent reads no result of a cleanup run. Give payload the reason the run should see.

Source: spec/conformance.md in the oarbank-sdk repository