Manifest (schema 1)
Every module bundle has a oarbank-module.toml at its root. It holds every fact the core needs before running module code: identity, compatibility, entry points, stages and resources, services and probes, result fields, dataset kinds, goldens, the declarative UI and the module CLI. Behaviour stays in code (the module protocol verbs).
Rule of thumb: if the scheduler, the installer or the console needs a fact to decide something, declare it in the manifest. If deciding needs the module’s judgement, it is a verb.
- Every key, with its type, default and stability label: manifest-reference.md, generated from the models.
- Schemas:
schemas/manifest-1.schema.json(lenient, what the core reads) andschemas/manifest-1.strict.schema.json(for editors: unknown keys are errors). - Validate a manifest with
oarbank-sdk check oarbank-module.toml. - Worked examples: toy (minimal, the reference module), the bench module’s manifest in its repository, oarbank-module-bench (typical), and render, an invented module the SDK’s tests use (a stage chain, a service and a probe, datasets, result fields, a CLI).
Names and interpreters
Section titled “Names and interpreters”- The module’s short name (“module-short”) is the last dot-separated part of
module.id(dev.example.primes→primes). It names the module in operation ids (mod.<short>.<verb>, with-as_), schema references (<short>/spec@N), reason codes (<short>/<code>), the release layout (modules/<short>/) andOARBANK_MODULE.
An exec (coordinator.exec, runner.exec, coordinator and runner variants, services, probes, workloads, cli.exec)
is an argv array, never a shell string.
- Element 0 is either:
- the token
python: the Python of the module’s own environment on every OS (bin/python, orScripts\python.exeon Windows), never whateverpythonis on the PATH. It needsruntime.kindpython(the host’s managed CPython with the SDK, plus the bundle’s requirements installed from wheels) oruv(a locked environment, wheels-only for every declared platform); or {bundle}/<PortablePath>: a native executable in the bundle (runtime.kind = "native"). On Windows it names its.exe; the host never appends extensions. Shebang scripts are valid only in darwin and linux variants.
- the token
.bat,.cmdand.ps1are never valid as element 0.{bundle}may appear in any element and is replaced by the bundle’s absolute native path. Nothing else is rewritten: relative paths are not resolved for you.- cwd: the bundle root for the coordinator, services and probes; the work directory for
run; the data directory fordoctor; the run directory for a workload’srunandstatus. oarbank_sdk.manifest.resolve_execis the reference resolver.
Platforms
Section titled “Platforms”requires.platformsis required: the node platform tokens the module runs on (platforms.md).requires.coordinator_platformslists the coordinator host platforms the coordinator side runs on (absent: any), and[requires.unsupported]gives reasons for what is not supported.requires.osgives per-OS version ranges (darwin,windows, andlinux = {kernel, glibc}).[runner.variants."<platform>"](or[runner.variants.<os>]) overridesexec,runtime,capabilities,stop_grace_s,gpuandenvfor that platform. The most specific key wins.[coordinator.variants.<key>]does the same for the coordinator host (exec,runtime,timeouts_s,concurrency,env), and[stages.variants.<key>]for a stage (timeout_s,requires.resources,retry).env,timeouts_sand resources merge key by key; every other field replaces (platforms.md).platformson services, probes, workloads andstages[].requiresrestricts them to some declared platforms.- One bundle carries every platform’s files: one digest, one approval, one
compat.[bundle.platform_files]says which files only some platforms’ nodes receive. [placement]keeps each campaign, group, dataset or pipeline on one platform class (platforms.md).- These per-platform and placement keys need
requires.core >= 2.2.
Sections
Section titled “Sections”| Section | What it declares |
|---|---|
manifest |
The schema major, 1. |
[module] |
id (reverse DNS; never reused), version (SemVer), compat (part of every job key), publisher, SPDX license, codeowners, stability. |
[requires] |
Supported core, agent and OS ranges, node and coordinator platforms, unsupported reasons, the protocol majors spoken, and must-understand features. experimental lists opt-ins. |
[coordinator] |
The module-protocol process: exec, runtime, optional-verb capabilities, concurrency, per-verb timeouts_s, host-callback permissions, the effects campaign.tick may request, env, and per-platform variants. |
[coordinator.move] |
The module’s part in a coordinator move: rules (a files prefix or a store collection, with class carry, rebuild or drop) and the effects the move verbs may request (module-protocol.md). |
[runner] |
The runner-protocol executable: exec, runtime, capabilities, stop_grace_s, checkpoint_grace_s, gpu, bandwidth_class, env, and per-platform variants. |
[[stages]] |
At least one. Each has name, optional after, and requires (node capabilities, reserved pools, needs_pools that must merely exist, and resources cpu/mem_gb/network), plus timeout_s and retry, per-platform variants, a placement constraint with its after stage, and, for a standalone stage, default (the stage a job runs when it names none) and its own determinism (none for work whose results depend on when it ran, such as ingesting a moving feed: the host never replicates, compares, caches or golden-tests it, and with no capability or pool required it runs before certification: Stages that run before certification). A stage’s checkpoint (max_mb, min_interval_s) keeps portable checkpoints (Portable checkpoints). One stage may be the cleanup stage, which no job runs (Cleanup stage). |
[[services]] |
Node helpers the agent manages through the service protocol: lifecycle, timeouts, restart policy, which pools and capabilities they provide, the memory, yield and pause flags, GPU use (gpu), endpoint for a service jobs reach through the agent (a warm model server), and the inbound listeners it serves. |
[sandbox] |
What the module’s node-side processes need beyond their own directories, each an operator approves per version: the network (net.mode, net.allow), inbound listeners the agent opens for it (net.inbound), host tools, the GPU, container images and image sets, folders (sandbox.md). |
[[probes]] |
Read-only capability checks (fingerprint only), each run every period_s. |
[[workloads]] |
Supervised long-running processes that are the work itself, run by the agent through the workload protocol on the nodes the owner assigns them to: exec, seat and identity, the secrets and listeners each receives, resources, gpu, control (what host protection may do besides stopping it), pause_tolerance_s, stop_timeout_s, restart, health and reports (Workloads). |
[settings] |
schema: a JSON Schema file whose properties are the module’s settings, each with its type, default, constraints, title, description and an x-oarbank annotation (scope fleet or node, required, unit, advanced, campaign). The core validates every value against it (Settings). Settings are visible to the owner: never put a credential in them. |
[[secrets]] |
Write-only credentials the owner sets through the core: name, description (Secrets). |
[placement] |
Which unit of work stays on one platform class (mix, unit), how it binds (bind) and what happens when its class has no eligible node (rebind, stranded_after_s). |
[results] |
The payload schema and its version, determinism and its determinism_scope, the digest (version, over), the objective value, the inline size limit, and declared fields (typed; indexed promotes a field to a sortable column; ui sets column, format and unit). |
[datasets] |
The dataset kinds the module registers (short names, scoped by the owning module), their typed attrs, and the platform_bound kinds. |
[bundle] |
executables globs (mode 755) and platform_files (glob → the platforms or OSes whose nodes receive the files). |
[goldens] |
The fixtures glob and the comparison mode (digest, or verb to call golden.compare). |
[ui] |
Declarative contributions only: a digest_line template, study_columns, icon. |
[cli] |
The module CLI. oarbank cli <module> [args...] runs it on the coordinator, sandboxed (its bundle read-only, a scratch directory writable, only the admin API reachable), with OARBANKD_URL and OARBANK_TOKEN: a one-hour token limited to the module’s own operations and reads. |
Cross-field rules
Section titled “Cross-field rules”The models enforce these, beyond the per-field types:
-
Stage names are unique.
aftermust name another stage, and stage dependencies form no cycle. -
Every pool a stage requires (
poolsorneeds_pools) is provided by a declared service. Every required capability is provided by a service or a probe. -
If services or probes are declared,
requires.service_protocollists at least one major. -
results.value.fieldis a declared result field. -
A module with more than one stage that jobs run (the cleanup stage does not count) implements
result.merge(listed incoordinator.capabilities). Thecampaign.tick.resultscapability requirescampaign.tick. -
goldens.compare = "verb"requires thegolden.comparecapability. -
With
runtime.kind = "uv", bothlockandpythonare set. -
A move rule names exactly one of
filesorstore. Arebuildrule requires themove.postflightcapability, andcoordinator.move.effectsrequires at least one move verb. -
Variant keys (runner, stages) and
bundle.platform_filesvalues are declared platforms or OSes of one. Coordinator variant keys are coordinator platforms or their OSes whenrequires.coordinator_platformsis set. -
requires.unsupportednever contradicts the allow-lists: arunnerkey is not a declared platform or the OS of one, andcoordinatorkeys needrequires.coordinator_platformsand are not in it. -
envnames (runner, coordinator, their variants) match^[A-Z][A-Z0-9_]*$and are never reserved:OARBANK_*,PATH,HOME,USERPROFILE,SYSTEMROOT,TEMP,TMP,TMPDIR,LOCALAPPDATA,APPDATA, the XDG base directories and the other variables the host sets (platforms.md). Stage variant resources keep the stage’s bounds. -
A key older cores would ignore needs a
requires.corerange whose lower bound is at least the core that understands it (versioning.md):- 2.2: the per-platform and placement keys (
requires.coordinator_platforms,requires.unsupported,requires.features,coordinator.env,coordinator.variants,runner.envand runner variantenv,stages[].variants,stages[].placement,[placement], adeterminism_scopeother thanglobalorplatform,bundle.platform_files,datasets.platform_bound); - 2.3:
stages[].determinism,stages[].default, the coordinator capabilitycampaign.tick.results, and the effectsdatasets.updateanddatasets.deletein any effects list (coordinator.campaign_effects,operations[].effects,coordinator.move.effects). - 2.4:
stages[].bootstrapanddatasets.pinned. - 2.5:
[[secrets]],stages[].secrets, the permissionsecrets:read:self,sandbox.container_sets, a stage reserving the agent’sgpupool,runner.gpu.apis_any(or a runner variant’s),services[].endpoint,services[].gpu,sandbox.folders,stages[].checkpoint,runner.checkpoint_grace_s, the runner capabilitycheckpointand the view cell typeartifact_ref. - 2.9: a
sandbox.tools[]entry with aversionor anarchother thanany(sandbox.md), and, in the settings schema, a key withx-oarbankscopenode, a required key (x-oarbank.requiredor the schema’srequiredlist) orx-oarbank.campaign(Settings).oarbank-sdk check,bundle buildandbundle verifyapply the settings part, since the schema is a bundle file. - 2.10:
sandbox.net.inbound,services[].listeners,[[workloads]],requires.workload_protocol, the coordinator capabilityworkload.reportandcoordinator.workload_effects. - 2.11:
stages[].cleanup.
Every entry of
requires.featuresis one this SDK knows. - 2.2: the per-platform and placement keys (
-
Every bundle path a node exec names (argv[0], or the script a
pythonexec runs) reaches each platform that runs it underbundle.platform_files.datasets.platform_boundkinds are declared kinds. -
stages[].defaultis set on at most one stage, a standalone one; when several stages are standalone, exactly one sets it.stages[].determinismis set only on standalone stages (neitherafteranother nor depended on): a chain is one evaluation and compares asresults.determinism. At least one stage compares (its effective determinism isexactorwithin_tolerance): goldens run only on such stages, and every module is certified on golden evidence. -
stages[].bootstrapis set only on a standalone stage that is not the default stage, whose effective determinism isnoneand which reserves and needs no pools. A module with a bootstrap stage pins at least one dataset ([[datasets.pinned]]), pins need a bootstrap stage, and itssandbox.net.modeisnoneoregress-allowlist. Pinned dataset ids are unique; each pin’skindis a declared kind, itsplatformis set exactly when the kind is platform-bound and is a declared platform, its file paths are unique, and no two pins hold the same files (Pinned datasets). -
Secrets: names are unique; a stage lists only declared secrets, each once, and a bootstrap stage lists none; every declared secret reaches something (a stage or a workload lists it, or the coordinator has
secrets:read:self). -
Container sets: names are unique; each
keyis a bundle file holding one ECDSA P-256 public key (checked byoarbank-sdk checkand bundle verification). A stage that reserves thegpupool also reservescontainers, and the runner declaresgpu.in_container = truewithgpu.usesharedorexclusive(sandbox.md). -
An endpoint service (
endpoint = true) provides at least one pool (a job reaches it through a pool its stage reserves) or lists at least one listener (rule 22), and itslifecycleison_demandoralways(the agent never starts amanualservice, so it could never hand it its channel). A service whosegpu.useis notnoneneeds[sandbox].devices.gpu = "compute". -
sandbox.foldersids are unique, each withaccessreadorwrite(sandbox.md). -
A stage with
checkpointneeds the runner capabilitycheckpoint, and the capability needs at least one such stage; a bootstrap stage never setscheckpoint. -
gpu.apis_any(the runner’s, a runner variant’s, a service’s) is set only withgpu.usesharedorexclusive: it places GPU work (runner-protocol.md). -
Inbound listeners (sandbox.md):
sandbox.net.inboundnames are unique; each entry is listed inlistenersby exactly one service or workload, and every name in a service’slistenersis an entry, once; a service withlistenersis an endpoint service withlifecycle = "always"(a listener is wanted for as long as the module is enabled on the node). An entry’sprotocolistcp,port_hint1024–65535,max_connsandmax_conns_per_ip1–65535,new_conns_per_ip_per_sandmax_bytes_per_sabove 0,idle_timeout_s1–86400.inboundneeds no network mode: answering an accepted connection is not egress. A workload may serve listeners too (rule 24). -
Lint (warnings, not errors): an unknown
mix; a stageplacementon a stage withoutafter; an unknowndeterminism_scope; a placement mix coarser thandeterminism_scopewhileresults.valueis set (values in one campaign would come from classes whose results are not comparable); stages receiving secrets while the network mode isegress-any; agpu.apis_anyentry no core detects yet (outsidecuda,directml,metal,opencl,rocm,vulkan: such work is placed nowhere); a listener withhandover = "listen_fd"whose service runs on a declared Windows platform (Windows nodes refuse it:LISTENER_HANDOVER_UNSUPPORTED).oarbank-sdk checkprints them;oarbank_sdk.manifest.lintreturns them. -
Workloads (Workloads):
- names are unique among workloads and services (both appear as
<module>/<name>in node reports); - a workload lists only declared secrets, each once;
identity = "per_node"lists at least one (its seat is named by the node’s values of them); gpu.useother thannoneneeds[sandbox].devices.gpu = "compute", andgpu.apis_anyneedsusesharedorexclusive;controllists each item once;pause_tolerance_sabove 0 needscooperative_pauseorfreeze_ok. It defaults to 20 for a workload that declares either of them and to 0 otherwise (such a workload is never paused: it is throttled where it can be, lowered, or stopped); it is at most 600, the pause ceiling;listenersname[sandbox].net.inboundentries (rule 22: each served by exactly one service or workload) with handoverconnections(a workload is handed its connections, never a listening socket);restart.backoff_max_sis at leastrestart.backoff_initial_s;- declared workloads need
requires.workload_protocol(and, rule 12,requires.core >= 2.10); reports = trueneeds the coordinator capabilityworkload.report, the capability needs a workload withreports = true, andcoordinator.workload_effectsneeds the capability;- lint: a workload under
sandbox.net.mode = "egress-any"while the module’s platforms include Linux (Linux nodes cannot enforce that mode, so the core places such workloads on macOS and Windows nodes only), and workloads that receive secrets underegress-any.
- names are unique among workloads and services (both appear as
-
stages[].cleanupis set on at most one stage: a standalone stage that is not the default stage and not a bootstrap stage, whose determinism isnoneor unset, which requires no capability, reserves and needs no pools, lists no secrets and sets nocheckpointorplacement. A module has at least one other stage. The cleanup stage is not counted as standalone (rule 14) or as a stage forresult.merge(rule 5), and it never compares (Cleanup stage).
A module may offer both forms of an evaluation. For example, render declares a single eval stage and a render → score chain; the operator’s pipeline setting picks the form for jobs that name no stage. A module may also declare standalone utility stages (an ingestion sync, a fetch that provisions tools) and enqueue jobs that name them; it then marks its evaluation stage default = true.
Stages that run before certification
Section titled “Stages that run before certification”A node runs a module’s work only once the module is certified there: its doctor is healthy and its goldens passed (golden jobs on the stages that compare). Certification is what lets a node’s results count: it shows that the node computes the module’s comparable results exactly as the goldens expect. Two kinds of stage need no certification, because the goldens say nothing about their results and no other node’s result is ever checked against theirs:
- a bootstrap stage (Pinned datasets): its output counts only when it is exactly the pinned datasets, which the host checks file by file;
- a stage that compares nothing and needs nothing a node must prove: its effective determinism is
none(never replicated, compared, cached or golden-tested, rule 14) and it requires no capability (requires.capabilities) and no pool (requires.pools,requires.needs_pools). An ingestionsyncthat pulls a moving feed is the usual case.
Manifest.certification_exempt(stage) says which stages these are. Their jobs run on any node where the module’s
runner starts (the agent ran its doctor and got a DoctorOutput), whatever the doctor’s health: a module that is
unhealthy or undetected on a node is not certified there, but its exempt stages still run. A failed doctor check
gates only the stages that need what it proves: a check named after a capability (a probe’s name, or a capability a
service provides) that fails takes that capability away from the module on that node, so a stage that requires it
waits there (STAGE_CAPABILITY_MISSING), and a stage that does not keeps running. Everything else is checked as for
any job (platforms, sandbox, secrets, placement, retries). A stage with determinism = "exact" or
"within_tolerance", or one that needs a capability or a pool, always waits for certification: its result is
replicated, compared and served from the result cache, and only certified nodes’ results may enter that.
Results of exempt stages are still judged by the module (result.evaluate) and fenced by job generation; they never
count toward certification. A node whose jobs keep failing trips its breaker as for any job, which revokes the module
there until its doctor runs again.
Pinned datasets
Section titled “Pinned datasets”A module that provisions its own tools or reference data (downloads from public origins) does it with a bootstrap stage: on a fresh fleet its goldens mount those datasets, so the jobs that fetch them must run before any node is certified.
[[stages]]name = "fetch"bootstrap = true # runs where the module's runner starts, before the goldens passdeterminism = "none" # never compared, cached or golden-testedtimeout_s = 3600requires = { resources = { cpu = 1, mem_gb = 1.0 } }
[[datasets.pinned]]dataset_id = "tool:gatk-4.5.0.0"kind = "tool"meta = { version = "4.5.0.0" } # optional: the registered dataset's metafiles = [{ path = "gatk-package-4.5.0.0-local.jar", sha256 = "<64 hex>", size = 1234567 }]- A bootstrap job runs with the bootstrap grants (sandbox.md): the module’s egress allowlist, its work directory and nothing else.
- Its result has an empty
payloadand one artifact per dataset it fetched, each holding exactly one pin’s files (runner-protocol.md). The host registers each such dataset itself (the pin’s id, kind, meta, platform and files, owned by the module) and calls no module verb on the result. Anything else is refused withpin_mismatch, counted against the job and never against the node, and nothing is registered. datasets.createof a pinned id succeeds only with the pinned contents, so the module’s own code may register a pinned dataset too.- The pins are part of the manifest, so the bundle digest an operator installs and approves covers them.
- A job runs a bootstrap stage only when it names it (
jobs.enqueuestage); its results never count toward certification.
Cleanup stage
Section titled “Cleanup stage”When a module leaves a node (the owner turned it off there, it is no longer in the node’s release, or the node was retired), the agent deletes what the module put there: its data directory, the datasets it cached and the container images it pulled (sandbox.md). The module’s processes can only write there, so the core needs no help from the module. A module that also holds state outside the node (a registration with an outside service, a lease, a remote resource) declares a cleanup stage to release it:
[[stages]]name = "release"cleanup = true # the agent runs it before it deletes the module's data on a nodetimeout_s = 120 # at most 600 srequires = { resources = { cpu = 0.5, mem_gb = 0.5 } }- The agent runs it on the node, once, just before the deletion, with the cleanup grants
(sandbox.md) and the spec the runner protocol describes
(runner-protocol.md). No job ever runs it:
jobs.enqueuerefuses a job that names it. - It is best effort. A run that fails, crashes or outlasts its timeout is logged and reported to the owner, and the deletion goes ahead anyway.
- It runs only on the platforms the stage’s
requires.platformsadmits (every declared platform when empty), and only while the node still has the module’s bundle. - Most modules need none: the core already deletes everything on the node.
Settings
Section titled “Settings”A module’s settings are the choices an owner makes for it: a VM size, a pipeline threshold, a gate’s numbers. The
module declares each one in the JSON Schema file [settings].schema names, one property per setting:
[settings]schema = "schemas/settings.schema.json"{ "type": "object", "properties": { "vm_mem_gb": {"type": "number", "minimum": 2, "maximum": 64, "default": 8, "title": "VM memory", "description": "Memory the module's VM gets on a node.", "x-oarbank": {"scope": "node", "unit": "GB"}}, "credits_per_round": {"type": "number", "minimum": 0, "title": "Credit pool per round", "description": "For the gate's expected value.", "x-oarbank": {"scope": "fleet", "unit": "credits", "required": true}} }}-
The property is the contract. Its
type(boolean,integer,number,string,arrayorobject, optionally with"null") and every JSON Schema constraint on it (minimum,maximum,enum,pattern,maxLength,items,format, …) are what the core checks each value against, on every write.defaultmust be valid itself.titleis the label the owner reads (at most 80 characters; the key’s name otherwise) anddescriptionits one line of help (at most 500). A name is lower-case letters, digits and_, starting with a letter, at most 64 characters; a schema declares at most 128 settings. -
x-oarbanksays how the core treats it:Field Default Meaning scope"fleet"fleet: one value for the whole fleet, which the module’s coordinator side reads (host.settings.get,initialize).node: a value the owner sets for the fleet, a node group or one node (the most specific wins); each node’s runners, doctor and services receive it inOARBANK_SETTINGS_FILErequiredfalse(or the schema’s ownrequiredlist)The owner must set it: until a value exists the module’s work waits ( SETTINGS_NOT_SET) and its readiness checklist shows the step “Required settings”. A required setting has nodefaultunitnone A short word shown beside the value ( GB,s,credits)advancedfalseShown under the module’s Advanced disclosure campaignfalseA campaign of the module may override it while the campaign runs ( campaigns.createorcampaigns.updatesettings, or the owner on the campaign’s page), bounded by the owner’s locks. The campaign’s value reaches that campaign’s jobs only: anodekey in their runners’OARBANK_SETTINGS_FILE, every key incampaign.tick(campaign.settings). Core 2.9 and later (requires.core >= 2.9, rule 12) -
What a process receives. Runners,
doctorand services get the node-scoped settings, resolved for their node: everynodekey with the owner’s value, else its default (a key with neither is absent), and nothing else, inOARBANK_SETTINGS_FILE(oarbank_sdk.settings.read()). Bootstrap jobs get{}. The coordinator side gets every key at fleet scope the same way. A change to one module’s setting never touches another module’s. -
Writes are validated. The owner sets values in the module’s Settings tab or with
oarbank settings set <key> <value> --module <module> [--node N | --group G]. A module’s own operations change its settings with themodule_settings.updateeffect (module-protocol.md): each key must be declared and each value valid, or the whole effect is refused;nullresets a key to inherit. -
Upgrades. The core reads the schema of the module’s current version (before it is enabled, the newest installed one). A key a new version drops keeps its values, unused, until the owner resets them or a later version declares it again; a value a new version’s schema no longer accepts is dropped, and the module event names it.
-
The core’s keys per module are not yours to declare:
enabled(the module on the fleet or on one node),services.disabled,pipelineandreplica_rateexist for every module and are set as[module] <key>.oarbank-sdk checkwarns about a property with one of those names. -
Credentials are never settings. A property with
writeOnly,x-secretorx-oarbank.secretis refused, and a name that looks like a credential gets a warning: declare it in[[secrets]].
oarbank-sdk check, bundle build and bundle verify refuse a schema the core would not read, and the conformance kit
checks the fixtures’ settings against it (conformance.md).
Which core. Cores before 2.9 stored a module’s settings as one opaque object, so a schema that relies on a node
key, a required key or a campaign override needs requires.core >= 2.9 (cross-field rule 12);
oarbank-sdk check, bundle build and bundle verify refuse it otherwise. Fleet keys with a unit or advanced
need no newer core.
Migrating a module written for SDK 1.5 or earlier. The core used to store a module’s settings as one opaque
object. Now: (1) declare every key the module reads or writes in its settings schema, (2) mark those its runners or
services need on a node "x-oarbank": {"scope": "node"} (only those reach OARBANK_SETTINGS_FILE), (3) move any
credential to [[secrets]], (4) drop a pipeline property (the core’s [module] pipeline replaces it). An
undeclared key in module_settings.update is refused, and values an owner set before are kept only for declared keys
that validate.
Secrets
Section titled “Secrets”A credential (an API key for a model provider, a token for a private package registry, a licence key) is a secret, never a setting:
[[secrets]]name = "llm_api_key"description = "API key for the model provider the agent harness calls"
[[stages]]name = "attempt"secrets = ["llm_api_key"] # only this stage's runner receives it- The owner sets it through the core (
oarbank secret set <module> <name> [--node N], or the module’s Settings page), for the module or for one node; a node’s own value wins. The core stores it encrypted, and no page, API read, plan, audit row, export or error message ever shows it: only whether it is set, a fingerprint and when it changed. - Delivery. Only the runners of stages that list it, and workloads that list it, get it, in
OARBANK_SECRETS_FILE: an owner-only file inside the job’s work directory, deleted with it (runner-protocol.md;oarbank_sdk.secrets.get), or in the workload’s private directory, deleted when the run ends (workload-protocol.md;oarbank_sdk.workload.secrets). Other stages, services, probes, doctor and bootstrap jobs never do. A job waits (SECRETS_NOT_SET) on a node with no value for a secret its stage lists, and so does a workload’s seat. - The coordinator side reads the module’s value with
host.secrets.getonly whencoordinator.permissionslistssecrets:read:self(module-protocol.md). Few modules need it; a value a verb puts in a spec reaches every stage, which the conformance kit refuses. - Redaction of exact values in captured logs is a safety net, not a guarantee: never log a value, write it into results or artifacts, or pass it on a command line.
- Pair it with an egress allowlist naming only the service the key is for, so a leaked key can reach only that host.
- Coordinator moves carry secrets re-encrypted for the new coordinator; the move preview lists them by name.
Workloads
Section titled “Workloads”Some work is a long-running process that is the work itself: a node of a peer-to-peer network, a pool worker, an inference provider. Declare it as a workload; the owner assigns it to nodes, and the agent runs it there through the workload protocol:
[requires]core = ">=2.10,<3"workload_protocol = [1]
[[workloads]]name = "worker"exec = ["python", "-I", "{bundle}/worker.py"] # the agent appends `run` or `status`; usually the SDK's supervisor shimseat = "node" # one seat per assigned node; "fleet": the owner asks for a count and the core places themidentity = "none" # "per_node": the seat is named by the node's values of its secrets (at most one node runs it)secrets = [] # declared secrets this workload receives (only it, of all the module's processes)resources = { cpu_cores = 2, mem_gb = 1.0 }control = ["cooperative_throttle", "freeze_ok"]pause_tolerance_s = 20 # the longest pause its peers survive: protection stops it instead of pausing longerstop_timeout_s = 30 # from the stop request to the killrestart = { backoff_initial_s = 10, backoff_max_s = 600, max_failures = 5, jitter = "full" }health = { period_s = 15, timeout_s = 5, failures = 3 } # the `status` op; absent: a running process is healthyreports = true # its report events reach the module's workload.report verb
[coordinator]capabilities = ["workload.report"]workload_effects = ["store.write"]- Not a job. A workload has no leases, results, goldens or retries elsewhere: it runs where it is assigned, in the module sandbox with the module’s approved grants, in a process container of its own, until it is unassigned or held.
- Seats.
seat = "node": the owner assigns it to nodes (workloads.assign).seat = "fleet": the owner asks for a count and the core places the seats once, on eligible nodes; it never moves one on its own. Withidentity = "per_node"a seat runs only under a coordinator lease with a fence, named by the node’s values of the workload’s secrets, so two nodes given the same key compete for one seat; every such secret must be set for the node itself. - Host protection treats a running workload as fleet work with its own ladder: no new start while work is not
admitted, then the
threadshint (cooperative_throttle), then lower, then pause (cooperative_pausethrough the control document, orfreeze_ok) for at mostpause_tolerance_s, then stop and hold for as long as the cause lasts. A workload that declares neither pause item is never paused. - Network-heavy.
resources.network = "heavy"says the workload moves a lot of data over the node’s internet line. At a site it counts against the site’s budget of network-heavy work, and while a protected process at the site is in an upload phase the agent pauses it on every node of the site (cooperative_pauseorfreeze_ok, at mostpause_tolerance_s), or stops and holds it until the phase ends (Network-heavy stages). - Secrets a workload lists reach only that workload, in an owner-only file deleted when its run ends (Secrets).
- Reports. With
reports = truethe workload’sreportevents reach the coordinator’sworkload.reportverb, whose effects are limited tocoordinator.workload_effects, the module’s own records (store.*,files.*) (module-protocol.md). oarbank_sdk.workload.Supervisorwraps a third-party daemon as a workload;oarbank-sdk conformruns the workload suite (conformance.md). The how-to: workloads.md.
Portable checkpoints
Section titled “Portable checkpoints”A long job on a machine people also use may be paused or moved by host protection. A runner that keeps portable checkpoints loses only the work since its last one:
[runner]capabilities = ["cooperative_pause", "checkpoint"]checkpoint_grace_s = 120 # from a checkpoint-then-stop request to the kill
[[stages]]name = "train"checkpoint = { max_mb = 4096, min_interval_s = 600 }- The runner announces each checkpoint with a
checkpointevent naming its files; the agent takes them away and uploads them (at mostmax_mb, at most one permin_interval_s), and the coordinator keeps the job’s latest. - When protection would evict the job, the agent asks for a checkpoint first (
stopwithcheckpointin the control document) and waitscheckpoint_grace_s. - The job’s next attempt, on any node, starts with the checkpoint under
<W>/checkpoint/andresumein its spec envelope (runner-protocol.md). A resumed attempt must give the same result as an uninterrupted one; the conformance kit replays an interrupted run to check it.
Network-heavy stages
Section titled “Network-heavy stages”Nodes that share one internet line slow each other’s large transfers, and an owner’s own program may have to upload on time. The owner groups such nodes into a site with an uplink budget; a stage whose jobs move a lot of data over the line themselves says so:
[[stages]]name = "publish"requires = { resources = { cpu = 1, mem_gb = 1.0, network = "heavy" } }- At a site, such a job is admitted only while the site runs fewer network-heavy jobs than its budget allows, and never
while a protected process at the site is in an upload phase (explain names
SITE_UPLINK_BUSYorSITE_UPLINK_BUDGET). - While such a phase lasts, the agent pauses the job on every node of the site: through
pausein the control document (cooperative_pause) or a freeze (freeze_ok); a runner that can do neither is released (preempt_site, requeued, never charged). A job paused longer than its node’s longest pause is released the same way. - The agent’s own transfers for a job (staging datasets, uploading results and checkpoints) are paced and paused by the
site without any declaration;
networkis for traffic the job makes itself. normal(the default) changes nothing. The spec envelope’sresourcescarries the value. Cores before 2.10 ignore it: the job then runs as any other.
Declarative UI: formats and templates
Section titled “Declarative UI: formats and templates”Module UI is data. Core templates render it and escape it, and no module HTML or JS is ever executed.
-
Field formats are one of these:
Format Renders as .Nffixed-point, N decimals .Neexponent notation, N decimals .N%percentage, N decimals dinteger ,dinteger with thousands separators sstring .Nsstring truncated to N characters N is one or two digits.
-
digest_lineis plain text with{field}or{field:FORMAT}placeholders (field names in lower snake case), and{{/}}for literal braces. Nothing else is allowed. -
iconcomes from a fixed set:cpu,gpu,dna,chart,flask,cube,bolt.
What a manifest cannot declare
Section titled “What a manifest cannot declare”- Host protection: exemptions, priority over the owner’s protected processes, or node selection by identity. Protection is owner-set only (see the public-surface statement).
- Secret values. A manifest declares secrets by name; only the owner sets their values.
Source: spec/manifest.md in the oarbank-sdk repository