Upgrading

Upgrade the coordinator first, then the nodes. Running jobs keep running through the upgrade, and nodes reconnect by themselves. The downloads are on the 2.11.1 release page; check each file against its SHA256SUMS-… file there (Get started).

From 2.10: read From 2.10 to 2.11. The coordinator updates the fleet itself.
From 2.9: install 2.11.1 as From 2.9 to 2.10 says, then read the 2.11 section for what is new. Nothing moves and no setting changes meaning.
From 2.8: install 2.11.1 directly. Its packages make the same one-time move of the coordinator as 2.9.0 did, so follow From 2.8 to 2.9 with the 2.11.1 files, then read the 2.10 and 2.11 sections for what is new.

From 2.10 to 2.11

Oarbank 2.11 says when a node runs its container VM and sizes it as set, approves a machine with what it starts as, and deletes what a module left on a node once it leaves. Nothing moves, and the new settings’ defaults keep 2.10’s behaviour, except where the list below says otherwise.

Before you start

Nodes that joined with 2.10.3 or 2.10.4 after their package was installed may have no update trust pinned: oarbank update plan 2.11.1 says “no update trust is pinned” for them, and the fleet update does not install their package until it is pinned. Run the command the plan shows once on each, or install the 2.11.1 node package on it by hand:

Platform	Command
macOS	`sudo /Library/Oarbank/bin/oarbank-launcher update-trust --from-agent`
Linux	`sudo oarbank-launcher update-trust --from-agent`
Windows (elevated prompt)	`oarbank-node update-trust --from-agent`

From 2.11 a node that joins pins its update trust itself.

Update

Update the fleet from the coordinator, as for any release: fetch (or import) 2.11.1, sign it in signing mode, and start the rollout (Update the fleet). The coordinator updates itself first, then the nodes’ packages. To update by hand instead, install the 2.11.1 coordinator package over the 2.10 one (Update the coordinator by hand), then each node’s.

What behaves differently after the upgrade

Container VMs: the new containers setting (node, group or fleet) decides when a node runs its container runtime. always, the default, keeps 2.10’s behaviour. on_demand is recommended where container work is occasional: the VM stays stopped until container work waits for the node. off suits small machines that also run your memory-heavy programs (Containers on nodes). Agents older than 2.11 ignore the setting, but the coordinator still keeps container work off a node set to off.
vm_mem_gb and vm_cpus now size the container VM. 2.10 ignored them. A value set earlier now takes effect: the agent restarts an idle VM at that size, within half the machine’s memory and its processors, and the node’s containers pool follows the size.
Approving machines: an approval can say what the new node starts as (paused, modules off, its own settings, labels and groups), and new_node_state and new_node_modules set it for every new node (Adding machines). Their defaults (active, all) change nothing.
Module data is deleted when a module leaves a node. A module turned off on a node loses what it left there after module_retention_days (7 by default); one no longer in the node’s release, or on a retired node, after at most a day. Turning it back on in time keeps everything. Raise module_retention_days before the upgrade if you turn modules off on nodes and want their data kept longer (When a module leaves a node). A node whose agent is older than 2.11 deletes nothing and refuses module_retention_days until it is updated.
The dataset cache is collected. A node’s cache from before 2.11 has no record of which module used each dataset: a dataset no job uses within the retention (7 days by default) is deleted, and staged again when a job needs it. disk_gb now caps the cache.
A retired node’s agent stops. It stops its jobs, services, workloads and listeners, deletes what its modules left after a day, and waits with E_RETIRED (Remove a node).
New reason codes: Reason codes.

Module authors (SDK 1.8)

The module SDK for this release is 1.8.0. A module that holds state outside the node (a registration with an outside service, a lease, a remote resource) can declare a cleanup stage, [[stages]] cleanup = true, which the node runs once before it deletes the module’s data there (When a module leaves a node). A manifest that sets stages[].cleanup needs requires.core >= 2.11 (Versions and compatibility, Cleanup stage).

From 2.9 to 2.10

Oarbank 2.10 adds module workloads, inbound listeners with router port mapping, sites for nodes that share an internet line, delegated processes, and measured co-tenancy with a GPU duty governor. Every one of them is off or idle until you use it: no workload runs until you assign it, no listener opens until you assign it to a node, no site exists until you create one, delegation is off by default, and gpu_jobs = "certified" is opt-in per node.

Upgrade the coordinator from 2.9

Run the 2.11.1 installer over the 2.9 one, as for any update (Updating and removing):

macOS: double-click oarbank-coordinator-2.11.1-macos-arm64.pkg. It writes the system services again and restarts them on the new build.
Linux: sudo apt install ./oarbank-coordinator-2.11.1-linux-amd64.deb (Fedora, RHEL: sudo dnf install ./oarbank-coordinator-2.11.1-linux-amd64.rpm).
Windows: run oarbank-coordinator-2.11.1-windows-x64.msi (or -arm64).

Check with oarbank coordinator status and oarbank verify. The console gains a Sites page, a Network card and a Workloads card on each node page, a Workloads tab on each module that declares workloads, and a Co-tenancy section on each node’s Protection page.

Upgrade the nodes from 2.9

Install the 2.11.1 node package on each node once, over the old one. This is the last time: the package brings the node’s update runner and pins the owner keys the node trusts for it, and from 2.10 the coordinator updates whole packages itself (Update the fleet).

To get a 2.10 or later agent onto nodes before you can reach them, upload, sign, try and promote it as before (Update only a node’s agent). The rest of the package, and fleet updates, still wait for the package:

oarbank agent upload oarbank-agent-2.11.1-linux-amd64
oarbank agent sign <build>
oarbank agent canary <build> --node <node>
oarbank agent promote <build>

The new features are the agent’s work, so a node acts on them only once it runs the 2.10 agent:

Feature	On a node still running a 2.9 agent
Inbound listeners	The node page’s Network card says the agent does not support them, and assigning one is refused until the node runs a 2.10 agent
Workloads, and modules that need core 2.10	Not run there: their work waits with `AGENT_TOO_OLD`
Sites	The node takes no transfer leases and does not pause its network-heavy jobs with the site; the coordinator still applies the site’s hold and budget when it grants work there
`gpu_jobs = "certified"`, the duty governor, co-tenancy runs	Not available: the node keeps the 2.9 GPU rules
Delegated processes, event-driven sources, `immediate` rules	Ignored

Install the 2.10 node package as well when you want the uninstaller to release router mappings and stop workloads (oarbank-launcher remove does it from 2.10), and on Macs whose Application Firewall is on: the package’s postinstall lists the agent as allowed, so a listener’s first connection raises no prompt.

What behaves differently after the upgrade

A GPU-using rule that protects progress_rate now holds GPU work back the way a gpu_share rule does, except under gpu_jobs = "always". Before, fleet GPU work could run beside such a program under the default when_no_gpu_protected. If GPU jobs that used to run beside it now wait, that is why: measure a duty and use gpu_jobs = "certified" (Prove co-tenancy is safe).
A node’s staging_mbps cap is now enforced on dataset staging, result and checkpoint uploads. A cap you set earlier and forgot now slows those transfers.
~ in a protection source now means the home of the person logged in at the machine, read by their session helper on a system install. Before, it meant the service account’s home, which held fleet work back for good.
New settings, all with defaults that change nothing: workloads (on: assigned workloads may run), delegated_processes (off), inbound_listeners (on: assigned listeners may open), listener_port_range, listener_probe_endpoint and listener_stun_server (Settings). A machine’s managed policy may turn workloads, delegated_processes and inbound_listeners off.
New reason codes and alerts: Reason codes.

Module authors (SDK 1.7)

SDK 1.7 came with 2.10. A manifest that declares inbound listeners ([sandbox].net.inbound, services[].listeners) or workloads ([[workloads]], workload.report) needs requires.core >= 2.10; network-heavy stages and duty specs need no floor (Versions and compatibility). The how-tos: Accept connections from the internet, Run a long-running workload and Share a GPU with a protected program.

From 2.8 to 2.9

Oarbank 2.9 changes how the coordinator runs on macOS and Linux (a system service instead of its owner’s per-user services), replaces per-node policy and caps with one settings model, and replaces the tool registry with host tools that each node detects. Running jobs keep running through the upgrade, and nodes reconnect by themselves.

Upgrade the coordinator first, then the nodes. The downloads are on the 2.9.0 release page; check each file against its SHA256SUMS-… file there (Get started).

Before you start

Note how the coordinator runs now: oarbank coordinator status, and the address nodes reach it on. The upgrade keeps the same address, port and identity.
Make sure the agent address exists before anyone logs in (a LAN address, or a VPN that connects at boot): from 2.9 the coordinator starts at boot (The coordinator service).
On macOS, be logged in at the Mac, as the person who set the coordinator up, when you install the package. That lets the package move the coordinator’s keys out of your login Keychain by itself.
Your owner signing keys are not touched: they stay in your home.

Upgrade the coordinator

Double-click oarbank-coordinator-2.9.0-macos-arm64.pkg and follow Installer (it asks for an administrator password, as any package does).
If you were logged in, the package moves the coordinator by itself: it exports the coordinator’s keys from your Keychain into the coordinator’s own key files (as you, in your session), stops the old LaunchAgents, copies the coordinator’s home to /Library/Application Support/Oarbank/coordinator (an APFS clone, so it is instant and takes no extra space), starts the two launchd daemons as _oarbankd, checks that they serve the same fleet, and removes the LaunchAgents. You are added to the owners’ group, _oarbankadmin.
If the package could not reach your Keychain (nobody logged in, an install over SSH or MDM), the coordinator keeps running as before, now on 2.9’s code and still using its Keychain keys. Oarbank Coordinator then shows Move the coordinator to a system service…, and the console and oarbank coordinator status say the move is waiting. Click it (macOS asks for an administrator once), or in Terminal, as yourself:
Terminal window
```
oarbank coordinator migrate
```

Install the package with sudo, as the person who set the coordinator up:

sudo apt install ./oarbank-coordinator-2.9.0-linux-amd64.deb      # Fedora, RHEL: sudo dnf install ./oarbank-coordinator-2.9.0-linux-amd64.rpm

The package moves the coordinator at once (its keys were already files): it stops the systemd user units, copies the home to /var/lib/oarbank/coordinator, starts the system units as oarbankd, checks them and removes the user units. The installing person is added to the owners’ group, oarbank-admin.
Log out and in again (or run newgrp oarbank-admin) so your oarbank reaches the coordinator without a token.

Lingering is no longer needed. The package leaves it as you set it; if you enabled it only for Oarbank, sudo loginctl disable-linger <user> turns it off.

The Windows coordinator was already a system service, so there is nothing to move. Install oarbank-coordinator-2.9.0-windows-x64.msi (or -arm64); it restarts both services on the new payload and puts oarbank on the system PATH. Open a new elevated prompt to use it.

A coordinator installed from an archive upgrades with the new archive’s helper, as before (Updating and removing); on macOS and Linux it installs the system services too.

Check the result

oarbank coordinator status             # form  system service — runs from boot, as _oarbankd (oarbankd on Linux)
oarbank coordinator migrate --status   # what is installed, and the migration record
oarbank fleet                          # every node back online

On macOS, Oarbank Coordinator’s window says “Runs as a system service”.
The console’s Coordinator page shows the same form; a coordinator still in the per-user form shows a warning and a banner across the console.
Every step of the move is recorded in /Library/Application Support/Oarbank/coordinator-migration.json (macOS) or /var/lib/oarbank/coordinator-migration.json (Linux): its state is done when it finished, needs-person while it waits for you (macOS), rolled-back with the error if it failed. It holds no secrets.
Open a new terminal and run oarbank --help: from 2.9 the command is on the PATH (The coordinator service).

If the move fails

Nothing of the old home is changed until the very last step. If any step after stopping the old services fails, the move removes the new system services, sets the new home aside as coordinator.failed-<time> for diagnosis, and starts the per-user coordinator again on its untouched home; the record says rolled-back with the error. Running the move again (oarbank coordinator migrate, or reinstalling the package) starts over; after done it does nothing.

If it rolled back, the coordinator keeps working in its 2.8 form on 2.9’s code. Open an issue on roeehrl/oarbank with the error from the record (it holds no secrets), your OS version and the output of oarbank coordinator migrate --status.

After a successful move

The old home stays, renamed coordinator.migrated-<time> and marked so that no coordinator starts on it: ~/Library/Application Support/Oarbank/coordinator.migrated-<time> on macOS, ~/.local/share/oarbank/coordinator.migrated-<time> (or under $XDG_DATA_HOME) on Linux. The old Keychain items stay in your Keychain on macOS. Delete both once you are content.
2.9 has no command that moves a coordinator back to the per-user form.
Other owners need to be added to the owners’ group (Who may use the CLI).
Removing the coordinator now works differently (Updating and removing).

Settings after the upgrade

When the 2.9 coordinator first opens its database, it converts the old per-node policy and caps into settings, once:

A node’s policy value becomes a node value only where it differs from what the node now inherits, so copies of the default disappear and real choices stay. Caps become node values.
Fleet-wide keys (notifications, console host names, the replica rate, the folder registry, dataset origins) become fleet values; the ntfy token moves to the secrets store.
Each module’s kill switch becomes its enabled setting set to off for the fleet; a module’s pipeline becomes its pipeline setting; disabled services become services.disabled.
Protection rules every node had in common become the fleet’s; the rest stay per node.
A module’s old settings become one value per key its settings schema declares; values the module does not declare, or that its schema refuses, are dropped and named in the coordinator’s settings_migrated event.

Then look at what you have: oarbank settings get --node <node> for a node, oarbank settings shadowed for values that change nothing.

Removed commands and operations

2.8	2.9
`oarbank node policy …`	`oarbank settings get\|set\|reset … --node <node>`
`oarbank node limits <node> --cpu-cores N --mem-gb N --jobs N …`	`oarbank settings set cpu_cores N --node <node>` (and `mem_gb`, `jobs`, `enforce`, …); `reset` removes one
`oarbank pipeline single\|split --module <m>`	`oarbank settings set pipeline split --module <m>`
`oarbank protection canary\|promote\|restore`	a canary group and `oarbank settings promote protection.rules --group <group>` (Host protection)
the tool registry (`settings.tools.update`) and its `trust` setting	host tool definitions: `oarbank tools define\|delete`, and `tool.<id>.path` per node
operations `settings.update`, `nodes.set_policy`, `nodes.set_caps`, `settings.notifications.update`, `modules.set_pipeline`, `protection.rules.canary`, `protection.rules.restore`	`settings.apply` (with its dry run), `settings.promote`, `settings.secrets.set`

Scripts that call the removed operations through oarbank op need updating; oarbank ops lists what exists.

Modules that used tool registry ids

Before 2.9 a module could ask for an operator-defined registry id such as java17. On upgrade, each registry entry becomes a host tool definition of the same id with its paths as search paths (an id named like a JDK becomes a JDK tool). If your fleet had no entry for an id a module asks for, the module’s jobs wait with TOOL_NOT_FOUND, and its readiness checklist says it asks for an unknown tool id and needs a new module version.

The module’s author publishes a version that asks for the built-in tool with a version constraint, for example { id = "jdk", version = ">=17", trust = "code-exec" }, with requires.core >= 2.9 (Ask for host tools). Install it, approve it once (its grants changed), enable it and sign the releases.

Nodes

Nodes update their agent from the coordinator; nobody needs to log in to them. Upload the 2.9.0 agent binary for each platform your nodes run, then sign, try and promote it:

oarbank agent upload oarbank-agent-2.9.0-linux-amd64
oarbank agent sign <build>
oarbank agent canary <build> --node <node>
oarbank agent promote <build>

Updating and removing has the details. Update every node: a 2.9 agent is what detects host tools (an older one reports none, so modules that ask for tools wait with TOOL_NOT_FOUND), reports which settings revision it applied, enforces managed settings, offers modules whose doctor is not healthy for the stages that need no certification, and brings up the container runtime on Macs.

The agent update replaces the agent only. Some 2.9 changes come with the node package:

Macs: install oarbank-agent-2.9.0-macos-arm64.pkg (Intel: -macos-x86_64.pkg) too. It brings the new Oarbank Node menu bar app (one setting, ShowStatusIcon), its administrator helper (joining and leaving as the system service now asks for an administrator as “Oarbank Node”, not as a script), and the node runtime that lets modules load native Python packages. Installing it keeps the node joined:
Terminal window
```
sudo installer -pkg oarbank-agent-2.9.0-macos-arm64.pkg -target /
```
Nodes that run containers also need Colima, Docker and Rosetta installed (Containers on nodes).
Windows: install oarbank-agent-2.9.0-windows-x64.msi (or -arm64) for the new tray model and container support installed after the MSI. Deploy the 2.9 ADMX template (oarbank-agent-2.9.0-windows-admx.zip) for the new policies.
Linux: the agent update is enough for the new agent; install the 2.9 package too for the new service definition (below).

A package upgrade renders the node’s service again. Installing the 2.9 package over a 2.8 node (pkg, deb or rpm) rewrites the agent’s launchd job or systemd unit with the new launcher’s settings (oarbank-launcher service refresh), keeping its program, account and environment, and restarts it. That is how a node gets 2.9’s 60-second stop time (below). A Windows MSI upgrade recreates the service. Nothing needs to rejoin.

Nodes stop their jobs cleanly

From 2.9, when a node’s agent is stopped or restarted (an upgrade, a reboot, launchctl, systemctl, the Windows service manager), it asks every job to stop as a release does (checkpointing first), ends what is left after at most 15 seconds, and releases each attempt as AGENT_STOPPED: the job is requeued and nothing is charged to the job or the node. Then it exits. The service definitions give it 60 seconds for this. An agent that is killed leaves no job processes behind: a watchdog, or the next agent before it takes work, ends them.

Module authors

The module SDK of 2.9 was 1.6.0. A manifest that uses a 2.9 feature (a host tool version or an arch other than any, or a settings schema with x-oarbank scope = "node", required or campaign) needs requires.core >= 2.9; see Versions and compatibility, Ask for host tools and Declare module settings.