# Upgrading

> Upgrade an Oarbank fleet to 2.11 on macOS, Linux and Windows from 2.10, 2.9 or 2.8: what changes, the one-time coordinator move and removed commands.

HTML version: https://docs.codonic.dev/oarbank/operate/upgrading

Upgrade the **coordinator first**, then the nodes. Running jobs keep running through the upgrade, and nodes reconnect by themselves. The downloads are on the [2.11.1 release page](https://github.com/roeehrl/oarbank/releases/tag/v2.11.1); check each file against its `SHA256SUMS-…` file there ([Get started](/oarbank/get-started#download)).

- **From 2.10:** read [From 2.10 to 2.11](#from-210-to-211). The coordinator updates the fleet itself.
- **From 2.9:** install 2.11.1 as [From 2.9 to 2.10](#from-29-to-210) says, then read the 2.11 section for what is new. Nothing moves and no setting changes meaning.
- **From 2.8:** install 2.11.1 directly. Its packages make the same one-time move of the coordinator as 2.9.0 did, so follow [From 2.8 to 2.9](#from-28-to-29) with the 2.11.1 files, then read the 2.10 and 2.11 sections for what is new.

## From 2.10 to 2.11

Oarbank 2.11 says when a node runs its container VM and sizes it as set, approves a machine with what it starts as, and deletes what a module left on a node once it leaves. Nothing moves, and the new settings’ defaults keep 2.10’s behaviour, except where the list below says otherwise.

### Before you start

Nodes that joined with 2.10.3 or 2.10.4 after their package was installed may have no update trust pinned: `oarbank update plan 2.11.1` says “no update trust is pinned” for them, and the fleet update does not install their package until it is pinned. Run the command the plan shows once on each, or install the 2.11.1 node package on it by hand:

| Platform                  | Command                                                                |
| ------------------------- | ---------------------------------------------------------------------- |
| macOS                     | `sudo /Library/Oarbank/bin/oarbank-launcher update-trust --from-agent` |
| Linux                     | `sudo oarbank-launcher update-trust --from-agent`                      |
| Windows (elevated prompt) | `oarbank-node update-trust --from-agent`                               |

From 2.11 a node that joins pins its update trust itself.

### Update

Update the fleet from the coordinator, as for any release: fetch (or import) 2.11.1, sign it in signing mode, and start the rollout ([Update the fleet](/oarbank/operate/update-and-remove#update-the-fleet)). The coordinator updates itself first, then the nodes’ packages. To update by hand instead, install the 2.11.1 coordinator package over the 2.10 one ([Update the coordinator by hand](/oarbank/operate/update-and-remove#update-the-coordinator-by-hand)), then each node’s.

### What behaves differently after the upgrade

- **Container VMs:** the new `containers` setting (node, group or fleet) decides when a node runs its container runtime. `always`, the default, keeps 2.10’s behaviour. `on_demand` is recommended where container work is occasional: the VM stays stopped until container work waits for the node. `off` suits small machines that also run your memory-heavy programs ([Containers on nodes](/oarbank/operate/containers)). Agents older than 2.11 ignore the setting, but the coordinator still keeps container work off a node set to `off`.
- **`vm_mem_gb` and `vm_cpus` now size the container VM.** 2.10 ignored them. A value set earlier now takes effect: the agent restarts an idle VM at that size, within half the machine’s memory and its processors, and the node’s `containers` pool follows the size.
- **Approving machines:** an approval can say what the new node starts as (paused, modules off, its own settings, labels and groups), and `new_node_state` and `new_node_modules` set it for every new node ([Adding machines](/oarbank/operate/adding-machines)). Their defaults (`active`, `all`) change nothing.
- **Module data is deleted when a module leaves a node.** A module turned off on a node loses what it left there after `module_retention_days` (7 by default); one no longer in the node’s release, or on a retired node, after at most a day. Turning it back on in time keeps everything. Raise `module_retention_days` before the upgrade if you turn modules off on nodes and want their data kept longer ([When a module leaves a node](/oarbank/operate/modules#when-a-module-leaves-a-node)). A node whose agent is older than 2.11 deletes nothing and refuses `module_retention_days` until it is updated.
- **The dataset cache is collected.** A node’s cache from before 2.11 has no record of which module used each dataset: a dataset no job uses within the retention (7 days by default) is deleted, and staged again when a job needs it. `disk_gb` now caps the cache.
- **A retired node’s agent stops.** It stops its jobs, services, workloads and listeners, deletes what its modules left after a day, and waits with `E_RETIRED` ([Remove a node](/oarbank/operate/update-and-remove#remove-a-node)).
- New reason codes: [Reason codes](/oarbank/reference/reason-codes#new-in-211).

### Module authors (SDK 1.8)

The module SDK for this release is 1.8.0. A module that holds state outside the node (a registration with an outside service, a lease, a remote resource) can declare a cleanup stage, `[[stages]] cleanup = true`, which the node runs once before it deletes the module’s data there ([When a module leaves a node](/oarbank/operate/modules#when-a-module-leaves-a-node)). A manifest that sets `stages[].cleanup` needs `requires.core >= 2.11` ([Versions and compatibility](/oarbank/operate/versions#which-sdk-features-need-which-core), [Cleanup stage](/oarbank/spec/manifest#cleanup-stage)).

## From 2.9 to 2.10

Oarbank 2.10 adds module [workloads](/oarbank/operate/workloads), [inbound listeners](/oarbank/operate/make-a-node-reachable) with router port mapping, [sites](/oarbank/operate/sites) for nodes that share an internet line, [delegated processes](/oarbank/operate/delegated-processes), and measured [co-tenancy with a GPU duty governor](/oarbank/operate/prove-co-tenancy). Every one of them is off or idle until you use it: no workload runs until you assign it, no listener opens until you assign it to a node, no site exists until you create one, delegation is off by default, and `gpu_jobs = "certified"` is opt-in per node.

### Upgrade the coordinator from 2.9

Run the 2.11.1 installer over the 2.9 one, as for any update ([Updating and removing](/oarbank/operate/update-and-remove#update-the-coordinator-by-hand)):

- **macOS:** double-click `oarbank-coordinator-2.11.1-macos-arm64.pkg`. It writes the system services again and restarts them on the new build.
- **Linux:** `sudo apt install ./oarbank-coordinator-2.11.1-linux-amd64.deb` (Fedora, RHEL: `sudo dnf install ./oarbank-coordinator-2.11.1-linux-amd64.rpm`).
- **Windows:** run `oarbank-coordinator-2.11.1-windows-x64.msi` (or `-arm64`).

Check with `oarbank coordinator status` and `oarbank verify`. The console gains a **Sites** page, a **Network** card and a **Workloads** card on each node page, a **Workloads** tab on each module that declares workloads, and a **Co-tenancy** section on each node’s **Protection** page.

### Upgrade the nodes from 2.9

Install the 2.11.1 node package on each node once, over the old one. This is the last time: the package brings the node’s update runner and pins the owner keys the node trusts for it, and from 2.10 the coordinator updates whole packages itself ([Update the fleet](/oarbank/operate/update-and-remove#update-the-fleet)).

To get a 2.10 or later agent onto nodes before you can reach them, upload, sign, try and promote it as before ([Update only a node’s agent](/oarbank/operate/update-and-remove#update-only-a-nodes-agent)). The rest of the package, and fleet updates, still wait for the package:

```sh
oarbank agent upload oarbank-agent-2.11.1-linux-amd64
oarbank agent sign <build>
oarbank agent canary <build> --node <node>
oarbank agent promote <build>
```

The new features are the agent’s work, so a node acts on them only once it runs the 2.10 agent:

| Feature                                                      | On a node still running a 2.9 agent                                                                                                                                           |
| ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Inbound listeners                                            | The node page’s **Network** card says the agent does not support them, and assigning one is refused until the node runs a 2.10 agent                                          |
| Workloads, and modules that need core 2.10                   | Not run there: their work waits with `AGENT_TOO_OLD`                                                                                                                          |
| Sites                                                        | The node takes no transfer leases and does not pause its network-heavy jobs with the site; the coordinator still applies the site’s hold and budget when it grants work there |
| `gpu_jobs = "certified"`, the duty governor, co-tenancy runs | Not available: the node keeps the 2.9 GPU rules                                                                                                                               |
| Delegated processes, event-driven sources, `immediate` rules | Ignored                                                                                                                                                                       |

Install the 2.10 node package as well when you want the uninstaller to release router mappings and stop workloads (`oarbank-launcher remove` does it from 2.10), and on Macs whose Application Firewall is on: the package’s postinstall lists the agent as allowed, so a listener’s first connection raises no prompt.

### What behaves differently after the upgrade

- **A GPU-using rule that protects `progress_rate` now holds GPU work back** the way a `gpu_share` rule does, except under `gpu_jobs = "always"`. Before, fleet GPU work could run beside such a program under the default `when_no_gpu_protected`. If GPU jobs that used to run beside it now wait, that is why: measure a duty and use `gpu_jobs = "certified"` ([Prove co-tenancy is safe](/oarbank/operate/prove-co-tenancy)).
- **A node’s `staging_mbps` cap is now enforced** on dataset staging, result and checkpoint uploads. A cap you set earlier and forgot now slows those transfers.
- **`~` in a protection source** now means the home of the person logged in at the machine, read by their session helper on a system install. Before, it meant the service account’s home, which held fleet work back for good.
- New settings, all with defaults that change nothing: `workloads` (on: assigned workloads may run), `delegated_processes` (off), `inbound_listeners` (on: assigned listeners may open), `listener_port_range`, `listener_probe_endpoint` and `listener_stun_server` ([Settings](/oarbank/operate/settings#the-keys)). A machine’s managed policy may turn `workloads`, `delegated_processes` and `inbound_listeners` off.
- New reason codes and alerts: [Reason codes](/oarbank/reference/reason-codes#new-in-210).

### Module authors (SDK 1.7)

SDK 1.7 came with 2.10. A manifest that declares inbound listeners (`[sandbox].net.inbound`, `services[].listeners`) or workloads (`[[workloads]]`, `workload.report`) needs `requires.core >= 2.10`; network-heavy stages and duty specs need no floor ([Versions and compatibility](/oarbank/operate/versions#which-sdk-features-need-which-core)). The how-tos: [Accept connections from the internet](/oarbank/build/inbound-listeners), [Run a long-running workload](/oarbank/build/workloads) and [Share a GPU with a protected program](/oarbank/build/gpu-duty).

## From 2.8 to 2.9

Note

This section describes the 2.9.0 files. From 2.8, use the 2.11.1 files with the same names: the 2.10 packages make the same move.

Oarbank 2.9 changes how the coordinator runs on macOS and Linux (a system service instead of its owner’s per-user services), replaces per-node policy and caps with one [settings](/oarbank/operate/settings) model, and replaces the tool registry with [host tools](/oarbank/operate/host-tools) that each node detects. Running jobs keep running through the upgrade, and nodes reconnect by themselves.

Upgrade the **coordinator first**, then the nodes. The downloads are on the [2.9.0 release page](https://github.com/roeehrl/oarbank/releases/tag/v2.9.0); check each file against its `SHA256SUMS-…` file there ([Get started](/oarbank/get-started#download)).

### Before you start

- Note how the coordinator runs now: `oarbank coordinator status`, and the address nodes reach it on. The upgrade keeps the same address, port and identity.
- Make sure the agent address exists before anyone logs in (a LAN address, or a VPN that connects at boot): from 2.9 the coordinator starts at boot ([The coordinator service](/oarbank/operate/coordinator#setting-up)).
- On macOS, be logged in at the Mac, as the person who set the coordinator up, when you install the package. That lets the package move the coordinator’s keys out of your login Keychain by itself.
- Your owner signing keys are not touched: they stay in your home.

### Upgrade the coordinator

- **macOS**

  1. Double-click `oarbank-coordinator-2.9.0-macos-arm64.pkg` and follow Installer (it asks for an administrator password, as any package does).

  2. If you were logged in, the package moves the coordinator by itself: it exports the coordinator’s keys from your Keychain into the coordinator’s own key files (as you, in your session), stops the old LaunchAgents, copies the coordinator’s home to `/Library/Application Support/Oarbank/coordinator` (an APFS clone, so it is instant and takes no extra space), starts the two launchd daemons as `_oarbankd`, checks that they serve the same fleet, and removes the LaunchAgents. You are added to the owners’ group, `_oarbankadmin`.

  3. If the package could not reach your Keychain (nobody logged in, an install over SSH or MDM), the coordinator keeps running as before, now on 2.9’s code and still using its Keychain keys. **Oarbank Coordinator** then shows **Move the coordinator to a system service…**, and the console and `oarbank coordinator status` say the move is waiting. Click it (macOS asks for an administrator once), or in Terminal, as yourself:

     ```sh
     oarbank coordinator migrate
     ```

- **Linux**

  1. Install the package with `sudo`, as the person who set the coordinator up:

     ```sh
     sudo apt install ./oarbank-coordinator-2.9.0-linux-amd64.deb      # Fedora, RHEL: sudo dnf install ./oarbank-coordinator-2.9.0-linux-amd64.rpm
     ```

  2. The package moves the coordinator at once (its keys were already files): it stops the systemd user units, copies the home to `/var/lib/oarbank/coordinator`, starts the system units as `oarbankd`, checks them and removes the user units. The installing person is added to the owners’ group, `oarbank-admin`.

  3. Log out and in again (or run `newgrp oarbank-admin`) so your `oarbank` reaches the coordinator without a token.

  Lingering is no longer needed. The package leaves it as you set it; if you enabled it only for Oarbank, `sudo loginctl disable-linger <user>` turns it off.

- **Windows**

  The Windows coordinator was already a system service, so there is nothing to move. Install `oarbank-coordinator-2.9.0-windows-x64.msi` (or `-arm64`); it restarts both services on the new payload and puts `oarbank` on the system PATH. Open a new elevated prompt to use it.

A coordinator installed from an archive upgrades with the new archive’s helper, as before ([Updating and removing](/oarbank/operate/update-and-remove#update-the-coordinator-by-hand)); on macOS and Linux it installs the system services too.

### Check the result

```sh
oarbank coordinator status             # form  system service — runs from boot, as _oarbankd (oarbankd on Linux)
oarbank coordinator migrate --status   # what is installed, and the migration record
oarbank fleet                          # every node back online
```

- On macOS, Oarbank Coordinator’s window says “Runs as a system service”.
- The console’s **Coordinator** page shows the same form; a coordinator still in the per-user form shows a warning and a banner across the console.
- Every step of the move is recorded in `/Library/Application Support/Oarbank/coordinator-migration.json` (macOS) or `/var/lib/oarbank/coordinator-migration.json` (Linux): its state is `done` when it finished, `needs-person` while it waits for you (macOS), `rolled-back` with the error if it failed. It holds no secrets.
- Open a new terminal and run `oarbank --help`: from 2.9 the command is on the PATH ([The coordinator service](/oarbank/operate/coordinator#the-oarbank-command-on-the-path)).

### If the move fails

Nothing of the old home is changed until the very last step. If any step after stopping the old services fails, the move removes the new system services, sets the new home aside as `coordinator.failed-<time>` for diagnosis, and starts the per-user coordinator again on its untouched home; the record says `rolled-back` with the error. Running the move again (`oarbank coordinator migrate`, or reinstalling the package) starts over; after `done` it does nothing.

If it rolled back, the coordinator keeps working in its 2.8 form on 2.9’s code. Open an issue on [roeehrl/oarbank](https://github.com/roeehrl/oarbank/issues) with the error from the record (it holds no secrets), your OS version and the output of `oarbank coordinator migrate --status`.

### After a successful move

- The old home stays, renamed `coordinator.migrated-<time>` and marked so that no coordinator starts on it: `~/Library/Application Support/Oarbank/coordinator.migrated-<time>` on macOS, `~/.local/share/oarbank/coordinator.migrated-<time>` (or under `$XDG_DATA_HOME`) on Linux. The old Keychain items stay in your Keychain on macOS. Delete both once you are content.
- 2.9 has no command that moves a coordinator back to the per-user form.
- Other owners need to be added to the owners’ group ([Who may use the CLI](/oarbank/operate/coordinator#who-may-use-the-cli)).
- Removing the coordinator now works differently ([Updating and removing](/oarbank/operate/update-and-remove#remove-the-coordinator)).

### Settings after the upgrade

When the 2.9 coordinator first opens its database, it converts the old per-node policy and caps into [settings](/oarbank/operate/settings), once:

- A node’s policy value becomes a node value only where it differs from what the node now inherits, so copies of the default disappear and real choices stay. Caps become node values.
- Fleet-wide keys (notifications, console host names, the replica rate, the folder registry, dataset origins) become fleet values; the ntfy token moves to the secrets store.
- Each module’s kill switch becomes its `enabled` setting set to off for the fleet; a module’s pipeline becomes its `pipeline` setting; disabled services become `services.disabled`.
- Protection rules every node had in common become the fleet’s; the rest stay per node.
- A module’s old settings become one value per key its settings schema declares; values the module does not declare, or that its schema refuses, are dropped and named in the coordinator’s `settings_migrated` event.

Then look at what you have: `oarbank settings get --node <node>` for a node, `oarbank settings shadowed` for values that change nothing.

#### Removed commands and operations

| 2.8                                                                                                                                                                                | 2.9                                                                                                                                                                        |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `oarbank node policy …`                                                                                                                                                            | `oarbank settings get\|set\|reset … --node <node>`                                                                                                                         |
| `oarbank node limits <node> --cpu-cores N --mem-gb N --jobs N …`                                                                                                                   | `oarbank settings set cpu_cores N --node <node>` (and `mem_gb`, `jobs`, `enforce`, …); `reset` removes one                                                                 |
| `oarbank pipeline single\|split --module <m>`                                                                                                                                      | `oarbank settings set pipeline split --module <m>`                                                                                                                         |
| `oarbank protection canary\|promote\|restore`                                                                                                                                      | a canary group and `oarbank settings promote protection.rules --group <group>` ([Host protection](/oarbank/operate/host-protection#roll-rules-out-through-a-canary-group)) |
| the tool registry (`settings.tools.update`) and its `trust` setting                                                                                                                | host tool definitions: `oarbank tools define\|delete`, and `tool.<id>.path` per node                                                                                       |
| operations `settings.update`, `nodes.set_policy`, `nodes.set_caps`, `settings.notifications.update`, `modules.set_pipeline`, `protection.rules.canary`, `protection.rules.restore` | `settings.apply` (with its dry run), `settings.promote`, `settings.secrets.set`                                                                                            |

Scripts that call the removed operations through `oarbank op` need updating; `oarbank ops` lists what exists.

### Modules that used tool registry ids

Before 2.9 a module could ask for an operator-defined registry id such as `java17`. On upgrade, each registry entry becomes a [host tool definition](/oarbank/operate/host-tools) of the same id with its paths as search paths (an id named like a JDK becomes a JDK tool). If your fleet had no entry for an id a module asks for, the module’s jobs wait with `TOOL_NOT_FOUND`, and its readiness checklist says it asks for an unknown tool id and needs a new module version.

The module’s author publishes a version that asks for the built-in tool with a version constraint, for example `{ id = "jdk", version = ">=17", trust = "code-exec" }`, with `requires.core >= 2.9` ([Ask for host tools](/oarbank/build/host-tools)). Install it, approve it once (its grants changed), enable it and sign the releases.

### Nodes

Nodes update their agent from the coordinator; nobody needs to log in to them. Upload the 2.9.0 agent binary for each platform your nodes run, then sign, try and promote it:

```sh
oarbank agent upload oarbank-agent-2.9.0-linux-amd64
oarbank agent sign <build>
oarbank agent canary <build> --node <node>
oarbank agent promote <build>
```

[Updating and removing](/oarbank/operate/update-and-remove#update-only-a-nodes-agent) has the details. Update every node: a 2.9 agent is what detects host tools (an older one reports none, so modules that ask for tools wait with `TOOL_NOT_FOUND`), reports which settings revision it applied, enforces managed settings, offers modules whose doctor is not healthy for the stages that need no certification, and brings up the container runtime on Macs.

The agent update replaces the agent only. Some 2.9 changes come with the node **package**:

- **Macs:** install `oarbank-agent-2.9.0-macos-arm64.pkg` (Intel: `-macos-x86_64.pkg`) too. It brings the new **Oarbank Node** menu bar app (one setting, `ShowStatusIcon`), its administrator helper (joining and leaving as the system service now asks for an administrator as “Oarbank Node”, not as a script), and the node runtime that lets modules load native Python packages. Installing it keeps the node joined:

  ```sh
  sudo installer -pkg oarbank-agent-2.9.0-macos-arm64.pkg -target /
  ```

  Nodes that run containers also need Colima, Docker and Rosetta installed ([Containers on nodes](/oarbank/operate/containers#macos)).

- **Windows:** install `oarbank-agent-2.9.0-windows-x64.msi` (or `-arm64`) for the new tray model and container support installed after the MSI. Deploy the 2.9 ADMX template (`oarbank-agent-2.9.0-windows-admx.zip`) for the new policies.

- **Linux:** the agent update is enough for the new agent; install the 2.9 package too for the new service definition (below).

**A package upgrade renders the node’s service again.** Installing the 2.9 package over a 2.8 node (pkg, deb or rpm) rewrites the agent’s launchd job or systemd unit with the new launcher’s settings (`oarbank-launcher service refresh`), keeping its program, account and environment, and restarts it. That is how a node gets 2.9’s 60-second stop time ([below](#nodes-stop-their-jobs-cleanly)). A Windows MSI upgrade recreates the service. Nothing needs to rejoin.

#### Nodes stop their jobs cleanly

From 2.9, when a node’s agent is stopped or restarted (an upgrade, a reboot, `launchctl`, `systemctl`, the Windows service manager), it asks every job to stop as a release does (checkpointing first), ends what is left after at most 15 seconds, and releases each attempt as `AGENT_STOPPED`: the job is requeued and nothing is charged to the job or the node. Then it exits. The service definitions give it 60 seconds for this. An agent that is killed leaves no job processes behind: a watchdog, or the next agent before it takes work, ends them.

Note

New managed policy keys in 2.9: `ShowStatusIcon` and a `Settings` dictionary that may only make settings stricter ([Adding machines](/oarbank/operate/adding-machines#many-machines-mdm-intune-ansible), [Settings](/oarbank/operate/settings#managed-on-this-machine)).

### Module authors

The module SDK of 2.9 was 1.6.0. A manifest that uses a 2.9 feature (a host tool `version` or an `arch` other than `any`, or a settings schema with `x-oarbank` `scope = "node"`, `required` or `campaign`) needs `requires.core >= 2.9`; see [Versions and compatibility](/oarbank/operate/versions#which-sdk-features-need-which-core), [Ask for host tools](/oarbank/build/host-tools) and [Declare module settings](/oarbank/build/module-settings).