# Containers on nodes > Run container jobs on Oarbank nodes: what macOS (Colima), Linux (Podman or Docker) and Windows (WSL) need, what the node sets up and how to check it. HTML version: https://docs.codonic.dev/oarbank/operate/containers A module never runs containers itself: the node’s agent runs them for it, through its broker, with images you approved by digest. Each node uses a container runtime of its own and offers container work (the `containers` pool) **only while that runtime is ready**. This page covers Oarbank 2.9.0. ## See where a node stands ```sh oarbank node show build-07 # containers: colima ready · or missing, with each piece and its fix · or failed, with why ``` The node page shows the same, and the module’s readiness checklist names the nodes a stage cannot run on because they have no container runtime. A node that lacks something is simply not offered container jobs; nothing waits on it. ## Choose when a node runs its container VM From Oarbank 2.11 the `containers` setting decides, per node, group or fleet (Settings → **Containers**): | Value | The runtime | Use it for | | ------------------------- | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | | `always` (default) | starts when a release needs containers and stays up, as in 2.9 and 2.10 | nodes that run container work all day | | `on_demand` (recommended) | stays stopped until container work waits for this node; stops again after `containers_idle_stop_s` (600 s) without any | most nodes: a stopped VM holds no memory | | `off` | never starts; the node reports `disabled` and takes no container work | small machines that also run your memory-heavy programs | ```sh oarbank settings set containers on_demand # the whole fleet oarbank settings set containers off --node mini-03 # never on this Mac oarbank settings set containers_idle_stop_s 300 --node mini-01 ``` With `on_demand`, a stopped node still counts as able to run container work. When such work waits and the nodes whose runtime is already up have no room for it, the coordinator tells **one** stopped node to start (not every one), and the work runs there once its VM is ready (about a minute; the node shows how long its last start took). A job waiting meanwhile explains “The node’s container runtime is stopped”. If a module’s container stage is short (a 50 s scoring step, say), one or two container nodes are enough: set the rest to `on_demand` or `off`. With `off`, the node’s container jobs explain `CONTAINERS_DISABLED` with the fix, and the node is certified for the module’s stages that need no containers only (as any node without a container runtime). Turning `off` on or off again certifies the node’s modules again. On Linux, whose container engine is the host’s and not a VM, `on_demand` behaves like `always`. ### The VM’s size `vm_mem_gb` and `vm_cpus` (Settings → Caps, advanced) size the VM, within half the machine’s memory and its processors; without them it is sized from the machine’s memory as below. When you change them, the agent restarts the VM at the new size as soon as no container job runs on it. The node’s container line shows both sizes until then, for example `containers = always · running · VM 4 GB, 6 CPUs (running at 8 GB, 6 CPUs: restarts at the new size when idle)`. The VM’s memory decides how many container jobs the node runs at once (one per 2.5 GB above 1.5 GB), so `vm_mem_gb 2` keeps container work off a node whose runtime stays up. Note Oarbank 2.10.4 and earlier ignored `vm_mem_gb` and `vm_cpus` for the container VM: every Mac got the size below whatever was set. 2.11 applies them. ## macOS From 2.9 a Mac node reports its container runtime and brings it up itself. You install the tools once; the agent does the rest. ### What you install On each Mac, as the administrator who owns Homebrew: ```sh brew install colima docker # Colima brings Lima; Docker Desktop is not needed softwareupdate --install-rosetta --agree-to-license # Apple silicon: linux/amd64 images run under Rosetta brew tap slp/krun && brew trust slp/krun && brew install krunkit # optional, Apple silicon: GPU containers (Vulkan) ``` Nothing else: no `colima start`, no Docker Desktop, no setting. The agent looks for the tools in `/opt/homebrew/bin`, `/usr/local/bin` and its PATH. A tool installed only in a person’s home is not one the node’s `_oarbank` account may run, and is reported as such. Homebrew asks you to trust the third-party tap for krunkit. ### What the node does itself 1. **No module wants containers yet:** no virtual machine is created. The node still reports what is missing (`absent`), so you can prepare it before deploying a container module. 2. **A release arrives with a module approved for containers:** the agent checks the prerequisites. If something is missing it reports `missing`, each piece with its fix, and starts nothing. Otherwise it creates and starts its own Colima profile, `oarbank`, in the background (`starting`): a Virtualization.framework VM with Rosetta for amd64 images, sized by `vm_mem_gb` and `vm_cpus` when you set them, otherwise from the Mac’s memory (8 GB on Macs with up to 32 GB, 12 GB up to 96 GB, 32 GB above), never more than half the memory, mounting only the agent’s work and module-data directories. With `containers` = `on_demand` this happens only when container work waits for the node. The first start downloads the VM image and takes a few minutes. 3. **Ready:** the node reports `ready` with the platforms it runs (`linux/arm64`, `linux/amd64`) and starts offering the `containers` pool. If the start fails, it reports `failed` with the end of Colima’s log, and tries again with the next release or agent restart. 4. **GPU containers** use a second profile, `oarbank-gpu`, on krunkit. The node offers them (`virtio-gpu:venus`, Vulkan in containers) only once the CPU runtime is ready and krunkit is installed on Apple silicon; that VM starts with the first GPU job, and a failed start withdraws the GPU until the next release or restart. The VMs live in the agent’s own home: `/Library/Application Support/Oarbank/agent/colima` for a system install, `~/.colima` for a personal one. **Never start or delete the `oarbank` or `oarbank-gpu` profiles yourself**; your own Colima profiles are untouched. Intel Macs get an x86_64 VM without Rosetta. ### Check a Mac on the spot Run the doctor as the agent’s account (its home is private to it): ```sh sudo -u _oarbank /Library/Oarbank/bin/oarbank-agent --home "/Library/Application Support/Oarbank/agent" containers doctor sudo -u _oarbank /Library/Oarbank/bin/oarbank-agent --home "/Library/Application Support/Oarbank/agent" containers doctor --probe --gpu ``` It prints the agent’s last report with the prerequisites as they are now, and exits 3 while anything is missing. `--probe` brings the runtime up and runs real containers through it; `--gpu` also tries the GPU profile. | Missing | Means | Fix | | ------------------------------------- | ------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | `colima`, `docker`, `lima` | not in `/opt/homebrew/bin`, `/usr/local/bin` or the PATH | `brew install colima docker` | | `account_access` | the tool exists, but the agent’s account may not run it (a prefix in a person’s home) | install with Homebrew in `/opt/homebrew` or `/usr/local`, or make the custom prefix readable (`chmod o+rx`) | | `rosetta` | Apple silicon without Rosetta 2 | `softwareupdate --install-rosetta --agree-to-license` | | `colima_home` | the agent’s account may not write its `colima` directory | give the account its home back (`chown -R`), or reinstall | | `socket_path` | a socket path would be longer than macOS allows (103 bytes) | use a shorter agent home | | `account` | the doctor was run by another account than the agent’s | run it with `sudo -u _oarbank` as above | | `krunkit`, `apple_silicon` (GPU only) | no GPU in containers | `brew tap slp/krun && brew trust slp/krun && brew install krunkit` | Known limitations in 2.9.0 Bringing the VM up from the node’s system service was built for 2.9 and tested in code, but has not yet been verified end to end on every kind of Mac. Still to be confirmed on real system installs: a Virtualization.framework VM created by the hidden `_oarbank` account from a launchd daemon, krunkit’s GPU access from a daemon, amd64 images under Rosetta in that VM, and the VM surviving an agent restart. If a Mac reports `failed` with a virtualization or Metal error, please report it on [roeehrl/oarbank](https://github.com/roeehrl/oarbank/issues) with the doctor’s output. ## Linux Containers use rootless Podman when it is installed, otherwise Docker Engine. With Podman, the `oarbank` account needs subordinate user and group ids: ```sh sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 oarbank ``` The node reports its runtime (`podman` or `docker`) as `installed`, or `absent`. GPU containers need the `oarbank` account in the groups that own the GPU device files ([Place work by GPU API](/oarbank/build/gpu-placement)). ## Windows Containers run in a WSL containers session the agent creates and owns (a VM of its own). They need Windows 10 version 2004 or later, WSL 2.9.3 or later and the Virtual Machine Platform, on a machine with hardware virtualization (nested virtualization in a VM). From 2.9 the MSI never installs them inside its own installation. `CONTAINERS=1` on the `msiexec` command line, the join page’s **Run container jobs** box, or a join code made for container jobs (`oarbank join-code --containers`) registers a one-shot task, `OarbankContainerSupport`, which runs as SYSTEM right after the installer has finished, installs the pieces, records the outcome and deletes itself. The MSI exits 0 either way. Oarbank Node and `oarbank-node status` then show “Restart Windows to finish container support” or “Container support failed: …” until it is done (the record is under `HKLM\SOFTWARE\Codonic\Oarbank\ContainerSupport`). - **Install later** ```powershell oarbank-agent containers install # as an administrator ``` - **Check** ```powershell oarbank-agent containers doctor # the runtime's state and each missing piece with its fix oarbank-agent containers doctor --probe # also runs a container through it ``` A Windows node runs container images of its own architecture. ## Container GPU APIs only beside a runtime From 2.9 a node lists GPU APIs in containers only when it has a container runtime: `oarbank node show` prints the runtime (or “containers: none”) next to them, and a node without a runtime never offers container GPU work, whatever an older doctor report said.