# Prove co-tenancy is safe > Prove co-tenancy is safe: measure what fleet work costs a protected program with the paired on/off harness, earn a certificate, and run GPU work beside it. HTML version: https://docs.codonic.dev/oarbank/operate/prove-co-tenancy Host protection keeps fleet work from harming the programs you protect. This page shows how to **measure** that it does: the paired on/off harness runs the fleet work under test beside one protected program in ON windows and holds it in OFF windows, in random order, and decides at your margin whether the program ran as fast with fleet work as without. A passing run is a **co-tenancy certificate**. ## Before a run: give the rule a progress source The harness measures the protected program’s own progress, so its rule needs `protect.metric = "progress_rate"` with a source: an event file the program writes, counted by event type. ```toml [[rule]] id = "trainer" match = { path_prefix = "/Applications/Example Trainer.app/" } tree = "descendants" immediate = true protect = { metric = "progress_rate", max_slowdown = 0.05, source = { jsonl = "~/Library/Application Support/Example/events-*.jsonl", event = "step_done" } } [[rule.during]] source = { jsonl = "~/Library/Application Support/Example/events-*.jsonl", event = "status", field = "state.phase", on = ["uploading"] } pause_fleet = { scope = "gpu" } tags = ["uplink"] ``` - `jsonl` may end in a glob: the newest matching file is followed, so a program that starts a new file on every restart is picked up at once. - `event` counts the lines whose `event` field (or `event_key`, a dotted path) equals it. - A `during` block’s phase comes from `enter`/`exit` events or from a `field` of one event type; `immediate = true` pauses fleet work the moment the phase begins, between protection’s two-second ticks. - `~` is your home. On a system install the node’s service account cannot read it: your **session helper** (the small helper that runs in your login) reads the file and passes on counts and phase states, never the file’s contents. Caution While nobody is logged in, no session helper reads a source in a home folder. Protection then treats the phase as on and the progress as unknown, and holds fleet work back under that rule; the node page says **source … needs the session helper**. Put a source outside your home, readable by the service account, if the program runs while you are logged out. Check the source reads before a run: the node’s **Protection** page, **Co-tenancy** section, shows a rate and who reads it (the agent or your session helper). ### Sources are watched, and `immediate` rules act at once Protection reads sources as they change, not only on its two-second tick: the agent watches the file (kqueue on macOS, inotify on Linux, a 250 ms check on Windows), follows a rename or a new file, and starts again from the top after a truncation. A source can be: | Kind | Keys | What it gives | | --------------------------------- | --------------------------------------------------- | ----------------------------------------------------------------------- | | Progress from a JSONL file | `jsonl`, `event`, `event_key` | a rate: matching lines per second | | Progress from a log | `path`, `log_regex` | a rate: regex matches per second | | Progress from a program you write | `exec` | a rate the program prints (run by the agent, never by a session helper) | | A phase from events | `jsonl`, `enter`, `exit`, `enter_soon`, `event_key` | on after an `enter` event, off after `exit` | | A phase from a field | `jsonl`, `event`, `field`, `on`, `soon` | on while the field holds one of the `on` values | | A phase while a program runs | `on_exec = { match, tree }` | on while a matching process of yours runs | A rule with `immediate = true` pauses the fleet work its `during` block names the moment the phase begins, between ticks, and releases it on the phase’s own end. The **Co-tenancy** section shows each phase and progress source with its state and who reads it, and the event path’s latency from the source’s write to the pause (p50 and p99). The node’s timeline records `PROTECTION_PHASE_ON` and `PROTECTION_PHASE_OFF` with how the change was seen. Protection never guesses in your favour: a progress source that cannot be read counts as harm, and a phase that cannot be read counts as on, so fleet work under the rule is held back. The node says so with its fix: `PROTECTION_SOURCE_UNREADABLE` (the agent cannot read the file) or `PROTECTION_SOURCE_NEEDS_HELPER` (the file is in a home folder and no session helper reports it). ## Run the harness ```sh oarbank protection ab start mac-01 --rule trainer --work cpu ``` | Option | Default | Meaning | | ----------------------- | --------- | ---------------------------------------------------------------------------------------------- | | `--work` | `gpu` | the fleet work under test: `gpu`, `cpu`, `all`, a module, or `/` | | `--duty` | none | the `gpu_duty` cooperative GPU work holds in ON windows | | `--blocks` | 12 | planned blocks of one ON and one OFF window; raised after the pilot when the margin needs more | | `--window`, `--washout` | 300, 60 s | each window, and the seconds discarded after each switch | | `--margin` | 0.02 | the slowdown that counts | | `--off` | `paused` | how OFF holds the work: `paused`, or `evicted` to free its memory too | | `--max-hours` | 8 | the run’s cap: a run that cannot reach the margin within it is refused after the pilot | It is a reviewed change (T2): you see the rule, its source, the work and the planned hours before anything starts. Protection stays active throughout, and a collapse of the program’s progress in an ON window aborts the run at once as a slowdown. Watch it with `oarbank protection ab status mac-01`, stop it with `oarbank protection ab abort `. Note Run the harness in a window you choose, for example overnight while the program works. A block is left out and repeated, never counted, when something outside the fleet changed: a phase of the rule, a restart of the program, sleep, the power source, the memory guard. Heat is not left out: if fleet work makes the machine throttle, that slowdown counts against it. ## Read the report `oarbank protection ab show `, or the run’s page in the console: - **No slowdown shown**: the program ran at, say, 99.4 % of its speed without fleet work (90 % interval 98.6–100.3 %), within the 2 % margin. The run becomes a certificate. - **Slowdown**: the whole interval is below 100 % and the estimate below the margin. - **Inconclusive**: the interval reaches below the margin; the report says about how many more blocks would decide it. The page plots each block’s OFF and ON rate side by side, shows `R` (ON ÷ OFF) with its interval against the margin, and lists the blocks left out and why. ## Certificates and certified GPU co-tenancy A certificate names the node, the rule, the fleet work and duty, `R` with its interval, the margin and the run. It is valid for 14 days, and only while the node’s hardware, OS major version and agent and the protected program are the ones it was earned under: `oarbank protection certificates mac-01` shows each as valid, expired or stale with why. To find how much GPU work a protected GPU program tolerates, sweep the duty: ```sh oarbank protection ab sweep mac-01 --rule trainer ``` It runs the harness at `gpu_duty` 0.1, 0.2, 0.35, 0.5, 0.65, 0.8 and 1.0 (or your `--levels`), one run each, and stops at the first level that does not pass. The highest passing level is the node’s certified duty for that rule. Then set the node’s `gpu_jobs` to `certified` in its protection section: ```toml [node] gpu_jobs = "certified" ``` ```sh oarbank protection preview mac-01 node-protection.toml oarbank protection set mac-01 node-protection.toml ``` `set` replaces the node’s own section, so keep its rules in the same file. `gpu_jobs` takes `never`, `when_no_gpu_protected` (the default: fleet GPU work runs only while no protected program uses the GPU), `certified` and `always`; across scopes the strictest wins. Without a valid certificate for every rule that holds GPU work back on the node, `certified` behaves as `when_no_gpu_protected`: GPU work waits, and the node shows `GPU_NOT_CERTIFIED` with the sweep as its remedy. A rule holds GPU work back while its program uses the GPU (more than 5 % of the GPU’s time) and the rule protects `gpu_share` or, from 2.10, `progress_rate`. Your own explicit pause always binds, certificate or not: a rule’s or a phase’s `pause_fleet` with scope `gpu`, `all` or `io` pauses GPU work and admits none. ## The duty governor With a certificate, fleet GPU work whose runner declares `cooperative_throttle` runs beside the protected program at a **GPU budget** the agent keeps under the certified duty, on the program’s own progress: - It first holds GPU work for a minute to measure the program without it (again every 30 minutes, and whenever the last measurement is that old), then starts at the certified duty. - Every minute it compares the program’s progress with that baseline. Harm at the rule’s target (`max_slowdown`, 5 % by default) halves the budget; harm under half the target lets it grow by 0.05, never above the certificate. - Harm over twice the target, the program’s progress falling under half its baseline, or the memory guard takes the budget to 0 at once for five minutes. A progress source that cannot be read takes it to 0 until it reads again. - Each cooperative GPU job gets an equal share as its `gpu_duty`. The agent checks each one’s GPU time every two seconds: one over its duty plus 0.1 for three samples is paused for a minute, and the third time within an hour it is stopped and its module gets no duty for an hour. A job whose GPU time cannot be read waits. ```sh oarbank protection duty mac-01 ``` ```text mac-01: gpu_duty budget 0.3 of ceiling 0.35 (running) beside trainer; 2 unit(s) at 0.15 each baseline trainer: 9.94/s, 312 s old, last pressure 0.21 dev.codonic.oarbank.llmserve/batch attempt 812 duty 0.15 measured 0.14 ok dev.codonic.oarbank.llmserve/batch attempt 813 duty 0.15 measured 0.16 ok ``` Without a node name it shows every node that has a budget. The protection page’s **GPU duty** card shows the same, with each unit’s asked and measured duty and its state (holds its duty, held, not trusted, cannot throttle, stopped). While a harness run measures GPU work, the governor stands down and the run alone decides what runs. GPU workloads take part like jobs: a running [workload](/oarbank/operate/workloads) that uses the GPU and declares `cooperative_throttle` gets an equal share of the budget, and the same compliance check holds and stops it through its own ladder. The node measures each unit’s GPU time from the operating system (per-process GPU time on macOS, the DRM driver’s counters or NVML on Linux, the GPU engine counters on Windows). A unit whose GPU time the node cannot read is held (`GPU_DUTY_UNVERIFIED`): on a node that reads no per-process GPU time, `certified` therefore runs no GPU work. What the timeline records: | Code | What happened | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | `GPU_DUTY_START` | the budget started beside the protected GPU work, at the certified ceiling or its last level | | `GPU_DUTY_BASELINE` | GPU work held for a minute to measure the program without it | | `GPU_DUTY_GROW` | the budget grew by 0.05 after a clean minute | | `GPU_DUTY_HALVE` | the program slowed to its target: the budget halved | | `GPU_DUTY_TAKE_BACK` | the budget went to 0 (harm over twice the target, a collapse, the memory guard, an unreadable source) | | `GPU_DUTY_CLAMP` | the budget fell to a lower certified ceiling (a new or an expired certificate) | | `GPU_DUTY_EXCEEDED` | a unit kept the GPU busier than its duty for three samples: held, or on the third time in an hour stopped and its module not trusted for an hour | | `GPU_DUTY_UNVERIFIED` | a unit’s GPU time could not be read: held | Module authors: a runner that declares `cooperative_throttle` holds `gpu_duty` with the SDK’s duty governor ([Share a GPU with a protected program](/oarbank/build/gpu-duty)).