Prove co-tenancy is safe

Host protection keeps fleet work from harming the programs you protect. This page shows how to measure that it does: the paired on/off harness runs the fleet work under test beside one protected program in ON windows and holds it in OFF windows, in random order, and decides at your margin whether the program ran as fast with fleet work as without. A passing run is a co-tenancy certificate.

Before a run: give the rule a progress source

The harness measures the protected program’s own progress, so its rule needs protect.metric = "progress_rate" with a source: an event file the program writes, counted by event type.

[[rule]]
id        = "trainer"
match     = { path_prefix = "/Applications/Example Trainer.app/" }
tree      = "descendants"
immediate = true
protect   = { metric = "progress_rate", max_slowdown = 0.05,
              source = { jsonl = "~/Library/Application Support/Example/events-*.jsonl", event = "step_done" } }
[[rule.during]]
source      = { jsonl = "~/Library/Application Support/Example/events-*.jsonl", event = "status",
                field = "state.phase", on = ["uploading"] }
pause_fleet = { scope = "gpu" }
tags        = ["uplink"]

• jsonl may end in a glob: the newest matching file is followed, so a program that starts a new file on every restart is picked up at once.
• event counts the lines whose event field (or event_key, a dotted path) equals it.
• A during block’s phase comes from enter/exit events or from a field of one event type; immediate = true pauses fleet work the moment the phase begins, between protection’s two-second ticks.
• ~ is your home. On a system install the node’s service account cannot read it: your session helper (the small helper that runs in your login) reads the file and passes on counts and phase states, never the file’s contents.

Caution

While nobody is logged in, no session helper reads a source in a home folder. Protection then treats the phase as on and the progress as unknown, and holds fleet work back under that rule; the node page says source … needs the session helper. Put a source outside your home, readable by the service account, if the program runs while you are logged out.

Check the source reads before a run: the node’s Protection page, Co-tenancy section, shows a rate and who reads it (the agent or your session helper).

Sources are watched, and immediate rules act at once

Protection reads sources as they change, not only on its two-second tick: the agent watches the file (kqueue on macOS, inotify on Linux, a 250 ms check on Windows), follows a rename or a new file, and starts again from the top after a truncation. A source can be:

Kind	Keys	What it gives
Progress from a JSONL file	jsonl, event, event_key	a rate: matching lines per second
Progress from a log	path, log_regex	a rate: regex matches per second
Progress from a program you write	exec	a rate the program prints (run by the agent, never by a session helper)
A phase from events	jsonl, enter, exit, enter_soon, event_key	on after an enter event, off after exit
A phase from a field	jsonl, event, field, on, soon	on while the field holds one of the on values
A phase while a program runs	on_exec = { match, tree }	on while a matching process of yours runs

A rule with immediate = true pauses the fleet work its during block names the moment the phase begins, between ticks, and releases it on the phase’s own end. The Co-tenancy section shows each phase and progress source with its state and who reads it, and the event path’s latency from the source’s write to the pause (p50 and p99). The node’s timeline records PROTECTION_PHASE_ON and PROTECTION_PHASE_OFF with how the change was seen.

Protection never guesses in your favour: a progress source that cannot be read counts as harm, and a phase that cannot be read counts as on, so fleet work under the rule is held back. The node says so with its fix: PROTECTION_SOURCE_UNREADABLE (the agent cannot read the file) or PROTECTION_SOURCE_NEEDS_HELPER (the file is in a home folder and no session helper reports it).

Run the harness

oarbank protection ab start mac-01 --rule trainer --work cpu

Option	Default	Meaning
--work	gpu	the fleet work under test: gpu, cpu, all, a module, or <module>/<workload>
--duty	none	the gpu_duty cooperative GPU work holds in ON windows
--blocks	12	planned blocks of one ON and one OFF window; raised after the pilot when the margin needs more
--window, --washout	300, 60 s	each window, and the seconds discarded after each switch
--margin	0.02	the slowdown that counts
--off	paused	how OFF holds the work: paused, or evicted to free its memory too
--max-hours	8	the run’s cap: a run that cannot reach the margin within it is refused after the pilot

It is a reviewed change (T2): you see the rule, its source, the work and the planned hours before anything starts. Protection stays active throughout, and a collapse of the program’s progress in an ON window aborts the run at once as a slowdown. Watch it with oarbank protection ab status mac-01, stop it with oarbank protection ab abort <run>.

Note

Run the harness in a window you choose, for example overnight while the program works. A block is left out and repeated, never counted, when something outside the fleet changed: a phase of the rule, a restart of the program, sleep, the power source, the memory guard. Heat is not left out: if fleet work makes the machine throttle, that slowdown counts against it.

Read the report

oarbank protection ab show <run>, or the run’s page in the console:

• No slowdown shown: the program ran at, say, 99.4 % of its speed without fleet work (90 % interval 98.6–100.3 %), within the 2 % margin. The run becomes a certificate.
• Slowdown: the whole interval is below 100 % and the estimate below the margin.
• Inconclusive: the interval reaches below the margin; the report says about how many more blocks would decide it.

The page plots each block’s OFF and ON rate side by side, shows R (ON ÷ OFF) with its interval against the margin, and lists the blocks left out and why.

Certificates and certified GPU co-tenancy

A certificate names the node, the rule, the fleet work and duty, R with its interval, the margin and the run. It is valid for 14 days, and only while the node’s hardware, OS major version and agent and the protected program are the ones it was earned under: oarbank protection certificates mac-01 shows each as valid, expired or stale with why.

To find how much GPU work a protected GPU program tolerates, sweep the duty:

oarbank protection ab sweep mac-01 --rule trainer

It runs the harness at gpu_duty 0.1, 0.2, 0.35, 0.5, 0.65, 0.8 and 1.0 (or your --levels), one run each, and stops at the first level that does not pass. The highest passing level is the node’s certified duty for that rule. Then set the node’s gpu_jobs to certified in its protection section:

[node]
gpu_jobs = "certified"

oarbank protection preview mac-01 node-protection.toml
oarbank protection set mac-01 node-protection.toml

set replaces the node’s own section, so keep its rules in the same file. gpu_jobs takes never, when_no_gpu_protected (the default: fleet GPU work runs only while no protected program uses the GPU), certified and always; across scopes the strictest wins. Without a valid certificate for every rule that holds GPU work back on the node, certified behaves as when_no_gpu_protected: GPU work waits, and the node shows GPU_NOT_CERTIFIED with the sweep as its remedy.

A rule holds GPU work back while its program uses the GPU (more than 5 % of the GPU’s time) and the rule protects gpu_share or, from 2.10, progress_rate. Your own explicit pause always binds, certificate or not: a rule’s or a phase’s pause_fleet with scope gpu, all or io pauses GPU work and admits none.

The duty governor

With a certificate, fleet GPU work whose runner declares cooperative_throttle runs beside the protected program at a GPU budget the agent keeps under the certified duty, on the program’s own progress:

• It first holds GPU work for a minute to measure the program without it (again every 30 minutes, and whenever the last measurement is that old), then starts at the certified duty.
• Every minute it compares the program’s progress with that baseline. Harm at the rule’s target (max_slowdown, 5 % by default) halves the budget; harm under half the target lets it grow by 0.05, never above the certificate.
• Harm over twice the target, the program’s progress falling under half its baseline, or the memory guard takes the budget to 0 at once for five minutes. A progress source that cannot be read takes it to 0 until it reads again.
• Each cooperative GPU job gets an equal share as its gpu_duty. The agent checks each one’s GPU time every two seconds: one over its duty plus 0.1 for three samples is paused for a minute, and the third time within an hour it is stopped and its module gets no duty for an hour. A job whose GPU time cannot be read waits.

oarbank protection duty mac-01

mac-01: gpu_duty budget 0.3 of ceiling 0.35 (running) beside trainer; 2 unit(s) at 0.15 each
  baseline trainer: 9.94/s, 312 s old, last pressure 0.21
  dev.codonic.oarbank.llmserve/batch attempt 812    duty 0.15   measured 0.14   ok
  dev.codonic.oarbank.llmserve/batch attempt 813    duty 0.15   measured 0.16   ok

Without a node name it shows every node that has a budget. The protection page’s GPU duty card shows the same, with each unit’s asked and measured duty and its state (holds its duty, held, not trusted, cannot throttle, stopped). While a harness run measures GPU work, the governor stands down and the run alone decides what runs.

GPU workloads take part like jobs: a running workload that uses the GPU and declares cooperative_throttle gets an equal share of the budget, and the same compliance check holds and stops it through its own ladder.

The node measures each unit’s GPU time from the operating system (per-process GPU time on macOS, the DRM driver’s counters or NVML on Linux, the GPU engine counters on Windows). A unit whose GPU time the node cannot read is held (GPU_DUTY_UNVERIFIED): on a node that reads no per-process GPU time, certified therefore runs no GPU work.

What the timeline records:

Code	What happened
GPU_DUTY_START	the budget started beside the protected GPU work, at the certified ceiling or its last level
GPU_DUTY_BASELINE	GPU work held for a minute to measure the program without it
GPU_DUTY_GROW	the budget grew by 0.05 after a clean minute
GPU_DUTY_HALVE	the program slowed to its target: the budget halved
GPU_DUTY_TAKE_BACK	the budget went to 0 (harm over twice the target, a collapse, the memory guard, an unreadable source)
GPU_DUTY_CLAMP	the budget fell to a lower certified ceiling (a new or an expired certificate)
GPU_DUTY_EXCEEDED	a unit kept the GPU busier than its duty for three samples: held, or on the third time in an hour stopped and its module not trusted for an hour
GPU_DUTY_UNVERIFIED	a unit’s GPU time could not be read: held

Module authors: a runner that declares cooperative_throttle holds gpu_duty with the SDK’s duty governor (Share a GPU with a protected program).