io-thread-controller discovers VMs through pluggable backends, refreshes
their I/O-worker state, asks the selected scaling engine for decisions, and
keeps the live inventory synchronized with backend discovery.
QEMU doesn't support dynamically change the number of threads, https://github.com/tmakatos/qemu/tree/io-thread-controller contains an AI-assisted PoC.
Backends and scaling engines are independent extension points around the controller-owned inventory:
+-------------------------+
| Controller |
| owns inventory + engine |
| refreshes and actuates |
+-----------+-------------+
|
+------------------+------------------+
| |
v v
+--------------------------------+ +--------------------------------+
| Backend (e.g. QEMU) | | Scaling engine |
| discovers zero or more VMs | | evaluates the refreshed fleet |
| creates one Instance per VM | | returns one scaling plan |
| supplies per-VM InstanceClient | | observes applied outcomes |
+-----------------+--------------+ +--------------------------------+
|
v
+---------------------------+
| Instance data record |
| one VM + InstanceClient |
| (operations for that VM) |
+---------------------------+
Scaling decisions pass through VM ownership, controller thread-count, vCPU,
and host-CPU guards before the controller changes the backend pool. One atomic
vm-ownership.json registry below /run/io-thread-controller preserves
managed and unmanaged classifications across daemon restarts.
startup
|
v
load config + select engine
|
v
+----------------> discover backend VMs <------------------+
| | |
| v |
| +------ reconcile live inventory <-----+ |
| | | |
| v | |
| wait for event | |
| / \ | |
| poll timer backend/inotify event | |
| | | | |
| v +--------------------------+ |
| refresh every VM concurrently |
| | |
| +-- refresh failed --> remove VM --> rediscover------+
| | on next event |
| v |
| engine evaluates the refreshed fleet |
| | |
| v |
+--- Hold |
| | |
+-- Scale --> guards reject -------------------------------+
| |
v |
set backend thread count |
| |
+---------+---------+ |
| | |
error success |
| | |
v v |
log failure update live count |
| | |
+---------+---------+ |
| |
+------------------------------------------+
The daemon reads its controller configuration from
/etc/io-thread-controller.json, or from the file given with --config. The
controller configuration names the scaling engine and the directories holding
the engine and backend files: the active engine reads
<engine_config_dir>/<engine>.json, and each backend reads its own file from
backend_config_dir. Every file is JSON, and unknown keys are rejected so that
misspelt settings fail loudly.
If the controller file does not exist, the daemon runs with built-in defaults,
which --dump-config prints. A missing engine file likewise selects the
engine's defaults. Settings omitted from a file that does exist take the
defaults listed below.
At startup the daemon logs the effective controller, engine, and backend configuration, after defaults, command-line overrides, and duration rounding have been applied.
Controller, engine, and backend JSON files may contain an include field. Its
path is resolved relative to the including file, loaded recursively, and merged
before the including file's keys are applied.
Durations are integer seconds in settings ending in _secs. Every duration,
including engine ones, must be a whole multiple of scale_poll_secs, and zero
is allowed. Otherwise the daemon refuses to start and lists each offending
setting with the nearest valid values. Changing only scale_poll_secs can
therefore invalidate other settings.
To try a new poll period without rewriting the other settings, set
round_up_durations or pass --round-up-durations. Each offending duration is
then rounded up to the next multiple with a warning, and the effective
configuration logged at startup shows the rounded values.
Percentages are written as numbers from 0 to 100 in settings ending in
_percent.
The first five settings are required when the controller file exists.
| Setting | Default | Meaning |
|---|---|---|
engine |
required | Scaling engine name, e.g. threshold. |
engine_config_dir |
required | Directory holding <engine>.json. |
backend_config_dir |
required | Directory holding backend files, e.g. qemu.json. |
vm_state_path |
required | VM ownership file. |
scale_poll_secs |
required | Poll period in seconds; must be at least 1. |
round_up_durations |
false |
Round durations up to poll multiples instead of rejecting them. |
min_thread_count |
1 |
Fewest worker threads the controller will set. |
max_thread_count |
8 |
Most worker threads the controller will set. |
host_cpu_scale_up_ceiling_percent |
90 |
Host CPU use that blocks scale-up; 0 disables. |
cooldown_secs |
30 |
Per-VM pause after a successful scale action. |
max_instances_adjusted_per_poll |
0 |
Most VMs resized per poll; 0 means no limit. |
dry_run |
false |
Log decisions without resizing pools. |
enable_per_vm_status_line |
false |
Write a status line per VM every poll. |
enable_per_vm_main_log |
false |
Send per-VM status lines to the main log instead of per-VM files. |
enable_aggregate_status_line |
true |
Write one fleet-wide status line every poll. |
print_status_header |
false |
Log a legend of the status-line fields at startup. |
refresh_cgroup_on_each_read |
false |
Re-read the VM's cgroup CPU limits on every refresh. |
experimental_psi_monitoring |
false |
Sample host pressure-stall information every poll. |
Every scale_poll_secs the controller refreshes each VM, asks the engine for
decisions, and applies them. The thread-count bounds are hard limits: an
engine proposal outside min_thread_count and max_thread_count, or above the
VM's vCPU count, is not applied. min_thread_count must be at least 1 and no
greater than max_thread_count. When the controller file does not exist,
min_thread_count defaults to 2 rather than 1.
host_cpu_scale_up_ceiling_percent protects the host: while overall host CPU
use is at or above it, scale-ups are blocked, but scale-downs and reverts still
apply. cooldown_secs starts after every successful scale-up, scale-down, or
revert and blocks further automatic scale-ups and scale-downs of that VM until
it expires; reverts are never blocked by it.
max_instances_adjusted_per_poll limits how disruptive a single poll can be.
When more VMs need resizing, scale-downs are applied before scale-ups, and the
remaining VMs are reconsidered on the next poll.
vm_state_path records which VMs the controller manages and which it leaves
to the operator, as classified by the backend when it first sees each VM. The
default, /run/io-thread-controller/vm-ownership.json, survives daemon
restarts but not a host reboot.
By default each VM's status line goes to
/var/log/io-thread-controller/vms/<vm>.log, together with that VM's log
events; see Status line for the format.
enable_per_vm_main_log sends the status lines to the main log instead.
dry_run and print_status_header can also be enabled with --dry-run and
--print-status-header. refresh_cgroup_on_each_read and
experimental_psi_monitoring are described under
Cgroup throttling and
Experimental PSI monitoring.
The threshold engine reads threshold.json and sizes each VM's worker pool to
the CPU load it observes.
| Setting | Default | Meaning |
|---|---|---|
scale_up_threshold_percent |
95 |
Average per-thread CPU above which a thread is added. |
scale_down_sustain_secs |
30 |
How long scale-down must stay possible before it happens. |
max_scale_down_step |
0 |
Most threads removed at once; 0 means no limit. |
scale_up_min_gain_percent |
5 |
IOPS gain a scale-up must deliver; 0 disables the check. |
scale_down_revert_drop_percent |
5 |
IOPS loss that reverts a scale-down; 0 disables the check. |
scale_validation_delay_secs |
20 |
How long to measure after an action before validating it. |
scale_validation_delay_per_thread_secs |
0 |
Extra validation time per thread in the new pool. |
block_scale_up_when_cgroup_throttled |
true |
Skip scale-up while the VM's cgroup is being CPU-throttled. |
post_revert_up_grace_secs |
10 |
Time after a reverted scale-up during which metrics are ignored. |
post_revert_up_recovery_secs |
20 |
Time after the grace period spent re-measuring IOPS. |
post_revert_up_backoff_max_secs |
300 |
Cap on the post-revert hold; 0 disables backoff. |
scale_down_backoff_max_secs |
1200 |
Cap on the scale-down sustain time; 0 disables backoff. |
Each poll, the engine compares the pool's average per-thread CPU use with
scale_up_threshold_percent. Above it, the engine proposes one more thread.
Otherwise it works out the smallest pool that would keep the average at or
below the threshold. If that is smaller than the current pool for
scale_down_sustain_secs, the engine shrinks the pool to it, by at most
max_scale_down_step threads at a time.
After a successful action, the engine measures IOPS for
scale_validation_delay_secs, plus scale_validation_delay_per_thread_secs
for each thread in the resulting pool, which gives larger pools more time to
settle. A scale-up is reverted unless IOPS gained at least
scale_up_min_gain_percent. A scale-down is reverted when IOPS fell by more
than scale_down_revert_drop_percent while CPU use is above the scale-up
threshold; if CPU use stayed low, demand simply left, and the smaller pool is
kept. Setting either percentage to zero disables that direction's validation.
Repeated failures back off per VM. After a reverted scale-up, scale-up is
blocked for post_revert_up_grace_secs, while the pool settles, plus
post_revert_up_recovery_secs, while the engine measures a fresh IOPS baseline
for the next attempt. That hold doubles with each consecutive failed scale-up,
up to post_revert_up_backoff_max_secs, and scale-down remains allowed
throughout. After a reverted scale-down, the next scale-down needs
scale_down_sustain_secs of low utilisation doubled once per consecutive
failure, up to scale_down_backoff_max_secs. A kept action resets its
direction's backoff, as does an applied action in the opposite direction.
Backends may select fallback /proc CPU samples with regular expressions and
an exact-name ignore list. Exact ignores take precedence; when no match
expressions are configured, every nonignored task is sampled.
refreshed
|
+-- below threshold / at controller maximum ----------> hold
|
+-- saturated --> propose current + 1
|
+-- backend ownership rejects -> hold
|
+-- host CPU guard -------------> hold
|
+-- target outside bounds ------> hold
|
+-- target > vCPUs -------------> hold
|
+-- backend error -------------> report failure
|
+-- backend accepts -----------> report success
An Instance is a backend-neutral data record for one VM, not a backend trait.
It stores VM identity, latest metrics, process-local override state, and an
InstanceClient trait object that performs operations on exactly that VM.
A Backend is the fleet-level adapter for a whole class of VMs. It owns
backend-wide configuration and discovers zero or more Instance values. The
two abstractions are separate because discovery has fleet scope, while metric
queries and thread-count changes need an independently stateful handle for each
VM. This keeps the controller independent of libvirt, QMP, control sockets, and
other transport details.
The QEMU backend discovers active VMs through libvirt and sends QMP commands
through virDomainQemuMonitorCommand. Its configuration is loaded from
backends.d/qemu.json. Named IOThreads and virtqueue mappings can be inspected
or changed through the backend CLI and D-Bus operations.
With refresh_cgroup_on_each_read enabled, each refresh reads the VM process's
cgroup v2 cpu.max and cpu.stat. The threshold engine setting
block_scale_up_when_cgroup_throttled suppresses scale-up when
throttled_usec increased since the preceding sample.
With experimental_psi_monitoring enabled, each tick samples the CPU, I/O, and
memory some.avg10 values from /proc/pressure and exposes them to the active
scaling engine. Sampling is disabled by default.
Every tick the daemon emits one INFO line per tracked VM and one aggregate line
on the status tracing target. With --print-status-header, it also emits a
#-prefixed legend on startup.
By default each VM's line is appended to
/var/log/io-thread-controller/vms/<vm>.log, so the line itself does not repeat
the VM identifier. Fields are positional, without keys, and separated by single
spaces. The examples are wrapped with backslashes for readability; each emitted
record occupies one line.
2026-01-01T00:00:00.000Z status=4 155593 0 0 154995 153840 151220 \
20394 0 90 87 358 40 80 120 45 - - - - 23 16 19
After the leading timestamp, the status= prefix distinguishes status records
from event= records, and the fields are:
- matched worker threads.
- current read, write, and other operations per second (3 fields).
- average total IOPS over rolling 1m, 5m, and 15m windows (3 fields).
- current read and write bandwidth in MB/s (2 fields).
- average, median, and total worker CPU percentage (3 fields).
- read latency p50, p95, p99, and average in microseconds (4 fields).
- write latency p50, p95, p99, and average in microseconds (4 fields).
- CPU microseconds per completed I/O over rolling 1m, 5m, and 15m windows (3 fields).
A dash (-) means no sample is available for that field.
The per-VM log also records event=vm_added when the daemon starts tracking
the VM and event=vm_removed reason=<not_discovered|refresh_failed> when it
stops. Every other log event that names the VM in a vm field, such as scaling
decisions, blocked or failed scale operations, and backend errors, is copied to
the VM's file as its message followed by its remaining key=value fields.
INFO status: 1 4 154995 153840 151220
status: is the tracing target, which separates status lines from the
controller and engine targets. The space-separated fields are:
- the current VM count,
- their combined IO thread count, and
- the aggregate IOPS over a rolling 1 minute window,
- the aggregate IOPS over a rolling 5 minute window,
- the aggregate IOPS over a rolling 15 minute window,
Each cell is the sum of the corresponding per-VM rolling rate.
When the threshold engine decides to change the thread count
you get one INFO line on the engine target using the same
vm=<id> convention as the status lines:
INFO vm=vm-1615a59c-... util=89.55% action="up" thr=4->5
vm is the same identifier used by status lines, util is
the per-thread utilisation percentage that triggered the decision, action
is up, down, or revert, and thr is the requested
<old>-><new> transition.
Failed actuations (backend rejected the SetThreadCount call)
surface as a WARN on the controller target with the same
vm=..., dir=..., from=..., to=..., error=... shape; the
success path is intentionally silent because the engine's
own INFO already describes the move.
The daemon owns:
- bus name:
com.nutanix.io_thread_controller1 - object path:
/com/nutanix/io_thread_controller1 - interface:
com.nutanix.io_thread_controller1
Debug builds expose SetThreadCount(vm, threads, sticky) and
SetAllThreadCounts(threads, sticky). The latter attempts every VM in the
daemon inventory and reports partial failures without undoing successful
updates. Setting sticky=true suppresses automatic scaling for each successful
VM until a later debug call clears it. The override is kept in memory and is
lost when the daemon restarts.
GetStats() returns a JSON fleet snapshot. For example:
{
"tick": 42,
"vms": [{
"vm": "vm-1",
"thread_count": 3,
"manual_scaling_sticky": false,
"scaling_allowed": true,
"vcpu_count": 8,
"per_thread_util": 0.72,
"read_io_count": 1000,
"write_io_count": 250,
"other_io_count": 0
}]
}busctl --system call \
com.nutanix.io_thread_controller1 \
/com/nutanix/io_thread_controller1 \
com.nutanix.io_thread_controller1 \
GetStatsGetStats is available in release builds and world-readable under the shipped
D-Bus policy.
busctl --system call \
com.nutanix.io_thread_controller1 \
/com/nutanix/io_thread_controller1 \
com.nutanix.io_thread_controller1 \
SetThreadCount sub <vm> 4 trueRelease builds do not expose this method. The shipped D-Bus policy restricts the debug method to root.
Backends that support named IOThreads can also expose
GetIoThreadVqMapping, AddIoThread, DelIoThread, and
SetIoThreadVqMapping.
--log-style=auto uses the compact systemd form when JOURNAL_STREAM is set
and the full human form otherwise. For example, journald already supplies the
timestamp and service name, so the daemon emits:
INFO vm=vm-1 thr=4 iops=12000/0/0 ...
Use --log-style=human to retain the timestamp and tracing target when running
interactively, or --log-style=systemd to force compact output for another log
collector.
iothread-tui is a ratatui-based operator dashboard for the
whole io-thread-controller fleet. It is a pure daemon
client: at startup it connects to
com.nutanix.io_thread_controller1 on the system D-Bus,
polls GetSnapshot() at the user's --refresh-secs cadence,
and renders whatever roster the daemon reports. A running
daemon is therefore mandatory — the TUI refuses to start
without one (fail-fast is friendlier than silently spawning
a poller that will time out on every tick).
Because the daemon already owns backend discovery,
reconciliation, and every perf counter, the TUI stays
backend-agnostic without linking backend implementations.
Add a new backend to the daemon and the TUI picks it up on the
next GetSnapshot reply, with no TUI rebuild required.
It ships as a second binary of the same crate. make (the
default target) builds both the daemon and the TUI; set
TUI=0 on the make line to skip it on lean build hosts:
make # builds io-thread-controller + iothread-tui
make TUI=0 # daemon only
cargo build --features tui # equivalent to the make default
./target/debug/iothread-tui --help
Add QEMU=1 (or --features qemu) for the upstream-QEMU
daemon backend; the TUI itself does not link libvirt because
it never sees a backend directly.
iothread-tui IOPS @ 4 KiB refresh 500ms vms=3 theme=dark [h]elp [q]uit [m]etric [p]ause [r]efresh [t]heme [a]ggregate 12345 plots ↑↓ focus s+/- threads
┌── perf (IOPS @ 4 KiB) y=200k ───────────────────────────────────────┐
│ 4→5 │
│ ╭╮ ╭─╮ ┌─╯ ╭───── │
│ ╱ ╲──────╯ ╰─╯ 3→2 ╱ │
│ ╱ ─────┘ │
│ ─── vm-1 ─── vm-2 ─── TOTAL (grey) │
└───────────────────────────────────────────────────────────────────────┘
┌── cpu (%) y=400 ────────────────────────────────────────────────────┐
│ ... │
└───────────────────────────────────────────────────────────────────────┘
┌── vms ────────────────────────────────────────────────────────────┐
│ id thr iops r/w/o bw r/w cpu% us/iop lat_r ... qd
│ 1615a59c…dc4a vc=4 4 61k/0/0 240/0 380 6 250/... 239
│ 8ac13990…7a11 vc=8 2 30k/0/0 120/0 195 7 200/... 123
│ TOTAL 6 91k/0/0 360/0 575 6 -/-/- 362
└───────────────────────────────────────────────────────────────────────┘
-
Up to five stackable plot panes share the top of the frame; each is individually toggleable and equal-height splits scale their Y-axis when more than one is on:
1— perf (IOPS or MB/s, toggle withm)2— latency (µs; four lines per vm — read avg + read p99 + write avg + write p99. The avg line is the histogram-derived arithmetic mean the daemon publishes onSnapshotLatency.avgso operators can see both the typical and the tail-case in the same pane.)3— cpu (per-pool%; a 4-thread pool at 100% reads as 400)4— queue depth (per-VM virtqueue depth)5— cpu (µs / iop) (efficiency: lower is better; NaN samples on idle ticks are skipped so the trace does not jump to infinity)
-
Per-VM traces are colour-cycled through the palette so overlapping lines stay legible. When at least two vms are tracked, a bold TOTAL row is appended to the dashboard, and
atoggles a grey aggregate line (sum across every live vm) in each plot; it is off by default. -
Scale events on the perf pane render as bold text
"<old>→<new>"— green for scale-ups, red for scale-downs — placed on the line so operators can see both the direction and the magnitude of every actuation without leaving the plot. -
Y-axis auto-collapses to kilos once the value crosses 10 000 (e.g.
12.3k,1.2M) so a plot going from idle to 6-digit IOPS reads cleanly at both scales. -
The dashboard's
cpu %column showsper_thread_util * num_threads * 100, i.e. total CPU across the whole pool. Thevc=Nsuffix on theidcolumn is the guest vCPU count the backend reported at discovery time, so operators can eyeball how close+will drive the pool to the vCPU cap. -
Dashboard columns are configurable. Press
cto open a modal checkbox picker; toggle each column on/off withSpace/Enterand move the cursor with↑/↓(orj/k). The default set isid · thr · iops · bw · cpu · us/iop · qd · queues; the read/write latency triples stay off by default because they are noisy on the narrow terminal widths operators typically SSH into.idis a required column (marked[·]in the picker) so an operator cannot accidentally hide the vm label. -
The
qdcolumn reads asavg/median/total, computed by the daemon across every SCSI command virtqueue on the freshest snapshot. A separatequeuescolumn shows the raw VQ count soavg/medianare easy to sanity-check. Both show-on backends that do not model VQs. -
A
*suffix on thethrcolumn means the vm is currently manual-sticky (the daemon'smanual_scaling_stickyflag is set). Rows for vms whose most recent daemon refresh failed (alive=false) render dimmed.
| Key | Effect |
|---|---|
q, Esc, Ctrl-C |
Quit |
h, ? |
Show/hide the modal help overlay |
m |
Toggle perf pane between IOPS and MB/s |
p, Space |
Pause: freezes the X-axis and the sampler / rediscovery loops |
r |
Prompt for a new refresh cadence in ms (50 – 60 000) |
c |
Open dashboard-columns modal (Space/Enter toggle, ↑↓ move, c/Esc close) |
t |
Cycle theme (dark / light) |
a |
Toggle the aggregate plot line (off by default) |
1 / 2 / 3 / 4 / 5 |
Toggle the perf / lat / cpu / qd / cpu-per-iop pane |
↑, k |
Focus previous vm — hides every other line |
↓, j |
Focus next vm; last slot is TOTAL while the aggregate line is shown |
Esc |
Clear focus (repaint every vm) |
+, = |
Ask the daemon to add one thread to the focused vm |
-, _ |
Ask the daemon to remove one thread from the focused vm |
A |
Set every VM in the daemon inventory to an exact count |
s |
Toggle sticky mode for + / - (title bar shows [STICKY]) |
+ / - / A always route through the daemon. Both
per-frame stats (via GetSnapshot) and manual actuations
(via SetThreadCount or SetAllThreadCounts) go over
com.nutanix.io_thread_controller1 on
/com/nutanix/io_thread_controller1, so the daemon is the
single source of truth for manual_scaling_sticky. A targets the complete
daemon inventory, including VMs hidden by --select. A wire error on either
verb logs at WARN on the iothread-tui tracing target (and shows up in the
daemon journal too) but otherwise leaves the UI running so the operator can
retry.
There is no direct-backend fallback path — a missing daemon
is a fatal condition the binary refuses to start with.
Sticky mode is armed by pressing s (toggle). The title bar
shows [STICKY] while it is on, and every subsequent +/-
carries sticky=true. Press s again to disarm. We
deliberately do not wire a Ctrl-+ / Ctrl-- variant —
tmux swallows and rewrites those key events depending on its
own bindings and they never reach the TUI reliably.
Focus hides every other line. With no focus, every
vm draws. Once you focus a vm (↑/↓, or select
TOTAL on a >=2-vm fleet with the aggregate line shown) the plots repaint with only
the highlighted line(s) — the plot becomes a solo view of the
thing you are tracking. Esc clears focus and brings every
line back. + / - are ignored on TOTAL because
broadcasting a scale-op to a room full of VMs is almost always
a mistake. Scale annotations (<old>→<new>) render on top
of the line with the direction colour in the foreground (green
up, red down) and the vm's own line colour painted
behind the glyph, so overlapping annotations across
close-together lines stay attributable at a glance.
Themes (t to cycle) swap the per-VM palette, axis
label colour, and aggregate/accent colours. Two themes ship:
dark and light, both with high-contrast axis and dashboard
colours. Vm colour assignment is theme-independent, so switching
themes recolours every trace consistently.
iothread-tui [OPTIONS]
--select <PATTERN> Substring of the per-VM id
the daemon publishes.
Repeatable; empty list = every
vm the daemon reports.
--refresh-secs <SECONDS> `GetSnapshot` poll cadence. Decimals
are accepted; values below 0.05 are
clamped. [default: 1.0]
--metric <iops|bandwidth> Perf pane metric at startup;
toggle at runtime with `m`.
[default: iops]
--iop-size <BYTES> Displayed alongside "IOPS" in the
perf legend (e.g. "IOPS @ 4 KiB").
Purely informational. [default: 4096]
--window-s <SECONDS> Rolling window shown on the X axis.
[default: 120]
Backend socket paths, libvirt URIs, and the rediscovery
cadence are all owned by the daemon now (see the
backends.d/ config directory). The TUI has no knobs for
any of them.
Watch every backend the daemon reports, at 500 ms cadence, plotting IOPS with the perf + CPU panes on:
iothread-tui --refresh-secs 0.5
# then press '3' to also open the cpu pane
Follow a single VM by UUID prefix, plotting MB/s over a 60 s window:
iothread-tui --select 1615a59c --metric bandwidth --window-s 60
Manually nudge the pool for the focused vm:
# in the TUI: ↓ ↓ to focus vm-3, then '+' three times.