Skip to content

Latest commit

 

History

History
676 lines (562 loc) · 31.2 KB

README.md

File metadata and controls

676 lines (562 loc) · 31.2 KB

io-thread-controller

io-thread-controller discovers VMs through pluggable backends, refreshes their I/O-worker state, asks the selected scaling engine for decisions, and keeps the live inventory synchronized with backend discovery.

Dynamic IO thread scaling in QEMU

QEMU doesn't support dynamically change the number of threads, https://github.com/tmakatos/qemu/tree/io-thread-controller contains an AI-assisted PoC.

Core control loop

Backends and scaling engines are independent extension points around the controller-owned inventory:

                         +-------------------------+
                         | Controller              |
                         | owns inventory + engine |
                         | refreshes and actuates  |
                         +-----------+-------------+
                                     |
                  +------------------+------------------+
                  |                                     |
                  v                                     v
+--------------------------------+     +--------------------------------+
| Backend        (e.g. QEMU)     |     | Scaling engine                 |
| discovers zero or more VMs     |     | evaluates the refreshed fleet |
| creates one Instance per VM    |     | returns one scaling plan       |
| supplies per-VM InstanceClient |     | observes applied outcomes      |
+-----------------+--------------+     +--------------------------------+
                  |
                  v
    +---------------------------+
    | Instance data record      |
    | one VM + InstanceClient   |
    | (operations for that VM)  |
    +---------------------------+

Scaling decisions pass through VM ownership, controller thread-count, vCPU, and host-CPU guards before the controller changes the backend pool. One atomic vm-ownership.json registry below /run/io-thread-controller preserves managed and unmanaged classifications across daemon restarts.

                          startup
                             |
                             v
                  load config + select engine
                             |
                             v
+----------------> discover backend VMs <------------------+
|                            |                             |
|                            v                             |
|             +------ reconcile live inventory <-----+     |
|             |                                      |     |
|             v                                      |     |
|        wait for event                              |     |
|       /              \                             |     |
| poll timer       backend/inotify event             |     |
|     |                   |                          |     |
|     v                   +--------------------------+     |
| refresh every VM concurrently                            |
|     |                                                    |
|     +-- refresh failed --> remove VM --> rediscover------+
|     |                                    on next event   |
|     v                                                    |
| engine evaluates the refreshed fleet                     |
|     |                                                    |
|     v                                                    |
+--- Hold                                                  |
|     |                                                    |
+-- Scale --> guards reject -------------------------------+
                |                                          |
                v                                          |
        set backend thread count                           |
                |                                          |
      +---------+---------+                                |
      |                   |                                |
    error              success                             |
      |                   |                                |
      v                   v                                |
 log failure       update live count                       |
      |                   |                                |
      +---------+---------+                                |
                |                                          |
                +------------------------------------------+

Configuration

The daemon reads its controller configuration from /etc/io-thread-controller.json, or from the file given with --config. The controller configuration names the scaling engine and the directories holding the engine and backend files: the active engine reads <engine_config_dir>/<engine>.json, and each backend reads its own file from backend_config_dir. Every file is JSON, and unknown keys are rejected so that misspelt settings fail loudly.

If the controller file does not exist, the daemon runs with built-in defaults, which --dump-config prints. A missing engine file likewise selects the engine's defaults. Settings omitted from a file that does exist take the defaults listed below.

At startup the daemon logs the effective controller, engine, and backend configuration, after defaults, command-line overrides, and duration rounding have been applied.

Includes

Controller, engine, and backend JSON files may contain an include field. Its path is resolved relative to the including file, loaded recursively, and merged before the including file's keys are applied.

Durations

Durations are integer seconds in settings ending in _secs. Every duration, including engine ones, must be a whole multiple of scale_poll_secs, and zero is allowed. Otherwise the daemon refuses to start and lists each offending setting with the nearest valid values. Changing only scale_poll_secs can therefore invalidate other settings.

To try a new poll period without rewriting the other settings, set round_up_durations or pass --round-up-durations. Each offending duration is then rounded up to the next multiple with a warning, and the effective configuration logged at startup shows the rounded values.

Percentages are written as numbers from 0 to 100 in settings ending in _percent.

Controller

The first five settings are required when the controller file exists.

Setting Default Meaning
engine required Scaling engine name, e.g. threshold.
engine_config_dir required Directory holding <engine>.json.
backend_config_dir required Directory holding backend files, e.g. qemu.json.
vm_state_path required VM ownership file.
scale_poll_secs required Poll period in seconds; must be at least 1.
round_up_durations false Round durations up to poll multiples instead of rejecting them.
min_thread_count 1 Fewest worker threads the controller will set.
max_thread_count 8 Most worker threads the controller will set.
host_cpu_scale_up_ceiling_percent 90 Host CPU use that blocks scale-up; 0 disables.
cooldown_secs 30 Per-VM pause after a successful scale action.
max_instances_adjusted_per_poll 0 Most VMs resized per poll; 0 means no limit.
dry_run false Log decisions without resizing pools.
enable_per_vm_status_line false Write a status line per VM every poll.
enable_per_vm_main_log false Send per-VM status lines to the main log instead of per-VM files.
enable_aggregate_status_line true Write one fleet-wide status line every poll.
print_status_header false Log a legend of the status-line fields at startup.
refresh_cgroup_on_each_read false Re-read the VM's cgroup CPU limits on every refresh.
experimental_psi_monitoring false Sample host pressure-stall information every poll.

Every scale_poll_secs the controller refreshes each VM, asks the engine for decisions, and applies them. The thread-count bounds are hard limits: an engine proposal outside min_thread_count and max_thread_count, or above the VM's vCPU count, is not applied. min_thread_count must be at least 1 and no greater than max_thread_count. When the controller file does not exist, min_thread_count defaults to 2 rather than 1.

host_cpu_scale_up_ceiling_percent protects the host: while overall host CPU use is at or above it, scale-ups are blocked, but scale-downs and reverts still apply. cooldown_secs starts after every successful scale-up, scale-down, or revert and blocks further automatic scale-ups and scale-downs of that VM until it expires; reverts are never blocked by it.

max_instances_adjusted_per_poll limits how disruptive a single poll can be. When more VMs need resizing, scale-downs are applied before scale-ups, and the remaining VMs are reconsidered on the next poll.

vm_state_path records which VMs the controller manages and which it leaves to the operator, as classified by the backend when it first sees each VM. The default, /run/io-thread-controller/vm-ownership.json, survives daemon restarts but not a host reboot.

By default each VM's status line goes to /var/log/io-thread-controller/vms/<vm>.log, together with that VM's log events; see Status line for the format. enable_per_vm_main_log sends the status lines to the main log instead.

dry_run and print_status_header can also be enabled with --dry-run and --print-status-header. refresh_cgroup_on_each_read and experimental_psi_monitoring are described under Cgroup throttling and Experimental PSI monitoring.

Threshold engine

The threshold engine reads threshold.json and sizes each VM's worker pool to the CPU load it observes.

Setting Default Meaning
scale_up_threshold_percent 95 Average per-thread CPU above which a thread is added.
scale_down_sustain_secs 30 How long scale-down must stay possible before it happens.
max_scale_down_step 0 Most threads removed at once; 0 means no limit.
scale_up_min_gain_percent 5 IOPS gain a scale-up must deliver; 0 disables the check.
scale_down_revert_drop_percent 5 IOPS loss that reverts a scale-down; 0 disables the check.
scale_validation_delay_secs 20 How long to measure after an action before validating it.
scale_validation_delay_per_thread_secs 0 Extra validation time per thread in the new pool.
block_scale_up_when_cgroup_throttled true Skip scale-up while the VM's cgroup is being CPU-throttled.
post_revert_up_grace_secs 10 Time after a reverted scale-up during which metrics are ignored.
post_revert_up_recovery_secs 20 Time after the grace period spent re-measuring IOPS.
post_revert_up_backoff_max_secs 300 Cap on the post-revert hold; 0 disables backoff.
scale_down_backoff_max_secs 1200 Cap on the scale-down sustain time; 0 disables backoff.

Each poll, the engine compares the pool's average per-thread CPU use with scale_up_threshold_percent. Above it, the engine proposes one more thread. Otherwise it works out the smallest pool that would keep the average at or below the threshold. If that is smaller than the current pool for scale_down_sustain_secs, the engine shrinks the pool to it, by at most max_scale_down_step threads at a time.

After a successful action, the engine measures IOPS for scale_validation_delay_secs, plus scale_validation_delay_per_thread_secs for each thread in the resulting pool, which gives larger pools more time to settle. A scale-up is reverted unless IOPS gained at least scale_up_min_gain_percent. A scale-down is reverted when IOPS fell by more than scale_down_revert_drop_percent while CPU use is above the scale-up threshold; if CPU use stayed low, demand simply left, and the smaller pool is kept. Setting either percentage to zero disables that direction's validation.

Repeated failures back off per VM. After a reverted scale-up, scale-up is blocked for post_revert_up_grace_secs, while the pool settles, plus post_revert_up_recovery_secs, while the engine measures a fresh IOPS baseline for the next attempt. That hold doubles with each consecutive failed scale-up, up to post_revert_up_backoff_max_secs, and scale-down remains allowed throughout. After a reverted scale-down, the next scale-down needs scale_down_sustain_secs of low utilisation doubled once per consecutive failure, up to scale_down_backoff_max_secs. A kept action resets its direction's backoff, as does an applied action in the opposite direction.

Thread-name filtering

Backends may select fallback /proc CPU samples with regular expressions and an exact-name ignore list. Exact ignores take precedence; when no match expressions are configured, every nonignored task is sampled.

Scale-up state machine

refreshed
   |
   +-- below threshold / at controller maximum ----------> hold
   |
   +-- saturated --> propose current + 1
                         |
                         +-- backend ownership rejects -> hold
                         |
                         +-- host CPU guard -------------> hold
                         |
                         +-- target outside bounds ------> hold
                         |
                         +-- target > vCPUs -------------> hold
                         |
                         +-- backend error -------------> report failure
                         |
                         +-- backend accepts -----------> report success

Controller architecture

An Instance is a backend-neutral data record for one VM, not a backend trait. It stores VM identity, latest metrics, process-local override state, and an InstanceClient trait object that performs operations on exactly that VM.

A Backend is the fleet-level adapter for a whole class of VMs. It owns backend-wide configuration and discovers zero or more Instance values. The two abstractions are separate because discovery has fleet scope, while metric queries and thread-count changes need an independently stateful handle for each VM. This keeps the controller independent of libvirt, QMP, control sockets, and other transport details.

QEMU backend

The QEMU backend discovers active VMs through libvirt and sends QMP commands through virDomainQemuMonitorCommand. Its configuration is loaded from backends.d/qemu.json. Named IOThreads and virtqueue mappings can be inspected or changed through the backend CLI and D-Bus operations.

Cgroup throttling

With refresh_cgroup_on_each_read enabled, each refresh reads the VM process's cgroup v2 cpu.max and cpu.stat. The threshold engine setting block_scale_up_when_cgroup_throttled suppresses scale-up when throttled_usec increased since the preceding sample.

Experimental PSI monitoring

With experimental_psi_monitoring enabled, each tick samples the CPU, I/O, and memory some.avg10 values from /proc/pressure and exposes them to the active scaling engine. Sampling is disabled by default.

Status line

Every tick the daemon emits one INFO line per tracked VM and one aggregate line on the status tracing target. With --print-status-header, it also emits a #-prefixed legend on startup.

Per-VM line

By default each VM's line is appended to /var/log/io-thread-controller/vms/<vm>.log, so the line itself does not repeat the VM identifier. Fields are positional, without keys, and separated by single spaces. The examples are wrapped with backslashes for readability; each emitted record occupies one line.

2026-01-01T00:00:00.000Z status=4 155593 0 0 154995 153840 151220 \
  20394 0 90 87 358 40 80 120 45 - - - - 23 16 19

After the leading timestamp, the status= prefix distinguishes status records from event= records, and the fields are:

  1. matched worker threads.
  2. current read, write, and other operations per second (3 fields).
  3. average total IOPS over rolling 1m, 5m, and 15m windows (3 fields).
  4. current read and write bandwidth in MB/s (2 fields).
  5. average, median, and total worker CPU percentage (3 fields).
  6. read latency p50, p95, p99, and average in microseconds (4 fields).
  7. write latency p50, p95, p99, and average in microseconds (4 fields).
  8. CPU microseconds per completed I/O over rolling 1m, 5m, and 15m windows (3 fields).

A dash (-) means no sample is available for that field.

The per-VM log also records event=vm_added when the daemon starts tracking the VM and event=vm_removed reason=<not_discovered|refresh_failed> when it stops. Every other log event that names the VM in a vm field, such as scaling decisions, blocked or failed scale operations, and backend errors, is copied to the VM's file as its message followed by its remaining key=value fields.

Aggregate line

INFO status: 1 4 154995 153840 151220

status: is the tracing target, which separates status lines from the controller and engine targets. The space-separated fields are:

  1. the current VM count,
  2. their combined IO thread count, and
  3. the aggregate IOPS over a rolling 1 minute window,
  4. the aggregate IOPS over a rolling 5 minute window,
  5. the aggregate IOPS over a rolling 15 minute window,

Each cell is the sum of the corresponding per-VM rolling rate.

Scaling verdicts

When the threshold engine decides to change the thread count you get one INFO line on the engine target using the same vm=<id> convention as the status lines:

INFO vm=vm-1615a59c-... util=89.55% action="up" thr=4->5

vm is the same identifier used by status lines, util is the per-thread utilisation percentage that triggered the decision, action is up, down, or revert, and thr is the requested <old>-><new> transition.

Failed actuations (backend rejected the SetThreadCount call) surface as a WARN on the controller target with the same vm=..., dir=..., from=..., to=..., error=... shape; the success path is intentionally silent because the engine's own INFO already describes the move.

D-Bus

The daemon owns:

  • bus name: com.nutanix.io_thread_controller1
  • object path: /com/nutanix/io_thread_controller1
  • interface: com.nutanix.io_thread_controller1

Debug builds expose SetThreadCount(vm, threads, sticky) and SetAllThreadCounts(threads, sticky). The latter attempts every VM in the daemon inventory and reports partial failures without undoing successful updates. Setting sticky=true suppresses automatic scaling for each successful VM until a later debug call clears it. The override is kept in memory and is lost when the daemon restarts.

GetStats() returns a JSON fleet snapshot. For example:

{
  "tick": 42,
  "vms": [{
    "vm": "vm-1",
    "thread_count": 3,
    "manual_scaling_sticky": false,
    "scaling_allowed": true,
    "vcpu_count": 8,
    "per_thread_util": 0.72,
    "read_io_count": 1000,
    "write_io_count": 250,
    "other_io_count": 0
  }]
}
busctl --system call \
  com.nutanix.io_thread_controller1 \
  /com/nutanix/io_thread_controller1 \
  com.nutanix.io_thread_controller1 \
  GetStats

GetStats is available in release builds and world-readable under the shipped D-Bus policy.

busctl --system call \
  com.nutanix.io_thread_controller1 \
  /com/nutanix/io_thread_controller1 \
  com.nutanix.io_thread_controller1 \
  SetThreadCount sub <vm> 4 true

Release builds do not expose this method. The shipped D-Bus policy restricts the debug method to root.

Backends that support named IOThreads can also expose GetIoThreadVqMapping, AddIoThread, DelIoThread, and SetIoThreadVqMapping.

Log style

--log-style=auto uses the compact systemd form when JOURNAL_STREAM is set and the full human form otherwise. For example, journald already supplies the timestamp and service name, so the daemon emits:

INFO vm=vm-1 thr=4 iops=12000/0/0 ...

Use --log-style=human to retain the timestamp and tracing target when running interactively, or --log-style=systemd to force compact output for another log collector.

iothread-tui

iothread-tui is a ratatui-based operator dashboard for the whole io-thread-controller fleet. It is a pure daemon client: at startup it connects to com.nutanix.io_thread_controller1 on the system D-Bus, polls GetSnapshot() at the user's --refresh-secs cadence, and renders whatever roster the daemon reports. A running daemon is therefore mandatory — the TUI refuses to start without one (fail-fast is friendlier than silently spawning a poller that will time out on every tick).

Because the daemon already owns backend discovery, reconciliation, and every perf counter, the TUI stays backend-agnostic without linking backend implementations. Add a new backend to the daemon and the TUI picks it up on the next GetSnapshot reply, with no TUI rebuild required.

It ships as a second binary of the same crate. make (the default target) builds both the daemon and the TUI; set TUI=0 on the make line to skip it on lean build hosts:

make                        # builds io-thread-controller + iothread-tui
make TUI=0                  # daemon only
cargo build --features tui  # equivalent to the make default
./target/debug/iothread-tui --help

Add QEMU=1 (or --features qemu) for the upstream-QEMU daemon backend; the TUI itself does not link libvirt because it never sees a backend directly.

Layout

iothread-tui  IOPS @ 4 KiB  refresh 500ms  vms=3  theme=dark   [h]elp [q]uit [m]etric [p]ause [r]efresh [t]heme [a]ggregate 12345 plots ↑↓ focus s+/- threads
┌── perf (IOPS @ 4 KiB)   y=200k ───────────────────────────────────────┐
│                          4→5                                          │
│         ╭╮        ╭─╮  ┌─╯          ╭─────                            │
│        ╱  ╲──────╯   ╰─╯     3→2   ╱                                  │
│       ╱                     ─────┘                                    │
│  ─── vm-1  ─── vm-2  ─── TOTAL (grey)                                 │
└───────────────────────────────────────────────────────────────────────┘
┌── cpu (%)   y=400 ────────────────────────────────────────────────────┐
│         ...                                                           │
└───────────────────────────────────────────────────────────────────────┘
┌── vms ────────────────────────────────────────────────────────────┐
│ id                    thr  iops r/w/o  bw r/w  cpu%  us/iop  lat_r ...  qd
│ 1615a59c…dc4a vc=4      4  61k/0/0     240/0    380  6       250/... 239
│ 8ac13990…7a11 vc=8      2  30k/0/0     120/0    195  7       200/... 123
│ TOTAL                   6  91k/0/0     360/0    575  6       -/-/-   362
└───────────────────────────────────────────────────────────────────────┘
  • Up to five stackable plot panes share the top of the frame; each is individually toggleable and equal-height splits scale their Y-axis when more than one is on:

    • 1 — perf (IOPS or MB/s, toggle with m)
    • 2 — latency (µs; four lines per vm — read avg + read p99 + write avg + write p99. The avg line is the histogram-derived arithmetic mean the daemon publishes on SnapshotLatency.avg so operators can see both the typical and the tail-case in the same pane.)
    • 3 — cpu (per-pool %; a 4-thread pool at 100% reads as 400)
    • 4 — queue depth (per-VM virtqueue depth)
    • 5 — cpu (µs / iop) (efficiency: lower is better; NaN samples on idle ticks are skipped so the trace does not jump to infinity)
  • Per-VM traces are colour-cycled through the palette so overlapping lines stay legible. When at least two vms are tracked, a bold TOTAL row is appended to the dashboard, and a toggles a grey aggregate line (sum across every live vm) in each plot; it is off by default.

  • Scale events on the perf pane render as bold text "<old>→<new>" — green for scale-ups, red for scale-downs — placed on the line so operators can see both the direction and the magnitude of every actuation without leaving the plot.

  • Y-axis auto-collapses to kilos once the value crosses 10 000 (e.g. 12.3k, 1.2M) so a plot going from idle to 6-digit IOPS reads cleanly at both scales.

  • The dashboard's cpu % column shows per_thread_util * num_threads * 100, i.e. total CPU across the whole pool. The vc=N suffix on the id column is the guest vCPU count the backend reported at discovery time, so operators can eyeball how close + will drive the pool to the vCPU cap.

  • Dashboard columns are configurable. Press c to open a modal checkbox picker; toggle each column on/off with Space / Enter and move the cursor with ↑ / ↓ (or j / k). The default set is id · thr · iops · bw · cpu · us/iop · qd · queues; the read/write latency triples stay off by default because they are noisy on the narrow terminal widths operators typically SSH into. id is a required column (marked [·] in the picker) so an operator cannot accidentally hide the vm label.

  • The qd column reads as avg/median/total, computed by the daemon across every SCSI command virtqueue on the freshest snapshot. A separate queues column shows the raw VQ count so avg/median are easy to sanity-check. Both show - on backends that do not model VQs.

  • A * suffix on the thr column means the vm is currently manual-sticky (the daemon's manual_scaling_sticky flag is set). Rows for vms whose most recent daemon refresh failed (alive=false) render dimmed.

Hotkeys

Key Effect
q, Esc, Ctrl-C Quit
h, ? Show/hide the modal help overlay
m Toggle perf pane between IOPS and MB/s
p, Space Pause: freezes the X-axis and the sampler / rediscovery loops
r Prompt for a new refresh cadence in ms (50 – 60 000)
c Open dashboard-columns modal (Space/Enter toggle, ↑↓ move, c/Esc close)
t Cycle theme (dark / light)
a Toggle the aggregate plot line (off by default)
1 / 2 / 3 / 4 / 5 Toggle the perf / lat / cpu / qd / cpu-per-iop pane
↑, k Focus previous vm — hides every other line
↓, j Focus next vm; last slot is TOTAL while the aggregate line is shown
Esc Clear focus (repaint every vm)
+, = Ask the daemon to add one thread to the focused vm
-, _ Ask the daemon to remove one thread from the focused vm
A Set every VM in the daemon inventory to an exact count
s Toggle sticky mode for + / - (title bar shows [STICKY])

+ / - / A always route through the daemon. Both per-frame stats (via GetSnapshot) and manual actuations (via SetThreadCount or SetAllThreadCounts) go over com.nutanix.io_thread_controller1 on /com/nutanix/io_thread_controller1, so the daemon is the single source of truth for manual_scaling_sticky. A targets the complete daemon inventory, including VMs hidden by --select. A wire error on either verb logs at WARN on the iothread-tui tracing target (and shows up in the daemon journal too) but otherwise leaves the UI running so the operator can retry. There is no direct-backend fallback path — a missing daemon is a fatal condition the binary refuses to start with.

Sticky mode is armed by pressing s (toggle). The title bar shows [STICKY] while it is on, and every subsequent +/- carries sticky=true. Press s again to disarm. We deliberately do not wire a Ctrl-+ / Ctrl-- variant — tmux swallows and rewrites those key events depending on its own bindings and they never reach the TUI reliably.

Focus hides every other line. With no focus, every vm draws. Once you focus a vm (↑/↓, or select TOTAL on a >=2-vm fleet with the aggregate line shown) the plots repaint with only the highlighted line(s) — the plot becomes a solo view of the thing you are tracking. Esc clears focus and brings every line back. + / - are ignored on TOTAL because broadcasting a scale-op to a room full of VMs is almost always a mistake. Scale annotations (<old>→<new>) render on top of the line with the direction colour in the foreground (green up, red down) and the vm's own line colour painted behind the glyph, so overlapping annotations across close-together lines stay attributable at a glance.

Themes (t to cycle) swap the per-VM palette, axis label colour, and aggregate/accent colours. Two themes ship: dark and light, both with high-contrast axis and dashboard colours. Vm colour assignment is theme-independent, so switching themes recolours every trace consistently.

Command line

iothread-tui [OPTIONS]

  --select <PATTERN>       Substring of the per-VM id
                           the daemon publishes.
                           Repeatable; empty list = every
                           vm the daemon reports.
  --refresh-secs <SECONDS> `GetSnapshot` poll cadence. Decimals
                           are accepted; values below 0.05 are
                           clamped. [default: 1.0]
  --metric <iops|bandwidth>  Perf pane metric at startup;
                             toggle at runtime with `m`.
                             [default: iops]
  --iop-size <BYTES>       Displayed alongside "IOPS" in the
                           perf legend (e.g. "IOPS @ 4 KiB").
                           Purely informational.  [default: 4096]
  --window-s <SECONDS>     Rolling window shown on the X axis.
                           [default: 120]

Backend socket paths, libvirt URIs, and the rediscovery cadence are all owned by the daemon now (see the backends.d/ config directory). The TUI has no knobs for any of them.

Examples

Watch every backend the daemon reports, at 500 ms cadence, plotting IOPS with the perf + CPU panes on:

iothread-tui --refresh-secs 0.5
# then press '3' to also open the cpu pane

Follow a single VM by UUID prefix, plotting MB/s over a 60 s window:

iothread-tui --select 1615a59c --metric bandwidth --window-s 60

Manually nudge the pool for the focused vm:

# in the TUI: ↓ ↓  to focus vm-3, then '+' three times.