Skip to content

Don't reconnect to known control sockets on every discovery - #137

Open
tmakatos wants to merge 3 commits into
mainfrom
control-socket-peer-cache
Open

tmakatos wants to merge 3 commits into
mainfrom
control-socket-peer-cache

Conversation

@tmakatos

@tmakatos tmakatos commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Discovery connected to every VM's control socket on every pass, only
to read the peer PID, even for VMs already tracked. Backends that
listen with a minimal backlog reject such a connection
with EAGAIN while an earlier one is still pending, which logged a
warning and dropped the VM from that discovery pass.

Cache the peer PID per socket path and reconnect only when the socket
file is replaced or the cached process is gone. EAGAIN on a socket that
isn't cached yet is transient, so log it at debug and retry on the next
pass.

AI-generated: everything
Signed-off-by: Thanos Makatos thanos.makatos@nutanix.com


Stack created with GitHub Stacks CLI • Give Feedback 💬

@tmakatos
tmakatos added this pull request to stack #139 October 21, 2026 22:08
@tmakatos
tmakatos marked this pull request as draft October 21, 2026 22:09
@tmakatos
tmakatos force-pushed the control-socket-peer-cache branch 3 times, most recently from 606a377 to 9760a36 Compare October 22, 2026 09:40
@tmakatos
tmakatos marked this pull request as ready for review October 22, 2026 09:41
@tmakatos
tmakatos force-pushed the control-socket-peer-cache branch from 9760a36 to 8fc04dd Compare October 22, 2026 10:14
@tmakatos
tmakatos removed this pull request from stack #139 October 22, 2026 10:22
@tmakatos
tmakatos added this pull request to stack #146 October 22, 2026 10:22
@tmakatos
tmakatos removed this pull request from stack #146 October 22, 2026 10:30
@tmakatos
tmakatos force-pushed the control-socket-peer-cache branch from 8fc04dd to 197c8cc Compare October 22, 2026 10:30
@tmakatos
tmakatos added this pull request to stack #147 October 22, 2026 10:30
@tmakatos
tmakatos removed this pull request from stack #147 October 22, 2026 10:47
@tmakatos
tmakatos force-pushed the control-socket-peer-cache branch from 197c8cc to 46b2f39 Compare October 22, 2026 10:47
@tmakatos
tmakatos added this pull request to stack #149 October 22, 2026 10:47
@tmakatos
tmakatos force-pushed the control-socket-peer-cache branch from 46b2f39 to 96b7e70 Compare October 22, 2026 10:52
@tmakatos
tmakatos force-pushed the control-socket-peer-cache branch from 96b7e70 to 46b2f39 Compare October 22, 2026 11:04
@tmakatos
tmakatos force-pushed the control-socket-peer-cache branch 3 times, most recently from 5402770 to c88b7fb Compare October 22, 2026 12:32
@tmakatos
tmakatos force-pushed the control-socket-peer-cache branch from c88b7fb to e2895c4 Compare October 22, 2026 13:20
@tmakatos
tmakatos force-pushed the control-socket-peer-cache branch from 80c49ec to 3e34d41 Compare October 22, 2026 13:40
@tmakatos
tmakatos force-pushed the control-socket-peer-cache branch from 3e34d41 to 3bd8a91 Compare October 22, 2026 13:43
@tmakatos
tmakatos force-pushed the control-socket-peer-cache branch 2 times, most recently from 27de12c to e38e016 Compare October 22, 2026 13:59
@tmakatos
tmakatos force-pushed the control-socket-peer-cache branch 2 times, most recently from 13f33fd to a44899d Compare October 22, 2026 16:45
@tmakatos
tmakatos force-pushed the control-socket-peer-cache branch 2 times, most recently from 67c8197 to 758c8f5 Compare October 23, 2026 07:23
Comment thread src/backends/mod.rs
Comment on lines +313 to +319
/// Control-socket peers identified by earlier discovery passes, keyed by
/// socket path.
///
/// Backends may listen with a minimal backlog, in which case connecting on
/// every pass just to read the peer credentials fails with `EAGAIN` whenever
/// an earlier connection has not been accepted yet.
static PEER_CACHE: LazyLock<Mutex<HashMap<PathBuf, CachedPeer>>> = LazyLock::new(Default::default);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would make it tougher to test, could we make an Inventory struct or similar which keeps this state and have cached_control_socket_peer_pid and discover_via_control_socket on that?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which aspect exactly makes it harder to test? The fact that it's static?

Comment thread src/backends/mod.rs Outdated
Comment thread src/backends/mod.rs

/// Return the start time of `pid`, or `None` if it no longer exists.
fn process_start_time(pid: i32) -> Option<u64> {
procfs::process::Process::new(pid)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably use new_with_root and the util::Path

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you talking about how it will be after you've merged your PR?

@tmakatos
tmakatos force-pushed the control-socket-peer-cache branch 5 times, most recently from 17b5fe2 to d9be6b8 Compare October 26, 2026 16:44
Base automatically changed from drop-span-new-events to main October 26, 2026 16:45
@tmakatos
tmakatos force-pushed the control-socket-peer-cache branch from d9be6b8 to bab5df6 Compare October 26, 2026 16:45
Discovery connected to every VM's control socket on every pass, only to
read the peer PID, even for VMs already tracked. Backends that listen
with a minimal backlog reject such a connection with EAGAIN while an
earlier one is still pending, which logged a warning and dropped the VM
from that discovery pass.

Cache the peer PID per socket path and reconnect only when the socket
file is replaced or the cached process is gone. EAGAIN on a socket that
isn't cached yet is transient, so log it at debug and retry on the next
pass.

AI-generated: everything
Signed-off-by: Thanos Makatos <thanos.makatos@nutanix.com>
Signed-off-by: Thanos Makatos <thanos.makatos@nutanix.com>
@tmakatos
tmakatos force-pushed the control-socket-peer-cache branch from bab5df6 to ef9721c Compare October 26, 2026 17:02
Signed-off-by: Thanos Makatos <thanos.makatos@nutanix.com>
@tmakatos
tmakatos removed this pull request from stack #149 October 26, 2026 18:18
@tmakatos
tmakatos added this pull request to stack #167 October 26, 2026 18:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants