Investigate possible stale SO_ERROR during uring nonblocking TCP connect

[russellromney]

Summary

While investigating psycopg/libpq nonblocking TCP connects under Quark, I observed a second possible uring TCP connect-state issue separate from the deterministic select(2) exception-fdset bug fixed in #1373.

This issue tracks the less-proven behavior so it can be isolated before proposing a fix.

Observed Behavior

In one minimal Quark run against a real host PostgreSQL server, the client sequence:

1. create TCP socket
2. set nonblocking
3. connect() / connect_ex() returns EINPROGRESS
4. immediately call getsockopt(SOL_SOCKET, SO_ERROR)

returned stale EINPROGRESS from Quark where the runc control returned 0.

That matters because libpq/psycopg checks SO_ERROR during its nonblocking connect flow.

Current Reproduction Status

The exact stale immediate SO_ERROR=EINPROGRESS symptom has not been reproduced deterministically yet.

What is reproducible:

• The select(2) exception-fdset bug is deterministic and tracked separately in fix select exception fdset clearing #1373.
• With an unpatched qkernel and postgres:18-alpine, a minimal loop showed:

summary bad_immediate=0/100 bad_select_except=100/100
psycopg_iter=7 error=ConnectionTimeout: connection timeout expired
psycopg_failures=1/20

After the select fix was restored:

summary bad_immediate=0/100 bad_select_except=0/100
psycopg_failures=0/20

So the remaining SO_ERROR concern may be timing/path dependent, or it may have been downstream of the select readiness issue in some runs.

Suggested Next Steps

• Build a focused repro that exercises Quark's uring TCPConnecting state directly.
• Stress the timing between nonblocking connect(), async connect completion, immediate SO_ERROR, and readiness registration.
• Compare against runc and Linux behavior for immediate SO_ERROR after loopback connect returns EINPROGRESS.
• Only propose a getsockopt(SO_ERROR) / PostConnect() change after this is reproduced independently from the select fdset bug.

[russellromney]

Update from a simpler harness: the stale SO_ERROR behavior can be reproduced without Postgres, psycopg, or any app code.

I wrote a standalone C loopback harness:

• create a TCP listener on 127.0.0.1:0
• fork an accept loop
• client repeatedly creates a nonblocking TCP socket
• connect()
• immediate getsockopt(SO_ERROR)
• select(readfds, writefds, exceptfds)
• second getsockopt(SO_ERROR)

Using qkernels built from the same Quark product base as the installed qvisor:

product-base unpatched qkernel

iter=1 connect_rc=1 immediate=115 read=0 write=0 except=1 after=0
iter=2 connect_rc=1 immediate=115 read=0 write=0 except=1 after=0
iter=3 connect_rc=1 immediate=115 read=0 write=0 except=1 after=0
iter=4 connect_rc=1 immediate=115 read=0 write=0 except=1 after=0
summary bad_immediate=50/50 bad_except=50/50 bad_missing_write=50/50 bad_after=0/50 child_status=0

product-base qkernel with only the select fdset fix

iter=1 connect_rc=1 immediate=115 read=0 write=0 except=1 after=0
iter=2 connect_rc=1 immediate=115 read=0 write=0 except=1 after=0
iter=3 connect_rc=1 immediate=115 read=0 write=0 except=1 after=0
iter=4 connect_rc=1 immediate=115 read=0 write=0 except=1 after=0
summary bad_immediate=5/5 bad_except=5/5 bad_missing_write=5/5 bad_after=0/5 child_status=0

So the stale immediate SO_ERROR=EINPROGRESS path is real and independent of PostgreSQL/libpq. It also persists after the select fdset fix.

I also tried a product-base qkernel with the earlier getsockopt(SO_ERROR) proof patch. That cleared immediate SO_ERROR, but readiness was still wrong in this compatible-variant matrix:

iter=1 connect_rc=2 immediate=0 read=1 write=0 except=1 after=0
summary bad_immediate=0/5 bad_except=5/5 bad_missing_write=5/5 bad_after=0/5 child_status=0

That suggests the earlier proof patch is not a clean upstream fix. It changes the cached error state, but does not correctly drive the full TCPConnecting -> connected readiness/state transition in all cases.

Next investigation target: instrument or trace these Quark paths around one nonblocking loopback connect:

• UringSocketOperations::Connect
• UringSocketOperations::GetSockOpt(SO_ERROR)
• UringSocketOperations::Readiness
• EventRegister / UpdateFD
• AsyncConnect::Process
• PostConnect()

The likely correct fix should probably centralize “complete pending connect from host state” instead of doing a one-off PostConnect() inside getsockopt(SO_ERROR).