Skip to content

fix(fsm): free a wedged write on teardown - #69

Open
gregwight wants to merge 1 commit into
jwhited:mainfrom
gregwight:free-wedged-write-on-teardown
Open

gregwight wants to merge 1 commit into
jwhited:mainfrom
gregwight:free-wedged-write-on-teardown

Conversation

@gregwight

Copy link
Copy Markdown

A peer that stops reading can make peer.stop() - and therefore Server.DeletePeer, Server.Close and every other exported Server method - block forever.

What happens

corebgp performs all of its own writes on the peer's FSM goroutine:

  • sendOpenAndSetHoldTimer - fsm.go:294
  • sendNotification - fsm.go:517
  • sendKeepAlive - fsm.go:527

A plugin's UPDATE write (updateMessageWriter.WriteUpdate, fsm.go:795) runs on whatever goroutine the plugin chooses, but shares the connection. There is no write deadline anywhere in the package, so a plugin write blocks for as long as the peer declines to read. Go holds the per-connection fd write lock across the whole of Write, so the FSM's next write queues behind the plugin's and never returns.

established()'s select (fsm.go:843-857) has three non-reader cases - closeCh, holdTimer.C and keepAliveTimer.C - and two of them write. So with a plugin write wedged:

  • the keepalive timer fires first (its interval is holdTime/3), the FSM blocks in sendKeepAlive, and closeCh is never evaluated again.
  • peer.stop() closes closeCh and then waits on <-p.doneCh, which never closes.
  • DeletePeer holds s.mu across p.stop() (server.go:230), and every exported Server method takes that mutex, so one wedged peer stops the whole server being managed.

Nothing times out. The same applies to openSent and openConfirm, which write from the FSM goroutine too (sendKeepAlive at fsm.go:635 and :709, sendNotification at six sites across the two states), so a peer that accepts a TCP connection and never reads a byte wedges the handshake identically.

Why a write deadline cannot fix it

internal/poll.FD.Write takes fd.writeLock() first and only then calls fd.pd.prepareWrite, which is where a deadline is consulted (internal/poll/fd_unix.go, lines 366 and 370 in go1.27). fdMutex.rwlock blocks in runtime_Semacquire and returns false only once mutexClosed is set (internal/poll/fd_mutex.go:125-165). A goroutine queued on the write lock therefore never reaches its own deadline: SetWriteDeadline bounds a write that holds the lock, not one waiting
for it. Closing the connection is the only thing that frees a queued writer - fdMutex.increfAndClose sets the closed bit and semreleases every waiter.

This was established by reading the Go source, not by experiment.

Prior art

GoBGP has the same unbounded writes and does not rely on deadlines to escape them. At v4.9.0, pkg/server/fsm.go:

  • UPDATEs and KEEPALIVEs go out on a dedicated sendMessageloop goroutine (:1776).
  • established() clears the write deadline for established data flow (:2001-2002).

What saves its administrative teardown is that the goroutine closing the connection is never the one blocked in a write. established()'s select stays free and closes the conn directly in the ctx.Done() default branch (:2065), or via sendNotification, which calls conn.Close() unconditionally after a 1s-deadline write attempt (:825-830). The reasonCh branch closes first and says why (:2079-2084): "tx goroutine might take long until it exits because it waits for
ctx.Done() or keepalive timer. So let kill it now."

The transferable part is the mechanism, not the deadline: something that is never blocked in a write must be able to close the connection. corebgp has nothing like that today, because its state machine and its control writes share one goroutine.

Line references above were verified by reading pkg/server/fsm.go at tag v4.9.0.

What this patch does

fsm.run() starts a watchdog goroutine for the FSM's lifetime. It waits for closeCh, then closes the connection if doneCh has not closed within closeGracePeriod:

go func() {
	select {
	case <-f.doneCh:
		return
	case <-f.closeCh:
	}
	select {
	case <-f.doneCh:
	case <-time.After(closeGracePeriod):
		if c := f.currentConn(); c != nil {
			c.Close()
		}
	}
}()

It is in run() rather than established() so that one mechanism covers openSent and openConfirm as well. doneCh already exists and is closed by run()'s defer, so no new channel was needed.

Once the connection is closed the wedged plugin write returns ErrNetClosing, the fd write lock is released, the FSM's queued write fails, established() returns, and the existing cleanupConnAndReader() closes the conn (a second Close is harmless), fires OnClose, and doneCh closes.

The grace period preserves current behaviour: a healthy peer still gets its CEASE from the FSM. Closing immediately on closeCh would drop the CEASE for every peer, healthy ones included. closeGracePeriod is an unexported constant of 1 second, matching GoBGP's notification deadline. It is deliberately not a PeerOption - it is an internal teardown detail rather than a protocol timer. peer_options.go already has the pattern (holdTime, idleHoldTime, connectRetryTime,
port, each with an exported DefaultX), so exposing it as WithCloseGracePeriod is trivial if you would rather it were configurable.

The weakened invariant

f.conn is currently owned by the FSM goroutine outright. The watchdog reads it, so this patch adds a mutex: the three post-construction assignments (connect() twice, and the nil in cleanupConnAndReader) go through setConn, and the watchdog reads through currentConn. The FSM's own reads stay direct, because they are on the same goroutine that performs every assignment.

That is the one invariant the change weakens, and it is unavoidable: "only the FSM touches the conn" is exactly what stops teardown working. The reader goroutine's existing unsynchronised reads of f.conn are unaffected - cleanupConnAndReader still joins the reader before the setConn(nil) in its defer.

Testing

fsm_teardown_test.go is a new in-package test file with no build tag, so it runs under the existing go test -race job. It uses net.Pipe() as the peer connection: an unbuffered pipe blocks a write the moment the remote end stops reading, it serialises writes behind a mutex held for the duration of a write (net/pipe.go, wrMu), and closing either end frees a write blocked on it. Those are the properties of a real connection's per-fd write lock that this bug depends on. net.Pipe does not model the fd lock's interaction with deadlines, which does not matter here because the fix sets none.

The harness runs a Server with one passive peer over a net.Pipe, behind a listener that yields the connection once, and drives a minimal OPEN/KEEPALIVE handshake from the peer end using the package's own encoders.

  • TestDeletePeerWithWedgedWrite - peer stops reading, a plugin UPDATE wedges, DeletePeer returns, the wedged write returns an error and OnClose fires.
  • TestDeletePeerSendsCeaseToReadingPeer - a peer that is still reading receives the CEASE. This guards the grace period against being dropped in favour of an immediate close.
  • TestServerCloseWithWedgedWrite - the same teardown through Server.Close().

Each test was checked against the unfixed code. With the fsm.go change reverted, TestDeletePeerWithWedgedWrite and TestServerCloseWithWedgedWrite fail on their timeout assertions ("DeletePeer did not return within 10s", "Server.Close did not return within 10s") rather than hanging the suite; TestDeletePeerSendsCeaseToReadingPeer still passes, as expected for a guard rather than a detector. With the grace period removed so the watchdog closes immediately on closeCh, TestDeletePeerSendsCeaseToReadingPeer fails with EOF in place of the CEASE - that one is a race by nature and failed in 2 of 3 runs.

goimports --format-only -d ., go vet ./..., staticcheck ./... and go test -v -race ./... are all clean. The local staticcheck is 2026.2.1 rather than the 2023.1.6 CI pins; it reported nothing either way.

Follow-ups, not in this PR

  1. A wedged session does not self-heal. With the FSM blocked in sendKeepAlive, holdTimer.C is never evaluated, so the session stays established-but-wedged until something calls DeletePeer. The structural fix is GoBGP's - writes off the state-machine goroutine - which is a far larger change and your call.
  2. DeletePeer holds s.mu across p.stop() (server.go:230-242), so one wedged peer blocks every exported Server method, and after this change it does so for the grace period on every wedged teardown. Deleting from the map and stopping outside the lock would confine the wait, but it lets AddPeer interleave with a peer's teardown - a change to Server's concurrency semantics that deserves its own review.

🤖 Generated with Claude Code

corebgp performs its own writes on the peer's FSM goroutine, so a plugin
write that blocks because the remote peer has stopped reading also stops
the FSM observing closeCh: the FSM's next write queues behind the plugin's
on the per-connection write lock and never returns. peer.stop() then waits
on the FSM's doneCh forever, and because DeletePeer holds the Server mutex
across it, one wedged peer blocks every exported Server method.

A write deadline cannot fix this. internal/poll.FD.Write takes the fd write
lock before consulting the deadline, so a goroutine queued on that lock
never reaches its own deadline. Closing the connection is the only thing
that frees a queued writer.

Each FSM now runs a watchdog that closes the connection if teardown has
been signalled and the FSM has not finished within closeGracePeriod. The
grace period leaves a healthy peer's CEASE to the FSM. conn assignment
moves behind a mutex so the watchdog can read it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant