Repository navigation
Conversation
corebgp performs its own writes on the peer's FSM goroutine, so a plugin write that blocks because the remote peer has stopped reading also stops the FSM observing closeCh: the FSM's next write queues behind the plugin's on the per-connection write lock and never returns. peer.stop() then waits on the FSM's doneCh forever, and because DeletePeer holds the Server mutex across it, one wedged peer blocks every exported Server method. A write deadline cannot fix this. internal/poll.FD.Write takes the fd write lock before consulting the deadline, so a goroutine queued on that lock never reaches its own deadline. Closing the connection is the only thing that frees a queued writer. Each FSM now runs a watchdog that closes the connection if teardown has been signalled and the FSM has not finished within closeGracePeriod. The grace period leaves a healthy peer's CEASE to the FSM. conn assignment moves behind a mutex so the watchdog can read it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A peer that stops reading can make
peer.stop()- and thereforeServer.DeletePeer,Server.Closeand every other exportedServermethod - block forever.What happens
corebgp performs all of its own writes on the peer's FSM goroutine:
sendOpenAndSetHoldTimer-fsm.go:294sendNotification-fsm.go:517sendKeepAlive-fsm.go:527A plugin's UPDATE write (
updateMessageWriter.WriteUpdate,fsm.go:795) runs on whatever goroutine the plugin chooses, but shares the connection. There is no write deadline anywhere in the package, so a plugin write blocks for as long as the peer declines to read. Go holds the per-connection fd write lock across the whole ofWrite, so the FSM's next write queues behind the plugin's and never returns.established()'s select (fsm.go:843-857) has three non-reader cases -closeCh,holdTimer.CandkeepAliveTimer.C- and two of them write. So with a plugin write wedged:sendKeepAlive, andcloseChis never evaluated again.peer.stop()closescloseChand then waits on<-p.doneCh, which never closes.DeletePeerholdss.muacrossp.stop()(server.go:230), and every exportedServermethod takes that mutex, so one wedged peer stops the whole server being managed.Nothing times out. The same applies to
openSentandopenConfirm, which write from the FSM goroutine too (sendKeepAliveatfsm.go:635and:709,sendNotificationat six sites across the two states), so a peer that accepts a TCP connection and never reads a byte wedges the handshake identically.Why a write deadline cannot fix it
internal/poll.FD.Writetakesfd.writeLock()first and only then callsfd.pd.prepareWrite, which is where a deadline is consulted (internal/poll/fd_unix.go, lines 366 and 370 in go1.27).fdMutex.rwlockblocks inruntime_Semacquireand returns false only oncemutexClosedis set (internal/poll/fd_mutex.go:125-165). A goroutine queued on the write lock therefore never reaches its own deadline:SetWriteDeadlinebounds a write that holds the lock, not one waitingfor it. Closing the connection is the only thing that frees a queued writer -
fdMutex.increfAndClosesets the closed bit and semreleases every waiter.This was established by reading the Go source, not by experiment.
Prior art
GoBGP has the same unbounded writes and does not rely on deadlines to escape them. At v4.9.0,
pkg/server/fsm.go:sendMessageloopgoroutine (:1776).established()clears the write deadline for established data flow (:2001-2002).What saves its administrative teardown is that the goroutine closing the connection is never the one blocked in a write.
established()'s select stays free and closes the conn directly in thectx.Done()default branch (:2065), or viasendNotification, which callsconn.Close()unconditionally after a 1s-deadline write attempt (:825-830). ThereasonChbranch closes first and says why (:2079-2084): "tx goroutine might take long until it exits because it waits forctx.Done() or keepalive timer. So let kill it now."
The transferable part is the mechanism, not the deadline: something that is never blocked in a write must be able to close the connection. corebgp has nothing like that today, because its state machine and its control writes share one goroutine.
Line references above were verified by reading
pkg/server/fsm.goat tag v4.9.0.What this patch does
fsm.run()starts a watchdog goroutine for the FSM's lifetime. It waits forcloseCh, then closes the connection ifdoneChhas not closed withincloseGracePeriod:It is in
run()rather thanestablished()so that one mechanism coversopenSentandopenConfirmas well.doneChalready exists and is closed byrun()'s defer, so no new channel was needed.Once the connection is closed the wedged plugin write returns
ErrNetClosing, the fd write lock is released, the FSM's queued write fails,established()returns, and the existingcleanupConnAndReader()closes the conn (a second Close is harmless), firesOnClose, anddoneChcloses.The grace period preserves current behaviour: a healthy peer still gets its CEASE from the FSM. Closing immediately on
closeChwould drop the CEASE for every peer, healthy ones included.closeGracePeriodis an unexported constant of 1 second, matching GoBGP's notification deadline. It is deliberately not aPeerOption- it is an internal teardown detail rather than a protocol timer.peer_options.goalready has the pattern (holdTime,idleHoldTime,connectRetryTime,port, each with an exportedDefaultX), so exposing it asWithCloseGracePeriodis trivial if you would rather it were configurable.The weakened invariant
f.connis currently owned by the FSM goroutine outright. The watchdog reads it, so this patch adds a mutex: the three post-construction assignments (connect()twice, and thenilincleanupConnAndReader) go throughsetConn, and the watchdog reads throughcurrentConn. The FSM's own reads stay direct, because they are on the same goroutine that performs every assignment.That is the one invariant the change weakens, and it is unavoidable: "only the FSM touches the conn" is exactly what stops teardown working. The reader goroutine's existing unsynchronised reads of
f.connare unaffected -cleanupConnAndReaderstill joins the reader before thesetConn(nil)in its defer.Testing
fsm_teardown_test.gois a new in-package test file with no build tag, so it runs under the existinggo test -racejob. It usesnet.Pipe()as the peer connection: an unbuffered pipe blocks a write the moment the remote end stops reading, it serialises writes behind a mutex held for the duration of a write (net/pipe.go,wrMu), and closing either end frees a write blocked on it. Those are the properties of a real connection's per-fd write lock that this bug depends on.net.Pipedoes not model the fd lock's interaction with deadlines, which does not matter here because the fix sets none.The harness runs a
Serverwith one passive peer over anet.Pipe, behind a listener that yields the connection once, and drives a minimal OPEN/KEEPALIVE handshake from the peer end using the package's own encoders.TestDeletePeerWithWedgedWrite- peer stops reading, a plugin UPDATE wedges,DeletePeerreturns, the wedged write returns an error andOnClosefires.TestDeletePeerSendsCeaseToReadingPeer- a peer that is still reading receives the CEASE. This guards the grace period against being dropped in favour of an immediate close.TestServerCloseWithWedgedWrite- the same teardown throughServer.Close().Each test was checked against the unfixed code. With the
fsm.gochange reverted,TestDeletePeerWithWedgedWriteandTestServerCloseWithWedgedWritefail on their timeout assertions ("DeletePeer did not return within 10s", "Server.Close did not return within 10s") rather than hanging the suite;TestDeletePeerSendsCeaseToReadingPeerstill passes, as expected for a guard rather than a detector. With the grace period removed so the watchdog closes immediately oncloseCh,TestDeletePeerSendsCeaseToReadingPeerfails with EOF in place of the CEASE - that one is a race by nature and failed in 2 of 3 runs.goimports --format-only -d .,go vet ./...,staticcheck ./...andgo test -v -race ./...are all clean. The local staticcheck is 2026.2.1 rather than the 2023.1.6 CI pins; it reported nothing either way.Follow-ups, not in this PR
sendKeepAlive,holdTimer.Cis never evaluated, so the session stays established-but-wedged until something callsDeletePeer. The structural fix is GoBGP's - writes off the state-machine goroutine - which is a far larger change and your call.DeletePeerholdss.muacrossp.stop()(server.go:230-242), so one wedged peer blocks every exportedServermethod, and after this change it does so for the grace period on every wedged teardown. Deleting from the map and stopping outside the lock would confine the wait, but it letsAddPeerinterleave with a peer's teardown - a change toServer's concurrency semantics that deserves its own review.🤖 Generated with Claude Code