Skip to content

Stop writeLoop flood after TCP dead-socket write errors#26

Open
Vishesh-Gupta wants to merge 1 commit into
merge-latest-quickfixfrom
fix/write-loop-dead-socket
Open

Stop writeLoop flood after TCP dead-socket write errors#26
Vishesh-Gupta wants to merge 1 commit into
merge-latest-quickfixfrom
fix/write-loop-dead-socket

Conversation

@Vishesh-Gupta

Copy link
Copy Markdown

Summary

  • writeLoop previously logged a write error and kept writing every remaining outbound FIX message to a dead TCP socket. Under load this produced thousands of write: broken pipe events per disconnect (seen on fixgateway-btgt acceptor in integration-v2).
  • On the first write failure: log once, close the connection so readLoop triggers a clean session disconnect, and drain messageOut without further writes (avoids blocking the session send path / deadlock).
  • Adds a unit test that floods 50 messages after a simulated broken pipe and asserts exactly one Write and one log event.

Notes

  • Branched from upstream v0.9.6 (module path github.com/quickfixgo/quickfix) so fixgateway can replace against this commit without a raw vendor/ edit.
  • Base is merge-latest-quickfix (upstream-synced) rather than master (old github.com/alpacahq/quickfix module path). Reviewers may prefer rebasing onto merge-latest-quickfix tip; the behavioral patch is confined to connection.go.

Test plan

  • go test -run TestWriteLoopStopsWritingAfterBrokenPipe -count=1
  • Deploy via fixgateway replace and reproduce a client disconnect under load
  • Confirm acceptor logs a single broken-pipe / write error event instead of a multi-thousand-line flood

writeLoop previously logged and kept writing after the first broken pipe,
producing thousands of events per disconnect under load. On write failure,
log once, close the connection to trigger disconnect, and drain messageOut
without further writes so the session send path does not deadlock.
@greptile-apps

greptile-apps Bot commented Jul 16, 2026

Copy link
Copy Markdown

Greptile Summary

This PR stops repeated writes after a TCP write failure. The main changes are:

  • Logs only the first socket write error.
  • Closes the connection to trigger session disconnection.
  • Drains queued outbound messages without writing them.
  • Adds coverage for a burst of messages after a broken pipe.

Confidence Score: 5/5

This looks safe to merge.

No blocking issues found in the changed code. Production callers pass closable network connections, so the write failure wakes the read loop and completes the existing disconnect flow. The new test covers the intended error suppression and draining behavior.

T-Rex T-Rex Logs

What T-Rex did

  • Ran the non-race test for TestWriteLoopStopsWritingAfterBrokenPipe; it completed with exit code 0.
  • Ran the race-enabled test for TestWriteLoopStopsWritingAfterBrokenPipe; it completed with exit code 0 and reported no race.

View all artifacts

T-Rex Ran code and verified through T-Rex

Important Files Changed

Filename Overview
connection.go Stops writes after the first failure, closes the network connection, and drains outbound messages until shutdown.
connection_test.go Adds focused coverage for single-write, single-log, connection-close, and channel-drain behavior.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant Session
participant WriteLoop
participant Socket
participant ReadLoop

Session->>WriteLoop: Send outbound message
WriteLoop->>Socket: Write message
Socket-->>WriteLoop: Write error
WriteLoop->>WriteLoop: Log once and mark failed
WriteLoop->>Socket: Close connection
Socket-->>ReadLoop: Read error or EOF
ReadLoop-->>Session: Close inbound channel
Session->>Session: Enter disconnected state
Session-->>WriteLoop: Close outbound channel
loop Until outbound channel closes
    WriteLoop->>WriteLoop: Drain without writing
end
WriteLoop-->>WriteLoop: Exit
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant Session
participant WriteLoop
participant Socket
participant ReadLoop

Session->>WriteLoop: Send outbound message
WriteLoop->>Socket: Write message
Socket-->>WriteLoop: Write error
WriteLoop->>WriteLoop: Log once and mark failed
WriteLoop->>Socket: Close connection
Socket-->>ReadLoop: Read error or EOF
ReadLoop-->>Session: Close inbound channel
Session->>Session: Enter disconnected state
Session-->>WriteLoop: Close outbound channel
loop Until outbound channel closes
    WriteLoop->>WriteLoop: Drain without writing
end
WriteLoop-->>WriteLoop: Exit
Loading

Reviews (1): Last reviewed commit: "Stop writeLoop flood after TCP dead-sock..." | Re-trigger Greptile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant