fix(node): fan out operations that call every node

An operation that calls every node has to finish inside the panel's 30s
write timeout. Reset all traffic, UpdatePanels and bulk inbound delete
walked the nodes one at a time, up to 10s per hanging node, so 15 hanging
nodes out of 150 kept each request running for 2m41s while the browser
had already been told it failed.

All three now fan out through fanoutInboundResults, bounded by
nodeFanoutConcurrency (32, the heartbeat's bound), and UpdatePanels keeps
its results in request order. Bulk delete still removes the rows one at a
time, since each rewrites shared routing references, and only fans out the
node pushes that delInbound now hands back.
This commit is contained in:
Sanaei
2026-09-15 20:07:18 +02:00
parent a84bbeab2e
commit eb11e8c85a
5 changed files with 210 additions and 15 deletions
+4
View File
@@ -34,6 +34,10 @@ const nodeBulkPushThreshold = 32
// committed and the node flagged dirty, so a slow node defers to the reconcile.
const nodeClientPushTimeout = 4 * time.Second
// nodeFanoutConcurrency bounds an operation that calls every node, as the heartbeat
// does: one at a time, a few hanging nodes outlast the request's write timeout.
const nodeFanoutConcurrency = 32
func nodePushContext() (context.Context, context.CancelFunc) {
return context.WithTimeout(context.Background(), nodeClientPushTimeout)
}