Editing, deleting or detaching a client on a master with several nodes took
one node round-trip per node, added end to end. Create and Attach already
fanned their per-inbound applies out through fanoutInboundClientAdds, but
Update, Delete, Detach and DeleteByEmail's record-less fallback still walked
their inbounds in a plain sequential loop, and each iteration blocks on a
node RPC (10s timeout, more when a node is slow or has just gone unreachable
and the heartbeat has not marked it offline yet).
Measured with a node runtime injecting 100ms per RPC, before:
nodes=1 create=101ms update=101ms delete=101ms
nodes=3 create=102ms update=303ms delete=302ms
nodes=5 create=202ms update=504ms delete=504ms
after, all three track create:
nodes=3 create=102ms update=102ms delete=101ms
nodes=5 create=203ms update=203ms delete=203ms
Generalize the existing helper into fanoutInboundApplies over an inboundApply
list and route the four remaining loops through it, so they inherit the same
concurrency cap, per-inbound panic recovery and joined errors. Each caller
still builds its payloads sequentially first: fillProtocolDefaults mints the
shared credentials on the first inbound and every later one reuses them, so
that order has to stay deterministic. Only the applies overlap; their DB work
still serializes through the single traffic writer, and the per-inbound
mutation lock is unchanged, which is exactly what Create has relied on.
Behaviour change: one failing inbound no longer aborts the remaining ones,
matching what Create already does. The error still names each failed inbound
and the record-level writes are still skipped when any inbound failed.
The snapshot merge on the same serialized writer was measured as a second
suspect and cleared: ~43ms per node at 500 clients, an order of magnitude
below the RPC serialization.
Sanaei