perf(clients): push a bulk client change to every node at once

63b46cd6 made a multi-inbound client create apply its inbounds concurrently,
and d34ec97f did the same for the single-client update, delete and detach. The
bulk operations were never converted, so they still walked their inbounds in a
plain sequential loop with a node RPC in each iteration — and those are what the
panel actually calls for a multi-select delete or an enable/disable, which is
why editing and deleting still felt slow on a master with several nodes.

Measured with a node runtime injecting 100ms per RPC, one client per node:

  nodes=1  update=101ms  bulkSetEnable=101ms  bulkAdjust=101ms  bulkDelete=101ms
  nodes=3  update=102ms  bulkSetEnable=302ms  bulkAdjust=304ms  bulkDelete=303ms
  nodes=5  update=203ms  bulkSetEnable=504ms  bulkAdjust=504ms  bulkDelete=504ms

after, all of them track the single-client ops:

  nodes=3  bulkSetEnable=103ms  bulkAdjust=101ms  bulkDelete=101ms
  nodes=5  bulkSetEnable=202ms  bulkAdjust=202ms  bulkDelete=202ms

Generalize the fanout into fanoutInboundResults over an arbitrary per-inbound
result type and route six loops through it: BulkDelete, BulkSetEnable,
BulkAdjust, BulkDetach, BulkAttach, BulkCreate, plus applyClientFieldByEmail —
the field edit behind the Telegram bot's enable/limit/expiry buttons and the
LDAP job. Each keeps its preparation sequential and overlaps only the node
pushes, inheriting the same concurrency cap and per-inbound panic recovery.

Two ordering details the sequential loops got for free and the fanout must do
itself: the three loops that ranged a map now walk sortedInboundIds, so which
inbound wins a per-email skip reason is the lowest id instead of whatever the
map yielded; and BulkAttach de-duplicates a repeated inbound id up front,
because the second pass used to see the client the first pass had just added.

The allocating paths stay serial when a tunnel inbound is involved. WireGuard
and AmneziaWG pick a free peer address by reading every inbound's used-set
before they write, so two overlapping allocations hand out the same address and
the in-transaction re-check refuses the loser — a bulk create of two clients
onto two wg inbounds returned created=1. addFanoutLimit drops those batches back
to one at a time; every other protocol keeps the full cap.

Eight tests: seven barrier tests that a sequential caller cannot satisfy (peak
pushes in flight is 1 without the change, 4 with it), and one that pins the
tunnel allocation.

internal/web/service/client_crud.go

@@ -7,6 +7,7 @@ import (
 	"fmt"
 	"net/netip"
 	"runtime/debug"
+	"slices"
 	"strings"
 	"sync"
 	"sync/atomic"
@@ -297,6 +298,60 @@ func (s *ClientService) fanoutInboundClientAdds(inboundSvc *InboundService, adds
 	return fanoutInboundApplies(applies)
 }
 
+// fanoutInboundResults runs one job per inbound with the node pushes
+// overlapping, so a bulk op costs one RPC round-trip instead of one per node.
+// limit is the caller's own cap: an op that allocates tunnel addresses passes 1,
+// because allocation reads a cross-inbound used-set before it writes.
+func fanoutInboundResults[T any](inboundIds []int, limit int, run func(i int) T) ([]T, []error) {
+	if limit < 1 {
+		limit = 1
+	}
+	out := make([]T, len(inboundIds))
+	errs := make([]error, len(inboundIds))
+	sem := make(chan struct{}, limit)
+	var wg sync.WaitGroup
+	for i := range inboundIds {
+		wg.Add(1)
+		sem <- struct{}{}
+		go func() {
+			defer wg.Done()
+			defer func() { <-sem }()
+			// Off the request goroutine gin's Recovery no longer covers this,
+			// so an unrecovered panic here would take the whole panel down.
+			defer func() {
+				if r := recover(); r != nil {
+					errs[i] = fmt.Errorf("inbound %d: panic: %v", inboundIds[i], r)
+					logger.Errorf("panic applying bulk client change to inbound %d: %v\n%s", inboundIds[i], r, debug.Stack())
+				}
+			}()
+			out[i] = run(i)
+		}()
+	}
+	wg.Wait()
+	return out, errs
+}
+
+// addFanoutLimit serializes an add that touches a tunnel inbound. WireGuard and
+// AmneziaWG pick a free peer address by reading every inbound's used-set first,
+// so two overlapping allocations hand out the same one and the second is refused.
+func addFanoutLimit(anyTunnel bool) int {
+	if anyTunnel {
+		return 1
+	}
+	return inboundFanoutConcurrency
+}
+
+// sortedInboundIds gives the fanout a stable order, so which inbound wins a
+// per-email report no longer depends on Go's map iteration order.
+func sortedInboundIds[V any](byInbound map[int]V) []int {
+	ids := make([]int, 0, len(byInbound))
+	for id := range byInbound {
+		ids = append(ids, id)
+	}
+	slices.Sort(ids)
+	return ids
+}
+
 // markInboundNodesDirty makes a half-applied client edit unobservable to a node
 // snapshot merge, which skips a node whose config is already flagged dirty.
 func markInboundNodesDirty(inboundIds []int) error {