Files
3x-ui/internal/web/service/client_locks.go
T
mrchatam 6f7a305239 fix(node): stop stale expiry sync from undoing client extensions (#6228) (#6231)
* fix(node): stop stale expiry sync from undoing client extensions (#6228)

After an expired client is extended on the master, a lagging node could
overwrite client_traffics with an older absolute expiry and latch
enable=false. Reject older absolute expiries on merge, ignore
expiry-stale disables when the master is not over quota, lift stale
lifecycle fields out of adopted settings, stamp reconcile fingerprints
from the pre-lift node blob, and mark the node dirty so the next tick
re-pushes.

* fix(node): lockstep client_traffics expiry/enable SQL with review fixes (#6228)

Make expiry merge keep any master absolute (node only activates when master
is unset/duration). Include this tick's up/down deltas in the enable
stale-disable quota check so a crossing-tick disable is not dropped.

* fix(node): add settings absolute helper for renew/lift guards (#6228)

Expose settingsClientAbsoluteExpiry so traffic merge can tell a real
node auto-renew (settings+stats later) from lagging ClientStats after a
master shorten. Trim lift godoc to the invariant.

* fix(node): authority-aware lifecycle merge for multi-node sync (#6228)

While config_dirty, accumulate traffic only — do not adopt node
expiry/enable/total/reset (and preserve dipped baselines so a false
renew cannot fire after clear). On clean ticks, master absolute expiry
wins; node auto-renew still goes through nodeClientRenewed when settings
also show the later deadline. Defer settings lifecycle lift until after
traffic deltas land, align SyncInbound via applyMasterClientLifecycle,
and avoid re-MarkNodeDirty when already dirty.

* fix(node): clear config_dirty only after the post-reconcile traffic merge (#6228)

After a successful ReconcileNode, keep the node dirty through the same
tick's SetRemoteTraffic so lagging ClientStats cannot clobber the
just-pushed master lifecycle, then ClearNodeDirty.

* test(node): cover dirty-gate, master-absolute, and renew false-positives (#6228)

Add regressions for extend/shorten while dirty, clean-sibling shorten,
settings vs lagging disable, renew recovery after dirty, renew with
matching settings, and shorten+Reset lagging stats not treated as renew.

* fix(node): address the review findings on the lifecycle merge (#6228)

The automated review on #6231 flagged a blocking regression and six smaller
issues. All of them are fixed here.

Blocking: making the master's absolute expiry always win left nodeClientRenewed
as the only channel for a node-side auto-renew, and that required a counter dip.
A client that used no traffic in the period never dips, so its renewal was
dropped, the master kept the expired deadline and disableInvalidClients removed
it with no way back (master-side autoRenewClients skips node inbounds). The node
bumps reset_count on every renewal, so that counter is now an independent
renewal signal and is persisted with the renewal so it keeps converging.

The deferred ClearNodeDirty made every reconcile-success tick merge in dirty
mode, which suppressed inbound adoption, new client_traffics rows, the orphan
sweeps and the whole SyncInbound record loop -- and left the node dirty forever
whenever SetRemoteTraffic errored. The clear goes back to where it was; a
separate justPushed flag now freezes only the client lifecycle merge for the
tick whose push just landed.

staleNodeDisable only recognised a lagging disable by an older expiry, so a
quota top-up (raise totalGB, leave the expiry alone) was re-latched to disabled
by the next lagging snapshot -- the #6228 symptom on a second axis. The
reviewer's suggestion of dropping the expiry precondition outright fails
TestNodeQuotaDisable_SameExpiryStillLatches, because the master's own counters
legitimately sit below a node's after a seeded-at-zero adoption. nodeDisableIsStale
instead compares the limits the node judged the client against with the master's
own: matching limits mean a genuine verdict that still latches (#4917), differing
limits mean the node has not seen the master's change yet. It also now measures
the master deadline against wall-clock now, so an expired master row no longer
looks "extended" merely because the node's copy is older still.

Also: the settings lift now writes enable in both directions, so a blob fetched
before a master disable cannot carry enable=true back into central settings and
on to the node; the renewal guard parses the inbound settings once per inbound
instead of once per renewing client; the adoption loop only writes settings when
they actually changed; and two comments that described mechanisms the code does
not use were corrected.

The test deadlines are now relative to the run: the merge compares against now,
so fixed timestamps would have rotted into the wrong side of it.

---------

Co-authored-by: mrchatam <287639636+mrchatam@users.noreply.github.com>
Co-authored-by: Sanaei <ho3ein.sanaei@gmail.com>
2026-08-23 23:17:56 +02:00

329 lines
7.9 KiB
Go

package service
import (
"encoding/json"
"strings"
"sync"
"time"
"github.com/mhsanaei/3x-ui/v3/internal/database/model"
"github.com/mhsanaei/3x-ui/v3/internal/logger"
"github.com/mhsanaei/3x-ui/v3/internal/xray"
"gorm.io/gorm"
)
// Short-lived tombstone of just-deleted client emails so that a node snapshot
// arriving between delete and node-side processing doesn't resurrect them.
var (
recentlyDeletedMu sync.Mutex
recentlyDeleted = map[string]time.Time{}
)
const deleteTombstoneTTL = 90 * time.Second
var (
inboundMutationLocksMu sync.Mutex
inboundMutationLocks = map[int]*sync.Mutex{}
)
func lockInbound(inboundId int) *sync.Mutex {
inboundMutationLocksMu.Lock()
m, ok := inboundMutationLocks[inboundId]
if !ok {
m = &sync.Mutex{}
inboundMutationLocks[inboundId] = m
}
inboundMutationLocksMu.Unlock()
m.Lock()
return m
}
func compactOrphans(db *gorm.DB, clients []any) []any {
if len(clients) == 0 {
return clients
}
emails := make([]string, 0, len(clients))
for _, c := range clients {
cm, ok := c.(map[string]any)
if !ok {
continue
}
if e, _ := cm["email"].(string); e != "" {
emails = append(emails, e)
}
}
if len(emails) == 0 {
return clients
}
existing := make(map[string]struct{}, len(emails))
const orphanChunk = 400
for start := 0; start < len(emails); start += orphanChunk {
end := min(start+orphanChunk, len(emails))
var found []string
if err := db.Model(&model.ClientRecord{}).Where("email IN ?", emails[start:end]).Pluck("email", &found).Error; err != nil {
logger.Warning("compactOrphans pluck:", err)
return clients
}
for _, e := range found {
existing[e] = struct{}{}
}
}
if len(existing) == len(emails) {
return clients
}
out := make([]any, 0, len(existing))
for _, c := range clients {
cm, ok := c.(map[string]any)
if !ok {
out = append(out, c)
continue
}
e, _ := cm["email"].(string)
if e == "" {
out = append(out, c)
continue
}
if _, ok := existing[e]; ok {
out = append(out, c)
}
}
return out
}
func tombstoneClientEmail(email string) {
if email == "" {
return
}
recentlyDeletedMu.Lock()
defer recentlyDeletedMu.Unlock()
recentlyDeleted[email] = time.Now()
cutoff := time.Now().Add(-deleteTombstoneTTL)
for e, ts := range recentlyDeleted {
if ts.Before(cutoff) {
delete(recentlyDeleted, e)
}
}
}
func withdrawClientTombstones(emails ...string) {
if len(emails) == 0 {
return
}
recentlyDeletedMu.Lock()
defer recentlyDeletedMu.Unlock()
for _, email := range emails {
if email != "" {
delete(recentlyDeleted, email)
}
}
}
func tombstoneClientEmails(emails []string) {
if len(emails) == 0 {
return
}
now := time.Now()
cutoff := now.Add(-deleteTombstoneTTL)
recentlyDeletedMu.Lock()
defer recentlyDeletedMu.Unlock()
for _, email := range emails {
if email != "" {
recentlyDeleted[email] = now
}
}
for e, ts := range recentlyDeleted {
if ts.Before(cutoff) {
delete(recentlyDeleted, e)
}
}
}
func isClientEmailTombstoned(email string) bool {
if email == "" {
return false
}
recentlyDeletedMu.Lock()
defer recentlyDeletedMu.Unlock()
ts, ok := recentlyDeleted[email]
if !ok {
return false
}
if time.Since(ts) > deleteTombstoneTTL {
delete(recentlyDeleted, email)
return false
}
return true
}
// dedupeSettingsClients collapses duplicate same-email client entries inside a
// settings JSON blob, keeping the first occurrence. Node snapshots produced by
// builds without the addInboundClient duplicate guard can carry duplicates
// (#5770); adopting them verbatim would copy the duplication into the central
// inbound. Returns the filtered JSON and whether anything was removed.
func dedupeSettingsClients(settings string) (string, bool) {
if settings == "" {
return settings, false
}
var parsed map[string]any
if err := json.Unmarshal([]byte(settings), &parsed); err != nil {
return settings, false
}
clients, _ := parsed["clients"].([]any)
if len(clients) < 2 {
return settings, false
}
seen := make(map[string]struct{}, len(clients))
kept := make([]any, 0, len(clients))
for _, c := range clients {
if cm, ok := c.(map[string]any); ok {
if email, _ := cm["email"].(string); email != "" {
key := strings.ToLower(email)
if _, dup := seen[key]; dup {
continue
}
seen[key] = struct{}{}
}
}
kept = append(kept, c)
}
if len(kept) == len(clients) {
return settings, false
}
parsed["clients"] = kept
b, err := json.MarshalIndent(parsed, "", " ")
if err != nil {
return settings, false
}
return string(b), true
}
// stripTombstonedClients drops just-deleted client entries from a node
// snapshot's settings JSON so adopting a stale snapshot can't re-add them to
// the central inbound while the delete tombstone is live. Returns the filtered
// JSON and whether anything was removed.
func stripTombstonedClients(settings string) (string, bool) {
if settings == "" {
return settings, false
}
var parsed map[string]any
if err := json.Unmarshal([]byte(settings), &parsed); err != nil {
return settings, false
}
clients, _ := parsed["clients"].([]any)
if len(clients) == 0 {
return settings, false
}
kept := make([]any, 0, len(clients))
for _, c := range clients {
if cm, ok := c.(map[string]any); ok {
if email, _ := cm["email"].(string); email != "" && isClientEmailTombstoned(email) {
continue
}
}
kept = append(kept, c)
}
if len(kept) == len(clients) {
return settings, false
}
parsed["clients"] = kept
b, err := json.MarshalIndent(parsed, "", " ")
if err != nil {
return settings, false
}
return string(b), true
}
// liftClientLifecycleInSettings rewrites adopted settings from master traffic
// so a lagging node blob cannot store pre-extension expiry/enable (#6228).
func liftClientLifecycleInSettings(settings string, trafficByEmail map[string]*xray.ClientTraffic) (string, bool) {
if settings == "" || len(trafficByEmail) == 0 {
return settings, false
}
var parsed map[string]any
if err := json.Unmarshal([]byte(settings), &parsed); err != nil {
return settings, false
}
clients, _ := parsed["clients"].([]any)
if len(clients) == 0 {
return settings, false
}
changed := false
for i := range clients {
cm, ok := clients[i].(map[string]any)
if !ok {
continue
}
email, _ := cm["email"].(string)
if email == "" {
continue
}
tr := trafficByEmail[email]
if tr == nil {
continue
}
nodeExpiry, hasExpiry := jsonClientInt64(cm["expiryTime"])
if !hasExpiry {
continue
}
merged := mergeActivationExpiry(tr.ExpiryTime, nodeExpiry)
if merged != nodeExpiry {
cm["expiryTime"] = merged
changed = true
}
// tr is the already-merged master row, authoritative in both directions:
// a lagging blob must not re-enable a disabled client either (#4917).
if nodeEnable, _ := cm["enable"].(bool); nodeEnable != tr.Enable {
cm["enable"] = tr.Enable
changed = true
}
clients[i] = cm
}
if !changed {
return settings, false
}
parsed["clients"] = clients
b, err := json.MarshalIndent(parsed, "", " ")
if err != nil {
return settings, false
}
return string(b), true
}
// settingsClientAbsoluteExpiries indexes the absolute (>0) expiryTime of every
// client in a settings blob. Never nil, so callers can cache it per inbound.
func settingsClientAbsoluteExpiries(settings string) map[string]int64 {
out := map[string]int64{}
if settings == "" {
return out
}
var parsed map[string]any
if err := json.Unmarshal([]byte(settings), &parsed); err != nil {
return out
}
clients, _ := parsed["clients"].([]any)
for _, c := range clients {
cm, ok := c.(map[string]any)
if !ok {
continue
}
email, _ := cm["email"].(string)
if email == "" {
continue
}
if exp, has := jsonClientInt64(cm["expiryTime"]); has && exp > 0 {
out[email] = exp
}
}
return out
}
func jsonClientInt64(v any) (int64, bool) {
// json.Unmarshal into map[string]any yields float64 for numbers.
n, ok := v.(float64)
if !ok {
return 0, false
}
return int64(n), true
}