feat(notifications): add a consecutive-failure threshold for outbound.down alerts (#5968)

Problem: a flaky outbound produces hundreds of false-positive "outbound down"
notifications overnight — each fires the moment xray's observatory reports a
single failed probe, and the next successful probe fires an "up".

applyObservatory forwarded every raw alive:true->false transition straight to
EventOutboundDown; xray's observatory has effectively no hysteresis, and nothing
on the panel side debounced it (the email/Telegram subscribers are pure
formatters).

Fix: debounce per outbound. outbound.down now fires only after
outboundDownThreshold consecutive FAILED probes (new setting, default 3);
outbound.up fires immediately on the first successful probe and only when a down
was actually notified. The threshold gates the event itself, so email and
Telegram share one knob (exposed next to the outbound.down toggle).

The streak counts genuinely new probes (last_try_time advancing), not sampler
polls — the sampler runs every 2s but the observatory re-probes per its
probeInterval, so counting samples would trip the threshold instantly.
outboundDownThreshold=1 reproduces the legacy notify-on-first-failure behaviour.

Tuning the observatory's probe interval/timeout is not a workaround: those
probes also drive the load balancer's outbound selection, so loosening them to
quiet notifications would slow real failover away from a genuinely dead
outbound. Notifications don't need observatory-grade latency, so the tolerance
belongs at the notification layer, leaving the observatory (and balancer)
untouched.

Adds TestApplyObservatoryDebounce covering the threshold, probe-vs-sample
counting, single-blip suppression and the legacy path.

Co-authored-by: Yuriy Khachaturian <y.khachaturian@souzmult.ru>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Yuri Khachaturyan
2026-07-21 16:59:07 +03:00
committed by GitHub
parent 9e117bbdd3
commit 2b1308ca29
13 changed files with 247 additions and 23 deletions
+12
View File
@@ -109,6 +109,11 @@
"ldapVlessField": {
"type": "string"
},
"outboundDownThreshold": {
"maximum": 100,
"minimum": 1,
"type": "integer"
},
"pageSize": {
"maximum": 1000,
"minimum": 0,
@@ -387,6 +392,7 @@
"ldapUserAttr",
"ldapUserFilter",
"ldapVlessField",
"outboundDownThreshold",
"pageSize",
"panelOutbound",
"remarkTemplate",
@@ -570,6 +576,11 @@
"ldapVlessField": {
"type": "string"
},
"outboundDownThreshold": {
"maximum": 100,
"minimum": 1,
"type": "integer"
},
"pageSize": {
"maximum": 1000,
"minimum": 0,
@@ -855,6 +866,7 @@
"ldapUserAttr",
"ldapUserFilter",
"ldapVlessField",
"outboundDownThreshold",
"pageSize",
"panelOutbound",
"remarkTemplate",