xinyin025/3x-ui
1
0
Fork 0
mirror of https://github.com/MHSanaei/3x-ui.git synced 2026-08-09 21:00:58 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(notifications): add a consecutive-failure threshold for outbound.down alerts (#5968)

Browse Source 
Problem: a flaky outbound produces hundreds of false-positive "outbound down"
notifications overnight — each fires the moment xray's observatory reports a
single failed probe, and the next successful probe fires an "up".

applyObservatory forwarded every raw alive:true->false transition straight to
EventOutboundDown; xray's observatory has effectively no hysteresis, and nothing
on the panel side debounced it (the email/Telegram subscribers are pure
formatters).

Fix: debounce per outbound. outbound.down now fires only after
outboundDownThreshold consecutive FAILED probes (new setting, default 3);
outbound.up fires immediately on the first successful probe and only when a down
was actually notified. The threshold gates the event itself, so email and
Telegram share one knob (exposed next to the outbound.down toggle).

The streak counts genuinely new probes (last_try_time advancing), not sampler
polls — the sampler runs every 2s but the observatory re-probes per its
probeInterval, so counting samples would trip the threshold instantly.
outboundDownThreshold=1 reproduces the legacy notify-on-first-failure behaviour.

Tuning the observatory's probe interval/timeout is not a workaround: those
probes also drive the load balancer's outbound selection, so loosening them to
quiet notifications would slow real failover away from a genuinely dead
outbound. Notifications don't need observatory-grade latency, so the tolerance
belongs at the notification layer, leaving the observatory (and balancer)
untouched.

Adds TestApplyObservatoryDebounce covering the threshold, probe-vs-sample
counting, single-blip suppression and the legacy path.

Co-authored-by: Yuriy Khachaturian <y.khachaturian@souzmult.ru>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Yuri Khachaturyan
2026-07-21 16:59:07 +03:00
committed by GitHub
parent 9e117bbdd3
commit 2b1308ca29
13 changed files with 247 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/web/service/setting.go
+14
View File
@@ -155,6 +155,9 @@ var defaultValueMap = map[string]string{
"smtpCpu": "80",
"smtpMemory": "80",
// Consecutive failed observatory probes before an outbound.down event fires
"outboundDownThreshold": "3",
// Email (SMTP) notifications
"smtpEnable": "false",
"smtpHost": "",
@@ -1136,6 +1139,17 @@ func (s *SettingService) SetSmtpMemory(value int) error {
return s.setInt("smtpMemory", value)
}
// GetOutboundDownThreshold returns how many consecutive failed observatory
// probes an outbound must accumulate before an outbound.down notification is
// emitted. 1 preserves the legacy "notify on the first failed probe" behaviour.
func (s *SettingService) GetOutboundDownThreshold() (int, error) {
return s.getInt("outboundDownThreshold")
}
func (s *SettingService) SetOutboundDownThreshold(value int) error {
return s.setInt("outboundDownThreshold", value)
}
// SecretClears marks redacted secrets the user explicitly emptied. Without a
// flag, a blank submitted secret means "unchanged" (the field is always served
// blank to the browser) and the stored value is preserved.