feat(notifications): add a consecutive-failure threshold for outbound.down alerts (#5968)

mirror of https://github.com/MHSanaei/3x-ui.git synced 2026-07-23 13:06:08 +00:00

Problem: a flaky outbound produces hundreds of false-positive "outbound down"
notifications overnight — each fires the moment xray's observatory reports a
single failed probe, and the next successful probe fires an "up".

applyObservatory forwarded every raw alive:true->false transition straight to
EventOutboundDown; xray's observatory has effectively no hysteresis, and nothing
on the panel side debounced it (the email/Telegram subscribers are pure
formatters).

Fix: debounce per outbound. outbound.down now fires only after
outboundDownThreshold consecutive FAILED probes (new setting, default 3);
outbound.up fires immediately on the first successful probe and only when a down
was actually notified. The threshold gates the event itself, so email and
Telegram share one knob (exposed next to the outbound.down toggle).

The streak counts genuinely new probes (last_try_time advancing), not sampler
polls — the sampler runs every 2s but the observatory re-probes per its
probeInterval, so counting samples would trip the threshold instantly.
outboundDownThreshold=1 reproduces the legacy notify-on-first-failure behaviour.

Tuning the observatory's probe interval/timeout is not a workaround: those
probes also drive the load balancer's outbound selection, so loosening them to
quiet notifications would slow real failover away from a genuinely dead
outbound. Notifications don't need observatory-grade latency, so the tolerance
belongs at the notification layer, leaving the observatory (and balancer)
untouched.

Adds TestApplyObservatoryDebounce covering the threshold, probe-vs-sample
counting, single-blip suppression and the legacy path.

Co-authored-by: Yuriy Khachaturian <y.khachaturian@souzmult.ru>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

This commit is contained in:

Yuri Khachaturyan

2026-07-21 16:59:07 +03:00

committed by

GitHub

parent 9e117bbdd3

commit 2b1308ca29

13 changed files with 247 additions and 23 deletions

									
										internal/web/entity/entity.go
									
		+2
		
												View File
												
				@@ -59,6 +59,8 @@ type AllSetting struct {

					SmtpCpu            int    `json:"smtpCpu" form:"smtpCpu" validate:"gte=0,lte=100"`

					SmtpMemory         int    `json:"smtpMemory" form:"smtpMemory" validate:"gte=0,lte=100"`

					OutboundDownThreshold int `json:"outboundDownThreshold" form:"outboundDownThreshold" validate:"gte=1,lte=100"`

					TimeLocation    string `json:"timeLocation" form:"timeLocation"`

					TwoFactorEnable bool   `json:"twoFactorEnable" form:"twoFactorEnable"`

					TwoFactorToken  string `json:"twoFactorToken" form:"twoFactorToken"`