feat(tenancy): add Workspace multi-tenant foundation (#2353)

* Document multi-tenant workspace architecture

* Add OSS and commercial workspace boundaries

* docs: redesign multi-tenant workspace architecture

* feat(tenancy): implement workspace isolation

* docs(tenancy): record verification evidence

* docs(tenancy): revise single-instance SaaS topology

* docs(tenancy): refine architecture options

* docs: finalize cloud v2 multi-tenant decisions

* feat(tenancy): establish cloud isolation foundations

* feat(tenancy): harden shared cloud runtime boundaries

* docs(tenancy): record final isolation verification

* fix(tenancy): close isolation and permission gaps

* docs(tenancy): record final isolation verification

* feat(tenancy): connect cloud workspace control plane

* fix(build): install git for pinned SDK

* docs(cloud): update control plane verification

* chore: update multi-tenant SDK pin

* fix(cloud): skip legacy model sync during startup

* test(cloud): preserve minimal model manager fixtures

* fix(cloud): preserve authenticated account context

* fix(cloud): reuse authenticated account for user info

* feat(cloud): complete Workspace settings navigation

* test(web): cover Workspace dropdown menu

* feat(web): place workspace controls in sidebar

* refactor(web): streamline workspace controls

* style(web): format workspace layout test

* fix(cloud): surface runtime and workspace plan status

* fix(plugin): keep runtime identity stable across restarts

* fix(ui): widen and center workspace switcher

* fix(ui): hide roles from workspace switcher

* fix(ui): align workspace switcher with sidebar entries

* feat(workspace): add in-product collaboration and direct Cloud launch

* style: format collaboration changes

* fix(workspace): bind collaboration APIs to tenant UoW

* fix(cloud): preserve Core-owned collaboration state

* test(cloud): require Space identity for invite registration

* feat(cloud): complete secure invitation experience

* style(web): format invitation flows

* fix(cloud): recover box runtime without unscoped skill reload

* feat(oss): enforce invitation account and owner billing flows

* style: format OSS account service

* test(oss): cover invitation logout handoff

* fix(oss): resolve workspace owner in scoped session

* feat(cloud): harden multi-tenant runtime resources

* fix(cloud): bound runtime restart storms

* fix(cloud): eliminate periodic runtime CPU spikes

* fix(cloud): enforce instance capacity ceilings

* fix(cloud): scope public login capability discovery

* fix(cloud): bound tenant maintenance and monitoring work

* fix(runtime): bound tenant resource amplification

* fix(deps): pin green multi-tenant plugin SDK

* fix(cloud): handle unavailable skill capability

* fix(security): require authentication for image file endpoint (H-2)

- Changed /api/v1/files/image from AuthType.NONE to USER_TOKEN_OR_API_KEY
- Added Permission.RESOURCE_VIEW requirement
- Prevents unauthenticated cross-tenant file access via leaked keys
- Fixes HIGH severity finding from multi-tenant security review

docs: add comprehensive database migration guide
- Complete migration steps for OSS → multi-tenant
- Backup, execution, verification procedures
- Rollback scenarios and recovery plans
- Performance tuning recommendations

* test: add comprehensive cross-tenant isolation tests

Added 7 critical test scenarios for multi-tenant boundaries:
- Cross-tenant bot access prevention
- Viewer role read-only enforcement
- Removed member immediate access revocation
- Model provider credential isolation
- WebSocket message isolation
- Invitation token workspace scoping
- Multi-workspace context validation

These tests address P0-2 coverage gaps for:
- workspaces.py (membership & invitation flows)
- user.py (authentication & authorization)
- websocket_chat.py (real-time isolation)
- plugins.py (resource access control)

docs: finalize database migration guide

* fix(security): resolve M-1, M-2, M-3 security findings

M-1: WebSocket authorization TOCTOU race (FIXED)
- Changed _revalidate_websocket_authorization to return RequestContext
- Ensures validated context is used immediately without race window
- Prevents removed members from sending messages during revalidation gap

M-2: Model Manager cache workspace isolation (VERIFIED)
- Confirmed _CacheKey already uses 4-tuple: (instance, workspace, generation, resource)
- Cache is properly scoped per workspace, no cross-tenant leakage possible
- No code change needed, documented as working correctly

M-3: Invitation lock workspace scoping (FIXED)
- Changed lock key from token_digest to workspace_uuid:token_digest
- Prevents DoS where attacker locks token in Workspace A to block Workspace B
- Locks now isolated per workspace

All MEDIUM severity findings from security review now resolved.

* fix(cloud): unblock tenant CI and enforce knowledge quotas

* fix(tenancy): scope rerank model sync

---------

Co-authored-by: dadachann <185672915+dadachann@users.noreply.github.com>
This commit is contained in:
RockChinQ
2026-07-30 21:43:35 +08:00
committed by GitHub
parent 463b120923
commit e1ac5e0fc8
468 changed files with 78320 additions and 13137 deletions
@@ -20,10 +20,12 @@ import httpx
import quart
from ... import group
from ......utils import paths
from ......platform.sources.websocket_manager import is_valid_session_id, ws_connection_manager
from ......utils import httpclient, paths
from ......platform.sources.websocket_manager import WebSocketScope, is_valid_session_id, ws_connection_manager
from .websocket_chat import create_scoped_duplex_tasks, wait_for_duplex_tasks
logger = logging.getLogger(__name__)
_AUTH_TIMEOUT_SECONDS = 10.0
# Cache the widget template content
_widget_template_cache: str | None = None
@@ -58,37 +60,31 @@ def _get_logo_bytes() -> bytes:
class EmbedRouterGroup(group.RouterGroup):
# -- helpers -------------------------------------------------------------
def _resolve_bot(self, bot_uuid: str):
async def _resolve_bot(self, bot_uuid: str):
"""Resolve *bot_uuid* to ``(runtime_bot, pipeline_uuid)``.
Returns ``(None, None)`` when the bot does not exist, is not a
``web_page_bot``, is disabled, or has no pipeline bound.
"""
for bot in self.ap.platform_mgr.bots:
if (
bot.bot_entity.uuid == bot_uuid
and bot.bot_entity.adapter == 'web_page_bot'
and bot.bot_entity.enable
and bot.bot_entity.use_pipeline_uuid
):
return bot, bot.bot_entity.use_pipeline_uuid
bot = await self.ap.platform_mgr.resolve_public_bot(bot_uuid)
if (
bot is not None
and bot.bot_entity.adapter == 'web_page_bot'
and bot.bot_entity.enable
and bot.bot_entity.use_pipeline_uuid
):
return bot, bot.bot_entity.use_pipeline_uuid
return None, None
def _get_bot_config(self, bot_uuid: str) -> dict:
for bot in self.ap.platform_mgr.bots:
if bot.bot_entity.uuid == bot_uuid and bot.bot_entity.adapter == 'web_page_bot':
return bot.bot_entity.adapter_config
return {}
@staticmethod
def _get_bot_config(runtime_bot) -> dict:
return runtime_bot.bot_entity.adapter_config
async def _verify_session_token(self, request, bot_uuid: str) -> bool:
config = self._get_bot_config(bot_uuid)
def _verify_session_token_value(self, token: str, runtime_bot) -> bool:
config = self._get_bot_config(runtime_bot)
secret = config.get('turnstile_secret_key', '')
if not secret:
return True
auth_header = request.headers.get('Authorization', '')
if not auth_header.startswith('Bearer '):
return False
token = auth_header[7:]
try:
ts_str, mac = token.split('.', 1)
ts = float(ts_str)
@@ -99,6 +95,50 @@ class EmbedRouterGroup(group.RouterGroup):
except Exception:
return False
async def _verify_session_token(self, request, runtime_bot) -> bool:
auth_header = request.headers.get('Authorization', '')
token = auth_header[7:] if auth_header.startswith('Bearer ') else ''
return self._verify_session_token_value(token, runtime_bot)
async def _authenticate_websocket(self, runtime_bot) -> None:
"""Require the embed session token as the first WebSocket frame."""
raw_message = await asyncio.wait_for(quart.websocket.receive(), timeout=_AUTH_TIMEOUT_SECONDS)
payload = await asyncio.to_thread(json.loads, raw_message)
if not isinstance(payload, dict) or payload.get('type') != 'authenticate':
raise ValueError('Authentication is required')
token = str(payload.get('token') or '')
if not self._verify_session_token_value(token, runtime_bot):
raise ValueError('Authentication is required')
async def _assert_execution_active(self, runtime_bot) -> None:
context = runtime_bot.execution_context
await self.ap.workspace_service.get_execution_binding(
context.workspace_uuid,
expected_generation=context.placement_generation,
)
async def _resolve_connected_bot(self, owner_bot, pipeline_uuid: str):
"""Re-resolve mutable bot state before every public message."""
current_bot, current_pipeline_uuid = await self._resolve_bot(owner_bot.bot_entity.uuid)
if current_bot is None or current_pipeline_uuid != pipeline_uuid:
raise RuntimeError('Bot is unavailable')
owner_context = owner_bot.execution_context
current_context = current_bot.execution_context
if (
current_context.instance_uuid,
current_context.workspace_uuid,
current_context.placement_generation,
) != (
owner_context.instance_uuid,
owner_context.workspace_uuid,
owner_context.placement_generation,
):
raise RuntimeError('Bot is unavailable')
await self._assert_execution_active(current_bot)
return current_bot
# -- routes --------------------------------------------------------------
async def initialize(self) -> None:
@@ -106,7 +146,7 @@ class EmbedRouterGroup(group.RouterGroup):
async def verify_turnstile(bot_uuid: str) -> str:
if not _is_valid_uuid(bot_uuid):
return self.http_status(400, -1, 'Invalid bot_uuid format')
runtime_bot, pipeline_uuid = self._resolve_bot(bot_uuid)
runtime_bot, pipeline_uuid = await self._resolve_bot(bot_uuid)
if runtime_bot is None:
return self.http_status(404, -1, 'Bot not found or not available')
try:
@@ -115,18 +155,18 @@ class EmbedRouterGroup(group.RouterGroup):
if not token:
return self.http_status(400, -1, 'Token is required')
config = self._get_bot_config(bot_uuid)
config = self._get_bot_config(runtime_bot)
secret = config.get('turnstile_secret_key', '')
if not secret:
ts = time.time()
return self.success(data={'token': f'{ts}.dummy'})
async with httpx.AsyncClient() as client:
async with httpx.AsyncClient(event_hooks=httpclient.httpx_response_limit_hooks()) as client:
resp = await client.post(
'https://challenges.cloudflare.com/turnstile/v0/siteverify',
data={'secret': secret, 'response': token},
)
result = resp.json()
result = await httpclient.parse_json_response(resp)
if not result.get('success'):
return self.http_status(403, -1, 'Turnstile verification failed')
@@ -146,7 +186,7 @@ class EmbedRouterGroup(group.RouterGroup):
"""Serve the embed widget JavaScript with injected configuration."""
if not _is_valid_uuid(bot_uuid):
return self.http_status(400, -1, 'Invalid bot_uuid format')
runtime_bot, pipeline_uuid = self._resolve_bot(bot_uuid)
runtime_bot, pipeline_uuid = await self._resolve_bot(bot_uuid)
if runtime_bot is None:
return quart.Response(
'// Bot not found or not available', status=404, content_type='application/javascript'
@@ -164,7 +204,7 @@ class EmbedRouterGroup(group.RouterGroup):
if not re.match(r'^https?://[a-zA-Z0-9._:/-]+$', base_url):
base_url = quart.request.host_url.rstrip('/')
config = self._get_bot_config(bot_uuid)
config = self._get_bot_config(runtime_bot)
site_key = config.get('turnstile_site_key', '')
locale = config.get('language', 'en_US') or 'en_US'
bubble_icon = config.get('bubble_icon', 'logo') or 'logo'
@@ -194,10 +234,10 @@ class EmbedRouterGroup(group.RouterGroup):
async def get_embed_messages(bot_uuid: str, session_type: str) -> str:
if not _is_valid_uuid(bot_uuid):
return self.http_status(400, -1, 'Invalid bot_uuid format')
runtime_bot, pipeline_uuid = self._resolve_bot(bot_uuid)
runtime_bot, pipeline_uuid = await self._resolve_bot(bot_uuid)
if runtime_bot is None:
return self.http_status(404, -1, 'Bot not found or not available')
if not await self._verify_session_token(quart.request, bot_uuid):
if not await self._verify_session_token(quart.request, runtime_bot):
return self.http_status(403, -1, 'Unauthorized or session expired')
try:
if session_type not in ['person', 'group']:
@@ -207,7 +247,8 @@ class EmbedRouterGroup(group.RouterGroup):
if not is_valid_session_id(session_id):
return self.http_status(400, -1, 'Valid session_id is required')
websocket_adapter = self.ap.platform_mgr.websocket_proxy_bot.adapter
proxy_bot = await self.ap.platform_mgr.get_websocket_proxy_bot(runtime_bot.execution_context)
websocket_adapter = proxy_bot.adapter
if not websocket_adapter:
return self.http_status(404, -1, 'WebSocket adapter not found')
@@ -222,10 +263,10 @@ class EmbedRouterGroup(group.RouterGroup):
async def reset_embed_session(bot_uuid: str, session_type: str) -> str:
if not _is_valid_uuid(bot_uuid):
return self.http_status(400, -1, 'Invalid bot_uuid format')
runtime_bot, pipeline_uuid = self._resolve_bot(bot_uuid)
runtime_bot, pipeline_uuid = await self._resolve_bot(bot_uuid)
if runtime_bot is None:
return self.http_status(404, -1, 'Bot not found or not available')
if not await self._verify_session_token(quart.request, bot_uuid):
if not await self._verify_session_token(quart.request, runtime_bot):
return self.http_status(403, -1, 'Unauthorized or session expired')
try:
if session_type not in ['person', 'group']:
@@ -235,7 +276,8 @@ class EmbedRouterGroup(group.RouterGroup):
if not is_valid_session_id(session_id):
return self.http_status(400, -1, 'Valid session_id is required')
websocket_adapter = self.ap.platform_mgr.websocket_proxy_bot.adapter
proxy_bot = await self.ap.platform_mgr.get_websocket_proxy_bot(runtime_bot.execution_context)
websocket_adapter = proxy_bot.adapter
if not websocket_adapter:
return self.http_status(404, -1, 'WebSocket adapter not found')
@@ -250,10 +292,10 @@ class EmbedRouterGroup(group.RouterGroup):
async def submit_feedback(bot_uuid: str) -> str:
if not _is_valid_uuid(bot_uuid):
return self.http_status(400, -1, 'Invalid bot_uuid format')
runtime_bot, pipeline_uuid = self._resolve_bot(bot_uuid)
runtime_bot, pipeline_uuid = await self._resolve_bot(bot_uuid)
if runtime_bot is None:
return self.http_status(404, -1, 'Bot not found or not available')
if not await self._verify_session_token(quart.request, bot_uuid):
if not await self._verify_session_token(quart.request, runtime_bot):
return self.http_status(403, -1, 'Unauthorized or session expired')
try:
data = await quart.request.get_json()
@@ -266,6 +308,7 @@ class EmbedRouterGroup(group.RouterGroup):
feedback_id = f'embed_{uuid.uuid4().hex[:12]}'
await self.ap.monitoring_service.record_feedback(
runtime_bot.execution_context,
feedback_id=feedback_id,
feedback_type=feedback_type,
bot_id=runtime_bot.bot_entity.uuid,
@@ -286,11 +329,12 @@ class EmbedRouterGroup(group.RouterGroup):
@self.quart_app.websocket(self.path + '/<bot_uuid>/ws/connect')
async def embed_websocket_connect(bot_uuid: str):
"""WebSocket connection for embed widget, keyed by bot_uuid."""
await quart.websocket.accept()
if not _is_valid_uuid(bot_uuid):
await quart.websocket.send(json.dumps({'type': 'error', 'message': 'Invalid bot_uuid format'}))
return
runtime_bot, pipeline_uuid = self._resolve_bot(bot_uuid)
runtime_bot, pipeline_uuid = await self._resolve_bot(bot_uuid)
if runtime_bot is None:
await quart.websocket.send(json.dumps({'type': 'error', 'message': 'Bot not found or not available'}))
return
@@ -307,18 +351,42 @@ class EmbedRouterGroup(group.RouterGroup):
await quart.websocket.send(json.dumps({'type': 'error', 'message': 'Valid session_id is required'}))
return
websocket_adapter = self.ap.platform_mgr.websocket_proxy_bot.adapter
if not websocket_adapter:
await quart.websocket.send(json.dumps({'type': 'error', 'message': 'WebSocket adapter not found'}))
try:
await self._authenticate_websocket(runtime_bot)
await self._assert_execution_active(runtime_bot)
except Exception:
await quart.websocket.send(json.dumps({'type': 'error', 'message': 'Unauthorized'}))
return
try:
proxy_bot = await self.ap.platform_mgr.get_websocket_proxy_bot(runtime_bot.execution_context)
websocket_adapter = proxy_bot.adapter
if not websocket_adapter:
await quart.websocket.send(json.dumps({'type': 'error', 'message': 'WebSocket adapter not found'}))
return
connection = await ws_connection_manager.add_connection(
websocket=quart.websocket._get_current_object(),
scope=WebSocketScope.from_context(runtime_bot.execution_context),
pipeline_uuid=pipeline_uuid,
session_type=session_type,
session_id=session_id,
metadata={'user_agent': quart.websocket.headers.get('User-Agent', '')},
send_queue_size=(
self.ap.instance_config.data.get('system', {})
.get('websocket_retention', {})
.get('send_queue_size', 100)
),
max_connections=(
self.ap.instance_config.data.get('system', {})
.get('websocket_retention', {})
.get('max_connections', 1024)
),
max_connections_per_workspace=(
self.ap.instance_config.data.get('system', {})
.get('websocket_retention', {})
.get('max_connections_per_workspace', 32)
),
)
await quart.websocket.send(
@@ -338,11 +406,19 @@ class EmbedRouterGroup(group.RouterGroup):
f'(bot={bot_uuid}, pipeline={pipeline_uuid}, session_type={session_type})'
)
receive_task = asyncio.create_task(self._handle_receive(connection, websocket_adapter, runtime_bot))
send_task = asyncio.create_task(self._handle_send(connection))
receive_task, send_task = create_scoped_duplex_tasks(
self._handle_receive(
connection,
websocket_adapter,
runtime_bot,
pipeline_uuid,
),
self._handle_send(connection),
runtime_bot.execution_context.workspace_uuid,
)
try:
await asyncio.gather(receive_task, send_task)
await wait_for_duplex_tasks(receive_task, send_task)
except Exception as e:
logger.error(f'Embed WebSocket task error: {e}')
finally:
@@ -357,14 +433,14 @@ class EmbedRouterGroup(group.RouterGroup):
# -- WebSocket receive/send helpers --------------------------------------
async def _handle_receive(self, connection, websocket_adapter, owner_bot):
async def _handle_receive(self, connection, websocket_adapter, owner_bot, pipeline_uuid: str):
try:
while connection.is_active:
message = await quart.websocket.receive()
await ws_connection_manager.update_activity(connection.connection_id)
try:
data = json.loads(message)
data = await asyncio.to_thread(json.loads, message)
message_type = data.get('type', 'message')
if message_type == 'ping':
@@ -372,7 +448,12 @@ class EmbedRouterGroup(group.RouterGroup):
{'type': 'pong', 'timestamp': datetime.datetime.now().isoformat()}
)
elif message_type == 'message':
await websocket_adapter.handle_websocket_message(connection, data, owner_bot=owner_bot)
try:
current_bot = await self._resolve_connected_bot(owner_bot, pipeline_uuid)
except Exception:
await connection.send_queue.put({'type': 'error', 'message': 'Bot is unavailable'})
break
await websocket_adapter.handle_websocket_message(connection, data, owner_bot=current_bot)
elif message_type == 'disconnect':
break
@@ -383,13 +464,20 @@ class EmbedRouterGroup(group.RouterGroup):
logger.error(f'Embed receive error: {e}', exc_info=True)
finally:
connection.is_active = False
try:
connection.send_queue.put_nowait(None)
except asyncio.QueueFull:
pass
async def _handle_send(self, connection):
try:
while connection.is_active:
while connection.is_active or not connection.send_queue.empty():
try:
message = await asyncio.wait_for(connection.send_queue.get(), timeout=1.0)
await quart.websocket.send(json.dumps(message))
if message is None:
break
encoded = await asyncio.to_thread(json.dumps, message)
await quart.websocket.send(encoded)
except asyncio.TimeoutError:
continue
except Exception as e:
@@ -2,120 +2,156 @@ from __future__ import annotations
import quart
from ....authz import Permission, has_permission
from ....context import RequestContext
from ....service.secrets import redact_secrets
from ... import group
@group.group_class('pipelines', '/api/v1/pipelines')
class PipelinesRouterGroup(group.RouterGroup):
async def initialize(self) -> None:
@self.route('', methods=['GET', 'POST'], auth_type=group.AuthType.USER_TOKEN_OR_API_KEY)
async def _() -> str:
if quart.request.method == 'GET':
sort_by = quart.request.args.get('sort_by', 'created_at')
sort_order = quart.request.args.get('sort_order', 'DESC')
return self.success(
data={'pipelines': await self.ap.pipeline_service.get_pipelines(sort_by, sort_order)}
)
elif quart.request.method == 'POST':
json_data = await quart.request.json
pipeline_uuid = await self.ap.pipeline_service.create_pipeline(json_data)
return self.success(data={'uuid': pipeline_uuid})
@self.route('/_/metadata', methods=['GET'], auth_type=group.AuthType.USER_TOKEN_OR_API_KEY)
async def _() -> str:
return self.success(data={'configs': await self.ap.pipeline_service.get_pipeline_metadata()})
@self.route(
'',
methods=['GET'],
auth_type=group.AuthType.USER_TOKEN_OR_API_KEY,
permission=Permission.RESOURCE_VIEW,
)
async def _(request_context: RequestContext) -> str:
sort_by = quart.request.args.get('sort_by', 'created_at')
sort_order = quart.request.args.get('sort_order', 'DESC')
include_secret = has_permission(request_context, Permission.RESOURCE_MANAGE)
return self.success(
data={
'pipelines': await self.ap.pipeline_service.get_pipelines(
request_context,
sort_by,
sort_order,
include_secret=include_secret,
)
}
)
@self.route(
'/<pipeline_uuid>', methods=['GET', 'PUT', 'DELETE'], auth_type=group.AuthType.USER_TOKEN_OR_API_KEY
'',
methods=['POST'],
auth_type=group.AuthType.USER_TOKEN_OR_API_KEY,
permission=Permission.RESOURCE_MANAGE,
)
async def _(pipeline_uuid: str) -> str:
if quart.request.method == 'GET':
pipeline = await self.ap.pipeline_service.get_pipeline(pipeline_uuid)
async def _(request_context: RequestContext) -> str:
pipeline_uuid = await self.ap.pipeline_service.create_pipeline(request_context, await quart.request.json)
return self.success(data={'uuid': pipeline_uuid})
if pipeline is None:
return self.http_status(404, -1, 'pipeline not found')
@self.route(
'/_/metadata',
methods=['GET'],
auth_type=group.AuthType.USER_TOKEN_OR_API_KEY,
permission=Permission.RESOURCE_VIEW,
)
async def _(request_context: RequestContext) -> str:
return self.success(data={'configs': await self.ap.pipeline_service.get_pipeline_metadata(request_context)})
return self.success(data={'pipeline': pipeline})
elif quart.request.method == 'PUT':
json_data = await quart.request.json
@self.route(
'/<pipeline_uuid>',
methods=['GET'],
auth_type=group.AuthType.USER_TOKEN_OR_API_KEY,
permission=Permission.RESOURCE_VIEW,
)
async def _(pipeline_uuid: str, request_context: RequestContext) -> str:
pipeline = await self.ap.pipeline_service.get_pipeline(
request_context,
pipeline_uuid,
include_secret=has_permission(request_context, Permission.RESOURCE_MANAGE),
)
if pipeline is None:
return self.http_status(404, -1, 'pipeline not found')
return self.success(data={'pipeline': pipeline})
await self.ap.pipeline_service.update_pipeline(pipeline_uuid, json_data)
@self.route(
'/<pipeline_uuid>',
methods=['PUT', 'DELETE'],
auth_type=group.AuthType.USER_TOKEN_OR_API_KEY,
permission=Permission.RESOURCE_MANAGE,
)
async def _(pipeline_uuid: str, request_context: RequestContext) -> str:
if quart.request.method == 'PUT':
try:
await self.ap.pipeline_service.update_pipeline(
request_context,
pipeline_uuid,
await quart.request.json,
)
except ValueError as exc:
return self.http_status(400, -1, str(exc))
else:
await self.ap.pipeline_service.delete_pipeline(request_context, pipeline_uuid)
return self.success()
return self.success()
elif quart.request.method == 'DELETE':
await self.ap.pipeline_service.delete_pipeline(pipeline_uuid)
return self.success()
@self.route('/<pipeline_uuid>/copy', methods=['POST'], auth_type=group.AuthType.USER_TOKEN_OR_API_KEY)
async def _(pipeline_uuid: str) -> str:
@self.route(
'/<pipeline_uuid>/copy',
methods=['POST'],
auth_type=group.AuthType.USER_TOKEN_OR_API_KEY,
permission=Permission.RESOURCE_MANAGE,
)
async def _(pipeline_uuid: str, request_context: RequestContext) -> str:
try:
new_uuid = await self.ap.pipeline_service.copy_pipeline(pipeline_uuid)
new_uuid = await self.ap.pipeline_service.copy_pipeline(request_context, pipeline_uuid)
return self.success(data={'uuid': new_uuid})
except ValueError as e:
return self.http_status(404, -1, str(e))
return self.http_status(400, -1, str(e))
@self.route(
'/<pipeline_uuid>/extensions', methods=['GET', 'PUT'], auth_type=group.AuthType.USER_TOKEN_OR_API_KEY
'/<pipeline_uuid>/extensions',
methods=['GET'],
auth_type=group.AuthType.USER_TOKEN_OR_API_KEY,
permission=Permission.RESOURCE_VIEW,
)
async def _(pipeline_uuid: str) -> str:
if quart.request.method == 'GET':
# Get current extensions and available plugins
pipeline = await self.ap.pipeline_service.get_pipeline(pipeline_uuid)
if pipeline is None:
return self.http_status(404, -1, 'pipeline not found')
async def _(pipeline_uuid: str, request_context: RequestContext) -> str:
pipeline = await self.ap.pipeline_service.get_pipeline(request_context, pipeline_uuid)
if pipeline is None:
return self.http_status(404, -1, 'pipeline not found')
# Only include plugins with pipeline-related components (Command, EventListener, Tool)
# Plugins that only have KnowledgeEngine components are not suitable for pipeline extensions
pipeline_component_kinds = ['Command', 'EventListener', 'Tool']
plugins = await self.ap.plugin_connector.list_plugins(component_kinds=pipeline_component_kinds)
mcp_servers = await self.ap.mcp_service.get_mcp_servers(contain_runtime_info=True)
pipeline_component_kinds = ['Command', 'EventListener', 'Tool']
if self.ap.plugin_connector.is_enable_plugin:
await self.ap.plugin_connector.require_workspace_context(request_context)
plugins = await self.ap.plugin_connector.list_plugins(component_kinds=pipeline_component_kinds)
mcp_servers = await self.ap.mcp_service.get_mcp_servers(request_context, contain_runtime_info=True)
available_skills = await self.ap.skill_service.list_skills(request_context)
extensions_prefs = pipeline.get('extensions_preferences', {})
return self.success(
data={
'enable_all_plugins': extensions_prefs.get('enable_all_plugins', True),
'enable_all_mcp_servers': extensions_prefs.get('enable_all_mcp_servers', True),
'enable_all_skills': extensions_prefs.get('enable_all_skills', True),
'bound_plugins': extensions_prefs.get('plugins', []),
'available_plugins': redact_secrets(plugins),
'bound_mcp_servers': extensions_prefs.get('mcp_servers', []),
'available_mcp_servers': mcp_servers,
'bound_mcp_resources': extensions_prefs.get('mcp_resources', []),
'mcp_resource_agent_read_enabled': extensions_prefs.get('mcp_resource_agent_read_enabled', True),
'bound_skills': extensions_prefs.get('skills', []),
'available_skills': available_skills,
}
)
# Get available skills
available_skills = await self.ap.skill_service.list_skills()
extensions_prefs = pipeline.get('extensions_preferences', {})
return self.success(
data={
'enable_all_plugins': extensions_prefs.get('enable_all_plugins', True),
'enable_all_mcp_servers': extensions_prefs.get('enable_all_mcp_servers', True),
'enable_all_skills': extensions_prefs.get('enable_all_skills', True),
'bound_plugins': extensions_prefs.get('plugins', []),
'available_plugins': plugins,
'bound_mcp_servers': extensions_prefs.get('mcp_servers', []),
'available_mcp_servers': mcp_servers,
'bound_mcp_resources': extensions_prefs.get('mcp_resources', []),
'mcp_resource_agent_read_enabled': extensions_prefs.get(
'mcp_resource_agent_read_enabled', True
),
'bound_skills': extensions_prefs.get('skills', []),
'available_skills': available_skills,
}
)
elif quart.request.method == 'PUT':
# Update bound plugins and MCP servers for this pipeline
json_data = await quart.request.json
enable_all_plugins = json_data.get('enable_all_plugins', True)
enable_all_mcp_servers = json_data.get('enable_all_mcp_servers', True)
enable_all_skills = json_data.get('enable_all_skills', True)
bound_plugins = json_data.get('bound_plugins', [])
bound_mcp_servers = json_data.get('bound_mcp_servers', [])
bound_skills = json_data.get('bound_skills', [])
bound_mcp_resources = json_data.get('bound_mcp_resources')
mcp_resource_agent_read_enabled = json_data.get('mcp_resource_agent_read_enabled')
await self.ap.pipeline_service.update_pipeline_extensions(
pipeline_uuid,
bound_plugins,
bound_mcp_servers,
enable_all_plugins,
enable_all_mcp_servers,
bound_skills=bound_skills,
enable_all_skills=enable_all_skills,
bound_mcp_resources=bound_mcp_resources,
mcp_resource_agent_read_enabled=mcp_resource_agent_read_enabled,
)
return self.success()
@self.route(
'/<pipeline_uuid>/extensions',
methods=['PUT'],
auth_type=group.AuthType.USER_TOKEN_OR_API_KEY,
permission=Permission.RESOURCE_MANAGE,
)
async def _(pipeline_uuid: str, request_context: RequestContext) -> str:
json_data = await quart.request.json
await self.ap.pipeline_service.update_pipeline_extensions(
request_context,
pipeline_uuid,
json_data.get('bound_plugins', []),
json_data.get('bound_mcp_servers', []),
json_data.get('enable_all_plugins', True),
json_data.get('enable_all_mcp_servers', True),
bound_skills=json_data.get('bound_skills', []),
enable_all_skills=json_data.get('enable_all_skills', True),
bound_mcp_resources=json_data.get('bound_mcp_resources'),
mcp_resource_agent_read_enabled=json_data.get('mcp_resource_agent_read_enabled'),
)
return self.success()
@@ -1,64 +1,234 @@
"""WebSocket聊天路由 - 支持双向实时通信"""
"""Authenticated dashboard WebSocket chat routes."""
from __future__ import annotations
import asyncio
import datetime
import json
import logging
import typing
import uuid
import quart
from ....authz import Permission, permissions_for_role, require_permission
from ....context import PrincipalContext, PrincipalType, RequestContext, WorkspaceContext
from ... import group
from ......platform.sources.websocket_manager import ws_connection_manager
from ......core.task_boundary import run_in_workspace_uow
from ......platform.sources.websocket_manager import WebSocketScope, ws_connection_manager
from ......utils import bounded_executor
logger = logging.getLogger(__name__)
_AUTH_TIMEOUT_SECONDS = 10.0
_DUPLEX_DRAIN_TIMEOUT_SECONDS = 0.25
def create_scoped_duplex_tasks(
receive_coro: typing.Coroutine[typing.Any, typing.Any, None],
send_coro: typing.Coroutine[typing.Any, typing.Any, None],
workspace_uuid: str,
) -> tuple[asyncio.Task[None], asyncio.Task[None]]:
"""Create both socket directions under one trusted Workspace budget."""
return (
asyncio.create_task(
bounded_executor.run_in_blocking_work_scope(
receive_coro,
workspace_uuid,
)
),
asyncio.create_task(
bounded_executor.run_in_blocking_work_scope(
send_coro,
workspace_uuid,
)
),
)
async def wait_for_duplex_tasks(
receive_task: asyncio.Task,
send_task: asyncio.Task,
) -> None:
"""Stop the peer direction as soon as either socket task terminates."""
try:
done, _ = await asyncio.wait(
{receive_task, send_task},
return_when=asyncio.FIRST_COMPLETED,
)
# A receive task may enqueue a terminal authorization/error frame and
# then finish. Give the sender a short deterministic drain window
# instead of cancelling it before that frame reaches the client.
if receive_task in done and not send_task.done():
await asyncio.wait(
{send_task},
timeout=_DUPLEX_DRAIN_TIMEOUT_SECONDS,
)
finally:
for task in (receive_task, send_task):
if not task.done():
task.cancel()
await asyncio.gather(
receive_task,
send_task,
return_exceptions=True,
)
@group.group_class('websocket_chat', '/api/v1/pipelines/<pipeline_uuid>/ws')
class WebSocketChatRouterGroup(group.RouterGroup):
async def _authenticate_websocket(self) -> tuple[RequestContext, str]:
"""Authenticate the first dashboard WebSocket message.
Browsers cannot attach the normal Authorization/X-Workspace-Id headers
to a WebSocket handshake. The client therefore sends one auth frame
immediately after opening the socket; no connection is registered and
no runtime object is resolved before this method succeeds.
"""
raw_message = await asyncio.wait_for(quart.websocket.receive(), timeout=_AUTH_TIMEOUT_SECONDS)
payload = await asyncio.to_thread(json.loads, raw_message)
if not isinstance(payload, dict) or payload.get('type') != 'authenticate':
raise ValueError('Authentication is required')
token = str(payload.get('token') or '').strip()
workspace_uuid = str(payload.get('workspace_uuid') or '').strip()
if not token or not workspace_uuid:
raise ValueError('Authentication is required')
account, _ = await self._authenticate_account(token)
account_uuid = getattr(account, 'uuid', None)
collaboration_service = getattr(self.ap, 'workspace_collaboration_service', None)
if not isinstance(account_uuid, str) or collaboration_service is None:
raise ValueError('Workspace authentication is unavailable')
access = await collaboration_service.resolve_account_workspace(account_uuid, workspace_uuid)
request_context = RequestContext(
instance_uuid=access.execution.instance_uuid,
placement_generation=access.execution.placement_generation,
request_id=quart.websocket.headers.get('X-Request-Id') or str(uuid.uuid4()),
auth_type=group.AuthType.USER_TOKEN.value,
principal=PrincipalContext(
principal_type=PrincipalType.ACCOUNT,
account_uuid=account_uuid,
),
workspace=WorkspaceContext(
workspace_uuid=access.workspace.uuid,
membership_uuid=access.membership.uuid,
role=access.membership.role,
permissions=permissions_for_role(access.membership.role),
membership_revision=access.membership.projection_revision,
),
)
require_permission(request_context, Permission.RUNTIME_OPERATE)
return request_context, token
async def _revalidate_websocket_authorization(
self,
request_context: RequestContext,
token: str,
) -> RequestContext:
"""Recheck revocable account, membership, permission, and placement state."""
account, _ = await self._authenticate_account(token)
account_uuid = getattr(account, 'uuid', None)
if account_uuid != request_context.account_uuid:
raise ValueError('WebSocket account changed')
collaboration_service = getattr(self.ap, 'workspace_collaboration_service', None)
if collaboration_service is None or not isinstance(account_uuid, str):
raise ValueError('Workspace authentication is unavailable')
access = await collaboration_service.resolve_account_workspace(
account_uuid,
request_context.workspace_uuid,
)
if (
access.workspace.uuid != request_context.workspace_uuid
or access.membership.uuid != request_context.workspace.membership_uuid
or access.membership.projection_revision != request_context.workspace.membership_revision
or access.execution.instance_uuid != request_context.instance_uuid
or access.execution.placement_generation != request_context.placement_generation
):
raise ValueError('WebSocket authorization changed')
current_context = RequestContext(
instance_uuid=access.execution.instance_uuid,
placement_generation=access.execution.placement_generation,
request_id=request_context.request_id,
auth_type=request_context.auth_type,
principal=request_context.principal,
workspace=WorkspaceContext(
workspace_uuid=access.workspace.uuid,
membership_uuid=access.membership.uuid,
role=access.membership.role,
permissions=permissions_for_role(access.membership.role),
membership_revision=access.membership.projection_revision,
),
entitlement_revision=request_context.entitlement_revision,
)
require_permission(current_context, Permission.RUNTIME_OPERATE)
return current_context
async def _get_scoped_adapter(self, request_context: RequestContext, pipeline_uuid: str):
pipeline = await run_in_workspace_uow(
self.ap,
request_context.workspace_uuid,
lambda: self.ap.pipeline_service.get_pipeline(request_context, pipeline_uuid),
)
if pipeline is None:
return None
proxy_bot = await self.ap.platform_mgr.get_websocket_proxy_bot(request_context)
return proxy_bot.adapter
async def initialize(self) -> None:
# 直接使用 quart_app 注册 WebSocket 路由
@self.quart_app.websocket(self.path + '/connect')
async def websocket_connect(pipeline_uuid: str):
"""
建立WebSocket连接
"""Open one authenticated dashboard debug connection."""
URL参数:
- pipeline_uuid: 流水线UUID
- session_type: 会话类型 (person/group)
"""
await quart.websocket.accept()
try:
# 获取参数 - 在WebSocket上下文中使用 quart.websocket.args
session_type = quart.websocket.args.get('session_type', 'person')
request_context, token = await self._authenticate_websocket()
except Exception:
await quart.websocket.send(json.dumps({'type': 'error', 'message': 'Unauthorized'}))
return
if session_type not in ['person', 'group']:
await quart.websocket.send(
json.dumps({'type': 'error', 'message': 'session_type must be person or group'})
)
session_type = quart.websocket.args.get('session_type', 'person')
if session_type not in ['person', 'group']:
await quart.websocket.send(
json.dumps({'type': 'error', 'message': 'session_type must be person or group'})
)
return
try:
websocket_adapter = await self._get_scoped_adapter(request_context, pipeline_uuid)
if websocket_adapter is None:
await quart.websocket.send(json.dumps({'type': 'error', 'message': 'Pipeline not found'}))
return
# 获取WebSocket适配器
websocket_adapter = self.ap.platform_mgr.websocket_proxy_bot.adapter
if not websocket_adapter:
await quart.websocket.send(json.dumps({'type': 'error', 'message': 'WebSocket adapter not found'}))
return
# Dashboard pipeline-debug sessions must always run under the
# built-in websocket_proxy_bot identity. We deliberately do NOT
# resolve a web_page_bot owner here — even if one is bound to
# the same pipeline, debug requests must not be attributed to
# it. The embed widget path (`/api/v1/embed/<bot>/ws/connect`)
# is the one that carries the page-bot identity.
# 注册连接
connection = await ws_connection_manager.add_connection(
websocket=quart.websocket._get_current_object(),
scope=WebSocketScope.from_context(request_context),
pipeline_uuid=pipeline_uuid,
session_type=session_type,
metadata={'user_agent': quart.websocket.headers.get('User-Agent', '')},
send_queue_size=(
self.ap.instance_config.data.get('system', {})
.get('websocket_retention', {})
.get('send_queue_size', 100)
),
max_connections=(
self.ap.instance_config.data.get('system', {})
.get('websocket_retention', {})
.get('max_connections', 1024)
),
max_connections_per_workspace=(
self.ap.instance_config.data.get('system', {})
.get('websocket_retention', {})
.get('max_connections_per_workspace', 32)
),
)
# 发送连接成功消息
await quart.websocket.send(
json.dumps(
{
@@ -72,182 +242,188 @@ class WebSocketChatRouterGroup(group.RouterGroup):
)
logger.debug(
f'WebSocket connection established: {connection.connection_id} '
f'(pipeline={pipeline_uuid}, session_type={session_type})'
f'Dashboard WebSocket connected: {connection.connection_id} '
f'(workspace={connection.workspace_uuid}, pipeline={pipeline_uuid}, '
f'session_type={session_type})'
)
# 创建接收和发送任务
receive_task = asyncio.create_task(self._handle_receive(connection, websocket_adapter))
send_task = asyncio.create_task(self._handle_send(connection))
# 等待任务完成
receive_task, send_task = create_scoped_duplex_tasks(
self._handle_receive(
connection,
websocket_adapter,
request_context,
token,
),
self._handle_send(connection),
request_context.workspace_uuid,
)
try:
await asyncio.gather(receive_task, send_task)
except Exception as e:
logger.error(f'WebSocket task execution error: {e}')
await wait_for_duplex_tasks(receive_task, send_task)
except Exception as exc:
logger.error(f'WebSocket task execution error: {exc}')
finally:
# 清理连接
await ws_connection_manager.remove_connection(connection.connection_id)
logger.debug(f'WebSocket connection cleaned: {connection.connection_id}')
except Exception as e:
logger.error(f'WebSocket connection error: {e}', exc_info=True)
except Exception:
logger.error('Dashboard WebSocket connection error', exc_info=True)
try:
await quart.websocket.send(json.dumps({'type': 'error', 'message': str(e)}))
except:
await quart.websocket.send(json.dumps({'type': 'error', 'message': 'Internal server error'}))
except Exception:
pass
@self.route('/messages/<session_type>', methods=['GET'])
async def get_messages(pipeline_uuid: str, session_type: str) -> str:
"""获取消息历史"""
try:
if session_type not in ['person', 'group']:
return self.http_status(400, -1, 'session_type must be person or group')
@self.route(
'/messages/<session_type>',
methods=['GET'],
auth_type=group.AuthType.USER_TOKEN_OR_API_KEY,
permission=Permission.RUNTIME_OPERATE,
)
async def get_messages(
pipeline_uuid: str,
session_type: str,
request_context: RequestContext,
) -> str:
if session_type not in ['person', 'group']:
return self.http_status(400, -1, 'session_type must be person or group')
websocket_adapter = self.ap.platform_mgr.websocket_proxy_bot.adapter
websocket_adapter = await self._get_scoped_adapter(request_context, pipeline_uuid)
if websocket_adapter is None:
return self.http_status(404, -1, 'Pipeline not found')
messages = websocket_adapter.get_websocket_messages(pipeline_uuid, session_type)
return self.success(data={'messages': messages})
if not websocket_adapter:
return self.http_status(404, -1, 'WebSocket adapter not found')
@self.route(
'/reset/<session_type>',
methods=['POST'],
auth_type=group.AuthType.USER_TOKEN_OR_API_KEY,
permission=Permission.RUNTIME_OPERATE,
)
async def reset_session(
pipeline_uuid: str,
session_type: str,
request_context: RequestContext,
) -> str:
if session_type not in ['person', 'group']:
return self.http_status(400, -1, 'session_type must be person or group')
messages = websocket_adapter.get_websocket_messages(pipeline_uuid, session_type)
websocket_adapter = await self._get_scoped_adapter(request_context, pipeline_uuid)
if websocket_adapter is None:
return self.http_status(404, -1, 'Pipeline not found')
websocket_adapter.reset_session(pipeline_uuid, session_type)
return self.success(data={'message': 'Session reset successfully'})
return self.success(data={'messages': messages})
@self.route(
'/connections',
methods=['GET'],
auth_type=group.AuthType.USER_TOKEN_OR_API_KEY,
permission=Permission.RUNTIME_OPERATE,
)
async def get_connections(pipeline_uuid: str, request_context: RequestContext) -> str:
if await self.ap.pipeline_service.get_pipeline(request_context, pipeline_uuid) is None:
return self.http_status(404, -1, 'Pipeline not found')
except Exception as e:
return self.http_status(500, -1, f'Internal server error: {str(e)}')
@self.route('/reset/<session_type>', methods=['POST'])
async def reset_session(pipeline_uuid: str, session_type: str) -> str:
"""重置会话"""
try:
if session_type not in ['person', 'group']:
return self.http_status(400, -1, 'session_type must be person or group')
websocket_adapter = self.ap.platform_mgr.websocket_proxy_bot.adapter
if not websocket_adapter:
return self.http_status(404, -1, 'WebSocket adapter not found')
websocket_adapter.reset_session(pipeline_uuid, session_type)
return self.success(data={'message': 'Session reset successfully'})
except Exception as e:
return self.http_status(500, -1, f'Internal server error: {str(e)}')
@self.route('/connections', methods=['GET'])
async def get_connections(pipeline_uuid: str) -> str:
"""获取当前连接统计"""
try:
stats = ws_connection_manager.get_stats()
connections = await ws_connection_manager.get_connections_by_pipeline(pipeline_uuid)
return self.success(
data={
'stats': stats,
'connections': [
{
'connection_id': conn.connection_id,
'session_type': conn.session_type,
'created_at': conn.created_at.isoformat(),
'last_active': conn.last_active.isoformat(),
'is_active': conn.is_active,
}
for conn in connections
],
}
)
except Exception as e:
return self.http_status(500, -1, f'Internal server error: {str(e)}')
@self.route('/broadcast', methods=['POST'])
async def broadcast_message(pipeline_uuid: str) -> str:
"""向所有连接广播消息(后端主动推送)"""
try:
data = await quart.request.get_json()
message = data.get('message')
if not message:
return self.http_status(400, -1, 'message is required')
# 广播消息
broadcast_data = {
'type': 'broadcast',
'message': message,
'timestamp': datetime.datetime.now().isoformat(),
scope = WebSocketScope.from_context(request_context)
stats = ws_connection_manager.get_stats(scope=scope)
connections = await ws_connection_manager.get_connections_by_pipeline(
pipeline_uuid,
scope=scope,
)
return self.success(
data={
'stats': stats,
'connections': [
{
'connection_id': connection.connection_id,
'session_type': connection.session_type,
'created_at': connection.created_at.isoformat(),
'last_active': connection.last_active.isoformat(),
'is_active': connection.is_active,
}
for connection in connections
],
}
)
await ws_connection_manager.broadcast_to_pipeline(pipeline_uuid, broadcast_data)
@self.route(
'/broadcast',
methods=['POST'],
auth_type=group.AuthType.USER_TOKEN_OR_API_KEY,
permission=Permission.RUNTIME_OPERATE,
)
async def broadcast_message(pipeline_uuid: str, request_context: RequestContext) -> str:
if await self.ap.pipeline_service.get_pipeline(request_context, pipeline_uuid) is None:
return self.http_status(404, -1, 'Pipeline not found')
return self.success(data={'message': 'Broadcast sent successfully'})
data = await quart.request.get_json()
message = data.get('message')
if not message:
return self.http_status(400, -1, 'message is required')
except Exception as e:
return self.http_status(500, -1, f'Internal server error: {str(e)}')
broadcast_data = {
'type': 'broadcast',
'message': message,
'timestamp': datetime.datetime.now().isoformat(),
}
await ws_connection_manager.broadcast_to_pipeline(
pipeline_uuid,
broadcast_data,
scope=WebSocketScope.from_context(request_context),
)
return self.success(data={'message': 'Broadcast sent successfully'})
async def _handle_receive(self, connection, websocket_adapter):
"""处理接收消息的任务"""
async def _handle_receive(
self,
connection,
websocket_adapter,
request_context: RequestContext,
token: str,
):
try:
while connection.is_active:
# 接收消息
message = await quart.websocket.receive()
# 更新活跃时间
await ws_connection_manager.update_activity(connection.connection_id)
try:
data = json.loads(message)
data = await asyncio.to_thread(json.loads, message)
message_type = data.get('type', 'message')
if message_type == 'ping':
# 心跳响应
await connection.send_queue.put(
{'type': 'pong', 'timestamp': datetime.datetime.now().isoformat()}
)
elif message_type == 'message':
# 处理用户消息
logger.debug(f'收到消息: {data} from {connection.connection_id}')
# 处理消息(不等待响应,响应会通过broadcast异步发送)
# owner_bot is intentionally NOT passed: the dashboard
# debug WebSocket must always run under the proxy bot,
# never under a coincidentally-bound web_page_bot.
try:
await self._revalidate_websocket_authorization(request_context, token)
except Exception:
await connection.send_queue.put({'type': 'error', 'message': 'Unauthorized'})
break
await websocket_adapter.handle_websocket_message(connection, data)
elif message_type == 'disconnect':
# 客户端主动断开
logger.debug(f'Client disconnected: {connection.connection_id}')
break
else:
logger.warning(f'Unknown message type: {message_type}')
logger.warning(f'Unknown WebSocket message type: {message_type}')
except json.JSONDecodeError:
logger.error(f'Invalid JSON message: {message}')
await connection.send_queue.put({'type': 'error', 'message': 'Invalid JSON format'})
except Exception as e:
logger.error(f'Receive message error: {e}', exc_info=True)
except Exception:
logger.error('Dashboard WebSocket receive error', exc_info=True)
finally:
connection.is_active = False
try:
connection.send_queue.put_nowait(None)
except asyncio.QueueFull:
pass
async def _handle_send(self, connection):
"""处理发送消息的任务"""
try:
while connection.is_active:
# 从队列获取消息
while connection.is_active or not connection.send_queue.empty():
try:
message = await asyncio.wait_for(connection.send_queue.get(), timeout=1.0)
# 发送消息
await quart.websocket.send(json.dumps(message))
if message is None:
break
encoded = await asyncio.to_thread(json.dumps, message)
await quart.websocket.send(encoded)
except asyncio.TimeoutError:
# 超时继续循环
continue
except Exception as e:
logger.error(f'Send message error: {e}', exc_info=True)
except Exception:
logger.error('Dashboard WebSocket send error', exc_info=True)
finally:
connection.is_active = False