feat(tenancy): add Workspace multi-tenant foundation (#2353)

* Document multi-tenant workspace architecture * Add OSS and commercial workspace boundaries * docs: redesign multi-tenant workspace architecture * feat(tenancy): implement workspace isolation * docs(tenancy): record verification evidence * docs(tenancy): revise single-instance SaaS topology * docs(tenancy): refine architecture options * docs: finalize cloud v2 multi-tenant decisions * feat(tenancy): establish cloud isolation foundations * feat(tenancy): harden shared cloud runtime boundaries * docs(tenancy): record final isolation verification * fix(tenancy): close isolation and permission gaps * docs(tenancy): record final isolation verification * feat(tenancy): connect cloud workspace control plane * fix(build): install git for pinned SDK * docs(cloud): update control plane verification * chore: update multi-tenant SDK pin * fix(cloud): skip legacy model sync during startup * test(cloud): preserve minimal model manager fixtures * fix(cloud): preserve authenticated account context * fix(cloud): reuse authenticated account for user info * feat(cloud): complete Workspace settings navigation * test(web): cover Workspace dropdown menu * feat(web): place workspace controls in sidebar * refactor(web): streamline workspace controls * style(web): format workspace layout test * fix(cloud): surface runtime and workspace plan status * fix(plugin): keep runtime identity stable across restarts * fix(ui): widen and center workspace switcher * fix(ui): hide roles from workspace switcher * fix(ui): align workspace switcher with sidebar entries * feat(workspace): add in-product collaboration and direct Cloud launch * style: format collaboration changes * fix(workspace): bind collaboration APIs to tenant UoW * fix(cloud): preserve Core-owned collaboration state * test(cloud): require Space identity for invite registration * feat(cloud): complete secure invitation experience * style(web): format invitation flows * fix(cloud): recover box runtime without unscoped skill reload * feat(oss): enforce invitation account and owner billing flows * style: format OSS account service * test(oss): cover invitation logout handoff * fix(oss): resolve workspace owner in scoped session * feat(cloud): harden multi-tenant runtime resources * fix(cloud): bound runtime restart storms * fix(cloud): eliminate periodic runtime CPU spikes * fix(cloud): enforce instance capacity ceilings * fix(cloud): scope public login capability discovery * fix(cloud): bound tenant maintenance and monitoring work * fix(runtime): bound tenant resource amplification * fix(deps): pin green multi-tenant plugin SDK * fix(cloud): handle unavailable skill capability * fix(security): require authentication for image file endpoint (H-2) - Changed /api/v1/files/image from AuthType.NONE to USER_TOKEN_OR_API_KEY - Added Permission.RESOURCE_VIEW requirement - Prevents unauthenticated cross-tenant file access via leaked keys - Fixes HIGH severity finding from multi-tenant security review docs: add comprehensive database migration guide - Complete migration steps for OSS → multi-tenant - Backup, execution, verification procedures - Rollback scenarios and recovery plans - Performance tuning recommendations * test: add comprehensive cross-tenant isolation tests Added 7 critical test scenarios for multi-tenant boundaries: - Cross-tenant bot access prevention - Viewer role read-only enforcement - Removed member immediate access revocation - Model provider credential isolation - WebSocket message isolation - Invitation token workspace scoping - Multi-workspace context validation These tests address P0-2 coverage gaps for: - workspaces.py (membership & invitation flows) - user.py (authentication & authorization) - websocket_chat.py (real-time isolation) - plugins.py (resource access control) docs: finalize database migration guide * fix(security): resolve M-1, M-2, M-3 security findings M-1: WebSocket authorization TOCTOU race (FIXED) - Changed _revalidate_websocket_authorization to return RequestContext - Ensures validated context is used immediately without race window - Prevents removed members from sending messages during revalidation gap M-2: Model Manager cache workspace isolation (VERIFIED) - Confirmed _CacheKey already uses 4-tuple: (instance, workspace, generation, resource) - Cache is properly scoped per workspace, no cross-tenant leakage possible - No code change needed, documented as working correctly M-3: Invitation lock workspace scoping (FIXED) - Changed lock key from token_digest to workspace_uuid:token_digest - Prevents DoS where attacker locks token in Workspace A to block Workspace B - Locks now isolated per workspace All MEDIUM severity findings from security review now resolved. * fix(cloud): unblock tenant CI and enforce knowledge quotas * fix(tenancy): scope rerank model sync --------- Co-authored-by: dadachann <185672915+dadachann@users.noreply.github.com>
2026-08-09 04:40:57 +00:00 · 2026-07-30 21:43:35 +08:00
parent 463b120923
commit e1ac5e0fc8
468 changed files with 78320 additions and 13137 deletions
@@ -20,10 +20,12 @@ import httpx
 import quart

 from ... import group
-from ......utils import paths
-from ......platform.sources.websocket_manager import is_valid_session_id, ws_connection_manager
+from ......utils import httpclient, paths
+from ......platform.sources.websocket_manager import WebSocketScope, is_valid_session_id, ws_connection_manager
+from .websocket_chat import create_scoped_duplex_tasks, wait_for_duplex_tasks

 logger = logging.getLogger(__name__)
+_AUTH_TIMEOUT_SECONDS = 10.0

 # Cache the widget template content
 _widget_template_cache: str | None = None
@@ -58,37 +60,31 @@ def _get_logo_bytes() -> bytes:
 class EmbedRouterGroup(group.RouterGroup):
    # -- helpers -------------------------------------------------------------

-    def _resolve_bot(self, bot_uuid: str):
+    async def _resolve_bot(self, bot_uuid: str):
        """Resolve *bot_uuid* to ``(runtime_bot, pipeline_uuid)``.

        Returns ``(None, None)`` when the bot does not exist, is not a
        ``web_page_bot``, is disabled, or has no pipeline bound.
        """
-        for bot in self.ap.platform_mgr.bots:
-            if (
-                bot.bot_entity.uuid == bot_uuid
-                and bot.bot_entity.adapter == 'web_page_bot'
-                and bot.bot_entity.enable
-                and bot.bot_entity.use_pipeline_uuid
-            ):
-                return bot, bot.bot_entity.use_pipeline_uuid
+        bot = await self.ap.platform_mgr.resolve_public_bot(bot_uuid)
+        if (
+            bot is not None
+            and bot.bot_entity.adapter == 'web_page_bot'
+            and bot.bot_entity.enable
+            and bot.bot_entity.use_pipeline_uuid
+        ):
+            return bot, bot.bot_entity.use_pipeline_uuid
        return None, None

-    def _get_bot_config(self, bot_uuid: str) -> dict:
-        for bot in self.ap.platform_mgr.bots:
-            if bot.bot_entity.uuid == bot_uuid and bot.bot_entity.adapter == 'web_page_bot':
-                return bot.bot_entity.adapter_config
-        return {}
+    @staticmethod
+    def _get_bot_config(runtime_bot) -> dict:
+        return runtime_bot.bot_entity.adapter_config

-    async def _verify_session_token(self, request, bot_uuid: str) -> bool:
-        config = self._get_bot_config(bot_uuid)
+    def _verify_session_token_value(self, token: str, runtime_bot) -> bool:
+        config = self._get_bot_config(runtime_bot)
        secret = config.get('turnstile_secret_key', '')
        if not secret:
            return True
-        auth_header = request.headers.get('Authorization', '')
-        if not auth_header.startswith('Bearer '):
-            return False
-        token = auth_header[7:]
        try:
            ts_str, mac = token.split('.', 1)
            ts = float(ts_str)
@@ -99,6 +95,50 @@ class EmbedRouterGroup(group.RouterGroup):
        except Exception:
            return False

+    async def _verify_session_token(self, request, runtime_bot) -> bool:
+        auth_header = request.headers.get('Authorization', '')
+        token = auth_header[7:] if auth_header.startswith('Bearer ') else ''
+        return self._verify_session_token_value(token, runtime_bot)
+
+    async def _authenticate_websocket(self, runtime_bot) -> None:
+        """Require the embed session token as the first WebSocket frame."""
+
+        raw_message = await asyncio.wait_for(quart.websocket.receive(), timeout=_AUTH_TIMEOUT_SECONDS)
+        payload = await asyncio.to_thread(json.loads, raw_message)
+        if not isinstance(payload, dict) or payload.get('type') != 'authenticate':
+            raise ValueError('Authentication is required')
+        token = str(payload.get('token') or '')
+        if not self._verify_session_token_value(token, runtime_bot):
+            raise ValueError('Authentication is required')
+
+    async def _assert_execution_active(self, runtime_bot) -> None:
+        context = runtime_bot.execution_context
+        await self.ap.workspace_service.get_execution_binding(
+            context.workspace_uuid,
+            expected_generation=context.placement_generation,
+        )
+
+    async def _resolve_connected_bot(self, owner_bot, pipeline_uuid: str):
+        """Re-resolve mutable bot state before every public message."""
+        current_bot, current_pipeline_uuid = await self._resolve_bot(owner_bot.bot_entity.uuid)
+        if current_bot is None or current_pipeline_uuid != pipeline_uuid:
+            raise RuntimeError('Bot is unavailable')
+
+        owner_context = owner_bot.execution_context
+        current_context = current_bot.execution_context
+        if (
+            current_context.instance_uuid,
+            current_context.workspace_uuid,
+            current_context.placement_generation,
+        ) != (
+            owner_context.instance_uuid,
+            owner_context.workspace_uuid,
+            owner_context.placement_generation,
+        ):
+            raise RuntimeError('Bot is unavailable')
+        await self._assert_execution_active(current_bot)
+        return current_bot
+
    # -- routes --------------------------------------------------------------

    async def initialize(self) -> None:
@@ -106,7 +146,7 @@ class EmbedRouterGroup(group.RouterGroup):
        async def verify_turnstile(bot_uuid: str) -> str:
            if not _is_valid_uuid(bot_uuid):
                return self.http_status(400, -1, 'Invalid bot_uuid format')
-            runtime_bot, pipeline_uuid = self._resolve_bot(bot_uuid)
+            runtime_bot, pipeline_uuid = await self._resolve_bot(bot_uuid)
            if runtime_bot is None:
                return self.http_status(404, -1, 'Bot not found or not available')
            try:
@@ -115,18 +155,18 @@ class EmbedRouterGroup(group.RouterGroup):
                if not token:
                    return self.http_status(400, -1, 'Token is required')

-                config = self._get_bot_config(bot_uuid)
+                config = self._get_bot_config(runtime_bot)
                secret = config.get('turnstile_secret_key', '')
                if not secret:
                    ts = time.time()
                    return self.success(data={'token': f'{ts}.dummy'})

-                async with httpx.AsyncClient() as client:
+                async with httpx.AsyncClient(event_hooks=httpclient.httpx_response_limit_hooks()) as client:
                    resp = await client.post(
                        'https://challenges.cloudflare.com/turnstile/v0/siteverify',
                        data={'secret': secret, 'response': token},
                    )
-                    result = resp.json()
+                    result = await httpclient.parse_json_response(resp)

                if not result.get('success'):
                    return self.http_status(403, -1, 'Turnstile verification failed')
@@ -146,7 +186,7 @@ class EmbedRouterGroup(group.RouterGroup):
            """Serve the embed widget JavaScript with injected configuration."""
            if not _is_valid_uuid(bot_uuid):
                return self.http_status(400, -1, 'Invalid bot_uuid format')
-            runtime_bot, pipeline_uuid = self._resolve_bot(bot_uuid)
+            runtime_bot, pipeline_uuid = await self._resolve_bot(bot_uuid)
            if runtime_bot is None:
                return quart.Response(
                    '// Bot not found or not available', status=404, content_type='application/javascript'
@@ -164,7 +204,7 @@ class EmbedRouterGroup(group.RouterGroup):
            if not re.match(r'^https?://[a-zA-Z0-9._:/-]+$', base_url):
                base_url = quart.request.host_url.rstrip('/')

-            config = self._get_bot_config(bot_uuid)
+            config = self._get_bot_config(runtime_bot)
            site_key = config.get('turnstile_site_key', '')
            locale = config.get('language', 'en_US') or 'en_US'
            bubble_icon = config.get('bubble_icon', 'logo') or 'logo'
@@ -194,10 +234,10 @@ class EmbedRouterGroup(group.RouterGroup):
        async def get_embed_messages(bot_uuid: str, session_type: str) -> str:
            if not _is_valid_uuid(bot_uuid):
                return self.http_status(400, -1, 'Invalid bot_uuid format')
-            runtime_bot, pipeline_uuid = self._resolve_bot(bot_uuid)
+            runtime_bot, pipeline_uuid = await self._resolve_bot(bot_uuid)
            if runtime_bot is None:
                return self.http_status(404, -1, 'Bot not found or not available')
-            if not await self._verify_session_token(quart.request, bot_uuid):
+            if not await self._verify_session_token(quart.request, runtime_bot):
                return self.http_status(403, -1, 'Unauthorized or session expired')
            try:
                if session_type not in ['person', 'group']:
@@ -207,7 +247,8 @@ class EmbedRouterGroup(group.RouterGroup):
                if not is_valid_session_id(session_id):
                    return self.http_status(400, -1, 'Valid session_id is required')

-                websocket_adapter = self.ap.platform_mgr.websocket_proxy_bot.adapter
+                proxy_bot = await self.ap.platform_mgr.get_websocket_proxy_bot(runtime_bot.execution_context)
+                websocket_adapter = proxy_bot.adapter
                if not websocket_adapter:
                    return self.http_status(404, -1, 'WebSocket adapter not found')

@@ -222,10 +263,10 @@ class EmbedRouterGroup(group.RouterGroup):
        async def reset_embed_session(bot_uuid: str, session_type: str) -> str:
            if not _is_valid_uuid(bot_uuid):
                return self.http_status(400, -1, 'Invalid bot_uuid format')
-            runtime_bot, pipeline_uuid = self._resolve_bot(bot_uuid)
+            runtime_bot, pipeline_uuid = await self._resolve_bot(bot_uuid)
            if runtime_bot is None:
                return self.http_status(404, -1, 'Bot not found or not available')
-            if not await self._verify_session_token(quart.request, bot_uuid):
+            if not await self._verify_session_token(quart.request, runtime_bot):
                return self.http_status(403, -1, 'Unauthorized or session expired')
            try:
                if session_type not in ['person', 'group']:
@@ -235,7 +276,8 @@ class EmbedRouterGroup(group.RouterGroup):
                if not is_valid_session_id(session_id):
                    return self.http_status(400, -1, 'Valid session_id is required')

-                websocket_adapter = self.ap.platform_mgr.websocket_proxy_bot.adapter
+                proxy_bot = await self.ap.platform_mgr.get_websocket_proxy_bot(runtime_bot.execution_context)
+                websocket_adapter = proxy_bot.adapter
                if not websocket_adapter:
                    return self.http_status(404, -1, 'WebSocket adapter not found')

@@ -250,10 +292,10 @@ class EmbedRouterGroup(group.RouterGroup):
        async def submit_feedback(bot_uuid: str) -> str:
            if not _is_valid_uuid(bot_uuid):
                return self.http_status(400, -1, 'Invalid bot_uuid format')
-            runtime_bot, pipeline_uuid = self._resolve_bot(bot_uuid)
+            runtime_bot, pipeline_uuid = await self._resolve_bot(bot_uuid)
            if runtime_bot is None:
                return self.http_status(404, -1, 'Bot not found or not available')
-            if not await self._verify_session_token(quart.request, bot_uuid):
+            if not await self._verify_session_token(quart.request, runtime_bot):
                return self.http_status(403, -1, 'Unauthorized or session expired')
            try:
                data = await quart.request.get_json()
@@ -266,6 +308,7 @@ class EmbedRouterGroup(group.RouterGroup):
                feedback_id = f'embed_{uuid.uuid4().hex[:12]}'

                await self.ap.monitoring_service.record_feedback(
+                    runtime_bot.execution_context,
                    feedback_id=feedback_id,
                    feedback_type=feedback_type,
                    bot_id=runtime_bot.bot_entity.uuid,
@@ -286,11 +329,12 @@ class EmbedRouterGroup(group.RouterGroup):
        @self.quart_app.websocket(self.path + '/<bot_uuid>/ws/connect')
        async def embed_websocket_connect(bot_uuid: str):
            """WebSocket connection for embed widget, keyed by bot_uuid."""
+            await quart.websocket.accept()
            if not _is_valid_uuid(bot_uuid):
                await quart.websocket.send(json.dumps({'type': 'error', 'message': 'Invalid bot_uuid format'}))
                return

-            runtime_bot, pipeline_uuid = self._resolve_bot(bot_uuid)
+            runtime_bot, pipeline_uuid = await self._resolve_bot(bot_uuid)
            if runtime_bot is None:
                await quart.websocket.send(json.dumps({'type': 'error', 'message': 'Bot not found or not available'}))
                return
@@ -307,18 +351,42 @@ class EmbedRouterGroup(group.RouterGroup):
                await quart.websocket.send(json.dumps({'type': 'error', 'message': 'Valid session_id is required'}))
                return

-            websocket_adapter = self.ap.platform_mgr.websocket_proxy_bot.adapter
-            if not websocket_adapter:
-                await quart.websocket.send(json.dumps({'type': 'error', 'message': 'WebSocket adapter not found'}))
+            try:
+                await self._authenticate_websocket(runtime_bot)
+                await self._assert_execution_active(runtime_bot)
+            except Exception:
+                await quart.websocket.send(json.dumps({'type': 'error', 'message': 'Unauthorized'}))
                return

            try:
+                proxy_bot = await self.ap.platform_mgr.get_websocket_proxy_bot(runtime_bot.execution_context)
+                websocket_adapter = proxy_bot.adapter
+                if not websocket_adapter:
+                    await quart.websocket.send(json.dumps({'type': 'error', 'message': 'WebSocket adapter not found'}))
+                    return
+
                connection = await ws_connection_manager.add_connection(
                    websocket=quart.websocket._get_current_object(),
+                    scope=WebSocketScope.from_context(runtime_bot.execution_context),
                    pipeline_uuid=pipeline_uuid,
                    session_type=session_type,
                    session_id=session_id,
                    metadata={'user_agent': quart.websocket.headers.get('User-Agent', '')},
+                    send_queue_size=(
+                        self.ap.instance_config.data.get('system', {})
+                        .get('websocket_retention', {})
+                        .get('send_queue_size', 100)
+                    ),
+                    max_connections=(
+                        self.ap.instance_config.data.get('system', {})
+                        .get('websocket_retention', {})
+                        .get('max_connections', 1024)
+                    ),
+                    max_connections_per_workspace=(
+                        self.ap.instance_config.data.get('system', {})
+                        .get('websocket_retention', {})
+                        .get('max_connections_per_workspace', 32)
+                    ),
                )

                await quart.websocket.send(
@@ -338,11 +406,19 @@ class EmbedRouterGroup(group.RouterGroup):
                    f'(bot={bot_uuid}, pipeline={pipeline_uuid}, session_type={session_type})'
                )

-                receive_task = asyncio.create_task(self._handle_receive(connection, websocket_adapter, runtime_bot))
-                send_task = asyncio.create_task(self._handle_send(connection))
+                receive_task, send_task = create_scoped_duplex_tasks(
+                    self._handle_receive(
+                        connection,
+                        websocket_adapter,
+                        runtime_bot,
+                        pipeline_uuid,
+                    ),
+                    self._handle_send(connection),
+                    runtime_bot.execution_context.workspace_uuid,
+                )

                try:
-                    await asyncio.gather(receive_task, send_task)
+                    await wait_for_duplex_tasks(receive_task, send_task)
                except Exception as e:
                    logger.error(f'Embed WebSocket task error: {e}')
                finally:
@@ -357,14 +433,14 @@ class EmbedRouterGroup(group.RouterGroup):

    # -- WebSocket receive/send helpers --------------------------------------

-    async def _handle_receive(self, connection, websocket_adapter, owner_bot):
+    async def _handle_receive(self, connection, websocket_adapter, owner_bot, pipeline_uuid: str):
        try:
            while connection.is_active:
                message = await quart.websocket.receive()
                await ws_connection_manager.update_activity(connection.connection_id)

                try:
-                    data = json.loads(message)
+                    data = await asyncio.to_thread(json.loads, message)
                    message_type = data.get('type', 'message')

                    if message_type == 'ping':
@@ -372,7 +448,12 @@ class EmbedRouterGroup(group.RouterGroup):
                            {'type': 'pong', 'timestamp': datetime.datetime.now().isoformat()}
                        )
                    elif message_type == 'message':
-                        await websocket_adapter.handle_websocket_message(connection, data, owner_bot=owner_bot)
+                        try:
+                            current_bot = await self._resolve_connected_bot(owner_bot, pipeline_uuid)
+                        except Exception:
+                            await connection.send_queue.put({'type': 'error', 'message': 'Bot is unavailable'})
+                            break
+                        await websocket_adapter.handle_websocket_message(connection, data, owner_bot=current_bot)
                    elif message_type == 'disconnect':
                        break

@@ -383,13 +464,20 @@ class EmbedRouterGroup(group.RouterGroup):
            logger.error(f'Embed receive error: {e}', exc_info=True)
        finally:
            connection.is_active = False
+            try:
+                connection.send_queue.put_nowait(None)
+            except asyncio.QueueFull:
+                pass

    async def _handle_send(self, connection):
        try:
-            while connection.is_active:
+            while connection.is_active or not connection.send_queue.empty():
                try:
                    message = await asyncio.wait_for(connection.send_queue.get(), timeout=1.0)
-                    await quart.websocket.send(json.dumps(message))
+                    if message is None:
+                        break
+                    encoded = await asyncio.to_thread(json.dumps, message)
+                    await quart.websocket.send(encoded)
                except asyncio.TimeoutError:
                    continue
        except Exception as e: