feat(tenancy): add Workspace multi-tenant foundation (#2353)

* Document multi-tenant workspace architecture * Add OSS and commercial workspace boundaries * docs: redesign multi-tenant workspace architecture * feat(tenancy): implement workspace isolation * docs(tenancy): record verification evidence * docs(tenancy): revise single-instance SaaS topology * docs(tenancy): refine architecture options * docs: finalize cloud v2 multi-tenant decisions * feat(tenancy): establish cloud isolation foundations * feat(tenancy): harden shared cloud runtime boundaries * docs(tenancy): record final isolation verification * fix(tenancy): close isolation and permission gaps * docs(tenancy): record final isolation verification * feat(tenancy): connect cloud workspace control plane * fix(build): install git for pinned SDK * docs(cloud): update control plane verification * chore: update multi-tenant SDK pin * fix(cloud): skip legacy model sync during startup * test(cloud): preserve minimal model manager fixtures * fix(cloud): preserve authenticated account context * fix(cloud): reuse authenticated account for user info * feat(cloud): complete Workspace settings navigation * test(web): cover Workspace dropdown menu * feat(web): place workspace controls in sidebar * refactor(web): streamline workspace controls * style(web): format workspace layout test * fix(cloud): surface runtime and workspace plan status * fix(plugin): keep runtime identity stable across restarts * fix(ui): widen and center workspace switcher * fix(ui): hide roles from workspace switcher * fix(ui): align workspace switcher with sidebar entries * feat(workspace): add in-product collaboration and direct Cloud launch * style: format collaboration changes * fix(workspace): bind collaboration APIs to tenant UoW * fix(cloud): preserve Core-owned collaboration state * test(cloud): require Space identity for invite registration * feat(cloud): complete secure invitation experience * style(web): format invitation flows * fix(cloud): recover box runtime without unscoped skill reload * feat(oss): enforce invitation account and owner billing flows * style: format OSS account service * test(oss): cover invitation logout handoff * fix(oss): resolve workspace owner in scoped session * feat(cloud): harden multi-tenant runtime resources * fix(cloud): bound runtime restart storms * fix(cloud): eliminate periodic runtime CPU spikes * fix(cloud): enforce instance capacity ceilings * fix(cloud): scope public login capability discovery * fix(cloud): bound tenant maintenance and monitoring work * fix(runtime): bound tenant resource amplification * fix(deps): pin green multi-tenant plugin SDK * fix(cloud): handle unavailable skill capability * fix(security): require authentication for image file endpoint (H-2) - Changed /api/v1/files/image from AuthType.NONE to USER_TOKEN_OR_API_KEY - Added Permission.RESOURCE_VIEW requirement - Prevents unauthenticated cross-tenant file access via leaked keys - Fixes HIGH severity finding from multi-tenant security review docs: add comprehensive database migration guide - Complete migration steps for OSS → multi-tenant - Backup, execution, verification procedures - Rollback scenarios and recovery plans - Performance tuning recommendations * test: add comprehensive cross-tenant isolation tests Added 7 critical test scenarios for multi-tenant boundaries: - Cross-tenant bot access prevention - Viewer role read-only enforcement - Removed member immediate access revocation - Model provider credential isolation - WebSocket message isolation - Invitation token workspace scoping - Multi-workspace context validation These tests address P0-2 coverage gaps for: - workspaces.py (membership & invitation flows) - user.py (authentication & authorization) - websocket_chat.py (real-time isolation) - plugins.py (resource access control) docs: finalize database migration guide * fix(security): resolve M-1, M-2, M-3 security findings M-1: WebSocket authorization TOCTOU race (FIXED) - Changed _revalidate_websocket_authorization to return RequestContext - Ensures validated context is used immediately without race window - Prevents removed members from sending messages during revalidation gap M-2: Model Manager cache workspace isolation (VERIFIED) - Confirmed _CacheKey already uses 4-tuple: (instance, workspace, generation, resource) - Cache is properly scoped per workspace, no cross-tenant leakage possible - No code change needed, documented as working correctly M-3: Invitation lock workspace scoping (FIXED) - Changed lock key from token_digest to workspace_uuid:token_digest - Prevents DoS where attacker locks token in Workspace A to block Workspace B - Locks now isolated per workspace All MEDIUM severity findings from security review now resolved. * fix(cloud): unblock tenant CI and enforce knowledge quotas * fix(tenancy): scope rerank model sync --------- Co-authored-by: dadachann <185672915+dadachann@users.noreply.github.com>
2026-08-09 12:40:59 +00:00 · 2026-07-30 21:43:35 +08:00
parent 463b120923
commit e1ac5e0fc8
468 changed files with 78320 additions and 13137 deletions
@@ -7,6 +7,7 @@ import time
 import typing
 import uuid
 import urllib.parse
+from contextlib import asynccontextmanager
 from typing import Awaitable, Callable, Optional
 import dingtalk_stream  # type: ignore
 import websockets
@@ -15,12 +16,42 @@ from .card_callback import DingTalkCardActionHandler
 from .dingtalkevent import DingTalkEvent
 import httpx
 import traceback
+from langbot.pkg.utils import httpclient


 _stdout_logger = logging.getLogger('langbot.dingtalk_api')


 DINGTALK_OPENAPI_BASE = 'https://api.dingtalk.com'
+_MAX_MEDIA_BYTES = 10 * 1024 * 1024
+_MAX_GATEWAY_MESSAGE_BYTES = 1024 * 1024
+
+
+def _read_local_media_limited(file_path: str) -> bytes:
+    if os.path.getsize(file_path) > _MAX_MEDIA_BYTES:
+        raise ValueError('DingTalk media exceeds the size limit')
+    with open(file_path, 'rb') as file:
+        body = file.read(_MAX_MEDIA_BYTES + 1)
+    if len(body) > _MAX_MEDIA_BYTES:
+        raise ValueError('DingTalk media exceeds the size limit')
+    return body
+
+
+async def _read_httpx_media_limited(response: httpx.Response) -> bytes:
+    content_length = response.headers.get('Content-Length')
+    if content_length is not None:
+        try:
+            if int(content_length) > _MAX_MEDIA_BYTES:
+                raise ValueError('DingTalk media exceeds the size limit')
+        except (TypeError, ValueError) as exc:
+            if 'exceeds' in str(exc):
+                raise
+    body = bytearray()
+    async for chunk in response.aiter_bytes():
+        body.extend(chunk)
+        if len(body) > _MAX_MEDIA_BYTES:
+            raise ValueError('DingTalk media exceeds the size limit')
+    return bytes(body)


 def _stringify_card_param_map(card_param_map: Optional[dict]) -> dict:
@@ -44,6 +75,8 @@ def _stringify_card_param_map(card_param_map: Optional[dict]) -> dict:


 class DingTalkClient:
+    _MAX_INBOUND_TASKS = 100
+
    def __init__(
        self,
        client_id: str,
@@ -86,6 +119,37 @@ class DingTalkClient:
        self.legacy_access_token = ''
        self.legacy_access_token_expiry_time: typing.Optional[float] = None
        self._stopped = False  # Flag to control the event loop
+        self._inbound_tasks: set[asyncio.Task] = set()
+        self._http_client: httpx.AsyncClient | None = None
+
+    @asynccontextmanager
+    async def _http_client_context(self):
+        """Reuse one connection pool while preserving existing call structure."""
+
+        if self._http_client is None or self._http_client.is_closed:
+            self._http_client = httpx.AsyncClient(event_hooks=httpclient.httpx_response_limit_hooks())
+        yield self._http_client
+
+    def _start_inbound_task(self, coro: typing.Coroutine) -> bool:
+        """Start one bounded inbound callback task."""
+
+        for task in tuple(self._inbound_tasks):
+            if task.done():
+                self._inbound_tasks.discard(task)
+        if len(self._inbound_tasks) >= self._MAX_INBOUND_TASKS:
+            coro.close()
+            return False
+
+        task = asyncio.create_task(coro)
+        self._inbound_tasks.add(task)
+
+        def done(done_task: asyncio.Task) -> None:
+            self._inbound_tasks.discard(done_task)
+            if not done_task.cancelled():
+                done_task.exception()
+
+        task.add_done_callback(done)
+        return True

    async def _on_card_action(self, payload: dict) -> None:
        """Dispatch a parsed card-action payload to the adapter callback."""
@@ -101,11 +165,11 @@ class DingTalkClient:
        url = 'https://api.dingtalk.com/v1.0/oauth2/accessToken'
        headers = {'Content-Type': 'application/json'}
        data = {'appKey': self.key, 'appSecret': self.secret}
-        async with httpx.AsyncClient() as client:
+        async with self._http_client_context() as client:
            try:
                response = await client.post(url, json=data, headers=headers)
                if response.status_code == 200:
-                    response_data = response.json()
+                    response_data = await httpclient.parse_json_response(response)
                    self.access_token = response_data.get('accessToken')
                    expires_in = int(response_data.get('expireIn', 7200))
                    self.access_token_expiry_time = time.time() + expires_in - 60
@@ -129,28 +193,28 @@ class DingTalkClient:
        url = 'https://api.dingtalk.com/v1.0/robot/messageFiles/download'
        params = {'downloadCode': download_code, 'robotCode': self.robot_code}
        headers = {'x-acs-dingtalk-access-token': self.access_token}
-        async with httpx.AsyncClient() as client:
+        async with self._http_client_context() as client:
            response = await client.post(url, headers=headers, json=params)
            if response.status_code == 200:
-                result = response.json()
+                result = await httpclient.parse_json_response(response)
                download_url = result.get('downloadUrl')
            else:
-                await self.logger.error(f'failed to get download url: {response.json()}')
+                error_payload = await httpclient.parse_json_response(response)
+                await self.logger.error(f'failed to get download url: {error_payload}')

        if download_url:
            return await self.download_url_to_base64(download_url)

    async def download_url_to_base64(self, download_url):
-        async with httpx.AsyncClient() as client:
-            response = await client.get(download_url)
-
-            if response.status_code == 200:
-                file_bytes = response.content
-                mime_type = response.headers.get('Content-Type', 'application/octet-stream')
-                base64_str = base64.b64encode(file_bytes).decode('utf-8')
-                return f'data:{mime_type};base64,{base64_str}'
-            else:
-                await self.logger.error(f'failed to get files: {response.json()}')
+        async with self._http_client_context() as client:
+            async with client.stream('GET', download_url) as response:
+                if response.status_code == 200:
+                    file_bytes = await _read_httpx_media_limited(response)
+                    mime_type = response.headers.get('Content-Type', 'application/octet-stream')
+                    base64_str = (await asyncio.to_thread(base64.b64encode, file_bytes)).decode('utf-8')
+                    return f'data:{mime_type};base64,{base64_str}'
+                error_body = await _read_httpx_media_limited(response)
+                await self.logger.error(f'failed to get files: {error_body[:300]!r}')

    async def get_audio_url(self, download_code: str):
        if not await self.check_access_token():
@@ -158,17 +222,19 @@ class DingTalkClient:
        url = 'https://api.dingtalk.com/v1.0/robot/messageFiles/download'
        params = {'downloadCode': download_code, 'robotCode': self.robot_code}
        headers = {'x-acs-dingtalk-access-token': self.access_token}
-        async with httpx.AsyncClient() as client:
+        async with self._http_client_context() as client:
            response = await client.post(url, headers=headers, json=params)
            if response.status_code == 200:
-                result = response.json()
+                result = await httpclient.parse_json_response(response)
                download_url = result.get('downloadUrl')
                if download_url:
                    return await self.download_url_to_base64(download_url)
                else:
-                    await self.logger.error(f'failed to get audio: {response.json()}')
+                    error_payload = await httpclient.parse_json_response(response)
+                    await self.logger.error(f'failed to get audio: {error_payload}')
            else:
-                raise Exception(f'Error: {response.status_code}, {response.text}')
+                body = await httpclient.response_text(response)
+                raise Exception(f'Error: {response.status_code}, {body}')

    async def get_file_url(self, download_code: str):
        if not await self.check_access_token():
@@ -176,17 +242,19 @@ class DingTalkClient:
        url = 'https://api.dingtalk.com/v1.0/robot/messageFiles/download'
        params = {'downloadCode': download_code, 'robotCode': self.robot_code}
        headers = {'x-acs-dingtalk-access-token': self.access_token}
-        async with httpx.AsyncClient() as client:
+        async with self._http_client_context() as client:
            response = await client.post(url, headers=headers, json=params)
            if response.status_code == 200:
-                result = response.json()
+                result = await httpclient.parse_json_response(response)
                download_url = result.get('downloadUrl')
                if download_url:
                    return download_url
                else:
-                    await self.logger.error(f'failed to get file: {response.json()}')
+                    error_payload = await httpclient.parse_json_response(response)
+                    await self.logger.error(f'failed to get file: {error_payload}')
            else:
-                raise Exception(f'Error: {response.status_code}, {response.text}')
+                body = await httpclient.response_text(response)
+                raise Exception(f'Error: {response.status_code}, {body}')

    async def update_incoming_message(self, message):
        """异步更新 DingTalkClient 中的 incoming_message"""
@@ -503,12 +571,13 @@ class DingTalkClient:
            len(content),
        )
        try:
-            async with httpx.AsyncClient() as client:
+            async with self._http_client_context() as client:
                response = await client.post(url, headers=headers, json=data)
+                response_body = await httpclient.response_text(response, max_chars=500)
                _stdout_logger.info(
                    'DingTalk send_proactive_message_to_one response: status=%d body=%s',
                    response.status_code,
-                    response.text[:500],
+                    response_body,
                )
                if response.status_code == 200:
                    return
@@ -535,7 +604,7 @@ class DingTalkClient:
            'msgParam': json.dumps({'content': content}),
        }
        try:
-            async with httpx.AsyncClient() as client:
+            async with self._http_client_context() as client:
                response = await client.post(url, headers=headers, json=data)
                if response.status_code == 200:
                    return
@@ -667,22 +736,23 @@ class DingTalkClient:
                'DingTalk createAndDeliver request body: %s',
                json.dumps(body, ensure_ascii=False)[:1500],
            )
-            async with httpx.AsyncClient() as client:
+            async with self._http_client_context() as client:
                response = await client.post(url, headers=headers, json=body, timeout=30.0)
+                response_body = await httpclient.response_text(response, max_chars=500)
                if response.status_code == 200:
                    _stdout_logger.info(
                        'DingTalk createAndDeliver response: %s',
-                        response.text[:500],
+                        response_body,
                    )
                    return True
                _stdout_logger.error(
                    'DingTalk createAndDeliver failed: status=%s body=%s',
                    response.status_code,
-                    response.text,
+                    response_body,
                )
                if self.logger:
                    await self.logger.error(
-                        f'DingTalk createAndDeliver failed: status={response.status_code} body={response.text}'
+                        f'DingTalk createAndDeliver failed: status={response.status_code} body={response_body}'
                    )
                return False
        except Exception:
@@ -725,13 +795,14 @@ class DingTalkClient:
            'Content-Type': 'application/json',
        }
        try:
-            async with httpx.AsyncClient() as client:
+            async with self._http_client_context() as client:
                response = await client.put(url, headers=headers, json=body, timeout=30.0)
                if response.status_code == 200:
                    return True
                if self.logger:
+                    response_body = await httpclient.response_text(response)
                    await self.logger.error(
-                        f'DingTalk card streaming failed: status={response.status_code} body={response.text}'
+                        f'DingTalk card streaming failed: status={response.status_code} body={response_body}'
                    )
                return False
        except Exception:
@@ -768,18 +839,19 @@ class DingTalkClient:
                out_track_id,
                json.dumps(body, ensure_ascii=False)[:1500],
            )
-            async with httpx.AsyncClient() as client:
+            async with self._http_client_context() as client:
                response = await client.put(url, headers=headers, json=body, timeout=30.0)
+                response_body = await httpclient.response_text(response, max_chars=300)
                _stdout_logger.info(
                    'DingTalk update_card_data response: status=%d body=%s',
                    response.status_code,
-                    response.text[:300],
+                    response_body,
                )
                if response.status_code == 200:
                    return True
                if self.logger:
                    await self.logger.error(
-                        f'DingTalk update card failed: status={response.status_code} body={response.text}'
+                        f'DingTalk update card failed: status={response.status_code} body={response_body}'
                    )
                return False
        except Exception:
@@ -808,17 +880,18 @@ class DingTalkClient:

        url = 'https://oapi.dingtalk.com/gettoken'
        try:
-            async with httpx.AsyncClient() as client:
+            async with self._http_client_context() as client:
                response = await client.get(url, params={'appkey': self.key, 'appsecret': self.secret}, timeout=15.0)
-            data = response.json() if response.status_code == 200 else {}
+            data = await httpclient.parse_json_response(response) if response.status_code == 200 else {}
            if data.get('errcode') == 0 and data.get('access_token'):
                self.legacy_access_token = data['access_token']
                expires_in = int(data.get('expires_in', 7200))
                self.legacy_access_token_expiry_time = now + expires_in - 60
                return self.legacy_access_token
            if self.logger:
+                response_body = await httpclient.response_text(response, max_chars=200)
                await self.logger.error(
-                    f'DingTalk legacy gettoken failed: status={response.status_code} body={response.text[:200]}'
+                    f'DingTalk legacy gettoken failed: status={response.status_code} body={response_body}'
                )
        except Exception:
            _stdout_logger.exception('DingTalk legacy gettoken error')
@@ -848,8 +921,7 @@ class DingTalkClient:

        url = 'https://oapi.dingtalk.com/media/upload'
        try:
-            with open(file_path, 'rb') as f:
-                file_bytes = f.read()
+            file_bytes = await asyncio.to_thread(_read_local_media_limited, file_path)
            file_name = os.path.basename(file_path)
            # Best-effort content-type guess; DingTalk accepts the major image
            # mime types and otherwise infers from the bytes.
@@ -857,20 +929,21 @@ class DingTalkClient:
            mime = {'png': 'image/png', 'jpg': 'image/jpeg', 'jpeg': 'image/jpeg', 'gif': 'image/gif'}.get(
                ext, 'application/octet-stream'
            )
-            async with httpx.AsyncClient() as client:
+            async with self._http_client_context() as client:
                response = await client.post(
                    url,
                    params={'access_token': token, 'type': 'image'},
                    files={'media': (file_name, file_bytes, mime)},
                    timeout=30.0,
                )
-            data = response.json() if response.status_code == 200 else {}
+            data = await httpclient.parse_json_response(response) if response.status_code == 200 else {}
            if data.get('errcode') == 0 and data.get('media_id'):
                _stdout_logger.info('DingTalk upload_image_media OK: media_id=%s', data['media_id'])
                return data['media_id']
            if self.logger:
+                response_body = await httpclient.response_text(response, max_chars=300)
                await self.logger.error(
-                    f'DingTalk upload_image_media failed: status={response.status_code} body={response.text[:300]}'
+                    f'DingTalk upload_image_media failed: status={response.status_code} body={response_body}'
                )
        except Exception:
            _stdout_logger.exception('DingTalk upload_image_media error')
@@ -897,15 +970,19 @@ class DingTalkClient:
                    continue

                uri = '%s?ticket=%s' % (connection['endpoint'], urllib.parse.quote_plus(connection['ticket']))
-                async with websockets.connect(uri) as websocket:
+                async with websockets.connect(uri, max_size=_MAX_GATEWAY_MESSAGE_BYTES) as websocket:
                    self.client.websocket = websocket
                    keepalive_task = asyncio.create_task(self._keepalive(websocket))
                    try:
                        async for raw_message in websocket:
                            if self._stopped:
                                break
-                            json_message = json.loads(raw_message)
-                            asyncio.create_task(self.client.background_task(json_message))
+                            json_message = await asyncio.to_thread(json.loads, raw_message)
+                            if not self._start_inbound_task(self.client.background_task(json_message)):
+                                if self.logger:
+                                    await self.logger.warning(
+                                        'DingTalk inbound task capacity reached; dropping message'
+                                    )
                    finally:
                        keepalive_task.cancel()
                        try:
@@ -948,5 +1025,15 @@ class DingTalkClient:
                await self.client.websocket.close()
            except Exception:
                pass
+        inbound_tasks = list(self._inbound_tasks)
+        for task in inbound_tasks:
+            if not task.done():
+                task.cancel()
+        if inbound_tasks:
+            await asyncio.gather(*inbound_tasks, return_exceptions=True)
+        self._inbound_tasks.clear()
        # Clear message handlers to prevent stale callbacks
        self._message_handlers = {'example': []}
+        if self._http_client is not None:
+            await self._http_client.aclose()
+            self._http_client = None