feat(tenancy): add Workspace multi-tenant foundation (#2353)

* Document multi-tenant workspace architecture * Add OSS and commercial workspace boundaries * docs: redesign multi-tenant workspace architecture * feat(tenancy): implement workspace isolation * docs(tenancy): record verification evidence * docs(tenancy): revise single-instance SaaS topology * docs(tenancy): refine architecture options * docs: finalize cloud v2 multi-tenant decisions * feat(tenancy): establish cloud isolation foundations * feat(tenancy): harden shared cloud runtime boundaries * docs(tenancy): record final isolation verification * fix(tenancy): close isolation and permission gaps * docs(tenancy): record final isolation verification * feat(tenancy): connect cloud workspace control plane * fix(build): install git for pinned SDK * docs(cloud): update control plane verification * chore: update multi-tenant SDK pin * fix(cloud): skip legacy model sync during startup * test(cloud): preserve minimal model manager fixtures * fix(cloud): preserve authenticated account context * fix(cloud): reuse authenticated account for user info * feat(cloud): complete Workspace settings navigation * test(web): cover Workspace dropdown menu * feat(web): place workspace controls in sidebar * refactor(web): streamline workspace controls * style(web): format workspace layout test * fix(cloud): surface runtime and workspace plan status * fix(plugin): keep runtime identity stable across restarts * fix(ui): widen and center workspace switcher * fix(ui): hide roles from workspace switcher * fix(ui): align workspace switcher with sidebar entries * feat(workspace): add in-product collaboration and direct Cloud launch * style: format collaboration changes * fix(workspace): bind collaboration APIs to tenant UoW * fix(cloud): preserve Core-owned collaboration state * test(cloud): require Space identity for invite registration * feat(cloud): complete secure invitation experience * style(web): format invitation flows * fix(cloud): recover box runtime without unscoped skill reload * feat(oss): enforce invitation account and owner billing flows * style: format OSS account service * test(oss): cover invitation logout handoff * fix(oss): resolve workspace owner in scoped session * feat(cloud): harden multi-tenant runtime resources * fix(cloud): bound runtime restart storms * fix(cloud): eliminate periodic runtime CPU spikes * fix(cloud): enforce instance capacity ceilings * fix(cloud): scope public login capability discovery * fix(cloud): bound tenant maintenance and monitoring work * fix(runtime): bound tenant resource amplification * fix(deps): pin green multi-tenant plugin SDK * fix(cloud): handle unavailable skill capability * fix(security): require authentication for image file endpoint (H-2) - Changed /api/v1/files/image from AuthType.NONE to USER_TOKEN_OR_API_KEY - Added Permission.RESOURCE_VIEW requirement - Prevents unauthenticated cross-tenant file access via leaked keys - Fixes HIGH severity finding from multi-tenant security review docs: add comprehensive database migration guide - Complete migration steps for OSS → multi-tenant - Backup, execution, verification procedures - Rollback scenarios and recovery plans - Performance tuning recommendations * test: add comprehensive cross-tenant isolation tests Added 7 critical test scenarios for multi-tenant boundaries: - Cross-tenant bot access prevention - Viewer role read-only enforcement - Removed member immediate access revocation - Model provider credential isolation - WebSocket message isolation - Invitation token workspace scoping - Multi-workspace context validation These tests address P0-2 coverage gaps for: - workspaces.py (membership & invitation flows) - user.py (authentication & authorization) - websocket_chat.py (real-time isolation) - plugins.py (resource access control) docs: finalize database migration guide * fix(security): resolve M-1, M-2, M-3 security findings M-1: WebSocket authorization TOCTOU race (FIXED) - Changed _revalidate_websocket_authorization to return RequestContext - Ensures validated context is used immediately without race window - Prevents removed members from sending messages during revalidation gap M-2: Model Manager cache workspace isolation (VERIFIED) - Confirmed _CacheKey already uses 4-tuple: (instance, workspace, generation, resource) - Cache is properly scoped per workspace, no cross-tenant leakage possible - No code change needed, documented as working correctly M-3: Invitation lock workspace scoping (FIXED) - Changed lock key from token_digest to workspace_uuid:token_digest - Prevents DoS where attacker locks token in Workspace A to block Workspace B - Locks now isolated per workspace All MEDIUM severity findings from security review now resolved. * fix(cloud): unblock tenant CI and enforce knowledge quotas * fix(tenancy): scope rerank model sync --------- Co-authored-by: dadachann <185672915+dadachann@users.noreply.github.com>
2026-08-09 12:40:59 +00:00 · 2026-07-30 21:43:35 +08:00
parent 463b120923
commit e1ac5e0fc8
468 changed files with 78320 additions and 13137 deletions
@@ -6,6 +6,9 @@ import sqlalchemy

 from ....core import app
 from ....entity.persistence import pipeline as persistence_pipeline
+from ....workspace.errors import WorkspaceNotFoundError
+from .secrets import contains_secret_placeholder, redact_secrets, restore_secret_placeholders
+from .tenant import TenantContext, require_workspace_uuid, scope_statement


 default_stage_order = [
@@ -30,7 +33,8 @@ class PipelineService:
    def __init__(self, ap: app.Application) -> None:
        self.ap = ap

-    async def get_pipeline_metadata(self) -> list[dict]:
+    async def get_pipeline_metadata(self, context: TenantContext) -> list[dict]:
+        require_workspace_uuid(context)
        return [
            self.ap.pipeline_config_meta_trigger,
            self.ap.pipeline_config_meta_safety,
@@ -38,8 +42,19 @@ class PipelineService:
            self.ap.pipeline_config_meta_output,
        ]

-    async def get_pipelines(self, sort_by: str = 'created_at', sort_order: str = 'DESC') -> list[dict]:
-        query = sqlalchemy.select(persistence_pipeline.LegacyPipeline)
+    async def get_pipelines(
+        self,
+        context: TenantContext,
+        sort_by: str = 'created_at',
+        sort_order: str = 'DESC',
+        *,
+        include_secret: bool = False,
+    ) -> list[dict]:
+        query = scope_statement(
+            sqlalchemy.select(persistence_pipeline.LegacyPipeline),
+            persistence_pipeline.LegacyPipeline,
+            context,
+        )

        if sort_by == 'created_at':
            if sort_order == 'DESC':
@@ -54,15 +69,26 @@ class PipelineService:

        result = await self.ap.persistence_mgr.execute_async(query)
        pipelines = result.all()
-        return [
+        serialized = [
            self.ap.persistence_mgr.serialize_model(persistence_pipeline.LegacyPipeline, pipeline)
            for pipeline in pipelines
        ]
+        return serialized if include_secret else [redact_secrets(pipeline) for pipeline in serialized]

-    async def get_pipeline(self, pipeline_uuid: str) -> dict | None:
+    async def get_pipeline(
+        self,
+        context: TenantContext,
+        pipeline_uuid: str,
+        *,
+        include_secret: bool = False,
+    ) -> dict | None:
        result = await self.ap.persistence_mgr.execute_async(
-            sqlalchemy.select(persistence_pipeline.LegacyPipeline).where(
-                persistence_pipeline.LegacyPipeline.uuid == pipeline_uuid
+            scope_statement(
+                sqlalchemy.select(persistence_pipeline.LegacyPipeline).where(
+                    persistence_pipeline.LegacyPipeline.uuid == pipeline_uuid
+                ),
+                persistence_pipeline.LegacyPipeline,
+                context,
            )
        )

@@ -71,20 +97,24 @@ class PipelineService:
        if pipeline is None:
            return None

-        return self.ap.persistence_mgr.serialize_model(persistence_pipeline.LegacyPipeline, pipeline)
+        serialized = self.ap.persistence_mgr.serialize_model(persistence_pipeline.LegacyPipeline, pipeline)
+        return serialized if include_secret else redact_secrets(serialized)

-    async def create_pipeline(self, pipeline_data: dict, default: bool = False) -> str:
+    async def create_pipeline(self, context: TenantContext, pipeline_data: dict, default: bool = False) -> str:
        from ....utils import paths as path_utils

+        workspace_uuid = require_workspace_uuid(context)
        # Check limitation
        limitation = self.ap.instance_config.data.get('system', {}).get('limitation', {})
        max_pipelines = limitation.get('max_pipelines', -1)
        if max_pipelines >= 0:
-            existing_pipelines = await self.get_pipelines()
+            existing_pipelines = await self.get_pipelines(context)
            if len(existing_pipelines) >= max_pipelines:
                raise ValueError(f'Maximum number of pipelines ({max_pipelines}) reached')

+        pipeline_data = pipeline_data.copy()
        pipeline_data['uuid'] = str(uuid.uuid4())
+        pipeline_data['workspace_uuid'] = workspace_uuid
        pipeline_data['for_version'] = self.ap.ver_mgr.get_current_version()
        pipeline_data['stages'] = default_stage_order.copy()
        pipeline_data['is_default'] = default
@@ -108,79 +138,122 @@ class PipelineService:
            sqlalchemy.insert(persistence_pipeline.LegacyPipeline).values(**pipeline_data)
        )

-        pipeline = await self.get_pipeline(pipeline_data['uuid'])
+        pipeline = await self.get_pipeline(context, pipeline_data['uuid'], include_secret=True)

-        await self.ap.pipeline_mgr.load_pipeline(pipeline)
+        await self.ap.pipeline_mgr.load_pipeline(context, pipeline)

        return pipeline_data['uuid']

-    async def update_pipeline(self, pipeline_uuid: str, pipeline_data: dict) -> None:
+    async def update_pipeline(self, context: TenantContext, pipeline_uuid: str, pipeline_data: dict) -> None:
+        workspace_uuid = require_workspace_uuid(context)
        pipeline_data = pipeline_data.copy()
-        for protected_field in ('uuid', 'for_version', 'stages', 'is_default'):
+        for protected_field in ('uuid', 'workspace_uuid', 'for_version', 'stages', 'is_default'):
            pipeline_data.pop(protected_field, None)

-        await self.ap.persistence_mgr.execute_async(
-            sqlalchemy.update(persistence_pipeline.LegacyPipeline)
-            .where(persistence_pipeline.LegacyPipeline.uuid == pipeline_uuid)
-            .values(**pipeline_data)
-        )
+        if 'config' in pipeline_data:
+            current_config = None
+            if contains_secret_placeholder(pipeline_data['config']):
+                current_pipeline = await self.get_pipeline(context, pipeline_uuid, include_secret=True)
+                if current_pipeline is None:
+                    raise WorkspaceNotFoundError('Pipeline not found')
+                current_config = current_pipeline.get('config', {})
+            pipeline_data['config'] = restore_secret_placeholders(
+                pipeline_data['config'],
+                current_config if current_config is not None else {},
+            )

-        pipeline = await self.get_pipeline(pipeline_uuid)
+        result = await self.ap.persistence_mgr.execute_async(
+            scope_statement(
+                sqlalchemy.update(persistence_pipeline.LegacyPipeline)
+                .where(persistence_pipeline.LegacyPipeline.uuid == pipeline_uuid)
+                .values(**pipeline_data),
+                persistence_pipeline.LegacyPipeline,
+                workspace_uuid,
+            )
+        )
+        if getattr(result, 'rowcount', None) == 0:
+            raise WorkspaceNotFoundError('Pipeline not found')
+
+        pipeline = await self.get_pipeline(context, pipeline_uuid, include_secret=True)
+        if pipeline is None:
+            raise WorkspaceNotFoundError('Pipeline not found')

        if 'name' in pipeline_data:
            from ....entity.persistence import bot as persistence_bot

            result = await self.ap.persistence_mgr.execute_async(
-                sqlalchemy.select(persistence_bot.Bot).where(persistence_bot.Bot.use_pipeline_uuid == pipeline_uuid)
+                scope_statement(
+                    sqlalchemy.select(persistence_bot.Bot).where(
+                        persistence_bot.Bot.use_pipeline_uuid == pipeline_uuid
+                    ),
+                    persistence_bot.Bot,
+                    workspace_uuid,
+                )
            )

            bots = result.all()

            for bot in bots:
                bot_data = {'use_pipeline_name': pipeline_data['name']}
-                await self.ap.bot_service.update_bot(bot.uuid, bot_data)
+                await self.ap.bot_service.update_bot(context, bot.uuid, bot_data)

-        await self.ap.pipeline_mgr.remove_pipeline(pipeline_uuid)
-        await self.ap.pipeline_mgr.load_pipeline(pipeline)
+        await self.ap.pipeline_mgr.remove_pipeline(context, pipeline_uuid)
+        await self.ap.pipeline_mgr.load_pipeline(context, pipeline)

        # update all conversation that use this pipeline
        for session in self.ap.sess_mgr.session_list:
-            if session.using_conversation is not None and session.using_conversation.pipeline_uuid == pipeline_uuid:
+            if (
+                session.using_conversation is not None
+                and session.using_conversation.pipeline_uuid == pipeline_uuid
+                and getattr(session, 'workspace_uuid', workspace_uuid) == workspace_uuid
+            ):
                session.using_conversation = None

-    async def delete_pipeline(self, pipeline_uuid: str) -> None:
-        await self.ap.persistence_mgr.execute_async(
-            sqlalchemy.delete(persistence_pipeline.LegacyPipeline).where(
-                persistence_pipeline.LegacyPipeline.uuid == pipeline_uuid
+    async def delete_pipeline(self, context: TenantContext, pipeline_uuid: str) -> None:
+        result = await self.ap.persistence_mgr.execute_async(
+            scope_statement(
+                sqlalchemy.delete(persistence_pipeline.LegacyPipeline).where(
+                    persistence_pipeline.LegacyPipeline.uuid == pipeline_uuid
+                ),
+                persistence_pipeline.LegacyPipeline,
+                context,
            )
        )
-        await self.ap.pipeline_mgr.remove_pipeline(pipeline_uuid)
+        if getattr(result, 'rowcount', None) == 0:
+            raise WorkspaceNotFoundError('Pipeline not found')
+        await self.ap.pipeline_mgr.remove_pipeline(context, pipeline_uuid)

-    async def copy_pipeline(self, pipeline_uuid: str) -> str:
+    async def copy_pipeline(self, context: TenantContext, pipeline_uuid: str) -> str:
        """Copy a pipeline with all its configurations"""
+        workspace_uuid = require_workspace_uuid(context)
        # Check limitation
        limitation = self.ap.instance_config.data.get('system', {}).get('limitation', {})
        max_pipelines = limitation.get('max_pipelines', -1)
        if max_pipelines >= 0:
-            existing_pipelines = await self.get_pipelines()
+            existing_pipelines = await self.get_pipelines(context)
            if len(existing_pipelines) >= max_pipelines:
                raise ValueError(f'Maximum number of pipelines ({max_pipelines}) reached')

        # Get the original pipeline
        result = await self.ap.persistence_mgr.execute_async(
-            sqlalchemy.select(persistence_pipeline.LegacyPipeline).where(
-                persistence_pipeline.LegacyPipeline.uuid == pipeline_uuid
+            scope_statement(
+                sqlalchemy.select(persistence_pipeline.LegacyPipeline).where(
+                    persistence_pipeline.LegacyPipeline.uuid == pipeline_uuid
+                ),
+                persistence_pipeline.LegacyPipeline,
+                workspace_uuid,
            )
        )

        original_pipeline = result.first()
        if original_pipeline is None:
-            raise ValueError(f'Pipeline {pipeline_uuid} not found')
+            raise WorkspaceNotFoundError(f'Pipeline {pipeline_uuid} not found')

        # Create new pipeline data
        new_uuid = str(uuid.uuid4())
        new_pipeline_data = {
            'uuid': new_uuid,
+            'workspace_uuid': workspace_uuid,
            'name': f'{original_pipeline.name} (Copy)',
            'description': original_pipeline.description,
            'for_version': self.ap.ver_mgr.get_current_version(),
@@ -207,13 +280,14 @@ class PipelineService:
        )

        # Load the new pipeline
-        pipeline = await self.get_pipeline(new_uuid)
-        await self.ap.pipeline_mgr.load_pipeline(pipeline)
+        pipeline = await self.get_pipeline(context, new_uuid, include_secret=True)
+        await self.ap.pipeline_mgr.load_pipeline(context, pipeline)

        return new_uuid

    async def update_pipeline_extensions(
        self,
+        context: TenantContext,
        pipeline_uuid: str,
        bound_plugins: list[dict],
        bound_mcp_servers: list[str] = None,
@@ -225,16 +299,21 @@ class PipelineService:
        mcp_resource_agent_read_enabled: bool | None = None,
    ) -> None:
        """Update the bound plugins and MCP servers for a pipeline"""
+        workspace_uuid = require_workspace_uuid(context)
        # Get current pipeline
        result = await self.ap.persistence_mgr.execute_async(
-            sqlalchemy.select(persistence_pipeline.LegacyPipeline).where(
-                persistence_pipeline.LegacyPipeline.uuid == pipeline_uuid
+            scope_statement(
+                sqlalchemy.select(persistence_pipeline.LegacyPipeline).where(
+                    persistence_pipeline.LegacyPipeline.uuid == pipeline_uuid
+                ),
+                persistence_pipeline.LegacyPipeline,
+                workspace_uuid,
            )
        )

        pipeline = result.first()
        if pipeline is None:
-            raise ValueError(f'Pipeline {pipeline_uuid} not found')
+            raise WorkspaceNotFoundError(f'Pipeline {pipeline_uuid} not found')

        # Update extensions_preferences
        extensions_preferences = pipeline.extensions_preferences or {}
@@ -252,12 +331,16 @@ class PipelineService:
            extensions_preferences['mcp_resources'] = bound_mcp_resources

        await self.ap.persistence_mgr.execute_async(
-            sqlalchemy.update(persistence_pipeline.LegacyPipeline)
-            .where(persistence_pipeline.LegacyPipeline.uuid == pipeline_uuid)
-            .values(extensions_preferences=extensions_preferences)
+            scope_statement(
+                sqlalchemy.update(persistence_pipeline.LegacyPipeline)
+                .where(persistence_pipeline.LegacyPipeline.uuid == pipeline_uuid)
+                .values(extensions_preferences=extensions_preferences),
+                persistence_pipeline.LegacyPipeline,
+                workspace_uuid,
+            )
        )

        # Reload pipeline to apply changes
-        await self.ap.pipeline_mgr.remove_pipeline(pipeline_uuid)
-        pipeline = await self.get_pipeline(pipeline_uuid)
-        await self.ap.pipeline_mgr.load_pipeline(pipeline)
+        await self.ap.pipeline_mgr.remove_pipeline(context, pipeline_uuid)
+        pipeline = await self.get_pipeline(context, pipeline_uuid, include_secret=True)
+        await self.ap.pipeline_mgr.load_pipeline(context, pipeline)