feat(tenancy): add Workspace multi-tenant foundation (#2353)

* Document multi-tenant workspace architecture

* Add OSS and commercial workspace boundaries

* docs: redesign multi-tenant workspace architecture

* feat(tenancy): implement workspace isolation

* docs(tenancy): record verification evidence

* docs(tenancy): revise single-instance SaaS topology

* docs(tenancy): refine architecture options

* docs: finalize cloud v2 multi-tenant decisions

* feat(tenancy): establish cloud isolation foundations

* feat(tenancy): harden shared cloud runtime boundaries

* docs(tenancy): record final isolation verification

* fix(tenancy): close isolation and permission gaps

* docs(tenancy): record final isolation verification

* feat(tenancy): connect cloud workspace control plane

* fix(build): install git for pinned SDK

* docs(cloud): update control plane verification

* chore: update multi-tenant SDK pin

* fix(cloud): skip legacy model sync during startup

* test(cloud): preserve minimal model manager fixtures

* fix(cloud): preserve authenticated account context

* fix(cloud): reuse authenticated account for user info

* feat(cloud): complete Workspace settings navigation

* test(web): cover Workspace dropdown menu

* feat(web): place workspace controls in sidebar

* refactor(web): streamline workspace controls

* style(web): format workspace layout test

* fix(cloud): surface runtime and workspace plan status

* fix(plugin): keep runtime identity stable across restarts

* fix(ui): widen and center workspace switcher

* fix(ui): hide roles from workspace switcher

* fix(ui): align workspace switcher with sidebar entries

* feat(workspace): add in-product collaboration and direct Cloud launch

* style: format collaboration changes

* fix(workspace): bind collaboration APIs to tenant UoW

* fix(cloud): preserve Core-owned collaboration state

* test(cloud): require Space identity for invite registration

* feat(cloud): complete secure invitation experience

* style(web): format invitation flows

* fix(cloud): recover box runtime without unscoped skill reload

* feat(oss): enforce invitation account and owner billing flows

* style: format OSS account service

* test(oss): cover invitation logout handoff

* fix(oss): resolve workspace owner in scoped session

* feat(cloud): harden multi-tenant runtime resources

* fix(cloud): bound runtime restart storms

* fix(cloud): eliminate periodic runtime CPU spikes

* fix(cloud): enforce instance capacity ceilings

* fix(cloud): scope public login capability discovery

* fix(cloud): bound tenant maintenance and monitoring work

* fix(runtime): bound tenant resource amplification

* fix(deps): pin green multi-tenant plugin SDK

* fix(cloud): handle unavailable skill capability

* fix(security): require authentication for image file endpoint (H-2)

- Changed /api/v1/files/image from AuthType.NONE to USER_TOKEN_OR_API_KEY
- Added Permission.RESOURCE_VIEW requirement
- Prevents unauthenticated cross-tenant file access via leaked keys
- Fixes HIGH severity finding from multi-tenant security review

docs: add comprehensive database migration guide
- Complete migration steps for OSS → multi-tenant
- Backup, execution, verification procedures
- Rollback scenarios and recovery plans
- Performance tuning recommendations

* test: add comprehensive cross-tenant isolation tests

Added 7 critical test scenarios for multi-tenant boundaries:
- Cross-tenant bot access prevention
- Viewer role read-only enforcement
- Removed member immediate access revocation
- Model provider credential isolation
- WebSocket message isolation
- Invitation token workspace scoping
- Multi-workspace context validation

These tests address P0-2 coverage gaps for:
- workspaces.py (membership & invitation flows)
- user.py (authentication & authorization)
- websocket_chat.py (real-time isolation)
- plugins.py (resource access control)

docs: finalize database migration guide

* fix(security): resolve M-1, M-2, M-3 security findings

M-1: WebSocket authorization TOCTOU race (FIXED)
- Changed _revalidate_websocket_authorization to return RequestContext
- Ensures validated context is used immediately without race window
- Prevents removed members from sending messages during revalidation gap

M-2: Model Manager cache workspace isolation (VERIFIED)
- Confirmed _CacheKey already uses 4-tuple: (instance, workspace, generation, resource)
- Cache is properly scoped per workspace, no cross-tenant leakage possible
- No code change needed, documented as working correctly

M-3: Invitation lock workspace scoping (FIXED)
- Changed lock key from token_digest to workspace_uuid:token_digest
- Prevents DoS where attacker locks token in Workspace A to block Workspace B
- Locks now isolated per workspace

All MEDIUM severity findings from security review now resolved.

* fix(cloud): unblock tenant CI and enforce knowledge quotas

* fix(tenancy): scope rerank model sync

---------

Co-authored-by: dadachann <185672915+dadachann@users.noreply.github.com>
This commit is contained in:
RockChinQ
2026-07-30 21:43:35 +08:00
committed by GitHub
parent 463b120923
commit e1ac5e0fc8
468 changed files with 78320 additions and 13137 deletions
+317 -43
View File
@@ -5,9 +5,21 @@ import typing
import enum
import quart
import traceback
import inspect
import uuid
from quart.typing import RouteCallable
from ....core import app
from ....utils import constants
from ....utils import bounded_executor
from ....workspace.collaboration import MembershipPermissionError, WorkspaceCollaborationError
from ....workspace.errors import WorkspaceNotFoundError
from ....cloud.entitlements import EntitlementUnavailableError
from ....core.errors import TaskCapacityError
from ..authz import AuthorizationError, Permission, permissions_for_role, require_permission
from ..context import PrincipalContext, PrincipalType, RequestContext, WorkspaceContext
if typing.TYPE_CHECKING:
from ....core.app import Application
# Maximum file upload size limit (10MB)
MAX_FILE_SIZE = 10 * 1024 * 1024 # 10MB
@@ -33,6 +45,7 @@ class AuthType(enum.Enum):
"""Authentication type"""
NONE = 'none'
ACCOUNT_TOKEN = 'account-token'
USER_TOKEN = 'user-token'
API_KEY = 'api-key'
USER_TOKEN_OR_API_KEY = 'user-token-or-api-key'
@@ -43,11 +56,11 @@ class RouterGroup(abc.ABC):
path: str
ap: app.Application
ap: Application
quart_app: quart.Quart
def __init__(self, ap: app.Application, quart_app: quart.Quart) -> None:
def __init__(self, ap: Application, quart_app: quart.Quart) -> None:
self.ap = ap
self.quart_app = quart_app
@@ -59,16 +72,38 @@ class RouterGroup(abc.ABC):
self,
rule: str,
auth_type: AuthType = AuthType.USER_TOKEN,
permission: Permission | str | None = None,
**options: typing.Any,
) -> typing.Callable[[RouteCallable], RouteCallable]: # decorator
"""Register a route"""
if auth_type == AuthType.ACCOUNT_TOKEN and permission is not None:
raise ValueError('Account-token routes cannot declare Workspace permissions')
def decorator(f: RouteCallable) -> RouteCallable:
nonlocal rule
rule = self.path + rule
async def handler_error(*args, **kwargs):
if auth_type == AuthType.USER_TOKEN:
request_context: RequestContext | None = None
if auth_type == AuthType.ACCOUNT_TOKEN:
authorization = quart.request.headers.get('Authorization', '')
if not authorization.startswith('Bearer '):
return self.http_status(401, -1, 'No valid user token provided')
token = authorization.removeprefix('Bearer ')
if not token:
return self.http_status(401, -1, 'No valid user token provided')
try:
account, user_email = await self._authenticate_account(token)
# Account-token routes deliberately stop before Workspace
# selection. They may bootstrap a selector, but cannot
# receive RequestContext or enforce Workspace permissions.
self._inject_handler_context(f, kwargs, user_email, None, account)
except Exception as e:
return self._auth_error_response(e)
elif auth_type == AuthType.USER_TOKEN:
# get token from Authorization header
token = quart.request.headers.get('Authorization', '').replace('Bearer ', '')
@@ -76,18 +111,15 @@ class RouterGroup(abc.ABC):
return self.http_status(401, -1, 'No valid user token provided')
try:
user_email = await self.ap.user_service.verify_jwt_token(token)
# check if this account exists
user = await self.ap.user_service.get_user_by_email(user_email)
if not user:
return self.http_status(401, -1, 'User not found')
# check if f accepts user_email parameter
if 'user_email' in f.__code__.co_varnames:
kwargs['user_email'] = user_email
account, user_email = await self._authenticate_account(token)
request_context = await self._resolve_account_context(account, auth_type)
if permission is not None:
if request_context is None:
raise AuthorizationError('Workspace authorization is unavailable')
require_permission(request_context, permission)
self._inject_handler_context(f, kwargs, user_email, request_context)
except Exception as e:
return self.http_status(401, -1, str(e))
return self._auth_error_response(e)
elif auth_type == AuthType.API_KEY:
# get API key from Authorization header or X-API-Key header
@@ -101,11 +133,12 @@ class RouterGroup(abc.ABC):
return self.http_status(401, -1, 'No valid API key provided')
try:
is_valid = await self.ap.apikey_service.verify_api_key(api_key)
if not is_valid:
return self.http_status(401, -1, 'Invalid API key')
request_context = await self._authenticate_api_key(api_key, auth_type)
if permission is not None:
require_permission(request_context, permission)
self._inject_handler_context(f, kwargs, None, request_context)
except Exception as e:
return self.http_status(401, -1, str(e))
return self._auth_error_response(e)
elif auth_type == AuthType.USER_TOKEN_OR_API_KEY:
# Try API key first (check X-API-Key header)
@@ -114,11 +147,12 @@ class RouterGroup(abc.ABC):
if api_key:
# API key authentication
try:
is_valid = await self.ap.apikey_service.verify_api_key(api_key)
if not is_valid:
return self.http_status(401, -1, 'Invalid API key')
request_context = await self._authenticate_api_key(api_key, auth_type)
if permission is not None:
require_permission(request_context, permission)
self._inject_handler_context(f, kwargs, None, request_context)
except Exception as e:
return self.http_status(401, -1, str(e))
return self._auth_error_response(e)
else:
# Try user token authentication (Authorization header)
token = quart.request.headers.get('Authorization', '').replace('Bearer ', '')
@@ -129,35 +163,89 @@ class RouterGroup(abc.ABC):
)
try:
user_email = await self.ap.user_service.verify_jwt_token(token)
# check if this account exists
user = await self.ap.user_service.get_user_by_email(user_email)
if not user:
return self.http_status(401, -1, 'User not found')
# check if f accepts user_email parameter
if 'user_email' in f.__code__.co_varnames:
kwargs['user_email'] = user_email
account, user_email = await self._authenticate_account(token)
request_context = await self._resolve_account_context(account, auth_type)
if permission is not None:
if request_context is None:
raise AuthorizationError('Workspace authorization is unavailable')
require_permission(request_context, permission)
self._inject_handler_context(f, kwargs, user_email, request_context)
except (AuthorizationError, WorkspaceNotFoundError, MembershipPermissionError) as e:
# Authentication succeeded and authorization was
# evaluated. Do not reinterpret a denied user token
# as an API key, which would mask the stable 403/404.
return self._auth_error_response(e)
except Exception:
# If user token fails, maybe it's an API key in Authorization header
try:
is_valid = await self.ap.apikey_service.verify_api_key(token)
if not is_valid:
return self.http_status(401, -1, 'Invalid authentication credentials')
request_context = await self._authenticate_api_key(token, auth_type)
if permission is not None:
require_permission(request_context, permission)
self._inject_handler_context(f, kwargs, None, request_context)
except Exception as e:
return self.http_status(401, -1, str(e))
return self._auth_error_response(e)
try:
if request_context is not None:
with bounded_executor.blocking_work_scope(request_context.workspace_uuid):
persistence_mgr = getattr(
self.ap,
'persistence_mgr',
None,
)
tenant_scope_descriptor = getattr(
type(persistence_mgr),
'tenant_scope',
None,
)
if callable(tenant_scope_descriptor):
# Authorization discovery is complete. Carry
# the trusted Workspace identity across the
# handler, but do not reserve a database
# connection while it waits on providers,
# runtimes, uploads, or streamed clients.
# Services that need atomic writes open a UoW.
async with persistence_mgr.tenant_scope(request_context.workspace_uuid):
return await f(*args, **kwargs)
return await f(*args, **kwargs)
return await f(*args, **kwargs)
except Exception as e: # 自动 500
traceback.print_exc()
# return self.http_status(500, -2, str(e))
return self.http_status(500, -2, str(e))
if isinstance(e, AuthorizationError):
return self.http_status(e.status_code, e.error_code, str(e))
if isinstance(e, WorkspaceNotFoundError):
return self.http_status(404, 'resource_not_found', 'Resource not found')
if isinstance(e, MembershipPermissionError):
return self.http_status(403, e.code, str(e))
if isinstance(e, WorkspaceCollaborationError):
return self.http_status(400, e.code, str(e))
if isinstance(e, TaskCapacityError):
return self.http_status(429, 'task_capacity_exceeded', str(e))
if isinstance(
e,
bounded_executor.BlockingWorkCapacityError,
):
return self.http_status(
429,
'blocking_work_capacity_exceeded',
str(e),
)
request_id = self.request_id()
logger = getattr(self.ap, 'logger', self.quart_app.logger)
logger.error(
f'Unhandled HTTP error request_id={request_id} '
f'method={quart.request.method} path={quart.request.path}\n{traceback.format_exc()}'
)
return self.internal_error_response(request_id)
new_f = handler_error
new_f.__name__ = (self.name + rule).replace('/', '__')
# Quart/Flask requires a unique endpoint name even when the same URL
# intentionally has separate handlers for different HTTP methods.
# Include the method set so CRUD routes can declare distinct
# permissions without colliding during application startup.
methods = options.get('methods') or ['GET']
method_suffix = '__'.join(sorted(str(method).upper() for method in methods))
new_f.__name__ = (self.name + rule + '__' + method_suffix).replace('/', '__')
new_f.__doc__ = f.__doc__
self.quart_app.route(rule, **options)(new_f)
@@ -165,6 +253,192 @@ class RouterGroup(abc.ABC):
return decorator
async def _authenticate_account(self, token: str) -> tuple[typing.Any, str]:
account: typing.Any = None
resolver = getattr(self.ap.user_service, 'get_authenticated_account', None)
if callable(resolver):
resolved = resolver(token)
if inspect.isawaitable(resolved):
account = await resolved
if isinstance(account, str) or account is None:
user_email = account or await self.ap.user_service.verify_jwt_token(token)
account = await self.ap.user_service.get_user_by_email(user_email)
if account is None:
raise ValueError('User not found')
return account, account.user
async def _resolve_account_context(
self,
account: typing.Any,
auth_type: AuthType,
) -> RequestContext | None:
collaboration_service = getattr(self.ap, 'workspace_collaboration_service', None)
account_uuid = getattr(account, 'uuid', None)
# Compatibility for isolated controller tests that do not wire the tenancy kernel.
if collaboration_service is None or not isinstance(account_uuid, str):
return None
requested_workspace_uuid = quart.request.headers.get('X-Workspace-Id')
access = await collaboration_service.resolve_account_workspace(account_uuid, requested_workspace_uuid)
entitlement_revision = await self._resolve_entitlement_revision(
access.execution.instance_uuid,
access.workspace.uuid,
)
request_context = RequestContext(
instance_uuid=access.execution.instance_uuid,
placement_generation=access.execution.placement_generation,
request_id=self.request_id(),
auth_type=auth_type.value,
principal=PrincipalContext(
principal_type=PrincipalType.ACCOUNT,
account_uuid=account_uuid,
),
workspace=WorkspaceContext(
workspace_uuid=access.workspace.uuid,
membership_uuid=access.membership.uuid,
role=access.membership.role,
permissions=permissions_for_role(access.membership.role),
membership_revision=access.membership.projection_revision,
),
entitlement_revision=entitlement_revision,
)
quart.g.request_context = request_context
quart.g.workspace_membership = access.membership
return request_context
async def _authenticate_api_key(self, api_key: str, auth_type: AuthType) -> RequestContext:
authenticator = getattr(self.ap.apikey_service, 'authenticate_api_key', None)
if callable(authenticator):
authenticated = authenticator(api_key)
if inspect.isawaitable(authenticated):
identity = await authenticated
if identity is not None:
entitlement_revision = await self._resolve_entitlement_revision(
identity.instance_uuid,
identity.workspace_uuid,
)
request_context = RequestContext(
instance_uuid=identity.instance_uuid,
placement_generation=identity.placement_generation,
request_id=self.request_id(),
auth_type=auth_type.value,
principal=PrincipalContext(
principal_type=PrincipalType.API_KEY,
api_key_uuid=identity.api_key_uuid,
),
workspace=WorkspaceContext(
workspace_uuid=identity.workspace_uuid,
membership_uuid=None,
role=None,
permissions=identity.permissions,
),
entitlement_revision=entitlement_revision,
)
quart.g.request_context = request_context
return request_context
if not await self.ap.apikey_service.verify_api_key(api_key):
raise ValueError('Invalid API key')
workspace_service = getattr(self.ap, 'workspace_service', None)
if workspace_service is None:
raise ValueError('API key Workspace binding is unavailable')
binding = await workspace_service.get_local_execution_binding()
request_context = RequestContext(
instance_uuid=binding.instance_uuid or constants.instance_id,
placement_generation=binding.placement_generation,
request_id=self.request_id(),
auth_type=auth_type.value,
principal=PrincipalContext(
principal_type=PrincipalType.API_KEY,
api_key_uuid='legacy-oss-api-key',
),
workspace=WorkspaceContext(
workspace_uuid=binding.workspace_uuid,
membership_uuid=None,
role=None,
permissions=frozenset(item.value for item in Permission),
),
)
quart.g.request_context = request_context
return request_context
async def _resolve_entitlement_revision(self, instance_uuid: str, workspace_uuid: str) -> int:
deployment = getattr(self.ap, 'deployment', None)
if deployment is None or not getattr(deployment, 'multi_workspace_enabled', False):
return 0
resolver = getattr(self.ap, 'entitlement_resolver', None)
if resolver is None:
raise EntitlementUnavailableError('Workspace entitlement resolver is unavailable')
if instance_uuid != resolver.instance_uuid:
raise EntitlementUnavailableError('Workspace entitlement targets another LangBot instance')
snapshot = await resolver.resolve(workspace_uuid)
return snapshot.entitlement_revision
@staticmethod
def _inject_handler_context(
handler: RouteCallable,
kwargs: dict[str, typing.Any],
user_email: str | None,
request_context: RequestContext | None,
account: typing.Any = None,
) -> None:
parameters = inspect.signature(handler).parameters
if user_email is not None and 'user_email' in parameters:
kwargs['user_email'] = user_email
if account is not None and 'account' in parameters:
kwargs['account'] = account
if request_context is not None:
if 'request_context' in parameters:
kwargs['request_context'] = request_context
elif 'ctx' in parameters:
kwargs['ctx'] = request_context
def _auth_error_response(self, error: Exception) -> typing.Any:
if isinstance(error, AuthorizationError):
return self.http_status(error.status_code, error.error_code, str(error))
if isinstance(error, WorkspaceNotFoundError):
return self.http_status(404, 'resource_not_found', 'Resource not found')
if isinstance(error, MembershipPermissionError):
return self.http_status(403, error.code, str(error))
if isinstance(error, EntitlementUnavailableError):
return self.http_status(403, 'entitlement_unavailable', str(error))
request_id = self.request_id()
logger = getattr(self.ap, 'logger', self.quart_app.logger)
logger.warning(f'Authentication failed request_id={request_id} error_type={type(error).__name__}: {error}')
return self.http_status(
401,
'invalid_authentication',
'Invalid authentication credentials',
)
def request_id(self) -> str:
"""Return one stable request ID for authentication, logs, and errors."""
request_context = getattr(quart.g, 'request_context', None)
request_id = getattr(request_context, 'request_id', None) or getattr(quart.g, 'request_id', None)
if not request_id:
candidate = str(quart.request.headers.get('X-Request-Id') or '').strip()
if not candidate or len(candidate) > 128 or any(ord(char) < 32 for char in candidate):
candidate = str(uuid.uuid4())
request_id = candidate
quart.g.request_id = request_id
return str(request_id)
def internal_error_response(self, request_id: str | None = None) -> typing.Tuple[quart.Response, int]:
"""Return a stable 500 response without exposing the underlying exception."""
resolved_request_id = request_id or self.request_id()
response = quart.jsonify(
{
'code': 'internal_error',
'msg': 'Internal server error',
'request_id': resolved_request_id,
}
)
response.headers['X-Request-Id'] = resolved_request_id
return response, 500
def success(self, data: typing.Any = None) -> quart.Response:
"""Return a 200 response"""
return quart.jsonify(
@@ -175,7 +449,7 @@ class RouterGroup(abc.ABC):
}
)
def fail(self, code: int, msg: str) -> quart.Response:
def fail(self, code: int | str, msg: str) -> quart.Response:
"""Return an error response"""
return quart.jsonify(
@@ -185,6 +459,6 @@ class RouterGroup(abc.ABC):
}
)
def http_status(self, status: int, code: int, msg: str) -> typing.Tuple[quart.Response, int]:
def http_status(self, status: int, code: int | str, msg: str) -> typing.Tuple[quart.Response, int]:
"""返回一个指定状态码的响应"""
return (self.fail(code, msg), status)