feat(tenancy): add Workspace multi-tenant foundation (#2353)

* Document multi-tenant workspace architecture

* Add OSS and commercial workspace boundaries

* docs: redesign multi-tenant workspace architecture

* feat(tenancy): implement workspace isolation

* docs(tenancy): record verification evidence

* docs(tenancy): revise single-instance SaaS topology

* docs(tenancy): refine architecture options

* docs: finalize cloud v2 multi-tenant decisions

* feat(tenancy): establish cloud isolation foundations

* feat(tenancy): harden shared cloud runtime boundaries

* docs(tenancy): record final isolation verification

* fix(tenancy): close isolation and permission gaps

* docs(tenancy): record final isolation verification

* feat(tenancy): connect cloud workspace control plane

* fix(build): install git for pinned SDK

* docs(cloud): update control plane verification

* chore: update multi-tenant SDK pin

* fix(cloud): skip legacy model sync during startup

* test(cloud): preserve minimal model manager fixtures

* fix(cloud): preserve authenticated account context

* fix(cloud): reuse authenticated account for user info

* feat(cloud): complete Workspace settings navigation

* test(web): cover Workspace dropdown menu

* feat(web): place workspace controls in sidebar

* refactor(web): streamline workspace controls

* style(web): format workspace layout test

* fix(cloud): surface runtime and workspace plan status

* fix(plugin): keep runtime identity stable across restarts

* fix(ui): widen and center workspace switcher

* fix(ui): hide roles from workspace switcher

* fix(ui): align workspace switcher with sidebar entries

* feat(workspace): add in-product collaboration and direct Cloud launch

* style: format collaboration changes

* fix(workspace): bind collaboration APIs to tenant UoW

* fix(cloud): preserve Core-owned collaboration state

* test(cloud): require Space identity for invite registration

* feat(cloud): complete secure invitation experience

* style(web): format invitation flows

* fix(cloud): recover box runtime without unscoped skill reload

* feat(oss): enforce invitation account and owner billing flows

* style: format OSS account service

* test(oss): cover invitation logout handoff

* fix(oss): resolve workspace owner in scoped session

* feat(cloud): harden multi-tenant runtime resources

* fix(cloud): bound runtime restart storms

* fix(cloud): eliminate periodic runtime CPU spikes

* fix(cloud): enforce instance capacity ceilings

* fix(cloud): scope public login capability discovery

* fix(cloud): bound tenant maintenance and monitoring work

* fix(runtime): bound tenant resource amplification

* fix(deps): pin green multi-tenant plugin SDK

* fix(cloud): handle unavailable skill capability

* fix(security): require authentication for image file endpoint (H-2)

- Changed /api/v1/files/image from AuthType.NONE to USER_TOKEN_OR_API_KEY
- Added Permission.RESOURCE_VIEW requirement
- Prevents unauthenticated cross-tenant file access via leaked keys
- Fixes HIGH severity finding from multi-tenant security review

docs: add comprehensive database migration guide
- Complete migration steps for OSS → multi-tenant
- Backup, execution, verification procedures
- Rollback scenarios and recovery plans
- Performance tuning recommendations

* test: add comprehensive cross-tenant isolation tests

Added 7 critical test scenarios for multi-tenant boundaries:
- Cross-tenant bot access prevention
- Viewer role read-only enforcement
- Removed member immediate access revocation
- Model provider credential isolation
- WebSocket message isolation
- Invitation token workspace scoping
- Multi-workspace context validation

These tests address P0-2 coverage gaps for:
- workspaces.py (membership & invitation flows)
- user.py (authentication & authorization)
- websocket_chat.py (real-time isolation)
- plugins.py (resource access control)

docs: finalize database migration guide

* fix(security): resolve M-1, M-2, M-3 security findings

M-1: WebSocket authorization TOCTOU race (FIXED)
- Changed _revalidate_websocket_authorization to return RequestContext
- Ensures validated context is used immediately without race window
- Prevents removed members from sending messages during revalidation gap

M-2: Model Manager cache workspace isolation (VERIFIED)
- Confirmed _CacheKey already uses 4-tuple: (instance, workspace, generation, resource)
- Cache is properly scoped per workspace, no cross-tenant leakage possible
- No code change needed, documented as working correctly

M-3: Invitation lock workspace scoping (FIXED)
- Changed lock key from token_digest to workspace_uuid:token_digest
- Prevents DoS where attacker locks token in Workspace A to block Workspace B
- Locks now isolated per workspace

All MEDIUM severity findings from security review now resolved.

* fix(cloud): unblock tenant CI and enforce knowledge quotas

* fix(tenancy): scope rerank model sync

---------

Co-authored-by: dadachann <185672915+dadachann@users.noreply.github.com>
This commit is contained in:
RockChinQ
2026-07-30 21:43:35 +08:00
committed by GitHub
parent 463b120923
commit e1ac5e0fc8
468 changed files with 78320 additions and 13137 deletions
+200 -40
View File
@@ -2,19 +2,42 @@
from __future__ import annotations
import contextvars
import io
import zipfile
from contextlib import asynccontextmanager
from types import SimpleNamespace
from unittest.mock import AsyncMock, Mock
import pytest
from langbot.pkg.api.http.context import ExecutionContext
from langbot.pkg.core.taskmgr import TaskCapacityError
from langbot.pkg.rag.knowledge.kbmgr import RuntimeKnowledgeBase
from langbot.pkg.storage.mgr import StorageMgr
from langbot.pkg.workspace.errors import WorkspaceNotFoundError
def _make_zip_bytes(entries: dict[str, bytes]) -> bytes:
WORKSPACE_A = '00000000-0000-0000-0000-00000000000a'
CONTEXT = ExecutionContext(
instance_uuid='instance-a',
workspace_uuid=WORKSPACE_A,
placement_generation=2,
)
def _upload_key(logical_key: str, *, context: ExecutionContext = CONTEXT) -> str:
return StorageMgr.scoped_object_key(
context,
owner_type='upload_document',
owner='account:test',
key=logical_key,
)
def _make_zip_bytes(entries: dict[str, bytes], *, compression: int = zipfile.ZIP_STORED) -> bytes:
buffer = io.BytesIO()
with zipfile.ZipFile(buffer, 'w') as zf:
with zipfile.ZipFile(buffer, 'w', compression=compression) as zf:
for name, content in entries.items():
zf.writestr(name, content)
zf.mkdir('emptydir')
@@ -25,26 +48,39 @@ def _make_app() -> Mock:
app = Mock()
app.logger = Mock()
app.task_mgr = Mock()
app.storage_mgr = Mock()
app.storage_mgr.storage_provider = Mock()
app.storage_mgr.storage_provider.exists = AsyncMock(return_value=True)
app.storage_mgr.storage_provider.load = AsyncMock()
app.storage_mgr.storage_provider.save = AsyncMock()
app.storage_mgr.storage_provider.size = AsyncMock(return_value=123)
app.storage_mgr.storage_provider.delete = AsyncMock()
storage_mgr = StorageMgr(app)
storage_mgr.storage_provider = Mock()
storage_mgr.storage_provider.exists = AsyncMock(return_value=True)
storage_mgr.storage_provider.load = AsyncMock()
storage_mgr.storage_provider.load_bounded = AsyncMock()
storage_mgr.storage_provider.save = AsyncMock()
storage_mgr.storage_provider.size = AsyncMock(return_value=123)
storage_mgr.storage_provider.delete = AsyncMock()
app.storage_mgr = storage_mgr
app.persistence_mgr = Mock()
app.persistence_mgr.execute_async = AsyncMock()
app.persistence_mgr.execute_async = AsyncMock(return_value=SimpleNamespace(rowcount=1))
app.plugin_connector = Mock()
app.plugin_connector.require_workspace_context = AsyncMock(side_effect=lambda context: context)
app.workspace_service = SimpleNamespace(
get_execution_binding=AsyncMock(
return_value=SimpleNamespace(
instance_uuid=CONTEXT.instance_uuid,
workspace_uuid=CONTEXT.workspace_uuid,
placement_generation=CONTEXT.placement_generation,
)
)
)
return app
def _make_kb(plugin_id: str | None = 'author/engine') -> RuntimeKnowledgeBase:
kb_entity = Mock()
kb_entity.uuid = 'test-kb-uuid'
kb_entity.workspace_uuid = WORKSPACE_A
kb_entity.collection_id = 'test-collection'
kb_entity.creation_settings = {}
kb_entity.knowledge_engine_plugin_id = plugin_id
return RuntimeKnowledgeBase(_make_app(), kb_entity)
return RuntimeKnowledgeBase(_make_app(), kb_entity, CONTEXT)
class TestStoreFile:
@@ -58,33 +94,89 @@ class TestStoreFile:
kb.ap.task_mgr.create_user_task = Mock(side_effect=create_user_task)
task_id = await kb.store_file('documents/test.pdf')
object_key = _upload_key('documents/test.pdf')
task_id = await kb.store_file(CONTEXT, object_key)
assert task_id == 'task-1'
kb.ap.storage_mgr.storage_provider.exists.assert_awaited_once_with('documents/test.pdf')
kb.ap.storage_mgr.storage_provider.exists.assert_awaited_once_with(object_key)
kb.ap.persistence_mgr.execute_async.assert_awaited_once()
call_kwargs = kb.ap.task_mgr.create_user_task.call_args.kwargs
assert call_kwargs['kind'] == 'knowledge-operation'
assert call_kwargs['name'] == 'knowledge-store-file-documents/test.pdf'
assert call_kwargs['label'] == 'Store file documents/test.pdf'
assert call_kwargs['name'] == f'knowledge-store-file-{object_key}'
assert call_kwargs['label'] == f'Store file {object_key}'
@pytest.mark.asyncio
async def test_store_file_raises_when_source_file_missing(self):
kb = _make_kb()
kb.ap.storage_mgr.storage_provider.exists = AsyncMock(return_value=False)
with pytest.raises(Exception, match='File missing.pdf not found'):
await kb.store_file('missing.pdf')
object_key = _upload_key('missing.pdf')
with pytest.raises(WorkspaceNotFoundError, match='Upload not found'):
await kb.store_file(CONTEXT, object_key)
kb.ap.persistence_mgr.execute_async.assert_not_awaited()
kb.ap.task_mgr.create_user_task.assert_not_called()
@pytest.mark.asyncio
async def test_store_file_rejects_cross_workspace_upload_key(self):
kb = _make_kb()
other_context = ExecutionContext(
instance_uuid=CONTEXT.instance_uuid,
workspace_uuid='00000000-0000-0000-0000-00000000000b',
placement_generation=CONTEXT.placement_generation,
)
with pytest.raises(WorkspaceNotFoundError, match='Upload not found'):
await kb.store_file(CONTEXT, _upload_key('stolen.pdf', context=other_context))
kb.ap.storage_mgr.storage_provider.exists.assert_not_awaited()
kb.ap.persistence_mgr.execute_async.assert_not_awaited()
@pytest.mark.asyncio
async def test_store_file_rejects_stale_generation_and_wrong_owner_type(self):
kb = _make_kb()
stale_context = ExecutionContext(
instance_uuid=CONTEXT.instance_uuid,
workspace_uuid=CONTEXT.workspace_uuid,
placement_generation=CONTEXT.placement_generation + 1,
)
plugin_key = StorageMgr.scoped_object_key(
CONTEXT,
owner_type='plugin_config',
owner='plugin:test',
key='config.pdf',
)
for object_key in (_upload_key('stale.pdf', context=stale_context), plugin_key, 'raw.pdf'):
with pytest.raises(WorkspaceNotFoundError, match='Upload not found'):
await kb.store_file(CONTEXT, object_key)
kb.ap.storage_mgr.storage_provider.exists.assert_not_awaited()
@pytest.mark.asyncio
async def test_store_file_rolls_back_pending_record_when_task_capacity_is_exhausted(self):
kb = _make_kb()
object_key = _upload_key('queued.pdf')
def reject(coro, **_kwargs):
coro.close()
raise TaskCapacityError('capacity')
kb.ap.task_mgr.create_user_task.side_effect = reject
with pytest.raises(TaskCapacityError, match='capacity'):
await kb.store_file(CONTEXT, object_key)
statements = [str(call.args[0]) for call in kb.ap.persistence_mgr.execute_async.await_args_list]
assert any(statement.startswith('INSERT') for statement in statements)
assert any(statement.startswith('DELETE') for statement in statements)
class TestStoreZipFile:
@pytest.mark.asyncio
async def test_store_zip_file_extracts_supported_files_and_skips_noise(self):
kb = _make_kb()
kb.ap.storage_mgr.storage_provider.load = AsyncMock(
kb.ap.storage_mgr.storage_provider.load_bounded = AsyncMock(
return_value=_make_zip_bytes(
{
'doc1.pdf': b'pdf',
@@ -99,63 +191,131 @@ class TestStoreZipFile:
)
kb.store_file = AsyncMock(side_effect=['task-pdf', 'task-txt', 'task-md', 'task-html'])
task_id = await kb._store_zip_file('archive.zip', parser_plugin_id='parser/plugin')
zip_key = _upload_key('archive.zip')
task_id = await kb._store_zip_file(CONTEXT, zip_key, parser_plugin_id='parser/plugin')
assert task_id == 'task-pdf'
assert kb.ap.storage_mgr.storage_provider.save.await_count == 4
saved_names = [call.args[0] for call in kb.ap.storage_mgr.storage_provider.save.await_args_list]
assert any(name.startswith('doc1_') and name.endswith('.pdf') for name in saved_names)
assert any(name.startswith('doc2_') and name.endswith('.txt') for name in saved_names)
assert any(name.startswith('subdir_doc3_') and name.endswith('.md') for name in saved_names)
assert any(name.startswith('page_') and name.endswith('.html') for name in saved_names)
assert not any('image' in name for name in saved_names)
assert not any('hidden' in name for name in saved_names)
assert not any('__MACOSX' in name for name in saved_names)
kb.ap.storage_mgr.storage_provider.delete.assert_awaited_once_with('archive.zip')
assert {name.rsplit('.', 1)[-1] for name in saved_names} == {'pdf', 'txt', 'md', 'html'}
for name in saved_names:
StorageMgr.require_scoped_object_key(
CONTEXT,
name,
expected_owner_type='upload_document',
)
forwarded_keys = [call.args[1] for call in kb.store_file.await_args_list]
assert forwarded_keys == saved_names
kb.ap.storage_mgr.storage_provider.delete.assert_awaited_once_with(zip_key)
@pytest.mark.asyncio
async def test_store_zip_file_raises_when_no_supported_files(self):
kb = _make_kb()
kb.ap.storage_mgr.storage_provider.load = AsyncMock(
kb.ap.storage_mgr.storage_provider.load_bounded = AsyncMock(
return_value=_make_zip_bytes({'image.png': b'png', 'video.mp4': b'video'})
)
kb.store_file = AsyncMock()
with pytest.raises(Exception, match='No supported files found'):
await kb._store_zip_file('archive.zip')
await kb._store_zip_file(CONTEXT, _upload_key('archive.zip'))
kb.store_file.assert_not_awaited()
kb.ap.storage_mgr.storage_provider.delete.assert_awaited_once_with('archive.zip')
kb.ap.storage_mgr.storage_provider.delete.assert_awaited_once_with(_upload_key('archive.zip'))
@pytest.mark.asyncio
async def test_store_zip_file_rejects_too_many_documents_before_extracting(self):
kb = _make_kb()
kb.ap.storage_mgr.storage_provider.load_bounded = AsyncMock(
return_value=_make_zip_bytes({f'doc-{index}.txt': b'text' for index in range(9)})
)
kb.store_file = AsyncMock()
with pytest.raises(ValueError, match='too many supported documents'):
await kb._store_zip_file(CONTEXT, _upload_key('archive.zip'))
kb.store_file.assert_not_awaited()
kb.ap.storage_mgr.storage_provider.save.assert_not_awaited()
kb.ap.storage_mgr.storage_provider.delete.assert_awaited_once_with(_upload_key('archive.zip'))
@pytest.mark.asyncio
async def test_store_zip_file_rejects_extreme_compression_ratio_before_extracting(self):
kb = _make_kb()
kb.ap.storage_mgr.storage_provider.load_bounded = AsyncMock(
return_value=_make_zip_bytes(
{'bomb.txt': b'A' * (1024 * 1024)},
compression=zipfile.ZIP_DEFLATED,
)
)
kb.store_file = AsyncMock()
with pytest.raises(ValueError, match='compression-ratio limit'):
await kb._store_zip_file(CONTEXT, _upload_key('archive.zip'))
kb.store_file.assert_not_awaited()
kb.ap.storage_mgr.storage_provider.save.assert_not_awaited()
kb.ap.storage_mgr.storage_provider.delete.assert_awaited_once_with(_upload_key('archive.zip'))
class TestStoreFileTask:
@pytest.mark.asyncio
async def test_store_file_task_opens_uow_before_first_database_helper(self):
kb = _make_kb()
active_workspace = contextvars.ContextVar('rag_task_workspace', default=None)
observed = []
@asynccontextmanager
async def tenant_uow(workspace_uuid):
token = active_workspace.set(workspace_uuid)
try:
yield
finally:
active_workspace.reset(token)
kb.ap.persistence_mgr.mode = SimpleNamespace(value='cloud_runtime')
kb.ap.persistence_mgr.tenant_uow = tenant_uow
async def assert_execution_context(_context):
observed.append(active_workspace.get())
kb._assert_execution_context = AsyncMock(side_effect=assert_execution_context)
kb._set_file_status = AsyncMock(side_effect=[True, True])
kb._ingest_document = AsyncMock(return_value={'status': 'completed'})
object_key = _upload_key('scoped.pdf')
file_obj = SimpleNamespace(uuid='file-uuid', file_name=object_key, extension='pdf')
await kb._store_file_task(CONTEXT, file_obj, Mock())
assert observed[0] == WORKSPACE_A
@pytest.mark.asyncio
async def test_store_file_task_marks_completed_and_cleans_storage(self):
kb = _make_kb()
kb._ingest_document = AsyncMock(return_value={'status': 'completed'})
file_obj = SimpleNamespace(uuid='file-uuid', file_name='test.pdf', extension='pdf')
object_key = _upload_key('test.pdf')
file_obj = SimpleNamespace(uuid='file-uuid', file_name=object_key, extension='pdf')
task_context = Mock()
await kb._store_file_task(file_obj, task_context)
await kb._store_file_task(CONTEXT, file_obj, task_context)
task_context.set_current_action.assert_called_once_with('Processing file')
kb.ap.storage_mgr.storage_provider.size.assert_awaited_once_with('test.pdf')
kb.ap.storage_mgr.storage_provider.size.assert_awaited_once_with(object_key)
kb._ingest_document.assert_awaited_once()
assert kb.ap.persistence_mgr.execute_async.await_count == 2
kb.ap.storage_mgr.storage_provider.delete.assert_awaited_once_with('test.pdf')
kb.ap.storage_mgr.storage_provider.delete.assert_awaited_once_with(object_key)
@pytest.mark.asyncio
async def test_store_file_task_marks_failed_and_cleans_storage(self):
kb = _make_kb()
kb._ingest_document = AsyncMock(return_value={'status': 'failed', 'error_message': 'parser failed'})
file_obj = SimpleNamespace(uuid='file-uuid', file_name='bad.pdf', extension='pdf')
object_key = _upload_key('bad.pdf')
file_obj = SimpleNamespace(uuid='file-uuid', file_name=object_key, extension='pdf')
task_context = Mock()
with pytest.raises(Exception, match='parser failed'):
await kb._store_file_task(file_obj, task_context)
await kb._store_file_task(CONTEXT, file_obj, task_context)
assert kb.ap.persistence_mgr.execute_async.await_count == 2
kb.ap.storage_mgr.storage_provider.delete.assert_awaited_once_with('bad.pdf')
kb.ap.storage_mgr.storage_provider.delete.assert_awaited_once_with(object_key)
class TestDeleteDocument:
@@ -163,7 +323,7 @@ class TestDeleteDocument:
async def test_delete_document_returns_false_when_no_plugin_id(self):
kb = _make_kb(plugin_id=None)
result = await kb._delete_document('doc-id')
result = await kb._delete_document(CONTEXT, 'doc-id')
assert result is False
@@ -172,7 +332,7 @@ class TestDeleteDocument:
kb = _make_kb()
kb.ap.plugin_connector.call_rag_delete_document = AsyncMock(return_value=True)
result = await kb._delete_document('doc-id')
result = await kb._delete_document(CONTEXT, 'doc-id')
assert result is True
kb.ap.plugin_connector.call_rag_delete_document.assert_awaited_once_with(
@@ -184,7 +344,7 @@ class TestDeleteDocument:
kb = _make_kb()
kb.ap.plugin_connector.call_rag_delete_document = AsyncMock(side_effect=Exception('plugin error'))
result = await kb._delete_document('doc-id')
result = await kb._delete_document(CONTEXT, 'doc-id')
assert result is False
kb.ap.logger.error.assert_called_once()