feat(tenancy): add Workspace multi-tenant foundation (#2353)

* Document multi-tenant workspace architecture

* Add OSS and commercial workspace boundaries

* docs: redesign multi-tenant workspace architecture

* feat(tenancy): implement workspace isolation

* docs(tenancy): record verification evidence

* docs(tenancy): revise single-instance SaaS topology

* docs(tenancy): refine architecture options

* docs: finalize cloud v2 multi-tenant decisions

* feat(tenancy): establish cloud isolation foundations

* feat(tenancy): harden shared cloud runtime boundaries

* docs(tenancy): record final isolation verification

* fix(tenancy): close isolation and permission gaps

* docs(tenancy): record final isolation verification

* feat(tenancy): connect cloud workspace control plane

* fix(build): install git for pinned SDK

* docs(cloud): update control plane verification

* chore: update multi-tenant SDK pin

* fix(cloud): skip legacy model sync during startup

* test(cloud): preserve minimal model manager fixtures

* fix(cloud): preserve authenticated account context

* fix(cloud): reuse authenticated account for user info

* feat(cloud): complete Workspace settings navigation

* test(web): cover Workspace dropdown menu

* feat(web): place workspace controls in sidebar

* refactor(web): streamline workspace controls

* style(web): format workspace layout test

* fix(cloud): surface runtime and workspace plan status

* fix(plugin): keep runtime identity stable across restarts

* fix(ui): widen and center workspace switcher

* fix(ui): hide roles from workspace switcher

* fix(ui): align workspace switcher with sidebar entries

* feat(workspace): add in-product collaboration and direct Cloud launch

* style: format collaboration changes

* fix(workspace): bind collaboration APIs to tenant UoW

* fix(cloud): preserve Core-owned collaboration state

* test(cloud): require Space identity for invite registration

* feat(cloud): complete secure invitation experience

* style(web): format invitation flows

* fix(cloud): recover box runtime without unscoped skill reload

* feat(oss): enforce invitation account and owner billing flows

* style: format OSS account service

* test(oss): cover invitation logout handoff

* fix(oss): resolve workspace owner in scoped session

* feat(cloud): harden multi-tenant runtime resources

* fix(cloud): bound runtime restart storms

* fix(cloud): eliminate periodic runtime CPU spikes

* fix(cloud): enforce instance capacity ceilings

* fix(cloud): scope public login capability discovery

* fix(cloud): bound tenant maintenance and monitoring work

* fix(runtime): bound tenant resource amplification

* fix(deps): pin green multi-tenant plugin SDK

* fix(cloud): handle unavailable skill capability

* fix(security): require authentication for image file endpoint (H-2)

- Changed /api/v1/files/image from AuthType.NONE to USER_TOKEN_OR_API_KEY
- Added Permission.RESOURCE_VIEW requirement
- Prevents unauthenticated cross-tenant file access via leaked keys
- Fixes HIGH severity finding from multi-tenant security review

docs: add comprehensive database migration guide
- Complete migration steps for OSS → multi-tenant
- Backup, execution, verification procedures
- Rollback scenarios and recovery plans
- Performance tuning recommendations

* test: add comprehensive cross-tenant isolation tests

Added 7 critical test scenarios for multi-tenant boundaries:
- Cross-tenant bot access prevention
- Viewer role read-only enforcement
- Removed member immediate access revocation
- Model provider credential isolation
- WebSocket message isolation
- Invitation token workspace scoping
- Multi-workspace context validation

These tests address P0-2 coverage gaps for:
- workspaces.py (membership & invitation flows)
- user.py (authentication & authorization)
- websocket_chat.py (real-time isolation)
- plugins.py (resource access control)

docs: finalize database migration guide

* fix(security): resolve M-1, M-2, M-3 security findings

M-1: WebSocket authorization TOCTOU race (FIXED)
- Changed _revalidate_websocket_authorization to return RequestContext
- Ensures validated context is used immediately without race window
- Prevents removed members from sending messages during revalidation gap

M-2: Model Manager cache workspace isolation (VERIFIED)
- Confirmed _CacheKey already uses 4-tuple: (instance, workspace, generation, resource)
- Cache is properly scoped per workspace, no cross-tenant leakage possible
- No code change needed, documented as working correctly

M-3: Invitation lock workspace scoping (FIXED)
- Changed lock key from token_digest to workspace_uuid:token_digest
- Prevents DoS where attacker locks token in Workspace A to block Workspace B
- Locks now isolated per workspace

All MEDIUM severity findings from security review now resolved.

* fix(cloud): unblock tenant CI and enforce knowledge quotas

* fix(tenancy): scope rerank model sync

---------

Co-authored-by: dadachann <185672915+dadachann@users.noreply.github.com>
This commit is contained in:
RockChinQ
2026-07-30 21:43:35 +08:00
committed by GitHub
parent 463b120923
commit e1ac5e0fc8
468 changed files with 78320 additions and 13137 deletions
+369 -13
View File
@@ -11,6 +11,7 @@ import pytest
from mcp import types as mcp_types
from mcp.shared.exceptions import McpError
from langbot.pkg.api.http.context import ExecutionContext
from langbot.pkg.provider.tools.loaders.mcp import (
MCP_RESOURCE_CONTEXT_QUERY_KEY,
MCP_RESOURCE_TRACE_QUERY_KEY,
@@ -23,10 +24,30 @@ from langbot.pkg.provider.tools.loaders.mcp import (
RuntimeMCPSession,
)
from langbot.pkg.telemetry import features as telemetry_features
from langbot.pkg.workspace.errors import WorkspaceGenerationMismatchError
TEST_EXECUTION_CONTEXT = ExecutionContext(
instance_uuid='instance-a',
workspace_uuid='workspace-a',
placement_generation=1,
query_uuid='query-a',
)
def _app() -> SimpleNamespace:
return SimpleNamespace(logger=Mock())
return SimpleNamespace(
logger=Mock(),
workspace_service=SimpleNamespace(
get_execution_binding=AsyncMock(
return_value=SimpleNamespace(
instance_uuid='instance-a',
workspace_uuid='workspace-a',
placement_generation=1,
)
)
),
)
def _connected_session(
@@ -35,8 +56,15 @@ def _connected_session(
uuid: str = 'srv-1',
resources: list[dict] | None = None,
templates: list[dict] | None = None,
execution_context: ExecutionContext = TEST_EXECUTION_CONTEXT,
) -> RuntimeMCPSession:
session = RuntimeMCPSession(name, {'uuid': uuid, 'mode': 'remote'}, True, _app())
session = RuntimeMCPSession(
name,
{'uuid': uuid, 'mode': 'remote'},
True,
_app(),
execution_context,
)
session.status = MCPSessionStatus.CONNECTED
session.session = SimpleNamespace(read_resource=AsyncMock())
session.resources = resources or [
@@ -56,8 +84,24 @@ def _connected_session(
return session
def _query() -> SimpleNamespace:
return SimpleNamespace(variables={})
def _query(variables: dict | None = None, context: ExecutionContext = TEST_EXECUTION_CONTEXT) -> SimpleNamespace:
return SimpleNamespace(
instance_uuid=context.instance_uuid,
workspace_uuid=context.workspace_uuid,
placement_generation=context.placement_generation,
bot_uuid=context.bot_uuid,
pipeline_uuid=context.pipeline_uuid,
query_uuid=context.query_uuid,
variables=variables or {},
)
def _register_session(loader: MCPLoader, session: RuntimeMCPSession) -> None:
loader._register_session(
session.execution_context,
session.server_name,
session,
)
def _http_status_error(status_code: int) -> httpx.HTTPStatusError:
@@ -84,6 +128,7 @@ async def test_invoke_mcp_tool_uses_configurable_request_timeout():
},
True,
_app(),
TEST_EXECUTION_CONTEXT,
)
session.session = SimpleNamespace(call_tool=AsyncMock(return_value=_tool_result()))
@@ -108,6 +153,7 @@ async def test_invoke_mcp_tool_zero_timeout_disables_request_deadline():
},
True,
_app(),
TEST_EXECUTION_CONTEXT,
)
session.session = SimpleNamespace(call_tool=AsyncMock(return_value=_tool_result()))
@@ -131,6 +177,7 @@ async def test_invoke_mcp_tool_timeout_is_not_retried_and_session_remains_usable
},
True,
_app(),
TEST_EXECUTION_CONTEXT,
)
timeout = McpError(
mcp_types.ErrorData(
@@ -165,6 +212,7 @@ def test_invalid_tool_call_timeout_falls_back_to_default(invalid_timeout):
},
True,
ap,
TEST_EXECUTION_CONTEXT,
)
assert session.tool_call_timeout_sec == MCP_TOOL_CALL_TIMEOUT_DEFAULT_SECONDS
@@ -178,6 +226,7 @@ async def test_remote_transport_falls_back_to_sse_for_compatible_http_status_in_
{'uuid': 'srv-1', 'mode': 'remote', 'url': 'https://example.com/mcp'},
True,
_app(),
TEST_EXECUTION_CONTEXT,
)
session._init_streamable_http_server = AsyncMock(
side_effect=ExceptionGroup('transport failed', [_http_status_error(405)])
@@ -197,6 +246,7 @@ async def test_remote_transport_does_not_fallback_for_auth_http_status():
{'uuid': 'srv-1', 'mode': 'remote', 'url': 'https://example.com/mcp'},
True,
_app(),
TEST_EXECUTION_CONTEXT,
)
error = _http_status_error(403)
session._init_streamable_http_server = AsyncMock(side_effect=error)
@@ -354,10 +404,18 @@ def test_resource_uri_allowed_supports_listed_templates_conservatively():
async def test_mcp_loader_can_hide_synthetic_resource_tools():
loader = MCPLoader(_app())
session = _connected_session()
loader.sessions = {'docs': session}
_register_session(loader, session)
with_resource_tools = await loader.get_tools(['srv-1'], include_resource_tools=True)
without_resource_tools = await loader.get_tools(['srv-1'], include_resource_tools=False)
with_resource_tools = await loader.get_tools(
TEST_EXECUTION_CONTEXT,
['srv-1'],
include_resource_tools=True,
)
without_resource_tools = await loader.get_tools(
TEST_EXECUTION_CONTEXT,
['srv-1'],
include_resource_tools=False,
)
assert {tool.name for tool in with_resource_tools} == {
MCP_TOOL_LIST_RESOURCES,
@@ -370,9 +428,9 @@ async def test_mcp_loader_can_hide_synthetic_resource_tools():
async def test_mcp_loader_refuses_resource_tool_calls_when_agent_read_disabled():
loader = MCPLoader(_app())
session = _connected_session()
loader.sessions = {'docs': session}
query = SimpleNamespace(
variables={
_register_session(loader, session)
query = _query(
{
'_pipeline_bound_mcp_servers': ['srv-1'],
'_pipeline_mcp_resource_agent_read_enabled': False,
}
@@ -411,9 +469,10 @@ async def test_build_resource_context_for_query_uses_only_bound_attached_text_re
)
]
)
loader.sessions = {'docs': docs, 'other': other}
query = SimpleNamespace(
variables={
_register_session(loader, docs)
_register_session(loader, other)
query = _query(
{
'_pipeline_bound_mcp_servers': ['srv-1'],
'_pipeline_mcp_resource_attachments': [
{'server_uuid': 'srv-1', 'server_name': 'docs', 'uri': 'file:///README.md', 'mode': 'pinned'},
@@ -433,6 +492,140 @@ async def test_build_resource_context_for_query_uses_only_bound_attached_text_re
other.session.read_resource.assert_not_called()
def test_mcp_loader_session_keys_do_not_collide_between_workspaces():
loader = MCPLoader(_app())
workspace_b = ExecutionContext(
instance_uuid='instance-a',
workspace_uuid='workspace-b',
placement_generation=1,
query_uuid='query-b',
)
session_a = _connected_session(name='docs', uuid='srv-a')
session_b = _connected_session(
name='docs',
uuid='srv-b',
execution_context=workspace_b,
)
_register_session(loader, session_a)
_register_session(loader, session_b)
assert len(loader.sessions) == 2
assert loader.get_session(TEST_EXECUTION_CONTEXT, 'docs') is session_a
assert loader.get_session(workspace_b, 'docs') is session_b
assert loader.get_session(TEST_EXECUTION_CONTEXT, 'docs') is not loader.get_session(workspace_b, 'docs')
@pytest.mark.asyncio
async def test_mcp_tool_result_is_discarded_when_generation_changes_during_call():
session = _connected_session()
binding = SimpleNamespace(
instance_uuid='instance-a',
workspace_uuid='workspace-a',
placement_generation=1,
)
session.ap.workspace_service.get_execution_binding.side_effect = [
binding,
binding,
WorkspaceGenerationMismatchError('generation changed during tool call'),
]
session.session = SimpleNamespace(call_tool=AsyncMock(return_value=SimpleNamespace(isError=False, content=[])))
with pytest.raises(WorkspaceGenerationMismatchError):
await session.invoke_mcp_tool('side_effecting_tool', {})
session.session.call_tool.assert_awaited_once_with(
'side_effecting_tool',
{},
read_timeout_seconds=timedelta(seconds=MCP_TOOL_CALL_TIMEOUT_DEFAULT_SECONDS),
)
@pytest.mark.asyncio
async def test_mcp_resource_cache_is_not_served_to_stale_generation():
session = _connected_session()
session._resource_cache[('file:///README.md', 10, None, False)] = {
'cached_at': 0,
'envelope': {'contents': [{'type': 'text', 'text': 'stale'}]},
}
session.ap.workspace_service.get_execution_binding.side_effect = WorkspaceGenerationMismatchError(
'stale generation'
)
with pytest.raises(WorkspaceGenerationMismatchError):
await session.read_resource_envelope('file:///README.md', max_bytes=10)
session.session.read_resource.assert_not_awaited()
@pytest.mark.asyncio
async def test_directory_projection_retires_idle_mcp_scope_without_db_poll():
loader = MCPLoader(_app())
sessions = []
for index in range(100):
context = ExecutionContext(
instance_uuid='instance-a',
workspace_uuid=f'workspace-{index}',
placement_generation=1,
)
session = RuntimeMCPSession(
f'server-{index}',
{'uuid': f'srv-{index}', 'mode': 'remote'},
True,
loader.ap,
context,
)
session.shutdown = AsyncMock()
loader._register_session(context, session.server_name, session)
sessions.append(session)
loader.reconcile_execution_projection('instance-a', {})
reconcile_task = loader._projection_reconcile_task
assert reconcile_task is not None
assert len(loader._pending_projection_retirements) == 100
# A second projection coalesces into the same worker instead of creating
# one timer or task per Workspace.
loader.reconcile_execution_projection('instance-a', {})
assert loader._projection_reconcile_task is reconcile_task
await asyncio.wait_for(reconcile_task, timeout=1)
assert loader.sessions == {}
assert loader._scope_generations == {}
assert loader._pending_projection_retirements == set()
assert sum(session.shutdown.await_count for session in sessions) == 100
loader.ap.workspace_service.get_execution_binding.assert_not_awaited()
@pytest.mark.asyncio
async def test_directory_projection_keeps_matching_and_unaffected_mcp_scopes():
loader = MCPLoader(_app())
matching = _connected_session()
other_context = ExecutionContext(
instance_uuid='instance-a',
workspace_uuid='workspace-b',
placement_generation=1,
)
unaffected = _connected_session(
name='other',
uuid='srv-2',
execution_context=other_context,
)
_register_session(loader, matching)
_register_session(loader, unaffected)
loader.reconcile_execution_projection(
'instance-a',
{'workspace-a': 1},
affected_workspace_uuids={'workspace-a'},
)
await asyncio.sleep(0)
assert loader.get_session(TEST_EXECUTION_CONTEXT, 'docs') is matching
assert loader.get_session(other_context, 'other') is unaffected
assert loader._projection_reconcile_task is None
@pytest.mark.asyncio
async def test_mcp_loader_shutdown_cancels_startup_tasks_and_closes_sessions_concurrently():
loader = MCPLoader(_app())
@@ -454,6 +647,7 @@ async def test_mcp_loader_shutdown_cancels_startup_tasks_and_closes_sessions_con
class Session:
def __init__(self, name: str):
self.name = name
self.server_name = name
async def shutdown(self):
started.add(self.name)
@@ -470,3 +664,165 @@ async def test_mcp_loader_shutdown_cancels_startup_tasks_and_closes_sessions_con
assert started == {'one', 'two'}
assert loader._hosted_mcp_tasks == []
assert loader.sessions == {}
@pytest.mark.asyncio
async def test_completed_mcp_host_tasks_do_not_accumulate():
loader = MCPLoader(_app())
task = asyncio.create_task(asyncio.sleep(0))
loader.track_hosted_task(task, TEST_EXECUTION_CONTEXT)
await task
await asyncio.sleep(0)
assert loader._hosted_mcp_tasks == []
assert loader._hosted_mcp_tasks_by_scope == {}
assert loader._scope_generations == {}
@pytest.mark.asyncio
async def test_generation_advance_cancels_host_tasks_and_closes_old_sessions():
loader = MCPLoader(_app())
old_session = SimpleNamespace(
server_name='old',
shutdown=AsyncMock(),
)
loader._register_session(
TEST_EXECUTION_CONTEXT,
old_session.server_name,
old_session,
)
async def pending_host():
await asyncio.Event().wait()
hosted_task = asyncio.create_task(pending_host())
loader.track_hosted_task(hosted_task, TEST_EXECUTION_CONTEXT)
await asyncio.sleep(0)
next_context = ExecutionContext(
instance_uuid=TEST_EXECUTION_CONTEXT.instance_uuid,
workspace_uuid=TEST_EXECUTION_CONTEXT.workspace_uuid,
placement_generation=2,
)
loader.ap.workspace_service.get_execution_binding = AsyncMock(
return_value=SimpleNamespace(
instance_uuid=next_context.instance_uuid,
workspace_uuid=next_context.workspace_uuid,
placement_generation=next_context.placement_generation,
)
)
await loader._assert_execution_active(next_context)
assert hosted_task.cancelled()
old_session.shutdown.assert_awaited_once_with()
assert loader.sessions == {}
assert loader._session_keys_by_scope == {}
assert loader._hosted_mcp_tasks_by_scope == {}
assert loader._scope_generations == {}
def test_session_lookup_uses_scope_index_without_global_iteration():
class NoGlobalIterationDict(dict):
def __iter__(self):
raise AssertionError('MCP lookup scanned every tenant session')
def items(self):
raise AssertionError('MCP lookup scanned every tenant session')
def values(self):
raise AssertionError('MCP lookup scanned every tenant session')
loader = MCPLoader(_app())
target_context = None
target_session = None
for index in range(1_000):
context = ExecutionContext(
instance_uuid='instance-a',
workspace_uuid=f'workspace-{index}',
placement_generation=1,
)
session = SimpleNamespace(server_name=f'server-{index}')
loader._register_session(context, session.server_name, session)
if index == 777:
target_context = context
target_session = session
loader._sessions = NoGlobalIterationDict(loader._sessions)
assert loader._sessions_for_context(target_context) == [target_session]
@pytest.mark.asyncio
async def test_mcp_startup_concurrency_is_instance_bounded():
app = _app()
app.instance_config = SimpleNamespace(data={'mcp': {'lifecycle_concurrency': 2}})
loader = MCPLoader(app)
active = 0
maximum_active = 0
release = asyncio.Event()
async def fake_host(_context, _config):
nonlocal active, maximum_active
active += 1
maximum_active = max(maximum_active, active)
if maximum_active == 2:
release.set()
await release.wait()
await asyncio.sleep(0)
active -= 1
loader._host_mcp_server = fake_host
await asyncio.gather(
*(
loader.host_mcp_server(
TEST_EXECUTION_CONTEXT,
{'name': f'server-{index}'},
)
for index in range(20)
)
)
assert loader._lifecycle_concurrency == 2
assert maximum_active == 2
@pytest.mark.asyncio
async def test_mcp_startup_dispatcher_does_not_create_every_server_task_at_once():
app = _app()
app.instance_config = SimpleNamespace(data={'mcp': {'lifecycle_concurrency': 2}})
loader = MCPLoader(app)
started = 0
first_batch_started = asyncio.Event()
release = asyncio.Event()
async def fake_host(_context, _config):
nonlocal started
started += 1
if started == 2:
first_batch_started.set()
await release.wait()
loader.host_mcp_server = fake_host
configs = [(TEST_EXECUTION_CONTEXT, {'name': f'server-{index}'}) for index in range(20)]
dispatch_task = asyncio.create_task(loader._host_server_configs_bounded(configs))
await asyncio.wait_for(first_batch_started.wait(), timeout=1)
assert started == 2
assert len(loader._hosted_mcp_tasks) == 2
release.set()
await dispatch_task
await asyncio.sleep(0)
assert started == 20
assert loader._hosted_mcp_tasks == []
assert loader._hosted_mcp_tasks_by_scope == {}
def test_invalid_mcp_lifecycle_concurrency_uses_safe_default():
app = _app()
app.instance_config = SimpleNamespace(data={'mcp': {'lifecycle_concurrency': True}})
assert MCPLoader(app)._lifecycle_concurrency == 16