feat(tenancy): add Workspace multi-tenant foundation (#2353)

* Document multi-tenant workspace architecture

* Add OSS and commercial workspace boundaries

* docs: redesign multi-tenant workspace architecture

* feat(tenancy): implement workspace isolation

* docs(tenancy): record verification evidence

* docs(tenancy): revise single-instance SaaS topology

* docs(tenancy): refine architecture options

* docs: finalize cloud v2 multi-tenant decisions

* feat(tenancy): establish cloud isolation foundations

* feat(tenancy): harden shared cloud runtime boundaries

* docs(tenancy): record final isolation verification

* fix(tenancy): close isolation and permission gaps

* docs(tenancy): record final isolation verification

* feat(tenancy): connect cloud workspace control plane

* fix(build): install git for pinned SDK

* docs(cloud): update control plane verification

* chore: update multi-tenant SDK pin

* fix(cloud): skip legacy model sync during startup

* test(cloud): preserve minimal model manager fixtures

* fix(cloud): preserve authenticated account context

* fix(cloud): reuse authenticated account for user info

* feat(cloud): complete Workspace settings navigation

* test(web): cover Workspace dropdown menu

* feat(web): place workspace controls in sidebar

* refactor(web): streamline workspace controls

* style(web): format workspace layout test

* fix(cloud): surface runtime and workspace plan status

* fix(plugin): keep runtime identity stable across restarts

* fix(ui): widen and center workspace switcher

* fix(ui): hide roles from workspace switcher

* fix(ui): align workspace switcher with sidebar entries

* feat(workspace): add in-product collaboration and direct Cloud launch

* style: format collaboration changes

* fix(workspace): bind collaboration APIs to tenant UoW

* fix(cloud): preserve Core-owned collaboration state

* test(cloud): require Space identity for invite registration

* feat(cloud): complete secure invitation experience

* style(web): format invitation flows

* fix(cloud): recover box runtime without unscoped skill reload

* feat(oss): enforce invitation account and owner billing flows

* style: format OSS account service

* test(oss): cover invitation logout handoff

* fix(oss): resolve workspace owner in scoped session

* feat(cloud): harden multi-tenant runtime resources

* fix(cloud): bound runtime restart storms

* fix(cloud): eliminate periodic runtime CPU spikes

* fix(cloud): enforce instance capacity ceilings

* fix(cloud): scope public login capability discovery

* fix(cloud): bound tenant maintenance and monitoring work

* fix(runtime): bound tenant resource amplification

* fix(deps): pin green multi-tenant plugin SDK

* fix(cloud): handle unavailable skill capability

* fix(security): require authentication for image file endpoint (H-2)

- Changed /api/v1/files/image from AuthType.NONE to USER_TOKEN_OR_API_KEY
- Added Permission.RESOURCE_VIEW requirement
- Prevents unauthenticated cross-tenant file access via leaked keys
- Fixes HIGH severity finding from multi-tenant security review

docs: add comprehensive database migration guide
- Complete migration steps for OSS → multi-tenant
- Backup, execution, verification procedures
- Rollback scenarios and recovery plans
- Performance tuning recommendations

* test: add comprehensive cross-tenant isolation tests

Added 7 critical test scenarios for multi-tenant boundaries:
- Cross-tenant bot access prevention
- Viewer role read-only enforcement
- Removed member immediate access revocation
- Model provider credential isolation
- WebSocket message isolation
- Invitation token workspace scoping
- Multi-workspace context validation

These tests address P0-2 coverage gaps for:
- workspaces.py (membership & invitation flows)
- user.py (authentication & authorization)
- websocket_chat.py (real-time isolation)
- plugins.py (resource access control)

docs: finalize database migration guide

* fix(security): resolve M-1, M-2, M-3 security findings

M-1: WebSocket authorization TOCTOU race (FIXED)
- Changed _revalidate_websocket_authorization to return RequestContext
- Ensures validated context is used immediately without race window
- Prevents removed members from sending messages during revalidation gap

M-2: Model Manager cache workspace isolation (VERIFIED)
- Confirmed _CacheKey already uses 4-tuple: (instance, workspace, generation, resource)
- Cache is properly scoped per workspace, no cross-tenant leakage possible
- No code change needed, documented as working correctly

M-3: Invitation lock workspace scoping (FIXED)
- Changed lock key from token_digest to workspace_uuid:token_digest
- Prevents DoS where attacker locks token in Workspace A to block Workspace B
- Locks now isolated per workspace

All MEDIUM severity findings from security review now resolved.

* fix(cloud): unblock tenant CI and enforce knowledge quotas

* fix(tenancy): scope rerank model sync

---------

Co-authored-by: dadachann <185672915+dadachann@users.noreply.github.com>
This commit is contained in:
RockChinQ
2026-07-30 21:43:35 +08:00
committed by GitHub
parent 463b120923
commit e1ac5e0fc8
468 changed files with 78320 additions and 13137 deletions
@@ -1,8 +1,10 @@
from __future__ import annotations
import base64
import contextlib
import os
import tempfile
import threading
from types import SimpleNamespace
from unittest.mock import AsyncMock, Mock
@@ -11,7 +13,16 @@ import pytest
import langbot_plugin.api.entities.builtin.resource.tool as resource_tool
from langbot.pkg.provider.tools.loaders.native import NativeToolLoader
from langbot.pkg.provider.tools.loaders import native as native_loader
from langbot.pkg.provider.tools.toolmgr import ToolManager
from langbot.pkg.api.http.context import ExecutionContext
_CONTEXT = ExecutionContext(
instance_uuid='instance-a',
workspace_uuid='workspace-a',
placement_generation=1,
)
class StubLoader:
@@ -44,7 +55,8 @@ class StubLoader:
for tool in self._tools
]
async def has_tool(self, name: str) -> bool:
async def has_tool(self, *args) -> bool:
name = args[-1]
return any(tool.name == name for tool in self._tools)
async def invoke_tool(self, name: str, parameters: dict, query):
@@ -72,7 +84,7 @@ async def test_tool_manager_omits_skill_authoring_tools_by_default():
manager.plugin_tool_loader = StubLoader([make_tool('plugin_tool')])
manager.mcp_tool_loader = StubLoader([make_tool('mcp_tool')])
tools = await manager.get_all_tools()
tools = await manager.get_all_tools(_CONTEXT)
assert [tool.name for tool in tools] == ['exec', 'plugin_tool', 'mcp_tool']
@@ -85,7 +97,7 @@ async def test_tool_manager_includes_skill_authoring_tools_when_requested():
manager.plugin_tool_loader = StubLoader([make_tool('plugin_tool')])
manager.mcp_tool_loader = StubLoader([make_tool('mcp_tool')])
tools = await manager.get_all_tools(include_skill_authoring=True)
tools = await manager.get_all_tools(_CONTEXT, include_skill_authoring=True)
assert [tool.name for tool in tools] == ['exec', 'activate', 'plugin_tool', 'mcp_tool']
@@ -102,7 +114,7 @@ async def test_tool_manager_catalog_labels_tool_sources():
)
manager.mcp_tool_loader = StubLoader([make_tool('mcp_tool')])
catalog = await manager.get_tool_catalog(include_skill_authoring=True)
catalog = await manager.get_tool_catalog(_CONTEXT, include_skill_authoring=True)
assert [(item['name'], item['source'], item['source_name']) for item in catalog] == [
('exec', 'builtin', 'LangBot'),
@@ -121,11 +133,55 @@ async def test_tool_manager_routes_native_tool_calls():
manager.plugin_tool_loader = StubLoader([make_tool('plugin_tool')])
manager.mcp_tool_loader = StubLoader([make_tool('mcp_tool')])
result = await manager.execute_func_call('exec', {'command': 'pwd'}, query=Mock())
query = SimpleNamespace(
_execution_context=_CONTEXT,
bot_uuid=None,
pipeline_uuid=None,
query_uuid=None,
)
result = await manager.execute_func_call('exec', {'command': 'pwd'}, query=query)
assert result == {'backend': 'fake'}
@pytest.mark.asyncio
async def test_tool_manager_hides_sandbox_and_skill_tools_without_workspace_entitlement():
box_service = SimpleNamespace(is_workspace_sandbox_available=AsyncMock(return_value=False))
manager = ToolManager(SimpleNamespace(box_service=box_service))
manager.native_tool_loader = StubLoader([make_tool('exec')])
manager.skill_tool_loader = StubLoader([make_tool('activate')])
manager.plugin_tool_loader = StubLoader([make_tool('plugin_tool')])
manager.mcp_tool_loader = StubLoader([make_tool('mcp_tool')])
tools = await manager.get_all_tools(_CONTEXT, include_skill_authoring=True)
catalog = await manager.get_tool_catalog(_CONTEXT, include_skill_authoring=True)
assert [tool.name for tool in tools] == ['plugin_tool', 'mcp_tool']
assert [item['name'] for item in catalog] == ['plugin_tool', 'mcp_tool']
assert box_service.is_workspace_sandbox_available.await_count == 2
@pytest.mark.asyncio
async def test_tool_manager_rechecks_workspace_entitlement_before_native_invocation():
box_service = SimpleNamespace(is_workspace_sandbox_available=AsyncMock(return_value=False))
manager = ToolManager(SimpleNamespace(box_service=box_service))
manager.native_tool_loader = StubLoader([make_tool('exec')], invoke_result={'unexpected': True})
manager.skill_tool_loader = StubLoader([])
manager.plugin_tool_loader = StubLoader([])
manager.mcp_tool_loader = StubLoader([])
query = SimpleNamespace(
_execution_context=_CONTEXT,
bot_uuid=None,
pipeline_uuid=None,
query_uuid=None,
)
with pytest.raises(Exception, match='exec'):
await manager.execute_func_call('exec', {'command': 'pwd'}, query=query)
box_service.is_workspace_sandbox_available.assert_awaited_once_with(_CONTEXT)
@pytest.mark.asyncio
async def test_native_tool_loader_hides_tools_when_box_unavailable():
loader = NativeToolLoader(SimpleNamespace(box_service=SimpleNamespace(available=False)))
@@ -139,7 +195,7 @@ async def test_native_tool_loader_hides_tools_when_box_unavailable():
async def test_native_tool_loader_exposes_all_tools_when_box_available():
box_service = SimpleNamespace(
available=True,
get_status=AsyncMock(return_value={'backend': {'available': True}}),
get_backend_status=AsyncMock(return_value={'backend': {'available': True}}),
)
loader = NativeToolLoader(SimpleNamespace(box_service=box_service, logger=Mock()))
await loader.initialize()
@@ -155,7 +211,7 @@ async def test_native_tool_loader_exposes_all_tools_when_box_available():
async def test_native_tool_loader_refreshes_after_box_recovers():
box_service = SimpleNamespace(
available=False,
get_status=AsyncMock(return_value={'backend': {'available': True}}),
get_backend_status=AsyncMock(return_value={'backend': {'available': True}}),
)
loader = NativeToolLoader(SimpleNamespace(box_service=box_service, logger=Mock()))
await loader.initialize()
@@ -166,20 +222,51 @@ async def test_native_tool_loader_refreshes_after_box_recovers():
assert [tool.name for tool in await loader.get_tools()] == ['exec', 'read', 'write', 'edit', 'glob', 'grep']
@pytest.mark.asyncio
async def test_native_tool_loader_rechecks_admission_at_the_final_invoke_boundary():
box_service = SimpleNamespace(
available=True,
require_workspace_sandbox=AsyncMock(side_effect=RuntimeError('entitlement expired')),
)
loader = NativeToolLoader(SimpleNamespace(box_service=box_service, logger=Mock()))
query = SimpleNamespace(
_execution_context=_CONTEXT,
bot_uuid=None,
pipeline_uuid=None,
query_uuid=None,
)
with pytest.raises(RuntimeError, match='entitlement expired'):
await loader.invoke_tool('read', {'path': '/workspace/private.txt'}, query)
box_service.require_workspace_sandbox.assert_awaited_once_with(_CONTEXT)
# ── read/write/edit file tool tests ─────────────────────────────
def _make_loader_with_workspace(tmpdir: str) -> tuple[NativeToolLoader, Mock]:
logger = Mock()
box_service = SimpleNamespace(available=True, default_workspace=tmpdir)
box_service = SimpleNamespace(
available=True,
default_workspace=tmpdir,
_tenant_workspace=Mock(return_value=tmpdir),
)
ap = SimpleNamespace(box_service=box_service, logger=logger)
return NativeToolLoader(ap), logger
def _make_query() -> Mock:
q = Mock()
q.query_id = 'test-query-1'
return q
def _make_query() -> SimpleNamespace:
return SimpleNamespace(
query_id='test-query-1',
query_uuid='test-query-1',
instance_uuid=_CONTEXT.instance_uuid,
workspace_uuid=_CONTEXT.workspace_uuid,
placement_generation=_CONTEXT.placement_generation,
bot_uuid=None,
pipeline_uuid=None,
variables={},
)
@pytest.mark.asyncio
@@ -373,6 +460,24 @@ async def test_edit_rejects_missing_string():
assert 'not found' in result['error'].lower()
@pytest.mark.asyncio
async def test_edit_rejects_oversized_host_file(monkeypatch):
with tempfile.TemporaryDirectory() as tmpdir:
loader, _ = _make_loader_with_workspace(tmpdir)
with open(os.path.join(tmpdir, 'large.txt'), 'wb') as f:
f.write(b'12345')
monkeypatch.setattr(native_loader, '_MAX_HOST_EDIT_FILE_BYTES', 4)
result = await loader.invoke_tool(
'edit',
{'path': '/workspace/large.txt', 'old_string': '1', 'new_string': 'x'},
_make_query(),
)
assert result['ok'] is False
assert 'edit limit' in result['error']
@pytest.mark.asyncio
async def test_path_escape_blocked():
with tempfile.TemporaryDirectory() as tmpdir:
@@ -382,6 +487,137 @@ async def test_path_escape_blocked():
await loader.invoke_tool('read', {'path': '/workspace/../../etc/passwd'}, _make_query())
@pytest.mark.parametrize(
('tool_name', 'parameters'),
[
('read', {'path': '/workspace/shared/tenant-b-only.txt'}),
(
'write',
{'path': '/workspace/shared/tenant-b-only.txt', 'content': 'overwritten by tenant a'},
),
(
'edit',
{
'path': '/workspace/shared/tenant-b-only.txt',
'old_string': 'tenant-b-secret',
'new_string': 'overwritten by tenant a',
},
),
('glob', {'path': '/workspace/shared', 'pattern': '*'}),
('grep', {'path': '/workspace/shared', 'pattern': 'tenant-b-secret'}),
],
)
@pytest.mark.asyncio
async def test_host_workspace_operations_do_not_follow_a_swapped_ancestor(
monkeypatch,
tool_name: str,
parameters: dict,
):
with tempfile.TemporaryDirectory() as tmpdir:
tenant_a = os.path.join(tmpdir, 'tenant-a')
tenant_b = os.path.join(tmpdir, 'tenant-b')
os.makedirs(os.path.join(tenant_a, 'shared'))
os.makedirs(os.path.join(tenant_b, 'shared'))
tenant_b_file = os.path.join(tenant_b, 'shared', 'tenant-b-only.txt')
with open(os.path.join(tenant_a, 'shared', 'tenant-a-only.txt'), 'w', encoding='utf-8') as file_obj:
file_obj.write('tenant-a-content')
with open(tenant_b_file, 'w', encoding='utf-8') as file_obj:
file_obj.write('tenant-b-secret')
box_service = SimpleNamespace(
available=True,
default_workspace=tmpdir,
_tenant_workspace=Mock(return_value=tenant_a),
)
loader = NativeToolLoader(SimpleNamespace(box_service=box_service, logger=Mock()))
original_open_host_root = native_loader._open_host_root
@contextlib.contextmanager
def open_host_root_after_swap(location, *, create):
with original_open_host_root(location, create=create) as root_fd:
original_ancestor = os.path.join(tenant_a, 'shared-original')
os.rename(os.path.join(tenant_a, 'shared'), original_ancestor)
os.symlink(os.path.join(tenant_b, 'shared'), os.path.join(tenant_a, 'shared'))
yield root_fd
monkeypatch.setattr(native_loader, '_open_host_root', open_host_root_after_swap)
try:
result = await loader.invoke_tool(tool_name, parameters, _make_query())
except ValueError as exc:
result = {'ok': False, 'error': str(exc)}
assert result.get('ok') is False
assert 'tenant-b-secret' not in repr(result)
assert 'tenant-b-only.txt' not in repr(result)
with open(tenant_b_file, encoding='utf-8') as file_obj:
assert file_obj.read() == 'tenant-b-secret'
@pytest.mark.asyncio
async def test_host_file_api_falls_back_to_tenant_box_when_openat_is_unavailable(monkeypatch):
with tempfile.TemporaryDirectory() as tmpdir:
box_service = SimpleNamespace(
available=True,
default_workspace=tmpdir,
_tenant_workspace=Mock(return_value=tmpdir),
execute_tool=AsyncMock(
return_value={
'ok': True,
'stdout': '{"ok": true, "content": "box-owned", "truncated": false}',
'stderr': '',
}
),
)
loader = NativeToolLoader(SimpleNamespace(box_service=box_service, logger=Mock()))
monkeypatch.setattr(native_loader, '_SECURE_HOST_FILE_OPS_AVAILABLE', False)
result = await loader.invoke_tool(
'read',
{'path': '/workspace/file.txt'},
_make_query(),
)
assert result['ok'] is True
assert result['content'] == 'box-owned'
command = box_service.execute_tool.await_args.args[0]['command']
assert 'path = "/workspace/file.txt"' in command
@pytest.mark.asyncio
async def test_box_workspace_edit_script_bounds_file_read_and_replacement(monkeypatch):
with tempfile.TemporaryDirectory() as tmpdir:
box_service = SimpleNamespace(
available=True,
default_workspace=tmpdir,
_tenant_workspace=Mock(return_value=tmpdir),
execute_tool=AsyncMock(
return_value={
'ok': True,
'stdout': '{"ok": false, "error": "File exceeds limit"}',
'stderr': '',
}
),
)
loader = NativeToolLoader(SimpleNamespace(box_service=box_service, logger=Mock()))
monkeypatch.setattr(native_loader, '_SECURE_HOST_FILE_OPS_AVAILABLE', False)
await loader.invoke_tool(
'edit',
{
'path': '/workspace/file.txt',
'old_string': 'old',
'new_string': 'new',
},
_make_query(),
)
command = box_service.execute_tool.await_args.args[0]['command']
assert f'os.path.getsize(path) > {native_loader._MAX_HOST_EDIT_FILE_BYTES}' in command
assert f'f.read({native_loader._MAX_HOST_EDIT_FILE_BYTES + 1})' in command
assert f"len(new_content.encode('utf-8')) > {native_loader._MAX_HOST_EDIT_FILE_BYTES}" in command
@pytest.mark.asyncio
async def test_box_availability_helper_handles_unavailable_and_errors():
from langbot.pkg.provider.tools.loaders.availability import is_box_backend_available
@@ -391,13 +627,13 @@ async def test_box_availability_helper_handles_unavailable_and_errors():
unavailable_backend = SimpleNamespace(
available=True,
get_status=AsyncMock(return_value={'backend': {'available': False}}),
get_backend_status=AsyncMock(return_value={'backend': {'available': False}}),
)
assert await is_box_backend_available(SimpleNamespace(box_service=unavailable_backend)) is False
failing_backend = SimpleNamespace(
available=True,
get_status=AsyncMock(side_effect=RuntimeError('box unavailable')),
get_backend_status=AsyncMock(side_effect=RuntimeError('box unavailable')),
)
assert await is_box_backend_available(SimpleNamespace(box_service=failing_backend)) is False
@@ -495,6 +731,34 @@ async def test_glob_caps_match_count_and_returns_preview():
assert result['truncated_by'] == 'matches'
@pytest.mark.asyncio
async def test_glob_runs_off_event_loop_and_caps_directory_walk(monkeypatch):
monkeypatch.setattr(native_loader, '_FILE_WALK_MAX_ENTRIES', 10)
with tempfile.TemporaryDirectory() as tmpdir:
loader, _ = _make_loader_with_workspace(tmpdir)
event_loop_thread = threading.get_ident()
observed_threads: list[int] = []
original = loader._glob_host_location
def observe(*args, **kwargs):
observed_threads.append(threading.get_ident())
return original(*args, **kwargs)
monkeypatch.setattr(loader, '_glob_host_location', observe)
for index in range(12):
with open(os.path.join(tmpdir, f'file-{index:03d}.txt'), 'w', encoding='utf-8') as f:
f.write(str(index))
result = await loader.invoke_tool('glob', {'path': '/workspace', 'pattern': '*.txt'}, _make_query())
assert result['ok'] is True
assert result['total'] == 10
assert result['truncated'] is True
assert result['truncated_by'] == 'scan'
assert observed_threads and observed_threads[0] != event_loop_thread
@pytest.mark.asyncio
async def test_grep_reports_invalid_regex_and_truncates_long_matching_lines():
with tempfile.TemporaryDirectory() as tmpdir:
@@ -512,3 +776,21 @@ async def test_grep_reports_invalid_regex_and_truncates_long_matching_lines():
assert result['truncated_by'] == 'line'
assert result['matches'][0]['file'] == '/workspace/data.txt'
assert result['matches'][0]['content'].endswith('... [truncated]')
@pytest.mark.asyncio
async def test_grep_interrupts_catastrophic_regex(monkeypatch):
monkeypatch.setattr(native_loader, '_GREP_REGEX_TIMEOUT_SECONDS', 0.001)
with tempfile.TemporaryDirectory() as tmpdir:
loader, _ = _make_loader_with_workspace(tmpdir)
with open(os.path.join(tmpdir, 'data.txt'), 'w', encoding='utf-8') as f:
f.write(('a' * 100_000) + '!')
result = await loader.invoke_tool(
'grep',
{'path': '/workspace', 'pattern': r'(a+)+$'},
_make_query(),
)
assert result == {'ok': False, 'error': 'Regex search timed out'}