feat(tenancy): add Workspace multi-tenant foundation (#2353)

* Document multi-tenant workspace architecture

* Add OSS and commercial workspace boundaries

* docs: redesign multi-tenant workspace architecture

* feat(tenancy): implement workspace isolation

* docs(tenancy): record verification evidence

* docs(tenancy): revise single-instance SaaS topology

* docs(tenancy): refine architecture options

* docs: finalize cloud v2 multi-tenant decisions

* feat(tenancy): establish cloud isolation foundations

* feat(tenancy): harden shared cloud runtime boundaries

* docs(tenancy): record final isolation verification

* fix(tenancy): close isolation and permission gaps

* docs(tenancy): record final isolation verification

* feat(tenancy): connect cloud workspace control plane

* fix(build): install git for pinned SDK

* docs(cloud): update control plane verification

* chore: update multi-tenant SDK pin

* fix(cloud): skip legacy model sync during startup

* test(cloud): preserve minimal model manager fixtures

* fix(cloud): preserve authenticated account context

* fix(cloud): reuse authenticated account for user info

* feat(cloud): complete Workspace settings navigation

* test(web): cover Workspace dropdown menu

* feat(web): place workspace controls in sidebar

* refactor(web): streamline workspace controls

* style(web): format workspace layout test

* fix(cloud): surface runtime and workspace plan status

* fix(plugin): keep runtime identity stable across restarts

* fix(ui): widen and center workspace switcher

* fix(ui): hide roles from workspace switcher

* fix(ui): align workspace switcher with sidebar entries

* feat(workspace): add in-product collaboration and direct Cloud launch

* style: format collaboration changes

* fix(workspace): bind collaboration APIs to tenant UoW

* fix(cloud): preserve Core-owned collaboration state

* test(cloud): require Space identity for invite registration

* feat(cloud): complete secure invitation experience

* style(web): format invitation flows

* fix(cloud): recover box runtime without unscoped skill reload

* feat(oss): enforce invitation account and owner billing flows

* style: format OSS account service

* test(oss): cover invitation logout handoff

* fix(oss): resolve workspace owner in scoped session

* feat(cloud): harden multi-tenant runtime resources

* fix(cloud): bound runtime restart storms

* fix(cloud): eliminate periodic runtime CPU spikes

* fix(cloud): enforce instance capacity ceilings

* fix(cloud): scope public login capability discovery

* fix(cloud): bound tenant maintenance and monitoring work

* fix(runtime): bound tenant resource amplification

* fix(deps): pin green multi-tenant plugin SDK

* fix(cloud): handle unavailable skill capability

* fix(security): require authentication for image file endpoint (H-2)

- Changed /api/v1/files/image from AuthType.NONE to USER_TOKEN_OR_API_KEY
- Added Permission.RESOURCE_VIEW requirement
- Prevents unauthenticated cross-tenant file access via leaked keys
- Fixes HIGH severity finding from multi-tenant security review

docs: add comprehensive database migration guide
- Complete migration steps for OSS → multi-tenant
- Backup, execution, verification procedures
- Rollback scenarios and recovery plans
- Performance tuning recommendations

* test: add comprehensive cross-tenant isolation tests

Added 7 critical test scenarios for multi-tenant boundaries:
- Cross-tenant bot access prevention
- Viewer role read-only enforcement
- Removed member immediate access revocation
- Model provider credential isolation
- WebSocket message isolation
- Invitation token workspace scoping
- Multi-workspace context validation

These tests address P0-2 coverage gaps for:
- workspaces.py (membership & invitation flows)
- user.py (authentication & authorization)
- websocket_chat.py (real-time isolation)
- plugins.py (resource access control)

docs: finalize database migration guide

* fix(security): resolve M-1, M-2, M-3 security findings

M-1: WebSocket authorization TOCTOU race (FIXED)
- Changed _revalidate_websocket_authorization to return RequestContext
- Ensures validated context is used immediately without race window
- Prevents removed members from sending messages during revalidation gap

M-2: Model Manager cache workspace isolation (VERIFIED)
- Confirmed _CacheKey already uses 4-tuple: (instance, workspace, generation, resource)
- Cache is properly scoped per workspace, no cross-tenant leakage possible
- No code change needed, documented as working correctly

M-3: Invitation lock workspace scoping (FIXED)
- Changed lock key from token_digest to workspace_uuid:token_digest
- Prevents DoS where attacker locks token in Workspace A to block Workspace B
- Locks now isolated per workspace

All MEDIUM severity findings from security review now resolved.

* fix(cloud): unblock tenant CI and enforce knowledge quotas

* fix(tenancy): scope rerank model sync

---------

Co-authored-by: dadachann <185672915+dadachann@users.noreply.github.com>

tests/unit_tests/vector/test_mgr.py

@@ -6,7 +6,9 @@ based on configuration, without actually creating real VDB instances.
 
 from __future__ import annotations
 
-from unittest.mock import MagicMock
+from unittest.mock import AsyncMock, MagicMock
+
+import pytest
 
 from tests.utils.import_isolation import isolated_sys_modules
 
@@ -138,6 +140,7 @@ class TestVectorDBManagerInitialization:
             mgr = VectorDBManager(mock_app)
 
             import asyncio
+
             asyncio.get_event_loop().run_until_complete(mgr.initialize())
 
             mock_valkey_class.assert_called_once_with(mock_app)
@@ -207,7 +210,10 @@ class TestVectorDBManagerInitialization:
             asyncio.get_event_loop().run_until_complete(mgr.initialize())
 
             mock_pgvector_class.assert_called_once_with(
-                mock_app, connection_string='postgresql://user:pass@host:5432/langbot'
+                mock_app,
+                connection_string='postgresql://user:pass@host:5432/langbot',
+                use_business_database=False,
+                allowed_dimensions=[384, 512, 768, 1024, 1536],
             )
 
     def test_initialize_pgvector_with_individual_params(self):
@@ -238,7 +244,14 @@ class TestVectorDBManagerInitialization:
             asyncio.get_event_loop().run_until_complete(mgr.initialize())
 
             mock_pgvector_class.assert_called_once_with(
-                mock_app, host='db.example.com', port=5433, database='vectordb', user='admin', password='secret'
+                mock_app,
+                host='db.example.com',
+                port=5433,
+                database='vectordb',
+                user='admin',
+                password='secret',
+                use_business_database=False,
+                allowed_dimensions=[384, 512, 768, 1024, 1536],
             )
 
     def test_initialize_pgvector_defaults(self):
@@ -260,7 +273,42 @@ class TestVectorDBManagerInitialization:
             asyncio.get_event_loop().run_until_complete(mgr.initialize())
 
             mock_pgvector_class.assert_called_once_with(
-                mock_app, host='localhost', port=5432, database='langbot', user='postgres', password='postgres'
+                mock_app,
+                host='localhost',
+                port=5432,
+                database='langbot',
+                user='postgres',
+                password='postgres',
+                use_business_database=False,
+                allowed_dimensions=[384, 512, 768, 1024, 1536],
+            )
+
+    def test_initialize_pgvector_with_shared_business_database(self):
+        vdb_config = {
+            'use': 'pgvector',
+            'pgvector': {
+                'use_business_database': True,
+                'allowed_dimensions': [768, 1536],
+            },
+        }
+        mock_app = self._create_mock_app(vdb_config)
+        mocks = self._make_vector_import_mocks()
+        mock_pgvector_class = MagicMock()
+        mocks['langbot.pkg.vector.vdbs.pgvector_db'].PgVectorDatabase = mock_pgvector_class
+
+        with isolated_sys_modules(mocks):
+            from langbot.pkg.vector.mgr import VectorDBManager
+
+            mgr = VectorDBManager(mock_app)
+
+            import asyncio
+
+            asyncio.get_event_loop().run_until_complete(mgr.initialize())
+
+            mock_pgvector_class.assert_called_once_with(
+                mock_app,
+                use_business_database=True,
+                allowed_dimensions=[768, 1536],
             )
 
     def test_initialize_unknown_backend_defaults_to_chroma(self):
@@ -337,3 +385,21 @@ class TestVectorDBManagerProxies:
 
             result = mgr.get_supported_search_types()
             assert result == ['vector', 'full_text']
+
+    @pytest.mark.asyncio
+    async def test_shutdown_closes_backend_and_releases_reference(self):
+        mock_app = MagicMock()
+        mocks = {'langbot.pkg.core.app': MagicMock()}
+
+        with isolated_sys_modules(mocks):
+            from langbot.pkg.vector.mgr import VectorDBManager
+
+            mgr = VectorDBManager(mock_app)
+            backend = MagicMock()
+            backend.close = AsyncMock()
+            mgr.vector_db = backend
+
+            await mgr.shutdown()
+
+            backend.close.assert_awaited_once_with()
+            assert mgr.vector_db is None

tests/unit_tests/vector/test_valkey_search_filter.py

@@ -31,6 +31,7 @@ def make_backend():
     # _ensure_client serializes creation through this lock; set it here since
     # __init__ (which normally creates it) is bypassed.
     backend._client_lock = asyncio.Lock()
+    backend._runtime_cache_limit = 1024
     return backend
 
 
@@ -267,9 +268,9 @@ class TestDeleteByFilterGuard:
         backend.ap = type('Ap', (), {'logger': AsyncMock()})()
         backend._ensure_client = AsyncMock(return_value=backend._client)
         backend._index_exists = AsyncMock(return_value=True)
-        # _search_keys must never be reached for an unusable filter.
-        backend._search_keys = AsyncMock(
-            side_effect=AssertionError('_search_keys must not be called for an unusable filter')
+        # The deletion scan must never be reached for an unusable filter.
+        backend._delete_search_results = AsyncMock(
+            side_effect=AssertionError('_delete_search_results must not be called for an unusable filter')
         )
 
         # Filter references only a non-indexed field -> maps to no FT conditions.
@@ -284,12 +285,57 @@ class TestDeleteByFilterGuard:
         backend.ap = type('Ap', (), {'logger': AsyncMock()})()
         backend._ensure_client = AsyncMock(return_value=backend._client)
         backend._index_exists = AsyncMock(return_value=True)
-        backend._search_keys = AsyncMock(return_value=['kb:col1:id1', 'kb:col1:id2'])
+        backend._delete_search_results = AsyncMock(return_value=2)
 
         deleted = await backend.delete_by_filter('col1', {'file_id': 'f1'})
 
         assert deleted == 2
-        backend._client.delete.assert_awaited_once_with(['kb:col1:id1', 'kb:col1:id2'])
+        backend._delete_search_results.assert_awaited_once_with(
+            backend._client,
+            backend._index_name('col1'),
+            '@file_id:{f1}',
+        )
+
+
+class TestBatchedDelete:
+    async def test_matching_keys_are_deleted_in_fixed_pages(self, monkeypatch):
+        mod = get_valkey_module()
+        backend = make_backend()
+        client = AsyncMock()
+        search = AsyncMock(
+            side_effect=[
+                [3, {b'key-1': {}, b'key-2': {}}],
+                [1, {b'key-3': {}}],
+            ]
+        )
+        monkeypatch.setattr(mod, '_DELETE_SCAN_BATCH', 2)
+        monkeypatch.setattr(mod, 'FtSearchLimit', lambda offset, limit: (offset, limit), raising=False)
+        monkeypatch.setattr(mod, 'FtSearchOptions', lambda **kwargs: kwargs, raising=False)
+        monkeypatch.setattr(mod, 'ft', type('FT', (), {'search': search})(), raising=False)
+
+        deleted = await backend._delete_search_results(client, 'idx:col1', '@file_id:{f1}')
+
+        assert deleted == 3
+        assert search.await_count == 2
+        assert [call.args[3]['limit'] for call in search.await_args_list] == [(0, 2), (0, 2)]
+        assert client.delete.await_args_list[0].args == (['key-1', 'key-2'],)
+        assert client.delete.await_args_list[1].args == (['key-3'],)
+
+    async def test_delete_rounds_have_a_hard_stop(self, monkeypatch):
+        mod = get_valkey_module()
+        backend = make_backend()
+        client = AsyncMock()
+        search = AsyncMock(return_value=[2, {b'key': {}}])
+        monkeypatch.setattr(mod, '_DELETE_SCAN_BATCH', 1)
+        monkeypatch.setattr(mod, '_MAX_DELETE_SCAN_ROUNDS', 2)
+        monkeypatch.setattr(mod, 'FtSearchLimit', lambda offset, limit: (offset, limit), raising=False)
+        monkeypatch.setattr(mod, 'FtSearchOptions', lambda **kwargs: kwargs, raising=False)
+        monkeypatch.setattr(mod, 'ft', type('FT', (), {'search': search})(), raising=False)
+
+        with pytest.raises(RuntimeError, match='exceeded 2 batches'):
+            await backend._delete_search_results(client, 'idx:col1', '@file_id:{f1}')
+
+        assert client.delete.await_count == 2
 
 
 class TestClose:

tests/unit_tests/vector/test_vdb_base.py

@@ -5,7 +5,12 @@ from __future__ import annotations
 from unittest.mock import AsyncMock
 import pytest
 
-from langbot.pkg.vector.vdb import SearchType, VectorDatabase
+from langbot.pkg.vector.vdb import (
+    SearchType,
+    VectorDatabase,
+    remember_bounded_mapping,
+    remember_bounded_set,
+)
 
 
 class TestSearchType:
@@ -29,6 +34,18 @@ class TestSearchType:
         assert SearchType('hybrid') == SearchType.HYBRID
 
 
+def test_runtime_cache_helpers_bound_mapping_and_set():
+    mapping = {}
+    values = set()
+
+    for index in range(100):
+        remember_bounded_mapping(mapping, str(index), object(), 8)
+        remember_bounded_set(values, str(index), 8)
+
+    assert len(mapping) == 8
+    assert len(values) == 8
+
+
 class TestVectorDatabaseAbstractMethods:
     """Tests for VectorDatabase abstract methods."""