feat(tenancy): add Workspace multi-tenant foundation (#2353)

* Document multi-tenant workspace architecture

* Add OSS and commercial workspace boundaries

* docs: redesign multi-tenant workspace architecture

* feat(tenancy): implement workspace isolation

* docs(tenancy): record verification evidence

* docs(tenancy): revise single-instance SaaS topology

* docs(tenancy): refine architecture options

* docs: finalize cloud v2 multi-tenant decisions

* feat(tenancy): establish cloud isolation foundations

* feat(tenancy): harden shared cloud runtime boundaries

* docs(tenancy): record final isolation verification

* fix(tenancy): close isolation and permission gaps

* docs(tenancy): record final isolation verification

* feat(tenancy): connect cloud workspace control plane

* fix(build): install git for pinned SDK

* docs(cloud): update control plane verification

* chore: update multi-tenant SDK pin

* fix(cloud): skip legacy model sync during startup

* test(cloud): preserve minimal model manager fixtures

* fix(cloud): preserve authenticated account context

* fix(cloud): reuse authenticated account for user info

* feat(cloud): complete Workspace settings navigation

* test(web): cover Workspace dropdown menu

* feat(web): place workspace controls in sidebar

* refactor(web): streamline workspace controls

* style(web): format workspace layout test

* fix(cloud): surface runtime and workspace plan status

* fix(plugin): keep runtime identity stable across restarts

* fix(ui): widen and center workspace switcher

* fix(ui): hide roles from workspace switcher

* fix(ui): align workspace switcher with sidebar entries

* feat(workspace): add in-product collaboration and direct Cloud launch

* style: format collaboration changes

* fix(workspace): bind collaboration APIs to tenant UoW

* fix(cloud): preserve Core-owned collaboration state

* test(cloud): require Space identity for invite registration

* feat(cloud): complete secure invitation experience

* style(web): format invitation flows

* fix(cloud): recover box runtime without unscoped skill reload

* feat(oss): enforce invitation account and owner billing flows

* style: format OSS account service

* test(oss): cover invitation logout handoff

* fix(oss): resolve workspace owner in scoped session

* feat(cloud): harden multi-tenant runtime resources

* fix(cloud): bound runtime restart storms

* fix(cloud): eliminate periodic runtime CPU spikes

* fix(cloud): enforce instance capacity ceilings

* fix(cloud): scope public login capability discovery

* fix(cloud): bound tenant maintenance and monitoring work

* fix(runtime): bound tenant resource amplification

* fix(deps): pin green multi-tenant plugin SDK

* fix(cloud): handle unavailable skill capability

* fix(security): require authentication for image file endpoint (H-2)

- Changed /api/v1/files/image from AuthType.NONE to USER_TOKEN_OR_API_KEY
- Added Permission.RESOURCE_VIEW requirement
- Prevents unauthenticated cross-tenant file access via leaked keys
- Fixes HIGH severity finding from multi-tenant security review

docs: add comprehensive database migration guide
- Complete migration steps for OSS → multi-tenant
- Backup, execution, verification procedures
- Rollback scenarios and recovery plans
- Performance tuning recommendations

* test: add comprehensive cross-tenant isolation tests

Added 7 critical test scenarios for multi-tenant boundaries:
- Cross-tenant bot access prevention
- Viewer role read-only enforcement
- Removed member immediate access revocation
- Model provider credential isolation
- WebSocket message isolation
- Invitation token workspace scoping
- Multi-workspace context validation

These tests address P0-2 coverage gaps for:
- workspaces.py (membership & invitation flows)
- user.py (authentication & authorization)
- websocket_chat.py (real-time isolation)
- plugins.py (resource access control)

docs: finalize database migration guide

* fix(security): resolve M-1, M-2, M-3 security findings

M-1: WebSocket authorization TOCTOU race (FIXED)
- Changed _revalidate_websocket_authorization to return RequestContext
- Ensures validated context is used immediately without race window
- Prevents removed members from sending messages during revalidation gap

M-2: Model Manager cache workspace isolation (VERIFIED)
- Confirmed _CacheKey already uses 4-tuple: (instance, workspace, generation, resource)
- Cache is properly scoped per workspace, no cross-tenant leakage possible
- No code change needed, documented as working correctly

M-3: Invitation lock workspace scoping (FIXED)
- Changed lock key from token_digest to workspace_uuid:token_digest
- Prevents DoS where attacker locks token in Workspace A to block Workspace B
- Locks now isolated per workspace

All MEDIUM severity findings from security review now resolved.

* fix(cloud): unblock tenant CI and enforce knowledge quotas

* fix(tenancy): scope rerank model sync

---------

Co-authored-by: dadachann <185672915+dadachann@users.noreply.github.com>

docker/docker-compose.yaml

@@ -14,6 +14,13 @@ services:
     restart: on-failure
     environment:
       - TZ=Asia/Shanghai
+      # Shared with the langbot service and sent only as a WebSocket handshake
+      # header. Generate with: openssl rand -hex 32
+      - LANGBOT_PLUGIN_RUNTIME_CONTROL_TOKEN=${LANGBOT_PLUGIN_RUNTIME_CONTROL_TOKEN:-}
+      # Process-wide admission for every asyncio.to_thread() call.
+      - LANGBOT_BLOCKING_EXECUTOR_MAX_WORKERS=${LANGBOT_BLOCKING_EXECUTOR_MAX_WORKERS:-8}
+      - LANGBOT_BLOCKING_EXECUTOR_MAX_PENDING=${LANGBOT_BLOCKING_EXECUTOR_MAX_PENDING:-128}
+      - LANGBOT_BLOCKING_EXECUTOR_MAX_INFLIGHT_PER_SCOPE=${LANGBOT_BLOCKING_EXECUTOR_MAX_INFLIGHT_PER_SCOPE:-4}
     command: ["uv", "run", "--no-sync", "-m", "langbot_plugin.cli.__init__", "rt"]
     networks:
       - langbot_network
@@ -40,9 +47,19 @@ services:
     restart: on-failure
     environment:
       - TZ=Asia/Shanghai
+      # Shared control-plane secret used to authenticate both the RPC socket
+      # and managed-process relay. Generate once (for example with
+      # ``openssl rand -hex 32``) and export it before enabling this profile.
+      # An empty value is accepted by Compose so Box can remain optional, but
+      # the Box runtime itself fails closed when the profile is started.
+      - LANGBOT_BOX_CONTROL_TOKEN=${LANGBOT_BOX_CONTROL_TOKEN:-}
+      # Box has its own process-wide blocking-work budget.
+      - LANGBOT_BLOCKING_EXECUTOR_MAX_WORKERS=${LANGBOT_BLOCKING_EXECUTOR_MAX_WORKERS:-8}
+      - LANGBOT_BLOCKING_EXECUTOR_MAX_PENDING=${LANGBOT_BLOCKING_EXECUTOR_MAX_PENDING:-128}
+      - LANGBOT_BLOCKING_EXECUTOR_MAX_INFLIGHT_PER_SCOPE=${LANGBOT_BLOCKING_EXECUTOR_MAX_INFLIGHT_PER_SCOPE:-4}
       # The Box runtime does NOT read box.local.* from config.yaml or env; it
-      # receives its configuration from LangBot via the INIT RPC action.
-      # Do not add LANGBOT_BOX_* / BOX__* here — they would be silently ignored.
+      # receives its functional configuration from LangBot via the INIT RPC
+      # action. Do not add BOX__* here because those would be ignored.
     # Launched through the same CLI entry point as the plugin runtime
     # (`langbot_plugin.cli.__init__ <subcommand>`). WebSocket is the default
     # control transport — mirrors `rt`, which also runs with no flag. Pass
@@ -60,6 +77,17 @@ services:
     restart: on-failure
     environment:
       - TZ=Asia/Shanghai
+      # Must match langbot_plugin_runtime. Empty/missing values make the
+      # external control channel fail closed.
+      - LANGBOT_PLUGIN_RUNTIME_CONTROL_TOKEN=${LANGBOT_PLUGIN_RUNTIME_CONTROL_TOKEN:-}
+      # Must match the value supplied to langbot_box. The token is sent only
+      # in WebSocket handshake headers, never in URLs or action payloads.
+      - LANGBOT_BOX_CONTROL_TOKEN=${LANGBOT_BOX_CONTROL_TOKEN:-}
+      # Core process-wide blocking-work admission. These are native config
+      # overrides and are persisted with the effective data/config.yaml.
+      - SYSTEM__BLOCKING_EXECUTOR__MAX_WORKERS=${LANGBOT_BLOCKING_EXECUTOR_MAX_WORKERS:-8}
+      - SYSTEM__BLOCKING_EXECUTOR__MAX_PENDING=${LANGBOT_BLOCKING_EXECUTOR_MAX_PENDING:-128}
+      - SYSTEM__BLOCKING_EXECUTOR__MAX_INFLIGHT_PER_SCOPE=${LANGBOT_BLOCKING_EXECUTOR_MAX_INFLIGHT_PER_SCOPE:-4}
       # Unified env-override convention: SECTION__SUBSECTION__KEY overrides the
       # matching config.yaml field (see LoadConfigStage). These map onto
       # box.* and are forwarded to the Box runtime via INIT RPC.

docker/kubernetes.yaml

@@ -4,6 +4,10 @@
 # Full deployment guide (zh/en/ja): https://docs.langbot.app -> Installation -> Kubernetes
 #
 # Usage:
+#   kubectl -n langbot create secret generic langbot-plugin-runtime-control \
+#     --from-literal=token="$(openssl rand -hex 32)"
+#   kubectl -n langbot create secret generic langbot-box-control \
+#     --from-literal=token="$(openssl rand -hex 32)"
 #   kubectl apply -f kubernetes.yaml
 #
 # Prerequisites:
@@ -87,6 +91,12 @@ metadata:
 data:
   TZ: "Asia/Shanghai"
   PLUGIN__RUNTIME_WS_URL: "ws://langbot-plugin-runtime:5400/control/ws"
+  SYSTEM__BLOCKING_EXECUTOR__MAX_WORKERS: "8"
+  SYSTEM__BLOCKING_EXECUTOR__MAX_PENDING: "128"
+  SYSTEM__BLOCKING_EXECUTOR__MAX_INFLIGHT_PER_SCOPE: "4"
+  LANGBOT_BLOCKING_EXECUTOR_MAX_WORKERS: "8"
+  LANGBOT_BLOCKING_EXECUTOR_MAX_PENDING: "128"
+  LANGBOT_BLOCKING_EXECUTOR_MAX_INFLIGHT_PER_SCOPE: "4"
   # Box sandbox runtime endpoint. LangBot connects to the Box runtime over
   # WebSocket. The hostname MUST match the langbot-box Service name. Note the
   # in-container default ("langbot_box") uses an underscore, which is an
@@ -127,6 +137,26 @@ spec:
             configMapKeyRef:
               name: langbot-config
               key: TZ
+        - name: LANGBOT_PLUGIN_RUNTIME_CONTROL_TOKEN
+          valueFrom:
+            secretKeyRef:
+              name: langbot-plugin-runtime-control
+              key: token
+        - name: LANGBOT_BLOCKING_EXECUTOR_MAX_WORKERS
+          valueFrom:
+            configMapKeyRef:
+              name: langbot-config
+              key: LANGBOT_BLOCKING_EXECUTOR_MAX_WORKERS
+        - name: LANGBOT_BLOCKING_EXECUTOR_MAX_PENDING
+          valueFrom:
+            configMapKeyRef:
+              name: langbot-config
+              key: LANGBOT_BLOCKING_EXECUTOR_MAX_PENDING
+        - name: LANGBOT_BLOCKING_EXECUTOR_MAX_INFLIGHT_PER_SCOPE
+          valueFrom:
+            configMapKeyRef:
+              name: langbot-config
+              key: LANGBOT_BLOCKING_EXECUTOR_MAX_INFLIGHT_PER_SCOPE
         volumeMounts:
         - name: plugin-data
           mountPath: /app/data/plugins
@@ -248,9 +278,28 @@ spec:
             configMapKeyRef:
               name: langbot-config
               key: TZ
+        - name: LANGBOT_BOX_CONTROL_TOKEN
+          valueFrom:
+            secretKeyRef:
+              name: langbot-box-control
+              key: token
+        - name: LANGBOT_BLOCKING_EXECUTOR_MAX_WORKERS
+          valueFrom:
+            configMapKeyRef:
+              name: langbot-config
+              key: LANGBOT_BLOCKING_EXECUTOR_MAX_WORKERS
+        - name: LANGBOT_BLOCKING_EXECUTOR_MAX_PENDING
+          valueFrom:
+            configMapKeyRef:
+              name: langbot-config
+              key: LANGBOT_BLOCKING_EXECUTOR_MAX_PENDING
+        - name: LANGBOT_BLOCKING_EXECUTOR_MAX_INFLIGHT_PER_SCOPE
+          valueFrom:
+            configMapKeyRef:
+              name: langbot-config
+              key: LANGBOT_BLOCKING_EXECUTOR_MAX_INFLIGHT_PER_SCOPE
         # The Box runtime does NOT read box.local.* / BOX__* from its own env;
-        # it receives its configuration from LangBot via the INIT RPC action.
-        # Do not add BOX__* here — they would be silently ignored.
+        # it receives its functional configuration from LangBot via INIT.
         volumeMounts:
         # Box workspace root — identical path on node, box, and sandbox
         # containers (see the IMPORTANT note above).
@@ -276,7 +325,9 @@ spec:
           failureThreshold: 3
         readinessProbe:
           httpGet:
-            path: /healthz
+            # Unlike liveness, readiness validates the configured backend and
+            # all strict managed-mode isolation guarantees.
+            path: /readyz
             port: 5410
           initialDelaySeconds: 10
           periodSeconds: 5
@@ -360,6 +411,26 @@ spec:
             configMapKeyRef:
               name: langbot-config
               key: PLUGIN__RUNTIME_WS_URL
+        - name: LANGBOT_PLUGIN_RUNTIME_CONTROL_TOKEN
+          valueFrom:
+            secretKeyRef:
+              name: langbot-plugin-runtime-control
+              key: token
+        - name: SYSTEM__BLOCKING_EXECUTOR__MAX_WORKERS
+          valueFrom:
+            configMapKeyRef:
+              name: langbot-config
+              key: SYSTEM__BLOCKING_EXECUTOR__MAX_WORKERS
+        - name: SYSTEM__BLOCKING_EXECUTOR__MAX_PENDING
+          valueFrom:
+            configMapKeyRef:
+              name: langbot-config
+              key: SYSTEM__BLOCKING_EXECUTOR__MAX_PENDING
+        - name: SYSTEM__BLOCKING_EXECUTOR__MAX_INFLIGHT_PER_SCOPE
+          valueFrom:
+            configMapKeyRef:
+              name: langbot-config
+              key: SYSTEM__BLOCKING_EXECUTOR__MAX_INFLIGHT_PER_SCOPE
         # Box (sandbox) runtime endpoint. Connects LangBot to the langbot-box
         # Service over WebSocket. Remove this (and the langbot-box Deployment)
         # and set BOX__ENABLED=false if you do not want the sandbox.
@@ -368,6 +439,13 @@ spec:
             configMapKeyRef:
               name: langbot-config
               key: BOX__RUNTIME__ENDPOINT
+        # Same Secret as langbot-box. It authenticates the RPC and managed-
+        # process relay handshakes and is never put in a URL or RPC payload.
+        - name: LANGBOT_BOX_CONTROL_TOKEN
+          valueFrom:
+            secretKeyRef:
+              name: langbot-box-control
+              key: token
         # box.local.* config — forwarded to the Box runtime via INIT RPC. The
         # host_root MUST match the box-root hostPath mountPath below AND the box
         # Deployment's box-root mountPath, so that skill package paths resolve
@@ -400,7 +478,7 @@ spec:
         # Liveness probe to restart container if it becomes unresponsive
         livenessProbe:
           httpGet:
-            path: /
+            path: /healthz
             port: 5300
           initialDelaySeconds: 60
           periodSeconds: 10
@@ -409,7 +487,7 @@ spec:
         # Readiness probe to know when container is ready to accept traffic
         readinessProbe:
           httpGet:
-            path: /
+            path: /healthz
             port: 5300
           initialDelaySeconds: 30
           periodSeconds: 5