feat(tenancy): add Workspace multi-tenant foundation (#2353)

* Document multi-tenant workspace architecture * Add OSS and commercial workspace boundaries * docs: redesign multi-tenant workspace architecture * feat(tenancy): implement workspace isolation * docs(tenancy): record verification evidence * docs(tenancy): revise single-instance SaaS topology * docs(tenancy): refine architecture options * docs: finalize cloud v2 multi-tenant decisions * feat(tenancy): establish cloud isolation foundations * feat(tenancy): harden shared cloud runtime boundaries * docs(tenancy): record final isolation verification * fix(tenancy): close isolation and permission gaps * docs(tenancy): record final isolation verification * feat(tenancy): connect cloud workspace control plane * fix(build): install git for pinned SDK * docs(cloud): update control plane verification * chore: update multi-tenant SDK pin * fix(cloud): skip legacy model sync during startup * test(cloud): preserve minimal model manager fixtures * fix(cloud): preserve authenticated account context * fix(cloud): reuse authenticated account for user info * feat(cloud): complete Workspace settings navigation * test(web): cover Workspace dropdown menu * feat(web): place workspace controls in sidebar * refactor(web): streamline workspace controls * style(web): format workspace layout test * fix(cloud): surface runtime and workspace plan status * fix(plugin): keep runtime identity stable across restarts * fix(ui): widen and center workspace switcher * fix(ui): hide roles from workspace switcher * fix(ui): align workspace switcher with sidebar entries * feat(workspace): add in-product collaboration and direct Cloud launch * style: format collaboration changes * fix(workspace): bind collaboration APIs to tenant UoW * fix(cloud): preserve Core-owned collaboration state * test(cloud): require Space identity for invite registration * feat(cloud): complete secure invitation experience * style(web): format invitation flows * fix(cloud): recover box runtime without unscoped skill reload * feat(oss): enforce invitation account and owner billing flows * style: format OSS account service * test(oss): cover invitation logout handoff * fix(oss): resolve workspace owner in scoped session * feat(cloud): harden multi-tenant runtime resources * fix(cloud): bound runtime restart storms * fix(cloud): eliminate periodic runtime CPU spikes * fix(cloud): enforce instance capacity ceilings * fix(cloud): scope public login capability discovery * fix(cloud): bound tenant maintenance and monitoring work * fix(runtime): bound tenant resource amplification * fix(deps): pin green multi-tenant plugin SDK * fix(cloud): handle unavailable skill capability * fix(security): require authentication for image file endpoint (H-2) - Changed /api/v1/files/image from AuthType.NONE to USER_TOKEN_OR_API_KEY - Added Permission.RESOURCE_VIEW requirement - Prevents unauthenticated cross-tenant file access via leaked keys - Fixes HIGH severity finding from multi-tenant security review docs: add comprehensive database migration guide - Complete migration steps for OSS → multi-tenant - Backup, execution, verification procedures - Rollback scenarios and recovery plans - Performance tuning recommendations * test: add comprehensive cross-tenant isolation tests Added 7 critical test scenarios for multi-tenant boundaries: - Cross-tenant bot access prevention - Viewer role read-only enforcement - Removed member immediate access revocation - Model provider credential isolation - WebSocket message isolation - Invitation token workspace scoping - Multi-workspace context validation These tests address P0-2 coverage gaps for: - workspaces.py (membership & invitation flows) - user.py (authentication & authorization) - websocket_chat.py (real-time isolation) - plugins.py (resource access control) docs: finalize database migration guide * fix(security): resolve M-1, M-2, M-3 security findings M-1: WebSocket authorization TOCTOU race (FIXED) - Changed _revalidate_websocket_authorization to return RequestContext - Ensures validated context is used immediately without race window - Prevents removed members from sending messages during revalidation gap M-2: Model Manager cache workspace isolation (VERIFIED) - Confirmed _CacheKey already uses 4-tuple: (instance, workspace, generation, resource) - Cache is properly scoped per workspace, no cross-tenant leakage possible - No code change needed, documented as working correctly M-3: Invitation lock workspace scoping (FIXED) - Changed lock key from token_digest to workspace_uuid:token_digest - Prevents DoS where attacker locks token in Workspace A to block Workspace B - Locks now isolated per workspace All MEDIUM severity findings from security review now resolved. * fix(cloud): unblock tenant CI and enforce knowledge quotas * fix(tenancy): scope rerank model sync --------- Co-authored-by: dadachann <185672915+dadachann@users.noreply.github.com>
2026-08-09 04:40:57 +00:00 · 2026-07-30 21:43:35 +08:00
parent 463b120923
commit e1ac5e0fc8
468 changed files with 78320 additions and 13137 deletions
@@ -0,0 +1,271 @@
+# Cloud v2 multi-tenant verification report
+
+Date: 2026-07-24
+
+Status: `FOUNDATION AND CONTROL PLANE VERIFIED — PRODUCTION ACTIVATION REMAINS GATED`
+
+This report records the implementation and verification evidence for one
+logical LangBot instance serving multiple Workspace tenants. It covers the
+open-source Core, the shared Plugin/Box runtimes, the closed Space adapter, and
+the Space Cloud v2 modular-monolith control plane. It does not claim that the
+production Cloud deployment may enable `CLOUD_V2_ENABLED` yet.
+
+## Repository refs and scope
+
+- LangBot Core: branch `feat/multi-tenants`, commit
+  `e8a09b7537ef285a967f24add05fdb9bb557b97e`
+- langbot-plugin-sdk: branch `feat/multi-tenants`, head
+  `ca545d079ca1657a5d4efb4e31bfeafe1a374a46`
+- langbot-space: branch `feat/cloud-v2-control-plane`, head
+  `ce41ff370e94a405f70e2fbb2f99b0946e0e0387`
+- Closed Core adapter: `langbot-space/cloud-adapter`
+- SDK protocol/package version: `0.4.18`
+
+Core pins SDK commit `e7d946af4a6b1494fbe74627c1815ace19ac8991`;
+the SDK branch head adds CI-only follow-up. Cloud v2 is a greenfield
+multi-Workspace deployment and does not provision one Pod, database, queue, or
+Runtime per tenant. Legacy Space Pods remain a compatibility surface only.
+
+## Implemented product and security boundaries
+
+### Workspace and identity
+
+- SaaS has one stable `instance_uuid`; `workspace_uuid` is the tenant key.
+- Registering an Account creates its personal Workspace and owner Membership in
+  the same PostgreSQL transaction.
+- A Workspace may contain multiple users with owner, admin, and member roles.
+  Invitation acceptance is token-hash based, email-bound, single-use,
+  concurrency-safe, and member-limit checked while locked.
+- Community remains exactly one local Workspace while supporting multiple
+  Accounts and fixed RBAC roles. Installing the closed adapter is required to
+  inject the SaaS policy; configuration alone cannot activate it.
+- Disabled or deleted Accounts cannot use password/code/OAuth login, sessions,
+  refresh/access tokens, or personal access tokens.
+
+### Closed Space control plane
+
+- `CLOUD_V2_ENABLED` defaults to false. Invalid or incomplete configuration
+  fails startup, and disabled endpoints return 503.
+- Space owns Workspace/Membership/Invitation, versioned plans, subscriptions,
+  entitlements, usage events, directory outbox, and signed instance manifests.
+- Free and Pro are Workspace plans. Pro projects one managed global sandbox;
+  Free projects none. Both force stdio MCP off in the first release.
+- Subscription changes use Workspace advisory locking, expected entitlement
+  revision, one-live-order uniqueness, idempotent usage ingestion, period-end
+  downgrade, renewal, and lazy expiry settlement.
+- Account provisioning and quota projection into New API use PostgreSQL
+  transactional outboxes, replica-safe claims, retry/backoff, revision fencing,
+  and database-enforced identity/user/token ownership. No additional service is
+  introduced.
+- EPay and Stripe callbacks are bound to the locked order's provider identity,
+  amount, currency, channel/session, expiry, and provider transaction ID.
+  Replays are idempotent, EPay is CNY-only, credential rotation is supported by
+  encrypted per-order snapshots, and permanent fulfillment conflicts are
+  retained for reconciliation.
+- Directory snapshot and per-Workspace delta reads use repeatable-read,
+  read-only transactions with a high-water cursor. Core stores a replica-local
+  consumer cursor and shared PostgreSQL projection state, snapshot coverage,
+  and inbox rows.
+- The `/cloud` page selects a Workspace and shows its independent subscription,
+  entitlement, limits, and usage. When Cloud v2 is disabled or the backend does
+  not expose the feature field, the complete legacy Welcome/Pod UI is used.
+
+### PostgreSQL and pgvector
+
+- SaaS business data uses one PostgreSQL shared schema with application scope
+  plus forced RLS. Cloud directory writes are separated from local tenant
+  writes.
+- Projected Account and Membership revisions are monotonic. Tombstones remove
+  memberships, and stale revisions cannot resurrect them.
+- pgvector shares the business PostgreSQL database and remains
+  Workspace-scoped.
+- Space runs versioned SQL migrations before application seeds. A fresh
+  database, a partial pre-migration database, and an existing-baseline path
+  converge without startup-time `AutoMigrate`.
+
+### Shared Plugin Runtime
+
+- One instance-scoped trusted Supervisor serves multiple Workspaces.
+- Every enabled installation has its own nsjail process bound to
+  `(instance, workspace, execution generation, installation, runtime revision,
+  artifact digest)`.
+- Verified same-digest code and dependency files are mounted read-only and may
+  be shared; home, tmp, data, process namespace, registration capability, and
+  cgroup are private.
+- Instance configuration owns CPU, memory, PID, open-file, and file-size
+  limits. Plugin manifests cannot increase them.
+- Memory includes swap: nsjail receives `memory.max` and `swap.max=0`.
+- Unexpected worker exit is recovered by a completion callback with bounded
+  per-installation exponential backoff. Remove, reconcile, Runtime shutdown,
+  and container SIGTERM perform a graceful-to-SIGKILL bounded reap.
+
+### Shared Box Runtime and MCP
+
+- One shared Box control plane serves Workspace-bound logical sessions.
+- Cloud grants allow at most one persistent `global` session for an entitled
+  Workspace and no managed processes in the first release.
+- Cloud is fixed to nsjail and network-off. Core and Box prove the shared
+  durable Workspace mount with an authenticated marker challenge.
+- stdio MCP is independently gated and forced off for Cloud v2.
+- The current nsjail Box backend does not provide hard byte/inode quotas, so
+  Cloud readiness correctly fails closed instead of silently using a soft
+  directory scan.
+
+## Automated verification
+
+### LangBot Core
+
+```text
+uv run --no-sync pytest -q
+  2590 passed, 32 skipped, 177 warnings
+
+real PostgreSQL migration, pgvector, and release-migrator suites
+  21 passed, 11 warnings
+
+uv run --no-sync ruff check .
+uv lock --check
+git diff --check
+  passed
+```
+
+The full suite ran without the closed adapter installed, proving the open-source
+single-Workspace/multi-user path remains standalone. Focused closed-adapter,
+directory projection, runtime connector, Box cleanup, and configuration suites
+also passed with the adapter installed.
+
+### Plugin SDK and real Linux runtime
+
+```text
+SDK full suite
+  1226 passed, 22 existing warnings
+
+Ruff check and format check
+git diff --check
+  passed
+```
+
+A privileged Linux test container with host cgroup namespace ran one shared
+Runtime and two Workspace installations:
+
+- both workers referenced the same artifact inode;
+- home, tmp, and data inodes were distinct;
+- each plugin saw only PID 1 in its private PID namespace;
+- a tampered binding and an unknown installation were rejected;
+- the control token was absent from worker environments;
+- cgroups were distinct with `memory.max=134217728`, `memory.swap.max=0`,
+  `pids.max=32`, and `cpu.max=500000 1000000`;
+- touching 256 MiB exited with code 137 without swap growth;
+- the 32nd fork failed with `EAGAIN`;
+- reconcile and container SIGTERM removed the worker cgroups.
+
+The same run started the Runtime from a non-root working directory, covering
+absolute nsjail mount-source normalization.
+
+### Space backend, adapter, and frontend
+
+```text
+MIGRATIONS_TEST_DSN=... MIGRATIONS_TEST_DSN_FRESH=... \
+  go test -count=1 ./...
+go vet ./...
+  passed against PostgreSQL 16
+
+fresh PostgreSQL app startup, partial-baseline migration,
+Cloud v2 migration rerun, and control-plane integration
+  passed
+
+closed adapter pytest and Ruff
+  passed
+
+pnpm exec tsc --noEmit
+pnpm check:i18n
+pnpm check:cloud-checkout-currency
+  passed; 7 checkout/currency cases
+```
+
+The PostgreSQL checks started from an empty database and verified all 34
+registered migrations in order, Cloud v2 Free/Pro seeds, legacy plan seeds,
+Cloud columns/indexes, payment callback constraints, New API outbox/ownership
+constraints, and repeatable reruns.
+
+## Cross-service and browser E2E
+
+### Signed Space-to-Core directory projection
+
+Using an isolated Space PostgreSQL database and a migrated Core PostgreSQL
+database:
+
+1. Space issued a signed manifest for the fixed LangBot instance.
+2. Space returned a directory snapshot at cursor 5 containing two Workspaces,
+   two projected Accounts, and owner/admin memberships.
+3. Core verified the signature, instance, release, validity window, and
+   capability before injecting the Cloud Workspace policy.
+4. Core stored both Workspaces, all active Memberships, both projected
+   Accounts, snapshot coverage, inbox entries, and cursor 5.
+5. Account-field-only projection revisions and Workspace directory revisions
+   remained independent, and cross-Workspace Account conflicts failed closed.
+
+### Real browser Cloud v2 flow
+
+A real local browser operated the Space frontend and backend:
+
+1. An existing legacy-Pod owner logged in and saw the automatically created
+   personal Workspace on Free.
+2. The page showed Free and Pro Workspace plans while retaining the legacy Pro
+   instance card, its Online state, URL, version, billing period, and actions.
+3. Annual Pro checkout through the configured EPay/Alipay rail was re-quoted
+   from the displayed USD plan price to `¥490.00 CNY`.
+4. The browser reached the EPay gateway with `money=490.00`; no USD amount was
+   sent through the CNY-only rail.
+5. A valid signed `TRADE_SUCCESS` callback returned `success`. An exact replay
+   also returned `success`, leaving one successful order and one Pro Workspace
+   subscription.
+6. Refreshing payment state cleared the pending indicator. The page then showed
+   the Pro annual period and managed-sandbox entitlement while the legacy Pod
+   remained Online.
+
+The browser run used the real Space UI and HTTP handlers. Its disposable local
+development harness added a same-origin Next.js rewrite only in the temporary
+worktree; that harness change was removed after the run.
+
+The legacy feature-flag branch was then covered by API/static checks and the
+production build: false or missing `cloud_v2_enabled` renders the original
+Welcome/Pod client; a failed web-config request renders an explicit retry
+instead of guessing a deployment mode.
+
+### Deliberate Core startup failure
+
+Core completed signed manifest verification and directory projection, then
+stopped at the Box readiness gate because the current nsjail backend cannot
+prove hard Workspace byte/inode quota enforcement. Connector shutdown and
+reconnect tasks were cleanly reaped; no event-loop or never-awaited coroutine
+warning remained.
+
+This is a successful fail-closed acceptance result, not a passing production
+Cloud boot.
+
+## Remaining production activation gates
+
+Cloud v2 must remain disabled until these gates are closed:
+
+- Box provides and proves hard byte and inode quotas for Workspace, Skill,
+  root, home, and tmp storage.
+- Plugin installation writable data receives an operator-owned hard total
+  disk quota.
+- Plugin and future networked Box workloads have tenant-safe egress/SSRF
+  policy.
+- Plugin Runtime adds jitter, global restart concurrency limiting, and a
+  Runtime-level circuit breaker, then passes systemic-failure injection.
+- M0 rolls Core and Plugin Runtime together until authenticated Runtime takeover
+  or an owner lease/fencing protocol exists.
+- Payment operations add scheduled reconciliation and alerting for stale
+  `processing` orders and persisted permanent fulfillment conflicts.
+- Cloud v2 subscription service periods are stored immutably and included in
+  recognized-revenue reporting.
+- Production migration Job, backup/rollback, PostgreSQL credential/network
+  boundaries, horizontal-replica fault injection, Workspace release/export,
+  deletion, and restore semantics are completed.
+
+These gates intentionally add no tenant-specific service. They are implemented
+inside the existing Space, Core, Plugin Runtime, Box Runtime, and PostgreSQL
+components to preserve the architecture goal: near-zero static cost for a new
+Workspace.