# Multi-tenant isolation kernel verification report Date: 2026-07-20 Status: `CORE ISOLATION FOUNDATION VERIFIED — SAAS ACTIVATION REMAINS DISABLED` This report records the final implementation and verification evidence for the shared Workspace isolation foundation. It does not claim that the closed SaaS Control Plane, billing system, or greenfield Cloud v2 deployment is production-ready. ## Repository refs and scope - LangBot branch: `feat/multi-tenants` - LangBot implementation commit: `90a977488212d3f3f50fb92d94e3fcae60f27eab` - LangBot base: `origin/master` at `6baeb032a7f76c65337b51c7b58593de4687a61c` - Plugin SDK branch: `feat/multi-tenants` - Plugin SDK commit and LangBot dependency pin: `95a1805af2038c745de2c018a00db1305089a32e` - No tracked `langbot-space` change was made. Cloud v2 does not extend or preserve the legacy per-account instance/pod deployment topology. The SDK protocol is versioned as 0.4.15 and is intentionally consumed from the exact pushed Git commit above. Publishing the release and replacing the Git pin with a registry pin remain merge-to-master release work; no registry release was performed here. ## Implemented boundaries ### Workspace, identity, and authorization - Community bootstraps exactly one local Workspace per LangBot instance while allowing multiple Accounts, memberships, invitations, and role-based permissions inside it. A mutable edition setting cannot activate SaaS multi-Workspace routing. - Each Account registration resolves or creates its local singleton Workspace. Invitation acceptance is one-time, email-bound, and survives the signed-out login transition without exposing the invitation secret in URLs after the initial fragment capture. - The Account-token Workspace discovery route is intentionally separate from Workspace-authorized routes. `X-Workspace-Id` becomes authority only after membership resolution; API keys and public resources derive Workspace from their trusted owner. - Ordinary monitoring uses `resource.view`. Export remains `data.export`, privileged runtime/system audit remains `audit.view`, and frontend controls follow the same permission split. ### Persistence and PostgreSQL - PostgreSQL uses one shared business schema with application scope plus exact `ENABLE` and `FORCE ROW LEVEL SECURITY` policies. Tenant UoWs keep `SET LOCAL` and business SQL on one owned root transaction and one task. - The public tenant Session accepts only structured SQLAlchemy query/DML trees. Raw/textual SQL, textual modifiers, custom AST/compiler nodes, unapproved functions/operators/types/casts, hidden dialect values, foreign binds, loader/execution options, Session event callbacks, ORM SQL-expression attributes, transaction control, and live-result escape fail closed and make the root transaction rollback-only. - The SQL guard is a trusted-Core misuse boundary, not an in-process Python sandbox. Mapped metadata and registered compilers are trusted boot-time code; plugins remain out of process. SQLAlchemy dialect-private container fields are pinned and covered by regression tests before dependency upgrades. - Legacy local RAG restore remains structured and tenant-fenced while accepting SQLite string-valued timestamps and both historical PostgreSQL `TEXT` and fresh-schema `JSON` settings columns. - The one-shot release migrator owns DDL and grants a distinct runtime role only the required business-table DML, `alembic_version` read access, sequence access, database `CONNECT`, and schema `USAGE`. Runtime startup reruns the role, ACL, schema, session, extension, routine, parameter, and RLS audit. - pgvector is stored in the same business database with Workspace-scoped keys, checked dimensions, release-created partial ANN indexes, and RLS. ### Shared Plugin Runtime and Box - One instance-scoped Plugin Runtime supervisor serves multiple Workspaces. Every enabled installation still owns one nsjail worker with an immutable instance/Workspace/generation/installation/revision/artifact binding. - Verified same-digest code and dependency environments may be mounted read-only and reused. Plugin processes and writable installation state are never merged, including for the same plugin and version. - Runtime limits come only from instance configuration with native environment overrides. Plugin manifests cannot raise limits. Shared mode rejects legacy global plugin directories and legacy lifecycle control paths. - One shared Box control plane admits at most one persistent logical `global` sandbox for an entitled Workspace. Admission is entitlement-gated, generation-fenced, shared-volume-challenged, quota-aware, and fixed to nsjail. Plain nsjail correctly fails Cloud readiness without a hard quota provider. - stdio MCP has an independent gate and is forced off by Cloud bootstrap even when Box is available. ## Automated verification ### LangBot backend and static checks ```text uv run pytest -q --tb=short 2527 passed, 32 skipped, 177 warnings in 73.37s focused tenant UoW and legacy RAG compatibility suite 120 passed, 10 warnings in 1.30s uv run ruff check . passed changed Python format check 15 files already formatted git diff --check passed ``` The full suite includes startup E2E, the fresh SQLite registration/Workspace journey, API authorization, storage, RAG, Plugin, MCP, and 18 Docker-backed Box integration cases. ### Real PostgreSQL 16 and pgvector ```text test_migrations_postgres.py test_pgvector_postgres.py test_release_migration_postgres.py 21 passed, 11 warnings in 11.61s ``` The real-database suite covers fresh and upgraded RLS schemas, two-Workspace isolation, pgvector CRUD and ANN indexes, least-privilege runtime bootstrap, release locking, historical/fresh RAG settings column compatibility, and transaction-scope escape rejection. Negative tests inject and reject role membership, grant options, persistent GUCs, extra schemas and relations, column ACLs, explicit routine and parameter ACLs, runtime-owned and `SECURITY DEFINER` routines, extra/runtime-owned extensions, and foreign data wrappers, servers, and user mappings. Alembic has exactly one head: `0013_tenant_pgvector`. ### Plugin SDK and Box Runtime ```text Plugin SDK full suite 1160 passed, 18 warnings Plugin SDK Ruff and action consistency checks passed Docker-backed Box integration suite 18 passed, 33 warnings ``` The LangBot environment resolves `langbot-plugin==0.4.15` from the same exact SDK commit recorded above. The SDK local ref and remote `refs/heads/feat/multi-tenants` were independently verified equal. ### Frontend ```text pnpm lint 0 errors, 34 existing warnings VITE_API_BASE_URL=/ pnpm build passed; 3168 modules transformed pnpm exec playwright test --project=chromium 43 passed in 22.2s ``` The frontend suite includes Viewer monitoring visibility, privileged control hiding, stable terminal invitation states, email-mismatch login handoff, and a temporary invitation-accept failure that retains the authenticated session and succeeds on retry. ### Packaging and deployment configuration - `uv lock --check` resolved 277 packages and passed. - Changed TypeScript/TSX files passed Prettier verification. - `docker compose -f docker/docker-compose.yaml config --quiet` passed with the existing warning that the top-level `version` field is obsolete. - The staged high-signal credential scan found no private key or provider token. ## Real browser E2E A production frontend and backend were run against an isolated temporary SQLite data directory and operated through a real local Chrome session: 1. The first Account registered and automatically received the default singleton Workspace as owner. 2. Skipping the setup wizard reached `/home/monitoring`; reload and a new tab recovered the authenticated Account, membership, and Workspace selection. 3. The owner invited a Viewer. After signing out, the second Account registered through the invitation and joined the same Workspace. 4. Owner and Admin could read overview/export/audit/API-key surfaces and reach resource-write validation. Runtime debug mutation remained disabled. 5. Developer retained ordinary resource management and monitoring, while export, audit, and API-key management returned 403. 6. Operator and Viewer retained ordinary monitoring reads while resource writes, export, audit, API-key management, and runtime mutation returned 403. 7. The UI matched the server matrix: Add/API Key/Export controls appeared only with their permissions, while ordinary monitoring refresh remained visible to read-only members. 8. A second Workspace creation attempt returned 403 `edition_limit`, proving Community remains single-Workspace and multi-user. 9. Used, revoked, expired, and email-mismatched invitation paths produced stable visible states. The mismatch survived login without a contradictory generic success toast. The server was stopped, port 15321 was verified free, and temporary artifacts were moved to the system Trash for recoverable cleanup. No invitation secret, password, token, or provider credential is recorded in this report. ## Deliberately incomplete SaaS activation gates Multi-Workspace activation remains unavailable in the open-source bootstrap. Cloud v2 must stay disabled until all of the following are implemented and independently verified: - the closed Control Plane owns the Account/Workspace directory, memberships, invitations, monotonic execution generations, entitlements, usage, subscriptions, and billing; - ordinary tenant writes hold a generation-aware fence through commit, and a generation-stamped outbox or equivalent atomic publish fence protects external side effects; - durable object references survive generation cutovers without stranding files, images, plugin artifacts, or knowledge-base content; - every tenant-configurable outbound path has complete SSRF/egress policy, and Plugin Runtime has production Linux namespace, cgroup v2, and network isolation evidence; - an unexpectedly exited enabled plugin worker is restored by a completion callback with bounded backoff and cross-tenant restart-storm isolation; - plugin installation data has an operator-owned hard disk quota, not a directory-scan approximation; - Box supplies and proves hard byte/inode quotas for Workspace, Skill, root, home, and temporary storage, and the production shared volume passes the authenticated marker challenge; - production PostgreSQL uses a dedicated endpoint/cluster, or a tested HBA/proxy policy proves the runtime credential can connect only to the target business database; - the release migrator is deployed as a one-shot Job with credential issuance, backup, rollback, failure/retry, and orchestration procedures; - OAuth exchange, directory projection, leases, snapshots, event replay, and outbox state use durable shared stores suitable for horizontally scaled replicas; - Workspace release, export, deletion, restore, and disaster-recovery semantics are decided and implemented; and - production Runtime restart, crash, partition, revocation, and fault-injection acceptance is completed against the closed Control Plane and greenfield Cloud v2 deployment. The following product work is also deliberately deferred: WebUI configuration for tenant-provided remote E2B, multi-replica lease storage and sharding rules, artifact signing/garbage collection, custom roles, SSO, and SCIM. These are explicit release gates, not hidden fallbacks. The current branches are a verified isolation foundation on which the closed SaaS system can be built; they are not a production Cloud activation.