LangBot/docs/multi-tenant/verification-report.md

Multi-tenant isolation kernel verification report

Date: 2026-07-20

Status: CORE ISOLATION FOUNDATION VERIFIED — SAAS ACTIVATION REMAINS DISABLED

This report records the final implementation and verification evidence for the shared Workspace isolation foundation. It does not claim that the closed SaaS Control Plane, billing system, or greenfield Cloud v2 deployment is production-ready.

Repository refs and scope

• LangBot branch: feat/multi-tenants
• LangBot implementation commit: 90a977488212d3f3f50fb92d94e3fcae60f27eab
• LangBot base: origin/master at 6baeb032a7f76c65337b51c7b58593de4687a61c
• Plugin SDK branch: feat/multi-tenants
• Plugin SDK commit and LangBot dependency pin: 95a1805af2038c745de2c018a00db1305089a32e
• No tracked langbot-space change was made. Cloud v2 does not extend or preserve the legacy per-account instance/pod deployment topology.

The SDK protocol is versioned as 0.4.15 and is intentionally consumed from the exact pushed Git commit above. Publishing the release and replacing the Git pin with a registry pin remain merge-to-master release work; no registry release was performed here.

Implemented boundaries

Workspace, identity, and authorization

• Community bootstraps exactly one local Workspace per LangBot instance while allowing multiple Accounts, memberships, invitations, and role-based permissions inside it. A mutable edition setting cannot activate SaaS multi-Workspace routing.
• Each Account registration resolves or creates its local singleton Workspace. Invitation acceptance is one-time, email-bound, and survives the signed-out login transition without exposing the invitation secret in URLs after the initial fragment capture.
• The Account-token Workspace discovery route is intentionally separate from Workspace-authorized routes. X-Workspace-Id becomes authority only after membership resolution; API keys and public resources derive Workspace from their trusted owner.
• Ordinary monitoring uses resource.view. Export remains data.export, privileged runtime/system audit remains audit.view, and frontend controls follow the same permission split.

Persistence and PostgreSQL

• PostgreSQL uses one shared business schema with application scope plus exact ENABLE and FORCE ROW LEVEL SECURITY policies. Tenant UoWs keep SET LOCAL and business SQL on one owned root transaction and one task.
• The public tenant Session accepts only structured SQLAlchemy query/DML trees. Raw/textual SQL, textual modifiers, custom AST/compiler nodes, unapproved functions/operators/types/casts, hidden dialect values, foreign binds, loader/execution options, Session event callbacks, ORM SQL-expression attributes, transaction control, and live-result escape fail closed and make the root transaction rollback-only.
• The SQL guard is a trusted-Core misuse boundary, not an in-process Python sandbox. Mapped metadata and registered compilers are trusted boot-time code; plugins remain out of process. SQLAlchemy dialect-private container fields are pinned and covered by regression tests before dependency upgrades.
• Legacy local RAG restore remains structured and tenant-fenced while accepting SQLite string-valued timestamps and both historical PostgreSQL TEXT and fresh-schema JSON settings columns.
• The one-shot release migrator owns DDL and grants a distinct runtime role only the required business-table DML, alembic_version read access, sequence access, database CONNECT, and schema USAGE. Runtime startup reruns the role, ACL, schema, session, extension, routine, parameter, and RLS audit.
• pgvector is stored in the same business database with Workspace-scoped keys, checked dimensions, release-created partial ANN indexes, and RLS.

Shared Plugin Runtime and Box

• One instance-scoped Plugin Runtime supervisor serves multiple Workspaces. Every enabled installation still owns one nsjail worker with an immutable instance/Workspace/generation/installation/revision/artifact binding.
• Verified same-digest code and dependency environments may be mounted read-only and reused. Plugin processes and writable installation state are never merged, including for the same plugin and version.
• Runtime limits come only from instance configuration with native environment overrides. Plugin manifests cannot raise limits. Shared mode rejects legacy global plugin directories and legacy lifecycle control paths.
• One shared Box control plane admits at most one persistent logical global sandbox for an entitled Workspace. Admission is entitlement-gated, generation-fenced, shared-volume-challenged, quota-aware, and fixed to nsjail. Plain nsjail correctly fails Cloud readiness without a hard quota provider.
• stdio MCP has an independent gate and is forced off by Cloud bootstrap even when Box is available.

Automated verification

LangBot backend and static checks

uv run pytest -q --tb=short
  2527 passed, 32 skipped, 177 warnings in 73.37s

focused tenant UoW and legacy RAG compatibility suite
  120 passed, 10 warnings in 1.30s

uv run ruff check .
  passed

changed Python format check
  15 files already formatted

git diff --check
  passed

The full suite includes startup E2E, the fresh SQLite registration/Workspace journey, API authorization, storage, RAG, Plugin, MCP, and 18 Docker-backed Box integration cases.

Real PostgreSQL 16 and pgvector

test_migrations_postgres.py
test_pgvector_postgres.py
test_release_migration_postgres.py
  21 passed, 11 warnings in 11.61s

The real-database suite covers fresh and upgraded RLS schemas, two-Workspace isolation, pgvector CRUD and ANN indexes, least-privilege runtime bootstrap, release locking, historical/fresh RAG settings column compatibility, and transaction-scope escape rejection. Negative tests inject and reject role membership, grant options, persistent GUCs, extra schemas and relations, column ACLs, explicit routine and parameter ACLs, runtime-owned and SECURITY DEFINER routines, extra/runtime-owned extensions, and foreign data wrappers, servers, and user mappings.

Alembic has exactly one head: 0013_tenant_pgvector.

Plugin SDK and Box Runtime

Plugin SDK full suite
  1160 passed, 18 warnings

Plugin SDK Ruff and action consistency checks
  passed

Docker-backed Box integration suite
  18 passed, 33 warnings

The LangBot environment resolves langbot-plugin==0.4.15 from the same exact SDK commit recorded above. The SDK local ref and remote refs/heads/feat/multi-tenants were independently verified equal.

Frontend

pnpm lint
  0 errors, 34 existing warnings

VITE_API_BASE_URL=/ pnpm build
  passed; 3168 modules transformed

pnpm exec playwright test --project=chromium
  43 passed in 22.2s

The frontend suite includes Viewer monitoring visibility, privileged control hiding, stable terminal invitation states, email-mismatch login handoff, and a temporary invitation-accept failure that retains the authenticated session and succeeds on retry.

Packaging and deployment configuration

• uv lock --check resolved 277 packages and passed.
• Changed TypeScript/TSX files passed Prettier verification.
• docker compose -f docker/docker-compose.yaml config --quiet passed with the existing warning that the top-level version field is obsolete.
• The staged high-signal credential scan found no private key or provider token.

Real browser E2E

A production frontend and backend were run against an isolated temporary SQLite data directory and operated through a real local Chrome session:

1. The first Account registered and automatically received the default singleton Workspace as owner.
2. Skipping the setup wizard reached /home/monitoring; reload and a new tab recovered the authenticated Account, membership, and Workspace selection.
3. The owner invited a Viewer. After signing out, the second Account registered through the invitation and joined the same Workspace.
4. Owner and Admin could read overview/export/audit/API-key surfaces and reach resource-write validation. Runtime debug mutation remained disabled.
5. Developer retained ordinary resource management and monitoring, while export, audit, and API-key management returned 403.
6. Operator and Viewer retained ordinary monitoring reads while resource writes, export, audit, API-key management, and runtime mutation returned 403.
7. The UI matched the server matrix: Add/API Key/Export controls appeared only with their permissions, while ordinary monitoring refresh remained visible to read-only members.
8. A second Workspace creation attempt returned 403 edition_limit, proving Community remains single-Workspace and multi-user.
9. Used, revoked, expired, and email-mismatched invitation paths produced stable visible states. The mismatch survived login without a contradictory generic success toast.

The server was stopped, port 15321 was verified free, and temporary artifacts were moved to the system Trash for recoverable cleanup. No invitation secret, password, token, or provider credential is recorded in this report.

Deliberately incomplete SaaS activation gates

Multi-Workspace activation remains unavailable in the open-source bootstrap. Cloud v2 must stay disabled until all of the following are implemented and independently verified:

• the closed Control Plane owns the Account/Workspace directory, memberships, invitations, monotonic execution generations, entitlements, usage, subscriptions, and billing;
• ordinary tenant writes hold a generation-aware fence through commit, and a generation-stamped outbox or equivalent atomic publish fence protects external side effects;
• durable object references survive generation cutovers without stranding files, images, plugin artifacts, or knowledge-base content;
• every tenant-configurable outbound path has complete SSRF/egress policy, and Plugin Runtime has production Linux namespace, cgroup v2, and network isolation evidence;
• an unexpectedly exited enabled plugin worker is restored by a completion callback with bounded backoff and cross-tenant restart-storm isolation;
• plugin installation data has an operator-owned hard disk quota, not a directory-scan approximation;
• Box supplies and proves hard byte/inode quotas for Workspace, Skill, root, home, and temporary storage, and the production shared volume passes the authenticated marker challenge;
• production PostgreSQL uses a dedicated endpoint/cluster, or a tested HBA/proxy policy proves the runtime credential can connect only to the target business database;
• the release migrator is deployed as a one-shot Job with credential issuance, backup, rollback, failure/retry, and orchestration procedures;
• OAuth exchange, directory projection, leases, snapshots, event replay, and outbox state use durable shared stores suitable for horizontally scaled replicas;
• Workspace release, export, deletion, restore, and disaster-recovery semantics are decided and implemented; and
• production Runtime restart, crash, partition, revocation, and fault-injection acceptance is completed against the closed Control Plane and greenfield Cloud v2 deployment.

The following product work is also deliberately deferred: WebUI configuration for tenant-provided remote E2B, multi-replica lease storage and sharding rules, artifact signing/garbage collection, custom roles, SSO, and SCIM.

These are explicit release gates, not hidden fallbacks. The current branches are a verified isolation foundation on which the closed SaaS system can be built; they are not a production Cloud activation.