LangBot/docs/multi-tenant/verification-report.md

Cloud v2 multi-tenant verification report

Date: 2026-07-24

Status: FOUNDATION AND CONTROL PLANE VERIFIED — PRODUCTION ACTIVATION REMAINS GATED

This report records the implementation and verification evidence for one logical LangBot instance serving multiple Workspace tenants. It covers the open-source Core, the shared Plugin/Box runtimes, the closed Space adapter, and the Space Cloud v2 modular-monolith control plane. It does not claim that the production Cloud deployment may enable CLOUD_V2_ENABLED yet.

Repository refs and scope

• LangBot Core: branch feat/multi-tenants
• langbot-plugin-sdk: branch feat/multi-tenants, commit e7d946af4a6b1494fbe74627c1815ace19ac8991
• langbot-space: branch feat/cloud-v2-control-plane
• Closed Core adapter: langbot-space/cloud-adapter
• SDK protocol/package version: 0.4.18

Core pins the exact pushed SDK commit above. Cloud v2 is a greenfield multi-Workspace deployment and does not provision one Pod, database, queue, or Runtime per tenant. Legacy Space Pods remain a compatibility surface only.

Implemented product and security boundaries

Workspace and identity

• SaaS has one stable instance_uuid; workspace_uuid is the tenant key.
• Registering an Account creates its personal Workspace and owner Membership in the same PostgreSQL transaction.
• A Workspace may contain multiple users with owner, admin, and member roles. Invitation acceptance is token-hash based, email-bound, single-use, concurrency-safe, and member-limit checked while locked.
• Community remains exactly one local Workspace while supporting multiple Accounts and fixed RBAC roles. Installing the closed adapter is required to inject the SaaS policy; configuration alone cannot activate it.
• Disabled or deleted Accounts cannot use password/code/OAuth login, sessions, refresh/access tokens, or personal access tokens.

Closed Space control plane

• CLOUD_V2_ENABLED defaults to false. Invalid or incomplete configuration fails startup, and disabled endpoints return 503.
• Space owns Workspace/Membership/Invitation, versioned plans, subscriptions, entitlements, usage events, directory outbox, and signed instance manifests.
• Free and Pro are Workspace plans. Pro projects one managed global sandbox; Free projects none. Both force stdio MCP off in the first release.
• Subscription changes use Workspace advisory locking, expected entitlement revision, one-live-order uniqueness, idempotent usage ingestion, period-end downgrade, renewal, and lazy expiry settlement.
• Directory snapshot and per-Workspace delta reads use repeatable-read, read-only transactions with a high-water cursor. Core stores a replica-local consumer cursor and shared PostgreSQL projection state, snapshot coverage, and inbox rows.
• The /cloud page selects a Workspace and shows its independent subscription, entitlement, limits, and usage. When Cloud v2 is disabled or the backend does not expose the feature field, the complete legacy Welcome/Pod UI is used.

PostgreSQL and pgvector

• SaaS business data uses one PostgreSQL shared schema with application scope plus forced RLS. Cloud directory writes are separated from local tenant writes.
• Projected Account and Membership revisions are monotonic. Tombstones remove memberships, and stale revisions cannot resurrect them.
• pgvector shares the business PostgreSQL database and remains Workspace-scoped.
• Space runs versioned SQL migrations before application seeds. A fresh database, a partial pre-migration database, and an existing-baseline path converge without startup-time AutoMigrate.

Shared Plugin Runtime

• One instance-scoped trusted Supervisor serves multiple Workspaces.
• Every enabled installation has its own nsjail process bound to (instance, workspace, execution generation, installation, runtime revision, artifact digest).
• Verified same-digest code and dependency files are mounted read-only and may be shared; home, tmp, data, process namespace, registration capability, and cgroup are private.
• Instance configuration owns CPU, memory, PID, open-file, and file-size limits. Plugin manifests cannot increase them.
• Memory includes swap: nsjail receives memory.max and swap.max=0.
• Unexpected worker exit is recovered by a completion callback with bounded per-installation exponential backoff. Remove, reconcile, Runtime shutdown, and container SIGTERM perform a graceful-to-SIGKILL bounded reap.

Shared Box Runtime and MCP

• One shared Box control plane serves Workspace-bound logical sessions.
• Cloud grants allow at most one persistent global session for an entitled Workspace and no managed processes in the first release.
• Cloud is fixed to nsjail and network-off. Core and Box prove the shared durable Workspace mount with an authenticated marker challenge.
• stdio MCP is independently gated and forced off for Cloud v2.
• The current nsjail Box backend does not provide hard byte/inode quotas, so Cloud readiness correctly fails closed instead of silently using a soft directory scan.

Automated verification

LangBot Core

uv run --no-sync pytest -q
  2590 passed, 32 skipped, 177 warnings

real PostgreSQL migration, pgvector, and release-migrator suites
  21 passed, 11 warnings

uv run --no-sync ruff check .
uv lock --check
git diff --check
  passed

The full suite ran without the closed adapter installed, proving the open-source single-Workspace/multi-user path remains standalone. Focused closed-adapter, directory projection, runtime connector, Box cleanup, and configuration suites also passed with the adapter installed.

Plugin SDK and real Linux runtime

SDK full suite
  1226 passed, 22 existing warnings

Ruff check and format check
git diff --check
  passed

A privileged Linux test container with host cgroup namespace ran one shared Runtime and two Workspace installations:

• both workers referenced the same artifact inode;
• home, tmp, and data inodes were distinct;
• each plugin saw only PID 1 in its private PID namespace;
• a tampered binding and an unknown installation were rejected;
• the control token was absent from worker environments;
• cgroups were distinct with memory.max=134217728, memory.swap.max=0, pids.max=32, and cpu.max=500000 1000000;
• touching 256 MiB exited with code 137 without swap growth;
• the 32nd fork failed with EAGAIN;
• reconcile and container SIGTERM removed the worker cgroups.

The same run started the Runtime from a non-root working directory, covering absolute nsjail mount-source normalization.

Space backend, adapter, and frontend

go test ./...
  passed

fresh PostgreSQL app startup, partial-baseline migration,
Cloud v2 migration rerun, and control-plane integration
  passed

closed adapter pytest and Ruff
  passed

pnpm exec tsc --noEmit
targeted ESLint for changed Cloud files
pnpm build
  passed; 47 routes generated

The PostgreSQL checks started from an empty database and verified all 32 registered migrations in order, Cloud v2 Free/Pro seeds, legacy plan seeds, Cloud columns/indexes, and repeatable reruns.

Cross-service and browser E2E

Signed Space-to-Core directory projection

Using an isolated Space PostgreSQL database and a migrated Core PostgreSQL database:

1. Space issued a signed manifest for the fixed LangBot instance.
2. Space returned a directory snapshot at cursor 5 containing two Workspaces, two projected Accounts, and owner/admin memberships.
3. Core verified the signature, instance, release, validity window, and capability before injecting the Cloud Workspace policy.
4. Core stored both Workspaces, all active Memberships, both projected Accounts, snapshot coverage, inbox entries, and cursor 5.
5. Account-field-only projection revisions and Workspace directory revisions remained independent, and cross-Workspace Account conflicts failed closed.

Real browser Cloud v2 flow

A real local browser operated the Space frontend and backend:

1. An owner logged in and saw the automatically created personal Workspace.
2. A second Account invited the owner as admin to another Workspace.
3. The selector showed both Workspaces and switched between them without stale subscription or usage state crossing the selection.
4. Each Workspace showed its own Free entitlement and limits.
5. Pro checkout opened for the selected Workspace. With no configured payment channels, confirmation was visibly disabled and no order was created.
6. No browser errors occurred; only the pre-existing i18next warning for the short en locale was observed.

The legacy feature-flag branch was then covered by API/static checks and the production build: false or missing cloud_v2_enabled renders the original Welcome/Pod client; a failed web-config request renders an explicit retry instead of guessing a deployment mode.

Deliberate Core startup failure

Core completed signed manifest verification and directory projection, then stopped at the Box readiness gate because the current nsjail backend cannot prove hard Workspace byte/inode quota enforcement. Connector shutdown and reconnect tasks were cleanly reaped; no event-loop or never-awaited coroutine warning remained.

This is a successful fail-closed acceptance result, not a passing production Cloud boot.

Remaining production activation gates

Cloud v2 must remain disabled until these gates are closed:

• Box provides and proves hard byte and inode quotas for Workspace, Skill, root, home, and tmp storage.
• Plugin installation writable data receives an operator-owned hard total disk quota.
• Plugin and future networked Box workloads have tenant-safe egress/SSRF policy.
• Plugin Runtime adds jitter, global restart concurrency limiting, and a Runtime-level circuit breaker, then passes systemic-failure injection.
• M0 rolls Core and Plugin Runtime together until authenticated Runtime takeover or an owner lease/fencing protocol exists.
• Account registration moves New API provisioning to an in-process, transactional outbox reconciler keyed by Space Account UUID.
• EPay and Stripe fulfillment bind provider identity, amount, currency, channel/session, payment status, and expiry to the locked order transaction; provider transaction IDs are unique and replay-safe.
• Payment-session creation and stale processing orders gain in-process reconciliation.
• Cloud v2 subscription service periods are stored immutably and included in recognized-revenue reporting.
• Production migration Job, backup/rollback, PostgreSQL credential/network boundaries, horizontal-replica fault injection, Workspace release/export, deletion, and restore semantics are completed.

These gates intentionally add no tenant-specific service. They are implemented inside the existing Space, Core, Plugin Runtime, Box Runtime, and PostgreSQL components to preserve the architecture goal: near-zero static cost for a new Workspace.