LangBot/docs/multi-tenant/implementation-checklist.md

Multi-tenant implementation checklist

Status interpretation, 2026-09-05: this checklist retains the original multi-tenant implementation and activation evidence. The kernel is now also present on dev/4.11.x; the branch names below are historical workstream labels. SDK installation-worker restart coordination is implemented in runtime/plugin/restart_coordinator.py, with focused tests passing on the inspected workspace. An unchecked production gate does not necessarily mean its code is absent. Production fault injection, egress, hard storage quotas, and generation-aware business transaction/outbox gates remain separate. See current 4.11 status and Cloud activation gates.

This checklist turns the Workspace architecture into implementation and verification gates. Exact commands and observed results are recorded in the verification report.

Scope guard

• LangBot uses branch feat/multi-tenants.
• langbot-plugin-sdk uses branch feat/multi-tenants.
• langbot-space implements the greenfield Cloud v2 modular-monolith control plane without extending the legacy per-account Pod topology; the old Pod UI remains available when Cloud v2 is disabled and only retained Pods appear in the v2 view.
• Unrelated untracked files in either repository remain untouched.
• Open-source startup cannot enable SaaS multi-workspace through edition flags or unsigned configuration.

SaaS activation gates

These items intentionally remain incomplete. Some require additional Core transaction/cutover primitives and others require the closed Control Plane or deployment. The feature branch delivers the Core isolation kernel, not the closed SaaS product or a production Cloud v2 deployment. Checked implementation items later in this document do not supersede these gates.

• The closed Control Plane owns the global Account, Workspace, Membership, and Invitation directory.
• The closed Control Plane execution-ownership module issues monotonic generations and owner leases for projected Workspaces.
• Core verifies a signed InstanceManifest before the closed bootstrap can inject CloudWorkspacePolicy.
• Tenant database writes hold a generation-aware shared transaction fence through commit, while execution-owner cutovers take the exclusive fence.
• Business writes and non-transactional side effects use a generation-stamped outbox or equivalent publish fence.
• Durable object references survive an execution-generation change through stable published keys or an explicitly atomic key/reference migration.
• The SaaS runtime pools enforce tenant-safe egress and SSRF controls for Webhooks, providers, MCP servers, and every tenant-configurable outbound URL.
• Entitlement checks, usage aggregation, and subscription lifecycle are implemented in the closed Control Plane; production activation still requires provider callback amount/currency/session/expiry binding inside the locked fulfillment transaction.
• Account registration persists a new_api.provision_account outbox item with the Account and personal Workspace, and an in-process reconciler provisions New API idempotently after commit.
• EPay and Stripe callbacks bind provider identity, amount, currency, channel/session, payment status, and expiry to the locked order before entitlement fulfillment.
• OAuth state and directory projection use an atomic shared store suitable for horizontally scaled SaaS services.
• A greenfield Cloud v2 deployment is designed and validated independently of the legacy Space deployment scheme.
• The Plugin Runtime shared profile refuses to run without delegated cgroup v2 CPU, memory-plus-swap, and PID limits, all verified in a real Linux container; production tenant-safe egress remains incomplete.
• The Plugin Runtime Supervisor automatically restores an unexpectedly exited enabled worker with bounded per-installation backoff.
• Jitter, a global restart concurrency limit, and a Runtime-level circuit breaker prevent a systemic failure from creating a cross-tenant restart storm.
• Until authenticated Runtime takeover or an owner lease/fence exists, the M0 deployment rolls Core and Plugin Runtime together and forbids an independent Core-only rollout.
• Plugin installation data has an operator-owned hard disk quota provider that atomically rejects writes over the limit; directory scans are not accepted as enforcement.
• The Box deployment provides an operator-owned quota provider that proves hard byte and inode limits for Workspace, Skill, root, tmp, and home storage.
• Core and Box Runtime mount the same durable volume and pass the authenticated marker challenge during startup and reconnect.
• Production provisions distinct migrator/runtime credentials and runs the implemented same-host/port/database release command as a one-shot Job, with tested orchestration retry, backup, and rollback procedures.
• Production PostgreSQL uses a dedicated cluster/endpoint, or a tested HBA/proxy policy proves the cluster-wide runtime credential can connect only to the target business database.
• Any future direct-migrator/pooler-runtime endpoint split is admitted only by a migrator-owned, runtime-read-only database cluster identity that the runtime role cannot spoof.
• Legacy pgvector migration failure and retry integration paths prove exact source-table RLS/FORCE restoration; the non-superuser, non-BYPASSRLS success path is already covered below.
• Multi-workspace is enabled in SaaS only after all closed Control Plane, deployment, and security gates pass.

1. Persistence foundation

Account and directory

• User has a stable, unique account UUID and explicit status.
• Existing email and password behavior remains compatible during migration.
• Workspace table represents the instance-local tenant.
• WorkspaceMembership has a unique Workspace and Account pair.
• WorkspaceInvitation stores only a token hash and supports expiry, revoke, and one-time accept.
• WorkspaceExecutionState stores generation, state, source, and write fence.
• OSS initialization creates exactly one Workspace and one owner membership atomically.
• OSS refuses a second Workspace while allowing multiple members.

Migration

• Alembic migration upgrades SQLite.
• Alembic migration upgrades PostgreSQL.
• Existing first user becomes owner of the default Workspace.
• Existing tenant resources are backfilled with the default Workspace UUID.
• SQLite destructive boundaries create verified, revision-aware backups and atomically restore after failure.
• Migration can resume safely after interruption.
• New installs and upgraded installs produce the same tenancy-kernel schema.
• The first Cloud release pins migrator and runtime sessions to public with current_schemas(false) containing only that business schema; runtime-role/database search_path overrides are rejected.
• Cloud sessions require session_replication_role=origin, row_security=on, and lo_compat_privileges=off; every persistent pg_db_role_setting applicable to the runtime role or current business database is rejected.
• The release migrator grants the runtime role exact business-table DML, alembic_version read-only access, and business-sequence USAGE/SELECT, with no WITH GRANT OPTION or non-business object grants.
• Every Cloud runtime startup revalidates the login role, current user, schema, effective/direct ACLs, ownership, memberships in all directions, column ACLs, routines, extensions, foreign objects, parameter ACLs, and other-schema access before serving traffic.
• The business database requires vector, permits only plpgsql/vector extensions, forbids runtime extension ownership, and contains no foreign data wrapper, foreign server, or user mapping.
• The runtime role and PUBLIC have no explicit routine or parameter ACL; the runtime owns no routine and cannot effectively execute any SECURITY DEFINER routine, including extension-owned routines.
• PostgreSQL's default PUBLIC TEMP is documented and tested as a dedicated-business-database v1 compatibility exception; the migrator never grants TEMP directly to the runtime role.
• Legacy pgvector migration succeeds as a non-superuser, non-BYPASSRLS source-table owner and restores mixed source-table RLS/FORCE states exactly.
• Legacy pgvector migration still needs explicit failure-and-retry integration coverage before SaaS activation.

Runtime transaction enforcement

• Each tenant UoW owns one task, root transaction, database bind, and transaction-local scope.
• A scoped Session and every captured bound method become permanently unusable when the owning UoW exits.
• Public transaction/session control, raw/textual SQL, connection/bind escape, nested transactions, execution/loader options, foreign binds, live results, unapproved functions/operators/casts/types, INSERT FROM SELECT, hidden ON CONFLICT and batch-value expressions, forced-unquoted identifiers, and custom AST/compiler nodes fail closed and make the UoW rollback-only.
• ORM SessionEvents fail before a registered callback can receive the synchronous Session or transaction connection; rollback cleanup cannot execute the rejected listener.
• ORM flush, implicit autoflush, and commit reject SQL expressions assigned to mapped attributes before compilation.
• Tenant relationship loading uses eager loading or explicit async refresh; synchronous object-session access and AsyncAttrs.awaitable_attrs are not supported tenant APIs.
• The UoW guard is documented as a trusted-Core misuse boundary rather than an in-process Python sandbox; mapped metadata/compiler registration is trusted, plugins remain out of process, and SQLAlchemy upgrades must rerun the private-container regression suite.

2. Authentication and authorization

Identity

• JWT sub uses account UUID, with a bounded compatibility path for legacy email tokens.
• Disabled or deleted accounts cannot authenticate.
• Local password and Space-linked account flows support more than one local Account.
• Public registration closes after initialization by default.
• Invitation registration works without requiring SMTP.
• An unknown Space OAuth subject cannot claim an existing Account by email; explicit account-bound binding is required.

Request context

• PrincipalContext identifies Account, API Key, or trusted runtime principal.
• WorkspaceContext contains Workspace, Membership, role, permissions, and revision.
• RequestContext contains instance UUID, Workspace context, auth type, request ID, and generation.
• ExecutionContext propagates Workspace and generation to runtime work.
• SaaS-style requests never fall back to the first or most recent Workspace.
• OSS may resolve the single Workspace when the selector is omitted.
• Account-token bootstrap can list only the authenticated Account's active memberships before a Workspace selector exists.

Fixed RBAC

• owner, admin, developer, operator, and viewer permissions match the architecture matrix.
• Invitation cannot grant owner.
• The last owner cannot be removed or demoted.
• Cross-Workspace resources return 404.
• Same-Workspace permission failures return 403.

3. Workspace and member APIs

• GET /api/v1/workspaces returns the OSS singleton Workspace.
• POST /api/v1/workspaces returns edition_limit in OSS.
• Current Workspace endpoint returns the authenticated Membership.
• Member list is permission scoped.
• Invitation create, revoke, inspect, and accept are atomic.
• Member role update and removal enforce owner rules.
• Invitation tokens travel in a request body and are redacted from logs.
• Relevant MCP tools and in-repo skills are updated with the same contract.

4. Tenant-scoped persistence and services

Each row type must have a non-null Workspace UUID, scoped indexes, scoped uniqueness, and scoped CRUD tests.

• Bots and bot admins.
• Legacy pipelines and pipeline run records.
• Model providers.
• LLM models.
• Embedding models.
• Rerank models.
• Plugin installations, settings, and configuration.
• MCP servers and resource preferences.
• Knowledge bases, files, and chunks.
• Vector collections and handles.
• Monitoring messages, calls, sessions, errors, embeddings, and feedback.
• API keys and scopes.
• Webhooks and public route resolution.
• Binary storage and Workspace storage.
• Workspace metadata, separated from system metadata.

Service and API rules

• Every tenant Service receives RequestContext or an explicit Workspace UUID.
• No tenant Service treats context None as global access.
• Every applicable get, list, create, update, delete, copy, export, and bulk operation is scoped.
• Parent-child references use the same Workspace.
• API Key authentication derives Workspace from the key, not a header.
• Webhook and Bot public routes derive Workspace from a trusted resource.
• Background jobs carry Workspace and generation explicitly.

5. Runtime isolation

Core runtime

• RuntimeBot carries Workspace UUID and execution generation (currently stored in the compatibility field placement_generation).
• RuntimePipeline carries Workspace UUID and execution generation (currently stored in the compatibility field placement_generation).
• Query and Event carry Workspace UUID without making it an authorization source.
• Session key includes Workspace UUID, Bot UUID, launcher type, and launcher ID.
• QueryPool and manager indexes cannot collide across Workspaces.
• Query and aggregation cache keys and locks include Workspace UUID.
• Runtime transports, cached results, object operations, and long-lived tasks revalidate WorkspaceExecutionState generation at side-effect boundaries.
• Ordinary tenant database writes hold the generation fence in the same transaction until commit; this remains a SaaS activation gate.

Plugin

• Plugin installation and configuration are Workspace scoped.
• Runtime control actions carry trusted Workspace binding and execution generation (wire-compatible as placement_generation).
• The Plugin Runtime supervisor is instance-scoped and intentionally serves multiple Workspaces.
• Every plugin process is bound to exactly one Workspace, installation, generation, revision, and verified artifact digest.
• Same-digest plugin code may be cached once, while worker processes and writable data remain isolated.
• Same-digest plugin dependencies are prepared once in a Runtime-owned immutable environment and mounted read-only into each isolated worker; dependency failure is surfaced before launch and recorded per installation without blocking other desired-state recovery.
• Host API derives Workspace from the connection, installation, and trusted action context, not plugin input.
• Plugin get_bots, models, tools, vector, RAG, configuration, and messaging calls are scoped.
• Plugin Workspace storage no longer uses owner default.
• Plugin page APIs check Membership and installation ownership.
• Local plugin launches use short-lived, one-use registration capabilities bound to manifest identity.

MCP, RAG, and Box

• MCP runtime key contains instance UUID, Workspace UUID, execution generation, and server UUID.
• Same-named MCP servers in two Workspaces do not share sessions.
• Pipeline cannot reference another Workspace's MCP resource.
• RAG collection names and handles are server-derived and Workspace scoped.
• Legacy global vector migration is available only to the local OSS singleton Workspace.
• Object storage paths include instance, Workspace, and execution generation for the fixed-generation OSS runtime.
• Object storage revalidates generation before touching a provider or resolving an opaque key.
• Cloud cutover uses generation-scoped staging plus stable published object references, rather than making the staging generation the durable identity.
• Box persistent and ephemeral namespaces include the required instance, Workspace, and generation scope.
• Same-named Box sessions and processes cannot collide across Workspaces or execution generations.
• Box relay and process I/O reject or retire stale generations.
• External paths and privileged mounts cannot be supplied by an untrusted plugin.
• Cloud attachment host I/O uses query UUIDs and link-free dirfd operations with bounded inode traversal.
• Cloud Skill package paths are Runtime-owned, Workspace-scoped, read-only mounts; Python env/cache stays tenant-writable.
• Skill ZIP preview/install rejects path escape, links, non-regular files, duplicate entries, excessive compression ratio, entry count, per-file size, and total size.
• Cloud Box code paths and automated tests require the authenticated marker challenge before startup or reconnect can proceed.
• Cloud Box readiness fails until hard Workspace, Skill, ephemeral-storage, and inode quota capabilities are available.

6. SDK and protocol

• Public Query, Event, Session, and context entities carry backward-compatible Workspace data.
• Action RPC request models carry trusted Workspace binding where required.
• Action enums and callers remain consistent.
• Old plugins continue to deserialize compatible events.
• Plugins cannot select an arbitrary Workspace through a Host API argument.
• Runtime storage uses the bound Workspace UUID.
• SDK API tests pass.
• Runtime tests pass.
• Action consistency script passes.

7. Frontend

• Every browser tenant API request carries the current Workspace selector after bootstrap.
• OSS automatically selects the singleton Workspace.
• OSS does not show Create Workspace or a misleading switcher.
• Workspace settings show current Workspace information.
• Members page lists roles and permissions.
• Invitation creation shows a one-time link when SMTP is unavailable.
• Invitation acceptance supports a signed-out user flow.
• Role controls are hidden or disabled consistently with backend permissions.
• Switching accounts clears stale Workspace query cache and local state.
• User-facing strings support en_US, zh_Hans, and ja_JP.

8. Automated verification

Persistence and authorization

• SQLite fresh install.
• SQLite upgrade from pre-tenant schema, including verified failure recovery.
• PostgreSQL fresh install.
• PostgreSQL upgrade from pre-tenant schema.
• All fixed roles have positive and negative permission-matrix tests.
• Concurrent invitation acceptance creates one Membership.
• Concurrent owner changes never leave zero owners.

Cross-tenant isolation

• Two Workspaces are created through a test-only policy.
• Applicable resource operations and parent-child references have cross-Workspace negative coverage.
• Resource UUID guessing cannot cross Workspace.
• API Key cannot cross Workspace.
• Plugin cannot enumerate or invoke another Workspace's resources.
• Sessions, caches, locks, MCP, RAG, Box, storage, and monitoring do not collide.
• Background jobs cannot execute without an explicit Workspace and execution generation.

Security and revocation

• Space login and binding use purpose-bound, one-time opaque OAuth state; caller-supplied state is rejected.
• OAuth redirects trust only server-configured WebUI or webhook origins, never request Host or Origin headers.
• Dashboard WebSockets revalidate authentication, Membership, resource, permission, and generation per message.
• Public embed WebSockets re-resolve Bot availability and execution binding per message.
• Runtime, storage, Plugin Runtime, MCP, RAG, and Box reject a stale execution generation.
• Unhandled API and webhook failures return a generic error plus request ID without exception text.
• URL user information and sensitive query parameters are redacted before configuration is serialized or logged.

Regression

• LangBot unit tests pass.
• LangBot integration tests pass.
• Frontend lint completes without errors and the production build passes.
• SDK focused and full relevant tests pass.
• LangBot is pinned to the exact pushed SDK commit and cross-repo tests pass against that revision.

9. Real browser E2E

• Start from a clean local data directory.
• First user initializes the singleton Workspace as owner.
• Owner creates an invitation link.
• A second signed-out browser identity accepts the invitation and registers.
• owner, admin, developer, operator, and viewer UI permissions match backend enforcement.
• Direct API calls cannot bypass hidden controls.
• Account switch does not expose prior account or Workspace data.
• Refresh and a new browser tab recover the correct Workspace safely.
• OSS rejects a second Workspace with edition_limit; same-name and same-identifier isolation is covered by the test-only multi-Workspace policy because OSS deliberately has no multi-Workspace browser surface.
• Explicit error states are visible for expired, revoked, reused, and email-mismatched invitations.

10. Completion evidence

• LangBot and SDK branch refs are recorded in the verification report.
• Space contains the closed adapter package and Cloud v2 control plane, billing, migration, and Workspace UI changes; unrelated pre-existing files remain unstaged.
• Migration output is captured for SQLite and PostgreSQL.
• Test commands and results are recorded.
• Browser E2E actions and observed results are recorded.
• No remaining tenant table, global Service query, owner default, or unscoped runtime key is found by the final audit.