docs(tenancy): record final isolation verification

docs/multi-tenant/verification-report.md

@@ -2,10 +2,10 @@
 
 Date: 2026-07-20
 
-Status: `CORE ISOLATION KERNEL VERIFIED — SAAS ACTIVATION REMAINS DISABLED`
+Status: `CORE ISOLATION FOUNDATION VERIFIED — SAAS ACTIVATION REMAINS DISABLED`
 
 This report records the final implementation and verification evidence for the
-shared Workspace isolation kernel. It does not claim that the closed SaaS
+shared Workspace isolation foundation. It does not claim that the closed SaaS
 Control Plane, billing system, or greenfield Cloud v2 deployment is
 production-ready.
 
@@ -13,77 +13,108 @@ production-ready.
 
 - LangBot branch: `feat/multi-tenants`
 - LangBot implementation commit:
-  `a47bfe8167e12d1853667b3d059cfb940ff0643e`
+  `90a977488212d3f3f50fb92d94e3fcae60f27eab`
 - LangBot base: `origin/master` at
   `6baeb032a7f76c65337b51c7b58593de4687a61c`
 - Plugin SDK branch: `feat/multi-tenants`
 - Plugin SDK commit and LangBot dependency pin:
-  `044536f3720a2a8424d92f78934b9623da9f7f1d`
+  `95a1805af2038c745de2c018a00db1305089a32e`
-- Space remained on `main` at
+- No tracked `langbot-space` change was made. Cloud v2 does not extend or
-  `5058b039c2629e3b01ebd7358369b50e7df62a1c`; this implementation made no
+  preserve the legacy per-account instance/pod deployment topology.
-  tracked Space change and does not reuse the legacy Space deployment model.
 
 The SDK protocol is versioned as 0.4.15 and is intentionally consumed from the
-exact pushed Git commit on this feature branch. Publishing 0.4.15 and replacing
+exact pushed Git commit above. Publishing the release and replacing the Git
-the Git pin with a registry pin remain merge-to-master release steps.
+pin with a registry pin remain merge-to-master release work; no registry
+release was performed here.
 
 ## Implemented boundaries
 
-- OSS bootstraps exactly one Workspace per instance while allowing multiple
+### Workspace, identity, and authorization
-  Accounts, memberships, invitations, and role-based permissions inside it.
+
-  Local configuration or an edition flag cannot activate SaaS multi-Workspace
+- Community bootstraps exactly one local Workspace per LangBot instance while
-  routing.
+  allowing multiple Accounts, memberships, invitations, and role-based
-- The Cloud bootstrap remains a closed-policy injection point. The closed entry
+  permissions inside it. A mutable edition setting cannot activate SaaS
-  point owns Manifest signature verification; Core accepts only a verified
+  multi-Workspace routing.
-  deployment receipt and validates its instance, expiry, generation,
+- Each Account registration resolves or creates its local singleton Workspace.
-  capability, entitlement-adapter, and runtime-configuration invariants.
+  Invitation acceptance is one-time, email-bound, and survives the signed-out
+  login transition without exposing the invitation secret in URLs after the
+  initial fragment capture.
+- The Account-token Workspace discovery route is intentionally separate from
+  Workspace-authorized routes. `X-Workspace-Id` becomes authority only after
+  membership resolution; API keys and public resources derive Workspace from
+  their trusted owner.
+- Ordinary monitoring uses `resource.view`. Export remains `data.export`,
+  privileged runtime/system audit remains `audit.view`, and frontend controls
+  follow the same permission split.
+
+### Persistence and PostgreSQL
+
 - PostgreSQL uses one shared business schema with application scope plus exact
   `ENABLE` and `FORCE ROW LEVEL SECURITY` policies. Tenant UoWs keep `SET LOCAL`
-  and SQL on one transaction; transaction-free scopes avoid holding pool
+  and business SQL on one owned root transaction and one task.
-  connections across model or network waits.
+- The public tenant Session accepts only structured SQLAlchemy query/DML trees.
+  Raw/textual SQL, textual modifiers, custom AST/compiler nodes, unapproved
+  functions/operators/types/casts, hidden dialect values, foreign binds,
+  loader/execution options, Session event callbacks, ORM SQL-expression
+  attributes, transaction control, and live-result escape fail closed and make
+  the root transaction rollback-only.
+- The SQL guard is a trusted-Core misuse boundary, not an in-process Python
+  sandbox. Mapped metadata and registered compilers are trusted boot-time code;
+  plugins remain out of process. SQLAlchemy dialect-private container fields
+  are pinned and covered by regression tests before dependency upgrades.
+- Legacy local RAG restore remains structured and tenant-fenced while accepting
+  SQLite string-valued timestamps and both historical PostgreSQL `TEXT` and
+  fresh-schema `JSON` settings columns.
 - The one-shot release migrator owns DDL and grants a distinct runtime role only
-  business-table DML, `alembic_version` read access, required sequence access,
+  the required business-table DML, `alembic_version` read access, sequence
-  database `CONNECT`, and schema `USAGE`. Runtime startup reruns the complete
+  access, database `CONNECT`, and schema `USAGE`. Runtime startup reruns the
   role, ACL, schema, session, extension, routine, parameter, and RLS audit.
 - pgvector is stored in the same business database with Workspace-scoped keys,
-  checked dimensions, release-created partial ANN indexes, and RLS. Legacy
+  checked dimensions, release-created partial ANN indexes, and RLS.
-  vector migration can temporarily suspend source-table RLS as a non-superuser,
+
-  non-`BYPASSRLS` table owner and restores each source table's prior state.
+### Shared Plugin Runtime and Box
-- One shared Plugin Runtime supervisor serves the logical instance. Every
+
-  enabled installation still owns one nsjail worker and one immutable
+- One instance-scoped Plugin Runtime supervisor serves multiple Workspaces.
-  Workspace/installation/generation/revision/digest binding. Verified
+  Every enabled installation still owns one nsjail worker with an immutable
-  same-digest code and dependency environments are read-only shared; processes
+  instance/Workspace/generation/installation/revision/artifact binding.
-  and writable state are never merged.
+- Verified same-digest code and dependency environments may be mounted
+  read-only and reused. Plugin processes and writable installation state are
+  never merged, including for the same plugin and version.
+- Runtime limits come only from instance configuration with native environment
+  overrides. Plugin manifests cannot raise limits. Shared mode rejects legacy
+  global plugin directories and legacy lifecycle control paths.
 - One shared Box control plane admits at most one persistent logical `global`
-  sandbox for an entitled Workspace. Cloud admission is entitlement-gated,
+  sandbox for an entitled Workspace. Admission is entitlement-gated,
   generation-fenced, shared-volume-challenged, quota-aware, and fixed to
-  nsjail. Plain nsjail correctly fails Cloud readiness without a hard storage
+  nsjail. Plain nsjail correctly fails Cloud readiness without a hard quota
-  quota provider.
+  provider.
-- stdio MCP has an independent policy gate and is forced off by the Cloud
+- stdio MCP has an independent gate and is forced off by Cloud bootstrap even
-  bootstrap even when Box is available.
+  when Box is available.
 
 ## Automated verification
 
 ### LangBot backend and static checks
 
 ```text
-uv run pytest tests/unit_tests tests/integration -q
+uv run pytest -q --tb=short
-  2374 passed, 30 skipped, 148 warnings in 26.76s
+  2527 passed, 32 skipped, 177 warnings in 73.37s
+
+focused tenant UoW and legacy RAG compatibility suite
+  120 passed, 10 warnings in 1.30s
 
 uv run ruff check .
   passed
 
-changed and untracked Python format check
+changed Python format check
-  112 files already formatted
+  15 files already formatted
 
 git diff --check
   passed
 ```
 
-The fresh SQLite journey test covers first-Account registration, automatic
+The full suite includes startup E2E, the fresh SQLite registration/Workspace
-singleton Workspace creation, authenticated current-Workspace bootstrap,
+journey, API authorization, storage, RAG, Plugin, MCP, and 18 Docker-backed Box
-wizard-state persistence, and a 403 `edition_limit` response when an OSS owner
+integration cases.
-attempts to create a second Workspace.
 
 ### Real PostgreSQL 16 and pgvector
 
@@ -91,16 +122,15 @@ attempts to create a second Workspace.
 test_migrations_postgres.py
 test_pgvector_postgres.py
 test_release_migration_postgres.py
-test_plugin_identity_migration.py
+  21 passed, 11 warnings in 11.61s
-  20 passed, 11 warnings in 13.51s
 ```
 
 The real-database suite covers fresh and upgraded RLS schemas, two-Workspace
-isolation, pgvector CRUD and ANN indexes, plugin installation identity,
+isolation, pgvector CRUD and ANN indexes, least-privilege runtime bootstrap,
-operator/runtime credential separation, advisory locking, and actual Cloud
+release locking, historical/fresh RAG settings column compatibility, and
-runtime initialization as the least-privilege role. Negative tests inject and
+transaction-scope escape rejection. Negative tests inject and reject role
-reject role membership, grant options, persistent GUCs, extra schemas and
+membership, grant options, persistent GUCs, extra schemas and relations,
-relations, column ACLs, explicit routine and parameter ACLs, runtime-owned and
+column ACLs, explicit routine and parameter ACLs, runtime-owned and
 `SECURITY DEFINER` routines, extra/runtime-owned extensions, and foreign data
 wrappers, servers, and user mappings.
 
@@ -110,61 +140,75 @@ Alembic has exactly one head: `0013_tenant_pgvector`.
 
 ```text
 Plugin SDK full suite
-  1154 passed, 18 warnings in 11.99s
+  1160 passed, 18 warnings
 
 Plugin SDK Ruff and action consistency checks
   passed
 
 Docker-backed Box integration suite
-  18 passed, 33 warnings in 13.89s
+  18 passed, 33 warnings
 ```
 
-The installed LangBot environment resolves `langbot-plugin==0.4.15` from the
+The LangBot environment resolves `langbot-plugin==0.4.15` from the same exact
-same exact SDK commit recorded above.
+SDK commit recorded above. The SDK local ref and remote
+`refs/heads/feat/multi-tenants` were independently verified equal.
 
 ### Frontend
 
 ```text
 pnpm lint
-  0 errors, 34 warnings
+  0 errors, 34 existing warnings
 
 VITE_API_BASE_URL=/ pnpm build
   passed; 3168 modules transformed
 
 pnpm exec playwright test --project=chromium
-  40 passed in 21.9s
+  43 passed in 22.2s
 ```
 
+The frontend suite includes Viewer monitoring visibility, privileged control
+hiding, stable terminal invitation states, email-mismatch login handoff, and a
+temporary invitation-accept failure that retains the authenticated session and
+succeeds on retry.
+
 ### Packaging and deployment configuration
 
 - `uv lock --check` resolved 277 packages and passed.
-- The configuration template, migration workflow, and Docker Compose file
+- Changed TypeScript/TSX files passed Prettier verification.
-  passed YAML parsing.
 - `docker compose -f docker/docker-compose.yaml config --quiet` passed with the
   existing warning that the top-level `version` field is obsolete.
-- Migration CI now runs the PostgreSQL RLS, pgvector, release-role, and plugin
+- The staged high-signal credential scan found no private key or provider
-  identity suites against PostgreSQL 16 with pgvector.
+  token.
-- The staged high-signal credential scan found no private key or provider token.
 
 ## Real browser E2E
 
 A production frontend and backend were run against an isolated temporary
-SQLite data directory and operated in a real local Google Chrome Guest window:
+SQLite data directory and operated through a real local Chrome session:
 
-1. The first Account registered and received the automatically created default
+1. The first Account registered and automatically received the default
    singleton Workspace as owner.
-2. Skipping the setup wizard reached `/home/monitoring`; reload preserved the
+2. Skipping the setup wizard reached `/home/monitoring`; reload and a new tab
-   dashboard and authenticated Workspace state.
+   recovered the authenticated Account, membership, and Workspace selection.
-3. Account Settings showed the OSS single-Workspace/multi-user model and owner
+3. The owner invited a Viewer. After signing out, the second Account registered
-   membership. The owner created an invitation for another Account.
+   through the invitation and joined the same Workspace.
-4. After refresh, the pending invitation remained while its one-time raw secret
+4. Owner and Admin could read overview/export/audit/API-key surfaces and reach
-   was no longer displayed.
+   resource-write validation. Runtime debug mutation remained disabled.
-5. The related Workspace, user, system, invitation, and monitoring requests
+5. Developer retained ordinary resource management and monitoring, while
-   returned successful responses in the backend log.
+   export, audit, and API-key management returned 403.
+6. Operator and Viewer retained ordinary monitoring reads while resource
+   writes, export, audit, API-key management, and runtime mutation returned 403.
+7. The UI matched the server matrix: Add/API Key/Export controls appeared only
+   with their permissions, while ordinary monitoring refresh remained visible
+   to read-only members.
+8. A second Workspace creation attempt returned 403 `edition_limit`, proving
+   Community remains single-Workspace and multi-user.
+9. Used, revoked, expired, and email-mismatched invitation paths produced
+   stable visible states. The mismatch survived login without a contradictory
+   generic success toast.
 
-The server was stopped, port 15321 was verified free, and the two temporary
+The server was stopped, port 15321 was verified free, and temporary artifacts
-test directories were moved to the system Trash for recoverable cleanup. No
+were moved to the system Trash for recoverable cleanup. No invitation secret,
-invitation secret is recorded in this report.
+password, token, or provider credential is recorded in this report.
 
 ## Deliberately incomplete SaaS activation gates
 
@@ -178,28 +222,36 @@ independently verified:
 - ordinary tenant writes hold a generation-aware fence through commit, and a
   generation-stamped outbox or equivalent atomic publish fence protects
   external side effects;
-- durable object references survive execution-generation cutovers without
+- durable object references survive generation cutovers without stranding
-  stranding files, images, plugin artifacts, or knowledge-base content;
+  files, images, plugin artifacts, or knowledge-base content;
-- Plugin Runtime deployment provides delegated cgroup v2 plus tenant-safe
+- every tenant-configurable outbound path has complete SSRF/egress policy, and
-  network isolation and egress policy. The current shared network namespace is
+  Plugin Runtime has production Linux namespace, cgroup v2, and network
-  not an acceptable public-SaaS isolation boundary;
+  isolation evidence;
-- Box deployment supplies and proves hard byte and inode quotas for Workspace,
+- an unexpectedly exited enabled plugin worker is restored by a completion
-  Skill, root, home, and temporary storage; stock nsjail intentionally fails
+  callback with bounded backoff and cross-tenant restart-storm isolation;
-  this readiness contract;
+- plugin installation data has an operator-owned hard disk quota, not a
-- production PostgreSQL uses a dedicated cluster/endpoint, or a tested
+  directory-scan approximation;
-  HBA/proxy policy proves that the cluster-wide runtime credential can connect
+- Box supplies and proves hard byte/inode quotas for Workspace, Skill, root,
-  only to the target business database;
+  home, and temporary storage, and the production shared volume passes the
+  authenticated marker challenge;
+- production PostgreSQL uses a dedicated endpoint/cluster, or a tested
+  HBA/proxy policy proves the runtime credential can connect only to the target
+  business database;
 - the release migrator is deployed as a one-shot Job with credential issuance,
-  backup, rollback, and orchestration retry procedures. A future direct
+  backup, rollback, failure/retry, and orchestration procedures;
-  migrator/pooler runtime endpoint split additionally requires an immutable
+- OAuth exchange, directory projection, leases, snapshots, event replay, and
-  database-backed cluster identity;
+  outbox state use durable shared stores suitable for horizontally scaled
-- OAuth exchange and directory projection state use an atomic shared store for
+  replicas;
-  horizontally scaled replicas;
+- Workspace release, export, deletion, restore, and disaster-recovery semantics
-- Workspace release, data export, deletion, and single-Workspace restore
+  are decided and implemented; and
-  semantics are decided and implemented; and
+- production Runtime restart, crash, partition, revocation, and fault-injection
-- the greenfield Cloud v2 deployment is validated without inheriting the old
+  acceptance is completed against the closed Control Plane and greenfield
-  Space deployment topology.
+  Cloud v2 deployment.
 
-These are explicit release gates, not hidden fallbacks. The current branch is a
+The following product work is also deliberately deferred: WebUI configuration
-verified isolation foundation on which the closed SaaS system can be built; it
+for tenant-provided remote E2B, multi-replica lease storage and sharding rules,
-is not a production Cloud activation.
+artifact signing/garbage collection, custom roles, SSO, and SCIM.
+
+These are explicit release gates, not hidden fallbacks. The current branches
+are a verified isolation foundation on which the closed SaaS system can be
+built; they are not a production Cloud activation.