fix(tenancy): close isolation and permission gaps

docs/multi-tenant/implementation-checklist.md

@@ -21,16 +21,18 @@ closed SaaS product or a production Cloud v2 deployment. Checked implementation
 items later in this document do not supersede these gates.
 
 - [ ] The closed Control Plane owns the global Account, Workspace, Membership, and Invitation directory.
-- [ ] The closed placement service issues monotonic generations and leases for projected Workspaces.
+- [ ] The closed Control Plane execution-ownership module issues monotonic generations and owner leases for projected Workspaces.
 - [ ] Core verifies a signed `InstanceManifest` before the closed bootstrap can inject `CloudWorkspacePolicy`.
-- [ ] Tenant database writes hold a generation-aware shared transaction fence through commit, while placement cutovers take the exclusive fence.
+- [ ] Tenant database writes hold a generation-aware shared transaction fence through commit, while execution-owner cutovers take the exclusive fence.
 - [ ] Business writes and non-transactional side effects use a generation-stamped outbox or equivalent publish fence.
-- [ ] Durable object references survive a placement-generation change through stable published keys or an explicitly atomic key/reference migration.
+- [ ] Durable object references survive an execution-generation change through stable published keys or an explicitly atomic key/reference migration.
 - [ ] The SaaS runtime pools enforce tenant-safe egress and SSRF controls for Webhooks, providers, MCP servers, and every tenant-configurable outbound URL.
 - [ ] Entitlement checks, usage aggregation, subscription lifecycle, and billing are implemented in the closed Control Plane.
 - [ ] OAuth state and directory projection use an atomic shared store suitable for horizontally scaled SaaS services.
 - [ ] A greenfield Cloud v2 deployment is designed and validated independently of the legacy Space deployment scheme.
 - [ ] The Plugin Runtime deployment provides delegated cgroup v2 and tenant-safe egress; the shared profile refuses to run without hard CPU, memory, and PID limits.
+- [ ] The Plugin Runtime Supervisor automatically restores an unexpectedly exited enabled worker with bounded backoff and cannot create a cross-tenant restart storm.
+- [ ] Plugin installation data has an operator-owned hard disk quota provider that atomically rejects writes over the limit; directory scans are not accepted as enforcement.
 - [ ] The Box deployment provides an operator-owned quota provider that proves hard byte and inode limits for Workspace, Skill, root, tmp, and home storage.
 - [ ] Core and Box Runtime mount the same durable volume and pass the authenticated marker challenge during startup and reconnect.
 - [ ] Production provisions distinct migrator/runtime credentials and runs the implemented same-host/port/database release command as a one-shot Job, with tested orchestration retry, backup, and rollback procedures.
@@ -71,6 +73,16 @@ items later in this document do not supersede these gates.
 - [x] Legacy pgvector migration succeeds as a non-superuser, non-`BYPASSRLS` source-table owner and restores mixed source-table RLS/FORCE states exactly.
 - [ ] Legacy pgvector migration still needs explicit failure-and-retry integration coverage before SaaS activation.
 
+### Runtime transaction enforcement
+
+- [x] Each tenant UoW owns one task, root transaction, database bind, and transaction-local scope.
+- [x] A scoped Session and every captured bound method become permanently unusable when the owning UoW exits.
+- [x] Public transaction/session control, raw/textual SQL, connection/bind escape, nested transactions, execution/loader options, foreign binds, live results, unapproved functions/operators/casts/types, `INSERT FROM SELECT`, hidden `ON CONFLICT` and batch-value expressions, forced-unquoted identifiers, and custom AST/compiler nodes fail closed and make the UoW rollback-only.
+- [x] ORM `SessionEvents` fail before a registered callback can receive the synchronous Session or transaction connection; rollback cleanup cannot execute the rejected listener.
+- [x] ORM flush, implicit autoflush, and commit reject SQL expressions assigned to mapped attributes before compilation.
+- [x] Tenant relationship loading uses eager loading or explicit async `refresh`; synchronous object-session access and `AsyncAttrs.awaitable_attrs` are not supported tenant APIs.
+- [x] The UoW guard is documented as a trusted-Core misuse boundary rather than an in-process Python sandbox; mapped metadata/compiler registration is trusted, plugins remain out of process, and SQLAlchemy upgrades must rerun the private-container regression suite.
+
 ## 2. Authentication and authorization
 
 ### Identity
@@ -145,8 +157,8 @@ Each row type must have a non-null Workspace UUID, scoped indexes, scoped unique
 
 ### Core runtime
 
-- [x] RuntimeBot carries Workspace UUID and placement generation.
-- [x] RuntimePipeline carries Workspace UUID and placement generation.
+- [x] RuntimeBot carries Workspace UUID and execution generation (currently stored in the compatibility field `placement_generation`).
+- [x] RuntimePipeline carries Workspace UUID and execution generation (currently stored in the compatibility field `placement_generation`).
 - [x] Query and Event carry Workspace UUID without making it an authorization source.
 - [x] Session key includes Workspace UUID, Bot UUID, launcher type, and launcher ID.
 - [x] QueryPool and manager indexes cannot collide across Workspaces.
@@ -157,7 +169,7 @@ Each row type must have a non-null Workspace UUID, scoped indexes, scoped unique
 ### Plugin
 
 - [x] Plugin installation and configuration are Workspace scoped.
-- [x] Runtime control actions carry trusted Workspace binding and placement generation.
+- [x] Runtime control actions carry trusted Workspace binding and execution generation (wire-compatible as `placement_generation`).
 - [x] The Plugin Runtime supervisor is instance-scoped and intentionally serves multiple Workspaces.
 - [x] Every plugin process is bound to exactly one Workspace, installation, generation, revision, and verified artifact digest.
 - [x] Same-digest plugin code may be cached once, while worker processes and writable data remain isolated.
@@ -170,22 +182,22 @@ Each row type must have a non-null Workspace UUID, scoped indexes, scoped unique
 
 ### MCP, RAG, and Box
 
-- [x] MCP runtime key contains instance UUID, Workspace UUID, placement generation, and server UUID.
+- [x] MCP runtime key contains instance UUID, Workspace UUID, execution generation, and server UUID.
 - [x] Same-named MCP servers in two Workspaces do not share sessions.
 - [x] Pipeline cannot reference another Workspace's MCP resource.
 - [x] RAG collection names and handles are server-derived and Workspace scoped.
 - [x] Legacy global vector migration is available only to the local OSS singleton Workspace.
-- [x] Object storage paths include instance, Workspace, and placement generation for the fixed-generation OSS runtime.
+- [x] Object storage paths include instance, Workspace, and execution generation for the fixed-generation OSS runtime.
 - [x] Object storage revalidates generation before touching a provider or resolving an opaque key.
 - [ ] Cloud cutover uses generation-scoped staging plus stable published object references, rather than making the staging generation the durable identity.
 - [x] Box persistent and ephemeral namespaces include the required instance, Workspace, and generation scope.
-- [x] Same-named Box sessions and processes cannot collide across Workspaces or placement generations.
+- [x] Same-named Box sessions and processes cannot collide across Workspaces or execution generations.
 - [x] Box relay and process I/O reject or retire stale generations.
 - [x] External paths and privileged mounts cannot be supplied by an untrusted plugin.
 - [x] Cloud attachment host I/O uses query UUIDs and link-free dirfd operations with bounded inode traversal.
 - [x] Cloud Skill package paths are Runtime-owned, Workspace-scoped, read-only mounts; Python env/cache stays tenant-writable.
 - [x] Skill ZIP preview/install rejects path escape, links, non-regular files, duplicate entries, excessive compression ratio, entry count, per-file size, and total size.
-- [x] Cloud Box startup proves Core and Runtime see the same durable volume.
+- [x] Cloud Box code paths and automated tests require the authenticated marker challenge before startup or reconnect can proceed.
 - [x] Cloud Box readiness fails until hard Workspace, Skill, ephemeral-storage, and inode quota capabilities are available.
 
 ## 6. SDK and protocol
@@ -233,7 +245,7 @@ Each row type must have a non-null Workspace UUID, scoped indexes, scoped unique
 - [x] API Key cannot cross Workspace.
 - [x] Plugin cannot enumerate or invoke another Workspace's resources.
 - [x] Sessions, caches, locks, MCP, RAG, Box, storage, and monitoring do not collide.
-- [x] Background jobs cannot execute without an explicit Workspace and placement generation.
+- [x] Background jobs cannot execute without an explicit Workspace and execution generation.
 
 ### Security and revocation
 
@@ -241,7 +253,7 @@ Each row type must have a non-null Workspace UUID, scoped indexes, scoped unique
 - [x] OAuth redirects trust only server-configured WebUI or webhook origins, never request `Host` or `Origin` headers.
 - [x] Dashboard WebSockets revalidate authentication, Membership, resource, permission, and generation per message.
 - [x] Public embed WebSockets re-resolve Bot availability and execution binding per message.
-- [x] Runtime, storage, Plugin Runtime, MCP, RAG, and Box reject a stale placement generation.
+- [x] Runtime, storage, Plugin Runtime, MCP, RAG, and Box reject a stale execution generation.
 - [x] Unhandled API and webhook failures return a generic error plus request ID without exception text.
 - [x] URL user information and sensitive query parameters are redacted before configuration is serialized or logged.
 

docs/multi-tenant/implementation-decisions.md

@@ -2,8 +2,9 @@
 
 This log records implementation choices made while delivering the Workspace architecture. It is intended to make trade-offs auditable without interrupting implementation for routine decisions.
 
-> Open challenges that may supersede parts of this log are tracked in
-> [pending-architecture-decisions.md](./pending-architecture-decisions.md). They remain proposals until a decision is recorded here.
+> Architecture decisions, activation gates, and still-open follow-ups are tracked in
+> [pending-architecture-decisions.md](./pending-architecture-decisions.md). Sections marked as decided there are authoritative;
+> this file records the concrete implementation choices and compatibility names used to realize them.
 
 ## 2026-07-18
 
@@ -11,8 +12,8 @@ This log records implementation choices made while delivering the Workspace arch
 
 - Decision: Community builds create exactly one Workspace per LangBot instance and allow multiple Accounts through invitations.
 - Reason: This preserves a simple self-hosted deployment while making authorization and ownership explicit. Creating a second Workspace is an edition error, not a hidden fallback.
-- SaaS boundary: Multi-Workspace directory, placement, entitlement, and billing are the responsibility of a separate closed SaaS Control Plane. Core consumes a validated projection and remains the final isolation and authorization enforcement point; it does not become the SaaS system of record or billing engine.
-- Deployment boundary: Cloud v2 is a greenfield deployment design. The previous Space instance/pod deployment scheme is not migrated, preserved for compatibility, or extended by this implementation. Existing Space OAuth, marketplace, and payment concepts may be reused only through explicit adapters where they still fit; `langbot-space` itself is not changed.
+- SaaS boundary: Multi-Workspace directory, execution ownership, entitlement, and billing are the responsibility of a separate closed SaaS Control Plane. Core consumes a validated projection and remains the final isolation and authorization enforcement point; it does not become the SaaS system of record or billing engine.
+- Deployment boundary: Cloud v2 is a greenfield deployment design. The previous per-account instance/pod scheme is not migrated, preserved for compatibility, or extended by this implementation. Existing OAuth, marketplace, and payment concepts may be reused only through explicit adapters where they still fit; `langbot-space` itself is not changed.
 
 ### Workspace selection is trusted only after authentication
 
@@ -37,10 +38,10 @@ This log records implementation choices made while delivering the Workspace arch
 
 ### Plugin Runtime is shared; every plugin process is single-Workspace
 
-- Decision: One instance-scoped Plugin Runtime control plane serves all Workspaces in the logical LangBot instance. Each plugin installation still launches as its own nsjail worker with an immutable binding containing `instance_uuid`, `workspace_uuid`, `placement_generation`, `installation_uuid`, `runtime_revision`, and verified artifact digest. A worker never routes actions for another Workspace or installation, and plugin-supplied scope fields are stripped.
+- Decision: One instance-scoped Plugin Runtime control plane serves all Workspaces in the logical LangBot instance. Each running plugin installation has its own nsjail worker with an immutable binding containing `instance_uuid`, `workspace_uuid`, `execution_generation` (stored as the compatibility field `placement_generation` until the schema rename), `installation_uuid`, `runtime_revision`, and verified artifact digest; enabled-resident is the desired semantic. A worker never routes actions for another Workspace or installation, and plugin-supplied scope fields are stripped.
 - Isolation: Plugin code is mounted read-only. Home, tmp, and data paths are installation-scoped; process, file-descriptor, file-size, CPU, memory, and PID limits come only from `data/config.yaml` (including native environment overrides), never from a plugin manifest. Cloud requires nsjail and delegated cgroup v2 hard limits or fails closed.
 - Cost boundary: Identical verified package bytes share one digest-addressed code cache. A dependency environment is keyed by the artifact and requirements digests, Python ABI, Runtime version, and installer schema, then atomically published read-only for reuse. Installations and processes are not merged, even for the same plugin and version. Registration creates database desired state only; a worker is launched only for an enabled installation.
-- Recovery: PostgreSQL installation desired state and durable binary storage are authoritative. Runtime reconnect performs an instance-wide full reconciliation, removes stale workers, and can replay a verified package after Runtime-local cache loss. Dependency preparation failure is recorded per installation with `dependency_prepare_failed`; it prevents that worker launch without blocking recovery of other desired installations, and the same revision can be retried.
+- Recovery: PostgreSQL installation desired state and durable binary storage are authoritative. Runtime reconnect performs an instance-wide full reconciliation, removes stale workers, and can replay a verified package after Runtime-local cache loss. Dependency preparation failure is recorded per installation with `dependency_prepare_failed`; it prevents that worker launch without blocking recovery of other desired installations, and the same revision can be retried. Enabled-resident is the desired semantic, but the current Supervisor restores an unexpectedly exited worker only on the next Core apply/reconcile; a completion callback, bounded backoff, and cross-tenant restart-storm isolation remain Cloud activation gates.
 - Compatibility: Older SDK payloads and legacy `data/plugins` remain an OSS-only bridge. Shared mode requires complete bindings and rejects incomplete context.
 - Reason: Sharing the supervisor and immutable code cache removes per-Workspace service cost without turning an untrusted plugin process into a cross-tenant router.
 
@@ -68,7 +69,7 @@ This log records implementation choices made while delivering the Workspace arch
 ### SaaS execution state is a validated Core projection
 
 - Decision: Core can resolve both local and `cloud_projection` Workspaces, but only from an explicit Workspace UUID and an active, unfenced `WorkspaceExecutionState` for the current instance and matching source. OSS-only bootstrap paths additionally require `source=local`.
-- Reason: The closed control plane owns placement decisions, while Core remains the enforcement point for instance binding, generation, and write fences.
+- Reason: The closed control plane owns execution ownership and generation decisions, while Core remains the enforcement point for instance binding, generation, and write fences.
 
 ### API-key secrets are one-time and Workspace-bound
 
@@ -88,7 +89,7 @@ This log records implementation choices made while delivering the Workspace arch
 ### Cloud directory writes stay outside Core
 
 - Decision: The open-source Core startup always installs `SingleWorkspacePolicy`, creates or repairs one local Workspace, and permits local membership/invitation workflows. Changing mutable configuration such as `system.edition` cannot activate multi-Workspace routing. The future closed Cloud bootstrap will install `CloudWorkspacePolicy` only after verifying a signed `InstanceManifest`; that policy requires an explicit projected Workspace selector, does not create Workspaces, and rejects invitation or membership mutations with `control_plane_required`; member reads use the versioned local projection.
-- Ownership split: The closed Control Plane owns the global Account/Workspace/Membership/Invitation directory, placement and lease generations, entitlements, subscription state, usage aggregation, and billing decisions. Core owns request authorization, resource scoping, execution-generation validation, and fail-closed enforcement. Provisioning and invoice computation do not belong in open-source Core.
+- Ownership split: The closed Control Plane owns the global Account/Workspace/Membership/Invitation directory, execution ownership and generation, entitlements, subscription state, usage aggregation, and billing decisions. Core owns request authorization, resource scoping, execution-generation validation, and fail-closed enforcement. Provisioning and invoice computation do not belong in open-source Core.
 - Reason: The closed control plane is authoritative for SaaS Account, Workspace, Membership, and Invitation state. Allowing Core to mutate the same directory would create split-brain ownership and would make an ownerless compatibility Workspace a dangerous fallback.
 - Release gate: Multi-Workspace activation is deliberately unavailable in the open-source bootstrap. Production Cloud v2 must implement the signed `InstanceManifest` verifier and closed bootstrap described in the architecture document before it can inject `CloudWorkspacePolicy`; `edition=cloud`, an environment variable, or any unsigned local configuration is never a valid activation credential.
 
@@ -110,7 +111,7 @@ This log records implementation choices made while delivering the Workspace arch
 
 ### Dashboard WebSocket sessions are tenant runtime objects
 
-- Decision: A dashboard WebSocket sends an authentication frame immediately after upgrade. The server validates Account, Membership, permission, Pipeline ownership, instance, Workspace, and placement generation before registering the connection. Connection indexes, sessions, broadcasts, attachments, and resets include the complete execution scope.
+- Decision: A dashboard WebSocket sends an authentication frame immediately after upgrade. The server validates Account, Membership, permission, Pipeline ownership, instance, Workspace, and execution generation before registering the connection. Connection indexes, sessions, broadcasts, attachments, and resets include the complete execution scope.
 - Reason: Browser WebSocket APIs cannot attach the normal authorization headers, and a process-global `pipeline_uuid` or `session_type` index can collide across Workspaces.
 
 ### Read permissions never imply secret permissions
@@ -120,7 +121,7 @@ This log records implementation choices made while delivering the Workspace arch
 
 ### Temporary credential exchanges are bound to their initiator
 
-- Decision: Lark, Weixin, DingTalk, WeComBot, and QQOfficial one-click registration sessions require `resource.manage` and store the initiating instance, Workspace, placement generation, and principal. Status and cancellation by any other scope return the same 404 as an unknown session.
+- Decision: Lark, Weixin, DingTalk, WeComBot, and QQOfficial one-click registration sessions require `resource.manage` and store the initiating instance, Workspace, execution generation, and principal. Status and cancellation by any other scope return the same 404 as an unknown session.
 - Reason: Random session IDs reduce guessing probability but do not authorize access to credentials returned by a completed exchange.
 
 ### Uploaded images and documents use different storage capabilities
@@ -153,18 +154,18 @@ This log records implementation choices made while delivering the Workspace arch
 - Compatibility: In-memory SQLite cannot provide this recovery guarantee and is rejected for destructive production migration boundaries; it remains usable in tests that create the final schema directly.
 - Reason: SQLite batch table rebuilds can leave an installation between schemas if a process or migration fails. A verified pre-boundary image makes retry behavior recoverable instead of merely idempotent in the happy path.
 
-### Placement generation is an execution revocation capability
+### Execution generation is an execution revocation capability
 
-- Decision: RuntimeBot, RuntimePipeline, background tasks, object storage, Plugin Runtime, MCP, RAG, and Box operations carry the complete instance, Workspace, and placement-generation scope. They revalidate the active execution binding before accessing a provider or transport; long-running calls validate again before accepting results. A stale generation is fenced before it can read, write, or reuse a cached object.
+- Decision: RuntimeBot, RuntimePipeline, background tasks, object storage, Plugin Runtime, MCP, RAG, and Box operations carry the complete instance, Workspace, and execution-generation scope. The current schema and wire compatibility field remains `placement_generation` until a coordinated rename. They revalidate the active execution binding before accessing a provider or transport; long-running calls validate again before accepting results. A stale generation is fenced before it can read, write, or reuse a cached object.
 - Plugin boundary: Each locally launched plugin receives a short-lived, one-use registration capability bound to the expected manifest identity and execution scope. The production child environment does not inherit the reusable debug credential, and Host APIs derive scope from the trusted connection and action context.
-- Box boundary: Persistent skill content remains Workspace-scoped, while session/process state and relay requests also include placement generation. A generation change retires matching live sessions and closes a stale relay before further stdin, stdout, or file operations.
-- Transaction boundary: Request admission and runtime side effects are fenced in this branch, but ordinary tenant database mutations do not yet hold a generation-aware lock through commit. The closed Cloud bootstrap must remain disabled until Core provides the shared-write/exclusive-cutover transaction primitive and a generation-stamped outbox (or an equivalent atomic publish fence). The OSS singleton policy has a fixed local generation and cannot trigger a placement cutover.
+- Box boundary: Persistent skill content remains Workspace-scoped, while session/process state and relay requests also include execution generation. A generation change retires matching live sessions and closes a stale relay before further stdin, stdout, or file operations.
+- Transaction boundary: Request admission and runtime side effects are fenced in this branch, but ordinary tenant database mutations do not yet hold a generation-aware lock through commit. The closed Cloud bootstrap must remain disabled until Core provides the shared-write/exclusive-cutover transaction primitive and a generation-stamped outbox (or an equivalent atomic publish fence). The OSS singleton policy has a fixed local generation and cannot trigger an execution-owner cutover.
 - Durable-object boundary: Current opaque storage keys include generation and therefore fail closed after a generation change. That is safe for OSS's fixed generation, but a Cloud cutover must not strand durable KB files, images, or plugin references. Cloud v2 must publish stable final object identities from generation-scoped staging, or perform an atomic object-and-reference migration before activating the new generation.
-- Reason: Workspace UUID prevents cross-tenant collisions, but it cannot revoke work after a Workspace is moved or fenced. Placement generation is the monotonic lease that makes old runtimes unusable.
+- Reason: Workspace UUID prevents cross-tenant collisions, but it cannot revoke work after execution ownership changes or is fenced. Execution generation is the monotonic revocation value that makes old runtimes unusable; it does not express membership in a product-level deployment entity.
 
 ### Long-lived WebSockets continuously revalidate authority
 
-- Decision: Dashboard WebSockets re-authenticate the Account, Membership, permission, resource ownership, instance, Workspace, and placement generation for every inbound message, not only during the initial frame. A changed role, removed Membership, or fenced placement takes effect without waiting for reconnect.
+- Decision: Dashboard WebSockets re-authenticate the Account, Membership, permission, resource ownership, instance, Workspace, and execution generation for every inbound message, not only during the initial frame. A changed role, removed Membership, or fenced execution binding takes effect without waiting for reconnect.
 - Public embed boundary: The embed connection re-resolves its Bot before every message and rejects a Bot that was disabled, deleted, moved, or rebound. The public connection may identify a Bot, but it cannot make the initial Bot object an indefinite authorization capability.
 - Reason: Authorization and resource state can change while a socket remains open. Connection-time validation alone leaves a revocation gap.
 
@@ -181,7 +182,7 @@ This log records implementation choices made while delivering the Workspace arch
 
 ### Box is one shared control plane with one admitted sandbox per Workspace
 
-- Decision: The logical instance has one shared Box Runtime. A closed entitlement adapter projects generic `managed_sandbox` capability and `managed_sandbox_sessions` limit; Core and Runtime never branch on a plan name. An eligible Workspace receives at most one persistent logical `global` session, while each ordinary command remains a one-shot nsjail process. Managed processes and network are disabled in the first Cloud release.
+- Decision: The logical instance has one shared Box Runtime control plane, implemented by one Runtime replica in M0. A closed entitlement adapter projects generic `managed_sandbox` capability and `managed_sandbox_sessions` limit; Core and Runtime never branch on a plan name. An eligible Workspace receives at most one persistent logical `global` session, while each ordinary command remains a one-shot nsjail process. Managed processes and network are disabled in the first Cloud release.
 - Storage boundary: Core and Runtime prove they see the same durable volume with an authenticated random-marker challenge. Attachments use opaque query UUID directories and link-free dirfd operations. Skill packages remain in the Runtime-owned Workspace store and enter a sandbox only as a read-only logical-name mount; Python environments and caches live in the tenant's writable Workspace.
 - Resource boundary: Cloud readiness requires cgroup v2 plus hard byte and inode limits for Workspace files, Skill storage, root, tmp, and home. The existing directory scan is only a compatibility soft check. Plain nsjail reports these storage capabilities as unavailable, so Cloud Box intentionally cannot start until the greenfield deployment supplies and verifies a real quota provider.
 - Archive boundary: Skill ZIP processing is bounded by compressed input, entry count, per-entry size, total uncompressed size, and compression ratio, and rejects links, non-regular entries, duplicates, and path escape before streaming extraction.
@@ -208,3 +209,36 @@ This log records implementation choices made while delivering the Workspace arch
 
 - Decision: `mcp.stdio.enabled` is independent of Box availability and entitlement. OSS defaults it on for compatibility; Cloud requires it off at bootstrap and enforces the same gate on create, update, test, startup loading, and final runtime execution.
 - Reason: Treating Box availability as stdio permission would silently create another persistent `mcp-shared` sandbox for each Workspace and bypass the one-sandbox subscription and cost boundary.
+
+## 2026-07-20
+
+### The tenant UoW owns its task, root transaction, bind, and scope
+
+- Decision: A tenant UoW creates one task-owned `TenantScopedAsyncSession` and one root transaction. Public commit, rollback, close, connection, bind, nested-transaction, synchronous-Session, live-streaming, raw SQL, public execution options, and public `set_config` paths fail closed and mark the transaction rollback-only. ORM objects cannot expose a usable synchronous Session, captured methods cannot run in child tasks, an explicit foreign bind is rejected, and a captured Session is permanently retired when its UoW exits rather than being reset for reuse. Tenant scope is installed only through a private UoW capability; pgvector index-plan `SET LOCAL`/`EXPLAIN` diagnostics use a test/operator connection rather than the business Session API.
+- SQL boundary: Public UoW calls accept only structured SQLAlchemy query and DML trees. `TextClause`, literal SQL columns, textual labels, prefixes/suffixes/hints, statement execution options, `VALUES` roots, `INSERT FROM SELECT`, `EXTRACT`, literal-execute parameters, unknown/custom AST nodes, forced-unquoted identifiers, named `ON CONFLICT` constraints, unknown dialect post-values clauses, and untrusted casts/types fail closed. PostgreSQL/SQLite `ON CONFLICT DO UPDATE` and batch-insert containers are traversed explicitly because SQLAlchemy's standard visitor omits their executable values. Function classes are exactly allowlisted as `count`, `coalesce`, `sum`, `now`, `length`, and `nullif`; the only custom operator/cast admitted is the validated pgvector cosine operator and `Vector` cast.
+- Legacy migration boundary: The local-only RAG backup restore uses explicit table and column objects, never raw SQL, but deliberately leaves legacy values untyped. This preserves SQLite's string-valued `DATETIME` rows and both the historical PostgreSQL `TEXT` and fresh-schema `JSON` settings columns while keeping every value bound rather than interpolated.
+- ORM boundary: SQLAlchemy `SessionEvents` are unsupported on a tenant-scoped Session. If a listener is registered before or during a UoW, the operation fails before the callback executes and cleanup proceeds against an empty dispatch surface. Public `get`, `get_one`, `refresh`, and `merge` reject caller-supplied loader, bind, lock, shard, and execution options. Flush, implicit autoflush, and commit reject a SQL expression assigned to a mapped attribute before it can reach the compiler. Tenant code uses the async Session directly; relationships use eager loading or explicit `await session.refresh(entity, [attribute])`. LangBot's persistence base does not expose `AsyncAttrs.awaitable_attrs` as a supported tenant API.
+- Compiler trust boundary: This guard prevents accidental scope/transaction escape by trusted LangBot Core code; it is not an in-process Python sandbox. Registered SQLAlchemy compilers and mapped schema metadata are trusted boot-time code. The fail-closed traversal of dialect containers necessarily covers SQLAlchemy private fields, so dependency upgrades require the regression suite and remain pinned until verified. Untrusted plugins cannot import or call this Session because they remain isolated in Plugin Runtime child processes.
+- Result boundary: A caught database or boundary failure rolls back the root transaction and cancels after-commit work. Buffered results contain only already-authorized rows and no live connection; live database results cannot escape the UoW operation.
+- Reason: `SET LOCAL` plus RLS protects a tenant only while every statement stays on the same owned connection and callers cannot end the transaction, replace the GUC, recover the synchronous proxy, or route a statement through another bind.
+
+### Parallel request work re-enters tenant scope explicitly
+
+- Decision: Child coroutines created by request-level `asyncio.gather` open their own explicit transaction-free tenant scope before calling persistence-backed Plugin, MCP, or Skill operations. They never inherit the parent's active database Session.
+- Reason: Python copies ContextVars into child tasks, but SQLAlchemy Sessions are not task-safe and task identity is part of the tenant UoW boundary.
+
+### Ordinary monitoring is readable; audit and export remain privileged
+
+- Decision: Workspace monitoring dashboards, Bot logs, sessions, messages, calls, errors, and feedback require `resource.view`. Monitoring export requires `data.export`; system/runtime audit logs keep `audit.view`. Frontend tabs and controls use the same split.
+- Reason: A Viewer needs useful read-only product observability, while bulk data extraction and privileged runtime/system logs are separate capabilities.
+
+### Invitation failures survive login without contradictory success state
+
+- Decision: Invitation terminal codes map to stable browser states. A login-mediated email mismatch preserves the fragment-captured secret only in session storage, returns to the acceptance page with the stable error code, and suppresses the generic login-success toast. A transient acceptance failure retains the authenticated session and offers the same one-time invitation for retry. Invalid Bearer tokens on acceptance map to the normal authentication error instead of an internal failure.
+- Reason: Invitation acceptance crosses signed-out and authenticated states; losing or masking the domain error makes recovery ambiguous and can present contradictory UI feedback.
+
+### Shared Plugin Runtime starts only from verified desired state
+
+- Decision: SDK shared mode waits for immutable runtime configuration before inspecting plugin state, never scans or launches legacy `data/plugins`, and rejects legacy install/restart/delete/upgrade control actions. Worker RPC files use installation-private directories with aggregate size enforcement, and resident nsjail workers explicitly disable the default 600-second wall-time limit.
+- Remaining gate: Unexpected worker exit is still recovered only by the next Core apply/reconcile. Completion callbacks, bounded backoff, hard installation disk quota, and production Linux/cgroup/egress evidence remain Cloud activation requirements.
+- Reason: A shared supervisor reduces per-Workspace services only if legacy global paths, writable transfer state, and lifecycle defaults cannot bypass installation isolation.

docs/multi-tenant/pending-architecture-decisions.md

@@ -28,7 +28,7 @@
 
 | 编号  | 结论                                                                                                                                          | 首期状态                                                             |
 | ----- | --------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
-| D-001 | 一个共享 Plugin Runtime 控制面；每个运行中的 plugin installation 独占一个 nsjail 子进程；只有 digest 相同且已验证的代码 artifact 可以只读共享 | `IMPLEMENTED — real Linux/egress deployment gate pending`            |
+| D-001 | 一个共享 Plugin Runtime 控制面；每个运行中的 plugin installation 独占一个 nsjail 子进程；只有 digest 相同且已验证的代码 artifact 可以只读共享 | `FOUNDATION IMPLEMENTED — Linux/egress, crash recovery and disk-quota gates pending` |
 | D-002 | 一个共享 Box Runtime；Cloud 固定使用 nsjail；符合套餐的 Workspace 最多一个持久 `global` 逻辑 sandbox，普通执行按需启动 nsjail 进程            | `IMPLEMENTED FAIL-CLOSED — hard filesystem quota provider pending`   |
 | D-003 | SaaS 业务数据使用 PostgreSQL shared schema、应用层作用域和 RLS 双重隔离；pgvector 使用同一 PostgreSQL，作为 SaaS 默认向量后端                 | `PARTIALLY IMPLEMENTED — transaction/outbox/deployment gates remain` |
 | D-004 | stdio MCP 与 Box availability 解耦；Cloud v2 首期强制关闭 stdio MCP，避免为每个 Workspace 创建额外的 `mcp-shared` persistent sandbox          | `IMPLEMENTED`                                                        |
@@ -82,12 +82,12 @@ Core 不能继承 Plugin Runtime 所需的 nsjail/cgroup 权限。Box Runtime
 
 ## 3. D-001：Plugin Runtime 多租户控制面
 
-状态：`IMPLEMENTED — Cloud deployment verification pending`
+状态：`FOUNDATION IMPLEMENTED — Cloud deployment and worker-recovery gates pending`
 
 ### 3.1 已实现的基础
 
-- Plugin Runtime 控制连接只绑定稳定实例身份；一个 Supervisor 通过完整 installation binding 管理多个 Workspace。
-- 每个 enabled installation 使用独立 nsjail worker；代码只读，home/tmp/data 私有，shared profile 不读取 artifact `.env`。
+- Plugin Runtime 控制连接只绑定稳定实例身份；一个逻辑共享控制面通过完整 installation binding 管理多个 Workspace，M0 由一个 Supervisor replica 承担。
+- 每个运行中的 installation 使用独立 nsjail worker；enabled-resident 是 desired semantics。代码只读，home/tmp/data 私有，shared profile 不读取 artifact `.env`。
 - 实例级 `PluginWorkerPolicy` 由 Core 的 `data/config.yaml` 下发，支持原生环境变量覆写；manifest 不能覆盖。
 - `installation_uuid`、`artifact_digest` 和 `runtime_revision` 已持久化并进入 desired-state、注册、Host API 和 generation/revision fence。
 - 已验证 `.lbpkg` 先进入 Workspace-scoped durable binary storage；Runtime 本地缓存丢失后可由 Core replay。
@@ -110,13 +110,14 @@ Core 不能继承 Plugin Runtime 所需的 nsjail/cgroup 权限。Box Runtime
 
 ### 3.3 首期执行模型
 
-- 整个 SaaS 实例共享一个可信 Plugin Runtime Supervisor；新 Workspace 不创建专属 Runtime、连接、卷或进程。
+- 整个 SaaS 实例共享一个可信 Plugin Runtime 逻辑控制面，M0 运行一个 Supervisor replica；新 Workspace 不创建专属 Runtime、连接、卷或进程。
 - Supervisor 的控制连接只绑定稳定 `instance_uuid` 和短期 Runtime identity，不绑定某个 Workspace。
   每条 installation desired-state 命令都携带并验证完整的 installation binding；每个 worker action context 在注册后永久绑定该 tuple。
 - 安装并启用插件后，Supervisor 在自己的 Runtime 容器内直接启动一个 nsjail 子进程；
   不再为每个插件创建 nested container、Pod、sidecar 或租户级 Runtime service。
-- 为兼容现有事件监听和定时任务语义，首期继续采用“enabled installation 保持 resident”，不做 idle eviction；
-  停用、删除、revision/generation 变化或 entitlement 撤销时停止并按需重建。
+- desired semantics 要求 enabled installation 保持 resident，不做 idle eviction；停用、删除、revision/generation 变化或 entitlement 撤销时停止并按需重建。
+  当前实现只会在 Runtime 重连或 Core apply/reconcile 时恢复意外退出的 worker，尚缺 completion callback、有界 backoff 和跨租户重启风暴抑制；
+  这属于 Cloud 激活门禁，不能把 desired semantics 描述成已经具备即时自愈。
 - 子进程使用一次性 registration capability 向 Supervisor 注册；capability 由可信 desired state 派生并绑定完整 installation tuple，
   不是插件直接建立 Core Host connection，也不能只绑定 author/name/path。Supervisor/Core 据此注入 tenant context，
   丢弃插件 payload 中自带的 scope 字段。
@@ -193,8 +194,9 @@ installation data 的总空间硬配额需要 filesystem project quota 或独立
 | MVP 不引入 | Runtime 专用数据库、Redis、Kafka 或独立 scheduler  | 当前无容量证据，会增加组件和运维面                       |
 | 后续演进   | 多 Supervisor replica、owner lease、dedicated pool | 保留接口，达到容量或可用性阈值后再决定具体存储与调度方式 |
 
-后续仍需决定的只有：Core/Supervisor 是否共置、artifact/venv cache 的签名/来源/撤销/GC 规范、v1 connection 的兼容期限，
-以及进入多 replica 后的 lease TTL、fencing token 和 owner 转移顺序。这些不改变“每 installation 一个隔离进程”的首期边界。
+架构扩展项包括：Core/Supervisor 是否共置、artifact/venv cache 的签名/来源/撤销/GC 规范、installation data hard-quota provider、
+v1 connection 的兼容期限，以及进入多 replica 后的 lease TTL、fencing token 和 owner 转移顺序。
+这些不改变“每个运行中的 installation 一个隔离进程”的首期边界。
 
 ### 3.7 验收条件
 
@@ -203,8 +205,10 @@ installation data 的总空间硬配额需要 filesystem project quota 或独立
 - 插件不能读取其他 installation 文件、枚举或 signal 其他进程，也不能修改共享代码/依赖目录。
 - CPU、内存、PID、open files 和单文件上限在真实 nsjail/cgroup 环境中生效；超额只终止或拒绝对应 installation。
 - 修改 manifest 不能改变任何资源上限。
+- installation data 的总空间硬配额在写入边界原子拒绝超额，并证明目录扫描不是生产 enforcement。
 - 旧 generation/revision 的回调、消息、副作用和存储访问全部失败关闭。
 - Runtime 重启能从业务 desired state 恢复，不依赖本地进程表作为权威真相。
+- 意外退出的 enabled worker 由 completion callback 触发带有界 backoff 的自动恢复；连续失败只影响对应 installation，不能形成跨租户重启风暴。
 - requirements 中存在 Runtime 基础镜像未预装的包时，Supervisor 仍能先完成共享依赖环境准备再启动 worker；
   安装失败不会留下持续重启的半启动进程，也不会影响同 digest 已就绪环境的其他 installation。
 
@@ -253,7 +257,7 @@ installation data 的总空间硬配额需要 filesystem project quota 或独立
 
 ### 4.3 Cloud nsjail 执行模型
 
-- 整个逻辑 SaaS 实例共享一个 Box Runtime 控制面；不创建每 Workspace Box service、worker pool、PVC、bucket、scheduler、Redis 或 Box 数据库。
+- 整个逻辑 SaaS 实例共享一个 Box Runtime 逻辑控制面，M0 运行一个 Runtime replica；不创建每 Workspace Box service、worker pool、PVC、bucket、scheduler、Redis 或 Box 数据库。
 - Cloud 显式固定 `box.backend: nsjail`。sandbox 直接作为 Box Runtime 容器内的 nsjail 子进程运行，
   不创建 nested Docker container、独立 Pod、microVM 或 warm pool，也不挂宿主机 `docker.sock`。
 - 符合 entitlement 的 Workspace 首次使用时懒创建一个逻辑 session，内部固定 ID 为 `global`，并强制 `persistent=True`；