feat(tenancy): connect cloud workspace control plane

This commit is contained in:
Junyan Qin
2026-07-24 19:11:33 +08:00
parent d7cdd206c2
commit 98f45aa88e
40 changed files with 4159 additions and 452 deletions
+11 -7
View File
@@ -8,7 +8,7 @@ verification gates. Exact commands and observed results are recorded in the
- [x] LangBot uses branch feat/multi-tenants.
- [x] langbot-plugin-sdk uses branch feat/multi-tenants.
- [x] langbot-space has no changes made by this implementation; Cloud v2 does not extend the legacy Space deployment scheme.
- [x] langbot-space implements the greenfield Cloud v2 modular-monolith control plane without extending the legacy per-account Pod topology; the old Pod UI remains available when Cloud v2 is disabled and only retained Pods appear in the v2 view.
- [x] Unrelated untracked files in either repository remain untouched.
- [x] Open-source startup cannot enable SaaS multi-workspace through edition flags or unsigned configuration.
@@ -20,18 +20,22 @@ deployment. The feature branch delivers the Core isolation kernel, not the
closed SaaS product or a production Cloud v2 deployment. Checked implementation
items later in this document do not supersede these gates.
- [ ] The closed Control Plane owns the global Account, Workspace, Membership, and Invitation directory.
- [x] The closed Control Plane owns the global Account, Workspace, Membership, and Invitation directory.
- [ ] The closed Control Plane execution-ownership module issues monotonic generations and owner leases for projected Workspaces.
- [ ] Core verifies a signed `InstanceManifest` before the closed bootstrap can inject `CloudWorkspacePolicy`.
- [x] Core verifies a signed `InstanceManifest` before the closed bootstrap can inject `CloudWorkspacePolicy`.
- [ ] Tenant database writes hold a generation-aware shared transaction fence through commit, while execution-owner cutovers take the exclusive fence.
- [ ] Business writes and non-transactional side effects use a generation-stamped outbox or equivalent publish fence.
- [ ] Durable object references survive an execution-generation change through stable published keys or an explicitly atomic key/reference migration.
- [ ] The SaaS runtime pools enforce tenant-safe egress and SSRF controls for Webhooks, providers, MCP servers, and every tenant-configurable outbound URL.
- [ ] Entitlement checks, usage aggregation, subscription lifecycle, and billing are implemented in the closed Control Plane.
- [ ] Entitlement checks, usage aggregation, and subscription lifecycle are implemented in the closed Control Plane; production activation still requires provider callback amount/currency/session/expiry binding inside the locked fulfillment transaction.
- [ ] Account registration persists a `new_api.provision_account` outbox item with the Account and personal Workspace, and an in-process reconciler provisions New API idempotently after commit.
- [ ] EPay and Stripe callbacks bind provider identity, amount, currency, channel/session, payment status, and expiry to the locked order before entitlement fulfillment.
- [ ] OAuth state and directory projection use an atomic shared store suitable for horizontally scaled SaaS services.
- [ ] A greenfield Cloud v2 deployment is designed and validated independently of the legacy Space deployment scheme.
- [ ] The Plugin Runtime deployment provides delegated cgroup v2 and tenant-safe egress; the shared profile refuses to run without hard CPU, memory, and PID limits.
- [ ] The Plugin Runtime Supervisor automatically restores an unexpectedly exited enabled worker with bounded backoff and cannot create a cross-tenant restart storm.
- [ ] The Plugin Runtime shared profile refuses to run without delegated cgroup v2 CPU, memory-plus-swap, and PID limits, all verified in a real Linux container; production tenant-safe egress remains incomplete.
- [x] The Plugin Runtime Supervisor automatically restores an unexpectedly exited enabled worker with bounded per-installation backoff.
- [ ] Jitter, a global restart concurrency limit, and a Runtime-level circuit breaker prevent a systemic failure from creating a cross-tenant restart storm.
- [ ] Until authenticated Runtime takeover or an owner lease/fence exists, the M0 deployment rolls Core and Plugin Runtime together and forbids an independent Core-only rollout.
- [ ] Plugin installation data has an operator-owned hard disk quota provider that atomically rejects writes over the limit; directory scans are not accepted as enforcement.
- [ ] The Box deployment provides an operator-owned quota provider that proves hard byte and inode limits for Workspace, Skill, root, tmp, and home storage.
- [ ] Core and Box Runtime mount the same durable volume and pass the authenticated marker challenge during startup and reconnect.
@@ -281,7 +285,7 @@ Each row type must have a non-null Workspace UUID, scoped indexes, scoped unique
## 10. Completion evidence
- [x] LangBot and SDK branch refs are recorded in the verification report.
- [x] Space git diff is empty relative to the pre-work snapshot.
- [x] Space contains the closed adapter package and Cloud v2 control plane, billing, migration, and Workspace UI changes; unrelated pre-existing files remain unstaged.
- [x] Migration output is captured for SQLite and PostgreSQL.
- [x] Test commands and results are recorded.
- [x] Browser E2E actions and observed results are recorded.
+33 -5
View File
@@ -13,7 +13,7 @@ This log records implementation choices made while delivering the Workspace arch
- Decision: Community builds create exactly one Workspace per LangBot instance and allow multiple Accounts through invitations.
- Reason: This preserves a simple self-hosted deployment while making authorization and ownership explicit. Creating a second Workspace is an edition error, not a hidden fallback.
- SaaS boundary: Multi-Workspace directory, execution ownership, entitlement, and billing are the responsibility of a separate closed SaaS Control Plane. Core consumes a validated projection and remains the final isolation and authorization enforcement point; it does not become the SaaS system of record or billing engine.
- Deployment boundary: Cloud v2 is a greenfield deployment design. The previous per-account instance/pod scheme is not migrated, preserved for compatibility, or extended by this implementation. Existing OAuth, marketplace, and payment concepts may be reused only through explicit adapters where they still fit; `langbot-space` itself is not changed.
- Deployment boundary: Cloud v2 is a greenfield deployment design. The previous per-account instance/pod scheme is not migrated or extended and remains available only for existing subscriptions. Existing OAuth, marketplace, and payment rails are reused through explicit adapters where they still fit; new Workspace, subscription, entitlement, directory, and usage modules live alongside the legacy Pod flow in `langbot-space`.
### Workspace selection is trusted only after authentication
@@ -41,7 +41,7 @@ This log records implementation choices made while delivering the Workspace arch
- Decision: One instance-scoped Plugin Runtime control plane serves all Workspaces in the logical LangBot instance. Each running plugin installation has its own nsjail worker with an immutable binding containing `instance_uuid`, `workspace_uuid`, `execution_generation` (stored as the compatibility field `placement_generation` until the schema rename), `installation_uuid`, `runtime_revision`, and verified artifact digest; enabled-resident is the desired semantic. A worker never routes actions for another Workspace or installation, and plugin-supplied scope fields are stripped.
- Isolation: Plugin code is mounted read-only. Home, tmp, and data paths are installation-scoped; process, file-descriptor, file-size, CPU, memory, and PID limits come only from `data/config.yaml` (including native environment overrides), never from a plugin manifest. Cloud requires nsjail and delegated cgroup v2 hard limits or fails closed.
- Cost boundary: Identical verified package bytes share one digest-addressed code cache. A dependency environment is keyed by the artifact and requirements digests, Python ABI, Runtime version, and installer schema, then atomically published read-only for reuse. Installations and processes are not merged, even for the same plugin and version. Registration creates database desired state only; a worker is launched only for an enabled installation.
- Recovery: PostgreSQL installation desired state and durable binary storage are authoritative. Runtime reconnect performs an instance-wide full reconciliation, removes stale workers, and can replay a verified package after Runtime-local cache loss. Dependency preparation failure is recorded per installation with `dependency_prepare_failed`; it prevents that worker launch without blocking recovery of other desired installations, and the same revision can be retried. Enabled-resident is the desired semantic, but the current Supervisor restores an unexpectedly exited worker only on the next Core apply/reconcile; a completion callback, bounded backoff, and cross-tenant restart-storm isolation remain Cloud activation gates.
- Recovery: PostgreSQL installation desired state and durable binary storage are authoritative. Runtime reconnect performs an instance-wide full reconciliation, removes stale workers, and can replay a verified package after Runtime-local cache loss. Dependency preparation failure is recorded per installation with `dependency_prepare_failed`; it prevents that worker launch without blocking recovery of other desired installations, and the same revision can be retried. The installation Supervisor now restores an unexpectedly exited enabled worker through a completion callback with bounded exponential backoff. Jitter, global restart concurrency limits, and a Runtime-wide circuit breaker are still required to prove that an infrastructure-wide failure cannot create a cross-tenant restart storm.
- Compatibility: Older SDK payloads and legacy `data/plugins` remain an OSS-only bridge. Shared mode requires complete bindings and rejects incomplete context.
- Reason: Sharing the supervisor and immutable code cache removes per-Workspace service cost without turning an untrusted plugin process into a cross-tenant router.
@@ -83,8 +83,8 @@ This log records implementation choices made while delivering the Workspace arch
### Unreleased SDK protocol is pinned reproducibly without publishing
- Decision: The SDK tenancy protocol is versioned as 0.4.15. This task does not create a GitHub release or publish PyPI because the user authorized pushing code, not a package release. After the SDK feature branch is final, LangBot's feature branch temporarily pins the exact pushed SDK Git commit. Before merging to master, the release gate is to publish `langbot-plugin==0.4.15` and replace the Git pin with the registry pin.
- Reason: PyPI 0.4.14 does not contain `ActionContext`; pinning it makes clean installs fail, while pinning an unpublished 0.4.15 makes dependency sync impossible. An exact Git commit is reproducible and keeps the feature branch testable without expanding release authority.
- Decision: The SDK tenancy protocol is versioned as 0.4.18. This task does not create a GitHub release or publish PyPI because the user authorized pushing code, not a package release. After the SDK feature branch is final, LangBot's feature branch temporarily pins the exact pushed SDK Git commit. Before merging to master, the release gate is to publish `langbot-plugin==0.4.18` and replace the Git pin with the registry pin.
- Reason: The current registry release does not contain the complete tenant action context and shared Runtime hardening. An exact Git commit is reproducible and keeps the feature branch testable without expanding release authority.
### Cloud directory writes stay outside Core
@@ -240,5 +240,33 @@ This log records implementation choices made while delivering the Workspace arch
### Shared Plugin Runtime starts only from verified desired state
- Decision: SDK shared mode waits for immutable runtime configuration before inspecting plugin state, never scans or launches legacy `data/plugins`, and rejects legacy install/restart/delete/upgrade control actions. Worker RPC files use installation-private directories with aggregate size enforcement, and resident nsjail workers explicitly disable the default 600-second wall-time limit.
- Remaining gate: Unexpected worker exit is still recovered only by the next Core apply/reconcile. Completion callbacks, bounded backoff, hard installation disk quota, and production Linux/cgroup/egress evidence remain Cloud activation requirements.
- Remaining gate: completion-callback recovery and bounded per-installation backoff are implemented. Cross-tenant restart-storm suppression, hard installation disk quota, and production egress policy remain Cloud activation requirements; Linux nsjail/cgroup CPU, memory-plus-swap, PID, namespace, and cgroup-reaping behavior have real-container evidence.
- Reason: A shared supervisor reduces per-Workspace services only if legacy global paths, writable transfer state, and lifecycle defaults cannot bypass installation isolation.
## 2026-07-24
### Cloud v2 remains a modular monolith with one closed adapter
- Decision: Workspace directory, plans, subscriptions, payment fulfillment, entitlements, usage, and signed runtime feeds are implemented as modules in the existing Space service and PostgreSQL database. The only separately packaged closed component is the thin Core bootstrap/control-plane adapter; it verifies signed data and owns no business state.
- Compatibility: The legacy Pod card and fulfillment path remain visible only to accounts with old subscriptions. New purchases create Workspace subscriptions and never provision a per-user LangBot Pod.
- Reason: A separate tenancy or billing service would add deployment, queue, network, and consistency cost without providing an isolation boundary. The signed adapter keeps SaaS logic closed while Core retains the ORM, RLS, authorization, and runtime enforcement boundary.
### Registration creates data, not infrastructure
- Decision: Account registration and personal Workspace creation share one transaction and include an active owner membership, Free subscription, entitlement snapshot, and outbox notifications. Repair is idempotent for older accounts. Empty Workspace creation starts no Plugin worker, Box sandbox, database, queue, bucket, or tenant service.
- Activation: This automatic Cloud v2 transaction is enabled only when Space has an explicit valid `CLOUD_V2_INSTANCE_UUID`. A legacy marketplace-only Space deployment with no Cloud v2 instance configured keeps its existing Account registration path; Cloud v2 internal endpoints still fail closed instead of inventing an instance identity.
- Billing boundary: Core and runtimes consume only generic capability and numeric-limit entitlements. They never branch on `free` or `pro`; plan names, prices, payment providers, and fulfillment stay in Space.
- Reason: This keeps the marginal cost of a new user close to a few PostgreSQL rows while preserving an immediate, usable Workspace.
### Directory bootstrap is full; steady-state projection is per Workspace
- Decision: Space generates the initial signed directory snapshot and its outbox high-water mark in one read-only PostgreSQL `REPEATABLE READ` transaction. After bootstrap, each event page names affected Workspaces and Core fetches one signed `directory.delta` for that set. Missing requested Workspaces are tombstones; unrelated Workspaces are untouched.
- Cursor boundary: A delta carries no event cursor. Each signed event page carries the transaction-consistent current high-water mark, while Core advances only to the final event it actually consumed, even when the authoritative delta already contains a later revision. Event payload Workspace and revision fields must exactly match the signed event envelope; readiness is renewed only after the replica-local cursor reaches the signed high-water and the shared projection is not ahead.
- Replica boundary: Every Core replica owns a process-local consumer cursor because its verified entitlement cache is process-local. Replicas share the PostgreSQL projection high-water mark and inbox. The state separately records which cursor range was atomically subsumed by a full snapshot, so a lagging replica can add missing receipts within that coverage and refresh its local caches without repeating tenant mutations; a missing receipt beyond snapshot coverage fails closed.
- Reason: A shared consumer cursor would let one replica starve another replica's local cache, while a full snapshot on every change would make steady-state cost grow with every registered Workspace.
### Cloud OAuth can authenticate only an existing projected Account
- Decision: In multi-Workspace Cloud mode, Space OAuth may refresh tokens only for an active `cloud_projection` Account whose Space subject UUID and normalized email exactly match. It cannot create an Account, relink by email, mutate directory identity, or choose a Workspace.
- Redirect boundary: When Cloud v2 configures its Core public URL, Space issues authorization codes only to that exact callback origin or the explicitly retained legacy managed-Pod domain. Community installations remain dynamic OAuth clients because they cannot pre-register with the public Space service: the consent screen displays their hostname, and only HTTPS or loopback HTTP with the fixed callback path is accepted. Remote HTTP, userinfo, fragments, arbitrary callback paths, and unrecognized query parameters always fail closed.
- Reason: Space is the SaaS identity and directory authority, but Core remains the authentication enforcement point. Projected-only matching avoids split-brain identities; redirect allowlisting prevents bearer authorization-code exfiltration.
@@ -2,7 +2,7 @@
状态:`DECIDED — Core isolation kernel implemented; SaaS activation gates remain`
创建日期:2026-07-19
最近更新:2026-07-20
最近更新:2026-07-24
本文记录 Cloud v2 多租户架构中已经确认的首期决策、明确淘汰的方案和仍需在后续阶段决定的扩展项。
本文同时记录实现状态。“实现完成”仅指开源 Core/SDK 的隔离内核和 fail-closed 门禁,
@@ -28,10 +28,11 @@
| 编号 | 结论 | 首期状态 |
| ----- | --------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| D-001 | 一个共享 Plugin Runtime 控制面;每个运行中的 plugin installation 独占一个 nsjail 子进程;只有 digest 相同且已验证的代码 artifact 可以只读共享 | `FOUNDATION IMPLEMENTED — Linux/egress, crash recovery and disk-quota gates pending` |
| D-001 | 一个共享 Plugin Runtime 控制面;每个运行中的 plugin installation 独占一个 nsjail 子进程;只有 digest 相同且已验证的代码 artifact 可以只读共享 | `FOUNDATION VERIFIED — egress, total disk-quota and restart-storm gates pending` |
| D-002 | 一个共享 Box RuntimeCloud 固定使用 nsjail;符合套餐的 Workspace 最多一个持久 `global` 逻辑 sandbox,普通执行按需启动 nsjail 进程 | `IMPLEMENTED FAIL-CLOSED — hard filesystem quota provider pending` |
| D-003 | SaaS 业务数据使用 PostgreSQL shared schema、应用层作用域和 RLS 双重隔离;pgvector 使用同一 PostgreSQL,作为 SaaS 默认向量后端 | `PARTIALLY IMPLEMENTED — transaction/outbox/deployment gates remain` |
| D-004 | stdio MCP 与 Box availability 解耦;Cloud v2 首期强制关闭 stdio MCP,避免为每个 Workspace 创建额外的 `mcp-shared` persistent sandbox | `IMPLEMENTED` |
| D-005 | 目录启动使用事务一致的全量快照,运行时按事件涉及的 Workspace 拉取增量;每个 Core replica 独立消费事件,共享 PostgreSQL 投影和 inbox | `IMPLEMENTED — production fault injection pending` |
Workspace 的具体创建、释放、数据导出和单 Workspace 恢复机制不在本轮决定;本文只保证这些后续能力不会改变稳定的
`workspace_uuid`,也不会要求重建租户专属部署。
@@ -76,13 +77,15 @@ flowchart LR
M1 是 M0 的透明扩容,M2 是相同架构下的资源等级;两者都不是新的 LangBot 实例、Cell 或 CloudInstance。
外部 API 只认识稳定的 `instance_uuid``workspace_uuid`,不认识 replica、worker、pool 或 shard。
Plugin Runtime 与 Core 在 M0 是否共用 Pod 仍可按发布和故障域决定;即使共用 Pod,也必须使用独立容器和 security context
Core 不能继承 Plugin Runtime 所需的 nsjail/cgroup 权限。Box Runtime 同样使用独立进程身份和安全配置,
Plugin Runtime 与 Core 在 M0 使用独立容器和 security contextCore 不能继承 Plugin Runtime 所需的
nsjail/cgroup 权限。当前 Runtime 在进程生命周期内绑定首次认证的 `runtime_id`,因此 M0 必须把 Core 与
Plugin Runtime 放在同一 rollout/restart unit 中协调重启;在实现受认证 takeover 或 owner lease/fencing 前,
不能单独滚动 Core 并让它接管仍存活的 Runtime。Box Runtime 同样使用独立进程身份和安全配置,
不与 Plugin Runtime 合并成一个高权限进程。
## 3. D-001Plugin Runtime 多租户控制面
状态:`FOUNDATION IMPLEMENTED — Cloud deployment and worker-recovery gates pending`
状态:`FOUNDATION VERIFIED — Cloud egress, total disk-quota and restart-storm gates pending`
### 3.1 已实现的基础
@@ -116,8 +119,8 @@ Core 不能继承 Plugin Runtime 所需的 nsjail/cgroup 权限。Box Runtime
- 安装并启用插件后,Supervisor 在自己的 Runtime 容器内直接启动一个 nsjail 子进程;
不再为每个插件创建 nested container、Pod、sidecar 或租户级 Runtime service。
- desired semantics 要求 enabled installation 保持 resident,不做 idle eviction;停用、删除、revision/generation 变化或 entitlement 撤销时停止并按需重建。
当前实现只会在 Runtime 重连或 Core apply/reconcile 时恢复意外退出的 worker尚缺 completion callback、有界 backoff 和跨租户重启风暴抑制;
这属于 Cloud 激活门禁,不能把 desired semantics 描述成已经具备即时自愈
当前 Supervisor 已通过 completion callback 和有界指数 backoff 恢复意外退出的 worker尚缺 jitter、全局重启并发上限和 Runtime 级 circuit breaker
因而系统性依赖或宿主故障下的跨租户重启风暴抑制仍是 Cloud 激活门禁
- 子进程使用一次性 registration capability 向 Supervisor 注册;capability 由可信 desired state 派生并绑定完整 installation tuple
不是插件直接建立 Core Host connection,也不能只绑定 author/name/path。Supervisor/Core 据此注入 tenant context
丢弃插件 payload 中自带的 scope 字段。
@@ -473,9 +476,9 @@ mcp:
- 直接调用 API、重放旧配置和启动 bootstrap 都失败关闭,且不会产生 `mcp-shared` session、nsjail 进程或额外配额占用。
- OSS 默认行为保持兼容;HTTP/SSE MCP 正常工作。
## 7. 项决策之间的关系
## 7. 项决策之间的关系
项决策共同遵循:
项决策共同遵循:
> 多租户共享可信控制面、连接池、只读 artifact 和基础容量;租户独占不可信执行进程、sandbox、secret、可写文件和数据作用域。
@@ -485,6 +488,7 @@ mcp:
- PostgreSQL 和 pgvector 共享数据库组件,但使用显式 tenant key、应用层 scope 和 RLS 防止共享存储变成共享权限。
- 订阅管理只在闭源 Control Plane 维护套餐与计费规则,并向开源 Core 投影签名/版本化 entitlement
Core/Runtime 执行通用 capability 和数值限额,不复制套餐名称或计费逻辑。
- 目录同步只在启动或恢复时读取全量快照;常态变更按 Workspace 聚合为签名增量,新增租户不会使每次目录事件退化为全实例重投影。
## 8. 当前不做分布式时仍保留的能力
@@ -497,6 +501,9 @@ mcp:
6. schema migration、任务扫描、监控聚合和运维接口不能假设永远只有一个 Core 进程。
7. 外部 API 不暴露 replica、worker 或 shard 标识;未来扩容不改变 Workspace URL、UUID 或客户端协议。
8. 只有出现容量、可用性、地域或合规需求时才增加 replica/shard;预留协议不等于现在部署额外组件。
9. 每个 Core replica 保存自己的事件消费 cursor,因为 entitlement cache 是进程本地状态;PostgreSQL 中的 projection high-water mark
、snapshot coverage 和 inbox 仍由所有 replica 共享,用于幂等投影和冲突检测。事件页携带签名 high-water,副本追平前不能续期
ready;不能让一个 replica 的共享 cursor 使其他 replica 跳过本地 cache 刷新。
## 9. 本轮明确不做的事情
+199 -204
View File
@@ -1,257 +1,252 @@
# Multi-tenant isolation kernel verification report
# Cloud v2 multi-tenant verification report
Date: 2026-07-20
Date: 2026-07-24
Status: `CORE ISOLATION FOUNDATION VERIFIED — SAAS ACTIVATION REMAINS DISABLED`
Status: `FOUNDATION AND CONTROL PLANE VERIFIED — PRODUCTION ACTIVATION REMAINS GATED`
This report records the final implementation and verification evidence for the
shared Workspace isolation foundation. It does not claim that the closed SaaS
Control Plane, billing system, or greenfield Cloud v2 deployment is
production-ready.
This report records the implementation and verification evidence for one
logical LangBot instance serving multiple Workspace tenants. It covers the
open-source Core, the shared Plugin/Box runtimes, the closed Space adapter, and
the Space Cloud v2 modular-monolith control plane. It does not claim that the
production Cloud deployment may enable `CLOUD_V2_ENABLED` yet.
## Repository refs and scope
- LangBot branch: `feat/multi-tenants`
- LangBot implementation commit:
`90a977488212d3f3f50fb92d94e3fcae60f27eab`
- LangBot base: `origin/master` at
`6baeb032a7f76c65337b51c7b58593de4687a61c`
- Plugin SDK branch: `feat/multi-tenants`
- Plugin SDK commit and LangBot dependency pin:
`95a1805af2038c745de2c018a00db1305089a32e`
- No tracked `langbot-space` change was made. Cloud v2 does not extend or
preserve the legacy per-account instance/pod deployment topology.
- LangBot Core: branch `feat/multi-tenants`
- langbot-plugin-sdk: branch `feat/multi-tenants`, commit
`e7d946af4a6b1494fbe74627c1815ace19ac8991`
- langbot-space: branch `feat/cloud-v2-control-plane`
- Closed Core adapter: `langbot-space/cloud-adapter`
- SDK protocol/package version: `0.4.18`
The SDK protocol is versioned as 0.4.15 and is intentionally consumed from the
exact pushed Git commit above. Publishing the release and replacing the Git
pin with a registry pin remain merge-to-master release work; no registry
release was performed here.
Core pins the exact pushed SDK commit above. Cloud v2 is a greenfield
multi-Workspace deployment and does not provision one Pod, database, queue, or
Runtime per tenant. Legacy Space Pods remain a compatibility surface only.
## Implemented boundaries
## Implemented product and security boundaries
### Workspace, identity, and authorization
### Workspace and identity
- Community bootstraps exactly one local Workspace per LangBot instance while
allowing multiple Accounts, memberships, invitations, and role-based
permissions inside it. A mutable edition setting cannot activate SaaS
multi-Workspace routing.
- Each Account registration resolves or creates its local singleton Workspace.
Invitation acceptance is one-time, email-bound, and survives the signed-out
login transition without exposing the invitation secret in URLs after the
initial fragment capture.
- The Account-token Workspace discovery route is intentionally separate from
Workspace-authorized routes. `X-Workspace-Id` becomes authority only after
membership resolution; API keys and public resources derive Workspace from
their trusted owner.
- Ordinary monitoring uses `resource.view`. Export remains `data.export`,
privileged runtime/system audit remains `audit.view`, and frontend controls
follow the same permission split.
- SaaS has one stable `instance_uuid`; `workspace_uuid` is the tenant key.
- Registering an Account creates its personal Workspace and owner Membership in
the same PostgreSQL transaction.
- A Workspace may contain multiple users with owner, admin, and member roles.
Invitation acceptance is token-hash based, email-bound, single-use,
concurrency-safe, and member-limit checked while locked.
- Community remains exactly one local Workspace while supporting multiple
Accounts and fixed RBAC roles. Installing the closed adapter is required to
inject the SaaS policy; configuration alone cannot activate it.
- Disabled or deleted Accounts cannot use password/code/OAuth login, sessions,
refresh/access tokens, or personal access tokens.
### Persistence and PostgreSQL
### Closed Space control plane
- PostgreSQL uses one shared business schema with application scope plus exact
`ENABLE` and `FORCE ROW LEVEL SECURITY` policies. Tenant UoWs keep `SET LOCAL`
and business SQL on one owned root transaction and one task.
- The public tenant Session accepts only structured SQLAlchemy query/DML trees.
Raw/textual SQL, textual modifiers, custom AST/compiler nodes, unapproved
functions/operators/types/casts, hidden dialect values, foreign binds,
loader/execution options, Session event callbacks, ORM SQL-expression
attributes, transaction control, and live-result escape fail closed and make
the root transaction rollback-only.
- The SQL guard is a trusted-Core misuse boundary, not an in-process Python
sandbox. Mapped metadata and registered compilers are trusted boot-time code;
plugins remain out of process. SQLAlchemy dialect-private container fields
are pinned and covered by regression tests before dependency upgrades.
- Legacy local RAG restore remains structured and tenant-fenced while accepting
SQLite string-valued timestamps and both historical PostgreSQL `TEXT` and
fresh-schema `JSON` settings columns.
- The one-shot release migrator owns DDL and grants a distinct runtime role only
the required business-table DML, `alembic_version` read access, sequence
access, database `CONNECT`, and schema `USAGE`. Runtime startup reruns the
role, ACL, schema, session, extension, routine, parameter, and RLS audit.
- pgvector is stored in the same business database with Workspace-scoped keys,
checked dimensions, release-created partial ANN indexes, and RLS.
- `CLOUD_V2_ENABLED` defaults to false. Invalid or incomplete configuration
fails startup, and disabled endpoints return 503.
- Space owns Workspace/Membership/Invitation, versioned plans, subscriptions,
entitlements, usage events, directory outbox, and signed instance manifests.
- Free and Pro are Workspace plans. Pro projects one managed global sandbox;
Free projects none. Both force stdio MCP off in the first release.
- Subscription changes use Workspace advisory locking, expected entitlement
revision, one-live-order uniqueness, idempotent usage ingestion, period-end
downgrade, renewal, and lazy expiry settlement.
- Directory snapshot and per-Workspace delta reads use repeatable-read,
read-only transactions with a high-water cursor. Core stores a replica-local
consumer cursor and shared PostgreSQL projection state, snapshot coverage,
and inbox rows.
- The `/cloud` page selects a Workspace and shows its independent subscription,
entitlement, limits, and usage. When Cloud v2 is disabled or the backend does
not expose the feature field, the complete legacy Welcome/Pod UI is used.
### Shared Plugin Runtime and Box
### PostgreSQL and pgvector
- One instance-scoped Plugin Runtime supervisor serves multiple Workspaces.
Every enabled installation still owns one nsjail worker with an immutable
instance/Workspace/generation/installation/revision/artifact binding.
- Verified same-digest code and dependency environments may be mounted
read-only and reused. Plugin processes and writable installation state are
never merged, including for the same plugin and version.
- Runtime limits come only from instance configuration with native environment
overrides. Plugin manifests cannot raise limits. Shared mode rejects legacy
global plugin directories and legacy lifecycle control paths.
- One shared Box control plane admits at most one persistent logical `global`
sandbox for an entitled Workspace. Admission is entitlement-gated,
generation-fenced, shared-volume-challenged, quota-aware, and fixed to
nsjail. Plain nsjail correctly fails Cloud readiness without a hard quota
provider.
- stdio MCP has an independent gate and is forced off by Cloud bootstrap even
when Box is available.
- SaaS business data uses one PostgreSQL shared schema with application scope
plus forced RLS. Cloud directory writes are separated from local tenant
writes.
- Projected Account and Membership revisions are monotonic. Tombstones remove
memberships, and stale revisions cannot resurrect them.
- pgvector shares the business PostgreSQL database and remains
Workspace-scoped.
- Space runs versioned SQL migrations before application seeds. A fresh
database, a partial pre-migration database, and an existing-baseline path
converge without startup-time `AutoMigrate`.
### Shared Plugin Runtime
- One instance-scoped trusted Supervisor serves multiple Workspaces.
- Every enabled installation has its own nsjail process bound to
`(instance, workspace, execution generation, installation, runtime revision,
artifact digest)`.
- Verified same-digest code and dependency files are mounted read-only and may
be shared; home, tmp, data, process namespace, registration capability, and
cgroup are private.
- Instance configuration owns CPU, memory, PID, open-file, and file-size
limits. Plugin manifests cannot increase them.
- Memory includes swap: nsjail receives `memory.max` and `swap.max=0`.
- Unexpected worker exit is recovered by a completion callback with bounded
per-installation exponential backoff. Remove, reconcile, Runtime shutdown,
and container SIGTERM perform a graceful-to-SIGKILL bounded reap.
### Shared Box Runtime and MCP
- One shared Box control plane serves Workspace-bound logical sessions.
- Cloud grants allow at most one persistent `global` session for an entitled
Workspace and no managed processes in the first release.
- Cloud is fixed to nsjail and network-off. Core and Box prove the shared
durable Workspace mount with an authenticated marker challenge.
- stdio MCP is independently gated and forced off for Cloud v2.
- The current nsjail Box backend does not provide hard byte/inode quotas, so
Cloud readiness correctly fails closed instead of silently using a soft
directory scan.
## Automated verification
### LangBot backend and static checks
### LangBot Core
```text
uv run pytest -q --tb=short
2527 passed, 32 skipped, 177 warnings in 73.37s
uv run --no-sync pytest -q
2590 passed, 32 skipped, 177 warnings
focused tenant UoW and legacy RAG compatibility suite
120 passed, 10 warnings in 1.30s
uv run ruff check .
passed
changed Python format check
15 files already formatted
real PostgreSQL migration, pgvector, and release-migrator suites
21 passed, 11 warnings
uv run --no-sync ruff check .
uv lock --check
git diff --check
passed
```
The full suite includes startup E2E, the fresh SQLite registration/Workspace
journey, API authorization, storage, RAG, Plugin, MCP, and 18 Docker-backed Box
integration cases.
The full suite ran without the closed adapter installed, proving the open-source
single-Workspace/multi-user path remains standalone. Focused closed-adapter,
directory projection, runtime connector, Box cleanup, and configuration suites
also passed with the adapter installed.
### Real PostgreSQL 16 and pgvector
### Plugin SDK and real Linux runtime
```text
test_migrations_postgres.py
test_pgvector_postgres.py
test_release_migration_postgres.py
21 passed, 11 warnings in 11.61s
SDK full suite
1226 passed, 22 existing warnings
Ruff check and format check
git diff --check
passed
```
The real-database suite covers fresh and upgraded RLS schemas, two-Workspace
isolation, pgvector CRUD and ANN indexes, least-privilege runtime bootstrap,
release locking, historical/fresh RAG settings column compatibility, and
transaction-scope escape rejection. Negative tests inject and reject role
membership, grant options, persistent GUCs, extra schemas and relations,
column ACLs, explicit routine and parameter ACLs, runtime-owned and
`SECURITY DEFINER` routines, extra/runtime-owned extensions, and foreign data
wrappers, servers, and user mappings.
A privileged Linux test container with host cgroup namespace ran one shared
Runtime and two Workspace installations:
Alembic has exactly one head: `0013_tenant_pgvector`.
- both workers referenced the same artifact inode;
- home, tmp, and data inodes were distinct;
- each plugin saw only PID 1 in its private PID namespace;
- a tampered binding and an unknown installation were rejected;
- the control token was absent from worker environments;
- cgroups were distinct with `memory.max=134217728`, `memory.swap.max=0`,
`pids.max=32`, and `cpu.max=500000 1000000`;
- touching 256 MiB exited with code 137 without swap growth;
- the 32nd fork failed with `EAGAIN`;
- reconcile and container SIGTERM removed the worker cgroups.
### Plugin SDK and Box Runtime
The same run started the Runtime from a non-root working directory, covering
absolute nsjail mount-source normalization.
### Space backend, adapter, and frontend
```text
Plugin SDK full suite
1160 passed, 18 warnings
Plugin SDK Ruff and action consistency checks
go test ./...
passed
Docker-backed Box integration suite
18 passed, 33 warnings
fresh PostgreSQL app startup, partial-baseline migration,
Cloud v2 migration rerun, and control-plane integration
passed
closed adapter pytest and Ruff
passed
pnpm exec tsc --noEmit
targeted ESLint for changed Cloud files
pnpm build
passed; 47 routes generated
```
The LangBot environment resolves `langbot-plugin==0.4.15` from the same exact
SDK commit recorded above. The SDK local ref and remote
`refs/heads/feat/multi-tenants` were independently verified equal.
The PostgreSQL checks started from an empty database and verified all 32
registered migrations in order, Cloud v2 Free/Pro seeds, legacy plan seeds,
Cloud columns/indexes, and repeatable reruns.
### Frontend
## Cross-service and browser E2E
```text
pnpm lint
0 errors, 34 existing warnings
### Signed Space-to-Core directory projection
VITE_API_BASE_URL=/ pnpm build
passed; 3168 modules transformed
Using an isolated Space PostgreSQL database and a migrated Core PostgreSQL
database:
pnpm exec playwright test --project=chromium
43 passed in 22.2s
```
1. Space issued a signed manifest for the fixed LangBot instance.
2. Space returned a directory snapshot at cursor 5 containing two Workspaces,
two projected Accounts, and owner/admin memberships.
3. Core verified the signature, instance, release, validity window, and
capability before injecting the Cloud Workspace policy.
4. Core stored both Workspaces, all active Memberships, both projected
Accounts, snapshot coverage, inbox entries, and cursor 5.
5. Account-field-only projection revisions and Workspace directory revisions
remained independent, and cross-Workspace Account conflicts failed closed.
The frontend suite includes Viewer monitoring visibility, privileged control
hiding, stable terminal invitation states, email-mismatch login handoff, and a
temporary invitation-accept failure that retains the authenticated session and
succeeds on retry.
### Real browser Cloud v2 flow
### Packaging and deployment configuration
A real local browser operated the Space frontend and backend:
- `uv lock --check` resolved 277 packages and passed.
- Changed TypeScript/TSX files passed Prettier verification.
- `docker compose -f docker/docker-compose.yaml config --quiet` passed with the
existing warning that the top-level `version` field is obsolete.
- The staged high-signal credential scan found no private key or provider
token.
1. An owner logged in and saw the automatically created personal Workspace.
2. A second Account invited the owner as admin to another Workspace.
3. The selector showed both Workspaces and switched between them without stale
subscription or usage state crossing the selection.
4. Each Workspace showed its own Free entitlement and limits.
5. Pro checkout opened for the selected Workspace. With no configured payment
channels, confirmation was visibly disabled and no order was created.
6. No browser errors occurred; only the pre-existing i18next warning for the
short `en` locale was observed.
## Real browser E2E
The legacy feature-flag branch was then covered by API/static checks and the
production build: false or missing `cloud_v2_enabled` renders the original
Welcome/Pod client; a failed web-config request renders an explicit retry
instead of guessing a deployment mode.
A production frontend and backend were run against an isolated temporary
SQLite data directory and operated through a real local Chrome session:
### Deliberate Core startup failure
1. The first Account registered and automatically received the default
singleton Workspace as owner.
2. Skipping the setup wizard reached `/home/monitoring`; reload and a new tab
recovered the authenticated Account, membership, and Workspace selection.
3. The owner invited a Viewer. After signing out, the second Account registered
through the invitation and joined the same Workspace.
4. Owner and Admin could read overview/export/audit/API-key surfaces and reach
resource-write validation. Runtime debug mutation remained disabled.
5. Developer retained ordinary resource management and monitoring, while
export, audit, and API-key management returned 403.
6. Operator and Viewer retained ordinary monitoring reads while resource
writes, export, audit, API-key management, and runtime mutation returned 403.
7. The UI matched the server matrix: Add/API Key/Export controls appeared only
with their permissions, while ordinary monitoring refresh remained visible
to read-only members.
8. A second Workspace creation attempt returned 403 `edition_limit`, proving
Community remains single-Workspace and multi-user.
9. Used, revoked, expired, and email-mismatched invitation paths produced
stable visible states. The mismatch survived login without a contradictory
generic success toast.
Core completed signed manifest verification and directory projection, then
stopped at the Box readiness gate because the current nsjail backend cannot
prove hard Workspace byte/inode quota enforcement. Connector shutdown and
reconnect tasks were cleanly reaped; no event-loop or never-awaited coroutine
warning remained.
The server was stopped, port 15321 was verified free, and temporary artifacts
were moved to the system Trash for recoverable cleanup. No invitation secret,
password, token, or provider credential is recorded in this report.
This is a successful fail-closed acceptance result, not a passing production
Cloud boot.
## Deliberately incomplete SaaS activation gates
## Remaining production activation gates
Multi-Workspace activation remains unavailable in the open-source bootstrap.
Cloud v2 must stay disabled until all of the following are implemented and
independently verified:
Cloud v2 must remain disabled until these gates are closed:
- the closed Control Plane owns the Account/Workspace directory, memberships,
invitations, monotonic execution generations, entitlements, usage,
subscriptions, and billing;
- ordinary tenant writes hold a generation-aware fence through commit, and a
generation-stamped outbox or equivalent atomic publish fence protects
external side effects;
- durable object references survive generation cutovers without stranding
files, images, plugin artifacts, or knowledge-base content;
- every tenant-configurable outbound path has complete SSRF/egress policy, and
Plugin Runtime has production Linux namespace, cgroup v2, and network
isolation evidence;
- an unexpectedly exited enabled plugin worker is restored by a completion
callback with bounded backoff and cross-tenant restart-storm isolation;
- plugin installation data has an operator-owned hard disk quota, not a
directory-scan approximation;
- Box supplies and proves hard byte/inode quotas for Workspace, Skill, root,
home, and temporary storage, and the production shared volume passes the
authenticated marker challenge;
- production PostgreSQL uses a dedicated endpoint/cluster, or a tested
HBA/proxy policy proves the runtime credential can connect only to the target
business database;
- the release migrator is deployed as a one-shot Job with credential issuance,
backup, rollback, failure/retry, and orchestration procedures;
- OAuth exchange, directory projection, leases, snapshots, event replay, and
outbox state use durable shared stores suitable for horizontally scaled
replicas;
- Workspace release, export, deletion, restore, and disaster-recovery semantics
are decided and implemented; and
- production Runtime restart, crash, partition, revocation, and fault-injection
acceptance is completed against the closed Control Plane and greenfield
Cloud v2 deployment.
- Box provides and proves hard byte and inode quotas for Workspace, Skill,
root, home, and tmp storage.
- Plugin installation writable data receives an operator-owned hard total
disk quota.
- Plugin and future networked Box workloads have tenant-safe egress/SSRF
policy.
- Plugin Runtime adds jitter, global restart concurrency limiting, and a
Runtime-level circuit breaker, then passes systemic-failure injection.
- M0 rolls Core and Plugin Runtime together until authenticated Runtime takeover
or an owner lease/fencing protocol exists.
- Account registration moves New API provisioning to an in-process,
transactional outbox reconciler keyed by Space Account UUID.
- EPay and Stripe fulfillment bind provider identity, amount, currency,
channel/session, payment status, and expiry to the locked order transaction;
provider transaction IDs are unique and replay-safe.
- Payment-session creation and stale `processing` orders gain in-process
reconciliation.
- Cloud v2 subscription service periods are stored immutably and included in
recognized-revenue reporting.
- Production migration Job, backup/rollback, PostgreSQL credential/network
boundaries, horizontal-replica fault injection, Workspace release/export,
deletion, and restore semantics are completed.
The following product work is also deliberately deferred: WebUI configuration
for tenant-provided remote E2B, multi-replica lease storage and sharding rules,
artifact signing/garbage collection, custom roles, SSO, and SCIM.
These are explicit release gates, not hidden fallbacks. The current branches
are a verified isolation foundation on which the closed SaaS system can be
built; they are not a production Cloud activation.
These gates intentionally add no tenant-specific service. They are implemented
inside the existing Space, Core, Plugin Runtime, Box Runtime, and PostgreSQL
components to preserve the architecture goal: near-zero static cost for a new
Workspace.
@@ -148,7 +148,8 @@ M1 是 M0 的透明扩容,M2 是相同架构下的资源等级。外部 API
### 4.4 组件边界
- Core、Plugin Runtime 和 Box Runtime 可以由发布与故障域决定是否共用 Pod,但必须保持独立进程身份、容器和 security context。
- Core、Plugin Runtime 和 Box Runtime 必须保持独立进程身份、容器和 security context。M0 中 Core 与 Plugin Runtime
需要处于同一 rollout/restart unit;在实现受认证 takeover 或 owner lease/fencing 前,Core 不能单独重启后接管仍存活的 Runtime。
- Core 不能继承 nsjail、cgroup 或 mount namespace 所需的高权限。
- Plugin Runtime 与 Box Runtime 不合并为一个高权限进程。
- MVP 不新增 Runtime 专用数据库、Box 专用数据库、Kafka、Redis、租户级 scheduler 或 artifact service。
@@ -321,7 +322,9 @@ delegated issuers and keyset revision
### 7.2 DirectoryEvent 与目录新鲜度
Control Plane 通过 transactional outbox 发布 Account、Workspace 和 Membership 的版本化事件。
Core 使用 inbox 按 `event_id` 去重,以 aggregate revision 拒绝旧写,并追踪连续应用水位。
Core 使用 inbox 按 `event_id` 去重,以 aggregate revision 拒绝旧写,并追踪连续应用水位。启动时读取一个 PostgreSQL
`REPEATABLE READ` 事务内生成的签名全量 snapshot;运行时先消费携带当前 high-water 的签名事件页,再只请求该页涉及的 Workspace 签名增量。
增量响应不携带新的事件 cursor,因此即使其内容已包含并发提交的后续 revision,也不能跳过尚未消费的事件。
要求:
@@ -329,6 +332,10 @@ Core 使用 inbox 按 `event_id` 去重,以 aggregate revision 拒绝旧写,
- 重复、乱序、延迟、断流和全量 replay 都安全。
- 删除使用 tombstone。
- 新实例先导入带 high watermark 的 snapshot,再消费增量。
- 常态目录更新成本与本页发生变化的 Workspace 数量相关,不得为每个 `directory.changed` 重新读取和投影全部 Workspace。
- 每个 Core replica 独立保存进程内消费 cursor,以确保各自的 entitlement cache 都看到事件;共享 PostgreSQL 保存投影
high-water mark、全量 snapshot coverage 和 inbox。同一事件被多个 replica 消费时,第二个 replica 验证已有 receipt
snapshot coverage 内缺少的 receipt 可以补写,coverage 之外缺失则失败关闭。只有本地 cursor 追平签名 high-water 后才续期 ready。
- projection 未就绪或落后于授权 lease 要求时,交互与自动化请求按策略失败关闭。
- SaaS pending Invitation、email 和 token hash 不进入 Core 投影。
@@ -630,8 +637,8 @@ installation 总磁盘配额需要可原子拒绝写入的 quota provider,不
- Runtime 重连执行实例范围 full reconciliation,清理 stale worker 并恢复 enabled installation。
- dependency preparation 失败记录在对应 installation,不启动半就绪 worker,也不阻塞其他 installation。
- desired semantics 要求 enabled installation 常驻,不做 idle eviction;是否按负载回收以后再决定。
- 当前 Supervisor 可在 Runtime 重连或 Core apply/reconcile 时恢复 desired state,但尚未为意外退出的 worker
实现带有界 backoff 的 completion callback;这项可靠性缺口在 Cloud 激活前必须补齐并验证
- 当前 Supervisor 已在意外退出时通过 completion callback 和有界指数 backoff 恢复 enabled worker
Cloud 激活前仍需加入 jitter、全局重启并发上限和 Runtime 级 circuit breaker,并验证系统性故障不会形成跨租户重启风暴
真实 Linux/nsjail/cgroup 与受控 egress 的 Cloud 部署验证尚未完成,是生产激活门禁。
@@ -813,7 +820,7 @@ SaaS
2. 普通业务写入贯穿 commit 的 generation-aware fence,以及与外部副作用同事务的 business outbox。
3. generation cutover 后稳定的 durable object identity 或原子对象引用迁移。
4. 所有 tenant-configurable outbound URL 的 SSRF 防护与 tenant-safe egressPlugin Runtime 还需在真实 Linux/nsjail/cgroup v2 环境验证 namespace、资源限制和文件隔离。
5. Plugin Runtime 意外退出的 enabled worker 实现 completion callback、有界 backoff 和自动恢复,并验证不会形成跨租户重启风暴
5. Plugin Runtime 已实现意外退出 worker completion callback、有界 backoff 和自动恢复;Cloud 激活前增加全局重启风暴抑制并完成故障注入验证
6. Plugin installation data 的 production hard disk quota provider,能够在写入边界原子拒绝超额,不能以目录扫描代替。
7. Box Runtime 的 production hard quota provider,包括 Workspace、Skill、root/tmp/home 的 byte 与 inode quota;真实部署还必须在启动和重连时通过共享卷 marker challenge。
8. PostgreSQL runtime credential 的专用 endpoint 或 HBA/proxy 跨 database 隔离证明、生产 migration/rollback 流程,以及 legacy pgvector migration 失败后精确恢复 RLS/FORCE 并可安全重试的集成证据。