mirror of
https://github.com/langbot-app/LangBot.git
synced 2026-07-25 05:46:13 +00:00
feat(tenancy): connect cloud workspace control plane
This commit is contained in:
@@ -8,7 +8,7 @@ verification gates. Exact commands and observed results are recorded in the
|
||||
|
||||
- [x] LangBot uses branch feat/multi-tenants.
|
||||
- [x] langbot-plugin-sdk uses branch feat/multi-tenants.
|
||||
- [x] langbot-space has no changes made by this implementation; Cloud v2 does not extend the legacy Space deployment scheme.
|
||||
- [x] langbot-space implements the greenfield Cloud v2 modular-monolith control plane without extending the legacy per-account Pod topology; the old Pod UI remains available when Cloud v2 is disabled and only retained Pods appear in the v2 view.
|
||||
- [x] Unrelated untracked files in either repository remain untouched.
|
||||
- [x] Open-source startup cannot enable SaaS multi-workspace through edition flags or unsigned configuration.
|
||||
|
||||
@@ -20,18 +20,22 @@ deployment. The feature branch delivers the Core isolation kernel, not the
|
||||
closed SaaS product or a production Cloud v2 deployment. Checked implementation
|
||||
items later in this document do not supersede these gates.
|
||||
|
||||
- [ ] The closed Control Plane owns the global Account, Workspace, Membership, and Invitation directory.
|
||||
- [x] The closed Control Plane owns the global Account, Workspace, Membership, and Invitation directory.
|
||||
- [ ] The closed Control Plane execution-ownership module issues monotonic generations and owner leases for projected Workspaces.
|
||||
- [ ] Core verifies a signed `InstanceManifest` before the closed bootstrap can inject `CloudWorkspacePolicy`.
|
||||
- [x] Core verifies a signed `InstanceManifest` before the closed bootstrap can inject `CloudWorkspacePolicy`.
|
||||
- [ ] Tenant database writes hold a generation-aware shared transaction fence through commit, while execution-owner cutovers take the exclusive fence.
|
||||
- [ ] Business writes and non-transactional side effects use a generation-stamped outbox or equivalent publish fence.
|
||||
- [ ] Durable object references survive an execution-generation change through stable published keys or an explicitly atomic key/reference migration.
|
||||
- [ ] The SaaS runtime pools enforce tenant-safe egress and SSRF controls for Webhooks, providers, MCP servers, and every tenant-configurable outbound URL.
|
||||
- [ ] Entitlement checks, usage aggregation, subscription lifecycle, and billing are implemented in the closed Control Plane.
|
||||
- [ ] Entitlement checks, usage aggregation, and subscription lifecycle are implemented in the closed Control Plane; production activation still requires provider callback amount/currency/session/expiry binding inside the locked fulfillment transaction.
|
||||
- [ ] Account registration persists a `new_api.provision_account` outbox item with the Account and personal Workspace, and an in-process reconciler provisions New API idempotently after commit.
|
||||
- [ ] EPay and Stripe callbacks bind provider identity, amount, currency, channel/session, payment status, and expiry to the locked order before entitlement fulfillment.
|
||||
- [ ] OAuth state and directory projection use an atomic shared store suitable for horizontally scaled SaaS services.
|
||||
- [ ] A greenfield Cloud v2 deployment is designed and validated independently of the legacy Space deployment scheme.
|
||||
- [ ] The Plugin Runtime deployment provides delegated cgroup v2 and tenant-safe egress; the shared profile refuses to run without hard CPU, memory, and PID limits.
|
||||
- [ ] The Plugin Runtime Supervisor automatically restores an unexpectedly exited enabled worker with bounded backoff and cannot create a cross-tenant restart storm.
|
||||
- [ ] The Plugin Runtime shared profile refuses to run without delegated cgroup v2 CPU, memory-plus-swap, and PID limits, all verified in a real Linux container; production tenant-safe egress remains incomplete.
|
||||
- [x] The Plugin Runtime Supervisor automatically restores an unexpectedly exited enabled worker with bounded per-installation backoff.
|
||||
- [ ] Jitter, a global restart concurrency limit, and a Runtime-level circuit breaker prevent a systemic failure from creating a cross-tenant restart storm.
|
||||
- [ ] Until authenticated Runtime takeover or an owner lease/fence exists, the M0 deployment rolls Core and Plugin Runtime together and forbids an independent Core-only rollout.
|
||||
- [ ] Plugin installation data has an operator-owned hard disk quota provider that atomically rejects writes over the limit; directory scans are not accepted as enforcement.
|
||||
- [ ] The Box deployment provides an operator-owned quota provider that proves hard byte and inode limits for Workspace, Skill, root, tmp, and home storage.
|
||||
- [ ] Core and Box Runtime mount the same durable volume and pass the authenticated marker challenge during startup and reconnect.
|
||||
@@ -281,7 +285,7 @@ Each row type must have a non-null Workspace UUID, scoped indexes, scoped unique
|
||||
## 10. Completion evidence
|
||||
|
||||
- [x] LangBot and SDK branch refs are recorded in the verification report.
|
||||
- [x] Space git diff is empty relative to the pre-work snapshot.
|
||||
- [x] Space contains the closed adapter package and Cloud v2 control plane, billing, migration, and Workspace UI changes; unrelated pre-existing files remain unstaged.
|
||||
- [x] Migration output is captured for SQLite and PostgreSQL.
|
||||
- [x] Test commands and results are recorded.
|
||||
- [x] Browser E2E actions and observed results are recorded.
|
||||
|
||||
@@ -13,7 +13,7 @@ This log records implementation choices made while delivering the Workspace arch
|
||||
- Decision: Community builds create exactly one Workspace per LangBot instance and allow multiple Accounts through invitations.
|
||||
- Reason: This preserves a simple self-hosted deployment while making authorization and ownership explicit. Creating a second Workspace is an edition error, not a hidden fallback.
|
||||
- SaaS boundary: Multi-Workspace directory, execution ownership, entitlement, and billing are the responsibility of a separate closed SaaS Control Plane. Core consumes a validated projection and remains the final isolation and authorization enforcement point; it does not become the SaaS system of record or billing engine.
|
||||
- Deployment boundary: Cloud v2 is a greenfield deployment design. The previous per-account instance/pod scheme is not migrated, preserved for compatibility, or extended by this implementation. Existing OAuth, marketplace, and payment concepts may be reused only through explicit adapters where they still fit; `langbot-space` itself is not changed.
|
||||
- Deployment boundary: Cloud v2 is a greenfield deployment design. The previous per-account instance/pod scheme is not migrated or extended and remains available only for existing subscriptions. Existing OAuth, marketplace, and payment rails are reused through explicit adapters where they still fit; new Workspace, subscription, entitlement, directory, and usage modules live alongside the legacy Pod flow in `langbot-space`.
|
||||
|
||||
### Workspace selection is trusted only after authentication
|
||||
|
||||
@@ -41,7 +41,7 @@ This log records implementation choices made while delivering the Workspace arch
|
||||
- Decision: One instance-scoped Plugin Runtime control plane serves all Workspaces in the logical LangBot instance. Each running plugin installation has its own nsjail worker with an immutable binding containing `instance_uuid`, `workspace_uuid`, `execution_generation` (stored as the compatibility field `placement_generation` until the schema rename), `installation_uuid`, `runtime_revision`, and verified artifact digest; enabled-resident is the desired semantic. A worker never routes actions for another Workspace or installation, and plugin-supplied scope fields are stripped.
|
||||
- Isolation: Plugin code is mounted read-only. Home, tmp, and data paths are installation-scoped; process, file-descriptor, file-size, CPU, memory, and PID limits come only from `data/config.yaml` (including native environment overrides), never from a plugin manifest. Cloud requires nsjail and delegated cgroup v2 hard limits or fails closed.
|
||||
- Cost boundary: Identical verified package bytes share one digest-addressed code cache. A dependency environment is keyed by the artifact and requirements digests, Python ABI, Runtime version, and installer schema, then atomically published read-only for reuse. Installations and processes are not merged, even for the same plugin and version. Registration creates database desired state only; a worker is launched only for an enabled installation.
|
||||
- Recovery: PostgreSQL installation desired state and durable binary storage are authoritative. Runtime reconnect performs an instance-wide full reconciliation, removes stale workers, and can replay a verified package after Runtime-local cache loss. Dependency preparation failure is recorded per installation with `dependency_prepare_failed`; it prevents that worker launch without blocking recovery of other desired installations, and the same revision can be retried. Enabled-resident is the desired semantic, but the current Supervisor restores an unexpectedly exited worker only on the next Core apply/reconcile; a completion callback, bounded backoff, and cross-tenant restart-storm isolation remain Cloud activation gates.
|
||||
- Recovery: PostgreSQL installation desired state and durable binary storage are authoritative. Runtime reconnect performs an instance-wide full reconciliation, removes stale workers, and can replay a verified package after Runtime-local cache loss. Dependency preparation failure is recorded per installation with `dependency_prepare_failed`; it prevents that worker launch without blocking recovery of other desired installations, and the same revision can be retried. The installation Supervisor now restores an unexpectedly exited enabled worker through a completion callback with bounded exponential backoff. Jitter, global restart concurrency limits, and a Runtime-wide circuit breaker are still required to prove that an infrastructure-wide failure cannot create a cross-tenant restart storm.
|
||||
- Compatibility: Older SDK payloads and legacy `data/plugins` remain an OSS-only bridge. Shared mode requires complete bindings and rejects incomplete context.
|
||||
- Reason: Sharing the supervisor and immutable code cache removes per-Workspace service cost without turning an untrusted plugin process into a cross-tenant router.
|
||||
|
||||
@@ -83,8 +83,8 @@ This log records implementation choices made while delivering the Workspace arch
|
||||
|
||||
### Unreleased SDK protocol is pinned reproducibly without publishing
|
||||
|
||||
- Decision: The SDK tenancy protocol is versioned as 0.4.15. This task does not create a GitHub release or publish PyPI because the user authorized pushing code, not a package release. After the SDK feature branch is final, LangBot's feature branch temporarily pins the exact pushed SDK Git commit. Before merging to master, the release gate is to publish `langbot-plugin==0.4.15` and replace the Git pin with the registry pin.
|
||||
- Reason: PyPI 0.4.14 does not contain `ActionContext`; pinning it makes clean installs fail, while pinning an unpublished 0.4.15 makes dependency sync impossible. An exact Git commit is reproducible and keeps the feature branch testable without expanding release authority.
|
||||
- Decision: The SDK tenancy protocol is versioned as 0.4.18. This task does not create a GitHub release or publish PyPI because the user authorized pushing code, not a package release. After the SDK feature branch is final, LangBot's feature branch temporarily pins the exact pushed SDK Git commit. Before merging to master, the release gate is to publish `langbot-plugin==0.4.18` and replace the Git pin with the registry pin.
|
||||
- Reason: The current registry release does not contain the complete tenant action context and shared Runtime hardening. An exact Git commit is reproducible and keeps the feature branch testable without expanding release authority.
|
||||
|
||||
### Cloud directory writes stay outside Core
|
||||
|
||||
@@ -240,5 +240,33 @@ This log records implementation choices made while delivering the Workspace arch
|
||||
### Shared Plugin Runtime starts only from verified desired state
|
||||
|
||||
- Decision: SDK shared mode waits for immutable runtime configuration before inspecting plugin state, never scans or launches legacy `data/plugins`, and rejects legacy install/restart/delete/upgrade control actions. Worker RPC files use installation-private directories with aggregate size enforcement, and resident nsjail workers explicitly disable the default 600-second wall-time limit.
|
||||
- Remaining gate: Unexpected worker exit is still recovered only by the next Core apply/reconcile. Completion callbacks, bounded backoff, hard installation disk quota, and production Linux/cgroup/egress evidence remain Cloud activation requirements.
|
||||
- Remaining gate: completion-callback recovery and bounded per-installation backoff are implemented. Cross-tenant restart-storm suppression, hard installation disk quota, and production egress policy remain Cloud activation requirements; Linux nsjail/cgroup CPU, memory-plus-swap, PID, namespace, and cgroup-reaping behavior have real-container evidence.
|
||||
- Reason: A shared supervisor reduces per-Workspace services only if legacy global paths, writable transfer state, and lifecycle defaults cannot bypass installation isolation.
|
||||
|
||||
## 2026-07-24
|
||||
|
||||
### Cloud v2 remains a modular monolith with one closed adapter
|
||||
|
||||
- Decision: Workspace directory, plans, subscriptions, payment fulfillment, entitlements, usage, and signed runtime feeds are implemented as modules in the existing Space service and PostgreSQL database. The only separately packaged closed component is the thin Core bootstrap/control-plane adapter; it verifies signed data and owns no business state.
|
||||
- Compatibility: The legacy Pod card and fulfillment path remain visible only to accounts with old subscriptions. New purchases create Workspace subscriptions and never provision a per-user LangBot Pod.
|
||||
- Reason: A separate tenancy or billing service would add deployment, queue, network, and consistency cost without providing an isolation boundary. The signed adapter keeps SaaS logic closed while Core retains the ORM, RLS, authorization, and runtime enforcement boundary.
|
||||
|
||||
### Registration creates data, not infrastructure
|
||||
|
||||
- Decision: Account registration and personal Workspace creation share one transaction and include an active owner membership, Free subscription, entitlement snapshot, and outbox notifications. Repair is idempotent for older accounts. Empty Workspace creation starts no Plugin worker, Box sandbox, database, queue, bucket, or tenant service.
|
||||
- Activation: This automatic Cloud v2 transaction is enabled only when Space has an explicit valid `CLOUD_V2_INSTANCE_UUID`. A legacy marketplace-only Space deployment with no Cloud v2 instance configured keeps its existing Account registration path; Cloud v2 internal endpoints still fail closed instead of inventing an instance identity.
|
||||
- Billing boundary: Core and runtimes consume only generic capability and numeric-limit entitlements. They never branch on `free` or `pro`; plan names, prices, payment providers, and fulfillment stay in Space.
|
||||
- Reason: This keeps the marginal cost of a new user close to a few PostgreSQL rows while preserving an immediate, usable Workspace.
|
||||
|
||||
### Directory bootstrap is full; steady-state projection is per Workspace
|
||||
|
||||
- Decision: Space generates the initial signed directory snapshot and its outbox high-water mark in one read-only PostgreSQL `REPEATABLE READ` transaction. After bootstrap, each event page names affected Workspaces and Core fetches one signed `directory.delta` for that set. Missing requested Workspaces are tombstones; unrelated Workspaces are untouched.
|
||||
- Cursor boundary: A delta carries no event cursor. Each signed event page carries the transaction-consistent current high-water mark, while Core advances only to the final event it actually consumed, even when the authoritative delta already contains a later revision. Event payload Workspace and revision fields must exactly match the signed event envelope; readiness is renewed only after the replica-local cursor reaches the signed high-water and the shared projection is not ahead.
|
||||
- Replica boundary: Every Core replica owns a process-local consumer cursor because its verified entitlement cache is process-local. Replicas share the PostgreSQL projection high-water mark and inbox. The state separately records which cursor range was atomically subsumed by a full snapshot, so a lagging replica can add missing receipts within that coverage and refresh its local caches without repeating tenant mutations; a missing receipt beyond snapshot coverage fails closed.
|
||||
- Reason: A shared consumer cursor would let one replica starve another replica's local cache, while a full snapshot on every change would make steady-state cost grow with every registered Workspace.
|
||||
|
||||
### Cloud OAuth can authenticate only an existing projected Account
|
||||
|
||||
- Decision: In multi-Workspace Cloud mode, Space OAuth may refresh tokens only for an active `cloud_projection` Account whose Space subject UUID and normalized email exactly match. It cannot create an Account, relink by email, mutate directory identity, or choose a Workspace.
|
||||
- Redirect boundary: When Cloud v2 configures its Core public URL, Space issues authorization codes only to that exact callback origin or the explicitly retained legacy managed-Pod domain. Community installations remain dynamic OAuth clients because they cannot pre-register with the public Space service: the consent screen displays their hostname, and only HTTPS or loopback HTTP with the fixed callback path is accepted. Remote HTTP, userinfo, fragments, arbitrary callback paths, and unrecognized query parameters always fail closed.
|
||||
- Reason: Space is the SaaS identity and directory authority, but Core remains the authentication enforcement point. Projected-only matching avoids split-brain identities; redirect allowlisting prevents bearer authorization-code exfiltration.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
状态:`DECIDED — Core isolation kernel implemented; SaaS activation gates remain`
|
||||
创建日期:2026-07-19
|
||||
最近更新:2026-07-20
|
||||
最近更新:2026-07-24
|
||||
|
||||
本文记录 Cloud v2 多租户架构中已经确认的首期决策、明确淘汰的方案和仍需在后续阶段决定的扩展项。
|
||||
本文同时记录实现状态。“实现完成”仅指开源 Core/SDK 的隔离内核和 fail-closed 门禁,
|
||||
@@ -28,10 +28,11 @@
|
||||
|
||||
| 编号 | 结论 | 首期状态 |
|
||||
| ----- | --------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
|
||||
| D-001 | 一个共享 Plugin Runtime 控制面;每个运行中的 plugin installation 独占一个 nsjail 子进程;只有 digest 相同且已验证的代码 artifact 可以只读共享 | `FOUNDATION IMPLEMENTED — Linux/egress, crash recovery and disk-quota gates pending` |
|
||||
| D-001 | 一个共享 Plugin Runtime 控制面;每个运行中的 plugin installation 独占一个 nsjail 子进程;只有 digest 相同且已验证的代码 artifact 可以只读共享 | `FOUNDATION VERIFIED — egress, total disk-quota and restart-storm gates pending` |
|
||||
| D-002 | 一个共享 Box Runtime;Cloud 固定使用 nsjail;符合套餐的 Workspace 最多一个持久 `global` 逻辑 sandbox,普通执行按需启动 nsjail 进程 | `IMPLEMENTED FAIL-CLOSED — hard filesystem quota provider pending` |
|
||||
| D-003 | SaaS 业务数据使用 PostgreSQL shared schema、应用层作用域和 RLS 双重隔离;pgvector 使用同一 PostgreSQL,作为 SaaS 默认向量后端 | `PARTIALLY IMPLEMENTED — transaction/outbox/deployment gates remain` |
|
||||
| D-004 | stdio MCP 与 Box availability 解耦;Cloud v2 首期强制关闭 stdio MCP,避免为每个 Workspace 创建额外的 `mcp-shared` persistent sandbox | `IMPLEMENTED` |
|
||||
| D-005 | 目录启动使用事务一致的全量快照,运行时按事件涉及的 Workspace 拉取增量;每个 Core replica 独立消费事件,共享 PostgreSQL 投影和 inbox | `IMPLEMENTED — production fault injection pending` |
|
||||
|
||||
Workspace 的具体创建、释放、数据导出和单 Workspace 恢复机制不在本轮决定;本文只保证这些后续能力不会改变稳定的
|
||||
`workspace_uuid`,也不会要求重建租户专属部署。
|
||||
@@ -76,13 +77,15 @@ flowchart LR
|
||||
M1 是 M0 的透明扩容,M2 是相同架构下的资源等级;两者都不是新的 LangBot 实例、Cell 或 CloudInstance。
|
||||
外部 API 只认识稳定的 `instance_uuid` 和 `workspace_uuid`,不认识 replica、worker、pool 或 shard。
|
||||
|
||||
Plugin Runtime 与 Core 在 M0 是否共用 Pod 仍可按发布和故障域决定;即使共用 Pod,也必须使用独立容器和 security context,
|
||||
Core 不能继承 Plugin Runtime 所需的 nsjail/cgroup 权限。Box Runtime 同样使用独立进程身份和安全配置,
|
||||
Plugin Runtime 与 Core 在 M0 使用独立容器和 security context,Core 不能继承 Plugin Runtime 所需的
|
||||
nsjail/cgroup 权限。当前 Runtime 在进程生命周期内绑定首次认证的 `runtime_id`,因此 M0 必须把 Core 与
|
||||
Plugin Runtime 放在同一 rollout/restart unit 中协调重启;在实现受认证 takeover 或 owner lease/fencing 前,
|
||||
不能单独滚动 Core 并让它接管仍存活的 Runtime。Box Runtime 同样使用独立进程身份和安全配置,
|
||||
不与 Plugin Runtime 合并成一个高权限进程。
|
||||
|
||||
## 3. D-001:Plugin Runtime 多租户控制面
|
||||
|
||||
状态:`FOUNDATION IMPLEMENTED — Cloud deployment and worker-recovery gates pending`
|
||||
状态:`FOUNDATION VERIFIED — Cloud egress, total disk-quota and restart-storm gates pending`
|
||||
|
||||
### 3.1 已实现的基础
|
||||
|
||||
@@ -116,8 +119,8 @@ Core 不能继承 Plugin Runtime 所需的 nsjail/cgroup 权限。Box Runtime
|
||||
- 安装并启用插件后,Supervisor 在自己的 Runtime 容器内直接启动一个 nsjail 子进程;
|
||||
不再为每个插件创建 nested container、Pod、sidecar 或租户级 Runtime service。
|
||||
- desired semantics 要求 enabled installation 保持 resident,不做 idle eviction;停用、删除、revision/generation 变化或 entitlement 撤销时停止并按需重建。
|
||||
当前实现只会在 Runtime 重连或 Core apply/reconcile 时恢复意外退出的 worker,尚缺 completion callback、有界 backoff 和跨租户重启风暴抑制;
|
||||
这属于 Cloud 激活门禁,不能把 desired semantics 描述成已经具备即时自愈。
|
||||
当前 Supervisor 已通过 completion callback 和有界指数 backoff 恢复意外退出的 worker;尚缺 jitter、全局重启并发上限和 Runtime 级 circuit breaker,
|
||||
因而系统性依赖或宿主故障下的跨租户重启风暴抑制仍是 Cloud 激活门禁。
|
||||
- 子进程使用一次性 registration capability 向 Supervisor 注册;capability 由可信 desired state 派生并绑定完整 installation tuple,
|
||||
不是插件直接建立 Core Host connection,也不能只绑定 author/name/path。Supervisor/Core 据此注入 tenant context,
|
||||
丢弃插件 payload 中自带的 scope 字段。
|
||||
@@ -473,9 +476,9 @@ mcp:
|
||||
- 直接调用 API、重放旧配置和启动 bootstrap 都失败关闭,且不会产生 `mcp-shared` session、nsjail 进程或额外配额占用。
|
||||
- OSS 默认行为保持兼容;HTTP/SSE MCP 正常工作。
|
||||
|
||||
## 7. 四项决策之间的关系
|
||||
## 7. 五项决策之间的关系
|
||||
|
||||
四项决策共同遵循:
|
||||
五项决策共同遵循:
|
||||
|
||||
> 多租户共享可信控制面、连接池、只读 artifact 和基础容量;租户独占不可信执行进程、sandbox、secret、可写文件和数据作用域。
|
||||
|
||||
@@ -485,6 +488,7 @@ mcp:
|
||||
- PostgreSQL 和 pgvector 共享数据库组件,但使用显式 tenant key、应用层 scope 和 RLS 防止共享存储变成共享权限。
|
||||
- 订阅管理只在闭源 Control Plane 维护套餐与计费规则,并向开源 Core 投影签名/版本化 entitlement;
|
||||
Core/Runtime 执行通用 capability 和数值限额,不复制套餐名称或计费逻辑。
|
||||
- 目录同步只在启动或恢复时读取全量快照;常态变更按 Workspace 聚合为签名增量,新增租户不会使每次目录事件退化为全实例重投影。
|
||||
|
||||
## 8. 当前不做分布式时仍保留的能力
|
||||
|
||||
@@ -497,6 +501,9 @@ mcp:
|
||||
6. schema migration、任务扫描、监控聚合和运维接口不能假设永远只有一个 Core 进程。
|
||||
7. 外部 API 不暴露 replica、worker 或 shard 标识;未来扩容不改变 Workspace URL、UUID 或客户端协议。
|
||||
8. 只有出现容量、可用性、地域或合规需求时才增加 replica/shard;预留协议不等于现在部署额外组件。
|
||||
9. 每个 Core replica 保存自己的事件消费 cursor,因为 entitlement cache 是进程本地状态;PostgreSQL 中的 projection high-water mark
|
||||
、snapshot coverage 和 inbox 仍由所有 replica 共享,用于幂等投影和冲突检测。事件页携带签名 high-water,副本追平前不能续期
|
||||
ready;不能让一个 replica 的共享 cursor 使其他 replica 跳过本地 cache 刷新。
|
||||
|
||||
## 9. 本轮明确不做的事情
|
||||
|
||||
|
||||
@@ -1,257 +1,252 @@
|
||||
# Multi-tenant isolation kernel verification report
|
||||
# Cloud v2 multi-tenant verification report
|
||||
|
||||
Date: 2026-07-20
|
||||
Date: 2026-07-24
|
||||
|
||||
Status: `CORE ISOLATION FOUNDATION VERIFIED — SAAS ACTIVATION REMAINS DISABLED`
|
||||
Status: `FOUNDATION AND CONTROL PLANE VERIFIED — PRODUCTION ACTIVATION REMAINS GATED`
|
||||
|
||||
This report records the final implementation and verification evidence for the
|
||||
shared Workspace isolation foundation. It does not claim that the closed SaaS
|
||||
Control Plane, billing system, or greenfield Cloud v2 deployment is
|
||||
production-ready.
|
||||
This report records the implementation and verification evidence for one
|
||||
logical LangBot instance serving multiple Workspace tenants. It covers the
|
||||
open-source Core, the shared Plugin/Box runtimes, the closed Space adapter, and
|
||||
the Space Cloud v2 modular-monolith control plane. It does not claim that the
|
||||
production Cloud deployment may enable `CLOUD_V2_ENABLED` yet.
|
||||
|
||||
## Repository refs and scope
|
||||
|
||||
- LangBot branch: `feat/multi-tenants`
|
||||
- LangBot implementation commit:
|
||||
`90a977488212d3f3f50fb92d94e3fcae60f27eab`
|
||||
- LangBot base: `origin/master` at
|
||||
`6baeb032a7f76c65337b51c7b58593de4687a61c`
|
||||
- Plugin SDK branch: `feat/multi-tenants`
|
||||
- Plugin SDK commit and LangBot dependency pin:
|
||||
`95a1805af2038c745de2c018a00db1305089a32e`
|
||||
- No tracked `langbot-space` change was made. Cloud v2 does not extend or
|
||||
preserve the legacy per-account instance/pod deployment topology.
|
||||
- LangBot Core: branch `feat/multi-tenants`
|
||||
- langbot-plugin-sdk: branch `feat/multi-tenants`, commit
|
||||
`e7d946af4a6b1494fbe74627c1815ace19ac8991`
|
||||
- langbot-space: branch `feat/cloud-v2-control-plane`
|
||||
- Closed Core adapter: `langbot-space/cloud-adapter`
|
||||
- SDK protocol/package version: `0.4.18`
|
||||
|
||||
The SDK protocol is versioned as 0.4.15 and is intentionally consumed from the
|
||||
exact pushed Git commit above. Publishing the release and replacing the Git
|
||||
pin with a registry pin remain merge-to-master release work; no registry
|
||||
release was performed here.
|
||||
Core pins the exact pushed SDK commit above. Cloud v2 is a greenfield
|
||||
multi-Workspace deployment and does not provision one Pod, database, queue, or
|
||||
Runtime per tenant. Legacy Space Pods remain a compatibility surface only.
|
||||
|
||||
## Implemented boundaries
|
||||
## Implemented product and security boundaries
|
||||
|
||||
### Workspace, identity, and authorization
|
||||
### Workspace and identity
|
||||
|
||||
- Community bootstraps exactly one local Workspace per LangBot instance while
|
||||
allowing multiple Accounts, memberships, invitations, and role-based
|
||||
permissions inside it. A mutable edition setting cannot activate SaaS
|
||||
multi-Workspace routing.
|
||||
- Each Account registration resolves or creates its local singleton Workspace.
|
||||
Invitation acceptance is one-time, email-bound, and survives the signed-out
|
||||
login transition without exposing the invitation secret in URLs after the
|
||||
initial fragment capture.
|
||||
- The Account-token Workspace discovery route is intentionally separate from
|
||||
Workspace-authorized routes. `X-Workspace-Id` becomes authority only after
|
||||
membership resolution; API keys and public resources derive Workspace from
|
||||
their trusted owner.
|
||||
- Ordinary monitoring uses `resource.view`. Export remains `data.export`,
|
||||
privileged runtime/system audit remains `audit.view`, and frontend controls
|
||||
follow the same permission split.
|
||||
- SaaS has one stable `instance_uuid`; `workspace_uuid` is the tenant key.
|
||||
- Registering an Account creates its personal Workspace and owner Membership in
|
||||
the same PostgreSQL transaction.
|
||||
- A Workspace may contain multiple users with owner, admin, and member roles.
|
||||
Invitation acceptance is token-hash based, email-bound, single-use,
|
||||
concurrency-safe, and member-limit checked while locked.
|
||||
- Community remains exactly one local Workspace while supporting multiple
|
||||
Accounts and fixed RBAC roles. Installing the closed adapter is required to
|
||||
inject the SaaS policy; configuration alone cannot activate it.
|
||||
- Disabled or deleted Accounts cannot use password/code/OAuth login, sessions,
|
||||
refresh/access tokens, or personal access tokens.
|
||||
|
||||
### Persistence and PostgreSQL
|
||||
### Closed Space control plane
|
||||
|
||||
- PostgreSQL uses one shared business schema with application scope plus exact
|
||||
`ENABLE` and `FORCE ROW LEVEL SECURITY` policies. Tenant UoWs keep `SET LOCAL`
|
||||
and business SQL on one owned root transaction and one task.
|
||||
- The public tenant Session accepts only structured SQLAlchemy query/DML trees.
|
||||
Raw/textual SQL, textual modifiers, custom AST/compiler nodes, unapproved
|
||||
functions/operators/types/casts, hidden dialect values, foreign binds,
|
||||
loader/execution options, Session event callbacks, ORM SQL-expression
|
||||
attributes, transaction control, and live-result escape fail closed and make
|
||||
the root transaction rollback-only.
|
||||
- The SQL guard is a trusted-Core misuse boundary, not an in-process Python
|
||||
sandbox. Mapped metadata and registered compilers are trusted boot-time code;
|
||||
plugins remain out of process. SQLAlchemy dialect-private container fields
|
||||
are pinned and covered by regression tests before dependency upgrades.
|
||||
- Legacy local RAG restore remains structured and tenant-fenced while accepting
|
||||
SQLite string-valued timestamps and both historical PostgreSQL `TEXT` and
|
||||
fresh-schema `JSON` settings columns.
|
||||
- The one-shot release migrator owns DDL and grants a distinct runtime role only
|
||||
the required business-table DML, `alembic_version` read access, sequence
|
||||
access, database `CONNECT`, and schema `USAGE`. Runtime startup reruns the
|
||||
role, ACL, schema, session, extension, routine, parameter, and RLS audit.
|
||||
- pgvector is stored in the same business database with Workspace-scoped keys,
|
||||
checked dimensions, release-created partial ANN indexes, and RLS.
|
||||
- `CLOUD_V2_ENABLED` defaults to false. Invalid or incomplete configuration
|
||||
fails startup, and disabled endpoints return 503.
|
||||
- Space owns Workspace/Membership/Invitation, versioned plans, subscriptions,
|
||||
entitlements, usage events, directory outbox, and signed instance manifests.
|
||||
- Free and Pro are Workspace plans. Pro projects one managed global sandbox;
|
||||
Free projects none. Both force stdio MCP off in the first release.
|
||||
- Subscription changes use Workspace advisory locking, expected entitlement
|
||||
revision, one-live-order uniqueness, idempotent usage ingestion, period-end
|
||||
downgrade, renewal, and lazy expiry settlement.
|
||||
- Directory snapshot and per-Workspace delta reads use repeatable-read,
|
||||
read-only transactions with a high-water cursor. Core stores a replica-local
|
||||
consumer cursor and shared PostgreSQL projection state, snapshot coverage,
|
||||
and inbox rows.
|
||||
- The `/cloud` page selects a Workspace and shows its independent subscription,
|
||||
entitlement, limits, and usage. When Cloud v2 is disabled or the backend does
|
||||
not expose the feature field, the complete legacy Welcome/Pod UI is used.
|
||||
|
||||
### Shared Plugin Runtime and Box
|
||||
### PostgreSQL and pgvector
|
||||
|
||||
- One instance-scoped Plugin Runtime supervisor serves multiple Workspaces.
|
||||
Every enabled installation still owns one nsjail worker with an immutable
|
||||
instance/Workspace/generation/installation/revision/artifact binding.
|
||||
- Verified same-digest code and dependency environments may be mounted
|
||||
read-only and reused. Plugin processes and writable installation state are
|
||||
never merged, including for the same plugin and version.
|
||||
- Runtime limits come only from instance configuration with native environment
|
||||
overrides. Plugin manifests cannot raise limits. Shared mode rejects legacy
|
||||
global plugin directories and legacy lifecycle control paths.
|
||||
- One shared Box control plane admits at most one persistent logical `global`
|
||||
sandbox for an entitled Workspace. Admission is entitlement-gated,
|
||||
generation-fenced, shared-volume-challenged, quota-aware, and fixed to
|
||||
nsjail. Plain nsjail correctly fails Cloud readiness without a hard quota
|
||||
provider.
|
||||
- stdio MCP has an independent gate and is forced off by Cloud bootstrap even
|
||||
when Box is available.
|
||||
- SaaS business data uses one PostgreSQL shared schema with application scope
|
||||
plus forced RLS. Cloud directory writes are separated from local tenant
|
||||
writes.
|
||||
- Projected Account and Membership revisions are monotonic. Tombstones remove
|
||||
memberships, and stale revisions cannot resurrect them.
|
||||
- pgvector shares the business PostgreSQL database and remains
|
||||
Workspace-scoped.
|
||||
- Space runs versioned SQL migrations before application seeds. A fresh
|
||||
database, a partial pre-migration database, and an existing-baseline path
|
||||
converge without startup-time `AutoMigrate`.
|
||||
|
||||
### Shared Plugin Runtime
|
||||
|
||||
- One instance-scoped trusted Supervisor serves multiple Workspaces.
|
||||
- Every enabled installation has its own nsjail process bound to
|
||||
`(instance, workspace, execution generation, installation, runtime revision,
|
||||
artifact digest)`.
|
||||
- Verified same-digest code and dependency files are mounted read-only and may
|
||||
be shared; home, tmp, data, process namespace, registration capability, and
|
||||
cgroup are private.
|
||||
- Instance configuration owns CPU, memory, PID, open-file, and file-size
|
||||
limits. Plugin manifests cannot increase them.
|
||||
- Memory includes swap: nsjail receives `memory.max` and `swap.max=0`.
|
||||
- Unexpected worker exit is recovered by a completion callback with bounded
|
||||
per-installation exponential backoff. Remove, reconcile, Runtime shutdown,
|
||||
and container SIGTERM perform a graceful-to-SIGKILL bounded reap.
|
||||
|
||||
### Shared Box Runtime and MCP
|
||||
|
||||
- One shared Box control plane serves Workspace-bound logical sessions.
|
||||
- Cloud grants allow at most one persistent `global` session for an entitled
|
||||
Workspace and no managed processes in the first release.
|
||||
- Cloud is fixed to nsjail and network-off. Core and Box prove the shared
|
||||
durable Workspace mount with an authenticated marker challenge.
|
||||
- stdio MCP is independently gated and forced off for Cloud v2.
|
||||
- The current nsjail Box backend does not provide hard byte/inode quotas, so
|
||||
Cloud readiness correctly fails closed instead of silently using a soft
|
||||
directory scan.
|
||||
|
||||
## Automated verification
|
||||
|
||||
### LangBot backend and static checks
|
||||
### LangBot Core
|
||||
|
||||
```text
|
||||
uv run pytest -q --tb=short
|
||||
2527 passed, 32 skipped, 177 warnings in 73.37s
|
||||
uv run --no-sync pytest -q
|
||||
2590 passed, 32 skipped, 177 warnings
|
||||
|
||||
focused tenant UoW and legacy RAG compatibility suite
|
||||
120 passed, 10 warnings in 1.30s
|
||||
|
||||
uv run ruff check .
|
||||
passed
|
||||
|
||||
changed Python format check
|
||||
15 files already formatted
|
||||
real PostgreSQL migration, pgvector, and release-migrator suites
|
||||
21 passed, 11 warnings
|
||||
|
||||
uv run --no-sync ruff check .
|
||||
uv lock --check
|
||||
git diff --check
|
||||
passed
|
||||
```
|
||||
|
||||
The full suite includes startup E2E, the fresh SQLite registration/Workspace
|
||||
journey, API authorization, storage, RAG, Plugin, MCP, and 18 Docker-backed Box
|
||||
integration cases.
|
||||
The full suite ran without the closed adapter installed, proving the open-source
|
||||
single-Workspace/multi-user path remains standalone. Focused closed-adapter,
|
||||
directory projection, runtime connector, Box cleanup, and configuration suites
|
||||
also passed with the adapter installed.
|
||||
|
||||
### Real PostgreSQL 16 and pgvector
|
||||
### Plugin SDK and real Linux runtime
|
||||
|
||||
```text
|
||||
test_migrations_postgres.py
|
||||
test_pgvector_postgres.py
|
||||
test_release_migration_postgres.py
|
||||
21 passed, 11 warnings in 11.61s
|
||||
SDK full suite
|
||||
1226 passed, 22 existing warnings
|
||||
|
||||
Ruff check and format check
|
||||
git diff --check
|
||||
passed
|
||||
```
|
||||
|
||||
The real-database suite covers fresh and upgraded RLS schemas, two-Workspace
|
||||
isolation, pgvector CRUD and ANN indexes, least-privilege runtime bootstrap,
|
||||
release locking, historical/fresh RAG settings column compatibility, and
|
||||
transaction-scope escape rejection. Negative tests inject and reject role
|
||||
membership, grant options, persistent GUCs, extra schemas and relations,
|
||||
column ACLs, explicit routine and parameter ACLs, runtime-owned and
|
||||
`SECURITY DEFINER` routines, extra/runtime-owned extensions, and foreign data
|
||||
wrappers, servers, and user mappings.
|
||||
A privileged Linux test container with host cgroup namespace ran one shared
|
||||
Runtime and two Workspace installations:
|
||||
|
||||
Alembic has exactly one head: `0013_tenant_pgvector`.
|
||||
- both workers referenced the same artifact inode;
|
||||
- home, tmp, and data inodes were distinct;
|
||||
- each plugin saw only PID 1 in its private PID namespace;
|
||||
- a tampered binding and an unknown installation were rejected;
|
||||
- the control token was absent from worker environments;
|
||||
- cgroups were distinct with `memory.max=134217728`, `memory.swap.max=0`,
|
||||
`pids.max=32`, and `cpu.max=500000 1000000`;
|
||||
- touching 256 MiB exited with code 137 without swap growth;
|
||||
- the 32nd fork failed with `EAGAIN`;
|
||||
- reconcile and container SIGTERM removed the worker cgroups.
|
||||
|
||||
### Plugin SDK and Box Runtime
|
||||
The same run started the Runtime from a non-root working directory, covering
|
||||
absolute nsjail mount-source normalization.
|
||||
|
||||
### Space backend, adapter, and frontend
|
||||
|
||||
```text
|
||||
Plugin SDK full suite
|
||||
1160 passed, 18 warnings
|
||||
|
||||
Plugin SDK Ruff and action consistency checks
|
||||
go test ./...
|
||||
passed
|
||||
|
||||
Docker-backed Box integration suite
|
||||
18 passed, 33 warnings
|
||||
fresh PostgreSQL app startup, partial-baseline migration,
|
||||
Cloud v2 migration rerun, and control-plane integration
|
||||
passed
|
||||
|
||||
closed adapter pytest and Ruff
|
||||
passed
|
||||
|
||||
pnpm exec tsc --noEmit
|
||||
targeted ESLint for changed Cloud files
|
||||
pnpm build
|
||||
passed; 47 routes generated
|
||||
```
|
||||
|
||||
The LangBot environment resolves `langbot-plugin==0.4.15` from the same exact
|
||||
SDK commit recorded above. The SDK local ref and remote
|
||||
`refs/heads/feat/multi-tenants` were independently verified equal.
|
||||
The PostgreSQL checks started from an empty database and verified all 32
|
||||
registered migrations in order, Cloud v2 Free/Pro seeds, legacy plan seeds,
|
||||
Cloud columns/indexes, and repeatable reruns.
|
||||
|
||||
### Frontend
|
||||
## Cross-service and browser E2E
|
||||
|
||||
```text
|
||||
pnpm lint
|
||||
0 errors, 34 existing warnings
|
||||
### Signed Space-to-Core directory projection
|
||||
|
||||
VITE_API_BASE_URL=/ pnpm build
|
||||
passed; 3168 modules transformed
|
||||
Using an isolated Space PostgreSQL database and a migrated Core PostgreSQL
|
||||
database:
|
||||
|
||||
pnpm exec playwright test --project=chromium
|
||||
43 passed in 22.2s
|
||||
```
|
||||
1. Space issued a signed manifest for the fixed LangBot instance.
|
||||
2. Space returned a directory snapshot at cursor 5 containing two Workspaces,
|
||||
two projected Accounts, and owner/admin memberships.
|
||||
3. Core verified the signature, instance, release, validity window, and
|
||||
capability before injecting the Cloud Workspace policy.
|
||||
4. Core stored both Workspaces, all active Memberships, both projected
|
||||
Accounts, snapshot coverage, inbox entries, and cursor 5.
|
||||
5. Account-field-only projection revisions and Workspace directory revisions
|
||||
remained independent, and cross-Workspace Account conflicts failed closed.
|
||||
|
||||
The frontend suite includes Viewer monitoring visibility, privileged control
|
||||
hiding, stable terminal invitation states, email-mismatch login handoff, and a
|
||||
temporary invitation-accept failure that retains the authenticated session and
|
||||
succeeds on retry.
|
||||
### Real browser Cloud v2 flow
|
||||
|
||||
### Packaging and deployment configuration
|
||||
A real local browser operated the Space frontend and backend:
|
||||
|
||||
- `uv lock --check` resolved 277 packages and passed.
|
||||
- Changed TypeScript/TSX files passed Prettier verification.
|
||||
- `docker compose -f docker/docker-compose.yaml config --quiet` passed with the
|
||||
existing warning that the top-level `version` field is obsolete.
|
||||
- The staged high-signal credential scan found no private key or provider
|
||||
token.
|
||||
1. An owner logged in and saw the automatically created personal Workspace.
|
||||
2. A second Account invited the owner as admin to another Workspace.
|
||||
3. The selector showed both Workspaces and switched between them without stale
|
||||
subscription or usage state crossing the selection.
|
||||
4. Each Workspace showed its own Free entitlement and limits.
|
||||
5. Pro checkout opened for the selected Workspace. With no configured payment
|
||||
channels, confirmation was visibly disabled and no order was created.
|
||||
6. No browser errors occurred; only the pre-existing i18next warning for the
|
||||
short `en` locale was observed.
|
||||
|
||||
## Real browser E2E
|
||||
The legacy feature-flag branch was then covered by API/static checks and the
|
||||
production build: false or missing `cloud_v2_enabled` renders the original
|
||||
Welcome/Pod client; a failed web-config request renders an explicit retry
|
||||
instead of guessing a deployment mode.
|
||||
|
||||
A production frontend and backend were run against an isolated temporary
|
||||
SQLite data directory and operated through a real local Chrome session:
|
||||
### Deliberate Core startup failure
|
||||
|
||||
1. The first Account registered and automatically received the default
|
||||
singleton Workspace as owner.
|
||||
2. Skipping the setup wizard reached `/home/monitoring`; reload and a new tab
|
||||
recovered the authenticated Account, membership, and Workspace selection.
|
||||
3. The owner invited a Viewer. After signing out, the second Account registered
|
||||
through the invitation and joined the same Workspace.
|
||||
4. Owner and Admin could read overview/export/audit/API-key surfaces and reach
|
||||
resource-write validation. Runtime debug mutation remained disabled.
|
||||
5. Developer retained ordinary resource management and monitoring, while
|
||||
export, audit, and API-key management returned 403.
|
||||
6. Operator and Viewer retained ordinary monitoring reads while resource
|
||||
writes, export, audit, API-key management, and runtime mutation returned 403.
|
||||
7. The UI matched the server matrix: Add/API Key/Export controls appeared only
|
||||
with their permissions, while ordinary monitoring refresh remained visible
|
||||
to read-only members.
|
||||
8. A second Workspace creation attempt returned 403 `edition_limit`, proving
|
||||
Community remains single-Workspace and multi-user.
|
||||
9. Used, revoked, expired, and email-mismatched invitation paths produced
|
||||
stable visible states. The mismatch survived login without a contradictory
|
||||
generic success toast.
|
||||
Core completed signed manifest verification and directory projection, then
|
||||
stopped at the Box readiness gate because the current nsjail backend cannot
|
||||
prove hard Workspace byte/inode quota enforcement. Connector shutdown and
|
||||
reconnect tasks were cleanly reaped; no event-loop or never-awaited coroutine
|
||||
warning remained.
|
||||
|
||||
The server was stopped, port 15321 was verified free, and temporary artifacts
|
||||
were moved to the system Trash for recoverable cleanup. No invitation secret,
|
||||
password, token, or provider credential is recorded in this report.
|
||||
This is a successful fail-closed acceptance result, not a passing production
|
||||
Cloud boot.
|
||||
|
||||
## Deliberately incomplete SaaS activation gates
|
||||
## Remaining production activation gates
|
||||
|
||||
Multi-Workspace activation remains unavailable in the open-source bootstrap.
|
||||
Cloud v2 must stay disabled until all of the following are implemented and
|
||||
independently verified:
|
||||
Cloud v2 must remain disabled until these gates are closed:
|
||||
|
||||
- the closed Control Plane owns the Account/Workspace directory, memberships,
|
||||
invitations, monotonic execution generations, entitlements, usage,
|
||||
subscriptions, and billing;
|
||||
- ordinary tenant writes hold a generation-aware fence through commit, and a
|
||||
generation-stamped outbox or equivalent atomic publish fence protects
|
||||
external side effects;
|
||||
- durable object references survive generation cutovers without stranding
|
||||
files, images, plugin artifacts, or knowledge-base content;
|
||||
- every tenant-configurable outbound path has complete SSRF/egress policy, and
|
||||
Plugin Runtime has production Linux namespace, cgroup v2, and network
|
||||
isolation evidence;
|
||||
- an unexpectedly exited enabled plugin worker is restored by a completion
|
||||
callback with bounded backoff and cross-tenant restart-storm isolation;
|
||||
- plugin installation data has an operator-owned hard disk quota, not a
|
||||
directory-scan approximation;
|
||||
- Box supplies and proves hard byte/inode quotas for Workspace, Skill, root,
|
||||
home, and temporary storage, and the production shared volume passes the
|
||||
authenticated marker challenge;
|
||||
- production PostgreSQL uses a dedicated endpoint/cluster, or a tested
|
||||
HBA/proxy policy proves the runtime credential can connect only to the target
|
||||
business database;
|
||||
- the release migrator is deployed as a one-shot Job with credential issuance,
|
||||
backup, rollback, failure/retry, and orchestration procedures;
|
||||
- OAuth exchange, directory projection, leases, snapshots, event replay, and
|
||||
outbox state use durable shared stores suitable for horizontally scaled
|
||||
replicas;
|
||||
- Workspace release, export, deletion, restore, and disaster-recovery semantics
|
||||
are decided and implemented; and
|
||||
- production Runtime restart, crash, partition, revocation, and fault-injection
|
||||
acceptance is completed against the closed Control Plane and greenfield
|
||||
Cloud v2 deployment.
|
||||
- Box provides and proves hard byte and inode quotas for Workspace, Skill,
|
||||
root, home, and tmp storage.
|
||||
- Plugin installation writable data receives an operator-owned hard total
|
||||
disk quota.
|
||||
- Plugin and future networked Box workloads have tenant-safe egress/SSRF
|
||||
policy.
|
||||
- Plugin Runtime adds jitter, global restart concurrency limiting, and a
|
||||
Runtime-level circuit breaker, then passes systemic-failure injection.
|
||||
- M0 rolls Core and Plugin Runtime together until authenticated Runtime takeover
|
||||
or an owner lease/fencing protocol exists.
|
||||
- Account registration moves New API provisioning to an in-process,
|
||||
transactional outbox reconciler keyed by Space Account UUID.
|
||||
- EPay and Stripe fulfillment bind provider identity, amount, currency,
|
||||
channel/session, payment status, and expiry to the locked order transaction;
|
||||
provider transaction IDs are unique and replay-safe.
|
||||
- Payment-session creation and stale `processing` orders gain in-process
|
||||
reconciliation.
|
||||
- Cloud v2 subscription service periods are stored immutably and included in
|
||||
recognized-revenue reporting.
|
||||
- Production migration Job, backup/rollback, PostgreSQL credential/network
|
||||
boundaries, horizontal-replica fault injection, Workspace release/export,
|
||||
deletion, and restore semantics are completed.
|
||||
|
||||
The following product work is also deliberately deferred: WebUI configuration
|
||||
for tenant-provided remote E2B, multi-replica lease storage and sharding rules,
|
||||
artifact signing/garbage collection, custom roles, SSO, and SCIM.
|
||||
|
||||
These are explicit release gates, not hidden fallbacks. The current branches
|
||||
are a verified isolation foundation on which the closed SaaS system can be
|
||||
built; they are not a production Cloud activation.
|
||||
These gates intentionally add no tenant-specific service. They are implemented
|
||||
inside the existing Space, Core, Plugin Runtime, Box Runtime, and PostgreSQL
|
||||
components to preserve the architecture goal: near-zero static cost for a new
|
||||
Workspace.
|
||||
|
||||
@@ -148,7 +148,8 @@ M1 是 M0 的透明扩容,M2 是相同架构下的资源等级。外部 API
|
||||
|
||||
### 4.4 组件边界
|
||||
|
||||
- Core、Plugin Runtime 和 Box Runtime 可以由发布与故障域决定是否共用 Pod,但必须保持独立进程身份、容器和 security context。
|
||||
- Core、Plugin Runtime 和 Box Runtime 必须保持独立进程身份、容器和 security context。M0 中 Core 与 Plugin Runtime
|
||||
需要处于同一 rollout/restart unit;在实现受认证 takeover 或 owner lease/fencing 前,Core 不能单独重启后接管仍存活的 Runtime。
|
||||
- Core 不能继承 nsjail、cgroup 或 mount namespace 所需的高权限。
|
||||
- Plugin Runtime 与 Box Runtime 不合并为一个高权限进程。
|
||||
- MVP 不新增 Runtime 专用数据库、Box 专用数据库、Kafka、Redis、租户级 scheduler 或 artifact service。
|
||||
@@ -321,7 +322,9 @@ delegated issuers and keyset revision
|
||||
### 7.2 DirectoryEvent 与目录新鲜度
|
||||
|
||||
Control Plane 通过 transactional outbox 发布 Account、Workspace 和 Membership 的版本化事件。
|
||||
Core 使用 inbox 按 `event_id` 去重,以 aggregate revision 拒绝旧写,并追踪连续应用水位。
|
||||
Core 使用 inbox 按 `event_id` 去重,以 aggregate revision 拒绝旧写,并追踪连续应用水位。启动时读取一个 PostgreSQL
|
||||
`REPEATABLE READ` 事务内生成的签名全量 snapshot;运行时先消费携带当前 high-water 的签名事件页,再只请求该页涉及的 Workspace 签名增量。
|
||||
增量响应不携带新的事件 cursor,因此即使其内容已包含并发提交的后续 revision,也不能跳过尚未消费的事件。
|
||||
|
||||
要求:
|
||||
|
||||
@@ -329,6 +332,10 @@ Core 使用 inbox 按 `event_id` 去重,以 aggregate revision 拒绝旧写,
|
||||
- 重复、乱序、延迟、断流和全量 replay 都安全。
|
||||
- 删除使用 tombstone。
|
||||
- 新实例先导入带 high watermark 的 snapshot,再消费增量。
|
||||
- 常态目录更新成本与本页发生变化的 Workspace 数量相关,不得为每个 `directory.changed` 重新读取和投影全部 Workspace。
|
||||
- 每个 Core replica 独立保存进程内消费 cursor,以确保各自的 entitlement cache 都看到事件;共享 PostgreSQL 保存投影
|
||||
high-water mark、全量 snapshot coverage 和 inbox。同一事件被多个 replica 消费时,第二个 replica 验证已有 receipt;
|
||||
snapshot coverage 内缺少的 receipt 可以补写,coverage 之外缺失则失败关闭。只有本地 cursor 追平签名 high-water 后才续期 ready。
|
||||
- projection 未就绪或落后于授权 lease 要求时,交互与自动化请求按策略失败关闭。
|
||||
- SaaS pending Invitation、email 和 token hash 不进入 Core 投影。
|
||||
|
||||
@@ -630,8 +637,8 @@ installation 总磁盘配额需要可原子拒绝写入的 quota provider,不
|
||||
- Runtime 重连执行实例范围 full reconciliation,清理 stale worker 并恢复 enabled installation。
|
||||
- dependency preparation 失败记录在对应 installation,不启动半就绪 worker,也不阻塞其他 installation。
|
||||
- desired semantics 要求 enabled installation 常驻,不做 idle eviction;是否按负载回收以后再决定。
|
||||
- 当前 Supervisor 可在 Runtime 重连或 Core apply/reconcile 时恢复 desired state,但尚未为意外退出的 worker
|
||||
实现带有界 backoff 的 completion callback;这项可靠性缺口在 Cloud 激活前必须补齐并验证。
|
||||
- 当前 Supervisor 已在意外退出时通过 completion callback 和有界指数 backoff 恢复 enabled worker。
|
||||
Cloud 激活前仍需加入 jitter、全局重启并发上限和 Runtime 级 circuit breaker,并验证系统性故障不会形成跨租户重启风暴。
|
||||
|
||||
真实 Linux/nsjail/cgroup 与受控 egress 的 Cloud 部署验证尚未完成,是生产激活门禁。
|
||||
|
||||
@@ -813,7 +820,7 @@ SaaS:
|
||||
2. 普通业务写入贯穿 commit 的 generation-aware fence,以及与外部副作用同事务的 business outbox。
|
||||
3. generation cutover 后稳定的 durable object identity 或原子对象引用迁移。
|
||||
4. 所有 tenant-configurable outbound URL 的 SSRF 防护与 tenant-safe egress;Plugin Runtime 还需在真实 Linux/nsjail/cgroup v2 环境验证 namespace、资源限制和文件隔离。
|
||||
5. Plugin Runtime 对意外退出的 enabled worker 实现 completion callback、有界 backoff 和自动恢复,并验证不会形成跨租户重启风暴。
|
||||
5. Plugin Runtime 已实现意外退出 worker 的 completion callback、有界 backoff 和自动恢复;Cloud 激活前增加全局重启风暴抑制并完成故障注入验证。
|
||||
6. Plugin installation data 的 production hard disk quota provider,能够在写入边界原子拒绝超额,不能以目录扫描代替。
|
||||
7. Box Runtime 的 production hard quota provider,包括 Workspace、Skill、root/tmp/home 的 byte 与 inode quota;真实部署还必须在启动和重连时通过共享卷 marker challenge。
|
||||
8. PostgreSQL runtime credential 的专用 endpoint 或 HBA/proxy 跨 database 隔离证明、生产 migration/rollback 流程,以及 legacy pgvector migration 失败后精确恢复 RLS/FORCE 并可安全重试的集成证据。
|
||||
|
||||
Reference in New Issue
Block a user