fix(cloud): bound runtime restart storms

docs/multi-tenant/cloud-runtime-soak-gate.md

@@ -80,6 +80,7 @@ uv run python scripts/cloud_runtime_soak.py \
 - 任一健康采样窗口的 event-loop recent max 超过 1 秒，或冷却尾段 recent p95 超过 250 ms。
 - 健康接口缺少 event-loop monitor、monitor 未持续运行，或其 sample counter 回退。
 - blocking executor rejection counter 增长。
+- Plugin Runtime restart circuit 的累计打开次数增长。
 
 负载结束后的冷却尾段还必须满足：
 
@@ -87,6 +88,8 @@ uv run python scripts/cloud_runtime_soak.py \
 - 平均 CPU 核数不超过 `--max-tail-cpu-cores`。
 - event-loop recent p95 不超过 `--max-event-loop-p95-lag-ms`。
 - blocking executor `pending` 至少回到过零；不能整个尾段持续积压。
+- Plugin Runtime restart coordinator 的 active launch、half-open probe 和
+  circuit open remaining time 必须回到零。
 - telemetry、QueryPool、MCP host/dispatch、Box creating/closing/background 等临时 gauge 不能继续增长。
 
 内存判定要求“增长量”和“斜率”同时越界，避免几 MiB allocator/page-cache 噪声在短窗口被外推成很大的每小时斜率。最终报告仍保留实际增长与斜率，人工审查时不能只看 verdict。

docs/multi-tenant/cloud-v2-pending-verification.md

@@ -24,7 +24,7 @@
 - PostgreSQL 16 + RLS 的 1,000 Workspace 真实启动测试，以及 5,000
   Workspace 三代替换合成探针已通过。
 - Core 已精确钉住 Plugin SDK 提交
-  `67be7c332ded4c1afa131b28232590cf9728f817`。最终验证必须使用包含该提交的
+  `a5a96b302a5808af84bbdd28833ce050176f87e9`。最终验证必须使用包含该提交的
   Core、Plugin Runtime 和 Box Runtime 镜像，不能混用旧 SDK。
 
 以上结果是进入生产候选验证的前提，不是 SaaS 上线批准。
@@ -35,10 +35,9 @@
 
 | 编号 | 阻断项 | 完成实现后的最低验收证据 |
 | --- | --- | --- |
-| B-01 | Plugin Runtime 缺少跨 installation 的重启风暴抑制，包括 jitter、全局重启并发上限和 Runtime 级 circuit breaker | 同时使大量 worker 因系统性故障退出，证明重启速率和并发受限、控制面仍可用，且单个故障 installation 不拖垮其他租户 |
-| B-02 | Cloud 插件缺少生产 egress policy | 证明插件只能访问允许的公网目标，不能访问 Core/Box/数据库、其他内部服务、loopback、link-local 或云 metadata endpoint |
-| B-03 | Plugin installation 与 Box Workspace/Skill/root/tmp/home 缺少真实的 byte 和 inode 硬配额 provider | 在写入边界原子拒绝超额；并发写入、重启和配额耗尽后不能越界，也不能用目录扫描或事后清理冒充硬配额 |
-| B-04 | 普通业务写入尚未具备贯穿 commit 的 generation-aware fence、同事务 business outbox，以及 generation cutover 后稳定的 durable-object 引用 | 在旧 generation 与新 generation 并发、事务提交竞态和重复投递下，旧 owner 不产生业务写入或外部副作用，outbox 可幂等恢复 |
+| B-01 | Cloud 插件缺少生产 egress policy | 证明插件只能访问允许的公网目标，不能访问 Core/Box/数据库、其他内部服务、loopback、link-local 或云 metadata endpoint |
+| B-02 | Plugin installation 与 Box Workspace/Skill/root/tmp/home 缺少真实的 byte 和 inode 硬配额 provider | 在写入边界原子拒绝超额；并发写入、重启和配额耗尽后不能越界，也不能用目录扫描或事后清理冒充硬配额 |
+| B-03 | 普通业务写入尚未具备贯穿 commit 的 generation-aware fence、同事务 business outbox，以及 generation cutover 后稳定的 durable-object 引用 | 在旧 generation 与新 generation 并发、事务提交竞态和重复投递下，旧 owner 不产生业务写入或外部副作用，outbox 可幂等恢复 |
 
 任一 B 类项目未关闭时，不得把 24 小时 soak 的通过结果解释为可以上线。
 
@@ -68,7 +67,7 @@
 
 ### V-02：Box 持久卷与硬存储配额
 
-在 B-03 的 quota provider 实现后，必须验证：
+在 B-02 的 quota provider 实现后，必须验证：
 
 1. Core 与 Box Runtime 通过随机 marker challenge 证明使用同一共享持久卷。
 2. Workspace、Skill store、ephemeral root/tmp/home 的 byte 和 inode quota
@@ -125,6 +124,9 @@
    plugin worker 和缓存写入全部失败关闭。
 6. 为未来多副本预留的 replica-local cursor 语义通过故障注入：
    一个副本追平不能使另一个副本跳过本地 cache 刷新。
+7. 同时使大量 plugin worker 因系统性故障退出，证明 restart launch 全局并发受限、
+   失败阈值触发 Runtime circuit、冷却后只有一个 half-open probe，且 probe 未稳定前
+   其他 installation 不会继续重启；24 小时门禁必须把 circuit 打开判为失败。
 
 ### V-06：套餐、Box 与 stdio MCP
 
@@ -169,7 +171,8 @@ PostgreSQL/pgvector 和代表性 Workspace 配置分布，测量：
 
 1. 注册、邀请、登录和 entitlement 刷新；
 2. plugin reconcile、依赖准备、调用、崩溃与重启；
-3. Dashboard/Embed/平台 WebSocket 建连、突发消息和断连；
+3. Dashboard/Embed/平台 WebSocket 建连、突发消息和断连；HTTP Bot 覆盖
+   高基数 session/idempotency、硬容量拒绝、空闲回收及 callback 堵塞；
 4. Box session、文件同步、并发 exec、输出与清理；
 5. PostgreSQL pool 接近容量、事务超时和恢复；
 6. Core、Plugin Runtime、Box 分别 SIGTERM 和恢复。
@@ -208,5 +211,5 @@ PID limit、blocking executor rejection、超阈值 CPU throttling/event-loop la
 3. 由报告明确给出 pass/fail，不能只依赖日志中“看起来正常”；
 4. 任一生产候选输入变化后，重跑受影响的验证。
 
-在 B-01 至 B-04 全部实现，且 V-01 至 V-09 均有当前生产候选版本的通过证据前，
+在 B-01 至 B-03 全部实现，且 V-01 至 V-09 均有当前生产候选版本的通过证据前，
 Cloud v2 状态保持 `NOT APPROVED FOR SAAS ACTIVATION`。

docs/multi-tenant/implementation-decisions.md

@@ -240,7 +240,7 @@ This log records implementation choices made while delivering the Workspace arch
 ### Shared Plugin Runtime starts only from verified desired state
 
 - Decision: SDK shared mode waits for immutable runtime configuration before inspecting plugin state, never scans or launches legacy `data/plugins`, and rejects legacy install/restart/delete/upgrade control actions. Worker RPC files use installation-private directories with aggregate size enforcement, and resident nsjail workers explicitly disable the default 600-second wall-time limit.
-- Remaining gate: completion-callback recovery and bounded per-installation backoff are implemented. Cross-tenant restart-storm suppression, hard installation disk quota, and production egress policy remain Cloud activation requirements; Linux nsjail/cgroup CPU, memory-plus-swap, PID, namespace, and cgroup-reaping behavior have real-container evidence.
+- Remaining gate: completion-callback recovery, jittered per-installation backoff, globally bounded restart launch admission, Runtime-level circuit breaking, one half-open probe, and worker ready timeout are implemented. Production cross-tenant fault injection must still prove the restart-storm controls. Hard installation disk quota and production egress policy remain Cloud activation requirements; Linux nsjail/cgroup CPU, memory-plus-swap, PID, namespace, and cgroup-reaping behavior have real-container evidence.
 - Reason: A shared supervisor reduces per-Workspace services only if legacy global paths, writable transfer state, and lifecycle defaults cannot bypass installation isolation.
 
 ## 2026-07-24

docs/multi-tenant/pending-architecture-decisions.md

@@ -28,7 +28,7 @@
 
 | 编号  | 结论                                                                                                                                          | 首期状态                                                             |
 | ----- | --------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
-| D-001 | 一个共享 Plugin Runtime 控制面；每个运行中的 plugin installation 独占一个 nsjail 子进程；只有 digest 相同且已验证的代码 artifact 可以只读共享 | `FOUNDATION VERIFIED — egress, total disk-quota and restart-storm gates pending` |
+| D-001 | 一个共享 Plugin Runtime 控制面；每个运行中的 plugin installation 独占一个 nsjail 子进程；只有 digest 相同且已验证的代码 artifact 可以只读共享 | `IMPLEMENTED — egress/disk-quota pending; restart-storm fault injection pending` |
 | D-002 | 一个共享 Box Runtime；Cloud 固定使用 nsjail；符合套餐的 Workspace 最多一个持久 `global` 逻辑 sandbox，普通执行按需启动 nsjail 进程            | `IMPLEMENTED FAIL-CLOSED — hard filesystem quota provider pending`   |
 | D-003 | SaaS 业务数据使用 PostgreSQL shared schema、应用层作用域和 RLS 双重隔离；pgvector 使用同一 PostgreSQL，作为 SaaS 默认向量后端                 | `PARTIALLY IMPLEMENTED — transaction/outbox/deployment gates remain` |
 | D-004 | stdio MCP 与 Box availability 解耦；Cloud v2 首期强制关闭 stdio MCP，避免为每个 Workspace 创建额外的 `mcp-shared` persistent sandbox          | `IMPLEMENTED`                                                        |
@@ -85,7 +85,7 @@ Plugin Runtime 放在同一 rollout/restart unit 中协调重启；在实现受
 
 ## 3. D-001：Plugin Runtime 多租户控制面
 
-状态：`FOUNDATION VERIFIED — Cloud egress, total disk-quota and restart-storm gates pending`
+状态：`IMPLEMENTED — Cloud egress/disk-quota pending; restart-storm fault injection pending`
 
 ### 3.1 已实现的基础
 
@@ -119,8 +119,10 @@ Plugin Runtime 放在同一 rollout/restart unit 中协调重启；在实现受
 - 安装并启用插件后，Supervisor 在自己的 Runtime 容器内直接启动一个 nsjail 子进程；
   不再为每个插件创建 nested container、Pod、sidecar 或租户级 Runtime service。
 - desired semantics 要求 enabled installation 保持 resident，不做 idle eviction；停用、删除、revision/generation 变化或 entitlement 撤销时停止并按需重建。
-  当前 Supervisor 已通过 completion callback 和有界指数 backoff 恢复意外退出的 worker；尚缺 jitter、全局重启并发上限和 Runtime 级 circuit breaker，
-  因而系统性依赖或宿主故障下的跨租户重启风暴抑制仍是 Cloud 激活门禁。
+  Supervisor 通过 completion callback、带 jitter 的有界指数 backoff 恢复意外退出的 worker。所有 restart launch 共用实例级并发槽；
+  在配置的失败窗口达到阈值后打开 Runtime 级 circuit breaker，冷却后只允许一个 half-open probe。
+  probe 必须完成初始化并持续稳定一个窗口后才能恢复其他 installation；未在 30 秒内 ready 的子进程会被取消回收。
+  生产候选环境仍需执行跨租户系统性故障注入，证明熔断、恢复和告警符合预期。
 - 子进程使用一次性 registration capability 向 Supervisor 注册；capability 由可信 desired state 派生并绑定完整 installation tuple，
   不是插件直接建立 Core Host connection，也不能只绑定 author/name/path。Supervisor/Core 据此注入 tenant context，
   丢弃插件 payload 中自带的 scope 字段。
@@ -178,6 +180,10 @@ plugin:
 - `PLUGIN__WORKER__MAX_PIDS`
 - `PLUGIN__WORKER__MAX_OPEN_FILES`
 - `PLUGIN__WORKER__MAX_FILE_SIZE_MB`
+- `PLUGIN__WORKER__MAX_CONCURRENT_RESTARTS`
+- `PLUGIN__WORKER__RESTART_FAILURE_THRESHOLD`
+- `PLUGIN__WORKER__RESTART_FAILURE_WINDOW_SECONDS`
+- `PLUGIN__WORKER__RESTART_CIRCUIT_OPEN_SECONDS`
 - `PLUGIN__WORKER__REQUIRE_HARD_LIMITS`
 
 Core 启动时校验配置并通过现有 `SET_RUNTIME_CONFIG` 下发不可变 `PluginWorkerPolicy`。

docs/multi-tenant/runtime-resource-audit-2026-07-28.md

@@ -17,7 +17,7 @@
 1. 在最终 Cloud 部署权限和 cgroup 拓扑下重复 nsjail、namespace 和 delegated cgroup v2 的 CPU、内存、swap、PID、文件句柄验证。本轮一次性 Linux 容器已经证明代码路径可工作，但普通容器和仅 `--privileged` 的 private cgroup namespace 都不满足条件。
 2. 为 Cloud Box 提供并验证硬文件系统 quota provider。普通 nsjail bind mount 不能证明总字节数和 inode 硬配额，当前严格 readiness 按设计会失败关闭。
 3. 使用最终生产配置分布继续做容量测试，并据此确定单实例 Workspace placement 上限。本轮真实 PostgreSQL 16 + RLS 启动测试已经覆盖 1,000 个各带 Provider、三类 Model、Bot、Pipeline、KnowledgeBase、MCP 和 Plugin setting 的 Workspace，启动加载耗时和 SQL 次数保持线性；5,000 Workspace 的合成三代替换探针也证明旧运行时会释放。仓库已新增可同时采集 Core/Plugin/Box HTTP、进程树和 cgroup v2 的 24 小时门禁工具，并在受 CPU、memory、swap、PID 硬限制的 Linux 容器中完成短时自检；但最终生产候选拓扑的 24 小时运行仍未执行。测试中的 fake adapter/requester/Plugin handler 仍不能替代真实平台 SDK、外部连接池和插件进程的容量数据；合法活跃租户本身仍会线性占用内存。
-SDK 已先行发布到分支提交 `67be7c332ded4c1afa131b28232590cf9728f817`，本提交集中的 LangBot
+SDK 已先行发布到分支提交 `a5a96b302a5808af84bbdd28833ce050176f87e9`，本提交集中的 LangBot
 `pyproject.toml` 和 `uv.lock` 已精确钉住该提交。最终镜像仍需按待验证清单记录并核对实际安装版本。
 
 ## 覆盖范围
@@ -93,6 +93,7 @@ SDK 已先行发布到分支提交 `67be7c332ded4c1afa131b28232590cf9728f817`，
 - 修复旧 QQ `repeat_seed('')` 空输入无限循环。
 - ZIP 校验/重打包、PIL、Base64、AES、JSON 解析、fsync probe、插件 artifact/依赖文件、Skill、S3、本地存储和维护目录扫描从事件循环移到线程。
 - 公开 Slack、QQ Official、HTTP Bot、公众号、WeCom/WeComCS 回调体显式限制为 1 MiB；JSON/XML 解码移出共享事件循环。QQ、DingTalk、Satori、WeCom AI 和 WeChatPad 网关帧同样设置 1 MiB 上限或在解码前拒绝超限消息；KOOK zlib 数据使用 10 MiB 解压后硬上限，阻断小压缩包制造的大内存解压。
+- HTTP Bot 的幂等键和 outbound session 现在均在写入前执行硬容量 admission；满额时只按固定 64 项预算检查最旧记录，不能先超限后整体清空，也不再在每条回复上扫描和排序全部 session。已有 session 继续 O(1) 访问，新 session 在没有可安全回收的空闲记录时失败关闭。
 - Dashboard、Embed 和 Plugin Runtime 的协议 JSON 编解码在线程执行；Dashboard/Embed 在接收端提交 terminal error 后为发送端保留有界 drain 窗口并使用内部 sentinel 唤醒，不会因“任一方向结束即取消”在撤权错误帧发出前关闭连接。
 - 租户配置的敏感词、内容忽略和群响应正则统一使用声明为直接依赖的 `regex` 引擎：最多 64 个 pattern、单 pattern 1,024 字符、输入 1 MiB、单次总匹配 CPU 预算 50 ms，并在线程中执行。超时、非法正则和替换放大均失败关闭；灾难性 `(a+)+$` 回归在 1 ms 测试预算内被中断。
 - 原生 `read/write/edit/glob/grep` 文件工具移出事件循环并继承 Workspace 阻塞预算。目录列举、递归 walk、grep 文件/总字符、单行、pattern、结果和 regex CPU 均有硬上限；glob 只用固定大小最小堆保留最新 100 项，不再先把全部命中路径驻留内存。Box 内执行的 glob/grep 脚本同样限制命中集合、扫描量和正则时间。
@@ -100,6 +101,9 @@ SDK 已先行发布到分支提交 `67be7c332ded4c1afa131b28232590cf9728f817`，
 - DashScope、TBox 等同步第三方 SDK 的调用和生成器迭代改为在线程执行；单个同步生成器最多消费 100,000 个事件。
 - Dashboard 和 Embed WebSocket 改为任一收发 task 结束即取消并等待另一方向，避免发送端退出后接收 task 永久阻塞；两方向 task 同时继承从认证结果或 RuntimeBot 得到的可信 Workspace 阻塞预算。
 - Plugin installation 生命周期全局串行化；不同租户的依赖 pip/nsjail 准备不会在安装高峰并发抢占 CPU。
+- Plugin installation 的意外退出除了每 installation 的 jittered exponential backoff，还经过 Runtime 全局 restart launch 并发槽和失败窗口熔断。
+  熔断冷却后只允许一个 half-open probe；probe 必须完成初始化并持续稳定后才恢复其他 installation。未在 30 秒内 ready 的 worker
+  会被取消回收。健康指标输出 active launch、窗口失败数、circuit 状态和累计打开次数，24 小时门禁把 circuit 打开或尾段启动槽不归零判为失败。
 - S3 同步 SDK 使用线程执行，并通过实例级 semaphore 限制并发；默认 `storage.s3.max_concurrency=16`，可通过实例配置和环境变量覆写。
 - Box 子进程 stderr 以 64 KiB 块读取，日志最多每秒输出 4 个摘录并汇总抑制数量，避免无换行或刷屏输出制造无界缓冲与日志放大。
 - Plugin worker 日志单行最多保留 64 KiB；Box managed-process stdout relay 以固定 64 KiB 块读取，不再依赖换行符，避免超长无换行输出触发 `StreamReader` limit 或堵塞子进程。
@@ -155,15 +159,15 @@ SDK 已先行发布到分支提交 `67be7c332ded4c1afa131b28232590cf9728f817`，
 | --- | --- |
 | LangBot Ruff + `git diff --check` | 通过 |
 | Plugin SDK Ruff + `git diff --check` | 通过 |
-| LangBot 全量测试（使用本地新 SDK，含 unit/integration/Box/E2E） | `2803 passed, 33 skipped` |
-| Plugin SDK 全量测试 | `1301 passed` |
+| LangBot 全量测试（使用远端精确钉住的新 SDK，含 unit/integration/Box/E2E） | `2808 passed, 33 skipped` |
+| Plugin SDK 全量测试 | `1312 passed` |
 | 真实 PostgreSQL 16 + pgvector 迁移/RLS/发布测试（严格资源告警） | `22 passed` |
 | 真实 PostgreSQL 16 + RLS populated Cloud 启动容量 | 500 Workspace `6.178s / CPU 3.026s`；当前 1,000 Workspace 复跑 `12.109s / CPU 5.967s` |
-| 当前 Core Dockerfile Linux 镜像构建与 `regex` 导入 | 通过，image SHA `8893a14053df`；仍使用待更新的旧 SDK pin |
+| 较早 Core Dockerfile Linux 镜像构建与 `regex` 导入 | 通过，image SHA `8893a14053df`；该镜像使用旧 SDK pin，已失效，最终候选必须重建 |
 | `ResourceWarning` + `PytestUnraisableExceptionWarning` 全量门禁 | Core 与 SDK 均通过，并已固化到 pytest 配置 |
 | Plugin SDK Box 专项测试（含全局扫描回归保护） | `669 passed` |
 | Docker Compose 渲染、Compose/Kubernetes YAML 解析与 diff 检查 | 通过 |
-| Cloud soak 门禁解析/采样/判定单元测试 | `23 passed` |
+| Cloud soak 门禁解析/采样/判定单元测试 | `24 passed` |
 | Core/Plugin SDK event-loop monitor 专项测试 | 两仓各 `7 passed`，包含真实 50 ms scheduler stall |
 | Cloud soak Linux 硬限制短时自检 | 通过；CPU `0.5`、memory+swap `256 MiB`、PID `128` 均从 cgroup v2 读回，冷却尾段 verdict `pass` |
 | Core 双阶段历史 churn 资源探针（使用本地新 SDK） | audit 通过，`11.275s` |

pyproject.toml

@@ -71,7 +71,7 @@ dependencies = [
     "chromadb>=1.0.0,<2.0.0",
     "qdrant-client (>=1.15.1,<2.0.0)",
     "pyseekdb==1.1.0.post3",
-    "langbot-plugin @ git+https://github.com/langbot-app/langbot-plugin-sdk.git@67be7c332ded4c1afa131b28232590cf9728f817",
+    "langbot-plugin @ git+https://github.com/langbot-app/langbot-plugin-sdk.git@a5a96b302a5808af84bbdd28833ce050176f87e9",
     "asyncpg>=0.30.0",
     "line-bot-sdk>=3.19.0",
     "matrix-nio>=0.25.2",

scripts/cloud_runtime_soak.py

@@ -72,7 +72,13 @@ REJECTION_SUFFIXES = (
     '.blocking_executor.global_rejected_total',
     '.blocking_executor.scope_rejected_total',
 )
-PENDING_SUFFIX = '.blocking_executor.pending'
+RUNTIME_FAILURE_COUNTER_SUFFIXES = ('.restart_coordinator.circuit_open_total',)
+DRAIN_GAUGE_SUFFIXES = (
+    '.blocking_executor.pending',
+    '.restart_coordinator.active_launches',
+    '.restart_coordinator.half_open_probe_inflight',
+    '.restart_coordinator.open_remaining_seconds',
+)
 TRANSIENT_GAUGE_SUFFIXES = (
     '.resources.telemetry_tasks',
     '.resources.query_pool.queued',
@@ -714,10 +720,22 @@ def evaluate_gate(
                         failures.append(f'{target_id} increased {metric} by {delta:g}')
                     elif delta is not None and delta < 0:
                         failures.append(f'{target_id} monotonic counter {metric} reset; the runtime may have restarted')
-            for metric in sorted(key for key in metric_keys if key.endswith(PENDING_SUFFIX)):
-                values = [sample.metrics[metric] for sample in tail_samples if metric in sample.metrics]
-                if values and min(values) > 0:
-                    failures.append(f'{target_id} kept {metric} above zero for the entire tail')
+            for suffix in RUNTIME_FAILURE_COUNTER_SUFFIXES:
+                for metric in sorted(key for key in metric_keys if key.endswith(suffix)):
+                    delta = _counter_delta(
+                        state.baseline_metrics or first,
+                        state.last_metrics or last,
+                        metric,
+                    )
+                    if delta is not None and delta > 0:
+                        failures.append(f'{target_id} increased {metric} by {delta:g}')
+                    elif delta is not None and delta < 0:
+                        failures.append(f'{target_id} monotonic counter {metric} reset; the runtime may have restarted')
+            for suffix in DRAIN_GAUGE_SUFFIXES:
+                for metric in sorted(key for key in metric_keys if key.endswith(suffix)):
+                    values = [sample.metrics[metric] for sample in tail_samples if metric in sample.metrics]
+                    if values and min(values) > 0:
+                        failures.append(f'{target_id} kept {metric} above zero for the entire tail')
             for suffix in TRANSIENT_GAUGE_SUFFIXES:
                 for metric in sorted(key for key in metric_keys if key.endswith(suffix)):
                     growth = _counter_delta(first, last, metric)

src/langbot/pkg/core/stages/load_config.py

@@ -32,6 +32,10 @@ _RUNTIME_POLICY_DEFAULTS = {
             'max_total_cpus': 8.0,
             'max_total_memory_mb': 8192,
             'max_installations': 10000,
+            'max_concurrent_restarts': 1,
+            'restart_failure_threshold': 8,
+            'restart_failure_window_seconds': 30.0,
+            'restart_circuit_open_seconds': 60.0,
             'require_hard_limits': False,
         }
     },

src/langbot/pkg/platform/sources/http_bot.py

@@ -28,6 +28,7 @@ See docs/platforms/http-bot.md for the full integration guide.
 from __future__ import annotations
 
 import asyncio
+import itertools
 import json
 import time
 import typing
@@ -64,12 +65,18 @@ _MAX_BODY = 1 * 1024 * 1024
 # Idempotency dedup window (seconds) and cap.
 _IDEMPOTENCY_TTL = 600
 _IDEMPOTENCY_MAX = 4096
+_IDEMPOTENCY_PRUNE_SCAN_MAX = 64
 _OUTBOUND_QUEUE_MAX = 100
 _OUTBOUND_IDLE_SECONDS = 60
 _OUTBOUND_STATE_MAX = 4096
+_OUTBOUND_PRUNE_SCAN_MAX = 64
 _INBOUND_TASK_MAX = 100
 
 
+class _OutboundStateCapacityError(RuntimeError):
+    """Raised when a new outbound session cannot be admitted safely."""
+
+
 class _SessionOutbound:
     """Per-session outbound state: ordered delivery queue + sequence counter."""
 
@@ -185,14 +192,34 @@ class HttpBotAdapter(abstract_platform_adapter.AbstractMessagePlatformAdapter):
         status, code = _ERR[kind]
         return quart.jsonify({'code': code, 'msg': detail or kind, 'data': None}), status
 
-    def _prune_idempotency(self) -> None:
-        now = time.time()
-        if len(self.idempotency_cache) > _IDEMPOTENCY_MAX:
-            self.idempotency_cache.clear()
-            return
-        expired = [k for k, ts in self.idempotency_cache.items() if now - ts > _IDEMPOTENCY_TTL]
-        for k in expired:
-            self.idempotency_cache.pop(k, None)
+    def _reserve_idempotency_key(self, key: str) -> str:
+        """Reserve a key without allowing unbounded state or full-map scans."""
+        now = time.monotonic()
+        accepted_at = self.idempotency_cache.get(key)
+        if accepted_at is not None:
+            if now - accepted_at <= _IDEMPOTENCY_TTL:
+                return 'duplicate'
+            self.idempotency_cache.pop(key, None)
+
+        if len(self.idempotency_cache) >= _IDEMPOTENCY_MAX:
+            self._prune_idempotency(now)
+        if len(self.idempotency_cache) >= _IDEMPOTENCY_MAX:
+            return 'overloaded'
+
+        self.idempotency_cache[key] = now
+        return 'accepted'
+
+    def _prune_idempotency(self, now: float | None = None) -> None:
+        """Remove at most a fixed number of oldest expired keys."""
+        current_time = time.monotonic() if now is None else now
+        oldest = itertools.islice(
+            self.idempotency_cache.items(),
+            _IDEMPOTENCY_PRUNE_SCAN_MAX,
+        )
+        for key, accepted_at in list(oldest):
+            if current_time - accepted_at <= _IDEMPOTENCY_TTL:
+                break
+            self.idempotency_cache.pop(key, None)
 
     def _start_inbound_task(self, coro: typing.Coroutine) -> asyncio.Task | None:
         self.inbound_tasks = {task for task in self.inbound_tasks if not task.done()}
@@ -299,10 +326,11 @@ class HttpBotAdapter(abstract_platform_adapter.AbstractMessagePlatformAdapter):
         # Idempotency.
         idem = request.headers.get(signing.HEADER_IDEMPOTENCY)
         if idem:
-            self._prune_idempotency()
-            if idem in self.idempotency_cache:
+            idempotency_result = self._reserve_idempotency_key(idem)
+            if idempotency_result == 'duplicate':
                 return self._err('duplicate', 'idempotency key already accepted')
-            self.idempotency_cache[idem] = time.time()
+            if idempotency_result == 'overloaded':
+                return self._err('overloaded', 'idempotency capacity reached; retry later')
 
         try:
             event, session_id, session_type, message_id = self._build_event(data)
@@ -397,8 +425,7 @@ class HttpBotAdapter(abstract_platform_adapter.AbstractMessagePlatformAdapter):
         return ''
 
     def _next_sequence(self, session_id: str, is_final: bool) -> int:
-        self._prune_outbound_states()
-        state = self.outbound_states.setdefault(session_id, _SessionOutbound())
+        state = self._outbound_state(session_id)
         state.last_active = time.monotonic()
         if state.last_was_final:
             state.sequence = 1
@@ -407,36 +434,42 @@ class HttpBotAdapter(abstract_platform_adapter.AbstractMessagePlatformAdapter):
         state.last_was_final = is_final
         return state.sequence
 
+    def _outbound_state(self, session_id: str) -> _SessionOutbound:
+        state = self.outbound_states.get(session_id)
+        if state is not None:
+            # Dicts retain insertion order. Moving active sessions to the end
+            # keeps bounded admission-time pruning focused on old entries.
+            self.outbound_states.pop(session_id)
+            self.outbound_states[session_id] = state
+            return state
+
+        if len(self.outbound_states) >= _OUTBOUND_STATE_MAX:
+            self._prune_outbound_states()
+        if len(self.outbound_states) >= _OUTBOUND_STATE_MAX:
+            raise _OutboundStateCapacityError(f'http_bot outbound session capacity reached ({_OUTBOUND_STATE_MAX})')
+
+        state = _SessionOutbound()
+        self.outbound_states[session_id] = state
+        return state
+
     def _prune_outbound_states(self) -> None:
         now = time.monotonic()
-        removable = [
-            (session_id, state)
-            for session_id, state in self.outbound_states.items()
-            if (state.worker is None or state.worker.done())
-            and state.queue.empty()
-            and now - state.last_active >= _OUTBOUND_IDLE_SECONDS
-        ]
-        for session_id, state in removable:
-            if self.outbound_states.get(session_id) is state:
-                self.outbound_states.pop(session_id, None)
-
-        overflow = len(self.outbound_states) - _OUTBOUND_STATE_MAX
-        if overflow <= 0:
-            return
-        idle = sorted(
-            (
-                (session_id, state)
-                for session_id, state in self.outbound_states.items()
-                if (state.worker is None or state.worker.done()) and state.queue.empty()
-            ),
-            key=lambda item: item[1].last_active,
+        oldest = itertools.islice(
+            self.outbound_states.items(),
+            _OUTBOUND_PRUNE_SCAN_MAX,
         )
-        for session_id, state in idle[:overflow]:
+        for session_id, state in list(oldest):
+            if (
+                (state.worker is not None and not state.worker.done())
+                or not state.queue.empty()
+                or now - state.last_active < _OUTBOUND_IDLE_SECONDS
+            ):
+                continue
             if self.outbound_states.get(session_id) is state:
                 self.outbound_states.pop(session_id, None)
 
     async def _enqueue_callback(self, session_id: str, payload: dict) -> None:
-        state = self.outbound_states.setdefault(session_id, _SessionOutbound())
+        state = self._outbound_state(session_id)
         state.last_active = time.monotonic()
         if state.worker is None or state.worker.done():
             state.worker = asyncio.create_task(self._outbound_worker(session_id, state))

src/langbot/pkg/plugin/connector.py

@@ -239,6 +239,10 @@ class PluginRuntimeConnector(ManagedRuntimeConnector):
             'max_total_cpus',
             'max_total_memory_mb',
             'max_installations',
+            'max_concurrent_restarts',
+            'restart_failure_threshold',
+            'restart_failure_window_seconds',
+            'restart_circuit_open_seconds',
         ):
             if field_name in PluginWorkerPolicy.model_fields and field_name in worker:
                 policy_data[field_name] = worker.get(field_name)

src/langbot/templates/config.yaml

@@ -227,6 +227,14 @@ plugin:
         # Includes disabled and historical installation fences retained to
         # reject stale desired-state replay.
         max_installations: 10000
+        # Restart storms are globally serialized by default. Repeated
+        # unexpected exits within the configured window open a Runtime-wide
+        # circuit; one half-open probe must remain stable before other
+        # installations may restart.
+        max_concurrent_restarts: 1
+        restart_failure_threshold: 8
+        restart_failure_window_seconds: 30.0
+        restart_circuit_open_seconds: 60.0
         # Cloud shared Runtime sets this to true and fails closed unless
         # delegated cgroup v2 controllers are available.
         require_hard_limits: false

tests/unit_tests/core/test_load_config.py

@@ -298,6 +298,10 @@ class TestApplyEnvOverridesToConfig:
                     'max_pids': 128,
                     'max_open_files': 256,
                     'max_file_size_mb': 512,
+                    'max_concurrent_restarts': 1,
+                    'restart_failure_threshold': 8,
+                    'restart_failure_window_seconds': 30.0,
+                    'restart_circuit_open_seconds': 60.0,
                 }
             },
             'mcp': {'stdio': {'enabled': True}},
@@ -308,6 +312,10 @@ class TestApplyEnvOverridesToConfig:
             'PLUGIN__WORKER__MAX_PIDS': '64',
             'PLUGIN__WORKER__MAX_OPEN_FILES': '128',
             'PLUGIN__WORKER__MAX_FILE_SIZE_MB': '256',
+            'PLUGIN__WORKER__MAX_CONCURRENT_RESTARTS': '2',
+            'PLUGIN__WORKER__RESTART_FAILURE_THRESHOLD': '12',
+            'PLUGIN__WORKER__RESTART_FAILURE_WINDOW_SECONDS': '45.5',
+            'PLUGIN__WORKER__RESTART_CIRCUIT_OPEN_SECONDS': '90.0',
             'MCP__STDIO__ENABLED': 'false',
         }
 
@@ -320,6 +328,10 @@ class TestApplyEnvOverridesToConfig:
             'max_pids': 64,
             'max_open_files': 128,
             'max_file_size_mb': 256,
+            'max_concurrent_restarts': 2,
+            'restart_failure_threshold': 12,
+            'restart_failure_window_seconds': 45.5,
+            'restart_circuit_open_seconds': 90.0,
         }
         assert result['mcp']['stdio']['enabled'] is False
 

tests/unit_tests/platform/test_http_bot_tenancy.py

@@ -1,6 +1,7 @@
 from __future__ import annotations
 
 import asyncio
+import time
 from types import SimpleNamespace
 
 import pytest
@@ -92,3 +93,58 @@ async def test_http_bot_bounds_inbound_listener_tasks(monkeypatch):
     await first
     await asyncio.sleep(0)
     assert adapter.inbound_tasks == set()
+
+
+def test_http_bot_outbound_state_has_a_hard_capacity(monkeypatch):
+    monkeypatch.setattr(http_bot_module, '_OUTBOUND_STATE_MAX', 2)
+    monkeypatch.setattr(http_bot_module, '_OUTBOUND_PRUNE_SCAN_MAX', 2)
+    adapter = _adapter(SimpleNamespace(), None)
+    first = adapter._outbound_state('first')
+    second = adapter._outbound_state('second')
+    first.queue.put_nowait({})
+    second.queue.put_nowait({})
+
+    with pytest.raises(RuntimeError, match='outbound session capacity reached'):
+        adapter._next_sequence('third', is_final=True)
+
+    assert len(adapter.outbound_states) == 2
+    assert adapter._next_sequence('first', is_final=True) == 1
+
+
+def test_http_bot_outbound_state_pruning_is_bounded_and_reclaims_stale(monkeypatch):
+    monkeypatch.setattr(http_bot_module, '_OUTBOUND_STATE_MAX', 2)
+    monkeypatch.setattr(http_bot_module, '_OUTBOUND_PRUNE_SCAN_MAX', 1)
+    monkeypatch.setattr(http_bot_module, '_OUTBOUND_IDLE_SECONDS', 10)
+    adapter = _adapter(SimpleNamespace(), None)
+    stale = adapter._outbound_state('stale')
+    stale.last_active = time.monotonic() - 11
+    adapter._outbound_state('active')
+
+    assert adapter._next_sequence('replacement', is_final=True) == 1
+    assert set(adapter.outbound_states) == {'active', 'replacement'}
+
+
+def test_http_bot_idempotency_cache_has_a_hard_capacity(monkeypatch):
+    monkeypatch.setattr(http_bot_module, '_IDEMPOTENCY_MAX', 2)
+    monkeypatch.setattr(http_bot_module, '_IDEMPOTENCY_PRUNE_SCAN_MAX', 1)
+    adapter = _adapter(SimpleNamespace(), None)
+
+    assert adapter._reserve_idempotency_key('first') == 'accepted'
+    assert adapter._reserve_idempotency_key('second') == 'accepted'
+    assert adapter._reserve_idempotency_key('third') == 'overloaded'
+    assert len(adapter.idempotency_cache) == 2
+    assert adapter._reserve_idempotency_key('first') == 'duplicate'
+
+
+def test_http_bot_idempotency_cache_reclaims_expired_oldest(monkeypatch):
+    monkeypatch.setattr(http_bot_module, '_IDEMPOTENCY_MAX', 2)
+    monkeypatch.setattr(http_bot_module, '_IDEMPOTENCY_PRUNE_SCAN_MAX', 1)
+    monkeypatch.setattr(http_bot_module, '_IDEMPOTENCY_TTL', 10)
+    adapter = _adapter(SimpleNamespace(), None)
+    adapter.idempotency_cache = {
+        'expired': time.monotonic() - 11,
+        'active': time.monotonic(),
+    }
+
+    assert adapter._reserve_idempotency_key('replacement') == 'accepted'
+    assert set(adapter.idempotency_cache) == {'active', 'replacement'}

tests/unit_tests/plugin/test_connector_ping.py

@@ -337,6 +337,10 @@ def test_worker_policy_is_loaded_only_from_instance_configuration():
                         'max_pids': 64,
                         'max_open_files': 128,
                         'max_file_size_mb': 32,
+                        'max_concurrent_restarts': 2,
+                        'restart_failure_threshold': 12,
+                        'restart_failure_window_seconds': 45,
+                        'restart_circuit_open_seconds': 90,
                         'require_hard_limits': True,
                     },
                     # A plugin-controlled value at any other path is ignored.
@@ -354,6 +358,10 @@ def test_worker_policy_is_loaded_only_from_instance_configuration():
     assert policy.max_pids == 64
     assert policy.max_open_files == 128
     assert policy.max_file_size_mb == 32
+    assert policy.max_concurrent_restarts == 2
+    assert policy.restart_failure_threshold == 12
+    assert policy.restart_failure_window_seconds == 45
+    assert policy.restart_circuit_open_seconds == 90
     assert policy.require_hard_limits is True
 
 

tests/unit_tests/utils/test_cloud_runtime_soak.py

@@ -212,7 +212,11 @@ def test_read_endpoint_snapshot_flattens_resource_metrics() -> None:
                     'blocking_executor': {
                         'pending': 0,
                         'global_rejected_total': 2,
-                    }
+                    },
+                    'restart_coordinator': {
+                        'active_launches': 1,
+                        'circuit_open_total': 3,
+                    },
                 },
             }
         )
@@ -226,6 +230,8 @@ def test_read_endpoint_snapshot_flattens_resource_metrics() -> None:
     assert metrics['http.ok'] == 1
     assert metrics['body.resources.blocking_executor.pending'] == 0
     assert metrics['body.resources.blocking_executor.global_rejected_total'] == 2
+    assert metrics['body.resources.restart_coordinator.active_launches'] == 1
+    assert metrics['body.resources.restart_coordinator.circuit_open_total'] == 3
 
 
 def test_read_endpoint_snapshot_fails_closed_on_not_ready() -> None:
@@ -382,6 +388,44 @@ def test_evaluate_gate_detects_executor_rejection_and_stuck_pending() -> None:
     assert any('pending above zero' in failure for failure in result.failures)
 
 
+def test_evaluate_gate_detects_restart_circuit_and_stuck_launch() -> None:
+    prefix = 'body.resources.restart_coordinator'
+    state = _state(
+        'endpoint',
+        [
+            _sample(
+                0,
+                **{
+                    f'{prefix}.active_launches': 1,
+                    f'{prefix}.half_open_probe_inflight': 1,
+                    f'{prefix}.open_remaining_seconds': 60,
+                    f'{prefix}.circuit_open_total': 0,
+                },
+            ),
+            _sample(
+                60,
+                **{
+                    f'{prefix}.active_launches': 1,
+                    f'{prefix}.half_open_probe_inflight': 1,
+                    f'{prefix}.open_remaining_seconds': 1,
+                    f'{prefix}.circuit_open_total': 1,
+                },
+            ),
+        ],
+    )
+
+    result = soak.evaluate_gate(
+        [state],
+        analysis_start_seconds=0,
+        thresholds=_thresholds(),
+    )
+
+    assert any('circuit_open_total by 1' in failure for failure in result.failures)
+    assert any('active_launches above zero' in failure for failure in result.failures)
+    assert any('half_open_probe_inflight above zero' in failure for failure in result.failures)
+    assert any('open_remaining_seconds above zero' in failure for failure in result.failures)
+
+
 def test_evaluate_gate_detects_event_loop_stall_and_sustained_lag() -> None:
     prefix = 'body.resources.event_loop'
     samples = [

uv.lock

@@ -2116,7 +2116,7 @@ requires-dist = [
     { name = "ebooklib", specifier = ">=0.18" },
     { name = "gewechat-client", specifier = ">=0.1.5" },
     { name = "html2text", specifier = ">=2024.2.26" },
-    { name = "langbot-plugin", git = "https://github.com/langbot-app/langbot-plugin-sdk.git?rev=67be7c332ded4c1afa131b28232590cf9728f817" },
+    { name = "langbot-plugin", git = "https://github.com/langbot-app/langbot-plugin-sdk.git?rev=a5a96b302a5808af84bbdd28833ce050176f87e9" },
     { name = "langchain", specifier = ">=1.3.9" },
     { name = "langchain-core", specifier = ">=1.3.3" },
     { name = "langchain-text-splitters", specifier = ">=1.1.2" },
@@ -2183,7 +2183,7 @@ dev = [
 [[package]]
 name = "langbot-plugin"
 version = "0.4.18"
-source = { git = "https://github.com/langbot-app/langbot-plugin-sdk.git?rev=67be7c332ded4c1afa131b28232590cf9728f817#67be7c332ded4c1afa131b28232590cf9728f817" }
+source = { git = "https://github.com/langbot-app/langbot-plugin-sdk.git?rev=a5a96b302a5808af84bbdd28833ce050176f87e9#a5a96b302a5808af84bbdd28833ce050176f87e9" }
 dependencies = [
     { name = "aiofiles" },
     { name = "aiohttp" },