fix(cloud): render empty skills without sandbox grant

Co-authored-by: dadachann <185672915+dadachann@users.noreply.github.com>

.github/workflows/frontend-tests.yml

@@ -37,6 +37,10 @@ jobs:
         working-directory: web
         run: pnpm install --frozen-lockfile
 
+      - name: Run frontend unit tests
+        working-directory: web
+        run: pnpm test:unit
+
       - name: Install Playwright browsers
         working-directory: web
         run: pnpm exec playwright install --with-deps chromium

.github/workflows/test-migrations.yml

@@ -44,7 +44,9 @@ jobs:
     runs-on: ubuntu-latest
     services:
       postgres:
-        image: postgres:16
+        # Release migration 0013 installs the pgvector extension in the shared
+        # business database; CI must exercise the same extension availability.
+        image: pgvector/pgvector:pg16
         env:
           POSTGRES_USER: langbot
           POSTGRES_PASSWORD: langbot
@@ -75,4 +77,10 @@ jobs:
       - name: Run PostgreSQL migration tests
         env:
           TEST_POSTGRES_URL: postgresql+asyncpg://langbot:langbot@localhost:5432/langbot_test
-        run: uv run pytest tests/integration/persistence/test_migrations_postgres.py -q --tb=short
+        run: >-
+          uv run pytest
+          tests/integration/persistence/test_migrations_postgres.py
+          tests/integration/persistence/test_pgvector_postgres.py
+          tests/integration/persistence/test_release_migration_postgres.py
+          tests/integration/persistence/test_plugin_identity_migration.py
+          -q --tb=short

.gitignore

@@ -48,6 +48,7 @@ coverage.xml
 .coverage
 src/langbot/web/
 testsdk/
+.qa/
 
 # Build artifacts
 /dist

AGENTS.md

@@ -1,160 +1,105 @@
 # AGENTS.md
 
-This file guides code agents (Claude Code, GitHub Copilot, OpenAI Codex, etc.) working in the LangBot project. `CLAUDE.md` is a symlink to this file.
+This file guides code agents working in the LangBot main repository. `CLAUDE.md` is a symlink to this file.
 
-## Project Overview
+Read `ARCHITECTURE.md` before non-trivial backend, frontend, runtime, plugin, Box, MCP, persistence, or cross-repo SDK changes. This file is the working checklist; `ARCHITECTURE.md` is the system map.
 
-LangBot is an open-source, LLM-native instant-messaging bot development platform. It aims to provide an out-of-the-box IM bot development experience with Agent, RAG, MCP and other LLM application capabilities, supporting mainstream global IM platforms and exposing rich APIs for custom development.
+## Quick Facts
 
-LangBot has a comprehensive web frontend — almost every operation can be performed through it.
+- Python backend: `>=3.11,<4.0`, dependencies managed by `uv`.
+- Frontend: `web/` is Vite + React Router 7 + shadcn/ui + Tailwind, managed by `pnpm`.
+- Backend framework: Quart served by Hypercorn on `api.port`, default `5300`.
+- Frontend dev server: `web/` on `3000`, with `VITE_API_BASE_URL` pointing at the backend.
+- Plugin/Box/runtime contracts live in sibling repo `langbot-plugin-sdk`, pinned as `langbot-plugin` in `pyproject.toml`.
 
-- **Python**: `>=3.11,<4.0`, dependencies managed by `uv`. Package version is in `pyproject.toml`.
-- **Frontend**: `web/` is a **Vite + React Router 7 + shadcn/ui + Tailwind CSS** SPA, managed by `pnpm`. (Note: this is NOT Next.js — the `dev` script is `vite`.)
-- **Backend framework**: Quart (the async flavour of Flask). The HTTP API and the pre-built web UI are both served by the backend on `http://127.0.0.1:5300`.
-
-## Repository Layout
-
-```
-LangBot/
-├── main.py                     # Entrypoint shim -> langbot.__main__.main()
-├── pyproject.toml              # Python project + deps (uv), pins langbot-plugin==<x.y.z>
-├── src/langbot/
-│   ├── __main__.py             # Real entrypoint, CLI args (--standalone-runtime, --standalone-box, --debug)
-│   ├── pkg/                    # Core backend package
-│   │   ├── api/                # HTTP API controllers + services (Quart)
-│   │   ├── core/               # App bootstrap, stages, task manager
-│   │   ├── platform/           # IM platform adapters, bot managers, session managers
-│   │   ├── provider/           # LLM providers, requesters, tool providers
-│   │   ├── pipeline/           # Pipelines, stages, query pool
-│   │   ├── plugin/             # Bridge connecting LangBot to the plugin runtime (see below)
-│   │   ├── box/                # Code-sandbox subsystem (Docker / nsjail / E2B backends)
-│   │   ├── skill/              # Skill subsystem
-│   │   ├── rag/ , vector/      # RAG + vector store
-│   │   ├── command/            # Built-in commands
-│   │   ├── persistence/        # ORM models + Alembic migrations (SQLite & PostgreSQL)
-│   │   ├── storage/            # Object/file storage abstractions
-│   │   ├── config/, entity/, discover/, utils/, telemetry/, survey/
-│   ├── libs/                   # Vendored SDKs (qq_official_api, wecom_api, etc.)
-│   └── templates/              # Config/component templates (e.g. templates/config.yaml)
-├── web/                        # Frontend SPA (Vite + React Router 7 + shadcn + Tailwind)
-└── docker/                     # docker-compose deployment files
-```
-
-## Development Environment Setup
-
-Full guide lives in the wiki: **["开发配置" / Dev Config](https://docs.langbot.app/zh/develop/dev-config)**. Summary:
-
-### Backend
-
-```bash
-pip install uv
-uv sync --dev          # uv creates a .venv/ for you; point your editor's interpreter at it
-uv run main.py         # serves API + web UI on http://127.0.0.1:5300
-```
-
-On first run the config file is generated at `data/config.yaml`. DB is SQLite by default (zero setup); PostgreSQL is supported. Migrations run automatically on startup.
-
-### Frontend
-
-Requires Node.js + [pnpm](https://pnpm.io/installation).
-
-```bash
-cd web
-cp .env.example .env   # Windows: copy .env.example .env
-pnpm install
-pnpm dev               # http://127.0.0.1:3000  (npm install / npm run dev also work)
-```
-
-`pnpm dev` reads `VITE_API_BASE_URL` from `web/.env` so the dev frontend can reach the backend on port `5300`. In production the frontend is pre-built into static files served by the backend on the same origin.
-
-### Code formatting
-
-The repo runs lint + format checks in CI. Install the pre-commit hooks so the same checks run locally before each commit:
+## Essential Commands
 
 ```bash
+uv sync --dev
+uv run main.py
 uv run pre-commit install
+
+cd web
+pnpm install
+pnpm dev
+pnpm build
 ```
 
-## Plugin System
-
-LangBot's plugin system (Plugin SDK, CLI `lbp`, Plugin Runtime, and the shared entity/API definitions) lives in a **separate repository**: [`langbot-plugin-sdk`](https://github.com/langbot-app/langbot-plugin-sdk). LangBot depends on it via the pinned `langbot-plugin` package in `pyproject.toml`.
-
-### Architecture (what to know inside this repo)
-
-- Plugins run as independent processes managed by the **Plugin Runtime**. The Runtime supports two control transports: `stdio` and `websocket`.
-- When LangBot is started directly by a user (not in a container), it spawns and connects to the Runtime over **stdio** (lightweight/personal use).
-- When LangBot runs in a container, it connects to a standalone Runtime over **WebSocket** (production).
-- The bridge code lives in `src/langbot/pkg/plugin/` (`connector.py`, `handler.py`).
-- Relevant config (`data/config.yaml`): `plugin.runtime_ws_url` (e.g. `ws://langbot_plugin_runtime:5400/control/ws`). Start LangBot with `--standalone-runtime` to make it connect to an externally-launched Runtime over WebSocket instead of spawning one over stdio.
-
-### Debugging the Plugin Runtime / CLI / SDK
-
-This is documented in detail in the **SDK repo's `AGENTS.md`** and in the wiki page **["调试插件运行时、CLI、SDK" / Plugin Runtime](https://docs.langbot.app/zh/develop/plugin-runtime)**. The short version:
-
-- Clone `LangBot` and `langbot-plugin-sdk` as siblings under one parent dir so the editor resolves shared entities.
-- Start a standalone Runtime from the SDK repo: `uv run --no-sync lbp rt` (control port `5400`, debug port `5401`).
-- To make LangBot use a locally-modified SDK: from the SDK dir, with LangBot's `.venv` active, run `uv pip install .`, then launch LangBot with `uv run --no-sync main.py --standalone-runtime` (keep `--no-sync` so your local SDK isn't overwritten).
-
-### Debugging the Box (sandbox) runtime
-
-The Box subsystem (`src/langbot/pkg/box/`) is the code sandbox. It picks the first available backend among **Docker / nsjail / E2B**. The standalone Box runtime is launched via the SDK CLI: `lbp box`. Backend selection details, the `lbp box` flags, and the SDK-side architecture are documented in the SDK repo's `AGENTS.md`.
-
-Relevant config (`data/config.yaml`, `box:` section): `box.enabled` (master switch — disabling it also disables the native sandbox tools, skill add/edit, and stdio-mode MCP servers), `box.backend` (`'local'` = Docker/nsjail auto-pick, or `'docker'` / `'nsjail'` / `'e2b'`; also settable via `BOX__BACKEND`), and `box.runtime.endpoint` (external Box runtime base URL, e.g. `ws://127.0.0.1:5410`; empty = local auto-managed runtime). Like the plugin runtime, LangBot can connect to an externally-launched Box runtime by setting that endpoint and starting with `--standalone-box`.
-
-> A common false "No supported sandbox backend (Docker / nsjail / E2B) is available" comes from Docker being installed and running but the current user not being in the `docker` group → `docker info` gets `permission denied` on the socket. Fix: `sudo usermod -aG docker <user>` and restart the backend in a shell that has the new group.
-
-## Development Standards
-
-- LangBot is a global project: **all code comments and docstrings must be in English**, and every user-facing string must support **i18n** (`en_US` + `zh_Hans` at minimum, plus `ja_JP` where the repo already has it).
-- LangBot is adopted in both toC and toB scenarios — always consider compatibility and security.
-- **Commit message format**: `<type>(<scope>): <subject>`
-  - `type`: one of `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`, `chore`, etc.
-  - `scope`: the affected package/module/file/class.
-  - `subject`: concise description of the change.
-
-### Database migrations (Alembic)
-
-LangBot uses [Alembic](https://alembic.sqlalchemy.org/) for migrations, supporting both SQLite and PostgreSQL from a single set of scripts. Migration files live in `src/langbot/pkg/persistence/alembic/versions/`.
-
-If you change ORM model definitions, generate a migration:
+Useful focused tests:
 
 ```bash
-# Run from the project root (requires data/config.yaml to exist)
-uv run python -m langbot.pkg.persistence.alembic_runner autogenerate "description of your change"
+uv run pytest tests/unit_tests -q
+uv run pytest tests/integration -q
+uv run pytest tests/integration/persistence -q
+uv run pytest tests/manual/mcp_smoke.py
+
+cd web
+pnpm lint
+pnpm test:e2e
 ```
 
-Review and edit the generated script before committing. Migrations execute automatically on startup. `autogenerate` detects schema changes (add/drop columns, tables, type changes) but **data migrations** (e.g. mutating JSON field contents) must be hand-written into the generated script. `env.py` sets `render_as_batch=True`, so SQLite's ALTER TABLE limits are handled automatically — no need to branch per database. More in the wiki ["开发配置"](https://docs.langbot.app/zh/develop/dev-config#数据库迁移).
+Run the narrowest useful test first, then broader checks when confidence is needed.
 
-When writing a migration, follow these rules:
+## Where to Look
 
-- **Revision id ≤ 32 characters.** PostgreSQL stores `alembic_version.version_num` as `varchar(32)`; a longer id raises `StringDataRightTruncationError` at runtime. Prefer short, descriptive ids like `0005_add_llm_context_length`.
-- **Guard every operation against missing tables/columns.** Fresh installs build the schema via `create_all()` and then stamp the Alembic baseline, so a migration may run against a table that already has the change — or, in tests, against an empty database. Check `inspector.get_table_names()` / `inspector.get_columns(...)` before `add_column` / `drop_column`, mirroring the existing migrations.
-- **Keep a single linear head.** Chain `down_revision` to the current head; do not create branches. Run the migration tests after adding one: `uv run pytest tests/integration/persistence/ -q` (the PostgreSQL test needs a running PG via `TEST_POSTGRES_URL`).
+- Architecture map: `ARCHITECTURE.md`.
+- Dev environment guide: https://docs.langbot.app/zh/develop/dev-config.
+- Plugin runtime / CLI / SDK debugging: https://docs.langbot.app/zh/develop/plugin-runtime.
+- API-key auth: `docs/API_KEY_AUTH.md`.
+- Box deep-dive notes: `docs/review/box-architecture.md` and related files.
+- In-repo skills: `skills/` is the single source of truth for LangBot agent skills.
+- SDK repo: `../langbot-plugin-sdk/` when changing shared entities, plugin APIs, action protocol, `lbp rt`, or `lbp box`.
 
-> **Legacy migration system (deprecated — do not extend).** The old 3.x migration system under `src/langbot/pkg/persistence/migrations/` (`DBMigration` subclasses in `dbmXXX_*.py`, run from `pkg/persistence/mgr.py`) is **frozen**. Do **not** add new `dbmXXX_*.py` files. The chain is capped at `required_database_version = 25` (`pkg/utils/constants.py`); those files only exist to upgrade pre-existing 3.x databases up to the Alembic baseline and are kept read-only. All new schema changes go through Alembic.
+## Cross-Repo SDK Work
 
-## Agent-Facing Surfaces (MCP + Skills)
+When changing SDK contracts used by LangBot:
 
-LangBot is built to be **agent-friendly**. Three surfaces let AI agents work
-with LangBot, and they MUST be kept in lockstep with the HTTP API:
+```bash
+# from langbot-plugin-sdk, with LangBot's .venv active
+uv pip install .
 
-1. **MCP server** — `src/langbot/pkg/api/mcp/` exposes a curated subset of the
-   API as MCP tools at `/mcp` (API-key authenticated, including the
-   `api.global_api_key` from config.yaml). `server.py` defines the tools (they
-   call the service layer directly); `mount.py` is the ASGI dispatcher.
-2. **In-repo skills** — `skills/` is the **single source of truth** for agent
-   skills (plugin/core/deploy/e2e/MCP-ops). Docs and the landing page link here
-   rather than embedding their own copies.
-3. **API-key auth** — `api.global_api_key` (config.yaml) authenticates the API
-   and MCP without a login session; see `docs/API_KEY_AUTH.md`.
+# from LangBot, preserve the locally installed SDK
+uv run --no-sync main.py
+```
 
-> **Maintenance rule (important).** When you add, remove, or change an HTTP API
-> endpoint that should be agent-accessible, you MUST update **both** the matching
-> MCP tool in `src/langbot/pkg/api/mcp/server.py` **and** the relevant skill under
-> `skills/` (especially `skills/skills/langbot-mcp-ops`). The API, the MCP tool
-> surface, and the skills are one system — drift between them is a bug.
+For standalone runtime debugging:
 
-## Some Principles
+```bash
+# in langbot-plugin-sdk
+uv run --no-sync lbp rt
+uv run --no-sync lbp box
+
+# in LangBot
+uv run --no-sync main.py --standalone-runtime
+uv run --no-sync main.py --standalone-box
+```
+
+Config keys to verify in `data/config.yaml` / `src/langbot/templates/config.yaml`:
+
+- Plugin runtime: `plugin.runtime_ws_url`, default Docker host `langbot_plugin_runtime:5400/control/ws`.
+- Box runtime: `box.enabled`, `box.backend`, `box.runtime.endpoint`, Docker host `langbot_box:5410`.
+- API/MCP auth: `api.global_api_key`.
+
+## Change Rules
+
+- HTTP API changes that should be agent-accessible must update the matching MCP tool in `src/langbot/pkg/api/mcp/server.py` and the relevant skill under `skills/` in the same pass.
+- New schema changes use Alembic under `src/langbot/pkg/persistence/alembic/versions/`; do not add legacy `dbmXXX` migrations.
+- New platform behavior belongs in platform adapters only for platform translation; pipeline/business logic belongs in `pkg/pipeline/` or services.
+- User-facing strings must support i18n (`en_US`, `zh_Hans`; include `ja_JP` where the repo already does).
+- Code comments and docstrings must be English.
+- Keep compatibility and security in mind; LangBot is used in both self-hosted/community and toB deployments.
+- Commit message format: `<type>(<scope>): <subject>`.
+
+## Runtime Pitfalls
+
+- Local stdio Plugin Runtime disconnects do not auto-reconnect; restart LangBot if that path breaks.
+- Orphan runtime processes on `5400`/`5401` commonly break plugin debugging.
+- Use `uv run --no-sync` after locally installing the SDK, or `uv` may restore the pinned package.
+- A false Box “no backend” often means Docker is running but the current user lacks Docker socket permission.
+- Do not confuse external MCP servers LangBot connects to (`pkg/provider/tools/loaders/mcp.py`) with LangBot's own `/mcp` server (`pkg/api/mcp/`).
+- `CLAUDE.md` is a symlink to this file; edit `AGENTS.md`, not the symlink.
+
+## Principles
 
 - Keep it simple, stupid.
 - Entities should not be multiplied unnecessarily.

ARCHITECTURE.md

@@ -0,0 +1,256 @@
+# Architecture
+
+This document is a map of LangBot's moving parts. It is intentionally more stable than a feature guide and more concrete than the README: when you need to change behavior, start here, then follow the file references into the code.
+
+For agent-specific working rules, see `AGENTS.md`. For plugin-runtime and Box-runtime implementation details, also read the sibling SDK repo: [`langbot-plugin-sdk`](https://github.com/langbot-app/langbot-plugin-sdk).
+
+## What LangBot Is
+
+LangBot is an open-source platform for building production IM bots backed by LLMs, agents, RAG, plugins, MCP tools, and a web management panel.
+
+At runtime, one LangBot process owns:
+
+- a Quart/Hypercorn HTTP service and the built web UI on `:5300`;
+- messaging-platform adapters such as Discord, Telegram, Slack, WeChat, QQ, WeCom, Lark, DingTalk, KOOK, LINE, Satori, Matrix, and HTTP/WebSocket bots;
+- a pipeline engine that turns inbound platform messages into LLM/tool/plugin work and replies;
+- persistence, storage, vector database, telemetry, monitoring, and configuration managers;
+- bridges to the Plugin Runtime and Box Runtime provided by `langbot-plugin-sdk`;
+- an MCP server at `/mcp` exposing a curated agent-facing subset of the service layer.
+
+## Repository Boundary
+
+LangBot is not a single-repo system.
+
+- `LangBot/` is the main product: backend, web UI, platform adapters, pipeline engine, HTTP API, MCP server, RAG, persistence, skills integration, and the bridge code that talks to runtimes.
+- `langbot-plugin-sdk/` is published as `langbot-plugin` and pinned in `LangBot/pyproject.toml`. It contains plugin developer APIs, shared entities, `lbp`, the Plugin Runtime (`lbp rt`), and the Box Runtime (`lbp box`).
+- Plugins import SDK APIs from `langbot_plugin.*`; the LangBot main process imports the same package for shared entities and runtime protocols.
+
+This split matters. If a change modifies SDK entities, component APIs, action protocols, `lbp rt`, or `lbp box`, verify the sibling SDK repo and install the local SDK into LangBot's virtualenv when testing cross-repo behavior.
+
+## Startup Path
+
+The process entrypoint is small and layered:
+
+1. `main.py` delegates to `langbot.__main__.main()`.
+2. `src/langbot/__main__.py` parses `--standalone-runtime`, `--standalone-box`, and `--debug`, checks dependencies, generates missing config/data files, and calls `pkg.core.boot.main()`.
+3. `pkg/core/boot.py` executes startup stages in order: `LoadConfigStage`, `GenKeysStage`, `SetupLoggerStage`, `BuildAppStage`, `ShowNotesStage`.
+4. `BuildAppStage` constructs the `Application` object by wiring managers, services, runtime connectors, and controllers.
+5. `Application.run()` starts the platform manager, query controller, HTTP controller, telemetry/cleanup loops, and plugin initialization.
+
+The central runtime object is `pkg/core/app.py::Application`. It is a service locator for long-lived managers. That is not elegant, but it is the current architectural center; most subsystems receive `ap: Application` and collaborate through it.
+
+## Top-Level Layout
+
+```text
+LangBot/
+├── main.py                         # Entrypoint shim
+├── pyproject.toml                  # Python package, deps, pinned langbot-plugin
+├── src/langbot/
+│   ├── __main__.py                 # CLI entrypoint and boot handoff
+│   ├── pkg/
+│   │   ├── core/                   # Application, boot stages, task manager
+│   │   ├── api/                    # HTTP API + MCP server mount
+│   │   ├── platform/               # IM adapters and runtime bot manager
+│   │   ├── pipeline/               # Message routing and pipeline stages
+│   │   ├── provider/               # LLM runners, model manager, tools
+│   │   ├── plugin/                 # LangBot-side Plugin Runtime connector/handler
+│   │   ├── box/                    # LangBot-side Box service/connector
+│   │   ├── skill/                  # Skill metadata/activation integration
+│   │   ├── rag/ , vector/          # Knowledge-base and vector DB integration
+│   │   ├── persistence/            # SQLAlchemy/SQLModel, Alembic, legacy migrations
+│   │   ├── storage/                # Local/S3 file storage abstraction
+│   │   └── config/, entity/, utils/, telemetry/, survey/
+│   ├── libs/                       # Vendored third-party platform SDKs
+│   └── templates/                  # Default config and component metadata
+├── web/                            # Vite + React Router + shadcn/ui + Tailwind SPA
+├── docker/                         # Deployment manifests
+├── skills/                         # In-repo agent skills, single source of truth
+└── tests/                          # Unit/integration/e2e/manual tests
+```
+
+## The Runtime Graph
+
+The most useful mental model is this graph:
+
+```text
+Platform adapter
+  → RuntimeBot
+  → MessageAggregator
+  → QueryPool
+  → Controller
+  → RuntimePipeline
+  → PipelineStage chain
+  → RequestRunner / ToolManager / PluginRuntimeConnector / BoxService
+  → response via adapter
+```
+
+The HTTP and MCP surfaces are parallel entrypoints into the same service layer:
+
+```text
+HTTP client / Web UI
+  → Quart route group
+  → api/http/service/*
+  → Application managers / persistence / runtime connectors
+
+MCP client
+  → /mcp mount
+  → api/mcp/server.py tools
+  → the same service layer directly
+```
+
+## Message Flow
+
+Inbound platform messages enter through adapter-specific SDK callbacks. The common path is:
+
+1. A platform adapter under `pkg/platform/sources/` converts platform-specific events into SDK message/event entities.
+2. `RuntimeBot` in `pkg/platform/botmgr.py` applies pipeline routing rules and either discards the message, pushes it to webhooks, or sends it to the message aggregator.
+3. `MessageAggregator` batches/normalizes messages before adding a `Query` to `QueryPool`.
+4. `Controller` in `pkg/pipeline/controller.py` selects queries subject to global pipeline concurrency and per-session concurrency.
+5. `RuntimePipeline` in `pkg/pipeline/pipelinemgr.py` runs configured pipeline stages using a responsibility-chain style executor that supports generator stages.
+6. The chat stage emits plugin events, calls a configured `RequestRunner`, handles streaming/non-streaming responses, records telemetry, and appends conversation history.
+7. Output stages send text, cards, chunks, files, or error notices back through the original platform adapter.
+
+Pipeline components are registered by decorators and package import side effects. When adding a new stage, loader, runner, or adapter, check the corresponding preregistration mechanism instead of inventing a second registry.
+
+## Platform Layer
+
+Platform code lives under `pkg/platform/`.
+
+- `botmgr.py` owns runtime bots, routing rules, event logging, webhook pushing, and adapter lifecycle.
+- `sources/` contains adapter implementations. Each adapter subclasses `langbot_plugin.api.definition.abstract.platform.adapter.AbstractMessagePlatformAdapter` from the SDK.
+- Platform entities such as `MessageChain`, `Image`, `At`, `Voice`, and events come from `langbot-plugin-sdk`, not from this repo.
+
+The platform layer should translate between external platform APIs and LangBot's shared message/event model. It should not contain LLM-provider logic or pipeline business logic.
+
+## Pipeline Layer
+
+Pipeline code lives under `pkg/pipeline/`.
+
+Important pieces:
+
+- `pool.py::QueryPool` stores pending queries and cached in-flight queries for plugin backward-compatible calls.
+- `controller.py::Controller` schedules query processing and enforces concurrency.
+- `pipelinemgr.py::RuntimePipeline` materializes database pipeline config into a runtime stage chain.
+- `process/handlers/chat.py::ChatMessageHandler` is the main LLM conversation handler.
+- Stage families include response rules, banned sessions, content filters, preprocessors, rate limits, message truncation, long text handling, response-back, command handling, and wrappers.
+
+Pipelines are configuration-driven. Prefer adding a stage or extending an existing stage family over hard-coding behavior in platform adapters.
+
+## Provider, RAG, and Tools
+
+Provider code lives under `pkg/provider/`.
+
+- `modelmgr/` manages configured model providers and requesters.
+- `runners/` implements request runners such as the local agent runner and external workflow integrations.
+- `tools/toolmgr.py` aggregates tools from native tools, plugin tools, external MCP servers, and skill-authoring tools.
+- `tools/loaders/mcp.py` is the MCP client side: external MCP servers that LangBot connects to for agent tools.
+- RAG lives across `pkg/rag/`, `pkg/vector/`, model services, and plugin KnowledgeEngine actions.
+
+Do not confuse LangBot's MCP client side with LangBot's own MCP server at `/mcp`; they are different surfaces.
+
+## Plugin System
+
+The plugin system crosses the repo boundary.
+
+In this repo:
+
+- `pkg/plugin/connector.py` connects LangBot to the Plugin Runtime over stdio or WebSocket.
+- `pkg/plugin/handler.py` exposes LangBot actions to the runtime and calls runtime actions for plugin operations.
+- `pkg/provider/tools/loaders/plugin.py` exposes plugin Tool components to LLM runners.
+- Pipeline handlers emit SDK events such as normal-message events and prompt-processing events.
+
+In `langbot-plugin-sdk`:
+
+- `src/langbot_plugin/api/` defines `BasePlugin`, component base classes, message/event entities, contexts, proxies, and manifests.
+- `src/langbot_plugin/runtime/` implements `lbp rt`, plugin discovery, dependency installation, process launching, and control/debug connections.
+- `src/langbot_plugin/entities/io/` defines the action protocol shared by LangBot, runtime, and plugin processes.
+
+The Plugin Runtime supports stdio and WebSocket control transports. Direct local LangBot runs usually spawn the runtime over stdio. Containerized/standalone deployments connect over WebSocket using `plugin.runtime_ws_url` and `--standalone-runtime`.
+
+## Box Runtime and Skills
+
+Box is the sandbox subsystem used by native agent tools, stdio MCP servers, skill authoring, and managed processes.
+
+In this repo:
+
+- `pkg/box/service.py` is the application-facing facade for exec, sessions, managed processes, skill CRUD, status, reconnects, quotas, mounts, and sandbox profiles.
+- `pkg/box/connector.py` connects to the Box Runtime over stdio, Windows subprocess+WebSocket, or remote WebSocket.
+- `pkg/provider/tools/loaders/native.py`, `mcp_stdio.py`, and skill loaders depend on Box availability.
+- `pkg/skill/manager.py` loads skills from the Box runtime, falling back to local `data/skills` when needed.
+
+Durable Box Workspace storage is shared across placement generations, but
+sandbox sessions and managed processes are generation-scoped. LangBot validates
+the current execution binding before an MCP stdio relay attach and sends the
+Workspace/generation binding in authenticated headers, so a placement cutover
+retires stale processes and closes already-attached relays.
+
+In `langbot-plugin-sdk`:
+
+- `src/langbot_plugin/box/server.py` implements `lbp box` and the WebSocket endpoints on `:5410`.
+- `src/langbot_plugin/box/runtime.py` owns sandbox sessions and managed processes.
+- `backend.py`, `nsjail_backend.py`, and `e2b_backend.py` implement sandbox backends.
+- `skill_store.py` manages skill packages from the Box side.
+
+Important config keys live under `box:` in `src/langbot/templates/config.yaml`: `box.enabled`, `box.backend`, `box.runtime.endpoint`, and `box.local.*`. Start LangBot with `--standalone-box` when connecting to an externally launched Box runtime.
+
+## HTTP API, Web UI, and MCP Server
+
+`pkg/api/http/controller/main.py` builds a Quart app, registers route groups, serves the built SPA, and wraps the ASGI app with the MCP dispatcher.
+
+- HTTP route groups live under `pkg/api/http/controller/groups/`.
+- Service-layer logic lives under `pkg/api/http/service/`.
+- The built web UI is served from the frontend build path with SPA fallback.
+- The MCP server lives under `pkg/api/mcp/` and is mounted at `/mcp`.
+
+The MCP server intentionally exposes a curated subset of the API. Tools call service classes directly rather than making HTTP requests back into LangBot.
+
+Maintenance rule: when adding, removing, or changing an HTTP endpoint that should be agent-accessible, update the matching MCP tool and the relevant in-repo skill under `skills/` in the same pass.
+
+## Persistence and Configuration
+
+Persistence is centered on `pkg/persistence/mgr.py`.
+
+- SQLite is the default database; PostgreSQL is supported.
+- Models live under `pkg/entity/persistence/`.
+- Fresh schemas are created from metadata, then legacy migrations run up to the frozen 3.x baseline, then Alembic migrations run to head.
+- New schema changes should use Alembic under `pkg/persistence/alembic/versions/`; do not extend the frozen legacy migration chain.
+
+Configuration starts from `src/langbot/templates/config.yaml` and is generated into `data/config.yaml` on first run. Most long-lived managers read from `ap.instance_config.data`.
+
+## Frontend
+
+The frontend lives in `web/` and is a Vite SPA using React Router 7, shadcn/ui, Tailwind CSS, and pnpm. It is not Next.js, despite some historical filenames.
+
+In development, `pnpm dev` serves the UI on `:3000` and reads `VITE_API_BASE_URL` to call the backend on `:5300`. In production, the built frontend is packaged into the Python distribution and served by the backend.
+
+Keep frontend API behavior aligned with `pkg/api/http/service/` and route groups. User-facing strings must go through the existing i18n setup.
+
+## Agent-Facing Surfaces
+
+LangBot is deliberately agent-friendly. The agent-facing surfaces are part of the architecture, not extra docs.
+
+- `skills/` is the single source of truth for in-repo skills.
+- `pkg/api/mcp/server.py` exposes the LangBot MCP server at `/mcp`.
+- `api.global_api_key` authenticates API/MCP access without a browser login.
+- `AGENTS.md` and `ARCHITECTURE.md` tell coding agents how the repo works.
+
+When one of these changes, update the others if the behavior or contract changed. API, MCP tools, and skills are one system; drift is a bug.
+
+## Where to Change Things
+
+- New HTTP API: add/adjust a service in `pkg/api/http/service/`, a route group in `pkg/api/http/controller/groups/`, tests, and MCP/skills if agent-accessible.
+- New platform adapter: add a `pkg/platform/sources/*` adapter, component metadata/templates as needed, i18n, docs, and tests/smoke coverage.
+- New pipeline behavior: add or extend a pipeline stage family under `pkg/pipeline/`; avoid putting pipeline rules in adapters.
+- New LLM provider/requester: work under `pkg/provider/modelmgr/` and related service/UI surfaces.
+- New LLM tool source: extend `pkg/provider/tools/loaders/` and `ToolManager` intentionally.
+- New plugin component/API/protocol: change `langbot-plugin-sdk` first or in lockstep, then update LangBot bridge code.
+- New Box capability: change both `pkg/box/` and `langbot-plugin-sdk/src/langbot_plugin/box/`, plus config and tests.
+- New database schema: add an Alembic migration, not a legacy `dbmXXX` migration.
+
+## Design Biases
+
+- Keep platform translation, pipeline orchestration, provider execution, and runtime protocols separate.
+- Reuse existing registries and service layers instead of adding parallel paths.
+- Prefer small, explicit agent surfaces over exposing every internal API.
+- Treat cross-repo contracts with the SDK as public interfaces.
+- Test behavior at the narrowest useful layer first, then add integration/e2e coverage for runtime or platform changes.

Dockerfile

@@ -38,7 +38,7 @@ COPY --from=node /app/web/dist ./web/dist
 COPY --from=nsjail-build /usr/local/bin/nsjail /usr/local/bin/nsjail
 
 RUN apt-get update \
-    && apt-get install -y --no-install-recommends gcc ca-certificates curl gnupg \
+    && apt-get install -y --no-install-recommends gcc ca-certificates curl git gnupg \
     # nsjail runtime libraries (the build toolchain stays in the nsjail-build
     # stage; only these shared libs are needed to execute the binary).
     && apt-get install -y --no-install-recommends libprotobuf32 libnl-route-3-200 \
@@ -63,8 +63,8 @@ RUN apt-get update \
     && rm -f /tmp/nodesource_setup.sh \
     && python -m pip install --no-cache-dir uv \
     && uv sync \
-    && apt-get purge -y --auto-remove curl gnupg \
+    && apt-get purge -y --auto-remove curl git gnupg \
     && rm -rf /var/lib/apt/lists/* \
     && touch /.dockerenv
 
-CMD [ "uv", "run", "--no-sync", "main.py" ]
+CMD [ "uv", "run", "--no-sync", "main.py" ]

README.md

@@ -5,7 +5,7 @@
 
 <div align="center">
 
-<a href="https://www.producthunt.com/products/langbot?utm_source=badge-follow&utm_medium=badge&utm_source=badge-langbot" target="_blank"><img src="https://api.producthunt.com/widgets/embed-image/v1/follow.svg?product_id=1077185&theme=light" alt="LangBot - Production&#0045;grade&#0032;IM&#0032;bot&#0032;made&#0032;easy&#0046; | Product Hunt" style="width: 250px; height: 54px;" width="250" height="54" /></a>
+<a href="https://www.producthunt.com/products/langbot/launches/langbot?embed=true&amp;utm_source=badge-featured&amp;utm_medium=badge&amp;utm_campaign=badge-langbot" target="_blank" rel="noopener noreferrer"><img alt="LangBot - Easy-to-use global IM bot platform designed for the LLM era | Product Hunt" width="250" height="54" src="https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=979554&amp;theme=light&amp;t=1782822143403"></a>
 
 <h3>Production-grade platform for building agentic IM bots.</h3>
 <h4>Quickly build, debug, and ship AI bots to Slack, Discord, Telegram, WeChat, and more.</h4>
@@ -51,7 +51,7 @@ LangBot is an **open-source, production-grade platform** for building AI-powered
 
 [→ Learn more about all features](https://link.langbot.app/en/docs/features)
 
-📍 Practical guides: [deploy a multi-platform AI bot in 5 minutes](https://blog.langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [connect DeepSeek to WeChat, Discord, and Telegram](https://blog.langbot.app/en/blog/connect-deepseek-to-wechat/), [run a Dify Agent in Discord, Telegram, and Slack](https://blog.langbot.app/en/blog/dify-agent-discord-telegram-slack/), and [build an n8n-powered chatbot](https://blog.langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
+📍 Practical guides: [deploy a multi-platform AI bot in 5 minutes](https://langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [connect DeepSeek to WeChat, Discord, and Telegram](https://langbot.app/en/blog/connect-deepseek-to-wechat/), [run a Dify Agent in Discord, Telegram, and Slack](https://langbot.app/en/blog/dify-agent-discord-telegram-slack/), and [build an n8n-powered chatbot](https://langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
 
 ---
 
@@ -92,6 +92,17 @@ docker compose --profile all up -d
 
 ---
 
+## Live Demo
+
+**Try it now:** https://demo.langbot.dev/
+
+- Email: `demo@langbot.app`
+- Password: `langbot123456`
+
+_Note: Public demo environment. Do not enter sensitive information._
+
+---
+
 ## Supported Platforms
 
 | Platform | Status | Notes |
@@ -136,7 +147,7 @@ docker compose --profile all up -d
 | [PPIO](https://ppinfra.com/user/register?invited_by=QJKFYD&utm_source=github_langbot)                             | GPU Platform | ✅     |
 | [ShengSuanYun](https://www.shengsuanyun.com/?from=CH_KYIPP758)                                                    | GPU Platform | ✅     |
 | [接口 AI](https://jiekou.ai/)                                                                                     | Gateway      | ✅     |
-| [302.AI](https://share.302.ai/SuTG99)                                                                             | Gateway      | ✅     |
+| [302.AI](https://share.302ai.cn/SuTG99)                                                                             | Gateway      | ✅     |
 | [Qiniu](https://www.qiniu.com/ai/agent)                                                                           | Gateway      | ✅     |
 
 [→ View all integrations](https://link.langbot.app/en/docs/features)
@@ -167,17 +178,6 @@ LangBot is **agent-friendly by design** — your coding agents (Claude Code, Cod
 
 ---
 
-## Live Demo
-
-**Try it now:** https://demo.langbot.dev/
-
-- Email: `demo@langbot.app`
-- Password: `langbot123456`
-
-_Note: Public demo environment. Do not enter sensitive information._
-
----
-
 ## Community
 
 [![Discord](https://img.shields.io/discord/1335141740050649118?logo=discord&label=Discord)](https://discord.gg/wdNEHETs87)
@@ -186,12 +186,6 @@ _Note: Public demo environment. Do not enter sensitive information._
 
 ---
 
-## Star History
-
-[![Star History Chart](https://api.star-history.com/svg?repos=langbot-app/LangBot&type=Date)](https://star-history.com/#langbot-app/LangBot&Date)
-
----
-
 ## Contributors
 
 Thanks to all [contributors](https://github.com/langbot-app/LangBot/graphs/contributors) who have helped make LangBot better:

README_CN.md

@@ -51,7 +51,7 @@ LangBot 是一个**开源的生产级平台**，用于构建 AI 驱动的即时
 
 [→ 了解更多功能特性](https://link.langbot.app/zh/docs/features)
 
-📍 实践指南：[5 分钟部署多平台 AI 机器人](https://blog.langbot.app/zh/blog/deploy-ai-bot-in-5-minutes/)、[将 DeepSeek 接入微信、企业微信与 Discord](https://blog.langbot.app/zh/blog/connect-deepseek-to-wechat/)、[让 Dify Agent 跑在 Discord、Telegram 和 Slack 上](https://blog.langbot.app/zh/blog/dify-agent-discord-telegram-slack/)，以及[用 n8n 构建多平台 AI 聊天机器人](https://blog.langbot.app/zh/blog/n8n-multi-platform-ai-chatbot/)。
+📍 实践指南：[5 分钟部署多平台 AI 机器人](https://langbot.app/zh/blog/deploy-ai-bot-in-5-minutes/)、[将 DeepSeek 接入微信、企业微信与 Discord](https://langbot.app/zh/blog/connect-deepseek-to-wechat/)、[让 Dify Agent 跑在 Discord、Telegram 和 Slack 上](https://langbot.app/zh/blog/dify-agent-discord-telegram-slack/)，以及[用 n8n 构建多平台 AI 聊天机器人](https://langbot.app/zh/blog/n8n-multi-platform-ai-chatbot/)。
 
 ---
 
@@ -92,6 +92,16 @@ docker compose --profile all up -d
 
 ---
 
+## 在线演示
+
+**立即体验：** https://demo.langbot.dev/
+- 邮箱：`demo@langbot.app`
+- 密码：`langbot123456`
+
+*注意：公开演示环境，请不要在其中填入任何敏感信息。*
+
+---
+
 ## 支持的平台
 
 | 平台 | 状态 | 备注 |
@@ -136,7 +146,7 @@ docker compose --profile all up -d
 | [优云智算](https://www.compshare.cn/?ytag=GPU_YY-gh_langbot) | GPU 平台 | ✅ |
 | [PPIO](https://ppinfra.com/user/register?invited_by=QJKFYD&utm_source=github_langbot) | GPU 平台 | ✅ |
 | [接口 AI](https://jiekou.ai/) | 聚合平台 | ✅ |
-| [302.AI](https://share.302.ai/SuTG99) | 聚合平台 | ✅ |
+| [302.AI](https://share.302ai.cn/SuTG99) | 聚合平台 | ✅ |
 | [小马算力](https://www.tokenpony.cn/453z1) | 聚合平台 | ✅ |
 | [百宝箱Tbox](https://www.tbox.cn/open) | 智能体平台 | ✅ |
 | [七牛云Qiniu](https://www.qiniu.com/ai/agent) | 聚合平台 | ✅ |
@@ -170,16 +180,6 @@ docker compose --profile all up -d
 
 ---
 
-## 在线演示
-
-**立即体验：** https://demo.langbot.dev/
-- 邮箱：`demo@langbot.app`
-- 密码：`langbot123456`
-
-*注意：公开演示环境，请不要在其中填入任何敏感信息。*
-
----
-
 ## 为 AI Agent 而生 🤖
 
 LangBot **从设计上就对 Agent 友好** —— 你的编码 Agent（Claude Code、Codex、Copilot、Cursor 等）可以一等公民般地操作、扩展和部署 LangBot：
@@ -203,12 +203,6 @@ LangBot **从设计上就对 Agent 友好** —— 你的编码 Agent（Claude C
 
 ---
 
-## Star 趋势
-
-[![Star History Chart](https://api.star-history.com/svg?repos=langbot-app/LangBot&type=Date)](https://star-history.com/#langbot-app/LangBot&Date)
-
----
-
 ## 贡献者
 
 感谢所有[贡献者](https://github.com/langbot-app/LangBot/graphs/contributors)对 LangBot 的帮助：

README_ES.md

@@ -5,7 +5,7 @@
 
 <div align="center">
 
-<a href="https://www.producthunt.com/products/langbot?utm_source=badge-follow&utm_medium=badge&utm_source=badge-langbot" target="_blank"><img src="https://api.producthunt.com/widgets/embed-image/v1/follow.svg?product_id=1077185&theme=light" alt="LangBot - Production&#0045;grade&#0032;IM&#0032;bot&#0032;made&#0032;easy&#0046; | Product Hunt" style="width: 250px; height: 54px;" width="250" height="54" /></a>
+<a href="https://www.producthunt.com/products/langbot/launches/langbot?embed=true&amp;utm_source=badge-featured&amp;utm_medium=badge&amp;utm_campaign=badge-langbot" target="_blank" rel="noopener noreferrer"><img alt="LangBot - Easy-to-use global IM bot platform designed for the LLM era | Product Hunt" width="250" height="54" src="https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=979554&amp;theme=light&amp;t=1782822143403"></a>
 
 <h3>Plataforma de grado de producción para construir bots de mensajería instantánea con agentes de IA.</h3>
 <h4>Construya, depure y despliegue bots de IA rápidamente en Slack, Discord, Telegram, WeChat y más.</h4>
@@ -50,7 +50,7 @@ LangBot es una **plataforma de código abierto y grado de producción** para con
 
 [→ Conocer más sobre todas las funcionalidades](https://link.langbot.app/en/docs/features)
 
-📍 Guías prácticas: [desplegar un bot de IA multiplataforma en 5 minutos](https://blog.langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [conectar DeepSeek a WeChat, Discord y Telegram](https://blog.langbot.app/en/blog/connect-deepseek-to-wechat/), [ejecutar un Dify Agent en Discord, Telegram y Slack](https://blog.langbot.app/en/blog/dify-agent-discord-telegram-slack/) y [crear un chatbot con n8n](https://blog.langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
+📍 Guías prácticas: [desplegar un bot de IA multiplataforma en 5 minutos](https://langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [conectar DeepSeek a WeChat, Discord y Telegram](https://langbot.app/en/blog/connect-deepseek-to-wechat/), [ejecutar un Dify Agent en Discord, Telegram y Slack](https://langbot.app/en/blog/dify-agent-discord-telegram-slack/) y [crear un chatbot con n8n](https://langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
 
 ---
 
@@ -91,6 +91,16 @@ docker compose --profile all up -d
 
 ---
 
+## Demo en Vivo
+
+**Pruébelo ahora:** https://demo.langbot.dev/
+- Correo electrónico: `demo@langbot.app`
+- Contraseña: `langbot123456`
+
+*Nota: Entorno de demostración público. No ingrese información confidencial.*
+
+---
+
 ## Plataformas Soportadas
 
 | Plataforma | Estado | Notas |
@@ -135,7 +145,7 @@ docker compose --profile all up -d
 | [PPIO](https://ppinfra.com/user/register?invited_by=QJKFYD&utm_source=github_langbot) | Plataforma GPU | ✅ |
 | [ShengSuanYun](https://www.shengsuanyun.com/?from=CH_KYIPP758) | Plataforma GPU | ✅ |
 | [接口 AI](https://jiekou.ai/) | Pasarela | ✅ |
-| [302.AI](https://share.302.ai/SuTG99) | Pasarela | ✅ |
+| [302.AI](https://share.302ai.cn/SuTG99) | Pasarela | ✅ |
 | [Qiniu](https://www.qiniu.com/ai/agent) | Pasarela | ✅ |
 
 [→ Ver todas las integraciones](https://link.langbot.app/en/docs/features)
@@ -153,14 +163,6 @@ docker compose --profile all up -d
 
 ---
 
-## Demo en Vivo
-
-**Pruébelo ahora:** https://demo.langbot.dev/
-- Correo electrónico: `demo@langbot.app`
-- Contraseña: `langbot123456`
-
-*Nota: Entorno de demostración público. No ingrese información confidencial.*
-
 ## Diseñado para Agentes de IA 🤖
 
 LangBot es **agent-friendly por diseño** —— tus agentes de codificación (Claude Code, Codex, Copilot, Cursor, …) pueden operar, extender y desplegar LangBot con soporte de primera clase:
@@ -182,12 +184,6 @@ LangBot es **agent-friendly por diseño** —— tus agentes de codificación (C
 
 ---
 
-## Historial de Stars
-
-[![Star History Chart](https://api.star-history.com/svg?repos=langbot-app/LangBot&type=Date)](https://star-history.com/#langbot-app/LangBot&Date)
-
----
-
 ## Colaboradores
 
 Gracias a todos los [colaboradores](https://github.com/langbot-app/LangBot/graphs/contributors) que han ayudado a mejorar LangBot:

README_FR.md

@@ -5,7 +5,7 @@
 
 <div align="center">
 
-<a href="https://www.producthunt.com/products/langbot?utm_source=badge-follow&utm_medium=badge&utm_source=badge-langbot" target="_blank"><img src="https://api.producthunt.com/widgets/embed-image/v1/follow.svg?product_id=1077185&theme=light" alt="LangBot - Production&#0045;grade&#0032;IM&#0032;bot&#0032;made&#0032;easy&#0046; | Product Hunt" style="width: 250px; height: 54px;" width="250" height="54" /></a>
+<a href="https://www.producthunt.com/products/langbot/launches/langbot?embed=true&amp;utm_source=badge-featured&amp;utm_medium=badge&amp;utm_campaign=badge-langbot" target="_blank" rel="noopener noreferrer"><img alt="LangBot - Easy-to-use global IM bot platform designed for the LLM era | Product Hunt" width="250" height="54" src="https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=979554&amp;theme=light&amp;t=1782822143403"></a>
 
 <h3>Plateforme de niveau production pour construire des bots de messagerie instantanée avec agents IA.</h3>
 <h4>Créez, déboguez et déployez rapidement des bots IA sur Slack, Discord, Telegram, WeChat et plus.</h4>
@@ -50,7 +50,7 @@ LangBot est une **plateforme open-source de niveau production** pour créer des
 
 [→ En savoir plus sur toutes les fonctionnalités](https://link.langbot.app/en/docs/features)
 
-📍 Guides pratiques : [déployer un bot IA multiplateforme en 5 minutes](https://blog.langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [connecter DeepSeek à WeChat, Discord et Telegram](https://blog.langbot.app/en/blog/connect-deepseek-to-wechat/), [exécuter un Dify Agent dans Discord, Telegram et Slack](https://blog.langbot.app/en/blog/dify-agent-discord-telegram-slack/) et [créer un chatbot avec n8n](https://blog.langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
+📍 Guides pratiques : [déployer un bot IA multiplateforme en 5 minutes](https://langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [connecter DeepSeek à WeChat, Discord et Telegram](https://langbot.app/en/blog/connect-deepseek-to-wechat/), [exécuter un Dify Agent dans Discord, Telegram et Slack](https://langbot.app/en/blog/dify-agent-discord-telegram-slack/) et [créer un chatbot avec n8n](https://langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
 
 ---
 
@@ -91,6 +91,16 @@ docker compose --profile all up -d
 
 ---
 
+## Démo en Ligne
+
+**Essayez maintenant :** https://demo.langbot.dev/
+- Email : `demo@langbot.app`
+- Mot de passe : `langbot123456`
+
+*Note : Environnement de démonstration public. Ne saisissez pas d'informations sensibles.*
+
+---
+
 ## Plateformes Supportées
 
 | Plateforme | Statut | Notes |
@@ -132,7 +142,7 @@ docker compose --profile all up -d
 | [ModelScope](https://modelscope.cn/docs/model-service/API-Inference/intro) | Passerelle | ✅ |
 | [GiteeAI](https://ai.gitee.com/) | Passerelle | ✅ |
 | [接口 AI](https://jiekou.ai/) | Passerelle | ✅ |
-| [302.AI](https://share.302.ai/SuTG99) | Passerelle | ✅ |
+| [302.AI](https://share.302ai.cn/SuTG99) | Passerelle | ✅ |
 | [CompShare](https://www.compshare.cn/?ytag=GPU_YY-gh_langbot) | Plateforme GPU | ✅ |
 | [PPIO](https://ppinfra.com/user/register?invited_by=QJKFYD&utm_source=github_langbot) | Plateforme GPU | ✅ |
 | [ShengSuanYun](https://www.shengsuanyun.com/?from=CH_KYIPP758) | Plateforme GPU | ✅ |
@@ -153,14 +163,6 @@ docker compose --profile all up -d
 
 ---
 
-## Démo en Ligne
-
-**Essayez maintenant :** https://demo.langbot.dev/
-- Email : `demo@langbot.app`
-- Mot de passe : `langbot123456`
-
-*Note : Environnement de démonstration public. Ne saisissez pas d'informations sensibles.*
-
 ## Conçu pour les agents IA 🤖
 
 LangBot est **agent-friendly par conception** —— vos agents de codage (Claude Code, Codex, Copilot, Cursor, …) peuvent exploiter, étendre et déployer LangBot avec un support de premier ordre :
@@ -182,12 +184,6 @@ LangBot est **agent-friendly par conception** —— vos agents de codage (Claud
 
 ---
 
-## Historique des Stars
-
-[![Star History Chart](https://api.star-history.com/svg?repos=langbot-app/LangBot&type=Date)](https://star-history.com/#langbot-app/LangBot&Date)
-
----
-
 ## Contributeurs
 
 Merci à tous les [contributeurs](https://github.com/langbot-app/LangBot/graphs/contributors) qui ont aidé à améliorer LangBot :

README_JP.md

@@ -5,7 +5,7 @@
 
 <div align="center">
 
-<a href="https://www.producthunt.com/products/langbot?utm_source=badge-follow&utm_medium=badge&utm_source=badge-langbot" target="_blank"><img src="https://api.producthunt.com/widgets/embed-image/v1/follow.svg?product_id=1077185&theme=light" alt="LangBot - Production&#0045;grade&#0032;IM&#0032;bot&#0032;made&#0032;easy&#0046; | Product Hunt" style="width: 250px; height: 54px;" width="250" height="54" /></a>
+<a href="https://www.producthunt.com/products/langbot/launches/langbot?embed=true&amp;utm_source=badge-featured&amp;utm_medium=badge&amp;utm_campaign=badge-langbot" target="_blank" rel="noopener noreferrer"><img alt="LangBot - Easy-to-use global IM bot platform designed for the LLM era | Product Hunt" width="250" height="54" src="https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=979554&amp;theme=light&amp;t=1782822143403"></a>
 
 <h3>AIエージェント搭載IMボットを構築するための本番グレードプラットフォーム。</h3>
 <h4>Slack、Discord、Telegram、WeChat などに AI ボットを素早く構築、デバッグ、デプロイ。</h4>
@@ -50,7 +50,7 @@ LangBot は、AI搭載のインスタントメッセージングボットを構
 
 [→ すべての機能について詳しく見る](https://link.langbot.app/ja/docs/features)
 
-📍 実践ガイド: [5分でマルチプラットフォームAIボットをデプロイ](https://blog.langbot.app/en/blog/deploy-ai-bot-in-5-minutes/)、[DeepSeekをWeChat・Discord・Telegramに接続](https://blog.langbot.app/en/blog/connect-deepseek-to-wechat/)、[Dify AgentをDiscord・Telegram・Slackで動かす](https://blog.langbot.app/en/blog/dify-agent-discord-telegram-slack/)、[n8n連携チャットボットを構築](https://blog.langbot.app/en/blog/n8n-multi-platform-ai-chatbot/)。
+📍 実践ガイド: [5分でマルチプラットフォームAIボットをデプロイ](https://langbot.app/en/blog/deploy-ai-bot-in-5-minutes/)、[DeepSeekをWeChat・Discord・Telegramに接続](https://langbot.app/en/blog/connect-deepseek-to-wechat/)、[Dify AgentをDiscord・Telegram・Slackで動かす](https://langbot.app/en/blog/dify-agent-discord-telegram-slack/)、[n8n連携チャットボットを構築](https://langbot.app/en/blog/n8n-multi-platform-ai-chatbot/)。
 
 ---
 
@@ -91,6 +91,16 @@ docker compose --profile all up -d
 
 ---
 
+## ライブデモ
+
+**今すぐ試す:** https://demo.langbot.dev/
+- メール: `demo@langbot.app`
+- パスワード: `langbot123456`
+
+*注意: 公開デモ環境です。機密情報を入力しないでください。*
+
+---
+
 ## 対応プラットフォーム
 
 | プラットフォーム | ステータス | 備考 |
@@ -135,7 +145,7 @@ docker compose --profile all up -d
 | [PPIO](https://ppinfra.com/user/register?invited_by=QJKFYD&utm_source=github_langbot) | GPUプラットフォーム | ✅ |
 | [ShengSuanYun](https://www.shengsuanyun.com/?from=CH_KYIPP758) | GPUプラットフォーム | ✅ |
 | [接口 AI](https://jiekou.ai/) | ゲートウェイ | ✅ |
-| [302.AI](https://share.302.ai/SuTG99) | ゲートウェイ | ✅ |
+| [302.AI](https://share.302ai.cn/SuTG99) | ゲートウェイ | ✅ |
 | [Qiniu](https://www.qiniu.com/ai/agent) | ゲートウェイ | ✅ |
 
 [→ すべての統合を表示](https://link.langbot.app/en/docs/features)
@@ -153,14 +163,6 @@ docker compose --profile all up -d
 
 ---
 
-## ライブデモ
-
-**今すぐ試す:** https://demo.langbot.dev/
-- メール: `demo@langbot.app`
-- パスワード: `langbot123456`
-
-*注意: 公開デモ環境です。機密情報を入力しないでください。*
-
 ## AI エージェントのために 🤖
 
 LangBot は **設計段階からエージェントフレンドリー** です。お使いのコーディングエージェント（Claude Code、Codex、Copilot、Cursor など）が、ファーストクラスのサポートで LangBot を操作・拡張・デプロイできます：
@@ -182,12 +184,6 @@ LangBot は **設計段階からエージェントフレンドリー** です。
 
 ---
 
-## Star 推移
-
-[![Star History Chart](https://api.star-history.com/svg?repos=langbot-app/LangBot&type=Date)](https://star-history.com/#langbot-app/LangBot&Date)
-
----
-
 ## コントリビューター
 
 LangBot をより良くするために貢献してくださったすべての[コントリビューター](https://github.com/langbot-app/LangBot/graphs/contributors)に感謝します:

README_KO.md

@@ -5,7 +5,7 @@
 
 <div align="center">
 
-<a href="https://www.producthunt.com/products/langbot?utm_source=badge-follow&utm_medium=badge&utm_source=badge-langbot" target="_blank"><img src="https://api.producthunt.com/widgets/embed-image/v1/follow.svg?product_id=1077185&theme=light" alt="LangBot - Production&#0045;grade&#0032;IM&#0032;bot&#0032;made&#0032;easy&#0046; | Product Hunt" style="width: 250px; height: 54px;" width="250" height="54" /></a>
+<a href="https://www.producthunt.com/products/langbot/launches/langbot?embed=true&amp;utm_source=badge-featured&amp;utm_medium=badge&amp;utm_campaign=badge-langbot" target="_blank" rel="noopener noreferrer"><img alt="LangBot - Easy-to-use global IM bot platform designed for the LLM era | Product Hunt" width="250" height="54" src="https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=979554&amp;theme=light&amp;t=1782822143403"></a>
 
 <h3>AI 에이전트 IM 봇 구축을 위한 프로덕션 등급 플랫폼.</h3>
 <h4>Slack, Discord, Telegram, WeChat 등에 AI 봇을 빠르게 구축, 디버그 및 배포.</h4>
@@ -50,7 +50,7 @@ LangBot은 AI 기반 인스턴트 메시징 봇을 구축하기 위한 **오픈
 
 [→ 모든 기능 자세히 보기](https://link.langbot.app/en/docs/features)
 
-📍 실전 가이드: [5분 만에 멀티 플랫폼 AI 봇 배포하기](https://blog.langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [DeepSeek를 WeChat, Discord, Telegram에 연결하기](https://blog.langbot.app/en/blog/connect-deepseek-to-wechat/), [Dify Agent를 Discord, Telegram, Slack에서 실행하기](https://blog.langbot.app/en/blog/dify-agent-discord-telegram-slack/), [n8n 기반 챗봇 만들기](https://blog.langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
+📍 실전 가이드: [5분 만에 멀티 플랫폼 AI 봇 배포하기](https://langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [DeepSeek를 WeChat, Discord, Telegram에 연결하기](https://langbot.app/en/blog/connect-deepseek-to-wechat/), [Dify Agent를 Discord, Telegram, Slack에서 실행하기](https://langbot.app/en/blog/dify-agent-discord-telegram-slack/), [n8n 기반 챗봇 만들기](https://langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
 
 ---
 
@@ -91,6 +91,16 @@ docker compose --profile all up -d
 
 ---
 
+## 라이브 데모
+
+**지금 체험:** https://demo.langbot.dev/
+- 이메일: `demo@langbot.app`
+- 비밀번호: `langbot123456`
+
+*참고: 공개 데모 환경입니다. 민감한 정보를 입력하지 마세요.*
+
+---
+
 ## 지원 플랫폼
 
 | 플랫폼 | 상태 | 비고 |
@@ -135,7 +145,7 @@ docker compose --profile all up -d
 | [PPIO](https://ppinfra.com/user/register?invited_by=QJKFYD&utm_source=github_langbot) | GPU 플랫폼 | ✅ |
 | [ShengSuanYun](https://www.shengsuanyun.com/?from=CH_KYIPP758) | GPU 플랫폼 | ✅ |
 | [接口 AI](https://jiekou.ai/) | 게이트웨이 | ✅ |
-| [302.AI](https://share.302.ai/SuTG99) | 게이트웨이 | ✅ |
+| [302.AI](https://share.302ai.cn/SuTG99) | 게이트웨이 | ✅ |
 | [Qiniu](https://www.qiniu.com/ai/agent) | 게이트웨이 | ✅ |
 
 [→ 모든 통합 보기](https://link.langbot.app/en/docs/features)
@@ -153,14 +163,6 @@ docker compose --profile all up -d
 
 ---
 
-## 라이브 데모
-
-**지금 체험:** https://demo.langbot.dev/
-- 이메일: `demo@langbot.app`
-- 비밀번호: `langbot123456`
-
-*참고: 공개 데모 환경입니다. 민감한 정보를 입력하지 마세요.*
-
 ## AI 에이전트를 위한 설계 🤖
 
 LangBot은 **설계 단계부터 에이전트 친화적**입니다 —— 코딩 에이전트(Claude Code, Codex, Copilot, Cursor 등)가 일급 지원으로 LangBot을 운영·확장·배포할 수 있습니다:
@@ -182,12 +184,6 @@ LangBot은 **설계 단계부터 에이전트 친화적**입니다 —— 코딩
 
 ---
 
-## Star 추이
-
-[![Star History Chart](https://api.star-history.com/svg?repos=langbot-app/LangBot&type=Date)](https://star-history.com/#langbot-app/LangBot&Date)
-
----
-
 ## 기여자
 
 LangBot을 더 나은 프로젝트로 만들어 주신 모든 [기여자](https://github.com/langbot-app/LangBot/graphs/contributors)분들께 감사드립니다:

README_RU.md

@@ -5,7 +5,7 @@
 
 <div align="center">
 
-<a href="https://www.producthunt.com/products/langbot?utm_source=badge-follow&utm_medium=badge&utm_source=badge-langbot" target="_blank"><img src="https://api.producthunt.com/widgets/embed-image/v1/follow.svg?product_id=1077185&theme=light" alt="LangBot - Production&#0045;grade&#0032;IM&#0032;bot&#0032;made&#0032;easy&#0046; | Product Hunt" style="width: 250px; height: 54px;" width="250" height="54" /></a>
+<a href="https://www.producthunt.com/products/langbot/launches/langbot?embed=true&amp;utm_source=badge-featured&amp;utm_medium=badge&amp;utm_campaign=badge-langbot" target="_blank" rel="noopener noreferrer"><img alt="LangBot - Easy-to-use global IM bot platform designed for the LLM era | Product Hunt" width="250" height="54" src="https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=979554&amp;theme=light&amp;t=1782822143403"></a>
 
 <h3>Платформа производственного уровня для создания агентных IM-ботов.</h3>
 <h4>Быстро создавайте, отлаживайте и развертывайте ИИ-ботов в Slack, Discord, Telegram, WeChat и других платформах.</h4>
@@ -50,7 +50,7 @@ LangBot — это **платформа с открытым исходным к
 
 [→ Подробнее обо всех возможностях](https://link.langbot.app/en/docs/features)
 
-📍 Практические руководства: [развернуть мультиплатформенного ИИ-бота за 5 минут](https://blog.langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [подключить DeepSeek к WeChat, Discord и Telegram](https://blog.langbot.app/en/blog/connect-deepseek-to-wechat/), [запустить Dify Agent в Discord, Telegram и Slack](https://blog.langbot.app/en/blog/dify-agent-discord-telegram-slack/) и [создать чат-бота на n8n](https://blog.langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
+📍 Практические руководства: [развернуть мультиплатформенного ИИ-бота за 5 минут](https://langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [подключить DeepSeek к WeChat, Discord и Telegram](https://langbot.app/en/blog/connect-deepseek-to-wechat/), [запустить Dify Agent в Discord, Telegram и Slack](https://langbot.app/en/blog/dify-agent-discord-telegram-slack/) и [создать чат-бота на n8n](https://langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
 
 ---
 
@@ -91,6 +91,16 @@ docker compose --profile all up -d
 
 ---
 
+## Демо
+
+**Попробуйте прямо сейчас:** https://demo.langbot.dev/
+- Email: `demo@langbot.app`
+- Пароль: `langbot123456`
+
+*Примечание: Публичная демо-среда. Не вводите конфиденциальную информацию.*
+
+---
+
 ## Поддерживаемые платформы
 
 | Платформа | Статус | Примечания |
@@ -131,7 +141,7 @@ docker compose --profile all up -d
 | [Volc Engine Ark](https://console.volcengine.com/ark/region:ark+cn-beijing/model?vendor=Bytedance&view=LIST_VIEW) | Шлюз | ✅ |
 | [ModelScope](https://modelscope.cn/docs/model-service/API-Inference/intro) | Шлюз | ✅ |
 | [GiteeAI](https://ai.gitee.com/) | Шлюз | ✅ |
-| [302.AI](https://share.302.ai/SuTG99) | Шлюз | ✅ |
+| [302.AI](https://share.302ai.cn/SuTG99) | Шлюз | ✅ |
 | [接口 AI](https://jiekou.ai/) | Шлюз | ✅ |
 | [CompShare](https://www.compshare.cn/?ytag=GPU_YY-gh_langbot) | Платформа GPU | ✅ |
 | [PPIO](https://ppinfra.com/user/register?invited_by=QJKFYD&utm_source=github_langbot) | Платформа GPU | ✅ |
@@ -153,14 +163,6 @@ docker compose --profile all up -d
 
 ---
 
-## Демо
-
-**Попробуйте прямо сейчас:** https://demo.langbot.dev/
-- Email: `demo@langbot.app`
-- Пароль: `langbot123456`
-
-*Примечание: Публичная демо-среда. Не вводите конфиденциальную информацию.*
-
 ## Создано для ИИ-агентов 🤖
 
 LangBot **дружелюбен к агентам по своей архитектуре** —— ваши кодинг-агенты (Claude Code, Codex, Copilot, Cursor и др.) могут управлять, расширять и развёртывать LangBot с первоклассной поддержкой:
@@ -182,12 +184,6 @@ LangBot **дружелюбен к агентам по своей архитек
 
 ---
 
-## История Stars
-
-[![Star History Chart](https://api.star-history.com/svg?repos=langbot-app/LangBot&type=Date)](https://star-history.com/#langbot-app/LangBot&Date)
-
----
-
 ## Участники
 
 Спасибо всем [участникам](https://github.com/langbot-app/LangBot/graphs/contributors), которые помогли сделать LangBot лучше:

README_TW.md

@@ -52,7 +52,7 @@ LangBot 是一個**開源的生產級平台**，用於建構 AI 驅動的即時
 
 [→ 了解更多功能特性](https://link.langbot.app/zh/docs/features)
 
-📍 實踐指南：[5 分鐘部署多平台 AI 機器人](https://blog.langbot.app/zh/blog/deploy-ai-bot-in-5-minutes/)、[將 DeepSeek 接入微信、企業微信與 Discord](https://blog.langbot.app/zh/blog/connect-deepseek-to-wechat/)、[讓 Dify Agent 跑在 Discord、Telegram 和 Slack 上](https://blog.langbot.app/zh/blog/dify-agent-discord-telegram-slack/)，以及[用 n8n 建構多平台 AI 聊天機器人](https://blog.langbot.app/zh/blog/n8n-multi-platform-ai-chatbot/)。
+📍 實踐指南：[5 分鐘部署多平台 AI 機器人](https://langbot.app/zh/blog/deploy-ai-bot-in-5-minutes/)、[將 DeepSeek 接入微信、企業微信與 Discord](https://langbot.app/zh/blog/connect-deepseek-to-wechat/)、[讓 Dify Agent 跑在 Discord、Telegram 和 Slack 上](https://langbot.app/zh/blog/dify-agent-discord-telegram-slack/)，以及[用 n8n 建構多平台 AI 聊天機器人](https://langbot.app/zh/blog/n8n-multi-platform-ai-chatbot/)。
 
 ---
 
@@ -93,6 +93,16 @@ docker compose --profile all up -d
 
 ---
 
+## 線上演示
+
+**立即體驗：** https://demo.langbot.dev/
+- 信箱：`demo@langbot.app`
+- 密碼：`langbot123456`
+
+*注意：公開演示環境，請不要在其中填入任何敏感資訊。*
+
+---
+
 ## 支援的平台
 
 | 平台 | 狀態 | 備註 |
@@ -137,7 +147,7 @@ docker compose --profile all up -d
 | [優雲智算](https://www.compshare.cn/?ytag=GPU_YY-gh_langbot) | GPU 平台 | ✅ |
 | [PPIO](https://ppinfra.com/user/register?invited_by=QJKFYD&utm_source=github_langbot) | GPU 平台 | ✅ |
 | [接口 AI](https://jiekou.ai/) | 聚合平台 | ✅ |
-| [302.AI](https://share.302.ai/SuTG99) | 聚合平台 | ✅ |
+| [302.AI](https://share.302ai.cn/SuTG99) | 聚合平台 | ✅ |
 | [Qiniu](https://www.qiniu.com/ai/agent) | 聚合平台 | ✅ |
 
 ### TTS（語音合成）
@@ -169,14 +179,6 @@ docker compose --profile all up -d
 
 ---
 
-## 線上演示
-
-**立即體驗：** https://demo.langbot.dev/
-- 信箱：`demo@langbot.app`
-- 密碼：`langbot123456`
-
-*注意：公開演示環境，請不要在其中填入任何敏感資訊。*
-
 ## 為 AI Agent 而生 🤖
 
 LangBot **從設計上就對 Agent 友善** —— 你的編碼 Agent（Claude Code、Codex、Copilot、Cursor 等）可以一等公民般地操作、擴充和部署 LangBot：
@@ -200,12 +202,6 @@ LangBot **從設計上就對 Agent 友善** —— 你的編碼 Agent（Claude C
 
 ---
 
-## Star 趨勢
-
-[![Star History Chart](https://api.star-history.com/svg?repos=langbot-app/LangBot&type=Date)](https://star-history.com/#langbot-app/LangBot&Date)
-
----
-
 ## 貢獻者
 
 感謝所有[貢獻者](https://github.com/langbot-app/LangBot/graphs/contributors)對 LangBot 的幫助：

README_VI.md

@@ -5,7 +5,7 @@
 
 <div align="center">
 
-<a href="https://www.producthunt.com/products/langbot?utm_source=badge-follow&utm_medium=badge&utm_source=badge-langbot" target="_blank"><img src="https://api.producthunt.com/widgets/embed-image/v1/follow.svg?product_id=1077185&theme=light" alt="LangBot - Production&#0045;grade&#0032;IM&#0032;bot&#0032;made&#0032;easy&#0046; | Product Hunt" style="width: 250px; height: 54px;" width="250" height="54" /></a>
+<a href="https://www.producthunt.com/products/langbot/launches/langbot?embed=true&amp;utm_source=badge-featured&amp;utm_medium=badge&amp;utm_campaign=badge-langbot" target="_blank" rel="noopener noreferrer"><img alt="LangBot - Easy-to-use global IM bot platform designed for the LLM era | Product Hunt" width="250" height="54" src="https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=979554&amp;theme=light&amp;t=1782822143403"></a>
 
 <h3>Nền tảng cấp sản xuất để xây dựng bot IM với AI agent.</h3>
 <h4>Xây dựng, gỡ lỗi và triển khai bot AI nhanh chóng trên Slack, Discord, Telegram, WeChat và nhiều nền tảng khác.</h4>
@@ -50,7 +50,7 @@ LangBot là một **nền tảng mã nguồn mở, cấp sản xuất** để x
 
 [→ Tìm hiểu thêm về tất cả tính năng](https://link.langbot.app/en/docs/features)
 
-📍 Hướng dẫn thực hành: [triển khai bot AI đa nền tảng trong 5 phút](https://blog.langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [kết nối DeepSeek với WeChat, Discord và Telegram](https://blog.langbot.app/en/blog/connect-deepseek-to-wechat/), [chạy Dify Agent trên Discord, Telegram và Slack](https://blog.langbot.app/en/blog/dify-agent-discord-telegram-slack/) và [xây dựng chatbot với n8n](https://blog.langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
+📍 Hướng dẫn thực hành: [triển khai bot AI đa nền tảng trong 5 phút](https://langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [kết nối DeepSeek với WeChat, Discord và Telegram](https://langbot.app/en/blog/connect-deepseek-to-wechat/), [chạy Dify Agent trên Discord, Telegram và Slack](https://langbot.app/en/blog/dify-agent-discord-telegram-slack/) và [xây dựng chatbot với n8n](https://langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
 
 ---
 
@@ -91,6 +91,16 @@ docker compose --profile all up -d
 
 ---
 
+## Demo trực tuyến
+
+**Thử ngay:** https://demo.langbot.dev/
+- Email: `demo@langbot.app`
+- Mật khẩu: `langbot123456`
+
+*Lưu ý: Môi trường demo công khai. Không nhập thông tin nhạy cảm.*
+
+---
+
 ## Nền tảng được hỗ trợ
 
 | Nền tảng | Trạng thái | Ghi chú |
@@ -135,7 +145,7 @@ docker compose --profile all up -d
 | [PPIO](https://ppinfra.com/user/register?invited_by=QJKFYD&utm_source=github_langbot) | Nền tảng GPU | ✅ |
 | [ShengSuanYun](https://www.shengsuanyun.com/?from=CH_KYIPP758) | Nền tảng GPU | ✅ |
 | [接口 AI](https://jiekou.ai/) | Cổng | ✅ |
-| [302.AI](https://share.302.ai/SuTG99) | Cổng | ✅ |
+| [302.AI](https://share.302ai.cn/SuTG99) | Cổng | ✅ |
 | [Qiniu](https://www.qiniu.com/ai/agent) | Cổng | ✅ |
 
 [→ Xem tất cả tích hợp](https://link.langbot.app/en/docs/features)
@@ -153,14 +163,6 @@ docker compose --profile all up -d
 
 ---
 
-## Demo trực tuyến
-
-**Thử ngay:** https://demo.langbot.dev/
-- Email: `demo@langbot.app`
-- Mật khẩu: `langbot123456`
-
-*Lưu ý: Môi trường demo công khai. Không nhập thông tin nhạy cảm.*
-
 ## Được xây dựng cho AI Agent 🤖
 
 LangBot **thân thiện với agent ngay từ thiết kế** —— các coding agent của bạn (Claude Code, Codex, Copilot, Cursor, …) có thể vận hành, mở rộng và triển khai LangBot với sự hỗ trợ hạng nhất:
@@ -182,12 +184,6 @@ LangBot **thân thiện với agent ngay từ thiết kế** —— các coding
 
 ---
 
-## Lịch sử Star
-
-[![Star History Chart](https://api.star-history.com/svg?repos=langbot-app/LangBot&type=Date)](https://star-history.com/#langbot-app/LangBot&Date)
-
----
-
 ## Người đóng góp
 
 Cảm ơn tất cả [người đóng góp](https://github.com/langbot-app/LangBot/graphs/contributors) đã giúp LangBot trở nên tốt hơn:

docker/docker-compose.yaml

@@ -14,6 +14,13 @@ services:
     restart: on-failure
     environment:
       - TZ=Asia/Shanghai
+      # Shared with the langbot service and sent only as a WebSocket handshake
+      # header. Generate with: openssl rand -hex 32
+      - LANGBOT_PLUGIN_RUNTIME_CONTROL_TOKEN=${LANGBOT_PLUGIN_RUNTIME_CONTROL_TOKEN:-}
+      # Process-wide admission for every asyncio.to_thread() call.
+      - LANGBOT_BLOCKING_EXECUTOR_MAX_WORKERS=${LANGBOT_BLOCKING_EXECUTOR_MAX_WORKERS:-8}
+      - LANGBOT_BLOCKING_EXECUTOR_MAX_PENDING=${LANGBOT_BLOCKING_EXECUTOR_MAX_PENDING:-128}
+      - LANGBOT_BLOCKING_EXECUTOR_MAX_INFLIGHT_PER_SCOPE=${LANGBOT_BLOCKING_EXECUTOR_MAX_INFLIGHT_PER_SCOPE:-4}
     command: ["uv", "run", "--no-sync", "-m", "langbot_plugin.cli.__init__", "rt"]
     networks:
       - langbot_network
@@ -40,9 +47,19 @@ services:
     restart: on-failure
     environment:
       - TZ=Asia/Shanghai
+      # Shared control-plane secret used to authenticate both the RPC socket
+      # and managed-process relay. Generate once (for example with
+      # ``openssl rand -hex 32``) and export it before enabling this profile.
+      # An empty value is accepted by Compose so Box can remain optional, but
+      # the Box runtime itself fails closed when the profile is started.
+      - LANGBOT_BOX_CONTROL_TOKEN=${LANGBOT_BOX_CONTROL_TOKEN:-}
+      # Box has its own process-wide blocking-work budget.
+      - LANGBOT_BLOCKING_EXECUTOR_MAX_WORKERS=${LANGBOT_BLOCKING_EXECUTOR_MAX_WORKERS:-8}
+      - LANGBOT_BLOCKING_EXECUTOR_MAX_PENDING=${LANGBOT_BLOCKING_EXECUTOR_MAX_PENDING:-128}
+      - LANGBOT_BLOCKING_EXECUTOR_MAX_INFLIGHT_PER_SCOPE=${LANGBOT_BLOCKING_EXECUTOR_MAX_INFLIGHT_PER_SCOPE:-4}
       # The Box runtime does NOT read box.local.* from config.yaml or env; it
-      # receives its configuration from LangBot via the INIT RPC action.
-      # Do not add LANGBOT_BOX_* / BOX__* here — they would be silently ignored.
+      # receives its functional configuration from LangBot via the INIT RPC
+      # action. Do not add BOX__* here because those would be ignored.
     # Launched through the same CLI entry point as the plugin runtime
     # (`langbot_plugin.cli.__init__ <subcommand>`). WebSocket is the default
     # control transport — mirrors `rt`, which also runs with no flag. Pass
@@ -60,13 +77,25 @@ services:
     restart: on-failure
     environment:
       - TZ=Asia/Shanghai
+      # Must match langbot_plugin_runtime. Empty/missing values make the
+      # external control channel fail closed.
+      - LANGBOT_PLUGIN_RUNTIME_CONTROL_TOKEN=${LANGBOT_PLUGIN_RUNTIME_CONTROL_TOKEN:-}
+      # Must match the value supplied to langbot_box. The token is sent only
+      # in WebSocket handshake headers, never in URLs or action payloads.
+      - LANGBOT_BOX_CONTROL_TOKEN=${LANGBOT_BOX_CONTROL_TOKEN:-}
+      # Core process-wide blocking-work admission. These are native config
+      # overrides and are persisted with the effective data/config.yaml.
+      - SYSTEM__BLOCKING_EXECUTOR__MAX_WORKERS=${LANGBOT_BLOCKING_EXECUTOR_MAX_WORKERS:-8}
+      - SYSTEM__BLOCKING_EXECUTOR__MAX_PENDING=${LANGBOT_BLOCKING_EXECUTOR_MAX_PENDING:-128}
+      - SYSTEM__BLOCKING_EXECUTOR__MAX_INFLIGHT_PER_SCOPE=${LANGBOT_BLOCKING_EXECUTOR_MAX_INFLIGHT_PER_SCOPE:-4}
       # Unified env-override convention: SECTION__SUBSECTION__KEY overrides the
       # matching config.yaml field (see LoadConfigStage). These map onto
-      # box.local.* and are forwarded to the Box runtime via INIT RPC.
+      # box.* and are forwarded to the Box runtime via INIT RPC.
       - BOX__LOCAL__HOST_ROOT=${LANGBOT_BOX_ROOT:-${PWD}/data/box}
       - BOX__LOCAL__DEFAULT_WORKSPACE=default
       - BOX__LOCAL__SKILLS_ROOT=skills
       - BOX__LOCAL__ALLOWED_MOUNT_ROOTS=${LANGBOT_BOX_ROOT:-${PWD}/data/box}
+      - BOX__DOCKER__CPU_LIMIT_ENABLED=${LANGBOT_BOX_DOCKER_CPU_LIMIT_ENABLED:-true}
     ports:
       - 5300:5300  # For web ui and webhook callback
       - 2280-2285:2280-2285  # For platform reverse connection

docker/kubernetes.yaml

@@ -4,6 +4,10 @@
 # Full deployment guide (zh/en/ja): https://docs.langbot.app -> Installation -> Kubernetes
 #
 # Usage:
+#   kubectl -n langbot create secret generic langbot-plugin-runtime-control \
+#     --from-literal=token="$(openssl rand -hex 32)"
+#   kubectl -n langbot create secret generic langbot-box-control \
+#     --from-literal=token="$(openssl rand -hex 32)"
 #   kubectl apply -f kubernetes.yaml
 #
 # Prerequisites:
@@ -87,6 +91,12 @@ metadata:
 data:
   TZ: "Asia/Shanghai"
   PLUGIN__RUNTIME_WS_URL: "ws://langbot-plugin-runtime:5400/control/ws"
+  SYSTEM__BLOCKING_EXECUTOR__MAX_WORKERS: "8"
+  SYSTEM__BLOCKING_EXECUTOR__MAX_PENDING: "128"
+  SYSTEM__BLOCKING_EXECUTOR__MAX_INFLIGHT_PER_SCOPE: "4"
+  LANGBOT_BLOCKING_EXECUTOR_MAX_WORKERS: "8"
+  LANGBOT_BLOCKING_EXECUTOR_MAX_PENDING: "128"
+  LANGBOT_BLOCKING_EXECUTOR_MAX_INFLIGHT_PER_SCOPE: "4"
   # Box sandbox runtime endpoint. LangBot connects to the Box runtime over
   # WebSocket. The hostname MUST match the langbot-box Service name. Note the
   # in-container default ("langbot_box") uses an underscore, which is an
@@ -127,6 +137,26 @@ spec:
             configMapKeyRef:
               name: langbot-config
               key: TZ
+        - name: LANGBOT_PLUGIN_RUNTIME_CONTROL_TOKEN
+          valueFrom:
+            secretKeyRef:
+              name: langbot-plugin-runtime-control
+              key: token
+        - name: LANGBOT_BLOCKING_EXECUTOR_MAX_WORKERS
+          valueFrom:
+            configMapKeyRef:
+              name: langbot-config
+              key: LANGBOT_BLOCKING_EXECUTOR_MAX_WORKERS
+        - name: LANGBOT_BLOCKING_EXECUTOR_MAX_PENDING
+          valueFrom:
+            configMapKeyRef:
+              name: langbot-config
+              key: LANGBOT_BLOCKING_EXECUTOR_MAX_PENDING
+        - name: LANGBOT_BLOCKING_EXECUTOR_MAX_INFLIGHT_PER_SCOPE
+          valueFrom:
+            configMapKeyRef:
+              name: langbot-config
+              key: LANGBOT_BLOCKING_EXECUTOR_MAX_INFLIGHT_PER_SCOPE
         volumeMounts:
         - name: plugin-data
           mountPath: /app/data/plugins
@@ -139,7 +169,8 @@ spec:
             cpu: "1000m"
         # Liveness probe to restart container if it becomes unresponsive
         livenessProbe:
-          tcpSocket:
+          httpGet:
+            path: /healthz
             port: 5400
           initialDelaySeconds: 30
           periodSeconds: 10
@@ -147,7 +178,8 @@ spec:
           failureThreshold: 3
         # Readiness probe to know when container is ready to accept traffic
         readinessProbe:
-          tcpSocket:
+          httpGet:
+            path: /healthz
             port: 5400
           initialDelaySeconds: 10
           periodSeconds: 5
@@ -246,9 +278,28 @@ spec:
             configMapKeyRef:
               name: langbot-config
               key: TZ
+        - name: LANGBOT_BOX_CONTROL_TOKEN
+          valueFrom:
+            secretKeyRef:
+              name: langbot-box-control
+              key: token
+        - name: LANGBOT_BLOCKING_EXECUTOR_MAX_WORKERS
+          valueFrom:
+            configMapKeyRef:
+              name: langbot-config
+              key: LANGBOT_BLOCKING_EXECUTOR_MAX_WORKERS
+        - name: LANGBOT_BLOCKING_EXECUTOR_MAX_PENDING
+          valueFrom:
+            configMapKeyRef:
+              name: langbot-config
+              key: LANGBOT_BLOCKING_EXECUTOR_MAX_PENDING
+        - name: LANGBOT_BLOCKING_EXECUTOR_MAX_INFLIGHT_PER_SCOPE
+          valueFrom:
+            configMapKeyRef:
+              name: langbot-config
+              key: LANGBOT_BLOCKING_EXECUTOR_MAX_INFLIGHT_PER_SCOPE
         # The Box runtime does NOT read box.local.* / BOX__* from its own env;
-        # it receives its configuration from LangBot via the INIT RPC action.
-        # Do not add BOX__* here — they would be silently ignored.
+        # it receives its functional configuration from LangBot via INIT.
         volumeMounts:
         # Box workspace root — identical path on node, box, and sandbox
         # containers (see the IMPORTANT note above).
@@ -265,14 +316,18 @@ spec:
             memory: "1Gi"
             cpu: "1000m"
         livenessProbe:
-          tcpSocket:
+          httpGet:
+            path: /healthz
             port: 5410
           initialDelaySeconds: 20
           periodSeconds: 10
           timeoutSeconds: 5
           failureThreshold: 3
         readinessProbe:
-          tcpSocket:
+          httpGet:
+            # Unlike liveness, readiness validates the configured backend and
+            # all strict managed-mode isolation guarantees.
+            path: /readyz
             port: 5410
           initialDelaySeconds: 10
           periodSeconds: 5
@@ -319,6 +374,10 @@ metadata:
     app: langbot
 spec:
   replicas: 1
+  # Plugin Runtime has a single active LangBot control owner. Recreate avoids
+  # two LangBot pods fighting over that connection during a rolling update.
+  strategy:
+    type: Recreate
   selector:
     matchLabels:
       app: langbot
@@ -352,6 +411,26 @@ spec:
             configMapKeyRef:
               name: langbot-config
               key: PLUGIN__RUNTIME_WS_URL
+        - name: LANGBOT_PLUGIN_RUNTIME_CONTROL_TOKEN
+          valueFrom:
+            secretKeyRef:
+              name: langbot-plugin-runtime-control
+              key: token
+        - name: SYSTEM__BLOCKING_EXECUTOR__MAX_WORKERS
+          valueFrom:
+            configMapKeyRef:
+              name: langbot-config
+              key: SYSTEM__BLOCKING_EXECUTOR__MAX_WORKERS
+        - name: SYSTEM__BLOCKING_EXECUTOR__MAX_PENDING
+          valueFrom:
+            configMapKeyRef:
+              name: langbot-config
+              key: SYSTEM__BLOCKING_EXECUTOR__MAX_PENDING
+        - name: SYSTEM__BLOCKING_EXECUTOR__MAX_INFLIGHT_PER_SCOPE
+          valueFrom:
+            configMapKeyRef:
+              name: langbot-config
+              key: SYSTEM__BLOCKING_EXECUTOR__MAX_INFLIGHT_PER_SCOPE
         # Box (sandbox) runtime endpoint. Connects LangBot to the langbot-box
         # Service over WebSocket. Remove this (and the langbot-box Deployment)
         # and set BOX__ENABLED=false if you do not want the sandbox.
@@ -360,6 +439,13 @@ spec:
             configMapKeyRef:
               name: langbot-config
               key: BOX__RUNTIME__ENDPOINT
+        # Same Secret as langbot-box. It authenticates the RPC and managed-
+        # process relay handshakes and is never put in a URL or RPC payload.
+        - name: LANGBOT_BOX_CONTROL_TOKEN
+          valueFrom:
+            secretKeyRef:
+              name: langbot-box-control
+              key: token
         # box.local.* config — forwarded to the Box runtime via INIT RPC. The
         # host_root MUST match the box-root hostPath mountPath below AND the box
         # Deployment's box-root mountPath, so that skill package paths resolve
@@ -392,7 +478,7 @@ spec:
         # Liveness probe to restart container if it becomes unresponsive
         livenessProbe:
           httpGet:
-            path: /
+            path: /healthz
             port: 5300
           initialDelaySeconds: 60
           periodSeconds: 10
@@ -401,7 +487,7 @@ spec:
         # Readiness probe to know when container is ready to accept traffic
         readinessProbe:
           httpGet:
-            path: /
+            path: /healthz
             port: 5300
           initialDelaySeconds: 30
           periodSeconds: 5

docs/API_KEY_AUTH.md

@@ -8,13 +8,21 @@ API keys can be managed through the web interface:
 
 1. Log in to the LangBot web interface
 2. Click the "API Keys" button at the bottom of the sidebar
-3. Create, view, copy, or delete API keys as needed
+3. Create an API key and copy its secret immediately
+4. Revoke keys that are no longer needed
+
+Database-backed API-key secrets are returned exactly once. LangBot stores only
+a SHA-256 lookup hash, so an existing secret cannot be displayed or recovered
+later. Each key belongs to one Workspace, has explicit permission scopes, and
+may have an expiry. The Workspace is derived from the authenticated key; an
+`X-Workspace-Id` header cannot redirect it to another tenant.
 
 ## Global API Key (config.yaml)
 
 In addition to web-UI-created keys (stored in the database, prefixed `lbk_`),
 LangBot supports a **global API key** defined directly in `data/config.yaml`.
-This is useful for automated deployments, infrastructure-as-code, and AI agents
+This is a Community-edition bootstrap option for automated deployments,
+infrastructure-as-code, and AI agents
 that need API/MCP access **without a login session and without creating a
 database record first**.
 
@@ -27,10 +35,12 @@ api:
 
 Behavior:
 
-- When `api.global_api_key` is a non-empty string, that exact value is accepted
-  anywhere a normal API key is accepted — the `X-API-Key` header or
-  `Authorization: Bearer <key>` — across the HTTP service API **and the MCP
-  server**.
+- In Community edition's singleton Workspace, a non-empty
+  `api.global_api_key` is bound to that Workspace and accepted across the HTTP
+  service API and the MCP server.
+- The global config key is rejected when multi-Workspace SaaS mode is enabled;
+  SaaS automation must use a database-backed Workspace key or a closed control
+  plane credential.
 - The global key does **not** require the `lbk_` prefix; use any sufficiently
   strong secret.
 - Leave it empty (`''`, the default) to disable it entirely; only database-backed
@@ -38,9 +48,10 @@ Behavior:
 - Existing installs are unaffected until you add the key — config completion only
   backfills top-level keys, and the lookup is defensive when the field is absent.
 
-> **Security:** the global key is stored in plaintext in `config.yaml`. Only
-> enable it on trusted/internal deployments, keep the file permissions tight,
-> always serve over HTTPS, and rotate the value if it may have leaked.
+> **Security:** the global key is stored in plaintext in `config.yaml` and has
+> the singleton Workspace's full fixed permission set. Only enable it on
+> trusted/internal Community deployments, keep file permissions tight, always
+> serve over HTTPS, and rotate it if it may have leaked.
 
 ## Using API Keys
 
@@ -60,7 +71,9 @@ Authorization: Bearer lbk_your_api_key_here
 
 ## Available APIs
 
-All existing LangBot APIs now support **both user token and API key authentication**. This means you can use API keys to access:
+Endpoints that declare API-key authentication accept either a user token or a
+Workspace API key. The key must include the permission required by the route.
+This includes:
 
 - **Model Management** - `/api/v1/provider/models/llm` and `/api/v1/provider/models/embedding`
 - **Bot Management** - `/api/v1/platform/bots`
@@ -227,6 +240,11 @@ or
 }
 ```
 
+### 403 Forbidden
+
+The key is valid for its Workspace but does not include the fixed permission
+required by the route.
+
 ### 500 Internal Server Error
 
 ```json
@@ -240,7 +258,7 @@ or
 
 1. **Keep API keys secure**: Store them securely and never commit them to version control
 2. **Use HTTPS**: Always use HTTPS in production to encrypt API key transmission
-3. **Rotate keys regularly**: Create new API keys periodically and delete old ones
+3. **Rotate keys regularly**: Create new API keys periodically and revoke old ones
 4. **Use descriptive names**: Give your API keys meaningful names to track their usage
 5. **Delete unused keys**: Remove API keys that are no longer needed
 6. **Use X-API-Key header**: Prefer using the `X-API-Key` header for clarity
@@ -317,7 +335,6 @@ curl -X POST \
 
 ## Notes
 
-- The same endpoints work for both the web UI (with user tokens) and external services (with API keys)
+- API-key-enabled endpoints use the same resource shapes as the web UI
 - No need to learn different API paths - use the existing API documentation with API key authentication
-- All endpoints that previously required user authentication now also accept API keys
-
+- API keys never select a Workspace from a request header; their persisted binding is authoritative

docs/VALKEY_SEARCH_INTEGRATION.md

@@ -0,0 +1,171 @@
+# Valkey Search Vector Database Integration
+
+This document describes how to use **Valkey Search** (the search/vector module bundled in
+`valkey/valkey-bundle`) as the vector database backend for LangBot's knowledge base (RAG)
+feature.
+
+## What is Valkey Search?
+
+**Valkey Search** is a module that adds vector similarity search and full-text search to
+[Valkey](https://valkey.io/), the open-source, BSD-licensed in-memory data store forked from
+Redis OSS. It is distributed in the `valkey/valkey-bundle` image alongside other modules
+(JSON, Bloom, LDAP).
+
+LangBot talks to Valkey through the official [`valkey-glide`](https://pypi.org/project/valkey-glide/)
+client (Rust core + async Python wrapper), using its native `ft` (search) command namespace.
+
+### Key Features
+
+- **Vector search**: ANN via HNSW or exact via FLAT, with COSINE / L2 / IP distance metrics
+- **Full-text search**: term, prefix and phrase matching over indexed text fields
+- **Hybrid search**: a metadata/text filter pre-selects candidates, then KNN ranks them
+- **In-memory speed**: vectors and documents are stored as Valkey HASH keys
+- **Auth + TLS**: optional username/password and TLS for production (toB / SaaS) deployments
+
+### Licensing
+
+- Valkey core and the Search module are **BSD-3-Clause**.
+- The `valkey-glide` client is **Apache-2.0**.
+
+Both are compatible with LangBot.
+
+## Installation
+
+Valkey Search support is included automatically on Linux and macOS. The official `valkey-glide`
+client does not currently publish a Windows package, so LangBot skips this optional dependency on
+Windows; LangBot remains usable there, but the Valkey Search backend is unavailable. To install the
+client manually on a supported platform:
+
+```bash
+pip install 'valkey-glide>=2.4.1,<3.0.0'
+```
+
+You also need a running Valkey server with the Search module loaded. The simplest way is the
+bundled image:
+
+```bash
+# Run valkey-bundle (includes the Search module) on host port 6380
+podman run -d --name valkey-test-langbot -p 6380:6379 valkey/valkey-bundle:9.1.0
+# (docker run ... works identically)
+```
+
+`valkey-bundle` ships multi-arch images (linux/amd64 + linux/arm64), so it runs on both CI
+(x86_64) and Apple-silicon dev machines.
+
+## Configuration
+
+Valkey Search is **opt-in and disabled by default** — the default `vdb.use` stays `chroma`,
+so existing single-process deployments are unaffected. To enable it, edit your `config.yaml`:
+
+```yaml
+vdb:
+  use: valkey_search
+  valkey_search:
+    host: 'localhost'
+    port: 6379            # use 6380 if you started the container as shown above
+    db: 0
+    password: ''          # optional (ACL / requirepass) — never logged
+    username: ''          # optional (ACL user)
+    tls: false            # optional (toB / SaaS)
+    index_algorithm: 'HNSW'   # HNSW | FLAT
+    distance_metric: 'COSINE' # COSINE | L2 | IP
+    request_timeout: 5000     # per-request timeout in ms
+```
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `host` | `localhost` | Valkey host |
+| `port` | `6379` | Valkey port |
+| `db` | `0` | Logical database id |
+| `password` | `''` | Optional auth password (empty = no auth). Never logged. |
+| `username` | `''` | Optional ACL username. Configuring a username without a password fails closed (raises) rather than connecting unauthenticated. |
+| `tls` | `false` | Enable TLS for the connection |
+| `index_algorithm` | `HNSW` | `HNSW` (approximate) or `FLAT` (exact) |
+| `distance_metric` | `COSINE` | `COSINE`, `L2`, or `IP` |
+| `request_timeout` | `5000` | Per-request timeout in milliseconds. The valkey-glide default (250ms) is too low for vector KNN under load; raise it further for remote/cross-AZ Valkey. |
+
+### Connection behavior
+
+The backend uses a **lazy** connection (`lazy_connect=True`): the client is created on first
+use and the connection is deferred to the first command. A misconfigured or unreachable Valkey
+server therefore does **not** block LangBot from booting — knowledge-base operations will error
+at call time instead, and you can recover by switching `vdb.use` back to another backend.
+
+The connection sets a fixed `client_name` of `langbot_vector_client` so it is identifiable in
+`CLIENT LIST` and monitoring dashboards.
+
+## Supported search types
+
+| Type | Behavior |
+|------|----------|
+| `vector` | Pure KNN over the embedding field |
+| `full_text` | Term/phrase match over the indexed `document` text field |
+| `hybrid` | Metadata/text filter **pre-selects** candidates, then KNN ranks them |
+
+### ⚠️ Important: `vector_weight` is NOT honored
+
+Valkey Search hybrid queries follow a **filter-then-KNN** model: the filter (and/or full-text
+clause) narrows the candidate set, and the KNN stage ranks the survivors by vector distance.
+There is **no native weighted score fusion** (unlike, e.g., SeekDB's RRF boost).
+
+For interface compatibility the backend still accepts a `vector_weight` argument, but it is
+**ignored** — passing different weights does not change result ordering. The first time a
+non-default weight is supplied, the backend logs a one-time warning.
+
+If weighted hybrid ranking is needed in the future, it can be added **application-side** (run
+vector KNN and full-text search separately and blend the scores). That is intentionally out of
+scope for this integration.
+
+## Metadata & filtering
+
+Documents are stored as Valkey HASH keys under the prefix `kb:{collection}:{id}` with fields:
+
+- `vector` — the embedding, packed as little-endian FLOAT32
+- `document` — the raw text (indexed as TEXT for full-text/hybrid search)
+- `file_id` — promoted to an indexed TAG field so it is filterable
+- `metadata_json` — the full metadata dict, preserved verbatim as JSON
+
+Only **indexed** fields are filterable. Currently that is `file_id`. Filters referencing
+non-indexed metadata keys are dropped with a warning (the same pragmatism used by the Milvus
+and pgvector backends). All other metadata still round-trips intact via `metadata_json`.
+
+Supported filter operators (canonical Chroma-style `where` syntax): `$eq`, `$ne`, `$gt`,
+`$gte`, `$lt`, `$lte`, `$in`, `$nin`. Multiple top-level keys are AND-ed.
+
+## Testing
+
+Unit tests (filter mapping, float32 packing, reply parsing, import guard) run in the fast lane
+with no server:
+
+```bash
+uv run pytest tests/unit_tests/vector/test_valkey_search_filter.py -q
+```
+
+Integration tests are **slow-gated** on `TEST_VALKEY_URL` and require a running server:
+
+```bash
+podman run -d --name valkey-test-langbot -p 6380:6379 valkey/valkey-bundle:9.1.0
+TEST_VALKEY_URL=valkey://localhost:6380 \
+    uv run pytest tests/integration/vector/test_valkey_search.py -m slow -q
+```
+
+The default upstream fast CI lane (`-m "not slow"`) skips these, matching the existing
+PostgreSQL migration-test precedent.
+
+## Troubleshooting
+
+| Symptom | Cause / fix |
+|---------|-------------|
+| Tests skip with "Valkey Search module not available" | The server is plain Valkey without the Search module. Use the `valkey/valkey-bundle` image. |
+| `ConnectionError` at call time | Check `host`/`port`/auth; remember `lazy_connect` defers errors to first use. |
+| Empty search results right after insert | The Search indexer is asynchronous; results become visible within a short delay. The integration tests poll/retry to account for this. |
+| Hybrid ranking ignores `vector_weight` | Expected — see the caveat above. |
+
+## Production considerations
+
+- **Cluster mode**: Valkey Search in cluster mode uses an additional coordination port. This
+  integration targets standalone mode; cluster support is a future consideration.
+- **Persistence**: configure Valkey RDB/AOF persistence if the knowledge base must survive
+  restarts; otherwise an in-memory store is ephemeral.
+- **Security**: set `password`/`username` and `tls: true` for any non-local deployment.
+  Credentials are never written to logs.

docs/multi-tenant/cloud-runtime-soak-gate.md

@@ -0,0 +1,114 @@
+# LangBot Cloud 24 小时资源 Soak 门禁
+
+`scripts/cloud_runtime_soak.py` 是生产候选拓扑的最终资源稳定性门禁。它不替代单元测试、历史 churn 探针或 nsjail 隔离测试；它把以下三类证据按同一时间轴采集并给出可机读的 pass/fail：
+
+- Core、Plugin Runtime 和 Box Runtime 的 HTTP liveness/readiness。
+- 三个 Python 进程的 event-loop recent max/p95 调度延迟。
+- Linux `/proc` 进程树的 current RSS、累计 CPU、线程、文件描述符和子进程数。
+- cgroup v2 的 `memory.current/peak/events`、swap、CPU usage/throttling、PID current/events 和实际硬限制。
+
+生产批准必须使用 cgroup 证据。`--pid` 只适合本地诊断，因为进程指标无法证明 OOM kill、PID limit 或 CPU throttling。
+
+## 运行位置
+
+建议把采集器放在独立的 node agent 或监控 sidecar 中，并只读挂载三个目标容器的 cgroup 路径。不要把采集器和样本文件放进被测容器自己的 cgroup/数据卷，否则采集器的 CPU、内存和 page cache 会污染目标数据。
+
+Kubernetes/containerd 生成的 cgroup 路径不是稳定 API。每次生产候选部署都必须从实际 pod/container ID 解析，不能从 pod 名猜路径。传入的每个目录都必须至少可读：
+
+- `memory.current`、`memory.events`
+- `cpu.stat`、`cpu.max`
+- `pids.current`、`pids.events`
+
+最终门禁应加 `--require-hard-limits`。该选项要求每个目标 cgroup 都能观察到有限的 CPU quota、memory、swap 和 PID 上限；任一值为 `max` 都失败。
+
+## 标准 24 小时命令
+
+```bash
+uv run python scripts/cloud_runtime_soak.py \
+  --duration 24h \
+  --startup-grace 5m \
+  --sample-interval 15s \
+  --cooldown 30m \
+  --analysis-window 30m \
+  --http-timeout 5s \
+  --max-memory-growth-mib 64 \
+  --max-memory-slope-mib-per-hour 32 \
+  --max-tail-cpu-cores 0.5 \
+  --max-throttled-period-ratio 0.25 \
+  --max-event-loop-lag-ms 1000 \
+  --max-event-loop-p95-lag-ms 250 \
+  --require-hard-limits \
+  --endpoint core=http://langbot:5300/healthz \
+  --endpoint plugin=http://langbot-plugin-runtime:5400/healthz \
+  --endpoint box=http://langbot-box:5410/readyz \
+  --cgroup core=/host-cgroup/CURRENT_CORE_CONTAINER \
+  --cgroup plugin=/host-cgroup/CURRENT_PLUGIN_RUNTIME_CONTAINER \
+  --cgroup box=/host-cgroup/CURRENT_BOX_RUNTIME_CONTAINER \
+  --samples-file artifacts/cloud-soak-samples.jsonl \
+  --report-file artifacts/cloud-soak-report.json \
+  --workload uv run python tests/load/cloud_candidate_workload.py
+```
+
+`--duration` 是包含启动观察、负载和冷却期的最大墙钟时间。工作负载必须在截止时间前退出并至少留出 30 分钟冷却；否则门禁会终止负载并失败。若负载由外部系统控制，可以省略 `--workload`，但必须保证最后 `--analysis-window` 完全无测试流量，该窗口才可解释为空闲尾段。
+
+工作负载命令的 stdout/stderr 会转发到采集器 stderr，不会混入 stdout 的最终 JSON 报告。命令以独立 process group 启动；超时或中断时整组收到 TERM，10 秒后仍未退出则收到 KILL。
+
+凭据只能通过 workload 进程环境或 secret mount 注入，不能放在命令参数中。最终报告只记录可执行文件名和参数个数，不保存参数正文；采集器也拒绝带 userinfo、query 或 fragment 的健康 URL。
+
+## 必须覆盖的负载
+
+同一候选版本至少要覆盖：
+
+1. 大批 Workspace 注册、成员邀请、登录和 entitlement 刷新。
+2. Plugin installation reconcile、依赖准备、正常调用、进程崩溃与重启。
+3. Dashboard/Embed/平台 WebSocket 建连、突发消息和批量断连。
+4. Box session、文件同步、并发 exec、managed-process 输出和清理。
+5. PostgreSQL pool 接近容量、事务超时和恢复。
+6. Core、Plugin Runtime、Box 分别收到 SIGTERM 后的优雅重启。
+
+工作负载不能把 API 过载拒绝当作成功吞掉。默认情况下，Core health 中 blocking executor 的 global/scope rejection counter 只要增长，门禁即失败；只有专门验证“过载会正确返回 429”的独立测试才可以使用 `--allow-rejections`，该次运行不能作为生产批准证据。
+
+## 判定规则
+
+整个有效观察期内出现以下任一情况即失败：
+
+- 健康接口请求失败、非 2xx、Core `code != 0`，或 Box `ready=false`。
+- `memory.events.high/max/oom/oom_kill/oom_group_kill` 增长。
+- `pids.events.max` 增长。
+- cgroup 单调计数器回退，表示目标很可能发生了未记录的重启或 cgroup 替换。
+- CPU throttled-period ratio 超过配置阈值。
+- 任一健康采样窗口的 event-loop recent max 超过 1 秒，或冷却尾段 recent p95 超过 250 ms。
+- 健康接口缺少 event-loop monitor、monitor 未持续运行，或其 sample counter 回退。
+- blocking executor rejection counter 增长。
+- Plugin Runtime restart circuit 的累计打开次数增长。
+- Core 目录 active Workspace、最近 snapshot/delta Workspace 或 membership 基数
+  超过各自配置上限，或 PostgreSQL `checked_out` 超过配置 pool 容量；相关 current/max
+  指标只出现一半或 max 非法也失败。
+
+负载结束后的冷却尾段还必须满足：
+
+- `memory.current`/RSS 的稳健首尾增长和线性斜率不能同时超过阈值。
+- 平均 CPU 核数不超过 `--max-tail-cpu-cores`。
+- event-loop recent p95 不超过 `--max-event-loop-p95-lag-ms`。
+- blocking executor `pending` 至少回到过零；不能整个尾段持续积压。
+- Plugin Runtime restart coordinator 的 active launch、half-open probe 和
+  circuit open remaining time 必须回到零，`gate_waiters` 必须至少归零一次。
+- Core 的 MCP projection retirement queue/worker 和 message aggregation
+  buffer/scope 必须至少归零一次。
+- telemetry、QueryPool、MCP host/dispatch、Box creating/closing/background 等临时 gauge 不能继续增长。
+
+内存判定要求“增长量”和“斜率”同时越界，避免几 MiB allocator/page-cache 噪声在短窗口被外推成很大的每小时斜率。最终报告仍保留实际增长与斜率，人工审查时不能只看 verdict。
+
+## 产物与退出码
+
+- `--samples-file`：逐样本 JSONL，写入后立即 flush，供时序图和故障定位。
+- `--report-file`：最终汇总、阈值、资源硬限制、OOM/PID/throttle delta、尾段斜率和 workload 状态。
+- stdout：与 report 文件相同的最终 JSON；workload 日志只写 stderr。
+
+退出码：
+
+- `0`：全部门禁通过。
+- `1`：采样完成但资源门禁失败。
+- `2`：CLI 参数或目标配置错误。
+
+必须保存原始 JSONL、最终报告、三个镜像 digest、LangBot/SDK commit、生产配置摘要和工作负载版本。滚动更新、节点迁移或镜像变化后，旧报告不能继续作为新候选版本的批准证据。

docs/multi-tenant/cloud-v2-pending-verification.md

@@ -0,0 +1,273 @@
+# Cloud v2 仍待验证事项
+
+状态：`NOT APPROVED FOR SAAS ACTIVATION`
+
+更新日期：2026-07-29
+
+本文是 Cloud v2 首期上线前的剩余验证清单。它只记录尚不能由当前代码审查、
+单元测试、集成测试、合成容量探针或短时 Linux 容器实验替代的证据。这里的项目
+不属于 2026-07-29 代码与本地测试资源审查的完成条件，也不会让该审查持续保持未完成；
+它们只在准备最终 SaaS 激活时重新进入验收范围。
+
+相关文档：
+
+- [多租户架构决策](./pending-architecture-decisions.md)
+- [实现决策记录](./implementation-decisions.md)
+- [Runtime 资源安全审查](./runtime-resource-audit-2026-07-28.md)
+- [24 小时资源 Soak 门禁](./cloud-runtime-soak-gate.md)
+
+## 1. 当前已形成的交付基线
+
+- LangBot Core 全量 `2855 passed, 33 skipped`，Plugin SDK 全量
+  `1328 passed`，闭源适配器 `40 passed`，Space Go 全量测试通过；三仓格式、
+  静态检查和 `git diff --check` 已通过。
+- Plugin Runtime 和 Box Runtime 的公开健康接口、event-loop lag 与有界
+  blocking executor 指标已经过真实进程短时验证。
+- 仓库 Dockerfile 构建的 Linux/cgroup v2 短时探针已证明 CPU、memory、
+  swap 和 PID 限制代码路径可工作。
+- PostgreSQL 16 + RLS 的 1,000 Workspace 真实启动测试，以及 5,000
+  Workspace 三代替换合成探针已通过。
+- Core 已精确钉住 Plugin SDK 提交
+  `1d65ed301a6afc52150a998043f73cd6032c8162`。最终验证必须使用包含该提交的
+  Core、Plugin Runtime 和 Box Runtime 镜像，不能混用旧 SDK。
+- 独立资源复核已经移除 Cloud MCP 每会话 5 秒查询执行绑定的轮询，改由签名目录
+  投影提交后向一个合并回收任务发布代次变化；工具与资源调用前后仍使用数据库
+  execution fence。Plugin restart 冷却等待者、MCP 投影回收、消息聚合 buffer/scope
+  均已纳入健康快照和 soak 归零门禁。
+- 单实例目录现在有一致的操作容量契约：Space 在注册事务内通过 PostgreSQL
+  advisory lock 串行执行 active Workspace check-and-create；Space 全量快照只返回
+  active Workspace，并在查询阶段限制 Workspace/membership 数量；闭源适配器限制
+  解压后的 HTTP 响应字节和签名目录基数；Core 在持有目录投影行锁的事务内再次
+  COUNT active Workspace，超限时整批回滚且不推进 cursor。任何一层都不截断权威数据。
+- Core Cloud PostgreSQL pool 的 `pool_size + max_overflow` 有绝对上限 100，
+  Cloud runtime 连接默认强制 60 秒 statement/idle-transaction timeout 和 5 秒
+  lock timeout；pool 使用量、超时累计数与目录 active/max 基数进入 `/healthz`。
+  Box Runtime 的 session、process、admission record、RPC 文件和 completed retention
+  配置也有不可被实例配置放大的绝对上限。
+- Space 的 concurrent registration 容量准入已在一次性 PostgreSQL 16 上真实执行：
+  两个 Account 同时争用最后一个 Workspace 槽位时，精确一个事务成功、一个事务
+  得到 capacity error，最终 active Workspace 数为 1。active-only snapshot 与
+  archived delta tombstone 的同一真实 PostgreSQL 集成流程也通过。
+- Core Cloud manager 已连接一次性 PostgreSQL 16，并从 `pg_settings` 读回
+  `statement_timeout=60000ms`、`lock_timeout=5000ms` 和
+  `idle_in_transaction_session_timeout=60000ms`；测试结束后引擎已显式 dispose。
+- 独立异常路径复核已补齐 HTTPX 超限/取消时的底层流关闭；Monitoring 查询、导出和
+  detail 物化量均有实例上限与绝对上限，detail 统计使用数据库聚合。Token statistics
+  不再拉取全部历史 LLM call 在 Python 中分桶，而由 PostgreSQL/SQLite 聚合并只返回
+  有界的最新时间桶和模型分组，截断状态在响应中显式可见。邀请、Monitoring 和 Storage
+  周期清理已合并为一个先等待首个 interval 的调度器，同一周期只进行一次 Workspace
+  discovery；数据库删除批次和本地/S3 文件候选也有每轮硬上限。
+
+以上结果是进入生产候选验证的前提，不是 SaaS 上线批准。
+
+## 2. 尚有实现前置条件的阻断项
+
+以下项目不是“再跑一次测试”即可关闭。必须先完成实现，再执行对应验收。
+
+| 编号 | 阻断项 | 完成实现后的最低验收证据 |
+| --- | --- | --- |
+| B-01 | Cloud 插件缺少生产 egress policy | 证明插件只能访问允许的公网目标，不能访问 Core/Box/数据库、其他内部服务、loopback、link-local 或云 metadata endpoint |
+| B-02 | Plugin installation 与 Box Workspace/Skill/root/tmp/home 缺少真实的 byte 和 inode 硬配额 provider | 在写入边界原子拒绝超额；并发写入、重启和配额耗尽后不能越界，也不能用目录扫描或事后清理冒充硬配额 |
+| B-03 | 普通业务写入尚未具备贯穿 commit 的 generation-aware fence、同事务 business outbox，以及 generation cutover 后稳定的 durable-object 引用 | 在旧 generation 与新 generation 并发、事务提交竞态和重复投递下，旧 owner 不产生业务写入或外部副作用，outbox 可幂等恢复 |
+
+任一 B 类项目未关闭时，不得把 24 小时 soak 的通过结果解释为可以上线。
+
+## 3. 最终部署环境验证
+
+### V-01：Plugin Runtime 与 Box 的 Linux 隔离
+
+必须在最终 Cloud Pod security context、容器 runtime 和 cgroup 拓扑中验证，
+不能使用开发机或权限不同的一次性容器替代。
+
+验证内容：
+
+1. nsjail 可以建立 mount、PID、IPC、UTS namespace 和 private `/proc`。
+2. delegated cgroup v2 对每个插件进程和 sandbox 强制 CPU、memory、
+   `memory.swap.max=0` 和 PID 上限。
+3. open files、process 和单文件大小 rlimit 生效。
+4. 插件不能枚举、读取或 signal Runtime 及其他 installation 的进程，
+   不能读取其他 installation 的 home/tmp/data，也不能修改共享只读
+   artifact/environment。
+5. 超额只杀死或拒绝当前 installation/sandbox；Runtime、其他租户和健康接口
+   继续工作。
+6. 进程退出、取消、超时、generation 切换和 Pod SIGTERM 后，cgroup、nsjail
+   目录、子进程和文件描述符均被回收。
+7. 硬限制或 namespace 能力缺失时 readiness 失败关闭，不允许降级成普通进程。
+
+通过证据必须包含实际容器安全配置、cgroup 文件值、探针原始输出和失败注入结果。
+
+### V-02：Box 持久卷与硬存储配额
+
+在 B-02 的 quota provider 实现后，必须验证：
+
+1. Core 与 Box Runtime 通过随机 marker challenge 证明使用同一共享持久卷。
+2. Workspace、Skill store、ephemeral root/tmp/home 的 byte 和 inode quota
+   都在写入点生效。
+3. 并发写、压缩包展开、文件同步、重启恢复和删除重建不能绕过配额。
+4. 配额耗尽只影响目标 Workspace，其他 Workspace 仍能执行。
+5. 任一硬存储能力缺失时 `/readyz` 返回非 2xx，Pod 不进入就绪流量。
+
+### V-03：PostgreSQL 与 pgvector 生产边界
+
+必须在最终 PostgreSQL endpoint、凭据和网络策略下验证：
+
+1. migrator 与 runtime 使用不同 role；runtime role 无 superuser、
+   `BYPASSRLS`、DDL、对象所有权、role membership 或额外 schema 权限。
+2. runtime credential 只能连接目标 business database。需要专用
+   cluster/endpoint，或经过测试的 HBA/proxy policy；database 内 catalog
+   audit 本身不能证明这一点。
+3. release migration Job 的 advisory lock、失败重试、回滚和精确 Alembic head
+   校验有效；应用启动角色不能执行 migration 或其他 DDL。
+4. 若使用 PgBouncer，transaction pooling、异常回滚和连接复用不会残留
+   tenant context。
+5. 故意遗漏应用层 Workspace filter 时，RLS 仍阻止跨租户读写。
+6. 两个 Workspace 使用相同 `vector_id`、猜测其他 Workspace ID、后台任务和
+   连接复用时，pgvector CRUD 均不能越权。
+7. dimension mismatch、extension/schema/ACL drift 或 runtime audit 失败时，
+   Core 启动失败且不回退到其他向量后端。
+
+### V-04：最终镜像和配置一致性
+
+生产候选验证前必须固定：
+
+- Core、Plugin Runtime、Box Runtime 的不可变镜像 digest；
+- LangBot 与 SDK commit；
+- `data/config.yaml` 的非敏感摘要和所有环境变量覆写；
+- Space 的 `CLOUD_V2_MAX_DIRECTORY_WORKSPACES` 必须与 Core
+  `cloud.directory.max_active_workspaces` 和
+  `cloud.directory.max_snapshot_workspaces` 一致；Space 的 membership 上限必须与
+  Core `cloud.directory.max_snapshot_memberships` 一致；
+- Core 的 PostgreSQL pool、statement/lock/idle-transaction timeout，以及 Plugin
+  Runtime/Box Runtime 的全部实例级资源上限；
+- PostgreSQL migration revision；
+- Cloud Adapter、Space control plane 和 workload 的版本。
+
+滚动更新、节点迁移、配置变化或任一镜像 digest 变化后，旧验证报告失效。
+
+## 4. 多租户行为与故障注入
+
+### V-05：目录、entitlement 与 generation
+
+在真实 Space control plane、闭源 Cloud Adapter 和 Core 之间验证：
+
+1. 注册自动创建个人 Workspace、owner membership、Free subscription、
+   entitlement snapshot 和 outbox 事件，且重试不重复创建。
+2. 邀请、成员变更、套餐变更和 Workspace 撤销只影响目标 Workspace。
+3. 全量快照与增量事件覆盖乱序、重复、断点续传、缺页、签名错误、
+   high-water gap、snapshot coverage 和消费者重启。
+4. Core、Plugin Runtime 或 Box 重启后，权威 desired state 可恢复；
+   本地进程表和缓存不是唯一真相。
+5. generation/revision 切换期间，旧 callback、RPC、WebSocket、Box relay、
+   plugin worker 和缓存写入全部失败关闭。
+6. 为未来多副本预留的 replica-local cursor 语义通过故障注入：
+   一个副本追平不能使另一个副本跳过本地 cache 刷新。
+7. 同时使大量 plugin worker 因系统性故障退出，证明 restart launch 全局并发受限、
+   失败阈值触发 Runtime circuit、冷却后只有一个 half-open probe，且 probe 未稳定前
+   其他 installation 不会继续重启；冷却计时器/状态等待者数量不得超过全局 restart
+   并发，取消 probe 不得把 circuit 永久卡在 half-open；24 小时门禁必须把 circuit
+   打开或 `gate_waiters` 未归零判为失败。
+8. 使用大量空闲 remote MCP session 做 generation 切换，证明目录投影只创建一个
+   合并回收任务，不产生每 session 周期数据库查询、计时器或同时唤醒；旧 session
+   最终关闭，`mcp_projection_retirements` 和
+   `mcp_projection_reconcile_active` 在冷却期归零。
+
+### V-06：套餐、Box 与 stdio MCP
+
+1. Free/非 Pro Workspace 不会自动获得 managed sandbox。
+2. 合资格 Workspace 最多只有一个持久 `global` sandbox，且新增 Workspace
+   不创建专属 Runtime、Pod、PVC、database、schema、role 或连接池。
+3. Cloud 即使 `box.enabled=true`，也不能 create/update/test/start stdio MCP；
+   旧记录和直接 API 调用同样失败关闭，且不创建 `mcp-shared` session。
+4. OSS 默认仍是单 Workspace、多用户，stdio MCP 保持兼容，多租户能力不会被
+   未签名配置或普通环境变量开启。
+
+### V-07：跨租户安全回归
+
+至少使用两个恶意测试 Workspace 验证：
+
+- 同 digest 插件只共享只读代码和依赖；进程、secret、日志及所有可写目录隔离。
+- 同 author/name/version 但 digest 不同的 artifact 不共享目录。
+- Plugin Host API、Box RPC、对象 key、WebSocket、RAG、storage、model/session
+  cache 和平台回调不能接受调用方伪造的 Workspace scope。
+- 撤销 entitlement、删除 installation 或 generation 切换后，已有长连接和
+  in-flight 请求不能继续访问旧权限。
+
+## 5. 生产候选容量与 24 小时门禁
+
+### V-08：真实容量曲线
+
+现有 fake adapter/requester/Plugin handler 探针不能替代真实容量数据。必须使用
+计划上线的平台 SDK、外部 HTTP/WebSocket 连接池、真实插件进程、真实
+PostgreSQL/pgvector 和代表性 Workspace 配置分布，测量：
+
+- 空 Workspace、活跃 Workspace、每个启用插件和每个 Pro sandbox 的边际
+  RSS、线程、文件描述符、连接和 PostgreSQL pool 成本；
+- 启动、目录重放、批量 reconcile 和故障恢复的耗时与峰值；
+- remote MCP 数量增加及目录 generation 批量切换时的数据库 QPS、回收队列和
+  event-loop lag，确认不存在与 session 数量成比例的空闲轮询；
+- 在最大 retention/backlog 和并发 Dashboard 请求下执行不带时间范围的 Monitoring
+  overview/token statistics，验证 SQL 分桶、statement timeout、响应截断和 cleanup
+  追赶不会形成 PostgreSQL CPU 尖峰或 Core RSS 增长；
+- 单实例可批准的 Workspace、活跃 Bot、plugin worker 和 sandbox 上限。
+
+容量上限必须写入生产配置与告警，不能只保留在测试报告中。
+代码中的默认值和绝对上限只是失控配置的最后防线，不等于生产容量结论。V-08 必须
+根据最终镜像的真实曲线把 Space 与 Core 的匹配上限调到已验证容量以内；如果最终
+批准值高于当前默认 1,000 active Workspace，必须重新执行目录启动、故障恢复和
+24 小时门禁。
+
+### V-09：24 小时资源 soak
+
+使用 [标准 24 小时命令](./cloud-runtime-soak-gate.md#标准-24-小时命令)，并强制
+`--require-hard-limits`。工作负载至少覆盖：
+
+1. 注册、邀请、登录和 entitlement 刷新；
+2. plugin reconcile、依赖准备、调用、崩溃与重启；
+3. Dashboard/Embed/平台 WebSocket 建连、突发消息和断连；HTTP Bot 覆盖
+   高基数 session/idempotency、硬容量拒绝、空闲回收及 callback 堵塞；
+   remote MCP 覆盖大量空闲连接、批量 generation 切换和合并回收；
+4. Box session、文件同步、并发 exec、输出与清理；
+5. PostgreSQL pool 接近容量、事务超时和恢复；
+6. Core、Plugin Runtime、Box 分别 SIGTERM 和恢复。
+
+最后至少保留 30 分钟无测试流量冷却。任一健康失败、OOM/memory pressure、
+PID limit、blocking executor rejection、超阈值 CPU throttling/event-loop lag、
+目录 `active_workspaces > max_active_workspaces`、数据库 pool 使用量超过配置容量、
+冷却尾段内存持续增长，或 Plugin restart `gate_waiters`、MCP 投影回收、
+消息聚合 buffer/scope 等临时 gauge 不回落都判为失败。
+标准 soak 工具已自动比较目录 active/最近批次与各自配置上限，并比较 PostgreSQL
+`checked_out` 与配置 pool 容量；名为 `core` 的标准 endpoint 缺少任一容量指标、
+current/max 只出现一半、数值非法或任一样本越界都会直接失败。
+
+必须归档：
+
+- 原始 `cloud-soak-samples.jsonl`；
+- 最终 `cloud-soak-report.json`；
+- 三个镜像 digest、Core/SDK commit；
+- 生产配置摘要、数据库 migration revision 和 workload 版本；
+- 故障注入时间线及关联日志/trace。
+
+## 6. 本轮不作为验收条件的后续事项
+
+以下能力已明确暂缓，不能混入当前验证结果，也不能以“尚未验证”为理由临时发明方案：
+
+- Workspace export、释放、delete、单 Workspace restore 和在线迁移；
+- Workspace 级 BYOK E2B WebUI 配置；
+- 多 Core/Plugin Runtime/Box replica 的 lease store 与调度实现；
+- PostgreSQL 多 shard、dedicated shard 和跨地域部署；
+- 多 CloudInstance、Cell Router 或 Workspace Placement。
+
+这些事项需要后续单独决策。首期实现仍需保留稳定 UUID、generation fence、
+幂等事件和无副本地址泄漏的协议边界。
+
+## 7. 关闭规则
+
+每个 B/V 项只能通过以下方式关闭：
+
+1. 记录被测 commit、镜像 digest、配置摘要和环境拓扑；
+2. 保存可复现命令、原始输出和失败注入证据；
+3. 由报告明确给出 pass/fail，不能只依赖日志中“看起来正常”；
+4. 任一生产候选输入变化后，重跑受影响的验证。
+
+在 B-01 至 B-03 全部实现，且 V-01 至 V-09 均有当前生产候选版本的通过证据前，
+Cloud v2 状态保持 `NOT APPROVED FOR SAAS ACTIVATION`。

docs/multi-tenant/implementation-checklist.md

@@ -0,0 +1,292 @@
+# Multi-tenant implementation checklist
+
+This checklist turns the Workspace architecture into implementation and
+verification gates. Exact commands and observed results are recorded in the
+[verification report](./verification-report.md).
+
+## Scope guard
+
+- [x] LangBot uses branch feat/multi-tenants.
+- [x] langbot-plugin-sdk uses branch feat/multi-tenants.
+- [x] langbot-space implements the greenfield Cloud v2 modular-monolith control plane without extending the legacy per-account Pod topology; the old Pod UI remains available when Cloud v2 is disabled and only retained Pods appear in the v2 view.
+- [x] Unrelated untracked files in either repository remain untouched.
+- [x] Open-source startup cannot enable SaaS multi-workspace through edition flags or unsigned configuration.
+
+## SaaS activation gates
+
+These items intentionally remain incomplete. Some require additional Core
+transaction/cutover primitives and others require the closed Control Plane or
+deployment. The feature branch delivers the Core isolation kernel, not the
+closed SaaS product or a production Cloud v2 deployment. Checked implementation
+items later in this document do not supersede these gates.
+
+- [x] The closed Control Plane owns the global Account, Workspace, Membership, and Invitation directory.
+- [ ] The closed Control Plane execution-ownership module issues monotonic generations and owner leases for projected Workspaces.
+- [x] Core verifies a signed `InstanceManifest` before the closed bootstrap can inject `CloudWorkspacePolicy`.
+- [ ] Tenant database writes hold a generation-aware shared transaction fence through commit, while execution-owner cutovers take the exclusive fence.
+- [ ] Business writes and non-transactional side effects use a generation-stamped outbox or equivalent publish fence.
+- [ ] Durable object references survive an execution-generation change through stable published keys or an explicitly atomic key/reference migration.
+- [ ] The SaaS runtime pools enforce tenant-safe egress and SSRF controls for Webhooks, providers, MCP servers, and every tenant-configurable outbound URL.
+- [ ] Entitlement checks, usage aggregation, and subscription lifecycle are implemented in the closed Control Plane; production activation still requires provider callback amount/currency/session/expiry binding inside the locked fulfillment transaction.
+- [ ] Account registration persists a `new_api.provision_account` outbox item with the Account and personal Workspace, and an in-process reconciler provisions New API idempotently after commit.
+- [ ] EPay and Stripe callbacks bind provider identity, amount, currency, channel/session, payment status, and expiry to the locked order before entitlement fulfillment.
+- [ ] OAuth state and directory projection use an atomic shared store suitable for horizontally scaled SaaS services.
+- [ ] A greenfield Cloud v2 deployment is designed and validated independently of the legacy Space deployment scheme.
+- [ ] The Plugin Runtime shared profile refuses to run without delegated cgroup v2 CPU, memory-plus-swap, and PID limits, all verified in a real Linux container; production tenant-safe egress remains incomplete.
+- [x] The Plugin Runtime Supervisor automatically restores an unexpectedly exited enabled worker with bounded per-installation backoff.
+- [ ] Jitter, a global restart concurrency limit, and a Runtime-level circuit breaker prevent a systemic failure from creating a cross-tenant restart storm.
+- [ ] Until authenticated Runtime takeover or an owner lease/fence exists, the M0 deployment rolls Core and Plugin Runtime together and forbids an independent Core-only rollout.
+- [ ] Plugin installation data has an operator-owned hard disk quota provider that atomically rejects writes over the limit; directory scans are not accepted as enforcement.
+- [ ] The Box deployment provides an operator-owned quota provider that proves hard byte and inode limits for Workspace, Skill, root, tmp, and home storage.
+- [ ] Core and Box Runtime mount the same durable volume and pass the authenticated marker challenge during startup and reconnect.
+- [ ] Production provisions distinct migrator/runtime credentials and runs the implemented same-host/port/database release command as a one-shot Job, with tested orchestration retry, backup, and rollback procedures.
+- [ ] Production PostgreSQL uses a dedicated cluster/endpoint, or a tested HBA/proxy policy proves the cluster-wide runtime credential can connect only to the target business database.
+- [ ] Any future direct-migrator/pooler-runtime endpoint split is admitted only by a migrator-owned, runtime-read-only database cluster identity that the runtime role cannot spoof.
+- [ ] Legacy pgvector migration failure and retry integration paths prove exact source-table RLS/FORCE restoration; the non-superuser, non-`BYPASSRLS` success path is already covered below.
+- [ ] Multi-workspace is enabled in SaaS only after all closed Control Plane, deployment, and security gates pass.
+
+## 1. Persistence foundation
+
+### Account and directory
+
+- [x] User has a stable, unique account UUID and explicit status.
+- [x] Existing email and password behavior remains compatible during migration.
+- [x] Workspace table represents the instance-local tenant.
+- [x] WorkspaceMembership has a unique Workspace and Account pair.
+- [x] WorkspaceInvitation stores only a token hash and supports expiry, revoke, and one-time accept.
+- [x] WorkspaceExecutionState stores generation, state, source, and write fence.
+- [x] OSS initialization creates exactly one Workspace and one owner membership atomically.
+- [x] OSS refuses a second Workspace while allowing multiple members.
+
+### Migration
+
+- [x] Alembic migration upgrades SQLite.
+- [x] Alembic migration upgrades PostgreSQL.
+- [x] Existing first user becomes owner of the default Workspace.
+- [x] Existing tenant resources are backfilled with the default Workspace UUID.
+- [x] SQLite destructive boundaries create verified, revision-aware backups and atomically restore after failure.
+- [x] Migration can resume safely after interruption.
+- [x] New installs and upgraded installs produce the same tenancy-kernel schema.
+- [x] The first Cloud release pins migrator and runtime sessions to `public` with `current_schemas(false)` containing only that business schema; runtime-role/database `search_path` overrides are rejected.
+- [x] Cloud sessions require `session_replication_role=origin`, `row_security=on`, and `lo_compat_privileges=off`; every persistent `pg_db_role_setting` applicable to the runtime role or current business database is rejected.
+- [x] The release migrator grants the runtime role exact business-table DML, `alembic_version` read-only access, and business-sequence `USAGE/SELECT`, with no `WITH GRANT OPTION` or non-business object grants.
+- [x] Every Cloud runtime startup revalidates the login role, current user, schema, effective/direct ACLs, ownership, memberships in all directions, column ACLs, routines, extensions, foreign objects, parameter ACLs, and other-schema access before serving traffic.
+- [x] The business database requires `vector`, permits only `plpgsql`/`vector` extensions, forbids runtime extension ownership, and contains no foreign data wrapper, foreign server, or user mapping.
+- [x] The runtime role and `PUBLIC` have no explicit routine or parameter ACL; the runtime owns no routine and cannot effectively execute any `SECURITY DEFINER` routine, including extension-owned routines.
+- [x] PostgreSQL's default `PUBLIC TEMP` is documented and tested as a dedicated-business-database v1 compatibility exception; the migrator never grants `TEMP` directly to the runtime role.
+- [x] Legacy pgvector migration succeeds as a non-superuser, non-`BYPASSRLS` source-table owner and restores mixed source-table RLS/FORCE states exactly.
+- [ ] Legacy pgvector migration still needs explicit failure-and-retry integration coverage before SaaS activation.
+
+### Runtime transaction enforcement
+
+- [x] Each tenant UoW owns one task, root transaction, database bind, and transaction-local scope.
+- [x] A scoped Session and every captured bound method become permanently unusable when the owning UoW exits.
+- [x] Public transaction/session control, raw/textual SQL, connection/bind escape, nested transactions, execution/loader options, foreign binds, live results, unapproved functions/operators/casts/types, `INSERT FROM SELECT`, hidden `ON CONFLICT` and batch-value expressions, forced-unquoted identifiers, and custom AST/compiler nodes fail closed and make the UoW rollback-only.
+- [x] ORM `SessionEvents` fail before a registered callback can receive the synchronous Session or transaction connection; rollback cleanup cannot execute the rejected listener.
+- [x] ORM flush, implicit autoflush, and commit reject SQL expressions assigned to mapped attributes before compilation.
+- [x] Tenant relationship loading uses eager loading or explicit async `refresh`; synchronous object-session access and `AsyncAttrs.awaitable_attrs` are not supported tenant APIs.
+- [x] The UoW guard is documented as a trusted-Core misuse boundary rather than an in-process Python sandbox; mapped metadata/compiler registration is trusted, plugins remain out of process, and SQLAlchemy upgrades must rerun the private-container regression suite.
+
+## 2. Authentication and authorization
+
+### Identity
+
+- [x] JWT sub uses account UUID, with a bounded compatibility path for legacy email tokens.
+- [x] Disabled or deleted accounts cannot authenticate.
+- [x] Local password and Space-linked account flows support more than one local Account.
+- [x] Public registration closes after initialization by default.
+- [x] Invitation registration works without requiring SMTP.
+- [x] An unknown Space OAuth subject cannot claim an existing Account by email; explicit account-bound binding is required.
+
+### Request context
+
+- [x] PrincipalContext identifies Account, API Key, or trusted runtime principal.
+- [x] WorkspaceContext contains Workspace, Membership, role, permissions, and revision.
+- [x] RequestContext contains instance UUID, Workspace context, auth type, request ID, and generation.
+- [x] ExecutionContext propagates Workspace and generation to runtime work.
+- [x] SaaS-style requests never fall back to the first or most recent Workspace.
+- [x] OSS may resolve the single Workspace when the selector is omitted.
+- [x] Account-token bootstrap can list only the authenticated Account's active memberships before a Workspace selector exists.
+
+### Fixed RBAC
+
+- [x] owner, admin, developer, operator, and viewer permissions match the architecture matrix.
+- [x] Invitation cannot grant owner.
+- [x] The last owner cannot be removed or demoted.
+- [x] Cross-Workspace resources return 404.
+- [x] Same-Workspace permission failures return 403.
+
+## 3. Workspace and member APIs
+
+- [x] GET /api/v1/workspaces returns the OSS singleton Workspace.
+- [x] POST /api/v1/workspaces returns edition_limit in OSS.
+- [x] Current Workspace endpoint returns the authenticated Membership.
+- [x] Member list is permission scoped.
+- [x] Invitation create, revoke, inspect, and accept are atomic.
+- [x] Member role update and removal enforce owner rules.
+- [x] Invitation tokens travel in a request body and are redacted from logs.
+- [x] Relevant MCP tools and in-repo skills are updated with the same contract.
+
+## 4. Tenant-scoped persistence and services
+
+Each row type must have a non-null Workspace UUID, scoped indexes, scoped uniqueness, and scoped CRUD tests.
+
+- [x] Bots and bot admins.
+- [x] Legacy pipelines and pipeline run records.
+- [x] Model providers.
+- [x] LLM models.
+- [x] Embedding models.
+- [x] Rerank models.
+- [x] Plugin installations, settings, and configuration.
+- [x] MCP servers and resource preferences.
+- [x] Knowledge bases, files, and chunks.
+- [x] Vector collections and handles.
+- [x] Monitoring messages, calls, sessions, errors, embeddings, and feedback.
+- [x] API keys and scopes.
+- [x] Webhooks and public route resolution.
+- [x] Binary storage and Workspace storage.
+- [x] Workspace metadata, separated from system metadata.
+
+### Service and API rules
+
+- [x] Every tenant Service receives RequestContext or an explicit Workspace UUID.
+- [x] No tenant Service treats context None as global access.
+- [x] Every applicable get, list, create, update, delete, copy, export, and bulk operation is scoped.
+- [x] Parent-child references use the same Workspace.
+- [x] API Key authentication derives Workspace from the key, not a header.
+- [x] Webhook and Bot public routes derive Workspace from a trusted resource.
+- [x] Background jobs carry Workspace and generation explicitly.
+
+## 5. Runtime isolation
+
+### Core runtime
+
+- [x] RuntimeBot carries Workspace UUID and execution generation (currently stored in the compatibility field `placement_generation`).
+- [x] RuntimePipeline carries Workspace UUID and execution generation (currently stored in the compatibility field `placement_generation`).
+- [x] Query and Event carry Workspace UUID without making it an authorization source.
+- [x] Session key includes Workspace UUID, Bot UUID, launcher type, and launcher ID.
+- [x] QueryPool and manager indexes cannot collide across Workspaces.
+- [x] Query and aggregation cache keys and locks include Workspace UUID.
+- [x] Runtime transports, cached results, object operations, and long-lived tasks revalidate WorkspaceExecutionState generation at side-effect boundaries.
+- [ ] Ordinary tenant database writes hold the generation fence in the same transaction until commit; this remains a SaaS activation gate.
+
+### Plugin
+
+- [x] Plugin installation and configuration are Workspace scoped.
+- [x] Runtime control actions carry trusted Workspace binding and execution generation (wire-compatible as `placement_generation`).
+- [x] The Plugin Runtime supervisor is instance-scoped and intentionally serves multiple Workspaces.
+- [x] Every plugin process is bound to exactly one Workspace, installation, generation, revision, and verified artifact digest.
+- [x] Same-digest plugin code may be cached once, while worker processes and writable data remain isolated.
+- [x] Same-digest plugin dependencies are prepared once in a Runtime-owned immutable environment and mounted read-only into each isolated worker; dependency failure is surfaced before launch and recorded per installation without blocking other desired-state recovery.
+- [x] Host API derives Workspace from the connection, installation, and trusted action context, not plugin input.
+- [x] Plugin get_bots, models, tools, vector, RAG, configuration, and messaging calls are scoped.
+- [x] Plugin Workspace storage no longer uses owner default.
+- [x] Plugin page APIs check Membership and installation ownership.
+- [x] Local plugin launches use short-lived, one-use registration capabilities bound to manifest identity.
+
+### MCP, RAG, and Box
+
+- [x] MCP runtime key contains instance UUID, Workspace UUID, execution generation, and server UUID.
+- [x] Same-named MCP servers in two Workspaces do not share sessions.
+- [x] Pipeline cannot reference another Workspace's MCP resource.
+- [x] RAG collection names and handles are server-derived and Workspace scoped.
+- [x] Legacy global vector migration is available only to the local OSS singleton Workspace.
+- [x] Object storage paths include instance, Workspace, and execution generation for the fixed-generation OSS runtime.
+- [x] Object storage revalidates generation before touching a provider or resolving an opaque key.
+- [ ] Cloud cutover uses generation-scoped staging plus stable published object references, rather than making the staging generation the durable identity.
+- [x] Box persistent and ephemeral namespaces include the required instance, Workspace, and generation scope.
+- [x] Same-named Box sessions and processes cannot collide across Workspaces or execution generations.
+- [x] Box relay and process I/O reject or retire stale generations.
+- [x] External paths and privileged mounts cannot be supplied by an untrusted plugin.
+- [x] Cloud attachment host I/O uses query UUIDs and link-free dirfd operations with bounded inode traversal.
+- [x] Cloud Skill package paths are Runtime-owned, Workspace-scoped, read-only mounts; Python env/cache stays tenant-writable.
+- [x] Skill ZIP preview/install rejects path escape, links, non-regular files, duplicate entries, excessive compression ratio, entry count, per-file size, and total size.
+- [x] Cloud Box code paths and automated tests require the authenticated marker challenge before startup or reconnect can proceed.
+- [x] Cloud Box readiness fails until hard Workspace, Skill, ephemeral-storage, and inode quota capabilities are available.
+
+## 6. SDK and protocol
+
+- [x] Public Query, Event, Session, and context entities carry backward-compatible Workspace data.
+- [x] Action RPC request models carry trusted Workspace binding where required.
+- [x] Action enums and callers remain consistent.
+- [x] Old plugins continue to deserialize compatible events.
+- [x] Plugins cannot select an arbitrary Workspace through a Host API argument.
+- [x] Runtime storage uses the bound Workspace UUID.
+- [x] SDK API tests pass.
+- [x] Runtime tests pass.
+- [x] Action consistency script passes.
+
+## 7. Frontend
+
+- [x] Every browser tenant API request carries the current Workspace selector after bootstrap.
+- [x] OSS automatically selects the singleton Workspace.
+- [x] OSS does not show Create Workspace or a misleading switcher.
+- [x] Workspace settings show current Workspace information.
+- [x] Members page lists roles and permissions.
+- [x] Invitation creation shows a one-time link when SMTP is unavailable.
+- [x] Invitation acceptance supports a signed-out user flow.
+- [x] Role controls are hidden or disabled consistently with backend permissions.
+- [x] Switching accounts clears stale Workspace query cache and local state.
+- [x] User-facing strings support en_US, zh_Hans, and ja_JP.
+
+## 8. Automated verification
+
+### Persistence and authorization
+
+- [x] SQLite fresh install.
+- [x] SQLite upgrade from pre-tenant schema, including verified failure recovery.
+- [x] PostgreSQL fresh install.
+- [x] PostgreSQL upgrade from pre-tenant schema.
+- [x] All fixed roles have positive and negative permission-matrix tests.
+- [x] Concurrent invitation acceptance creates one Membership.
+- [x] Concurrent owner changes never leave zero owners.
+
+### Cross-tenant isolation
+
+- [x] Two Workspaces are created through a test-only policy.
+- [x] Applicable resource operations and parent-child references have cross-Workspace negative coverage.
+- [x] Resource UUID guessing cannot cross Workspace.
+- [x] API Key cannot cross Workspace.
+- [x] Plugin cannot enumerate or invoke another Workspace's resources.
+- [x] Sessions, caches, locks, MCP, RAG, Box, storage, and monitoring do not collide.
+- [x] Background jobs cannot execute without an explicit Workspace and execution generation.
+
+### Security and revocation
+
+- [x] Space login and binding use purpose-bound, one-time opaque OAuth state; caller-supplied state is rejected.
+- [x] OAuth redirects trust only server-configured WebUI or webhook origins, never request `Host` or `Origin` headers.
+- [x] Dashboard WebSockets revalidate authentication, Membership, resource, permission, and generation per message.
+- [x] Public embed WebSockets re-resolve Bot availability and execution binding per message.
+- [x] Runtime, storage, Plugin Runtime, MCP, RAG, and Box reject a stale execution generation.
+- [x] Unhandled API and webhook failures return a generic error plus request ID without exception text.
+- [x] URL user information and sensitive query parameters are redacted before configuration is serialized or logged.
+
+### Regression
+
+- [x] LangBot unit tests pass.
+- [x] LangBot integration tests pass.
+- [x] Frontend lint completes without errors and the production build passes.
+- [x] SDK focused and full relevant tests pass.
+- [x] LangBot is pinned to the exact pushed SDK commit and cross-repo tests pass against that revision.
+
+## 9. Real browser E2E
+
+- [x] Start from a clean local data directory.
+- [x] First user initializes the singleton Workspace as owner.
+- [x] Owner creates an invitation link.
+- [x] A second signed-out browser identity accepts the invitation and registers.
+- [x] owner, admin, developer, operator, and viewer UI permissions match backend enforcement.
+- [x] Direct API calls cannot bypass hidden controls.
+- [x] Account switch does not expose prior account or Workspace data.
+- [x] Refresh and a new browser tab recover the correct Workspace safely.
+- [x] OSS rejects a second Workspace with `edition_limit`; same-name and same-identifier isolation is covered by the test-only multi-Workspace policy because OSS deliberately has no multi-Workspace browser surface.
+- [x] Explicit error states are visible for expired, revoked, reused, and email-mismatched invitations.
+
+## 10. Completion evidence
+
+- [x] LangBot and SDK branch refs are recorded in the verification report.
+- [x] Space contains the closed adapter package and Cloud v2 control plane, billing, migration, and Workspace UI changes; unrelated pre-existing files remain unstaged.
+- [x] Migration output is captured for SQLite and PostgreSQL.
+- [x] Test commands and results are recorded.
+- [x] Browser E2E actions and observed results are recorded.
+- [x] No remaining tenant table, global Service query, owner default, or unscoped runtime key is found by the final audit.

docs/multi-tenant/implementation-decisions.md

@@ -0,0 +1,305 @@
+# Multi-tenant implementation decisions
+
+This log records implementation choices made while delivering the Workspace architecture. It is intended to make trade-offs auditable without interrupting implementation for routine decisions.
+
+> Architecture decisions, activation gates, and still-open follow-ups are tracked in
+> [pending-architecture-decisions.md](./pending-architecture-decisions.md). Sections marked as decided there are authoritative;
+> this file records the concrete implementation choices and compatibility names used to realize them.
+
+## 2026-07-18
+
+### OSS remains a singleton Workspace with multiple Accounts
+
+- Decision: Community builds create exactly one Workspace per LangBot instance and allow multiple Accounts through invitations.
+- Reason: This preserves a simple self-hosted deployment while making authorization and ownership explicit. Creating a second Workspace is an edition error, not a hidden fallback.
+- SaaS boundary: Multi-Workspace directory, execution ownership, entitlement, and billing are the responsibility of a separate closed SaaS Control Plane. Core consumes a validated projection and remains the final isolation and authorization enforcement point; it does not become the SaaS system of record or billing engine.
+- Deployment boundary: Cloud v2 is a greenfield deployment design. The previous per-account instance/pod scheme is not migrated or extended and remains available only for existing subscriptions. Existing OAuth, marketplace, and payment rails are reused through explicit adapters where they still fit; new Workspace, subscription, entitlement, directory, and usage modules live alongside the legacy Pod flow in `langbot-space`.
+
+### Workspace selection is trusted only after authentication
+
+- Decision: Browser requests carry `X-Workspace-Id`, but the server resolves it against the authenticated Account membership. API keys, public Bot routes, webhooks, jobs, and plugin calls derive Workspace from their trusted owning resource or binding instead of trusting the header.
+- Reason: A selector is routing input, not authorization evidence.
+- Compatibility: Community builds may select the singleton Workspace when the header is omitted. A multi-Workspace-capable build must reject an omitted selector.
+
+### Stable Account UUID is the token subject
+
+- Decision: New JWTs use the stable Account UUID as `sub`; a bounded compatibility path accepts legacy email-subject tokens and rotates them when checked.
+- Reason: Email can change and therefore cannot be a durable authorization identity.
+
+### Fixed roles are authoritative in Core
+
+- Decision: `owner`, `admin`, `developer`, `operator`, and `viewer` map to a fixed permission matrix in LangBot Core. The last owner cannot be removed or demoted, and invitations cannot create an owner directly.
+- Reason: Core must remain the final authorization boundary in both OSS and SaaS deployments.
+
+### Cross-Workspace access is indistinguishable from absence
+
+- Decision: Resource lookups always include Workspace UUID. A guessed UUID belonging to another Workspace returns 404; a visible resource with insufficient same-Workspace permission returns 403.
+- Reason: This avoids leaking resource existence across tenants while preserving actionable same-tenant errors.
+
+### Plugin Runtime is shared; every plugin process is single-Workspace
+
+- Decision: One instance-scoped Plugin Runtime control plane serves all Workspaces in the logical LangBot instance. Each running plugin installation has its own nsjail worker with an immutable binding containing `instance_uuid`, `workspace_uuid`, `execution_generation` (stored as the compatibility field `placement_generation` until the schema rename), `installation_uuid`, `runtime_revision`, and verified artifact digest; enabled-resident is the desired semantic. A worker never routes actions for another Workspace or installation, and plugin-supplied scope fields are stripped.
+- Isolation: Plugin code is mounted read-only. Home, tmp, and data paths are installation-scoped; process, file-descriptor, file-size, CPU, memory, and PID limits come only from `data/config.yaml` (including native environment overrides), never from a plugin manifest. Cloud requires nsjail and delegated cgroup v2 hard limits or fails closed.
+- Cost boundary: Identical verified package bytes share one digest-addressed code cache. A dependency environment is keyed by the artifact and requirements digests, Python ABI, Runtime version, and installer schema, then atomically published read-only for reuse. Installations and processes are not merged, even for the same plugin and version. Registration creates database desired state only; a worker is launched only for an enabled installation.
+- Recovery: PostgreSQL installation desired state and durable binary storage are authoritative. Runtime reconnect performs an instance-wide full reconciliation, removes stale workers, and can replay a verified package after Runtime-local cache loss. Dependency preparation failure is recorded per installation with `dependency_prepare_failed`; it prevents that worker launch without blocking recovery of other desired installations, and the same revision can be retried. The installation Supervisor now restores an unexpectedly exited enabled worker through a completion callback with bounded exponential backoff. Jitter, global restart concurrency limits, and a Runtime-wide circuit breaker are still required to prove that an infrastructure-wide failure cannot create a cross-tenant restart storm.
+- Compatibility: Older SDK payloads and legacy `data/plugins` remain an OSS-only bridge. Shared mode requires complete bindings and rejects incomplete context.
+- Reason: Sharing the supervisor and immutable code cache removes per-Workspace service cost without turning an untrusted plugin process into a cross-tenant router.
+
+### Invitation delivery does not require SMTP
+
+- Decision: Core returns an invitation secret once for copy-and-share, persists only its hash, and supports expiry, revocation, and one-time acceptance.
+- Reason: Self-hosted OSS must support adding users without an email service while avoiding recoverable invitation secrets at rest.
+- Browser handling: The copyable invitation URL carries the secret in its fragment, which browsers do not send in HTTP requests or Referer headers. The acceptance page immediately removes the fragment and keeps the secret only in `sessionStorage` until login or acceptance completes; it is never placed in a path, query string, analytics event, or persistent local storage.
+
+### Schema rollout is additive before enforcement
+
+- Decision: Add Account/Workspace directory tables first, then add non-null Workspace ownership to every tenant resource with a deterministic default-Workspace backfill. Runtime and service enforcement is enabled only with matching migration and isolation tests.
+- Reason: A Workspace column alone is not isolation, and enforcing queries before data backfill would break upgraded installations.
+
+### Login capability discovery is instance-scoped, not account-scoped
+
+- Decision: The unauthenticated login bootstrap endpoint reports only which login mechanisms the instance supports. It does not inspect the first Account or expose whether that Account has a password. Both password and Space OAuth entry points are available on a multi-user instance; the submitted identity determines which mechanism is valid.
+- Reason: This avoids projecting the original owner's authentication type onto invited users and removes a public Account-state disclosure.
+
+### Space OAuth identity does not choose a SaaS Workspace
+
+- Decision: Space OAuth tokens remain Account credentials. In OSS singleton mode, an OAuth refresh may update the singleton Workspace's Space provider only when that Account's role can manage provider secrets. In SaaS multi-Workspace mode an OAuth callback without an authenticated Workspace selector never guesses which Workspace to mutate; explicit Workspace configuration or the closed control plane owns that linkage.
+- Reason: An Account may belong to several Workspaces, and authentication must not silently mutate a shared tenant secret.
+
+### SaaS execution state is a validated Core projection
+
+- Decision: Core can resolve both local and `cloud_projection` Workspaces, but only from an explicit Workspace UUID and an active, unfenced `WorkspaceExecutionState` for the current instance and matching source. OSS-only bootstrap paths additionally require `source=local`.
+- Reason: The closed control plane owns execution ownership and generation decisions, while Core remains the enforcement point for instance binding, generation, and write fences.
+
+### API-key secrets are one-time and Workspace-bound
+
+- Decision: Database API keys persist only a globally unique SHA-256 hash, an opaque UUID, one Workspace UUID, explicit fixed-permission scopes, status, expiry, creator, and last-used time. The raw secret is returned once. Authentication derives Workspace and generation from the key record and ignores Workspace selectors. Legacy plaintext keys are hashed during migration and receive a compatibility `*` scope. The plaintext config key works only for the OSS singleton Workspace and is disabled in multi-Workspace mode.
+- Reason: A bearer key is an identity and routing credential, not merely a password layered on top of caller-controlled tenant selection.
+
+### MCP tools inherit the authenticated API-key context
+
+- Decision: The MCP ASGI mount authenticates the API key once, binds an immutable per-request `RequestContext`, and every tool checks a fixed permission before calling tenant services with that same context.
+- Reason: Authenticating the transport without propagating Workspace identity into tool calls would leave the direct service path globally scoped.
+
+### Unreleased SDK protocol is pinned reproducibly without publishing
+
+- Decision: The SDK tenancy protocol is versioned as 0.4.18. This task does not create a GitHub release or publish PyPI because the user authorized pushing code, not a package release. After the SDK feature branch is final, LangBot's feature branch temporarily pins the exact pushed SDK Git commit. Before merging to master, the release gate is to publish `langbot-plugin==0.4.18` and replace the Git pin with the registry pin.
+- Reason: The current registry release does not contain the complete tenant action context and shared Runtime hardening. An exact Git commit is reproducible and keeps the feature branch testable without expanding release authority.
+
+### Cloud directory writes stay outside Core
+
+- Decision: The open-source Core startup always installs `SingleWorkspacePolicy`, creates or repairs one local Workspace, and permits local membership/invitation workflows. Changing mutable configuration such as `system.edition` cannot activate multi-Workspace routing. The future closed Cloud bootstrap will install `CloudWorkspacePolicy` only after verifying a signed `InstanceManifest`; that policy requires an explicit projected Workspace selector, does not create Workspaces, and rejects invitation or membership mutations with `control_plane_required`; member reads use the versioned local projection.
+- Ownership split: The closed Control Plane owns the global Account/Workspace/Membership/Invitation directory, execution ownership and generation, entitlements, subscription state, usage aggregation, and billing decisions. Core owns request authorization, resource scoping, execution-generation validation, and fail-closed enforcement. Provisioning and invoice computation do not belong in open-source Core.
+- Reason: The closed control plane is authoritative for SaaS Account, Workspace, Membership, and Invitation state. Allowing Core to mutate the same directory would create split-brain ownership and would make an ownerless compatibility Workspace a dangerous fallback.
+- Release gate: Multi-Workspace activation is deliberately unavailable in the open-source bootstrap. Production Cloud v2 must implement the signed `InstanceManifest` verifier and closed bootstrap described in the architecture document before it can inject `CloudWorkspacePolicy`; `edition=cloud`, an environment variable, or any unsigned local configuration is never a valid activation credential.
+
+### Workspace bootstrap is reactive and ordered before browser resource calls
+
+- Decision: The web application blocks Workspace-owned pages until Account and current Workspace bootstrap completes. A `useSyncExternalStore` Workspace store publishes permission changes to React consumers; direct mutation-only routes and controls are hidden or disabled when the fixed role lacks the required permission.
+- Reason: Mutating a module-level variable after the initial React render did not reliably re-render permission controls, and mounting resource pages before the selector was established could issue tenant requests without `X-Workspace-Id`.
+
+### JWTs are bound to one LangBot instance
+
+- Decision: New Core JWTs require `iss=langbot-core`, an audience derived from the immutable instance UUID, and an expiry. Legacy community tokens are accepted only when they have the historical issuer, carry no audience, and the active policy is the OSS singleton policy.
+- Reason: A token issued by one instance must not authenticate against another instance that happens to share a secret, and a compatibility decoder must not become an alternate path around the SaaS trust boundary.
+
+### Runtime control transports authenticate before protocol dispatch
+
+- Decision: External Plugin Runtime and Box WebSocket control channels require independent strong shared secrets in handshake headers. Locally managed child processes receive ephemeral secrets through their environment; secrets are not placed in URLs, process arguments, request payloads, or logs. Box additionally binds the first authenticated control channel to one trusted instance. Plugin Runtime debug and control credentials remain separate.
+- Reason: Workspace context inside an RPC payload is not trustworthy until the transport peer itself is authenticated. Separating control and debug credentials also limits accidental privilege reuse.
+- Deployment consequence: Docker Compose and Kubernetes wire one shared secret to each host/runtime pair. An empty external-runtime secret fails startup instead of silently exposing an unauthenticated socket.
+
+### Dashboard WebSocket sessions are tenant runtime objects
+
+- Decision: A dashboard WebSocket sends an authentication frame immediately after upgrade. The server validates Account, Membership, permission, Pipeline ownership, instance, Workspace, and execution generation before registering the connection. Connection indexes, sessions, broadcasts, attachments, and resets include the complete execution scope.
+- Reason: Browser WebSocket APIs cannot attach the normal authorization headers, and a process-global `pipeline_uuid` or `session_type` index can collide across Workspaces.
+
+### Read permissions never imply secret permissions
+
+- Decision: `resource.view` responses recursively redact Bot, Plugin, MCP, and provider credentials. Provider secrets require `provider_secret.manage`; Bot and Plugin configuration writes require `resource.manage`. Masked Plugin values can be round-tripped by a manager without overwriting the stored secret. Plugin Runtime debug credentials require `resource.manage`, not the operator-only `runtime.operate` permission.
+- Reason: A multi-user Workspace needs useful viewer access without turning every visible configuration endpoint into credential export. Plugin debug attachment can register executable code and is therefore a resource-management operation.
+
+### Temporary credential exchanges are bound to their initiator
+
+- Decision: Lark, Weixin, DingTalk, WeComBot, and QQOfficial one-click registration sessions require `resource.manage` and store the initiating instance, Workspace, execution generation, and principal. Status and cancellation by any other scope return the same 404 as an unknown session.
+- Reason: Random session IDs reduce guessing probability but do not authorize access to credentials returned by a completed exchange.
+
+### Uploaded images and documents use different storage capabilities
+
+- Decision: Browser images use the scoped `upload_image` owner type and may be resolved only through the opaque public-image route. RAG documents use `upload_document` and can be read, sized, or deleted only by an exact instance, Workspace, generation, and owner-type match. Legacy `upload` objects are cleanup-only.
+- Reason: Treating every upload as a public image made a leaked document key sufficient to bypass authenticated RAG access.
+
+## 2026-07-19
+
+### Space OAuth state is server-issued and single-use
+
+- Decision: Core issues an opaque, cryptographically random OAuth state for each Space login or Account-binding attempt, stores only its digest, and consumes it exactly once within a short expiry. Login and binding states are different capabilities; a binding state is additionally bound to the authenticated Account. Caller-supplied state, including a LangBot JWT, is rejected.
+- Redirect boundary: Callback redirects are accepted only for the known callback path and an origin declared by the server-side `api.webui_url` or `api.webhook_prefix`. Request `Host` and `Origin` headers never expand this allowlist.
+- Current deployment: The OSS state store is bounded and process-local, so a Core restart safely invalidates outstanding attempts. A horizontally scaled SaaS deployment must move this exchange to an atomic, shared Control Plane store before enabling the closed Cloud bootstrap.
+- Reason: OAuth state is a narrow, one-time CSRF and flow-binding capability. Reusing a bearer JWT or trusting caller-controlled Host or Origin data would turn an authorization redirect into an Account-token theft or open-redirect primitive.
+
+### OAuth provider subjects, not email addresses, bind Accounts
+
+- Decision: A known Space `account_uuid` may refresh the credentials of its already-bound local Account. An unknown provider subject that presents an email belonging to an existing Account is rejected, even when the normalized emails match. The Account owner must authenticate locally and use the one-time, account-bound binding flow.
+- Reason: Email is contact and display data, not a stable federated identity key. Email-only auto-linking would let provider verification drift or identity reassignment become a local Account takeover.
+
+### Workspace discovery is an account-only bootstrap capability
+
+- Decision: `ACCOUNT_TOKEN` validates the active Account JWT but intentionally cannot resolve a Workspace, receive `RequestContext`, or declare Workspace permissions. Its narrow bootstrap endpoint returns only active Workspace memberships belonging to that Account and never chooses the first Workspace when several exist. All tenant resource routes still require the explicit selector in multi-Workspace mode.
+- Reason: Requiring a Workspace header to discover the Account's Workspaces creates an authentication deadlock; allowing the bootstrap route to perform tenant actions would create an authorization bypass. Separating the two capabilities resolves the cycle without weakening tenant routes.
+
+### SQLite tenancy migrations have a verified recovery boundary
+
+- Decision: Before each destructive tenant-schema boundary, a file-backed SQLite installation creates an online-consistent backup with its source and target revisions, runs `PRAGMA quick_check`, writes a durable manifest, and fsyncs restrictive-permission files and directories. A failed boundary disposes the engine, removes stale journal sidecars, atomically restores the verified source revision, and verifies the restored database before startup continues.
+- Compatibility: In-memory SQLite cannot provide this recovery guarantee and is rejected for destructive production migration boundaries; it remains usable in tests that create the final schema directly.
+- Reason: SQLite batch table rebuilds can leave an installation between schemas if a process or migration fails. A verified pre-boundary image makes retry behavior recoverable instead of merely idempotent in the happy path.
+
+### Execution generation is an execution revocation capability
+
+- Decision: RuntimeBot, RuntimePipeline, background tasks, object storage, Plugin Runtime, MCP, RAG, and Box operations carry the complete instance, Workspace, and execution-generation scope. The current schema and wire compatibility field remains `placement_generation` until a coordinated rename. They revalidate the active execution binding before accessing a provider or transport; long-running calls validate again before accepting results. A stale generation is fenced before it can read, write, or reuse a cached object.
+- Plugin boundary: Each locally launched plugin receives a short-lived, one-use registration capability bound to the expected manifest identity and execution scope. The production child environment does not inherit the reusable debug credential, and Host APIs derive scope from the trusted connection and action context.
+- Box boundary: Persistent skill content remains Workspace-scoped, while session/process state and relay requests also include execution generation. A generation change retires matching live sessions and closes a stale relay before further stdin, stdout, or file operations.
+- Transaction boundary: Request admission and runtime side effects are fenced in this branch, but ordinary tenant database mutations do not yet hold a generation-aware lock through commit. The closed Cloud bootstrap must remain disabled until Core provides the shared-write/exclusive-cutover transaction primitive and a generation-stamped outbox (or an equivalent atomic publish fence). The OSS singleton policy has a fixed local generation and cannot trigger an execution-owner cutover.
+- Durable-object boundary: Current opaque storage keys include generation and therefore fail closed after a generation change. That is safe for OSS's fixed generation, but a Cloud cutover must not strand durable KB files, images, or plugin references. Cloud v2 must publish stable final object identities from generation-scoped staging, or perform an atomic object-and-reference migration before activating the new generation.
+- Reason: Workspace UUID prevents cross-tenant collisions, but it cannot revoke work after execution ownership changes or is fenced. Execution generation is the monotonic revocation value that makes old runtimes unusable; it does not express membership in a product-level deployment entity.
+
+### Long-lived WebSockets continuously revalidate authority
+
+- Decision: Dashboard WebSockets re-authenticate the Account, Membership, permission, resource ownership, instance, Workspace, and execution generation for every inbound message, not only during the initial frame. A changed role, removed Membership, or fenced execution binding takes effect without waiting for reconnect.
+- Public embed boundary: The embed connection re-resolves its Bot before every message and rejects a Bot that was disabled, deleted, moved, or rebound. The public connection may identify a Bot, but it cannot make the initial Bot object an indefinite authorization capability.
+- Reason: Authorization and resource state can change while a socket remains open. Connection-time validation alone leaves a revocation gap.
+
+### Legacy vector migration is an OSS-local compatibility path
+
+- Decision: Status, backup, execute, dismiss, and background entry points for legacy global vector collections require an active local Workspace binding under `SingleWorkspacePolicy`. A `cloud_projection` Workspace cannot observe or migrate the old global collection, even when it carries a legacy marker.
+- Reason: The legacy collection predates tenant ownership. Treating it as a SaaS fallback would expose one installation's historical vectors to an arbitrary projected Workspace.
+
+### External errors and persisted URLs are redacted centrally
+
+- Decision: Unhandled HTTP and webhook failures return a stable `internal_error` response and request ID, and expose that ID in `X-Request-Id`; the detailed exception is retained only in server logs correlated by the same ID. Explicit domain and validation errors keep their documented status and code.
+- Secret boundary: Shared sanitization removes URL user information and masks sensitive query parameters before provider or MCP configuration is serialized, logged, or shown to a reader. Masked placeholders can be round-tripped by an authorized manager without replacing the stored secret.
+- Reason: Tenant isolation is incomplete if framework exceptions, connection URLs, or configuration reads can export credentials across otherwise authorized interfaces.
+
+### Box is one shared control plane with one admitted sandbox per Workspace
+
+- Decision: The logical instance has one shared Box Runtime control plane, implemented by one Runtime replica in M0. A closed entitlement adapter projects generic `managed_sandbox` capability and `managed_sandbox_sessions` limit; Core and Runtime never branch on a plan name. An eligible Workspace receives at most one persistent logical `global` session, while each ordinary command remains a one-shot nsjail process. Managed processes and network are disabled in the first Cloud release.
+- Storage boundary: Core and Runtime prove they see the same durable volume with an authenticated random-marker challenge. Attachments use opaque query UUID directories and link-free dirfd operations. Skill packages remain in the Runtime-owned Workspace store and enter a sandbox only as a read-only logical-name mount; Python environments and caches live in the tenant's writable Workspace.
+- Resource boundary: Cloud readiness requires cgroup v2 plus hard byte and inode limits for Workspace files, Skill storage, root, tmp, and home. The existing directory scan is only a compatibility soft check. Plain nsjail reports these storage capabilities as unavailable, so Cloud Box intentionally cannot start until the greenfield deployment supplies and verifies a real quota provider.
+- Archive boundary: Skill ZIP processing is bounded by compressed input, entry count, per-entry size, total uncompressed size, and compression ratio, and rejects links, non-regular entries, duplicates, and path escape before streaming extraction.
+- Reason: A shared supervisor removes per-Workspace services and idle control-plane cost, but storage and process admission must still fail closed at the untrusted execution boundary.
+
+### Cloud business data and vectors share one PostgreSQL schema
+
+- Decision: SaaS uses one PostgreSQL business database and shared schema. Every tenant row has an explicit Workspace key; application scope is the first boundary and precise `ENABLE` plus `FORCE ROW LEVEL SECURITY` policies are the second. The Cloud runtime role must be non-owner and have neither superuser nor `BYPASSRLS`.
+- Vector boundary: pgvector is the Cloud default in the same business database. Vectors use `(workspace_uuid, knowledge_base_uuid, vector_id)` identity, an untyped vector column with explicit checked dimension, and release-created partial expression indexes for the enabled dimensions. Cloud never falls back to Chroma or performs vector DDL at runtime.
+- Transaction boundary: A tenant UoW binds `SET LOCAL` and SQL to one transaction. Long-running pipeline and streaming MCP execution carry a trusted transaction-free tenant scope; each database helper opens a short scoped transaction, avoiding a held pool connection during LLM or network waits. Detached tasks start only after commit and create their own short UoW; rollback cancels them.
+- Schema boundary: The first release has exactly one business schema, `public`. Both migrator and runtime sessions must report `current_schema() = 'public'` and `current_schemas(false) = ARRAY['public']`; the runtime role and business database must not carry a `search_path` override. Runtime startup validates this before using the prepared schema and reruns the complete catalog and privilege validation on every process start; it never runs DDL.
+- Session boundary: Both Cloud modes require `session_replication_role = 'origin'`, `row_security = 'on'`, and `lo_compat_privileges = 'off'`. Every persistent setting applicable to the runtime role or current business database in `pg_db_role_setting` is rejected, even if its present value appears safe; tenant context remains transaction-local application state rather than a persistent role/database override.
+- Grant boundary: The migrator grants the runtime role direct `CONNECT` on the dedicated business database and `USAGE` on `public`; exact `SELECT, INSERT, UPDATE, DELETE` on every allowlisted business table; `SELECT` only on `alembic_version`; and exact `USAGE, SELECT` on business-owned sequences. It grants neither `CREATE`, `TRUNCATE`, `REFERENCES`, `TRIGGER`, sequence `UPDATE`, nor any privilege with `WITH GRANT OPTION`, and grants nothing on other relations or schemas.
+- Role boundary: The runtime identity is a `LOGIN` role with no superuser, `BYPASSRLS`, `CREATEDB`, `CREATEROLE`, or replication attribute; no role membership in any direction, including acting as grantor; no ownership of the business database, `public` schema, relations, sequences, routines, or extensions; no column ACLs; and no use, create, or ownership in another non-system schema. Neither the runtime role nor `PUBLIC` may have an explicit routine or parameter ACL, and the runtime role may not effectively execute any `SECURITY DEFINER` routine, including an extension-owned one. PostgreSQL's default `TEMP` privilege inherited from `PUBLIC` is an explicit first-release compatibility decision for this dedicated business database, not a direct runtime-role grant.
+- Catalog boundary: The business database must contain `vector` and may contain no extension other than `plpgsql` and `vector`; the runtime role owns neither. It contains no foreign data wrapper, foreign server, or user mapping. These checks remove catalog-level escape paths without forbidding the ordinary implicit execution of non-`SECURITY DEFINER` built-in routines.
+- Migration boundary: In the first release, the migrator and runtime URLs must name the same normalized PostgreSQL host, port, and database while using different roles. The migrator owns the application schema, establishes the exact allowlist above, and validates both required access and every prohibited escalation path before releasing the advisory lock. An exact Alembic head, RLS checks, and pgvector table/index/constraint validation remain mandatory; concurrent Jobs fail explicitly and are retried by orchestration.
+- Deployment boundary: PostgreSQL roles are cluster-wide, while the in-database audit proves only the target business database contract. SaaS production must therefore use a dedicated PostgreSQL cluster or endpoint that exposes only this business database to the runtime credential, or enforce and test an HBA/proxy policy proving that the credential cannot connect to any other database. This external connectivity proof is still an incomplete SaaS activation gate.
+- Endpoint evolution: A future deployment may use a direct endpoint for migrations and a pooler endpoint for runtime traffic. That topology may relax literal host/port equality only after both endpoints are proven to reach the same database through a database-internal, migrator-owned cluster identity that the runtime role can read but cannot create, alter, or spoof.
+- Legacy pgvector boundary: Revision 0013 records the exact `ENABLE` and `FORCE ROW LEVEL SECURITY` state of each RLS-protected source table, temporarily suspends those source policies as their table owner inside the migration transaction, and restores every table to its recorded state in `finally`. The migrator does not require superuser or `BYPASSRLS` for this data move.
+- Activation gate: The shared schema, pgvector adapter, and database-local runtime audit are implemented, but the external cluster/endpoint or HBA/proxy connectivity proof remains deployment work. Ordinary business writes also do not yet hold a generation-aware fence through commit; a generation-stamped outbox (or equivalent atomic publish fence) and stable durable-object references across generation cutover remain required before SaaS activation.
+- Reason: Sharing one database and pool keeps marginal Workspace cost low, while transaction-local context and RLS prevent that shared storage from becoming shared authority.
+
+### stdio MCP has an independent deployment gate
+
+- Decision: `mcp.stdio.enabled` is independent of Box availability and entitlement. OSS defaults it on for compatibility; Cloud requires it off at bootstrap and enforces the same gate on create, update, test, startup loading, and final runtime execution.
+- Reason: Treating Box availability as stdio permission would silently create another persistent `mcp-shared` sandbox for each Workspace and bypass the one-sandbox subscription and cost boundary.
+
+## 2026-07-20
+
+### The tenant UoW owns its task, root transaction, bind, and scope
+
+- Decision: A tenant UoW creates one task-owned `TenantScopedAsyncSession` and one root transaction. Public commit, rollback, close, connection, bind, nested-transaction, synchronous-Session, live-streaming, raw SQL, public execution options, and public `set_config` paths fail closed and mark the transaction rollback-only. ORM objects cannot expose a usable synchronous Session, captured methods cannot run in child tasks, an explicit foreign bind is rejected, and a captured Session is permanently retired when its UoW exits rather than being reset for reuse. Tenant scope is installed only through a private UoW capability; pgvector index-plan `SET LOCAL`/`EXPLAIN` diagnostics use a test/operator connection rather than the business Session API.
+- SQL boundary: Public UoW calls accept only structured SQLAlchemy query and DML trees. `TextClause`, literal SQL columns, textual labels, prefixes/suffixes/hints, statement execution options, `VALUES` roots, `INSERT FROM SELECT`, `EXTRACT`, literal-execute parameters, unknown/custom AST nodes, forced-unquoted identifiers, named `ON CONFLICT` constraints, unknown dialect post-values clauses, and untrusted casts/types fail closed. PostgreSQL/SQLite `ON CONFLICT DO UPDATE` and batch-insert containers are traversed explicitly because SQLAlchemy's standard visitor omits their executable values. Function classes are exactly allowlisted as `count`, `coalesce`, `sum`, `now`, `length`, and `nullif`; the only custom operator/cast admitted is the validated pgvector cosine operator and `Vector` cast.
+- Legacy migration boundary: The local-only RAG backup restore uses explicit table and column objects, never raw SQL, but deliberately leaves legacy values untyped. This preserves SQLite's string-valued `DATETIME` rows and both the historical PostgreSQL `TEXT` and fresh-schema `JSON` settings columns while keeping every value bound rather than interpolated.
+- ORM boundary: SQLAlchemy `SessionEvents` are unsupported on a tenant-scoped Session. If a listener is registered before or during a UoW, the operation fails before the callback executes and cleanup proceeds against an empty dispatch surface. Public `get`, `get_one`, `refresh`, and `merge` reject caller-supplied loader, bind, lock, shard, and execution options. Flush, implicit autoflush, and commit reject a SQL expression assigned to a mapped attribute before it can reach the compiler. Tenant code uses the async Session directly; relationships use eager loading or explicit `await session.refresh(entity, [attribute])`. LangBot's persistence base does not expose `AsyncAttrs.awaitable_attrs` as a supported tenant API.
+- Compiler trust boundary: This guard prevents accidental scope/transaction escape by trusted LangBot Core code; it is not an in-process Python sandbox. Registered SQLAlchemy compilers and mapped schema metadata are trusted boot-time code. The fail-closed traversal of dialect containers necessarily covers SQLAlchemy private fields, so dependency upgrades require the regression suite and remain pinned until verified. Untrusted plugins cannot import or call this Session because they remain isolated in Plugin Runtime child processes.
+- Result boundary: A caught database or boundary failure rolls back the root transaction and cancels after-commit work. Buffered results contain only already-authorized rows and no live connection; live database results cannot escape the UoW operation.
+- Reason: `SET LOCAL` plus RLS protects a tenant only while every statement stays on the same owned connection and callers cannot end the transaction, replace the GUC, recover the synchronous proxy, or route a statement through another bind.
+
+### Parallel request work re-enters tenant scope explicitly
+
+- Decision: Child coroutines created by request-level `asyncio.gather` open their own explicit transaction-free tenant scope before calling persistence-backed Plugin, MCP, or Skill operations. They never inherit the parent's active database Session.
+- Reason: Python copies ContextVars into child tasks, but SQLAlchemy Sessions are not task-safe and task identity is part of the tenant UoW boundary.
+
+### Ordinary monitoring is readable; audit and export remain privileged
+
+- Decision: Workspace monitoring dashboards, Bot logs, sessions, messages, calls, errors, and feedback require `resource.view`. Monitoring export requires `data.export`; system/runtime audit logs keep `audit.view`. Frontend tabs and controls use the same split.
+- Reason: A Viewer needs useful read-only product observability, while bulk data extraction and privileged runtime/system logs are separate capabilities.
+
+### Invitation failures survive login without contradictory success state
+
+- Decision: Invitation terminal codes map to stable browser states. A login-mediated email mismatch preserves the fragment-captured secret only in session storage, returns to the acceptance page with the stable error code, and suppresses the generic login-success toast. A transient acceptance failure retains the authenticated session and offers the same one-time invitation for retry. Invalid Bearer tokens on acceptance map to the normal authentication error instead of an internal failure.
+- Reason: Invitation acceptance crosses signed-out and authenticated states; losing or masking the domain error makes recovery ambiguous and can present contradictory UI feedback.
+
+### Shared Plugin Runtime starts only from verified desired state
+
+- Decision: SDK shared mode waits for immutable runtime configuration before inspecting plugin state, never scans or launches legacy `data/plugins`, and rejects legacy install/restart/delete/upgrade control actions. Worker RPC files use installation-private directories with aggregate size enforcement, and resident nsjail workers explicitly disable the default 600-second wall-time limit.
+- Remaining gate: completion-callback recovery, jittered per-installation backoff, globally bounded restart launch admission, Runtime-level circuit breaking, one half-open probe, and worker ready timeout are implemented. Production cross-tenant fault injection must still prove the restart-storm controls. Hard installation disk quota and production egress policy remain Cloud activation requirements; Linux nsjail/cgroup CPU, memory-plus-swap, PID, namespace, and cgroup-reaping behavior have real-container evidence.
+- Reason: A shared supervisor reduces per-Workspace services only if legacy global paths, writable transfer state, and lifecycle defaults cannot bypass installation isolation.
+
+## 2026-07-24
+
+### Cloud v2 remains a modular monolith with one closed adapter
+
+- Decision: Workspace directory, plans, subscriptions, payment fulfillment, entitlements, usage, and signed runtime feeds are implemented as modules in the existing Space service and PostgreSQL database. The only separately packaged closed component is the thin Core bootstrap/control-plane adapter; it verifies signed data and owns no business state.
+- Compatibility: The legacy Pod card and fulfillment path remain visible only to accounts with old subscriptions. New purchases create Workspace subscriptions and never provision a per-user LangBot Pod.
+- Reason: A separate tenancy or billing service would add deployment, queue, network, and consistency cost without providing an isolation boundary. The signed adapter keeps SaaS logic closed while Core retains the ORM, RLS, authorization, and runtime enforcement boundary.
+
+### Registration creates data, not infrastructure
+
+- Decision: Account registration and personal Workspace creation share one transaction and include an active owner membership, Free subscription, entitlement snapshot, and outbox notifications. Repair is idempotent for older accounts. Empty Workspace creation starts no Plugin worker, Box sandbox, database, queue, bucket, or tenant service.
+- Activation: This automatic Cloud v2 transaction is enabled only when Space has an explicit valid `CLOUD_V2_INSTANCE_UUID`. A legacy marketplace-only Space deployment with no Cloud v2 instance configured keeps its existing Account registration path; Cloud v2 internal endpoints still fail closed instead of inventing an instance identity.
+- Billing boundary: Core and runtimes consume only generic capability and numeric-limit entitlements. They never branch on `free` or `pro`; plan names, prices, payment providers, and fulfillment stay in Space.
+- Reason: This keeps the marginal cost of a new user close to a few PostgreSQL rows while preserving an immediate, usable Workspace.
+
+### Directory bootstrap is full; steady-state projection is per Workspace
+
+- Decision: Space generates the initial signed directory snapshot and its outbox high-water mark in one read-only PostgreSQL `REPEATABLE READ` transaction. After bootstrap, each event page names affected Workspaces and Core fetches one signed `directory.delta` for that set. Missing requested Workspaces are tombstones; unrelated Workspaces are untouched.
+- Cursor boundary: A delta carries no event cursor. Each signed event page carries the transaction-consistent current high-water mark, while Core advances only to the final event it actually consumed, even when the authoritative delta already contains a later revision. Event payload Workspace and revision fields must exactly match the signed event envelope; readiness is renewed only after the replica-local cursor reaches the signed high-water and the shared projection is not ahead.
+- Replica boundary: Every Core replica owns a process-local consumer cursor because its verified entitlement cache is process-local. Replicas share the PostgreSQL projection high-water mark and inbox. The state separately records which cursor range was atomically subsumed by a full snapshot, so a lagging replica can add missing receipts within that coverage and refresh its local caches without repeating tenant mutations; a missing receipt beyond snapshot coverage fails closed.
+- Reason: A shared consumer cursor would let one replica starve another replica's local cache, while a full snapshot on every change would make steady-state cost grow with every registered Workspace.
+
+### Cloud OAuth can authenticate only an existing projected Account
+
+- Decision: In multi-Workspace Cloud mode, Space OAuth may refresh tokens only for an active `cloud_projection` Account whose Space subject UUID and normalized email exactly match. It cannot create an Account, relink by email, mutate directory identity, or choose a Workspace.
+- Redirect boundary: When Cloud v2 configures its Core public URL, Space issues authorization codes only to that exact callback origin or the explicitly retained legacy managed-Pod domain. Community installations remain dynamic OAuth clients because they cannot pre-register with the public Space service: the consent screen displays their hostname, and only HTTPS or loopback HTTP with the fixed callback path is accepted. Remote HTTP, userinfo, fragments, arbitrary callback paths, and unrecognized query parameters always fail closed.
+- Reason: Space is the SaaS identity and directory authority, but Core remains the authentication enforcement point. Projected-only matching avoids split-brain identities; redirect allowlisting prevents bearer authorization-code exfiltration.
+
+## 2026-07-29
+
+### Directory capacity is one instance-level admission contract
+
+- Decision: Space and Core share an explicitly configured operational ceiling for active
+  Workspace and snapshot membership cardinality. Space serializes new personal-Workspace
+  creation across replicas with a PostgreSQL transaction advisory lock and rejects the
+  registration transaction before the limit is crossed. Core independently counts the
+  projected active Workspace set while holding the per-instance directory projection row
+  lock. Exceeding any limit rolls back the complete projection and does not advance its
+  cursor; neither side truncates authoritative data.
+- Snapshot boundary: A full snapshot is current desired state and contains active
+  Workspaces only. Archived Workspace revisions are retained as bounded, targeted deltas
+  so Core can validate and apply monotonic tombstones without every bootstrap carrying
+  unbounded history.
+- Memory boundary: Space bounds database result cardinality before signing. The closed
+  adapter bounds decompressed response bytes before JSON/JWS parsing and validates
+  Workspace/membership cardinality before entitlement-cache fan-out. Core schema models
+  have absolute list ceilings, validate duplicates in one pass, and bulk-read Accounts in
+  bounded chunks rather than issuing two serial queries per Account. Manifest and
+  entitlement responses have smaller endpoint-specific byte ceilings; entitlement refresh
+  validates and releases batches of at most 16 raw responses instead of retaining the
+  entire directory fan-out.
+- Operations boundary: Core health exposes aggregate active/max directory cardinality and
+  PostgreSQL pool occupancy/timeouts. The default active Workspace ceiling is 1,000, with
+  a hard code ceiling of 5,000; this is a safety stop, not a production capacity claim.
+  Space and Core configuration must match, and the approved production value comes from
+  the real V-08 capacity curve plus the V-09 24-hour soak.
+- Reason: One LangBot instance should admit new tenants at the cheapest data-only boundary,
+  while still preventing legitimate registration growth or a malformed control-plane
+  response from causing an unbounded startup allocation, database connection storm, or
+  CPU spike.

docs/multi-tenant/pending-architecture-decisions.md

@@ -0,0 +1,525 @@
+# Cloud v2 多租户架构决策与待决策项
+
+状态：`DECIDED — Core isolation kernel implemented; SaaS activation gates remain`
+创建日期：2026-07-19
+最近更新：2026-07-24
+
+本文记录 Cloud v2 多租户架构中已经确认的首期决策、明确淘汰的方案和仍需在后续阶段决定的扩展项。
+本文同时记录实现状态。“实现完成”仅指开源 Core/SDK 的隔离内核和 fail-closed 门禁，
+不表示闭源 Control Plane、计费或 Cloud v2 部署已经可上线。最终实现选择同步记录在
+[implementation-decisions.md](./implementation-decisions.md)，剩余发布门禁记录在实施清单和验证报告。
+
+## 0. 已确认的 SaaS 拓扑前提
+
+1. SaaS 只有一个逻辑 LangBot 实例，全部 Workspace 都是该实例内的租户。
+2. 产品和领域模型中不引入 Cell 内多个 CloudInstance、Workspace Placement 或 Workspace 到 CloudInstance 的路由。
+3. 当前不实现分布式，但同一个逻辑实例未来可以运行多个 Core、Plugin Runtime 和 Box Runtime replica，
+   也可以增加 PostgreSQL shard；这些只是内部实现，不成为新的租户或产品实体。
+4. 所有副本共享稳定的 `instance_uuid`；`replica_id`、`worker_id` 和进程地址是短期运行身份，不能写进业务资源的永久主键。
+5. `workspace_uuid` 始终是数据、任务和运行时的租户键，也是未来内部路由与分片的候选键。
+6. generation/epoch 的语义是执行所有权、故障转移和任务撤销，不代表 Workspace 在多个 CloudInstance 之间 Placement。
+7. 注册 Account 时自动创建 Workspace，但只新增目录记录和业务行，不创建租户专属部署、数据库、队列或 Runtime。
+8. OSS 仍是单租户 LangBot 实例，但允许该 Workspace 内存在多个用户；只有 SaaS 开启多 Workspace 租户模式。
+
+“单个 LangBot 实例”表示单个逻辑服务和安全域，不等于永远只有一个 OS 进程或一个 Kubernetes replica。
+当前代码字段 `placement_generation` 在完成架构迁移前继续兼容，目标语义和候选命名是 `execution_generation`。
+
+### 0.1 本轮确认的首期决策
+
+| 编号  | 结论                                                                                                                                          | 首期状态                                                             |
+| ----- | --------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
+| D-001 | 一个共享 Plugin Runtime 控制面；每个运行中的 plugin installation 独占一个 nsjail 子进程；只有 digest 相同且已验证的代码 artifact 可以只读共享 | `IMPLEMENTED — egress/disk-quota pending; restart-storm fault injection pending` |
+| D-002 | 一个共享 Box Runtime；Cloud 固定使用 nsjail；符合套餐的 Workspace 最多一个持久 `global` 逻辑 sandbox，普通执行按需启动 nsjail 进程            | `IMPLEMENTED FAIL-CLOSED — hard filesystem quota provider pending`   |
+| D-003 | SaaS 业务数据使用 PostgreSQL shared schema、应用层作用域和 RLS 双重隔离；pgvector 使用同一 PostgreSQL，作为 SaaS 默认向量后端                 | `PARTIALLY IMPLEMENTED — transaction/outbox/deployment gates remain` |
+| D-004 | stdio MCP 与 Box availability 解耦；Cloud v2 首期强制关闭 stdio MCP，避免为每个 Workspace 创建额外的 `mcp-shared` persistent sandbox          | `IMPLEMENTED`                                                        |
+| D-005 | 目录启动使用事务一致的全量快照，运行时按事件涉及的 Workspace 拉取增量；每个 Core replica 独立消费事件，共享 PostgreSQL 投影和 inbox           | `IMPLEMENTED — production fault injection pending`                   |
+
+Workspace 的具体创建、释放、数据导出和单 Workspace 恢复机制不在本轮决定；本文只保证这些后续能力不会改变稳定的
+`workspace_uuid`，也不会要求重建租户专属部署。
+
+## 1. 本轮重构的最高目标
+
+> 共享可信控制面和基础设施池，隔离不可信执行单元；减少独立部署、扩缩容和运维组件，使新增 Account 或 Workspace 的静态成本接近零。
+
+这里的“减少组件”指减少独立 Deployment、Service、数据库、消息系统和租户专属常驻控制面，
+不是通过合并安全边界来减少必要的隔离进程。
+
+统一评估原则：
+
+1. 注册 Account、自动创建空 Workspace 时，不启动 Plugin worker 或 Box sandbox。
+2. 启用插件后，每个 installation 的常驻成本来自其独立安全边界；首次使用托管 sandbox 后，符合套餐的 Workspace 才承担一个持久逻辑 session 的成本。
+3. 可信 supervisor、artifact cache、数据库连接池和 Runtime 容量可以多租户共享。
+4. 一个不可信插件进程不能服务多个 installation；一个 sandbox/session 不能服务多个 Workspace。
+5. 默认使用共享 Runtime；dedicated 只作为未来高隔离、大客户或合规资源等级，不建立第二套外部协议。
+6. 没有明确容量证据前，不新增 Kafka、Redis、Runtime 专用数据库、Box 专用数据库或租户级调度服务。
+7. 多租户隔离必须覆盖身份、路由、存储、缓存、日志、配额、撤销和故障恢复，不能只给请求增加 `workspace_uuid`。
+
+## 2. 首期部署形态与未来演进
+
+```mermaid
+flowchart LR
+    Traffic["SaaS traffic"] --> Core["One logical LangBot instance<br/>1 Core replica in MVP"]
+    Core --> PluginRuntime["Shared Plugin Runtime<br/>trusted supervisor"]
+    Core --> BoxRuntime["Shared Box Runtime<br/>nsjail backend"]
+    PluginRuntime --> PA["Workspace A / installation 1<br/>isolated nsjail process"]
+    PluginRuntime --> PB["Workspace B / installation 2<br/>isolated nsjail process"]
+    BoxRuntime --> BA["Workspace A<br/>one persistent global logical session"]
+    BoxRuntime --> BB["Workspace B<br/>one persistent global logical session"]
+    Core --> PG["Shared PostgreSQL business schema<br/>RLS + pgvector"]
+```
+
+| 档位                       | 内部部署形态                                                                                                                                                | 新 Workspace 静态成本         | 启用条件                 |
+| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- | ------------------------ |
+| M0. 单副本 MVP             | 一个 Core、一个共享 Plugin Runtime、一个共享 Box Runtime、一个 PostgreSQL business database；插件按启用状态运行，托管 sandbox 按首次使用与 entitlement 创建 | 只新增 Workspace 业务行       | 当前已确认目标           |
+| M1. 同逻辑实例内部横向扩展 | Core、Plugin Runtime、Box Runtime 按容量增加 replica；运行所有权由内部 lease 和 generation fence 决定；PostgreSQL 可增加共享 shard                          | 不创建 Workspace 专属部署     | 出现容量或可用性证据后   |
+| M2. Dedicated 资源档位     | 特定 workload 使用独享 worker pool、sandbox class 或 PostgreSQL shard，但沿用相同身份、协议、schema 和控制面                                                | 仅由购买 dedicated 的客户承担 | 合规、数据驻留或超大负载 |
+
+M1 是 M0 的透明扩容，M2 是相同架构下的资源等级；两者都不是新的 LangBot 实例、Cell 或 CloudInstance。
+外部 API 只认识稳定的 `instance_uuid` 和 `workspace_uuid`，不认识 replica、worker、pool 或 shard。
+
+Plugin Runtime 与 Core 在 M0 使用独立容器和 security context，Core 不能继承 Plugin Runtime 所需的
+nsjail/cgroup 权限。当前 Runtime 在进程生命周期内绑定首次认证的 `runtime_id`，因此 M0 必须把 Core 与
+Plugin Runtime 放在同一 rollout/restart unit 中协调重启；在实现受认证 takeover 或 owner lease/fencing 前，
+不能单独滚动 Core 并让它接管仍存活的 Runtime。Box Runtime 同样使用独立进程身份和安全配置，
+不与 Plugin Runtime 合并成一个高权限进程。
+
+## 3. D-001：Plugin Runtime 多租户控制面
+
+状态：`IMPLEMENTED — Cloud egress/disk-quota pending; restart-storm fault injection pending`
+
+### 3.1 已实现的基础
+
+- Plugin Runtime 控制连接只绑定稳定实例身份；一个逻辑共享控制面通过完整 installation binding 管理多个 Workspace，M0 由一个 Supervisor replica 承担。
+- 每个运行中的 installation 使用独立 nsjail worker；enabled-resident 是 desired semantics。代码只读，home/tmp/data 私有，shared profile 不读取 artifact `.env`。
+- 实例级 `PluginWorkerPolicy` 由 Core 的 `data/config.yaml` 下发，支持原生环境变量覆写；manifest 不能覆盖。
+- `installation_uuid`、`artifact_digest` 和 `runtime_revision` 已持久化并进入 desired-state、注册、Host API 和 generation/revision fence。
+- 已验证 `.lbpkg` 先进入 Workspace-scoped durable binary storage；Runtime 本地缓存丢失后可由 Core replay。
+- 相同 digest 的代码和 Runtime 准备的只读依赖环境可以共享，但 worker、运行时写入、配置和数据不合并；
+  dependency preparation 在启动 worker 前完成，失败会进入明确的 installation failed 状态。
+- Cloud shared profile 强制 Linux nsjail；`plugin.worker.require_hard_limits=true` 时 cgroup v2 delegation 不可用会启动失败。
+
+### 3.2 已确认的不变量
+
+1. 每个运行中的 plugin installation 独占一个 worker process tree，任何时刻都不能与其他 installation 共用；
+   停用或删除的 installation 可以没有进程。
+2. 插件进程只绑定一个
+   `(instance_uuid, workspace_uuid, execution_generation, installation_uuid, runtime_revision, artifact_digest)`，且运行期间不可重绑。
+3. 插件不能通过 payload、Host API 参数、环境变量或重连选择 Workspace。
+4. 插件进程的 home、tmp、可写数据、secret、进程视图和配额必须按 installation 隔离。
+5. 只有 `artifact_digest` 相同且完整性已验证的代码文件和依赖环境可以只读共享；
+   同名同版本但 digest 不同的 artifact 不能共享。配置、持久数据和运行进程不能共享。
+6. generation、installation revision 或 capability 被撤销后，旧进程必须失去 Host API 和副作用权限。
+7. Supervisor 不在自身解释器中加载第三方插件代码。
+
+### 3.3 首期执行模型
+
+- 整个 SaaS 实例共享一个可信 Plugin Runtime 逻辑控制面，M0 运行一个 Supervisor replica；新 Workspace 不创建专属 Runtime、连接、卷或进程。
+- Supervisor 的控制连接只绑定稳定 `instance_uuid` 和短期 Runtime identity，不绑定某个 Workspace。
+  每条 installation desired-state 命令都携带并验证完整的 installation binding；每个 worker action context 在注册后永久绑定该 tuple。
+- 安装并启用插件后，Supervisor 在自己的 Runtime 容器内直接启动一个 nsjail 子进程；
+  不再为每个插件创建 nested container、Pod、sidecar 或租户级 Runtime service。
+- desired semantics 要求 enabled installation 保持 resident，不做 idle eviction；停用、删除、revision/generation 变化或 entitlement 撤销时停止并按需重建。
+  Supervisor 通过 completion callback、带 jitter 的有界指数 backoff 恢复意外退出的 worker。所有 restart launch 共用实例级并发槽；
+  在配置的失败窗口达到阈值后打开 Runtime 级 circuit breaker，冷却后只允许一个 half-open probe。
+  probe 必须完成初始化并持续稳定一个窗口后才能恢复其他 installation；未在 30 秒内 ready 的子进程会被取消回收。
+  生产候选环境仍需执行跨租户系统性故障注入，证明熔断、恢复和告警符合预期。
+- 子进程使用一次性 registration capability 向 Supervisor 注册；capability 由可信 desired state 派生并绑定完整 installation tuple，
+  不是插件直接建立 Core Host connection，也不能只绑定 author/name/path。Supervisor/Core 据此注入 tenant context，
+  丢弃插件 payload 中自带的 scope 字段。
+- Supervisor 的进程表、nsjail root/tmp 和 artifact cache 都是可重建运行态；PostgreSQL 中的 installation desired state 才是权威业务状态。
+- M0 不增加 Runtime 专用数据库、Redis、Kafka、scheduler 或 artifact service；Core 重连后向 Supervisor replay desired state。
+
+### 3.4 nsjail 和文件边界
+
+首期目标目录模型：
+
+```text
+data/plugin-runtime/
+├── artifacts/sha256/<artifact_digest>/code/   # digest 校验后只读共享
+├── environments/sha256/<environment_digest>/ # 原子发布、只读共享依赖环境
+└── installations/<installation_uuid>/
+    ├── home/                                  # 私有可写
+    ├── tmp/                                   # 私有可写、可清理
+    └── data/                                  # 私有持久数据
+```
+
+- artifact 只有在内容摘要和完整性校验一致时才允许共享，不能只凭 author/name/version 复用目录；
+  cache 可接受的签名/来源、撤销和 GC 规范属于后续发布规则，不改变本轮基于已验证 digest 的只读共享边界。
+- artifact 与按环境摘要构建的共享依赖环境以只读 mount 进入 nsjail；installation 的 home/tmp/data 使用独立可写 mount。
+  环境摘要包含 artifact、requirements、Python ABI、Runtime 版本和 installer schema。依赖只能从已验证 artifact 的 PEP 508 声明构建，
+  index/trusted-host 只由实例配置控制；构建在独立 nsjail 的临时路径中完成并在成功后原子发布，失败或并发安装不能留下可见半成品。
+- 插件 cwd 可以是其私有 mount namespace 内的只读 `/plugin`，不要求为每个 installation 复制代码；
+  必须私有的是 home/tmp/data 等所有可写路径。
+- nsjail 必须启用 mount、PID、IPC、UTS 和 private `/proc` 等必要 namespace，插件不能枚举或 signal 其他插件及 Runtime 进程，
+  不能读取 Runtime 文件系统、宿主机路径、其他 installation 目录或平台 metadata endpoint。
+- 公开 SaaS 禁止从插件 artifact 自动加载 `.env`。secret 只能由可信控制面按 installation 注入，且不能进入共享 artifact/cache。
+- 插件需要外网时使用受控 egress；不得通过共享 host network 访问 Core loopback、Box Runtime、数据库或其他内部服务。
+- Cloud 部署必须提供可用的 cgroup v2 delegation 和所需 namespace 权限；如果硬 CPU/内存/PID 限制不可用，
+  Plugin Runtime readiness 必须失败，不能只记录告警后降级为普通子进程。
+
+### 3.5 统一资源上限与配置
+
+首期资源规格完全由 LangBot 实例配置决定，manifest 不能声明、放宽或覆盖资源。以下数值是建议默认值，
+最终仍由同一实例的 `data/config.yaml` 统一配置：
+
+```yaml
+plugin:
+  worker:
+    max_cpus: 1.0
+    max_memory_mb: 512
+    max_pids: 128
+    max_open_files: 256
+    max_file_size_mb: 512
+    require_hard_limits: true # Cloud; OSS defaults false
+```
+
+配置文件路径为 `data/config.yaml`，沿用现有原生环境变量覆写：
+
+- `PLUGIN__WORKER__MAX_CPUS`
+- `PLUGIN__WORKER__MAX_MEMORY_MB`
+- `PLUGIN__WORKER__MAX_PIDS`
+- `PLUGIN__WORKER__MAX_OPEN_FILES`
+- `PLUGIN__WORKER__MAX_FILE_SIZE_MB`
+- `PLUGIN__WORKER__MAX_CONCURRENT_RESTARTS`
+- `PLUGIN__WORKER__RESTART_FAILURE_THRESHOLD`
+- `PLUGIN__WORKER__RESTART_FAILURE_WINDOW_SECONDS`
+- `PLUGIN__WORKER__RESTART_CIRCUIT_OPEN_SECONDS`
+- `PLUGIN__WORKER__REQUIRE_HARD_LIMITS`
+
+Core 启动时校验配置并通过现有 `SET_RUNTIME_CONFIG` 下发不可变 `PluginWorkerPolicy`。
+Runtime 不读取另一份环境变量配置，避免两个配置源不一致。CPU、内存和 PID 使用 cgroup 硬限制，
+open files/file size 使用 rlimit。Cloud deployment profile 固定使用 nsjail，不能通过插件 manifest 或 SaaS 环境变量降级为普通进程。
+installation data 的总空间硬配额需要 filesystem project quota 或独立 quota volume，不能用目录扫描伪装成硬限制；
+该字段在选定可原子拒绝写入的存储机制前不进入首期配置。
+
+### 3.6 淘汰与暂缓方案
+
+| 状态       | 方案                                               | 结论                                                     |
+| ---------- | -------------------------------------------------- | -------------------------------------------------------- |
+| 淘汰       | 每 Workspace 一个 Plugin Runtime                   | 部署、连接和固定内存随 Workspace 线性增长                |
+| 淘汰       | 一个插件进程服务多个 Workspace/installation        | 全局状态、本地文件和依赖无法形成可信租户边界             |
+| 淘汰       | 同 Workspace 多插件合并到一个 worker               | 与“每 installation 独立进程”冲突，扩大故障和权限边界     |
+| 淘汰       | manifest 自行声明 CPU、内存或更高限额              | 首期统一执行实例级最大值                                 |
+| MVP 不引入 | Runtime 专用数据库、Redis、Kafka 或独立 scheduler  | 当前无容量证据，会增加组件和运维面                       |
+| 后续演进   | 多 Supervisor replica、owner lease、dedicated pool | 保留接口，达到容量或可用性阈值后再决定具体存储与调度方式 |
+
+架构扩展项包括：Core/Supervisor 是否共置、artifact/venv cache 的签名/来源/撤销/GC 规范、installation data hard-quota provider、
+v1 connection 的兼容期限，以及进入多 replica 后的 lease TTL、fencing token 和 owner 转移顺序。
+这些不改变“每个运行中的 installation 一个隔离进程”的首期边界。
+
+### 3.7 验收条件
+
+- 两个 Workspace 安装 digest 相同且已验证的 artifact 时，共享目录仍为只读，进程、配置、data、home、tmp、日志和 Host API 完全隔离；
+  同名同版本但 digest 不同的 artifact 绝不共享目录。
+- 插件不能读取其他 installation 文件、枚举或 signal 其他进程，也不能修改共享代码/依赖目录。
+- CPU、内存、PID、open files 和单文件上限在真实 nsjail/cgroup 环境中生效；超额只终止或拒绝对应 installation。
+- 修改 manifest 不能改变任何资源上限。
+- installation data 的总空间硬配额在写入边界原子拒绝超额，并证明目录扫描不是生产 enforcement。
+- 旧 generation/revision 的回调、消息、副作用和存储访问全部失败关闭。
+- Runtime 重启能从业务 desired state 恢复，不依赖本地进程表作为权威真相。
+- 意外退出的 enabled worker 由 completion callback 触发带有界 backoff 的自动恢复；连续失败只影响对应 installation，不能形成跨租户重启风暴。
+- requirements 中存在 Runtime 基础镜像未预装的包时，Supervisor 仍能先完成共享依赖环境准备再启动 worker；
+  安装失败不会留下持续重启的半启动进程，也不会影响同 digest 已就绪环境的其他 installation。
+
+## 4. D-002：Box 多租户控制面和套餐边界
+
+状态：`IMPLEMENTED FAIL-CLOSED — production quota provider pending`
+
+### 4.1 已实现的基础
+
+- 共享 Box 控制连接可服务多个 Workspace；所有操作绑定 instance、Workspace 和 generation，Runtime namespace 由可信 context 派生。
+- 短期 `SandboxAdmissionGrant`、revision tombstone 和原子 session admission 强制每个合资格 Workspace 最多一个 `global` persistent session，managed process 固定为零。
+- Core 与 Runtime 使用认证 host-control challenge 校验同一个 durable volume，而不是比较路径字符串；不一致时启动和重连失败。
+- Cloud skill 只传逻辑名称；Runtime 从 Workspace-scoped store 解析只读包路径，Python env/cache 写入租户自己的 `/workspace/.skill-envs`。
+- ZIP 安装限制压缩输入、条目、单项、总解压量和压缩比，采用流式解压并拒绝 link、非普通文件、重复项和路径逃逸。
+- 附件 host path 使用 query UUID 和 dirfd/openat/O_NOFOLLOW；Cloud replica 启动不再全局清理其他请求目录，遍历和删除有 inode 预算。
+- grant-enforced readiness 强制 cgroup、namespace、mount、共享卷、Workspace hard quota、Skill hard quota、ephemeral storage 和 inode quota 全部被证明。
+  普通 nsjail backend 对尚未实现的硬磁盘能力明确返回 false，因此当前 Cloud Box 会按设计拒绝启动，直到新部署提供真实 quota provider。
+
+### 4.2 首期套餐与 entitlement 模型
+
+- 闭源订阅管理/Control Plane 负责把套餐映射为版本化 entitlement；Core 和 Box Runtime 不硬编码 `plan == pro`。
+- 首期复用 Cloud Control Plane（可结合现有 Space 的订阅模块）承载闭源套餐、计费和 entitlement 投影，
+  不再拆一个独立 billing/tenant microservice；开源 Core 只实现通用 capability 和数值限额。
+- 首期套餐投影为：Pro 的 `managed_sandbox_sessions = 1`，其他套餐为 `0`。建议 capability 形态：
+
+```json
+{
+  "features": {
+    "managed_sandbox": true,
+    "external_sandbox": false,
+    "mcp_stdio": false
+  },
+  "limits": {
+    "managed_sandbox_sessions": 1
+  }
+}
+```
+
+- `box.enabled` 只表示当前 LangBot 实例是否部署了 Box Runtime，不能替代 Workspace entitlement。
+- 工具发现层根据 entitlement 隐藏/禁用托管 sandbox。Core 校验 Control Plane 的 entitlement 后，
+  通过受认证控制连接向 Box Runtime 下发短期 `SandboxAdmissionGrant`，绑定
+  `instance_uuid + workspace_uuid + execution_generation + entitlement_revision + expires_at + max_sessions + max_managed_processes`。
+  Runtime 只验证和执行该内部 grant，不理解 Pro 等套餐名称，也不相信业务调用方提交的 plan、session ID 或 host path。
+- entitlement 缺失、过期或无法验证时失败关闭。并发创建必须用原子 admission 保证同一 Workspace 永远不超过一个 managed session。
+- entitlement 被撤销后停止 managed process 并关闭逻辑 session；Workspace 数据保留/删除策略随未来 Workspace 释放机制一并决定。
+
+### 4.3 Cloud nsjail 执行模型
+
+- 整个逻辑 SaaS 实例共享一个 Box Runtime 逻辑控制面，M0 运行一个 Runtime replica；不创建每 Workspace Box service、worker pool、PVC、bucket、scheduler、Redis 或 Box 数据库。
+- Cloud 显式固定 `box.backend: nsjail`。sandbox 直接作为 Box Runtime 容器内的 nsjail 子进程运行，
+  不创建 nested Docker container、独立 Pod、microVM 或 warm pool，也不挂宿主机 `docker.sock`。
+- 符合 entitlement 的 Workspace 首次使用时懒创建一个逻辑 session，内部固定 ID 为 `global`，并强制 `persistent=True`；
+  外部调用方不能选择或覆盖 session ID、persistence、host path 或 backend。
+- “全局 sandbox 一直存活”在当前机制中的精确定义是：每个合资格 Workspace 最多一个稳定的 `global` 逻辑 session，
+  它不被 TTL reaper 回收，其 `/workspace` 持久保存；普通命令仍按需启动并退出 nsjail 进程，不能承诺一个空闲 OS 进程永久驻留。
+- Box Runtime 重启后，旧进程、attach token、root/tmp/home 和内存 session 状态失效；下一次使用时以相同 Workspace namespace 懒重建
+  `global` session。持久 `/workspace` 必须继续存在，旧 generation 权限必须失败关闭。
+- 共享 Box Runtime 采用单 owner 的 M0 实现；未来多 replica 才引入 session owner lease 和跨 replica 路由，
+  但 session handle 永远不包含 replica 地址。
+- Cloud 首期强制 `network=off`，调用方和 WebUI 不能覆盖。当前 `network=on` 会关闭 nsjail 的独立 network namespace，
+  不能用于共享 SaaS。未来如需联网，必须先实现每 session 独立 netns 和受控 egress，再单独开放。
+- Cloud 首期禁止 `START_MANAGED_PROCESS`，`SandboxAdmissionGrant.max_managed_processes` 固定为 `0`；
+  普通 exec 在同一 Workspace 的 `global` session 内串行执行。未来开放 resident process 前必须增加数量和聚合 CPU/内存上限。
+
+### 4.4 文件与资源边界
+
+- 文件机制沿用当前 nsjail 方案：Box-owned durable volume 上的 Workspace 目录只 bind mount 到对应租户的 `/workspace`；
+  不在首期新增对象存储双向同步服务或文件服务。
+- `/workspace` 的持久性来自独立 durable host path，而不是 `persistent=True`；后者只禁止 TTL/普通 shutdown 回收逻辑 session。
+  Box Runtime 容器必须挂载持久卷，Workspace 数据不能只放在容器可写层。root/tmp/home 可以在 Runtime 重启时丢失。
+- Cloud MVP 要求 Core 与 Box Runtime 以相同路径挂载同一持久卷并沿用直接文件读写；现有 exec/base64 fallback 的单文件上限
+  低于正常附件上限，不能当作等价 Cloud 文件机制。无法共享路径时必须先扩展传输协议，否则 deployment readiness 失败。
+- 现有执行前后目录扫描只能提供软检查，不能阻止单次命令写满共享卷。生产 Cloud 的 Workspace 总空间上限必须由
+  Box-owned volume 的 filesystem project quota/subvolume quota 在写入点原子执行；Box Runtime 负责设置和验证，不能因 Core 看不到路径而跳过。
+- Cloud 的 nsjail CPU、内存、PID、单文件和总空间限制使用运维配置统一设置；套餐只决定 session 数量，
+  不允许 Workspace 放宽 sandbox 上限。
+- Box Runtime 必须在 cgroup v2 hard limit、namespace 和 mount 条件满足后才通过 readiness；不能在共享 SaaS 中告警后降级运行。
+- 同 Workspace 的 `global` session 可以复用持久 `/workspace`，但不同 Workspace 即使使用相同文件名、进程名或逻辑 session ID，
+  物理 namespace、路径、进程和 capability 也必须完全隔离；Cloud 首期没有可暴露端口或共享网络 namespace。
+
+### 4.5 非 Pro 和未来外部 E2B
+
+- 非 Pro Workspace 在首期没有 Cloud managed sandbox，直接调用内部 API 也必须被拒绝。
+- 未来允许 Workspace 在 WebUI 配置自己购买的远程 E2B endpoint/template/secret；它属于 tenant-owned external sandbox，
+  不消耗 Cloud 的 `managed_sandbox_sessions` 配额，也不能读取其他 Workspace 的凭证。
+- 当前 Box Runtime 只有实例级全局 backend 和 E2B credential，WebUI 也没有 Workspace 级配置，因此 BYOK E2B 明确不在首期实现。
+- 未来实现时在共享 Box Runtime 内增加按可信 Workspace context 选择 backend/provider 的 registry，
+  仍不创建租户专属 Box 控制面或新协议。
+
+### 4.6 淘汰与暂缓方案
+
+| 状态       | 方案                                             | 结论                                                     |
+| ---------- | ------------------------------------------------ | -------------------------------------------------------- |
+| 淘汰       | 每 Workspace 一个 Box service                    | 组件和空闲成本随 Workspace 线性增长                      |
+| 淘汰       | 多 Workspace 共享一个活 sandbox/session          | 不能承载不可信代码                                       |
+| 淘汰为 MVP | Docker、独立 Pod、microVM 或 warm pool           | Cloud v2 首期固定使用 Runtime 容器内 nsjail              |
+| 淘汰为 MVP | 非 Pro 使用 Cloud managed sandbox                | 首期数值 entitlement 为 0                                |
+| 后续演进   | 多 Box Runtime replica、dedicated pool、BYOK E2B | 保留 provider/ownership 接口，有真实容量或产品需求后实现 |
+
+### 4.7 验收条件
+
+- Pro entitlement 首次使用时懒创建一个 persistent `global` session；重复和并发请求都不能产生第二个 session。
+- 非 Pro、entitlement 缺失/过期及伪造 plan 的 API 直调全部失败关闭。
+- TTL 不回收 persistent session；Runtime 重启后进程和临时目录失效，但 `/workspace` 保留并能在下一次使用时安全重建。
+- 两个 Workspace 的文件、进程、session、attach token 和 generation 完全隔离；network/managed-process 请求在首期失败关闭。
+- Core 与 Box Runtime 通过随机 marker challenge 证明同一共享持久卷；只配置相同路径字符串不算通过。
+- Workspace、Skill store、ephemeral root/tmp/home 的 byte quota 与 inode quota 在写入点真实生效；现有目录扫描不被当作硬配额。
+- cgroup 或任一硬存储能力不可用时 Cloud Box Runtime readiness 失败；普通 nsjail 因此不会被误当成 production-ready provider。
+
+## 5. D-003：SaaS PostgreSQL 与 pgvector
+
+状态：`PARTIALLY IMPLEMENTED — shared schema/pgvector complete; SaaS transaction and deployment gates remain`
+
+当前分支已实现 PostgreSQL shared schema、transaction-local scope、FORCE RLS、Cloud runtime 非 DDL 模式、
+同业务数据库 pgvector、显式向量维度和 tenant-scoped vector 主键。一次性 migrator 使用独立凭据、advisory lock，
+负责建立并校验 runtime role 的最小权限，并完成全量 schema 验证；
+普通业务写入贯穿 commit 的 generation-aware fence、与外部副作用同事务的 outbox，以及 generation cutover 后稳定的 durable object 引用尚未实现。
+这些 Core 事务原语与生产 Job、凭据发放、备份和回滚流程，以及 runtime credential 的跨 database 连接隔离证明一起，
+都是 Cloud v2 的 SaaS activation gate。
+
+### 5.1 已确认的数据库边界
+
+- PostgreSQL 是 SaaS 的业务数据库，不把它扩展成通用 Runtime coordinator、Box session directory 或新控制面数据库。
+- M0 使用一个 PostgreSQL business database/shared schema 承载全部 Workspace。创建 Workspace 不创建 database、schema、role 或专属连接池。
+- 首版 migrator URL 和 runtime URL 必须归一化到同一 host、port 和 database，但必须使用不同 role。
+  未来如果 migrator 使用 direct endpoint、runtime 使用 pooler endpoint，只能在两个端点都验证同一个数据库内部 cluster identity 后放宽 host/port 相等。
+  该 identity 由 migrator 所有并固定，runtime role 只能读取，不能创建、修改或伪造。
+- 首版唯一业务 schema 固定为 `public`。migrator 和 runtime 连接都必须满足
+  `current_schema() = 'public'` 且 `current_schemas(false) = ARRAY['public']`；禁止 runtime role 级和 business database 级 `search_path` 覆写。
+- migrator 和 runtime session 的安全值固定为 `session_replication_role=origin`、`row_security=on`、
+  `lo_compat_privileges=off`。runtime role 或当前 business database 作用域内只要存在任意 `pg_db_role_setting` 持久化设置就失败关闭，
+  即使该设置当前看似等于安全值也不接受；tenant context 只能通过事务内 `SET LOCAL` 建立。
+- 业务行显式携带 `workspace_uuid`；Repository/Service 的应用层 scope 是第一道边界，PostgreSQL RLS 是第二道边界。
+- runtime role 的直接 ACL 固定为：business database 的 `CONNECT`、`public` 的 `USAGE`、全部 allowlisted business table 的
+  `SELECT/INSERT/UPDATE/DELETE`、`alembic_version` 的只读 `SELECT`，以及业务表自有 sequence 的 `USAGE/SELECT`。
+  不授予 database/schema `CREATE`、table `TRUNCATE/REFERENCES/TRIGGER`、sequence `UPDATE`、其他对象权限或任何 `WITH GRANT OPTION`。
+- runtime role 必须是 `LOGIN`，但不得具有 superuser、`BYPASSRLS`、`CREATEDB`、`CREATEROLE` 或 replication 属性；
+  不得在 role membership 中以 granted role、member 或 grantor 任一方向出现；不得拥有 database、schema、table、view、sequence、routine 或 extension，
+  不得持有 column ACL，也不得使用、创建或拥有其他非系统 schema。
+- business database 必须安装 `vector`，且 extension catalog 只允许 `plpgsql` 和 `vector`；不得存在 FDW、foreign server 或 user mapping。
+  runtime role 和 `PUBLIC` 都不得有显式 routine ACL 或 parameter `SET/ALTER SYSTEM` ACL；runtime role 不得有效执行任何
+  `SECURITY DEFINER` routine，包括被 allowlisted extension 收编的 routine。普通非 `SECURITY DEFINER` 内建函数的隐式执行权限不在此禁令内。
+- PostgreSQL 新 database 默认向 `PUBLIC` 提供的 `TEMP` 是首版在专用业务 database 上明确接受的兼容性决定，
+  不是 migrator 对 runtime role 的直接 grant；首版不得据此把业务 database 与不受信任工作负载混用。
+  migrator 在释放 advisory lock 前建立并校验上述精确 allowlist；Cloud runtime 每次启动都必须重新完成 schema、身份、有效权限和 catalog 负向校验，发现 drift 立即失败关闭。
+- PostgreSQL role 是 cluster-wide identity，当前 database 内的 catalog audit 不能证明同一 credential 无法连接 cluster 中的其他 database。
+  SaaS 生产环境必须使用仅向该 credential 暴露目标 business database 的专用 PostgreSQL cluster/endpoint，
+  或通过已验证的 HBA/proxy policy 证明该 credential 只能连接目标 business database；这项部署隔离仍是未完成的 activation gate。
+- 关键租户表使用 `FORCE ROW LEVEL SECURITY`；migration/repair/audit 使用独立受控 migrator role。
+- 每个租户事务通过 `SET LOCAL` 设置 tenant context，并由统一 `TenantUnitOfWork` 保证设置 context 和业务查询使用同一事务/连接。
+  禁止使用连接级 session variable 或 `search_path`，避免连接池、PgBouncer、异常回滚和后台任务串租户。
+- 一个 `TenantUnitOfWork` 只能访问一个 Workspace。业务写入与对应 business outbox 在同一事务中提交；
+  写入可以校验由执行层传入的 generation/fencing token，但 Runtime owner、lease 和 Box session directory 不由业务 PostgreSQL 承担。
+- SaaS schema、extension 和 policy 只由 release migration job 创建；应用启动角色不执行 `CREATE EXTENSION`、`create_all` 或自动 migration。
+- OSS 继续默认 SQLite，并保留自托管 PostgreSQL 选项；Cloud RLS 约束不让 OSS 强制依赖 PostgreSQL。
+
+### 5.2 pgvector 首期方案
+
+- SaaS 使用 pgvector 作为默认向量数据库，并与业务表使用同一个 PostgreSQL cluster/database；
+  vector schema 可以使用独立 adapter、受控 role 和有上限的 pool，但不新增 Chroma、Milvus 或独立向量数据库服务。
+- OSS 默认仍是 SQLite + Chroma，用户可以显式选择 pgvector；Cloud 配置 pgvector 失败时必须启动失败，不能静默回退到 Chroma。
+- 向量表必须显式保存 `workspace_uuid` 和 `knowledge_base_uuid`，并至少以
+  `(workspace_uuid, knowledge_base_uuid, vector_id)` 建立唯一键/主键和查询条件；服务端生成的 collection name/hash 不是安全边界。
+- pgvector adapter 复用相同的 tenant-context/RLS 契约，每次向量操作在自己的事务中执行 `SET LOCAL`；
+  是否复用普通业务 UoW、role 或 connection pool 由实现决定，但 adapter 不能丢弃 tenant metadata。
+- `vector(1536)` 不能继续作为无条件硬编码。首期使用无 typmod 的 `vector` 列和显式 `embedding_dimension`，
+  以 `CHECK (vector_dims(embedding) = embedding_dimension)` 校验；release migration 为允许的维度创建带 dimension predicate 的 expression/partial ANN index。
+  知识库/model 元数据必须选择已启用维度，写入和查询 mismatch 或未启用维度时失败关闭，不能截断、补齐、退化为无界扫描或换后端。
+- `vector` extension、表、索引和 RLS 由 release migration 创建。应用进程不在启动时执行 DDL。
+- 0013 如需搬迁 legacy pgvector 数据，migrator 作为源表 owner 先记录每个受保护源表的 `ENABLE/FORCE RLS` 状态，
+  仅在同一 migration transaction 内临时暂停 RLS，并在 `finally` 中精确恢复各表原状态。
+  该流程不依赖 superuser 或 `BYPASSRLS`，也不允许在迁移事务外留下已禁用的 RLS。
+
+### 5.3 候选拓扑与未来演进
+
+| 状态     | 方案                                      | 结论                                                                        |
+| -------- | ----------------------------------------- | --------------------------------------------------------------------------- |
+| 首期决定 | P0. shared database/shared schema         | 一个 pool、一套 migration；应用 scope + RLS                                 |
+| 后续演进 | P1. 多 shared database shard              | 每个 shard 仍承载多个 Workspace，并使用相同 schema；有容量/地域证据后再设计 |
+| 后续例外 | P2. dedicated shard                       | 只作为合规、驻留或超大 workload 的资源等级，不建立第二套代码路径            |
+| 淘汰     | schema/database per Workspace             | catalog、pool、migration、备份成本随 Workspace 线性增长                     |
+| 淘汰     | database/schema per replica/Cell/Instance | 把业务数据拓扑错误绑定到计算副本或已删除的产品实体                          |
+
+M0 不提前增加始终返回 `primary` 的 resolver、shard router 或 shard binding。
+P1 的 resolver、映射、在线迁移、连接池预算、shard-affine replica 和 dedicated shard 细节等到出现容量、地域或合规需求时再设计。
+在此之前，direct endpoint 与 pooler endpoint 分离只能通过数据库内部、runtime 不可伪造的 cluster identity 开启，不使用 DNS 名、数据库名或配置声明代替。
+
+### 5.4 备份与生命周期边界
+
+- PostgreSQL PITR 是 database/cluster 级恢复手段，不等同于单 Workspace 恢复。
+- Workspace 创建、释放、export、delete、单 Workspace restore 和在线迁移机制本轮暂缓，后续单独决策；
+  首期不以尚未设计的 export 能力作为数据库架构验收条件。
+- 除关系业务数据和 pgvector 的向量/检索字段外，大对象、插件 artifact 和 sandbox 文件仍存放在对象存储或 Runtime 持久卷；
+  PostgreSQL 保存相应业务元数据和稳定引用。
+- tenant-visible usage/billing 业务行可以进入 PostgreSQL；基础设施 log/metric/trace 不进入业务数据库。
+  高增长业务表在有数据量证据后再决定 retention、时间分区或分析存储。
+
+### 5.5 验收条件
+
+- 故意遗漏应用层 Workspace filter 时，RLS 仍阻止跨租户读写。
+- 连接池/事务池复用、异常回滚、并发请求和后台任务不会残留 tenant context；如部署 PgBouncer，也必须覆盖 transaction pooling。
+- migration 对 shared schema 只执行一次，不产生 Workspace 级 schema drift；应用启动角色不能执行 DDL。
+- 首版拒绝 host、port 或 database 不同的 migrator/runtime URL，并拒绝相同 role；migrator/runtime 都只解析到 `public`，
+  migrator 在迁移后完成精确 table/sequence/`alembic_version` ACL grant 和正反向 role 校验，runtime 每次启动重新校验。
+- runtime role 没有任一方向的 role membership、`WITH GRANT OPTION`、`search_path` 覆写、对象所有权、其他 schema 访问或非业务对象权限；
+  专用业务 database 上可继承 PostgreSQL 默认 `PUBLIC TEMP`，但 runtime role 没有直接 `TEMP` ACL。
+- migrator/runtime session GUC 保持 `session_replication_role=origin`、`row_security=on`、`lo_compat_privileges=off`，
+  runtime role/当前 database 没有任何 `pg_db_role_setting`；extension 仅为 `plpgsql/vector` 且 runtime 不拥有 extension，
+  database 中没有 FDW/server/user mapping、runtime 或 `PUBLIC` 显式 routine/parameter ACL、runtime-owned routine 或 runtime 可执行的 `SECURITY DEFINER` routine。
+- 生产 deployment 证明 cluster-wide runtime credential 只能连接目标 business database；专用 cluster/endpoint 或 HBA/proxy 隔离未经验证前不得启用 SaaS。
+- legacy pgvector 搬迁在非 superuser、非 `BYPASSRLS` 的 table-owner migrator 下可成功，成功、异常和重试路径都精确恢复所有源表的 RLS/FORCE 状态。
+- 业务写入和对应 business outbox 在同一事务内具备可证明的提交顺序；外部 generation/fencing token 校验失败时不产生写入。
+- pgvector 使用真实 PostgreSQL 集成测试覆盖：两个 Workspace 使用相同 `vector_id`、猜测其他 Workspace ID、
+  故意遗漏 scope、连接复用、CRUD 和后台任务，全部不能越权。
+- embedding dimension 不匹配或 pgvector extension 不可用时失败关闭，不回退到其他向量后端。
+- 新建 Workspace 只新增目录与业务行，不创建 database、schema、role 或专属连接池。
+
+## 6. D-004：stdio MCP 独立开关
+
+状态：`IMPLEMENTED`
+
+### 6.1 已修复的原问题
+
+- 修复前，stdio MCP 的启用条件只检查 transport 为 `stdio` 且 Box available，没有独立 feature gate。
+- 修复前，所有 stdio MCP 使用固定的 `mcp-shared` 逻辑 session，并强制 `persistent=True`。
+- 在该旧逻辑下，如果多租户 Cloud 只通过 `box.enabled` 开放能力，每个配置 stdio MCP 的 Workspace 都会额外保留一个 persistent sandbox，
+  绕过“每 Workspace 最多一个 managed `global` sandbox”的成本和套餐边界。
+
+### 6.2 首期决定
+
+新增独立实例配置：
+
+```yaml
+mcp:
+  stdio:
+    enabled: true
+```
+
+- OSS 默认 `true`，保持当前本地部署兼容；Cloud v2 通过 `MCP__STDIO__ENABLED=false` 强制关闭。
+- 该开关与 `box.enabled`、`managed_sandbox` entitlement 和 sandbox session 数量相互独立，不能从任一条件推导。
+- 后续如开放给特定套餐，可在实例开关之上再叠加 Workspace capability；实例开关为 `false` 时任何 entitlement 都不能绕过。
+- HTTP/SSE/其他远程 MCP transport 不受此开关影响。
+
+### 6.3 强制检查点
+
+开关必须同时覆盖：
+
+1. MCP create；
+2. MCP update 到 stdio；
+3. transient connection test；
+4. Core 启动时加载已有 stdio 配置；
+5. RuntimeMCPSession/loader 的最终执行门禁；
+6. WebUI transport selector 和错误提示。
+
+不能只在 WebUI 隐藏选项。Cloud 配置关闭时，已有 stdio 记录保留但不自动启动，并返回明确的 feature-disabled 错误；
+最终 gate 必须位于 Box 分支和 legacy host-stdio 分支之前，不能误报为 `box_unavailable`，
+也不得创建 `mcp-shared` session 或 stdio 子进程。
+
+### 6.4 验收条件
+
+- Cloud 即使 `box.enabled=true` 且 Workspace 拥有一个 managed sandbox，也无法 create/update/test/start 任何 stdio MCP。
+- 直接调用 API、重放旧配置和启动 bootstrap 都失败关闭，且不会产生 `mcp-shared` session、nsjail 进程或额外配额占用。
+- OSS 默认行为保持兼容；HTTP/SSE MCP 正常工作。
+
+## 7. 五项决策之间的关系
+
+五项决策共同遵循：
+
+> 多租户共享可信控制面、连接池、只读 artifact 和基础容量；租户独占不可信执行进程、sandbox、secret、可写文件和数据作用域。
+
+- Plugin Runtime 通过“共享 Supervisor + 每 installation 独立 nsjail 进程”降低控制面数量，同时保留进程级租户隔离。
+- Box 通过“共享 Runtime + 每个合资格 Workspace 一个持久逻辑 session + one-shot nsjail exec”避免每租户部署服务和空闲容器。
+- stdio MCP 独立关闭，防止从 Box availability 隐式产生第二套 persistent sandbox。
+- PostgreSQL 和 pgvector 共享数据库组件，但使用显式 tenant key、应用层 scope 和 RLS 防止共享存储变成共享权限。
+- 订阅管理只在闭源 Control Plane 维护套餐与计费规则，并向开源 Core 投影签名/版本化 entitlement；
+  Core/Runtime 执行通用 capability 和数值限额，不复制套餐名称或计费逻辑。
+- 目录同步只在启动或恢复时读取全量快照；常态变更按 Workspace 聚合为签名增量，新增租户不会使每次目录事件退化为全实例重投影。
+
+## 8. 当前不做分布式时仍保留的能力
+
+1. 所有运行时协议继续携带稳定 `instance_uuid`、`workspace_uuid` 和 execution generation；不能依赖进程地址表达身份。
+2. Core、Plugin Runtime 和 Box Runtime 的本地进程表不能成为 durable desired state 或撤销状态的唯一真相。
+3. 创建、重试、回调、outbox 和 worker 注册使用稳定 idempotency key；重复投递不能产生第二个 owner 或副作用。
+4. Plugin installation 和 Box session 使用稳定 owner abstraction；启用第二个 replica 前再实现带 expiry、CAS 和 fencing token 的 lease。
+   具体 lease store 后续决定，不预设复用业务 PostgreSQL，更不因此新增 Runtime/Box 数据库。
+5. Repository/UoW 不允许无边界跨 Workspace 事务；`workspace_uuid` 从第一天就是内部路由与分片候选键。
+6. schema migration、任务扫描、监控聚合和运维接口不能假设永远只有一个 Core 进程。
+7. 外部 API 不暴露 replica、worker 或 shard 标识；未来扩容不改变 Workspace URL、UUID 或客户端协议。
+8. 只有出现容量、可用性、地域或合规需求时才增加 replica/shard；预留协议不等于现在部署额外组件。
+9. 每个 Core replica 保存自己的事件消费 cursor，因为 entitlement cache 是进程本地状态；PostgreSQL 中的 projection high-water mark
+   、snapshot coverage 和 inbox 仍由所有 replica 共享，用于幂等投影和冲突检测。事件页携带签名 high-water，副本追平前不能续期
+   ready；不能让一个 replica 的共享 cursor 使其他 replica 跳过本地 cache 刷新。
+
+## 9. 本轮明确不做的事情
+
+- 不合并 plugin installation 进程，即使插件和版本完全相同；只允许共享摘要校验后的只读代码/依赖文件。
+- 不允许插件 manifest 声明或覆盖 CPU、内存、PID、文件或存储上限。
+- 不为每个 Workspace 创建 Plugin Runtime、Box service、database、schema、role、bucket、PVC 或消息队列。
+- 不在 Cloud v2 首期使用 Docker sandbox、microVM、warm pool 或非 Pro managed sandbox。
+- 不在首期实现 Workspace 级 BYOK E2B WebUI 配置。
+- 不在 Cloud v2 首期支持 stdio MCP，也不让 Box availability 隐式开启它。
+- 不把业务 PostgreSQL 用作未决定的 Runtime/Box 通用协调数据库，不在缺少容量证据时引入 Redis、Kafka 或新 scheduler。
+- 不在本轮实现 Workspace export、释放、单租户恢复或在线迁移；具体生命周期另行决策。
+- 不实现多个 CloudInstance、Workspace Placement 或 Cell Router；未来分布式只作为单逻辑实例内部的副本和分片能力。
+- 不修改旧 Space 部署模型；Cloud v2 继续按绿地方案设计。

docs/multi-tenant/runtime-resource-audit-2026-07-28.md

@@ -0,0 +1,348 @@
+# LangBot Cloud Runtime 资源安全审查
+
+日期：2026-07-28 至 2026-07-29
+
+审查分支：
+
+- LangBot：`feat/multi-tenants`，审查起点 `32abbb636f4455e965141d8d209b359dbfbb5aae`
+- Plugin SDK：`feat/multi-tenants`，审查起点 `0cddf3c2bea5939c67b71e488a719e9903c28d17`
+
+## 结论
+
+本轮已覆盖 LangBot Core、Plugin Runtime 和 Box Runtime 的主要常驻对象、后台任务、队列、网络客户端、进程生命周期及数据库连接池。本轮定位到的攻击者可控或历史累积状态均已补充容量、超时、淘汰或确定性清理边界；修正后的高基数探针没有观察到随历史请求继续增长的活跃缓存。按本轮“代码审查 + 本地可重复测试”的验收口径，审查已经完成，未发现仍未处理的严重内存泄漏或 CPU 抢占路径。该结论不等于证明任意生产负载下不存在资源问题。
+
+最终 Cloud 拓扑和生产环境不是本轮完成条件。代码级审查、跨仓全量测试和仓库 Dockerfile 构建的 Linux/cgroup v2 探针已经通过，但当前状态仍不能单独作为 Cloud 生产激活批准。完整的环境侧剩余清单见
+[Cloud v2 仍待验证事项](./cloud-v2-pending-verification.md)。其中与本轮资源审查直接相关、上线前还必须完成的项目包括：
+
+1. 在最终 Cloud 部署权限和 cgroup 拓扑下重复 nsjail、namespace 和 delegated cgroup v2 的 CPU、内存、swap、PID、文件句柄验证。本轮一次性 Linux 容器已经证明代码路径可工作，但普通容器和仅 `--privileged` 的 private cgroup namespace 都不满足条件。
+2. 为 Cloud Box 提供并验证硬文件系统 quota provider。普通 nsjail bind mount 不能证明总字节数和 inode 硬配额，当前严格 readiness 按设计会失败关闭。
+3. 使用最终生产配置分布继续做容量测试，并据此确定单实例 Workspace placement 上限。本轮真实 PostgreSQL 16 + RLS 启动测试已经覆盖 1,000 个各带 Provider、三类 Model、Bot、Pipeline、KnowledgeBase、MCP 和 Plugin setting 的 Workspace，启动加载耗时和 SQL 次数保持线性；5,000 Workspace 的合成三代替换探针也证明旧运行时会释放。仓库已新增可同时采集 Core/Plugin/Box HTTP、进程树和 cgroup v2 的 24 小时门禁工具，并在受 CPU、memory、swap、PID 硬限制的 Linux 容器中完成短时自检；但最终生产候选拓扑的 24 小时运行仍未执行。测试中的 fake adapter/requester/Plugin handler 仍不能替代真实平台 SDK、外部连接池和插件进程的容量数据；合法活跃租户本身仍会线性占用内存。
+SDK 已先行发布到分支提交 `1d65ed301a6afc52150a998043f73cd6032c8162`，本提交集中的 LangBot
+`pyproject.toml` 和 `uv.lock` 已精确钉住该提交。最终镜像仍需按待验证清单记录并核对实际安装版本。
+
+## 覆盖范围
+
+### LangBot
+
+- 启动、停机、全局任务管理和运行时配置。
+- PostgreSQL、Tenant UoW、RLS、迁移和共享 pgvector。
+- HTTP、MCP、WebSocket、上传下载、S3/本地存储、维护任务和异步用户任务。
+- QueryPool、Pipeline Controller、会话/对话、限流、第三方 Agent/LLM runner 和同步 SDK 桥接。
+- PlatformManager 以及 DingTalk、QQ、Lark、WeCom、WeComCS、WeChatPad、LINE、Kook、Satori、OpenClaw Weixin、Telegram、Discord、Matrix、HTTP/WebSocket 等适配器。
+- Plugin Runtime connector、插件包校验、Marketplace 下载、pip 安装输出和 desired-state reconcile。
+- Box connector、admission、session/process 生命周期和 RPC 文件。
+- RAG、向量后端、Skill、Storage、Telemetry 和日志缓存。
+
+### Plugin SDK
+
+- stdio/WebSocket transport、请求 waiter、action task 和文件传输。
+- Runtime control handler、Workspace/generation fence 和 EventContext。
+- 插件 artifact、Marketplace、pip 依赖安装、共享依赖环境、installation desired state、Supervisor 和 worker launcher。
+- nsjail 参数、cgroup/rlimit、进程注册 capability 和 Runtime shutdown。
+- Box admission、generation fence、session/process/reaper、RPC/relay WebSocket 和 nsjail backend。
+
+## 主要修复
+
+### 确定性生命周期
+
+- 修复 `Application.shutdown()` 使用 `contextlib.suppress` 却未导入 `contextlib` 的问题。原行为会在实际资源关闭分支直接 `NameError`，阻断后续 Plugin、Box、HTTP 和数据库释放。
+- `make_app()` 在任一启动 stage 或初始化失败时会关闭尚未返回给 `main()` 的半构建 Application；Telemetry、Box、Tool、Platform、Vector 和 HTTP manager 在初始化前即挂到 Application，避免初始化中途失败后清理器无法发现已经创建的连接、会话、子进程或后台任务。
+- MCP streamable-HTTP session manager 现在随 Application shutdown 显式退出。
+- MCP loader 按 `(instance, workspace, generation)` 管理 host task 和 session；任务完成会从注册表移除，代次推进会取消旧 host task 并关闭旧 session，reload/shutdown 会先清空现有运行时，避免 completed task 和旧代连接永久驻留。
+- Platform bot reload/remove/shutdown 统一串行化，旧 bot、代理、adapter 任务和进程会先停止再从注册表移除。
+- Model provider requester 新增异步关闭契约；provider reload/remove、Workspace generation 替换、全量 reload 和 Application shutdown 都会确定性关闭旧 requester，允许第三方 requester 安全持有自己的 HTTP client 或连接池。
+- Plugin Runtime、Box Runtime、stdio transport、adapter 连接和共享 HTTP client 均补齐 close/cancel/await。
+- HTTPX 有界流在超限异常或消费者取消时会立即关闭底层响应流；原来的 response hook
+  只在正常读完后由 HTTPX 自动关闭，持久客户端反复收到超大响应时可能积累未释放连接。
+  已消费响应在超限分支也会先 `aclose()` 再传播错误。
+- `Application.dispose()` 只允许一个可追踪 shutdown task；重复的信号、窗口关闭或调用方清理不会铺开多个并行停机流程。
+- Lark、微信、钉钉、企业微信和 QQ Official 的凭证交换后台任务统一进入 Application TaskManager，受全局/单 Workspace admission 约束并随应用停机取消；容量满时关闭尚未调度的 coroutine 并返回 429，不留下游离 task。
+- `TaskCapacityError` 已下沉到无 Application/controller 依赖的纯错误模块。原来的 HTTP 过载异常路径会在特定冷启动导入顺序下触发 TaskManager/controller 循环导入，把应返回的 429 变成框架 500。
+- S3 storage provider 在初始化失败或 Application shutdown 时关闭 botocore HTTP connection pool；Storage manager 在 provider 初始化前即挂到 Application，避免 bucket probe 失败后遗留 client。
+- 修复 Coze runner 每次请求创建 `aiohttp.ClientSession` 却不关闭的问题；现在 runner 的 `aclose()` 会确定性关闭底层 API client。
+- LINE SDK client 现在随 adapter 停止关闭；Plugin Runtime shutdown 回调只创建一个可追踪、可等待的后台任务，重复回调不会累积清理 task。
+- SDK `lbp publish` 现在用上下文管理器关闭插件包上传文件；原实现的成功、API 错误和 HTTP 错误返回路径都会遗留文件句柄。
+- Plugin worker、Box Docker/CLI backend、Box nsjail backend 和 nsjail 依赖安装在调用方取消时会 terminate/kill、读取管道并 `wait()` 回收子进程；Windows worker 的原生进程路径也进入相同的 `finally` 清理契约，避免取消安装、停机或超时后留下孤儿进程。
+- Box 服务入口现在从 Runtime initialize、aiohttp app/runner setup、端口 bind 到主循环共用一个外层清理边界；建站期间的非 `OSError` 也会关闭 Runtime 和 reaper。WebSocket 控制模式端口绑定失败会退出并交给编排器重启，不再在没有任何可用 RPC/health 端口时永久等待；stdio 模式仍允许仅 relay 绑定失败后继续控制通道。
+- Plugin artifact 解压、installation staging/activation/rollback/delete、共享依赖环境和 nsjail session 目录的关键文件系统变更均移出事件循环。已经开始的原子变更在取消时会先等待线程结束，再回滚临时目录、目标目录或旧 supervisor，不会让后台线程继续修改一个调用方已经认为清理完毕的路径。
+- Core 和 SDK 的阻塞 executor 新增独立的有界清理入口。普通工作在容量耗尽时仍快速拒绝；已经拥有资源的 close/unlink/rmtree/进程回收则等待有限 worker 槽位并保证原子操作结束后再传播取消，避免过载恰好导致清理任务被拒绝。SeekDB、Milvus、S3、LINE、WeChatPad、MCP staging 和 TBox 临时文件等关键路径已接入。
+
+### 有界队列、缓存和历史状态
+
+- QueryPool 同时限制全局与单 Workspace 的 queued/running query；调度后不再被过载淘汰；历史 scope counter 有上限。
+- Session、Conversation、WebSocket connection/proxy/message、rate-limit identity、task record/log、telemetry task、vector handle 和 adapter 私有队列均有容量或 LRU/TTL。
+- SessionManager 现在维护 Workspace 二级索引和带 revision 校验的最小过期堆。新会话只扫描目标 Workspace 的有界会话集，TTL 回收只消费已过期堆前缀，全局 idle 淘汰使用最小堆；高频命中产生的旧堆项按活跃会话的有界倍数压缩。原实现会在每个攻击者可制造的新 launcher id 上扫描并排序实例全部会话。
+- SDK 的 EventContext 和依赖准备锁使用 weak reference；generation、admission、installation、capability 和 completed-process 状态有上限。
+- Plugin restart circuit 打开期间只有 `max_concurrent_restarts` 个 supervisor
+  能持有冷却计时器/状态等待，其他 installation 睡眠在同一个 semaphore FIFO；
+  probe 状态变更在调用者取消时仍会完成，避免 half-open 永久占用。
+- Box nsjail 启动时只扫描一次 `/proc`，并流式删除遗留 session 目录；不再为每个
+  遗留目录重复扫描全部进程或先把全部目录物化到内存。
+- Box Skill discovery、目录列表和列表正文分别限制扫描 entry、package、返回 entry
+  与累计文本字节；BFS 使用 deque，拒绝 inode 洪泛导致的 O(N²) 或无界列表。
+- Core message aggregation 使用 `(instance, workspace, generation)` O(1) scope
+  counter 做准入，不再为每个新 launcher 扫描全实例 buffer。
+- Cloud remote MCP 的 idle execution fence 不再由每个 session 每 5 秒查询
+  Workspace/ExecutionState；签名目录投影事务提交后将失效 scope 合并进一个有界
+  cleanup worker。实际工具/资源调用前后仍保留数据库强校验。
+- 空 Workspace 不再预分配 Model generation scope、Plugin installation set 或 Box generation event；只有 Workspace 实际拥有对应运行时资源或等待任务时才创建这些对象。
+- Runtime RPC 文件同时限制单文件字节数和单连接未消费文件数量，连接关闭时清理连接拥有的临时文件。
+- Box Runtime 维护实例与 Workspace 到活跃 session 的二级索引；创建、删除、过期、撤销和 shutdown 共用同一清理路径，避免每个租户 RPC 都扫描实例中的全部 session。
+- Box Runtime 另外只索引“可过期 session”和“持有 managed process 的 session”。Cloud 的持久 `global` sandbox 不进入 TTL 索引，managed process 被禁用时进程索引为空；session 创建、周期 reaper、状态和 `/healthz` 因此不会随全部持久租户数线性扫描。测试把总 session 字典替换为禁止迭代的映射，仍能完成第二个持久 session 创建、reap、status 和 health。
+- Core MCP loader 同样维护 Workspace/generation 到 session、host task 的二级索引；请求、代次回收和动态配置不再扫描实例中的全部租户 MCP session，已完成 task 的 done callback 会同步移除所有索引。
+- Box admission 过期回收使用带 revision/generation 校验的最小堆，只访问已到期记录；重复续期产生的旧堆项会被忽略，堆大小超过活跃 grant 的有界倍数时压缩，不再在每次 RPC 上全表扫描所有租户 grant。
+- 旧 QQ message ID/object cache 和 stdio MCP Workspace copy lock 不再随历史请求无限增长。
+- LLM/Agent runner 的单次生成结果默认限制为 1 MiB，流式传输限制单事件 1 MiB、单请求累计 16 MiB，并限制最多 100,000 个流式事件，避免上游异常响应无限占用内存或 CPU。
+- Marketplace JSON 限制为 1 MiB、插件包限制为 64 MiB；pip stdout/stderr 各最多保留 1 MiB，超出部分继续 drain 但不驻留内存。
+- Plugin Runtime stdio/WebSocket 协议除 16 MiB 消息字节上限外，新增最多 4,096 个入站和出站碎片的对象数量上限；大消息的 UTF-8 编码、分片、拼接、JSON 编解码和 Pydantic 验证均在线程执行。WebSocket receive 异常使用稳定的 `ConnectionClosed` 类型，不再因库顶层未导出 `exceptions` 属性而在错误路径二次失败。
+- HTTPX response hook 和 aiohttp 有界读取统一把第三方响应限制为 10 MiB；JSON 解析及诊断文本转换在线程执行，错误正文最多保留 4 KiB，避免大 JSON 在共享事件循环集中解析。
+- 图片、data URI 和平台媒体默认限制为 10 MiB，Base64 在解码前先校验编码长度；Plugin binary storage 默认单值 10 MiB，并设置不可由错误配置绕过的 64 MiB 绝对上限。
+- Skill 文本单文件限制为 1 MiB，Plugin UI 文件限制为 4 MiB，host edit 文件限制为 1 MiB；Box Skill ZIP、插件 artifact 和 GitHub Skill archive 同时限制条目数、单文件、解压总量和压缩比。
+- SDK E2B 文件同步限制为最多 2,048 个目录项、1,024 个文件、单文件 10 MiB、总计 50 MiB；同步文件 IO 从事件循环移到线程。
+- Dify 待提交表单、用户 Space OAuth state 和 Cloud launch JTI replay cache 改为带 revision 校验的最小过期堆；Space credits 使用按时间有序的 LRU/TTL 队列。原实现会在每次攻击者可触发的请求上扫描整个历史缓存并在满容量时再次线性寻找最旧项；现在过期回收为摊销 `O(log N)` 或仅消费已过期前缀，旧堆项会忽略并按活跃状态的有界倍数压缩。
+- Cloud launch JTI cache 达到 4,096 个仍有效 token 时失败关闭，不再为了接纳新 token 淘汰仍有效的 replay 记录；否则攻击者可以在容量满后重放被提前遗忘的合法签名。
+- Entitlement resolver 现在跟随 Cloud directory 的权威 Workspace 活跃集合。全量目录投影会丢弃已 fenced/removed Workspace 的历史 entitlement snapshot，delta 批量更新不会对每个变化重复扫描；provider 请求进行中发生目录撤销时，返回前的第二次 active fence 会阻止旧结果重新写回缓存。
+- Cloud directory 的签名响应、Workspace、membership 和实例 active Workspace
+  均新增可配置操作上限与绝对上限。闭源适配器按流读取响应并在 JSON/JWS 解析前
+  拒绝超过 32 MiB 默认值的解压后正文；Manifest/entitlement/event endpoint 再
+  分别限制为 256 KiB、256 KiB 和 2 MiB。entitlement 刷新最多并发并驻留 16 个
+  原始响应，逐批验证成小型 snapshot 后释放；Core 在目录投影行锁保护的事务内检查最终
+  active 数量，超限时回滚 Workspace、Account、membership、inbox 和 cursor，不会
+  截断权威数据或让并发副本各自越过最后一个容量槽。
+- Space 全量目录只投影 active Workspace，历史 archived Workspace 只在最多 100 个
+  目标的签名 delta 中作为 tombstone 返回。注册创建新个人 Workspace 前通过
+  PostgreSQL transaction advisory lock 串行执行全局 active 数量准入；达到上限
+  返回 503，避免多个 Space 副本同时观察到最后一个空位。
+- Monitoring 分页、offset、CSV export 和 session/message detail 均在 service
+  边界执行实例配置上限与不可放大的绝对上限；detail 的完整统计改为 SQL aggregate，
+  只物化有界的 tool/LLM/error 明细并显式返回 `detail_truncated`。默认分页 1,000、
+  export 10,000、detail 2,000，绝对上限分别为 5,000、50,000、10,000。
+- Token statistics 的时间序列不再把筛选范围内的全部 LLM call 拉回 Python 分桶；
+  PostgreSQL 使用 `date_trunc`、SQLite 使用 `strftime` 在数据库中聚合，并只返回
+  最近 1,000 个时间桶（绝对上限 10,000）。模型分组复用分页上限并在 SQL 中按 token
+  排序、限制；两类结果都返回显式的 `*_truncated` 标志。
+- Monitoring 过期数据每表每轮默认最多删除 4 个批次、绝对最多 100 个批次；本地/S3
+  过期上传文件候选和每轮删除默认最多 1,000、绝对最多 10,000。单个历史数据量异常的
+  Workspace 不再能让一次维护循环无限物化候选或持续清空全部 backlog。
+- Workspace webhook 数量默认限制为 16、绝对限制为 64；管理查询和运行时 fan-out
+  都只物化有界结果。实例同时发送的 webhook 请求默认限制为 16、绝对限制为 128；
+  满载时直接跳过未获准目的地，不创建一批等待 semaphore 的 task。取消调用时会取消并
+  await 已创建的所有请求任务，归还实例槽位。
+- Local/S3 Storage 的对象读取在实际 IO 中只读取 `limit + 1` 字节，S3 body 在成功、
+  超限和异常分支都会关闭；默认单对象 10 MiB、绝对上限 64 MiB。所有 scoped load
+  以及 WebSocket attachment 都经过同一边界，写入也不能产生当前实例无法安全读取的对象。
+- Valkey Search 的批量删除改为固定页流式搜索、删除并累计计数，不再把全部匹配 key
+  保留在 Python 列表；每次删除后从 offset 0 继续，避免结果集缩短造成跳项，并设置
+  1,000 轮绝对终止条件。
+
+### CPU 和事件循环保护
+
+- 修复旧 QQ `repeat_seed('')` 空输入无限循环。
+- ZIP 校验/重打包、PIL、Base64、AES、JSON 解析、fsync probe、插件 artifact/依赖文件、Skill、S3、本地存储和维护目录扫描从事件循环移到线程。
+- 公开 Slack、QQ Official、HTTP Bot、公众号、WeCom/WeComCS 回调体显式限制为 1 MiB；JSON/XML 解码移出共享事件循环。QQ、DingTalk、Satori、WeCom AI 和 WeChatPad 网关帧同样设置 1 MiB 上限或在解码前拒绝超限消息；KOOK zlib 数据使用 10 MiB 解压后硬上限，阻断小压缩包制造的大内存解压。
+- HTTP Bot 的幂等键和 outbound session 现在均在写入前执行硬容量 admission；满额时只按固定 64 项预算检查最旧记录，不能先超限后整体清空，也不再在每条回复上扫描和排序全部 session。已有 session 继续 O(1) 访问，新 session 在没有可安全回收的空闲记录时失败关闭。
+- Dashboard、Embed 和 Plugin Runtime 的协议 JSON 编解码在线程执行；Dashboard/Embed 在接收端提交 terminal error 后为发送端保留有界 drain 窗口并使用内部 sentinel 唤醒，不会因“任一方向结束即取消”在撤权错误帧发出前关闭连接。
+- 租户配置的敏感词、内容忽略和群响应正则统一使用声明为直接依赖的 `regex` 引擎：最多 64 个 pattern、单 pattern 1,024 字符、输入 1 MiB、单次总匹配 CPU 预算 50 ms，并在线程中执行。超时、非法正则和替换放大均失败关闭；灾难性 `(a+)+$` 回归在 1 ms 测试预算内被中断。
+- 原生 `read/write/edit/glob/grep` 文件工具移出事件循环并继承 Workspace 阻塞预算。目录列举、递归 walk、grep 文件/总字符、单行、pattern、结果和 regex CPU 均有硬上限；glob 只用固定大小最小堆保留最新 100 项，不再先把全部命中路径驻留内存。Box 内执行的 glob/grep 脚本同样限制命中集合、扫描量和正则时间。
+- Dify、DingTalk、QQ、WeCom 等客户端复用连接并在生命周期结束时关闭，响应体和下载有字节上限。
+- DashScope、TBox 等同步第三方 SDK 的调用和生成器迭代改为在线程执行；单个同步生成器最多消费 100,000 个事件。
+- Dashboard 和 Embed WebSocket 改为任一收发 task 结束即取消并等待另一方向，避免发送端退出后接收 task 永久阻塞；两方向 task 同时继承从认证结果或 RuntimeBot 得到的可信 Workspace 阻塞预算。
+- Plugin installation 生命周期全局串行化；不同租户的依赖 pip/nsjail 准备不会在安装高峰并发抢占 CPU。
+- Plugin installation 的意外退出除了每 installation 的 jittered exponential backoff，还经过 Runtime 全局 restart launch 并发槽和失败窗口熔断。
+  熔断冷却后只允许一个 half-open probe；probe 必须完成初始化并持续稳定后才恢复其他 installation。未在 30 秒内 ready 的 worker
+  会被取消回收。健康指标输出 active launch、窗口失败数、circuit 状态和累计打开次数，24 小时门禁把 circuit 打开或尾段启动槽不归零判为失败。
+- S3 同步 SDK 使用线程执行，并通过实例级 semaphore 限制并发；默认 `storage.s3.max_concurrency=16`，可通过实例配置和环境变量覆写。
+- Box 子进程 stderr 以 64 KiB 块读取，日志最多每秒输出 4 个摘录并汇总抑制数量，避免无换行或刷屏输出制造无界缓冲与日志放大。
+- Plugin worker 日志单行最多保留 64 KiB；Box managed-process stdout relay 以固定 64 KiB 块读取，不再依赖换行符，避免超长无换行输出触发 `StreamReader` limit 或堵塞子进程。
+- Box generation fence 的代次更新改为只访问目标 Workspace 的 event 和 active-task 二级索引。原实现每次更新都会遍历全部 Workspace 的 fence/task 记录，10,000 个 Workspace 的第二阶段更新会退化为 O(N²) 并在 40 秒后仍未完成；修正后包括其他 SDK 高基数负载和本轮协议 offload 在内的当前完整双阶段探针耗时 `11.270s`。
+- Box session 枚举、旧 generation 回收和 admission 计数均通过 Workspace 索引执行；admission 过期回收通过最小堆执行，不再在每次 RPC 上产生 O(实例总 session/grant 数) 的扫描。
+- Model、Pipeline、RAG 和 Platform manager 均维护 Workspace 到运行时 key 的二级索引。Workspace generation 更新只清理目标 Workspace 的缓存和运行时，不再扫描实例内所有租户的 provider/model、pipeline、knowledge runtime 或 bot；回归测试使用禁止全局迭代的映射验证该边界。
+- Cloud heartbeat 直接读取已加载且有容量边界的 Pipeline、MCP、KnowledgeBase 和 Bot registry 计数，不再为每个活跃 Workspace 依次打开 Tenant UoW、执行四类 COUNT 查询；这消除了租户数增长后每日周期性形成的串行 SQL/CPU 尖峰。OSS 模式仍保留数据库统计语义。
+- 邀请、Monitoring 和 Storage 的三个周期清理 task 合并为一个
+  `resource-maintenance` 调度器。调度器先等待首个 interval，不与启动加载争抢资源；
+  同一到期周期只执行一次 active Workspace discovery，然后按 Workspace 串行运行
+  有界 job，单 Workspace 失败不跳过其他 Workspace。默认相同的一小时周期由此从
+  三次全租户发现和三个同时唤醒的任务收敛为一次发现和一个任务。
+- Cloud 启动阶段先生成一份经过部署适配器和目录投影校验的 Workspace binding 快照，Model、Platform、Pipeline、RAG 和 Plugin 初始化共用该快照，初始化完成后立即释放；避免启动期间为每个 manager 重复执行整批租户发现和投影校验。
+- Platform、Pipeline 和 RAG 的资源加载在使用已验证启动快照时不再为每个 Bot/Pipeline/KnowledgeBase 重新查询同一个 execution binding；常规请求和动态更新路径仍保留数据库 generation fence。
+- MCP 初始 host 和 shutdown burst 由实例级 semaphore/批次限制；默认 `mcp.lifecycle_concurrency=16`，支持 `MCP__LIFECYCLE_CONCURRENCY` 覆写并硬性限制最大 128。初始加载不再先为每个 server 创建一个等待 semaphore 的 task，而是由一个可取消 dispatcher 每批最多物化 `lifecycle_concurrency` 个子 task；同时去掉了 ORM server/config 的双份临时列表，避免大量租户启动时集中占用 CPU、内存、socket 和文件句柄。
+- Core、Plugin Runtime、Box Runtime 和独立 Plugin worker 的默认 `asyncio.to_thread()` executor 统一改为硬有界线程池。默认最多同时运行 8 个阻塞调用、排队 128 个，达到容量后立即抛出 admission 错误，不再使用 Python 默认 `ThreadPoolExecutor` 的无界工作队列保留任意数量的请求对象、Future 和闭包。每个可信 Workspace 的 running + queued 默认再限制为 4，并强制配置值不超过 worker 数的一半，避免单个租户先提交一整批同步工作占满全部 worker/FIFO 队列。Core 使用 `system.blocking_executor.max_workers/max_pending/max_inflight_per_scope`，原生支持对应的 `SYSTEM__BLOCKING_EXECUTOR__*` 覆写；SDK 进程使用 `LANGBOT_BLOCKING_EXECUTOR_MAX_*`，并分别限制全局最大值为 64/4096。
+- Core、Plugin Runtime 和 Box Runtime 各自运行固定 1 秒间隔、仅保留最近 120 个样本的 event-loop lag monitor；健康快照输出 current/recent max/recent p95/进程期最大延迟和累计样本数。Plugin Runtime 的两个 WebSocket 端口现在都在免认证 `/healthz` 返回同一份无凭据、无租户标识的聚合 JSON，Box `/readyz` 同时附带资源快照。24 小时门禁默认拒绝缺失或停止的 monitor、超过 1 秒的 recent max、超过 250 ms 的尾段 recent p95 及 sample counter 回退。
+- Workspace 阻塞预算由服务端认证后的 `RequestContext`、公开 bot 的 RuntimeBot、公开对象 key 中经 binding fence 验证的 Workspace、Platform/TaskManager 的 ExecutionContext，以及 SDK 入站 ActionContext 建立，不接受调用方伪造的租户 header。公开 webhook、公开对象下载、Dashboard/Embed WebSocket、普通 HTTP handler、Platform adapter 和 detached tenant task 均已覆盖。容量拒绝在 Core HTTP 路径返回稳定的 429，health/debug counter 分开报告 global 与 scope rejection。
+- Argon2 密码 hash/verify 只允许一个实例级在途操作，额外并发立即返回容量错误而不是在 asyncio semaphore 中无限积累等待请求；该 CPU/内存密集工作同时使用独立的 `system:authentication` 阻塞作用域。Cloud 本身仍禁用本地密码登录。
+- WeCom 扩展 API 的无限客户端超时改为 120 秒；平台 webhook 的 AES、媒体 Base64 与同步 SDK 调用均移出共享事件循环。
+- 长文本转图片限制为 100,000 字符、256 行、800 万 RGBA 像素和 10 MiB 输出；
+  超限时回退到 forward message。数字边界查找从重复 `count/find/sort` 改成线性扫描，
+  PIL image 使用显式关闭，压缩步长为零时也能终止。
+- Core 在每次 quota-enforced Box exec 前后遍历 Workspace 时使用非递归 DFS，并在
+  超过字节 quota 或默认 100,000/绝对 1,000,000 个目录项后立即停止；目录项洪泛
+  失败关闭，不再重复完整扫描 inode bomb。远程 outbox fallback 同时限制扫描项、
+  文件数、单文件和总字节，Python project manifest 使用分块 hash 并限制单文件 10 MiB。
+
+### 插件和 Box 资源隔离
+
+- Plugin worker 数量受 `max_workers`、`max_total_cpus / max_cpus` 和 `max_total_memory_mb / max_memory_mb` 的最小值约束。
+- Shared profile 强制 Linux 和 nsjail；Cloud 强制 `plugin.worker.require_hard_limits=true`；cgroup v2 delegation 不可用时拒绝启动。
+- 每个 worker 下发 CPU、memory、swap、PID cgroup 限制，以及 process、open-file、file-size rlimit；插件 manifest 不能提高限制。
+- Box nsjail 的 cgroup v2 路径现在同时设置 `memory.max` 和 `memory.swap.max=0`。修复前，48 MiB 沙盒可以把强制提交的 128 MiB 页面换出并正常退出，形成宿主 swap 抢占；修复后同一探针以 exit 137 被 cgroup 杀死。
+- 仓库 Docker Compose/Kubernetes 示例显式下发 Core、Plugin Runtime 和 Box Runtime 的 blocking executor 上限；Kubernetes Box readiness probe 从仅报告进程存活的 `/healthz` 改为 `/readyz`，使 backend 或 managed-mode 隔离检查失败时不会把 Pod 加入就绪流量。
+- 相同 digest 的已验证代码和依赖环境可只读共享，每个 installation 的 home/tmp/data 和进程独立。
+- SDK 在发布共享依赖环境前最多校验 100,000 个目录项和 2 GiB 常规文件元数据总量；
+  超限的 staging tree 会被原子清理而不会进入 worker。`requirements.txt` 和插件
+  `manifest.yaml` 都使用 `limit + 1` 有界读取，manifest 额外限制为 1 MiB。
+- Box session、managed process、completed process、admission record 和 RPC 文件均有实例级上限；Cloud entitlement 仍限制每个合资格 Workspace 一个 `global` session、零 managed process。
+- Box Runtime 对上述实例级配置再增加不可放大的硬上限：session 5,000、managed
+  process 1,024、completed process 10,000、admission record 250,000、RPC 单文件
+  100 MiB、completed retention 86,400 秒。初始化与远程 INIT 对错误类型、负数和
+  超上限均失败关闭，错误动态更新不会留下部分生效的 limit。
+
+### PostgreSQL
+
+- Cloud 强制 PostgreSQL 业务库、共享 pgvector 和允许的固定向量维度。
+- pgvector Cloud 模式复用业务数据库的同一个 AsyncEngine，不创建第二个连接池。
+- `database.postgresql` 新增并校验 `pool_size`、`max_overflow`、`pool_timeout_seconds`、`pool_recycle_seconds`；默认最大连接数为 `10 + 10`。
+- `pool_size + max_overflow` 的绝对上限为 100，timeout/recycle 也有绝对上限；
+  Cloud runtime 的 asyncpg 连接默认设置 60 秒 statement timeout、5 秒 lock
+  timeout 和 60 秒 idle-in-transaction timeout，并分别限制最大 300/60/300 秒。
+  一次性 release migration 不继承这些短 runtime timeout。
+- `/healthz` 输出 pool 配置容量、checked-in/out、overflow、pool admission timeout
+  累计数和 SQL timeout 配置；目录同时输出 active/max 与最近批次
+  Workspace/membership 数，供生产 soak 和告警核对。
+- Application shutdown 显式 dispose 业务引擎；standalone pgvector 仅关闭自己拥有的引擎。
+- PersistenceManager 提供统一异步 shutdown；Cloud 常驻进程的启动失败、正常停机和一次性 release migration 的成功/异常路径都会释放数据库引擎。真实 PG catalog 测试还覆盖了“入口已经关闭后测试再次复用 manager 会重开 pool”的第二生命周期，严格资源告警模式下无 asyncpg socket/transport 遗留。
+
+## 本轮采用的默认决策
+
+- 优先 fail closed 或淘汰最老的 idle cache，不允许攻击者控制的历史 key 无限驻留。
+- 插件依赖准备选择实例级串行化，以稳定 CPU/磁盘峰值；代价是批量安装耗时增加。
+- PostgreSQL 使用一个显式有界共享连接池；未拆分 pgvector pool。
+- 单实例目录的默认 active/full-snapshot Workspace 上限均为 1,000，membership
+  上限为 20,000，签名响应上限为 32 MiB；绝对上限分别为 5,000、100,000 和
+  64 MiB。Space 和 Core 必须配置为同一操作上限，生产值只能根据 V-08 容量曲线
+  向下调整或在重跑全部门禁后提高。
+- 第三方 runner 采用 1 MiB 单结果、16 MiB 单流总量和 100,000 个同步/异步事件的统一实例级安全上限；超限请求失败关闭。
+- S3 默认允许 16 个并发阻塞调用，最大配置值 128；在没有独立 worker service 的前提下限制线程池排队和上游连接压力。
+- MCP 生命周期默认并发 16、最大 128；该限制统一约束实例启动时的 session host 峰值和 shutdown 批次，不允许租户配置单独放大。
+- Core 与 SDK 各进程的通用阻塞 executor 默认使用 8 个 worker、128 个 pending 槽位、每 Workspace 4 个在途槽位；它是实例/进程级共享背压，不由 Workspace 或插件 manifest 调高，单 Workspace 配置硬性不得超过 worker 的一半。生产值应按容器 CPU 和上游阻塞时延校准，不能把 pending 当吞吐配置无限放大。
+- 插件包下载上限 64 MiB，pip stdout/stderr 保留上限各 1 MiB；这不会限制安装进程实际输出，只限制父进程内存中的诊断副本。
+- 通用远程响应和媒体默认上限 10 MiB；错误诊断正文只保留 4 KiB。Plugin binary storage 默认 10 MiB、绝对上限 64 MiB；Skill 文本、Plugin UI 和 host edit 分别限制为 1 MiB、4 MiB 和 1 MiB。
+- Storage scoped object 默认读写上限 10 MiB、代码绝对上限 64 MiB；Webhook 默认每
+  Workspace 16 个、实例 16 个同时出站请求，代码绝对上限分别为 64 和 128。Box
+  Workspace quota 扫描默认最多访问 100,000 个目录项、绝对最多 1,000,000 个。
+- SDK 共享依赖环境在发布前最多接受 100,000 个条目、2 GiB 常规文件元数据总量；
+  artifact manifest 与 requirements 各最多 1 MiB。这些是 Runtime 控制面在启动
+  worker 前的保护，不替代最终文件系统的 byte/inode 硬配额。
+- Monitoring 查询上限由 `monitoring.query_limits` 配置并支持原生环境变量覆写，但始终
+  受代码绝对上限约束；cleanup 的每表批次数和 Storage 每轮文件数同样采用实例配置加
+  绝对上限。时间序列默认/绝对上限为 1,000/10,000 个数据库聚合桶，模型分组复用分页
+  上限。提高这些值必须计入 V-08/V-09 的数据库 CPU 与 Core RSS 容量曲线。
+- Managed-process relay 保留 stdout 的原始换行，并按 64 KiB WebSocket frame 分块；不再承诺“一行对应一个 frame”。这是为无换行输出提供确定内存边界所需的协议收敛。
+- 本轮没有把 Pipeline、Model、KnowledgeBase 等合法租户资源改成 lazy runtime。该改动会改变启动和请求语义，留到 Workspace placement/释放机制一起设计。
+- 本轮没有为普通 nsjail 声称伪硬盘配额；严格 Cloud readiness 保持失败关闭。
+
+## 验证结果
+
+| 验证项 | 结果 |
+| --- | --- |
+| LangBot Ruff + `git diff --check` | 通过 |
+| Plugin SDK Ruff + `git diff --check` | 通过 |
+| LangBot 全量测试（含 unit/integration/Box/E2E） | `2855 passed, 33 skipped` |
+| Plugin SDK 全量测试 | `1328 passed` |
+| Space Go 全量测试与闭源 Cloud Adapter 测试 | Go `go test ./...` 通过；Adapter `40 passed` |
+| Space PostgreSQL 16 Cloud v2 目录与并发容量准入 | 通过；两个注册并发争用最后一个槽位时 `1 success / 1 capacity rejection / 1 active Workspace` |
+| Core PostgreSQL 16 Cloud runtime server timeout | 真实连接从 `pg_settings` 读回 `60000ms / 5000ms / 60000ms` 的 statement/lock/idle-transaction timeout，并显式 dispose |
+| 真实 PostgreSQL 16 + pgvector 迁移/RLS/发布测试（严格资源告警） | `22 passed` |
+| 真实 PostgreSQL 16 + RLS populated Cloud 启动容量 | 500 Workspace `6.178s / CPU 3.026s`；当前 1,000 Workspace 复跑 `12.109s / CPU 5.967s` |
+| 较早 Core Dockerfile Linux 镜像构建与 `regex` 导入 | 通过，image SHA `8893a14053df`；该镜像使用旧 SDK pin，已失效，最终候选必须重建 |
+| `ResourceWarning` + `PytestUnraisableExceptionWarning` 全量门禁 | Core 与 SDK 均通过，并已固化到 pytest 配置 |
+| Plugin SDK Box 专项测试（含全局扫描回归保护） | `669 passed` |
+| Docker Compose 渲染、Compose/Kubernetes YAML 解析与 diff 检查 | 通过 |
+| Cloud soak 门禁解析/采样/判定单元测试 | `27 passed` |
+| Core/Plugin SDK event-loop monitor 专项测试 | 两仓各 `7 passed`，包含真实 50 ms scheduler stall |
+| Cloud soak Linux 硬限制短时自检 | 通过；CPU `0.5`、memory+swap `256 MiB`、PID `128` 均从 cgroup v2 读回，冷却尾段 verdict `pass` |
+| Core 双阶段历史 churn 资源探针（使用当前本地 SDK 分支） | audit 通过，`12.559s` |
+| Core 5,000 个 populated Workspace 三代容量探针（使用当前本地 SDK 分支） | 当前复跑通过，最大替换耗时比 `1.405` |
+| Plugin SDK 双阶段资源探针 | audit 通过，`11.270s` |
+
+两个仓库新增了可重复执行的历史 churn 探针，Core 另有 populated Workspace 三代替换探针：
+
+```bash
+# LangBot Core
+PYTHONPATH=../langbot-plugin-sdk/src uv run python scripts/runtime_resource_probe.py --scale audit --json
+
+# LangBot Core：5,000 个带代表性资源的 Workspace
+PYTHONPATH=../langbot-plugin-sdk/src uv run python scripts/workspace_runtime_capacity_probe.py --scale audit --json
+
+# LangBot Core：真实 PostgreSQL 16 + RLS populated Workspace 启动
+TEST_POSTGRES_URL=postgresql+asyncpg://... \
+LANGBOT_PG_CAPACITY_WORKSPACES=1000 \
+uv run pytest \
+  tests/integration/persistence/test_migrations_postgres.py::TestPostgreSQLTenantRuntime::test_populated_cloud_startup_is_linear_and_task_bounded \
+  -q -W error::ResourceWarning --log-cli-level=INFO
+
+# langbot-plugin-sdk
+uv run python scripts/runtime_resource_probe.py --scale audit --json
+```
+
+Core audit 每个阶段执行 10,000 个空 Workspace 的真实 Model/Plugin manager 加载与 reconcile、25,000 次 Query、2,500 次 session churn、10,000 个限流身份、5,000 个 task 和 2,500 次 WebSocket churn。第一、第二阶段的保留状态完全一致：
+
+- 20,000 个历史空 Workspace：Model scope/provider/LLM、Plugin Workspace set/installation 均为 `0`。
+- 50,000 个历史 Query：活跃 query cache `0`，历史 scope counter `100`。
+- 5,000 个会话身份：session cache `200`。
+- 20,000 个限流身份：rate-limit container `10,000`。
+- 10,000 个历史 task：task record `200`。
+- 5,000 次 WebSocket churn：conversation 与 stream index 均为 `200`。
+- event-loop task、线程和文件描述符保持 `1 / 1 / 6`；使用当前本地 SDK 分支的复跑中，第二阶段相对第一阶段 RSS 增长 `2,228,224 bytes`、tracemalloc current 增长 `344,669 bytes`，总耗时 `12.559s`。Session 淘汰改为 Workspace 索引和最小堆后，同一 audit 工作量相对此前 `16.150s` 明显下降。
+
+Populated Workspace audit 为 5,000 个 Workspace 各加载一个 Provider、LLM、Embedding、Rerank、Pipeline、Bot、KnowledgeBase 和 MCP session，然后全部推进两个 generation：
+
+- 三个阶段的活跃 provider/model、pipeline、bot、knowledge 和 MCP registry 均精确维持 `5,000`，不存在按历史 generation 增长。
+- 到第三阶段，前两代的 requester、Bot adapter 和 MCP session 各 `10,000` 个全部收到确定性关闭；weak reference 断言旧代对象可被回收。
+- event-loop task、线程和文件描述符保持 `1 / 1 / 6`；使用远端精确钉住 SDK 的当前复跑中，第三阶段相对第二阶段 RSS 增长 `1,245,184 bytes`，tracemalloc current 仅增长 `2,061 bytes`。
+- 初始/第一次替换/第二次替换分别耗时 `1.893s / 2.549s / 2.659s`，最大替换耗时比为 `1.405`，未随历史代次出现 CPU 退化。
+- macOS RSS sample 从初始的 `154,648,576` 增至第一阶段 `368,181,248`、第二阶段 `389,087,232` 和第三阶段 `390,332,416 bytes`；第二次替换只比第一次替换增加约 1.19 MiB，但“合法活跃租户资源的线性容量”仍必须作为 placement 容量输入。这里使用轻量 fake adapter/requester，不应把第一阶段约 204 MiB 增量外推为生产每租户成本。
+
+Plugin SDK audit 每个阶段执行 25,000 次 loopback RPC、5,000 次安装 binding 激活/撤销、10,000 个 Workspace generation 更新和 2,500 次带 Workspace 上下文的 Box session 创建/删除。第一、第二阶段的保留状态完全一致：
+
+- RPC waiter、stream queue、action task 和活跃 installation binding 均为 `0`。
+- installation watermark 为有界的 `5,000`；Workspace generation record 为有界的 `10,000`，没有等待者时 generation event 为 `0`。
+- generation active task/index、Box session、Box Workspace session index、creating/closing/background task 和 session lock 均为 `0`。
+- event-loop task 和文件描述符保持 `1 / 7`；当前复跑第二阶段相对第一阶段 RSS peak 增长 `2,637,824 bytes`、tracemalloc current 增长 `289,746 bytes`，总耗时 `11.270s`。耗时增加来自本轮把大协议消息的 JSON/Pydantic、UTF-8 编码、分片和拼接移入有界线程池；结构状态和第二阶段 tracemalloc 增量保持平稳。
+
+第二轮反向静态审查另外枚举了 Core 的 50 个显式 task 创建点和 204 个线程、阻塞调用及子进程调用点，以及 SDK 的 28 个显式 task 创建点和 62 个线程、阻塞调用及子进程调用点。第三轮独立复核继续从高基数定时器、目录遍历、准入全表扫描和取消竞态反推，新增关闭了 Plugin restart 冷却唤醒群、MCP idle 数据库轮询、nsjail orphan 的 O(session × process) 启动扫描、message aggregation 的 O(buffer) 准入及 Skill inode/文本列表边界。显式 task 均具有持有者、完成回调或 `finally` 回收路径；所有生产入口在第一次 `asyncio.to_thread()` 前安装有界默认 executor。Core、Plugin Runtime 和 Box 的公开 `/healthz`（Box `/readyz` 亦同）会输出各自的 aggregate runtime/resource counter 和 event-loop lag，供 soak 对比活跃量、pending、累计 capacity rejection 与调度延迟；不输出 debug key、控制 token、租户或插件身份。Plugin Runtime 的授权 debug info 复用同一资源快照，避免公开/私有指标语义漂移。
+
+真实 PostgreSQL populated 启动门禁会先通过 release migration 创建最新 schema，再用无 `BYPASSRLS` 的临时 Cloud Runtime 角色启动。每个 Workspace 都含九类代表性资源，测试会走实际的 instance discovery、tenant UoW、启动 binding 快照和 Model/Platform/Pipeline/RAG/MCP/Plugin 加载路径：
+
+- 500 Workspace：启动加载 `6.178s`，进程 CPU `3.026s`。
+- 当前 1,000 Workspace 复跑：启动加载 `12.109s`，进程 CPU `5.967s`；相对此前 500 Workspace 的墙钟比为 `1.960`。
+- `model_providers`、`llm_models`、`embedding_models`、`rerank_models`、`bots`、`legacy_pipelines`、`knowledge_bases`、`mcp_servers`、`plugin_settings` 九张表的 SELECT 次数均精确等于 Workspace 数，没有重复的全租户发现或超线性资源扫描。
+- MCP host dispatcher、host task 和临时 Runtime 角色/asyncpg 连接在测试结束后均清空；严格 `ResourceWarning` 模式通过。
+
+探针要求第二阶段的结构状态与第一阶段精确相等，并对第二阶段 RSS 与 tracemalloc 增长设置失败阈值。macOS 的 RSS 来源是 `getrusage` peak，因此这里验证的是峰值增量边界而非“当前 RSS 回落”；最终 Linux 24 小时 soak 仍需采集 current RSS/PSS 和 cgroup `memory.current`。
+
+LangBot 全量测试的 33 个 skip 中，22 个是默认全量运行未提供 PostgreSQL/pgvector 而跳过的集成用例，10 个是未提供 Valkey，另 1 个是可选环境的 collection skip；真实 PostgreSQL 相关路径已由上表单独运行覆盖。Plugin SDK 的 26 个 warning 为现有 Pydantic v2 deprecation 与 aiohttp AppKey 建议；没有失败、未关闭资源或资源上限降级。Core 当前全量产生 194 个既有第三方/兼容性 warning；`ResourceWarning` 和 `PytestUnraisableExceptionWarning` 仍由 pytest 配置提升为错误，本轮没有此类泄漏告警。
+
+Linux Runtime 探针使用上述镜像并只读挂载本地最新 SDK 源码：
+
+- 普通容器：nsjail binary 可执行，但 namespace、mount、network 与 cgroup v2 检查均为 `false`，严格 readiness 按预期失败关闭。
+- `--privileged` + private cgroup namespace：namespace、mount、network 通过，但 cgroup v2 delegation 为 `false`，仍按预期不能进入 Cloud ready。
+- 一次性容器内建立可写 delegated cgroup 子树后：Plugin 与 Box cgroup 探针均为 `true`，nsjail namespace、mount、network 和 cgroup v2 均通过；硬文件系统与 inode quota 继续报告 `false`。
+- `cpus=0.1` 的 1.0 秒 process-CPU busy loop 实际耗时 `9.13s`；`memory_mb=48` 下逐页提交 128 MiB 以 exit `137` 终止；`pids_limit=8` 下批量 fork 返回 `EAGAIN`。这些结果验证了 CPU、memory+swap 和 PID 的实际内核执行路径。
+- 新增 `scripts/cloud_runtime_soak.py` 后，在同一 Linux 镜像的独立容器中设置 `--cpus 0.5 --memory 256m --memory-swap 256m --pids-limit 128`，工具从目标 cgroup 读回 quota `50000/100000 usec`、memory `268435456 bytes`、swap `0 bytes` 和 PID `128`。最终复跑中，32 MiB 子负载退出后的 4 秒冷却尾段 `memory.current` 稳健增长和斜率均为 `0`，平均 CPU `0.00132 cores`，OOM、memory pressure、PID max 和 throttle delta 均为 `0`，最终 verdict 为 `pass`。这只是采集器/判定器自检，不替代最终 24 小时生产候选运行。
+- 本地实际启动 Plugin Runtime 后，控制端口与 debug 端口的公开 `/healthz` 均返回相同聚合 JSON，event-loop monitor 为 running，且正文不含 debug key。采集器显式绕过进程级 HTTP proxy 后，对控制端口执行 6 秒短时 endpoint gate：无失败，观测到的 recent max/p95 均为 `2.233 ms`，verdict 为 `pass`。
+- 本地实际启动 Box Runtime（未创建 sandbox session）后，`/healthz` 与 `/readyz` 均返回 event-loop、blocking executor、session/process/task 聚合快照；monitor 为 running、样本持续增长，两个端点观测到的 recent max 均为 `2.265 ms`。SIGINT 后 aiohttp、Runtime、reaper 与 monitor 走统一清理路径并正常退出。
+
+## 上线配置与监控门禁
+
+最终 24 小时命令、运行位置、阈值语义、负载矩阵和产物要求见 [LangBot Cloud 24 小时资源 Soak 门禁](./cloud-runtime-soak-gate.md)。该工具默认把任一健康失败、OOM/memory pressure、PID limit、CPU throttling 超阈值、blocking executor rejection、冷却尾段内存持续增长或空闲 CPU 过高判为失败；生产运行必须使用 `--require-hard-limits`。
+
+至少需要监控并告警：
+
+- Core/Plugin Runtime/Box Runtime 的 RSS、CPU throttling、OOM、PID 数和 event-loop lag。
+- 各进程 blocking executor 的 running、pending、inflight、active scopes、`global_rejected_total` 和 `scope_rejected_total`；pending 持续不归零或 rejection 增长都应告警。
+- QueryPool、WebSocket、session、task、plugin worker、Box session 的当前量、容量拒绝和淘汰计数。
+- Plugin crash/restart 频率、dependency prepare 耗时和失败率。
+- PostgreSQL pool checked-out/overflow/wait timeout、事务耗时和连接错误。
+- 临时文件、artifact、dependency environment、Box Workspace volume 的字节数和 inode。
+
+生产 soak 应覆盖租户突发登录、批量插件 reconcile、插件崩溃重启、WebSocket 断连、Box 并发执行、PG pool 饱和和应用 SIGTERM；持续运行至少 24 小时，并验证负载停止后 RSS、task、socket、文件和子进程数量回到稳定基线。

docs/multi-tenant/verification-report.md

@@ -0,0 +1,271 @@
+# Cloud v2 multi-tenant verification report
+
+Date: 2026-07-24
+
+Status: `FOUNDATION AND CONTROL PLANE VERIFIED — PRODUCTION ACTIVATION REMAINS GATED`
+
+This report records the implementation and verification evidence for one
+logical LangBot instance serving multiple Workspace tenants. It covers the
+open-source Core, the shared Plugin/Box runtimes, the closed Space adapter, and
+the Space Cloud v2 modular-monolith control plane. It does not claim that the
+production Cloud deployment may enable `CLOUD_V2_ENABLED` yet.
+
+## Repository refs and scope
+
+- LangBot Core: branch `feat/multi-tenants`, commit
+  `e8a09b7537ef285a967f24add05fdb9bb557b97e`
+- langbot-plugin-sdk: branch `feat/multi-tenants`, head
+  `ca545d079ca1657a5d4efb4e31bfeafe1a374a46`
+- langbot-space: branch `feat/cloud-v2-control-plane`, head
+  `ce41ff370e94a405f70e2fbb2f99b0946e0e0387`
+- Closed Core adapter: `langbot-space/cloud-adapter`
+- SDK protocol/package version: `0.4.18`
+
+Core pins SDK commit `e7d946af4a6b1494fbe74627c1815ace19ac8991`;
+the SDK branch head adds CI-only follow-up. Cloud v2 is a greenfield
+multi-Workspace deployment and does not provision one Pod, database, queue, or
+Runtime per tenant. Legacy Space Pods remain a compatibility surface only.
+
+## Implemented product and security boundaries
+
+### Workspace and identity
+
+- SaaS has one stable `instance_uuid`; `workspace_uuid` is the tenant key.
+- Registering an Account creates its personal Workspace and owner Membership in
+  the same PostgreSQL transaction.
+- A Workspace may contain multiple users with owner, admin, and member roles.
+  Invitation acceptance is token-hash based, email-bound, single-use,
+  concurrency-safe, and member-limit checked while locked.
+- Community remains exactly one local Workspace while supporting multiple
+  Accounts and fixed RBAC roles. Installing the closed adapter is required to
+  inject the SaaS policy; configuration alone cannot activate it.
+- Disabled or deleted Accounts cannot use password/code/OAuth login, sessions,
+  refresh/access tokens, or personal access tokens.
+
+### Closed Space control plane
+
+- `CLOUD_V2_ENABLED` defaults to false. Invalid or incomplete configuration
+  fails startup, and disabled endpoints return 503.
+- Space owns Workspace/Membership/Invitation, versioned plans, subscriptions,
+  entitlements, usage events, directory outbox, and signed instance manifests.
+- Free and Pro are Workspace plans. Pro projects one managed global sandbox;
+  Free projects none. Both force stdio MCP off in the first release.
+- Subscription changes use Workspace advisory locking, expected entitlement
+  revision, one-live-order uniqueness, idempotent usage ingestion, period-end
+  downgrade, renewal, and lazy expiry settlement.
+- Account provisioning and quota projection into New API use PostgreSQL
+  transactional outboxes, replica-safe claims, retry/backoff, revision fencing,
+  and database-enforced identity/user/token ownership. No additional service is
+  introduced.
+- EPay and Stripe callbacks are bound to the locked order's provider identity,
+  amount, currency, channel/session, expiry, and provider transaction ID.
+  Replays are idempotent, EPay is CNY-only, credential rotation is supported by
+  encrypted per-order snapshots, and permanent fulfillment conflicts are
+  retained for reconciliation.
+- Directory snapshot and per-Workspace delta reads use repeatable-read,
+  read-only transactions with a high-water cursor. Core stores a replica-local
+  consumer cursor and shared PostgreSQL projection state, snapshot coverage,
+  and inbox rows.
+- The `/cloud` page selects a Workspace and shows its independent subscription,
+  entitlement, limits, and usage. When Cloud v2 is disabled or the backend does
+  not expose the feature field, the complete legacy Welcome/Pod UI is used.
+
+### PostgreSQL and pgvector
+
+- SaaS business data uses one PostgreSQL shared schema with application scope
+  plus forced RLS. Cloud directory writes are separated from local tenant
+  writes.
+- Projected Account and Membership revisions are monotonic. Tombstones remove
+  memberships, and stale revisions cannot resurrect them.
+- pgvector shares the business PostgreSQL database and remains
+  Workspace-scoped.
+- Space runs versioned SQL migrations before application seeds. A fresh
+  database, a partial pre-migration database, and an existing-baseline path
+  converge without startup-time `AutoMigrate`.
+
+### Shared Plugin Runtime
+
+- One instance-scoped trusted Supervisor serves multiple Workspaces.
+- Every enabled installation has its own nsjail process bound to
+  `(instance, workspace, execution generation, installation, runtime revision,
+  artifact digest)`.
+- Verified same-digest code and dependency files are mounted read-only and may
+  be shared; home, tmp, data, process namespace, registration capability, and
+  cgroup are private.
+- Instance configuration owns CPU, memory, PID, open-file, and file-size
+  limits. Plugin manifests cannot increase them.
+- Memory includes swap: nsjail receives `memory.max` and `swap.max=0`.
+- Unexpected worker exit is recovered by a completion callback with bounded
+  per-installation exponential backoff. Remove, reconcile, Runtime shutdown,
+  and container SIGTERM perform a graceful-to-SIGKILL bounded reap.
+
+### Shared Box Runtime and MCP
+
+- One shared Box control plane serves Workspace-bound logical sessions.
+- Cloud grants allow at most one persistent `global` session for an entitled
+  Workspace and no managed processes in the first release.
+- Cloud is fixed to nsjail and network-off. Core and Box prove the shared
+  durable Workspace mount with an authenticated marker challenge.
+- stdio MCP is independently gated and forced off for Cloud v2.
+- The current nsjail Box backend does not provide hard byte/inode quotas, so
+  Cloud readiness correctly fails closed instead of silently using a soft
+  directory scan.
+
+## Automated verification
+
+### LangBot Core
+
+```text
+uv run --no-sync pytest -q
+  2590 passed, 32 skipped, 177 warnings
+
+real PostgreSQL migration, pgvector, and release-migrator suites
+  21 passed, 11 warnings
+
+uv run --no-sync ruff check .
+uv lock --check
+git diff --check
+  passed
+```
+
+The full suite ran without the closed adapter installed, proving the open-source
+single-Workspace/multi-user path remains standalone. Focused closed-adapter,
+directory projection, runtime connector, Box cleanup, and configuration suites
+also passed with the adapter installed.
+
+### Plugin SDK and real Linux runtime
+
+```text
+SDK full suite
+  1226 passed, 22 existing warnings
+
+Ruff check and format check
+git diff --check
+  passed
+```
+
+A privileged Linux test container with host cgroup namespace ran one shared
+Runtime and two Workspace installations:
+
+- both workers referenced the same artifact inode;
+- home, tmp, and data inodes were distinct;
+- each plugin saw only PID 1 in its private PID namespace;
+- a tampered binding and an unknown installation were rejected;
+- the control token was absent from worker environments;
+- cgroups were distinct with `memory.max=134217728`, `memory.swap.max=0`,
+  `pids.max=32`, and `cpu.max=500000 1000000`;
+- touching 256 MiB exited with code 137 without swap growth;
+- the 32nd fork failed with `EAGAIN`;
+- reconcile and container SIGTERM removed the worker cgroups.
+
+The same run started the Runtime from a non-root working directory, covering
+absolute nsjail mount-source normalization.
+
+### Space backend, adapter, and frontend
+
+```text
+MIGRATIONS_TEST_DSN=... MIGRATIONS_TEST_DSN_FRESH=... \
+  go test -count=1 ./...
+go vet ./...
+  passed against PostgreSQL 16
+
+fresh PostgreSQL app startup, partial-baseline migration,
+Cloud v2 migration rerun, and control-plane integration
+  passed
+
+closed adapter pytest and Ruff
+  passed
+
+pnpm exec tsc --noEmit
+pnpm check:i18n
+pnpm check:cloud-checkout-currency
+  passed; 7 checkout/currency cases
+```
+
+The PostgreSQL checks started from an empty database and verified all 34
+registered migrations in order, Cloud v2 Free/Pro seeds, legacy plan seeds,
+Cloud columns/indexes, payment callback constraints, New API outbox/ownership
+constraints, and repeatable reruns.
+
+## Cross-service and browser E2E
+
+### Signed Space-to-Core directory projection
+
+Using an isolated Space PostgreSQL database and a migrated Core PostgreSQL
+database:
+
+1. Space issued a signed manifest for the fixed LangBot instance.
+2. Space returned a directory snapshot at cursor 5 containing two Workspaces,
+   two projected Accounts, and owner/admin memberships.
+3. Core verified the signature, instance, release, validity window, and
+   capability before injecting the Cloud Workspace policy.
+4. Core stored both Workspaces, all active Memberships, both projected
+   Accounts, snapshot coverage, inbox entries, and cursor 5.
+5. Account-field-only projection revisions and Workspace directory revisions
+   remained independent, and cross-Workspace Account conflicts failed closed.
+
+### Real browser Cloud v2 flow
+
+A real local browser operated the Space frontend and backend:
+
+1. An existing legacy-Pod owner logged in and saw the automatically created
+   personal Workspace on Free.
+2. The page showed Free and Pro Workspace plans while retaining the legacy Pro
+   instance card, its Online state, URL, version, billing period, and actions.
+3. Annual Pro checkout through the configured EPay/Alipay rail was re-quoted
+   from the displayed USD plan price to `¥490.00 CNY`.
+4. The browser reached the EPay gateway with `money=490.00`; no USD amount was
+   sent through the CNY-only rail.
+5. A valid signed `TRADE_SUCCESS` callback returned `success`. An exact replay
+   also returned `success`, leaving one successful order and one Pro Workspace
+   subscription.
+6. Refreshing payment state cleared the pending indicator. The page then showed
+   the Pro annual period and managed-sandbox entitlement while the legacy Pod
+   remained Online.
+
+The browser run used the real Space UI and HTTP handlers. Its disposable local
+development harness added a same-origin Next.js rewrite only in the temporary
+worktree; that harness change was removed after the run.
+
+The legacy feature-flag branch was then covered by API/static checks and the
+production build: false or missing `cloud_v2_enabled` renders the original
+Welcome/Pod client; a failed web-config request renders an explicit retry
+instead of guessing a deployment mode.
+
+### Deliberate Core startup failure
+
+Core completed signed manifest verification and directory projection, then
+stopped at the Box readiness gate because the current nsjail backend cannot
+prove hard Workspace byte/inode quota enforcement. Connector shutdown and
+reconnect tasks were cleanly reaped; no event-loop or never-awaited coroutine
+warning remained.
+
+This is a successful fail-closed acceptance result, not a passing production
+Cloud boot.
+
+## Remaining production activation gates
+
+Cloud v2 must remain disabled until these gates are closed:
+
+- Box provides and proves hard byte and inode quotas for Workspace, Skill,
+  root, home, and tmp storage.
+- Plugin installation writable data receives an operator-owned hard total
+  disk quota.
+- Plugin and future networked Box workloads have tenant-safe egress/SSRF
+  policy.
+- Plugin Runtime adds jitter, global restart concurrency limiting, and a
+  Runtime-level circuit breaker, then passes systemic-failure injection.
+- M0 rolls Core and Plugin Runtime together until authenticated Runtime takeover
+  or an owner lease/fencing protocol exists.
+- Payment operations add scheduled reconciliation and alerting for stale
+  `processing` orders and persisted permanent fulfillment conflicts.
+- Cloud v2 subscription service periods are stored immutably and included in
+  recognized-revenue reporting.
+- Production migration Job, backup/rollback, PostgreSQL credential/network
+  boundaries, horizontal-replica fault injection, Workspace release/export,
+  deletion, and restore semantics are completed.
+
+These gates intentionally add no tenant-specific service. They are implemented
+inside the existing Space, Core, Plugin Runtime, Box Runtime, and PostgreSQL
+components to preserve the architecture goal: near-zero static cost for a new
+Workspace.

docs/multi-tenant/workspace-multi-user-architecture.md

@@ -0,0 +1,941 @@
+# LangBot Workspace 多用户与 SaaS 多租户架构
+
+状态：`ARCHITECTURE BASELINE — isolation kernel implemented; SaaS activation gates remain`
+
+本文描述 Cloud v2 的目标架构和安全边界。详细的 Runtime、Box、PostgreSQL、pgvector 与 stdio MCP 决策以
+[pending-architecture-decisions.md](./pending-architecture-decisions.md) 为权威来源；已经落地的实现选择记录在
+[implementation-decisions.md](./implementation-decisions.md)。
+
+“隔离内核已实现”仅表示开源 Core/SDK 已具备多租户数据和运行时隔离所需的基础能力，
+不表示闭源控制面、计费、生产部署或 Cloud v2 已经可以上线。
+
+## 1. 架构决策摘要
+
+Cloud v2 采用以下模型：
+
+> SaaS 对外只有一个逻辑 LangBot 实例，全部 Workspace 都是该实例内的租户；
+> 开源 Core 提供完整隔离内核，闭源 Cloud Control Plane 管理 SaaS 目录、订阅、权益和计费。
+
+核心决策如下：
+
+1. `Workspace` 是数据、成员、权限、用量和不可信执行的租户边界，不是一个 Pod、namespace、数据库或独立 LangBot 部署。
+2. SaaS 注册 Account 时自动创建个人 Workspace；这只新增目录与业务记录，不创建租户专属服务、数据库、队列或 Runtime。
+3. OSS 每个 LangBot 实例只能存在一个 Workspace，但该 Workspace 可以有多个 Account、邀请和固定角色。
+4. SaaS 才允许一个 Account 拥有或加入多个 Workspace，并在 WebUI 中切换当前 Workspace。
+5. MVP 可以各运行一个 Core、Plugin Runtime 和 Box Runtime 进程；未来增加副本或 PostgreSQL shard 仍属于同一个逻辑实例的内部扩展，不改变产品模型和外部 API。
+6. 一个共享 Plugin Runtime 控制面管理所有 Workspace，但每个运行中的 plugin installation 独占一个 nsjail 进程；enabled-resident 是 desired semantics，只读代码和依赖可按已验证摘要共享。
+7. 一个共享 Box Runtime 管理所有 Workspace；首期符合 entitlement 的 Workspace 最多拥有一个持久 `global` 逻辑 sandbox，实际命令继续以 nsjail 子进程执行。
+8. SaaS 业务数据使用 PostgreSQL shared schema、应用层 scope 与 RLS 双重隔离；pgvector 位于同一个业务数据库并作为 SaaS 默认向量后端。
+9. stdio MCP 有独立实例开关，Cloud v2 首期强制关闭，不能由 Box availability 或套餐能力隐式开启。
+10. 闭源 Control Plane 可以作为模块化单体复用现有账户、支付和运营能力，但历史 Cloud 的租户专属部署模型不进入新架构。
+11. Workspace 创建、释放、export、单 Workspace restore 和在线迁移的具体流程仍待后续决策。
+
+本轮重构的最高目标是：
+
+> 共享可信控制面和基础设施池，隔离不可信执行单元；减少独立部署和常驻组件，使新增 Account 或空 Workspace 的静态成本接近零。
+
+减少组件数量不意味着合并安全边界。插件进程、sandbox、secret、可写文件和租户数据仍必须严格隔离。
+
+## 2. 范围与非目标
+
+### 2.1 本方案覆盖
+
+- OSS 单 Workspace 多用户、邀请和固定 RBAC。
+- SaaS 多 Workspace 账户、成员和 Workspace 切换模型。
+- HTTP、WebSocket、API Key、Bot、Webhook、后台任务和内部调用的可信 Workspace 上下文。
+- Bot、Pipeline、Provider、Knowledge、Plugin、MCP、RAG、Session、Storage 和 Monitoring 的租户隔离。
+- Plugin Runtime 与 Box Runtime 的共享控制面和进程级隔离。
+- SaaS PostgreSQL shared schema、RLS 与 pgvector 边界。
+- 开源 Core 与闭源 Control Plane 的职责、协议和故障边界。
+- 当前单副本运行和未来同一逻辑实例内横向扩展的兼容约束。
+- 分阶段实施、激活门禁和验收策略。
+
+### 2.2 本方案不覆盖
+
+- 兼容或原地升级历史 Cloud 的租户专属部署方案。
+- 为每个 Workspace 创建独立服务、数据库、schema、role、bucket、PVC、队列或 Runtime。
+- 当前阶段实现多副本调度、跨地域 active-active 或 PostgreSQL 在线分片迁移。
+- 第一版自定义角色、SAML、SCIM 或企业离线授权。
+- 第一版 Workspace 级 BYOK E2B WebUI 配置。
+- Cloud v2 首期 stdio MCP。
+- Workspace export、释放、单租户恢复和在线迁移的具体产品流程。
+
+历史客户数据、账户和财务记录如需迁移，应单独立项；旧部署拓扑不作为本架构的设计约束。
+
+## 3. 术语与不变量
+
+### 3.1 术语
+
+| 术语 | 定义 |
+| --- | --- |
+| Account | 登录主体。OSS 中是实例本地账户；SaaS 中是全局账户 |
+| Workspace | 逻辑 LangBot 实例内的租户，是资源、成员、权限、用量和不可信执行的首要边界 |
+| Membership | Account 与 Workspace 的关系，包含固定角色、状态和权限版本 |
+| Invitation | 邀请一个 Account 或邮箱加入 Workspace 的一次性凭证 |
+| Logical Instance | 对外唯一的 LangBot 服务与安全域，拥有稳定 `instance_uuid`，不等同于某个进程或 Pod |
+| Replica | Core、Plugin Runtime 或 Box Runtime 的短期内部运行副本，不是产品实体 |
+| Execution Generation | Workspace 执行所有权和撤销的单调代数，用于隔离旧任务、旧连接和故障转移 |
+| Billing Account | SaaS 付款主体，可以为一个或多个 Workspace 付费 |
+| Entitlement | Control Plane 签发、Core 与 Runtime 本地执行的功能和数值额度快照 |
+| Cloud Control Plane | 闭源 SaaS 控制面，管理全局身份、Workspace 目录、订阅、权益、计费和生命周期 |
+| LangBot Core | 开源数据面，执行 Bot、Pipeline、Plugin、MCP、RAG 等业务并实施最终授权与隔离 |
+
+当前代码中的 `placement_generation` 字段在迁移完成前保留兼容；其架构语义和目标命名均为
+`execution_generation`，不表达 Workspace 属于某个产品级部署单元。
+
+### 3.2 必须始终成立的不变量
+
+1. SaaS 只有一个稳定 `instance_uuid`；所有副本共享该身份。
+2. `replica_id`、`worker_id`、Pod 名称、进程地址和数据库连接地址都是短期运行信息，不能进入业务资源的永久主键或外部 URL。
+3. `workspace_uuid` 是租户数据、任务、缓存、文件、日志、用量和运行时隔离的稳定键，也是未来内部路由与分片的候选键。
+4. OSS 一个实例最多一个 Workspace；SaaS 才能激活多个 Workspace。
+5. 一个 Workspace 可以有多个 Account；一个 SaaS Account 可以加入多个 Workspace。
+6. 所有租户业务资源都具有非空 `workspace_uuid`，并使用 `(workspace_uuid, resource_uuid)` 定位。
+7. Workspace 选择器只是路由输入，不是授权凭证；服务端必须重新验证 Account、Membership、资源所有权和权限。
+8. API Key、Bot、Webhook、后台任务、Plugin 与 Box 调用从可信所有权或绑定派生 Workspace，不能信任调用方自报 scope。
+9. SaaS 缺少有效 Workspace 上下文时必须失败关闭，不能回退到第一个、最近或 OSS 默认 Workspace。
+10. Core 是资源访问、运行时授权和 entitlement 执行的最后一道边界；Control Plane 不同步代理每条消息或普通资源请求。
+11. 一个不可信插件进程只能属于一个 installation；一个 sandbox/session 只能属于一个 Workspace。
+12. execution generation 失效后，旧任务、连接、回调和副作用必须被拒绝。
+13. 本地进程表、缓存和临时目录都可重建，不能成为 desired state、撤销状态或业务数据的唯一真相。
+14. 创建空 Workspace 不启动插件 worker、sandbox 或租户专属常驻组件。
+15. 未来横向扩展不能改变 Workspace UUID、外部 API、权限模型或隔离语义。
+
+## 4. 产品与部署模型
+
+### 4.1 SaaS 逻辑拓扑
+
+```mermaid
+flowchart LR
+    User["Browser / API / Bot traffic"] --> Edge["SaaS Edge"]
+    User --> CP["Closed Cloud Control Plane<br/>directory + subscription + billing"]
+    Edge --> Core["One logical LangBot instance<br/>Core replica pool; MVP = 1"]
+    CP -->|"signed manifest, directory projection,<br/>entitlement and desired state"| Core
+    Core -->|"usage outbox and observed state"| CP
+    Core --> PG["Shared PostgreSQL business database<br/>RLS + pgvector"]
+    Core --> PluginRT["Shared Plugin Runtime<br/>trusted supervisor"]
+    Core --> BoxRT["Shared Box Runtime<br/>trusted supervisor"]
+    PluginRT --> PluginA["Workspace A installation<br/>isolated nsjail process"]
+    PluginRT --> PluginB["Workspace B installation<br/>isolated nsjail process"]
+    BoxRT --> SandboxA["Workspace A<br/>persistent global logical sandbox"]
+    BoxRT --> SandboxB["Workspace B<br/>persistent global logical sandbox"]
+    Core --> ObjectStore["Shared durable object storage<br/>Workspace-scoped keys"]
+```
+
+这里的“一个逻辑实例”是一个服务、安全域和稳定身份，不是“永远只有一个 OS 进程”。
+MVP 不实现分布式，但从第一天保留内部扩展所需的身份、幂等、generation 和 owner 抽象。
+
+### 4.2 容量演进
+
+| 阶段 | 内部部署形态 | 新 Workspace 静态成本 | 启用条件 |
+| --- | --- | --- | --- |
+| M0 单副本 MVP | 一个 Core、一个共享 Plugin Runtime、一个共享 Box Runtime、一个 PostgreSQL business database | 只新增目录和业务行 | 当前目标 |
+| M1 同逻辑实例横向扩展 | 按容量增加 Core/Runtime 副本；使用 owner lease、fencing 和 generation；PostgreSQL 可增加 shared shard | 不创建 Workspace 专属部署 | 出现容量或可用性证据后 |
+| M2 Dedicated 资源等级 | 特定 workload 使用独享 worker pool、sandbox class 或 database shard，但沿用相同身份、协议和 schema | 仅购买该等级的客户承担 | 合规、驻留或超大负载需求 |
+
+M1 是 M0 的透明扩容，M2 是相同架构下的资源等级。外部 API 只认识稳定的
+`instance_uuid` 和 `workspace_uuid`，不认识 replica、worker、pool 或 shard。
+
+### 4.3 当前不做分布式时必须预留的能力
+
+1. 运行时协议携带稳定 `instance_uuid`、`workspace_uuid` 和 `execution_generation`，不依赖进程地址表达身份。
+2. Plugin installation 和 Box session 使用稳定 owner 抽象；启用第二个副本前再实现带 expiry、CAS 和 fencing token 的 lease。
+3. 创建、重试、回调、worker 注册和 outbox 使用稳定 idempotency key，重复投递不能产生第二个 owner 或副作用。
+4. Repository/UoW 不允许无边界跨 Workspace 事务；`workspace_uuid` 可直接作为未来 shard key。
+5. schema migration、后台任务扫描、监控聚合和运维接口不能假设永远只有一个 Core 进程。
+6. Runtime 重启通过 durable desired state reconciliation 恢复，不依赖原进程或本地 cache。
+7. 只有出现容量、可用性、地域或合规证据后才增加副本、lease store 或 shard router；预留协议不等于提前部署组件。
+
+### 4.4 组件边界
+
+- Core、Plugin Runtime 和 Box Runtime 必须保持独立进程身份、容器和 security context。M0 中 Core 与 Plugin Runtime
+  需要处于同一 rollout/restart unit；在实现受认证 takeover 或 owner lease/fencing 前，Core 不能单独重启后接管仍存活的 Runtime。
+- Core 不能继承 nsjail、cgroup 或 mount namespace 所需的高权限。
+- Plugin Runtime 与 Box Runtime 不合并为一个高权限进程。
+- MVP 不新增 Runtime 专用数据库、Box 专用数据库、Kafka、Redis、租户级 scheduler 或 artifact service。
+- 可信 supervisor、数据库连接池、只读 artifact cache 和基础容量可以多租户共享。
+
+## 5. OSS 与 SaaS 产品行为
+
+### 5.1 能力矩阵
+
+| 能力 | OSS | SaaS |
+| --- | --- | --- |
+| Workspace 数量 | 实例固定一个 | Account 可拥有或加入多个，受 ProductPolicy 约束 |
+| Workspace 成员 | 多用户 | 多用户，受 entitlement 约束 |
+| 邀请成员 | 支持 | 支持 |
+| 固定 RBAC | 支持 | 支持 |
+| 自定义角色 | 不支持 | 后续商业能力 |
+| Workspace 创建 | 首次初始化创建唯一 Workspace | 注册自动创建个人 Workspace；后续创建受 ProductPolicy 约束 |
+| Workspace 切换 | 无需展示 | 支持 |
+| 订阅与计费 | 无远端依赖 | 闭源 Control Plane 管理 |
+| 租户隔离 | 完整实现 | 完整实现 |
+
+OSS edition policy 应表达为：
+
+```text
+workspace_limit = 1
+members_enabled = true
+invitations_enabled = true
+fixed_rbac_enabled = true
+multi_workspace_enabled = false
+```
+
+不能用 `member_limit = 1`、关闭邀请或移除 RBAC 来实现单租户限制。
+
+### 5.2 OSS 初始化和邀请
+
+首次初始化在一个事务中完成：
+
+1. 创建本地 Account。
+2. 创建实例唯一 Workspace。
+3. 创建 owner Membership。
+4. 创建默认 Pipeline、metadata 等 Workspace 初始资源。
+5. 标记实例初始化完成。
+
+初始化后默认关闭公开注册。后续用户由 owner/admin 创建一次性 Invitation，注册或登录后接受邀请并加入唯一 Workspace。
+OSS 后续注册不创建第二个 Workspace。未配置 SMTP 时，系统返回只展示一次的邀请链接供管理员通过可信渠道发送。
+
+### 5.3 SaaS 注册和邀请
+
+普通注册由 Control Plane 通过幂等工作流完成：
+
+1. 创建或确认全局 Account 与 AuthIdentity。
+2. 创建 personal Workspace 和 owner Membership。
+3. 创建初始 Subscription/Entitlement 投影。
+4. 完成 verified email、速率限制和基础风控。
+5. 将 Account、Workspace 和 Membership 投影到 Core。
+6. Core 达到要求的目录 revision 后返回可访问 route。
+
+注册只创建逻辑记录，不启动 Runtime 或租户专属基础设施。
+
+通过邀请注册的新用户也创建自己的 personal Workspace，同时加入受邀 Workspace；已注册用户接受邀请时只新增目标 Membership。
+个人 Workspace 与团队 Workspace 的付费关系必须由 ProductPolicy 明确，不允许代码根据名称或创建路径隐式推断。
+
+### 5.4 Invitation 安全规则
+
+- token 使用至少 256-bit 加密安全随机数，数据库只保存 hash。
+- token 具有 `expires_at`、`accepted_at`、`revoked_at`，只能使用一次。
+- Membership 创建与 token 消费在同一事务中提交。
+- Invitation 不能授予 owner；owner 转移使用独立流程。
+- SaaS 接受邀请时必须验证目标邮箱；OAuth 邮箱相同不能跳过 token 和显式确认。
+- Workspace 必须始终至少有一个 active owner；admin 不能移除或降级 owner。
+- 浏览器邀请链接把 secret 放在 URL fragment 中，页面读取后立即清除 fragment，并只短期保存在 `sessionStorage`。
+
+### 5.5 固定 RBAC
+
+Core 权威定义 `owner`、`admin`、`developer`、`operator` 和 `viewer` 固定角色。
+权限按能力划分，例如资源查看、资源管理、运行操作、成员管理、provider secret 管理、审计查看和数据导出。
+
+规则：
+
+- 普通资源可见性不自动授予 secret 可见性。
+- 跨 Workspace 猜测资源 UUID 返回 404，不泄露存在性。
+- 同 Workspace 资源存在但缺少权限时返回 403。
+- 最后一个 owner 不能被删除或降级。
+- 前端隐藏或禁用无权限入口只改善体验；后端仍必须执行所有授权检查。
+
+## 6. 开源与闭源职责边界
+
+### 6.1 LangBot Core OSS
+
+Core 负责：
+
+- 本地 Account、Workspace、Membership 和 OSS Invitation。
+- 固定 RBAC 与单 Workspace edition policy。
+- 业务资源及其 Workspace scope。
+- HTTP、WebSocket、后台任务和运行时请求上下文。
+- Plugin、MCP、RAG、Box、Session、Storage 和 Monitoring 隔离。
+- SaaS Account/Workspace/Membership 的版本化执行投影。
+- InstanceManifest、EntitlementSnapshot 和 Runtime 控制通道验证。
+- 通用 capability 与数值 quota enforcement。
+- UsageEvent/business outbox 和基础安全审计。
+
+Core 是 Bot、Pipeline、Model、Knowledge、Plugin installation、MCP configuration 和 Monitoring 数据的权威来源，
+也是每个业务和运行时请求的最终授权边界。
+
+### 6.2 Closed Cloud Control Plane
+
+Control Plane 负责：
+
+- SaaS 全局 Account、AuthIdentity、Session、OIDC 和后续 SSO。
+- SaaS Workspace、Membership 和 Invitation 的权威目录。
+- Workspace 创建、暂停、归档和删除工作流。
+- BillingAccount、Product、PlanVersion、Price、Subscription、Invoice、Refund 和 provider event。
+- Entitlement 计算、签名与版本。
+- Usage ledger、聚合、额度和欠费策略。
+- 实例 manifest、release、capacity、内部 desired state 和 observed state。
+- SaaS 运营后台、平台角色和高级审计。
+
+首期不把这些职责拆成多个租户、计费和调度微服务。推荐以一个独立于 Core 的闭源模块化单体承载，
+并通过模块边界复用已有账户、OAuth、支付、邮件和运营能力。历史 Cloud 的租户专属部署代码不复用。
+
+Control Plane 不保存 Bot、Pipeline、Model 或 Knowledge 等业务内容，也不代理普通消息执行。
+
+### 6.3 SaaS Adapter
+
+Core 中只保留薄的协议适配层：
+
+- 验证 InstanceManifest、Account token 和 JWKS。
+- 消费 DirectoryEvent 并写入本地投影。
+- 缓存并验证 EntitlementSnapshot。
+- 将 UsageEvent 写入 durable outbox。
+- 接收 execution desired state 并上报 observed state。
+
+适配层不得 monkey patch ORM、绕过 Core 权限检查或在普通资源请求中同步调用 Control Plane。
+
+### 6.4 Source of Truth
+
+| 数据 | OSS | SaaS |
+| --- | --- | --- |
+| Account、Workspace、Membership | Core 本地数据库 | Control Plane 权威，Core 保存版本化投影 |
+| Invitation | Core 本地数据库 | Control Plane 权威，不向 Core 投影 pending secret |
+| Bot、Pipeline、Model、KB、Plugin、MCP | Core | Core |
+| Subscription、Payment、Invoice、Usage ledger | 无远端依赖 | Control Plane |
+| Feature 和 quota | 本地 edition policy | Control Plane 签发，Core/Runtime 验证执行 |
+| Execution generation | OSS 固定本地值 | Control Plane desired state，Core 执行 |
+| 运行时授权 | Core | Core 根据本地投影和 entitlement 执行 |
+
+SaaS 不维护两套可写目录。Control Plane 是目录权威写模型；Core 只保存带 revision 的执行投影。
+
+## 7. 控制面协议
+
+### 7.1 InstanceManifest
+
+仅设置 `system.edition=cloud`、环境变量或前端 feature flag 不得启用 SaaS 多 Workspace。
+Cloud bootstrap 必须验证由预置根信任签名的 InstanceManifest，并据此安装闭源 Workspace policy。
+
+Manifest 至少绑定：
+
+```text
+iss, aud, sub, jti, iat, nbf, exp
+instance_uuid
+release
+capabilities
+tenant_isolation_version
+execution_generation
+delegated issuers and keyset revision
+```
+
+签名错误、audience 不匹配、过期、generation 回滚或信任链缺失时必须失败关闭，不能降级为 OSS 默认 Workspace。
+
+### 7.2 DirectoryEvent 与目录新鲜度
+
+Control Plane 通过 transactional outbox 发布 Account、Workspace 和 Membership 的版本化事件。
+Core 使用 inbox 按 `event_id` 去重，以 aggregate revision 拒绝旧写，并追踪连续应用水位。启动时读取一个 PostgreSQL
+`REPEATABLE READ` 事务内生成的签名全量 snapshot；运行时先消费携带当前 high-water 的签名事件页，再只请求该页涉及的 Workspace 签名增量。
+增量响应不携带新的事件 cursor，因此即使其内容已包含并发提交的后续 revision，也不能跳过尚未消费的事件。
+
+要求：
+
+- 事件和 batch 经过实例绑定的强认证与签名。
+- 重复、乱序、延迟、断流和全量 replay 都安全。
+- 删除使用 tombstone。
+- 新实例先导入带 high watermark 的 snapshot，再消费增量。
+- 常态目录更新成本与本页发生变化的 Workspace 数量相关，不得为每个 `directory.changed` 重新读取和投影全部 Workspace。
+- 每个 Core replica 独立保存进程内消费 cursor，以确保各自的 entitlement cache 都看到事件；共享 PostgreSQL 保存投影
+  high-water mark、全量 snapshot coverage 和 inbox。同一事件被多个 replica 消费时，第二个 replica 验证已有 receipt；
+  snapshot coverage 内缺少的 receipt 可以补写，coverage 之外缺失则失败关闭。只有本地 cursor 追平签名 high-water 后才续期 ready。
+- projection 未就绪或落后于授权 lease 要求时，交互与自动化请求按策略失败关闭。
+- SaaS pending Invitation、email 和 token hash 不进入 Core 投影。
+
+MVP 可采用一个共享、原子且可恢复的 Control Plane store；未来多副本不能继续使用进程内状态承担一次性 token 或目录水位。
+
+### 7.3 EntitlementSnapshot
+
+Entitlement 使用版本化签名快照，至少绑定：
+
+```text
+instance_uuid
+workspace_uuid
+plan_revision
+entitlement_revision
+status
+features
+limits
+nbf, exp, grace_until
+```
+
+Core 校验 issuer、audience、subject、instance、revision、时间和签名；旧 revision 不覆盖新快照。
+套餐名称和价格规则只存在于闭源 Control Plane，Core 与 Runtime 只理解通用 capability 和数值限额。
+
+Control Plane 故障时，已缓存且仍有效的快照可继续执行；过期后只能进入明确、有限的 grace 模式或失败关闭。
+
+### 7.4 UsageEvent 与 outbox
+
+用量事件 append-only、至少一次投递，Control Plane 按 `event_id` 去重。事件至少包含：
+
+```text
+event_id
+instance_uuid
+workspace_uuid
+execution_generation
+meter
+quantity_integer
+unit
+source
+occurred_at
+entitlement_revision
+schema_version
+```
+
+Core 不计算账单金额，也不在普通请求中同步扣费。业务写入与相应 business outbox 必须在同一事务中提交；
+generation-aware write fence 与 outbox 原子性尚是 SaaS 激活门禁。
+
+### 7.5 Desired state 与 observed state
+
+闭源控制面发布版本化的 release、capacity 和 execution desired state，Core/Runtime 幂等 reconcile 并上报 observed state。
+desired state 只描述同一逻辑实例内部的执行所有权和容量，不产生新的产品级实例或租户实体。
+
+Workspace 安全状态由 directory revision 决定，订阅状态由 entitlement revision 决定，执行撤销由
+execution generation 决定。三者取最严格有效状态，但任何通道都不能修改另一个通道的权威字段。
+
+## 8. 身份、鉴权与请求上下文
+
+### 8.1 上下文模型
+
+租户业务入口统一解析不可变的 `RequestContext`：
+
+```python
+@dataclass(frozen=True)
+class RequestContext:
+    instance_uuid: str
+    workspace_uuid: str
+    execution_generation: int
+    principal_type: str
+    principal_uuid: str
+    permissions: frozenset[str]
+    auth_method: str
+    entitlement_revision: int | None
+    request_id: str
+```
+
+不同入口的 Workspace 来源：
+
+| 入口 | Workspace 来源 |
+| --- | --- |
+| Browser Account token | `X-Workspace-Id` 只作候选；服务端校验 Membership |
+| API Key | key 记录绑定的 Workspace，忽略 caller selector |
+| Public Bot / Webhook | Bot 或 webhook route 的可信所有权 |
+| Background job | durable payload 中的完整 scope，执行前重新验证 generation |
+| Plugin Host API | 认证控制连接和 immutable action context |
+| Box operation | 已验证 entitlement、admission grant 和 Runtime namespace |
+| System operation | 显式、最小能力的 SystemContext，禁止隐式全局上下文 |
+
+禁止从模块全局变量、进程默认 Workspace、请求 payload 或“第一个 Workspace”推断 scope。
+
+### 8.2 Account token 与 Workspace discovery
+
+- 新 JWT 使用稳定 Account UUID 作为 `sub`，并绑定 issuer、当前 `instance_uuid` audience 和 expiry。
+- 账户级 Workspace discovery 是一个窄 bootstrap capability，只列出该 Account 的 active Membership，不能执行租户业务。
+- multi-Workspace 模式下，tenant route 缺少 selector 必须拒绝；OSS singleton 模式可由 policy 选择唯一 Workspace。
+- Account token 不直接证明任一 Workspace 权限；Membership 必须在服务端解析并验证状态与 revision。
+
+### 8.3 API Key、WebSocket 与长任务
+
+- API Key 只持久化 hash，raw secret 仅返回一次；记录绑定 Workspace、固定 scopes、状态、expiry 和 creator。
+- Dashboard WebSocket 在升级后认证，并在每条入站消息前重新验证 Account、Membership、权限、资源所有权和 generation。
+- 长时间 LLM、MCP、Plugin 或 Box 调用在产生副作用或接受结果前再次校验 execution generation。
+- 临时凭证交换绑定发起者、Workspace、instance 和 generation；其他 scope 查询返回与不存在相同的 404。
+
+### 8.4 错误语义
+
+| 场景 | 语义 |
+| --- | --- |
+| 未认证或 token 无效 | 401 |
+| 同 Workspace 资源存在但权限不足 | 403 |
+| 资源不存在或属于其他 Workspace | 404 |
+| edition / entitlement / quota 禁止 | 稳定领域错误码，不伪装为 500 |
+| execution generation 过期 | fail closed，并停止旧运行态 |
+| 未处理异常 | 稳定 `internal_error` + request ID；细节只进入服务端日志 |
+
+## 9. Core 数据模型
+
+### 9.1 Account、Workspace 与 Membership
+
+核心实体至少包含：
+
+```text
+Account
+  uuid
+  email_normalized
+  display_name
+  status
+  auth bindings
+
+Workspace
+  uuid
+  name
+  status
+  source: local | cloud_projection
+  directory_revision
+
+WorkspaceExecutionState
+  workspace_uuid
+  instance_uuid
+  execution_generation
+  status
+  write_fenced_at
+  revision
+
+WorkspaceMembership
+  workspace_uuid
+  account_uuid
+  role
+  status
+  directory_revision
+```
+
+约束：
+
+- Membership 对 `(workspace_uuid, account_uuid)` 唯一。
+- Workspace 的 source 不允许通过可变本地配置从 local 升级成 cloud projection。
+- Cloud projection 只有在 manifest、instance binding、目录 revision 和 execution state 均有效时才可路由。
+- OSS bootstrap 只创建或修复 local singleton Workspace。
+
+### 9.2 Invitation
+
+OSS Invitation 存在 Core 本地数据库；SaaS Invitation 只存在于闭源目录。
+
+```text
+WorkspaceInvitation
+  uuid
+  workspace_uuid
+  email_normalized
+  role
+  token_hash
+  expires_at
+  accepted_at
+  revoked_at
+  created_by
+```
+
+数据库约束必须保证同一 Workspace 与邮箱只有一个有效邀请，并保证 token hash 全局唯一。
+
+### 9.3 业务资源
+
+所有租户资源显式包含 `workspace_uuid`，包括但不限于：
+
+- Bot、Pipeline、Provider、Model、Knowledge Base 和 vector record。
+- Plugin installation、MCP configuration、API Key 和 webhook binding。
+- Query、Message、Session、Monitoring、Usage 和 AuditEvent。
+- Upload、ObjectRef、Skill、Runtime desired state 和 temporary credential session。
+
+唯一键、索引、缓存 key、object key、日志维度和幂等键都必须包含 Workspace scope。
+服务层不得暴露可绕过 Workspace 条件的普通 `get(id)`、`list()` 或 `delete(id)`。
+
+### 9.4 防御性约束
+
+- tenant table 的 `workspace_uuid` 非空并有外键。
+- SaaS PostgreSQL 关键表启用并强制 RLS。
+- 需要全局唯一的 opaque token 使用 hash 唯一索引，不依赖 Workspace 内唯一。
+- owner 保底、Membership revision、invitation one-shot 等规则同时由 service 和数据库事务保护。
+- 任何跨 Workspace 运维操作必须走显式受审计的 system capability，不得复用普通 repository。
+
+## 10. PostgreSQL、pgvector 与存储
+
+### 10.1 数据库边界
+
+- OSS 继续默认 SQLite，并可显式选择自托管 PostgreSQL。
+- SaaS 使用一个 PostgreSQL business database、一个 `public` shared schema 和共享连接池。
+- 创建 Workspace 不创建 database、schema、role 或专属连接池。
+- 每个 tenant transaction 使用 `SET LOCAL` 建立 scope，并由统一 TenantUnitOfWork 保证 context 与 SQL 使用同一事务和连接。
+- 应用层 Workspace scope 是第一道边界，`ENABLE` + `FORCE ROW LEVEL SECURITY` 是第二道边界。
+- runtime role 必须是非 owner、最小权限、无 superuser、无 `BYPASSRLS`、无 role membership 和跨 schema 权限。
+- schema、extension、policy 和 ACL 只由独立 release migrator 创建与验证；Cloud runtime 不执行 DDL。
+- PostgreSQL 仅承载业务数据和 pgvector，不成为 Plugin/Box 通用协调数据库、进程目录或新的控制面数据库。
+
+首期 migrator 和 runtime URL 必须连接同一个 host、port、database，但使用不同 role。
+生产部署还必须证明 runtime credential 无法连接 PostgreSQL 集群中的其他 database；专用 endpoint 或经验证的 HBA/proxy 隔离仍是激活门禁。
+
+### 10.2 Transaction 与后台任务
+
+- 一个 TenantUnitOfWork 只绑定一个 Workspace、一个 execution generation 和一个事务所有者任务。
+- 子任务不能继承并提交、回滚或关闭父任务的 tenant session。
+- 长时间 LLM 或网络等待不持有数据库连接；每次数据库 helper 打开短事务。
+- detached task 只在父事务提交后启动，并自行建立新 scope；父事务回滚时取消待启动任务。
+- generation-aware write fence 必须保持到 commit，并与 business outbox 原子提交；该能力完成前不得激活 SaaS 写流量。
+
+### 10.3 pgvector
+
+- SaaS 默认使用同一业务 PostgreSQL 中的 pgvector，不静默回退到 Chroma。
+- 向量身份至少为 `(workspace_uuid, knowledge_base_uuid, vector_id)`。
+- 向量操作使用相同 tenant context 与 RLS 契约。
+- embedding 维度显式存储和校验；不匹配时失败关闭，不截断、补齐或改用无界扫描。
+- extension、表、constraint 和 ANN index 由 release migration 创建。
+- OSS 默认仍可使用 SQLite + Chroma；选择 pgvector 时遵守相同 scope。
+
+### 10.4 Object storage
+
+- 大对象、plugin artifact、upload、knowledge 文件和 sandbox 文件不作为 PostgreSQL blob 存储。
+- durable object key 和 metadata 都包含 Workspace scope；临时 staging 可包含 generation，但稳定业务引用不能因未来 generation 切换而永久失效。
+- 现有 generation-scoped opaque key 在固定 generation 的 OSS 中安全，但 Cloud cutover 前必须实现稳定 final identity 或原子引用迁移。
+- public image 与 private document 使用不同 capability；不能把通用 upload key 当作公开读取凭证。
+
+## 11. Plugin Runtime
+
+### 11.1 共享 supervisor、独立 worker
+
+整个逻辑实例共享一个可信 Plugin Runtime 逻辑控制面；M0 由一个 supervisor replica 承担。新 Workspace 不创建专属 Runtime、连接、卷或进程。
+
+每个运行中的 plugin installation 独占一个 nsjail worker process tree；enabled-resident 是 desired semantics。worker 运行期间永久绑定：
+
+```text
+instance_uuid
+workspace_uuid
+execution_generation
+installation_uuid
+runtime_revision
+artifact_digest
+```
+
+插件不能通过 payload、Host API 参数、环境变量或重连改变该绑定。Supervisor 不在自身解释器中加载第三方插件代码。
+停用、删除、revision/generation 变化或 entitlement 撤销时，旧 worker 必须停止并失去 Host API 权限。
+
+### 11.2 文件和进程边界
+
+```text
+data/plugin-runtime/
+├── artifacts/sha256/<artifact_digest>/code/   # 已验证、只读共享
+├── environments/sha256/<environment_digest>/ # 原子发布、只读共享
+└── installations/<installation_uuid>/
+    ├── home/                                  # 私有可写
+    ├── tmp/                                   # 私有可写
+    └── data/                                  # 私有持久数据
+```
+
+- 同插件同版本只有在 package digest 完全相同且完整性已验证时才共享只读代码。
+- dependency environment key 包含 artifact/requirements digest、Python ABI、Runtime version 和 installer schema。
+- installation 进程、配置、secret、home、tmp、data 和日志永不合并。
+- namespace、private `/proc`、mount、PID、IPC、UTS、cgroup 与 rlimit 阻止读取其他文件、枚举或 signal 其他进程。
+- Cloud 不从 artifact 自动加载 `.env`；secret 只由可信控制面按 installation 注入。
+- 插件 egress 必须阻止访问 Core loopback、Box Runtime、数据库和平台 metadata endpoint。
+
+### 11.3 统一资源上限
+
+资源限制只来自实例级 `data/config.yaml`，并支持现有环境变量覆写；plugin manifest 不能声明、放宽或覆盖。
+
+```yaml
+plugin:
+  worker:
+    max_cpus: 1.0
+    max_memory_mb: 512
+    max_pids: 128
+    max_open_files: 256
+    max_file_size_mb: 512
+    require_hard_limits: true
+```
+
+CPU、内存和 PID 使用 cgroup 硬限制，open files 和单文件大小使用 rlimit。
+Cloud deployment profile 强制 nsjail；硬限制不可用时 readiness 失败，不能降级为普通子进程。
+installation 总磁盘配额需要可原子拒绝写入的 quota provider，不能以目录扫描冒充硬限制。
+
+### 11.4 Desired state 与恢复
+
+- PostgreSQL 中的 installation desired state 与 durable binary storage 是权威状态。
+- Runtime 本地进程表、nsjail 目录、artifact/venv cache 都可重建。
+- Runtime 重连执行实例范围 full reconciliation，清理 stale worker 并恢复 enabled installation。
+- dependency preparation 失败记录在对应 installation，不启动半就绪 worker，也不阻塞其他 installation。
+- desired semantics 要求 enabled installation 常驻，不做 idle eviction；是否按负载回收以后再决定。
+- 当前 Supervisor 已在意外退出时通过 completion callback 和有界指数 backoff 恢复 enabled worker。
+  Cloud 激活前仍需加入 jitter、全局重启并发上限和 Runtime 级 circuit breaker，并验证系统性故障不会形成跨租户重启风暴。
+
+真实 Linux/nsjail/cgroup 与受控 egress 的 Cloud 部署验证尚未完成，是生产激活门禁。
+
+## 12. Box Runtime 与 stdio MCP
+
+### 12.1 共享 Box 控制面
+
+整个逻辑实例共享一个可信 Box Runtime 逻辑控制面；M0 由一个 Runtime replica 承担。Core 与 Runtime 控制通道绑定稳定 instance identity，
+每个 operation 绑定 `workspace_uuid`、`execution_generation`、session revision 和短期 admission grant。
+
+首期 entitlement 模型：
+
+```json
+{
+  "features": {
+    "managed_sandbox": true,
+    "external_sandbox": false
+  },
+  "limits": {
+    "managed_sandbox_sessions": 1
+  }
+}
+```
+
+闭源订阅模块把套餐映射为该通用 capability；Core 与 Runtime 不判断 `plan == pro`。
+预期 Pro 得到 `managed_sandbox_sessions = 1`，其他套餐为 `0`。
+
+### 12.2 Sandbox 模型
+
+- 合资格 Workspace 首次使用时懒创建一个持久 `global` 逻辑 session。
+- `global` 表示 Workspace 内默认逻辑 sandbox，不表示跨 Workspace 共享。
+- session TTL 不自动回收；Runtime 重启后进程和临时目录失效，但 `/workspace` 持久数据保留。
+- 每次普通命令在 Box Runtime 容器内启动一个 one-shot nsjail 子进程。
+- 首期禁止 managed background process 和 network，避免 session 被当成常驻共享主机。
+- Core 与 Runtime 通过认证 random-marker challenge 证明看到同一 durable volume，不能只比较路径字符串。
+- 文件同步、attachment 和 skill mount 沿用现有 nsjail 机制，但所有 host path 解析必须由可信 Workspace context 派生并防止 symlink/path escape。
+
+Cloud readiness 必须证明 cgroup、namespace、mount、Workspace/Skill/ephemeral byte quota 和 inode quota 均为硬限制。
+当前普通 nsjail backend 不具备全部硬磁盘能力，因此 Cloud Box 应失败关闭，直到绿地部署提供并验证真实 quota provider；
+不能把软目录扫描写成“生产已就绪”。
+
+### 12.3 外部 E2B
+
+非 Pro 用户后续可在 WebUI 配置 Workspace 自有的远程 E2B sandbox。该功能尚未实现，首期不纳入。
+未来 credential 必须属于 Workspace、加密存储且读取受 secret 权限保护，不消耗 Cloud managed sandbox 配额。
+
+### 12.4 stdio MCP 独立开关
+
+```yaml
+mcp:
+  stdio:
+    enabled: true
+```
+
+- OSS 默认 `true` 保持兼容。
+- Cloud v2 通过 `MCP__STDIO__ENABLED=false` 强制关闭。
+- 该 gate 独立于 `box.enabled`、managed sandbox entitlement 和 session quota。
+- gate 同时覆盖 create、update、test、bootstrap load 和最终 Runtime execution。
+- 已有 stdio 配置在 gate 关闭时保留但不启动，并返回明确的 feature-disabled 错误。
+- HTTP/SSE 等远程 MCP transport 不受影响。
+
+## 13. HTTP API 与 WebUI
+
+### 13.1 Core API
+
+OSS 与 SaaS 执行面共用通用 Workspace API：
+
+```text
+GET    /api/v1/workspaces
+GET    /api/v1/workspaces/{workspace_uuid}
+GET    /api/v1/workspaces/{workspace_uuid}/members
+POST   /api/v1/workspaces/{workspace_uuid}/invitations
+PATCH  /api/v1/workspaces/{workspace_uuid}/members/{account_uuid}
+DELETE /api/v1/workspaces/{workspace_uuid}/members/{account_uuid}
+```
+
+Cloud policy 下，目录 mutation 由闭源 Control Plane 负责；Core 对本地创建、邀请和成员修改返回稳定的
+`control_plane_required`，只提供执行投影的安全读取。
+
+所有 tenant resource route 必须经过统一 decorator/middleware：
+
+1. 认证 principal。
+2. 解析可信 Workspace。
+3. 校验 Workspace/ExecutionState。
+4. 校验 Membership 或资源绑定。
+5. 校验 permission 和 entitlement。
+6. 创建 RequestContext 与 TenantUnitOfWork。
+
+### 13.2 SaaS Control Plane API
+
+SaaS 产品 API 包含：
+
+```text
+POST /cloud/workspaces
+GET  /cloud/workspaces
+POST /cloud/workspaces/{workspace_uuid}/invitations
+POST /cloud/invitations/{token}/accept
+GET  /cloud/workspaces/{workspace_uuid}/subscription
+POST /cloud/workspaces/{workspace_uuid}/checkout
+GET  /cloud/workspaces/{workspace_uuid}/usage
+```
+
+这些 API 管理目录、产品和计费，不直接操作 Bot/Pipeline 等 Core 业务资源。
+
+### 13.3 WebUI
+
+OSS：
+
+- 首次注册进入唯一 Workspace。
+- owner/admin 可邀请成员并管理固定角色。
+- 不展示 Workspace 切换器和创建第二 Workspace 的入口。
+
+SaaS：
+
+- 登录先获取 Account 级 Workspace 列表，再显式选择当前 Workspace。
+- 当前 Workspace UUID 保存在受控客户端状态中；所有 tenant request 自动附带 selector。
+- 切换 Account 或 Workspace 时清理缓存、WebSocket、上传、表单、错误和 optimistic state，不能显示前一租户数据。
+- 页面 refresh、新 tab 和邀请跳转恢复同一个经过授权的 Workspace；失效 Membership 不回退到其他 Workspace。
+- UI 权限变化必须响应式更新，但 API 仍是最终授权边界。
+
+## 14. 故障、安全与降级
+
+### 14.1 Fail-closed 场景
+
+以下情况必须拒绝新的租户业务和副作用：
+
+- Cloud manifest 缺失、签名失败、audience 错误或回滚。
+- Account token、Membership、Workspace status 或 execution generation 无效。
+- 目录投影未就绪或落后于有效 lease 要求。
+- Entitlement 缺失、过期且不在明确 grace 范围内。
+- Runtime 控制通道认证失败或实例绑定不一致。
+- Plugin nsjail/cgroup hard limit 在 Cloud profile 下不可用。
+- Box 的任一硬存储或 namespace capability 无法证明。
+- PostgreSQL RLS、runtime role、schema、catalog 或 endpoint 隔离校验失败。
+- stdio MCP 在 Cloud profile 下被尝试启用。
+
+不能把上述错误静默降级为 OSS singleton、普通子进程、Chroma、软 quota 或 caller-supplied Workspace。
+
+### 14.2 撤销语义
+
+- Membership 删除或降权必须影响下一次 HTTP 请求，并使长连接在下一条消息前重新授权。
+- Workspace 暂停禁止新交互、自动化工作负载和新副作用；恢复只允许当前 generation。
+- entitlement 到期按 capability 明确停止新创建或新执行，不隐式删除已有数据。
+- generation 变化使旧 worker、session、callback、cached runtime object 和 outbox publisher 失效。
+- 控制面暂时不可达时，只能在有效签名快照和本地投影允许的范围内继续；过期后失败关闭。
+
+### 14.3 安全清单
+
+- 所有 identifier 使用不可猜 UUID，但不把随机性当成授权。
+- 所有 token/secret 只存 hash 或加密值，raw secret 一次展示。
+- 日志、trace、metric、cache 和 object key 都包含 Workspace 维度并过滤 secret。
+- Provider、Bot、Plugin、MCP 配置的 read response 递归遮蔽 credential。
+- Runtime control、debug、registration 和 attachment capability 分离，不能复用万能 secret。
+- untrusted code 不访问 Core loopback、数据库、其他 Runtime、宿主文件系统或 metadata endpoint。
+- bulk operation、后台扫描和 monitoring 聚合使用显式 tenant/system capability。
+- 所有跨 Workspace 运维操作记录 principal、reason、scope、request ID 和结果。
+
+## 15. 实现状态与 SaaS 激活门禁
+
+### 15.1 已实现的隔离内核
+
+当前分支已经实现或具备基础的部分包括：
+
+- OSS singleton Workspace、多 Account、Invitation 和固定 RBAC。
+- trusted RequestContext、Workspace-scoped repository 和资源所有权检查。
+- tenant-aware Plugin SDK protocol 与 Runtime installation binding。
+- shared Plugin Runtime / Box Runtime 控制协议和 execution generation fence。
+- stdio MCP 独立 gate。
+- PostgreSQL shared schema、transaction-local scope、FORCE RLS 与 pgvector adapter。
+- Cloud bootstrap 默认不可由普通配置激活，并对缺失安全能力失败关闭。
+
+这些是代码能力边界，不等于完成闭源 SaaS 产品或生产部署验收。
+
+### 15.2 尚未完成的激活门禁
+
+以下事项完成并取得真实环境证据前，不得宣称 Cloud v2 production-ready：
+
+1. 闭源 Control Plane 的全局目录、注册、邀请、订阅、计费、entitlement 签发和签名 manifest bootstrap；横向扩展前 OAuth exchange 与目录投影还必须使用原子共享存储。
+2. 普通业务写入贯穿 commit 的 generation-aware fence，以及与外部副作用同事务的 business outbox。
+3. generation cutover 后稳定的 durable object identity 或原子对象引用迁移。
+4. 所有 tenant-configurable outbound URL 的 SSRF 防护与 tenant-safe egress；Plugin Runtime 还需在真实 Linux/nsjail/cgroup v2 环境验证 namespace、资源限制和文件隔离。
+5. Plugin Runtime 已实现意外退出 worker 的 completion callback、有界 backoff 和自动恢复；Cloud 激活前增加全局重启风暴抑制并完成故障注入验证。
+6. Plugin installation data 的 production hard disk quota provider，能够在写入边界原子拒绝超额，不能以目录扫描代替。
+7. Box Runtime 的 production hard quota provider，包括 Workspace、Skill、root/tmp/home 的 byte 与 inode quota；真实部署还必须在启动和重连时通过共享卷 marker challenge。
+8. PostgreSQL runtime credential 的专用 endpoint 或 HBA/proxy 跨 database 隔离证明、生产 migration/rollback 流程，以及 legacy pgvector migration 失败后精确恢复 RLS/FORCE 并可安全重试的集成证据。
+9. 闭源目录事件、lease、snapshot、entitlement 和 usage/outbox 的重放、断流与灾难恢复验证。
+10. 真实浏览器多 Account/RBAC/邀请/刷新场景已完成；仍需生产 Runtime 重启、worker crash、断流、异常回滚和闭源 Control Plane 的 fault-injection 验收。
+
+### 15.3 有意暂缓的产品决策
+
+- Workspace 创建后的休眠、释放、删除和保留策略。
+- Workspace export 与单 Workspace restore。
+- 非 Pro Workspace 的 BYOK E2B WebUI。
+- 多副本 owner lease 的 store、TTL、fencing token 和转移顺序。
+- PostgreSQL shard resolver、在线迁移和 dedicated shard 产品规则。
+- artifact/cache 的签名来源、撤销、GC 和磁盘配额机制。
+- custom roles、SSO、SCIM 和企业合规能力。
+
+暂缓项不得被实现代码用隐式默认值提前固化。
+
+## 16. 实施顺序
+
+### Phase 0：契约和基线
+
+- 固定术语、RequestContext、角色矩阵、edition policy 和错误语义。
+- 建立升级备份、回滚和跨租户负向测试基线。
+
+### Phase 1：OSS tenancy kernel
+
+- Account、Workspace、Membership、Invitation。
+- singleton bootstrap、多用户邀请、RBAC 和前端权限。
+
+### Phase 2：数据与入口隔离
+
+- 为所有资源补充 Workspace scope。
+- HTTP、API Key、Bot、Webhook、WebSocket、后台任务和 storage 统一上下文。
+- SQLite migration recovery 与 PostgreSQL RLS 集成测试。
+
+### Phase 3：Runtime 与 SDK 隔离
+
+- Plugin installation binding、nsjail、资源上限和 artifact replay。
+- Box admission、session namespace、skill/attachment 文件边界。
+- MCP gate、RAG/vector 与 long-running generation revalidation。
+
+### Phase 4：闭源 SaaS 控制面
+
+- signed manifest bootstrap。
+- 全局目录、注册、邀请、Subscription、Entitlement 和 Usage ledger。
+- projection、lease、outbox、reconciliation 和运维后台。
+
+### Phase 5：生产部署激活
+
+- 真实 Linux Plugin/Box hard isolation。
+- PostgreSQL credential、migration、backup 和 rollback 验证。
+- 完整浏览器/API/Runtime E2E 和故障注入。
+- 所有激活门禁通过后才开启多 Workspace Cloud policy。
+
+### Phase 6：同逻辑实例内部扩展
+
+- 有容量证据后增加副本、owner lease 和 fencing。
+- 有地域、合规或规模证据后增加 shared/dedicated shard。
+- 保持外部身份、API 和 Workspace URL 不变。
+
+## 17. 测试与验收
+
+### 17.1 数据隔离
+
+- 两个 Workspace 使用相同 resource UUID、name、vector ID 和 cache key，不发生冲突或越权。
+- 故意遗漏应用层 Workspace filter 时，PostgreSQL RLS 仍阻止跨租户读写。
+- 连接池复用、异常回滚、子任务、后台任务和 transaction pooling 不残留 tenant context。
+- 跨 Workspace 猜测返回 404；同租户缺权限返回 403。
+
+### 17.2 产品行为
+
+- OSS 首个 Account 创建唯一 Workspace；第二个 Account 只能通过邀请加入；创建第二 Workspace 返回 edition error。
+- 邀请覆盖有效、已使用、撤销、过期、邮箱不匹配和并发接受。
+- owner/admin/developer/operator/viewer 的 API 和 WebUI 权限一致。
+- SaaS 普通注册和邀请注册都创建个人 Workspace，但不创建专属部署或 Runtime。
+
+### 17.3 Runtime
+
+- 两个 Workspace 安装同一已验证 artifact 时只共享只读 code/env，进程、secret、home/tmp/data、日志和 Host API 完全隔离。
+- cgroup、rlimit、namespace、egress 和 generation fence 在真实 Linux 环境生效。
+- Runtime restart/cache loss 通过 durable desired state 与 binary storage 恢复。
+- 两个 Workspace 的 Box session、files、process、skill、attachment 和 quota 完全隔离。
+- stdio MCP gate 对 UI、API、bootstrap 和最终 execution 同时生效。
+
+### 17.4 Control Plane 与故障
+
+- DirectoryEvent 重复、乱序、缺口、snapshot + replay 和过期 lease 均安全。
+- Entitlement 旧 revision、签名错误、过期和撤销均失败关闭。
+- UsageEvent 重放不重复计费；业务事务回滚不发送副作用。
+- Runtime、Core 或 Control Plane 重启不创建重复 Workspace、worker 或 sandbox。
+- manifest、数据库安全校验或 hard quota 缺失时实例保持不可激活，而不是静默降级。
+
+### 17.5 浏览器端到端
+
+真实浏览器至少覆盖：
+
+1. clean database 首位 owner 注册与 singleton Workspace bootstrap。
+2. owner 创建邀请，第二个用户注册/登录并接受。
+3. 角色在 viewer/operator/developer/admin 间变化时，导航、控制项和 API 结果同步变化。
+4. Account/Workspace 切换清空前一 scope 状态，refresh 和新 tab 恢复正确 Workspace。
+5. 第二 Workspace edition limit，以及 invitation used/revoked/expired/email mismatch 的可见错误。
+6. 直接 API 越权、伪造 selector 和跨租户 UUID 猜测不能绕过 UI。
+
+## 18. 最终结论
+
+Cloud v2 的产品模型只有一个逻辑 LangBot 实例和实例内多个 Workspace。
+当前选择单副本 MVP 是为了减少组件和新增租户成本，不是把单进程假设写进业务身份或协议。
+未来需要容量或高可用时，在同一逻辑实例内部增加 Core/Runtime 副本和 PostgreSQL shard，
+Workspace 的 UUID、权限、数据边界和外部 API 均保持不变。
+
+开源 Core 必须完整实现安全的 Workspace 隔离和 OSS 单 Workspace 多用户；闭源 Control Plane
+管理 SaaS 的全局目录、订阅、权益、计费和生命周期。共享可信控制面、连接池、只读 artifact 和数据库组件，
+同时让每个不可信插件进程、sandbox、secret、可写文件和 tenant transaction 保持独占边界，
+才能在不增加每租户部署的前提下最大化降低新增用户成本。
+
+在闭源控制面、事务 fence/outbox、真实 Runtime hard isolation、Box hard quota 和 PostgreSQL 生产隔离等门禁完成之前，
+本架构仍处于隔离内核阶段，不应被描述为可上线的 SaaS 多租户部署。

docs/review/mcp-resources-pr-2215-review.md

@@ -0,0 +1,196 @@
+# MCP Resources PR #2215 Review
+
+> 更新日期: 2026-06-29
+> 分支: `mcp_resources`
+> PR: langbot-app/LangBot#2215
+> 主题: MCP Resources 在 LangBot 中的产品价值、AgentRunner 集成方式与后续架构方向
+
+## 结论
+
+PR #2215 对 LangBot 有明确价值：它补齐了 MCP 协议中 Resources 这一重要能力，让 MCP server 不再只暴露 tools，也可以暴露文档、代码片段、配置、日志、图片等上下文资源。管理端可以发现和预览资源，Agent 也可以通过当前实现按需列出和读取资源。
+
+但当前 AgentRunner 层的接入方式更接近一个可用的第一阶段方案，而不是最终架构。现在 MCP Resources 被包装成两个 synthetic tools：
+
+- `langbot_mcp_list_resources`
+- `langbot_mcp_read_resource`
+
+这让模型可以通过 function calling 主动探索资源，落地成本低，也复用了已有 `ToolManager` / `LocalAgentRunner` 的工具调用链路。不过从 MCP 规范和主流实现来看，Resources 更适合作为一种一等上下文来源，而不是长期隐藏在工具列表里。
+
+建议保留当前 synthetic tools 作为探索能力，同时把后续主线设计调整为：MCP Resources 是 pipeline / conversation / message 级别可选择、可固定、可审计的上下文输入。
+
+## 当前实现判断
+
+当前 AgentRunner 集成路径如下：
+
+```text
+Pipeline 绑定 MCP server
+  -> query.variables['_pipeline_bound_mcp_servers']
+  -> Preproc 为 local-agent 加载工具
+  -> ToolManager.get_all_tools()
+  -> MCPLoader 注入 synthetic resource tools
+  -> LocalAgentRunner 将工具 schema 传给模型
+  -> 模型发起 list/read tool call
+  -> ToolManager.execute_func_call()
+  -> MCPLoader 调 MCP session.list_resources/read_resource
+  -> tool result 回灌给模型
+```
+
+这个路径的优点是：
+
+- 复用现有工具调用机制，改动范围小。
+- Agent 可以按需探索资源，不需要每轮预先读取所有资源。
+- 可以沿用 pipeline 绑定的 MCP server 范围，避免越权读取未绑定 server。
+- 对已有 MCP tools 行为影响较小。
+
+主要问题是：
+
+- Resources 在语义上被降级成 tools，和 MCP 规范里的 resource primitive 不完全一致。
+- 模型必须先理解并主动调用 `list/read`，资源不会自然成为上下文。
+- pipeline 不能配置“默认携带某些资源”或“本轮附加某些资源”。
+- UI 资源 tab 目前是管理端预览能力，和 Agent 上下文选择没有打通。
+- 对 blob、图片、大文件、结构化资源的处理还比较粗糙。
+- 缺少 resource templates、订阅更新、缓存、chunk、token budget、trace 与审计策略。
+
+## 主流项目做法
+
+### MCP 官方规范
+
+MCP Resources 是 server 暴露上下文数据的协议能力。规范没有要求 resources 必须以 tool call 形式给模型使用，而是把如何选择、过滤、读取和纳入上下文交给 Host application。
+
+这意味着比较正统的集成方式是：LangBot 作为 Host，在 pipeline、会话或消息层决定哪些 resources 进入模型上下文。
+
+参考: https://modelcontextprotocol.io/specification/2025-06-18/server/resources
+
+### VS Code Copilot
+
+VS Code 把 MCP Resources 做成 chat context 的一部分。用户可以通过 `Add Context > MCP Resources` 或命令浏览 MCP resources，并把选中的资源附加到一次 chat request。
+
+这是目前最值得 LangBot 参考的产品形态：资源不是模型工具，而是用户和 Host 可控的上下文附件。
+
+参考: https://code.visualstudio.com/docs/agent-customization/mcp-servers
+
+### Anthropic SDK
+
+Anthropic 的 client-side MCP helpers 提供资源读取和转换能力，例如把 MCP resource 转为 Claude message content 或 file。也就是说，应用先读取 resource，再显式放进模型消息。
+
+这同样是 application-owned context injection，而不是把 resource 伪装成模型工具。
+
+参考: https://platform.claude.com/docs/en/agents-and-tools/mcp-connector
+
+### LangChain MCP Adapters
+
+LangChain 把 MCP Resources 更像 data loader / document input 来处理，可以把资源加载成 `Blob`，再进入 LangChain 的文档、检索或上下文处理链路。
+
+这说明 Resources 很适合作为知识源、文档源或上下文源，而不只是即时工具调用。
+
+参考: https://docs.langchain.com/oss/python/langchain/mcp
+
+### OpenAI Agents SDK
+
+OpenAI Agents SDK 主路径仍偏向 MCP tools，但底层 MCP server API 已经有 `list_resources`、`list_resource_templates`、`read_resource` 等能力。当前形态说明 resources 是 client 能力，但并未默认变成 agent-visible tools。
+
+参考: https://openai.github.io/openai-agents-python/mcp/
+
+### Cline
+
+Cline 会拉取 MCP tools、resources、resourceTemplates、prompts，并通过类似 `access_mcp_resource` 的内置访问方式让模型读取资源。这个方向和 LangBot 当前 synthetic tools 比较接近。
+
+这种模式适合让 Agent 自主探索，但更像 Host 自定义的模型访问协议，不应成为唯一集成路径。
+
+参考: https://github.com/cline/cline/blob/main/src/services/mcp/McpHub.ts
+
+## 建议架构方向
+
+### 1. 保留探索型工具
+
+保留当前两个 synthetic tools：
+
+- `langbot_mcp_list_resources`
+- `langbot_mcp_read_resource`
+
+它们适合处理“用户没有显式选择资源，但 Agent 判断需要探索 MCP server 上下文”的场景。后续可以优化工具描述、返回格式、资源大小限制和错误信息。
+
+### 2. 增加一等 Resource Context
+
+新增一个 Host 层资源上下文概念，例如：
+
+```text
+PipelineResourceBinding
+ConversationResourceAttachment
+MessageResourceAttachment
+```
+
+Preproc 或独立的 `ResourceContextProvider` 在模型调用前读取这些资源，按 MIME 类型、大小、token budget 转为模型可消费的上下文。
+
+### 3. 打通 UI 与 Agent 上下文
+
+当前 MCP 详情页的 Resources tab 可以继续作为资源发现和预览入口。建议增加操作：
+
+- 添加到本轮上下文
+- 固定到当前 pipeline
+- 固定到当前 bot / conversation
+- 查看资源读取历史和错误
+
+这样 UI 资源管理能力才能真正影响 Agent 行为。
+
+### 4. 支持 resource templates
+
+MCP resource templates 允许 server 暴露参数化资源，例如：
+
+```text
+repo://{owner}/{repo}/file/{path}
+log://{service}/{date}
+```
+
+LangBot 后续应支持模板发现、参数填写、实例化和绑定。否则只能使用静态 resources，覆盖面会受限。
+
+### 5. 增加资源处理策略
+
+建议补齐：
+
+- 文本资源 token budget 与截断策略。
+- 大文件 chunk 与摘要策略。
+- 图片/blob 的模型能力判断与 fallback。
+- MIME 类型白名单与安全限制。
+- 缓存与过期策略。
+- `resources/listChanged` 或订阅更新。
+- resource read trace，便于审计 Agent 读取了什么上下文。
+
+## 推荐落地顺序
+
+### Phase 1: 完成当前 PR 可用性
+
+- 保留 synthetic tools。
+- 明确文档说明当前 Agent 集成是 tool-mediated。
+- 完善资源工具描述，降低模型误用概率。
+- 给 read/list 增加大小限制和更清晰的 MIME 处理。
+- 前端 Resources tab 与 Tools tab 分离，保持管理端清晰。
+
+### Phase 2: 做 Host-owned context attachments
+
+- 在 pipeline 或 conversation 层新增 resource attachment 配置。
+- Preproc 读取已绑定 resources，注入模型上下文。
+- UI 支持“添加到上下文 / 固定到 pipeline”。
+- 记录每轮实际注入的 resource URI 和 token 消耗。
+
+### Phase 3: 做完整 MCP Resources 能力
+
+- 支持 resource templates。
+- 支持资源订阅更新。
+- 支持 chunk、summary、RAG 化接入。
+- 为 DifyAgentRunner、LocalAgentRunner 等不同 runner 定义统一资源上下文接口。
+
+## 最终建议
+
+PR #2215 可以作为 MCP Resources 的第一阶段实现继续推进。它让 LangBot 快速拥有“资源发现、预览、按需读取”的闭环，也给 Agent 探索资源提供了可运行路径。
+
+但在正式设计上，不建议把 “Resources == Tools” 固化为长期抽象。LangBot 更应该把 MCP Resources 定位为上下文来源，与 tools、prompts、knowledge base 并列：
+
+```text
+Tools      -> Agent 可以执行的动作
+Resources  -> Host/用户/Agent 可以选择的上下文数据
+Prompts    -> 可复用的任务模板
+Knowledge  -> 可检索、可索引的长期知识
+```
+
+这样既尊重 MCP 协议语义，也能让 LangBot 在 Agent 工作流、企业知识接入和多 MCP server 管理上走得更稳。

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "langbot"
-version = "4.10.4"
+version = "4.10.6"
 description = "Production-grade platform for building agentic IM bots"
 readme = "README.md"
 license-files = ["LICENSE"]
@@ -39,6 +39,7 @@ dependencies = [
     "quart>=0.20.0",
     "quart-cors>=0.8.0",
     "requests>=2.33.0",
+    "regex>=2026.1.15",
     "slack-sdk>=3.35.0",
     "alembic>=1.15.0",
     "sqlalchemy[asyncio]>=2.0.40",
@@ -70,7 +71,7 @@ dependencies = [
     "chromadb>=1.0.0,<2.0.0",
     "qdrant-client (>=1.15.1,<2.0.0)",
     "pyseekdb==1.1.0.post3",
-    "langbot-plugin==0.4.6",
+    "langbot-plugin @ git+https://github.com/langbot-app/langbot-plugin-sdk.git@1d65ed301a6afc52150a998043f73cd6032c8162",
     "asyncpg>=0.30.0",
     "line-bot-sdk>=3.19.0",
     "matrix-nio>=0.25.2",
@@ -80,6 +81,7 @@ dependencies = [
     "pgvector>=0.4.1",
     "botocore>=1.42.39",
     "litellm>=1.0.0",
+    "valkey-glide>=2.4.1,<3.0.0; sys_platform != 'win32'", # No Windows wheels are published
 ]
 keywords = [
     "bot",

pytest.ini

@@ -13,6 +13,12 @@ testpaths = tests
 # Asyncio configuration
 asyncio_mode = auto
 
+# Resource leaks are often reported during object finalization and wrapped by
+# pytest. Keep both forms fatal so --disable-warnings cannot hide them.
+filterwarnings =
+    error::ResourceWarning
+    error::pytest.PytestUnraisableExceptionWarning
+
 # Output options
 addopts =
     -v

scripts/runtime_resource_probe.py

@@ -0,0 +1,466 @@
+#!/usr/bin/env python3
+"""Exercise long-lived Core registries and verify that they reach a plateau.
+
+This probe is intentionally separate from the default test suite because the
+audit profile creates tens of thousands of historical identities. It uses the
+real admission, eviction, and cleanup code while replacing external platform
+objects that are irrelevant to registry retention.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import gc
+import json
+import time
+import tracemalloc
+from dataclasses import asdict, dataclass
+from types import SimpleNamespace
+from unittest.mock import patch
+
+import psutil
+
+from langbot.pkg.api.http.context import ExecutionContext
+
+# Import the Application graph before taskmgr. The production boot path has
+# this same ordering; importing taskmgr first exposes its historical cycle
+# through HTTP route annotations.
+from langbot.pkg.core import app as _core_app  # noqa: F401
+from langbot.pkg.core.taskmgr import AsyncTaskManager
+from langbot.pkg.pipeline.pool import QueryPool
+from langbot.pkg.pipeline.ratelimit.algos.fixedwin import FixedWindowAlgo
+from langbot.pkg.plugin.connector import PluginRuntimeConnector
+from langbot.pkg.platform.sources.websocket_adapter import (
+    WebSocketMessage,
+    WebSocketSession,
+)
+from langbot.pkg.provider.modelmgr.modelmgr import ModelManager
+from langbot.pkg.provider.session.sessionmgr import SessionManager
+from langbot_plugin.api.entities.builtin.provider.session import LauncherTypes
+
+
+@dataclass(frozen=True, slots=True)
+class ProbeScale:
+    query_churn_per_phase: int
+    session_churn_per_phase: int
+    rate_limit_churn_per_phase: int
+    task_churn_per_phase: int
+    websocket_churn_per_phase: int
+    empty_workspace_churn_per_phase: int
+
+
+SCALES = {
+    'quick': ProbeScale(
+        query_churn_per_phase=2_500,
+        session_churn_per_phase=500,
+        rate_limit_churn_per_phase=10_000,
+        task_churn_per_phase=1_000,
+        websocket_churn_per_phase=500,
+        empty_workspace_churn_per_phase=1_000,
+    ),
+    'audit': ProbeScale(
+        query_churn_per_phase=25_000,
+        session_churn_per_phase=2_500,
+        rate_limit_churn_per_phase=10_000,
+        task_churn_per_phase=5_000,
+        websocket_churn_per_phase=2_500,
+        empty_workspace_churn_per_phase=10_000,
+    ),
+}
+
+
+class _ProbeQuery:
+    """Small weak-referenceable stand-in for SDK Query construction."""
+
+    def __init__(self, **values):
+        self.__dict__.update(values)
+
+
+class _EmptyResult:
+    def all(self) -> list:
+        return []
+
+
+class _EmptyPluginRuntimeHandler:
+    async def reconcile_plugin_installations(self, _states: tuple) -> dict:
+        return {
+            'applied': [],
+            'removed': [],
+            'missing_artifacts': [],
+            'failed_installations': [],
+        }
+
+    def unregister_installation_binding(self, _binding) -> None:
+        raise AssertionError('An empty Workspace exposed an installation binding')
+
+
+@dataclass(frozen=True, slots=True)
+class ProcessSample:
+    rss_bytes: int
+    traced_current_bytes: int
+    traced_peak_bytes: int
+    asyncio_tasks: int
+    threads: int
+    open_fds: int | None
+
+
+def _sample_process() -> ProcessSample:
+    gc.collect()
+    process = psutil.Process()
+    try:
+        open_fds = process.num_fds()
+    except (AttributeError, psutil.Error):
+        open_fds = None
+    traced_current, traced_peak = tracemalloc.get_traced_memory()
+    return ProcessSample(
+        rss_bytes=process.memory_info().rss,
+        traced_current_bytes=traced_current,
+        traced_peak_bytes=traced_peak,
+        asyncio_tasks=len(asyncio.all_tasks()),
+        threads=process.num_threads(),
+        open_fds=open_fds,
+    )
+
+
+def _execution_context(index: int, *, query_uuid: str | None = None) -> ExecutionContext:
+    return ExecutionContext(
+        instance_uuid='runtime-resource-probe',
+        workspace_uuid=f'workspace-{index}',
+        placement_generation=1,
+        bot_uuid='probe-bot',
+        pipeline_uuid='probe-pipeline',
+        query_uuid=query_uuid,
+    )
+
+
+class CoreRuntimeProbe:
+    """Own the same manager instances across two equal churn phases."""
+
+    def __init__(self) -> None:
+        self.query_pool = QueryPool(max_queries=100, max_queries_per_workspace=1)
+        app = SimpleNamespace(
+            event_loop=asyncio.get_running_loop(),
+            persistence_mgr=None,
+            instance_config=SimpleNamespace(
+                data={
+                    'concurrency': {'session': 1},
+                    'system': {
+                        'session_retention': {
+                            'idle_ttl_seconds': 86_400,
+                            'max_entries': 200,
+                            'max_entries_per_workspace': 200,
+                            'max_conversations_per_session': 20,
+                            'max_messages_per_conversation': 100,
+                        },
+                        'task_retention': {
+                            'completed_limit': 200,
+                            'max_log_chars': 4_096,
+                            'max_active_user_tasks': 256,
+                            'max_active_user_tasks_per_workspace': 8,
+                        },
+                    },
+                }
+            ),
+        )
+        self.session_manager = SessionManager(app)
+        self.task_manager = AsyncTaskManager(app)
+        self.rate_limit = FixedWindowAlgo(SimpleNamespace())
+        self.websocket_session = WebSocketSession(
+            'resource-probe',
+            max_conversations=200,
+            max_messages=100,
+        )
+        logger = SimpleNamespace(
+            debug=lambda *_args, **_kwargs: None,
+            info=lambda *_args, **_kwargs: None,
+            warning=lambda *_args, **_kwargs: None,
+            error=lambda *_args, **_kwargs: None,
+        )
+        self.empty_model_queries = 0
+
+        async def execute_empty(_statement):
+            self.empty_model_queries += 1
+            return _EmptyResult()
+
+        model_app = SimpleNamespace(
+            logger=logger,
+            persistence_mgr=SimpleNamespace(execute_async=execute_empty),
+        )
+        self.empty_model_manager = ModelManager(model_app)
+
+        async def runtime_disconnect_callback(_connector) -> None:
+            return None
+
+        plugin_app = SimpleNamespace(
+            instance_config=SimpleNamespace(data={'plugin': {'enable': True}}),
+            deployment=SimpleNamespace(mode='cloud'),
+            logger=logger,
+        )
+        self.empty_plugin_connector = PluginRuntimeConnector(
+            plugin_app,
+            runtime_disconnect_callback,
+        )
+        self.empty_plugin_connector.handler = _EmptyPluginRuntimeHandler()
+
+        async def validate_context(context):
+            return context
+
+        async def load_desired_states(_context):
+            return []
+
+        self.empty_plugin_connector._validate_execution_context = validate_context
+        self.empty_plugin_connector._load_workspace_desired_states = load_desired_states
+
+    async def initialize(self) -> None:
+        await self.rate_limit.initialize()
+
+    async def run_phase(self, scale: ProbeScale, phase: int) -> None:
+        offsets = {
+            'query': (phase - 1) * scale.query_churn_per_phase,
+            'session': (phase - 1) * scale.session_churn_per_phase,
+            'rate': (phase - 1) * scale.rate_limit_churn_per_phase,
+            'task': (phase - 1) * scale.task_churn_per_phase,
+            'websocket': (phase - 1) * scale.websocket_churn_per_phase,
+            'empty_workspace': ((phase - 1) * scale.empty_workspace_churn_per_phase),
+        }
+        await self._churn_queries(offsets['query'], scale.query_churn_per_phase)
+        await self._churn_sessions(offsets['session'], scale.session_churn_per_phase)
+        await self._churn_rate_limits(offsets['rate'], scale.rate_limit_churn_per_phase)
+        await self._churn_tasks(offsets['task'], scale.task_churn_per_phase)
+        self._churn_websocket_history(
+            offsets['websocket'],
+            scale.websocket_churn_per_phase,
+        )
+        await self._churn_empty_workspaces(
+            offsets['empty_workspace'],
+            scale.empty_workspace_churn_per_phase,
+        )
+        await asyncio.sleep(0)
+
+    async def _churn_queries(self, start: int, count: int) -> None:
+        def make_query(**values):
+            return _ProbeQuery(**values)
+
+        with patch(
+            'langbot.pkg.pipeline.pool.pipeline_query.Query',
+            side_effect=make_query,
+        ):
+            for index in range(start, start + count):
+                context = _execution_context(index)
+                query = await self.query_pool.add_query(
+                    bot_uuid='probe-bot',
+                    launcher_type=LauncherTypes.PERSON,
+                    launcher_id=f'launcher-{index}',
+                    sender_id=f'sender-{index}',
+                    message_event=SimpleNamespace(),
+                    message_chain=SimpleNamespace(),
+                    adapter=None,
+                    pipeline_uuid='probe-pipeline',
+                    execution_context=context,
+                )
+                removed = await self.query_pool.remove_query(query)
+                if not removed:
+                    raise AssertionError('Query cleanup failed')
+
+    async def _churn_sessions(self, start: int, count: int) -> None:
+        for index in range(start, start + count):
+            workspace_index = index % 100
+            context = _execution_context(
+                workspace_index,
+                query_uuid=f'session-query-{index}',
+            )
+            query = SimpleNamespace(
+                launcher_type=LauncherTypes.PERSON,
+                launcher_id=f'launcher-{index}',
+                sender_id=f'sender-{index}',
+                bot_uuid='probe-bot',
+                pipeline_uuid='probe-pipeline',
+                query_uuid=context.query_uuid,
+                _execution_context=context,
+            )
+            await self.session_manager.get_session(query)
+
+    async def _churn_rate_limits(self, start: int, count: int) -> None:
+        for index in range(start, start + count):
+            context = _execution_context(
+                index % 1_000,
+                query_uuid=f'rate-query-{index}',
+            )
+            query = SimpleNamespace(
+                bot_uuid='probe-bot',
+                pipeline_uuid='probe-pipeline',
+                _execution_context=context,
+                pipeline_config={
+                    'safety': {
+                        'rate-limit': {
+                            'window-length': 60,
+                            'limitation': 100_000,
+                            'strategy': 'drop',
+                        }
+                    }
+                },
+            )
+            admitted = await self.rate_limit.require_access(
+                query,
+                LauncherTypes.PERSON,
+                f'rate-identity-{index}',
+            )
+            if not admitted:
+                raise AssertionError('Rate-limit registry rejected bounded churn')
+
+    async def _churn_tasks(self, start: int, count: int) -> None:
+        async def complete_immediately() -> None:
+            return None
+
+        for batch_start in range(start, start + count, 256):
+            batch_size = min(256, start + count - batch_start)
+            wrappers = [
+                self.task_manager.create_task(
+                    complete_immediately(),
+                    name=f'resource-probe-{batch_start + offset}',
+                )
+                for offset in range(batch_size)
+            ]
+            await asyncio.gather(*(wrapper.task for wrapper in wrappers))
+            await asyncio.sleep(0)
+
+    def _churn_websocket_history(self, start: int, count: int) -> None:
+        for index in range(start, start + count):
+            conversation_key = f'conversation-{index}'
+            response_id = f'response-{index}'
+            indexes = self.websocket_session.get_stream_message_indexes(conversation_key)
+            indexes[response_id] = 0
+            self.websocket_session.append_message(
+                conversation_key,
+                WebSocketMessage(
+                    id=self.websocket_session.next_message_id(conversation_key),
+                    role='assistant',
+                    content='probe',
+                    message_chain=[],
+                    timestamp='1970-01-01T00:00:00+00:00',
+                    is_final=True,
+                ),
+            )
+
+    async def _churn_empty_workspaces(self, start: int, count: int) -> None:
+        for index in range(start, start + count):
+            await self.empty_model_manager._load_workspace_models(_execution_context(index))
+        await self.empty_plugin_connector.reconcile_projected_workspaces(
+            _execution_context(index) for index in range(start, start + count)
+        )
+
+    def retained_state(self) -> dict[str, int]:
+        return {
+            'query_cached': len(self.query_pool.cached_queries),
+            'query_queued': len(self.query_pool.queries),
+            'query_active_workspaces': len(self.query_pool.active_query_count_by_workspace),
+            'query_scope_counters': len(self.query_pool.query_count_by_scope),
+            'sessions': len(self.session_manager.session_list),
+            'session_index': len(self.session_manager._session_index),
+            'rate_limit_containers': len(self.rate_limit.containers),
+            'task_records': len(self.task_manager.tasks),
+            'websocket_conversations': len(self.websocket_session.message_lists),
+            'websocket_stream_indexes': len(self.websocket_session.stream_message_indexes),
+            'empty_model_scopes': len(self.empty_model_manager._scope_generations),
+            'empty_model_providers': len(self.empty_model_manager.provider_dict),
+            'empty_model_llms': len(self.empty_model_manager.llm_model_dict),
+            'empty_plugin_workspace_sets': len(self.empty_plugin_connector._workspace_installations),
+            'empty_plugin_installations': len(self.empty_plugin_connector._known_desired_states),
+        }
+
+    def assert_bounded(self) -> None:
+        state = self.retained_state()
+        expected_maximums = {
+            'query_cached': 0,
+            'query_queued': 0,
+            'query_active_workspaces': 0,
+            'query_scope_counters': 100,
+            'sessions': 200,
+            'session_index': 200,
+            'rate_limit_containers': 10_000,
+            'task_records': 200,
+            'websocket_conversations': 200,
+            'websocket_stream_indexes': 200,
+            'empty_model_scopes': 0,
+            'empty_model_providers': 0,
+            'empty_model_llms': 0,
+            'empty_plugin_workspace_sets': 0,
+            'empty_plugin_installations': 0,
+        }
+        violations = {key: (state[key], maximum) for key, maximum in expected_maximums.items() if state[key] > maximum}
+        if violations:
+            raise AssertionError(f'Core retained-state limits failed: {violations}')
+
+
+async def _run(args: argparse.Namespace) -> dict:
+    scale = SCALES[args.scale]
+    tracemalloc.start()
+    started_at = time.monotonic()
+    probe = CoreRuntimeProbe()
+    await probe.initialize()
+
+    baseline = _sample_process()
+    await probe.run_phase(scale, 1)
+    probe.assert_bounded()
+    phase_one = _sample_process()
+    state_one = probe.retained_state()
+
+    await probe.run_phase(scale, 2)
+    probe.assert_bounded()
+    phase_two = _sample_process()
+    state_two = probe.retained_state()
+
+    if state_two != state_one:
+        raise AssertionError(f'Core retained state did not plateau: phase_one={state_one}, phase_two={state_two}')
+    traced_growth = phase_two.traced_current_bytes - phase_one.traced_current_bytes
+    rss_growth = phase_two.rss_bytes - phase_one.rss_bytes
+    max_traced_growth = int(args.max_traced_growth_mib * 1024 * 1024)
+    max_rss_growth = int(args.max_rss_growth_mib * 1024 * 1024)
+    if traced_growth > max_traced_growth:
+        raise AssertionError(f'Second-phase traced memory grew by {traced_growth} bytes (limit {max_traced_growth})')
+    if rss_growth > max_rss_growth:
+        raise AssertionError(f'Second-phase RSS grew by {rss_growth} bytes (limit {max_rss_growth})')
+
+    return {
+        'component': 'langbot-core',
+        'scale': args.scale,
+        'work_per_phase': asdict(scale),
+        'elapsed_seconds': round(time.monotonic() - started_at, 3),
+        'samples': {
+            'baseline': asdict(baseline),
+            'phase_one': asdict(phase_one),
+            'phase_two': asdict(phase_two),
+        },
+        'second_phase_growth': {
+            'rss_bytes': rss_growth,
+            'traced_current_bytes': traced_growth,
+        },
+        'retained_state': {
+            'phase_one': state_one,
+            'phase_two': state_two,
+        },
+        'passed': True,
+    }
+
+
+def _parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--scale', choices=tuple(SCALES), default='quick')
+    parser.add_argument('--max-traced-growth-mib', type=float, default=8.0)
+    parser.add_argument('--max-rss-growth-mib', type=float, default=64.0)
+    parser.add_argument('--json', action='store_true', help='Print compact JSON')
+    return parser.parse_args()
+
+
+def main() -> None:
+    args = _parse_args()
+    result = asyncio.run(_run(args))
+    if args.json:
+        print(json.dumps(result, sort_keys=True))
+    else:
+        print(json.dumps(result, indent=2, sort_keys=True))
+
+
+if __name__ == '__main__':
+    main()

scripts/workspace_runtime_capacity_probe.py

@@ -0,0 +1,561 @@
+#!/usr/bin/env python3
+"""Measure populated Workspace runtime replacement cost and retention.
+
+Unlike ``runtime_resource_probe.py``, which stresses historical request keys
+and empty tenants, this probe keeps one representative Provider, LLM,
+Embedding model, Rerank model, Pipeline, Bot, and Knowledge Base per Workspace.
+It then advances every Workspace to a new placement generation and verifies
+that old runtime objects are closed and collectible while active registry
+cardinality remains constant.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import gc
+import json
+import time
+import tracemalloc
+import weakref
+from dataclasses import asdict, dataclass
+from types import SimpleNamespace
+
+import psutil
+
+from langbot.pkg.api.http.context import ExecutionContext
+
+# Match the production import order; importing a leaf manager first exposes a
+# historical annotation cycle that the application graph resolves.
+from langbot.pkg.core import app as _core_app  # noqa: F401
+from langbot.pkg.entity.persistence import bot as persistence_bot
+from langbot.pkg.entity.persistence import model as persistence_model
+from langbot.pkg.entity.persistence import pipeline as persistence_pipeline
+from langbot.pkg.entity.persistence import rag as persistence_rag
+from langbot.pkg.pipeline.pipelinemgr import PipelineManager
+from langbot.pkg.platform.botmgr import PlatformManager
+from langbot.pkg.provider.modelmgr import requester
+from langbot.pkg.provider.modelmgr.modelmgr import ModelManager
+from langbot.pkg.provider.tools.loaders.mcp import MCPLoader
+from langbot.pkg.rag.knowledge.kbmgr import RAGManager
+from langbot.pkg.workspace.entities import WorkspaceExecutionBinding
+
+
+@dataclass(frozen=True, slots=True)
+class ProbeScale:
+    workspaces: int
+
+
+SCALES = {
+    'quick': ProbeScale(workspaces=250),
+    'audit': ProbeScale(workspaces=5_000),
+}
+
+
+@dataclass(frozen=True, slots=True)
+class ProcessSample:
+    rss_bytes: int
+    traced_current_bytes: int
+    traced_peak_bytes: int
+    asyncio_tasks: int
+    threads: int
+    open_fds: int | None
+
+
+class _ProbeLogger:
+    def debug(self, *_args, **_kwargs) -> None:
+        return None
+
+    def info(self, *_args, **_kwargs) -> None:
+        return None
+
+    def warning(self, *_args, **_kwargs) -> None:
+        return None
+
+    def error(self, *_args, **_kwargs) -> None:
+        return None
+
+
+class _ProbeWorkspaceService:
+    instance_uuid = 'runtime-capacity-probe'
+
+    def __init__(self) -> None:
+        self.generations: dict[str, int] = {}
+        self.binding_lookups = 0
+
+    async def get_execution_binding(
+        self,
+        workspace_uuid: str,
+        *,
+        expected_generation: int | None = None,
+    ) -> WorkspaceExecutionBinding:
+        self.binding_lookups += 1
+        generation = self.generations[workspace_uuid]
+        if expected_generation is not None and expected_generation != generation:
+            raise AssertionError(f'stale probe generation {expected_generation} != {generation}')
+        return WorkspaceExecutionBinding(
+            instance_uuid=self.instance_uuid,
+            workspace_uuid=workspace_uuid,
+            placement_generation=generation,
+            write_fenced=False,
+            state='active',
+        )
+
+
+class _ProbeRequester(requester.ProviderAPIRequester):
+    name = 'capacity-probe'
+    closed = 0
+
+    async def invoke_llm(
+        self,
+        query,
+        model,
+        messages,
+        funcs=None,
+        extra_args=None,
+        remove_think=False,
+    ):
+        return None
+
+    async def aclose(self) -> None:
+        type(self).closed += 1
+
+
+class _ProbeAdapter:
+    killed = 0
+
+    def __init__(self, _config, _logger) -> None:
+        self.listeners = []
+
+    def register_listener(self, event_type, listener) -> None:
+        self.listeners.append((event_type, listener))
+
+    async def kill(self) -> None:
+        type(self).killed += 1
+
+
+class _ProbeMCPSession:
+    closed = 0
+
+    def __init__(self, server_name: str) -> None:
+        self.server_name = server_name
+
+    async def shutdown(self) -> None:
+        type(self).closed += 1
+
+
+def _sample_process() -> ProcessSample:
+    gc.collect()
+    process = psutil.Process()
+    try:
+        open_fds = process.num_fds()
+    except (AttributeError, psutil.Error):
+        open_fds = None
+    traced_current, traced_peak = tracemalloc.get_traced_memory()
+    return ProcessSample(
+        rss_bytes=process.memory_info().rss,
+        traced_current_bytes=traced_current,
+        traced_peak_bytes=traced_peak,
+        asyncio_tasks=len(asyncio.all_tasks()),
+        threads=process.num_threads(),
+        open_fds=open_fds,
+    )
+
+
+class PopulatedWorkspaceProbe:
+    def __init__(self) -> None:
+        _ProbeRequester.closed = 0
+        _ProbeAdapter.killed = 0
+        _ProbeMCPSession.closed = 0
+        self.workspace_service = _ProbeWorkspaceService()
+        self.logger = _ProbeLogger()
+        self.app = SimpleNamespace(
+            logger=self.logger,
+            workspace_service=self.workspace_service,
+            persistence_mgr=SimpleNamespace(
+                mode=SimpleNamespace(value='cloud_runtime'),
+            ),
+            pipeline_config_meta_trigger={'name': 'trigger', 'stages': []},
+            pipeline_config_meta_safety={'name': 'safety', 'stages': []},
+            pipeline_config_meta_ai={'name': 'ai', 'stages': []},
+            pipeline_config_meta_output={'name': 'output', 'stages': []},
+            task_mgr=SimpleNamespace(
+                cancel_by_scope=lambda *_args, **_kwargs: None,
+                cancel_task=lambda *_args, **_kwargs: None,
+            ),
+        )
+        self.model_manager = ModelManager(self.app)
+        self.model_manager.requester_dict = {
+            _ProbeRequester.name: _ProbeRequester,
+        }
+        self.pipeline_manager = PipelineManager(self.app)
+        self.pipeline_manager.stage_dict = {}
+        self.rag_manager = RAGManager(self.app)
+        self.mcp_loader = MCPLoader(self.app)
+        self.platform_manager = PlatformManager(self.app)
+        self.platform_manager.adapter_dict = {
+            'capacity-probe': _ProbeAdapter,
+        }
+        self.generation_refs: dict[
+            int,
+            list[weakref.ReferenceType],
+        ] = {}
+
+    def _context(
+        self,
+        workspace_uuid: str,
+        generation: int,
+        *,
+        bot_uuid: str | None = None,
+        pipeline_uuid: str | None = None,
+    ) -> ExecutionContext:
+        return ExecutionContext(
+            instance_uuid=self.workspace_service.instance_uuid,
+            workspace_uuid=workspace_uuid,
+            placement_generation=generation,
+            bot_uuid=bot_uuid,
+            pipeline_uuid=pipeline_uuid,
+        )
+
+    async def load_generation(self, workspaces: int, generation: int) -> None:
+        for index in range(workspaces):
+            workspace_uuid = f'workspace-{index}'
+            provider_uuid = f'provider-{index}'
+            llm_uuid = f'llm-{index}'
+            embedding_uuid = f'embedding-{index}'
+            rerank_uuid = f'rerank-{index}'
+            pipeline_uuid = f'pipeline-{index}'
+            bot_uuid = f'bot-{index}'
+            kb_uuid = f'knowledge-{index}'
+            mcp_server_name = f'mcp-{index}'
+            self.workspace_service.generations[workspace_uuid] = generation
+            context = self._context(workspace_uuid, generation)
+
+            runtime_provider = await self.model_manager.load_provider(
+                context,
+                persistence_model.ModelProvider(
+                    uuid=provider_uuid,
+                    workspace_uuid=workspace_uuid,
+                    name='Capacity Provider',
+                    requester=_ProbeRequester.name,
+                    base_url='https://capacity.invalid',
+                    api_keys=['probe'],
+                ),
+            )
+            await self.model_manager.cache_provider(context, runtime_provider)
+
+            runtime_llm = await self.model_manager.load_llm_model_with_provider(
+                context,
+                persistence_model.LLMModel(
+                    uuid=llm_uuid,
+                    workspace_uuid=workspace_uuid,
+                    name='Capacity LLM',
+                    provider_uuid=provider_uuid,
+                    abilities=['func_call'],
+                    extra_args={'temperature': 0.1},
+                ),
+                runtime_provider,
+            )
+            await self.model_manager.cache_llm_model(context, runtime_llm)
+            runtime_embedding = await self.model_manager.load_embedding_model_with_provider(
+                context,
+                persistence_model.EmbeddingModel(
+                    uuid=embedding_uuid,
+                    workspace_uuid=workspace_uuid,
+                    name='Capacity Embedding',
+                    provider_uuid=provider_uuid,
+                    extra_args={'dimensions': 1_024},
+                ),
+                runtime_provider,
+            )
+            await self.model_manager.cache_embedding_model(
+                context,
+                runtime_embedding,
+            )
+            runtime_rerank = await self.model_manager.load_rerank_model_with_provider(
+                context,
+                persistence_model.RerankModel(
+                    uuid=rerank_uuid,
+                    workspace_uuid=workspace_uuid,
+                    name='Capacity Rerank',
+                    provider_uuid=provider_uuid,
+                    extra_args={},
+                ),
+                runtime_provider,
+            )
+            await self.model_manager.cache_rerank_model(
+                context,
+                runtime_rerank,
+            )
+
+            pipeline_context = self._context(
+                workspace_uuid,
+                generation,
+                pipeline_uuid=pipeline_uuid,
+            )
+            await self.pipeline_manager.load_pipeline(
+                pipeline_context,
+                persistence_pipeline.LegacyPipeline(
+                    uuid=pipeline_uuid,
+                    workspace_uuid=workspace_uuid,
+                    name='Capacity Pipeline',
+                    description='',
+                    for_version='probe',
+                    is_default=True,
+                    stages=[],
+                    config={},
+                    extensions_preferences={},
+                ),
+                _binding_validated=True,
+            )
+            runtime_pipeline = self.pipeline_manager._pipelines_by_key[
+                (
+                    self.workspace_service.instance_uuid,
+                    workspace_uuid,
+                    pipeline_uuid,
+                )
+            ]
+
+            runtime_kb = await self.rag_manager.load_knowledge_base(
+                context,
+                persistence_rag.KnowledgeBase(
+                    uuid=kb_uuid,
+                    workspace_uuid=workspace_uuid,
+                    name='Capacity Knowledge',
+                    description='',
+                    knowledge_engine_plugin_id=None,
+                    collection_id=kb_uuid,
+                    creation_settings={},
+                    retrieval_settings={},
+                ),
+                _binding_validated=True,
+            )
+
+            await self.mcp_loader._assert_execution_active(context)
+            runtime_mcp = _ProbeMCPSession(mcp_server_name)
+            self.mcp_loader._register_session(
+                context,
+                mcp_server_name,
+                runtime_mcp,
+            )
+
+            bot_context = self._context(
+                workspace_uuid,
+                generation,
+                bot_uuid=bot_uuid,
+            )
+            runtime_bot = await self.platform_manager.load_bot(
+                bot_context,
+                persistence_bot.Bot(
+                    uuid=bot_uuid,
+                    workspace_uuid=workspace_uuid,
+                    name='Capacity Bot',
+                    description='',
+                    adapter='capacity-probe',
+                    adapter_config={},
+                    enable=True,
+                    use_pipeline_uuid=pipeline_uuid,
+                    pipeline_routing_rules=[],
+                ),
+                _binding_validated=True,
+            )
+
+            self.generation_refs.setdefault(generation, []).extend(
+                (
+                    weakref.ref(runtime_provider),
+                    weakref.ref(runtime_llm),
+                    weakref.ref(runtime_embedding),
+                    weakref.ref(runtime_rerank),
+                    weakref.ref(runtime_pipeline),
+                    weakref.ref(runtime_kb),
+                    weakref.ref(runtime_mcp),
+                    weakref.ref(runtime_bot),
+                )
+            )
+
+        await asyncio.sleep(0)
+
+    def retained_state(self) -> dict[str, int]:
+        return {
+            'model_providers': len(self.model_manager.provider_dict),
+            'llm_models': len(self.model_manager.llm_model_dict),
+            'embedding_models': len(self.model_manager.embedding_model_dict),
+            'rerank_models': len(self.model_manager.rerank_model_dict),
+            'model_scopes': len(self.model_manager._scope_generations),
+            'pipelines': len(self.pipeline_manager._pipelines_by_key),
+            'pipeline_scopes': len(self.pipeline_manager._scope_generations),
+            'knowledge_bases': len(self.rag_manager.knowledge_bases),
+            'knowledge_scopes': len(self.rag_manager._scope_generations),
+            'mcp_sessions': len(self.mcp_loader.sessions),
+            'mcp_scopes': len(self.mcp_loader._scope_generations),
+            'bots': len(self.platform_manager._bots_by_key),
+            'bot_scopes': len(self.platform_manager._scope_generations),
+            'requesters_closed': _ProbeRequester.closed,
+            'adapters_killed': _ProbeAdapter.killed,
+            'mcp_sessions_closed': _ProbeMCPSession.closed,
+            'binding_lookups': self.workspace_service.binding_lookups,
+        }
+
+    def assert_generation_state(
+        self,
+        workspaces: int,
+        generation: int,
+    ) -> None:
+        state = self.retained_state()
+        cardinality_keys = (
+            'model_providers',
+            'llm_models',
+            'embedding_models',
+            'rerank_models',
+            'model_scopes',
+            'pipelines',
+            'pipeline_scopes',
+            'knowledge_bases',
+            'knowledge_scopes',
+            'mcp_sessions',
+            'mcp_scopes',
+            'bots',
+            'bot_scopes',
+        )
+        invalid = {key: value for key in cardinality_keys if (value := state[key]) != workspaces}
+        if invalid:
+            raise AssertionError(f'populated Workspace cardinality mismatch: {invalid}')
+        expected_retired = (generation - 1) * workspaces
+        if state['requesters_closed'] != expected_retired:
+            raise AssertionError(f'retired requester count {state["requesters_closed"]} != {expected_retired}')
+        if state['adapters_killed'] != expected_retired:
+            raise AssertionError(f'retired adapter count {state["adapters_killed"]} != {expected_retired}')
+        if state['mcp_sessions_closed'] != expected_retired:
+            raise AssertionError(f'retired MCP session count {state["mcp_sessions_closed"]} != {expected_retired}')
+
+    def assert_generation_collected(self, generation: int) -> None:
+        gc.collect()
+        references = self.generation_refs.pop(generation)
+        retained = sum(reference() is not None for reference in references)
+        if retained:
+            raise AssertionError(f'{retained} generation-{generation} runtime objects remain reachable')
+
+
+async def _run(args: argparse.Namespace) -> dict:
+    scale = SCALES[args.scale]
+    tracemalloc.start()
+    probe = PopulatedWorkspaceProbe()
+    baseline = _sample_process()
+
+    phase_one_started = time.monotonic()
+    await probe.load_generation(scale.workspaces, 1)
+    phase_one_seconds = time.monotonic() - phase_one_started
+    probe.assert_generation_state(scale.workspaces, 1)
+    phase_one = _sample_process()
+    phase_one_state = probe.retained_state()
+
+    phase_two_started = time.monotonic()
+    await probe.load_generation(scale.workspaces, 2)
+    phase_two_seconds = time.monotonic() - phase_two_started
+    probe.assert_generation_state(scale.workspaces, 2)
+    probe.assert_generation_collected(1)
+    phase_two = _sample_process()
+    phase_two_state = probe.retained_state()
+
+    phase_three_started = time.monotonic()
+    await probe.load_generation(scale.workspaces, 3)
+    phase_three_seconds = time.monotonic() - phase_three_started
+    probe.assert_generation_state(scale.workspaces, 3)
+    probe.assert_generation_collected(2)
+    phase_three = _sample_process()
+    phase_three_state = probe.retained_state()
+
+    cardinality_keys = (
+        'model_providers',
+        'llm_models',
+        'embedding_models',
+        'rerank_models',
+        'model_scopes',
+        'pipelines',
+        'pipeline_scopes',
+        'knowledge_bases',
+        'knowledge_scopes',
+        'mcp_sessions',
+        'mcp_scopes',
+        'bots',
+        'bot_scopes',
+    )
+    if any(
+        phase_two_state[key] != phase_one_state[key] or phase_three_state[key] != phase_one_state[key]
+        for key in cardinality_keys
+    ):
+        raise AssertionError(
+            'populated Workspace registries did not plateau: '
+            f'phase_one={phase_one_state}, phase_two={phase_two_state}, '
+            f'phase_three={phase_three_state}'
+        )
+
+    traced_growth = phase_three.traced_current_bytes - phase_two.traced_current_bytes
+    rss_growth = phase_three.rss_bytes - phase_two.rss_bytes
+    max_traced_growth = int(args.max_traced_growth_mib * 1024 * 1024)
+    max_rss_growth = int(args.max_rss_growth_mib * 1024 * 1024)
+    if traced_growth > max_traced_growth:
+        raise AssertionError(f'replacement traced memory grew by {traced_growth} bytes (limit {max_traced_growth})')
+    if rss_growth > max_rss_growth:
+        raise AssertionError(f'replacement RSS grew by {rss_growth} bytes (limit {max_rss_growth})')
+    phase_ratio = max(
+        phase_two_seconds,
+        phase_three_seconds,
+    ) / max(phase_one_seconds, 0.000_001)
+    if phase_ratio > args.max_replacement_time_ratio:
+        raise AssertionError(f'replacement phase ratio {phase_ratio:.3f} exceeds {args.max_replacement_time_ratio:.3f}')
+
+    return {
+        'component': 'langbot-populated-workspaces',
+        'scale': args.scale,
+        'workspaces': scale.workspaces,
+        'passed': True,
+        'phase_seconds': {
+            'initial': round(phase_one_seconds, 3),
+            'replacement_one': round(phase_two_seconds, 3),
+            'replacement_two': round(phase_three_seconds, 3),
+            'maximum_replacement_ratio': round(phase_ratio, 3),
+        },
+        'samples': {
+            'baseline': asdict(baseline),
+            'phase_one': asdict(phase_one),
+            'phase_two': asdict(phase_two),
+            'phase_three': asdict(phase_three),
+        },
+        'replacement_growth': {
+            'rss_bytes': rss_growth,
+            'traced_current_bytes': traced_growth,
+        },
+        'retained_state': {
+            'phase_one': phase_one_state,
+            'phase_two': phase_two_state,
+            'phase_three': phase_three_state,
+        },
+    }
+
+
+def _parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--scale', choices=tuple(SCALES), default='quick')
+    parser.add_argument('--max-traced-growth-mib', type=float, default=16.0)
+    parser.add_argument('--max-rss-growth-mib', type=float, default=64.0)
+    parser.add_argument(
+        '--max-replacement-time-ratio',
+        type=float,
+        default=3.0,
+    )
+    parser.add_argument('--json', action='store_true')
+    return parser.parse_args()
+
+
+def main() -> None:
+    args = _parse_args()
+    result = asyncio.run(_run(args))
+    if args.json:
+        print(json.dumps(result, sort_keys=True))
+    else:
+        print(json.dumps(result, indent=2, sort_keys=True))
+
+
+if __name__ == '__main__':
+    main()

skills/README.md

@@ -26,7 +26,7 @@ and LangBot's own Local Agent) working with the LangBot ecosystem.
 
 ## Quick start (for an AI agent)
 
-1. Read this README, `AGENTS.md`, and `qa-agent-docs/` to understand the layout.
+1. Read this README, `AGENTS.md`, and `docs/user-guide.md` to understand the layout.
 2. Read `skills/.env` for shared local defaults. On a new machine, copy
    `skills/.env.example` to `skills/.env.local` (gitignored) and override
    machine-specific values there. Never commit secrets.
@@ -48,6 +48,7 @@ bin/lbs env show     # inspect resolved env defaults (redacted)
 bin/lbs env doctor   # diagnose local environment readiness
 bin/lbs case list --ready
 bin/lbs test plan <case-id>
+bin/lbs suite plan langbot-debug-chat-load-gate
 ```
 
 ## Maintenance rule

skills/docs/user-guide.md

@@ -0,0 +1,171 @@
+# LangBot QA Skills User Guide
+
+Use this guide as the first operational path after reading `README.md` and
+`AGENTS.md`.
+
+## 1. Configure Local Inputs
+
+Read `skills/.env`, then create `skills/.env.local` for machine-local values.
+Do not commit `.env.local`, browser profiles, reports, tokens, API keys, OAuth
+state, or provider credentials.
+
+Minimum local fields for live browser QA:
+
+```bash
+LANGBOT_REPO=/path/to/LangBot
+LANGBOT_WEB_REPO=/path/to/LangBot/web
+LANGBOT_BACKEND_URL=http://127.0.0.1:5300
+LANGBOT_FRONTEND_URL=http://127.0.0.1:3000
+LANGBOT_DEV_FRONTEND_URL=http://127.0.0.1:3000
+LANGBOT_BROWSER_PROFILE=/path/to/langbot-browser-profile
+LANGBOT_CHROMIUM_EXECUTABLE=/path/to/chromium-or-playwright-chrome
+LANGBOT_E2E_LOGIN_USER=qa-local@example.com
+```
+
+`LANGBOT_E2E_LOGIN_USER` is a local QA account. The setup automation uses the
+LangBot recovery key from the active checkout to initialize or refresh that
+local account and write a browser `localStorage` token. It does not need the
+user's GitHub or Space credentials.
+
+## 2. Check Readiness
+
+From `skills/`:
+
+```bash
+bin/lbs env show
+bin/lbs env doctor
+bin/lbs validate
+bin/lbs index --check
+```
+
+`env doctor` should report reachable backend and frontend URLs before live
+browser cases are run. Missing Space provider credentials are not a LangBot
+product pass; classify them as `env_issue` and configure the local Space
+provider before measuring Debug Chat performance.
+
+## 3. Start Services
+
+Start the backend from `LANGBOT_REPO`:
+
+```bash
+cd "$LANGBOT_REPO"
+uv run main.py
+```
+
+Start the standalone frontend from `LANGBOT_WEB_REPO` and point it at the
+backend:
+
+```bash
+cd "$LANGBOT_WEB_REPO"
+VITE_API_BASE_URL="$LANGBOT_BACKEND_URL" pnpm dev --host 0.0.0.0
+```
+
+If `VITE_API_BASE_URL` is missing, browser tests can load the Vite page but send
+API requests to the frontend port, which produces false UI failures.
+
+## 4. Prepare User-Path Fixtures
+
+For local-agent Debug Chat cases and the user-path performance gate:
+
+```bash
+node scripts/e2e/ensure-local-agent-pipeline.mjs --write-env
+```
+
+The script:
+
+- refreshes the local QA login and browser token;
+- marks the local wizard as skipped;
+- creates or updates a local QA pipeline;
+- scans Space LLM models, tests candidates, and switches to the first working
+  Space model with tested fallback models;
+- writes `LANGBOT_PIPELINE_URL`, `LANGBOT_PIPELINE_NAME`, and local-agent
+  pipeline/model variables into `skills/.env.local`;
+- returns `env_issue` when no Space model can be scanned or tested.
+
+Useful model controls:
+
+```bash
+LANGBOT_E2E_MODEL_TEST_LIMIT=8
+LANGBOT_E2E_MODEL_FALLBACK_COUNT=3
+LANGBOT_E2E_SKIP_MODEL_UUIDS=uuid-a,uuid-b
+LANGBOT_E2E_SKIP_MODEL_NAMES=model-a,model-b
+LANGBOT_E2E_SCAN_SPACE_MODELS=true
+```
+
+The setup writes a current-runtime compatibility `max-round` value into the
+pipeline config because this backend still reads that field directly during
+message truncation. Do not treat it as a long-term QA contract.
+
+## 5. Run Gates
+
+Fast contract gate, no live service required:
+
+```bash
+bin/lbs suite run langbot-performance-contract-gate --run-id langbot-contract-local
+```
+
+Live backend gate:
+
+```bash
+bin/lbs suite run langbot-live-backend-gate --run-id langbot-backend-local
+```
+
+Browser-visible user-path performance gate:
+
+```bash
+bin/lbs suite plan langbot-user-path-performance-gate
+bin/lbs suite run langbot-user-path-performance-gate --run-id langbot-user-path-local --include-manual-check
+```
+
+Controlled Debug Chat message-path load gate (manual/non-required; run fake-provider cases serially when they share `LANGBOT_FAKE_PROVIDER_URL`):
+
+```bash
+bin/lbs suite plan langbot-debug-chat-load-gate
+bin/lbs test run langbot-fake-provider-debug-chat-load --run-id langbot-fake-load-local
+bin/lbs test run langbot-fake-provider-debug-chat-slow-load --run-id langbot-fake-slow-local
+bin/lbs test run langbot-fake-provider-debug-chat-fault-recovery --run-id langbot-fake-fault-local
+bin/lbs test run langbot-space-debug-chat-concurrency-smoke --run-id langbot-space-smoke-local
+```
+
+Cross-pipeline Debug Chat isolation is a separate manual regression gate because
+current releases may fail it due to product bug #2286:
+
+```bash
+bin/lbs suite plan langbot-debug-chat-isolation-gate
+bin/lbs suite run langbot-debug-chat-isolation-gate --run-id langbot-debug-chat-isolation-local --include-manual-check
+```
+
+Start with `langbot-fake-provider-debug-chat-load`. It launches a local
+OpenAI-compatible fake provider, creates the matching provider/model/pipeline,
+then sends concurrent WebSocket Debug Chat messages through the real backend.
+Use `langbot-fake-provider-debug-chat-slow-load` to measure the same path under
+deterministic streaming latency. Use
+`langbot-fake-provider-debug-chat-fault-recovery` to inject bounded provider
+HTTP failures and confirm later Debug Chat requests recover. Use the separate
+`langbot-debug-chat-isolation-gate` to verify that concurrent Debug Chat traffic
+on two pipelines does not leak assistant responses across pipeline boundaries;
+current releases may fail that gate because of #2286, so keep it out of the
+normal load gate until the product fix lands.
+Use `langbot-space-debug-chat-concurrency-smoke` only as a low-volume live
+provider smoke; it includes Space/model/network latency and should be compared
+against the fake-provider baseline before attributing failures to LangBot.
+
+`manual_check` means the agent must confirm the declared preconditions for that
+run window. When setup automation is declared, run output may stop early with
+`env_issue`; fix that environment input before treating the product path as
+measured.
+
+## 6. Read Results
+
+Suite reports live under `skills/reports/`. Evidence lives under
+`skills/reports/evidence/<run-id>/`.
+
+For performance cases, inspect:
+
+- `metrics.json` for p50/p95/p99, error rate, and total duration;
+- `automation-result.json` for threshold decisions and artifacts;
+- `console.log` and `network.log` for frontend/API failures;
+- backend logs for provider, runner, WebSocket, or persistence failures.
+
+Do not call a user-path performance result a LangBot overhead regression until
+provider/tool/network time has been separated or ruled out.

skills/schemas/case.schema.json

@@ -48,7 +48,18 @@
     },
     "type": {
       "type": "string",
-      "enum": ["smoke", "regression", "feature", "provider", "exploratory"]
+      "enum": [
+        "smoke",
+        "regression",
+        "feature",
+        "provider",
+        "exploratory",
+        "contract",
+        "performance",
+        "reliability",
+        "chaos",
+        "security"
+      ]
     },
     "priority": {
       "type": "string",
@@ -102,7 +113,11 @@
           "backend_log",
           "frontend_log",
           "api_diagnostic",
-          "filesystem"
+          "filesystem",
+          "metrics",
+          "trace",
+          "profile",
+          "resource_log"
         ]
       },
       "minItems": 1
@@ -188,9 +203,101 @@
       "type": "string",
       "enum": ["person", "group"]
     },
+    "automation_debug_chat_response_p95_ms": {
+      "type": "string"
+    },
+    "automation_debug_chat_max_error_rate": {
+      "type": "string"
+    },
+    "automation_debug_chat_load_requests": {
+      "type": "string"
+    },
+    "automation_debug_chat_load_concurrency": {
+      "type": "string"
+    },
+    "automation_debug_chat_load_timeout_ms": {
+      "type": "string"
+    },
+    "automation_debug_chat_load_response_p95_ms": {
+      "type": "string"
+    },
+    "automation_debug_chat_load_first_response_p95_ms": {
+      "type": "string"
+    },
+    "automation_debug_chat_load_max_error_rate": {
+      "type": "string"
+    },
+    "automation_debug_chat_load_min_error_rate": {
+      "type": "string"
+    },
+    "automation_debug_chat_load_min_error_count": {
+      "type": "string"
+    },
+    "automation_debug_chat_load_min_ok_count": {
+      "type": "string"
+    },
+    "automation_debug_chat_load_min_provider_fault_count": {
+      "type": "string"
+    },
+    "automation_debug_chat_load_expected_prefix": {
+      "type": "string"
+    },
+    "automation_debug_chat_load_prompt_template": {
+      "type": "string"
+    },
+    "automation_debug_chat_load_stream": {
+      "type": "string",
+      "enum": ["0", "1", "false", "true"]
+    },
+    "automation_debug_chat_load_reset": {
+      "type": "string",
+      "enum": ["0", "1", "false", "true"]
+    },
+    "automation_debug_chat_load_fail_on_final_mismatch": {
+      "type": "string",
+      "enum": ["0", "1", "false", "true"]
+    },
+    "automation_fake_provider_response_text": {
+      "type": "string"
+    },
+    "automation_fake_provider_first_token_delay_ms": {
+      "type": "string"
+    },
+    "automation_fake_provider_chunk_delay_ms": {
+      "type": "string"
+    },
+    "automation_fake_provider_chunk_count": {
+      "type": "string"
+    },
+    "automation_fake_provider_fail_first_n": {
+      "type": "string"
+    },
+    "automation_fake_provider_fail_every_n": {
+      "type": "string"
+    },
+    "automation_fake_provider_fault_status": {
+      "type": "string"
+    },
+    "automation_fake_provider_fail_after_first_chunk": {
+      "type": "string",
+      "enum": ["0", "1", "false", "true"]
+    },
+    "automation_fake_provider_dynamic_response": {
+      "type": "string",
+      "enum": ["0", "1", "false", "true"]
+    },
     "automation_filesystem_checks_json": {
       "type": "string"
     },
+    "metrics_thresholds_json": {
+      "type": "string"
+    },
+    "load_profile_json": {
+      "type": "string"
+    },
+    "fault_model_json": {
+      "type": "string"
+    },
     "automation_pipeline_url_env": {
       "type": "string",
       "pattern": "^[A-Z][A-Z0-9_]*$"

skills/schemas/suite.schema.json

@@ -18,7 +18,17 @@
     },
     "type": {
       "type": "string",
-      "enum": ["smoke", "regression", "release_gate", "exploratory"]
+      "enum": [
+        "smoke",
+        "regression",
+        "release_gate",
+        "exploratory",
+        "contract",
+        "performance",
+        "reliability",
+        "chaos",
+        "security"
+      ]
     },
     "priority": {
       "type": "string",

skills/scripts/e2e/ensure-fake-provider-cross-pipelines.mjs

@@ -0,0 +1,205 @@
+#!/usr/bin/env node
+
+import { spawn } from "node:child_process";
+import { mkdir, readFile, writeFile } from "node:fs/promises";
+import { dirname, resolve } from "node:path";
+import { env } from "node:process";
+import {
+  appendLine,
+  ensureEvidence,
+  evidencePaths,
+  loadEnvFiles,
+  redact,
+  writeResult,
+} from "./lib/langbot-e2e.mjs";
+
+const caseId = "ensure-fake-provider-cross-pipelines";
+const DEFAULT_PIPELINE_A_NAME = "LangBot QA Fake Provider Debug Chat A";
+const DEFAULT_PIPELINE_B_NAME = "LangBot QA Fake Provider Debug Chat B";
+
+await loadEnvFiles();
+const paths = evidencePaths(caseId);
+await ensureEvidence(paths);
+
+const writeEnv = process.argv.includes("--write-env");
+const envLocalPath = resolve("skills/.env.local");
+const pipelineAName = env.LANGBOT_FAKE_PROVIDER_PIPELINE_A_NAME || DEFAULT_PIPELINE_A_NAME;
+const pipelineBName = env.LANGBOT_FAKE_PROVIDER_PIPELINE_B_NAME || DEFAULT_PIPELINE_B_NAME;
+
+const result = {
+  source: "setup_automation",
+  case_id: caseId,
+  run_id: paths.runId,
+  status: "fail",
+  reason: "",
+  pipeline_a: {
+    name: pipelineAName,
+    id: "",
+    url: "",
+  },
+  pipeline_b: {
+    name: pipelineBName,
+    id: "",
+    url: "",
+  },
+  fake_provider: {
+    url: "",
+    base_url: "",
+    pid: null,
+  },
+  wrote_env: false,
+  evidence: {
+    console_log: paths.consoleLog,
+    automation_result_json: paths.automationResultJson,
+    result_json: paths.resultJson,
+  },
+  evidence_collected: ["api_diagnostic", "filesystem"],
+};
+
+try {
+  console.error(`[langbot-qa] configuring cross-pipeline QA fixtures: pipeline_a=\"${pipelineAName}\", pipeline_b=\"${pipelineBName}\"`);
+  console.error("[langbot-qa] run these fake-provider setup/probe commands serially when they share LANGBOT_FAKE_PROVIDER_URL.");
+  if (pipelineAName === pipelineBName) {
+    throw new Error("LANGBOT_FAKE_PROVIDER_PIPELINE_A_NAME and LANGBOT_FAKE_PROVIDER_PIPELINE_B_NAME must be different.");
+  }
+
+  const setupA = await runPipelineSetup(pipelineAName, "A");
+  const setupB = await runPipelineSetup(pipelineBName, "B");
+  result.pipeline_a = {
+    name: setupA.pipeline_name || pipelineAName,
+    id: setupA.pipeline_id || "",
+    url: setupA.pipeline_url || "",
+  };
+  result.pipeline_b = {
+    name: setupB.pipeline_name || pipelineBName,
+    id: setupB.pipeline_id || "",
+    url: setupB.pipeline_url || "",
+  };
+  result.fake_provider = {
+    url: setupB.fake_provider?.url || setupA.fake_provider?.url || "",
+    base_url: setupB.fake_provider?.base_url || setupA.fake_provider?.base_url || "",
+    pid: setupB.fake_provider?.pid ?? setupA.fake_provider?.pid ?? null,
+  };
+
+  if (!result.pipeline_a.url || !result.pipeline_b.url || !result.fake_provider.url) {
+    throw new Error("Cross-pipeline fake provider setup did not return both pipeline URLs and provider URL.");
+  }
+
+  if (writeEnv) {
+    await upsertEnvLocal(envLocalPath, {
+      LANGBOT_FAKE_PROVIDER_URL: result.fake_provider.url,
+      LANGBOT_FAKE_PROVIDER_BASE_URL: result.fake_provider.base_url,
+      LANGBOT_FAKE_PROVIDER_PID: result.fake_provider.pid ? String(result.fake_provider.pid) : "",
+      LANGBOT_FAKE_PROVIDER_PIPELINE_A_URL: result.pipeline_a.url,
+      LANGBOT_FAKE_PROVIDER_PIPELINE_A_NAME: result.pipeline_a.name,
+      LANGBOT_FAKE_PROVIDER_PIPELINE_B_URL: result.pipeline_b.url,
+      LANGBOT_FAKE_PROVIDER_PIPELINE_B_NAME: result.pipeline_b.name,
+    });
+    result.wrote_env = true;
+  }
+
+  result.status = "pass";
+  result.reason = "Fake provider cross-pipeline fixtures are configured.";
+} catch (error) {
+  result.status = looksLikeEnvIssue(error) ? "env_issue" : "fail";
+  result.reason = safeReason(error.message);
+} finally {
+  await writeResult(paths, result);
+  console.log(JSON.stringify(result, null, 2));
+}
+
+process.exit(result.status === "pass" ? 0 : result.status === "env_issue" ? 2 : 1);
+
+function runPipelineSetup(pipelineName, label) {
+  return new Promise((resolvePromise, rejectPromise) => {
+    const child = spawn(process.execPath, ["scripts/e2e/ensure-fake-provider-pipeline.mjs"], {
+      cwd: resolve("."),
+      env: {
+        ...env,
+        LANGBOT_FAKE_PROVIDER_PIPELINE_NAME: pipelineName,
+        LANGBOT_FAKE_PROVIDER_FIRST_TOKEN_DELAY_MS: env.LANGBOT_FAKE_PROVIDER_FIRST_TOKEN_DELAY_MS || "25",
+        LANGBOT_FAKE_PROVIDER_CHUNK_DELAY_MS: env.LANGBOT_FAKE_PROVIDER_CHUNK_DELAY_MS || "10",
+        LANGBOT_FAKE_PROVIDER_CHUNK_COUNT: env.LANGBOT_FAKE_PROVIDER_CHUNK_COUNT || "0",
+        LANGBOT_FAKE_PROVIDER_FAIL_FIRST_N: "0",
+        LANGBOT_FAKE_PROVIDER_FAIL_EVERY_N: "0",
+        LANGBOT_FAKE_PROVIDER_FAULT_STATUS: env.LANGBOT_FAKE_PROVIDER_FAULT_STATUS || "500",
+        LANGBOT_FAKE_PROVIDER_FAIL_AFTER_FIRST_CHUNK: "false",
+        LANGBOT_FAKE_PROVIDER_DYNAMIC_RESPONSE: "true",
+      },
+      stdio: ["ignore", "pipe", "pipe"],
+    });
+
+    let stdout = "";
+    let stderr = "";
+    child.stdout.on("data", (chunk) => {
+      const text = chunk.toString();
+      stdout += text;
+      appendLine(paths.consoleLog, `[setup ${label} stdout] ${text.trimEnd()}`).catch(() => {});
+    });
+    child.stderr.on("data", (chunk) => {
+      const text = chunk.toString();
+      stderr += text;
+      appendLine(paths.consoleLog, `[setup ${label} stderr] ${text.trimEnd()}`).catch(() => {});
+    });
+    child.on("error", rejectPromise);
+    child.on("close", (code) => {
+      const parsed = parseJsonOutput(stdout);
+      if (code !== 0 || parsed.status !== "pass") {
+        rejectPromise(new Error(parsed.reason || stderr || `Fake provider pipeline setup ${label} exited with ${code}.`));
+        return;
+      }
+      resolvePromise(parsed);
+    });
+  });
+}
+
+function parseJsonOutput(text) {
+  const trimmed = String(text || "").trim();
+  if (!trimmed) return {};
+  try {
+    return JSON.parse(trimmed);
+  } catch {
+    const start = trimmed.indexOf("{");
+    const end = trimmed.lastIndexOf("}");
+    if (start >= 0 && end > start) {
+      try {
+        return JSON.parse(trimmed.slice(start, end + 1));
+      } catch {
+        return {};
+      }
+    }
+    return {};
+  }
+}
+
+async function upsertEnvLocal(path, updates) {
+  await mkdir(dirname(path), { recursive: true });
+  let text = "";
+  try {
+    text = await readFile(path, "utf8");
+  } catch {
+    text = "";
+  }
+  const lines = text.split(/\r?\n/);
+  const seen = new Set();
+  const next = lines.map((line) => {
+    const trimmed = line.trim();
+    const match = trimmed.match(/^([A-Z][A-Z0-9_]*)=/);
+    if (!match || updates[match[1]] === undefined) return line;
+    seen.add(match[1]);
+    return `${match[1]}=${updates[match[1]]}`;
+  });
+  for (const [key, value] of Object.entries(updates)) {
+    if (!seen.has(key)) next.push(`${key}=${value}`);
+  }
+  await writeFile(path, `${next.join("\n").replace(/\n+$/, "")}\n`, "utf8");
+}
+
+function looksLikeEnvIssue(error) {
+  const message = String(error?.message || error || "");
+  return /fetch failed|ECONNREFUSED|ENOTFOUND|LANGBOT_.*not configured|Could not read recovery_key|Backend did not respond/i.test(message);
+}
+
+function safeReason(value) {
+  return redact(String(value || "")).slice(0, 1000);
+}

skills/scripts/e2e/ensure-fake-provider-pipeline.mjs

@@ -0,0 +1,635 @@
+#!/usr/bin/env node
+
+import { spawn } from "node:child_process";
+import { open, readFile, mkdir, writeFile } from "node:fs/promises";
+import { dirname, resolve } from "node:path";
+import { env } from "node:process";
+import {
+  apiJson,
+  ensureEvidence,
+  evidencePaths,
+  loadEnvFiles,
+  redact,
+  resetAndAuthLocalUser,
+  writeResult,
+} from "./lib/langbot-e2e.mjs";
+
+const RUNNER_ID = "local-agent";
+const DEFAULT_LOCAL_PASSWORD = "LangBotE2ELocalPass!2026";
+const DEFAULT_PIPELINE_NAME = "LangBot QA Fake Provider Debug Chat";
+const DEFAULT_PROVIDER_NAME = "LangBot QA Fake OpenAI Provider";
+const QA_RESOURCE_DESCRIPTION = "Managed by LangBot skills QA automation for controlled fake-provider Debug Chat tests. Safe to delete when local QA fixtures are no longer needed.";
+const DEFAULT_MODEL_NAME = "gpt-4o-mini";
+const DEFAULT_REQUESTER = "openai-chat-completions";
+
+const caseId = "ensure-fake-provider-pipeline";
+
+await loadEnvFiles();
+const paths = evidencePaths(caseId);
+await ensureEvidence(paths);
+
+const writeEnv = process.argv.includes("--write-env");
+const frontendUrl = env.LANGBOT_FRONTEND_URL || "";
+const backendUrl = env.LANGBOT_BACKEND_URL || "";
+const envLocalPath = resolve("skills/.env.local");
+const repoRoot = resolve(env.LANGBOT_REPO || "..");
+const fakeStateDir = resolve(env.LANGBOT_FAKE_PROVIDER_STATE_DIR || resolve(repoRoot, ".qa/fake-provider"));
+const fakeStatePath = resolve(fakeStateDir, "state.json");
+const fakeStdoutPath = resolve(fakeStateDir, "fake-provider.stdout.log");
+const fakeStderrPath = resolve(fakeStateDir, "fake-provider.stderr.log");
+const pipelineName = env.LANGBOT_FAKE_PROVIDER_PIPELINE_NAME || DEFAULT_PIPELINE_NAME;
+const providerName = env.LANGBOT_FAKE_PROVIDER_NAME || DEFAULT_PROVIDER_NAME;
+const requester = env.LANGBOT_FAKE_PROVIDER_REQUESTER || DEFAULT_REQUESTER;
+const modelName = env.LANGBOT_FAKE_PROVIDER_MODEL_NAME || DEFAULT_MODEL_NAME;
+
+const result = {
+  source: "automation",
+  case_id: caseId,
+  run_id: paths.runId,
+  status: "fail",
+  reason: "",
+  frontend_url: frontendUrl,
+  backend_url: backendUrl,
+  fake_provider: {
+    url: "",
+    base_url: "",
+    pid: null,
+    reused: false,
+    config: {},
+    state_file: fakeStatePath,
+    stdout_log: fakeStdoutPath,
+    stderr_log: fakeStderrPath,
+  },
+  provider: {
+    uuid: "",
+    name: providerName,
+    requester,
+    created: false,
+    updated: false,
+  },
+  model: {
+    uuid: "",
+    name: modelName,
+    created: false,
+    updated: false,
+    test_status: "not_run",
+    test_reason: "",
+  },
+  pipeline_id: "",
+  pipeline_name: pipelineName,
+  pipeline_url: "",
+  created: false,
+  updated: false,
+  wrote_env: false,
+  evidence: {
+    console_log: paths.consoleLog,
+    network_log: paths.networkLog,
+    automation_result_json: paths.automationResultJson,
+    result_json: paths.resultJson,
+  },
+  evidence_collected: ["api_diagnostic", "network", "filesystem"],
+};
+
+try {
+  console.error(`[langbot-qa] configuring QA-owned fake-provider fixtures: provider=\"${providerName}\", pipeline=\"${pipelineName}\"`);
+  console.error("[langbot-qa] this setup may create or update local QA provider/model/pipeline resources on the selected backend.");
+  if (!backendUrl) {
+    result.status = "env_issue";
+    throw new Error("LANGBOT_BACKEND_URL is not configured.");
+  }
+  if (!frontendUrl) {
+    result.status = "env_issue";
+    throw new Error("LANGBOT_FRONTEND_URL is not configured.");
+  }
+
+  const fakeProvider = await ensureFakeProvider();
+  const setupConfig = await configureFakeProvider(fakeProvider.url, healthyFakeProviderConfig(), true);
+  result.fake_provider = {
+    ...result.fake_provider,
+    ...fakeProvider,
+    config: setupConfig.config || healthyFakeProviderConfig(),
+  };
+
+  const user = env.LANGBOT_E2E_LOGIN_USER || "";
+  const password = env.LANGBOT_E2E_LOGIN_PASSWORD || DEFAULT_LOCAL_PASSWORD;
+  if (!user) {
+    result.status = "env_issue";
+    throw new Error("LANGBOT_E2E_LOGIN_USER is required so this setup can create/update the fake provider pipeline.");
+  }
+
+  const auth = await resetAndAuthLocalUser({ backendUrl, user, password });
+  const wizard = await skipWizard({ backendUrl, token: auth.token });
+  if (wizard.status !== "pass") {
+    result.status = "fail";
+    throw new Error(wizard.reason || "Failed to mark the local QA wizard as skipped.");
+  }
+
+  const provider = await ensureProvider({
+    backendUrl,
+    token: auth.token,
+    name: providerName,
+    requester,
+    baseUrl: fakeProvider.base_url,
+  });
+  result.provider = provider;
+
+  const model = await ensureModel({
+    backendUrl,
+    token: auth.token,
+    providerUuid: provider.uuid,
+    name: modelName,
+  });
+  result.model = model;
+
+  const pipeline = await ensurePipeline({
+    backendUrl,
+    token: auth.token,
+    name: pipelineName,
+    modelUuid: model.uuid,
+  });
+  Object.assign(result, pipeline);
+  result.pipeline_url = `${frontendUrl.replace(/\/$/, "")}/home/pipelines?id=${encodeURIComponent(pipeline.pipeline_id)}`;
+
+  const runConfig = await configureFakeProvider(fakeProvider.url, targetFakeProviderConfig(), true);
+  result.fake_provider.config = runConfig.config || targetFakeProviderConfig();
+
+  if (writeEnv) {
+    await upsertEnvLocal(envLocalPath, {
+      LANGBOT_E2E_LOGIN_USER: user,
+      LANGBOT_FAKE_PROVIDER_URL: fakeProvider.url,
+      LANGBOT_FAKE_PROVIDER_BASE_URL: fakeProvider.base_url,
+      LANGBOT_FAKE_PROVIDER_PID: fakeProvider.pid ? String(fakeProvider.pid) : "",
+      LANGBOT_FAKE_PROVIDER_PROVIDER_UUID: provider.uuid,
+      LANGBOT_FAKE_PROVIDER_MODEL_UUID: model.uuid,
+      LANGBOT_FAKE_PROVIDER_PIPELINE_URL: result.pipeline_url,
+      LANGBOT_FAKE_PROVIDER_PIPELINE_NAME: pipelineName,
+    });
+    result.wrote_env = true;
+  }
+
+  result.status = "pass";
+  result.reason = `Fake provider pipeline is configured with ${requester}/${modelName}.`;
+} catch (error) {
+  result.status = result.status === "env_issue" ? "env_issue" : "fail";
+  result.reason = result.reason || safeReason(error.message);
+} finally {
+  await writeResult(paths, result);
+  console.log(JSON.stringify(result, null, 2));
+}
+
+process.exit(result.status === "pass" ? 0 : result.status === "env_issue" ? 2 : 1);
+
+async function ensureFakeProvider() {
+  const envUrl = normalizeProviderRootUrl(env.LANGBOT_FAKE_PROVIDER_URL || "");
+  if (envUrl && await fakeProviderHealthy(envUrl) && await fakeProviderConfigurable(envUrl)) {
+    return {
+      url: envUrl,
+      base_url: `${envUrl}/v1`,
+      pid: null,
+      reused: true,
+    };
+  }
+
+  const state = await readState(fakeStatePath);
+  const stateUrl = normalizeProviderRootUrl(state.url || "");
+  if (stateUrl && await fakeProviderHealthy(stateUrl)) {
+    if (await fakeProviderConfigurable(stateUrl)) {
+      return {
+        url: stateUrl,
+        base_url: state.base_url || `${stateUrl}/v1`,
+        pid: Number.isInteger(state.pid) ? state.pid : null,
+        reused: true,
+      };
+    }
+    if (Number.isInteger(state.pid)) await stopProcess(state.pid);
+  }
+
+  await mkdir(fakeStateDir, { recursive: true });
+  await writeFile(fakeStatePath, `${JSON.stringify({ status: "starting", started_at: new Date().toISOString() }, null, 2)}\n`, "utf8");
+  const stdout = await open(fakeStdoutPath, "a");
+  const stderr = await open(fakeStderrPath, "a");
+  const scriptPath = resolve("scripts/e2e/fake-openai-provider.mjs");
+  const host = env.LANGBOT_FAKE_PROVIDER_HOST || "127.0.0.1";
+  const port = env.LANGBOT_FAKE_PROVIDER_PORT || "0";
+  const child = spawn(process.execPath, [
+    scriptPath,
+    `--host=${host}`,
+    `--port=${port}`,
+    `--state-file=${fakeStatePath}`,
+  ], {
+    cwd: resolve("."),
+    detached: true,
+    env: {
+      ...env,
+      LANGBOT_FAKE_PROVIDER_MODEL_NAME: modelName,
+    },
+    stdio: ["ignore", stdout.fd, stderr.fd],
+  });
+  child.unref();
+  await stdout.close();
+  await stderr.close();
+
+  const started = await waitForFakeProviderState(fakeStatePath, child.pid, 10_000);
+  if (!started.url || !await fakeProviderHealthy(started.url) || !await fakeProviderConfigurable(started.url)) {
+    throw new Error(`Fake provider did not become healthy. See ${fakeStderrPath}`);
+  }
+
+  return {
+    url: started.url,
+    base_url: started.base_url || `${started.url}/v1`,
+    pid: child.pid ?? started.pid ?? null,
+    reused: false,
+  };
+}
+
+async function configureFakeProvider(rootUrl, config, resetRequestCount) {
+  const response = await fetch(`${normalizeProviderRootUrl(rootUrl)}/__qa/config`, {
+    method: "POST",
+    headers: { "content-type": "application/json" },
+    body: JSON.stringify({
+      config,
+      reset_request_count: resetRequestCount,
+    }),
+    signal: AbortSignal.timeout(3000),
+  });
+  const json = await response.json().catch(() => ({}));
+  if (!response.ok || json.ok !== true) {
+    throw new Error(`Fake provider config failed with HTTP ${response.status}.`);
+  }
+  return json;
+}
+
+async function fakeProviderHealthy(rootUrl) {
+  try {
+    const response = await fetch(`${rootUrl.replace(/\/$/, "")}/healthz`, {
+      signal: AbortSignal.timeout(2000),
+    });
+    if (!response.ok) return false;
+    const json = await response.json().catch(() => ({}));
+    return json.ok === true;
+  } catch {
+    return false;
+  }
+}
+
+async function fakeProviderConfigurable(rootUrl) {
+  try {
+    const response = await fetch(`${rootUrl.replace(/\/$/, "")}/__qa/config`, {
+      signal: AbortSignal.timeout(2000),
+    });
+    if (!response.ok) return false;
+    const json = await response.json().catch(() => ({}));
+    return json.ok === true && json.config && typeof json.config === "object";
+  } catch {
+    return false;
+  }
+}
+
+async function stopProcess(pid) {
+  try {
+    process.kill(pid, "SIGTERM");
+  } catch {
+    return;
+  }
+  await sleep(500);
+}
+
+async function waitForFakeProviderState(path, expectedPid, timeoutMs) {
+  const startedAt = Date.now();
+  let lastState = {};
+  while (Date.now() - startedAt < timeoutMs) {
+    const state = await readState(path);
+    if (state.url && (!expectedPid || state.pid === expectedPid)) return state;
+    lastState = state;
+    await sleep(150);
+  }
+  return lastState;
+}
+
+async function readState(path) {
+  try {
+    return JSON.parse(await readFile(path, "utf8"));
+  } catch {
+    return {};
+  }
+}
+
+function normalizeProviderRootUrl(value) {
+  const trimmed = String(value || "").trim().replace(/\/$/, "");
+  return trimmed.endsWith("/v1") ? trimmed.slice(0, -3) : trimmed;
+}
+
+function healthyFakeProviderConfig() {
+  return {
+    response_text: "OK",
+    first_token_delay_ms: 25,
+    chunk_delay_ms: 10,
+    chunk_count: 0,
+    fault_status: 500,
+    fail_first_n: 0,
+    fail_every_n: 0,
+    fail_after_first_chunk: false,
+    dynamic_response: true,
+  };
+}
+
+function targetFakeProviderConfig() {
+  return {
+    response_text: env.LANGBOT_FAKE_PROVIDER_RESPONSE_TEXT || "OK",
+    first_token_delay_ms: nonNegativeInteger(env.LANGBOT_FAKE_PROVIDER_FIRST_TOKEN_DELAY_MS, 25),
+    chunk_delay_ms: nonNegativeInteger(env.LANGBOT_FAKE_PROVIDER_CHUNK_DELAY_MS, 10),
+    chunk_count: nonNegativeInteger(env.LANGBOT_FAKE_PROVIDER_CHUNK_COUNT, 0),
+    fault_status: httpFaultStatus(env.LANGBOT_FAKE_PROVIDER_FAULT_STATUS, 500),
+    fail_first_n: nonNegativeInteger(env.LANGBOT_FAKE_PROVIDER_FAIL_FIRST_N, 0),
+    fail_every_n: nonNegativeInteger(env.LANGBOT_FAKE_PROVIDER_FAIL_EVERY_N, 0),
+    fail_after_first_chunk: envBool(env.LANGBOT_FAKE_PROVIDER_FAIL_AFTER_FIRST_CHUNK, false),
+    dynamic_response: envBool(env.LANGBOT_FAKE_PROVIDER_DYNAMIC_RESPONSE, true),
+  };
+}
+
+async function skipWizard({ backendUrl, token }) {
+  const response = await apiJson(backendUrl, "/api/v1/system/wizard/completed", {
+    method: "POST",
+    token,
+    body: { status: "skipped" },
+  });
+  const ok = response.status < 400 && response.json.code === 0;
+  return {
+    status: ok ? "pass" : "fail",
+    http_status: response.status,
+    code: response.json.code ?? null,
+    reason: ok ? "Wizard marked skipped for local QA." : response.json.msg || "Wizard status update failed.",
+  };
+}
+
+async function ensureProvider({ backendUrl, token, name, requester, baseUrl }) {
+  const list = await apiJson(backendUrl, "/api/v1/provider/providers", { token });
+  if (isApiFailure(list)) {
+    throw new Error(list.json.msg || "Failed to list providers.");
+  }
+  const providers = list.json.data?.providers || [];
+  const existing = providers.find((provider) => (
+    provider.name === name
+      || (provider.requester === requester && String(provider.base_url || "").replace(/\/$/, "") === baseUrl.replace(/\/$/, ""))
+  ));
+  const body = {
+    name,
+    requester,
+    base_url: baseUrl,
+    api_keys: [env.LANGBOT_FAKE_PROVIDER_API_KEY || "langbot-fake-provider-key"],
+  };
+
+  if (existing?.uuid) {
+    const update = await apiJson(backendUrl, `/api/v1/provider/providers/${encodeURIComponent(existing.uuid)}`, {
+      method: "PUT",
+      token,
+      body,
+    });
+    if (isApiFailure(update)) {
+      throw new Error(update.json.msg || "Failed to update fake provider.");
+    }
+    return {
+      uuid: existing.uuid,
+      name,
+      requester,
+      created: false,
+      updated: true,
+    };
+  }
+
+  const create = await apiJson(backendUrl, "/api/v1/provider/providers", {
+    method: "POST",
+    token,
+    body,
+  });
+  const uuid = create.json.data?.uuid || "";
+  if (isApiFailure(create) || !uuid) {
+    throw new Error(create.json.msg || "Failed to create fake provider.");
+  }
+  return {
+    uuid,
+    name,
+    requester,
+    created: true,
+    updated: false,
+  };
+}
+
+async function ensureModel({ backendUrl, token, providerUuid, name }) {
+  const list = await apiJson(backendUrl, `/api/v1/provider/models/llm?provider_uuid=${encodeURIComponent(providerUuid)}`, { token });
+  if (isApiFailure(list)) {
+    throw new Error(list.json.msg || "Failed to list fake provider models.");
+  }
+  const models = list.json.data?.models || [];
+  const existing = models.find((model) => model.name === name);
+  const body = {
+    name,
+    provider_uuid: providerUuid,
+    abilities: [],
+    context_length: positiveInteger(env.LANGBOT_FAKE_PROVIDER_CONTEXT_LENGTH, 8192),
+    extra_args: {},
+    prefered_ranking: 0,
+  };
+  let modelUuid = existing?.uuid || "";
+  let created = false;
+  let updated = false;
+
+  if (modelUuid) {
+    const update = await apiJson(backendUrl, `/api/v1/provider/models/llm/${encodeURIComponent(modelUuid)}`, {
+      method: "PUT",
+      token,
+      body,
+    });
+    if (isApiFailure(update)) {
+      throw new Error(update.json.msg || "Failed to update fake provider model.");
+    }
+    updated = true;
+  } else {
+    const create = await apiJson(backendUrl, "/api/v1/provider/models/llm", {
+      method: "POST",
+      token,
+      body,
+    });
+    modelUuid = create.json.data?.uuid || "";
+    if (isApiFailure(create) || !modelUuid) {
+      throw new Error(create.json.msg || "Failed to create fake provider model.");
+    }
+    created = true;
+  }
+
+  const test = await apiJson(backendUrl, `/api/v1/provider/models/llm/${encodeURIComponent(modelUuid)}/test`, {
+    method: "POST",
+    token,
+    body: { extra_args: {} },
+  });
+  if (isApiFailure(test)) {
+    throw new Error(safeReason(test.json.msg || test.json.message || "Fake provider model test failed."));
+  }
+
+  return {
+    uuid: modelUuid,
+    name,
+    created,
+    updated,
+    test_status: "pass",
+    test_reason: "",
+  };
+}
+
+async function ensurePipeline({ backendUrl, token, name, modelUuid }) {
+  const list = await apiJson(backendUrl, "/api/v1/pipelines", { token });
+  if (isApiFailure(list)) {
+    throw new Error(list.json.msg || "Failed to list pipelines.");
+  }
+  const pipelines = list.json.data?.pipelines || [];
+  let pipeline = pipelines.find((item) => item.name === name) || null;
+  let created = false;
+
+  if (!pipeline) {
+    const create = await apiJson(backendUrl, "/api/v1/pipelines", {
+      method: "POST",
+      token,
+      body: {
+        name,
+        description: QA_RESOURCE_DESCRIPTION,
+        emoji: "QA",
+      },
+    });
+    const pipelineId = create.json.data?.uuid || "";
+    if (isApiFailure(create) || !pipelineId) {
+      throw new Error(create.json.msg || "Failed to create fake provider pipeline.");
+    }
+    created = true;
+    pipeline = { uuid: pipelineId };
+  }
+
+  const loaded = await apiJson(backendUrl, `/api/v1/pipelines/${encodeURIComponent(pipeline.uuid)}`, { token });
+  pipeline = loaded.json.data?.pipeline || null;
+  if (isApiFailure(loaded) || !pipeline?.uuid) {
+    throw new Error(loaded.json.msg || "Failed to load fake provider pipeline.");
+  }
+
+  const config = pipeline.config && typeof pipeline.config === "object" ? pipeline.config : {};
+  const ai = config.ai && typeof config.ai === "object" ? config.ai : {};
+  const existingLocalAgentConfig = ai["local-agent"] && typeof ai["local-agent"] === "object"
+    ? ai["local-agent"]
+    : {};
+  const localAgentConfig = {
+    timeout: 60,
+    prompt: [{ role: "system", content: "You are a deterministic QA assistant. Reply exactly as instructed." }],
+    "remove-think": false,
+    "knowledge-bases": [],
+    "box-session-id-template": "{launcher_type}_{launcher_id}",
+    "retrieval-top-k": 5,
+    "rerank-model": "",
+    "rerank-top-k": 5,
+    "max-tool-iterations": 20,
+    "tool-execution-mode": "parallel",
+    "max-tool-result-chars": 20000,
+    "context-history-fetch-limit": 20,
+    "context-window-tokens": 8192,
+    "context-reserve-tokens": 1024,
+    "context-keep-recent-tokens": 2048,
+    "context-summary-tokens": 1024,
+    ...existingLocalAgentConfig,
+    // Current backend truncation still reads this field directly.
+    "max-round": positiveInteger(existingLocalAgentConfig["max-round"], 10),
+    model: {
+      primary: modelUuid,
+      fallbacks: [],
+    },
+  };
+  const updatedConfig = {
+    ...config,
+    ai: {
+      ...ai,
+      runner: {
+        ...(ai.runner && typeof ai.runner === "object" ? ai.runner : {}),
+        id: RUNNER_ID,
+        runner: RUNNER_ID,
+        "expire-time": 0,
+      },
+      "local-agent": localAgentConfig,
+    },
+  };
+
+  const update = await apiJson(backendUrl, `/api/v1/pipelines/${encodeURIComponent(pipeline.uuid)}`, {
+    method: "PUT",
+    token,
+    body: {
+      name,
+      description: QA_RESOURCE_DESCRIPTION,
+      emoji: "QA",
+      config: updatedConfig,
+    },
+  });
+  if (isApiFailure(update)) {
+    throw new Error(update.json.msg || "Failed to update fake provider pipeline.");
+  }
+
+  return {
+    pipeline_id: pipeline.uuid,
+    pipeline_name: name,
+    created,
+    updated: true,
+  };
+}
+
+function isApiFailure(response) {
+  return response.status >= 400 || (response.json.code !== undefined && response.json.code !== 0);
+}
+
+function positiveInteger(value, fallback) {
+  const parsed = Number(value);
+  return Number.isInteger(parsed) && parsed > 0 ? parsed : fallback;
+}
+
+function nonNegativeInteger(value, fallback) {
+  const parsed = Number(value);
+  return Number.isInteger(parsed) && parsed >= 0 ? parsed : fallback;
+}
+
+function httpFaultStatus(value, fallback) {
+  const parsed = Number(value);
+  return Number.isInteger(parsed) && parsed >= 400 && parsed <= 599 ? parsed : fallback;
+}
+
+function envBool(value, fallback) {
+  if (value === undefined || value === "") return fallback;
+  if (/^(1|true|yes|on)$/i.test(String(value))) return true;
+  if (/^(0|false|no|off)$/i.test(String(value))) return false;
+  return fallback;
+}
+
+function sleep(ms) {
+  return new Promise((resolve) => setTimeout(resolve, ms));
+}
+
+function safeReason(value) {
+  return redact(String(value || "")).slice(0, 1000);
+}
+
+async function upsertEnvLocal(path, updates) {
+  await mkdir(dirname(path), { recursive: true });
+  let text = "";
+  try {
+    text = await readFile(path, "utf8");
+  } catch {
+    text = "";
+  }
+  const lines = text.split(/\r?\n/);
+  const seen = new Set();
+  const next = lines.map((line) => {
+    const trimmed = line.trim();
+    const equals = trimmed.indexOf("=");
+    if (equals <= 0 || trimmed.startsWith("#")) return line;
+    const key = trimmed.slice(0, equals).trim();
+    if (!(key in updates)) return line;
+    seen.add(key);
+    return `${key}=${updates[key]}`;
+  });
+  for (const [key, value] of Object.entries(updates)) {
+    if (!seen.has(key)) next.push(`${key}=${value}`);
+  }
+  await writeFile(path, `${next.filter((line, index) => line !== "" || index < next.length - 1).join("\n")}\n`, "utf8");
+}

skills/scripts/e2e/ensure-local-agent-pipeline.mjs

@@ -10,6 +10,7 @@ import {
   ensureEvidence,
   evidencePaths,
   loadEnvFiles,
+  redact,
   resetAndAuthLocalUser,
   safeScreenshot,
   setBrowserToken,
@@ -17,9 +18,12 @@ import {
   writeResult,
 } from "./lib/langbot-e2e.mjs";
 
-const RUNNER_ID = "plugin:langbot/local-agent/default";
+const RUNNER_ID = "local-agent";
+const SPACE_PROVIDER_UUID = "00000000-0000-0000-0000-000000000000";
 const DEFAULT_PIPELINE_NAME = "Agent QA Local Agent Debug Chat";
 const DEFAULT_LOCAL_PASSWORD = "LangBotE2ELocalPass!2026";
+const DEFAULT_MODEL_TEST_LIMIT = 8;
+const DEFAULT_MODEL_FALLBACK_COUNT = 3;
 const caseId = "ensure-local-agent-pipeline";
 
 await loadEnvFiles();
@@ -45,11 +49,18 @@ const result = {
   pipeline_url: "",
   runner_id: RUNNER_ID,
   selected_model_id: "",
+  selected_model_name: "",
+  fallback_model_ids: [],
   model_count: 0,
+  space_model_count: 0,
+  scanned_space_model_count: 0,
+  tested_model_count: 0,
+  model_tests: [],
   created: false,
   updated: false,
   wrote_env: false,
   auth: null,
+  wizard: null,
   browser_token_check: null,
   page_signal: "",
   evidence: {
@@ -71,6 +82,7 @@ try {
   const user = env.LANGBOT_E2E_LOGIN_USER || "";
   const password = env.LANGBOT_E2E_LOGIN_PASSWORD || DEFAULT_LOCAL_PASSWORD;
   if (!user) {
+    result.status = "env_issue";
     throw new Error("LANGBOT_E2E_LOGIN_USER is required so this setup can create/update the pipeline via backend API.");
   }
 
@@ -81,6 +93,13 @@ try {
     backend_token_check: auth.check,
   };
 
+  const wizard = await skipWizard({ backendUrl, token: auth.token });
+  result.wizard = wizard;
+  if (wizard.status !== "pass") {
+    result.status = "fail";
+    throw new Error(wizard.reason || "Failed to mark the local QA wizard as skipped.");
+  }
+
   const prepared = await ensureLocalAgentPipeline({
     backendUrl,
     token: auth.token,
@@ -99,6 +118,10 @@ try {
       LANGBOT_PIPELINE_NAME: result.pipeline_name || pipelineName,
       LANGBOT_LOCAL_AGENT_PIPELINE_URL: result.pipeline_url,
       LANGBOT_LOCAL_AGENT_PIPELINE_NAME: result.pipeline_name || pipelineName,
+      ...(result.selected_model_id ? {
+        LANGBOT_LOCAL_AGENT_MODEL_UUID: result.selected_model_id,
+        LANGBOT_E2E_MODEL_UUID: result.selected_model_id,
+      } : {}),
     });
     result.wrote_env = true;
   }
@@ -127,6 +150,21 @@ try {
 
 process.exit(result.status === "pass" ? 0 : result.status === "env_issue" ? 2 : 1);
 
+async function skipWizard({ backendUrl, token }) {
+  const response = await apiJson(backendUrl, "/api/v1/system/wizard/completed", {
+    method: "POST",
+    token,
+    body: { status: "skipped" },
+  });
+  const ok = response.status < 400 && response.json.code === 0;
+  return {
+    status: ok ? "pass" : "fail",
+    http_status: response.status,
+    code: response.json.code ?? null,
+    reason: ok ? "Wizard marked skipped for local QA." : response.json.msg || "Wizard status update failed.",
+  };
+}
+
 async function ensureLocalAgentPipeline({ backendUrl, token, pipelineName, runnerId }) {
   const [pipelineList, modelList] = await Promise.all([
     apiJson(backendUrl, "/api/v1/pipelines", { token }),
@@ -149,7 +187,19 @@ async function ensureLocalAgentPipeline({ backendUrl, token, pipelineName, runne
   }
 
   const models = modelList.json.data?.models || [];
-  const selectedModel = models.find((model) => model.uuid) || null;
+  const skippedModelIds = new Set(
+    String(env.LANGBOT_E2E_SKIP_MODEL_UUIDS || "")
+      .split(",")
+      .map((item) => item.trim())
+      .filter(Boolean),
+  );
+  const skippedModelNames = new Set(
+    String(env.LANGBOT_E2E_SKIP_MODEL_NAMES || "")
+      .split(",")
+      .map((item) => item.trim())
+      .filter(Boolean),
+  );
+  const spaceModels = models.filter((model) => isSpaceModel(model) && !skippedModelIds.has(model.uuid));
   const pipelines = pipelineList.json.data?.pipelines || [];
   let pipeline = pipelines.find((item) => item.name === pipelineName) || null;
   let created = false;
@@ -170,6 +220,7 @@ async function ensureLocalAgentPipeline({ backendUrl, token, pipelineName, runne
         reason: createdResponse.json.msg || "Failed to create pipeline.",
         create_status: createdResponse.status,
         model_count: models.length,
+        space_model_count: spaceModels.length,
       };
     }
     const pipelineId = createdResponse.json.data?.uuid || "";
@@ -183,6 +234,7 @@ async function ensureLocalAgentPipeline({ backendUrl, token, pipelineName, runne
       status: "fail",
       reason: "Pipeline was not created or resolved.",
       model_count: models.length,
+      space_model_count: spaceModels.length,
     };
   }
 
@@ -194,27 +246,37 @@ async function ensureLocalAgentPipeline({ backendUrl, token, pipelineName, runne
       get_status: loaded.status,
       pipeline_id: pipeline.uuid,
       model_count: models.length,
+      space_model_count: spaceModels.length,
     };
   }
   pipeline = loaded.json.data.pipeline;
 
   const config = pipeline.config && typeof pipeline.config === "object" ? pipeline.config : {};
   const ai = config.ai && typeof config.ai === "object" ? config.ai : {};
-  const runnerConfig = ai.runner_config && typeof ai.runner_config === "object" ? ai.runner_config : {};
-  const rawExistingLocalAgentConfig = runnerConfig[runnerId] && typeof runnerConfig[runnerId] === "object"
-    ? runnerConfig[runnerId]
+  const rawExistingLocalAgentConfig = ai["local-agent"] && typeof ai["local-agent"] === "object"
+    ? ai["local-agent"]
     : {};
   const existingLocalAgentConfig = rawExistingLocalAgentConfig;
   const existingModel = existingLocalAgentConfig.model && typeof existingLocalAgentConfig.model === "object"
     ? existingLocalAgentConfig.model
     : {};
   const requestedModelId = env.LANGBOT_LOCAL_AGENT_MODEL_UUID || env.LANGBOT_E2E_MODEL_UUID || "";
-  const selectedModelId = requestedModelId || existingModel.primary || selectedModel?.uuid || "";
+  const selected = await selectWorkingSpaceModel({
+    backendUrl,
+    token,
+    models,
+    skippedModelIds,
+    skippedModelNames,
+    requestedModelId,
+    existingModelId: existingModel.primary || "",
+  });
+  const selectedModelId = selected.selected_model_id || "";
   const localAgentConfig = {
     timeout: 300,
     prompt: [{ role: "system", content: "You are a helpful assistant." }],
     "remove-think": false,
     "knowledge-bases": [],
+    "box-session-id-template": "{launcher_type}_{launcher_id}",
     "retrieval-top-k": 5,
     "rerank-model": "",
     "rerank-top-k": 5,
@@ -227,9 +289,11 @@ async function ensureLocalAgentPipeline({ backendUrl, token, pipelineName, runne
     "context-keep-recent-tokens": 20000,
     "context-summary-tokens": 8000,
     ...existingLocalAgentConfig,
+    // Current backend truncation still reads this field directly.
+    "max-round": positiveInteger(existingLocalAgentConfig["max-round"], 10),
     model: {
       primary: selectedModelId,
-      fallbacks: requestedModelId ? [] : Array.isArray(existingModel.fallbacks) ? existingModel.fallbacks : [],
+      fallbacks: selected.fallback_model_ids || [],
     },
   };
   const updatedConfig = {
@@ -239,12 +303,10 @@ async function ensureLocalAgentPipeline({ backendUrl, token, pipelineName, runne
       runner: {
         ...(ai.runner && typeof ai.runner === "object" ? ai.runner : {}),
         id: runnerId,
+        runner: runnerId,
         "expire-time": 0,
       },
-      runner_config: {
-        ...runnerConfig,
-        [runnerId]: localAgentConfig,
-      },
+      "local-agent": localAgentConfig,
     },
   };
 
@@ -265,19 +327,31 @@ async function ensureLocalAgentPipeline({ backendUrl, token, pipelineName, runne
       update_status: updateResponse.status,
       pipeline_id: pipeline.uuid,
       model_count: models.length,
+      space_model_count: spaceModels.length,
+      scanned_space_model_count: selected.scanned_space_model_count,
+      tested_model_count: selected.tested_model_count,
+      model_tests: selected.model_tests,
       selected_model_id: selectedModelId,
+      selected_model_name: selected.selected_model_name,
+      fallback_model_ids: selected.fallback_model_ids,
     };
   }
 
   return {
     status: selectedModelId ? "pass" : "env_issue",
     reason: selectedModelId
-      ? "Local-agent pipeline is configured for Debug Chat."
-      : "Pipeline was created but no LLM model is configured in this LangBot instance.",
+      ? `Local-agent pipeline is configured for Debug Chat with Space model ${selected.selected_model_name || selectedModelId} and ${selected.fallback_model_ids.length} fallback(s).`
+      : selected.reason || "No working Space LLM model is configured in this LangBot instance.",
     pipeline_id: pipeline.uuid,
-    pipeline_name: pipeline.name,
+    pipeline_name: pipelineName,
     model_count: models.length,
+    space_model_count: spaceModels.length,
+    scanned_space_model_count: selected.scanned_space_model_count,
+    tested_model_count: selected.tested_model_count,
+    model_tests: selected.model_tests,
     selected_model_id: selectedModelId,
+    selected_model_name: selected.selected_model_name,
+    fallback_model_ids: selected.fallback_model_ids,
     created,
     updated: true,
   };
@@ -287,6 +361,229 @@ function isApiFailure(response) {
   return response.status >= 400 || (response.json.code !== undefined && response.json.code !== 0);
 }
 
+function isSpaceModel(model) {
+  const provider = model?.provider && typeof model.provider === "object" ? model.provider : {};
+  return model?.provider_uuid === SPACE_PROVIDER_UUID
+    || provider.uuid === SPACE_PROVIDER_UUID
+    || provider.requester === "space-chat-completions"
+    || provider.name === "LangBot Models";
+}
+
+async function selectWorkingSpaceModel({
+  backendUrl,
+  token,
+  models,
+  skippedModelIds,
+  skippedModelNames,
+  requestedModelId,
+  existingModelId,
+}) {
+  const modelTests = [];
+  const testLimit = positiveInteger(env.LANGBOT_E2E_MODEL_TEST_LIMIT, DEFAULT_MODEL_TEST_LIMIT);
+  const fallbackCount = positiveInteger(env.LANGBOT_E2E_MODEL_FALLBACK_COUNT, DEFAULT_MODEL_FALLBACK_COUNT);
+  const workingModels = [];
+  const spaceModels = rankModels(models.filter((model) => (
+    model.uuid
+      && isSpaceModel(model)
+      && !skippedModelIds.has(model.uuid)
+      && !skippedModelNames.has(model.name)
+  )));
+  const requestedModel = requestedModelId
+    ? spaceModels.find((model) => model.uuid === requestedModelId) || null
+    : null;
+  const existingModel = existingModelId
+    ? spaceModels.find((model) => model.uuid === existingModelId) || null
+    : null;
+  const candidates = uniqueCandidates([
+    ...(requestedModel ? [existingCandidate(requestedModel, "requested")] : []),
+    ...(existingModel ? [existingCandidate(existingModel, "existing-pipeline")] : []),
+    ...spaceModels.map((model) => existingCandidate(model, "configured-space")),
+  ]);
+
+  let scanResult = { status: "skipped", models: [], reason: "" };
+  if (env.LANGBOT_E2E_SCAN_SPACE_MODELS !== "false") {
+    scanResult = await scanSpaceModels({ backendUrl, token });
+    if (scanResult.status === "pass") {
+      const knownNames = new Set(spaceModels.map((model) => model.name));
+      candidates.push(...scanResult.models
+        .filter((model) => model.name && !knownNames.has(model.name) && !skippedModelNames.has(model.name))
+        .map((model) => scannedCandidate(model)));
+    }
+  }
+
+  const unique = uniqueCandidates(candidates);
+  for (const candidate of unique.slice(0, testLimit)) {
+    const test = await ensureAndTestModel({ backendUrl, token, candidate });
+    modelTests.push(test);
+    if (test.status === "pass" && test.model_uuid) {
+      workingModels.push(test);
+      if (workingModels.length >= fallbackCount + 1) break;
+    }
+  }
+
+  if (workingModels.length > 0) {
+    const [primary, ...fallbacks] = workingModels;
+    return {
+      status: "pass",
+      reason: "",
+      selected_model_id: primary.model_uuid,
+      selected_model_name: primary.model_name,
+      fallback_model_ids: fallbacks.map((model) => model.model_uuid),
+      scanned_space_model_count: scanResult.models.length,
+      tested_model_count: modelTests.length,
+      model_tests: modelTests,
+    };
+  }
+
+  const baseReason = unique.length === 0
+    ? scanResult.reason || "No Space LLM model candidates are available."
+    : `No working Space LLM model found after testing ${modelTests.length} candidate(s).`;
+  return {
+    status: "env_issue",
+    reason: requestedModelId && !requestedModel
+      ? `Requested Space LLM model ${requestedModelId} is missing or skipped; ${baseReason}`
+      : baseReason,
+    selected_model_id: "",
+    selected_model_name: "",
+    fallback_model_ids: [],
+    scanned_space_model_count: scanResult.models.length,
+    tested_model_count: modelTests.length,
+    model_tests: modelTests,
+  };
+}
+
+async function scanSpaceModels({ backendUrl, token }) {
+  const response = await apiJson(
+    backendUrl,
+    `/api/v1/provider/providers/${encodeURIComponent(SPACE_PROVIDER_UUID)}/scan-models?type=llm`,
+    { token },
+  );
+  if (isApiFailure(response)) {
+    return {
+      status: "env_issue",
+      models: [],
+      reason: safeReason(response.json.msg || response.json.message || "Failed to scan Space LLM models."),
+    };
+  }
+  return {
+    status: "pass",
+    models: response.json.data?.models || [],
+    reason: "",
+  };
+}
+
+async function ensureAndTestModel({ backendUrl, token, candidate }) {
+  let modelUuid = candidate.uuid || "";
+  let created = false;
+  if (!modelUuid) {
+    const create = await apiJson(backendUrl, "/api/v1/provider/models/llm", {
+      method: "POST",
+      token,
+      body: {
+        name: candidate.name,
+        provider_uuid: SPACE_PROVIDER_UUID,
+        abilities: candidate.abilities || [],
+        context_length: candidate.context_length ?? null,
+        extra_args: {},
+        prefered_ranking: positiveInteger(candidate.prefered_ranking, 0),
+      },
+    });
+    modelUuid = create.json.data?.uuid || "";
+    if (isApiFailure(create) || !modelUuid) {
+      return modelTestResult(candidate, {
+        status: "fail",
+        reason: safeReason(create.json.msg || "Failed to create scanned Space model."),
+        http_status: create.status,
+      });
+    }
+    created = true;
+  }
+
+  const test = await apiJson(backendUrl, `/api/v1/provider/models/llm/${encodeURIComponent(modelUuid)}/test`, {
+    method: "POST",
+    token,
+    body: { extra_args: {} },
+  });
+  const passed = !isApiFailure(test);
+  if (!passed && created) {
+    await apiJson(backendUrl, `/api/v1/provider/models/llm/${encodeURIComponent(modelUuid)}`, {
+      method: "DELETE",
+      token,
+    }).catch(() => {});
+  }
+  return modelTestResult(candidate, {
+    status: passed ? "pass" : "fail",
+    reason: passed ? "" : safeReason(test.json.msg || test.json.message || "Space model test failed."),
+    http_status: test.status,
+    model_uuid: modelUuid,
+    created,
+  });
+}
+
+function modelTestResult(candidate, details) {
+  return {
+    source: candidate.source,
+    model_uuid: details.model_uuid || candidate.uuid || "",
+    model_name: candidate.name,
+    status: details.status,
+    reason: details.reason || "",
+    http_status: details.http_status ?? null,
+    created: Boolean(details.created),
+  };
+}
+
+function existingCandidate(model, source) {
+  return {
+    source,
+    uuid: model.uuid,
+    name: model.name,
+    abilities: model.abilities || [],
+    context_length: model.context_length,
+    prefered_ranking: model.prefered_ranking,
+  };
+}
+
+function scannedCandidate(model) {
+  return {
+    source: "scanned-space",
+    uuid: "",
+    name: model.name || model.id,
+    abilities: model.abilities || [],
+    context_length: model.context_length,
+    prefered_ranking: model.prefered_ranking,
+  };
+}
+
+function uniqueCandidates(candidates) {
+  const seen = new Set();
+  const result = [];
+  for (const candidate of candidates) {
+    const key = candidate.uuid ? `uuid:${candidate.uuid}` : `name:${candidate.name}`;
+    if (!candidate.name || seen.has(key)) continue;
+    seen.add(key);
+    result.push(candidate);
+  }
+  return result;
+}
+
+function rankModels(models) {
+  return [...models].sort((left, right) => {
+    const leftRank = Number.isFinite(Number(left.prefered_ranking)) ? Number(left.prefered_ranking) : 9999;
+    const rightRank = Number.isFinite(Number(right.prefered_ranking)) ? Number(right.prefered_ranking) : 9999;
+    if (leftRank !== rightRank) return leftRank - rightRank;
+    return String(left.name || "").localeCompare(String(right.name || ""));
+  });
+}
+
+function positiveInteger(value, fallback) {
+  const parsed = Number(value);
+  return Number.isInteger(parsed) && parsed > 0 ? parsed : fallback;
+}
+
+function safeReason(value) {
+  return redact(String(value || "")).slice(0, 1000);
+}
+
 async function upsertEnvLocal(path, updates) {
   let text = "";
   try {

skills/scripts/e2e/fake-openai-provider.mjs

@@ -0,0 +1,496 @@
+#!/usr/bin/env node
+
+import { createServer } from "node:http";
+import { mkdir, writeFile } from "node:fs/promises";
+import { dirname, resolve } from "node:path";
+import { env, exit } from "node:process";
+
+const args = parseArgs(process.argv.slice(2));
+const host = args.host || env.LANGBOT_FAKE_PROVIDER_HOST || "127.0.0.1";
+const port = integer(args.port ?? env.LANGBOT_FAKE_PROVIDER_PORT, 0);
+const stateFile = args["state-file"] || env.LANGBOT_FAKE_PROVIDER_STATE_FILE || "";
+const modelName = env.LANGBOT_FAKE_PROVIDER_MODEL_NAME || "gpt-4o-mini";
+const config = {
+  response_text: env.LANGBOT_FAKE_PROVIDER_RESPONSE_TEXT || "OK",
+  first_token_delay_ms: integer(env.LANGBOT_FAKE_PROVIDER_FIRST_TOKEN_DELAY_MS, 25),
+  chunk_delay_ms: integer(env.LANGBOT_FAKE_PROVIDER_CHUNK_DELAY_MS, 10),
+  chunk_count: integer(env.LANGBOT_FAKE_PROVIDER_CHUNK_COUNT, 0),
+  fault_status: integer(env.LANGBOT_FAKE_PROVIDER_FAULT_STATUS, 500),
+  fail_first_n: integer(env.LANGBOT_FAKE_PROVIDER_FAIL_FIRST_N, 0),
+  fail_every_n: integer(env.LANGBOT_FAKE_PROVIDER_FAIL_EVERY_N, 0),
+  fail_after_first_chunk: bool(env.LANGBOT_FAKE_PROVIDER_FAIL_AFTER_FIRST_CHUNK, false),
+  dynamic_response: !/^(0|false|no|off)$/i.test(env.LANGBOT_FAKE_PROVIDER_DYNAMIC_RESPONSE || ""),
+  request_log_limit: integer(env.LANGBOT_FAKE_PROVIDER_REQUEST_LOG_LIMIT, 500),
+};
+
+let requestCount = 0;
+const recentRequests = [];
+
+const server = createServer(async (request, response) => {
+  const startedAt = Date.now();
+  const startedPerf = performance.now();
+  let requestRecord = null;
+  const url = new URL(request.url || "/", `http://${request.headers.host || `${host}:${port}`}`);
+  try {
+    if (request.method === "GET" && url.pathname === "/healthz") {
+      sendJson(response, 200, {
+        ok: true,
+        model: modelName,
+        config,
+        request_count: requestCount,
+        recent_request_count: recentRequests.length,
+      });
+      return;
+    }
+
+    if (request.method === "GET" && url.pathname === "/__qa/config") {
+      sendJson(response, 200, {
+        ok: true,
+        model: modelName,
+        config,
+        request_count: requestCount,
+        recent_requests: recentRequests,
+      });
+      return;
+    }
+
+    if (request.method === "POST" && url.pathname === "/__qa/config") {
+      const body = await readJson(request);
+      applyConfig(body.config && typeof body.config === "object" ? body.config : body);
+      if (body.reset_request_count !== false) resetRequestState();
+      sendJson(response, 200, {
+        ok: true,
+        model: modelName,
+        config,
+        request_count: requestCount,
+      });
+      return;
+    }
+
+    if (request.method === "POST" && url.pathname === "/__qa/reset") {
+      resetRequestState();
+      sendJson(response, 200, {
+        ok: true,
+        model: modelName,
+        config,
+        request_count: requestCount,
+      });
+      return;
+    }
+
+    if (request.method === "GET" && ["/models", "/v1/models"].includes(url.pathname)) {
+      sendJson(response, 200, {
+        object: "list",
+        data: [
+          {
+            id: modelName,
+            object: "model",
+            created: 1,
+            owned_by: "langbot-qa",
+            type: "llm",
+          },
+        ],
+      });
+      return;
+    }
+
+    if (request.method === "POST" && ["/chat/completions", "/v1/chat/completions"].includes(url.pathname)) {
+      requestCount += 1;
+      const body = await readJson(request);
+      const requestId = `chatcmpl-langbot-fake-${requestCount}`;
+      const shouldFail = requestCount <= config.fail_first_n
+        || (config.fail_every_n > 0 && requestCount % config.fail_every_n === 0);
+      const replyText = responseTextForBody(body);
+      requestRecord = recordRequest({
+        id: requestId,
+        request_number: requestCount,
+        path: url.pathname,
+        stream: Boolean(body.stream),
+        model: body.model || "",
+        message_count: Array.isArray(body.messages) ? body.messages.length : 0,
+        should_fail: shouldFail,
+        status: "running",
+        http_status: null,
+        expected_text: replyText,
+        response_text_preview: previewText(replyText),
+        started_at: new Date(startedAt).toISOString(),
+        started_epoch_ms: startedAt,
+        configured_first_token_delay_ms: config.first_token_delay_ms,
+        configured_chunk_delay_ms: config.chunk_delay_ms,
+        configured_chunk_count: config.chunk_count,
+      });
+
+      if (shouldFail) {
+        await sleep(config.first_token_delay_ms);
+        sendJson(response, config.fault_status, {
+          error: {
+            message: `LangBot fake provider injected HTTP ${config.fault_status}`,
+            type: "fake_provider_fault",
+            code: "fake_provider_fault",
+          },
+        });
+        finishRequestRecord(requestRecord, startedPerf, {
+          status: "http_fault",
+          http_status: config.fault_status,
+        });
+        return;
+      }
+
+      if (body.stream) {
+        await streamCompletion(response, {
+          requestId,
+          model: body.model || modelName,
+          content: replyText,
+          failAfterFirstChunk: config.fail_after_first_chunk,
+          requestRecord,
+          startedPerf,
+        });
+      } else {
+        await sleep(config.first_token_delay_ms + config.chunk_delay_ms);
+        sendJson(response, 200, completionPayload({
+          requestId,
+          model: body.model || modelName,
+          content: replyText,
+        }));
+        markRequestTiming(requestRecord, "first_chunk", startedPerf);
+        markRequestTiming(requestRecord, "first_content_chunk", startedPerf);
+        requestRecord.content_chunk_count = 1;
+        finishRequestRecord(requestRecord, startedPerf, {
+          status: "ok",
+          http_status: 200,
+        });
+      }
+      return;
+    }
+
+    sendJson(response, 404, {
+      error: {
+        message: `No fake provider route for ${request.method} ${url.pathname}`,
+        type: "not_found",
+      },
+    });
+  } catch (error) {
+    if (requestRecord) {
+      finishRequestRecord(requestRecord, startedPerf, {
+        status: "fake_provider_error",
+        http_status: 500,
+        error: error instanceof Error ? error.message : String(error),
+      });
+    }
+    sendJson(response, 500, {
+      error: {
+        message: error instanceof Error ? error.message : String(error),
+        type: "fake_provider_error",
+      },
+    });
+  } finally {
+    const durationMs = Date.now() - startedAt;
+    if (url.pathname !== "/healthz") {
+      console.log(JSON.stringify({
+        at: new Date().toISOString(),
+        method: request.method,
+        path: url.pathname,
+        duration_ms: durationMs,
+      }));
+    }
+  }
+});
+
+server.listen(port, host, async () => {
+  const address = server.address();
+  const selectedPort = typeof address === "object" && address ? address.port : port;
+  const url = `http://${host}:${selectedPort}`;
+  const state = {
+    status: "ready",
+    pid: process.pid,
+    url,
+    base_url: `${url}/v1`,
+    model: modelName,
+    started_at: new Date().toISOString(),
+  };
+  if (stateFile) {
+    const path = resolve(stateFile);
+    await mkdir(dirname(path), { recursive: true });
+    await writeFile(path, `${JSON.stringify(state, null, 2)}\n`, "utf8");
+  }
+  console.log(JSON.stringify(state));
+});
+
+server.on("error", (error) => {
+  console.error(JSON.stringify({
+    status: "error",
+    reason: error instanceof Error ? error.message : String(error),
+  }));
+  exit(1);
+});
+
+process.on("SIGTERM", () => {
+  server.close(() => exit(0));
+});
+
+function parseArgs(argv) {
+  const result = {};
+  for (const item of argv) {
+    const match = item.match(/^--([^=]+)(?:=(.*))?$/);
+    if (!match) continue;
+    result[match[1]] = match[2] ?? "1";
+  }
+  return result;
+}
+
+function integer(value, fallback) {
+  const parsed = Number.parseInt(String(value ?? ""), 10);
+  return Number.isFinite(parsed) && parsed >= 0 ? parsed : fallback;
+}
+
+function bool(value, fallback) {
+  if (value === undefined || value === "") return fallback;
+  if (/^(1|true|yes|on)$/i.test(String(value))) return true;
+  if (/^(0|false|no|off)$/i.test(String(value))) return false;
+  return fallback;
+}
+
+function sleep(ms) {
+  return new Promise((resolve) => setTimeout(resolve, Math.max(0, ms)));
+}
+
+async function readJson(request) {
+  let text = "";
+  for await (const chunk of request) text += chunk.toString();
+  if (!text) return {};
+  return JSON.parse(text);
+}
+
+function sendJson(response, status, payload) {
+  const text = `${JSON.stringify(payload)}\n`;
+  response.writeHead(status, {
+    "content-type": "application/json",
+    "content-length": Buffer.byteLength(text),
+  });
+  response.end(text);
+}
+
+function completionPayload({ requestId, model, content }) {
+  const completionTokens = tokenEstimate(content);
+  return {
+    id: requestId,
+    object: "chat.completion",
+    created: Math.floor(Date.now() / 1000),
+    model,
+    choices: [
+      {
+        index: 0,
+        message: {
+          role: "assistant",
+          content,
+        },
+        finish_reason: "stop",
+      },
+    ],
+    usage: {
+      prompt_tokens: 8,
+      completion_tokens: completionTokens,
+      total_tokens: 8 + completionTokens,
+    },
+  };
+}
+
+async function streamCompletion(response, {
+  requestId,
+  model,
+  content,
+  failAfterFirstChunk: failMidStream,
+  requestRecord,
+  startedPerf,
+}) {
+  response.writeHead(200, {
+    "content-type": "text/event-stream; charset=utf-8",
+    "cache-control": "no-cache",
+    "connection": "keep-alive",
+  });
+
+  await sleep(config.first_token_delay_ms);
+  markRequestTiming(requestRecord, "first_chunk", startedPerf);
+  writeSse(response, {
+    id: requestId,
+    object: "chat.completion.chunk",
+    created: Math.floor(Date.now() / 1000),
+    model,
+    choices: [{ index: 0, delta: { role: "assistant" }, finish_reason: null }],
+  });
+
+  const chunks = splitContent(content);
+  for (let index = 0; index < chunks.length; index += 1) {
+    await sleep(config.chunk_delay_ms);
+    if (index === 0) markRequestTiming(requestRecord, "first_content_chunk", startedPerf);
+    requestRecord.content_chunk_count = (requestRecord.content_chunk_count || 0) + 1;
+    writeSse(response, {
+      id: requestId,
+      object: "chat.completion.chunk",
+      created: Math.floor(Date.now() / 1000),
+      model,
+      choices: [{ index: 0, delta: { content: chunks[index] }, finish_reason: null }],
+    });
+    if (failMidStream && index === 0) {
+      finishRequestRecord(requestRecord, startedPerf, {
+        status: "mid_stream_disconnect",
+        http_status: 200,
+      });
+      response.destroy(new Error("LangBot fake provider injected mid-stream disconnect"));
+      return;
+    }
+  }
+
+  await sleep(config.chunk_delay_ms);
+  const completionTokens = tokenEstimate(content);
+  writeSse(response, {
+    id: requestId,
+    object: "chat.completion.chunk",
+    created: Math.floor(Date.now() / 1000),
+    model,
+    choices: [{ index: 0, delta: {}, finish_reason: "stop" }],
+    usage: {
+      prompt_tokens: 8,
+      completion_tokens: completionTokens,
+      total_tokens: 8 + completionTokens,
+    },
+  });
+  response.write("data: [DONE]\n\n");
+  response.end();
+  finishRequestRecord(requestRecord, startedPerf, {
+    status: "ok",
+    http_status: 200,
+  });
+}
+
+function writeSse(response, payload) {
+  response.write(`data: ${JSON.stringify(payload)}\n\n`);
+}
+
+function splitContent(content) {
+  const text = String(content);
+  const requested = config.chunk_count;
+  if (requested <= 1 || text.length <= 1) return [text];
+  const chunkSize = Math.max(1, Math.ceil(text.length / requested));
+  const chunks = [];
+  for (let index = 0; index < text.length; index += chunkSize) {
+    chunks.push(text.slice(index, index + chunkSize));
+  }
+  return chunks;
+}
+
+function tokenEstimate(content) {
+  return Math.max(1, Math.ceil(String(content || "").length / 4));
+}
+
+function responseTextForBody(body) {
+  if (!config.dynamic_response) {
+    return config.response_text;
+  }
+  const messages = Array.isArray(body.messages) ? body.messages : [];
+  const lastUser = [...messages].reverse().find((message) => message?.role === "user");
+  const text = flattenContent(lastUser?.content || "");
+  const quoted = text.match(/["'“”](.{1,80}?)["'“”]/);
+  if (quoted?.[1]) return quoted[1].trim();
+  const exact = text.match(/(?:reply|回复|输出|return)\s+(?:exactly\s+)?([A-Za-z0-9_.:@-]{1,80})/i);
+  if (exact?.[1]) return exact[1].trim().replace(/[。.!?]+$/, "");
+  const only = text.match(/只回复\s*([A-Za-z0-9_.:@-]{1,80})/);
+  if (only?.[1]) return only[1].trim().replace(/[。.!?]+$/, "");
+  return config.response_text;
+}
+
+function flattenContent(content) {
+  if (typeof content === "string") return content;
+  if (Array.isArray(content)) {
+    return content
+      .map((item) => {
+        if (typeof item === "string") return item;
+        if (item && typeof item === "object") return item.text || "";
+        return "";
+      })
+      .join("\n");
+  }
+  return "";
+}
+
+function recordRequest(entry) {
+  const item = {
+    ...entry,
+    at: new Date().toISOString(),
+    finished_at: null,
+    finished_epoch_ms: null,
+    duration_ms: null,
+    first_chunk_at: null,
+    first_chunk_epoch_ms: null,
+    first_chunk_ms: null,
+    first_content_chunk_at: null,
+    first_content_chunk_epoch_ms: null,
+    first_content_chunk_ms: null,
+    content_chunk_count: 0,
+  };
+  recentRequests.push(item);
+  while (recentRequests.length > config.request_log_limit) recentRequests.shift();
+  return item;
+}
+
+function markRequestTiming(entry, key, startedPerf) {
+  if (!entry || entry[`${key}_at`]) return;
+  const now = Date.now();
+  entry[`${key}_at`] = new Date(now).toISOString();
+  entry[`${key}_epoch_ms`] = now;
+  entry[`${key}_ms`] = rounded(performance.now() - startedPerf);
+}
+
+function finishRequestRecord(entry, startedPerf, updates = {}) {
+  if (!entry || entry.finished_at) return;
+  const now = Date.now();
+  Object.assign(entry, updates);
+  entry.finished_at = new Date(now).toISOString();
+  entry.finished_epoch_ms = now;
+  entry.duration_ms = rounded(performance.now() - startedPerf);
+}
+
+function rounded(value) {
+  return Number(value.toFixed(3));
+}
+
+function previewText(value) {
+  return String(value || "").slice(0, 120);
+}
+
+function resetRequestState() {
+  requestCount = 0;
+  recentRequests.length = 0;
+}
+
+function applyConfig(updates) {
+  if (!updates || typeof updates !== "object") return;
+  assignString(updates, "response_text");
+  assignNonNegativeInteger(updates, "first_token_delay_ms");
+  assignNonNegativeInteger(updates, "chunk_delay_ms");
+  assignNonNegativeInteger(updates, "chunk_count");
+  assignNonNegativeInteger(updates, "fail_first_n");
+  assignNonNegativeInteger(updates, "fail_every_n");
+  assignNonNegativeInteger(updates, "request_log_limit");
+  if (updates.fault_status !== undefined) {
+    const parsed = Number.parseInt(String(updates.fault_status), 10);
+    if (Number.isInteger(parsed) && parsed >= 400 && parsed <= 599) config.fault_status = parsed;
+  }
+  assignBoolean(updates, "fail_after_first_chunk");
+  assignBoolean(updates, "dynamic_response");
+}
+
+function assignString(updates, key) {
+  if (updates[key] !== undefined) config[key] = String(updates[key]);
+}
+
+function assignNonNegativeInteger(updates, key) {
+  if (updates[key] === undefined) return;
+  const parsed = Number.parseInt(String(updates[key]), 10);
+  if (Number.isInteger(parsed) && parsed >= 0) config[key] = parsed;
+}
+
+function assignBoolean(updates, key) {
+  if (updates[key] === undefined) return;
+  config[key] = bool(updates[key], config[key]);
+}

skills/scripts/e2e/lib/langbot-e2e.mjs

@@ -72,6 +72,7 @@ export async function writeResult(paths, result) {
 }
 
 export async function loadEnvFiles(paths = ["skills/.env", "skills/.env.local"]) {
+  const processEnvKeys = new Set(Object.keys(env));
   for (const path of paths) {
     let text = "";
     try {
@@ -86,7 +87,7 @@ export async function loadEnvFiles(paths = ["skills/.env", "skills/.env.local"])
       if (equals <= 0) continue;
       const key = trimmed.slice(0, equals).trim();
       const value = trimmed.slice(equals + 1).trim().replace(/^["']|["']$/g, "");
-      if (!(key in env)) env[key] = value;
+      if (!processEnvKeys.has(key)) env[key] = value;
     }
   }
 }

skills/scripts/e2e/pipeline-debug-chat.mjs

@@ -54,6 +54,7 @@ const debugChatSessionType = env.LANGBOT_E2E_DEBUG_CHAT_SESSION_TYPE || "person"
 const pipelineConfigDiagnosticPath = resolve(paths.evidenceDir, "pipeline-config-diagnostic.json");
 const debugChatResetDiagnosticPath = resolve(paths.evidenceDir, "debug-chat-reset-diagnostic.json");
 const pipelineConfigRestoreDiagnosticPath = resolve(paths.evidenceDir, "pipeline-config-restore-diagnostic.json");
+const metricsPath = resolve(paths.evidenceDir, "metrics.json");
 const startedAt = new Date();
 
 let browser;
@@ -80,10 +81,11 @@ let result = {
     console_log: paths.consoleLog,
     network_log: paths.networkLog,
     screenshot: paths.screenshot,
+    metrics_json: metricsPath,
     automation_result_json: paths.automationResultJson,
     result_json: paths.resultJson,
   },
-  evidence_collected: ["ui", "screenshot", "console", "network"],
+  evidence_collected: ["ui", "screenshot", "console", "network", "metrics"],
 };
 
 function boolFromEnv(value, defaultValue) {
@@ -103,6 +105,29 @@ function parseJsonEnv(key, fallback) {
   }
 }
 
+function positiveNumberEnv(key, fallback) {
+  const value = Number(env[key] || "");
+  return Number.isFinite(value) && value >= 0 ? value : fallback;
+}
+
+function percentile(values, percentileValue) {
+  if (values.length === 0) return 0;
+  const sorted = [...values].sort((a, b) => a - b);
+  const index = Math.min(sorted.length - 1, Math.ceil((percentileValue / 100) * sorted.length) - 1);
+  return Number(sorted[index].toFixed(3));
+}
+
+function stats(values) {
+  if (values.length === 0) return { min: 0, p50: 0, p95: 0, p99: 0, max: 0 };
+  return {
+    min: Number(Math.min(...values).toFixed(3)),
+    p50: percentile(values, 50),
+    p95: percentile(values, 95),
+    p99: percentile(values, 99),
+    max: Number(Math.max(...values).toFixed(3)),
+  };
+}
+
 function promptStepsFromEnv() {
   const rawSteps = parseJsonEnv("LANGBOT_E2E_PROMPTS_JSON", null);
   if (rawSteps === null) {
@@ -658,6 +683,7 @@ try {
       } else {
         for (let index = 0; index < promptSteps.length; index += 1) {
           const step = promptSteps[index];
+          const promptStartedAt = Date.now();
           const chatResult = await runDebugChatPrompt(page, {
             prompt: step.prompt,
             expectedText: step.expectedText,
@@ -665,11 +691,13 @@ try {
             imagePath: index === 0 ? imagePath : "",
             failureSignals: failureSignals.length > 0 ? failureSignals : undefined,
           });
+          const promptDurationMs = Date.now() - promptStartedAt;
           result.chat_results.push({
             index,
             expected_text: step.expectedText,
             status: chatResult.status,
             reason: chatResult.reason,
+            response_duration_ms: promptDurationMs,
             min_expected_count: chatResult.min_expected_count,
             final_count: chatResult.final_count,
             before_assistant_expected_count: chatResult.before_assistant_expected_count,
@@ -714,6 +742,56 @@ try {
   const finishedAt = new Date();
   result.finished_at = finishedAt.toISOString();
   result.finished_at_local = localIsoWithOffset(finishedAt);
+  result.duration_ms = finishedAt.getTime() - startedAt.getTime();
+  const responseDurations = result.chat_results
+    .map((item) => item.response_duration_ms)
+    .filter((value) => Number.isFinite(value));
+  const passedPrompts = result.chat_results.filter((item) => item.status === "pass").length;
+  const attemptedPrompts = result.chat_results.length;
+  const errorRate = attemptedPrompts === 0 ? 1 : Number(((attemptedPrompts - passedPrompts) / attemptedPrompts).toFixed(4));
+  const responseStats = stats(responseDurations);
+  const responseP95BudgetMs = positiveNumberEnv(
+    "LANGBOT_E2E_DEBUG_CHAT_RESPONSE_P95_MS",
+    positiveNumberEnv("LANGBOT_DEBUG_CHAT_RESPONSE_P95_MS", safeResponseTimeoutMs),
+  );
+  const maxErrorRate = positiveNumberEnv("LANGBOT_E2E_DEBUG_CHAT_MAX_ERROR_RATE", 0);
+  const metrics = {
+    probe: caseId,
+    url: result.url,
+    prompt_count: result.prompt_count,
+    attempted_prompt_count: attemptedPrompts,
+    passed_prompt_count: passedPrompts,
+    error_rate: errorRate,
+    response_duration_ms: responseStats,
+    total_duration_ms: result.duration_ms,
+    chat_results: result.chat_results,
+  };
+  result.metrics_summary = {
+    prompt_count: metrics.prompt_count,
+    attempted_prompt_count: metrics.attempted_prompt_count,
+    passed_prompt_count: metrics.passed_prompt_count,
+    error_rate: metrics.error_rate,
+    response_p50_ms: metrics.response_duration_ms.p50,
+    response_p95_ms: metrics.response_duration_ms.p95,
+    total_duration_ms: metrics.total_duration_ms,
+  };
+  result.thresholds_summary = {
+    response_p95_ms: {
+      actual: metrics.response_duration_ms.p95,
+      max: responseP95BudgetMs,
+      pass: attemptedPrompts > 0 && metrics.response_duration_ms.p95 <= responseP95BudgetMs,
+    },
+    error_rate: {
+      actual: metrics.error_rate,
+      max: maxErrorRate,
+      pass: metrics.error_rate <= maxErrorRate,
+    },
+  };
+  await writeFile(metricsPath, `${JSON.stringify(metrics, null, 2)}\n`, "utf8");
+  if (result.status === "pass" && !Object.values(result.thresholds_summary).every((item) => item.pass)) {
+    result.status = "fail";
+    result.reason = "Debug Chat performance breached response latency or error-rate thresholds.";
+  }
   const existingEvidence = {};
   for (const [key, value] of Object.entries(result.evidence)) {
     if (typeof value !== "string") continue;

skills/skills.index.json

@@ -130,6 +130,7 @@
         "references/local-agent-runner.md",
         "references/mcp-stdio-testing.md",
         "references/model-provider-testing.md",
+        "references/performance-reliability-testing.md",
         "references/pipeline-debug-chat.md",
         "references/plugin-e2e-smoke.md",
         "references/sandbox-skill-authoring.md",
@@ -150,6 +151,16 @@
         "agent-runner-release-preflight",
         "agent-runner-runtime-chaos",
         "dify-agent-debug-chat",
+        "langbot-fake-provider-debug-chat-cross-pipeline-isolation",
+        "langbot-fake-provider-debug-chat-fault-recovery",
+        "langbot-fake-provider-debug-chat-load",
+        "langbot-fake-provider-debug-chat-slow-load",
+        "langbot-fault-taxonomy-contract",
+        "langbot-live-backend-latency",
+        "langbot-live-backend-log-health",
+        "langbot-live-control-plane-api",
+        "langbot-overhead-accounting-contract",
+        "langbot-space-debug-chat-concurrency-smoke",
         "langrag-kb-retrieve",
         "langrag-parser-golden-e2e",
         "langrag-sentinel-kb-discover",
@@ -165,6 +176,7 @@
         "mcp-stdio-register",
         "mcp-stdio-tool-call",
         "pipeline-debug-chat",
+        "pipeline-debug-chat-performance",
         "plugin-e2e-smoke",
         "provider-deepseek",
         "qa-plugin-smoke-live-install",
@@ -486,6 +498,316 @@
             "backend_log"
           ]
         },
+        {
+          "id": "langbot-fake-provider-debug-chat-cross-pipeline-isolation",
+          "title": "LangBot Debug Chat fake-provider cross-pipeline isolation probe",
+          "mode": "probe",
+          "area": "reliability",
+          "type": "reliability",
+          "priority": "p1",
+          "risk": "high",
+          "ci_eligible": false,
+          "tags": [
+            "reliability",
+            "debug-chat",
+            "websocket",
+            "fake-provider",
+            "isolation",
+            "concurrency",
+            "metrics"
+          ],
+          "automation": "skills/langbot-testing/probes/langbot-debug-chat-cross-pipeline-isolation.mjs",
+          "setup_automation": [
+            "node:scripts/e2e/ensure-fake-provider-cross-pipelines.mjs --write-env"
+          ],
+          "setup_provides_env": [
+            "LANGBOT_FAKE_PROVIDER_URL",
+            "LANGBOT_FAKE_PROVIDER_BASE_URL",
+            "LANGBOT_FAKE_PROVIDER_PID",
+            "LANGBOT_FAKE_PROVIDER_PIPELINE_A_URL",
+            "LANGBOT_FAKE_PROVIDER_PIPELINE_A_NAME",
+            "LANGBOT_FAKE_PROVIDER_PIPELINE_B_URL",
+            "LANGBOT_FAKE_PROVIDER_PIPELINE_B_NAME"
+          ],
+          "evidence_required": [
+            "metrics",
+            "network",
+            "api_diagnostic",
+            "filesystem"
+          ]
+        },
+        {
+          "id": "langbot-fake-provider-debug-chat-fault-recovery",
+          "title": "LangBot Debug Chat fake-provider fault recovery probe",
+          "mode": "probe",
+          "area": "reliability",
+          "type": "chaos",
+          "priority": "p1",
+          "risk": "high",
+          "ci_eligible": false,
+          "tags": [
+            "reliability",
+            "chaos",
+            "debug-chat",
+            "websocket",
+            "fake-provider",
+            "fault-injection",
+            "metrics"
+          ],
+          "automation": "skills/langbot-testing/probes/langbot-debug-chat-concurrency.mjs",
+          "setup_automation": [
+            "node:scripts/e2e/ensure-fake-provider-pipeline.mjs --write-env"
+          ],
+          "setup_provides_env": [
+            "LANGBOT_FAKE_PROVIDER_URL",
+            "LANGBOT_FAKE_PROVIDER_BASE_URL",
+            "LANGBOT_FAKE_PROVIDER_PID",
+            "LANGBOT_FAKE_PROVIDER_PROVIDER_UUID",
+            "LANGBOT_FAKE_PROVIDER_MODEL_UUID",
+            "LANGBOT_FAKE_PROVIDER_PIPELINE_URL",
+            "LANGBOT_FAKE_PROVIDER_PIPELINE_NAME"
+          ],
+          "evidence_required": [
+            "metrics",
+            "network",
+            "api_diagnostic",
+            "filesystem"
+          ]
+        },
+        {
+          "id": "langbot-fake-provider-debug-chat-load",
+          "title": "LangBot Debug Chat controlled fake-provider load probe",
+          "mode": "probe",
+          "area": "performance",
+          "type": "performance",
+          "priority": "p1",
+          "risk": "medium",
+          "ci_eligible": false,
+          "tags": [
+            "performance",
+            "debug-chat",
+            "websocket",
+            "fake-provider",
+            "load",
+            "metrics"
+          ],
+          "automation": "skills/langbot-testing/probes/langbot-debug-chat-concurrency.mjs",
+          "setup_automation": [
+            "node:scripts/e2e/ensure-fake-provider-pipeline.mjs --write-env"
+          ],
+          "setup_provides_env": [
+            "LANGBOT_FAKE_PROVIDER_URL",
+            "LANGBOT_FAKE_PROVIDER_BASE_URL",
+            "LANGBOT_FAKE_PROVIDER_PID",
+            "LANGBOT_FAKE_PROVIDER_PROVIDER_UUID",
+            "LANGBOT_FAKE_PROVIDER_MODEL_UUID",
+            "LANGBOT_FAKE_PROVIDER_PIPELINE_URL",
+            "LANGBOT_FAKE_PROVIDER_PIPELINE_NAME"
+          ],
+          "evidence_required": [
+            "metrics",
+            "network",
+            "api_diagnostic",
+            "filesystem"
+          ]
+        },
+        {
+          "id": "langbot-fake-provider-debug-chat-slow-load",
+          "title": "LangBot Debug Chat slow fake-provider load probe",
+          "mode": "probe",
+          "area": "performance",
+          "type": "performance",
+          "priority": "p1",
+          "risk": "medium",
+          "ci_eligible": false,
+          "tags": [
+            "performance",
+            "debug-chat",
+            "websocket",
+            "fake-provider",
+            "slow-provider",
+            "load",
+            "metrics"
+          ],
+          "automation": "skills/langbot-testing/probes/langbot-debug-chat-concurrency.mjs",
+          "setup_automation": [
+            "node:scripts/e2e/ensure-fake-provider-pipeline.mjs --write-env"
+          ],
+          "setup_provides_env": [
+            "LANGBOT_FAKE_PROVIDER_URL",
+            "LANGBOT_FAKE_PROVIDER_BASE_URL",
+            "LANGBOT_FAKE_PROVIDER_PID",
+            "LANGBOT_FAKE_PROVIDER_PROVIDER_UUID",
+            "LANGBOT_FAKE_PROVIDER_MODEL_UUID",
+            "LANGBOT_FAKE_PROVIDER_PIPELINE_URL",
+            "LANGBOT_FAKE_PROVIDER_PIPELINE_NAME"
+          ],
+          "evidence_required": [
+            "metrics",
+            "network",
+            "api_diagnostic",
+            "filesystem"
+          ]
+        },
+        {
+          "id": "langbot-fault-taxonomy-contract",
+          "title": "LangBot fault taxonomy and cleanup contract",
+          "mode": "probe",
+          "area": "reliability",
+          "type": "chaos",
+          "priority": "p1",
+          "risk": "medium",
+          "ci_eligible": true,
+          "tags": [
+            "reliability",
+            "chaos",
+            "contract",
+            "synthetic"
+          ],
+          "automation": "skills/langbot-testing/probes/langbot-fault-taxonomy-contract.mjs",
+          "setup_automation": [],
+          "setup_provides_env": [],
+          "evidence_required": [
+            "metrics",
+            "filesystem"
+          ]
+        },
+        {
+          "id": "langbot-live-backend-latency",
+          "title": "LangBot live backend basic latency probe",
+          "mode": "probe",
+          "area": "performance",
+          "type": "performance",
+          "priority": "p1",
+          "risk": "medium",
+          "ci_eligible": false,
+          "tags": [
+            "performance",
+            "live-backend",
+            "latency",
+            "metrics"
+          ],
+          "automation": "skills/langbot-testing/probes/langbot-live-backend-latency.mjs",
+          "setup_automation": [],
+          "setup_provides_env": [],
+          "evidence_required": [
+            "metrics",
+            "network",
+            "api_diagnostic",
+            "filesystem"
+          ]
+        },
+        {
+          "id": "langbot-live-backend-log-health",
+          "title": "LangBot live backend log health probe",
+          "mode": "probe",
+          "area": "reliability",
+          "type": "reliability",
+          "priority": "p1",
+          "risk": "medium",
+          "ci_eligible": false,
+          "tags": [
+            "reliability",
+            "live-backend",
+            "backend-log",
+            "metrics"
+          ],
+          "automation": "skills/langbot-testing/probes/langbot-live-backend-log-health.mjs",
+          "setup_automation": [],
+          "setup_provides_env": [],
+          "evidence_required": [
+            "metrics",
+            "backend_log",
+            "filesystem"
+          ]
+        },
+        {
+          "id": "langbot-live-control-plane-api",
+          "title": "LangBot live control-plane API probe",
+          "mode": "probe",
+          "area": "performance",
+          "type": "performance",
+          "priority": "p1",
+          "risk": "medium",
+          "ci_eligible": false,
+          "tags": [
+            "performance",
+            "reliability",
+            "live-backend",
+            "control-plane",
+            "metrics"
+          ],
+          "automation": "skills/langbot-testing/probes/langbot-live-control-plane-api.mjs",
+          "setup_automation": [],
+          "setup_provides_env": [],
+          "evidence_required": [
+            "metrics",
+            "network",
+            "api_diagnostic",
+            "filesystem"
+          ]
+        },
+        {
+          "id": "langbot-overhead-accounting-contract",
+          "title": "LangBot overhead accounting metrics contract",
+          "mode": "probe",
+          "area": "performance",
+          "type": "performance",
+          "priority": "p1",
+          "risk": "medium",
+          "ci_eligible": true,
+          "tags": [
+            "performance",
+            "metrics",
+            "contract",
+            "synthetic"
+          ],
+          "automation": "skills/langbot-testing/probes/langbot-overhead-accounting-contract.mjs",
+          "setup_automation": [],
+          "setup_provides_env": [],
+          "evidence_required": [
+            "metrics",
+            "resource_log",
+            "filesystem"
+          ]
+        },
+        {
+          "id": "langbot-space-debug-chat-concurrency-smoke",
+          "title": "LangBot Debug Chat real Space-provider concurrency smoke",
+          "mode": "probe",
+          "area": "performance",
+          "type": "performance",
+          "priority": "p1",
+          "risk": "high",
+          "ci_eligible": false,
+          "tags": [
+            "performance",
+            "debug-chat",
+            "websocket",
+            "space",
+            "live-provider",
+            "smoke",
+            "metrics"
+          ],
+          "automation": "skills/langbot-testing/probes/langbot-debug-chat-concurrency.mjs",
+          "setup_automation": [
+            "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env"
+          ],
+          "setup_provides_env": [
+            "LANGBOT_PIPELINE_URL",
+            "LANGBOT_PIPELINE_NAME",
+            "LANGBOT_LOCAL_AGENT_PIPELINE_URL",
+            "LANGBOT_LOCAL_AGENT_PIPELINE_NAME",
+            "LANGBOT_LOCAL_AGENT_MODEL_UUID",
+            "LANGBOT_E2E_MODEL_UUID"
+          ],
+          "evidence_required": [
+            "metrics",
+            "network",
+            "api_diagnostic",
+            "filesystem"
+          ]
+        },
         {
           "id": "langrag-kb-retrieve",
           "title": "LangRAG knowledge base ingests and retrieves a sentinel document",
@@ -911,6 +1233,38 @@
             "backend_log"
           ]
         },
+        {
+          "id": "pipeline-debug-chat-performance",
+          "title": "Pipeline Debug Chat user-path performance probe",
+          "mode": "agent-browser",
+          "area": "pipeline",
+          "type": "performance",
+          "priority": "p1",
+          "risk": "medium",
+          "ci_eligible": false,
+          "tags": [
+            "performance",
+            "pipeline",
+            "debug-chat",
+            "user-path",
+            "metrics"
+          ],
+          "automation": "scripts/e2e/pipeline-debug-chat.mjs",
+          "setup_automation": [
+            "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env"
+          ],
+          "setup_provides_env": [
+            "LANGBOT_PIPELINE_URL",
+            "LANGBOT_PIPELINE_NAME"
+          ],
+          "evidence_required": [
+            "ui",
+            "screenshot",
+            "console",
+            "network",
+            "metrics"
+          ]
+        },
         {
           "id": "plugin-e2e-smoke",
           "title": "Plugin system installs a local plugin and exposes tool/page APIs",
@@ -1059,6 +1413,12 @@
       "suites": [
         "agent-runner-release-gate",
         "core-smoke",
+        "langbot-debug-chat-isolation-gate",
+        "langbot-debug-chat-load-gate",
+        "langbot-live-backend-gate",
+        "langbot-performance-contract-gate",
+        "langbot-performance-reliability-gate",
+        "langbot-user-path-performance-gate",
         "local-agent-gate"
       ],
       "suite_summaries": [
@@ -1121,6 +1481,113 @@
             "local-agent-basic-debug-chat"
           ]
         },
+        {
+          "id": "langbot-debug-chat-isolation-gate",
+          "title": "LangBot Debug Chat isolation gate",
+          "description": "Manual/non-required cross-pipeline Debug Chat isolation gate. Current releases may fail this gate because of product bug #2286; use it as regression evidence after the routing fix lands.",
+          "type": "reliability",
+          "priority": "p1",
+          "tags": [
+            "reliability",
+            "debug-chat",
+            "websocket",
+            "isolation",
+            "concurrency"
+          ],
+          "cases": [
+            "langbot-fake-provider-debug-chat-cross-pipeline-isolation"
+          ]
+        },
+        {
+          "id": "langbot-debug-chat-load-gate",
+          "title": "LangBot Debug Chat load gate",
+          "description": "Manual/non-required message-path load checks for Pipeline Debug Chat: controlled fake-provider baseline, slow-provider and fault-recovery profiles, plus optional real Space-provider smoke. Cross-pipeline isolation is split into langbot-debug-chat-isolation-gate because current releases may fail it due to product bug #2286.",
+          "type": "performance",
+          "priority": "p1",
+          "tags": [
+            "performance",
+            "debug-chat",
+            "websocket",
+            "load"
+          ],
+          "cases": [
+            "langbot-fake-provider-debug-chat-load",
+            "langbot-fake-provider-debug-chat-slow-load",
+            "langbot-fake-provider-debug-chat-fault-recovery",
+            "langbot-space-debug-chat-concurrency-smoke"
+          ]
+        },
+        {
+          "id": "langbot-live-backend-gate",
+          "title": "LangBot live backend reliability gate",
+          "description": "Live backend control-plane responsiveness and runtime log health checks for a locally running LangBot instance.",
+          "type": "reliability",
+          "priority": "p1",
+          "tags": [
+            "performance",
+            "reliability",
+            "live-backend",
+            "metrics"
+          ],
+          "cases": [
+            "langbot-live-backend-latency",
+            "langbot-live-control-plane-api",
+            "langbot-live-backend-log-health"
+          ]
+        },
+        {
+          "id": "langbot-performance-contract-gate",
+          "title": "LangBot performance contract gate",
+          "description": "Fast synthetic contract checks for performance metric accounting and non-destructive reliability fault taxonomy.",
+          "type": "contract",
+          "priority": "p1",
+          "tags": [
+            "performance",
+            "reliability",
+            "contract",
+            "metrics"
+          ],
+          "cases": [
+            "langbot-overhead-accounting-contract",
+            "langbot-fault-taxonomy-contract"
+          ]
+        },
+        {
+          "id": "langbot-performance-reliability-gate",
+          "title": "LangBot performance and reliability starter gate",
+          "description": "Starter gate for LangBot performance accounting, live backend control-plane latency, and non-destructive fault taxonomy checks.",
+          "type": "reliability",
+          "priority": "p1",
+          "tags": [
+            "performance",
+            "reliability",
+            "metrics",
+            "chaos"
+          ],
+          "cases": [
+            "langbot-overhead-accounting-contract",
+            "langbot-fault-taxonomy-contract",
+            "langbot-live-backend-latency",
+            "langbot-live-control-plane-api",
+            "langbot-live-backend-log-health"
+          ]
+        },
+        {
+          "id": "langbot-user-path-performance-gate",
+          "title": "LangBot user-path performance gate",
+          "description": "Browser-visible performance checks for user-facing LangBot paths such as Pipeline Debug Chat.",
+          "type": "performance",
+          "priority": "p1",
+          "tags": [
+            "performance",
+            "browser",
+            "debug-chat",
+            "user-path"
+          ],
+          "cases": [
+            "pipeline-debug-chat-performance"
+          ]
+        },
         {
           "id": "local-agent-gate",
           "title": "Local Agent runner regression gate",
@@ -1265,6 +1732,7 @@
         "sandbox-native-tools-unavailable",
         "socks-proxy-without-socksio",
         "survey-widget-blocks-debug-chat",
+        "telemetry-proxy-noise",
         "tool-name-collision-between-mcp-and-plugin",
         "uv-run-resyncs-local-sdk"
       ],
@@ -1449,6 +1917,14 @@
             "mcp-stdio-tool-call"
           ]
         },
+        {
+          "id": "telemetry-proxy-noise",
+          "title": "Telemetry posting fails through the proxy while the target flow succeeds",
+          "category": "env_issue",
+          "related_cases": [
+            "langbot-space-debug-chat-concurrency-smoke"
+          ]
+        },
         {
           "id": "tool-name-collision-between-mcp-and-plugin",
           "title": "MCP and plugin expose the same tool name",

skills/skills/.env.example

@@ -26,6 +26,23 @@ LANGBOT_NO_PROXY=localhost,127.0.0.1,::1
 LANGBOT_PIPELINE_URL=
 LANGBOT_PIPELINE_NAME=
 
+# Optional fake OpenAI-compatible provider controls for Debug Chat load tests.
+# Leave URL empty to let setup automation start a local provider and write the
+# selected URL to skills/.env.local.
+LANGBOT_FAKE_PROVIDER_URL=
+LANGBOT_FAKE_PROVIDER_HOST=127.0.0.1
+LANGBOT_FAKE_PROVIDER_PORT=
+LANGBOT_FAKE_PROVIDER_MODEL_NAME=gpt-4o-mini
+LANGBOT_FAKE_PROVIDER_RESPONSE_TEXT=OK
+LANGBOT_FAKE_PROVIDER_FIRST_TOKEN_DELAY_MS=25
+LANGBOT_FAKE_PROVIDER_CHUNK_DELAY_MS=10
+LANGBOT_FAKE_PROVIDER_CHUNK_COUNT=0
+LANGBOT_FAKE_PROVIDER_FAIL_FIRST_N=0
+LANGBOT_FAKE_PROVIDER_FAIL_EVERY_N=0
+LANGBOT_FAKE_PROVIDER_FAULT_STATUS=500
+LANGBOT_FAKE_PROVIDER_FAIL_AFTER_FIRST_CHUNK=false
+LANGBOT_FAKE_PROVIDER_DYNAMIC_RESPONSE=true
+
 # Optional case-specific runner targets. Prefer these for runner-specific cases
 # so the automation cannot silently test the wrong runner.
 LANGBOT_LOCAL_AGENT_PIPELINE_URL=

skills/skills/langbot-deploy/SKILL.md

@@ -27,6 +27,17 @@ The `all` / `box` profile starts three services:
 - `langbot_box` — Box sandbox runtime (`:5410`). Uses the host Docker socket to
   spawn sandbox containers, so the **Box root host path and in-container path
   must be identical** (`BOX__LOCAL__HOST_ROOT=${LANGBOT_BOX_ROOT:-${PWD}/data/box}`).
+  Its RPC and managed-process relay require a shared
+  `LANGBOT_BOX_CONTROL_TOKEN` (at least 32 non-whitespace characters) in both
+  the LangBot and Box containers. Generate it once with `openssl rand -hex 32`;
+  never put it in `box.runtime.endpoint` or commit it to config.
+
+Every Compose deployment also needs one
+`LANGBOT_PLUGIN_RUNTIME_CONTROL_TOKEN` shared by `langbot` and
+`langbot_plugin_runtime`. Generate it with `openssl rand -hex 32` and export it
+before `docker compose up`; the external Plugin Runtime fails closed when the
+token is empty or weak. Kubernetes uses the `langbot-plugin-runtime-control`
+Secret shown in `docker/kubernetes.yaml`.
 
 With Box off, the dashboard/skills list stays visible (read-only) but sandbox
 tools, skill add/edit, and stdio MCP are disabled. Set `box.enabled: false`

skills/skills/langbot-dev/SKILL.md

@@ -65,10 +65,23 @@ Route auth is declared per-route via `AuthType` in
 - `API_KEY` — `X-API-Key` or `Authorization: Bearer <key>`.
 - `USER_TOKEN_OR_API_KEY` — either.
 
-API keys are verified by `apikey_service.verify_api_key()`, which accepts:
-1. the **global key** from `config.yaml` `api.global_api_key` (no DB, no login,
-   no `lbk_` prefix required), then
-2. **web-UI keys** (DB-stored, `lbk_` prefix).
+Authenticated routes receive an immutable `RequestContext` containing the
+principal, authorized Workspace membership, fixed-role permissions, instance,
+request id, and placement generation. A browser's `X-Workspace-Id` is only a
+selector and is always checked against the Account membership. Tenant services
+must accept this context (or an explicit trusted execution context) and fail
+closed when it is absent.
+
+API-key authentication accepts:
+
+1. the **global key** from `config.yaml` `api.global_api_key` only for a
+   community instance with exactly one local Workspace, then
+2. **web-UI keys** whose one-time `lbk_` secret is stored only as a hash and is
+   bound to one Workspace, explicit scopes, status, and optional expiry.
+
+An API key derives its Workspace from the key record and ignores a caller's
+Workspace selector. Public Bot/Webhook routes similarly derive Workspace from
+the opaque owning resource rather than a header.
 
 Route groups self-register via `@group.group_class(name, path)` and are
 discovered by `importutil.import_modules_in_pkg`.

skills/skills/langbot-env-setup/references/service-startup.md

@@ -53,7 +53,7 @@ Start the new frontend from the web repo:
 
 ```bash
 cd "$LANGBOT_WEB_REPO"
-npm run dev
+VITE_API_BASE_URL="$LANGBOT_BACKEND_URL" pnpm dev --host 0.0.0.0
 ```
 
 Healthy startup includes:
@@ -68,6 +68,10 @@ Quick check:
 curl -I --max-time 3 "$LANGBOT_FRONTEND_URL"
 ```
 
+If `VITE_API_BASE_URL` is missing, Vite still serves the page but frontend API
+calls may go to the frontend port instead of the backend port. That produces
+false browser failures in login, wizard, pipeline, and Debug Chat cases.
+
 ## Completion Signal
 
 Environment setup is not complete until the required frontend/backend URLs are reachable and the chosen browser-control path can open the WebUI.

skills/skills/langbot-mcp-ops/SKILL.md

@@ -29,13 +29,19 @@ Authorization: Bearer <api-key>
 
 Two kinds of key are accepted:
 
-1. **Web-UI key** — created in the web UI (sidebar → API Keys), prefixed `lbk_`,
-   stored in the database.
+1. **Web-UI key** — created in the web UI (sidebar → API Keys), prefixed `lbk_`.
+   The secret is shown once; only its SHA-256 hash is stored. Each key is bound
+   to one Workspace and has explicit scopes, status, optional expiry, and
+   last-used metadata. The key determines the Workspace; callers cannot switch
+   it with `X-Workspace-Id`.
 2. **Global API key** — set in `data/config.yaml` under `api.global_api_key`.
    Requires no login session and no DB record; does not need the `lbk_` prefix.
-   Leave empty to disable. See the `langbot-deploy` skill for config details.
+   It is accepted only by a community instance with exactly one local
+   Workspace and is disabled for SaaS multi-Workspace operation. Leave empty to
+   disable. See the `langbot-deploy` skill for config details.
 
-Requests without a valid key get `401 Unauthorized`.
+Invalid, revoked, or expired keys get `401 Unauthorized`. A valid key whose
+scopes do not authorize a tool gets `403 Forbidden`.
 
 ## Client configuration
 
@@ -66,7 +72,9 @@ The tools wrap the LangBot service layer. Current tools (v1):
 
 Mutating tools (`create_*`, `update_*`) take a JSON object matching the same
 shape as the corresponding HTTP API request body. Discover resources with the
-`list_*` / `get_*` tools before mutating; identifiers are UUIDs.
+`list_*` / `get_*` tools before mutating; identifiers are UUIDs. Reads require
+`resource.view`; mutations require `resource.manage`. All service calls inherit
+the immutable Workspace context authenticated at the MCP transport boundary.
 
 ## How to use
 
@@ -93,7 +101,8 @@ shape as the corresponding HTTP API request body. Discover resources with the
 - `/mcp` is the **server** LangBot exposes. The `/api/v1/mcp` routes are the
   **client** side (managing external MCP servers LangBot connects to). Don't
   confuse them.
-- A `401` means the key is wrong, missing, or (for the global key)
-  `api.global_api_key` is empty in config.yaml.
+- A `401` means the key is wrong, missing, revoked, expired, or (for the global
+  key) `api.global_api_key` is empty or the instance is not an OSS singleton.
+- A `403` means the key is valid but lacks the permission required by the tool.
 - The global key is plaintext in config.yaml — only enable it on trusted/internal
   deployments and serve over HTTPS.

skills/skills/langbot-testing/SKILL.md

@@ -21,6 +21,7 @@ Use this skill when an agent needs to verify LangBot behavior through the WebUI
 - **Sandbox-backed skill authoring**: read `references/sandbox-skill-authoring.md`.
 - **LangRAG knowledge bases**: read `references/langrag-knowledge-base.md`.
 - **MCP stdio tool testing**: read `references/mcp-stdio-testing.md`.
+- **Performance, reliability, or chaos probes**: read `references/performance-reliability-testing.md`.
 - **Drive a live instance over MCP (not raw HTTP)**: use the `langbot-mcp-ops` skill — the instance exposes an MCP server at `http://<host>:5300/mcp` (reuses API keys). Useful for setting up bots/pipelines/models as test fixtures programmatically.
 - **Known failures and fixes**: read `references/troubleshooting.md`.
 - **Reusable test groups**: run `bin/lbs suite list` and `bin/lbs suite plan <suite-id>` before manually assembling a case set.
@@ -36,6 +37,8 @@ Use this skill when an agent needs to verify LangBot behavior through the WebUI
 - Use an authenticated browser profile prepared by `langbot-env-setup`.
 - Do not expose API keys, OAuth secrets, tokens, or localStorage token values in output.
 - A WebUI test is not complete until the visible UI result is checked against backend logs or network behavior.
+- A performance result is not complete without `metrics` evidence and a clear split between LangBot overhead and external provider/tool/network time.
+- A chaos or reliability result is not complete until the fault scope, cleanup, and recovery checks are recorded.
 - For a suite, use `bin/lbs suite start <suite-id>` to create the suite evidence root, per-case directories, and `suite-start.json`/`suite-start.md` handoff files; use `bin/lbs test result <case-id>` to write final per-case `result.json`, then run `bin/lbs suite report <suite-id> --evidence-dir <dir>`.
 - Do not mark a case `pass` until `test result --evidence` covers every value in the case's `evidence_required`.
 - For runner-specific Debug Chat cases, use the case-specific pipeline env declared by `automation_pipeline_url_env` / `automation_pipeline_name_env`; do not silently reuse a generic `LANGBOT_PIPELINE_URL`.

skills/skills/langbot-testing/cases/langbot-fake-provider-debug-chat-cross-pipeline-isolation.yaml

@@ -0,0 +1,84 @@
+id: langbot-fake-provider-debug-chat-cross-pipeline-isolation
+title: "LangBot Debug Chat fake-provider cross-pipeline isolation probe"
+mode: probe
+area: reliability
+type: reliability
+priority: p1
+risk: high
+ci_eligible: false
+tags:
+  - reliability
+  - debug-chat
+  - websocket
+  - fake-provider
+  - isolation
+  - concurrency
+  - metrics
+skills:
+  - langbot-env-setup
+  - langbot-testing
+env:
+  - LANGBOT_BACKEND_URL
+  - LANGBOT_FRONTEND_URL
+  - LANGBOT_E2E_LOGIN_USER
+automation: skills/langbot-testing/probes/langbot-debug-chat-cross-pipeline-isolation.mjs
+automation_env:
+  - LANGBOT_BACKEND_URL
+  - LANGBOT_E2E_LOGIN_USER
+  - LANGBOT_FAKE_PROVIDER_URL
+  - LANGBOT_FAKE_PROVIDER_PIPELINE_A_URL
+  - LANGBOT_FAKE_PROVIDER_PIPELINE_A_NAME
+  - LANGBOT_FAKE_PROVIDER_PIPELINE_B_URL
+  - LANGBOT_FAKE_PROVIDER_PIPELINE_B_NAME
+automation_debug_chat_load_requests: "6"
+automation_debug_chat_load_concurrency: "4"
+automation_debug_chat_load_timeout_ms: "30000"
+automation_debug_chat_load_response_p95_ms: "5000"
+automation_debug_chat_load_max_error_rate: "0"
+automation_debug_chat_load_prompt_template: '请只回复 "{expected}"，不要解释，不要添加其他字符。'
+automation_debug_chat_load_stream: "true"
+automation_debug_chat_load_reset: "true"
+metrics_thresholds_json: '{"cross_pipeline_leak_count":{"max":0},"response_p95_ms":{"max":5000},"error_rate":{"max":0}}'
+load_profile_json: '{"requests_per_pipeline":6,"pipelines":2,"concurrency":4,"path":"Pipeline Debug Chat WebSocket","provider":"controlled fake OpenAI-compatible provider","metric":"cross-pipeline response isolation and send-to-final-assistant-response"}'
+setup_automation:
+  - "node:scripts/e2e/ensure-fake-provider-cross-pipelines.mjs --write-env"
+setup_provides_env:
+  - LANGBOT_FAKE_PROVIDER_URL
+  - LANGBOT_FAKE_PROVIDER_BASE_URL
+  - LANGBOT_FAKE_PROVIDER_PID
+  - LANGBOT_FAKE_PROVIDER_PIPELINE_A_URL
+  - LANGBOT_FAKE_PROVIDER_PIPELINE_A_NAME
+  - LANGBOT_FAKE_PROVIDER_PIPELINE_B_URL
+  - LANGBOT_FAKE_PROVIDER_PIPELINE_B_NAME
+steps:
+  - "Start or reuse the local fake OpenAI-compatible provider."
+  - "Create or update two local-agent pipelines that both point at the controlled fake provider."
+  - "Reset both Debug Chat sessions and the fake-provider request log."
+  - "Open concurrent WebSocket Debug Chat connections to both pipelines and send unique pipeline-scoped response tokens."
+checks:
+  - "automation-result.json status is pass only when every request receives its own expected token and cross_pipeline_leak_count is zero."
+  - "metrics_summary includes by_pipeline status counts, fake-provider request count, and LangBot/provider timing estimates."
+  - "samples.json contains per-request pipeline labels so any leak can be attributed to the receiving pipeline."
+evidence_required:
+  - metrics
+  - network
+  - api_diagnostic
+  - filesystem
+diagnostics:
+  - "This probe targets Debug Chat isolation under concurrent traffic from two pipelines."
+  - "It is designed to expose regressions where global pipeline state causes one pipeline's assistant response to be delivered to another pipeline's Debug Chat session."
+  - "Same-pipeline foreign responses are tolerated because Debug Chat intentionally broadcasts within the same pipeline/session; cross-pipeline tokens are never tolerated."
+  - "Known product bug: current releases may fail this probe because Debug Chat replies can read singleton WebSocket proxy pipeline state after another pipeline overwrites it. See https://github.com/langbot-app/LangBot/issues/2286."
+expected_failures:
+  - "https://github.com/langbot-app/LangBot/issues/2286"
+success_patterns:
+  - "Debug Chat cross-pipeline isolation probe passed"
+failure_patterns:
+  - "cross_pipeline_leak"
+  - "Timed out after"
+  - "WebSocket connection error"
+  - "Final assistant response did not include"
+troubleshooting:
+  - backend-not-listening
+  - debug-chat-history-contaminates-automation
+  - local-agent-model-route-unavailable

skills/skills/langbot-testing/cases/langbot-fake-provider-debug-chat-fault-recovery.yaml

@@ -0,0 +1,95 @@
+id: langbot-fake-provider-debug-chat-fault-recovery
+title: "LangBot Debug Chat fake-provider fault recovery probe"
+mode: probe
+area: reliability
+type: chaos
+priority: p1
+risk: high
+ci_eligible: false
+tags:
+  - reliability
+  - chaos
+  - debug-chat
+  - websocket
+  - fake-provider
+  - fault-injection
+  - metrics
+skills:
+  - langbot-env-setup
+  - langbot-testing
+env:
+  - LANGBOT_BACKEND_URL
+  - LANGBOT_FRONTEND_URL
+  - LANGBOT_E2E_LOGIN_USER
+automation: skills/langbot-testing/probes/langbot-debug-chat-concurrency.mjs
+automation_env:
+  - LANGBOT_BACKEND_URL
+  - LANGBOT_E2E_LOGIN_USER
+  - LANGBOT_FAKE_PROVIDER_PIPELINE_URL
+  - LANGBOT_FAKE_PROVIDER_PIPELINE_NAME
+automation_pipeline_url_env: LANGBOT_FAKE_PROVIDER_PIPELINE_URL
+automation_pipeline_name_env: LANGBOT_FAKE_PROVIDER_PIPELINE_NAME
+automation_debug_chat_load_requests: "6"
+automation_debug_chat_load_concurrency: "1"
+automation_debug_chat_load_timeout_ms: "15000"
+automation_debug_chat_load_response_p95_ms: "5000"
+automation_debug_chat_load_max_error_rate: "0"
+automation_debug_chat_load_min_ok_count: "6"
+automation_debug_chat_load_min_provider_fault_count: "2"
+automation_debug_chat_load_expected_prefix: "FAULTQA"
+automation_debug_chat_load_prompt_template: '请只回复 "{expected}"，不要解释，不要添加其他字符。'
+automation_debug_chat_load_stream: "true"
+automation_debug_chat_load_reset: "true"
+automation_debug_chat_load_fail_on_final_mismatch: "true"
+automation_fake_provider_first_token_delay_ms: "25"
+automation_fake_provider_chunk_delay_ms: "10"
+automation_fake_provider_chunk_count: "0"
+automation_fake_provider_fail_first_n: "2"
+automation_fake_provider_fail_every_n: "0"
+automation_fake_provider_fault_status: "503"
+metrics_thresholds_json: '{"response_p95_ms":{"max":5000},"error_rate":{"max":0},"ok_count_min":{"min":6},"fake_provider_fault_count_min":{"min":2}}'
+fault_model_json: '{"provider_fault":"HTTP 503 for first 2 fake-provider chat completions after reset","expected_behavior":"LangBot retries or otherwise recovers from bounded provider failures so every Debug Chat request receives its expected response without backend crash."}'
+load_profile_json: '{"requests":6,"concurrency":1,"path":"Pipeline Debug Chat WebSocket","provider":"controlled fake OpenAI-compatible provider","classification":"fault-recovery-not-throughput-benchmark"}'
+setup_automation:
+  - "node:scripts/e2e/ensure-fake-provider-pipeline.mjs --write-env"
+setup_provides_env:
+  - LANGBOT_FAKE_PROVIDER_URL
+  - LANGBOT_FAKE_PROVIDER_BASE_URL
+  - LANGBOT_FAKE_PROVIDER_PID
+  - LANGBOT_FAKE_PROVIDER_PROVIDER_UUID
+  - LANGBOT_FAKE_PROVIDER_MODEL_UUID
+  - LANGBOT_FAKE_PROVIDER_PIPELINE_URL
+  - LANGBOT_FAKE_PROVIDER_PIPELINE_NAME
+steps:
+  - "Configure the local fake provider to return HTTP 503 for the first two chat completions after reset."
+  - "Create or update the LangBot provider, model, and local-agent pipeline that points at the fake provider."
+  - "Reset the target Debug Chat session and fake-provider request counter."
+  - "Send a sequential Debug Chat batch and verify later requests recover after the injected provider faults."
+checks:
+  - "automation-result.json status is pass when the fake provider records at least two injected faults, every Debug Chat request succeeds, and total user-visible error rate stays at zero."
+  - "metrics_summary includes fake_provider_fault_count and status_counts for the same run window."
+  - "backend logs show request handling for the same run window without unexpected Traceback or task-leak findings."
+evidence_required:
+  - metrics
+  - network
+  - api_diagnostic
+  - filesystem
+diagnostics:
+  - "This is a fault-recovery probe, not a throughput benchmark."
+  - "Provider faults may be retried inside the provider/requester path; judge this case by fake_provider_fault_count plus user-visible success/error metrics."
+  - "The profile uses concurrency 1 because Debug Chat broadcasts assistant responses to every connection in a session, and failed responses do not carry the unique success token needed for concurrent attribution."
+success_patterns:
+  - "Debug Chat WebSocket concurrency probe passed"
+  - "Streaming completed"
+failure_patterns:
+  - "fake_provider_fault"
+  - "HTTP 503"
+  - "Timed out after"
+  - "All models failed during streaming setup"
+expected_failures:
+  - "fake_provider_fault"
+  - "HTTP 503"
+troubleshooting:
+  - backend-not-listening
+  - debug-chat-history-contaminates-automation
+  - local-agent-model-route-unavailable

skills/skills/langbot-testing/cases/langbot-fake-provider-debug-chat-load.yaml

@@ -0,0 +1,81 @@
+id: langbot-fake-provider-debug-chat-load
+title: "LangBot Debug Chat controlled fake-provider load probe"
+mode: probe
+area: performance
+type: performance
+priority: p1
+risk: medium
+ci_eligible: false
+tags:
+  - performance
+  - debug-chat
+  - websocket
+  - fake-provider
+  - load
+  - metrics
+skills:
+  - langbot-env-setup
+  - langbot-testing
+env:
+  - LANGBOT_BACKEND_URL
+  - LANGBOT_FRONTEND_URL
+  - LANGBOT_E2E_LOGIN_USER
+automation: skills/langbot-testing/probes/langbot-debug-chat-concurrency.mjs
+automation_env:
+  - LANGBOT_BACKEND_URL
+  - LANGBOT_E2E_LOGIN_USER
+  - LANGBOT_FAKE_PROVIDER_PIPELINE_URL
+  - LANGBOT_FAKE_PROVIDER_PIPELINE_NAME
+automation_pipeline_url_env: LANGBOT_FAKE_PROVIDER_PIPELINE_URL
+automation_pipeline_name_env: LANGBOT_FAKE_PROVIDER_PIPELINE_NAME
+automation_debug_chat_load_requests: "12"
+automation_debug_chat_load_concurrency: "4"
+automation_debug_chat_load_timeout_ms: "30000"
+automation_debug_chat_load_response_p95_ms: "5000"
+automation_debug_chat_load_first_response_p95_ms: "3000"
+automation_debug_chat_load_max_error_rate: "0"
+automation_debug_chat_load_expected_prefix: "FAKEQA"
+automation_debug_chat_load_prompt_template: '请只回复 "{expected}"，不要解释，不要添加其他字符。'
+automation_debug_chat_load_stream: "true"
+automation_debug_chat_load_reset: "true"
+metrics_thresholds_json: '{"response_p95_ms":{"max":5000},"first_response_p95_ms":{"max":3000},"error_rate":{"max":0}}'
+load_profile_json: '{"requests":12,"concurrency":4,"path":"Pipeline Debug Chat WebSocket","provider":"controlled fake OpenAI-compatible provider","metric":"send-to-final-assistant-response"}'
+setup_automation:
+  - "node:scripts/e2e/ensure-fake-provider-pipeline.mjs --write-env"
+setup_provides_env:
+  - LANGBOT_FAKE_PROVIDER_URL
+  - LANGBOT_FAKE_PROVIDER_BASE_URL
+  - LANGBOT_FAKE_PROVIDER_PID
+  - LANGBOT_FAKE_PROVIDER_PROVIDER_UUID
+  - LANGBOT_FAKE_PROVIDER_MODEL_UUID
+  - LANGBOT_FAKE_PROVIDER_PIPELINE_URL
+  - LANGBOT_FAKE_PROVIDER_PIPELINE_NAME
+steps:
+  - "Start or reuse the local fake OpenAI-compatible provider."
+  - "Create or update the LangBot provider, model, and local-agent pipeline that points at the fake provider."
+  - "Reset the target Debug Chat session."
+  - "Open concurrent WebSocket Debug Chat connections and send unique deterministic prompts through the real backend pipeline."
+checks:
+  - "automation-result.json status is pass when every request receives its own expected assistant response."
+  - "metrics_summary includes request count, concurrency, p50/p95 response latency, first response latency, throughput, and error rate."
+  - "thresholds_summary shows response_p95_ms, first_response_p95_ms, and error_rate pass."
+evidence_required:
+  - metrics
+  - network
+  - api_diagnostic
+  - filesystem
+diagnostics:
+  - "This probe removes external model latency from the measurement; it still exercises the live LangBot backend, provider requester, local-agent runner, pipeline, and Debug Chat WebSocket adapter."
+  - "Use this as the repeatable message-path baseline before comparing against Space or another real provider."
+success_patterns:
+  - "Debug Chat WebSocket concurrency probe passed"
+  - "Streaming completed"
+failure_patterns:
+  - "WebSocket connection error"
+  - "Timed out after"
+  - "Final assistant response did not include"
+  - "All models failed during streaming setup"
+troubleshooting:
+  - backend-not-listening
+  - debug-chat-history-contaminates-automation
+  - local-agent-model-route-unavailable

skills/skills/langbot-testing/cases/langbot-fake-provider-debug-chat-slow-load.yaml

@@ -0,0 +1,88 @@
+id: langbot-fake-provider-debug-chat-slow-load
+title: "LangBot Debug Chat slow fake-provider load probe"
+mode: probe
+area: performance
+type: performance
+priority: p1
+risk: medium
+ci_eligible: false
+tags:
+  - performance
+  - debug-chat
+  - websocket
+  - fake-provider
+  - slow-provider
+  - load
+  - metrics
+skills:
+  - langbot-env-setup
+  - langbot-testing
+env:
+  - LANGBOT_BACKEND_URL
+  - LANGBOT_FRONTEND_URL
+  - LANGBOT_E2E_LOGIN_USER
+automation: skills/langbot-testing/probes/langbot-debug-chat-concurrency.mjs
+automation_env:
+  - LANGBOT_BACKEND_URL
+  - LANGBOT_E2E_LOGIN_USER
+  - LANGBOT_FAKE_PROVIDER_PIPELINE_URL
+  - LANGBOT_FAKE_PROVIDER_PIPELINE_NAME
+automation_pipeline_url_env: LANGBOT_FAKE_PROVIDER_PIPELINE_URL
+automation_pipeline_name_env: LANGBOT_FAKE_PROVIDER_PIPELINE_NAME
+automation_debug_chat_load_requests: "8"
+automation_debug_chat_load_concurrency: "4"
+automation_debug_chat_load_timeout_ms: "45000"
+automation_debug_chat_load_response_p95_ms: "10000"
+automation_debug_chat_load_first_response_p95_ms: "7000"
+automation_debug_chat_load_max_error_rate: "0"
+automation_debug_chat_load_expected_prefix: "SLOWQA"
+automation_debug_chat_load_prompt_template: '请只回复 "{expected}"，不要解释，不要添加其他字符。'
+automation_debug_chat_load_stream: "true"
+automation_debug_chat_load_reset: "true"
+automation_fake_provider_first_token_delay_ms: "1000"
+automation_fake_provider_chunk_delay_ms: "250"
+automation_fake_provider_chunk_count: "4"
+automation_fake_provider_fail_first_n: "0"
+automation_fake_provider_fail_every_n: "0"
+automation_fake_provider_fault_status: "500"
+metrics_thresholds_json: '{"response_p95_ms":{"max":10000},"first_response_p95_ms":{"max":7000},"error_rate":{"max":0}}'
+load_profile_json: '{"requests":8,"concurrency":4,"path":"Pipeline Debug Chat WebSocket","provider":"controlled slow fake OpenAI-compatible provider","metric":"send-to-final-assistant-response","provider_profile":{"first_token_delay_ms":1000,"chunk_delay_ms":250,"chunk_count":4}}'
+setup_automation:
+  - "node:scripts/e2e/ensure-fake-provider-pipeline.mjs --write-env"
+setup_provides_env:
+  - LANGBOT_FAKE_PROVIDER_URL
+  - LANGBOT_FAKE_PROVIDER_BASE_URL
+  - LANGBOT_FAKE_PROVIDER_PID
+  - LANGBOT_FAKE_PROVIDER_PROVIDER_UUID
+  - LANGBOT_FAKE_PROVIDER_MODEL_UUID
+  - LANGBOT_FAKE_PROVIDER_PIPELINE_URL
+  - LANGBOT_FAKE_PROVIDER_PIPELINE_NAME
+steps:
+  - "Configure the local fake provider with deterministic slow streaming latency."
+  - "Create or update the LangBot provider, model, and local-agent pipeline that points at the fake provider."
+  - "Reset the target Debug Chat session."
+  - "Open concurrent WebSocket Debug Chat connections and send unique deterministic prompts through the real backend pipeline."
+checks:
+  - "automation-result.json status is pass when every request receives its own expected assistant response."
+  - "metrics_summary shows zero errors under the slow-provider profile."
+  - "thresholds_summary shows response_p95_ms, first_response_p95_ms, and error_rate pass."
+evidence_required:
+  - metrics
+  - network
+  - api_diagnostic
+  - filesystem
+diagnostics:
+  - "This probe keeps the model deterministic while injecting provider latency, so it catches backend timeout, streaming, and WebSocket backpressure issues without Space variability."
+  - "Compare with langbot-fake-provider-debug-chat-load to separate fixed LangBot overhead from provider-latency amplification."
+success_patterns:
+  - "Debug Chat WebSocket concurrency probe passed"
+  - "Streaming completed"
+failure_patterns:
+  - "WebSocket connection error"
+  - "Timed out after"
+  - "Final assistant response did not include"
+  - "All models failed during streaming setup"
+troubleshooting:
+  - backend-not-listening
+  - debug-chat-history-contaminates-automation
+  - local-agent-model-route-unavailable

skills/skills/langbot-testing/cases/langbot-fault-taxonomy-contract.yaml

@@ -0,0 +1,35 @@
+id: langbot-fault-taxonomy-contract
+title: "LangBot fault taxonomy and cleanup contract"
+mode: probe
+area: reliability
+type: chaos
+priority: p1
+risk: medium
+ci_eligible: true
+tags:
+  - reliability
+  - chaos
+  - contract
+  - synthetic
+skills:
+  - langbot-testing
+automation: skills/langbot-testing/probes/langbot-fault-taxonomy-contract.mjs
+fault_model_json: '{"kind":"taxonomy-contract","destructive":false,"scenarios":["provider-timeout","plugin-runtime-disconnect","mcp-stdio-server-exit","operator-missing-login","transient-marketplace-timeout"]}'
+steps:
+  - "Run `rtk bin/lbs test run langbot-fault-taxonomy-contract --dry-run` first; remove `--dry-run` after checking the evidence directory."
+  - "Automation validates that representative fault scenarios declare target, injected fault, expected status, recovery check, and cleanup."
+  - "Review metrics.json, fault-model.json, and automation-result.json under LBS_EVIDENCE_DIR."
+checks:
+  - "automation-result.json status is pass."
+  - "Every scenario has an expected status in pass, fail, blocked, env_issue, or flaky."
+  - "Every scenario declares a cleanup action and recovery check."
+evidence_required:
+  - metrics
+  - filesystem
+diagnostics:
+  - "This is a non-destructive taxonomy contract probe; it does not inject real runtime faults."
+  - "Use it as a gate before adding live chaos cases that kill runtimes, route traffic through a proxy, or disrupt a backend dependency."
+success_patterns:
+  - "Fault taxonomy contract declares status"
+failure_patterns:
+  - "missing required scenario fields"

skills/skills/langbot-testing/cases/langbot-live-backend-latency.yaml

@@ -0,0 +1,42 @@
+id: langbot-live-backend-latency
+title: "LangBot live backend basic latency probe"
+mode: probe
+area: performance
+type: performance
+priority: p1
+risk: medium
+ci_eligible: false
+tags:
+  - performance
+  - live-backend
+  - latency
+  - metrics
+skills:
+  - langbot-testing
+env:
+  - LANGBOT_BACKEND_URL
+automation: skills/langbot-testing/probes/langbot-live-backend-latency.mjs
+metrics_thresholds_json: '{"backend_p95_ms":{"max":1000},"error_rate":{"max":0}}'
+load_profile_json: '{"requests":12,"concurrency":2,"endpoints":["/healthz"]}'
+steps:
+  - "Confirm the selected LangBot backend is the intended test target."
+  - "Run `rtk bin/lbs test run langbot-live-backend-latency --dry-run` first; remove `--dry-run` after checking LANGBOT_BACKEND_URL and evidence directory."
+  - "Automation sends a small request batch to LANGBOT_BACKEND_URL/healthz and records latency, status counts, and network errors."
+checks:
+  - "automation-result.json status is pass when the backend responds and p95/error-rate thresholds pass."
+  - "automation-result.json status is env_issue when the backend is not reachable."
+  - "metrics.json and network.log are written under LBS_EVIDENCE_DIR."
+evidence_required:
+  - metrics
+  - network
+  - api_diagnostic
+  - filesystem
+diagnostics:
+  - "This probe measures backend health endpoint reachability latency only; it does not cover model/provider, browser, Debug Chat, RAG, or plugin runtime latency."
+success_patterns:
+  - "Live backend latency probe passed"
+failure_patterns:
+  - "Backend did not respond"
+  - "breached latency or error-rate thresholds"
+troubleshooting:
+  - socks-proxy-without-socksio

skills/skills/langbot-testing/cases/langbot-live-backend-log-health.yaml

@@ -0,0 +1,45 @@
+id: langbot-live-backend-log-health
+title: "LangBot live backend log health probe"
+mode: probe
+area: reliability
+type: reliability
+priority: p1
+risk: medium
+ci_eligible: false
+tags:
+  - reliability
+  - live-backend
+  - backend-log
+  - metrics
+skills:
+  - langbot-testing
+env:
+  - LANGBOT_BACKEND_URL
+automation: skills/langbot-testing/probes/langbot-live-backend-log-health.mjs
+metrics_thresholds_json: '{"fail_count":{"max":0}}'
+load_profile_json: '{"lookback_seconds":300,"log_source":"LANGBOT_BACKEND_LOG or latest LANGBOT_REPO/data/logs/langbot-*.log"}'
+steps:
+  - "Confirm the selected LangBot backend log belongs to the intended test target."
+  - "Run `rtk bin/lbs test run langbot-live-backend-log-health --dry-run` first; remove `--dry-run` after checking evidence directory and log source."
+  - "Automation scans the recent backend log window for fail-severity runtime findings such as Traceback, ImportError, ERROR, unclosed sessions, and unawaited coroutines."
+checks:
+  - "automation-result.json status is pass only when fail_count is 0."
+  - "metrics_summary includes scanned_line_count, fail_count, warning_count, and finding_count."
+  - "findings.json and scanned-backend.log are written under LBS_EVIDENCE_DIR."
+evidence_required:
+  - metrics
+  - backend_log
+  - filesystem
+diagnostics:
+  - "Set LANGBOT_BACKEND_LOG to an explicit log path when the latest log file is not the run target."
+  - "Set LANGBOT_BACKEND_LOG_SINCE or LANGBOT_BACKEND_LOG_LOOKBACK_SECONDS to control the scan window."
+  - "This probe measures runtime log health; it does not prove user-facing Debug Chat, plugin, model, or RAG behavior."
+success_patterns:
+  - "Live backend log health passed"
+failure_patterns:
+  - "Traceback"
+  - "ImportError"
+  - "ERROR"
+  - "unclosed"
+troubleshooting:
+  - socks-proxy-without-socksio

skills/skills/langbot-testing/cases/langbot-live-control-plane-api.yaml

@@ -0,0 +1,44 @@
+id: langbot-live-control-plane-api
+title: "LangBot live control-plane API probe"
+mode: probe
+area: performance
+type: performance
+priority: p1
+risk: medium
+ci_eligible: false
+tags:
+  - performance
+  - reliability
+  - live-backend
+  - control-plane
+  - metrics
+skills:
+  - langbot-testing
+env:
+  - LANGBOT_BACKEND_URL
+automation: skills/langbot-testing/probes/langbot-live-control-plane-api.mjs
+metrics_thresholds_json: '{"error_rate":{"max":0},"response_shape_failures":{"max":0},"healthz_p95_ms":{"max":500},"system_info_p95_ms":{"max":1000}}'
+load_profile_json: '{"requests":20,"concurrency":4,"endpoints":["/healthz","/api/v1/system/info"],"auth_required":false}'
+steps:
+  - "Confirm the selected LangBot backend is the intended test target."
+  - "Run `rtk bin/lbs test run langbot-live-control-plane-api --dry-run` first; remove `--dry-run` after checking LANGBOT_BACKEND_URL and evidence directory."
+  - "Automation sends a small request batch to /healthz and /api/v1/system/info, then validates status code, JSON shape, and latency budgets."
+checks:
+  - "automation-result.json status is pass when every control-plane request returns HTTP 200, JSON code 0, and required response fields."
+  - "metrics_summary includes per-endpoint p50/p95 latency, error rate, status counts, and response_shape_failures."
+  - "thresholds_summary shows error_rate, response_shape_failures, healthz_p95_ms, and system_info_p95_ms all pass."
+evidence_required:
+  - metrics
+  - network
+  - api_diagnostic
+  - filesystem
+diagnostics:
+  - "This probe measures unauthenticated backend control-plane readiness; it does not cover authenticated UI flows, Debug Chat, model calls, plugins, or RAG."
+  - "A system_info shape failure usually means the API contract or startup state changed and should be investigated before treating latency as healthy."
+success_patterns:
+  - "Live control-plane API probe passed"
+failure_patterns:
+  - "Backend did not respond"
+  - "breached shape, latency, or error-rate thresholds"
+troubleshooting:
+  - socks-proxy-without-socksio

skills/skills/langbot-testing/cases/langbot-overhead-accounting-contract.yaml

@@ -0,0 +1,37 @@
+id: langbot-overhead-accounting-contract
+title: "LangBot overhead accounting metrics contract"
+mode: probe
+area: performance
+type: performance
+priority: p1
+risk: medium
+ci_eligible: true
+tags:
+  - performance
+  - metrics
+  - contract
+  - synthetic
+skills:
+  - langbot-testing
+automation: skills/langbot-testing/probes/langbot-overhead-accounting-contract.mjs
+metrics_thresholds_json: '{"sample_count":{"min":50},"langbot_overhead_p95_ms":{"max":25},"accounting_gap_max_ms":{"max":0.001}}'
+load_profile_json: '{"kind":"synthetic-overhead-accounting","samples":80,"external_latency_segments":["provider","external_tool","network"]}'
+steps:
+  - "Run `rtk bin/lbs test run langbot-overhead-accounting-contract --dry-run` first; remove `--dry-run` after checking the evidence directory."
+  - "Automation generates deterministic message-path latency samples and separates LangBot overhead from provider/tool/network latency."
+  - "Review metrics.json, thresholds.json, resource-log.json, and automation-result.json under LBS_EVIDENCE_DIR."
+checks:
+  - "automation-result.json status is pass."
+  - "metrics_summary includes sample_count, langbot_overhead_p95_ms, e2e_latency_p95_ms, external_latency_p95_ms, and accounting_gap_max_ms."
+  - "thresholds_summary shows sample_count, langbot_overhead_p95_ms, and accounting_gap_max_ms all pass."
+evidence_required:
+  - metrics
+  - resource_log
+  - filesystem
+diagnostics:
+  - "This is a synthetic contract probe for the QA harness; it is not live product performance."
+  - "Use it to verify that reports can carry overhead accounting metrics before running live backend or browser performance probes."
+success_patterns:
+  - "Overhead accounting contract passed"
+failure_patterns:
+  - "breached one or more thresholds"

skills/skills/langbot-testing/cases/langbot-space-debug-chat-concurrency-smoke.yaml

@@ -0,0 +1,84 @@
+id: langbot-space-debug-chat-concurrency-smoke
+title: "LangBot Debug Chat real Space-provider concurrency smoke"
+mode: probe
+area: performance
+type: performance
+priority: p1
+risk: high
+ci_eligible: false
+tags:
+  - performance
+  - debug-chat
+  - websocket
+  - space
+  - live-provider
+  - smoke
+  - metrics
+skills:
+  - langbot-env-setup
+  - langbot-testing
+env:
+  - LANGBOT_BACKEND_URL
+  - LANGBOT_FRONTEND_URL
+  - LANGBOT_E2E_LOGIN_USER
+automation: skills/langbot-testing/probes/langbot-debug-chat-concurrency.mjs
+automation_env:
+  - LANGBOT_BACKEND_URL
+  - LANGBOT_E2E_LOGIN_USER
+  - LANGBOT_LOCAL_AGENT_PIPELINE_URL
+  - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL
+automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+automation_debug_chat_load_requests: "3"
+automation_debug_chat_load_concurrency: "2"
+automation_debug_chat_load_timeout_ms: "120000"
+automation_debug_chat_load_response_p95_ms: "120000"
+automation_debug_chat_load_max_error_rate: "0"
+automation_debug_chat_load_expected_prefix: "SPACEQA"
+automation_debug_chat_load_prompt_template: '请只回复 "{expected}"，不要解释，不要添加其他字符。'
+automation_debug_chat_load_stream: "true"
+automation_debug_chat_load_reset: "true"
+metrics_thresholds_json: '{"response_p95_ms":{"max":120000},"error_rate":{"max":0}}'
+load_profile_json: '{"requests":3,"concurrency":2,"path":"Pipeline Debug Chat WebSocket","provider":"LangBot Space model route","metric":"send-to-final-assistant-response","classification":"smoke-not-benchmark"}'
+setup_automation:
+  - "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env"
+setup_provides_env:
+  - LANGBOT_PIPELINE_URL
+  - LANGBOT_PIPELINE_NAME
+  - LANGBOT_LOCAL_AGENT_PIPELINE_URL
+  - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+  - LANGBOT_LOCAL_AGENT_MODEL_UUID
+  - LANGBOT_E2E_MODEL_UUID
+preconditions:
+  - "The selected local LangBot instance is safe for a low-volume real Space model smoke run."
+  - "Treat Space/provider/network failures as environment or dependency findings until fake-provider baseline evidence separates LangBot overhead."
+steps:
+  - "Prepare a local-agent pipeline with a tested Space model and fallback models."
+  - "Reset the target Debug Chat session."
+  - "Open a small number of concurrent WebSocket Debug Chat connections and send unique deterministic prompts through the live Space provider path."
+checks:
+  - "automation-result.json status is pass when every request receives its own expected assistant response."
+  - "metrics_summary includes request count, concurrency, p95 response latency, throughput, and error rate."
+  - "The report classifies the result as a live-provider smoke, not a stable LangBot overhead benchmark."
+evidence_required:
+  - metrics
+  - network
+  - api_diagnostic
+  - filesystem
+diagnostics:
+  - "This probe measures real user-path latency through Space and includes provider latency, model behavior, and network effects."
+  - "Compare with langbot-fake-provider-debug-chat-load before attributing slow or failed runs to LangBot itself."
+success_patterns:
+  - "Debug Chat WebSocket concurrency probe passed"
+  - "Streaming completed"
+failure_patterns:
+  - "invalid api key"
+  - "WebSocket connection error"
+  - "Timed out after"
+  - "Final assistant response did not include"
+  - "All models failed during streaming setup"
+troubleshooting:
+  - local-agent-model-route-unavailable
+  - marketplace-network-flaky
+  - proxy-env-mismatch
+  - telemetry-proxy-noise

skills/skills/langbot-testing/cases/pipeline-debug-chat-performance.yaml

@@ -0,0 +1,80 @@
+id: pipeline-debug-chat-performance
+title: "Pipeline Debug Chat user-path performance probe"
+mode: agent-browser
+area: pipeline
+type: performance
+priority: p1
+risk: medium
+ci_eligible: false
+tags:
+  - performance
+  - pipeline
+  - debug-chat
+  - user-path
+  - metrics
+skills:
+  - langbot-env-setup
+  - langbot-testing
+env:
+  - LANGBOT_FRONTEND_URL
+  - LANGBOT_BACKEND_URL
+env_any:
+  - LANGBOT_PIPELINE_URL|LANGBOT_PIPELINE_NAME
+automation: scripts/e2e/pipeline-debug-chat.mjs
+automation_env:
+  - LANGBOT_FRONTEND_URL
+  - LANGBOT_BACKEND_URL
+  - LANGBOT_BROWSER_PROFILE
+  - LANGBOT_CHROMIUM_EXECUTABLE
+  - LANGBOT_E2E_PROMPT
+  - LANGBOT_E2E_EXPECTED_TEXT
+  - LANGBOT_E2E_RESPONSE_TIMEOUT_MS
+automation_env_any:
+  - LANGBOT_PIPELINE_URL|LANGBOT_PIPELINE_NAME
+automation_prompt: "请只回复 OK，用于性能测试。"
+automation_expected_text: "OK"
+automation_response_timeout_ms: "120000"
+automation_reset_debug_chat: "true"
+automation_debug_chat_response_p95_ms: "120000"
+automation_debug_chat_max_error_rate: "0"
+metrics_thresholds_json: '{"response_p95_ms":{"max":120000},"error_rate":{"max":0}}'
+load_profile_json: '{"prompts":1,"browser":true,"path":"Pipeline Debug Chat","metric":"send-to-visible-completion"}'
+setup_automation:
+  - "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env"
+setup_provides_env:
+  - LANGBOT_PIPELINE_URL
+  - LANGBOT_PIPELINE_NAME
+preconditions:
+  - "LANGBOT_PIPELINE_URL or LANGBOT_PIPELINE_NAME points to the pipeline intended for this Debug Chat performance run."
+  - "The target pipeline is safe to reset Debug Chat history for this run."
+  - "The target pipeline has a known-good runner/model; provider latency should be interpreted separately from LangBot overhead."
+steps:
+  - "Open LANGBOT_FRONTEND_URL with the prepared browser profile."
+  - "Open the target pipeline and select Debug Chat."
+  - "Reset Debug Chat history through the backend API when configured."
+  - "Send the deterministic prompt and wait for the expected assistant response."
+checks:
+  - "automation-result.json status is pass when the expected assistant response appears."
+  - "metrics_summary includes response_p50_ms, response_p95_ms, error_rate, and total_duration_ms."
+  - "thresholds_summary shows response_p95_ms and error_rate pass."
+evidence_required:
+  - ui
+  - screenshot
+  - console
+  - network
+  - metrics
+diagnostics:
+  - "This case measures browser-visible send-to-completion latency; it does not split provider latency from LangBot overhead."
+  - "Use backend logs and provider diagnostics to explain slow runs before calling them LangBot regressions."
+success_patterns:
+  - "Processing request from person_websocket"
+  - "Streaming completed"
+failure_patterns:
+  - "Action invoke_llm_stream call timed out"
+  - "Task exception was never retrieved"
+  - "All models failed during streaming setup"
+troubleshooting:
+  - debug-chat-history-contaminates-automation
+  - local-agent-model-route-unavailable
+  - plugin-runtime-timeout
+  - proxy-env-mismatch

skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/.gitignore

@@ -1 +1,3 @@
-dist/
+dist/*
+!dist/
+!dist/qa-plugin-smoke-0.1.0.lbpkg

skills/skills/langbot-testing/probes/langbot-debug-chat-concurrency.mjs

@@ -0,0 +1,837 @@
+#!/usr/bin/env node
+
+import crypto from "node:crypto";
+import net from "node:net";
+import tls from "node:tls";
+import { mkdir, writeFile } from "node:fs/promises";
+import { join, resolve } from "node:path";
+import { env, exit } from "node:process";
+import {
+  apiJson,
+  appendLine,
+  ensureEvidence,
+  evidencePaths,
+  loadEnvFiles,
+  localIsoWithOffset,
+  redact,
+  resetAndAuthLocalUser,
+  writeResult,
+} from "../../../scripts/e2e/lib/langbot-e2e.mjs";
+import {
+  buildProviderTimingMetrics,
+  summarizeFakeProviderState,
+} from "./lib/fake-provider-timing.mjs";
+
+const DEFAULT_LOCAL_PASSWORD = "LangBotE2ELocalPass!2026";
+
+await loadEnvFiles();
+const caseId = env.LBS_CASE_ID || "langbot-debug-chat-concurrency";
+const paths = evidencePaths(caseId);
+await ensureEvidence(paths);
+
+const startedAt = new Date();
+const metricsPath = resolve(paths.evidenceDir, "metrics.json");
+const samplesPath = resolve(paths.evidenceDir, "samples.json");
+const fakeProviderStatePath = resolve(paths.evidenceDir, "fake-provider-state.json");
+const resetDiagnosticPath = resolve(paths.evidenceDir, "debug-chat-reset-diagnostic.json");
+const backendUrl = env.LANGBOT_BACKEND_URL || "";
+const fakeProviderUrl = env.LANGBOT_FAKE_PROVIDER_URL || "";
+const pipelineUrl = env.LANGBOT_E2E_PIPELINE_URL || env.LANGBOT_PIPELINE_URL || "";
+const pipelineName = env.LANGBOT_E2E_PIPELINE_NAME || env.LANGBOT_PIPELINE_NAME || "";
+const sessionType = env.LANGBOT_DEBUG_CHAT_LOAD_SESSION_TYPE || env.LANGBOT_E2E_DEBUG_CHAT_SESSION_TYPE || "person";
+const totalRequests = positiveInteger(env.LANGBOT_DEBUG_CHAT_LOAD_REQUESTS, defaultRequests(caseId));
+const concurrency = Math.min(totalRequests, positiveInteger(env.LANGBOT_DEBUG_CHAT_LOAD_CONCURRENCY, defaultConcurrency(caseId)));
+const timeoutMs = positiveInteger(env.LANGBOT_DEBUG_CHAT_LOAD_TIMEOUT_MS, defaultTimeout(caseId));
+const expectedPrefix = env.LANGBOT_DEBUG_CHAT_LOAD_EXPECTED_PREFIX || "LBQA";
+const promptTemplate = env.LANGBOT_DEBUG_CHAT_LOAD_PROMPT_TEMPLATE
+  || "请只回复 \"{expected}\"，不要解释，不要添加其他字符。";
+const stream = bool(env.LANGBOT_DEBUG_CHAT_LOAD_STREAM, true);
+const resetBeforeRun = bool(env.LANGBOT_DEBUG_CHAT_LOAD_RESET, true);
+const responseP95BudgetMs = positiveNumber(env.LANGBOT_DEBUG_CHAT_LOAD_RESPONSE_P95_MS, defaultP95Budget(caseId));
+const firstResponseP95BudgetMs = positiveNumber(env.LANGBOT_DEBUG_CHAT_LOAD_FIRST_RESPONSE_P95_MS, 0);
+const maxErrorRate = positiveNumber(env.LANGBOT_DEBUG_CHAT_LOAD_MAX_ERROR_RATE, 0);
+const minErrorRate = positiveNumber(env.LANGBOT_DEBUG_CHAT_LOAD_MIN_ERROR_RATE, 0);
+const minErrorCount = nonNegativeInteger(env.LANGBOT_DEBUG_CHAT_LOAD_MIN_ERROR_COUNT, 0);
+const minOkCount = nonNegativeInteger(env.LANGBOT_DEBUG_CHAT_LOAD_MIN_OK_COUNT, 0);
+const minProviderFaultCount = nonNegativeInteger(env.LANGBOT_DEBUG_CHAT_LOAD_MIN_PROVIDER_FAULT_COUNT, 0);
+const failOnFinalMismatch = bool(env.LANGBOT_DEBUG_CHAT_LOAD_FAIL_ON_FINAL_MISMATCH, false);
+const failureSignals = textList(env.LANGBOT_E2E_FAILURE_SIGNALS || env.LANGBOT_DEBUG_CHAT_LOAD_FAILURE_SIGNALS || "");
+
+const result = {
+  source: "automation",
+  case_id: caseId,
+  run_id: paths.runId,
+  status: "fail",
+  reason: "",
+  started_at: startedAt.toISOString(),
+  started_at_local: localIsoWithOffset(startedAt),
+  finished_at: "",
+  finished_at_local: "",
+  duration_ms: 0,
+  backend_url: backendUrl,
+  pipeline_url: pipelineUrl,
+  pipeline_name: pipelineName,
+  pipeline_id: "",
+  session_type: sessionType,
+  load_profile: {
+    requests: totalRequests,
+    concurrency,
+    timeout_ms: timeoutMs,
+    stream,
+    reset_before_run: resetBeforeRun,
+    fail_on_final_mismatch: failOnFinalMismatch,
+  },
+  evidence: {
+    network_log: paths.networkLog,
+    metrics_json: metricsPath,
+    samples_json: samplesPath,
+    fake_provider_state_json: fakeProviderStatePath,
+    debug_chat_reset_diagnostic_json: resetDiagnosticPath,
+    automation_result_json: paths.automationResultJson,
+    result_json: paths.resultJson,
+  },
+  evidence_collected: ["metrics", "network", "api_diagnostic", "filesystem"],
+};
+
+try {
+  if (!backendUrl) {
+    result.status = "env_issue";
+    throw new Error("LANGBOT_BACKEND_URL is not configured.");
+  }
+  if (!["person", "group"].includes(sessionType)) {
+    throw new Error(`LANGBOT_DEBUG_CHAT_LOAD_SESSION_TYPE must be person or group, got ${sessionType}.`);
+  }
+  const backendReady = await backendReachable(backendUrl);
+  if (!backendReady) {
+    result.status = "env_issue";
+    throw new Error(`Backend did not respond at ${backendUrl}.`);
+  }
+
+  const user = env.LANGBOT_E2E_LOGIN_USER || "";
+  const password = env.LANGBOT_E2E_LOGIN_PASSWORD || DEFAULT_LOCAL_PASSWORD;
+  if (!user) {
+    result.status = "env_issue";
+    throw new Error("LANGBOT_E2E_LOGIN_USER is required so this probe can resolve/reset the Debug Chat session.");
+  }
+  const auth = await resetAndAuthLocalUser({ backendUrl, user, password });
+
+  const pipeline = await resolvePipeline({ backendUrl, token: auth.token, pipelineUrl, pipelineName });
+  result.pipeline_id = pipeline.id;
+  result.pipeline_name = pipeline.name || pipelineName;
+  if (!result.pipeline_url && env.LANGBOT_FRONTEND_URL) {
+    result.pipeline_url = `${env.LANGBOT_FRONTEND_URL.replace(/\/$/, "")}/home/pipelines?id=${encodeURIComponent(pipeline.id)}`;
+  }
+
+  if (resetBeforeRun) {
+    const reset = await apiJson(backendUrl, `/api/v1/pipelines/${encodeURIComponent(pipeline.id)}/ws/reset/${encodeURIComponent(sessionType)}`, {
+      method: "POST",
+      token: auth.token,
+    });
+    const resetDiagnostic = {
+      status: isApiFailure(reset) ? "fail" : "ready",
+      http_status: reset.status,
+      code: reset.json.code ?? null,
+      reason: isApiFailure(reset) ? reset.json.msg || "Debug Chat reset failed." : "Debug Chat session reset.",
+    };
+    await writeFile(resetDiagnosticPath, `${JSON.stringify(resetDiagnostic, null, 2)}\n`, "utf8");
+    if (resetDiagnostic.status === "fail") {
+      throw new Error(resetDiagnostic.reason);
+    }
+  }
+
+  const wsUrl = websocketUrl(backendUrl, pipeline.id, sessionType);
+  const loadStartedAt = performance.now();
+  const samples = await runLoad({
+    wsUrl,
+    totalRequests,
+    concurrency,
+    timeoutMs,
+    promptTemplate,
+    expectedPrefix,
+    stream,
+    failOnFinalMismatch,
+    failureSignals,
+  });
+  const loadDurationMs = performance.now() - loadStartedAt;
+  const fakeProviderState = await readFakeProviderState(fakeProviderUrl);
+  if (fakeProviderState) {
+    await writeFile(fakeProviderStatePath, `${JSON.stringify(fakeProviderState, null, 2)}\n`, "utf8");
+  }
+  const metrics = buildMetrics({
+    samples,
+    totalRequests,
+    concurrency,
+    timeoutMs,
+    loadDurationMs,
+    backendUrl,
+    pipelineId: pipeline.id,
+    sessionType,
+    fakeProviderState,
+  });
+  const thresholds = buildThresholds(metrics);
+  const passed = Object.values(thresholds).every((item) => item.pass);
+  result.status = passed ? "pass" : "fail";
+  result.reason = passed
+    ? "Debug Chat WebSocket concurrency probe passed all thresholds."
+    : "Debug Chat WebSocket concurrency probe breached latency or error-rate thresholds.";
+  result.metrics_summary = {
+    requests: metrics.total_requests,
+    concurrency: metrics.concurrency,
+    ok_count: metrics.ok_count,
+    error_count: metrics.error_count,
+    timeout_count: metrics.timeout_count,
+    error_rate: metrics.error_rate,
+    response_p50_ms: metrics.response_duration_ms.p50,
+    response_p95_ms: metrics.response_duration_ms.p95,
+    first_assistant_event_p95_ms: metrics.first_assistant_event_ms.p95,
+    first_assistant_content_p95_ms: metrics.first_assistant_content_ms.p95,
+    first_response_p95_ms: metrics.first_response_ms.p95,
+    throughput_rps: metrics.throughput_rps,
+    status_counts: metrics.status_counts,
+    fake_provider_request_count: metrics.fake_provider?.request_count ?? null,
+    fake_provider_fault_count: metrics.fake_provider?.fault_count ?? null,
+    fake_provider_duration_p95_ms: metrics.provider_timing?.provider_duration_ms.p95 ?? null,
+    langbot_overhead_estimate_p95_ms: metrics.provider_timing?.langbot_overhead_estimate_ms.p95 ?? null,
+    send_to_provider_start_p95_ms: metrics.provider_timing?.send_to_provider_start_ms.p95 ?? null,
+    provider_finish_to_ws_final_p95_ms: metrics.provider_timing?.provider_finish_to_ws_final_ms.p95 ?? null,
+    provider_timing_matched_request_count: metrics.provider_timing?.matched_request_count ?? null,
+  };
+  result.thresholds_summary = thresholds;
+  result.artifacts = {
+    metrics_json: metricsPath,
+    samples_json: samplesPath,
+    fake_provider_state_json: fakeProviderState ? fakeProviderStatePath : "",
+    network_log: paths.networkLog,
+    automation_result_json: paths.automationResultJson,
+    result_json: paths.resultJson,
+  };
+
+  await writeFile(metricsPath, `${JSON.stringify({ ...metrics, thresholds }, null, 2)}\n`, "utf8");
+  await writeFile(samplesPath, `${JSON.stringify(samples, null, 2)}\n`, "utf8");
+} catch (error) {
+  if (!["env_issue", "blocked"].includes(result.status)) {
+    result.status = looksLikeEnvIssue(error) ? "env_issue" : "fail";
+  }
+  result.reason = result.reason || safeReason(error.message);
+} finally {
+  const finishedAt = new Date();
+  result.finished_at = finishedAt.toISOString();
+  result.finished_at_local = localIsoWithOffset(finishedAt);
+  result.duration_ms = finishedAt.getTime() - startedAt.getTime();
+  await mkdir(paths.evidenceDir, { recursive: true });
+  await writeResult(paths, result);
+  console.log(JSON.stringify(result, null, 2));
+}
+
+exit(result.status === "pass" ? 0 : result.status === "env_issue" || result.status === "blocked" ? 2 : 1);
+
+function defaultRequests(id) {
+  return id.includes("space") ? 3 : 12;
+}
+
+function defaultConcurrency(id) {
+  return id.includes("space") ? 1 : 4;
+}
+
+function defaultTimeout(id) {
+  return id.includes("space") ? 120_000 : 30_000;
+}
+
+function defaultP95Budget(id) {
+  return id.includes("space") ? 120_000 : 5_000;
+}
+
+function positiveInteger(value, fallback) {
+  const parsed = Number.parseInt(String(value || ""), 10);
+  return Number.isInteger(parsed) && parsed > 0 ? parsed : fallback;
+}
+
+function nonNegativeInteger(value, fallback) {
+  const parsed = Number.parseInt(String(value ?? ""), 10);
+  return Number.isInteger(parsed) && parsed >= 0 ? parsed : fallback;
+}
+
+function positiveNumber(value, fallback) {
+  const parsed = Number(value || "");
+  return Number.isFinite(parsed) && parsed >= 0 ? parsed : fallback;
+}
+
+function bool(value, fallback) {
+  if (value === undefined || value === "") return fallback;
+  if (/^(1|true|yes|on)$/i.test(String(value))) return true;
+  if (/^(0|false|no|off)$/i.test(String(value))) return false;
+  return fallback;
+}
+
+function textList(value) {
+  return String(value || "")
+    .split(/\r?\n|,/)
+    .map((item) => item.trim())
+    .filter(Boolean);
+}
+
+async function backendReachable(baseUrl) {
+  try {
+    const response = await fetch(`${baseUrl.replace(/\/$/, "")}/healthz`, {
+      signal: AbortSignal.timeout(3000),
+    });
+    return response.status < 500;
+  } catch {
+    return false;
+  }
+}
+
+async function readFakeProviderState(rootUrl) {
+  if (!rootUrl) return null;
+  try {
+    const response = await fetch(`${normalizeProviderRootUrl(rootUrl)}/__qa/config`, {
+      signal: AbortSignal.timeout(3000),
+    });
+    const json = await response.json().catch(() => ({}));
+    return {
+      status: response.ok && json.ok === true ? "loaded" : "unavailable",
+      url: normalizeProviderRootUrl(rootUrl),
+      http_status: response.status,
+      model: json.model || "",
+      config: json.config || {},
+      request_count: Number.isFinite(json.request_count) ? json.request_count : null,
+      recent_requests: Array.isArray(json.recent_requests) ? json.recent_requests : [],
+    };
+  } catch (error) {
+    return {
+      status: "unavailable",
+      url: normalizeProviderRootUrl(rootUrl),
+      reason: safeReason(error.message),
+      request_count: null,
+      recent_requests: [],
+    };
+  }
+}
+
+function normalizeProviderRootUrl(value) {
+  const trimmed = String(value || "").trim().replace(/\/$/, "");
+  return trimmed.endsWith("/v1") ? trimmed.slice(0, -3) : trimmed;
+}
+
+function pipelineIdFromUrl(url) {
+  if (!url) return "";
+  try {
+    const parsed = new URL(url);
+    return parsed.searchParams.get("id") || "";
+  } catch {
+    return "";
+  }
+}
+
+async function resolvePipeline({ backendUrl, token, pipelineUrl, pipelineName }) {
+  const idFromUrl = pipelineIdFromUrl(pipelineUrl);
+  if (idFromUrl) {
+    const response = await apiJson(backendUrl, `/api/v1/pipelines/${encodeURIComponent(idFromUrl)}`, { token });
+    const pipeline = response.json.data?.pipeline;
+    if (isApiFailure(response) || !pipeline?.uuid) {
+      throw new Error(response.json.msg || `Could not load pipeline ${idFromUrl}.`);
+    }
+    return { id: pipeline.uuid, name: pipeline.name || "" };
+  }
+  if (!pipelineName) {
+    throw new Error("Set LANGBOT_E2E_PIPELINE_URL or LANGBOT_E2E_PIPELINE_NAME before running this probe.");
+  }
+  const response = await apiJson(backendUrl, "/api/v1/pipelines", { token });
+  if (isApiFailure(response)) {
+    throw new Error(response.json.msg || "Failed to list pipelines.");
+  }
+  const pipeline = (response.json.data?.pipelines || []).find((item) => item.name === pipelineName);
+  if (!pipeline?.uuid) {
+    throw new Error(`Could not find pipeline named ${pipelineName}.`);
+  }
+  return { id: pipeline.uuid, name: pipeline.name || pipelineName };
+}
+
+function isApiFailure(response) {
+  return response.status >= 400 || (response.json.code !== undefined && response.json.code !== 0);
+}
+
+function websocketUrl(baseUrl, pipelineId, sessionType) {
+  const parsed = new URL(baseUrl);
+  parsed.protocol = parsed.protocol === "https:" ? "wss:" : "ws:";
+  parsed.pathname = `/api/v1/pipelines/${encodeURIComponent(pipelineId)}/ws/connect`;
+  parsed.search = `?session_type=${encodeURIComponent(sessionType)}`;
+  return parsed.toString();
+}
+
+async function runLoad(options) {
+  const samples = [];
+  let nextIndex = 0;
+  const workers = Array.from({ length: options.concurrency }, async () => {
+    while (nextIndex < options.totalRequests) {
+      const index = nextIndex;
+      nextIndex += 1;
+      const sample = await runSingleRequest({ ...options, index });
+      samples.push(sample);
+    }
+  });
+  await Promise.all(workers);
+  return samples.sort((left, right) => left.index - right.index);
+}
+
+function expectedForIndex(prefix, index) {
+  return `${prefix}-${String(index + 1).padStart(4, "0")}`;
+}
+
+function promptForIndex(template, expected) {
+  return template.replaceAll("{expected}", expected);
+}
+
+function runSingleRequest({
+  wsUrl,
+  index,
+  timeoutMs,
+  promptTemplate,
+  expectedPrefix,
+  stream,
+  failOnFinalMismatch,
+  failureSignals,
+}) {
+  return new Promise((resolve) => {
+    const expected = expectedForIndex(expectedPrefix, index);
+    const prompt = promptForIndex(promptTemplate, expected);
+    const sample = {
+      index,
+      status: "running",
+      ok: false,
+      expected_text: expected,
+      prompt,
+      response_text: "",
+      started_at: new Date().toISOString(),
+      started_epoch_ms: Date.now(),
+      connected_at: null,
+      connected_epoch_ms: null,
+      sent_at: null,
+      sent_epoch_ms: null,
+      first_assistant_event_at: null,
+      first_assistant_event_epoch_ms: null,
+      first_assistant_event_ms: null,
+      first_assistant_content_at: null,
+      first_assistant_content_epoch_ms: null,
+      first_assistant_content_ms: null,
+      first_response_at: null,
+      first_response_epoch_ms: null,
+      connected_ms: null,
+      first_response_ms: null,
+      response_duration_ms: null,
+      finished_at: null,
+      finished_epoch_ms: null,
+      event_count: 0,
+      foreign_response_count: 0,
+      last_foreign_response_text: "",
+      error: "",
+      close_code: null,
+      close_reason: "",
+    };
+    let closed = false;
+    let connectedAt = 0;
+    let sentAt = 0;
+    const startedAt = performance.now();
+    let client = null;
+    const timer = setTimeout(() => {
+      finish("timeout", `Timed out after ${timeoutMs} ms.`);
+    }, timeoutMs);
+
+    client = openRawWebSocket(wsUrl, {
+      onOpen() {
+        connectedAt = performance.now();
+        const now = Date.now();
+        sample.connected_at = new Date(now).toISOString();
+        sample.connected_epoch_ms = now;
+        sample.connected_ms = rounded(connectedAt - startedAt);
+      },
+      onMessage(text) {
+        sample.event_count += 1;
+        let data;
+        try {
+          data = JSON.parse(String(text || ""));
+        } catch (error) {
+          finish("error", `Invalid WebSocket JSON: ${error.message}`);
+          return;
+        }
+        appendLine(paths.networkLog, JSON.stringify({
+          request_index: index,
+          type: data.type,
+          session_type: data.session_type || "",
+          role: data.data?.role || "",
+          is_final: data.data?.is_final ?? null,
+          content_preview: redact(String(data.data?.content || data.message || "").slice(0, 200)),
+        })).catch(() => {});
+
+        if (data.type === "connected") {
+          sentAt = performance.now();
+          const now = Date.now();
+          sample.sent_at = new Date(now).toISOString();
+          sample.sent_epoch_ms = now;
+          client.send(JSON.stringify({
+            type: "message",
+            message: [{ type: "Plain", text: prompt }],
+            stream,
+          }));
+          return;
+        }
+        if (data.type === "error") {
+          finish("error", data.message || "WebSocket error message.");
+          return;
+        }
+        if (data.type !== "response" || data.data?.role !== "assistant") return;
+
+        const content = String(data.data.content || "");
+        markFirstAssistantEvent(sample, sentAt);
+        if (content) sample.response_text = content;
+        if (content) markFirstAssistantContent(sample, sentAt);
+        if (content.includes(expected) && sample.first_response_ms === null && sentAt > 0) {
+          const now = Date.now();
+          sample.first_response_at = new Date(now).toISOString();
+          sample.first_response_epoch_ms = now;
+          sample.first_response_ms = rounded(performance.now() - sentAt);
+        }
+        if (data.data.is_final === true) {
+          const ok = sample.response_text.includes(expected);
+          if (ok) {
+            if (sample.first_response_ms === null && sentAt > 0) {
+              sample.first_response_ms = rounded(performance.now() - sentAt);
+            }
+            finish("pass", "");
+          } else if (matchesFailureSignal(sample.response_text, failureSignals)) {
+            finish("app_error", `Assistant final response matched a failure signal: ${sample.response_text}`);
+          } else if (failOnFinalMismatch && !containsLoadToken(sample.response_text, expectedPrefix)) {
+            finish("mismatch", `Final assistant response did not include ${expected}: ${sample.response_text}`);
+          } else {
+            sample.foreign_response_count += 1;
+            sample.last_foreign_response_text = sample.response_text;
+          }
+        }
+      },
+      onError(error) {
+        finish("connection_error", `WebSocket connection error: ${error.message}`);
+      },
+      onClose(event) {
+        sample.close_code = event.code;
+        sample.close_reason = event.reason || "";
+        if (!closed) finish("closed", `WebSocket closed before final assistant response: ${event.code}`);
+      },
+    });
+
+    function finish(status, reason) {
+      if (closed) return;
+      closed = true;
+      clearTimeout(timer);
+      sample.status = status;
+      sample.ok = status === "pass";
+      sample.error = status === "timeout" && sample.foreign_response_count > 0
+        ? `${reason || ""} Saw ${sample.foreign_response_count} foreign assistant response(s); last=${sample.last_foreign_response_text}`
+        : reason || "";
+      if (sentAt > 0) sample.response_duration_ms = rounded(performance.now() - sentAt);
+      else sample.response_duration_ms = rounded(performance.now() - startedAt);
+      const now = Date.now();
+      sample.finished_at = new Date(now).toISOString();
+      sample.finished_epoch_ms = now;
+      try {
+        client?.close();
+      } catch {
+        // Closing a failed socket should not hide the sample result.
+      }
+      resolve(sample);
+    }
+  });
+}
+
+function markFirstAssistantEvent(sample, sentAt) {
+  if (sample.first_assistant_event_ms !== null || sentAt <= 0) return;
+  const now = Date.now();
+  sample.first_assistant_event_at = new Date(now).toISOString();
+  sample.first_assistant_event_epoch_ms = now;
+  sample.first_assistant_event_ms = rounded(performance.now() - sentAt);
+}
+
+function markFirstAssistantContent(sample, sentAt) {
+  if (sample.first_assistant_content_ms !== null || sentAt <= 0) return;
+  const now = Date.now();
+  sample.first_assistant_content_at = new Date(now).toISOString();
+  sample.first_assistant_content_epoch_ms = now;
+  sample.first_assistant_content_ms = rounded(performance.now() - sentAt);
+}
+
+function containsLoadToken(text, prefix) {
+  const escaped = String(prefix).replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
+  return new RegExp(`${escaped}-\\d{4}`).test(String(text || ""));
+}
+
+function matchesFailureSignal(text, signals) {
+  const lower = String(text || "").toLowerCase();
+  return signals.some((signal) => lower.includes(signal.toLowerCase()));
+}
+
+function openRawWebSocket(wsUrl, handlers) {
+  const parsed = new URL(wsUrl);
+  const secure = parsed.protocol === "wss:";
+  const port = Number(parsed.port || (secure ? 443 : 80));
+  const host = parsed.hostname;
+  const path = `${parsed.pathname}${parsed.search}`;
+  const key = crypto.randomBytes(16).toString("base64");
+  const socket = secure
+    ? tls.connect({ host, port, servername: host })
+    : net.connect({ host, port });
+  let opened = false;
+  let closed = false;
+  let buffer = Buffer.alloc(0);
+
+  socket.setNoDelay(true);
+  socket.on("connect", () => {
+    const originProtocol = secure ? "https" : "http";
+    const request = [
+      `GET ${path} HTTP/1.1`,
+      `Host: ${parsed.host}`,
+      "Upgrade: websocket",
+      "Connection: Upgrade",
+      `Sec-WebSocket-Key: ${key}`,
+      "Sec-WebSocket-Version: 13",
+      `Origin: ${originProtocol}://${parsed.host}`,
+      "",
+      "",
+    ].join("\r\n");
+    socket.write(request);
+  });
+  socket.on("data", (chunk) => {
+    buffer = Buffer.concat([buffer, chunk]);
+    if (!opened) {
+      const headerEnd = buffer.indexOf("\r\n\r\n");
+      if (headerEnd === -1) return;
+      const headerText = buffer.slice(0, headerEnd).toString("utf8");
+      buffer = buffer.slice(headerEnd + 4);
+      if (!/^HTTP\/1\.1 101\b/i.test(headerText)) {
+        handlers.onError(new Error(`Handshake failed: ${headerText.split("\r\n")[0] || "missing status"}`));
+        socket.destroy();
+        return;
+      }
+      opened = true;
+      handlers.onOpen();
+    }
+    processFrames();
+  });
+  socket.on("error", (error) => {
+    if (!closed) handlers.onError(error);
+  });
+  socket.on("close", () => {
+    if (closed) return;
+    closed = true;
+    handlers.onClose({ code: null, reason: "" });
+  });
+
+  function processFrames() {
+    while (true) {
+      const frame = readFrame(buffer);
+      if (!frame) return;
+      buffer = buffer.slice(frame.consumed);
+      if (frame.opcode === 0x1) {
+        handlers.onMessage(frame.payload.toString("utf8"));
+      } else if (frame.opcode === 0x8) {
+        const code = frame.payload.length >= 2 ? frame.payload.readUInt16BE(0) : null;
+        const reason = frame.payload.length > 2 ? frame.payload.slice(2).toString("utf8") : "";
+        closed = true;
+        handlers.onClose({ code, reason });
+        socket.end();
+        return;
+      } else if (frame.opcode === 0x9) {
+        writeFrame(socket, 0xA, frame.payload);
+      }
+    }
+  }
+
+  return {
+    send(text) {
+      if (closed || !opened) return;
+      writeFrame(socket, 0x1, Buffer.from(text, "utf8"));
+    },
+    close() {
+      if (closed) return;
+      closed = true;
+      if (!socket.destroyed) {
+        if (opened) writeFrame(socket, 0x8, Buffer.alloc(0));
+        setTimeout(() => socket.end(), 50).unref();
+      }
+    },
+  };
+}
+
+function readFrame(buffer) {
+  if (buffer.length < 2) return null;
+  const first = buffer[0];
+  const second = buffer[1];
+  const opcode = first & 0x0f;
+  const masked = Boolean(second & 0x80);
+  let length = second & 0x7f;
+  let offset = 2;
+  if (length === 126) {
+    if (buffer.length < offset + 2) return null;
+    length = buffer.readUInt16BE(offset);
+    offset += 2;
+  } else if (length === 127) {
+    if (buffer.length < offset + 8) return null;
+    const high = buffer.readUInt32BE(offset);
+    const low = buffer.readUInt32BE(offset + 4);
+    length = high * 2 ** 32 + low;
+    offset += 8;
+  }
+  let mask = null;
+  if (masked) {
+    if (buffer.length < offset + 4) return null;
+    mask = buffer.slice(offset, offset + 4);
+    offset += 4;
+  }
+  if (buffer.length < offset + length) return null;
+  let payload = buffer.slice(offset, offset + length);
+  if (mask) {
+    payload = Buffer.from(payload);
+    for (let index = 0; index < payload.length; index += 1) {
+      payload[index] ^= mask[index % 4];
+    }
+  }
+  return {
+    opcode,
+    payload,
+    consumed: offset + length,
+  };
+}
+
+function writeFrame(socket, opcode, payload) {
+  const body = Buffer.isBuffer(payload) ? payload : Buffer.from(payload || "");
+  const mask = crypto.randomBytes(4);
+  const headerLength = body.length < 126 ? 2 : body.length <= 0xffff ? 4 : 10;
+  const header = Buffer.alloc(headerLength);
+  header[0] = 0x80 | opcode;
+  if (body.length < 126) {
+    header[1] = 0x80 | body.length;
+  } else if (body.length <= 0xffff) {
+    header[1] = 0x80 | 126;
+    header.writeUInt16BE(body.length, 2);
+  } else {
+    header[1] = 0x80 | 127;
+    header.writeUInt32BE(Math.floor(body.length / 2 ** 32), 2);
+    header.writeUInt32BE(body.length >>> 0, 6);
+  }
+  const masked = Buffer.from(body);
+  for (let index = 0; index < masked.length; index += 1) {
+    masked[index] ^= mask[index % 4];
+  }
+  socket.write(Buffer.concat([header, mask, masked]));
+}
+
+function rounded(value) {
+  return Number(value.toFixed(3));
+}
+
+function percentile(values, percentileValue) {
+  if (values.length === 0) return 0;
+  const sorted = [...values].sort((a, b) => a - b);
+  const index = Math.min(sorted.length - 1, Math.ceil((percentileValue / 100) * sorted.length) - 1);
+  return rounded(sorted[index]);
+}
+
+function stats(values) {
+  if (values.length === 0) return { min: 0, p50: 0, p95: 0, p99: 0, max: 0 };
+  return {
+    min: rounded(Math.min(...values)),
+    p50: percentile(values, 50),
+    p95: percentile(values, 95),
+    p99: percentile(values, 99),
+    max: rounded(Math.max(...values)),
+  };
+}
+
+function buildMetrics({ samples, totalRequests, concurrency, timeoutMs, loadDurationMs, backendUrl, pipelineId, sessionType, fakeProviderState }) {
+  const okSamples = samples.filter((sample) => sample.ok);
+  const statusCounts = {};
+  for (const sample of samples) {
+    statusCounts[sample.status] = (statusCounts[sample.status] || 0) + 1;
+  }
+  const errorCount = samples.length - okSamples.length;
+  return {
+    probe: caseId,
+    backend_url: backendUrl,
+    pipeline_id: pipelineId,
+    session_type: sessionType,
+    total_requests: totalRequests,
+    completed_requests: samples.length,
+    concurrency,
+    timeout_ms: timeoutMs,
+    ok_count: okSamples.length,
+    error_count: errorCount,
+    timeout_count: samples.filter((sample) => sample.status === "timeout").length,
+    error_rate: samples.length === 0 ? 1 : rounded(errorCount / samples.length),
+    load_duration_ms: rounded(loadDurationMs),
+    throughput_rps: loadDurationMs <= 0 ? 0 : rounded(okSamples.length / (loadDurationMs / 1000)),
+    status_counts: statusCounts,
+    connected_ms: stats(samples.map((sample) => sample.connected_ms).filter(Number.isFinite)),
+    first_assistant_event_ms: stats(samples.map((sample) => sample.first_assistant_event_ms).filter(Number.isFinite)),
+    first_assistant_content_ms: stats(samples.map((sample) => sample.first_assistant_content_ms).filter(Number.isFinite)),
+    first_response_ms: stats(okSamples.map((sample) => sample.first_response_ms).filter(Number.isFinite)),
+    response_duration_ms: stats(okSamples.map((sample) => sample.response_duration_ms).filter(Number.isFinite)),
+    fake_provider: summarizeFakeProviderState(fakeProviderState),
+    provider_timing: buildProviderTimingMetrics(samples, fakeProviderState),
+    samples,
+  };
+}
+
+function buildThresholds(metrics) {
+  const thresholds = {
+    error_rate: { actual: metrics.error_rate, max: maxErrorRate, pass: metrics.error_rate <= maxErrorRate },
+    response_p95_ms: {
+      actual: metrics.response_duration_ms.p95,
+      max: responseP95BudgetMs,
+      pass: metrics.ok_count > 0 && metrics.response_duration_ms.p95 <= responseP95BudgetMs,
+    },
+  };
+  if (minErrorRate > 0) {
+    thresholds.error_rate_min = {
+      actual: metrics.error_rate,
+      min: minErrorRate,
+      pass: metrics.error_rate >= minErrorRate,
+    };
+  }
+  if (minErrorCount > 0) {
+    thresholds.error_count_min = {
+      actual: metrics.error_count,
+      min: minErrorCount,
+      pass: metrics.error_count >= minErrorCount,
+    };
+  }
+  if (minOkCount > 0) {
+    thresholds.ok_count_min = {
+      actual: metrics.ok_count,
+      min: minOkCount,
+      pass: metrics.ok_count >= minOkCount,
+    };
+  }
+  if (minProviderFaultCount > 0) {
+    const actual = metrics.fake_provider?.fault_count ?? 0;
+    thresholds.fake_provider_fault_count_min = {
+      actual,
+      min: minProviderFaultCount,
+      pass: actual >= minProviderFaultCount,
+    };
+  }
+  if (firstResponseP95BudgetMs > 0) {
+    thresholds.first_response_p95_ms = {
+      actual: metrics.first_response_ms.p95,
+      max: firstResponseP95BudgetMs,
+      pass: metrics.ok_count > 0 && metrics.first_response_ms.p95 <= firstResponseP95BudgetMs,
+    };
+  }
+  return thresholds;
+}
+
+function looksLikeEnvIssue(error) {
+  const message = String(error?.message || error || "");
+  return /fetch failed|ECONNREFUSED|ENOTFOUND|LANGBOT_.*not configured|Could not read recovery_key|Backend did not respond/i.test(message);
+}
+
+function safeReason(value) {
+  return redact(String(value || "")).slice(0, 1000);
+}

skills/skills/langbot-testing/probes/langbot-debug-chat-cross-pipeline-isolation.mjs

@@ -0,0 +1,861 @@
+#!/usr/bin/env node
+
+import crypto from "node:crypto";
+import net from "node:net";
+import tls from "node:tls";
+import { mkdir, writeFile } from "node:fs/promises";
+import { resolve } from "node:path";
+import { env, exit } from "node:process";
+import {
+  apiJson,
+  appendLine,
+  ensureEvidence,
+  evidencePaths,
+  loadEnvFiles,
+  localIsoWithOffset,
+  redact,
+  resetAndAuthLocalUser,
+  writeResult,
+} from "../../../scripts/e2e/lib/langbot-e2e.mjs";
+import {
+  buildProviderTimingMetrics,
+  summarizeFakeProviderState,
+} from "./lib/fake-provider-timing.mjs";
+
+const DEFAULT_LOCAL_PASSWORD = "LangBotE2ELocalPass!2026";
+
+await loadEnvFiles();
+const caseId = env.LBS_CASE_ID || "langbot-debug-chat-cross-pipeline-isolation";
+const paths = evidencePaths(caseId);
+await ensureEvidence(paths);
+
+const startedAt = new Date();
+const metricsPath = resolve(paths.evidenceDir, "metrics.json");
+const samplesPath = resolve(paths.evidenceDir, "samples.json");
+const fakeProviderStatePath = resolve(paths.evidenceDir, "fake-provider-state.json");
+const resetDiagnosticPath = resolve(paths.evidenceDir, "debug-chat-reset-diagnostic.json");
+const backendUrl = env.LANGBOT_BACKEND_URL || "";
+const fakeProviderUrl = env.LANGBOT_FAKE_PROVIDER_URL || "";
+const sessionType = env.LANGBOT_DEBUG_CHAT_LOAD_SESSION_TYPE || env.LANGBOT_E2E_DEBUG_CHAT_SESSION_TYPE || "person";
+const requestsPerPipeline = positiveInteger(env.LANGBOT_DEBUG_CHAT_LOAD_REQUESTS, 6);
+const concurrency = Math.min(requestsPerPipeline * 2, positiveInteger(env.LANGBOT_DEBUG_CHAT_LOAD_CONCURRENCY, 4));
+const timeoutMs = positiveInteger(env.LANGBOT_DEBUG_CHAT_LOAD_TIMEOUT_MS, 30_000);
+const stream = bool(env.LANGBOT_DEBUG_CHAT_LOAD_STREAM, true);
+const resetBeforeRun = bool(env.LANGBOT_DEBUG_CHAT_LOAD_RESET, true);
+const responseP95BudgetMs = positiveNumber(env.LANGBOT_DEBUG_CHAT_LOAD_RESPONSE_P95_MS, 5_000);
+const maxErrorRate = positiveNumber(env.LANGBOT_DEBUG_CHAT_LOAD_MAX_ERROR_RATE, 0);
+const promptTemplate = env.LANGBOT_DEBUG_CHAT_LOAD_PROMPT_TEMPLATE
+  || "请只回复 \"{expected}\"，不要解释，不要添加其他字符。";
+const failureSignals = textList(env.LANGBOT_E2E_FAILURE_SIGNALS || env.LANGBOT_DEBUG_CHAT_LOAD_FAILURE_SIGNALS || "");
+
+const pipelineTargets = [
+  {
+    label: "A",
+    expectedPrefix: "PIPEA",
+    otherPrefix: "PIPEB",
+    url: env.LANGBOT_FAKE_PROVIDER_PIPELINE_A_URL || "",
+    name: env.LANGBOT_FAKE_PROVIDER_PIPELINE_A_NAME || "",
+  },
+  {
+    label: "B",
+    expectedPrefix: "PIPEB",
+    otherPrefix: "PIPEA",
+    url: env.LANGBOT_FAKE_PROVIDER_PIPELINE_B_URL || "",
+    name: env.LANGBOT_FAKE_PROVIDER_PIPELINE_B_NAME || "",
+  },
+];
+
+const result = {
+  source: "automation",
+  case_id: caseId,
+  run_id: paths.runId,
+  status: "fail",
+  reason: "",
+  started_at: startedAt.toISOString(),
+  started_at_local: localIsoWithOffset(startedAt),
+  finished_at: "",
+  finished_at_local: "",
+  duration_ms: 0,
+  backend_url: backendUrl,
+  session_type: sessionType,
+  pipelines: [],
+  load_profile: {
+    requests_per_pipeline: requestsPerPipeline,
+    total_requests: requestsPerPipeline * 2,
+    concurrency,
+    timeout_ms: timeoutMs,
+    stream,
+    reset_before_run: resetBeforeRun,
+  },
+  evidence: {
+    network_log: paths.networkLog,
+    metrics_json: metricsPath,
+    samples_json: samplesPath,
+    fake_provider_state_json: fakeProviderStatePath,
+    debug_chat_reset_diagnostic_json: resetDiagnosticPath,
+    automation_result_json: paths.automationResultJson,
+    result_json: paths.resultJson,
+  },
+  evidence_collected: ["metrics", "network", "api_diagnostic", "filesystem"],
+};
+
+try {
+  if (!backendUrl) {
+    result.status = "env_issue";
+    throw new Error("LANGBOT_BACKEND_URL is not configured.");
+  }
+  if (!["person", "group"].includes(sessionType)) {
+    throw new Error(`LANGBOT_DEBUG_CHAT_LOAD_SESSION_TYPE must be person or group, got ${sessionType}.`);
+  }
+  for (const target of pipelineTargets) {
+    if (!target.url && !target.name) {
+      result.status = "env_issue";
+      throw new Error(`Set LANGBOT_FAKE_PROVIDER_PIPELINE_${target.label}_URL or LANGBOT_FAKE_PROVIDER_PIPELINE_${target.label}_NAME.`);
+    }
+  }
+
+  const backendReady = await backendReachable(backendUrl);
+  if (!backendReady) {
+    result.status = "env_issue";
+    throw new Error(`Backend did not respond at ${backendUrl}.`);
+  }
+
+  const user = env.LANGBOT_E2E_LOGIN_USER || "";
+  const password = env.LANGBOT_E2E_LOGIN_PASSWORD || DEFAULT_LOCAL_PASSWORD;
+  if (!user) {
+    result.status = "env_issue";
+    throw new Error("LANGBOT_E2E_LOGIN_USER is required so this probe can resolve/reset Debug Chat sessions.");
+  }
+  const auth = await resetAndAuthLocalUser({ backendUrl, user, password });
+  const pipelines = [];
+  for (const target of pipelineTargets) {
+    const pipeline = await resolvePipeline({
+      backendUrl,
+      token: auth.token,
+      pipelineUrl: target.url,
+      pipelineName: target.name,
+    });
+    pipelines.push({
+      ...target,
+      id: pipeline.id,
+      name: pipeline.name || target.name,
+      wsUrl: websocketUrl(backendUrl, pipeline.id, sessionType),
+    });
+  }
+  result.pipelines = pipelines.map((pipeline) => ({
+    label: pipeline.label,
+    id: pipeline.id,
+    name: pipeline.name,
+    url: pipeline.url,
+  }));
+
+  if (resetBeforeRun) {
+    const resetDiagnostics = [];
+    for (const pipeline of pipelines) {
+      const reset = await apiJson(backendUrl, `/api/v1/pipelines/${encodeURIComponent(pipeline.id)}/ws/reset/${encodeURIComponent(sessionType)}`, {
+        method: "POST",
+        token: auth.token,
+      });
+      resetDiagnostics.push({
+        pipeline_label: pipeline.label,
+        pipeline_id: pipeline.id,
+        status: isApiFailure(reset) ? "fail" : "ready",
+        http_status: reset.status,
+        code: reset.json.code ?? null,
+        reason: isApiFailure(reset) ? reset.json.msg || "Debug Chat reset failed." : "Debug Chat session reset.",
+      });
+    }
+    await writeFile(resetDiagnosticPath, `${JSON.stringify(resetDiagnostics, null, 2)}\n`, "utf8");
+    const failedReset = resetDiagnostics.find((item) => item.status === "fail");
+    if (failedReset) throw new Error(failedReset.reason);
+  }
+  await resetFakeProvider(fakeProviderUrl);
+
+  const jobs = [];
+  for (let index = 0; index < requestsPerPipeline; index += 1) {
+    for (const pipeline of pipelines) {
+      jobs.push({ ...pipeline, index });
+    }
+  }
+
+  const loadStartedAt = performance.now();
+  const samples = await runLoad({
+    jobs,
+    concurrency,
+    timeoutMs,
+    promptTemplate,
+    stream,
+    failureSignals,
+  });
+  const loadDurationMs = performance.now() - loadStartedAt;
+  const fakeProviderState = await readFakeProviderState(fakeProviderUrl);
+  if (fakeProviderState) {
+    await writeFile(fakeProviderStatePath, `${JSON.stringify(fakeProviderState, null, 2)}\n`, "utf8");
+  }
+  const metrics = buildMetrics({
+    samples,
+    requestsPerPipeline,
+    concurrency,
+    timeoutMs,
+    loadDurationMs,
+    backendUrl,
+    sessionType,
+    fakeProviderState,
+  });
+  const thresholds = buildThresholds(metrics);
+  const passed = Object.values(thresholds).every((item) => item.pass);
+  result.status = passed ? "pass" : "fail";
+  result.reason = passed
+    ? "Debug Chat cross-pipeline isolation probe passed all thresholds."
+    : "Debug Chat cross-pipeline isolation probe found leaks, errors, or latency threshold breaches.";
+  result.metrics_summary = {
+    requests_per_pipeline: metrics.requests_per_pipeline,
+    total_requests: metrics.total_requests,
+    concurrency: metrics.concurrency,
+    ok_count: metrics.ok_count,
+    error_count: metrics.error_count,
+    cross_pipeline_leak_count: metrics.cross_pipeline_leak_count,
+    timeout_count: metrics.timeout_count,
+    error_rate: metrics.error_rate,
+    response_p95_ms: metrics.response_duration_ms.p95,
+    first_response_p95_ms: metrics.first_response_ms.p95,
+    throughput_rps: metrics.throughput_rps,
+    status_counts: metrics.status_counts,
+    by_pipeline: metrics.by_pipeline,
+    fake_provider_request_count: metrics.fake_provider?.request_count ?? null,
+    fake_provider_duration_p95_ms: metrics.provider_timing?.provider_duration_ms.p95 ?? null,
+    langbot_overhead_estimate_p95_ms: metrics.provider_timing?.langbot_overhead_estimate_ms.p95 ?? null,
+    send_to_provider_start_p95_ms: metrics.provider_timing?.send_to_provider_start_ms.p95 ?? null,
+    provider_finish_to_ws_final_p95_ms: metrics.provider_timing?.provider_finish_to_ws_final_ms.p95 ?? null,
+  };
+  result.thresholds_summary = thresholds;
+  result.artifacts = {
+    metrics_json: metricsPath,
+    samples_json: samplesPath,
+    fake_provider_state_json: fakeProviderState ? fakeProviderStatePath : "",
+    network_log: paths.networkLog,
+    automation_result_json: paths.automationResultJson,
+    result_json: paths.resultJson,
+  };
+
+  await writeFile(metricsPath, `${JSON.stringify({ ...metrics, thresholds }, null, 2)}\n`, "utf8");
+  await writeFile(samplesPath, `${JSON.stringify(samples, null, 2)}\n`, "utf8");
+} catch (error) {
+  if (!["env_issue", "blocked"].includes(result.status)) {
+    result.status = looksLikeEnvIssue(error) ? "env_issue" : "fail";
+  }
+  result.reason = result.reason || safeReason(error.message);
+} finally {
+  const finishedAt = new Date();
+  result.finished_at = finishedAt.toISOString();
+  result.finished_at_local = localIsoWithOffset(finishedAt);
+  result.duration_ms = finishedAt.getTime() - startedAt.getTime();
+  await mkdir(paths.evidenceDir, { recursive: true });
+  await writeResult(paths, result);
+  console.log(JSON.stringify(result, null, 2));
+}
+
+exit(result.status === "pass" ? 0 : result.status === "env_issue" || result.status === "blocked" ? 2 : 1);
+
+async function backendReachable(baseUrl) {
+  try {
+    const response = await fetch(`${baseUrl.replace(/\/$/, "")}/healthz`, {
+      signal: AbortSignal.timeout(3000),
+    });
+    return response.status < 500;
+  } catch {
+    return false;
+  }
+}
+
+async function resetFakeProvider(rootUrl) {
+  if (!rootUrl) return;
+  try {
+    await fetch(`${normalizeProviderRootUrl(rootUrl)}/__qa/reset`, {
+      method: "POST",
+      signal: AbortSignal.timeout(3000),
+    });
+  } catch {
+    // Missing fake-provider diagnostics should not hide the isolation result.
+  }
+}
+
+async function readFakeProviderState(rootUrl) {
+  if (!rootUrl) return null;
+  try {
+    const response = await fetch(`${normalizeProviderRootUrl(rootUrl)}/__qa/config`, {
+      signal: AbortSignal.timeout(3000),
+    });
+    const json = await response.json().catch(() => ({}));
+    return {
+      status: response.ok && json.ok === true ? "loaded" : "unavailable",
+      url: normalizeProviderRootUrl(rootUrl),
+      http_status: response.status,
+      model: json.model || "",
+      config: json.config || {},
+      request_count: Number.isFinite(json.request_count) ? json.request_count : null,
+      recent_requests: Array.isArray(json.recent_requests) ? json.recent_requests : [],
+    };
+  } catch (error) {
+    return {
+      status: "unavailable",
+      url: normalizeProviderRootUrl(rootUrl),
+      reason: safeReason(error.message),
+      request_count: null,
+      recent_requests: [],
+    };
+  }
+}
+
+function normalizeProviderRootUrl(value) {
+  const trimmed = String(value || "").trim().replace(/\/$/, "");
+  return trimmed.endsWith("/v1") ? trimmed.slice(0, -3) : trimmed;
+}
+
+function pipelineIdFromUrl(url) {
+  if (!url) return "";
+  try {
+    const parsed = new URL(url);
+    return parsed.searchParams.get("id") || "";
+  } catch {
+    return "";
+  }
+}
+
+async function resolvePipeline({ backendUrl, token, pipelineUrl, pipelineName }) {
+  const idFromUrl = pipelineIdFromUrl(pipelineUrl);
+  if (idFromUrl) {
+    const response = await apiJson(backendUrl, `/api/v1/pipelines/${encodeURIComponent(idFromUrl)}`, { token });
+    const pipeline = response.json.data?.pipeline;
+    if (isApiFailure(response) || !pipeline?.uuid) {
+      throw new Error(response.json.msg || `Could not load pipeline ${idFromUrl}.`);
+    }
+    return { id: pipeline.uuid, name: pipeline.name || "" };
+  }
+  if (!pipelineName) {
+    throw new Error("Set pipeline URL or name before running this probe.");
+  }
+  const response = await apiJson(backendUrl, "/api/v1/pipelines", { token });
+  if (isApiFailure(response)) {
+    throw new Error(response.json.msg || "Failed to list pipelines.");
+  }
+  const pipeline = (response.json.data?.pipelines || []).find((item) => item.name === pipelineName);
+  if (!pipeline?.uuid) {
+    throw new Error(`Could not find pipeline named ${pipelineName}.`);
+  }
+  return { id: pipeline.uuid, name: pipeline.name || pipelineName };
+}
+
+function isApiFailure(response) {
+  return response.status >= 400 || (response.json.code !== undefined && response.json.code !== 0);
+}
+
+function websocketUrl(baseUrl, pipelineId, sessionTypeValue) {
+  const parsed = new URL(baseUrl);
+  parsed.protocol = parsed.protocol === "https:" ? "wss:" : "ws:";
+  parsed.pathname = `/api/v1/pipelines/${encodeURIComponent(pipelineId)}/ws/connect`;
+  parsed.search = `?session_type=${encodeURIComponent(sessionTypeValue)}`;
+  return parsed.toString();
+}
+
+async function runLoad(options) {
+  const samples = [];
+  const queue = [...options.jobs];
+  const workers = Array.from({ length: options.concurrency }, async () => {
+    while (queue.length > 0) {
+      const job = queue.shift();
+      if (!job) continue;
+      const sample = await runSingleRequest({ ...options, job });
+      samples.push(sample);
+    }
+  });
+  await Promise.all(workers);
+  return samples.sort((left, right) => (
+    left.pipeline_label.localeCompare(right.pipeline_label) || left.index - right.index
+  ));
+}
+
+function expectedForIndex(prefix, index) {
+  return `${prefix}-${String(index + 1).padStart(4, "0")}`;
+}
+
+function promptForIndex(template, expected) {
+  return template.replaceAll("{expected}", expected);
+}
+
+function runSingleRequest({
+  job,
+  timeoutMs,
+  promptTemplate,
+  stream,
+  failureSignals,
+}) {
+  return new Promise((resolvePromise) => {
+    const expected = expectedForIndex(job.expectedPrefix, job.index);
+    const prompt = promptForIndex(promptTemplate, expected);
+    const sample = {
+      index: job.index,
+      pipeline_label: job.label,
+      pipeline_id: job.id,
+      pipeline_name: job.name,
+      status: "running",
+      ok: false,
+      expected_text: expected,
+      expected_prefix: job.expectedPrefix,
+      other_prefix: job.otherPrefix,
+      prompt,
+      response_text: "",
+      started_at: new Date().toISOString(),
+      started_epoch_ms: Date.now(),
+      connected_at: null,
+      connected_epoch_ms: null,
+      sent_at: null,
+      sent_epoch_ms: null,
+      first_assistant_event_at: null,
+      first_assistant_event_epoch_ms: null,
+      first_assistant_event_ms: null,
+      first_assistant_content_at: null,
+      first_assistant_content_epoch_ms: null,
+      first_assistant_content_ms: null,
+      first_response_at: null,
+      first_response_epoch_ms: null,
+      connected_ms: null,
+      first_response_ms: null,
+      response_duration_ms: null,
+      finished_at: null,
+      finished_epoch_ms: null,
+      event_count: 0,
+      same_pipeline_foreign_response_count: 0,
+      cross_pipeline_leak_count: 0,
+      last_foreign_response_text: "",
+      error: "",
+      close_code: null,
+      close_reason: "",
+    };
+    let closed = false;
+    let connectedAt = 0;
+    let sentAt = 0;
+    const startedPerf = performance.now();
+    let client = null;
+    const timer = setTimeout(() => {
+      finish("timeout", `Timed out after ${timeoutMs} ms.`);
+    }, timeoutMs);
+
+    client = openRawWebSocket(job.wsUrl, {
+      onOpen() {
+        connectedAt = performance.now();
+        const now = Date.now();
+        sample.connected_at = new Date(now).toISOString();
+        sample.connected_epoch_ms = now;
+        sample.connected_ms = rounded(connectedAt - startedPerf);
+      },
+      onMessage(text) {
+        sample.event_count += 1;
+        let data;
+        try {
+          data = JSON.parse(String(text || ""));
+        } catch (error) {
+          finish("error", `Invalid WebSocket JSON: ${error.message}`);
+          return;
+        }
+        appendLine(paths.networkLog, JSON.stringify({
+          pipeline_label: job.label,
+          request_index: job.index,
+          type: data.type,
+          session_type: data.session_type || "",
+          role: data.data?.role || "",
+          is_final: data.data?.is_final ?? null,
+          content_preview: redact(String(data.data?.content || data.message || "").slice(0, 200)),
+        })).catch(() => {});
+
+        if (data.type === "connected") {
+          sentAt = performance.now();
+          const now = Date.now();
+          sample.sent_at = new Date(now).toISOString();
+          sample.sent_epoch_ms = now;
+          client.send(JSON.stringify({
+            type: "message",
+            message: [{ type: "Plain", text: prompt }],
+            stream,
+          }));
+          return;
+        }
+        if (data.type === "error") {
+          finish("error", data.message || "WebSocket error message.");
+          return;
+        }
+        if (data.type !== "response" || data.data?.role !== "assistant") return;
+
+        const content = String(data.data.content || "");
+        markFirstAssistantEvent(sample, sentAt);
+        if (content) sample.response_text = content;
+        if (content) markFirstAssistantContent(sample, sentAt);
+        if (containsPipelineToken(content, job.otherPrefix)) {
+          sample.cross_pipeline_leak_count += 1;
+          finish("cross_pipeline_leak", `Pipeline ${job.label} received response from ${job.otherPrefix}: ${content}`);
+          return;
+        }
+        if (content.includes(expected) && sample.first_response_ms === null && sentAt > 0) {
+          const now = Date.now();
+          sample.first_response_at = new Date(now).toISOString();
+          sample.first_response_epoch_ms = now;
+          sample.first_response_ms = rounded(performance.now() - sentAt);
+        }
+        if (data.data.is_final === true) {
+          const ok = sample.response_text.includes(expected);
+          if (ok) {
+            if (sample.first_response_ms === null && sentAt > 0) {
+              const now = Date.now();
+              sample.first_response_at = new Date(now).toISOString();
+              sample.first_response_epoch_ms = now;
+              sample.first_response_ms = rounded(performance.now() - sentAt);
+            }
+            finish("pass", "");
+          } else if (matchesFailureSignal(sample.response_text, failureSignals)) {
+            finish("app_error", `Assistant final response matched a failure signal: ${sample.response_text}`);
+          } else if (containsPipelineToken(sample.response_text, job.expectedPrefix)) {
+            sample.same_pipeline_foreign_response_count += 1;
+            sample.last_foreign_response_text = sample.response_text;
+          } else {
+            finish("mismatch", `Final assistant response did not include ${expected}: ${sample.response_text}`);
+          }
+        }
+      },
+      onError(error) {
+        finish("connection_error", `WebSocket connection error: ${error.message}`);
+      },
+      onClose(event) {
+        sample.close_code = event.code;
+        sample.close_reason = event.reason || "";
+        if (!closed) finish("closed", `WebSocket closed before final assistant response: ${event.code}`);
+      },
+    });
+
+    function finish(status, reason) {
+      if (closed) return;
+      closed = true;
+      clearTimeout(timer);
+      sample.status = status;
+      sample.ok = status === "pass";
+      sample.error = status === "timeout" && sample.same_pipeline_foreign_response_count > 0
+        ? `${reason || ""} Saw ${sample.same_pipeline_foreign_response_count} same-pipeline foreign assistant response(s); last=${sample.last_foreign_response_text}`
+        : reason || "";
+      if (sentAt > 0) sample.response_duration_ms = rounded(performance.now() - sentAt);
+      else sample.response_duration_ms = rounded(performance.now() - startedPerf);
+      const now = Date.now();
+      sample.finished_at = new Date(now).toISOString();
+      sample.finished_epoch_ms = now;
+      try {
+        client?.close();
+      } catch {
+        // Closing a failed socket should not hide the sample result.
+      }
+      resolvePromise(sample);
+    }
+  });
+}
+
+function markFirstAssistantEvent(sample, sentAt) {
+  if (sample.first_assistant_event_ms !== null || sentAt <= 0) return;
+  const now = Date.now();
+  sample.first_assistant_event_at = new Date(now).toISOString();
+  sample.first_assistant_event_epoch_ms = now;
+  sample.first_assistant_event_ms = rounded(performance.now() - sentAt);
+}
+
+function markFirstAssistantContent(sample, sentAt) {
+  if (sample.first_assistant_content_ms !== null || sentAt <= 0) return;
+  const now = Date.now();
+  sample.first_assistant_content_at = new Date(now).toISOString();
+  sample.first_assistant_content_epoch_ms = now;
+  sample.first_assistant_content_ms = rounded(performance.now() - sentAt);
+}
+
+function containsPipelineToken(text, prefix) {
+  const escaped = String(prefix).replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
+  return new RegExp(`${escaped}-\\d{4}`).test(String(text || ""));
+}
+
+function matchesFailureSignal(text, signals) {
+  const lower = String(text || "").toLowerCase();
+  return signals.some((signal) => lower.includes(signal.toLowerCase()));
+}
+
+function openRawWebSocket(wsUrl, handlers) {
+  const parsed = new URL(wsUrl);
+  const secure = parsed.protocol === "wss:";
+  const port = Number(parsed.port || (secure ? 443 : 80));
+  const host = parsed.hostname;
+  const path = `${parsed.pathname}${parsed.search}`;
+  const key = crypto.randomBytes(16).toString("base64");
+  const socket = secure
+    ? tls.connect({ host, port, servername: host })
+    : net.connect({ host, port });
+  let opened = false;
+  let closed = false;
+  let buffer = Buffer.alloc(0);
+
+  socket.setNoDelay(true);
+  socket.on("connect", () => {
+    const originProtocol = secure ? "https" : "http";
+    const request = [
+      `GET ${path} HTTP/1.1`,
+      `Host: ${parsed.host}`,
+      "Upgrade: websocket",
+      "Connection: Upgrade",
+      `Sec-WebSocket-Key: ${key}`,
+      "Sec-WebSocket-Version: 13",
+      `Origin: ${originProtocol}://${parsed.host}`,
+      "",
+      "",
+    ].join("\r\n");
+    socket.write(request);
+  });
+  socket.on("data", (chunk) => {
+    buffer = Buffer.concat([buffer, chunk]);
+    if (!opened) {
+      const headerEnd = buffer.indexOf("\r\n\r\n");
+      if (headerEnd === -1) return;
+      const headerText = buffer.slice(0, headerEnd).toString("utf8");
+      buffer = buffer.slice(headerEnd + 4);
+      if (!/^HTTP\/1\.1 101\b/i.test(headerText)) {
+        handlers.onError(new Error(`Handshake failed: ${headerText.split("\r\n")[0] || "missing status"}`));
+        socket.destroy();
+        return;
+      }
+      opened = true;
+      handlers.onOpen();
+    }
+    processFrames();
+  });
+  socket.on("error", (error) => {
+    if (!closed) handlers.onError(error);
+  });
+  socket.on("close", () => {
+    if (closed) return;
+    closed = true;
+    handlers.onClose({ code: null, reason: "" });
+  });
+
+  function processFrames() {
+    while (true) {
+      const frame = readFrame(buffer);
+      if (!frame) return;
+      buffer = buffer.slice(frame.consumed);
+      if (frame.opcode === 0x1) {
+        handlers.onMessage(frame.payload.toString("utf8"));
+      } else if (frame.opcode === 0x8) {
+        const code = frame.payload.length >= 2 ? frame.payload.readUInt16BE(0) : null;
+        const reason = frame.payload.length > 2 ? frame.payload.slice(2).toString("utf8") : "";
+        closed = true;
+        handlers.onClose({ code, reason });
+        socket.end();
+        return;
+      } else if (frame.opcode === 0x9) {
+        writeFrame(socket, 0xA, frame.payload);
+      }
+    }
+  }
+
+  return {
+    send(text) {
+      if (closed || !opened) return;
+      writeFrame(socket, 0x1, Buffer.from(text, "utf8"));
+    },
+    close() {
+      if (closed) return;
+      closed = true;
+      if (!socket.destroyed) {
+        if (opened) writeFrame(socket, 0x8, Buffer.alloc(0));
+        setTimeout(() => socket.end(), 50).unref();
+      }
+    },
+  };
+}
+
+function readFrame(buffer) {
+  if (buffer.length < 2) return null;
+  const first = buffer[0];
+  const second = buffer[1];
+  const opcode = first & 0x0f;
+  const masked = Boolean(second & 0x80);
+  let length = second & 0x7f;
+  let offset = 2;
+  if (length === 126) {
+    if (buffer.length < offset + 2) return null;
+    length = buffer.readUInt16BE(offset);
+    offset += 2;
+  } else if (length === 127) {
+    if (buffer.length < offset + 8) return null;
+    const high = buffer.readUInt32BE(offset);
+    const low = buffer.readUInt32BE(offset + 4);
+    length = high * 2 ** 32 + low;
+    offset += 8;
+  }
+  let mask = null;
+  if (masked) {
+    if (buffer.length < offset + 4) return null;
+    mask = buffer.slice(offset, offset + 4);
+    offset += 4;
+  }
+  if (buffer.length < offset + length) return null;
+  let payload = buffer.slice(offset, offset + length);
+  if (mask) {
+    payload = Buffer.from(payload);
+    for (let index = 0; index < payload.length; index += 1) {
+      payload[index] ^= mask[index % 4];
+    }
+  }
+  return {
+    opcode,
+    payload,
+    consumed: offset + length,
+  };
+}
+
+function writeFrame(socket, opcode, payload) {
+  const body = Buffer.isBuffer(payload) ? payload : Buffer.from(payload || "");
+  const mask = crypto.randomBytes(4);
+  const headerLength = body.length < 126 ? 2 : body.length <= 0xffff ? 4 : 10;
+  const header = Buffer.alloc(headerLength);
+  header[0] = 0x80 | opcode;
+  if (body.length < 126) {
+    header[1] = 0x80 | body.length;
+  } else if (body.length <= 0xffff) {
+    header[1] = 0x80 | 126;
+    header.writeUInt16BE(body.length, 2);
+  } else {
+    header[1] = 0x80 | 127;
+    header.writeUInt32BE(Math.floor(body.length / 2 ** 32), 2);
+    header.writeUInt32BE(body.length >>> 0, 6);
+  }
+  const masked = Buffer.from(body);
+  for (let index = 0; index < masked.length; index += 1) {
+    masked[index] ^= mask[index % 4];
+  }
+  socket.write(Buffer.concat([header, mask, masked]));
+}
+
+function buildMetrics({ samples, requestsPerPipeline, concurrency, timeoutMs, loadDurationMs, backendUrl, sessionType, fakeProviderState }) {
+  const okSamples = samples.filter((sample) => sample.ok);
+  const statusCounts = {};
+  const byPipeline = {};
+  for (const sample of samples) {
+    statusCounts[sample.status] = (statusCounts[sample.status] || 0) + 1;
+    if (!byPipeline[sample.pipeline_label]) {
+      byPipeline[sample.pipeline_label] = {
+        ok_count: 0,
+        error_count: 0,
+        cross_pipeline_leak_count: 0,
+        timeout_count: 0,
+      };
+    }
+    if (sample.ok) byPipeline[sample.pipeline_label].ok_count += 1;
+    else byPipeline[sample.pipeline_label].error_count += 1;
+    byPipeline[sample.pipeline_label].cross_pipeline_leak_count += sample.cross_pipeline_leak_count || 0;
+    if (sample.status === "timeout") byPipeline[sample.pipeline_label].timeout_count += 1;
+  }
+  const errorCount = samples.length - okSamples.length;
+  return {
+    probe: caseId,
+    backend_url: backendUrl,
+    session_type: sessionType,
+    requests_per_pipeline: requestsPerPipeline,
+    total_requests: requestsPerPipeline * 2,
+    completed_requests: samples.length,
+    concurrency,
+    timeout_ms: timeoutMs,
+    ok_count: okSamples.length,
+    error_count: errorCount,
+    timeout_count: samples.filter((sample) => sample.status === "timeout").length,
+    cross_pipeline_leak_count: samples.reduce((count, sample) => count + (sample.cross_pipeline_leak_count || 0), 0),
+    error_rate: samples.length === 0 ? 1 : rounded(errorCount / samples.length),
+    load_duration_ms: rounded(loadDurationMs),
+    throughput_rps: loadDurationMs <= 0 ? 0 : rounded(okSamples.length / (loadDurationMs / 1000)),
+    status_counts: statusCounts,
+    by_pipeline: byPipeline,
+    connected_ms: stats(samples.map((sample) => sample.connected_ms).filter(Number.isFinite)),
+    first_assistant_event_ms: stats(samples.map((sample) => sample.first_assistant_event_ms).filter(Number.isFinite)),
+    first_assistant_content_ms: stats(samples.map((sample) => sample.first_assistant_content_ms).filter(Number.isFinite)),
+    first_response_ms: stats(okSamples.map((sample) => sample.first_response_ms).filter(Number.isFinite)),
+    response_duration_ms: stats(okSamples.map((sample) => sample.response_duration_ms).filter(Number.isFinite)),
+    fake_provider: summarizeFakeProviderState(fakeProviderState),
+    provider_timing: buildProviderTimingMetrics(samples, fakeProviderState),
+    samples,
+  };
+}
+
+function buildThresholds(metrics) {
+  return {
+    cross_pipeline_leak_count: {
+      actual: metrics.cross_pipeline_leak_count,
+      max: 0,
+      pass: metrics.cross_pipeline_leak_count === 0,
+    },
+    error_rate: {
+      actual: metrics.error_rate,
+      max: maxErrorRate,
+      pass: metrics.error_rate <= maxErrorRate,
+    },
+    response_p95_ms: {
+      actual: metrics.response_duration_ms.p95,
+      max: responseP95BudgetMs,
+      pass: metrics.ok_count > 0 && metrics.response_duration_ms.p95 <= responseP95BudgetMs,
+    },
+  };
+}
+
+function positiveInteger(value, fallback) {
+  const parsed = Number.parseInt(String(value || ""), 10);
+  return Number.isInteger(parsed) && parsed > 0 ? parsed : fallback;
+}
+
+function positiveNumber(value, fallback) {
+  const parsed = Number(value || "");
+  return Number.isFinite(parsed) && parsed >= 0 ? parsed : fallback;
+}
+
+function bool(value, fallback) {
+  if (value === undefined || value === "") return fallback;
+  if (/^(1|true|yes|on)$/i.test(String(value))) return true;
+  if (/^(0|false|no|off)$/i.test(String(value))) return false;
+  return fallback;
+}
+
+function textList(value) {
+  return String(value || "")
+    .split(/\r?\n|,/)
+    .map((item) => item.trim())
+    .filter(Boolean);
+}
+
+function rounded(value) {
+  return Number(value.toFixed(3));
+}
+
+function percentile(values, percentileValue) {
+  if (values.length === 0) return 0;
+  const sorted = [...values].sort((a, b) => a - b);
+  const index = Math.min(sorted.length - 1, Math.ceil((percentileValue / 100) * sorted.length) - 1);
+  return rounded(sorted[index]);
+}
+
+function stats(values) {
+  if (values.length === 0) return { min: 0, p50: 0, p95: 0, p99: 0, max: 0 };
+  return {
+    min: rounded(Math.min(...values)),
+    p50: percentile(values, 50),
+    p95: percentile(values, 95),
+    p99: percentile(values, 99),
+    max: rounded(Math.max(...values)),
+  };
+}
+
+function looksLikeEnvIssue(error) {
+  const message = String(error?.message || error || "");
+  return /fetch failed|ECONNREFUSED|ENOTFOUND|LANGBOT_.*not configured|Could not read recovery_key|Backend did not respond/i.test(message);
+}
+
+function safeReason(value) {
+  return redact(String(value || "")).slice(0, 1000);
+}

skills/skills/langbot-testing/probes/langbot-fault-taxonomy-contract.mjs

@@ -0,0 +1,159 @@
+#!/usr/bin/env node
+
+import { mkdir, writeFile } from "node:fs/promises";
+import { join, resolve } from "node:path";
+import { env, exit } from "node:process";
+
+function pad(value, size = 2) {
+  return String(value).padStart(size, "0");
+}
+
+function localIsoWithOffset(date = new Date()) {
+  const offsetMinutes = -date.getTimezoneOffset();
+  const sign = offsetMinutes >= 0 ? "+" : "-";
+  const absolute = Math.abs(offsetMinutes);
+  return [
+    `${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())}`,
+    `T${pad(date.getHours())}:${pad(date.getMinutes())}:${pad(date.getSeconds())}.${pad(date.getMilliseconds(), 3)}`,
+    `${sign}${pad(Math.floor(absolute / 60))}:${pad(absolute % 60)}`,
+  ].join("");
+}
+
+function timestampSlug(date = new Date()) {
+  return date.toISOString().replace(/\.\d{3}Z$/, "Z").replace(/[^0-9A-Za-z]+/g, "-").replace(/^-|-$/g, "");
+}
+
+const scenarios = [
+  {
+    id: "provider-timeout",
+    target: "provider",
+    injected_fault: "fake provider request exceeds the configured timeout",
+    expected_status: "env_issue",
+    recovery_check: "provider route is reachable or the case remains outside product pass/fail",
+    cleanup: "stop fake provider or reset proxy route",
+  },
+  {
+    id: "plugin-runtime-disconnect",
+    target: "plugin-runtime",
+    injected_fault: "runtime control channel disconnects during an action",
+    expected_status: "fail",
+    recovery_check: "runtime reconnects and a deterministic plugin action succeeds",
+    cleanup: "restart the local plugin runtime process",
+  },
+  {
+    id: "mcp-stdio-server-exit",
+    target: "mcp",
+    injected_fault: "stdio server exits mid-call",
+    expected_status: "fail",
+    recovery_check: "server can be registered again and exposes the expected tool",
+    cleanup: "remove temporary MCP server registration",
+  },
+  {
+    id: "operator-missing-login",
+    target: "webui",
+    injected_fault: "browser profile is not authenticated",
+    expected_status: "blocked",
+    recovery_check: "authenticated profile can open the same WebUI origin",
+    cleanup: "no product cleanup; refresh local login state",
+  },
+  {
+    id: "transient-marketplace-timeout",
+    target: "marketplace",
+    injected_fault: "marketplace request times out once and then succeeds",
+    expected_status: "flaky",
+    recovery_check: "rerun passes with the same product revision and no code change",
+    cleanup: "clear retry-only evidence and keep the run classified as flaky",
+  },
+];
+
+function validateScenario(scenario) {
+  const missing = ["id", "target", "injected_fault", "expected_status", "recovery_check", "cleanup"]
+    .filter((key) => !scenario[key]);
+  const allowedStatuses = new Set(["pass", "fail", "blocked", "env_issue", "flaky"]);
+  return {
+    id: scenario.id,
+    pass: missing.length === 0 && allowedStatuses.has(scenario.expected_status),
+    missing,
+    expected_status: scenario.expected_status,
+  };
+}
+
+async function main() {
+  const root = resolve(env.LBS_ROOT || process.cwd());
+  const caseId = "langbot-fault-taxonomy-contract";
+  const runId = env.LBS_RUN_ID || `${timestampSlug()}-${caseId}`;
+  const evidenceDir = resolve(env.LBS_EVIDENCE_DIR || join(root, "reports", "evidence", runId));
+  await mkdir(evidenceDir, { recursive: true });
+
+  const startedAt = new Date();
+  const validations = scenarios.map(validateScenario);
+  const statusCounts = {};
+  for (const scenario of scenarios) {
+    statusCounts[scenario.expected_status] = (statusCounts[scenario.expected_status] || 0) + 1;
+  }
+  const metrics = {
+    probe: caseId,
+    scenario_count: scenarios.length,
+    status_counts: statusCounts,
+    scenarios,
+    validations,
+  };
+  const thresholds = {
+    scenario_count: { actual: scenarios.length, min: 5, pass: scenarios.length >= 5 },
+    invalid_scenario_count: {
+      actual: validations.filter((item) => !item.pass).length,
+      max: 0,
+      pass: validations.every((item) => item.pass),
+    },
+    cleanup_declared_count: {
+      actual: scenarios.filter((item) => item.cleanup).length,
+      min: scenarios.length,
+      pass: scenarios.every((item) => item.cleanup),
+    },
+  };
+  const status = Object.values(thresholds).every((item) => item.pass) ? "pass" : "fail";
+  const metricsPath = join(evidenceDir, "metrics.json");
+  const faultModelPath = join(evidenceDir, "fault-model.json");
+  const automationResultPath = join(evidenceDir, "automation-result.json");
+  const resultPath = join(evidenceDir, "result.json");
+
+  await writeFile(metricsPath, `${JSON.stringify(metrics, null, 2)}\n`, "utf8");
+  await writeFile(faultModelPath, `${JSON.stringify({ scenarios }, null, 2)}\n`, "utf8");
+
+  const finishedAt = new Date();
+  const result = {
+    source: "automation",
+    case_id: caseId,
+    run_id: runId,
+    status,
+    reason: status === "pass"
+      ? "Fault taxonomy contract declares status, recovery, and cleanup for every scenario."
+      : "Fault taxonomy contract is missing required scenario fields.",
+    started_at: startedAt.toISOString(),
+    started_at_local: localIsoWithOffset(startedAt),
+    finished_at: finishedAt.toISOString(),
+    finished_at_local: localIsoWithOffset(finishedAt),
+    duration_ms: finishedAt.getTime() - startedAt.getTime(),
+    metrics_summary: {
+      scenario_count: metrics.scenario_count,
+      status_counts: metrics.status_counts,
+      invalid_scenario_count: thresholds.invalid_scenario_count.actual,
+    },
+    thresholds_summary: thresholds,
+    artifacts: {
+      metrics_json: metricsPath,
+      fault_model_json: faultModelPath,
+      automation_result_json: automationResultPath,
+      result_json: resultPath,
+    },
+    evidence_collected: ["metrics", "filesystem"],
+  };
+
+  const resultText = `${JSON.stringify(result, null, 2)}\n`;
+  await writeFile(automationResultPath, resultText, "utf8");
+  await writeFile(resultPath, resultText, "utf8");
+  console.log(JSON.stringify(result, null, 2));
+  exit(status === "pass" ? 0 : 1);
+}
+
+await main();

skills/skills/langbot-testing/probes/langbot-live-backend-latency.mjs

@@ -0,0 +1,212 @@
+#!/usr/bin/env node
+
+import { mkdir, writeFile } from "node:fs/promises";
+import { join, resolve } from "node:path";
+import { env, exit } from "node:process";
+
+function pad(value, size = 2) {
+  return String(value).padStart(size, "0");
+}
+
+function localIsoWithOffset(date = new Date()) {
+  const offsetMinutes = -date.getTimezoneOffset();
+  const sign = offsetMinutes >= 0 ? "+" : "-";
+  const absolute = Math.abs(offsetMinutes);
+  return [
+    `${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())}`,
+    `T${pad(date.getHours())}:${pad(date.getMinutes())}:${pad(date.getSeconds())}.${pad(date.getMilliseconds(), 3)}`,
+    `${sign}${pad(Math.floor(absolute / 60))}:${pad(absolute % 60)}`,
+  ].join("");
+}
+
+function timestampSlug(date = new Date()) {
+  return date.toISOString().replace(/\.\d{3}Z$/, "Z").replace(/[^0-9A-Za-z]+/g, "-").replace(/^-|-$/g, "");
+}
+
+function percentile(values, percentileValue) {
+  if (values.length === 0) return 0;
+  const sorted = [...values].sort((a, b) => a - b);
+  const index = Math.min(sorted.length - 1, Math.ceil((percentileValue / 100) * sorted.length) - 1);
+  return Number(sorted[index].toFixed(3));
+}
+
+function stats(values) {
+  if (values.length === 0) return { min: 0, p50: 0, p95: 0, p99: 0, max: 0 };
+  return {
+    min: Number(Math.min(...values).toFixed(3)),
+    p50: percentile(values, 50),
+    p95: percentile(values, 95),
+    p99: percentile(values, 99),
+    max: Number(Math.max(...values).toFixed(3)),
+  };
+}
+
+function parseJsonList(value, fallback) {
+  if (!value) return fallback;
+  try {
+    const parsed = JSON.parse(value);
+    return Array.isArray(parsed) && parsed.every((item) => typeof item === "string") ? parsed : fallback;
+  } catch {
+    return fallback;
+  }
+}
+
+function joinUrl(baseUrl, path) {
+  const base = baseUrl.replace(/\/+$/, "");
+  const suffix = path.startsWith("/") ? path : `/${path}`;
+  return `${base}${suffix}`;
+}
+
+async function fetchOnce(url, timeoutMs) {
+  const controller = new AbortController();
+  const timeout = setTimeout(() => controller.abort(), timeoutMs);
+  const started = performance.now();
+  try {
+    const response = await fetch(url, { method: "GET", signal: controller.signal });
+    await response.arrayBuffer();
+    const latencyMs = performance.now() - started;
+    return {
+      url,
+      ok: response.status < 500,
+      status: response.status,
+      latency_ms: Number(latencyMs.toFixed(3)),
+      error: "",
+    };
+  } catch (error) {
+    const latencyMs = performance.now() - started;
+    return {
+      url,
+      ok: false,
+      status: 0,
+      latency_ms: Number(latencyMs.toFixed(3)),
+      error: error instanceof Error ? error.message : String(error),
+    };
+  } finally {
+    clearTimeout(timeout);
+  }
+}
+
+async function runBatches(urls, totalRequests, concurrency, timeoutMs) {
+  const queue = Array.from({ length: totalRequests }, (_, index) => urls[index % urls.length]);
+  const results = [];
+  while (queue.length > 0) {
+    const batch = queue.splice(0, concurrency);
+    results.push(...await Promise.all(batch.map((url) => fetchOnce(url, timeoutMs))));
+  }
+  return results;
+}
+
+async function main() {
+  const root = resolve(env.LBS_ROOT || process.cwd());
+  const caseId = "langbot-live-backend-latency";
+  const runId = env.LBS_RUN_ID || `${timestampSlug()}-${caseId}`;
+  const evidenceDir = resolve(env.LBS_EVIDENCE_DIR || join(root, "reports", "evidence", runId));
+  await mkdir(evidenceDir, { recursive: true });
+
+  const startedAt = new Date();
+  const backendUrl = env.LANGBOT_BACKEND_URL || "";
+  const endpoints = parseJsonList(env.LANGBOT_PERF_ENDPOINTS_JSON, ["/healthz"]);
+  const totalRequests = Number(env.LANGBOT_PERF_REQUESTS || "12");
+  const concurrency = Number(env.LANGBOT_PERF_CONCURRENCY || "2");
+  const timeoutMs = Number(env.LANGBOT_PERF_TIMEOUT_MS || "5000");
+  const p95BudgetMs = Number(env.LANGBOT_PERF_BACKEND_P95_MS || "1000");
+  const maxErrorRate = Number(env.LANGBOT_PERF_MAX_ERROR_RATE || "0");
+  const metricsPath = join(evidenceDir, "metrics.json");
+  const networkLogPath = join(evidenceDir, "network.log");
+  const automationResultPath = join(evidenceDir, "automation-result.json");
+  const resultPath = join(evidenceDir, "result.json");
+
+  let status = "fail";
+  let reason = "";
+  let results = [];
+  if (!backendUrl) {
+    status = "env_issue";
+    reason = "LANGBOT_BACKEND_URL is not configured.";
+  } else {
+    const urls = endpoints.map((path) => joinUrl(backendUrl, path));
+    results = await runBatches(urls, totalRequests, concurrency, timeoutMs);
+    const okCount = results.filter((item) => item.ok).length;
+    const errorCount = results.length - okCount;
+    const errorRate = results.length === 0 ? 1 : errorCount / results.length;
+    const latencies = results.filter((item) => item.ok).map((item) => item.latency_ms);
+    const latencyStats = stats(latencies);
+    const allConnectionFailures = results.length > 0 && results.every((item) => item.status === 0);
+    if (allConnectionFailures) {
+      status = "env_issue";
+      reason = `Backend did not respond at ${backendUrl}.`;
+    } else if (latencyStats.p95 <= p95BudgetMs && errorRate <= maxErrorRate) {
+      status = "pass";
+      reason = "Live backend latency probe passed all thresholds.";
+    } else {
+      status = "fail";
+      reason = "Live backend latency probe breached latency or error-rate thresholds.";
+    }
+  }
+
+  const statusCounts = {};
+  for (const item of results) {
+    const key = item.status === 0 ? "network_error" : String(item.status);
+    statusCounts[key] = (statusCounts[key] || 0) + 1;
+  }
+  const okResults = results.filter((item) => item.ok);
+  const metrics = {
+    probe: caseId,
+    backend_url: backendUrl,
+    endpoints,
+    total_requests: totalRequests,
+    concurrency,
+    timeout_ms: timeoutMs,
+    ok_count: okResults.length,
+    error_count: results.length - okResults.length,
+    error_rate: results.length === 0 ? 1 : Number(((results.length - okResults.length) / results.length).toFixed(4)),
+    latency_ms: stats(okResults.map((item) => item.latency_ms)),
+    status_counts: statusCounts,
+  };
+  const thresholds = {
+    backend_p95_ms: { actual: metrics.latency_ms.p95, max: p95BudgetMs, pass: metrics.latency_ms.p95 <= p95BudgetMs },
+    error_rate: { actual: metrics.error_rate, max: maxErrorRate, pass: metrics.error_rate <= maxErrorRate },
+  };
+
+  await writeFile(metricsPath, `${JSON.stringify({ ...metrics, samples: results }, null, 2)}\n`, "utf8");
+  await writeFile(networkLogPath, results.map((item) => JSON.stringify(item)).join("\n") + (results.length > 0 ? "\n" : ""), "utf8");
+
+  const finishedAt = new Date();
+  const result = {
+    source: "automation",
+    case_id: caseId,
+    run_id: runId,
+    status,
+    reason,
+    started_at: startedAt.toISOString(),
+    started_at_local: localIsoWithOffset(startedAt),
+    finished_at: finishedAt.toISOString(),
+    finished_at_local: localIsoWithOffset(finishedAt),
+    duration_ms: finishedAt.getTime() - startedAt.getTime(),
+    url: backendUrl,
+    metrics_summary: {
+      requests: metrics.total_requests,
+      concurrency: metrics.concurrency,
+      ok_count: metrics.ok_count,
+      error_rate: metrics.error_rate,
+      latency_p50_ms: metrics.latency_ms.p50,
+      latency_p95_ms: metrics.latency_ms.p95,
+      status_counts: metrics.status_counts,
+    },
+    thresholds_summary: thresholds,
+    artifacts: {
+      metrics_json: metricsPath,
+      network_log: networkLogPath,
+      automation_result_json: automationResultPath,
+      result_json: resultPath,
+    },
+    evidence_collected: ["metrics", "network", "api_diagnostic", "filesystem"],
+  };
+
+  const resultText = `${JSON.stringify(result, null, 2)}\n`;
+  await writeFile(automationResultPath, resultText, "utf8");
+  await writeFile(resultPath, resultText, "utf8");
+  console.log(JSON.stringify(result, null, 2));
+  exit(status === "pass" ? 0 : status === "env_issue" ? 2 : 1);
+}
+
+await main();

skills/skills/langbot-testing/probes/langbot-live-backend-log-health.mjs

@@ -0,0 +1,205 @@
+#!/usr/bin/env node
+
+import { existsSync, readdirSync, statSync } from "node:fs";
+import { mkdir, readFile, writeFile } from "node:fs/promises";
+import { join, resolve } from "node:path";
+import { env, exit } from "node:process";
+
+function pad(value, size = 2) {
+  return String(value).padStart(size, "0");
+}
+
+function localIsoWithOffset(date = new Date()) {
+  const offsetMinutes = -date.getTimezoneOffset();
+  const sign = offsetMinutes >= 0 ? "+" : "-";
+  const absolute = Math.abs(offsetMinutes);
+  return [
+    `${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())}`,
+    `T${pad(date.getHours())}:${pad(date.getMinutes())}:${pad(date.getSeconds())}.${pad(date.getMilliseconds(), 3)}`,
+    `${sign}${pad(Math.floor(absolute / 60))}:${pad(absolute % 60)}`,
+  ].join("");
+}
+
+function timestampSlug(date = new Date()) {
+  return date.toISOString().replace(/\.\d{3}Z$/, "Z").replace(/[^0-9A-Za-z]+/g, "-").replace(/^-|-$/g, "");
+}
+
+function repoRootFromEnv(root) {
+  return env.LANGBOT_REPO ? resolve(env.LANGBOT_REPO) : resolve(root, "..");
+}
+
+function latestBackendLog(root) {
+  const explicit = env.LANGBOT_BACKEND_LOG;
+  if (explicit) return resolve(explicit);
+
+  const logsDir = join(repoRootFromEnv(root), "data", "logs");
+  if (!existsSync(logsDir)) return "";
+  const candidates = readdirSync(logsDir)
+    .filter((name) => /^langbot-.*\.log$/.test(name))
+    .map((name) => join(logsDir, name))
+    .filter((path) => {
+      try {
+        return statSync(path).isFile();
+      } catch {
+        return false;
+      }
+    })
+    .sort((left, right) => statSync(right).mtimeMs - statSync(left).mtimeMs);
+  return candidates[0] || "";
+}
+
+function parseSince(startedAt) {
+  if (env.LANGBOT_BACKEND_LOG_SINCE) return new Date(env.LANGBOT_BACKEND_LOG_SINCE);
+  const lookbackSeconds = Number(env.LANGBOT_BACKEND_LOG_LOOKBACK_SECONDS || "300");
+  return new Date(startedAt.getTime() - lookbackSeconds * 1000);
+}
+
+function parseTimestamp(line, year) {
+  const localMatch = line.match(/^\[(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})\.(\d{3})\]/);
+  if (localMatch) {
+    const [, month, day, hour, minute, second, millisecond] = localMatch;
+    return new Date(`${year}-${month}-${day}T${hour}:${minute}:${second}.${millisecond}+08:00`);
+  }
+
+  const accessMatch = line.match(/^\[(\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2}) ([+-]\d{4})\]/);
+  if (accessMatch) {
+    const [, fullYear, month, day, hour, minute, second, offset] = accessMatch;
+    const normalizedOffset = `${offset.slice(0, 3)}:${offset.slice(3)}`;
+    return new Date(`${fullYear}-${month}-${day}T${hour}:${minute}:${second}${normalizedOffset}`);
+  }
+
+  return null;
+}
+
+function findingForLine(line, number) {
+  const rules = [
+    { severity: "fail", kind: "python_traceback", pattern: /\bTraceback(?: \(most recent call last\))?/i },
+    { severity: "fail", kind: "unretrieved_task_exception", pattern: /Task exception was never retrieved/i },
+    { severity: "fail", kind: "unawaited_coroutine", pattern: /RuntimeWarning:\s+coroutine .* was never awaited/i },
+    { severity: "fail", kind: "unclosed_client_session", pattern: /Unclosed client session/i },
+    { severity: "fail", kind: "unclosed_connector", pattern: /Unclosed connector/i },
+    { severity: "fail", kind: "import_error", pattern: /\bImportError\b/i },
+    { severity: "fail", kind: "error_log", pattern: /\b(?:ERROR|CRITICAL)\b/ },
+    { severity: "warning", kind: "warning_log", pattern: /\bWARNING\b/ },
+  ];
+
+  for (const rule of rules) {
+    if (rule.pattern.test(line)) {
+      return {
+        severity: rule.severity,
+        kind: rule.kind,
+        line: number,
+        excerpt: line,
+      };
+    }
+  }
+  return null;
+}
+
+function scanLines(text, since, year) {
+  const findings = [];
+  const scanned = [];
+  let includeContinuation = false;
+  const lines = text.split(/\r?\n/);
+  for (const [index, line] of lines.entries()) {
+    const number = index + 1;
+    const timestamp = parseTimestamp(line, year);
+    if (timestamp) includeContinuation = timestamp >= since;
+    if (!includeContinuation) continue;
+    scanned.push({ number, text: line });
+    const finding = findingForLine(line, number);
+    if (finding) findings.push(finding);
+  }
+  return { findings, scanned, total_lines: lines.length };
+}
+
+async function main() {
+  const root = resolve(env.LBS_ROOT || process.cwd());
+  const caseId = "langbot-live-backend-log-health";
+  const runId = env.LBS_RUN_ID || `${timestampSlug()}-${caseId}`;
+  const evidenceDir = resolve(env.LBS_EVIDENCE_DIR || join(root, "reports", "evidence", runId));
+  await mkdir(evidenceDir, { recursive: true });
+
+  const startedAt = new Date();
+  const since = parseSince(startedAt);
+  const logPath = latestBackendLog(root);
+  const metricsPath = join(evidenceDir, "metrics.json");
+  const findingsPath = join(evidenceDir, "findings.json");
+  const scannedLogPath = join(evidenceDir, "scanned-backend.log");
+  const automationResultPath = join(evidenceDir, "automation-result.json");
+  const resultPath = join(evidenceDir, "result.json");
+
+  let status = "fail";
+  let reason = "";
+  let scan = { findings: [], scanned: [], total_lines: 0 };
+  if (!logPath || !existsSync(logPath)) {
+    status = "env_issue";
+    reason = "No LangBot backend log file was found. Set LANGBOT_BACKEND_LOG or LANGBOT_REPO.";
+  } else {
+    const text = await readFile(logPath, "utf8");
+    scan = scanLines(text, since, startedAt.getFullYear());
+    const failCount = scan.findings.filter((item) => item.severity === "fail").length;
+    status = failCount === 0 ? "pass" : "fail";
+    reason = status === "pass"
+      ? "Live backend log health passed; no fail-severity findings in the scanned window."
+      : "Live backend log health found fail-severity backend log findings.";
+  }
+
+  const warningCount = scan.findings.filter((item) => item.severity === "warning").length;
+  const failCount = scan.findings.filter((item) => item.severity === "fail").length;
+  const metrics = {
+    probe: caseId,
+    backend_log: logPath,
+    since: since.toISOString(),
+    scanned_line_count: scan.scanned.length,
+    total_line_count: scan.total_lines,
+    fail_count: failCount,
+    warning_count: warningCount,
+    finding_count: scan.findings.length,
+  };
+  const thresholds = {
+    fail_count: { actual: failCount, max: 0, pass: failCount === 0 },
+  };
+
+  await writeFile(metricsPath, `${JSON.stringify(metrics, null, 2)}\n`, "utf8");
+  await writeFile(findingsPath, `${JSON.stringify(scan.findings, null, 2)}\n`, "utf8");
+  await writeFile(scannedLogPath, scan.scanned.map((item) => `${item.number}: ${item.text}`).join("\n") + (scan.scanned.length > 0 ? "\n" : ""), "utf8");
+
+  const finishedAt = new Date();
+  const result = {
+    source: "automation",
+    case_id: caseId,
+    run_id: runId,
+    status,
+    reason,
+    started_at: startedAt.toISOString(),
+    started_at_local: localIsoWithOffset(startedAt),
+    finished_at: finishedAt.toISOString(),
+    finished_at_local: localIsoWithOffset(finishedAt),
+    duration_ms: finishedAt.getTime() - startedAt.getTime(),
+    url: logPath,
+    metrics_summary: {
+      scanned_line_count: metrics.scanned_line_count,
+      fail_count: metrics.fail_count,
+      warning_count: metrics.warning_count,
+      finding_count: metrics.finding_count,
+    },
+    thresholds_summary: thresholds,
+    artifacts: {
+      metrics_json: metricsPath,
+      findings_json: findingsPath,
+      scanned_backend_log: scannedLogPath,
+      automation_result_json: automationResultPath,
+      result_json: resultPath,
+    },
+    evidence_collected: ["metrics", "backend_log", "filesystem"],
+  };
+
+  const resultText = `${JSON.stringify(result, null, 2)}\n`;
+  await writeFile(automationResultPath, resultText, "utf8");
+  await writeFile(resultPath, resultText, "utf8");
+  console.log(JSON.stringify(result, null, 2));
+  exit(status === "pass" ? 0 : status === "env_issue" ? 2 : 1);
+}
+
+await main();

skills/skills/langbot-testing/probes/langbot-live-control-plane-api.mjs

@@ -0,0 +1,311 @@
+#!/usr/bin/env node
+
+import { mkdir, writeFile } from "node:fs/promises";
+import { join, resolve } from "node:path";
+import { env, exit } from "node:process";
+
+function pad(value, size = 2) {
+  return String(value).padStart(size, "0");
+}
+
+function localIsoWithOffset(date = new Date()) {
+  const offsetMinutes = -date.getTimezoneOffset();
+  const sign = offsetMinutes >= 0 ? "+" : "-";
+  const absolute = Math.abs(offsetMinutes);
+  return [
+    `${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())}`,
+    `T${pad(date.getHours())}:${pad(date.getMinutes())}:${pad(date.getSeconds())}.${pad(date.getMilliseconds(), 3)}`,
+    `${sign}${pad(Math.floor(absolute / 60))}:${pad(absolute % 60)}`,
+  ].join("");
+}
+
+function timestampSlug(date = new Date()) {
+  return date.toISOString().replace(/\.\d{3}Z$/, "Z").replace(/[^0-9A-Za-z]+/g, "-").replace(/^-|-$/g, "");
+}
+
+function percentile(values, percentileValue) {
+  if (values.length === 0) return 0;
+  const sorted = [...values].sort((a, b) => a - b);
+  const index = Math.min(sorted.length - 1, Math.ceil((percentileValue / 100) * sorted.length) - 1);
+  return Number(sorted[index].toFixed(3));
+}
+
+function stats(values) {
+  if (values.length === 0) return { min: 0, p50: 0, p95: 0, p99: 0, max: 0 };
+  return {
+    min: Number(Math.min(...values).toFixed(3)),
+    p50: percentile(values, 50),
+    p95: percentile(values, 95),
+    p99: percentile(values, 99),
+    max: Number(Math.max(...values).toFixed(3)),
+  };
+}
+
+function joinUrl(baseUrl, path) {
+  const base = baseUrl.replace(/\/+$/, "");
+  const suffix = path.startsWith("/") ? path : `/${path}`;
+  return `${base}${suffix}`;
+}
+
+function parseJsonObject(value, fallback) {
+  if (!value) return fallback;
+  try {
+    const parsed = JSON.parse(value);
+    return parsed && typeof parsed === "object" && !Array.isArray(parsed) ? parsed : fallback;
+  } catch {
+    return fallback;
+  }
+}
+
+function controlPlaneEndpoints() {
+  return [
+    {
+      id: "healthz",
+      path: "/healthz",
+      expected_status: 200,
+      expected_code: 0,
+      p95_budget_ms: Number(env.LANGBOT_PERF_HEALTHZ_P95_MS || "500"),
+      required_data_fields: [],
+    },
+    {
+      id: "system_info",
+      path: "/api/v1/system/info",
+      expected_status: 200,
+      expected_code: 0,
+      p95_budget_ms: Number(env.LANGBOT_PERF_SYSTEM_INFO_P95_MS || "1000"),
+      required_data_fields: ["version", "edition", "enable_marketplace"],
+    },
+  ];
+}
+
+async function fetchEndpoint(backendUrl, endpoint, timeoutMs) {
+  const url = joinUrl(backendUrl, endpoint.path);
+  const controller = new AbortController();
+  const timeout = setTimeout(() => controller.abort(), timeoutMs);
+  const started = performance.now();
+  let bodyText = "";
+  let json = null;
+  let jsonValid = false;
+  let error = "";
+
+  try {
+    const response = await fetch(url, {
+      method: "GET",
+      headers: { "accept": "application/json" },
+      signal: controller.signal,
+    });
+    bodyText = await response.text();
+    try {
+      json = bodyText ? JSON.parse(bodyText) : null;
+      jsonValid = json !== null;
+    } catch (parseError) {
+      error = parseError instanceof Error ? parseError.message : String(parseError);
+    }
+
+    const data = json && typeof json === "object" && json.data && typeof json.data === "object" ? json.data : {};
+    const missingFields = endpoint.required_data_fields.filter((field) => !(field in data));
+    const statusOk = response.status === endpoint.expected_status;
+    const codeOk = !json || typeof json !== "object" ? false : json.code === endpoint.expected_code;
+    const shapeOk = jsonValid && missingFields.length === 0;
+    const latencyMs = performance.now() - started;
+    return {
+      endpoint_id: endpoint.id,
+      path: endpoint.path,
+      url,
+      status: response.status,
+      ok: statusOk && codeOk && shapeOk,
+      status_ok: statusOk,
+      code_ok: codeOk,
+      json_valid: jsonValid,
+      missing_fields: missingFields,
+      response_code: json && typeof json === "object" ? json.code : null,
+      latency_ms: Number(latencyMs.toFixed(3)),
+      error,
+    };
+  } catch (fetchError) {
+    const latencyMs = performance.now() - started;
+    return {
+      endpoint_id: endpoint.id,
+      path: endpoint.path,
+      url,
+      status: 0,
+      ok: false,
+      status_ok: false,
+      code_ok: false,
+      json_valid: false,
+      missing_fields: endpoint.required_data_fields,
+      response_code: null,
+      latency_ms: Number(latencyMs.toFixed(3)),
+      error: fetchError instanceof Error ? fetchError.message : String(fetchError),
+    };
+  } finally {
+    clearTimeout(timeout);
+  }
+}
+
+async function runBatches(backendUrl, endpoints, totalRequests, concurrency, timeoutMs) {
+  const queue = Array.from({ length: totalRequests }, (_, index) => endpoints[index % endpoints.length]);
+  const results = [];
+  while (queue.length > 0) {
+    const batch = queue.splice(0, concurrency);
+    results.push(...await Promise.all(batch.map((endpoint) => fetchEndpoint(backendUrl, endpoint, timeoutMs))));
+  }
+  return results;
+}
+
+function endpointMetrics(endpoints, results) {
+  return Object.fromEntries(endpoints.map((endpoint) => {
+    const samples = results.filter((item) => item.endpoint_id === endpoint.id);
+    const okSamples = samples.filter((item) => item.ok);
+    return [
+      endpoint.id,
+      {
+        path: endpoint.path,
+        requests: samples.length,
+        ok_count: okSamples.length,
+        error_rate: samples.length === 0 ? 1 : Number(((samples.length - okSamples.length) / samples.length).toFixed(4)),
+        latency_ms: stats(okSamples.map((item) => item.latency_ms)),
+        p95_budget_ms: endpoint.p95_budget_ms,
+      },
+    ];
+  }));
+}
+
+async function main() {
+  const root = resolve(env.LBS_ROOT || process.cwd());
+  const caseId = "langbot-live-control-plane-api";
+  const runId = env.LBS_RUN_ID || `${timestampSlug()}-${caseId}`;
+  const evidenceDir = resolve(env.LBS_EVIDENCE_DIR || join(root, "reports", "evidence", runId));
+  await mkdir(evidenceDir, { recursive: true });
+
+  const startedAt = new Date();
+  const backendUrl = env.LANGBOT_BACKEND_URL || "";
+  const endpoints = controlPlaneEndpoints();
+  const configuredBudgets = parseJsonObject(env.LANGBOT_CONTROL_PLANE_P95_BUDGETS_JSON, {});
+  for (const endpoint of endpoints) {
+    const budget = configuredBudgets[endpoint.id];
+    if (typeof budget === "number" && Number.isFinite(budget)) endpoint.p95_budget_ms = budget;
+  }
+  const totalRequests = Number(env.LANGBOT_CONTROL_PLANE_REQUESTS || "20");
+  const concurrency = Number(env.LANGBOT_CONTROL_PLANE_CONCURRENCY || "4");
+  const timeoutMs = Number(env.LANGBOT_CONTROL_PLANE_TIMEOUT_MS || "5000");
+  const maxErrorRate = Number(env.LANGBOT_CONTROL_PLANE_MAX_ERROR_RATE || "0");
+  const metricsPath = join(evidenceDir, "metrics.json");
+  const endpointsPath = join(evidenceDir, "endpoints.json");
+  const networkLogPath = join(evidenceDir, "network.log");
+  const automationResultPath = join(evidenceDir, "automation-result.json");
+  const resultPath = join(evidenceDir, "result.json");
+
+  let status = "fail";
+  let reason = "";
+  let results = [];
+  if (!backendUrl) {
+    status = "env_issue";
+    reason = "LANGBOT_BACKEND_URL is not configured.";
+  } else {
+    results = await runBatches(backendUrl, endpoints, totalRequests, concurrency, timeoutMs);
+    const allConnectionFailures = results.length > 0 && results.every((item) => item.status === 0);
+    if (allConnectionFailures) {
+      status = "env_issue";
+      reason = `Backend did not respond at ${backendUrl}.`;
+    }
+  }
+
+  const okResults = results.filter((item) => item.ok);
+  const statusCounts = {};
+  for (const item of results) {
+    const key = item.status === 0 ? "network_error" : String(item.status);
+    statusCounts[key] = (statusCounts[key] || 0) + 1;
+  }
+  const perEndpoint = endpointMetrics(endpoints, results);
+  const responseShapeFailures = results.filter((item) => !item.json_valid || item.missing_fields.length > 0 || !item.code_ok).length;
+  const errorRate = results.length === 0 ? 1 : Number(((results.length - okResults.length) / results.length).toFixed(4));
+  const thresholds = {
+    error_rate: { actual: errorRate, max: maxErrorRate, pass: errorRate <= maxErrorRate },
+    response_shape_failures: { actual: responseShapeFailures, max: 0, pass: responseShapeFailures === 0 },
+  };
+  for (const endpoint of endpoints) {
+    const actual = perEndpoint[endpoint.id].latency_ms.p95;
+    thresholds[`${endpoint.id}_p95_ms`] = {
+      actual,
+      max: endpoint.p95_budget_ms,
+      pass: actual <= endpoint.p95_budget_ms,
+    };
+  }
+
+  if (status !== "env_issue") {
+    const passed = Object.values(thresholds).every((item) => item.pass);
+    status = passed ? "pass" : "fail";
+    reason = passed
+      ? "Live control-plane API probe passed all thresholds."
+      : "Live control-plane API probe breached shape, latency, or error-rate thresholds.";
+  }
+
+  const metrics = {
+    probe: caseId,
+    backend_url: backendUrl,
+    total_requests: totalRequests,
+    concurrency,
+    timeout_ms: timeoutMs,
+    ok_count: okResults.length,
+    error_count: results.length - okResults.length,
+    error_rate: errorRate,
+    status_counts: statusCounts,
+    response_shape_failures: responseShapeFailures,
+    endpoints: perEndpoint,
+  };
+
+  await writeFile(metricsPath, `${JSON.stringify({ ...metrics, samples: results }, null, 2)}\n`, "utf8");
+  await writeFile(endpointsPath, `${JSON.stringify(endpoints, null, 2)}\n`, "utf8");
+  await writeFile(networkLogPath, results.map((item) => JSON.stringify(item)).join("\n") + (results.length > 0 ? "\n" : ""), "utf8");
+
+  const finishedAt = new Date();
+  const result = {
+    source: "automation",
+    case_id: caseId,
+    run_id: runId,
+    status,
+    reason,
+    started_at: startedAt.toISOString(),
+    started_at_local: localIsoWithOffset(startedAt),
+    finished_at: finishedAt.toISOString(),
+    finished_at_local: localIsoWithOffset(finishedAt),
+    duration_ms: finishedAt.getTime() - startedAt.getTime(),
+    url: backendUrl,
+    metrics_summary: {
+      requests: metrics.total_requests,
+      concurrency: metrics.concurrency,
+      ok_count: metrics.ok_count,
+      error_rate: metrics.error_rate,
+      response_shape_failures: metrics.response_shape_failures,
+      endpoints: Object.fromEntries(Object.entries(metrics.endpoints).map(([id, value]) => [
+        id,
+        {
+          path: value.path,
+          ok_count: value.ok_count,
+          error_rate: value.error_rate,
+          latency_p50_ms: value.latency_ms.p50,
+          latency_p95_ms: value.latency_ms.p95,
+        },
+      ])),
+      status_counts: metrics.status_counts,
+    },
+    thresholds_summary: thresholds,
+    artifacts: {
+      metrics_json: metricsPath,
+      endpoints_json: endpointsPath,
+      network_log: networkLogPath,
+      automation_result_json: automationResultPath,
+      result_json: resultPath,
+    },
+    evidence_collected: ["metrics", "network", "api_diagnostic", "filesystem"],
+  };
+
+  const resultText = `${JSON.stringify(result, null, 2)}\n`;
+  await writeFile(automationResultPath, resultText, "utf8");
+  await writeFile(resultPath, resultText, "utf8");
+  console.log(JSON.stringify(result, null, 2));
+  exit(status === "pass" ? 0 : status === "env_issue" ? 2 : 1);
+}
+
+await main();

skills/skills/langbot-testing/probes/langbot-overhead-accounting-contract.mjs

@@ -0,0 +1,162 @@
+#!/usr/bin/env node
+
+import { mkdir, writeFile } from "node:fs/promises";
+import { join, resolve } from "node:path";
+import { env, exit } from "node:process";
+
+function pad(value, size = 2) {
+  return String(value).padStart(size, "0");
+}
+
+function localIsoWithOffset(date = new Date()) {
+  const offsetMinutes = -date.getTimezoneOffset();
+  const sign = offsetMinutes >= 0 ? "+" : "-";
+  const absolute = Math.abs(offsetMinutes);
+  return [
+    `${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())}`,
+    `T${pad(date.getHours())}:${pad(date.getMinutes())}:${pad(date.getSeconds())}.${pad(date.getMilliseconds(), 3)}`,
+    `${sign}${pad(Math.floor(absolute / 60))}:${pad(absolute % 60)}`,
+  ].join("");
+}
+
+function timestampSlug(date = new Date()) {
+  return date.toISOString().replace(/\.\d{3}Z$/, "Z").replace(/[^0-9A-Za-z]+/g, "-").replace(/^-|-$/g, "");
+}
+
+function percentile(values, percentileValue) {
+  if (values.length === 0) return 0;
+  const sorted = [...values].sort((a, b) => a - b);
+  const index = Math.min(sorted.length - 1, Math.ceil((percentileValue / 100) * sorted.length) - 1);
+  return Number(sorted[index].toFixed(3));
+}
+
+function stats(values) {
+  return {
+    min: Number(Math.min(...values).toFixed(3)),
+    p50: percentile(values, 50),
+    p95: percentile(values, 95),
+    p99: percentile(values, 99),
+    max: Number(Math.max(...values).toFixed(3)),
+  };
+}
+
+function threshold(actual, limit, operator) {
+  const pass = operator === "<=" ? actual <= limit : actual >= limit;
+  return { actual, [operator === "<=" ? "max" : "min"]: limit, pass };
+}
+
+function makeSample(index) {
+  const ingress = 1 + (index % 5) * 0.22;
+  const pipeline = 2.8 + (index % 7) * 0.31;
+  const persistence = 1.1 + (index % 4) * 0.2;
+  const pluginIpc = 1.9 + (index % 6) * 0.27;
+  const rag = index % 3 === 0 ? 4.4 : 0.8 + (index % 5) * 0.18;
+  const streaming = 1.5 + (index % 8) * 0.24;
+  const provider = 80 + (index % 13) * 11;
+  const externalTool = index % 4 === 0 ? 25 + (index % 9) * 3 : 0;
+  const network = 8 + (index % 10) * 1.7;
+  const overhead = ingress + pipeline + persistence + pluginIpc + rag + streaming;
+  const external = provider + externalTool + network;
+  const total = overhead + external;
+  return {
+    index,
+    segments_ms: {
+      ingress,
+      pipeline,
+      persistence,
+      plugin_ipc: pluginIpc,
+      rag,
+      streaming,
+      provider,
+      external_tool: externalTool,
+      network,
+    },
+    langbot_overhead_ms: Number(overhead.toFixed(3)),
+    external_latency_ms: Number(external.toFixed(3)),
+    e2e_latency_ms: Number(total.toFixed(3)),
+    accounting_gap_ms: Number((total - external - overhead).toFixed(6)),
+  };
+}
+
+async function main() {
+  const root = resolve(env.LBS_ROOT || process.cwd());
+  const caseId = "langbot-overhead-accounting-contract";
+  const runId = env.LBS_RUN_ID || `${timestampSlug()}-${caseId}`;
+  const evidenceDir = resolve(env.LBS_EVIDENCE_DIR || join(root, "reports", "evidence", runId));
+  await mkdir(evidenceDir, { recursive: true });
+
+  const startedAt = new Date();
+  const sampleCount = Number(env.LANGBOT_PERF_CONTRACT_SAMPLES || "80");
+  const overheadP95BudgetMs = Number(env.LANGBOT_PERF_OVERHEAD_P95_MS || "25");
+  const samples = Array.from({ length: sampleCount }, (_, index) => makeSample(index));
+  const overheads = samples.map((sample) => sample.langbot_overhead_ms);
+  const e2e = samples.map((sample) => sample.e2e_latency_ms);
+  const external = samples.map((sample) => sample.external_latency_ms);
+  const gaps = samples.map((sample) => Math.abs(sample.accounting_gap_ms));
+  const memory = process.memoryUsage();
+
+  const metrics = {
+    probe: caseId,
+    sample_count: sampleCount,
+    langbot_overhead_ms: stats(overheads),
+    e2e_latency_ms: stats(e2e),
+    external_latency_ms: stats(external),
+    accounting_gap_max_ms: Number(Math.max(...gaps).toFixed(6)),
+    samples,
+  };
+  const thresholds = {
+    sample_count: threshold(sampleCount, 50, ">="),
+    langbot_overhead_p95_ms: threshold(metrics.langbot_overhead_ms.p95, overheadP95BudgetMs, "<="),
+    accounting_gap_max_ms: threshold(metrics.accounting_gap_max_ms, 0.001, "<="),
+  };
+  const status = Object.values(thresholds).every((item) => item.pass) ? "pass" : "fail";
+  const metricsPath = join(evidenceDir, "metrics.json");
+  const thresholdsPath = join(evidenceDir, "thresholds.json");
+  const resourceLogPath = join(evidenceDir, "resource-log.json");
+  const automationResultPath = join(evidenceDir, "automation-result.json");
+  const resultPath = join(evidenceDir, "result.json");
+
+  await writeFile(metricsPath, `${JSON.stringify(metrics, null, 2)}\n`, "utf8");
+  await writeFile(thresholdsPath, `${JSON.stringify(thresholds, null, 2)}\n`, "utf8");
+  await writeFile(resourceLogPath, `${JSON.stringify({ memory, pid: process.pid }, null, 2)}\n`, "utf8");
+
+  const finishedAt = new Date();
+  const result = {
+    source: "automation",
+    case_id: caseId,
+    run_id: runId,
+    status,
+    reason: status === "pass"
+      ? "Overhead accounting contract passed all thresholds."
+      : "Overhead accounting contract breached one or more thresholds.",
+    started_at: startedAt.toISOString(),
+    started_at_local: localIsoWithOffset(startedAt),
+    finished_at: finishedAt.toISOString(),
+    finished_at_local: localIsoWithOffset(finishedAt),
+    duration_ms: finishedAt.getTime() - startedAt.getTime(),
+    metrics_summary: {
+      sample_count: metrics.sample_count,
+      langbot_overhead_p95_ms: metrics.langbot_overhead_ms.p95,
+      e2e_latency_p95_ms: metrics.e2e_latency_ms.p95,
+      external_latency_p95_ms: metrics.external_latency_ms.p95,
+      accounting_gap_max_ms: metrics.accounting_gap_max_ms,
+    },
+    thresholds_summary: thresholds,
+    artifacts: {
+      metrics_json: metricsPath,
+      thresholds_json: thresholdsPath,
+      resource_log_json: resourceLogPath,
+      automation_result_json: automationResultPath,
+      result_json: resultPath,
+    },
+    evidence_collected: ["metrics", "resource_log", "filesystem"],
+  };
+
+  const resultText = `${JSON.stringify(result, null, 2)}\n`;
+  await writeFile(automationResultPath, resultText, "utf8");
+  await writeFile(resultPath, resultText, "utf8");
+  console.log(JSON.stringify(result, null, 2));
+  exit(status === "pass" ? 0 : 1);
+}
+
+await main();

skills/skills/langbot-testing/probes/lib/fake-provider-timing.mjs

@@ -0,0 +1,134 @@
+export function summarizeFakeProviderState(state) {
+  if (!state) return null;
+  const recentRequests = Array.isArray(state.recent_requests) ? state.recent_requests : [];
+  const chatRequests = recentRequests.filter((request) => String(request?.path || "").includes("/chat/completions"));
+  const successfulRequests = chatRequests.filter((request) => request?.status === "ok");
+  const faultRequests = chatRequests.filter((request) => (
+    request?.should_fail === true
+      || request?.status === "http_fault"
+      || (Number.isFinite(request?.http_status) && request.http_status >= 400)
+  ));
+
+  return {
+    status: state.status || "unknown",
+    url: state.url || "",
+    request_count: Number.isFinite(state.request_count) ? state.request_count : recentRequests.length,
+    recent_request_count: recentRequests.length,
+    chat_request_count: chatRequests.length,
+    fault_count: faultRequests.length,
+    streamed_request_count: chatRequests.filter((request) => request?.stream === true).length,
+    duration_ms: stats(chatRequests.map((request) => numberOrNull(request?.duration_ms)).filter(Number.isFinite)),
+    successful_duration_ms: stats(successfulRequests.map((request) => numberOrNull(request?.duration_ms)).filter(Number.isFinite)),
+    first_chunk_ms: stats(successfulRequests.map((request) => numberOrNull(request?.first_chunk_ms)).filter(Number.isFinite)),
+    first_content_chunk_ms: stats(successfulRequests.map((request) => numberOrNull(request?.first_content_chunk_ms)).filter(Number.isFinite)),
+    content_chunk_count: stats(successfulRequests.map((request) => numberOrNull(request?.content_chunk_count)).filter(Number.isFinite)),
+    config: state.config || {},
+  };
+}
+
+export function buildProviderTimingMetrics(samples, state) {
+  const recentRequests = Array.isArray(state?.recent_requests) ? state.recent_requests : [];
+  const byExpectedText = new Map();
+  for (const request of recentRequests) {
+    const expected = String(request?.expected_text || "");
+    if (!expected) continue;
+    if (!byExpectedText.has(expected)) byExpectedText.set(expected, []);
+    byExpectedText.get(expected).push(request);
+  }
+
+  const segments = [];
+  const missingExpectedText = [];
+  for (const sample of samples) {
+    const expected = String(sample?.expected_text || "");
+    if (!expected) continue;
+    const request = (byExpectedText.get(expected) || []).shift();
+    if (!request) {
+      missingExpectedText.push(expected);
+      continue;
+    }
+    const segment = buildTimingSegment(sample, request);
+    if (segment) segments.push(segment);
+  }
+
+  const values = (key) => segments.map((segment) => numberOrNull(segment[key])).filter(Number.isFinite);
+  return {
+    matched_request_count: segments.length,
+    missing_provider_match_count: missingExpectedText.length,
+    missing_expected_text: missingExpectedText.slice(0, 20),
+    send_to_provider_start_ms: stats(values("send_to_provider_start_ms")),
+    provider_duration_ms: stats(values("provider_duration_ms")),
+    provider_finish_to_ws_final_ms: stats(values("provider_finish_to_ws_final_ms")),
+    langbot_overhead_estimate_ms: stats(values("langbot_overhead_estimate_ms")),
+    e2e_minus_provider_ms: stats(values("e2e_minus_provider_ms")),
+    provider_first_content_to_ws_first_content_ms: stats(values("provider_first_content_to_ws_first_content_ms")),
+    segments,
+  };
+}
+
+function buildTimingSegment(sample, request) {
+  const sentEpochMs = numberOrNull(sample.sent_epoch_ms);
+  const finishedEpochMs = numberOrNull(sample.finished_epoch_ms);
+  const providerStartedEpochMs = numberOrNull(request.started_epoch_ms);
+  const providerFinishedEpochMs = numberOrNull(request.finished_epoch_ms);
+  const providerFirstContentEpochMs = numberOrNull(request.first_content_chunk_epoch_ms);
+  const wsFirstContentEpochMs = numberOrNull(sample.first_assistant_content_epoch_ms);
+  const responseDurationMs = numberOrNull(sample.response_duration_ms);
+  const providerDurationMs = numberOrNull(request.duration_ms);
+
+  const sendToProviderStartMs = finiteDelta(providerStartedEpochMs, sentEpochMs);
+  const providerFinishToWsFinalMs = finiteDelta(finishedEpochMs, providerFinishedEpochMs);
+  const e2eMinusProviderMs = Number.isFinite(responseDurationMs) && Number.isFinite(providerDurationMs)
+    ? rounded(responseDurationMs - providerDurationMs)
+    : null;
+  const overheadEstimateMs = Number.isFinite(sendToProviderStartMs) && Number.isFinite(providerFinishToWsFinalMs)
+    ? rounded(sendToProviderStartMs + providerFinishToWsFinalMs)
+    : e2eMinusProviderMs;
+
+  return {
+    sample_index: sample.index,
+    pipeline_label: sample.pipeline_label || "",
+    expected_text: sample.expected_text || "",
+    provider_request_id: request.id || "",
+    provider_request_number: request.request_number ?? null,
+    response_duration_ms: responseDurationMs,
+    provider_duration_ms: providerDurationMs,
+    send_to_provider_start_ms: sendToProviderStartMs,
+    provider_finish_to_ws_final_ms: providerFinishToWsFinalMs,
+    langbot_overhead_estimate_ms: overheadEstimateMs,
+    e2e_minus_provider_ms: e2eMinusProviderMs,
+    provider_first_content_to_ws_first_content_ms: finiteDelta(wsFirstContentEpochMs, providerFirstContentEpochMs),
+    provider_status: request.status || "",
+    provider_http_status: request.http_status ?? null,
+  };
+}
+
+function finiteDelta(left, right) {
+  return Number.isFinite(left) && Number.isFinite(right) ? rounded(left - right) : null;
+}
+
+export function stats(values) {
+  if (values.length === 0) return { min: 0, p50: 0, p95: 0, p99: 0, max: 0 };
+  return {
+    min: rounded(Math.min(...values)),
+    p50: percentile(values, 50),
+    p95: percentile(values, 95),
+    p99: percentile(values, 99),
+    max: rounded(Math.max(...values)),
+  };
+}
+
+export function percentile(values, percentileValue) {
+  if (values.length === 0) return 0;
+  const sorted = [...values].sort((a, b) => a - b);
+  const index = Math.min(sorted.length - 1, Math.ceil((percentileValue / 100) * sorted.length) - 1);
+  return rounded(sorted[index]);
+}
+
+export function rounded(value) {
+  return Number(value.toFixed(3));
+}
+
+function numberOrNull(value) {
+  const number = Number(value);
+  return Number.isFinite(number) ? number : null;
+}

skills/skills/langbot-testing/references/performance-reliability-testing.md

@@ -0,0 +1,285 @@
+# Performance And Reliability Testing
+
+Use this reference when a QA request asks whether LangBot is fast enough,
+stable under load, or resilient to controlled faults.
+
+These probes are manual/non-required QA gates unless a case or suite explicitly
+states otherwise. They depend on a live local backend, mutable QA fixtures, and
+operator-selected environment variables, so do not promote them to required CI
+checks until fake-provider isolation, ownership markers, and cleanup are in
+place.
+
+## Scope
+
+Treat `skills/` as the QA control plane:
+
+- Cases define intent, readiness, thresholds, and required evidence.
+- Probe scripts collect metrics, traces, resource logs, and artifacts.
+- Reports classify the same run as `pass`, `fail`, `blocked`,
+  `env_issue`, or `flaky`.
+
+Do not turn `skills/` into a load generator or chaos engine. Call a focused
+tool from a `mode: probe` case when the test needs one, for example k6,
+Locust, pytest-benchmark, Playwright trace collection, Toxiproxy, Docker, or a
+Kubernetes disruption tool.
+
+## LangBot Performance Model
+
+For LangBot, performance is the cost LangBot adds around external systems:
+
+```text
+LangBot overhead = end-to-end latency - provider latency - external tool latency - network/fault injection latency
+```
+
+Measure user experience and internal composition separately:
+
+- WebUI load and interaction latency.
+- Debug Chat send-to-first-visible-token and send-to-completion latency.
+- Pipeline, RAG, plugin runtime, MCP, AgentRunner, and persistence segment
+  latency.
+- Queue wait time, concurrency, throughput, timeout rate, and p95/p99 latency.
+- Startup, plugin install, knowledge-base ingestion, migration, and recovery
+  time.
+
+Do not report a single message round-trip time as "LangBot performance" unless
+the report also explains external provider/tool/network time.
+
+## Evidence Contract
+
+Performance and reliability cases should declare the evidence they need:
+
+- `metrics`: machine-readable latency, throughput, error-rate, or recovery
+  metrics, usually `metrics.json`.
+- `resource_log`: CPU, memory, process, connection, queue, or file descriptor
+  samples.
+- `trace`: browser, HTTP, database, or runtime trace artifacts.
+- `profile`: CPU, memory, or flamegraph profile artifacts.
+- `backend_log`, `network`, `api_diagnostic`, and `filesystem` as supporting
+  evidence when relevant.
+
+Automation should write `automation-result.json` with these fields when
+available:
+
+```json
+{
+  "status": "pass",
+  "reason": "Probe passed all thresholds.",
+  "metrics_summary": {
+    "langbot_overhead_p95_ms": 12.4,
+    "error_rate": 0
+  },
+  "thresholds_summary": {
+    "langbot_overhead_p95_ms": { "actual": 12.4, "max": 50, "pass": true }
+  },
+  "artifacts": {
+    "metrics_json": "/path/to/metrics.json"
+  },
+  "evidence_collected": ["metrics", "filesystem"]
+}
+```
+
+Synthetic contract probes are useful for checking the QA harness, but they are
+not live product performance results. Label them as contract probes in the case
+title, checks, and report.
+
+## Chaos And Reliability Rules
+
+Chaos tests must be narrow and reversible:
+
+- Declare the fault model in `fault_model_json`.
+- Record blast radius, target component, injection method, duration, and abort
+  conditions.
+- Capture recovery checks and cleanup steps in the case.
+- Classify unavailable dependencies as `env_issue` unless the target behavior
+  is LangBot's handling of that dependency failure.
+- Do not run destructive fault injection against a shared or production-like
+  instance without explicit operator approval.
+
+Recommended first fault models:
+
+- Provider timeout or HTTP 429 from a fake provider endpoint.
+- Plugin runtime disconnect/reconnect in a local instance.
+- MCP stdio server exits mid-call.
+- RAG parser fixture fails once and recovers on retry.
+- Backend API endpoint returns 5xx from a controlled local proxy.
+
+## Starter Live Probes
+
+The starter gate separates QA-harness contracts from live product checks:
+
+- `langbot-overhead-accounting-contract` verifies that reports can carry
+  overhead accounting metrics. It uses deterministic synthetic samples and is
+  not live product performance.
+- `langbot-fault-taxonomy-contract` verifies that fault scenarios declare
+  expected status, recovery, and cleanup before destructive chaos tests are
+  added.
+- `langbot-live-backend-latency` checks the unauthenticated `/healthz`
+  endpoint for basic backend responsiveness.
+- `langbot-live-control-plane-api` checks `/healthz` and
+  `/api/v1/system/info` for HTTP 200, JSON `code: 0`, response shape, and
+  per-endpoint p95 latency.
+- `langbot-live-backend-log-health` scans the recent backend log window for
+  fail-severity runtime findings. It is the reliability guard that should fail
+  the gate when HTTP probes pass but backend logs contain Traceback, ImportError,
+  ERROR, unclosed sessions, or unawaited coroutine signals.
+
+Do not treat these starter live probes as Debug Chat or model-provider
+performance. They are control-plane readiness checks; user-facing performance
+needs browser/WebSocket/message-path measurements.
+
+## Debug Chat Load And Fake Provider Baseline
+
+Use `langbot-fake-provider-debug-chat-load` before real-provider load checks.
+The setup automation starts a local OpenAI-compatible fake provider, registers
+it as a normal LangBot provider/model, configures a local-agent pipeline, resets
+Debug Chat, and then drives concurrent WebSocket messages through the live
+backend.
+
+This is not a mocked backend test. It still exercises:
+
+- provider/model persistence and runtime reload;
+- LiteLLM OpenAI-compatible requester path;
+- local-agent runner selection and pipeline execution;
+- Debug Chat WebSocket adapter and broadcast behavior;
+- backend concurrency, timeout, and error-rate accounting.
+
+The fake provider is deterministic and can inject controlled latency or faults
+with `LANGBOT_FAKE_PROVIDER_*` variables, so it is the baseline for LangBot
+message-path overhead. A fake-provider process keeps process-global config,
+request counters, and recent request history; run fake-provider probes serially
+or give each run its own provider instance. Concurrent probes against the same
+fake-provider URL can reset or reconfigure each other's metrics.
+
+The probe uses unique expected response tokens per
+request because Debug Chat broadcasts messages to every connection in the same
+session; unique tokens prevent one connection from counting another
+connection's response as its own.
+
+When the fake provider is used, reports also include provider-side timing in
+`metrics.json`:
+
+- `fake_provider.duration_ms` and `fake_provider.first_content_chunk_ms`
+  measure the controlled provider itself.
+- `provider_timing.send_to_provider_start_ms` estimates WebSocket ingress,
+  pipeline dispatch, runner setup, and requester time before the provider
+  receives the request.
+- `provider_timing.provider_finish_to_ws_final_ms` estimates the path from
+  provider completion back to the final Debug Chat WebSocket response.
+- `provider_timing.langbot_overhead_estimate_ms` is the sum of those two
+  LangBot-side segments when wall-clock timestamps can be matched by the
+  unique expected response token.
+
+After the baseline passes, run `langbot-fake-provider-debug-chat-slow-load` to
+keep the same live backend path while injecting deterministic streaming latency.
+Run `langbot-fake-provider-debug-chat-fault-recovery` to inject bounded HTTP
+provider failures and require both observed failures and later successful
+requests. The fault-recovery case is deliberately sequential because failed
+Debug Chat responses do not carry a unique success token that can be attributed
+to one concurrent connection.
+
+Run `langbot-fake-provider-debug-chat-cross-pipeline-isolation` separately via
+`langbot-debug-chat-isolation-gate`. Current LangBot releases may fail it because
+of product bug [#2286](https://github.com/langbot-app/LangBot/issues/2286), where
+Debug Chat replies can read singleton WebSocket proxy pipeline state after a
+later message overwrites it. Treat that failure as regression evidence for the
+product fix rather than as a fake-provider latency finding.
+
+Use `langbot-space-debug-chat-concurrency-smoke` after the fake-provider
+baseline. It runs a deliberately small real Space-provider batch and reports
+user-visible latency, not pure LangBot overhead. Space/model/network failures
+are dependency findings until the fake baseline shows the same symptom.
+If a Space smoke passes but log guard finds telemetry posting Tracebacks,
+classify that separately as `telemetry-proxy-noise` instead of clearing the
+proxy or treating the Debug Chat path as failed.
+
+Useful commands:
+
+```bash
+rtk bin/lbs test run langbot-fake-provider-debug-chat-load --run-id langbot-fake-load-local
+rtk bin/lbs test run langbot-fake-provider-debug-chat-slow-load --run-id langbot-fake-slow-local
+rtk bin/lbs test run langbot-fake-provider-debug-chat-fault-recovery --run-id langbot-fake-fault-local
+rtk bin/lbs suite run langbot-debug-chat-isolation-gate --run-id langbot-debug-chat-isolation-local --include-manual-check
+rtk bin/lbs test run langbot-space-debug-chat-concurrency-smoke --run-id langbot-space-smoke-local
+rtk bin/lbs suite run langbot-debug-chat-load-gate --run-id langbot-debug-chat-load-local --include-manual-check
+```
+
+## Gate Layers
+
+Use the smallest gate that answers the quality question:
+
+- `langbot-performance-contract-gate`: fast synthetic checks for report shape,
+  threshold accounting, and fault taxonomy. Good for PR feedback when no live
+  service is running.
+- `langbot-live-backend-gate`: live backend `/healthz`,
+  `/api/v1/system/info`, and backend log health. Good after starting a local
+  LangBot backend.
+- `langbot-user-path-performance-gate`: browser-visible user path performance,
+  starting with Pipeline Debug Chat send-to-visible-completion latency. Run it
+  only when the browser profile and target pipeline are ready.
+- `langbot-debug-chat-load-gate`: manual WebSocket Debug Chat load checks,
+  starting with controlled fake-provider baseline, slow-provider, and
+  fault-recovery profiles, plus an optional low-volume real Space-provider
+  smoke. Run fake-provider cases serially when they share a provider URL.
+- `langbot-debug-chat-isolation-gate`: manual cross-pipeline Debug Chat
+  isolation regression gate. Current releases may fail because of #2286; keep it
+  separate from the normal load gate until that product fix lands.
+- `langbot-performance-reliability-gate`: combined starter gate for synthetic
+  contracts plus live backend checks.
+
+Keep environment diagnostics separate from product regressions. For example, a
+SOCKS proxy without Python `socksio` support should be fixed or clearly
+classified by `bin/lbs env doctor`; do not hide the resulting backend
+Traceback in reports.
+
+## Debug Chat Performance
+
+`pipeline-debug-chat-performance` reuses the browser Debug Chat automation and
+adds `metrics.json`, `metrics_summary`, and `thresholds_summary` to
+`automation-result.json`.
+
+Current metric:
+
+```text
+response_duration_ms = prompt send -> expected assistant response visible and stable
+```
+
+This is a user-path metric, not pure LangBot overhead. If it regresses, inspect
+provider latency, model route health, plugin/runtime logs, WebSocket behavior,
+and browser console/network evidence before attributing the whole duration to
+LangBot.
+
+### User-Path Gate Runbook
+
+1. Start the backend and frontend. The frontend must be launched with
+   `VITE_API_BASE_URL="$LANGBOT_BACKEND_URL"` so browser API calls reach the
+   backend.
+2. Run `node scripts/e2e/ensure-local-agent-pipeline.mjs --write-env`. The
+   setup refreshes the local QA login, skips the wizard, prepares a Debug Chat
+   pipeline, scans Space models, tests candidates, writes tested fallback
+   models, and writes the selected pipeline/model env values to
+   `skills/.env.local`.
+3. If setup returns `env_issue`, read `model_tests` and provider errors first.
+   A missing Space key, failed Space scan, or unavailable model route is not a
+   LangBot performance regression.
+4. Run
+   `bin/lbs suite run langbot-user-path-performance-gate --include-manual-check`.
+5. Interpret `response_p95_ms` as browser-visible send-to-completion time. It
+   includes provider latency; use backend logs and model test evidence to
+   separate LangBot overhead from the external model route.
+
+The setup keeps a `max-round` value in the generated pipeline config only
+because the current backend truncator still reads that field directly. Do not
+use it as a quality requirement for future local-agent behavior.
+
+## Running The First Gate
+
+Start with the reusable suite:
+
+```bash
+rtk bin/lbs suite plan langbot-performance-reliability-gate
+rtk bin/lbs suite start langbot-performance-reliability-gate --run-id langbot-perf-rel-local
+```
+
+Run synthetic contract probes first. Run live probes only after the selected
+backend/frontend instance is reachable and the run owner accepts any fault
+scope.

skills/skills/langbot-testing/suites/langbot-debug-chat-isolation-gate.yaml

@@ -0,0 +1,13 @@
+id: langbot-debug-chat-isolation-gate
+title: "LangBot Debug Chat isolation gate"
+description: "Manual/non-required cross-pipeline Debug Chat isolation gate. Current releases may fail this gate because of product bug #2286; use it as regression evidence after the routing fix lands."
+type: reliability
+priority: p1
+tags:
+  - reliability
+  - debug-chat
+  - websocket
+  - isolation
+  - concurrency
+cases:
+  - langbot-fake-provider-debug-chat-cross-pipeline-isolation

skills/skills/langbot-testing/suites/langbot-debug-chat-load-gate.yaml

@@ -0,0 +1,15 @@
+id: langbot-debug-chat-load-gate
+title: "LangBot Debug Chat load gate"
+description: "Manual/non-required message-path load checks for Pipeline Debug Chat: controlled fake-provider baseline, slow-provider and fault-recovery profiles, plus optional real Space-provider smoke. Cross-pipeline isolation is split into langbot-debug-chat-isolation-gate because current releases may fail it due to product bug #2286."
+type: performance
+priority: p1
+tags:
+  - performance
+  - debug-chat
+  - websocket
+  - load
+cases:
+  - langbot-fake-provider-debug-chat-load
+  - langbot-fake-provider-debug-chat-slow-load
+  - langbot-fake-provider-debug-chat-fault-recovery
+  - langbot-space-debug-chat-concurrency-smoke

skills/skills/langbot-testing/suites/langbot-live-backend-gate.yaml

@@ -0,0 +1,14 @@
+id: langbot-live-backend-gate
+title: "LangBot live backend reliability gate"
+description: "Live backend control-plane responsiveness and runtime log health checks for a locally running LangBot instance."
+type: reliability
+priority: p1
+tags:
+  - performance
+  - reliability
+  - live-backend
+  - metrics
+cases:
+  - langbot-live-backend-latency
+  - langbot-live-control-plane-api
+  - langbot-live-backend-log-health

skills/skills/langbot-testing/suites/langbot-performance-contract-gate.yaml

@@ -0,0 +1,13 @@
+id: langbot-performance-contract-gate
+title: "LangBot performance contract gate"
+description: "Fast synthetic contract checks for performance metric accounting and non-destructive reliability fault taxonomy."
+type: contract
+priority: p1
+tags:
+  - performance
+  - reliability
+  - contract
+  - metrics
+cases:
+  - langbot-overhead-accounting-contract
+  - langbot-fault-taxonomy-contract

skills/skills/langbot-testing/suites/langbot-performance-reliability-gate.yaml

@@ -0,0 +1,16 @@
+id: langbot-performance-reliability-gate
+title: "LangBot performance and reliability starter gate"
+description: "Starter gate for LangBot performance accounting, live backend control-plane latency, and non-destructive fault taxonomy checks."
+type: reliability
+priority: p1
+tags:
+  - performance
+  - reliability
+  - metrics
+  - chaos
+cases:
+  - langbot-overhead-accounting-contract
+  - langbot-fault-taxonomy-contract
+  - langbot-live-backend-latency
+  - langbot-live-control-plane-api
+  - langbot-live-backend-log-health