mirror of
https://github.com/langbot-app/LangBot.git
synced 2026-07-30 16:16:07 +00:00
Compare commits
8 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 huanghuoguoguo
|
2b03095d4e | ||
|
huanghuoguoguo
|
c7d4885bfc | ||
|
huanghuoguoguo
|
4b34d4cffd | ||
|
huanghuoguoguo
|
96f5b5e365 | ||
|
huanghuoguoguo
|
73be17b02c | ||
|
huanghuoguoguo
|
e5a5188442 | ||
|
huanghuoguoguo
|
190028d5ab | ||
|
huanghuoguoguo
|
cede35b31b |
@@ -48,7 +48,6 @@ coverage.xml
|
||||
.coverage
|
||||
src/langbot/web/
|
||||
testsdk/
|
||||
.qa/
|
||||
|
||||
# Build artifacts
|
||||
/dist
|
||||
|
||||
@@ -1,105 +1,160 @@
|
||||
# AGENTS.md
|
||||
|
||||
This file guides code agents working in the LangBot main repository. `CLAUDE.md` is a symlink to this file.
|
||||
This file guides code agents (Claude Code, GitHub Copilot, OpenAI Codex, etc.) working in the LangBot project. `CLAUDE.md` is a symlink to this file.
|
||||
|
||||
Read `ARCHITECTURE.md` before non-trivial backend, frontend, runtime, plugin, Box, MCP, persistence, or cross-repo SDK changes. This file is the working checklist; `ARCHITECTURE.md` is the system map.
|
||||
## Project Overview
|
||||
|
||||
## Quick Facts
|
||||
LangBot is an open-source, LLM-native instant-messaging bot development platform. It aims to provide an out-of-the-box IM bot development experience with Agent, RAG, MCP and other LLM application capabilities, supporting mainstream global IM platforms and exposing rich APIs for custom development.
|
||||
|
||||
- Python backend: `>=3.11,<4.0`, dependencies managed by `uv`.
|
||||
- Frontend: `web/` is Vite + React Router 7 + shadcn/ui + Tailwind, managed by `pnpm`.
|
||||
- Backend framework: Quart served by Hypercorn on `api.port`, default `5300`.
|
||||
- Frontend dev server: `web/` on `3000`, with `VITE_API_BASE_URL` pointing at the backend.
|
||||
- Plugin/Box/runtime contracts live in sibling repo `langbot-plugin-sdk`, pinned as `langbot-plugin` in `pyproject.toml`.
|
||||
LangBot has a comprehensive web frontend — almost every operation can be performed through it.
|
||||
|
||||
## Essential Commands
|
||||
- **Python**: `>=3.11,<4.0`, dependencies managed by `uv`. Package version is in `pyproject.toml`.
|
||||
- **Frontend**: `web/` is a **Vite + React Router 7 + shadcn/ui + Tailwind CSS** SPA, managed by `pnpm`. (Note: this is NOT Next.js — the `dev` script is `vite`.)
|
||||
- **Backend framework**: Quart (the async flavour of Flask). The HTTP API and the pre-built web UI are both served by the backend on `http://127.0.0.1:5300`.
|
||||
|
||||
## Repository Layout
|
||||
|
||||
```
|
||||
LangBot/
|
||||
├── main.py # Entrypoint shim -> langbot.__main__.main()
|
||||
├── pyproject.toml # Python project + deps (uv), pins langbot-plugin==<x.y.z>
|
||||
├── src/langbot/
|
||||
│ ├── __main__.py # Real entrypoint, CLI args (--standalone-runtime, --standalone-box, --debug)
|
||||
│ ├── pkg/ # Core backend package
|
||||
│ │ ├── api/ # HTTP API controllers + services (Quart)
|
||||
│ │ ├── core/ # App bootstrap, stages, task manager
|
||||
│ │ ├── platform/ # IM platform adapters, bot managers, session managers
|
||||
│ │ ├── provider/ # LLM providers, requesters, tool providers
|
||||
│ │ ├── pipeline/ # Pipelines, stages, query pool
|
||||
│ │ ├── plugin/ # Bridge connecting LangBot to the plugin runtime (see below)
|
||||
│ │ ├── box/ # Code-sandbox subsystem (Docker / nsjail / E2B backends)
|
||||
│ │ ├── skill/ # Skill subsystem
|
||||
│ │ ├── rag/ , vector/ # RAG + vector store
|
||||
│ │ ├── command/ # Built-in commands
|
||||
│ │ ├── persistence/ # ORM models + Alembic migrations (SQLite & PostgreSQL)
|
||||
│ │ ├── storage/ # Object/file storage abstractions
|
||||
│ │ ├── config/, entity/, discover/, utils/, telemetry/, survey/
|
||||
│ ├── libs/ # Vendored SDKs (qq_official_api, wecom_api, etc.)
|
||||
│ └── templates/ # Config/component templates (e.g. templates/config.yaml)
|
||||
├── web/ # Frontend SPA (Vite + React Router 7 + shadcn + Tailwind)
|
||||
└── docker/ # docker-compose deployment files
|
||||
```
|
||||
|
||||
## Development Environment Setup
|
||||
|
||||
Full guide lives in the wiki: **["开发配置" / Dev Config](https://docs.langbot.app/zh/develop/dev-config)**. Summary:
|
||||
|
||||
### Backend
|
||||
|
||||
```bash
|
||||
uv sync --dev
|
||||
uv run main.py
|
||||
uv run pre-commit install
|
||||
pip install uv
|
||||
uv sync --dev # uv creates a .venv/ for you; point your editor's interpreter at it
|
||||
uv run main.py # serves API + web UI on http://127.0.0.1:5300
|
||||
```
|
||||
|
||||
On first run the config file is generated at `data/config.yaml`. DB is SQLite by default (zero setup); PostgreSQL is supported. Migrations run automatically on startup.
|
||||
|
||||
### Frontend
|
||||
|
||||
Requires Node.js + [pnpm](https://pnpm.io/installation).
|
||||
|
||||
```bash
|
||||
cd web
|
||||
cp .env.example .env # Windows: copy .env.example .env
|
||||
pnpm install
|
||||
pnpm dev
|
||||
pnpm build
|
||||
pnpm dev # http://127.0.0.1:3000 (npm install / npm run dev also work)
|
||||
```
|
||||
|
||||
Useful focused tests:
|
||||
`pnpm dev` reads `VITE_API_BASE_URL` from `web/.env` so the dev frontend can reach the backend on port `5300`. In production the frontend is pre-built into static files served by the backend on the same origin.
|
||||
|
||||
### Code formatting
|
||||
|
||||
The repo runs lint + format checks in CI. Install the pre-commit hooks so the same checks run locally before each commit:
|
||||
|
||||
```bash
|
||||
uv run pytest tests/unit_tests -q
|
||||
uv run pytest tests/integration -q
|
||||
uv run pytest tests/integration/persistence -q
|
||||
uv run pytest tests/manual/mcp_smoke.py
|
||||
|
||||
cd web
|
||||
pnpm lint
|
||||
pnpm test:e2e
|
||||
uv run pre-commit install
|
||||
```
|
||||
|
||||
Run the narrowest useful test first, then broader checks when confidence is needed.
|
||||
## Plugin System
|
||||
|
||||
## Where to Look
|
||||
LangBot's plugin system (Plugin SDK, CLI `lbp`, Plugin Runtime, and the shared entity/API definitions) lives in a **separate repository**: [`langbot-plugin-sdk`](https://github.com/langbot-app/langbot-plugin-sdk). LangBot depends on it via the pinned `langbot-plugin` package in `pyproject.toml`.
|
||||
|
||||
- Architecture map: `ARCHITECTURE.md`.
|
||||
- Dev environment guide: https://docs.langbot.app/zh/develop/dev-config.
|
||||
- Plugin runtime / CLI / SDK debugging: https://docs.langbot.app/zh/develop/plugin-runtime.
|
||||
- API-key auth: `docs/API_KEY_AUTH.md`.
|
||||
- Box deep-dive notes: `docs/review/box-architecture.md` and related files.
|
||||
- In-repo skills: `skills/` is the single source of truth for LangBot agent skills.
|
||||
- SDK repo: `../langbot-plugin-sdk/` when changing shared entities, plugin APIs, action protocol, `lbp rt`, or `lbp box`.
|
||||
### Architecture (what to know inside this repo)
|
||||
|
||||
## Cross-Repo SDK Work
|
||||
- Plugins run as independent processes managed by the **Plugin Runtime**. The Runtime supports two control transports: `stdio` and `websocket`.
|
||||
- When LangBot is started directly by a user (not in a container), it spawns and connects to the Runtime over **stdio** (lightweight/personal use).
|
||||
- When LangBot runs in a container, it connects to a standalone Runtime over **WebSocket** (production).
|
||||
- The bridge code lives in `src/langbot/pkg/plugin/` (`connector.py`, `handler.py`).
|
||||
- Relevant config (`data/config.yaml`): `plugin.runtime_ws_url` (e.g. `ws://langbot_plugin_runtime:5400/control/ws`). Start LangBot with `--standalone-runtime` to make it connect to an externally-launched Runtime over WebSocket instead of spawning one over stdio.
|
||||
|
||||
When changing SDK contracts used by LangBot:
|
||||
### Debugging the Plugin Runtime / CLI / SDK
|
||||
|
||||
This is documented in detail in the **SDK repo's `AGENTS.md`** and in the wiki page **["调试插件运行时、CLI、SDK" / Plugin Runtime](https://docs.langbot.app/zh/develop/plugin-runtime)**. The short version:
|
||||
|
||||
- Clone `LangBot` and `langbot-plugin-sdk` as siblings under one parent dir so the editor resolves shared entities.
|
||||
- Start a standalone Runtime from the SDK repo: `uv run --no-sync lbp rt` (control port `5400`, debug port `5401`).
|
||||
- To make LangBot use a locally-modified SDK: from the SDK dir, with LangBot's `.venv` active, run `uv pip install .`, then launch LangBot with `uv run --no-sync main.py --standalone-runtime` (keep `--no-sync` so your local SDK isn't overwritten).
|
||||
|
||||
### Debugging the Box (sandbox) runtime
|
||||
|
||||
The Box subsystem (`src/langbot/pkg/box/`) is the code sandbox. It picks the first available backend among **Docker / nsjail / E2B**. The standalone Box runtime is launched via the SDK CLI: `lbp box`. Backend selection details, the `lbp box` flags, and the SDK-side architecture are documented in the SDK repo's `AGENTS.md`.
|
||||
|
||||
Relevant config (`data/config.yaml`, `box:` section): `box.enabled` (master switch — disabling it also disables the native sandbox tools, skill add/edit, and stdio-mode MCP servers), `box.backend` (`'local'` = Docker/nsjail auto-pick, or `'docker'` / `'nsjail'` / `'e2b'`; also settable via `BOX__BACKEND`), and `box.runtime.endpoint` (external Box runtime base URL, e.g. `ws://127.0.0.1:5410`; empty = local auto-managed runtime). Like the plugin runtime, LangBot can connect to an externally-launched Box runtime by setting that endpoint and starting with `--standalone-box`.
|
||||
|
||||
> A common false "No supported sandbox backend (Docker / nsjail / E2B) is available" comes from Docker being installed and running but the current user not being in the `docker` group → `docker info` gets `permission denied` on the socket. Fix: `sudo usermod -aG docker <user>` and restart the backend in a shell that has the new group.
|
||||
|
||||
## Development Standards
|
||||
|
||||
- LangBot is a global project: **all code comments and docstrings must be in English**, and every user-facing string must support **i18n** (`en_US` + `zh_Hans` at minimum, plus `ja_JP` where the repo already has it).
|
||||
- LangBot is adopted in both toC and toB scenarios — always consider compatibility and security.
|
||||
- **Commit message format**: `<type>(<scope>): <subject>`
|
||||
- `type`: one of `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`, `chore`, etc.
|
||||
- `scope`: the affected package/module/file/class.
|
||||
- `subject`: concise description of the change.
|
||||
|
||||
### Database migrations (Alembic)
|
||||
|
||||
LangBot uses [Alembic](https://alembic.sqlalchemy.org/) for migrations, supporting both SQLite and PostgreSQL from a single set of scripts. Migration files live in `src/langbot/pkg/persistence/alembic/versions/`.
|
||||
|
||||
If you change ORM model definitions, generate a migration:
|
||||
|
||||
```bash
|
||||
# from langbot-plugin-sdk, with LangBot's .venv active
|
||||
uv pip install .
|
||||
|
||||
# from LangBot, preserve the locally installed SDK
|
||||
uv run --no-sync main.py
|
||||
# Run from the project root (requires data/config.yaml to exist)
|
||||
uv run python -m langbot.pkg.persistence.alembic_runner autogenerate "description of your change"
|
||||
```
|
||||
|
||||
For standalone runtime debugging:
|
||||
Review and edit the generated script before committing. Migrations execute automatically on startup. `autogenerate` detects schema changes (add/drop columns, tables, type changes) but **data migrations** (e.g. mutating JSON field contents) must be hand-written into the generated script. `env.py` sets `render_as_batch=True`, so SQLite's ALTER TABLE limits are handled automatically — no need to branch per database. More in the wiki ["开发配置"](https://docs.langbot.app/zh/develop/dev-config#数据库迁移).
|
||||
|
||||
```bash
|
||||
# in langbot-plugin-sdk
|
||||
uv run --no-sync lbp rt
|
||||
uv run --no-sync lbp box
|
||||
When writing a migration, follow these rules:
|
||||
|
||||
# in LangBot
|
||||
uv run --no-sync main.py --standalone-runtime
|
||||
uv run --no-sync main.py --standalone-box
|
||||
```
|
||||
- **Revision id ≤ 32 characters.** PostgreSQL stores `alembic_version.version_num` as `varchar(32)`; a longer id raises `StringDataRightTruncationError` at runtime. Prefer short, descriptive ids like `0005_add_llm_context_length`.
|
||||
- **Guard every operation against missing tables/columns.** Fresh installs build the schema via `create_all()` and then stamp the Alembic baseline, so a migration may run against a table that already has the change — or, in tests, against an empty database. Check `inspector.get_table_names()` / `inspector.get_columns(...)` before `add_column` / `drop_column`, mirroring the existing migrations.
|
||||
- **Keep a single linear head.** Chain `down_revision` to the current head; do not create branches. Run the migration tests after adding one: `uv run pytest tests/integration/persistence/ -q` (the PostgreSQL test needs a running PG via `TEST_POSTGRES_URL`).
|
||||
|
||||
Config keys to verify in `data/config.yaml` / `src/langbot/templates/config.yaml`:
|
||||
> **Legacy migration system (deprecated — do not extend).** The old 3.x migration system under `src/langbot/pkg/persistence/migrations/` (`DBMigration` subclasses in `dbmXXX_*.py`, run from `pkg/persistence/mgr.py`) is **frozen**. Do **not** add new `dbmXXX_*.py` files. The chain is capped at `required_database_version = 25` (`pkg/utils/constants.py`); those files only exist to upgrade pre-existing 3.x databases up to the Alembic baseline and are kept read-only. All new schema changes go through Alembic.
|
||||
|
||||
- Plugin runtime: `plugin.runtime_ws_url`, default Docker host `langbot_plugin_runtime:5400/control/ws`.
|
||||
- Box runtime: `box.enabled`, `box.backend`, `box.runtime.endpoint`, Docker host `langbot_box:5410`.
|
||||
- API/MCP auth: `api.global_api_key`.
|
||||
## Agent-Facing Surfaces (MCP + Skills)
|
||||
|
||||
## Change Rules
|
||||
LangBot is built to be **agent-friendly**. Three surfaces let AI agents work
|
||||
with LangBot, and they MUST be kept in lockstep with the HTTP API:
|
||||
|
||||
- HTTP API changes that should be agent-accessible must update the matching MCP tool in `src/langbot/pkg/api/mcp/server.py` and the relevant skill under `skills/` in the same pass.
|
||||
- New schema changes use Alembic under `src/langbot/pkg/persistence/alembic/versions/`; do not add legacy `dbmXXX` migrations.
|
||||
- New platform behavior belongs in platform adapters only for platform translation; pipeline/business logic belongs in `pkg/pipeline/` or services.
|
||||
- User-facing strings must support i18n (`en_US`, `zh_Hans`; include `ja_JP` where the repo already does).
|
||||
- Code comments and docstrings must be English.
|
||||
- Keep compatibility and security in mind; LangBot is used in both self-hosted/community and toB deployments.
|
||||
- Commit message format: `<type>(<scope>): <subject>`.
|
||||
1. **MCP server** — `src/langbot/pkg/api/mcp/` exposes a curated subset of the
|
||||
API as MCP tools at `/mcp` (API-key authenticated, including the
|
||||
`api.global_api_key` from config.yaml). `server.py` defines the tools (they
|
||||
call the service layer directly); `mount.py` is the ASGI dispatcher.
|
||||
2. **In-repo skills** — `skills/` is the **single source of truth** for agent
|
||||
skills (plugin/core/deploy/e2e/MCP-ops). Docs and the landing page link here
|
||||
rather than embedding their own copies.
|
||||
3. **API-key auth** — `api.global_api_key` (config.yaml) authenticates the API
|
||||
and MCP without a login session; see `docs/API_KEY_AUTH.md`.
|
||||
|
||||
## Runtime Pitfalls
|
||||
> **Maintenance rule (important).** When you add, remove, or change an HTTP API
|
||||
> endpoint that should be agent-accessible, you MUST update **both** the matching
|
||||
> MCP tool in `src/langbot/pkg/api/mcp/server.py` **and** the relevant skill under
|
||||
> `skills/` (especially `skills/skills/langbot-mcp-ops`). The API, the MCP tool
|
||||
> surface, and the skills are one system — drift between them is a bug.
|
||||
|
||||
- Local stdio Plugin Runtime disconnects do not auto-reconnect; restart LangBot if that path breaks.
|
||||
- Orphan runtime processes on `5400`/`5401` commonly break plugin debugging.
|
||||
- Use `uv run --no-sync` after locally installing the SDK, or `uv` may restore the pinned package.
|
||||
- A false Box “no backend” often means Docker is running but the current user lacks Docker socket permission.
|
||||
- Do not confuse external MCP servers LangBot connects to (`pkg/provider/tools/loaders/mcp.py`) with LangBot's own `/mcp` server (`pkg/api/mcp/`).
|
||||
- `CLAUDE.md` is a symlink to this file; edit `AGENTS.md`, not the symlink.
|
||||
|
||||
## Principles
|
||||
## Some Principles
|
||||
|
||||
- Keep it simple, stupid.
|
||||
- Entities should not be multiplied unnecessarily.
|
||||
|
||||
-250
@@ -1,250 +0,0 @@
|
||||
# Architecture
|
||||
|
||||
This document is a map of LangBot's moving parts. It is intentionally more stable than a feature guide and more concrete than the README: when you need to change behavior, start here, then follow the file references into the code.
|
||||
|
||||
For agent-specific working rules, see `AGENTS.md`. For plugin-runtime and Box-runtime implementation details, also read the sibling SDK repo: [`langbot-plugin-sdk`](https://github.com/langbot-app/langbot-plugin-sdk).
|
||||
|
||||
## What LangBot Is
|
||||
|
||||
LangBot is an open-source platform for building production IM bots backed by LLMs, agents, RAG, plugins, MCP tools, and a web management panel.
|
||||
|
||||
At runtime, one LangBot process owns:
|
||||
|
||||
- a Quart/Hypercorn HTTP service and the built web UI on `:5300`;
|
||||
- messaging-platform adapters such as Discord, Telegram, Slack, WeChat, QQ, WeCom, Lark, DingTalk, KOOK, LINE, Satori, Matrix, and HTTP/WebSocket bots;
|
||||
- a pipeline engine that turns inbound platform messages into LLM/tool/plugin work and replies;
|
||||
- persistence, storage, vector database, telemetry, monitoring, and configuration managers;
|
||||
- bridges to the Plugin Runtime and Box Runtime provided by `langbot-plugin-sdk`;
|
||||
- an MCP server at `/mcp` exposing a curated agent-facing subset of the service layer.
|
||||
|
||||
## Repository Boundary
|
||||
|
||||
LangBot is not a single-repo system.
|
||||
|
||||
- `LangBot/` is the main product: backend, web UI, platform adapters, pipeline engine, HTTP API, MCP server, RAG, persistence, skills integration, and the bridge code that talks to runtimes.
|
||||
- `langbot-plugin-sdk/` is published as `langbot-plugin` and pinned in `LangBot/pyproject.toml`. It contains plugin developer APIs, shared entities, `lbp`, the Plugin Runtime (`lbp rt`), and the Box Runtime (`lbp box`).
|
||||
- Plugins import SDK APIs from `langbot_plugin.*`; the LangBot main process imports the same package for shared entities and runtime protocols.
|
||||
|
||||
This split matters. If a change modifies SDK entities, component APIs, action protocols, `lbp rt`, or `lbp box`, verify the sibling SDK repo and install the local SDK into LangBot's virtualenv when testing cross-repo behavior.
|
||||
|
||||
## Startup Path
|
||||
|
||||
The process entrypoint is small and layered:
|
||||
|
||||
1. `main.py` delegates to `langbot.__main__.main()`.
|
||||
2. `src/langbot/__main__.py` parses `--standalone-runtime`, `--standalone-box`, and `--debug`, checks dependencies, generates missing config/data files, and calls `pkg.core.boot.main()`.
|
||||
3. `pkg/core/boot.py` executes startup stages in order: `LoadConfigStage`, `GenKeysStage`, `SetupLoggerStage`, `BuildAppStage`, `ShowNotesStage`.
|
||||
4. `BuildAppStage` constructs the `Application` object by wiring managers, services, runtime connectors, and controllers.
|
||||
5. `Application.run()` starts the platform manager, query controller, HTTP controller, telemetry/cleanup loops, and plugin initialization.
|
||||
|
||||
The central runtime object is `pkg/core/app.py::Application`. It is a service locator for long-lived managers. That is not elegant, but it is the current architectural center; most subsystems receive `ap: Application` and collaborate through it.
|
||||
|
||||
## Top-Level Layout
|
||||
|
||||
```text
|
||||
LangBot/
|
||||
├── main.py # Entrypoint shim
|
||||
├── pyproject.toml # Python package, deps, pinned langbot-plugin
|
||||
├── src/langbot/
|
||||
│ ├── __main__.py # CLI entrypoint and boot handoff
|
||||
│ ├── pkg/
|
||||
│ │ ├── core/ # Application, boot stages, task manager
|
||||
│ │ ├── api/ # HTTP API + MCP server mount
|
||||
│ │ ├── platform/ # IM adapters and runtime bot manager
|
||||
│ │ ├── pipeline/ # Message routing and pipeline stages
|
||||
│ │ ├── provider/ # LLM runners, model manager, tools
|
||||
│ │ ├── plugin/ # LangBot-side Plugin Runtime connector/handler
|
||||
│ │ ├── box/ # LangBot-side Box service/connector
|
||||
│ │ ├── skill/ # Skill metadata/activation integration
|
||||
│ │ ├── rag/ , vector/ # Knowledge-base and vector DB integration
|
||||
│ │ ├── persistence/ # SQLAlchemy/SQLModel, Alembic, legacy migrations
|
||||
│ │ ├── storage/ # Local/S3 file storage abstraction
|
||||
│ │ └── config/, entity/, utils/, telemetry/, survey/
|
||||
│ ├── libs/ # Vendored third-party platform SDKs
|
||||
│ └── templates/ # Default config and component metadata
|
||||
├── web/ # Vite + React Router + shadcn/ui + Tailwind SPA
|
||||
├── docker/ # Deployment manifests
|
||||
├── skills/ # In-repo agent skills, single source of truth
|
||||
└── tests/ # Unit/integration/e2e/manual tests
|
||||
```
|
||||
|
||||
## The Runtime Graph
|
||||
|
||||
The most useful mental model is this graph:
|
||||
|
||||
```text
|
||||
Platform adapter
|
||||
→ RuntimeBot
|
||||
→ MessageAggregator
|
||||
→ QueryPool
|
||||
→ Controller
|
||||
→ RuntimePipeline
|
||||
→ PipelineStage chain
|
||||
→ RequestRunner / ToolManager / PluginRuntimeConnector / BoxService
|
||||
→ response via adapter
|
||||
```
|
||||
|
||||
The HTTP and MCP surfaces are parallel entrypoints into the same service layer:
|
||||
|
||||
```text
|
||||
HTTP client / Web UI
|
||||
→ Quart route group
|
||||
→ api/http/service/*
|
||||
→ Application managers / persistence / runtime connectors
|
||||
|
||||
MCP client
|
||||
→ /mcp mount
|
||||
→ api/mcp/server.py tools
|
||||
→ the same service layer directly
|
||||
```
|
||||
|
||||
## Message Flow
|
||||
|
||||
Inbound platform messages enter through adapter-specific SDK callbacks. The common path is:
|
||||
|
||||
1. A platform adapter under `pkg/platform/sources/` converts platform-specific events into SDK message/event entities.
|
||||
2. `RuntimeBot` in `pkg/platform/botmgr.py` applies pipeline routing rules and either discards the message, pushes it to webhooks, or sends it to the message aggregator.
|
||||
3. `MessageAggregator` batches/normalizes messages before adding a `Query` to `QueryPool`.
|
||||
4. `Controller` in `pkg/pipeline/controller.py` selects queries subject to global pipeline concurrency and per-session concurrency.
|
||||
5. `RuntimePipeline` in `pkg/pipeline/pipelinemgr.py` runs configured pipeline stages using a responsibility-chain style executor that supports generator stages.
|
||||
6. The chat stage emits plugin events, calls a configured `RequestRunner`, handles streaming/non-streaming responses, records telemetry, and appends conversation history.
|
||||
7. Output stages send text, cards, chunks, files, or error notices back through the original platform adapter.
|
||||
|
||||
Pipeline components are registered by decorators and package import side effects. When adding a new stage, loader, runner, or adapter, check the corresponding preregistration mechanism instead of inventing a second registry.
|
||||
|
||||
## Platform Layer
|
||||
|
||||
Platform code lives under `pkg/platform/`.
|
||||
|
||||
- `botmgr.py` owns runtime bots, routing rules, event logging, webhook pushing, and adapter lifecycle.
|
||||
- `sources/` contains adapter implementations. Each adapter subclasses `langbot_plugin.api.definition.abstract.platform.adapter.AbstractMessagePlatformAdapter` from the SDK.
|
||||
- Platform entities such as `MessageChain`, `Image`, `At`, `Voice`, and events come from `langbot-plugin-sdk`, not from this repo.
|
||||
|
||||
The platform layer should translate between external platform APIs and LangBot's shared message/event model. It should not contain LLM-provider logic or pipeline business logic.
|
||||
|
||||
## Pipeline Layer
|
||||
|
||||
Pipeline code lives under `pkg/pipeline/`.
|
||||
|
||||
Important pieces:
|
||||
|
||||
- `pool.py::QueryPool` stores pending queries and cached in-flight queries for plugin backward-compatible calls.
|
||||
- `controller.py::Controller` schedules query processing and enforces concurrency.
|
||||
- `pipelinemgr.py::RuntimePipeline` materializes database pipeline config into a runtime stage chain.
|
||||
- `process/handlers/chat.py::ChatMessageHandler` is the main LLM conversation handler.
|
||||
- Stage families include response rules, banned sessions, content filters, preprocessors, rate limits, message truncation, long text handling, response-back, command handling, and wrappers.
|
||||
|
||||
Pipelines are configuration-driven. Prefer adding a stage or extending an existing stage family over hard-coding behavior in platform adapters.
|
||||
|
||||
## Provider, RAG, and Tools
|
||||
|
||||
Provider code lives under `pkg/provider/`.
|
||||
|
||||
- `modelmgr/` manages configured model providers and requesters.
|
||||
- `runners/` implements request runners such as the local agent runner and external workflow integrations.
|
||||
- `tools/toolmgr.py` aggregates tools from native tools, plugin tools, external MCP servers, and skill-authoring tools.
|
||||
- `tools/loaders/mcp.py` is the MCP client side: external MCP servers that LangBot connects to for agent tools.
|
||||
- RAG lives across `pkg/rag/`, `pkg/vector/`, model services, and plugin KnowledgeEngine actions.
|
||||
|
||||
Do not confuse LangBot's MCP client side with LangBot's own MCP server at `/mcp`; they are different surfaces.
|
||||
|
||||
## Plugin System
|
||||
|
||||
The plugin system crosses the repo boundary.
|
||||
|
||||
In this repo:
|
||||
|
||||
- `pkg/plugin/connector.py` connects LangBot to the Plugin Runtime over stdio or WebSocket.
|
||||
- `pkg/plugin/handler.py` exposes LangBot actions to the runtime and calls runtime actions for plugin operations.
|
||||
- `pkg/provider/tools/loaders/plugin.py` exposes plugin Tool components to LLM runners.
|
||||
- Pipeline handlers emit SDK events such as normal-message events and prompt-processing events.
|
||||
|
||||
In `langbot-plugin-sdk`:
|
||||
|
||||
- `src/langbot_plugin/api/` defines `BasePlugin`, component base classes, message/event entities, contexts, proxies, and manifests.
|
||||
- `src/langbot_plugin/runtime/` implements `lbp rt`, plugin discovery, dependency installation, process launching, and control/debug connections.
|
||||
- `src/langbot_plugin/entities/io/` defines the action protocol shared by LangBot, runtime, and plugin processes.
|
||||
|
||||
The Plugin Runtime supports stdio and WebSocket control transports. Direct local LangBot runs usually spawn the runtime over stdio. Containerized/standalone deployments connect over WebSocket using `plugin.runtime_ws_url` and `--standalone-runtime`.
|
||||
|
||||
## Box Runtime and Skills
|
||||
|
||||
Box is the sandbox subsystem used by native agent tools, stdio MCP servers, skill authoring, and managed processes.
|
||||
|
||||
In this repo:
|
||||
|
||||
- `pkg/box/service.py` is the application-facing facade for exec, sessions, managed processes, skill CRUD, status, reconnects, quotas, mounts, and sandbox profiles.
|
||||
- `pkg/box/connector.py` connects to the Box Runtime over stdio, Windows subprocess+WebSocket, or remote WebSocket.
|
||||
- `pkg/provider/tools/loaders/native.py`, `mcp_stdio.py`, and skill loaders depend on Box availability.
|
||||
- `pkg/skill/manager.py` loads skills from the Box runtime, falling back to local `data/skills` when needed.
|
||||
|
||||
In `langbot-plugin-sdk`:
|
||||
|
||||
- `src/langbot_plugin/box/server.py` implements `lbp box` and the WebSocket endpoints on `:5410`.
|
||||
- `src/langbot_plugin/box/runtime.py` owns sandbox sessions and managed processes.
|
||||
- `backend.py`, `nsjail_backend.py`, and `e2b_backend.py` implement sandbox backends.
|
||||
- `skill_store.py` manages skill packages from the Box side.
|
||||
|
||||
Important config keys live under `box:` in `src/langbot/templates/config.yaml`: `box.enabled`, `box.backend`, `box.runtime.endpoint`, and `box.local.*`. Start LangBot with `--standalone-box` when connecting to an externally launched Box runtime.
|
||||
|
||||
## HTTP API, Web UI, and MCP Server
|
||||
|
||||
`pkg/api/http/controller/main.py` builds a Quart app, registers route groups, serves the built SPA, and wraps the ASGI app with the MCP dispatcher.
|
||||
|
||||
- HTTP route groups live under `pkg/api/http/controller/groups/`.
|
||||
- Service-layer logic lives under `pkg/api/http/service/`.
|
||||
- The built web UI is served from the frontend build path with SPA fallback.
|
||||
- The MCP server lives under `pkg/api/mcp/` and is mounted at `/mcp`.
|
||||
|
||||
The MCP server intentionally exposes a curated subset of the API. Tools call service classes directly rather than making HTTP requests back into LangBot.
|
||||
|
||||
Maintenance rule: when adding, removing, or changing an HTTP endpoint that should be agent-accessible, update the matching MCP tool and the relevant in-repo skill under `skills/` in the same pass.
|
||||
|
||||
## Persistence and Configuration
|
||||
|
||||
Persistence is centered on `pkg/persistence/mgr.py`.
|
||||
|
||||
- SQLite is the default database; PostgreSQL is supported.
|
||||
- Models live under `pkg/entity/persistence/`.
|
||||
- Fresh schemas are created from metadata, then legacy migrations run up to the frozen 3.x baseline, then Alembic migrations run to head.
|
||||
- New schema changes should use Alembic under `pkg/persistence/alembic/versions/`; do not extend the frozen legacy migration chain.
|
||||
|
||||
Configuration starts from `src/langbot/templates/config.yaml` and is generated into `data/config.yaml` on first run. Most long-lived managers read from `ap.instance_config.data`.
|
||||
|
||||
## Frontend
|
||||
|
||||
The frontend lives in `web/` and is a Vite SPA using React Router 7, shadcn/ui, Tailwind CSS, and pnpm. It is not Next.js, despite some historical filenames.
|
||||
|
||||
In development, `pnpm dev` serves the UI on `:3000` and reads `VITE_API_BASE_URL` to call the backend on `:5300`. In production, the built frontend is packaged into the Python distribution and served by the backend.
|
||||
|
||||
Keep frontend API behavior aligned with `pkg/api/http/service/` and route groups. User-facing strings must go through the existing i18n setup.
|
||||
|
||||
## Agent-Facing Surfaces
|
||||
|
||||
LangBot is deliberately agent-friendly. The agent-facing surfaces are part of the architecture, not extra docs.
|
||||
|
||||
- `skills/` is the single source of truth for in-repo skills.
|
||||
- `pkg/api/mcp/server.py` exposes the LangBot MCP server at `/mcp`.
|
||||
- `api.global_api_key` authenticates API/MCP access without a browser login.
|
||||
- `AGENTS.md` and `ARCHITECTURE.md` tell coding agents how the repo works.
|
||||
|
||||
When one of these changes, update the others if the behavior or contract changed. API, MCP tools, and skills are one system; drift is a bug.
|
||||
|
||||
## Where to Change Things
|
||||
|
||||
- New HTTP API: add/adjust a service in `pkg/api/http/service/`, a route group in `pkg/api/http/controller/groups/`, tests, and MCP/skills if agent-accessible.
|
||||
- New platform adapter: add a `pkg/platform/sources/*` adapter, component metadata/templates as needed, i18n, docs, and tests/smoke coverage.
|
||||
- New pipeline behavior: add or extend a pipeline stage family under `pkg/pipeline/`; avoid putting pipeline rules in adapters.
|
||||
- New LLM provider/requester: work under `pkg/provider/modelmgr/` and related service/UI surfaces.
|
||||
- New LLM tool source: extend `pkg/provider/tools/loaders/` and `ToolManager` intentionally.
|
||||
- New plugin component/API/protocol: change `langbot-plugin-sdk` first or in lockstep, then update LangBot bridge code.
|
||||
- New Box capability: change both `pkg/box/` and `langbot-plugin-sdk/src/langbot_plugin/box/`, plus config and tests.
|
||||
- New database schema: add an Alembic migration, not a legacy `dbmXXX` migration.
|
||||
|
||||
## Design Biases
|
||||
|
||||
- Keep platform translation, pipeline orchestration, provider execution, and runtime protocols separate.
|
||||
- Reuse existing registries and service layers instead of adding parallel paths.
|
||||
- Prefer small, explicit agent surfaces over exposing every internal API.
|
||||
- Treat cross-repo contracts with the SDK as public interfaces.
|
||||
- Test behavior at the narrowest useful layer first, then add integration/e2e coverage for runtime or platform changes.
|
||||
@@ -52,15 +52,6 @@ RUN apt-get update \
|
||||
&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian $(. /etc/os-release && echo \"$VERSION_CODENAME\") stable" > /etc/apt/sources.list.d/docker.list \
|
||||
&& apt-get update \
|
||||
&& apt-get install -y --no-install-recommends docker-ce-cli \
|
||||
# Install Node.js LTS so the sandbox (nsjail/Docker box) can run npx-based
|
||||
# stdio MCP servers. node/npx land in /usr/bin, which is on the nsjail
|
||||
# read-only mount whitelist (_READONLY_SYSTEM_MOUNTS), so they are bound
|
||||
# into the sandbox chroot automatically. Without node, any npx-launched
|
||||
# MCP server exits with return_code=127 (command not found).
|
||||
&& curl -fsSL https://deb.nodesource.com/setup_22.x -o /tmp/nodesource_setup.sh \
|
||||
&& bash /tmp/nodesource_setup.sh \
|
||||
&& apt-get install -y --no-install-recommends nodejs \
|
||||
&& rm -f /tmp/nodesource_setup.sh \
|
||||
&& python -m pip install --no-cache-dir uv \
|
||||
&& uv sync \
|
||||
&& apt-get purge -y --auto-remove curl gnupg \
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
|
||||
<div align="center">
|
||||
|
||||
<a href="https://www.producthunt.com/products/langbot/launches/langbot?embed=true&utm_source=badge-featured&utm_medium=badge&utm_campaign=badge-langbot" target="_blank" rel="noopener noreferrer"><img alt="LangBot - Easy-to-use global IM bot platform designed for the LLM era | Product Hunt" width="250" height="54" src="https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=979554&theme=light&t=1782822143403"></a>
|
||||
<a href="https://www.producthunt.com/products/langbot?utm_source=badge-follow&utm_medium=badge&utm_source=badge-langbot" target="_blank"><img src="https://api.producthunt.com/widgets/embed-image/v1/follow.svg?product_id=1077185&theme=light" alt="LangBot - Production-grade IM bot made easy. | Product Hunt" style="width: 250px; height: 54px;" width="250" height="54" /></a>
|
||||
|
||||
<h3>Production-grade platform for building agentic IM bots.</h3>
|
||||
<h4>Quickly build, debug, and ship AI bots to Slack, Discord, Telegram, WeChat, and more.</h4>
|
||||
@@ -55,12 +55,6 @@ LangBot is an **open-source, production-grade platform** for building AI-powered
|
||||
|
||||
---
|
||||
|
||||
## 😎 Stay Updated
|
||||
|
||||
Click the Star and Watch buttons in the top-right corner of the repository to get the latest updates.
|
||||
|
||||
|
||||
|
||||
## Quick Start
|
||||
|
||||
### ☁️ LangBot Cloud (Recommended)
|
||||
@@ -80,7 +74,7 @@ uvx langbot
|
||||
```bash
|
||||
git clone https://github.com/langbot-app/LangBot
|
||||
cd LangBot/docker
|
||||
docker compose --profile all up -d
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### One-Click Cloud Deploy
|
||||
@@ -136,7 +130,7 @@ docker compose --profile all up -d
|
||||
| [PPIO](https://ppinfra.com/user/register?invited_by=QJKFYD&utm_source=github_langbot) | GPU Platform | ✅ |
|
||||
| [ShengSuanYun](https://www.shengsuanyun.com/?from=CH_KYIPP758) | GPU Platform | ✅ |
|
||||
| [接口 AI](https://jiekou.ai/) | Gateway | ✅ |
|
||||
| [302.AI](https://share.302ai.cn/SuTG99) | Gateway | ✅ |
|
||||
| [302.AI](https://share.302.ai/SuTG99) | Gateway | ✅ |
|
||||
| [Qiniu](https://www.qiniu.com/ai/agent) | Gateway | ✅ |
|
||||
|
||||
[→ View all integrations](https://link.langbot.app/en/docs/features)
|
||||
|
||||
+2
-8
@@ -55,12 +55,6 @@ LangBot 是一个**开源的生产级平台**，用于构建 AI 驱动的即时
|
||||
|
||||
---
|
||||
|
||||
## 😎 保持更新
|
||||
|
||||
点击[仓库首页](https://github.com/langbot-app/LangBot)右上角 Star 和 Watch 按钮，获取最新动态。
|
||||
|
||||
|
||||
|
||||
## 快速开始
|
||||
|
||||
### ☁️ LangBot Cloud（推荐）
|
||||
@@ -80,7 +74,7 @@ uvx langbot
|
||||
```bash
|
||||
git clone https://github.com/langbot-app/LangBot
|
||||
cd LangBot/docker
|
||||
docker compose --profile all up -d
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### 一键云部署
|
||||
@@ -136,7 +130,7 @@ docker compose --profile all up -d
|
||||
| [优云智算](https://www.compshare.cn/?ytag=GPU_YY-gh_langbot) | GPU 平台 | ✅ |
|
||||
| [PPIO](https://ppinfra.com/user/register?invited_by=QJKFYD&utm_source=github_langbot) | GPU 平台 | ✅ |
|
||||
| [接口 AI](https://jiekou.ai/) | 聚合平台 | ✅ |
|
||||
| [302.AI](https://share.302ai.cn/SuTG99) | 聚合平台 | ✅ |
|
||||
| [302.AI](https://share.302.ai/SuTG99) | 聚合平台 | ✅ |
|
||||
| [小马算力](https://www.tokenpony.cn/453z1) | 聚合平台 | ✅ |
|
||||
| [百宝箱Tbox](https://www.tbox.cn/open) | 智能体平台 | ✅ |
|
||||
| [七牛云Qiniu](https://www.qiniu.com/ai/agent) | 聚合平台 | ✅ |
|
||||
|
||||
+3
-9
@@ -5,7 +5,7 @@
|
||||
|
||||
<div align="center">
|
||||
|
||||
<a href="https://www.producthunt.com/products/langbot/launches/langbot?embed=true&utm_source=badge-featured&utm_medium=badge&utm_campaign=badge-langbot" target="_blank" rel="noopener noreferrer"><img alt="LangBot - Easy-to-use global IM bot platform designed for the LLM era | Product Hunt" width="250" height="54" src="https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=979554&theme=light&t=1782822143403"></a>
|
||||
<a href="https://www.producthunt.com/products/langbot?utm_source=badge-follow&utm_medium=badge&utm_source=badge-langbot" target="_blank"><img src="https://api.producthunt.com/widgets/embed-image/v1/follow.svg?product_id=1077185&theme=light" alt="LangBot - Production-grade IM bot made easy. | Product Hunt" style="width: 250px; height: 54px;" width="250" height="54" /></a>
|
||||
|
||||
<h3>Plataforma de grado de producción para construir bots de mensajería instantánea con agentes de IA.</h3>
|
||||
<h4>Construya, depure y despliegue bots de IA rápidamente en Slack, Discord, Telegram, WeChat y más.</h4>
|
||||
@@ -54,12 +54,6 @@ LangBot es una **plataforma de código abierto y grado de producción** para con
|
||||
|
||||
---
|
||||
|
||||
## 😎 Manténgase Actualizado
|
||||
|
||||
Haga clic en los botones Star y Watch en la esquina superior derecha del repositorio para obtener las últimas actualizaciones.
|
||||
|
||||
|
||||
|
||||
## Inicio Rápido
|
||||
|
||||
### ☁️ LangBot Cloud (Recomendado)
|
||||
@@ -79,7 +73,7 @@ uvx langbot
|
||||
```bash
|
||||
git clone https://github.com/langbot-app/LangBot
|
||||
cd LangBot/docker
|
||||
docker compose --profile all up -d
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### Despliegue en la Nube con un Clic
|
||||
@@ -135,7 +129,7 @@ docker compose --profile all up -d
|
||||
| [PPIO](https://ppinfra.com/user/register?invited_by=QJKFYD&utm_source=github_langbot) | Plataforma GPU | ✅ |
|
||||
| [ShengSuanYun](https://www.shengsuanyun.com/?from=CH_KYIPP758) | Plataforma GPU | ✅ |
|
||||
| [接口 AI](https://jiekou.ai/) | Pasarela | ✅ |
|
||||
| [302.AI](https://share.302ai.cn/SuTG99) | Pasarela | ✅ |
|
||||
| [302.AI](https://share.302.ai/SuTG99) | Pasarela | ✅ |
|
||||
| [Qiniu](https://www.qiniu.com/ai/agent) | Pasarela | ✅ |
|
||||
|
||||
[→ Ver todas las integraciones](https://link.langbot.app/en/docs/features)
|
||||
|
||||
+3
-9
@@ -5,7 +5,7 @@
|
||||
|
||||
<div align="center">
|
||||
|
||||
<a href="https://www.producthunt.com/products/langbot/launches/langbot?embed=true&utm_source=badge-featured&utm_medium=badge&utm_campaign=badge-langbot" target="_blank" rel="noopener noreferrer"><img alt="LangBot - Easy-to-use global IM bot platform designed for the LLM era | Product Hunt" width="250" height="54" src="https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=979554&theme=light&t=1782822143403"></a>
|
||||
<a href="https://www.producthunt.com/products/langbot?utm_source=badge-follow&utm_medium=badge&utm_source=badge-langbot" target="_blank"><img src="https://api.producthunt.com/widgets/embed-image/v1/follow.svg?product_id=1077185&theme=light" alt="LangBot - Production-grade IM bot made easy. | Product Hunt" style="width: 250px; height: 54px;" width="250" height="54" /></a>
|
||||
|
||||
<h3>Plateforme de niveau production pour construire des bots de messagerie instantanée avec agents IA.</h3>
|
||||
<h4>Créez, déboguez et déployez rapidement des bots IA sur Slack, Discord, Telegram, WeChat et plus.</h4>
|
||||
@@ -54,12 +54,6 @@ LangBot est une **plateforme open-source de niveau production** pour créer des
|
||||
|
||||
---
|
||||
|
||||
## 😎 Restez à Jour
|
||||
|
||||
Cliquez sur les boutons Star et Watch dans le coin supérieur droit du dépôt pour obtenir les dernières mises à jour.
|
||||
|
||||
|
||||
|
||||
## Démarrage Rapide
|
||||
|
||||
### ☁️ LangBot Cloud (Recommandé)
|
||||
@@ -79,7 +73,7 @@ uvx langbot
|
||||
```bash
|
||||
git clone https://github.com/langbot-app/LangBot
|
||||
cd LangBot/docker
|
||||
docker compose --profile all up -d
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### Déploiement Cloud en un Clic
|
||||
@@ -132,7 +126,7 @@ docker compose --profile all up -d
|
||||
| [ModelScope](https://modelscope.cn/docs/model-service/API-Inference/intro) | Passerelle | ✅ |
|
||||
| [GiteeAI](https://ai.gitee.com/) | Passerelle | ✅ |
|
||||
| [接口 AI](https://jiekou.ai/) | Passerelle | ✅ |
|
||||
| [302.AI](https://share.302ai.cn/SuTG99) | Passerelle | ✅ |
|
||||
| [302.AI](https://share.302.ai/SuTG99) | Passerelle | ✅ |
|
||||
| [CompShare](https://www.compshare.cn/?ytag=GPU_YY-gh_langbot) | Plateforme GPU | ✅ |
|
||||
| [PPIO](https://ppinfra.com/user/register?invited_by=QJKFYD&utm_source=github_langbot) | Plateforme GPU | ✅ |
|
||||
| [ShengSuanYun](https://www.shengsuanyun.com/?from=CH_KYIPP758) | Plateforme GPU | ✅ |
|
||||
|
||||
+3
-9
@@ -5,7 +5,7 @@
|
||||
|
||||
<div align="center">
|
||||
|
||||
<a href="https://www.producthunt.com/products/langbot/launches/langbot?embed=true&utm_source=badge-featured&utm_medium=badge&utm_campaign=badge-langbot" target="_blank" rel="noopener noreferrer"><img alt="LangBot - Easy-to-use global IM bot platform designed for the LLM era | Product Hunt" width="250" height="54" src="https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=979554&theme=light&t=1782822143403"></a>
|
||||
<a href="https://www.producthunt.com/products/langbot?utm_source=badge-follow&utm_medium=badge&utm_source=badge-langbot" target="_blank"><img src="https://api.producthunt.com/widgets/embed-image/v1/follow.svg?product_id=1077185&theme=light" alt="LangBot - Production-grade IM bot made easy. | Product Hunt" style="width: 250px; height: 54px;" width="250" height="54" /></a>
|
||||
|
||||
<h3>AIエージェント搭載IMボットを構築するための本番グレードプラットフォーム。</h3>
|
||||
<h4>Slack、Discord、Telegram、WeChat などに AI ボットを素早く構築、デバッグ、デプロイ。</h4>
|
||||
@@ -54,12 +54,6 @@ LangBot は、AI搭載のインスタントメッセージングボットを構
|
||||
|
||||
---
|
||||
|
||||
## 😎 最新情報を入手
|
||||
|
||||
リポジトリの右上にある Star と Watch ボタンをクリックして、最新の更新を取得してください。
|
||||
|
||||
|
||||
|
||||
## クイックスタート
|
||||
|
||||
### ☁️ LangBot Cloud（推奨）
|
||||
@@ -79,7 +73,7 @@ uvx langbot
|
||||
```bash
|
||||
git clone https://github.com/langbot-app/LangBot
|
||||
cd LangBot/docker
|
||||
docker compose --profile all up -d
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### ワンクリッククラウドデプロイ
|
||||
@@ -135,7 +129,7 @@ docker compose --profile all up -d
|
||||
| [PPIO](https://ppinfra.com/user/register?invited_by=QJKFYD&utm_source=github_langbot) | GPUプラットフォーム | ✅ |
|
||||
| [ShengSuanYun](https://www.shengsuanyun.com/?from=CH_KYIPP758) | GPUプラットフォーム | ✅ |
|
||||
| [接口 AI](https://jiekou.ai/) | ゲートウェイ | ✅ |
|
||||
| [302.AI](https://share.302ai.cn/SuTG99) | ゲートウェイ | ✅ |
|
||||
| [302.AI](https://share.302.ai/SuTG99) | ゲートウェイ | ✅ |
|
||||
| [Qiniu](https://www.qiniu.com/ai/agent) | ゲートウェイ | ✅ |
|
||||
|
||||
[→ すべての統合を表示](https://link.langbot.app/en/docs/features)
|
||||
|
||||
+3
-9
@@ -5,7 +5,7 @@
|
||||
|
||||
<div align="center">
|
||||
|
||||
<a href="https://www.producthunt.com/products/langbot/launches/langbot?embed=true&utm_source=badge-featured&utm_medium=badge&utm_campaign=badge-langbot" target="_blank" rel="noopener noreferrer"><img alt="LangBot - Easy-to-use global IM bot platform designed for the LLM era | Product Hunt" width="250" height="54" src="https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=979554&theme=light&t=1782822143403"></a>
|
||||
<a href="https://www.producthunt.com/products/langbot?utm_source=badge-follow&utm_medium=badge&utm_source=badge-langbot" target="_blank"><img src="https://api.producthunt.com/widgets/embed-image/v1/follow.svg?product_id=1077185&theme=light" alt="LangBot - Production-grade IM bot made easy. | Product Hunt" style="width: 250px; height: 54px;" width="250" height="54" /></a>
|
||||
|
||||
<h3>AI 에이전트 IM 봇 구축을 위한 프로덕션 등급 플랫폼.</h3>
|
||||
<h4>Slack, Discord, Telegram, WeChat 등에 AI 봇을 빠르게 구축, 디버그 및 배포.</h4>
|
||||
@@ -54,12 +54,6 @@ LangBot은 AI 기반 인스턴트 메시징 봇을 구축하기 위한 **오픈
|
||||
|
||||
---
|
||||
|
||||
## 😎 최신 정보 받기
|
||||
|
||||
리포지토리 오른쪽 상단의 Star 및 Watch 버튼을 클릭하여 최신 업데이트를 받으세요.
|
||||
|
||||
|
||||
|
||||
## 빠른 시작
|
||||
|
||||
### ☁️ LangBot Cloud (추천)
|
||||
@@ -79,7 +73,7 @@ uvx langbot
|
||||
```bash
|
||||
git clone https://github.com/langbot-app/LangBot
|
||||
cd LangBot/docker
|
||||
docker compose --profile all up -d
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### 원클릭 클라우드 배포
|
||||
@@ -135,7 +129,7 @@ docker compose --profile all up -d
|
||||
| [PPIO](https://ppinfra.com/user/register?invited_by=QJKFYD&utm_source=github_langbot) | GPU 플랫폼 | ✅ |
|
||||
| [ShengSuanYun](https://www.shengsuanyun.com/?from=CH_KYIPP758) | GPU 플랫폼 | ✅ |
|
||||
| [接口 AI](https://jiekou.ai/) | 게이트웨이 | ✅ |
|
||||
| [302.AI](https://share.302ai.cn/SuTG99) | 게이트웨이 | ✅ |
|
||||
| [302.AI](https://share.302.ai/SuTG99) | 게이트웨이 | ✅ |
|
||||
| [Qiniu](https://www.qiniu.com/ai/agent) | 게이트웨이 | ✅ |
|
||||
|
||||
[→ 모든 통합 보기](https://link.langbot.app/en/docs/features)
|
||||
|
||||
+3
-9
@@ -5,7 +5,7 @@
|
||||
|
||||
<div align="center">
|
||||
|
||||
<a href="https://www.producthunt.com/products/langbot/launches/langbot?embed=true&utm_source=badge-featured&utm_medium=badge&utm_campaign=badge-langbot" target="_blank" rel="noopener noreferrer"><img alt="LangBot - Easy-to-use global IM bot platform designed for the LLM era | Product Hunt" width="250" height="54" src="https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=979554&theme=light&t=1782822143403"></a>
|
||||
<a href="https://www.producthunt.com/products/langbot?utm_source=badge-follow&utm_medium=badge&utm_source=badge-langbot" target="_blank"><img src="https://api.producthunt.com/widgets/embed-image/v1/follow.svg?product_id=1077185&theme=light" alt="LangBot - Production-grade IM bot made easy. | Product Hunt" style="width: 250px; height: 54px;" width="250" height="54" /></a>
|
||||
|
||||
<h3>Платформа производственного уровня для создания агентных IM-ботов.</h3>
|
||||
<h4>Быстро создавайте, отлаживайте и развертывайте ИИ-ботов в Slack, Discord, Telegram, WeChat и других платформах.</h4>
|
||||
@@ -54,12 +54,6 @@ LangBot — это **платформа с открытым исходным к
|
||||
|
||||
---
|
||||
|
||||
## 😎 Оставайтесь в курсе
|
||||
|
||||
Нажмите кнопки Star и Watch в правом верхнем углу репозитория, чтобы получать последние обновления.
|
||||
|
||||
|
||||
|
||||
## Быстрый старт
|
||||
|
||||
### ☁️ LangBot Cloud (Рекомендуется)
|
||||
@@ -79,7 +73,7 @@ uvx langbot
|
||||
```bash
|
||||
git clone https://github.com/langbot-app/LangBot
|
||||
cd LangBot/docker
|
||||
docker compose --profile all up -d
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### Облачное развертывание одним кликом
|
||||
@@ -131,7 +125,7 @@ docker compose --profile all up -d
|
||||
| [Volc Engine Ark](https://console.volcengine.com/ark/region:ark+cn-beijing/model?vendor=Bytedance&view=LIST_VIEW) | Шлюз | ✅ |
|
||||
| [ModelScope](https://modelscope.cn/docs/model-service/API-Inference/intro) | Шлюз | ✅ |
|
||||
| [GiteeAI](https://ai.gitee.com/) | Шлюз | ✅ |
|
||||
| [302.AI](https://share.302ai.cn/SuTG99) | Шлюз | ✅ |
|
||||
| [302.AI](https://share.302.ai/SuTG99) | Шлюз | ✅ |
|
||||
| [接口 AI](https://jiekou.ai/) | Шлюз | ✅ |
|
||||
| [CompShare](https://www.compshare.cn/?ytag=GPU_YY-gh_langbot) | Платформа GPU | ✅ |
|
||||
| [PPIO](https://ppinfra.com/user/register?invited_by=QJKFYD&utm_source=github_langbot) | Платформа GPU | ✅ |
|
||||
|
||||
+2
-8
@@ -56,12 +56,6 @@ LangBot 是一個**開源的生產級平台**，用於建構 AI 驅動的即時
|
||||
|
||||
---
|
||||
|
||||
## 😎 保持更新
|
||||
|
||||
點擊倉庫右上角 Star 和 Watch 按鈕，獲取最新動態。
|
||||
|
||||
|
||||
|
||||
## 快速開始
|
||||
|
||||
### ☁️ LangBot Cloud（推薦）
|
||||
@@ -81,7 +75,7 @@ uvx langbot
|
||||
```bash
|
||||
git clone https://github.com/langbot-app/LangBot
|
||||
cd LangBot/docker
|
||||
docker compose --profile all up -d
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### 一鍵雲端部署
|
||||
@@ -137,7 +131,7 @@ docker compose --profile all up -d
|
||||
| [優雲智算](https://www.compshare.cn/?ytag=GPU_YY-gh_langbot) | GPU 平台 | ✅ |
|
||||
| [PPIO](https://ppinfra.com/user/register?invited_by=QJKFYD&utm_source=github_langbot) | GPU 平台 | ✅ |
|
||||
| [接口 AI](https://jiekou.ai/) | 聚合平台 | ✅ |
|
||||
| [302.AI](https://share.302ai.cn/SuTG99) | 聚合平台 | ✅ |
|
||||
| [302.AI](https://share.302.ai/SuTG99) | 聚合平台 | ✅ |
|
||||
| [Qiniu](https://www.qiniu.com/ai/agent) | 聚合平台 | ✅ |
|
||||
|
||||
### TTS（語音合成）
|
||||
|
||||
+3
-9
@@ -5,7 +5,7 @@
|
||||
|
||||
<div align="center">
|
||||
|
||||
<a href="https://www.producthunt.com/products/langbot/launches/langbot?embed=true&utm_source=badge-featured&utm_medium=badge&utm_campaign=badge-langbot" target="_blank" rel="noopener noreferrer"><img alt="LangBot - Easy-to-use global IM bot platform designed for the LLM era | Product Hunt" width="250" height="54" src="https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=979554&theme=light&t=1782822143403"></a>
|
||||
<a href="https://www.producthunt.com/products/langbot?utm_source=badge-follow&utm_medium=badge&utm_source=badge-langbot" target="_blank"><img src="https://api.producthunt.com/widgets/embed-image/v1/follow.svg?product_id=1077185&theme=light" alt="LangBot - Production-grade IM bot made easy. | Product Hunt" style="width: 250px; height: 54px;" width="250" height="54" /></a>
|
||||
|
||||
<h3>Nền tảng cấp sản xuất để xây dựng bot IM với AI agent.</h3>
|
||||
<h4>Xây dựng, gỡ lỗi và triển khai bot AI nhanh chóng trên Slack, Discord, Telegram, WeChat và nhiều nền tảng khác.</h4>
|
||||
@@ -54,12 +54,6 @@ LangBot là một **nền tảng mã nguồn mở, cấp sản xuất** để x
|
||||
|
||||
---
|
||||
|
||||
## 😎 Cập nhật Mới nhất
|
||||
|
||||
Nhấp vào các nút Star và Watch ở góc trên bên phải của kho lưu trữ để nhận các bản cập nhật mới nhất.
|
||||
|
||||
|
||||
|
||||
## Bắt đầu nhanh
|
||||
|
||||
### ☁️ LangBot Cloud (Khuyên dùng)
|
||||
@@ -79,7 +73,7 @@ uvx langbot
|
||||
```bash
|
||||
git clone https://github.com/langbot-app/LangBot
|
||||
cd LangBot/docker
|
||||
docker compose --profile all up -d
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### Triển khai đám mây một cú nhấp
|
||||
@@ -135,7 +129,7 @@ docker compose --profile all up -d
|
||||
| [PPIO](https://ppinfra.com/user/register?invited_by=QJKFYD&utm_source=github_langbot) | Nền tảng GPU | ✅ |
|
||||
| [ShengSuanYun](https://www.shengsuanyun.com/?from=CH_KYIPP758) | Nền tảng GPU | ✅ |
|
||||
| [接口 AI](https://jiekou.ai/) | Cổng | ✅ |
|
||||
| [302.AI](https://share.302ai.cn/SuTG99) | Cổng | ✅ |
|
||||
| [302.AI](https://share.302.ai/SuTG99) | Cổng | ✅ |
|
||||
| [Qiniu](https://www.qiniu.com/ai/agent) | Cổng | ✅ |
|
||||
|
||||
[→ Xem tất cả tích hợp](https://link.langbot.app/en/docs/features)
|
||||
|
||||
@@ -1,163 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Compare YAML node definitions with frontend node-configs."""
|
||||
|
||||
import yaml
|
||||
import os
|
||||
import re
|
||||
import json
|
||||
|
||||
# 1. Parse YAML files
|
||||
yaml_dir = 'src/langbot/templates/metadata/nodes'
|
||||
yaml_nodes = {}
|
||||
|
||||
for filename in sorted(os.listdir(yaml_dir)):
|
||||
if filename.endswith('.yaml'):
|
||||
filepath = os.path.join(yaml_dir, filename)
|
||||
with open(filepath, 'r') as f:
|
||||
data = yaml.safe_load(f)
|
||||
node_name = data.get('name', filename.replace('.yaml', ''))
|
||||
yaml_nodes[node_name] = {
|
||||
'category': data.get('category', ''),
|
||||
'inputs': [i['name'] for i in data.get('inputs', [])],
|
||||
'outputs': [o['name'] for o in data.get('outputs', [])],
|
||||
'config': [c['name'] for c in data.get('config', [])]
|
||||
}
|
||||
|
||||
# 2. Parse frontend node-configs TypeScript files
|
||||
node_configs_dir = 'web/src/app/home/workflows/components/workflow-editor/node-configs'
|
||||
|
||||
frontend_nodes = {}
|
||||
|
||||
def parse_ts_file(filepath):
|
||||
"""Parse a TypeScript file to extract node configurations."""
|
||||
with open(filepath, 'r') as f:
|
||||
content = f.read()
|
||||
|
||||
# Find all node type definitions
|
||||
# Pattern: nodeType: 'xxx'
|
||||
node_type_pattern = r"nodeType:\s*'([^']+)'"
|
||||
node_types = re.findall(node_type_pattern, content)
|
||||
|
||||
# For each node type, extract inputs, outputs, and config
|
||||
for node_type in node_types:
|
||||
# Find the config object for this node type
|
||||
# Look for the section between this nodeType and the next one or end of object
|
||||
pattern = rf"nodeType:\s*'({re.escape(node_type)})'.*?(?=nodeType:|export\s+(const|function)|$)"
|
||||
match = re.search(pattern, content, re.DOTALL)
|
||||
|
||||
if match:
|
||||
section = match.group(0)
|
||||
|
||||
# Extract inputs
|
||||
inputs = re.findall(r"createInput\('([^']+)'", section)
|
||||
|
||||
# Extract outputs
|
||||
outputs = re.findall(r"createOutput\('([^']+)'", section)
|
||||
|
||||
# Extract config names
|
||||
config_names = re.findall(r"name:\s*'([^']+)'", section)
|
||||
# Remove duplicates while preserving order
|
||||
seen = set()
|
||||
unique_config = []
|
||||
for c in config_names:
|
||||
if c not in seen:
|
||||
seen.add(c)
|
||||
unique_config.append(c)
|
||||
|
||||
frontend_nodes[node_type] = {
|
||||
'inputs': inputs,
|
||||
'outputs': outputs,
|
||||
'config': unique_config
|
||||
}
|
||||
|
||||
# Parse all config files
|
||||
for filename in os.listdir(node_configs_dir):
|
||||
if filename.endswith('.ts') and filename != 'types.ts' and filename != 'index.ts':
|
||||
filepath = os.path.join(node_configs_dir, filename)
|
||||
parse_ts_file(filepath)
|
||||
|
||||
# 3. Compare and report differences
|
||||
print("=" * 80)
|
||||
print("WORKFLOW NODE COMPARISON REPORT: YAML vs Frontend")
|
||||
print("=" * 80)
|
||||
|
||||
all_node_types = sorted(set(list(yaml_nodes.keys()) + list(frontend_nodes.keys())))
|
||||
|
||||
discrepancies = []
|
||||
|
||||
for node_type in all_node_types:
|
||||
yaml_def = yaml_nodes.get(node_type)
|
||||
frontend_def = frontend_nodes.get(node_type)
|
||||
|
||||
node_discrepancies = []
|
||||
|
||||
if not yaml_def:
|
||||
print(f"\n⚠️ {node_type}: ONLY in frontend (not in YAML)")
|
||||
continue
|
||||
if not frontend_def:
|
||||
print(f"\n⚠️ {node_type}: ONLY in YAML (not in frontend)")
|
||||
continue
|
||||
|
||||
# Compare inputs
|
||||
yaml_inputs = set(yaml_def['inputs'])
|
||||
frontend_inputs = set(frontend_def['inputs'])
|
||||
if yaml_inputs != frontend_inputs:
|
||||
only_yaml = yaml_inputs - frontend_inputs
|
||||
only_frontend = frontend_inputs - yaml_inputs
|
||||
node_discrepancies.append({
|
||||
'type': 'inputs',
|
||||
'only_yaml': list(only_yaml),
|
||||
'only_frontend': list(only_frontend)
|
||||
})
|
||||
|
||||
# Compare outputs
|
||||
yaml_outputs = set(yaml_def['outputs'])
|
||||
frontend_outputs = set(frontend_def['outputs'])
|
||||
if yaml_outputs != frontend_outputs:
|
||||
only_yaml = yaml_outputs - frontend_outputs
|
||||
only_frontend = frontend_outputs - yaml_outputs
|
||||
node_discrepancies.append({
|
||||
'type': 'outputs',
|
||||
'only_yaml': list(only_yaml),
|
||||
'only_frontend': list(only_frontend)
|
||||
})
|
||||
|
||||
# Compare config
|
||||
yaml_config = set(yaml_def['config'])
|
||||
frontend_config = set(frontend_def['config'])
|
||||
if yaml_config != frontend_config:
|
||||
only_yaml = yaml_config - frontend_config
|
||||
only_frontend = frontend_config - yaml_config
|
||||
node_discrepancies.append({
|
||||
'type': 'config',
|
||||
'only_yaml': list(only_yaml),
|
||||
'only_frontend': list(only_frontend)
|
||||
})
|
||||
|
||||
if node_discrepancies:
|
||||
print(f"\n❌ {node_type} ({yaml_def['category']}): HAS DISCREPANCIES")
|
||||
for d in node_discrepancies:
|
||||
print(f" {d['type']}:")
|
||||
if d['only_yaml']:
|
||||
print(f" Only in YAML: {d['only_yaml']}")
|
||||
if d['only_frontend']:
|
||||
print(f" Only in Frontend: {d['only_frontend']}")
|
||||
discrepancies.append((node_type, node_discrepancies))
|
||||
else:
|
||||
print(f"\n✅ {node_type} ({yaml_def['category']}): OK")
|
||||
|
||||
print(f"\n{'=' * 80}")
|
||||
print(f"SUMMARY: {len(discrepancies)} nodes with discrepancies out of {len(all_node_types)} total")
|
||||
print(f"{'=' * 80}")
|
||||
|
||||
# Output as JSON for further processing
|
||||
output = {
|
||||
'yaml_nodes': {k: v for k, v in yaml_nodes.items()},
|
||||
'frontend_nodes': {k: v for k, v in frontend_nodes.items()},
|
||||
'discrepancies': {k: v for k, v in discrepancies}
|
||||
}
|
||||
|
||||
with open('node_comparison.json', 'w') as f:
|
||||
json.dump(output, f, indent=2)
|
||||
|
||||
print(f"\nDetailed comparison saved to node_comparison.json")
|
||||
@@ -62,12 +62,11 @@ services:
|
||||
- TZ=Asia/Shanghai
|
||||
# Unified env-override convention: SECTION__SUBSECTION__KEY overrides the
|
||||
# matching config.yaml field (see LoadConfigStage). These map onto
|
||||
# box.* and are forwarded to the Box runtime via INIT RPC.
|
||||
# box.local.* and are forwarded to the Box runtime via INIT RPC.
|
||||
- BOX__LOCAL__HOST_ROOT=${LANGBOT_BOX_ROOT:-${PWD}/data/box}
|
||||
- BOX__LOCAL__DEFAULT_WORKSPACE=default
|
||||
- BOX__LOCAL__SKILLS_ROOT=skills
|
||||
- BOX__LOCAL__ALLOWED_MOUNT_ROOTS=${LANGBOT_BOX_ROOT:-${PWD}/data/box}
|
||||
- BOX__DOCKER__CPU_LIMIT_ENABLED=${LANGBOT_BOX_DOCKER_CPU_LIMIT_ENABLED:-true}
|
||||
ports:
|
||||
- 5300:5300 # For web ui and webhook callback
|
||||
- 2280-2285:2280-2285 # For platform reverse connection
|
||||
|
||||
@@ -1,575 +0,0 @@
|
||||
# HTTP Bot Adapter — Design Document
|
||||
|
||||
> Status: **Implemented** · Branch: `feat/http-bot-adapter` · Author: LangBot core
|
||||
>
|
||||
> A first-class, **standalone** message-platform adapter (`http_bot`) that lets
|
||||
> any external system (e.g. LangBot Space ticketing, an internal back-office, a
|
||||
> CRM, a custom web app) talk to a LangBot pipeline over plain HTTP — **inbound**
|
||||
> by POSTing messages in, **outbound** by receiving replies on a callback URL —
|
||||
> with full support for the pipeline's native N→1 aggregation and 1→M
|
||||
> multi-reply semantics, and **without** holding a long-lived WebSocket
|
||||
> connection.
|
||||
>
|
||||
> **Shipped in this branch:**
|
||||
> - `src/langbot/pkg/platform/sources/http_bot.yaml` — adapter manifest (auto-discovered)
|
||||
> - `src/langbot/pkg/platform/sources/http_bot.py` — `HttpBotAdapter`
|
||||
> - `src/langbot/pkg/platform/sources/http_bot_signing.py` — HMAC helpers
|
||||
> - `src/langbot/pkg/platform/sources/http_bot.svg` — icon
|
||||
> - `docs/platforms/http-bot.md` — integration guide
|
||||
> - `docs/http-bot-openapi.json` — machine-readable contract
|
||||
> - `examples/http-bot/` — Python + TypeScript reference clients
|
||||
>
|
||||
> **Final decisions (resolving the original open questions):**
|
||||
> 1. Callback URL is **config-only** — never accepted per-message (SSRF closed).
|
||||
> 2. **Session reset is provided** — `POST /bots/<uuid>/reset` keyed by `session_id`.
|
||||
> 3. Reference **clients are provided** — `examples/http-bot/client.py` + `client.ts`.
|
||||
> 4. **Sync convenience mode is included** — `POST /bots/<uuid>/sync` (opt-in, lossy).
|
||||
|
||||
---
|
||||
|
||||
## 1. Background & Motivation
|
||||
|
||||
### 1.1 The concrete need
|
||||
|
||||
LangBot Space wants to use a LangBot pipeline as the brain for **ticket
|
||||
handling**. The integration is **server-to-server**: Space's backend pushes a
|
||||
user's ticket messages into LangBot and renders LangBot's replies back into the
|
||||
ticket thread.
|
||||
|
||||
This interaction is **not** request/response shaped:
|
||||
|
||||
- **N → 1**: a user may fire several messages in a row ("the app crashed" …
|
||||
"when I click export" … "here's a screenshot"). The pipeline's
|
||||
**message aggregation** feature should debounce and merge these into one turn.
|
||||
- **1 → N**: a single turn may yield **multiple** outbound messages — a tool/
|
||||
function call narrating progress, a plugin emitting several cards, a streamed
|
||||
answer split into chunks.
|
||||
|
||||
### 1.2 Why the existing options don't fit
|
||||
|
||||
LangBot today exposes exactly one externally-reachable way to drive a pipeline
|
||||
that is **not** tied to a specific IM vendor: the **WebSocket** path
|
||||
(`/api/v1/pipelines/<uuid>/ws/connect` for dashboard debug, and
|
||||
`/api/v1/embed/<bot_uuid>/ws/connect` for the embeddable web widget).
|
||||
|
||||
For a server-to-server integration the WebSocket path has real friction:
|
||||
|
||||
| Problem | Detail |
|
||||
|---|---|
|
||||
| Long-lived connection | Caller must maintain a socket, heartbeats, and reconnect logic for what is fundamentally a fire-and-collect workload. |
|
||||
| Session identity | Inbound messages are keyed by the transient `connection_id` (`websocket_{connection_id}`); the caller **cannot supply a stable, business-meaningful session id** (e.g. a ticket number). Multi-ticket isolation is not expressible. |
|
||||
| Auth mismatch | The debug socket is gated by the **dashboard JWT** (must not be handed to an external service); the embed socket is gated by **Cloudflare Turnstile** (a *browser* human-check that a backend cannot satisfy). Neither is a server-to-server credential. |
|
||||
| In-memory, single-process state | Session history lives in process memory and is lost on restart. |
|
||||
|
||||
> **Key realisation.** The N→1 / 1→M behaviour the caller wants is **not**
|
||||
> provided by WebSocket — it is provided by the **pipeline** (aggregation +
|
||||
> the adapter being free to call `reply_message` any number of times). It is
|
||||
> therefore **transport-independent**. We can deliver the exact same semantics
|
||||
> over a far lighter HTTP transport.
|
||||
|
||||
### 1.3 Why a *new, standalone* adapter (not a refactor of an existing one)
|
||||
|
||||
The brief is explicit: **do not reuse / fork an existing vendor adapter.** The
|
||||
vendor adapters (`lark`, `wecom`, `qqofficial`, `slack`, …) carry vendor-specific
|
||||
signature schemes, payload shapes, and message-segment mappings. Bending one of
|
||||
them into a "generic" mode would couple a public integration surface to one
|
||||
vendor's quirks and make the developer experience worse for everyone.
|
||||
|
||||
Instead we ship `http_bot` as a clean, independent adapter whose **entire
|
||||
contract is LangBot's own** — documented, versioned, and designed front-to-back
|
||||
around *integrator* developer experience.
|
||||
|
||||
---
|
||||
|
||||
## 2. Goals & Non-Goals
|
||||
|
||||
### Goals
|
||||
|
||||
- **G1** A standalone `http_bot` adapter, selectable like any other platform
|
||||
adapter in the dashboard, with its own config schema and docs.
|
||||
- **G2** **Inbound**: external systems POST messages to a stable LangBot URL,
|
||||
carrying a **caller-defined `session_id`** that maps 1:1 to a LangBot session.
|
||||
- **G3** **Outbound**: LangBot delivers each reply by POSTing to a
|
||||
caller-configured **callback URL**; one turn may produce **many** callbacks.
|
||||
- **G4** Preserve pipeline-native **N→1 aggregation** and **1→M multi-reply**.
|
||||
- **G5** Server-to-server **auth**: shared-secret HMAC request signing both
|
||||
directions (no JWT, no Turnstile, no long-lived socket).
|
||||
- **G6** **Great DX**: copy-pasteable curl, a tiny reference client, an OpenAPI
|
||||
fragment, idempotency, clear error envelope, and a local echo-server recipe.
|
||||
|
||||
### Non-Goals
|
||||
|
||||
- Not replacing or deprecating the WebSocket / embed widget path (that remains
|
||||
the right tool for *browser*, real-time, streaming chat UIs).
|
||||
- Not a synchronous "one request → one response" RPC (explicitly rejected: it
|
||||
cannot express 1→M; see §9 for the optional sync convenience mode).
|
||||
- No built-in message **persistence/replay** in v1 (callbacks are at-least-once
|
||||
best-effort; durability is the caller's responsibility — see §8).
|
||||
- No multi-tenant API-key management UI in v1 (one secret per bot; see §11).
|
||||
|
||||
---
|
||||
|
||||
## 3. How LangBot routes a message (the parts we plug into)
|
||||
|
||||
Understanding the existing flow is what makes this adapter cheap. A message
|
||||
flows through these stages (verified against current `master`):
|
||||
|
||||
```
|
||||
INBOUND OUTBOUND
|
||||
external POST ─┐ ┌─ reply_message()
|
||||
▼ │ reply_message_chunk()
|
||||
POST /bots/<bot_uuid> (unified webhook router, AuthType.NONE)
|
||||
│ webhooks.py → adapter.handle_unified_webhook(bot_uuid, path, request)
|
||||
▼ │
|
||||
HttpBotAdapter.handle_unified_webhook │ (called 0..N times
|
||||
• verify HMAC signature │ per turn by the
|
||||
• parse {session_id, message[]} │ pipeline / plugins)
|
||||
• build FriendMessage / GroupMessage │
|
||||
• fire registered listener ───────────────┐ │
|
||||
│ │ │
|
||||
▼ ▼ │
|
||||
botmgr.on_friend_message / on_group_message │
|
||||
• (optional) webhook_pusher fan-out │
|
||||
• msg_aggregator.add_message(...) ── N→1 debounce ──►│
|
||||
│ │
|
||||
▼ │
|
||||
query_pool → pipeline.run() ─── invokes adapter ─────┘
|
||||
reply methods 1..M times
|
||||
```
|
||||
|
||||
Two framework facts we rely on:
|
||||
|
||||
1. **N→1 aggregation is free.** `botmgr` hands every inbound event to
|
||||
`self.ap.msg_aggregator.add_message(...)`, which debounces per
|
||||
`session_id` and merges consecutive messages into one pipeline turn
|
||||
(`pkg/pipeline/aggregator.py`). The adapter does nothing special.
|
||||
|
||||
2. **1→M is free.** The pipeline (and any plugin in the chain) calls
|
||||
`adapter.reply_message()` / `reply_message_chunk()` **as many times as it
|
||||
wants** per turn. The adapter's only job is to deliver each call outward.
|
||||
For `http_bot` that means: **one outbound callback POST per call.**
|
||||
|
||||
3. **A unified inbound route already exists.** `WebhookRouterGroup`
|
||||
(`pkg/api/http/controller/groups/webhooks.py`) maps
|
||||
`POST /bots/<bot_uuid>[/<path>]` (auth `NONE`) to
|
||||
`adapter.handle_unified_webhook(bot_uuid, path, request)`. `http_bot`
|
||||
implements that method and is reachable **without registering any new
|
||||
route** — it does its own signature verification, exactly like the vendor
|
||||
webhook adapters do.
|
||||
|
||||
> Net new code is essentially: one `http_bot.py` adapter, one `http_bot.yaml`
|
||||
> schema, signing helpers, and docs. No router, aggregator, or pipeline changes.
|
||||
|
||||
---
|
||||
|
||||
## 4. Architecture Overview
|
||||
|
||||
```
|
||||
┌────────────────────┐ (1) inbound: POST signed message
|
||||
│ External system │ ──────────────────────────────────────────────► ┌──────────────────────┐
|
||||
│ (LangBot Space, │ POST /bots/<bot_uuid> │ LangBot │
|
||||
│ CRM, web app …) │ X-LB-Signature, X-LB-Timestamp │ │
|
||||
│ │ { session_id, message:[...] } │ HttpBotAdapter │
|
||||
│ - callback server │ ◄────────────────────────────────────────────── │ (platform/sources) │
|
||||
│ (receives │ (4) outbound: POST signed reply(s) │ │
|
||||
│ replies) │ POST <callback_url> │ pipeline + aggregator│
|
||||
└────────────────────┘ X-LB-Signature, X-LB-Timestamp └──────────────────────┘
|
||||
{ session_id, sequence, is_final,
|
||||
message:[...] } (sent 1..M times)
|
||||
```
|
||||
|
||||
- The adapter is **stateless across requests** at the HTTP layer; session
|
||||
continuity is carried by `session_id` and resolved by LangBot's normal
|
||||
session manager.
|
||||
- **Inbound** and **outbound** are **independent HTTP exchanges**. LangBot does
|
||||
not answer the inbound POST with the pipeline result; it `202 Accepts` it and
|
||||
later POSTs the reply(s) to the callback URL. This is what makes 1→M natural.
|
||||
|
||||
---
|
||||
|
||||
## 5. Configuration Schema (`http_bot.yaml`)
|
||||
|
||||
Follows the existing `MessagePlatformAdapter` manifest convention (cf.
|
||||
`slack.yaml`). Fields:
|
||||
|
||||
| field | type | required | purpose |
|
||||
|---|---|---|---|
|
||||
| `inbound_secret` | string (secret) | yes | HMAC key the **caller** uses to sign inbound POSTs; LangBot verifies. |
|
||||
| `callback_url` | string (url) | no* | Where LangBot POSTs replies. *Optional if the caller supplies `callback_url` per-message (see §6.1); a static default lives here. |
|
||||
| `outbound_secret` | string (secret) | no | HMAC key LangBot uses to sign outbound callbacks; caller verifies. Defaults to `inbound_secret` if empty. |
|
||||
| `default_session_type` | enum `person`/`group` | no | Default when a message omits `session_type`. Default `person`. |
|
||||
| `signature_required` | bool | no | If `false`, skip inbound signature check (dev only; logs a warning). Default `true`. |
|
||||
| `callback_timeout` | int (seconds) | no | Per-callback HTTP timeout. Default `15`. |
|
||||
| `callback_max_retries` | int | no | Retries on 5xx/timeout with backoff. Default `3`. |
|
||||
| `webhook_url` | webhook-url (display) | — | Read-only field rendering the inbound URL `…/bots/<bot_uuid>` for copy-paste, like other webhook adapters. |
|
||||
|
||||
Manifest sketch (i18n labels elided for brevity):
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: MessagePlatformAdapter
|
||||
metadata:
|
||||
name: http_bot
|
||||
label: { en_US: "HTTP Bot", zh_Hans: "HTTP 通用接入" }
|
||||
description:
|
||||
en_US: "Integrate any backend over plain HTTP. Push messages in, receive replies on a callback URL. Server-to-server, no long-lived connection."
|
||||
zh_Hans: "通过 HTTP 接入任意后端系统。推入消息、在回调地址接收回复。面向服务间集成，无需长连接。"
|
||||
icon: http_bot.svg
|
||||
spec:
|
||||
categories: [popular, global]
|
||||
help_links:
|
||||
zh: https://docs.langbot.app/zh/platforms/http-bot
|
||||
en: https://docs.langbot.app/en/platforms/http-bot
|
||||
config:
|
||||
- { name: inbound_secret, type: string, required: true, default: "" }
|
||||
- { name: callback_url, type: string, required: false, default: "" }
|
||||
- { name: outbound_secret, type: string, required: false, default: "" }
|
||||
- { name: default_session_type, type: select, required: false, default: "person",
|
||||
options: [person, group] }
|
||||
- { name: signature_required, type: boolean, required: false, default: true }
|
||||
- { name: callback_timeout, type: integer, required: false, default: 15 }
|
||||
- { name: callback_max_retries, type: integer, required: false, default: 3 }
|
||||
- { name: webhook_url, type: webhook-url, required: false, default: "" }
|
||||
execution:
|
||||
python:
|
||||
path: ./http_bot.py
|
||||
attr: HttpBotAdapter
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. The HTTP Contract (this is the DX surface)
|
||||
|
||||
### 6.1 Inbound — push a message into LangBot
|
||||
|
||||
```
|
||||
POST /bots/{bot_uuid}
|
||||
Content-Type: application/json
|
||||
X-LB-Timestamp: 1718000000
|
||||
X-LB-Signature: sha256=<hex hmac>
|
||||
X-LB-Idempotency-Key: <uuid> # optional, dedup window
|
||||
```
|
||||
|
||||
Body:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"session_id": "ticket-10293", // REQUIRED. Caller-defined. Maps 1:1 to a LangBot session.
|
||||
"session_type": "person", // optional, "person" | "group"; default from config
|
||||
"sender": { // optional metadata, surfaced to pipeline/plugins
|
||||
"id": "user-5567",
|
||||
"name": "Alice"
|
||||
},
|
||||
"message": [ // REQUIRED. A LangBot MessageChain (list of segments).
|
||||
{ "type": "Plain", "text": "Export keeps failing on the dashboard." },
|
||||
{ "type": "Image", "url": "https://.../screenshot.png" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Response (LangBot does **not** block on the pipeline):
|
||||
|
||||
```jsonc
|
||||
// 202 Accepted
|
||||
{
|
||||
"code": 0,
|
||||
"msg": "accepted",
|
||||
"data": {
|
||||
"session_id": "ticket-10293",
|
||||
"accepted_message_id": "in_01H....", // server-assigned id for this inbound message
|
||||
"aggregating": true // true if buffered by the aggregator
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**N→1 in practice.** Fire three POSTs with the same `session_id` inside the
|
||||
aggregation window → the pipeline runs **once** with the three messages merged.
|
||||
No special flag needed; this is the aggregator's default behaviour when enabled
|
||||
on the pipeline.
|
||||
|
||||
### 6.2 Outbound — LangBot delivers replies to your callback
|
||||
|
||||
For each `reply_message` / `reply_message_chunk` the pipeline emits, LangBot
|
||||
POSTs to `callback_url`:
|
||||
|
||||
```
|
||||
POST {callback_url}
|
||||
Content-Type: application/json
|
||||
X-LB-Timestamp: 1718000001
|
||||
X-LB-Signature: sha256=<hex hmac over body>
|
||||
```
|
||||
|
||||
Body:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"session_id": "ticket-10293", // echoes the inbound session
|
||||
"reply_to": "in_01H....", // the inbound message id this answers
|
||||
"sequence": 1, // 1-based ordinal within this turn (for 1→M ordering)
|
||||
"is_final": false, // false for intermediate/streamed parts
|
||||
"stream": false, // true when this is a streamed chunk
|
||||
"message": [
|
||||
{ "type": "Plain", "text": "Looking into it — checking your export logs…" }
|
||||
],
|
||||
"timestamp": "2026-06-22T09:00:01Z"
|
||||
}
|
||||
```
|
||||
|
||||
**1→M in practice.** A turn that fires a function call then a final answer
|
||||
produces e.g.:
|
||||
|
||||
```
|
||||
POST callback → { sequence: 1, is_final: false, message: ["Checking logs…"] }
|
||||
POST callback → { sequence: 2, is_final: false, message: ["Found 2 failed exports."] }
|
||||
POST callback → { sequence: 3, is_final: true, message: ["Fixed. Try again now."] }
|
||||
```
|
||||
|
||||
The caller stitches by `session_id` + `sequence`, and knows the turn is complete
|
||||
when `is_final: true` arrives.
|
||||
|
||||
Your callback endpoint should return `200` quickly. A non-2xx triggers retry
|
||||
with backoff (`callback_max_retries`).
|
||||
|
||||
### 6.3 Error envelope (inbound)
|
||||
|
||||
Consistent, machine-readable; never leak internals:
|
||||
|
||||
```jsonc
|
||||
{ "code": 40101, "msg": "invalid signature", "data": null }
|
||||
```
|
||||
|
||||
| HTTP | code | meaning |
|
||||
|---|---|---|
|
||||
| 202 | 0 | accepted |
|
||||
| 400 | 40001 | malformed body / missing `session_id` or `message` |
|
||||
| 401 | 40101 | bad/expired signature |
|
||||
| 403 | 40301 | bot disabled |
|
||||
| 404 | 40401 | bot_uuid not found / not an `http_bot` adapter |
|
||||
| 409 | 40901 | duplicate idempotency key (already accepted) |
|
||||
| 413 | 41301 | message too large |
|
||||
| 500 | 50001 | internal error |
|
||||
|
||||
---
|
||||
|
||||
## 7. Signing scheme (both directions)
|
||||
|
||||
Symmetric, dependency-free HMAC-SHA256 — trivial to implement in any language.
|
||||
|
||||
```
|
||||
signing_string = "{timestamp}.{raw_request_body}"
|
||||
signature = "sha256=" + hex(HMAC_SHA256(secret, signing_string))
|
||||
```
|
||||
|
||||
Verification rules:
|
||||
|
||||
- Reject if `|now - timestamp| > 300s` (replay window).
|
||||
- Constant-time compare (`hmac.compare_digest`).
|
||||
- Inbound verified with `inbound_secret`; outbound signed with
|
||||
`outbound_secret` (falls back to `inbound_secret`).
|
||||
- `signature_required: false` bypasses verification **and logs a warning** —
|
||||
intended only for local development behind a trusted network.
|
||||
|
||||
Reference (Python, ~6 lines):
|
||||
|
||||
```python
|
||||
import hmac, hashlib, time
|
||||
|
||||
def sign(secret: str, body: bytes, ts: int | None = None) -> tuple[str, str]:
|
||||
ts = ts or int(time.time())
|
||||
mac = hmac.new(secret.encode(), f"{ts}.".encode() + body, hashlib.sha256)
|
||||
return str(ts), "sha256=" + mac.hexdigest()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Delivery semantics & reliability
|
||||
|
||||
- **Inbound**: `202 Accepted` means *queued*, not *processed*. Use
|
||||
`X-LB-Idempotency-Key` to make client retries safe (dedup window, e.g. 10 min).
|
||||
- **Outbound**: **at-least-once**, best-effort. Retries on timeout/5xx with
|
||||
exponential backoff up to `callback_max_retries`. Callbacks for one
|
||||
`session_id` are delivered **in `sequence` order** (serialised per session);
|
||||
across sessions they may interleave.
|
||||
- **No persistence in v1**: if LangBot restarts mid-turn, in-flight callbacks
|
||||
may be lost. Durable replay is deferred (see §13). Callers needing exactly-once
|
||||
should dedup on `(session_id, reply_to, sequence)`.
|
||||
- **Backpressure**: the adapter must not block the pipeline on slow callbacks —
|
||||
outbound POSTs run on a per-session ordered queue with the configured timeout.
|
||||
|
||||
---
|
||||
|
||||
## 9. Optional: synchronous convenience mode (v1.1, behind a flag)
|
||||
|
||||
Some simple callers genuinely want "POST a message, get the reply in the HTTP
|
||||
response" and don't care about streaming/multi-part. We can offer an **opt-in**
|
||||
sync endpoint that internally waits for `is_final` and **collapses** all 1→M
|
||||
parts into one array:
|
||||
|
||||
```
|
||||
POST /bots/{bot_uuid}/sync → 200 { session_id, message: [ ...all parts concatenated... ] }
|
||||
```
|
||||
|
||||
Implemented by attaching a per-request future that resolves on the final reply,
|
||||
with a hard timeout. This is a **convenience wrapper** over the same machinery,
|
||||
explicitly documented as lossy for streaming/ordering. Not in v1 core.
|
||||
|
||||
---
|
||||
|
||||
## 10. Adapter implementation sketch (`platform/sources/http_bot.py`)
|
||||
|
||||
Implements `AbstractMessagePlatformAdapter`. Key methods:
|
||||
|
||||
```python
|
||||
class HttpBotAdapter(AbstractMessagePlatformAdapter):
|
||||
listeners: dict = pydantic.Field(default_factory=dict, exclude=True)
|
||||
|
||||
# --- inbound -------------------------------------------------------
|
||||
async def handle_unified_webhook(self, bot_uuid, path, request):
|
||||
body = await request.get_body()
|
||||
if self.config.get("signature_required", True):
|
||||
if not self._verify(request, body):
|
||||
return jsonify({"code": 40101, "msg": "invalid signature"}), 401
|
||||
data = json.loads(body)
|
||||
session_id = data["session_id"] # caller-defined identity
|
||||
session_type = data.get("session_type", self.config.get("default_session_type", "person"))
|
||||
chain = MessageChain.model_validate(data["message"])
|
||||
event = self._build_event(session_type, session_id, data.get("sender"), chain)
|
||||
# remember where to send replies for this session
|
||||
self._callback_for[session_id] = data.get("callback_url") or self.config.get("callback_url")
|
||||
# fire the registered listener → botmgr → msg_aggregator (N→1) → pipeline
|
||||
if type(event) in self.listeners:
|
||||
asyncio.create_task(self.listeners[type(event)](event, self))
|
||||
return jsonify({"code": 0, "msg": "accepted",
|
||||
"data": {"session_id": session_id, "accepted_message_id": event.message_id}}), 202
|
||||
|
||||
# --- outbound (called 1..M times per turn by the pipeline) ---------
|
||||
async def reply_message(self, message_source, message, quote_origin=False):
|
||||
return await self._post_callback(message_source, message, is_final=True, stream=False)
|
||||
|
||||
async def reply_message_chunk(self, message_source, bot_message, message,
|
||||
quote_origin=False, is_final=False):
|
||||
return await self._post_callback(message_source, message, is_final=is_final, stream=True)
|
||||
|
||||
async def is_stream_output_supported(self) -> bool:
|
||||
return True
|
||||
|
||||
def register_listener(self, event_type, func): self.listeners[event_type] = func
|
||||
def unregister_listener(self, event_type, func): self.listeners.pop(event_type, None)
|
||||
async def run_async(self): pass # nothing to poll; purely webhook-driven
|
||||
async def kill(self): pass
|
||||
```
|
||||
|
||||
`_post_callback` resolves the session's callback URL, assigns the next
|
||||
`sequence`, signs the body, and enqueues an ordered, retrying POST.
|
||||
|
||||
Session→callback mapping is kept in a small in-memory dict keyed by
|
||||
`session_id` (acceptable for v1; a turn's callback URL is captured at inbound
|
||||
time so replies always have a destination even if config later changes).
|
||||
|
||||
---
|
||||
|
||||
## 11. Security considerations
|
||||
|
||||
- **Inbound route is `AuthType.NONE`** at the framework level (same as all
|
||||
webhook adapters) — the adapter **must** enforce HMAC itself. Default
|
||||
`signature_required: true`.
|
||||
- **Timestamp window** (±300s) + idempotency key blunt replay.
|
||||
- **SSRF on callback_url**: validate scheme (`https` in prod), and consider an
|
||||
allow-list / block of private CIDRs since LangBot initiates the POST. Document
|
||||
this; enforce in code where feasible.
|
||||
- **Secret storage**: secrets live in the bot's `adapter_config` like every
|
||||
other adapter credential; surfaced as `type: string`/secret in the dashboard.
|
||||
- **One secret per bot** in v1. Per-caller key rotation / multiple keys is a
|
||||
future enhancement (§13).
|
||||
|
||||
---
|
||||
|
||||
## 12. Developer Experience (explicit deliverables)
|
||||
|
||||
The whole point of a standalone adapter is that **integrating is pleasant**. v1
|
||||
ships:
|
||||
|
||||
1. **`docs/platforms/http-bot.md`** — task-oriented integration guide:
|
||||
create the bot → copy inbound URL → set secret → stand up a callback
|
||||
endpoint → send first message → handle 1→M.
|
||||
2. **Copy-paste curl** for the first message (with a working signing one-liner).
|
||||
3. **Reference clients** (≤50 LOC each) in `examples/http-bot/`:
|
||||
`client.py` (push + a Flask/Quart callback receiver) and `client.ts`.
|
||||
4. **OpenAPI fragment** `docs/http-bot-openapi.json` describing inbound +
|
||||
callback shapes, so integrators can codegen.
|
||||
5. **Local echo recipe**: a one-command callback server that prints every
|
||||
reply, so a developer sees N→1 and 1→M working in under five minutes.
|
||||
6. **Postman/Hoppscotch collection** (nice-to-have).
|
||||
|
||||
DX acceptance check: *a developer who has never seen LangBot can, from the docs
|
||||
alone, push a message and observe a multi-part reply on their callback within
|
||||
10 minutes.*
|
||||
|
||||
### Quickstart (curl)
|
||||
|
||||
```bash
|
||||
BOT=https://your-langbot/bots/2f1c....
|
||||
SECRET=supersecret
|
||||
BODY='{"session_id":"ticket-10293","message":[{"type":"Plain","text":"hello"}]}'
|
||||
TS=$(date +%s)
|
||||
SIG="sha256=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -r | cut -d' ' -f1)"
|
||||
curl -sS -X POST "$BOT" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "X-LB-Timestamp: $TS" \
|
||||
-H "X-LB-Signature: $SIG" \
|
||||
-d "$BODY"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 13. Future work
|
||||
|
||||
- **Durable outbound queue** (persist + replay across restarts; exactly-once).
|
||||
- **Per-caller API keys** with rotation and scopes (multi-tenant Space usage).
|
||||
- **Sync convenience endpoint** (§9) once core is stable.
|
||||
- **Server-Sent Events outbound option** for callers that *do* want a stream but
|
||||
not a full duplex socket — single GET, server pushes chunks.
|
||||
- **Dashboard "test console"** for `http_bot` (send a message, watch callbacks)
|
||||
mirroring the existing WebSocket debug panel.
|
||||
|
||||
---
|
||||
|
||||
## 14. Rollout / task breakdown
|
||||
|
||||
| # | Task | Touches |
|
||||
|---|---|---|
|
||||
| 1 | `http_bot.yaml` manifest + icon | `platform/sources/` |
|
||||
| 2 | `HttpBotAdapter` (inbound verify, event build, outbound queue) | `platform/sources/http_bot.py` |
|
||||
| 3 | Signing helper module (shared) | `platform/sources/` or `utils/` |
|
||||
| 4 | i18n strings (en/zh/ja) | adapter yaml + web locale |
|
||||
| 5 | Integration docs `docs/platforms/http-bot.md` | `docs/` |
|
||||
| 6 | OpenAPI fragment + reference clients | `docs/`, `examples/http-bot/` |
|
||||
| 7 | Tests: signature verify, N→1 aggregation, 1→M ordering, retry | `tests/` |
|
||||
| 8 | (opt) SSRF guard for callback_url | adapter |
|
||||
|
||||
No changes required to: the unified webhook router, the aggregator, the query
|
||||
pool, or the pipeline. That is the design's main payoff.
|
||||
|
||||
---
|
||||
|
||||
## 15. Resolved decisions
|
||||
|
||||
1. **Callback URL trust** — **config-only.** The inbound message may not carry a
|
||||
`callback_url`; replies always go to the bot-config URL. Closes the SSRF
|
||||
vector where a leaked inbound secret could redirect replies.
|
||||
2. **Session lifecycle** — **`POST /bots/<uuid>/reset`** (body `{session_id,
|
||||
session_type?}`) drops the matching session from the session manager; the
|
||||
next message starts a fresh conversation. Implemented via sub-path routing in
|
||||
`handle_unified_webhook`.
|
||||
3. **Group semantics** — for `session_type: group`, `session_id` is the group/
|
||||
launcher id; `sender.id` (and optional `sender.group_name`) identify the
|
||||
member. A Space ticket maps to one `session_id`.
|
||||
4. **Backpressure** — bounded per-session outbound queue (maxlen 1000); on
|
||||
overflow the oldest reply is dropped and a warning logged, so a persistently
|
||||
down callback can never exhaust memory.
|
||||
|
||||
### Still open / deferred (see §13)
|
||||
|
||||
- Durable outbound queue (persist + replay across restarts).
|
||||
- Per-caller API keys with rotation/scopes for multi-tenant Space usage.
|
||||
- SSE outbound option and a dashboard test console.
|
||||
@@ -0,0 +1,149 @@
|
||||
# Agent-owned Context 协议设计
|
||||
|
||||
本文档描述插件化 AgentRunner 场景下的上下文边界**设计理由**。结论先行：LangBot 不应成为最终 agentic context manager；它提供 context substrate，AgentRunner 或其背后的 runtime 自己决定如何管理历史、压缩、召回和 KV cache。
|
||||
|
||||
> 涉及的数据结构（`AgentRunContext`、`ContextAccess`、`AgentRunAPIProxy` 等）唯一定义在 [PROTOCOL_V1.md](./PROTOCOL_V1.md)。本文只讲语义和约束，不重抄 schema。
|
||||
|
||||
## 1. 设计原则
|
||||
|
||||
### 1.1 Agent 拥有上下文策略
|
||||
|
||||
不同 runner 背后的 runtime 差异很大：
|
||||
|
||||
- 官方 local-agent 可能依赖 LangBot 的模型、工具、知识库和存储。
|
||||
- Claude Code SDK / Codex 类 runtime 有自己的 session、transcript、tool loop 和上下文压缩。
|
||||
- Pi Agent SDK 或外部 agent 平台可能只需要当前事件和一个外部 conversation key。
|
||||
|
||||
因此 LangBot 不应强行决定最终传给模型的历史窗口。Host 只提供：当前事件的完整结构化信息、稳定身份和会话引用、可授权读取的 history / event / state API、sandbox/workspace 文件能力、可投影给外部 harness 的 scoped context / SDK-owned MCP bridge / resource handles、payload hard cap 和权限 guardrail。
|
||||
|
||||
### 1.2 Host 不定义通用历史窗口
|
||||
|
||||
历史窗口策略不是 AgentRunner 协议或 Query entry adapter 的核心概念。Host 只提供 history pull API、cursor、hard cap 和权限边界；runner 自己决定是否读取、读取多少、如何截断和压缩。
|
||||
|
||||
正确的问题不是"LangBot 每轮裁几轮历史给 agent"，而是：
|
||||
|
||||
- 这类 runner 是否自管 context？
|
||||
- 事件到来时 host 应 inline 哪些最小信息？
|
||||
- agent 需要更多上下文时通过什么 API 拉取？
|
||||
- host 如何保证安全、可审计和可分页？
|
||||
|
||||
### 1.3 Host 保存事实源，Agent 管理 working context
|
||||
|
||||
三类数据要分开：
|
||||
|
||||
- `EventLog`: Host 保存原始事件、工具调用、投递结果、错误和系统事件。
|
||||
- `Transcript`: Host 从 EventLog 投影出的对话视图，用于 UI、审计和按需历史读取。
|
||||
- `Working context`: Agent 本轮实际送进模型或 runtime 的上下文，由 AgentRunner 决定。
|
||||
|
||||
LangBot 不提供 host-side inline history window。简单 runner 如果需要历史窗口，应在 runner 内部通过 Host history API 拉取并裁剪。
|
||||
|
||||
## 2. Event 到来时传什么
|
||||
|
||||
默认 `AgentRunContext`（PROTOCOL_V1 §5.2）应尽量小且稳定。默认规则：
|
||||
|
||||
- Host MUST NOT inline full history by default.
|
||||
- Host SHOULD inline only current event / input and context handles.
|
||||
- Runner owns working-context assembly.
|
||||
- Runner MAY use Host history / event / state / storage API and sandbox/workspace file tools when authorized.
|
||||
- Official runners MUST consume Host infrastructure through the same public API as third-party runners.
|
||||
|
||||
### 2.1 必须 inline 的内容
|
||||
|
||||
当前 event 的类型/id/时间/source；当前输入文本和结构化内容；附件/文件/图片的 metadata、path 或 URL；actor / subject / conversation / thread / bot / workspace；delivery 能力；已授权资源列表；context cursors 和可用 API 能力；Agent/runner config。这些是 agent 决定下一步所需的最低信息。
|
||||
|
||||
### 2.2 默认不 inline 的内容
|
||||
|
||||
完整历史消息、大文件全文、大工具结果、全量知识库内容、平台原始 payload 大对象、每轮重新生成的大段 summary。这些会破坏跨进程序列化成本、泄露范围、KV cache 稳定性，也会迫使 host 替 agent 做 context 策略。
|
||||
|
||||
### 2.3 不提供 Host Inline History Window
|
||||
|
||||
`AgentRunContext` 不包含 `bootstrap` 字段。Host 不下发历史窗口，也不通过 Pipeline 配置决定窗口大小。runner 若需要类似 `recent_tail` 的策略，应在自己的 manifest/config schema 中声明参数，并在 runner 内部通过 history API 读取、裁剪和压缩。Host 只负责权限、分页、hard cap 和事实源。
|
||||
|
||||
## 3. ContextAccess 的作用
|
||||
|
||||
`ContextAccess`（PROTOCOL_V1 §5.8）是 host 交给 agent 的上下文读取入口描述，告诉 agent：当前事件位于哪条 conversation / thread、若需要更多历史从哪个 cursor 开始拉、host inline 了什么没 inline 什么、当前 run 有哪些 context API 权限。
|
||||
|
||||
## 4. Agent 如何获取更多上下文
|
||||
|
||||
所有 API 都走 `AgentRunAPIProxy`（PROTOCOL_V1 §8），由 host 用 `run_id` 校验。
|
||||
|
||||
外部 harness 不能直接访问 LangBot 资源。无论是 history、event、state、model、tool、knowledge base，还是 LangBot skills，都必须通过 SDK runtime 转发到 Host API，并由 Host 按 active `run_id`、runner identity、binding resource policy 和 caller plugin identity 校验。当前运行文件进入授权 sandbox/workspace 后，再由 runner 用 read/write/exec 类工具按需访问。harness 自己的 native tools 只属于 harness 执行环境，不能绕过 SDK runtime 访问 LangBot 内部资源。
|
||||
|
||||
### 4.1 History
|
||||
|
||||
```python
|
||||
await api.history_page(conversation_id=ctx.context.conversation_id,
|
||||
before_cursor=ctx.context.latest_cursor,
|
||||
limit=50, direction="backward", include_attachments=False)
|
||||
```
|
||||
|
||||
返回 `HistoryPage`（schema 见 PROTOCOL_V1 §8）。
|
||||
|
||||
约束：`limit` 有 host hard cap；默认只能读当前 conversation / thread；跨会话读取需 binding policy / run authorization snapshot 授权；可返回 attachment ref，不默认返回大文件内容。
|
||||
|
||||
### 4.2 Search
|
||||
|
||||
```python
|
||||
await api.history_search(query="用户之前提到的数据库连接信息",
|
||||
filters={"conversation_id": ..., "event_types": ["message.received"]},
|
||||
top_k=10)
|
||||
```
|
||||
|
||||
Search 可先用数据库全文索引，后续接 embedding recall。它是 host 检索能力，不等于 agent 的长期记忆策略。
|
||||
|
||||
### 4.3 Event / State
|
||||
|
||||
- Event API（`events.get` / `events.page`）用于读取非消息事件、工具事件、系统事件。Agent 不应把所有事件都当成 user/assistant message。
|
||||
- State API（`state.get` / `set`）是可选寄宿能力。自管 runtime 可以完全不用；依附 LangBot 的官方 runner 可以使用，例如 `external.session_id`、`summary.checkpoint`。
|
||||
|
||||
### 4.4 大文件与工具协作
|
||||
|
||||
大文件、多模态输入和工具产物不要内联进 prompt 或 tool result：message/content 里只放小文本和必要摘要；当前事件附件由 Host staged 到授权 sandbox/workspace，并在 input attachment 中给出轻量 metadata/path。工具之间传递大结果时传 sandbox path 或 attachment ref，不传完整 blob。Host 只保证当前 run 授权范围，默认不允许插件直接读任意本地路径；临时文件由 sandbox 生命周期和清理机制管理。
|
||||
|
||||
### 4.5 External harness context projection
|
||||
|
||||
外部 harness 的总体边界以 [HOST_SDK_INFRASTRUCTURE.md](./HOST_SDK_INFRASTRUCTURE.md) §4.8 为准。本节只描述 context projection 的推荐形态。
|
||||
|
||||
Claude Code、Codex、Kimi Code 这类 runtime 通常已有自己的 session、工具 loop、MCP 加载、上下文压缩和工作目录。LangBot 不应把它们改造成"host prompt assembler"，而应提供可审计的事件和资源投影。推荐 projection 形态：
|
||||
|
||||
- `agent-context.json`：结构化 JSON，包含 `run_id`、`event`、`actor`、`subject`、`input`、`delivery`、`resources`、`context`、`state`、`runtime`。
|
||||
- `LANGBOT_CONTEXT.md`：人类可读摘要。
|
||||
- `resources`：只包含本次 run 授权后的资源句柄和能力摘要，不暴露 Host 内部私有对象、secret 或资源内容。
|
||||
- `skills`：LangBot skills 不是直接投影给 harness native tool loop 的文件能力，而是**一组被授权的 tool**。发现走 `list_skills`（或 `langbot_list_assets` 增加 skills 一类），激活/注册走 `activate` / `register_skill`，包内操作走 native exec/read/write，统一通过 `ctx.resources.tools`、`AgentRunAPIProxy` 或 SDK-owned MCP bridge 暴露。Host 不向 prompt 注入 skill 索引（无 progressive-disclosure 注入）；harness 通过调用发现工具主动查询 skill 清单。`agent-context.json` 的 `skills` 字段仅作发现工具的数据来源与可选 `suggested_skill_prompt` 的输入。
|
||||
- `MCP config`：只投影 per-run、scoped 的 SDK-owned bridge 或外部 MCP 连接配置；LangBot 资源访问必须回到 SDK runtime / Host API，不允许 harness 通过自带 MCP/native tool 直接读 Host 内部资源。
|
||||
- `state pointers`：外部 session id、working directory、checkpoint 等小型 JSON 状态通过 Host state API 保存。
|
||||
|
||||
当前官方外部 harness 路径由 ACP / Claude Code / Codex 等 runner 插件承担（现状见 OFFICIAL_RUNNER_PLUGINS §7）。这类 projection 是"把 LangBot 事实源和授权资源句柄交给 harness"，不是"把 LangBot 资源本体或内部权限交给 harness"，也不是"由 LangBot 决定最终模型上下文"。
|
||||
|
||||
## 5. Runner 上下文边界
|
||||
|
||||
Host 只给当前事件、当前输入和 context handles。Runner 是否能拉取历史、事件、state 或 storage、是否能访问 sandbox/workspace 文件，以运行时 `ctx.context.available_apis` 和工具授权为准；runner 自己决定是否拉取历史、是否搜索、何时摘要、如何构造最终 prompt。
|
||||
|
||||
## 6. KV cache 友好的上下文管理
|
||||
|
||||
支持 Claude Code SDK、Codex、Pi Agent SDK 等 runtime 时，必须避免每轮由 LangBot 重组大块 prompt：
|
||||
|
||||
- 稳定 session key：`workspace/bot/binding/runner/conversation/thread`。
|
||||
- 每轮只传 delta：当前 event、attachment refs/path、少量 runtime metadata。
|
||||
- 历史 append-only：不要每轮改写同一段 history 文本。
|
||||
- Summary checkpoint 稳定：只有压缩发生时产生新 checkpoint。
|
||||
- 大文件和工具结果写入 sandbox/workspace。
|
||||
- Tool/context API schema 稳定，数据通过 API 拉取而非塞入 prompt。
|
||||
- 对自管 runtime，优先让它复用自身 session/cache，而不是强制 LangBot 每轮重放 transcript。
|
||||
- 模型窗口元信息应作为 resource/runtime metadata 暴露给 runner，由 runner 决定预算和压缩策略。
|
||||
|
||||
稳定 session key 的用途是隔离外部 runtime 的 resume/cache/state，不是改变 PROTOCOL_V1 §13 定义的 Agent 复用和 dispatch 边界。只有当某个外部 harness 的同一 native session 不支持并发 turn 时，runner 或 future runtime control plane 才应按 external session key 做 turn-level 串行化。
|
||||
|
||||
对长期运行的 external harness / daemon，推荐运行形态是 reader 与 writer 分离：一个 session reader 独占读取 stdout/SSE/native event stream，并把 native event 转成 `AgentRunResult` 或 task progress；用户输入只作为 turn write 进入该 session。当前一次性 CLI subprocess runner 可以继续在单次 `run(ctx)` 内同步收集 stdout，但后续改成长连接时不应让多个 request 同时读取同一 native stream。
|
||||
|
||||
## 7. Host guardrail
|
||||
|
||||
Agent 自管 context 不代表无限制访问。LangBot 仍必须控制：每次 run 的 active `run_id`、runner identity、当前 binding 的 resource policy、conversation / actor / subject scope、page size / sandbox file read size / API rate limit、跨会话读取权限、数据脱敏和敏感变量过滤、审计日志。Host 不负责"最佳上下文策略"，但负责"不越权、不爆内存、不不可审计"。
|
||||
|
||||
外部 harness 的 native tools、shell、MCP 或 skill 机制不构成 LangBot 资源授权边界。只要访问的是 LangBot 持有的资源，就必须经 SDK runtime 转发并接受 Host 校验；完整边界见 HOST_SDK §4.8。
|
||||
|
||||
## 8. 官方 runner 与业务编排边界
|
||||
|
||||
官方 runner 插件可以把状态寄宿在 LangBot，但必须和第三方 runner 一样通过公开 Host API 消费。LangBot core 不内置官方 agent 的业务流程（prompt 组装、tool loop、RAG 编排、summary/compaction、"local-agent 专用"状态字段）。
|
||||
|
||||
官方 local-agent 应作为"依附 LangBot 基础设施的复杂 runner 参考实现"：transcript/history 通过 `api.history_page()` / `api.history_search()` 读取，summary/checkpoint/外部 session id/用户偏好通过 `api.state_get()` / `api.state_set()` 或 storage 方法保存，图片/文件/工具大结果通过 sandbox/workspace read/write 工具访问，模型/工具/知识库通过 `api.invoke_llm()` / `api.call_tool()` / `api.retrieve_knowledge()` 调用。这样 LangBot 保持为通用 agent host，不变成内置 agent 框架。具体迁移要求见 [OFFICIAL_RUNNER_PLUGINS.md](./OFFICIAL_RUNNER_PLUGINS.md)。
|
||||
@@ -0,0 +1,227 @@
|
||||
# Agent Runner QA 指南
|
||||
|
||||
本文档是 agent-runner 插件化下一轮测试的唯一 QA 入口。它合并并取代旧的 Phase 1 验收矩阵与 2026-05-18 / 2026-05-29 两份本地 QA 报告。
|
||||
|
||||
目标不是保留完整历史流水账，而是指导测试 agent 用最小但高价值的路径判断当前分支是否仍然健康。
|
||||
|
||||
## 1. 测试边界
|
||||
|
||||
当前主线验证的是 AgentRunner Protocol v1：
|
||||
|
||||
```text
|
||||
event -> binding -> runner.run(ctx) -> result stream
|
||||
```
|
||||
|
||||
本指南验证：
|
||||
|
||||
- Host 能通过当前 Query entry adapter 进入 event-first `run(event, binding)` 主链路。
|
||||
- Runner 来自插件 registry，而不是旧内置 runner 分支。
|
||||
- `local-agent` 能消费 Host 模型、工具、知识库、history、state、sandbox 文件等基础设施。
|
||||
- 外部 harness runner（ACP / Claude Code / Codex 等直接 runner 插件）能消费 event-first context，并把外部 session 指针写回 host-owned state。
|
||||
- 错误、权限裁剪、无输出、timeout 等路径不会破坏主聊天流程。
|
||||
|
||||
本指南不验证：
|
||||
|
||||
- Runtime Control Plane v2。
|
||||
- EventGateway / EventRouter 完整落地由外部 EBA 分支联调；本指南只验证本分支 Host 底座。
|
||||
- 发布级 path isolation、secret filtering、MCP allowlist、资源配额和 workspace cleanup。
|
||||
- 所有外部服务 runner 的真实凭据联调。
|
||||
|
||||
这些属于后续能力或发布门槛，分别见 [RUNTIME_CONTROL_PLANE_V2.md](./RUNTIME_CONTROL_PLANE_V2.md) 与 [SECURITY_HARDENING.md](./SECURITY_HARDENING.md)。
|
||||
|
||||
## 2. 状态定义
|
||||
|
||||
测试报告只使用以下状态：
|
||||
|
||||
| 状态 | 含义 |
|
||||
| --- | --- |
|
||||
| PASS | 按步骤执行，用户可见行为和日志证据都满足通过条件。 |
|
||||
| FAIL | 环境可用，但行为不满足通过条件。 |
|
||||
| BLOCKED | 凭据、CLI、外部服务、测试数据或本地配置缺失导致无法执行。必须写清阻塞原因。 |
|
||||
| N/A | 当前 runner 或平台明确不支持该能力。必须引用 manifest、文档或配置说明。 |
|
||||
|
||||
不能使用“看起来正常”“大概通过”“基本没问题”等模糊状态。
|
||||
|
||||
## 3. 执行顺序
|
||||
|
||||
推荐按以下顺序执行，前一层失败时不要继续扩大测试面：
|
||||
|
||||
1. Host / SDK / runner 单测。
|
||||
2. WebUI 登录与 Pipeline Debug Chat 基础 smoke。
|
||||
3. `local-agent` 高价值场景。
|
||||
4. 外部 code-agent harness smoke。
|
||||
5. 权限和错误路径补充检查。
|
||||
6. 汇总 PASS / FAIL / BLOCKED，并给出下一步建议。
|
||||
|
||||
用户可见流程必须通过 WebUI 或真实消息平台验证。API / curl 只能作为诊断证据，不能单独让 UI case PASS。
|
||||
|
||||
## 4. 必跑基线
|
||||
|
||||
### 4.1 单测基线
|
||||
|
||||
在 LangBot 仓库运行：
|
||||
|
||||
```bash
|
||||
uv run --frozen pytest tests/unit_tests/agent
|
||||
```
|
||||
|
||||
如果本次改动只触及默认配置或 API service，也至少补跑相关目标测试，例如：
|
||||
|
||||
```bash
|
||||
uv run pytest tests/unit_tests/api/test_pipeline_service_defaults.py
|
||||
```
|
||||
|
||||
通过条件：
|
||||
|
||||
- agent 单测全 PASS，或失败项已确认与本次 agent-runner 路径无关。
|
||||
- 若失败来自 `context_builder`、`orchestrator`、`session_registry`、`resource_builder`、`plugin/handler.py` 的 run action 权限路径，不应进入 UI smoke。
|
||||
|
||||
### 4.2 环境基线
|
||||
|
||||
用 `langbot-skills` 做环境检查：
|
||||
|
||||
```bash
|
||||
cd "$LANGBOT_SKILLS_REPO"
|
||||
bin/lbs env doctor
|
||||
bin/lbs case list
|
||||
```
|
||||
|
||||
`LANGBOT_SKILLS_REPO` 指向当前工作区里的 `langbot-skills` 仓库。优先使用已有 case，而不是临时发明测试路径。
|
||||
|
||||
推荐首批 case：
|
||||
|
||||
- `webui-login-state`
|
||||
- `pipeline-debug-chat`
|
||||
- `local-agent-basic-debug-chat`
|
||||
- `local-agent-rag-debug-chat`（改动涉及 RAG / knowledge）
|
||||
- `local-agent-plugin-tool-call-debug-chat`（改动涉及 tool / resource policy）
|
||||
|
||||
## 5. WebUI 主链路 Smoke
|
||||
|
||||
### 5.1 Runner registry
|
||||
|
||||
步骤：
|
||||
|
||||
1. 打开 WebUI Pipeline 配置页。
|
||||
2. 查看 AI runner 下拉列表。
|
||||
3. 选择 `plugin:langbot/local-agent/default`。
|
||||
4. 保存并刷新页面。
|
||||
|
||||
通过条件：
|
||||
|
||||
- runner 选项来自插件 registry。
|
||||
- 保存后配置仍为 `ai.runner.id` + `ai.runner_config[id]`。
|
||||
- `runner_config` 表示 Agent/runner config，不表示插件实例状态。
|
||||
- 不读取或回写旧 `ai.runner.runner` 字段。
|
||||
- 不出现旧内置 runner stage 名（例如裸 `local-agent`）作为当前选中项或配置 surface。
|
||||
- 插件没有循环重启或 metadata 加载失败。
|
||||
|
||||
### 5.2 主聊天路径
|
||||
|
||||
步骤：
|
||||
|
||||
1. 使用绑定 `plugin:langbot/local-agent/default` 的 Pipeline。
|
||||
2. 在 Debug Chat 发送确定性普通文本。
|
||||
3. 查看 WebUI 回复和后端日志。
|
||||
|
||||
通过条件：
|
||||
|
||||
- 用户可见回复正常。
|
||||
- 后端日志显示走 `AgentRunOrchestrator` / `RUN_AGENT`。
|
||||
- 不走旧内置 local-agent 主执行分支。
|
||||
- conversation transcript 写入用户消息和助手消息。
|
||||
|
||||
## 6. `local-agent` 高价值测试
|
||||
|
||||
只保留最能覆盖架构边界的场景。
|
||||
|
||||
| ID | 场景 | 操作 | 通过条件 |
|
||||
| --- | --- | --- | --- |
|
||||
| LA-01 | 绑定 prompt | 配置 system prompt 后发送文本。 | runner 使用 `ctx.config.prompt`，不读取 `ctx.adapter.extra["prompt"]`；回复体现绑定 prompt。 |
|
||||
| LA-02 | history API | 连续两轮对话，第二轮引用第一轮 marker。 | runner 通过 Host history API 或自管上下文读取历史，不依赖 inline history window。 |
|
||||
| LA-03 | 流式 / 非流式 | 分别用支持流式和关闭流式的路径发送文本。 | 流式 UI 不重复、不空白；非流式只输出最终消息。 |
|
||||
| LA-04 | 工具调用 | 绑定测试工具，发送会触发工具的 prompt。 | `ctx.resources.tools` 只包含授权工具；工具调用 started/completed；最终回复包含工具结果。 |
|
||||
| LA-05 | RAG | 绑定测试知识库，发送命中文档的 prompt。 | `ctx.resources.knowledge_bases` 包含所选知识库；runner 通过授权 API 检索；回复使用检索内容。 |
|
||||
| LA-06 | 多模态 | 发送图片输入。 | `ctx.input.contents` 保留图片；支持视觉模型时正常处理，不支持时受控失败。 |
|
||||
| LA-07 | fallback / 错误 | 模拟 primary 模型失败或 runner 抛错。 | fallback 或 `run.failed` 行为受控；后续请求不受影响。 |
|
||||
| LA-08 | 无输出保护 | 测试 runner 完成但不产出消息。 | 不产生空白成功回复；按受控失败或明确缺陷处理。 |
|
||||
| LA-09 | steering / 运行中追加消息 | 使用支持 steering 的 runner，第一条消息触发长 run；run 未结束时在同 conversation 追加第二条消息。 | 第二条消息被 active run claim，不启动并发 run；runner 通过 `steering_pull` 看到追加输入；EventLog 有 `queued` -> `steering.injected`，若未消费则有 `steering.dropped` 终态；后续普通消息仍可处理。 |
|
||||
|
||||
Rerank、remove-think、文件输入等场景只在本次改动直接涉及时补测，不作为每轮必跑项。
|
||||
|
||||
## 7. Code-agent Harness Smoke
|
||||
|
||||
这些测试用于验证 ACP、Claude Code、Codex 这类自管 runtime 能走同一条 Host 协议路径。若目标 harness 没有 CLI/daemon、登录态、代理配置或远端 workspace，标记 BLOCKED，不要伪造 PASS。
|
||||
|
||||
Smoke 前应优先保留一层轻量单测或 fixture 测试：session 创建/复用、消息发送、结果解析、`run_id` 注入和 LangBot MCP gateway 必须有稳定测试覆盖。WebUI smoke 证明真实链路可用，但不能替代转换层和错误映射测试。
|
||||
|
||||
### 7.1 外部 harness runner
|
||||
|
||||
步骤：
|
||||
|
||||
1. 确认目标 harness（例如 ACP daemon、Claude Code 或 Codex）在对应机器上可执行且已登录。
|
||||
2. 绑定目标 runner，例如 `plugin:langbot/acp-agent-runner/default`、`plugin:langbot/claude-code-agent/default` 或 `plugin:langbot/codex-agent/default`。
|
||||
3. 配置 runner 必要字段，例如 remote target、workspace、provider、startup timeout、reuse session 等。
|
||||
4. 在 Debug Chat 执行一次确定性真实 smoke。
|
||||
5. 检查 LangBot MCP gateway、`run_id` 回填和 host-owned state。
|
||||
|
||||
通过条件：
|
||||
|
||||
- WebUI 可见回复包含预期 sentinel。
|
||||
- 发送给 harness 的消息包含当前 LangBot `run_id` 和可访问资源摘要。
|
||||
- Harness 通过 gateway 调用 `langbot_history_page`、`langbot_retrieve_knowledge` 或 `langbot_call_tool` 时必须携带正确 `run_id`；错误 run id 被拒绝。
|
||||
- `external.session_id` 写入 host-owned state。
|
||||
- 外部 harness 错误、timeout、empty output 都转成受控 `run.failed`。
|
||||
- resume 到同一 external session 时，全局锁边界符合 PROTOCOL_V1 §13。
|
||||
|
||||
### 7.2 API 型外部 runner
|
||||
|
||||
Dify、n8n、Coze、DashScope、Langflow、Tbox 等外部服务 runner 不作为每轮必跑项。只有在本次改动触及对应 runner 或凭据已经可用时执行 smoke。
|
||||
|
||||
通过条件：
|
||||
|
||||
- runner 可选，配置可保存。
|
||||
- 请求成功，或外部服务错误被清晰返回。
|
||||
- 外部服务凭据缺失时标记 BLOCKED，并记录缺失项。
|
||||
|
||||
## 8. 权限与隔离补充
|
||||
|
||||
以下优先用单测 / targeted fixture 覆盖，不要求每次通过 UI 人工构造恶意 runner。
|
||||
|
||||
| 场景 | 推荐证据 |
|
||||
| --- | --- |
|
||||
| 未授权模型调用被拒绝 | `plugin/handler.py` run action 权限测试或目标单测。 |
|
||||
| 未授权工具调用被拒绝 | `ctx.resources.tools` 与 host action 拒绝日志。 |
|
||||
| 未授权知识库检索被拒绝 | `ctx.resources.knowledge_bases` 与 host action 拒绝日志。 |
|
||||
| run_id 结束后复用被拒绝 | session registry 注销测试。 |
|
||||
| 插件身份不匹配被拒绝 | `caller_plugin_identity` mismatch 测试。 |
|
||||
| 绑定插件身份的 run_id 省略 caller identity 被拒绝 | `_validate_run_authorization(..., caller_plugin_identity=None)` 返回错误。 |
|
||||
| 未注册 Runtime 连接伪造插件身份被剥离 | SDK runtime forwarding 测试：请求自带 `caller_plugin_identity` 时，未注册连接转发前必须 `pop`，已注册连接必须覆盖为真实插件身份。 |
|
||||
| storage/state scope 越权被拒绝 | state/storage proxy 单测。 |
|
||||
| steering claim 异常不杀 consumer loop | controller 单测：无效 runner / registry 异常只让当前消息回到普通 session 槽位路径，消息消费循环继续。 |
|
||||
| steering queue 未消费有终态 | session registry / orchestrator 单测：队列有上限；run unregister 时未 pull 项写 `steering.dropped` 审计。 |
|
||||
|
||||
如果这些单测失败，不能用 WebUI 正常回复替代。
|
||||
|
||||
## 9. 证据要求
|
||||
|
||||
每轮测试报告至少记录：
|
||||
|
||||
- LangBot commit、SDK commit、相关 runner 插件 commit。
|
||||
- Pipeline UUID/name、runner id、关键 runner config 摘要。
|
||||
- WebUI 截图或 Playwright 操作记录。
|
||||
- 后端日志中对应 query id / run id 的关键行。
|
||||
- `langbot-skills` case/report 路径。
|
||||
- 外部 harness runner 的 context 文件、session id、working directory、CLI 错误摘要。
|
||||
- FAIL/BLOCKED 的复现步骤和归属仓库建议。
|
||||
|
||||
报告结论必须回答：
|
||||
|
||||
- 是否建议继续进入下一阶段测试。
|
||||
- 是否存在主聊天路径阻塞。
|
||||
- 是否只是凭据 / 外部服务 / 本机 CLI 缺失导致 BLOCKED。
|
||||
- 是否需要进入 [SECURITY_HARDENING.md](./SECURITY_HARDENING.md) 的发布级验收。
|
||||
|
||||
## 10. 历史高价值记录
|
||||
|
||||
历史高价值记录与当前 runner 验收状态见 [STATUS.md](./STATUS.md)。本指南只保留可重复执行的测试步骤和证据要求。
|
||||
@@ -0,0 +1,92 @@
|
||||
# Event Based Agent 接入设计
|
||||
|
||||
> 本文记录 EBA 如何接入当前 AgentRunner Protocol v1 / Host 底座。EventGateway、EventRouter、Event subscription/notification 由外部 EBA 分支实现并联调；本分支只保留 event-first 入口和 envelope/binding models。
|
||||
>
|
||||
> 数据结构唯一定义在 [PROTOCOL_V1.md](./PROTOCOL_V1.md)（runner 可见）与 [HOST_SDK_INFRASTRUCTURE.md](./HOST_SDK_INFRASTRUCTURE.md)（Host 内部模型）；本文只讲 EBA 语义，不重抄 schema。
|
||||
> 与当前 runner 外化分支、后续 Agent Platform / Runtime Control Plane 的边界见 [EXTENSION_SCOPE_MATRIX.md](./EXTENSION_SCOPE_MATRIX.md)。
|
||||
|
||||
本文描述 EBA 接入时，事件如何进入 LangBot、如何触发 AgentRunner，以及如何复用插件化 agent 基础设施。本分支不实现完整 EventBus / EventRouter / Platform API；这些能力正在外部 EBA 分支联调。这里的目标是把协议边界说清楚，避免当前消息入口继续绑死 Pipeline 和用户文本消息。
|
||||
|
||||
## 1. 设计目标
|
||||
|
||||
- 消息、撤回、入群、好友申请、定时任务、API 调用都能抽象为 host event。
|
||||
- EventRouter 可以根据 event type、bot、workspace、conversation、actor、subject 解析 `AgentBinding`。
|
||||
- AgentRunner 通过同一套 orchestrator 被调用。
|
||||
- 非消息事件不伪造成用户文本消息。
|
||||
- 平台动作执行通过显式 capability / permission / result type 预留，不混入普通文本回复。
|
||||
|
||||
## 2. 事件不是消息
|
||||
|
||||
`message.received` 只是事件的一种。协议不应假设：一定有用户文本、一定有 conversation history、一定要返回一条聊天消息、actor 一定等于 sender、subject 一定等于当前消息。
|
||||
|
||||
| event_type | actor | subject | input |
|
||||
| --- | --- | --- | --- |
|
||||
| `message.received` | 发消息的人 | 当前消息 | 文本、图片、文件等 |
|
||||
| `message.recalled` | 撤回操作者，未知时为系统 | 被撤回消息 | 通常为空 |
|
||||
| `group.member_joined` | 新成员或邀请人 | 群/成员关系 | 通常为空 |
|
||||
| `friend.request_received` | 申请人 | 好友申请 | 验证消息或申请理由 |
|
||||
| `schedule.triggered` | 系统 | 定时任务 | 任务 payload |
|
||||
| `api.invoked` | API caller | API request | request payload |
|
||||
|
||||
## 3. 稳定事件名
|
||||
|
||||
先保留的稳定事件名（作为插件协议的一部分保持稳定）：
|
||||
|
||||
- `message.received`
|
||||
- `message.recalled`
|
||||
- `group.member_joined`
|
||||
- `friend.request_received`
|
||||
|
||||
平台原始事件名只能进入 `ctx.event.source_event_type` / `raw_ref`，不能成为 `ctx.event.event_type` 的公共契约。
|
||||
|
||||
## 4. Event Envelope 与 Binding
|
||||
|
||||
- 入口事件用 `AgentEventEnvelope`（HOST_SDK §4.1）承载；顶层字段使用 LangBot 稳定协议名，平台原始事件名和原始 payload 放 `metadata` / `raw_ref`。
|
||||
- 触发关系用 `AgentBinding`（HOST_SDK §4.2）表达。EBA 阶段 binding 通过 `event_types`、`scope`、`filters` 决定哪些事件触发当前 bot / channel 绑定的 Agent。
|
||||
|
||||
EBA dispatch 基数、Agent 复用和 fan-out 边界以 PROTOCOL_V1 §13 为准；本节只说明外部 EBA 分支的 EventRouter 如何产出当前 v1 主线需要的 binding。
|
||||
|
||||
Binding scope 示例：workspace 全局、bot 级、platform channel 级、conversation / group / thread 级、user / actor 级。旧 Pipeline 可迁移为 `message.received` 的临时 binding source，但目标持久配置应是 Agent，不是 Pipeline。
|
||||
|
||||
Event Source 可包括：`platform_adapter`（飞书、QQ、微信、Telegram 等）、`webui`、`http_api`、`scheduler`、`system`。EventRouter 不应写死平台 adapter 的类名。
|
||||
|
||||
## 5. EventRouter 调用链
|
||||
|
||||
```text
|
||||
Platform Adapter / WebUI / API
|
||||
-> Event Gateway normalize payload
|
||||
-> EventLog append raw event
|
||||
-> EventRouter resolve one effective AgentBinding
|
||||
-> AgentRunOrchestrator.run(event, binding)
|
||||
-> AgentRunContextBuilder.build(event, binding)
|
||||
-> PluginRuntimeConnector.run_agent()
|
||||
-> AgentRunResult stream
|
||||
-> DeliveryController render / platform action
|
||||
```
|
||||
|
||||
约束：必须复用现有 orchestrator，不能为 EBA 单独实现另一套 plugin runner 调用协议；非消息事件不能绕过 resource authorization；delivery 和 platform action 走统一权限模型；外部 harness runner 也通过同一套 envelope/binding/context/result 协议接入，不为 Claude Code / Codex / Kimi 单独发明队列协议。observer / fan-out / parallel arbitration 的额外语义仍按 PROTOCOL_V1 §13 处理。
|
||||
|
||||
## 6. 平台动作执行
|
||||
|
||||
EBA 后 `action.requested`（PROTOCOL_V1 §7.3，当前仅 telemetry 不执行）将用于请求 host 执行平台动作：
|
||||
|
||||
```json
|
||||
{ "type": "action.requested",
|
||||
"data": { "action": "friend.request.accept",
|
||||
"target": {"platform": "wechat", "request_id": "..."},
|
||||
"payload": {"reason": "policy matched"} } }
|
||||
```
|
||||
|
||||
Host 必须校验：binding / platform action policy 是否授权该 action、actor / bot / workspace 是否允许、是否需要人工审批，以及当前 run session / caller identity 是否匹配。EBA 还可能预留 `delivery.requested`（请求投递到某 surface）。
|
||||
|
||||
Delivery 方面，event 不一定回复到当前聊天窗口：消息事件通常带 reply target；系统事件可能没有默认 reply target，需要 runner 返回 `action.requested` 或由 binding 的 delivery policy 决定投递位置（`DeliveryContext` 见 PROTOCOL_V1 §5.7）。
|
||||
|
||||
## 7. 与 Context 协议的关系
|
||||
|
||||
EBA 事件进入 AgentRunner 时仍遵循 [AGENT_CONTEXT_PROTOCOL.md](./AGENT_CONTEXT_PROTOCOL.md)：inline 当前事件、大 payload 用 raw/staged file ref、不默认 inline 完整 history、agent 按需通过 API 拉取、Host 保留 EventLog 和权限 guardrail。非消息事件可以被投影进 Transcript，但不能强制伪装为 user message；AgentRunner 根据 event type 自己决定是否纳入模型上下文。
|
||||
|
||||
## 8. EBA 分支联调内容
|
||||
|
||||
外部 EBA 分支负责联调 EventGateway 完整实现、EventRouter 与 BindingResolver 集成、`AgentBinding` 持久模型和 UI、`DeliveryContext` 完整实现、platform action permission model 和执行器、真实平台事件接入。
|
||||
|
||||
当前底座已完成：① 把当前 Pipeline 消息入口适配成 `message.received` event → ② 增加 `AgentBinding` 抽象，先由 current config 生成 → ③ context builder 改为从 event + binding 构造 → ④ 引入 EventLog / Transcript。外部 EBA 分支在此基础上联调：⑤ 非消息事件协议测试与真实事件来源 → ⑥ 真实 EventRouter、binding persistence / UI 和 platform action。
|
||||
@@ -0,0 +1,51 @@
|
||||
# AgentRunner 外化扩展边界矩阵
|
||||
|
||||
本文用于回答一个问题：本分支只做 AgentRunner 外化时，哪些能力已经作为扩展底座完成，哪些由外部 EBA / Agent Platform / Runtime Control Plane 分支接入，后续分支接入时应该走哪个扩展点。
|
||||
|
||||
结论：本分支不实现完整 Agent Platform，也不实现完整 EBA。EBA 完整事件网关与事件路由由外部 EBA 分支联调。本分支必须把 runner 外化的 Host / SDK 边界做干净，让外部分支只需要接入持久模型、事件路由或 runtime task，而不需要重写 `AgentRunner Protocol v1`。
|
||||
|
||||
调度基数、Agent 复用、插件实例无状态、Pipeline adapter 和 fan-out 边界的单一事实源是 [PROTOCOL_V1.md](./PROTOCOL_V1.md) §13；本矩阵只说明后续能力应该接入哪个扩展点。
|
||||
|
||||
## 1. 分支边界
|
||||
|
||||
| 范围 | 本分支职责 | 不在本分支做 |
|
||||
| --- | --- | --- |
|
||||
| AgentRunner Protocol v1 | 定义 Host 调用 runner 的稳定合同：discovery、`AgentRunContext`、result stream、Host pull API、错误和权限边界。 | 不定义 Agent Platform 的产品数据库模型；不定义 runtime task queue。 |
|
||||
| Host runner 外化底座 | 提供 `AgentEventEnvelope`、`AgentBinding` 运行投影、`run(event, binding)`、resource authorization、run-scoped session、EventLog / Transcript / State / sandbox 文件边界。 | 不实现 EventGateway、scheduler、integration provider、Agent 管控面 UI。 |
|
||||
| 当前 Pipeline 入口 | 通过 `QueryEntryAdapter` 把旧 Query / Pipeline config 投影成 event + binding，作为迁移期入口。 | 不继续把 Pipeline 当作长期 agent 配置中心。 |
|
||||
| 官方 runner 插件 | 作为协议消费者验证 local-agent / 外部 harness runner 能接入 Host 基础设施。 | 不让官方 runner 的内部实现反向决定 Host / SDK 协议形态。 |
|
||||
|
||||
## 2. 扩展矩阵
|
||||
|
||||
| 能力 | 当前分支状态 | 后续归属 | 后续接入方式 | 禁止事项 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Product `Agent` | 已有运行期 `AgentConfig` / `AgentBinding` 投影；还没有正式持久化产品对象。 | Agent Platform / binding persistence UI。 | 持久 Agent 保存 runner id、runner config、resource/state/delivery policy；运行前投影为 `AgentBinding`。 | 不把持久 Agent schema 加进 SDK 协议；插件实例边界见 PROTOCOL_V1 §13。 |
|
||||
| Bot / channel 绑定 Agent | 已有单次运行前的 `AgentBinding` 解析投影；目标调度语义见 PROTOCOL_V1 §13。 | EBA / Agent Platform。 | EventRouter 根据 bot、channel、workspace、conversation、event type 解析有效 `AgentBinding`。 | 不在本矩阵重定义 fan-out / observer 语义；需要时按 §3 新增设计。 |
|
||||
| Agent session / run | 当前只有 `run_id` 和 active `AgentRunSessionRegistry`，用于权限校验和生命周期。 | Agent Platform / Runtime Control Plane。 | 如需要可新增持久 `AgentRun` / `AgentSession` / task 表，但执行仍回到 `run(event, binding)` 或 runtime-managed 等价入口。 | 不把持久 session 字段塞进 `AgentRunContext` 顶层；不要求所有 runner 长期持有 LangBot session。 |
|
||||
| EventLog / Transcript / Sandbox files | 已完成 Host-owned store、history pull API 和 sandbox 文件边界；runner 不直接写 DB。 | 本分支持续维护底座；Agent Platform 可复用。 | 外部 EBA、scheduler、integration、runtime task 都写同一套 EventLog / Transcript；当前 run 文件通过 sandbox/workspace staging 共享。 | 不让 runner / sandbox 直接访问 Host DB；不把大 payload 内联进 prompt。 |
|
||||
| Host-owned state / storage | 已有 state snapshot、`state.updated` 处理和 State API；storage 作为授权能力保留。 | 本分支持续维护底座；Runtime / Platform 可复用。 | 外部 session id、working directory、checkpoint 等小 JSON 用 state；当前 run 大对象用 sandbox/workspace 文件。 | 不把跨轮次状态存在插件实例内；不绕过 run-scoped authorization。 |
|
||||
| EventGateway / EventRouter | 本分支只提供 event-first envelope 和 `run(event, binding)` 入口。 | EBA 分支（联调中）。 | EventGateway 规范化平台/WebUI/API/scheduler 事件；EventRouter 解析一个 binding；调用现有 orchestrator。 | 不为 EBA 新增另一套 runner 调用协议；不把非消息事件伪装成 user message。 |
|
||||
| Scheduler / Automation | 不实现。文档中只把 `scheduler` 作为 future event source。 | EBA / Agent Platform。 | 定时任务触发 `schedule.triggered` host event，复用 EventGateway -> EventRouter -> `run(event, binding)`。 | 不直接调用某个 runner 插件；不绕过 EventLog / authorization。 |
|
||||
| Integration provider | 不实现。IM platform adapter 仍是当前平台接入系统。 | EBA / Agent Platform。 | OAuth/webhook/outbound provider 应先转成 canonical host event 或 platform action，再交给 AgentRunner。 | 不把 Linear/Slack/GitHub 等 provider 私有 payload 扩散到 runner 协议顶层。 |
|
||||
| Platform action / delivery | `action.requested` 已预留但当前仅 telemetry，不执行。`DeliveryContext` 只作为上下文/策略投影。 | EBA / platform action executor。 | 后续 executor 校验 runner capability、binding policy、actor/bot/workspace 权限和审批后执行。 | 不让 runner 直接调用平台 adapter 私有 API；不把平台动作伪装成文本回复副作用。 |
|
||||
| Runtime registry / worker / task queue | 不实现。当前官方外部 harness 通过 ACP、远端 daemon、本机 subprocess 或外部 HTTP API runner 调用目标运行环境，不在本分支维护通用 worker。 | Runtime Control Plane v2。 | 第一阶段先补 Host-owned `AgentRun` / `AgentRunEvent` / run control primitives；完整 runtime registry、heartbeat、task queue、daemon claim、progress/audit 是后续可选阶段。 | 不把 heartbeat/task/warm pool 放进 Protocol v1；不让管理插件拥有 runtime/task 事实源。 |
|
||||
| Warm pool / reconcile / diagnose | 不实现。 | Runtime Control Plane v2 / deployment layer。 | 作为 task/runtime 的运维能力，围绕 Host-owned runtime/task/audit 表实现。 | 不把 runtime 运维语义写进普通 runner 协议；不把 pod/task 细节泄漏给普通 runner。 |
|
||||
| Agent memory | 不实现通用长期记忆产品层；提供 history/state/storage 和 sandbox 文件基础能力。 | Agent Platform 或具体 runner/plugin。 | 平台 memory 可通过 Host storage/state 或独立产品表实现，runner 通过授权 API 拉取。 | 不在 Host core 内置通用 agentic memory 策略；不默认把 memory 全量 inline 到 context。 |
|
||||
| External harness native session | ACP / Claude Code / Codex 等 runner 支持 external session id state handoff 和 LangBot resource projection。 | 官方 runner 后续增强；Runtime Control Plane v2 可接管执行。 | 外部 harness 调用继续走 `runner.run(ctx)`；如后续引入长连接/daemon 模式，按 external session key 串行 turn，reader 独占 native stream。 | 不把具体 provider native wire 变成 LangBot 协议；全局锁边界见 PROTOCOL_V1 §13。 |
|
||||
|
||||
## 3. 后续分支接入规则
|
||||
|
||||
外部 EBA、Agent Platform 或 Runtime Control Plane 分支接入时，默认遵守以下规则：
|
||||
|
||||
- 新入口只生产或解析 Host 内部模型：`AgentEventEnvelope`、持久 Agent 投影出的 `AgentBinding`、以及必要的 delivery/resource/state policy。
|
||||
- runner 调用仍走 `AgentRunOrchestrator.run(event, binding)`，除非 Runtime Control Plane 明确引入 runtime-managed 执行模式；即便如此，runner 可见合同仍应保持 Protocol v1。
|
||||
- Host-owned facts 继续写入 EventLog / Transcript / State，当前 run 文件继续走 sandbox/workspace；产品层可以新增更高阶视图，但不能替代这些事实源。
|
||||
- 新能力如果需要持久化，优先加 Host-owned 表或 service；不要把事实源藏在插件 storage 或 runner subprocess 内。
|
||||
- 新 result type 可以按 Protocol v1 的演进规则增加；不能用入口 adapter 私有字段绕过 schema。
|
||||
- 任何 fan-out、observer agent、parallel arbitration、platform action execution 都必须单独定义 delivery、state conflict、approval 和 audit 语义。
|
||||
|
||||
## 4. 与 Agent Platform 产品层的关系
|
||||
|
||||
这里的 Agent Platform 指面向 agent 产品层的实体拆分：`Agent` 描述可配置 agent，`Session` / `SessionMessage` 描述会话事实，`Automation` 描述自动触发，`IntegrationBinding` 描述外部集成连接，`Memory` 描述长期记忆，`WarmTask` 描述预热/后台任务。这些拆分对 LangBot 后续产品层有参考价值，但不能直接搬进本分支。
|
||||
|
||||
LangBot 当前分支的对应目标是更底层的：把 IM/WebUI/API 等入口统一投影到 Host event，把 Agent / binding 配置统一投影到 runner binding，把 runner 能力统一收束到 Protocol v1。完整 Agent Platform 可以在这个底座之上构建，而不应反过来污染本分支的 runner 外化边界。
|
||||
@@ -0,0 +1,263 @@
|
||||
# LangBot Host 与 SDK 基础设施设计
|
||||
|
||||
本文档描述 LangBot 作为 agent host 的内部能力与分层架构，以及 Host 内部模型。
|
||||
|
||||
- SDK ↔ Host 的协议数据结构（`AgentRunContext`、`AgentRunnerManifest`、`AgentRunResult`、`AgentRunAPIProxy` 等）的**唯一定义在** [PROTOCOL_V1.md](./PROTOCOL_V1.md)；本文只引用，不重抄。
|
||||
- 测试执行入口和 smoke 记录见 [AGENT_RUNNER_QA_GUIDE.md](./AGENT_RUNNER_QA_GUIDE.md)；安全发布门槛见 [SECURITY_HARDENING.md](./SECURITY_HARDENING.md)。
|
||||
- 本文定义的 Host 内部模型（`AgentEventEnvelope`、`AgentBinding`、`AgentRunnerDescriptor`）不属于 SDK 协议字段。
|
||||
|
||||
## 1. 目标
|
||||
|
||||
LangBot 要转为 agent host，而不是内置 runner 容器：
|
||||
|
||||
- 接收 IM、WebUI、API 和外部 EBA 分支 EventRouter 产生的事件。
|
||||
- 根据事件、bot、workspace、scope 解析应该调用的 Agent / agent binding。
|
||||
- 发现、校验和调用插件提供的 AgentRunner。
|
||||
- 为每次 run 提供受限资源、状态、存储、上下文引用和生命周期控制。
|
||||
- 接收 AgentRunner 返回的事件流，投递到 IM、WebUI 或其他 output surface。
|
||||
|
||||
## 2. 非目标
|
||||
|
||||
- 不把 Pipeline 当作长期架构中心。
|
||||
- 不要求所有 AgentRunner 依赖 LangBot 的上下文管理。
|
||||
- 不要求官方 local-agent 的旧行为反向塑造 host 协议。
|
||||
- 不在 host 中实现通用 agentic prompt assembler。
|
||||
- 不强制 runner 使用 LangBot state / storage；只提供可选、受控的寄宿能力。
|
||||
- 不实现 EventGateway / EventRouter：它们由外部 EBA 分支提供并联调。本分支只定义 host-side envelope/binding models 和 `run(event, binding)` 入口。
|
||||
|
||||
## 3. 分层架构
|
||||
|
||||
```text
|
||||
IM / WebUI / API / EventRouter (external EBA branch)
|
||||
|
|
||||
v
|
||||
Event Gateway (external EBA branch)
|
||||
|
|
||||
v
|
||||
AgentBindingResolver
|
||||
|
|
||||
v
|
||||
AgentRunOrchestrator
|
||||
|-- AgentRunnerRegistry
|
||||
|-- AgentResourceBuilder
|
||||
|-- AgentContextBuilder
|
||||
|-- AgentRunSessionRegistry
|
||||
|-- PersistentStateStore / EventLogStore / TranscriptStore
|
||||
|-- Sandbox / workspace file tools
|
||||
v
|
||||
Plugin Runtime / AgentRunner
|
||||
|
|
||||
v
|
||||
AgentRunResult stream
|
||||
|
|
||||
v
|
||||
Delivery / Renderer / Platform API
|
||||
```
|
||||
|
||||
目标产品模型、单绑定调度、Agent 复用、插件实例无状态和 fan-out 边界以 [PROTOCOL_V1.md](./PROTOCOL_V1.md) §13 为准。本文只说明 Host 如何把当前入口投影为内部模型。当前 Pipeline 只应接入在 Query entry adapter 位置：它可以继续产生 `message.received` 并投影出临时 `AgentConfig` / `AgentBinding`，但不应再拥有 runner 选择、上下文裁剪和业务 agent 执行的核心语义。EventGateway / EventRouter 由外部 EBA 分支实现并联调。
|
||||
|
||||
## 4. LangBot 侧能力
|
||||
|
||||
### 4.1 Event Gateway / EventRouter（External EBA Branch Integration Point）
|
||||
|
||||
> EventGateway / EventRouter 由外部 EBA 分支实现并联调，不在本分支范围。本分支只保留 event-first 入口和 envelope/binding models。
|
||||
|
||||
Event Gateway 将把入口统一成 host event（IM 平台消息、WebUI debug chat、API 触发、后续非消息事件），输出稳定的 `AgentEventEnvelope`（Host 内部模型）：
|
||||
|
||||
```python
|
||||
class AgentEventEnvelope(BaseModel):
|
||||
event_id: str
|
||||
event_type: str
|
||||
event_time: int | None
|
||||
source: str
|
||||
bot_id: str | None
|
||||
workspace_id: str | None
|
||||
conversation_id: str | None
|
||||
thread_id: str | None
|
||||
actor: ActorRef | None
|
||||
subject: SubjectRef | None
|
||||
input: AgentInput # 见 PROTOCOL_V1 §5.6
|
||||
delivery: DeliveryContext # 见 PROTOCOL_V1 §5.7
|
||||
raw_ref: RawEventRef | None
|
||||
metadata: dict[str, Any] = {}
|
||||
```
|
||||
|
||||
`AgentEventEnvelope` 是 Host 内部入口模型；投影给 runner 的是 `ctx.event`（PROTOCOL_V1 §5.4）。原始平台 payload 存为 raw event 或 staged file reference，不扩散到 runner 协议顶层。
|
||||
|
||||
**当前 adapter source**：`QueryEntryAdapter.query_to_event(query)` 从 Query 生成 `AgentEventEnvelope`。
|
||||
|
||||
### 4.2 AgentConfig 与 AgentBinding
|
||||
|
||||
`AgentConfig` 是迁移期的 Host 内部 Agent 配置投影（不暴露给 SDK）。当前 Query entry adapter 从 Pipeline config 投影出它；未来持久 Agent 也应先投影成这个运行期配置，再由 BindingResolver 结合事件和 scope 解析为 `AgentBinding`。
|
||||
|
||||
```python
|
||||
class AgentConfig(BaseModel):
|
||||
agent_id: str | None = None
|
||||
runner_id: str
|
||||
runner_config: dict[str, Any] = {}
|
||||
resource_policy: ResourcePolicy = ResourcePolicy()
|
||||
state_policy: StatePolicy = StatePolicy()
|
||||
delivery_policy: DeliveryPolicy = DeliveryPolicy()
|
||||
event_types: list[str] = ["message.received"]
|
||||
enabled: bool = True
|
||||
metadata: dict[str, Any] = {}
|
||||
```
|
||||
|
||||
`AgentBinding` 是"什么事件调用哪个 AgentRunner、带什么 Agent 配置"的 Host 内部运行投影（不暴露给 SDK）。它是 EventRouter / 当前 QueryEntryAdapter 在一次运行前解析出的有效绑定。
|
||||
|
||||
```python
|
||||
class AgentBinding(BaseModel):
|
||||
binding_id: str
|
||||
enabled: bool
|
||||
scope: BindingScope
|
||||
event_types: list[str]
|
||||
filters: list[EventFilter] = [] # EBA 阶段使用，见 EVENT_BASED_AGENT
|
||||
runner_id: str
|
||||
runner_config: dict[str, Any]
|
||||
resource_policy: ResourcePolicy
|
||||
state_policy: StatePolicy
|
||||
delivery_policy: DeliveryPolicy
|
||||
```
|
||||
|
||||
BindingResolver 的基数、fan-out 和冲突处理约束见 PROTOCOL_V1 §13；本节只定义 Host 内部投影形态。
|
||||
|
||||
**当前 adapter source**：`QueryEntryAdapter.config_to_agent_config(query, runner_id)`
|
||||
先把 current config 投影为迁移期 `AgentConfig`，再由
|
||||
`AgentBindingResolver.resolve_one(event, [agent_config])` 解析出唯一
|
||||
`AgentBinding`。Pipeline 当前只是迁移期 Agent config source（AI runner config
|
||||
→ runner_config、extension preference → resource_policy、output settings →
|
||||
delivery_policy），但新设计不再把这些字段命名为 Pipeline 专属概念。
|
||||
|
||||
### 4.3 AgentRunnerRegistry
|
||||
|
||||
Registry 收集 runner descriptor（来自插件 runtime、开发期本地插件）：
|
||||
|
||||
```python
|
||||
class AgentRunnerDescriptor(BaseModel):
|
||||
id: str
|
||||
source: Literal["plugin"]
|
||||
label: I18nObject
|
||||
description: I18nObject | None = None
|
||||
plugin_author: str
|
||||
plugin_name: str
|
||||
runner_name: str
|
||||
capabilities: AgentRunnerCapabilities # 见 PROTOCOL_V1 §4.3
|
||||
permissions: AgentRunnerPermissions # 见 PROTOCOL_V1 §4.4
|
||||
config_schema: list[DynamicFormItemSchema]
|
||||
plugin_version: str | None = None
|
||||
raw_manifest: dict[str, Any] = {}
|
||||
```
|
||||
|
||||
职责：调用 `plugin_connector.list_agent_runners()` 拉取 runner、校验 typed `AgentRunnerManifest`、输出 descriptor、缓存 discovery 结果并提供 `refresh()`。单个插件 manifest 失败只记 warning，不影响其它 runner。`plugin:author/name/runner` 是稳定 id 格式；插件实例边界见 PROTOCOL_V1 §13。
|
||||
|
||||
Host 内置 runner / adapter 不能作为 `AgentRunnerDescriptor.source` 绕过插件
|
||||
runtime、`run_id`、`ctx.resources` 和 `AgentRunAPIProxy` 权限链。若需要
|
||||
开发期调试 adapter，应放在 Host 内部测试入口，不进入可选 runner 列表。
|
||||
|
||||
刷新触发点：插件安装/卸载/升级/重启后；Pipeline metadata 请求时发现缓存为空；可选 TTL（优先保证正确性）。
|
||||
|
||||
### 4.4 AgentRunOrchestrator
|
||||
|
||||
Orchestrator 是唯一运行入口：
|
||||
|
||||
```text
|
||||
run(event, binding)
|
||||
-> resolve runner descriptor
|
||||
-> build resources
|
||||
-> build context
|
||||
-> register run session
|
||||
-> call plugin runtime
|
||||
-> normalize result stream
|
||||
-> update state
|
||||
-> unregister run session
|
||||
```
|
||||
|
||||
它负责：`run_id` 生成和生命周期、timeout/deadline/cancellation、插件异常隔离、result schema 校验和大小限制、`state.updated` 处理、delivery backpressure 和 telemetry。
|
||||
|
||||
典型 run 时序：
|
||||
|
||||
```text
|
||||
QueryEntryAdapter / EventRouter
|
||||
-> AgentRunOrchestrator.run(event, binding)
|
||||
-> AgentRunnerRegistry.resolve(runner_id)
|
||||
-> AgentResourceBuilder.freeze_snapshot(binding, event)
|
||||
-> AgentRunSessionRegistry.register(run_id, runner_id, snapshot)
|
||||
-> AgentContextBuilder.build(event, binding, snapshot)
|
||||
-> PluginRuntimeConnector.run_agent(ctx)
|
||||
-> AgentRunAPIProxy action
|
||||
-> validate active run session + caller identity + snapshot
|
||||
-> Host API / Store
|
||||
<- AgentRunResult stream
|
||||
-> apply state.updated to PersistentStateStore
|
||||
-> write message.completed to Transcript
|
||||
-> keep current-run files and large tool outputs in sandbox/workspace
|
||||
-> render delivery or raise RunnerExecutionError
|
||||
-> AgentRunSessionRegistry.unregister(run_id)
|
||||
```
|
||||
|
||||
`run_from_query()` 保留为 Query entry adapter 入口，但内部转换成 event + binding 后走统一 `run()`。约束：`ChatMessageHandler` 不解析 `plugin:*`、不实例化 wrapper、不知道 runner 组件细节；`PipelineService` 从 registry 读取 metadata，不直接访问插件 runtime；跨请求持久化状态必须走授权 storage / 外部服务。
|
||||
|
||||
### 4.5 Resource Authorization
|
||||
|
||||
LangBot 在每次 run 前生成 `ctx.resources`（PROTOCOL_V1 §6），来自 manifest permissions 与 binding policy 的交集：
|
||||
|
||||
1. `descriptor.permissions` 声明 runner 需要的 LangBot 资源访问上限。
|
||||
2. binding / resource policy 允许的资源范围。
|
||||
3. Agent/runner config 中选择的模型、知识库、文件等资源。
|
||||
4. 当前 event / actor / bot / workspace 的实际权限。
|
||||
5. `ctx.context.available_apis` 暴露的 pull API 能力。
|
||||
|
||||
这次裁剪结果必须冻结为 run-scoped authorization snapshot，并由
|
||||
`AgentRunSessionRegistry` 按 `run_id` 保存。`ctx.resources` 是投影给 runner
|
||||
看的同一份授权结果；运行期每个 proxy action 只依据该 snapshot 校验 active
|
||||
run session、caller plugin identity、resource id、scope、payload size、rate
|
||||
limit 和 deadline。Handler 不应重新执行授权裁剪，否则 build-time 与 runtime
|
||||
授权逻辑会漂移。
|
||||
|
||||
SDK 侧本地校验只用于开发体验，host 侧 run authorization snapshot 才是安全边界。`spec.capabilities` 只帮助 Host 判断 runner 是否需要 tool / knowledge 等资源投影，不能替代 permissions 或 binding policy。skill 不由独立 capability 决定是否投影——它通过统一 tool 授权（`resource_policy.allowed_tool_names`）消费，`skill_authoring` 仅作为「一键授权这组 skill tool + sandbox」的便捷开关。
|
||||
|
||||
资源裁剪应通用，不写死 local-agent。selector 与资源的映射示例：`model-fallback-selector` → primary/fallback LLM、`llm-model-selector` → LLM、`rerank-model-selector` → rerank 模型、`knowledge-base-multi-selector` → 知识库；新增 selector 时在 resource builder 中统一扩展。
|
||||
|
||||
构造 `ctx.resources.tools` 时，Host 一次塞齐每个工具的完整 schema（`ToolResource.parameters`），runner 不需再逐个 `get_tool_detail` 拉取，减少 N 次往返。
|
||||
|
||||
执行/文件/skill/MCP 等能力的接入方向：先由 Host / sandbox 封装成普通 scoped tool，再通过 `ctx.resources.tools` 和 SDK runtime 转发进入 runner；runner 不应识别或硬编码执行环境 provider。外部 harness 的 native tools 不能直接访问 LangBot 资源。skill 的整个生命周期都走统一 tool：发现走 `list_skills` / `langbot_list_assets`，激活/注册走 `activate` / `register_skill`，包内操作走 native exec/read/write——runner 不需要独立的 skill 渲染或门控。
|
||||
|
||||
### 4.6 State / Storage
|
||||
|
||||
LangBot 可提供 host-owned state 让 runner 寄宿状态（conversation / actor / subject / runner / binding / workspace state），但**不是强制**。Host 只需提供：授权开关、scope key、get/set/list/delete API（见 PROTOCOL_V1 §8）、持久化 backend、审计和清理策略。外部 agent runtime 可维护自己的 session 和 memory。进程内 state store 只能作为过渡实现，不能作为正式生产语义。
|
||||
|
||||
部分 host-owned state 由 Host 自身直接写：例如 `activate` tool 在 Host 侧执行时，把已激活 skill 写入 conversation scope 的 `host.activated_skills`。host 直接写与 runner `state.updated` 写到同一 key 时按 **last-write-wins** 合并，runner 可覆盖。
|
||||
|
||||
### 4.7 EventLog / Transcript / Sandbox Files（事实源）
|
||||
|
||||
- `EventLog`: durable append-only，保存原始事件、系统事件、工具调用、投递结果、错误。
|
||||
- `Transcript`: 从 EventLog 投影出的对话视图，用于 UI、审计和按需历史读取。
|
||||
- `Sandbox / workspace files`: 当前 run 的上传文件、平台附件、工具大结果和临时产物。Host 负责 staging 与授权边界，runner 通过 read/write/exec 类工具按需访问。
|
||||
|
||||
三类数据与 working context 的边界、读取约束见 [AGENT_CONTEXT_PROTOCOL.md](./AGENT_CONTEXT_PROTOCOL.md)。AgentRunner 可读取这些能力，但不被迫使用 LangBot 作为唯一记忆系统。
|
||||
|
||||
### 4.8 External harness resource projection
|
||||
|
||||
Claude Code、Codex、Kimi Code 等外部 harness runner 可能不直接调用 LangBot 的 model/tool loop，而是把 LangBot 事件和授权资源句柄投影到自己的 harness 执行。Host 侧仍保持统一边界：Host 负责构造 event-first context、资源授权、state/storage、EventLog/Transcript、sandbox/workspace 文件边界和审计；Host 或 binding policy 决定哪些 MCP bridge、skill-backed tool、sandbox path、history/state 句柄可投影给 runner；runner plugin 把 scoped projection 转成目标 harness 可消费形式；所有 LangBot 资源访问必须经 SDK runtime / `AgentRunAPIProxy` / SDK-owned MCP bridge 转发并接受 Host 校验；外部 harness 负责自己的 native session、tool loop、压缩、权限模式和 resume，但不能用 native tools 绕过 Host 授权。
|
||||
|
||||
投影的具体形态（context 文件、resource handles、LangBot MCP gateway、state pointers）见 AGENT_CONTEXT_PROTOCOL §4.5；当前 code-agent harness runner 形态见 OFFICIAL_RUNNER_PLUGINS §7。发布级隔离要求见 SECURITY_HARDENING。
|
||||
|
||||
## 5. SDK 侧协议
|
||||
|
||||
SDK 组件入口如下；所有数据结构定义见 PROTOCOL_V1。
|
||||
|
||||
```python
|
||||
class AgentRunner(BaseComponent):
|
||||
__kind__ = "AgentRunner"
|
||||
|
||||
@classmethod
|
||||
def get_config_schema(cls) -> list[dict]: ...
|
||||
|
||||
async def run(self, ctx: AgentRunContext) -> AsyncGenerator[AgentRunResult, None]: ...
|
||||
# ctx: PROTOCOL_V1 §5.2 ; AgentRunResult: PROTOCOL_V1 §7
|
||||
```
|
||||
|
||||
- Manifest / capabilities / effective access：PROTOCOL_V1 §4。Capabilities 来自组件 manifest 的 `spec.capabilities`，不是 SDK 基类 classmethod。
|
||||
- `AgentRunContext`：PROTOCOL_V1 §5.2。`messages` / `bootstrap` 不是协议字段。
|
||||
- `AgentRunResult`：PROTOCOL_V1 §7。
|
||||
- `AgentRunAPIProxy`：PROTOCOL_V1 §8，是 runner 访问 host 能力的唯一入口，所有请求带 `run_id`。
|
||||
@@ -0,0 +1,138 @@
|
||||
# 官方 AgentRunner 插件迁移计划
|
||||
|
||||
本文档描述内置 `RequestRunner` 迁出 LangBot 后，官方 runner 插件如何组织、迁移和验收。它是 [HOST_SDK_INFRASTRUCTURE.md](./HOST_SDK_INFRASTRUCTURE.md) 和 [AGENT_CONTEXT_PROTOCOL.md](./AGENT_CONTEXT_PROTOCOL.md) 的下游落地计划，不是 LangBot 宿主协议的设计前提。QA 入口和 smoke 记录见 [AGENT_RUNNER_QA_GUIDE.md](./AGENT_RUNNER_QA_GUIDE.md)。
|
||||
|
||||
官方 `local-agent` 可以外移，也可以重写。设计重点不是保留旧内置 runner 的内部结构，而是验证一个依附 LangBot host 基础设施的官方 agent 能否完整工作。同时，LangBot host 协议必须服务 Claude Code SDK、Codex、Pi Agent SDK、外部 Agent 平台等自管 context/runtime 的 runner，不能被官方插件的实现细节绑死。
|
||||
|
||||
## 1. 仓库组织
|
||||
|
||||
官方 runner 插件与 LangBot 主仓库、SDK 仓库以不同节奏迭代：LangBot 主仓库只维护宿主协议和调度，SDK 仓库维护 AgentRunner 组件和 runtime 协议，官方 runner 插件承载业务 runner 的具体实现和第三方平台适配。
|
||||
|
||||
当前推荐"官方插件可独立发布，必要时共享 SDK helper"。开发期采用本地多目录布局：
|
||||
|
||||
```text
|
||||
langbot-app/
|
||||
langbot-local-agent/ # plugin:langbot/local-agent/default
|
||||
manifest.yaml
|
||||
components/agent_runner/default.{yaml,py}
|
||||
langbot-agent-runner/ # 外部服务 runner 仓库
|
||||
acp-agent-runner/ claude-code-agent/ codex-agent/ dify-agent/ n8n-agent/ ...
|
||||
```
|
||||
|
||||
后续可聚合进 monorepo，也可继续独立发布——这个选择不影响协议设计。重复逻辑优先沉淀到 SDK 或明确的共享 helper 包，不要把宿主私有结构泄漏给插件。旧 `src/langbot/pkg/provider/runners/*` 只作为历史行为对齐基准；当前未发布分支不提供旧内置 runner 的运行时 fallback。
|
||||
|
||||
## 2. 插件命名和 runner id
|
||||
|
||||
| 旧 runner | 官方插件 | runner id |
|
||||
| --- | --- | --- |
|
||||
| `local-agent` | `langbot/local-agent` | `plugin:langbot/local-agent/default` |
|
||||
| `dify-service-api` | `langbot/dify-agent` | `plugin:langbot/dify-agent/default` |
|
||||
| `n8n-service-api` | `langbot/n8n-agent` | `plugin:langbot/n8n-agent/default` |
|
||||
| `coze-api` | `langbot/coze-agent` | `plugin:langbot/coze-agent/default` |
|
||||
| - | `langbot/acp-agent-runner` | `plugin:langbot/acp-agent-runner/default` |
|
||||
| - | `langbot/claude-code-agent` | `plugin:langbot/claude-code-agent/default` |
|
||||
| - | `langbot/codex-agent` | `plugin:langbot/codex-agent/default` |
|
||||
| `dashscope-app-api` | `langbot/dashscope-agent` | `plugin:langbot/dashscope-agent/default` |
|
||||
| `deerflow-api` | `langbot/deerflow-agent` | `plugin:langbot/deerflow-agent/default` |
|
||||
| `langflow-api` | `langbot/langflow-agent` | `plugin:langbot/langflow-agent/default` |
|
||||
| `tbox-app-api` | `langbot/tbox-agent` | `plugin:langbot/tbox-agent/default` |
|
||||
| `weknora-api` | `langbot/weknora-agent` | `plugin:langbot/weknora-agent/default` |
|
||||
|
||||
每个插件可后续提供多个 runner，但迁移目标的默认 runner 统一叫 `default`。
|
||||
|
||||
## 3. 迁移批次
|
||||
|
||||
- **Batch 1（打通协议）**：`local-agent`（能力最完整基准）、`acp-agent-runner` / `claude-code-agent` / `codex-agent`（外部 code-agent harness 路径）、`dify-agent`（传统 service API runner）。
|
||||
- **Batch 2（外部 workflow）**：`n8n-agent`、`langflow-agent`（webhook/workflow 输入输出、timeout、外部 conversation id）。
|
||||
- **Batch 3（平台 Agent API）**：`coze-agent`、`dashscope-agent`、`tbox-agent`、`deerflow-agent`、`weknora-agent`（平台特有响应格式、引用资料、文件/图片输入、外部 thread/session 状态）。
|
||||
|
||||
## 4. 每个官方插件的组件要求
|
||||
|
||||
每个插件至少包含一个 `AgentRunner` 组件，manifest 示例：
|
||||
|
||||
```yaml
|
||||
apiVersion: langbot/v1
|
||||
kind: AgentRunner
|
||||
metadata:
|
||||
name: default
|
||||
label: { en_US: Dify Agent, zh_Hans: Dify Agent }
|
||||
description:
|
||||
en_US: Run a Dify application as a LangBot AgentRunner.
|
||||
zh_Hans: 将 Dify 应用作为 LangBot AgentRunner 运行。
|
||||
spec:
|
||||
config: []
|
||||
capabilities: # 字段语义见 PROTOCOL_V1 §4.3
|
||||
streaming: true
|
||||
execution:
|
||||
python: { path: ./main.py, attr: DefaultAgentRunner }
|
||||
```
|
||||
|
||||
## 5. local-agent 插件方向
|
||||
|
||||
`local-agent` 是官方插件中能力最完整的消费者，但不是宿主协议的设计中心。它需要证明：一个主要依附 LangBot host 能力的 agent runner 可以通过公开协议完成模型、工具、知识库、状态、history、sandbox 文件访问、上下文压缩和消息投递。
|
||||
|
||||
迁移或重写需覆盖旧内置 runner 的用户可见能力：model primary/fallback 选择、prompt、knowledge-bases、rerank-model、rerank-top-k、function calling、streaming、multimodal input、conversation history、monitoring metadata。
|
||||
|
||||
责任边界与 Host API 消费方式见 AGENT_CONTEXT_PROTOCOL §8。关键约束：
|
||||
|
||||
- 从 `ctx.config` 读取静态绑定 `prompt`，**不**读取 `ctx.adapter.extra["prompt"]`；不消费 Query entry adapter 生成的历史窗口。
|
||||
- 通过 `AgentRunAPIProxy.history` 拉取 transcript，而不是依赖 host 每轮强塞历史窗口。
|
||||
- `ctx.input.contents` 保留图片/文件等多模态内容；RAG 只替换/插入文本部分，不丢图片/文件。
|
||||
- 不能绕过 `ctx.resources` 调用未授权模型、工具或知识库。
|
||||
- manifest 声明功能能力、LangBot 资源 permissions 和配置表单；实际授权来自 manifest permissions 与 binding resource policy、runner config、`ctx.context.available_apis` 和 Host run session snapshot 的交集。
|
||||
|
||||
### 5.1 Native Execution / Skills 后续接入
|
||||
|
||||
本阶段不把 sandbox/skills 做成 AgentRunner 协议字段。后续 sandbox/skills 分支合并后，命令执行、文件操作、skill、MCP managed process 应先由 Host / sandbox 封装成 scoped tools，再通过 `ctx.resources.tools` 和 SDK runtime 转发暴露给 runner。这让 local-agent 只消费授权后的 Host 基础设施，而不是直接持有宿主机执行能力。
|
||||
|
||||
## 6. 外部 runner 插件要求
|
||||
|
||||
外部平台 runner 迁移遵循：旧配置字段尽量保持同名便于 migration 复制；输出统一转换为 `AgentRunResult`；外部 API timeout 从 runner config 读取；平台 conversation id 存 plugin storage 或 context runtime state，不依赖 LangBot 内置 conversation uuid 私有结构；流式按平台能力声明，没有流式就只发 `message.completed`。
|
||||
|
||||
### 6.1 Code-agent harness runner
|
||||
|
||||
Claude Code、Codex、Kimi Code 这类 runner 不一定通过 LangBot 的模型/工具 loop 执行，可以依赖自己的 harness，但仍必须遵守统一 Host 边界。总体边界见 [HOST_SDK_INFRASTRUCTURE.md](./HOST_SDK_INFRASTRUCTURE.md) §4.8；context projection 形态见 [AGENT_CONTEXT_PROTOCOL.md](./AGENT_CONTEXT_PROTOCOL.md) §4.5；发布级要求见 [SECURITY_HARDENING.md](./SECURITY_HARDENING.md)。
|
||||
|
||||
本文件只补充官方 runner 的实现要求：输入来自 `ctx.event` / `ctx.input`，不依赖 Pipeline 私有 `Query`；外部 session id / workspace / checkpoint 写入 Host state 或 plugin storage；插件实例边界见 PROTOCOL_V1 §13；CLI / subprocess runner 必须处理 timeout、取消、空输出、非零退出和 stderr 映射。
|
||||
|
||||
实现结构应把 provider-native output 解析与 LangBot result stream 组装分开：Claude stream-json、Codex JSONL、Kimi / OpenCode 事件等只在 runner adapter 内解析，输出统一归一为 `AgentRunResult`（`message.completed` / `message.delta`、`state.updated`、`run.completed` / `run.failed`）。文件和工具大结果留在当前 run 的 sandbox/workspace，通过消息 metadata、attachment ref 或 path 指向。未知 native event 不应导致 run 崩溃；应记录诊断 metadata 或 warning。新增 harness 时优先补 native fixture -> `AgentRunResult` 的转换测试，再接 WebUI smoke。
|
||||
|
||||
并发约束应按外部 session 粒度表达，而不是按 Agent / runner id / 插件实例表达；Agent 复用和全局锁边界见 PROTOCOL_V1 §13。若 runner 使用 `external.session_id` / `thread_id` resume 到同一 native session，且该 harness 不支持并发 turn，runner 应按稳定 external session key 串行写入；一次性 subprocess runner 可以只在单次 `run(ctx)` 内处理，长连接/daemon runner 则应采用 reader 独占 native stream、turn writer 串行写入的结构。
|
||||
|
||||
### 6.2 LangBot MCP gateway
|
||||
|
||||
外部 harness 不能直接持有进程内的 `plugin_runtime_handler`，也不能用自己的 native tools 直接访问 LangBot 资源。外部 harness runner 应通过稳定 HTTP MCP gateway 或 SDK-owned bridge 把 harness 的工具请求转回 SDK runtime / Host API：
|
||||
|
||||
- Gateway 由 runner 插件启动，暴露稳定的 `langbot_history_page`、`langbot_retrieve_knowledge`、`langbot_call_tool` 等最小工具面。
|
||||
- Harness 每次调用必须携带当前 LangBot `run_id`；Host 仍按 run session、caller identity 和授权快照校验。
|
||||
- Gateway 只转发 LangBot 资产访问，不承担外部 harness 的文件、进程或 native tool 权限边界。
|
||||
|
||||
第一批工具保持很小：history page、knowledge retrieve、authorized tool call。新增工具必须先有 Host action 权限与 run-scoped authorization，再由 gateway 投影。
|
||||
|
||||
## 7. Code-agent harness runner 当前形态
|
||||
|
||||
外部 code-agent harness 由直接 runner 插件承接，例如 `acp-agent-runner`、`claude-code-agent`、`codex-agent`，每个 runner 负责把目标 harness 的 native session、workspace、MCP bridge 和输出事件转换为统一 `AgentRunResult`。本地 smoke 验收入口与记录见 [AGENT_RUNNER_QA_GUIDE.md](./AGENT_RUNNER_QA_GUIDE.md)。
|
||||
|
||||
当前形态：
|
||||
|
||||
- Runner ID 示例：`plugin:langbot/acp-agent-runner/default`、`plugin:langbot/claude-code-agent/default`、`plugin:langbot/codex-agent/default`。
|
||||
- Runner 可通过 ACP、远端 daemon、本机 subprocess 或外部 HTTP API 调用 harness；harness 的安装、登录态、workspace 和 provider-native 权限由该运行环境负责。
|
||||
- Runner 会把当前 LangBot `run_id`、可访问资源摘要和 gateway 使用规则注入本次消息；harness 通过 gateway 回填 `run_id` 后访问 LangBot 资产。
|
||||
- 外部 session id / workspace / checkpoint 写回 Host state 或 plugin storage，后续轮次可复用目标 harness 会话。
|
||||
|
||||
### 7.1 当前限制
|
||||
|
||||
这不是发布级安全边界实现；LangBot 只约束 LangBot 持有资产的访问，外部 harness 的文件、进程、workspace、provider-native MCP 和模型凭据由对应 runner 的运行环境承担。当前 `run_id` 可由系统提示词、ACP metadata 或 runner 自有 session metadata 传递给 harness 并由 gateway 校验。runtime 管控面方向见 [RUNTIME_CONTROL_PLANE_V2.md](./RUNTIME_CONTROL_PLANE_V2.md)。
|
||||
|
||||
## 8. 发布和安装策略
|
||||
|
||||
最终 LangBot 安装/升级时需保证官方 runner 插件可用，可选方案：首次启动检测缺失并提示安装；打包发行版预装；migration 前检查插件存在性。当前分支未发布，因此不把历史配置兼容或旧内置 runner fallback 写入运行时协议面。建议顺序：开发阶段用本地路径插件 → 发布前支持 marketplace 安装 → 若发布升级需要迁移历史配置，再在 release gate 中实现一次性 migration 并要求官方插件已可用。
|
||||
|
||||
## 9. 验收标准
|
||||
|
||||
- 每个目标 runner 都有对应官方 AgentRunner 插件和稳定 runner id；当前配置只使用 `ai.runner.id` + `ai.runner_config[id]`。
|
||||
- LangBot 主聊天路径不再通过 `RequestRunner` 执行业务 runner。
|
||||
- 官方插件测试覆盖非流式、流式、错误、timeout、配置缺失。
|
||||
- `local-agent` 能完成模型 fallback、tool calling、知识库检索、多模态输入、静态绑定 prompt 消费、history API 拉取、rerank。
|
||||
- 外部 code-agent harness runner 能消费 event-first context、投影 scoped resources、保存 external session state，并通过 WebUI Debug Chat smoke。
|
||||
- `local-agent` 覆盖旧内置 runner 的用户可见核心能力；代码结构和运行路径不需要相同。
|
||||
@@ -0,0 +1,736 @@
|
||||
# LangBot AgentRunner Protocol v1
|
||||
|
||||
本文档是 LangBot Host 与插件 SDK / Runtime / AgentRunner 之间协议合同的**唯一规范来源（single source of truth）**。
|
||||
|
||||
- 本文件描述当前 Protocol v1 稳定合同，不混入验收流水。当前实现状态见 [STATUS.md](./STATUS.md)，测试执行入口见 [AGENT_RUNNER_QA_GUIDE.md](./AGENT_RUNNER_QA_GUIDE.md)，安全发布门槛见 [SECURITY_HARDENING.md](./SECURITY_HARDENING.md)。
|
||||
- 本文件之外的任何文档**不得重新定义这里的数据结构**，只能引用，例如"见 PROTOCOL_V1 §4.2"。
|
||||
- Host 内部模型（`AgentEventEnvelope`、`AgentBinding`、Descriptor、各 Store）不属于 SDK 协议，定义在 [HOST_SDK_INFRASTRUCTURE.md](./HOST_SDK_INFRASTRUCTURE.md)。
|
||||
|
||||
## 1. 协议目标
|
||||
|
||||
Protocol v1 只解决四件事：
|
||||
|
||||
- LangBot 如何发现插件提供的 AgentRunner。
|
||||
- LangBot 如何把一次事件调用封装成 `AgentRunContext`。
|
||||
- AgentRunner 如何以事件流形式返回运行结果。
|
||||
- AgentRunner 如何通过受限 API 访问 LangBot host 能力。
|
||||
|
||||
Protocol v1 **不定义**：
|
||||
|
||||
- LangBot 内部如何持久化 `AgentBinding`（见 HOST_SDK）。
|
||||
- AgentRunner 内部如何组装 prompt、压缩历史、管理 memory（见 [AGENT_CONTEXT_PROTOCOL.md](./AGENT_CONTEXT_PROTOCOL.md)）。
|
||||
- 官方 runner 的具体实现（见 [OFFICIAL_RUNNER_PLUGINS.md](./OFFICIAL_RUNNER_PLUGINS.md)）。
|
||||
- Pipeline 的长期配置模型。
|
||||
- 发布级安全 hardening 的完整实现（见 [SECURITY_HARDENING.md](./SECURITY_HARDENING.md)）。
|
||||
|
||||
## 2. 参与方
|
||||
|
||||
| 名称 | 职责 |
|
||||
| --- | --- |
|
||||
| LangBot Host | 事件入口、绑定解析、权限、资源、存储、生命周期、结果投递。 |
|
||||
| Plugin Runtime | 加载插件，响应 Host 的 runner discovery 和 run 调用。 |
|
||||
| AgentRunner | 插件提供的 agent 执行组件。 |
|
||||
| AgentRunAPIProxy | AgentRunner 访问 Host 能力的受限 API。 |
|
||||
| AgentBinding | Host 内部的事件到 runner 绑定配置，不直接暴露给 SDK（见 HOST_SDK §4.2）。 |
|
||||
|
||||
产品层的 `Agent` 替代旧 Pipeline 承载 agent 配置：bot / IM channel
|
||||
绑定一个 Agent，一个 Agent 可以被多个 bot / channel 复用。Host 内部的
|
||||
`AgentBinding` 是一次事件运行前解析出的有效绑定，只影响 Host 构造出的
|
||||
`ctx.config`、`ctx.resources`、`ctx.context` 和 `ctx.delivery`。SDK 不需要知道
|
||||
Agent / binding 的持久化形态。
|
||||
|
||||
外部 harness runner（Claude Code、Codex、Kimi Code 等）也是 `AgentRunner`：它们消费 event-first `AgentRunContext`、返回 `AgentRunResult`，并通过 Host 授权的 state/storage API 保存跨轮次指针；当前运行文件和工具大结果进入 sandbox/workspace。它们内部可以继续使用自己的 session、tool loop、MCP、上下文压缩和权限模型。
|
||||
|
||||
## 3. 协议演进
|
||||
|
||||
当前 AgentRunner 合同不暴露显式 `protocol_version` 字段。协议演进先按字段级兼容规则处理：
|
||||
|
||||
- 新增可选字段保持向后兼容。
|
||||
- 删除字段或改变既有字段语义，需要在 SDK 发布前完成；发布后应走新的显式兼容方案。
|
||||
- 结果流演进：Host **必须忽略未知 result type 并记录 warning**（除非该 type 明确要求强校验）。SDK envelope 接收入站未知 `type` 字符串，runner 侧可按原字符串转发或忽略；新增 result type 不提升大版本。
|
||||
- SDK 入站 context 类实体偏宽松，用于兼容 Host 附加的非核心字段；manifest、result payload、page/result 返回与错误模型偏严格，未知字段默认禁止。安全边界仍在 Host，SDK 校验只提升开发体验。
|
||||
|
||||
## 4. Discovery 协议
|
||||
|
||||
### 4.1 LIST_AGENT_RUNNERS
|
||||
|
||||
Host 调用 Plugin Runtime 获取当前插件暴露的 runner 列表，请求无额外 payload。返回：
|
||||
|
||||
```python
|
||||
class ListAgentRunnersResponse(BaseModel):
|
||||
runners: list[AgentRunnerDiscovery]
|
||||
|
||||
class AgentRunnerDiscovery(BaseModel):
|
||||
plugin_author: str
|
||||
plugin_name: str
|
||||
runner_name: str
|
||||
manifest: AgentRunnerManifest
|
||||
```
|
||||
|
||||
`manifest` 是 SDK typed `AgentRunnerManifest`，由 Runtime 从插件组件 manifest 解析并校验后返回。`plugin_author` / `plugin_name` / `runner_name` 保留为 transport 寻址字段；Host 以它们生成稳定 runner id，并把 `manifest.id` 校验为 `plugin:author/name/runner`。单个 runner manifest 解析失败时 Runtime/Host 记录 warning 并跳过该 runner，不影响同一插件或其它插件的 runner discovery。
|
||||
|
||||
### 4.2 AgentRunnerManifest
|
||||
|
||||
这里的 manifest 指 Runtime 返回给 Host 的 typed runner manifest：
|
||||
|
||||
```python
|
||||
class AgentRunnerManifest(BaseModel):
|
||||
id: str
|
||||
name: str
|
||||
label: I18nObject
|
||||
description: I18nObject | None = None
|
||||
capabilities: AgentRunnerCapabilities = AgentRunnerCapabilities()
|
||||
permissions: AgentRunnerPermissions = AgentRunnerPermissions()
|
||||
config_schema: list[DynamicFormItemSchema] = []
|
||||
metadata: dict[str, Any] = {}
|
||||
```
|
||||
|
||||
- runner id 由 Host 生成，格式 `plugin:author/name/runner`。
|
||||
- `name` 是插件内 runner 名称，例如 `default`。
|
||||
- `config_schema` 只描述绑定配置表单，不代表插件实例状态。
|
||||
- `capabilities` 是 Host 用于 UI 和资源投影的 typed bool model；它不是权限授予。
|
||||
- `permissions` 是 runner 申请的 LangBot 资源访问上限；实际授权仍必须与 binding policy 求交。
|
||||
- `metadata` 只放展示、诊断、非稳定扩展信息。
|
||||
|
||||
### 4.3 Capabilities
|
||||
|
||||
```python
|
||||
class AgentRunnerCapabilities(BaseModel):
|
||||
streaming: bool = False
|
||||
tool_calling: bool = False
|
||||
knowledge_retrieval: bool = False
|
||||
multimodal_input: bool = False
|
||||
skill_authoring: bool = False
|
||||
interrupt: bool = False
|
||||
steering: bool = False
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
```
|
||||
|
||||
- `streaming`: runner 可以返回 `message.delta`。
|
||||
- `tool_calling`: runner 可能调用 Host tool API。
|
||||
- `knowledge_retrieval`: runner 可能调用 Host knowledge API。
|
||||
- `multimodal_input`: runner 可以处理非纯文本 input / attachment。
|
||||
- `skill_authoring`:（降级为便捷开关，非访问硬前提）声明该 runner 期望使用 LangBot skill 工具链。skill 本身通过**统一 tool 授权**获得——发现走 `list_skills` / `langbot_list_assets`，激活/注册走 `activate` / `register_skill`，操作走 native exec/read/write，全部计入 `resource_policy.allowed_tool_names`。该 capability 仅作为「一键授权这组 skill tool + sandbox」的便捷开关，不再单独决定 skill 是否可用。
|
||||
- `interrupt`: runner 支持取消或中断。
|
||||
- `steering`: runner 支持在 turn 边界通过 Host pull API 消费同 conversation 在途追加消息。
|
||||
|
||||
Capabilities 字段全部是 `bool`，未知 key 禁止进入 typed manifest。早期草案里的上下文/会话类 capability 已删除；对应语义由 event-first context 和 runner-owned context 原则表达。
|
||||
|
||||
### 4.4 Permissions 与 Effective Access
|
||||
|
||||
```python
|
||||
class AgentRunnerPermissions(BaseModel):
|
||||
models: list[Literal["invoke", "stream", "rerank"]] = []
|
||||
tools: list[Literal["detail", "call"]] = []
|
||||
knowledge_bases: list[Literal["list", "retrieve"]] = []
|
||||
history: list[Literal["page", "search"]] = []
|
||||
events: list[Literal["get", "page"]] = []
|
||||
storage: list[Literal["plugin", "workspace"]] = []
|
||||
files: list[Literal["config", "knowledge"]] = []
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
```
|
||||
|
||||
平台动作执行不属于当前 permissions。Platform action executor / EBA action 分支落地前，runner 只能返回 `action.requested` telemetry，Host 不执行平台动作。
|
||||
|
||||
Runner 实际可用 LangBot 资源来自 Host 在 run 前冻结的授权快照：
|
||||
|
||||
```text
|
||||
effective_access = manifest.permissions ∩ binding.resource_policy ∩ current scope/config
|
||||
```
|
||||
|
||||
具体落地：
|
||||
|
||||
1. `AgentResourceBuilder` 先用 manifest permissions 与 binding resource policy / runner config 求交，生成 `ctx.resources`。
|
||||
2. `AgentContextBuilder` 用 manifest permissions 与 binding state/storage policy 求交，生成 `ctx.context.available_apis`。
|
||||
3. `AgentRunSessionRegistry` 冻结 run-scoped resources 与 available APIs。
|
||||
4. Runtime handler / `AgentRunAPIProxy` 按 active `run_id`、runner identity、caller plugin identity、resource id、scope、payload size、rate limit 和 deadline 校验每次调用。
|
||||
|
||||
反承诺：manifest permissions **只约束 LangBot 持有的资源访问**。它不承诺限制外部 harness 的 native shell、文件系统、CLI、MCP、网络或本机权限；这些能力由 operator/runtime/sandbox 另行约束，见 HOST_SDK §4.8 与 SECURITY_HARDENING。
|
||||
|
||||
默认原则：
|
||||
|
||||
- Host 不得默认 inline 全量历史。
|
||||
- Host 只 inline 当前 event / input 和 context handles。
|
||||
- Runner 拥有 working context assembly。
|
||||
- Runner 可在授权后通过 Host history / event / state API 拉取更多上下文，并通过授权 sandbox/workspace 工具访问当前运行文件。
|
||||
- 历史窗口策略不属于 Protocol v1 字段，也不属于 Host 通用语义。
|
||||
|
||||
context 边界的设计理由见 [AGENT_CONTEXT_PROTOCOL.md](./AGENT_CONTEXT_PROTOCOL.md)。
|
||||
|
||||
## 5. Run 协议
|
||||
|
||||
### 5.1 RUN_AGENT
|
||||
|
||||
Host 调用 Runtime：
|
||||
|
||||
```python
|
||||
class AgentRunRequest(BaseModel):
|
||||
runner_id: str
|
||||
runner_name: str
|
||||
context: AgentRunContext
|
||||
```
|
||||
|
||||
Runtime 返回 `AgentRunResult` 异步流。底层 transport 可继续用 `plugin_author` / `plugin_name` / `runner_name` 定位组件，但协议语义以 `runner_id` 和 `context` 为准。
|
||||
|
||||
### 5.2 AgentRunContext
|
||||
|
||||
这是 SDK 看到的**唯一权威 context 定义**。
|
||||
|
||||
```python
|
||||
class AgentRunContext(BaseModel):
|
||||
run_id: str
|
||||
trigger: AgentTrigger
|
||||
event: AgentEventContext
|
||||
conversation: ConversationContext | None = None
|
||||
actor: ActorContext | None = None
|
||||
subject: SubjectContext | None = None
|
||||
input: AgentInput
|
||||
delivery: DeliveryContext
|
||||
resources: AgentResources
|
||||
context: ContextAccess
|
||||
state: AgentRunState
|
||||
runtime: AgentRuntimeContext
|
||||
config: dict[str, Any] = {}
|
||||
adapter: AdapterContext | None = None
|
||||
metadata: dict[str, Any] = {}
|
||||
```
|
||||
|
||||
核心约束：
|
||||
|
||||
- `event` 是必选字段，Protocol v1 是 event-first。
|
||||
- `input` 表示当前事件的主输入，不等于历史消息。
|
||||
- `bootstrap` / `messages` **不是协议字段**；Host 不内联历史窗口。
|
||||
- `adapter` 只放入口 adapter 的非核心元数据，runner 不应依赖它做长期能力。
|
||||
- `config` 是 Agent/runner config，不是插件实例状态。
|
||||
|
||||
### 5.3 AgentTrigger
|
||||
|
||||
```python
|
||||
class AgentTrigger(BaseModel):
|
||||
type: str
|
||||
source: Literal["platform", "webui", "api", "scheduler", "system", "host_adapter"]
|
||||
timestamp: int | None = None
|
||||
```
|
||||
|
||||
`trigger.type` 应与 `event.event_type` 一致或更粗粒度。例如入口适配器触发消息时：
|
||||
|
||||
```json
|
||||
{ "type": "message.received", "source": "host_adapter" }
|
||||
```
|
||||
|
||||
### 5.4 AgentEventContext
|
||||
|
||||
```python
|
||||
class AgentEventContext(BaseModel):
|
||||
event_id: str
|
||||
event_type: str
|
||||
event_time: int | None = None
|
||||
source: str
|
||||
source_event_type: str | None = None
|
||||
raw_ref: RawEventRef | None = None
|
||||
data: dict[str, Any] = {}
|
||||
```
|
||||
|
||||
- `event_type` 使用 LangBot 稳定协议名，例如 `message.received`。稳定事件名清单见 [EVENT_BASED_AGENT.md](./EVENT_BASED_AGENT.md)。
|
||||
- 平台原始事件名放入 `source_event_type`。
|
||||
- 大型原始 payload 必须放入 `raw_ref` 或 staged file，不应直接塞入 `data`。
|
||||
|
||||
### 5.5 Conversation / Actor / Subject
|
||||
|
||||
```python
|
||||
class ConversationContext(BaseModel):
|
||||
conversation_id: str | None = None
|
||||
thread_id: str | None = None
|
||||
launcher_type: str | None = None
|
||||
launcher_id: str | None = None
|
||||
sender_id: str | None = None
|
||||
bot_id: str | None = None
|
||||
workspace_id: str | None = None
|
||||
session_id: str | None = None
|
||||
|
||||
class ActorContext(BaseModel):
|
||||
actor_type: str
|
||||
actor_id: str | None = None
|
||||
actor_name: str | None = None
|
||||
metadata: dict[str, Any] = {}
|
||||
|
||||
class SubjectContext(BaseModel):
|
||||
subject_type: str
|
||||
subject_id: str | None = None
|
||||
data: dict[str, Any] = {}
|
||||
```
|
||||
|
||||
示例：
|
||||
|
||||
- 消息事件：actor 是发消息的人，subject 是当前消息。
|
||||
- 入群事件：actor 是新成员或邀请人，subject 是群/成员关系。
|
||||
- 定时事件：actor 可以是 system，subject 是 schedule。
|
||||
|
||||
### 5.6 AgentInput
|
||||
|
||||
```python
|
||||
class AgentInput(BaseModel):
|
||||
text: str | None = None
|
||||
contents: list[ContentElement] = []
|
||||
attachments: list[InputAttachment] = []
|
||||
```
|
||||
|
||||
- 文本、多模态、附件都属于当前 event input。
|
||||
- 大文件、图片、音频、工具大结果应进入授权 sandbox/workspace，input attachment 只携带轻量 metadata/path/url/content。
|
||||
- 平台原始消息链不属于 SDK `AgentInput`；需要诊断时放在 Host 内部 envelope 或 `ctx.adapter.extra` 的一次性兼容字段中，不作为长期 runner 合同。
|
||||
|
||||
### 5.7 DeliveryContext
|
||||
|
||||
```python
|
||||
class DeliveryContext(BaseModel):
|
||||
surface: str
|
||||
reply_target: dict[str, Any] | None = None
|
||||
supports_streaming: bool = False
|
||||
supports_edit: bool = False
|
||||
supports_reaction: bool = False
|
||||
max_message_size: int | None = None
|
||||
platform_capabilities: dict[str, Any] = {}
|
||||
```
|
||||
|
||||
Runner 可参考 delivery 能力决定返回 `message.delta`、`message.completed` 或 `action.requested`。
|
||||
|
||||
### 5.8 ContextAccess
|
||||
|
||||
```python
|
||||
class ContextAccess(BaseModel):
|
||||
conversation_id: str | None = None
|
||||
thread_id: str | None = None
|
||||
latest_cursor: str | None = None
|
||||
event_seq: int | None = None
|
||||
transcript_seq: int | None = None
|
||||
has_history_before: bool = False
|
||||
inline_policy: InlineContextPolicy
|
||||
available_apis: ContextAPICapabilities
|
||||
|
||||
class InlineContextPolicy(BaseModel):
|
||||
mode: Literal["none", "current_event", "recent_tail", "summary_tail"]
|
||||
delivered_count: int = 0
|
||||
source_total_count: int | None = None
|
||||
messages_complete: bool = False
|
||||
reason: str | None = None
|
||||
|
||||
class ContextAPICapabilities(BaseModel):
|
||||
prompt_get: bool = False
|
||||
history_page: bool = False
|
||||
history_search: bool = False
|
||||
event_get: bool = False
|
||||
event_page: bool = False
|
||||
state: bool = False
|
||||
storage: bool = False
|
||||
steering_pull: bool = False
|
||||
```
|
||||
|
||||
`ContextAccess` 告诉 runner：Host inline 了什么、没 inline 什么、需要更多上下文时走哪些 API。它是 runner 按需读取上下文的入口说明，不是 Host 的业务上下文编排策略。
|
||||
|
||||
### 5.9 AgentRuntimeContext
|
||||
|
||||
```python
|
||||
class AgentRuntimeContext(BaseModel):
|
||||
langbot_version: str | None = None
|
||||
trace_id: str | None = None
|
||||
deadline_at: float | None = None
|
||||
metadata: dict[str, Any] = {}
|
||||
```
|
||||
|
||||
### 5.10 AgentRunState
|
||||
|
||||
```python
|
||||
class AgentRunState(BaseModel):
|
||||
conversation: dict[str, Any] = {}
|
||||
actor: dict[str, Any] = {}
|
||||
subject: dict[str, Any] = {}
|
||||
runner: dict[str, Any] = {}
|
||||
```
|
||||
|
||||
State 是可选 host-owned snapshot。Runner 也可以完全自管状态。
|
||||
|
||||
## 6. Resources
|
||||
|
||||
```python
|
||||
class ToolResource(BaseModel):
|
||||
tool_name: str
|
||||
tool_type: str | None = None
|
||||
description: str | None = None
|
||||
parameters: dict[str, Any] | None = None # 完整 JSON schema，由 Host 一次塞齐
|
||||
operations: list[Literal["detail", "call"]] = []
|
||||
|
||||
class SkillResource(BaseModel):
|
||||
skill_name: str
|
||||
display_name: str | None = None
|
||||
description: str | None = None
|
||||
|
||||
class AgentResources(BaseModel):
|
||||
models: list[ModelResource] = []
|
||||
tools: list[ToolResource] = []
|
||||
knowledge_bases: list[KnowledgeBaseResource] = []
|
||||
skills: list[SkillResource] = []
|
||||
storage: StorageResource = StorageResource()
|
||||
platform_capabilities: dict[str, Any] = {}
|
||||
```
|
||||
|
||||
`tools` 携带每个授权工具的完整 schema（`parameters`），由 Host 在构造 `ctx.resources` 时一次塞齐，runner 不需再逐个调用 `get_tool_detail` 拉取，减少 N 次往返。
|
||||
|
||||
`skills` 是本次 run 中 pipeline-visible 的 skill facts（`skill_name`、`display_name`、`description`）。**skill 通过统一 tool 形式消费，不是独立资源类别**：发现走 `list_skills` tool（或 `langbot_list_assets` 增加 skills 一类），激活走 `activate`，操作走 native exec/read/write。Host **不**把 skill 索引注入 system prompt，也不做 progressive-disclosure 注入；LLM 通过调用发现工具主动查询 skill 清单。Host **可选**在 ctx 提供预渲染的 `suggested_skill_prompt`（首轮延迟优化，runner 可忽略 / override），但它不是访问前提。`skills` 字段本身仅作为发现工具的数据来源与该可选预渲染的输入。
|
||||
|
||||
资源列表是本次 run 的授权结果。History / Event / State / Storage 访问通过 `ctx.context.available_apis` 和 Host 侧 run session 校验控制，不作为可枚举 resource list 暴露。Runner 只能通过 `AgentRunAPIProxy` 访问这些能力。当前事件的文件和工具大结果优先进入授权 sandbox/workspace，由 runner 通过 read/write/exec 类工具按需读取。
|
||||
|
||||
## 7. Result Stream
|
||||
|
||||
### 7.1 AgentRunResult envelope
|
||||
|
||||
```python
|
||||
JSONValue = str | int | float | bool | None | list["JSONValue"] | dict[str, "JSONValue"]
|
||||
|
||||
ResultType = Literal[
|
||||
"message.delta",
|
||||
"message.completed",
|
||||
"tool.call.started",
|
||||
"tool.call.completed",
|
||||
"state.updated",
|
||||
"action.requested",
|
||||
"run.completed",
|
||||
"run.failed",
|
||||
]
|
||||
|
||||
class AgentRunResult(BaseModel):
|
||||
run_id: str
|
||||
type: AgentRunResultType | str
|
||||
data: dict[str, Any] = {}
|
||||
usage: LLMTokenUsage | None = None
|
||||
sequence: int | None = None
|
||||
timestamp: int | None = None
|
||||
```
|
||||
|
||||
SDK 当前实现是单一 envelope：`type` 枚举 + `data` dict。Payload 由 SDK typed model 构造并 dump，但 wire 不改成 discriminated union；这样新旧版本偏斜时 Host 仍可按 §3 忽略未知 `type`。
|
||||
|
||||
`usage` 是 runner 可选上报的 token 使用量，沿用 SDK `LLMTokenUsage`：
|
||||
|
||||
```python
|
||||
class LLMTokenUsage(BaseModel):
|
||||
prompt_tokens: int | None = None
|
||||
completion_tokens: int | None = None
|
||||
total_tokens: int | None = None
|
||||
# provider-specific detail/cached/reasoning counters are preserved as extra fields
|
||||
```
|
||||
|
||||
约束：
|
||||
|
||||
- 运行时能观测到 provider/runtime usage 时，SHOULD 在 terminal `run.completed.usage` 上报本次 run 的最终聚合 token usage。
|
||||
- `run.failed.usage` MAY 上报失败前已经产生的部分 usage。
|
||||
- 不能观测 usage 的 runner 合法地省略该字段；缺失表示 unknown，Host 不得按 0 处理。
|
||||
- ACP 等外部协议不保证统一 usage；ACP runner 只能在具体 provider/native event 提供 usage 时填充本字段。
|
||||
- cost 不作为 runner result 的权威字段。Host 后续应基于 usage、model identity、时间和自身价格表计算账单成本；provider 原始 cost 如需保留，可放在 `usage` extra 字段中作为非权威 telemetry。
|
||||
|
||||
Host 边界分级校验：
|
||||
|
||||
- `message.delta`、`message.completed`、`state.updated`、`action.requested`、`run.completed`、`run.failed` 属于会影响投递或 Host 副作用的严格 payload；校验失败时丢弃该 result 并记录 warning。
|
||||
- `tool.call.started`、`tool.call.completed` 当前只作为 telemetry，payload 宽松兼容。
|
||||
- 未知 `type` 忽略并记录 warning。
|
||||
|
||||
### 7.2 稳定 result payloads
|
||||
|
||||
| type | `data` payload |
|
||||
| --- | --- |
|
||||
| `message.delta` | `{ "chunk": MessageChunk }` |
|
||||
| `message.completed` | `{ "message": Message }` |
|
||||
| `tool.call.started` | `{ "tool_call_id": str, "tool_name": str, "parameters": dict }` |
|
||||
| `tool.call.completed` | `{ "tool_call_id": str, "tool_name": str, "result": dict \| None, "error": str \| None }` |
|
||||
| `state.updated` | `{ "scope": "conversation" \| "actor" \| "subject" \| "runner", "key": str, "value": JSONValue }` |
|
||||
| `action.requested` | `{ "action": str, "target": dict \| None, "payload": dict \| None }` |
|
||||
| `run.completed` | `{ "finish_reason": str, "message"?: Message }` |
|
||||
| `run.failed` | `{ "code": str, "error": str, "retryable": bool }` |
|
||||
|
||||
Runner 生成的大文件、工具输出和临时产物不通过 result event 回传；应写入当前 run 的授权 sandbox/workspace，再用消息文本、metadata 或 attachment reference 指向它们。
|
||||
|
||||
### 7.3 稳定 result types
|
||||
|
||||
| type | 说明 | 当前消费 |
|
||||
| --- | --- | --- |
|
||||
| `message.delta` | 流式消息片段。 | ✅ |
|
||||
| `message.completed` | 完整消息。 | ✅ |
|
||||
| `tool.call.started` | 工具调用开始的可观测事件。 | telemetry |
|
||||
| `tool.call.completed` | 工具调用完成的可观测事件。 | telemetry |
|
||||
| `state.updated` | runner 请求更新 host-owned state。 | ✅ |
|
||||
| `action.requested` | runner 请求 Host 执行平台动作。 | **reserved / 仅 telemetry，不执行** |
|
||||
| `run.completed` | run 正常结束。 | ✅ |
|
||||
| `run.failed` | run 失败。 | ✅ |
|
||||
|
||||
`action.requested` 是为 EBA 和 platform API 保留的协议表面：本分支 Host 收到后只记 telemetry，**不执行**，runner 作者不应在当前 Host 底座中依赖其副作用。真实执行器由外部 EBA / platform action 分支接入；执行模型见 EVENT_BASED_AGENT §6。
|
||||
|
||||
Host 必须校验 `state.updated` 的 scope、key、value 大小和 JSON 可序列化性。本分支 `action.requested` 仍只记录 telemetry。
|
||||
|
||||
除 runner 经 `state.updated` 写之外，Host 自身也可直接写部分 host-owned state。例如 `activate` tool 在 Host 侧执行时，直接把已激活 skill 写入 conversation scope 的 `host.activated_skills` 快照。当 host 直接写与 runner `state.updated` 写到同一 key 时，按 **last-write-wins** 合并——runner 可以覆盖 host 写的快照。
|
||||
|
||||
### 7.4 Stream delivery semantics
|
||||
|
||||
- Host 按 Runtime stream 顺序消费 result。当前 v1 不定义跨连接 replay，也不承诺 at-least-once；从 Host 视角，收到的 result 最多应用一次。
|
||||
- `sequence` 是单个 `run_id` 内的结果序号。in-process / stdio 这类天然有序的在线 stream 可以省略；任何会缓冲、重放、跨进程队列或 runtime-managed task 的 transport 必须提供从 1 开始严格递增的 `sequence`。
|
||||
- Host 看到已提供 `sequence` 的 result 时，应按 `(run_id, sequence)` 做重复检测，并在缺号或乱序时记录 warning；除非 transport 明确声明 replay 语义，Host 不应自行等待缺失序号重排用户可见输出。
|
||||
- `run.failed.data.retryable` 只表示整次 run 理论上可由上层重试；Protocol v1 不自动重试 run，也不自动重试 proxy action。
|
||||
- History / Event / Transcript cursor 是 opaque token。runner 不得解析 cursor，也不得假设 cursor 在不同 API、conversation、thread 或 retention window 之间可比较；当前实现即使返回数字字符串，也只是实现细节。
|
||||
|
||||
### 7.5 示例
|
||||
|
||||
```json
|
||||
{ "type": "message.delta", "data": { "chunk": { "role": "assistant", "content": "hel" } } }
|
||||
{ "type": "message.completed", "data": { "message": { "role": "assistant", "content": "hello" } } }
|
||||
{ "type": "state.updated", "data": { "scope": "conversation", "key": "external.session_id", "value": "abc" } }
|
||||
{ "type": "action.requested", "data": { "action": "message.edit", "target": {"message_id": "..."}, "payload": {"text": "..."} } }
|
||||
```
|
||||
|
||||
## 8. AgentRunAPIProxy
|
||||
|
||||
所有 proxy action 必须携带 `run_id`。Host 必须校验：active run session 存在、caller plugin identity 匹配、resource 在本次 `ctx.resources` 中授权、scope 不越界、payload size / rate limit / deadline 合法。
|
||||
|
||||
```python
|
||||
# Model
|
||||
await api.invoke_llm(llm_model_uuid, messages, funcs=None, extra_args=None)
|
||||
await api.invoke_llm_with_usage(llm_model_uuid, messages, funcs=None, extra_args=None)
|
||||
async for chunk in api.invoke_llm_stream(llm_model_uuid, messages, funcs=None, extra_args=None):
|
||||
...
|
||||
async for event in api.invoke_llm_stream_events(llm_model_uuid, messages, funcs=None, extra_args=None):
|
||||
...
|
||||
await api.invoke_rerank(rerank_model_id, query, documents, top_k=None)
|
||||
|
||||
# Tool
|
||||
await api.get_tool_detail(tool_name)
|
||||
await api.call_tool(tool_name, parameters)
|
||||
|
||||
# Knowledge
|
||||
await api.retrieve_knowledge(kb_id, query_text, top_k=5, filters=None)
|
||||
|
||||
# History（返回 Transcript projection，不返回原始平台 payload）
|
||||
await api.get_prompt()
|
||||
await api.history_page(conversation_id=None, before_cursor=None, after_cursor=None,
|
||||
limit=50, direction="backward", include_attachments=False)
|
||||
await api.history_search(query, filters=None, top_k=10)
|
||||
|
||||
# Event（返回稳定 event envelope 或受限 raw ref，不默认返回大 payload）
|
||||
await api.event_get(event_id)
|
||||
await api.event_page(conversation_id=None, event_types=None, before_cursor=None, limit=50)
|
||||
await api.steering_pull(mode="all", limit=None)
|
||||
|
||||
# State / Storage
|
||||
await api.state_get(scope, key); await api.state_set(scope, key, value); await api.state_delete(scope, key)
|
||||
await api.state_list(scope, prefix=None, limit=100)
|
||||
await api.get_plugin_storage(key); await api.set_plugin_storage(key, value); await api.delete_plugin_storage(key)
|
||||
await api.get_plugin_storage_keys()
|
||||
await api.get_workspace_storage(key); await api.set_workspace_storage(key, value); await api.delete_workspace_storage(key)
|
||||
await api.get_workspace_storage_keys()
|
||||
|
||||
# Host info
|
||||
await api.get_langbot_version()
|
||||
```
|
||||
|
||||
`invoke_llm()` / `invoke_llm_stream()` 的第一个参数在 SDK 中命名为
|
||||
`llm_model_uuid`，wire payload 字段也是 `llm_model_uuid`。该值对 runner
|
||||
仍是 opaque identifier，不应解析其内部格式。
|
||||
|
||||
`invoke_llm()` 和 `invoke_llm_stream()` 保持兼容：前者返回 `Message`，后者只
|
||||
yield `MessageChunk`。需要 provider 真实 token 计量的 runner 应使用
|
||||
`invoke_llm_with_usage()` 或 `invoke_llm_stream_events()`。Host response 可在
|
||||
原有 `{message: ...}` / `{chunk: ...}` 外额外携带可选 `usage` 字段；streaming
|
||||
场景允许在所有 chunk 之后追加一个 usage-only event。`usage` 至少保留
|
||||
OpenAI-compatible 的 `prompt_tokens`、`completion_tokens`、`total_tokens`，
|
||||
若 provider 返回 `prompt_tokens_details` / `completion_tokens_details` 或
|
||||
cache token counters，Host / SDK 不应丢弃这些字段。没有 usage 的 provider
|
||||
必须继续返回成功响应，SDK 将 usage 置为 `None`。
|
||||
|
||||
`get_prompt()` 返回当前 query-backed run 的 Host effective prompt messages：
|
||||
`list[Message]` 的 JSON 形式。该能力只在 `ctx.context.available_apis.prompt_get`
|
||||
为 true 时可用；没有 query 缓存、prompt 已过期或非 query entry run 时 Host
|
||||
可以返回错误或空列表。Runner 应在不可用时回退到自己的 config/prompt 策略。
|
||||
|
||||
`steering_pull(mode="all")` 是推荐默认：Host 按 claim 顺序返回全部 pending steering 输入并清空对应队列。`mode="one-at-a-time"` 仅用于 runner 主动节流，每次返回一条。Host 不合并多条用户消息；runner 负责在 turn 边界决定模型侧格式。
|
||||
|
||||
Steering 审计使用 EventLog 而不是 Transcript schema 扩展：被 active run 吸收的原始 `message.received` 事件保留原事件类型，并在 `metadata.steering` 标记 `status="queued"`、`trigger_behavior="absorbed_into_active_run"`、`claimed_by_run_id`、`claimed_runner_id`、`claimed_at`。Runner 成功 pull 后，Host 追加 `steering.injected` EventLog 记录，`metadata.steering.status="injected"` 并引用 `source_event_id`。若 run 结束时仍有已 claim 但未 pull 的 steering 输入，Host 追加 `steering.dropped` EventLog 记录，`metadata.steering.status="dropped"` 并引用 `source_event_id`；这不是用户消息事实的删除，只是 dispatch 终态。Transcript 继续只表示会话事实，不承担 dispatch 行为标记。
|
||||
|
||||
`state` 与 `storage` 的建议边界：`state` 放小型 JSON（conversation / actor / subject / runner），`storage` 放 blob 或较大数据（插件私有数据、workspace 数据、checkpoint）。
|
||||
|
||||
Compaction checkpoint 的推荐 state 约定：
|
||||
|
||||
- scope: `conversation`
|
||||
- key: `runner.compaction.checkpoint`
|
||||
- value:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": "langbot.local_agent.compaction_checkpoint.v1",
|
||||
"summary": "<conversation_summary>...</conversation_summary>",
|
||||
"covers_until": "transcript-cursor-or-seq",
|
||||
"tokens_before": 12345,
|
||||
"created_at": 1710000000,
|
||||
"conversation_id": "conv-..."
|
||||
}
|
||||
```
|
||||
|
||||
`covers_until` 是摘要覆盖到的 transcript 游标锚点。Runner 读取 checkpoint 后应只拉取该游标之后的 transcript；若 checkpoint 缺失、schema 不匹配、conversation 不匹配或游标不可用，应回退到无 checkpoint 的尾部历史拉取行为。
|
||||
|
||||
Proxy 返回数据结构也属于本协议：
|
||||
|
||||
```python
|
||||
class TranscriptItem(BaseModel):
|
||||
transcript_id: str
|
||||
event_id: str
|
||||
conversation_id: str | None = None
|
||||
thread_id: str | None = None
|
||||
role: str
|
||||
item_type: str = "message"
|
||||
content: str | None = None
|
||||
content_json: dict[str, Any] | None = None
|
||||
attachment_refs: list[dict[str, Any]] = []
|
||||
seq: int | None = None
|
||||
cursor: str | None = None
|
||||
created_at: int | None = None
|
||||
metadata: dict[str, Any] = {}
|
||||
|
||||
class HistoryPage(BaseModel):
|
||||
items: list[TranscriptItem] = []
|
||||
next_cursor: str | None = None
|
||||
prev_cursor: str | None = None
|
||||
has_more: bool = False
|
||||
total_count: int | None = None
|
||||
|
||||
class HistorySearchResult(BaseModel):
|
||||
items: list[TranscriptItem] = []
|
||||
total_count: int | None = None
|
||||
query: str
|
||||
|
||||
class AgentEventRecord(BaseModel):
|
||||
event_id: str
|
||||
event_type: str
|
||||
event_time: int | None = None
|
||||
source: str
|
||||
bot_id: str | None = None
|
||||
workspace_id: str | None = None
|
||||
conversation_id: str | None = None
|
||||
thread_id: str | None = None
|
||||
actor_type: str | None = None
|
||||
actor_id: str | None = None
|
||||
actor_name: str | None = None
|
||||
subject_type: str | None = None
|
||||
subject_id: str | None = None
|
||||
input_summary: str | None = None
|
||||
input_ref: str | None = None
|
||||
raw_ref: str | None = None
|
||||
seq: int | None = None
|
||||
cursor: str | None = None
|
||||
created_at: int | None = None
|
||||
metadata: dict[str, Any] = {}
|
||||
|
||||
class EventPage(BaseModel):
|
||||
items: list[AgentEventRecord] = []
|
||||
next_cursor: str | None = None
|
||||
prev_cursor: str | None = None
|
||||
has_more: bool = False
|
||||
total_count: int | None = None
|
||||
|
||||
class SteeringInputItem(BaseModel):
|
||||
claimed_run_id: str
|
||||
runner_id: str
|
||||
claimed_at: int | None = None
|
||||
event: AgentEventContext
|
||||
input: AgentInput
|
||||
conversation: ConversationContext | None = None
|
||||
actor: ActorContext | None = None
|
||||
subject: SubjectContext | None = None
|
||||
metadata: dict[str, Any] = {}
|
||||
|
||||
class SteeringPullResult(BaseModel):
|
||||
items: list[SteeringInputItem] = []
|
||||
```
|
||||
|
||||
## 9. 错误模型
|
||||
|
||||
```python
|
||||
class AgentAPIError(BaseModel):
|
||||
code: str
|
||||
message: str
|
||||
retryable: bool = False
|
||||
details: dict[str, Any] = {}
|
||||
```
|
||||
|
||||
| code | 说明 |
|
||||
| --- | --- |
|
||||
| `unauthorized` | 未授权访问资源或 scope。 |
|
||||
| `not_found` | 资源不存在或对当前 runner 不可见。 |
|
||||
| `deadline_exceeded` | 超过 run deadline。 |
|
||||
| `payload_too_large` | 请求或响应过大。 |
|
||||
| `rate_limited` | Host 限流。 |
|
||||
| `invalid_argument` | 参数错误。 |
|
||||
| `runtime_error` | Host 或下游能力错误。 |
|
||||
|
||||
SDK runner-facing proxy 在 Host 返回结构化错误或畸形响应时抛出 `AgentAPIException`，其中 `error` 字段为 `AgentAPIError`。Legacy transport 只返回字符串错误时，SDK 使用 `host.action_error` 包装，避免 runner 继续依赖裸 `KeyError` 或字符串匹配。
|
||||
|
||||
Runner 失败使用 `run.failed`：
|
||||
|
||||
```json
|
||||
{ "type": "run.failed", "data": { "code": "runner.error", "error": "failed to call external agent", "retryable": false } }
|
||||
```
|
||||
|
||||
## 10. Timeout 与 Cancellation
|
||||
|
||||
- Host 在 `ctx.runtime.deadline_at` 下发总 deadline；SDK proxy 必须用该 deadline 限制单次 action timeout。
|
||||
- Host 可以取消 active run；Runtime 应尽力中断 runner。
|
||||
- Protocol v1 的 run 绑定当前 Host 进程和当前 runtime channel，不保证跨 Host 重启恢复。Host 重启、runtime channel 断开或 run session 丢失时，Runtime / external harness connector 必须 fail-fast 并尽力取消仍在执行的 runner，不得继续使用旧 `run_id` 调用 Host API。
|
||||
- Runner 支持中断时应返回或触发 `run.failed`，code 为 `cancelled`。
|
||||
- Host 必须 unregister active run session。
|
||||
|
||||
## 11. Security 与 Guardrail（协议层）
|
||||
|
||||
Protocol v1 的安全边界在 Host：
|
||||
|
||||
- Runner 不能直接访问未授权 model/tool/kb/history/storage/sandbox。
|
||||
- SDK 本地校验只提升开发体验，不能替代 Host 校验。
|
||||
- 所有 resource id 对 runner 来说都是 opaque。
|
||||
- 默认只能访问当前 conversation / thread 的 history；跨会话、workspace 级访问必须额外授权。
|
||||
- 大 payload 不应塞进 result event；当前 run 的文件和工具大结果应进入授权 sandbox/workspace，由 read/write/exec 类工具按需访问。
|
||||
- Host 必须记录 run_id、runner_id、action、resource、scope、result。
|
||||
|
||||
Host 不负责业务编排：不拼接全量历史、不替 runner 做 prompt assembly、不内置 agent memory / tool loop / 上下文压缩策略。这些由官方或第三方 AgentRunner 插件实现。
|
||||
|
||||
外部 harness runner 的边界统一见 HOST_SDK §4.8。简言之：harness native permission mode、allowed/disallowed tools、shell/MCP 权限只是额外执行约束，不能替代 Host 对 LangBot 资源的授权。
|
||||
|
||||
> 发布级路径隔离、MCP allowlist、secret redaction、配额、workspace 清理等**不属于** v1 协议闭环，是生产默认启用前的 release gate，见 [SECURITY_HARDENING.md](./SECURITY_HARDENING.md)。
|
||||
|
||||
## 12. Pipeline Adapter 边界
|
||||
|
||||
Pipeline 是当前入口 adapter，不是协议中心。目标产品模型中 Agent 会替代
|
||||
Pipeline 承载 runner config、resource policy 和 delivery policy；当前 Query
|
||||
entry adapter 只是迁移桥。它负责：
|
||||
|
||||
- 从 `Query` 构造 `AgentEventContext` 和临时 `AgentBinding`（见 HOST_SDK §4.2）。
|
||||
- 从当前 Agent/runner config 构造 `ctx.config`。
|
||||
- 将 Query-only 字段放入 `ctx.adapter`，例如 filtered params 放 `ctx.adapter.extra["params"]`。
|
||||
|
||||
约束：
|
||||
|
||||
- adapter **不**定义历史窗口、prompt 组装或 agentic context 策略。
|
||||
- `ctx.adapter.extra` 只允许承载一次性、JSON-safe、入口相关的非核心元数据，例如 `params`；不得承载 `prompt`、history window、RAG 结果、tool schema 或授权资源。
|
||||
- 静态绑定 prompt 属于 `ctx.config.prompt`。preprocessing / hook 后的动态有效指令不通过 `ctx.adapter.extra` 主动推送；后续如需要保留这类能力，应通过 Host prompt/instruction pull API 暴露（占位见 HOST_SDK §4.8）。
|
||||
- 新 runner 不应长期依赖 `adapter`，应只依赖 event-first context 和 Host API。
|
||||
|
||||
## 13. 已确认约束
|
||||
|
||||
- v1 / EBA 主线是 `one event -> one AgentBinding -> one run_id -> one runner`。
|
||||
- 一个 bot / IM channel 在同一时间只绑定一个负责 agentic 处理的 Agent；一个 Agent 可以被多个 bot / channel 复用。
|
||||
- 如果配置层出现多个匹配 AgentBinding，BindingResolver 必须按明确规则选出一个或拒绝配置，不应默认 fan-out。
|
||||
- observer agent、多 runner fan-out、并行裁决、result 合并等能力需要单独设计 delivery、state、platform action 和 audit 语义，不属于当前 v1 契约。
|
||||
- `AgentRunnerDescriptor.source` 只允许 `plugin`；Host 内置 adapter 不能作为 runner source 绕过插件/runtime/proxy 权限链。
|
||||
- `ctx.resources` 与 proxy action 校验必须来自同一个 run authorization snapshot；runtime handler 不应重新执行资源裁剪。
|
||||
- v1 不要求 Agent、AgentRunner 插件实例或 runner id 全局串行。多个 bot / channel 可复用同一个 Agent；并发隔离依赖 `run_id`、binding、conversation / thread scope 和 Host authorization snapshot。
|
||||
- 外部 harness runner 当前是 MVP / dev path，证明协议可接入，不代表发布级安全边界或 Docker 生产可用性完成。
|
||||
|
||||
## 14. 开放问题
|
||||
|
||||
- `AgentBinding` 是否需要进入 SDK 文档作为只读诊断信息，还是完全 Host 内部。
|
||||
- State 与 Storage 的边界是否需要更强类型。
|
||||
- platform action 的审批模型如何表达。
|
||||
- Host 侧 scoped MCP / workspace projection 是否需要从 runner config 上移为一等 resource projection API。（skill 一项已收敛：skill 全 tool 化，作为被授权 tool 暴露，不再是独立 projection。）
|
||||
@@ -0,0 +1,154 @@
|
||||
# Agent Runner 插件化文档入口
|
||||
|
||||
本文档是 agent-runner 插件化工作的路由页。具体设计拆到独立文档中维护，避免把 LangBot 宿主架构、SDK 协议、上下文管理、EBA 接入边界和官方 runner 迁移混在同一份 README 里。
|
||||
|
||||
## 背景与问题
|
||||
|
||||
旧 runner 路径主要围绕 Pipeline / Query 和 `pkg/provider/runners` 内置实现展开，扩展外部 agent runtime 时容易把 runner 选择、上下文裁剪、资源授权和消息投递绑在同一条聊天链路里。这个分支要把 LangBot 收敛成 Agent Host：Host 负责事件、绑定、授权、事实源和结果投递；AgentRunner 作为插件或外部 harness 消费统一协议并自主管理 prompt / history / memory。
|
||||
|
||||
## 文档维护原则（单一事实源）
|
||||
|
||||
- **协议数据结构（schema）唯一定义在 [PROTOCOL_V1.md](./PROTOCOL_V1.md)。** 其他文档不得重抄 schema，只能引用，例如"见 PROTOCOL_V1 §4.2"。
|
||||
- 当前实现状态、spec 差距与 runner 验收状态归 [STATUS.md](./STATUS.md)；测试执行入口归 [AGENT_RUNNER_QA_GUIDE.md](./AGENT_RUNNER_QA_GUIDE.md)，安全发布门槛归 [SECURITY_HARDENING.md](./SECURITY_HARDENING.md)。
|
||||
- Host 内部模型（`AgentEventEnvelope`、`AgentBinding`、Descriptor、各 Store）定义在 [HOST_SDK_INFRASTRUCTURE.md](./HOST_SDK_INFRASTRUCTURE.md)，不属于 SDK 协议。
|
||||
- 其余专题文档只讲"为什么/边界/怎么用"，避免重复叙述。
|
||||
|
||||
## 本分支目标
|
||||
|
||||
**本分支目标：AgentRunner 外化 / 插件化基础设施**
|
||||
|
||||
本分支只做 LangBot 作为 Agent Host 的基础能力建设，为后续用 `Agent`
|
||||
替代 Pipeline 承载 agent 配置打底：
|
||||
|
||||
- LangBot 与 SDK 的稳定协议合同（Protocol v1）
|
||||
- Host-side `AgentEventEnvelope` / `AgentBinding` 模型
|
||||
- `run(event, binding)` event-first 入口
|
||||
- `QueryEntryAdapter`：Query → AgentEventEnvelope + AgentBinding
|
||||
- EventLog / Transcript / PersistentStateStore
|
||||
- History / Event / State pull APIs
|
||||
- Sandbox/workspace read/write/exec 文件能力，用于当前 run 的上传文件、工具大结果和临时产物
|
||||
- SDK runtime forwarding pull APIs + `caller_plugin_identity` 验证路径
|
||||
|
||||
## 本分支不实现
|
||||
|
||||
以下能力由其他分支负责，本分支只保留 integration point。EBA 完整事件网关与事件路由当前由外部 EBA 分支联调：
|
||||
|
||||
- **EventGateway / EventRouter**：完整事件网关实现、事件路由、事件持久化管理
|
||||
- **Event subscription / Event notification**：事件订阅、推送通知
|
||||
- **BindingResolver persistence UI**：绑定配置的持久化 UI 和 event router 集成（如由其他模块负责）
|
||||
- **Scheduler / Background event source**：定时任务、后台事件源
|
||||
- **完整 Agent Platform / daemon control plane**：Host-owned `AgentRun` / `AgentRunEvent`、run control primitives、最小 runtime heartbeat/claim lease 已作为 v2 foundation 落地；业务队列、Platform UI、daemon supervisor、runtime wakeup channel 和分布式 runtime 管控仍不属于 Protocol v1 主线。
|
||||
|
||||
EventGateway / EventRouter 在本文档中描述为 **external EBA branch integration point**，由外部 EBA 分支提供并联调。本分支只定义 host-side envelope/binding models 和 `run(event, binding)` orchestrator 入口。
|
||||
|
||||
本分支与外部 EBA / Agent Platform / Runtime Control Plane 的扩展边界见 [EXTENSION_SCOPE_MATRIX.md](./EXTENSION_SCOPE_MATRIX.md)。
|
||||
|
||||
## 目标产品模型
|
||||
|
||||
未来产品层应把 `Agent` 理解为 Pipeline 的替代物：原先 bot 绑定 Pipeline，Pipeline 携带 agent/provider/RAG/tool 等配置；后续应改为 bot 或 IM channel 绑定一个 Agent，Agent 携带 runner id、runner config、resource/state/delivery policy 等 agent 配置。
|
||||
|
||||
调度基数、Agent 复用、插件实例无状态、Pipeline adapter 和 fan-out 边界的规范来源是 [PROTOCOL_V1.md](./PROTOCOL_V1.md) §13；README 不复写这些约束。
|
||||
|
||||
## 当前入口关系
|
||||
|
||||
**当前 Pipeline 是入口 adapter，不再是 agent runner 设计核心。**
|
||||
|
||||
主入口仍可由 Pipeline 触发，但内部已转换成 event-first path：`run_from_query()` 经 `QueryEntryAdapter` 把 `Query` 转换为 `AgentEventEnvelope` + `AgentBinding`，再委托到统一的 `run(event, binding, ...)`。Pipeline path 因此获得了 event-first host capabilities（EventLog / Transcript / PersistentStateStore 写入，History / Event / State pull API 和 sandbox/workspace 文件读写能力可用）。
|
||||
|
||||
下一轮测试路径、状态定义和 smoke 记录见 [AGENT_RUNNER_QA_GUIDE.md](./AGENT_RUNNER_QA_GUIDE.md)。
|
||||
|
||||
## 术语表
|
||||
|
||||
| 术语 | 含义 |
|
||||
| --- | --- |
|
||||
| Protocol v1 | Host 调用 AgentRunner 的 runner 可见合同：discovery、`AgentRunContext`、result stream、Host pull API 和错误模型。 |
|
||||
| Agent | 目标产品层配置对象，保存 runner id、runner config 和资源/状态/投递策略；不等于插件实例。 |
|
||||
| AgentConfig | Host 内部迁移期配置投影，由当前 Pipeline config 或未来持久 Agent 生成。 |
|
||||
| AgentBinding / binding | Host 在一次事件运行前解析出的有效绑定，决定调用哪个 runner 以及带什么策略。 |
|
||||
| envelope | Host 内部事件封装，即 `AgentEventEnvelope`；runner 看到的是由它投影出的 `ctx.event`。 |
|
||||
| descriptor / manifest | runner discovery 的能力和配置描述；manifest 来自插件，descriptor 是 Host 校验后的注册表视图。 |
|
||||
| EBA | Event Based Agent，把消息、撤回、入群、定时任务等都统一成 host event 的接入方向；完整网关和路由在外部 EBA 分支联调。 |
|
||||
| harness runner | ACP、Claude Code、Codex 等已有自身 session / tool loop / MCP / 压缩机制的外部 runtime adapter。 |
|
||||
| projection | Host 把内部事实源、授权资源或配置裁剪成 runner / harness 可消费视图的过程。 |
|
||||
| Runtime Control Plane | v2 Host 能力层，当前已落地 Host-owned run/result ledger、run control primitives、最小 runtime heartbeat/claim lease；完整 daemon worker 管控、task wakeup 和 Agent Platform 产品形态不是 Protocol v1 主线。 |
|
||||
|
||||
## 设计文档
|
||||
|
||||
| 文档 | 关注点 |
|
||||
| --- | --- |
|
||||
| [PROTOCOL_V1.md](./PROTOCOL_V1.md) | **🔒 唯一 schema 事实源**。LangBot Host 与 SDK / Runtime / AgentRunner 的协议合同：版本协商、discovery、run context、result stream、proxy actions、错误和 adapter 边界。 |
|
||||
| [HOST_SDK_INFRASTRUCTURE.md](./HOST_SDK_INFRASTRUCTURE.md) | LangBot 宿主能力与分层架构、Host 内部模型（`AgentEventEnvelope` / `AgentBinding` / Descriptor / 各 Store）、runner 发现、绑定、资源授权、状态、存储、生命周期和调用链。 |
|
||||
| [AGENT_CONTEXT_PROTOCOL.md](./AGENT_CONTEXT_PROTOCOL.md) | Agent-owned context 方向：事件到来时 LangBot 传什么，agent 如何按需拉取更多历史 / state、如何访问 sandbox/workspace 文件，以及如何支持 KV cache 友好的上下文管理。 |
|
||||
| [EXTENSION_SCOPE_MATRIX.md](./EXTENSION_SCOPE_MATRIX.md) | AgentRunner 外化与外部 EBA / Agent Platform / Runtime Control Plane 的扩展边界矩阵，说明哪些是本分支底座、哪些由外部分支接入。 |
|
||||
| [EVENT_BASED_AGENT.md](./EVENT_BASED_AGENT.md) | EBA 接入边界：事件模型、事件来源、触发绑定、非消息事件如何复用 AgentRunner 调度；完整 EventGateway / EventRouter 由外部 EBA 分支联调。 |
|
||||
| [RUNTIME_CONTROL_PLANE_V2.md](./RUNTIME_CONTROL_PLANE_V2.md) | Agent Platform v2 / runtime 管控面决策：`AgentRun` / `AgentRunEvent` / run control 已作为 Host 事实源落地，最小 runtime heartbeat/claim lease 已落地；完整 runtime registry / daemon 管控仍是后续可选阶段。 |
|
||||
| [OFFICIAL_RUNNER_PLUGINS.md](./OFFICIAL_RUNNER_PLUGINS.md) | 官方 runner 插件迁移，包括 local-agent 和外部 runner。它是下游落地计划，不是 LangBot 基础能力设计的前置约束。 |
|
||||
| [RUN_STEERING_AND_CHECKPOINT.md](./RUN_STEERING_AND_CHECKPOINT.md) | 运行中消息注入（steering / follow-up）与压缩摘要持久化（compaction checkpoint）的设计与落地状态记录；schema 仍以 PROTOCOL_V1 为准。 |
|
||||
| [STATUS.md](./STATUS.md) | 当前实现状态、spec 与实现已知差距、runner 验收状态和历史高价值记录。 |
|
||||
| [AGENT_RUNNER_QA_GUIDE.md](./AGENT_RUNNER_QA_GUIDE.md) | Agent Runner QA 指南：保留最高价值测试路径，指导 agent 开展下一轮 WebUI / runner smoke 验证。 |
|
||||
| [SECURITY_HARDENING.md](./SECURITY_HARDENING.md) | 安全发布级 hardening 的后续发布门槛：路径隔离、权限边界、secret、资源配额、MCP / skill 投影和审计。 |
|
||||
|
||||
## 工作拆分
|
||||
|
||||
### 1. LangBot + SDK 基础设施
|
||||
|
||||
目标是把 LangBot 从内置 runner 执行器变成 agent host：
|
||||
|
||||
- LangBot 与 SDK 的稳定协议合同
|
||||
- runner manifest / descriptor / registry
|
||||
- Agent / binding 配置解析
|
||||
- run orchestration 和生命周期管理
|
||||
- resource authorization 与 `run_id` 级权限校验
|
||||
- host-owned state / storage / event log / transcript 能力
|
||||
- sandbox/workspace 文件 staging 与 read/write/exec 能力
|
||||
- SDK `AgentRunner`、`AgentRunContext`、`AgentRunResult`、`AgentRunAPIProxy`
|
||||
|
||||
协议合同详见 [PROTOCOL_V1.md](./PROTOCOL_V1.md)。
|
||||
|
||||
详见 [HOST_SDK_INFRASTRUCTURE.md](./HOST_SDK_INFRASTRUCTURE.md)。
|
||||
|
||||
### 2. Agent-owned context
|
||||
|
||||
LangBot 不应成为最终 agentic context manager。它应提供事实源、默认上下文引用和按需读取 API；agent 或其背后的 runtime 负责历史剪裁、摘要、召回和 KV cache 策略。
|
||||
|
||||
Host 不定义通用历史窗口字段或策略；runner 通过 Host pull API 按需拉取历史并自行管理 working context。
|
||||
|
||||
详见 [AGENT_CONTEXT_PROTOCOL.md](./AGENT_CONTEXT_PROTOCOL.md)。
|
||||
|
||||
### 3. Event Based Agent（External Branch）
|
||||
|
||||
消息只是事件的一种。外部 EBA 分支中的 `message.received`、`message.recalled`、`group.member_joined`、`friend.request_received` 等事件都应能通过统一事件 envelope 触发 AgentRunner。
|
||||
|
||||
EBA dispatch 的基数和 fan-out 边界仍以 PROTOCOL_V1 §13 为准；本文档只列出本分支提供给外部 EBA 分支复用的入口点。
|
||||
|
||||
**本分支不实现 EBA 完整能力，只提供：**
|
||||
- event-first envelope (`AgentEventEnvelope`)
|
||||
- AgentBinding model
|
||||
- `run(event, binding)` 入口
|
||||
- QueryEntryAdapter（当前 AgentEventEnvelope / AgentBinding 的 Query entry adapter source）
|
||||
|
||||
详见 [EVENT_BASED_AGENT.md](./EVENT_BASED_AGENT.md)。
|
||||
|
||||
### 4. 官方 runner 插件
|
||||
|
||||
官方 `local-agent` 和外部 runner 迁移是下游工作。它们需要依附 LangBot 提供的宿主能力，但不应反过来决定宿主协议。
|
||||
|
||||
`local-agent` 可以外移，也可以重写。验收重点是它能完整消费 LangBot 的模型、工具、知识库、存储、事件、history API 和 result stream，而不是保留旧内置 runner 的内部结构。
|
||||
|
||||
详见 [OFFICIAL_RUNNER_PLUGINS.md](./OFFICIAL_RUNNER_PLUGINS.md)。
|
||||
|
||||
### 5. Runtime Control Plane v2（Foundation Partial）
|
||||
|
||||
当前 AgentRunner v1 主线仍以 `event -> binding -> runner.run(ctx) -> result stream` 为 runner 可见合同。Host 侧已经新增持久 `AgentRun` / `AgentRunEvent`、result persistence、cancel/finalize/query 等通用 run control primitives，并提供受权限保护的最小 runtime register/heartbeat/list、claim/renew/release 和 reconcile 原语。
|
||||
|
||||
在这些 Host 能力之上，可以构建独立 agent 管控面插件；插件负责 UI、策略和编排体验，runtime/task 的事实源仍由 Host 持有。完整 daemon supervisor、任务唤醒/长轮询/WebSocket、跨 Host 分布式锁、provider 登录态诊断和产品化业务队列仍是后续工作。
|
||||
|
||||
详见 [RUNTIME_CONTROL_PLANE_V2.md](./RUNTIME_CONTROL_PLANE_V2.md)。
|
||||
|
||||
## 约束事实源
|
||||
|
||||
本分支已确认约束不在 README 重写：
|
||||
|
||||
- Runner 可见协议、result stream 和调度边界见 [PROTOCOL_V1.md](./PROTOCOL_V1.md)。
|
||||
- Host 内部 `AgentConfig` / `AgentBinding` 投影见 [HOST_SDK_INFRASTRUCTURE.md](./HOST_SDK_INFRASTRUCTURE.md)。
|
||||
- 外部 EBA / Agent Platform / Runtime Control Plane 接入边界见 [EXTENSION_SCOPE_MATRIX.md](./EXTENSION_SCOPE_MATRIX.md)。
|
||||
@@ -0,0 +1,541 @@
|
||||
# Agent Platform / Runtime Control Plane Decision Note
|
||||
|
||||
本文档记录 AgentRunner 插件化之后，LangBot 如何继续演进成 Agent Platform 基础设施层。这里讨论的是 Host capability layer，不是 `AgentRunner Protocol v2`，也不是把某个具体 Agent Platform 产品写进 LangBot core。
|
||||
|
||||
> 本文是当前决策版。协议数据结构仍以 [PROTOCOL_V1.md](./PROTOCOL_V1.md) 为准；测试执行入口见 [AGENT_RUNNER_QA_GUIDE.md](./AGENT_RUNNER_QA_GUIDE.md)；扩展边界见 [EXTENSION_SCOPE_MATRIX.md](./EXTENSION_SCOPE_MATRIX.md)。
|
||||
>
|
||||
> 实现状态说明：本文描述的是 Runtime Control Plane v2 的目标能力和分阶段落地建议。当前 AgentRunner 插件化主线已经具备 event-first context、run-scoped authorization、EventLog / Transcript / State / sandbox 文件等 Host capability，并已落地持久 `AgentRun` / `AgentRunEvent` ledger、run control actions、最小 runtime heartbeat/claim lease 和 admin reconcile 原语。完整 Agent Platform 产品形态、daemon supervisor、runtime wakeup channel 和分布式 runtime 管控仍未完成。当前实现状态以 [STATUS.md](./STATUS.md) 为准。
|
||||
|
||||
## 1. 当前决策
|
||||
|
||||
LangBot 后续定位应更像 **Agent Host / infrastructure provider / transfer layer**，而不是把某个完整 Agent Platform 产品固化进 core。
|
||||
|
||||
结论：
|
||||
|
||||
- **Agent Platform 产品形态做成插件**。插件负责 agent 管理、策略、业务队列、UI、编排、多 agent 协作和产品体验。
|
||||
- **Agent Platform 所需的基础事实源做进 Host**。当前 Host 已保存 event、state、transcript、sandbox 文件边界、active run 权限快照、持久 run/result ledger、审计关联和通用控制状态。
|
||||
- **最小 runtime registry / heartbeat / claim lease 已作为 Host 原语落地，但不等于完整 daemon worker 管控**。远程 harness / daemon 的进程托管、wakeup channel、provider 登录态诊断和分布式调度仍可以先由 AgentRunner 插件和 SDK remote layer 自己维护。
|
||||
- **不把业务调度写进 Host**。Host 提供通用 run/result/control primitives，Platform 插件决定哪些事件触发哪些 agent、如何排队、如何分配、是否 fan-out。
|
||||
|
||||
推荐分层：
|
||||
|
||||
```text
|
||||
LangBot Host
|
||||
Current base: EventLog / runtime AgentBinding / State / Transcript / sandbox files / active run authorization
|
||||
Current v2 foundation: Run / RunEvent / audit / result persistence / control primitives / minimal runtime heartbeat and claim lease
|
||||
Planned: Agent / Binding persistence / daemon supervisor / wakeup channel / distributed runtime operations
|
||||
|
||||
Agent Platform plugin
|
||||
Agent management UI / project-task model / event routing policy
|
||||
Business queue / multi-agent orchestration / runtime selection policy
|
||||
|
||||
AgentRunner plugin / external harness runtime
|
||||
Connects ACP / remote daemon / local subprocess / HTTP API
|
||||
Executes and converts provider-native events to AgentRunResult
|
||||
```
|
||||
|
||||
## 2. Platform 与非 Platform 的区别
|
||||
|
||||
当前 LangBot 已经具备 Agent Host 的核心特征：
|
||||
|
||||
- 抹平不同 AgentRunner。
|
||||
- 从 IM / Pipeline 入口触发 runner。
|
||||
- 有 event-first context 方向。
|
||||
- 有 Host-owned EventLog / Transcript / State 和 sandbox/workspace 文件边界。
|
||||
- 有 runner config 下发和 active run-scoped authorization。
|
||||
- 有 `run_id` 串联 event、transcript、state、sandbox 文件和内存授权上下文。
|
||||
|
||||
这还不是完整 Agent Platform。完整 Platform 至少还需要：
|
||||
|
||||
- 可管理的 agent 资产：agent profile、binding、resource policy、runner config、可用状态。
|
||||
- 可观察的执行生命周期：run status、result stream、失败原因、文件引用、审计、回放。
|
||||
- 可运营的控制面：取消、重试、排队、并发、超时、恢复、诊断。
|
||||
- 可产品化的调度体验：事件订阅、路由策略、任务板、多 agent 协作、项目/工作区视图。
|
||||
|
||||
因此，区别不只是“有没有调度”，而是是否具备：
|
||||
|
||||
```text
|
||||
managed agent assets + observable run lifecycle + operational run control
|
||||
```
|
||||
|
||||
Host 负责这些能力的通用事实源和安全边界；Platform 插件负责把它们组装成具体产品。
|
||||
|
||||
### 2.1 当前实现边界
|
||||
|
||||
当前代码中的 `run_id` 已经连接 active run 授权、持久 run ledger 和多个 Host 事实源：
|
||||
|
||||
- `EventLog` 保存输入事件和审计入口，并记录 `run_id` / `runner_id`。
|
||||
- `Transcript` 保存对话历史投影，并用 `run_id` 关联 assistant 输出。
|
||||
- Sandbox/workspace 保存当前运行输入文件和 runner 产物，并用 `run_id` 做访问边界的一部分。
|
||||
- `PersistentStateStore` 保存 runner state，但不等同于 run lifecycle。
|
||||
- `AgentRunSessionRegistry` 保存 active run 的内存态授权快照，用于 proxy action 校验；进程结束或 run 结束后不作为可回放事实源。
|
||||
- `AgentRun` 保存 run lifecycle、scope、authorization snapshot、queue/claim 状态、cancel intent、usage/cost 和 metadata。
|
||||
- `AgentRunEvent` 保存 runner/result/admin event stream，按 `run_id + sequence` 做可回放分页。
|
||||
- `AgentRuntime` 保存最小 runtime registry / heartbeat 事实，用于 runtime list、stale mark 和 claim lease reconcile。
|
||||
|
||||
因此本文后续提到的 `AgentRun` / `AgentRunEvent`、`run_append_result`、`run_finalize`、`run_cancel`、`runtime_register`、`runtime_heartbeat`、`run_claim` 等基础原语已经存在。仍未完成的是独立 platform `run_create` action、Host-owned Agent / Binding 持久模型、业务队列产品形态、daemon supervisor、runtime wakeup channel、跨 Host 分布式锁和 provider/runtime 诊断面。
|
||||
|
||||
## 3. 基础概念
|
||||
|
||||
### 3.1 Event
|
||||
|
||||
Event 表示“发生了什么”：
|
||||
|
||||
```text
|
||||
message.received
|
||||
github.issue.opened
|
||||
scheduler.tick
|
||||
user.approved
|
||||
system.webhook.received
|
||||
```
|
||||
|
||||
EBA 负责把外部输入标准化成 event。Event 本身不是 queue，也不等同于一次 agent 执行。当前 `EventLog` 记录的是输入事件和审计事实；未来 `AgentRunEvent` 记录的是某次 run 的输出事件流，二者不能混用。
|
||||
|
||||
### 3.2 Run
|
||||
|
||||
Run 表示“某个 agent / binding / runner 针对某个 event 的一次执行”。
|
||||
|
||||
Run 应由 Host 持久化，成为执行状态、结果、权限和审计的事实源：
|
||||
|
||||
```text
|
||||
run_id
|
||||
event_id
|
||||
agent_id / binding_id
|
||||
runner_id
|
||||
status
|
||||
created_at / started_at / finished_at
|
||||
error / failure_reason
|
||||
delivery target
|
||||
metadata
|
||||
```
|
||||
|
||||
当前 `AgentRunSessionRegistry` 只保存 active run 的内存态授权信息，不足以支撑 Platform 的回放、审计、取消、重试和异步执行。
|
||||
|
||||
### 3.3 RunEvent / RunResult
|
||||
|
||||
RunEvent 是一次 run 过程中产生的结果事件流，对应 runner 返回的 `AgentRunResult`。它不同于 EBA/EventLog 的输入事件：
|
||||
|
||||
```text
|
||||
message.delta
|
||||
message.completed
|
||||
tool.call.started
|
||||
tool.call.completed
|
||||
state.updated
|
||||
action.requested
|
||||
run.completed
|
||||
run.failed
|
||||
```
|
||||
|
||||
Host 应保存这些输出事件，按 `run_id + sequence` 可回放。Transcript、State 可以由这些 result event 触发写入现有 store，并保留能回溯到 `AgentRunEvent` 的关联。文件和工具大结果留在当前 run 的 sandbox/workspace 中，不作为 result event blob 回传。
|
||||
|
||||
### 3.4 Queue
|
||||
|
||||
Queue 不是 EBA 的替代品。
|
||||
|
||||
EBA 负责产生 event；queue 负责处理“这个 event 对应的执行 work item 何时执行、谁来执行、如何取消/重试/恢复”。
|
||||
|
||||
队列可以分两层：
|
||||
|
||||
- **业务队列**：由 Platform 插件管理，例如项目任务、优先级、agent team、workflow、人工审批。
|
||||
- **执行队列 / run queue**：可选 Host 原语，例如 queued / running / completed / failed / cancelled、claim lease、dispatch timeout、orphan recovery。
|
||||
|
||||
第一阶段不要求 Host 内置完整执行队列。Platform 插件可以先管理业务队列；在 Phase 1 / Phase 2 能力落地前，插件仍只能通过现有 `AgentRunOrchestrator.run(...)` 同步执行路径和现有 Host stores 获得有限的 run 关联能力。
|
||||
|
||||
### 3.5 Runtime / Daemon
|
||||
|
||||
Runtime / daemon 表示执行位置或执行能力，例如某台机器上的 Claude Code / Codex CLI。
|
||||
|
||||
当前决策：
|
||||
|
||||
- Host 不在第一阶段维护完整 runtime registry。
|
||||
- AgentRunner 插件可以通过 SDK remote layer 与 daemon 保持连接、心跳和执行通道。
|
||||
- 外部 harness / agent 不应直接访问 LangBot Host 或数据库。访问 LangBot 资源必须通过 daemon / AgentRunner plugin / SDK runtime / `AgentRunAPIProxy` / scoped MCP bridge，并接受 run-scoped authorization 校验。
|
||||
- 如果后续多个插件都需要共享 runtime 状态，再把薄的 `RuntimeLease` / registry 下沉为 Host 通用能力。
|
||||
|
||||
## 4. Host 应新增的最小能力
|
||||
|
||||
第一阶段最重要的不是 daemon registry，而是让 Host 成为 run/result 的事实源。
|
||||
|
||||
### 4.1 AgentRun Store
|
||||
|
||||
新增持久 `AgentRun`：
|
||||
|
||||
```text
|
||||
id / run_id
|
||||
event_id
|
||||
agent_id
|
||||
binding_id
|
||||
runner_id
|
||||
conversation_id / thread_id
|
||||
workspace_id / bot_id
|
||||
status
|
||||
status_reason
|
||||
created_at / started_at / finished_at / updated_at
|
||||
deadline_at
|
||||
cancel_requested_at
|
||||
usage_json
|
||||
cost_json
|
||||
metadata_json
|
||||
```
|
||||
|
||||
建议 status 至少包含：
|
||||
|
||||
```text
|
||||
created
|
||||
running
|
||||
completed
|
||||
failed
|
||||
cancelled
|
||||
timeout
|
||||
```
|
||||
|
||||
如果后续加执行队列，再引入：
|
||||
|
||||
```text
|
||||
queued
|
||||
claimed
|
||||
dispatching
|
||||
```
|
||||
|
||||
### 4.2 AgentRunEvent Store
|
||||
|
||||
新增持久 `AgentRunEvent`：
|
||||
|
||||
```text
|
||||
id
|
||||
run_id
|
||||
sequence
|
||||
type
|
||||
data_json
|
||||
usage_json
|
||||
created_at
|
||||
source
|
||||
metadata_json
|
||||
```
|
||||
|
||||
约束：
|
||||
|
||||
- 同一 `run_id` 内 `sequence` 单调递增。
|
||||
- append 必须幂等，支持远程 daemon / plugin 重试。
|
||||
- 未知 result type 可保存但 Host 只对已知类型执行副作用。
|
||||
- 大 payload 仍应进入 sandbox/workspace，不直接塞入 result event。
|
||||
- `usage_json` 保存 `AgentRunResult.usage` 原样结构；缺失表示 unknown，不等于 0。
|
||||
|
||||
### 4.3 Run Control API
|
||||
|
||||
Host 提供通用控制原语：
|
||||
|
||||
```text
|
||||
run.create
|
||||
run.get
|
||||
run.list
|
||||
run.events.page
|
||||
run.cancel
|
||||
run.append_result
|
||||
run.finalize
|
||||
```
|
||||
|
||||
语义：
|
||||
|
||||
- `run.create` 创建 Host-owned run 和授权快照。
|
||||
- `run.append_result` 只允许受信 SDK/runtime 路径调用，必须绑定 run 创建时固化的授权快照，写入 `AgentRunEvent` 并触发 transcript/state/delivery 副作用。
|
||||
- `run.finalize` 关闭 run，更新 terminal status。
|
||||
- `run.cancel` 设置取消意图；同步 runner 通过 context/deadline 感知，远程 runner 通过插件/daemon 通道感知。
|
||||
|
||||
第一阶段可以只暴露给插件 runtime action，不一定先做公开 HTTP API。
|
||||
|
||||
### 4.4 Result Persistence In Orchestrator
|
||||
|
||||
当前 `AgentRunOrchestrator.run()` 已经处理：
|
||||
|
||||
```text
|
||||
event -> binding -> context -> runner invocation -> result normalization
|
||||
```
|
||||
|
||||
需要补齐：
|
||||
|
||||
- run 开始时创建 `AgentRun`。
|
||||
- 每个 `AgentRunResult` 进入 `AgentRunEvent`。
|
||||
- `run.completed` / 正常 generator 结束时标记 completed。
|
||||
- `run.failed` / exception / timeout 标记 failed 或 timeout。
|
||||
- terminal result 携带 usage 时，写入 `AgentRunEvent.usage_json` 并汇总到 `AgentRun.usage_json`。
|
||||
- `state.updated`、transcript 写入继续走现有 journal，但应与 `AgentRunEvent` 有可追踪关系。
|
||||
|
||||
### 4.5 Usage / Cost Accounting
|
||||
|
||||
SDK 侧 `AgentRunResult` 已提供可选 `usage` 字段，用于把不同 runner / external harness / provider-native event 的 token usage 归一到同一个 run result envelope。
|
||||
|
||||
语义：
|
||||
|
||||
- `run.completed.usage` SHOULD 表示本次 run 的最终聚合 token usage。
|
||||
- `run.failed.usage` MAY 表示失败前已知的部分 token usage。
|
||||
- 没有 usage 表示 upstream runtime 没有报告或 adapter 暂未接入；Host 不得按 0 计费或按 0 判断上下文消耗。
|
||||
- Host 应把 event-level usage 原样写入 `AgentRunEvent.usage_json`，并在 terminal event 或 finalize 阶段汇总到 `AgentRun.usage_json`。
|
||||
- cost 应由 Host 根据 usage、runner/model identity、发生时间和价格表计算，写入 `AgentRun.cost_json`；runner/provider 上报的 cost 只能作为非权威 telemetry 保留在 metadata 或 usage extra 中。
|
||||
|
||||
这层约束先解决协议位置和持久化位置；具体 ACP、remote daemon、local subprocess runner 如何从 native event 中抽取 usage，可在各插件后续适配。
|
||||
|
||||
### 4.6 Authorization Snapshot
|
||||
|
||||
异步或远程执行时，run 创建时必须固化授权快照：
|
||||
|
||||
- runner identity
|
||||
- binding identity
|
||||
- caller plugin identity
|
||||
- resource policy
|
||||
- allowed tools/models/files/knowledge bases/storage scopes
|
||||
- state scopes
|
||||
- conversation/thread/workspace scope
|
||||
|
||||
后续 append result、state API、history API 和 sandbox/workspace 文件访问都以这个 snapshot 校验，不重新扩大权限。
|
||||
|
||||
## 5. SDK 侧应新增的最小能力
|
||||
|
||||
SDK 不需要马上定义完整 daemon registry，但需要让插件和 runner 使用 Host run/result 能力。
|
||||
|
||||
### 5.1 Entities
|
||||
|
||||
新增或补齐：
|
||||
|
||||
```text
|
||||
AgentRun
|
||||
AgentRunStatus
|
||||
AgentRunEvent
|
||||
RunEventPage
|
||||
RunCreateRequest / RunCreateResult
|
||||
RunAppendResultRequest
|
||||
```
|
||||
|
||||
这些是 Host control primitives，不替代 `AgentRunContext` / `AgentRunResult`。
|
||||
|
||||
### 5.2 Proxy Methods
|
||||
|
||||
在 SDK proxy 中提供：
|
||||
|
||||
```python
|
||||
create_run(...)
|
||||
get_run(run_id)
|
||||
list_runs(...)
|
||||
page_run_events(run_id, cursor=None, limit=...)
|
||||
cancel_run(run_id)
|
||||
append_run_result(run_id, result, sequence=None)
|
||||
finalize_run(run_id, status, error=None)
|
||||
```
|
||||
|
||||
访问边界：
|
||||
|
||||
- 普通 AgentRunner 在同步 `run(ctx)` 内不一定需要直接调用这些 API；Host orchestrator 可自动记录。
|
||||
- Platform 插件可以创建/查询/取消 run。
|
||||
- AgentRunner 插件或 daemon bridge 可以 append/finalize 自己负责的 run。
|
||||
- 外部 harness 仍不能直接调用 Host；必须经 SDK runtime / proxy / bridge。
|
||||
|
||||
### 5.3 Plugin-Daemon Heartbeat
|
||||
|
||||
远程 daemon 的初始心跳可以是 SDK / AgentRunner plugin 私有能力：
|
||||
|
||||
```text
|
||||
daemon <-> AgentRunner plugin / SDK remote layer <-> LangBot plugin runtime <-> Host
|
||||
```
|
||||
|
||||
Host 第一阶段只需要知道：
|
||||
|
||||
- 相关插件是否在线。
|
||||
- run 是否有 progress/result。
|
||||
- run 是否超时或取消。
|
||||
|
||||
如果后续需要跨插件共享 daemon 可用性，再把 heartbeat/registry 下沉为 Host 能力。
|
||||
|
||||
## 6. Platform 插件应负责什么
|
||||
|
||||
Agent Platform 插件可以负责：
|
||||
|
||||
- 管理哪些 agent 可用。
|
||||
- 维护产品层 agent profile、项目、任务板、workflow、team。
|
||||
- 订阅 EBA event，决定哪些 event 触发哪些 agent。
|
||||
- 维护业务 queue：优先级、重试策略、人工审批、分配规则。
|
||||
- 选择 runner / runtime / daemon。
|
||||
- 在 Run Control API 落地后，调用 Host run API 创建、取消、查询执行。
|
||||
- 展示 run status、result stream、文件引用、失败原因和审计。
|
||||
|
||||
Platform 插件不应负责：
|
||||
|
||||
- 在 Host Run Ledger 落地后，私有保存通用 run/result 事实源。
|
||||
- 绕过 Host 直接写 transcript/state 或越权访问 sandbox/workspace 文件。
|
||||
- 让外部 harness 直接访问 LangBot DB 或 Host 内部资源。
|
||||
- 把某个业务队列语义强塞进 AgentRunner Protocol v1。
|
||||
|
||||
## 7. 与 EBA 的关系
|
||||
|
||||
EBA 做好后，事件流可以进入两种路径。
|
||||
|
||||
直接执行路径：
|
||||
|
||||
```text
|
||||
EventGateway
|
||||
-> EventRouter resolves AgentBinding
|
||||
-> AgentRunOrchestrator.run(event, binding)
|
||||
-> Host records AgentRun / AgentRunEvent (after Run Ledger lands)
|
||||
-> delivery
|
||||
```
|
||||
|
||||
Platform 插件编排路径：
|
||||
|
||||
```text
|
||||
EventGateway
|
||||
-> Platform plugin receives/subscribes event
|
||||
-> plugin applies policy / business queue
|
||||
-> plugin creates Host run (after Run Control API lands)
|
||||
-> runner/plugin/daemon executes
|
||||
-> Host records result and state
|
||||
-> plugin displays / Host delivers
|
||||
```
|
||||
|
||||
这两条路径最终应共享 Host run/result/state 事实源和 sandbox/workspace 文件边界。当前阶段可共享的是 event/transcript/state、sandbox 文件和同步执行链路；持久 run/result ledger 需要 Runtime Control Plane v2 Phase 1 补齐。区别在于是否有 Platform 插件参与产品化调度和业务队列。
|
||||
|
||||
## 8. 与 AgentRunner Protocol v1 的关系
|
||||
|
||||
本设计不改变 v1 的 runner 可见合同：
|
||||
|
||||
```text
|
||||
AgentRunContext -> AgentRunner.run(ctx) -> AgentRunResult stream
|
||||
```
|
||||
|
||||
必须保持：
|
||||
|
||||
- `AgentRunContext` 不塞入 daemon/worker/pod 细节。
|
||||
- `AgentRunResult` 仍是 runner 输出的统一事件流。
|
||||
- 普通 runner 不需要知道 task queue / runtime registry。
|
||||
- 远程 harness 可以自管 session、tool loop、MCP、上下文压缩，但访问 LangBot 资源必须通过 SDK proxy / bridge。
|
||||
- Runtime-managed execution 是 placement / transport 选择，不是普通 runner 协议的强制概念。
|
||||
|
||||
## 9. 分阶段实施建议
|
||||
|
||||
### Phase 1: Run Ledger（Foundation Implemented）
|
||||
|
||||
目标：Host 成为执行状态和结果事实源。
|
||||
|
||||
范围：
|
||||
|
||||
- `AgentRun` 表。
|
||||
- `AgentRunEvent` 表。
|
||||
- Orchestrator 自动创建/更新 run。
|
||||
- Journal 持久化每个 `AgentRunResult`。
|
||||
- Run 查询和事件分页 API。
|
||||
- SDK entities + proxy 方法。
|
||||
|
||||
复杂度：中等。
|
||||
|
||||
预计改动：
|
||||
|
||||
```text
|
||||
Host: 12-20 个文件
|
||||
SDK: 4-8 个文件
|
||||
Tests: 8-15 个文件
|
||||
```
|
||||
|
||||
### Phase 2: Platform Plugin Queue On Host Run Primitives（Control Primitives Partially Implemented; Product Queue Pending）
|
||||
|
||||
目标：Platform 插件管理业务 queue，Host 提供 run/result/cancel 原语。
|
||||
|
||||
范围：
|
||||
|
||||
- `run.create`
|
||||
- `run.cancel`
|
||||
- `run.append_result`
|
||||
- `run.finalize`
|
||||
- result append 的 sequence/idempotency。
|
||||
- 受权限保护的远程 append/finalize。
|
||||
- Platform 插件可基于 Host run 构建任务板和调度体验。
|
||||
|
||||
复杂度：中等偏高。
|
||||
|
||||
预计改动：
|
||||
|
||||
```text
|
||||
Host: 20-35 个文件
|
||||
SDK: 8-14 个文件
|
||||
Tests: 15-25 个文件
|
||||
```
|
||||
|
||||
### Phase 3: Optional Host Execution Queue / Claim Lease（Claim Lease Primitive Implemented; Full Queue Pending）
|
||||
|
||||
目标：当多个插件重复实现 claim/cancel/retry/recovery 时，再下沉执行队列到 Host。
|
||||
|
||||
范围：
|
||||
|
||||
- `queued/running/completed/failed/cancelled` 状态机扩展。
|
||||
- `claim_run` / `lease_until`。
|
||||
- dispatch timeout。
|
||||
- retry / orphan recovery。
|
||||
- cancel propagation。
|
||||
- 并发 claim 防重。
|
||||
|
||||
复杂度：高。
|
||||
|
||||
预计改动：
|
||||
|
||||
```text
|
||||
Host: 35-55 个文件
|
||||
SDK: 12-20 个文件
|
||||
Tests: 25-40 个文件
|
||||
```
|
||||
|
||||
### Phase 4: Optional Runtime Registry（Minimal Registry Implemented; Full Daemon Control Pending）
|
||||
|
||||
目标：当 Host 需要统一管理多个 daemon / worker 时，再引入 runtime registry。
|
||||
|
||||
范围：
|
||||
|
||||
- runtime register / heartbeat / deregister。
|
||||
- capability report：provider、version、login status、workspace access、slot。
|
||||
- runtime online/offline。
|
||||
- runtime scoped auth。
|
||||
- runtime audit。
|
||||
- runtime gone recovery。
|
||||
- task wakeup / long polling / websocket。
|
||||
- 多 Host 实例下的 relay / distributed lock。
|
||||
|
||||
复杂度：很高。
|
||||
|
||||
预计改动：
|
||||
|
||||
```text
|
||||
Host: 55-80+ 个文件
|
||||
SDK: 18-30 个文件
|
||||
Tests: 40+ 个文件
|
||||
```
|
||||
|
||||
不建议现在直接进入此阶段。
|
||||
|
||||
## 10. 设计原则
|
||||
|
||||
- 先把 run/result 事实源做进 Host，再谈完整 runtime control plane。
|
||||
- Agent Platform 产品做插件；Host 做基础设施。
|
||||
- Host 不写业务调度策略，但要保存通用状态、结果、权限和审计。
|
||||
- EBA event 不是 queue；queue 是执行生命周期问题。
|
||||
- 业务 queue 可以先在 Platform 插件里；执行 queue 只有在复用需求明确后再下沉 Host。
|
||||
- Daemon registry 不应污染 AgentRunner Protocol v1。
|
||||
- 外部 harness 不直接访问 LangBot Host 或 DB。
|
||||
- 所有 LangBot 资源访问必须走 SDK runtime / `AgentRunAPIProxy` / scoped MCP bridge。
|
||||
- Docker / remote / local subprocess 只是 runtime placement，不是 runner 协议差异。
|
||||
|
||||
## 11. 非目标
|
||||
|
||||
当前阶段不做：
|
||||
|
||||
- 完整 Multica 式 runtime registry。
|
||||
- Host 内置项目管理、任务板、agent team、workflow 产品逻辑。
|
||||
- 把 daemon heartbeat / worker liveness 放进 `AgentRunContext`。
|
||||
- 把业务 queue 定义为 AgentRunner Protocol 字段。
|
||||
- 让 Platform 插件私有保存 run/result 事实源。
|
||||
- 让外部 agent/harness 直连 Host 内部资源。
|
||||
|
||||
## 12. 待定问题
|
||||
|
||||
- Host 是否需要最小持久 `Agent` / `Binding` 模型，还是继续由 Pipeline / Platform 插件投影运行期 `AgentBinding`。
|
||||
- Platform 插件创建 run 时，是否传完整 `AgentBinding` snapshot，还是引用 Host-owned binding id。
|
||||
- `AgentRunEvent` 与现有 `EventLog` / `Transcript` 的查询关系：直接 join，还是通过专门 view 聚合。
|
||||
- `run.append_result` 的认证粒度：runner plugin identity、run token、scoped capability token，或 SDK runtime 内部 channel。
|
||||
- 取消语义：同步 runner、external harness runtime/session 如何统一感知 cancel。
|
||||
- 何时把插件私有 daemon heartbeat 提升为 Host `RuntimeLease`。
|
||||
- 若未来 Host 做 claim lease，Platform 插件业务 queue 与 Host execution queue 如何避免双队列混乱。
|
||||
@@ -0,0 +1,154 @@
|
||||
# Run Steering 与 Compaction Checkpoint（Design Note）
|
||||
|
||||
本文档记录两项 Host/runner 协作能力：**运行中消息注入（steering / follow-up）**和
|
||||
**压缩摘要持久化（compaction checkpoint）**。两者来自官方 local-agent 对照
|
||||
Pi agent harness（`pi-mono/packages/agent`，下称 pi-agent-core）的差距分析：
|
||||
local-agent 已移植 Pi 的事件生命周期、并行工具语义、hook 扩展点和压缩预算模型，
|
||||
这两项需要 Host 协议、授权与 runner turn 边界协同才能闭环。
|
||||
|
||||
> 本文是设计备忘，不是 schema 事实源。涉及的数据结构最终落到
|
||||
> [PROTOCOL_V1.md](./PROTOCOL_V1.md)；上下文边界语义以
|
||||
> [AGENT_CONTEXT_PROTOCOL.md](./AGENT_CONTEXT_PROTOCOL.md) 为准；
|
||||
> run 持久化与控制原语以 [RUNTIME_CONTROL_PLANE_V2.md](./RUNTIME_CONTROL_PLANE_V2.md) 为准。
|
||||
|
||||
## 1. Run Steering / Follow-up（运行中消息注入）
|
||||
|
||||
### 1.1 问题
|
||||
|
||||
IM 场景下用户在 agent 运行中追加消息非常常见（补充信息、纠正方向、"算了别查了"）。
|
||||
当前主线是 `one event -> one AgentBinding -> one run_id -> one runner`
|
||||
（PROTOCOL_V1 §13）：同会话的新消息要么等待当前 run 结束后触发新 run，
|
||||
要么并发触发独立 run。两种行为都无法把新消息送进**正在执行的 tool loop**，
|
||||
用户体验是"agent 自顾自跑完过期任务，然后才看到新消息"。
|
||||
|
||||
cancel（PROTOCOL_V1 §10）不解决这个问题：cancel 丢弃已完成的工作；
|
||||
steering 是在保留当前进度的前提下改变后续方向。
|
||||
|
||||
### 1.2 Pi 的参考语义
|
||||
|
||||
pi-agent-core 区分两个队列，注入时机都在 turn 边界，不打断进行中的模型流或工具执行：
|
||||
|
||||
- **steering**：运行中插入。当前 assistant 消息的全部 tool call 完成后、
|
||||
下一次模型调用前，注入排队的用户消息；模型在下一 turn 看到它们。
|
||||
- **follow-up**：排队后续工作。仅当没有 pending tool call 且没有 steering 消息、
|
||||
run 即将自然结束时检查；若有排队消息则注入并继续下一 turn，而不是结束 run。
|
||||
|
||||
两个队列各自支持 `one-at-a-time`（每次注入一条）和 `all`（一次注入全部）模式。
|
||||
|
||||
### 1.3 设计方向
|
||||
|
||||
职责划分遵循既有原则：Host 拥有事件路由和会话事实源，runner 拥有 turn 边界。
|
||||
|
||||
- **Host 侧**：BindingResolver / dispatch 层识别"同 conversation 存在 active run
|
||||
且 runner 声明支持 steering"的新消息事件，将其写入 run-scoped steering queue，
|
||||
并标记该事件已被在途 run 认领（不再触发新 run，避免破坏 §13 的基数约束）。
|
||||
事件仍照常进 EventLog / Transcript（事实源不变，改变的只是触发行为）。
|
||||
- **Runner 侧**：在 turn 边界（tool batch 完成后、下一次模型调用前，以及 run
|
||||
即将自然结束前）通过 run-scoped pull API 拉取 pending steering 输入，
|
||||
注入 working context。local-agent 的 `AgentLoopHooks.prepare_next_turn` /
|
||||
`should_stop_after_turn` 已预留了对应的注入点。
|
||||
- **能力协商**：runner manifest 声明 `steering` capability（参照 PROTOCOL_V1 §4.3）；
|
||||
未声明的 runner 保持现状（新消息按现有规则另起 run）。
|
||||
- **回执**：被 steering 消费的事件通过 EventLog 审计。原始 `message.received`
|
||||
记录在 `metadata.steering` 标记 queued/absorbed 与 `claimed_by_run_id`；
|
||||
runner 成功 pull 后，Host 追加 `steering.injected` 记录并引用源事件。
|
||||
run 结束时仍未被 pull 的已 claim 输入，Host 追加 `steering.dropped` 记录作为
|
||||
dispatch 终态；原始 Transcript 事实不删除。
|
||||
Transcript 继续只表示会话事实，不扩展 dispatch 行为字段。
|
||||
|
||||
已落地的协议面（最终定义归 PROTOCOL_V1）：
|
||||
|
||||
1. `ContextAccess.available_apis` 增加 steering pull 能力位。
|
||||
2. `AgentRunAPIProxy` 增加 steering 拉取 action：默认 `mode=all`，Host 保序返回全部
|
||||
pending 输入；`one-at-a-time` 仅作为 runner 主动节流选项。
|
||||
3. dispatch 层的"认领"规则：`message.received` 可被同 conversation 的 active run
|
||||
吸收，原事件写 EventLog / Transcript，dispatch 行为写入 EventLog metadata。
|
||||
4. Host 对单 run steering queue 设置内存上限，队列满时不再 claim 新消息，消息回到
|
||||
正常 dispatch 路径，避免 active run 无限吞入同会话输入。
|
||||
|
||||
### 1.4 边界
|
||||
|
||||
- 不引入 Host 替 runner 做 prompt 拼接：Host 只递队列，注入位置和格式由 runner 决定。
|
||||
- 不与 observer / fan-out 混淆：steering 仍是单 run 内的输入补充，不产生第二个 runner。
|
||||
- 远程 / 外部 harness runner（claude-code、codex 等）若其底层 session 自带
|
||||
steering 能力，adapter 可以直接转发；协议面保持一致。
|
||||
|
||||
## 2. Compaction Checkpoint 持久化
|
||||
|
||||
### 2.1 问题
|
||||
|
||||
local-agent 当前是无状态 runner：每次 run 重新拉取 transcript 尾部
|
||||
（默认 50 条）、重新估算 token、重新生成压缩摘要。后果：
|
||||
|
||||
- 长会话中每 run 重复压缩计算，摘要每次重新生成，不同 run 之间措辞漂移，
|
||||
对 provider KV cache 不友好（AGENT_CONTEXT_PROTOCOL §"Summary checkpoint 稳定"
|
||||
已写明期望：只有压缩发生时才产生新 checkpoint）。
|
||||
- 历史一旦超过 fetch limit，更早的内容永久不可见——没有 checkpoint 记录
|
||||
"已压缩到哪里、压缩出了什么"。
|
||||
|
||||
pi-agent-core 把 compaction 条目持久化进 session tree：摘要带
|
||||
`tokensBefore` 和覆盖范围，后续 turn 直接复用，只在再次越过阈值时增量压缩。
|
||||
|
||||
### 2.2 现状盘点
|
||||
|
||||
协议面和主消费路径已具备：
|
||||
|
||||
- State / Storage API 已定义（PROTOCOL_V1 §8 "State / Storage"），
|
||||
且 AGENT_CONTEXT_PROTOCOL 已点名 `summary.checkpoint` 是 state 的预期用法。
|
||||
- Host 会根据 binding state policy 暴露 `ContextAccess.available_apis.state`。
|
||||
- local-agent 会在 state API 可用时读取/写入 `runner.compaction.checkpoint`；
|
||||
缺失、schema 不匹配、conversation 不匹配或游标失败时回退尾部历史拉取。
|
||||
- LLM 生成摘要**不依赖**本项 Host 能力——runner 用已授权的 `invoke_llm`
|
||||
即可生成；checkpoint 只解决"存下来、下次复用"。
|
||||
|
||||
### 2.3 设计方向
|
||||
|
||||
- **存放位置**：state，scope=`conversation`（小 JSON，符合 PROTOCOL_V1 §8
|
||||
对 state/storage 的边界建议）。若未来摘要膨胀，超出部分放 storage 并在
|
||||
state 中留引用。
|
||||
- **key 约定**：`runner.compaction.checkpoint`（runner 命名空间内）。
|
||||
- **内容约定**（schema 落 PROTOCOL_V1 或 runner 文档，此处只列语义）：
|
||||
- `schema_version`
|
||||
- `summary`：压缩摘要文本（LLM 生成或确定性生成）
|
||||
- `covers_until`：已被摘要覆盖的 transcript 游标（seq / message id），
|
||||
是增量压缩和"从哪继续拉历史"的锚点
|
||||
- `tokens_before` / `created_at`：诊断与失效判断
|
||||
- **消费流程**：run 开始时读 checkpoint → 只拉取 `covers_until` 之后的
|
||||
transcript → 压缩触发时基于旧摘要增量生成新摘要、写回新 checkpoint。
|
||||
checkpoint 缺失或解析失败时回退到现行为（全量拉尾部），保证向后兼容。
|
||||
- **失效规则**：`covers_until` 在 Host transcript 中不存在（会话被清理 / 重置）
|
||||
即作废；runner 不得信任跨 conversation 的 checkpoint。
|
||||
- **授权**：Host 对声明需要 state 的 runner binding 开启
|
||||
`available_apis.state`；校验沿用现有 run-scoped state 校验
|
||||
（scope、key、value 大小、JSON 可序列化，见 PROTOCOL_V1 §7.2 对
|
||||
`state.updated` 的要求）。
|
||||
|
||||
### 2.4 相关但独立的工作
|
||||
|
||||
- **tokenizer / usage metadata 透传**：runner 目前用 chars/4 启发式估 token，
|
||||
对 CJK 偏低 3-4 倍，压缩触发系统性偏晚。Host 应在模型响应或
|
||||
`ctx.runtime.metadata` 透传 provider usage（prompt/completion tokens）与
|
||||
model context window（LiteLLM model-info 工作）。该项不阻塞 checkpoint
|
||||
落地，但决定压缩触发的准确性。
|
||||
|
||||
## 3. 实施拆分
|
||||
|
||||
| 项 | 归属 | 依赖 |
|
||||
| --- | --- | --- |
|
||||
| steering queue、事件认领、基础审计 | LangBot Host（dispatch / binding 层） | 已落地，含队列上限与未消费 dropped 终态 |
|
||||
| steering pull API + capability 位 | PROTOCOL_V1 + SDK proxy | 已落地 |
|
||||
| turn 边界拉取与注入 | langbot-local-agent | 已落地 |
|
||||
| local-agent 对 state API 的 checkpoint 读写 | langbot-local-agent | 已落地 |
|
||||
| checkpoint key / 内容 / 失效约定 | PROTOCOL_V1 + local-agent README | 已落地 |
|
||||
| LLM 压缩摘要生成 | langbot-local-agent | 已落地（`invoke_llm`，失败回退确定性摘要） |
|
||||
| usage / context-window metadata 透传 | LangBot Host（model 层） | LiteLLM model-info |
|
||||
|
||||
剩余工作应优先补 usage / context-window metadata。streaming delivery 衔接依赖
|
||||
`ctx.delivery` 编辑/追加语义，不建议在协议能力缺失时硬编码。
|
||||
|
||||
## 4. 开放问题
|
||||
|
||||
- streaming delivery 下 steering 注入后，前序 turn 已流出的内容与新 turn
|
||||
输出在 IM 消息编辑面的衔接（涉及 `ctx.delivery` 能力，待 delivery 演进定）。
|
||||
- checkpoint 是否需要 Host 侧主动失效通知（如会话清空时删除对应 state key）。
|
||||
当前实现靠 runner 读取时校验并回退，功能不阻塞。
|
||||
@@ -0,0 +1,211 @@
|
||||
# Agent Runner Security Boundary
|
||||
|
||||
本文档记录 agent-runner 插件化后的安全边界和最小护栏。
|
||||
|
||||
## 状态
|
||||
|
||||
**当前结论：不采用高强度监管模型。**
|
||||
|
||||
LangBot 的目标不是托管一个强隔离、不可信 code runner 平台。AgentRunner 插件，尤其是 ACP / Claude Code / Codex / OpenCode / Kimi Code 这类外部 harness，默认视为 **operator-owned execution**：用户或部署者显式配置并承担其文件系统、进程、网络、workspace、provider 登录态和 native tool 风险。
|
||||
|
||||
LangBot 需要负责的是保护 **LangBot 自己持有的资源**，包括模型、知识库、LangBot tools、history、event、state、plugin/workspace storage、sandbox/workspace 文件访问等。只要这些资源访问是 run-scoped、permission-scoped、可校验、可诊断的，当前阶段即可接受。
|
||||
|
||||
这意味着：
|
||||
|
||||
- 不要求 LangBot 在应用层实现完整 OS sandbox、VM、cgroup、seccomp、CPU / memory / network quota。
|
||||
- 不要求为 ACP runner 做复杂审批流；用户选择 ACP runner 即表示显式 opt-in。
|
||||
- 不要求在非 Docker 进程部署里做强监管；只要文档明确风险归属即可。
|
||||
- Docker / K8s 可以提供部署级隔离，但不是 LangBot agent-runner 协议发布的前置条件。
|
||||
- 不能宣传 LangBot 已经提供 managed sandbox；除非未来真的提供受管执行环境。
|
||||
|
||||
## 责任边界
|
||||
|
||||
### LangBot Host 负责
|
||||
|
||||
- **资源授权**：根据 runner manifest permissions、binding resource policy、run scope 生成本次 run 可访问的资源快照。
|
||||
- **运行期校验**：所有带 `run_id` 的 SDK / Host action 必须校验 active run session、caller plugin identity、resource id 和 operation。
|
||||
- **Scoped projection**：只把授权后的资源摘要、MCP server config、context、attachment/path ref、state snapshot 投影给 runner。
|
||||
- **LangBot 文件路径约束**：LangBot 自己 staged 和读取的文件必须限制在声明 root 内，防止 path escape。
|
||||
- **基础 secret 策略**：不要主动把 LangBot 持有的 API key / token / secret 投影给 runner；日志和错误里做常见 secret 字段脱敏。
|
||||
- **基础运行约束**：提供 timeout、取消传播、输出大小限制或错误映射的基础能力。
|
||||
- **audit-lite**：记录 event、run id、runner id、binding、资源授权摘要、关键失败、state/file/transcript 事实。
|
||||
|
||||
### Runner Plugin 负责
|
||||
|
||||
- 遵守 Host 下发的 `ctx.resources`、`ctx.context.available_apis`、runner config 和 state policy。
|
||||
- 把 LangBot 资源投影成目标平台可消费的形式，例如 MCP config、context prompt、HTTP header、run token。
|
||||
- 不绕过 SDK / Host action 直接访问 LangBot 内部资源。
|
||||
- 对自己启动的外部进程做合理封装，包括参数构造、timeout、取消、输出解析和错误映射。
|
||||
- 清楚记录自身 README 中的 provider 风险、部署假设和限制。
|
||||
|
||||
### 部署者 / 用户负责
|
||||
|
||||
- ACP / external harness 的 workspace 内容、文件系统访问、进程权限、网络访问、provider-native tool 权限。
|
||||
- Docker / K8s 的 image、volume、secret、network policy、resource limit、namespace、service account 配置。
|
||||
- 本机进程部署时的 OS 用户权限、PATH、HOME、CLI 登录态、全局配置和外部 MCP 配置。
|
||||
- 是否允许 runner 对某个目录执行真实写操作。
|
||||
|
||||
### 外部 Harness 负责
|
||||
|
||||
Claude Code、Codex、OpenCode、Kimi Code、Gemini CLI 等外部工具继续使用自己的权限模型、MCP 加载策略、session/resume、sandbox 或 approval 能力。LangBot 不承诺约束这些工具对其所在容器或宿主 OS 用户本来可访问资源的能力。
|
||||
|
||||
## 部署场景策略
|
||||
|
||||
| 场景 | LangBot 策略 | 不由 LangBot 承担 |
|
||||
| --- | --- | --- |
|
||||
| 普通进程部署 | 文档提示 operator-owned execution；Host 只保护 LangBot 资源。 | 阻止外部 CLI 读取同一 OS 用户可访问的文件、进程、HOME、全局 CLI 配置。 |
|
||||
| Docker / K8s 部署 | 继续使用相同 Host 资源边界；容器隔离由部署环境提供。 | 应用层重复实现容器/VM/cgroup/seccomp/network quota。 |
|
||||
| ACP runner | 用户显式选择 runner 和 workspace；LangBot 注入 scoped MCP / run token。 | ACP CLI native tools、workspace 写入、provider 登录态和外部 MCP 行为。 |
|
||||
| 外部 SaaS runner，例如 Dify | LangBot 通过 run token / gateway 限制 LangBot 资产访问。 | SaaS 平台内部 agent 执行策略、模型工具消息格式、平台侧日志。 |
|
||||
| 未来 managed runner | 只有当 LangBot 明确提供受管执行环境时，才需要单独定义强隔离 SLA。 | 当前协议闭环不承诺 managed sandbox。 |
|
||||
|
||||
## 最小护栏
|
||||
|
||||
以下是当前阶段需要维持的最小要求。它们是保护 LangBot 资源边界的要求，不是完整监管外部进程的要求。
|
||||
|
||||
### Resource Permission Boundary
|
||||
|
||||
每次 run 前必须冻结授权快照：
|
||||
|
||||
- runner manifest permissions 是资源访问上限。
|
||||
- binding resource policy / runner config 决定本次实际授权。
|
||||
- runtime action 按 `run_id` + `caller_plugin_identity` + resource id + operation 校验。
|
||||
- manifest permissions 只约束 LangBot 持有资源，不约束 external harness native tools。
|
||||
|
||||
当前实现方向是正确的：`AgentRunSessionRegistry` 保存 run-scoped snapshot，`plugin/handler.py` 对模型、工具、知识库、history、state、storage 等 action 做运行期校验，sandbox/workspace 文件访问由 scoped tool 边界控制。
|
||||
|
||||
**Skill 读写门控（不可弱化）**：pipeline-visible 的 skill 一次性以 `rw` 挂进同一 sandbox，mount 层不区分「可见」与「已激活」；写类 native 操作（write/edit/exec）只放行 activated skill，读类放行 visible + activated——这层区分等同资产授权语义，必须保留。skill 全 tool 化后尤其注意：「都是 tool」不等于「只控资产授权即可」，native 层的 visible/activated 门控不能砍。可弱化的只是 realpath 越界字符串检查（有 chroot/namespace 兜底）。
|
||||
|
||||
### MCP / Asset Gateway Boundary
|
||||
|
||||
LangBot MCP / asset gateway 只暴露当前 run 授权的工具面：
|
||||
|
||||
- `langbot_list_assets`
|
||||
- `langbot_get_current_event`
|
||||
- `langbot_history_page`
|
||||
- `langbot_retrieve_knowledge`
|
||||
- `langbot_get_tool_detail`
|
||||
- `langbot_call_tool`
|
||||
|
||||
外部平台需要使用短期 `run_token` 或 Authorization bearer token。token 缺失、错误或过期时必须拒绝访问。
|
||||
|
||||
不要求当前阶段实现 admin 级 MCP allowlist、dangerous tool approval 或复杂审批流。是否注册外部 MCP provider 是部署者/用户行为。
|
||||
|
||||
### Workspace / Path Boundary
|
||||
|
||||
LangBot 只需要约束自己管理的路径：
|
||||
|
||||
- Host staged 文件必须校验 `realpath` 和 root containment。
|
||||
- Attachment/file metadata 不应暴露 Host-only storage key / host path。
|
||||
- Context 文件、sandbox/workspace 文件如由 LangBot 创建，应放在可清理的位置。
|
||||
|
||||
用户配置给 ACP runner 的 workspace 不属于 LangBot 的强监管范围。Docker/K8s 下依赖 volume 挂载边界；普通进程部署下依赖 OS 用户权限和用户自担风险。
|
||||
|
||||
### Secret Handling
|
||||
|
||||
这里的 secret 指 API key、provider token、run token、MCP token、platform secret、数据库密码等。
|
||||
|
||||
当前阶段只要求基础策略：
|
||||
|
||||
- LangBot 不主动把自己持有的 secret 投影给 runner，除非这是 runner config 明确需要的外部服务凭据。
|
||||
- run token 是短期、run-scoped 的，不应长期保存。
|
||||
- 日志、错误、transcript、attachment/file metadata 尽量避免打印常见 secret 字段。
|
||||
- 配置 UI / API 返回时继续沿用现有 secret masking 规则。
|
||||
|
||||
不要求当前阶段实现完整 DLP、全链路敏感数据追踪、secret lineage 或自动轮换体系。
|
||||
|
||||
### Process / Runtime Bounds
|
||||
|
||||
LangBot 需要提供基本可控性：
|
||||
|
||||
- Host run deadline / runner timeout。
|
||||
- runner 侧请求 timeout。
|
||||
- generator close / cancel 传播。
|
||||
- 输出和 inline payload size 上限。
|
||||
- 错误映射为受控 runner failure。
|
||||
|
||||
不要求 LangBot 为外部 harness 实现 CPU、内存、磁盘、网络、进程树强隔离。需要这些能力时由 Docker/K8s、systemd、容器平台或用户机器策略提供。
|
||||
|
||||
### UI / Admin Surface
|
||||
|
||||
前端可以展示 runner 权限摘要，但它是信息披露，不是审批系统。
|
||||
|
||||
权限摘要指 runner manifest 声明的 LangBot 资源权限，例如：
|
||||
|
||||
- `tools.detail`
|
||||
- `tools.call`
|
||||
- `knowledge_bases.retrieve`
|
||||
- `history.page`
|
||||
- `storage.plugin`
|
||||
|
||||
当前阶段不要求强制弹窗、管理员审批、dangerous tool approval 或生产禁用开关。可以在 runner 配置区展示简短提示：此 runner 能访问哪些 LangBot 资源，外部 harness 执行风险由用户/部署者承担。
|
||||
|
||||
### Audit Lite
|
||||
|
||||
需要记录足够排查问题的事实：
|
||||
|
||||
- run id、runner id、binding、event。
|
||||
- 授权资源摘要。
|
||||
- state update、file write/read event、transcript message。
|
||||
- MCP / pull API 拒绝时的 warning。
|
||||
- steering queued / injected / dropped。
|
||||
|
||||
不要求当前阶段建立独立安全审计产品、审批记录系统或 SIEM 级事件模型。
|
||||
|
||||
## 降级后的检查表
|
||||
|
||||
| 项目 | 当前要求 | 状态判断 |
|
||||
| --- | --- | --- |
|
||||
| Path isolation | 只约束 LangBot 管理的 context/sandbox 文件路径；runner workspace 归用户/部署环境。 | Minimal required |
|
||||
| Permission boundary | 必须保护 LangBot 资源；不约束外部 CLI native 能力。 | Required |
|
||||
| Secret handling | 基础不投影、基础 masking、run token 短期化。 | Basic required |
|
||||
| MCP policy | run-scoped token + scoped tool surface；无复杂审批。 | Required |
|
||||
| Skill access policy | skill 通过 Host 授权 tool 暴露（发现 / activate / register / native exec 走统一 tool 授权）；**native 层 visible（只读）vs activated（可写）门控不可弱化**——所有 pipeline-visible skill 以 `rw` 挂进同一 sandbox，读写区分全靠 native 层；harness-native skill 文件不作为 LangBot 安全边界。 | Required |
|
||||
| Process isolation | 由 Docker/K8s/用户机器负责。 | Out of scope |
|
||||
| State lifecycle | scope 隔离、JSON size limit、基础 cleanup primitive。 | Basic required |
|
||||
| Audit | 记录运行事实和拒绝原因。 | Audit-lite |
|
||||
| UI / Admin control | 权限摘要可展示；不要求审批流。 | Optional |
|
||||
| Test matrix | 覆盖 run auth、MCP token、permission deny、timeout、sandbox path、state size。 | Focused tests |
|
||||
|
||||
## 当前实现快照
|
||||
|
||||
截至 2026-06-15，已有实现覆盖：
|
||||
|
||||
- SDK typed AgentRunner manifest、capabilities、permissions。
|
||||
- Host resource builder 按 manifest permissions 和 binding policy 生成 `ctx.resources`。
|
||||
- Active run session snapshot 和 `caller_plugin_identity` 校验。
|
||||
- History / event / state / tool / knowledge runtime action 的 run-scoped 校验。
|
||||
- Sandbox file path `realpath` + root containment。
|
||||
- Persistent state scope 隔离和 JSON size limit。
|
||||
- SDK-owned MCP bridge 和 long-lived asset gateway。
|
||||
- Dify / ACP runner 对 LangBot asset gateway 的接入。
|
||||
- Runner timeout、Dify HTTP timeout、ACP startup / initialize / request timeout。
|
||||
|
||||
仍可继续优化但不阻塞当前发布的事项：
|
||||
|
||||
- 前端展示 runner LangBot 资源权限摘要。
|
||||
- 常见 secret 字段 redaction 收敛成统一 helper。
|
||||
- Context/sandbox file TTL cleanup 调度。
|
||||
- 更完整的 MCP 调用 audit。
|
||||
- 更好的文档提示：ACP runner 是 operator-owned execution。
|
||||
|
||||
## 非目标
|
||||
|
||||
以下不属于当前 agent-runner pluginization 的安全目标：
|
||||
|
||||
- 防止 ACP / external harness 修改其 workspace。
|
||||
- 防止外部 CLI 读取同一容器或 OS 用户本来可读的文件。
|
||||
- 管控 external harness 的 provider-native tools、approval、MCP、browser、shell。
|
||||
- 在 LangBot 应用层实现 VM / container / cgroup / seccomp / network policy。
|
||||
- 为 Docker/K8s 部署替代平台自身的 secret、volume、network、resource limit 管理。
|
||||
- 实现企业级审批系统、SIEM、DLP 或安全运营面板。
|
||||
|
||||
## 发布口径
|
||||
|
||||
可以对外说明：
|
||||
|
||||
> AgentRunner 插件通过 run-scoped authorization 和 scoped MCP gateway 保护 LangBot 持有资源。外部 code harness 的执行环境由用户或部署平台负责隔离；LangBot 当前不提供 managed sandbox。
|
||||
|
||||
不能对外说明：
|
||||
|
||||
> LangBot 已经安全沙箱化 Claude Code / Codex / OpenCode 等外部 runner。
|
||||
@@ -0,0 +1,59 @@
|
||||
# AgentRunner Pluginization Status
|
||||
|
||||
本文档是 `docs/agent-runner-pluginization/` 的状态事实源。协议 schema 仍以 [PROTOCOL_V1.md](./PROTOCOL_V1.md) 为准；测试步骤以 [AGENT_RUNNER_QA_GUIDE.md](./AGENT_RUNNER_QA_GUIDE.md) 为准；安全发布门槛以 [SECURITY_HARDENING.md](./SECURITY_HARDENING.md) 为准。
|
||||
|
||||
状态快照日期：2026-06-20。
|
||||
|
||||
## 实现状态
|
||||
|
||||
| 领域 | 状态 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| SDK manifest schema | Done | `AgentRunnerManifest` 包含 typed `capabilities` / `permissions`；未知 capability / permission key 禁止进入 typed model。 |
|
||||
| Runner discovery | Done | Runtime 返回 typed manifest；Host registry 校验单个 runner，失败 warning + skip，不影响其它 runner。 |
|
||||
| Host resource authorization | Done | `ctx.resources` 和 `ctx.context.available_apis` 由 manifest permissions 与 binding policy / run scope 求交后生成。 |
|
||||
| Run authorization snapshot | Done | active run session 冻结 run-scoped resources 与 available APIs；runtime handler 按 snapshot 校验 pull API。 |
|
||||
| Result payload validation | Done | Wire 保持 `{type, data}`；Host 对投递/副作用类 payload 严格校验，tool-call telemetry 宽松，未知 type 忽略并 warning。 |
|
||||
| Old built-in runners | Done | 旧 `src/langbot/pkg/provider/runners/*` 与 `RequestRunner` 路径已从本分支删除。 |
|
||||
| Official runner manifests | Done | `local-agent`、ACP / Claude Code / Codex 外部 harness runner、外部服务 runner 已重新声明真实生效的 LangBot resource permissions。 |
|
||||
| Skill 链路 | Broken → Redesigning | 分支上 skill 激活链端到端悬空：`activate` 调用未定义的 `persist_activated_skill`（运行即 `AttributeError`）、`host.activated_skills` 只读不写、skill awareness 既未注入也未被 runner 消费。已拍板改为 **skill 全 tool 化**：发现走 `list_skills` / `langbot_list_assets` 加 skills 一类，`activate` / `register_skill` 走统一 tool 授权，`skill_authoring` capability 降级为便捷开关，host 直接写 `host.activated_skills`（last-write-wins）。 |
|
||||
| Runtime Control Plane v2 foundation | Partial | Host-owned `AgentRun` / `AgentRunEvent` ledger、orchestrator 自动建账、result event persistence、run get/list/event page/cancel/append/finalize actions 已落地；`agent_run:admin` / `runtime:admin` 控制权限、最小 runtime register/heartbeat/list/reconcile 和 run claim/renew/release 原语已落地。完整 Agent Platform 产品形态、daemon supervisor、任务唤醒/长轮询/WebSocket、分布式 runtime 管控仍未完成。 |
|
||||
| Security boundary | Done | 当前口径降级为轻量边界：LangBot 保护自身持有资源；external harness 的 OS / process / network / workspace 风险由用户或部署环境承担；managed sandbox 不是当前承诺。 |
|
||||
| Steering control path | Done | claim 异常不再逃逸 consumer loop；queue 有上限；未 pull 的 claimed 输入在 run 结束时写 `steering.dropped` 审计终态。 |
|
||||
| SDK v1 contract closure | Done | SDK 提供 `AgentAPIError` / `AgentAPIException`、typed `SteeringPullResult`、未知 result type 宽容解析、result `sequence` 注入与取消传播。 |
|
||||
|
||||
## Spec 与实现已知差距
|
||||
|
||||
- `action.requested` 仍只作为 telemetry / reserved surface；platform action executor 不在本分支执行。
|
||||
- EventGateway / EventRouter 完整实现由外部 EBA 分支联调；本分支只提供 event-first host envelope / binding / run 入口。
|
||||
- State 与 storage 的长期类型边界仍可继续收窄；当前合同只要求 JSON-safe state 与受控 storage API。
|
||||
- `ToolResource` 当前只带 `tool_name` / `tool_type` / `description` / `operations`，不含 `parameters` full schema；runner（如 local-agent `build_llm_tools`）需逐个 `get_tool_detail` 往返。拟在 `ToolResource` 增补 `parameters`，由 Host 在构造 `ctx.resources` 时一次塞齐。
|
||||
- EventLog / Transcript 已提供显式 cleanup primitive；长期 retention 默认值、TTL 调度接入和 sandbox/workspace 文件清理仍是运维收尾项，应在 Runtime Control Plane 产品化前补齐。
|
||||
- External harness 的 native shell / filesystem / CLI / MCP 权限不受 manifest permissions 约束；manifest permissions 只约束 LangBot 持有的资源访问。
|
||||
- LangBot 当前不承诺 managed sandbox；external harness 的 OS/process/network quota、workspace GC、provider-native tool 权限由用户或部署环境承担。
|
||||
- Runtime Control Plane v2 当前只落地 Host 事实源和控制原语；还没有内置 Agent Platform UI、业务队列、daemon 进程托管、runtime wakeup channel、跨 Host 分布式锁或 provider 登录态诊断。
|
||||
|
||||
## Runner 验收状态
|
||||
|
||||
| Runner | 状态 | 最近证据 |
|
||||
| --- | --- | --- |
|
||||
| `plugin:langbot/local-agent/default` | Unit-pass; UI smoke pending | 2026-06-10 本地 pytest / ruff 通过；WebUI smoke 由人工统一执行。 |
|
||||
| `plugin:langbot/acp-agent-runner/default` / `plugin:langbot/claude-code-agent/default` / `plugin:langbot/codex-agent/default` | Unit-pass; E2E pending | 通过 runner 仓库单测覆盖 session、run_id 注入和 LangBot MCP gateway；真实 harness E2E 取决于对应运行环境、CLI/daemon 可用性和 provider 登录态。 |
|
||||
| Dify / n8n / Coze / DashScope / Langflow / Tbox / DeerFlow / WeKnora | Unit-pass; credential smoke optional | 2026-06-13 plugin layout / parser tests 通过；真实服务凭据 smoke 非每轮必跑。 |
|
||||
|
||||
## Host / SDK 验收状态
|
||||
|
||||
| 范围 | 状态 | 最近证据 |
|
||||
| --- | --- | --- |
|
||||
| LangBot Runtime Control Plane v2 foundation | Unit-pass; product E2E pending | 2026-06-16 `tests/unit_tests/agent/test_run_ledger_store.py`、`test_run_ledger_api_auth.py`、`test_orchestrator_integration.py` 通过，覆盖 ledger、admin permissions、runtime heartbeat、claim/reconcile、orchestrator 持久化和取消传播。 |
|
||||
| SDK AgentRunner control entities / proxy | Unit-pass | 2026-06-16 SDK agent-runner 相关单测通过，覆盖 typed run ledger entities、AgentRunAPIProxy、MCP bridge、runtime manager 与 pull API handlers。 |
|
||||
|
||||
## 历史高价值记录
|
||||
|
||||
历史报告已合并为本状态页和 QA 指南，不再保留单独进度文档。后续若需要追溯，优先查看 `langbot-skills/reports/` 下的原始执行报告。
|
||||
|
||||
截至 2026-05-29，已有本地 smoke 证明：
|
||||
|
||||
- `local-agent` 可以通过 Pipeline Debug Chat 走插件化 `AgentRunOrchestrator` 主链路。
|
||||
- 外部 harness runner 可以通过同一条 `run(event, binding)` 路径执行；当前官方实现已收敛到 ACP / Claude Code / Codex 等直接 runner 插件。
|
||||
|
||||
这些记录只证明本地协议闭环可用，不代表 LangBot 提供 managed sandbox 或 external harness OS 级隔离。
|
||||
@@ -1,713 +0,0 @@
|
||||
# Workflow 系统开发者文档
|
||||
|
||||
本文档面向 LangBot 开发者，详细介绍 Workflow 系统的技术架构、核心组件和扩展方法。
|
||||
|
||||
## 目录
|
||||
|
||||
- [系统架构概述](#系统架构概述)
|
||||
- [目录结构](#目录结构)
|
||||
- [核心组件](#核心组件)
|
||||
- [后端模块](#后端模块)
|
||||
- [前端组件](#前端组件)
|
||||
- [数据库表结构](#数据库表结构)
|
||||
- [API 接口文档](#api-接口文档)
|
||||
- [如何添加新节点类型](#如何添加新节点类型)
|
||||
- [调试功能实现](#调试功能实现)
|
||||
|
||||
---
|
||||
|
||||
## 系统架构概述
|
||||
|
||||
Workflow 系统采用前后端分离架构，主要包含以下层次：
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ 前端层 (React) │
|
||||
│ ┌─────────────┬──────────────┬──────────────┬───────────┐ │
|
||||
│ │ 可视化编辑器 │ 节点面板 │ 属性面板 │ 调试器 │ │
|
||||
│ │ ReactFlow │ NodePalette │ PropertyPanel│ Debugger │ │
|
||||
│ └─────────────┴──────────────┴──────────────┴───────────┘ │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ API 层 (Quart) │
|
||||
│ ┌─────────────┬──────────────┬──────────────────────────┐ │
|
||||
│ │ Workflow API│ Debug API │ Node Types API │ │
|
||||
│ └─────────────┴──────────────┴──────────────────────────┘ │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ 核心引擎层 (Python) │
|
||||
│ ┌─────────────┬──────────────┬──────────────┬───────────┐ │
|
||||
│ │ Executor │ Registry │ Node │ Entities │ │
|
||||
│ │ 执行引擎 │ 节点注册表 │ 节点基类 │ 数据结构 │ │
|
||||
│ └─────────────┴──────────────┴──────────────┴───────────┘ │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ 存储层 (SQLAlchemy) │
|
||||
│ ┌─────────────┬──────────────┬──────────────────────────┐ │
|
||||
│ │ Workflow │ Executions │ Triggers │ │
|
||||
│ └─────────────┴──────────────┴──────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 目录结构
|
||||
|
||||
### 后端代码结构
|
||||
|
||||
```
|
||||
LangBot/src/langbot/pkg/
|
||||
├── workflow/ # Workflow 核心模块
|
||||
│ ├── __init__.py # 模块初始化，导出公共接口
|
||||
│ ├── entities.py # 数据实体定义
|
||||
│ ├── executor.py # 执行引擎
|
||||
│ ├── node.py # 节点基类和装饰器
|
||||
│ ├── registry.py # 节点类型注册表
|
||||
│ └── nodes/ # 内置节点实现
|
||||
│ ├── __init__.py # 注册所有内置节点
|
||||
│ ├── trigger.py # 触发节点
|
||||
│ ├── process.py # 处理节点
|
||||
│ ├── control.py # 控制节点
|
||||
│ └── action.py # 动作节点
|
||||
├── entity/persistence/
|
||||
│ └── workflow.py # 数据库模型
|
||||
├── api/http/
|
||||
│ ├── controller/groups/workflows/
|
||||
│ │ └── workflows.py # API 路由控制器
|
||||
│ └── service/
|
||||
│ └── workflow.py # 业务逻辑服务
|
||||
└── persistence/migrations/
|
||||
└── dbm026_workflow_tables.py # 数据库迁移
|
||||
```
|
||||
|
||||
### 前端代码结构
|
||||
|
||||
```
|
||||
LangBot/web/src/app/home/workflows/
|
||||
├── page.tsx # Workflow 列表页
|
||||
├── WorkflowDetailContent.tsx # 详情页内容
|
||||
├── store/
|
||||
│ └── useWorkflowStore.ts # Zustand 状态管理
|
||||
└── components/
|
||||
├── workflow-editor/ # 可视化编辑器
|
||||
│ ├── index.ts # 导出
|
||||
│ ├── WorkflowEditorComponent.tsx # 主编辑器组件
|
||||
│ ├── WorkflowNodeComponent.tsx # 自定义节点组件
|
||||
│ ├── NodePalette.tsx # 节点面板
|
||||
│ ├── PropertyPanel.tsx # 属性面板
|
||||
│ └── node-configs/ # 节点配置元数据
|
||||
│ ├── types.ts # 配置类型定义
|
||||
│ ├── trigger-configs.ts
|
||||
│ ├── ai-configs.ts
|
||||
│ ├── process-configs.ts
|
||||
│ ├── control-configs.ts
|
||||
│ ├── action-configs.ts
|
||||
│ ├── integration-configs.ts
|
||||
│ └── index.ts # 配置汇总
|
||||
├── workflow-debugger/ # 调试器组件
|
||||
│ ├── index.ts
|
||||
│ └── WorkflowDebugger.tsx
|
||||
├── workflow-form/ # 表单组件
|
||||
│ └── WorkflowFormComponent.tsx
|
||||
└── workflow-executions/ # 执行历史组件
|
||||
└── WorkflowExecutionsTab.tsx
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 核心组件
|
||||
|
||||
### 后端模块
|
||||
|
||||
#### 1. 执行引擎 (WorkflowExecutor)
|
||||
|
||||
位置：[`executor.py`](../../src/langbot/pkg/workflow/executor.py)
|
||||
|
||||
执行引擎负责工作流的实际执行，包括：
|
||||
|
||||
- **拓扑排序**：确定节点执行顺序
|
||||
- **节点执行**：调用各节点的 execute 方法
|
||||
- **控制流处理**：处理条件分支、循环、并行执行
|
||||
- **错误处理**：支持重试机制
|
||||
|
||||
```python
|
||||
class WorkflowExecutor:
|
||||
async def execute(
|
||||
self,
|
||||
workflow: WorkflowDefinition,
|
||||
context: ExecutionContext,
|
||||
start_node_id: Optional[str] = None
|
||||
) -> ExecutionContext:
|
||||
"""执行工作流"""
|
||||
# 1. 构建执行图
|
||||
# 2. 初始化节点状态
|
||||
# 3. 找到起始节点
|
||||
# 4. 按拓扑顺序执行
|
||||
```
|
||||
|
||||
**调试执行器 (DebugWorkflowExecutor)**
|
||||
|
||||
继承自 WorkflowExecutor，增加了调试支持：
|
||||
|
||||
- 断点支持
|
||||
- 单步执行
|
||||
- 暂停/继续
|
||||
- 实时日志
|
||||
|
||||
```python
|
||||
class DebugWorkflowExecutor(WorkflowExecutor):
|
||||
async def execute_debug(
|
||||
self,
|
||||
workflow: WorkflowDefinition,
|
||||
context: ExecutionContext,
|
||||
debug_state: DebugExecutionState,
|
||||
) -> ExecutionContext:
|
||||
"""调试模式执行"""
|
||||
```
|
||||
|
||||
#### 2. 节点注册表 (NodeTypeRegistry)
|
||||
|
||||
位置：[`registry.py`](../../src/langbot/pkg/workflow/registry.py)
|
||||
|
||||
单例模式管理所有节点类型：
|
||||
|
||||
```python
|
||||
class NodeTypeRegistry:
|
||||
_instance: Optional['NodeTypeRegistry'] = None
|
||||
|
||||
def register(self, node_type: str, node_class: type[WorkflowNode]):
|
||||
"""注册节点类型"""
|
||||
|
||||
def create_instance(self, node_type: str, node_id: str, config: dict) -> WorkflowNode:
|
||||
"""创建节点实例"""
|
||||
|
||||
def list_all(self) -> list[dict]:
|
||||
"""获取所有节点类型的 Schema"""
|
||||
```
|
||||
|
||||
#### 3. 节点基类 (WorkflowNode)
|
||||
|
||||
位置：[`node.py`](../../src/langbot/pkg/workflow/node.py)
|
||||
|
||||
所有节点必须继承此基类：
|
||||
|
||||
```python
|
||||
class WorkflowNode(abc.ABC):
|
||||
# 节点元数据
|
||||
type_name: str = ""
|
||||
name: str = ""
|
||||
description: str = ""
|
||||
category: str = "misc"
|
||||
icon: str = ""
|
||||
|
||||
# 端口定义
|
||||
inputs: list[NodePort] = []
|
||||
outputs: list[NodePort] = []
|
||||
|
||||
# 配置 Schema
|
||||
config_schema: list[NodeConfig] = []
|
||||
|
||||
@abc.abstractmethod
|
||||
async def execute(
|
||||
self,
|
||||
inputs: dict[str, Any],
|
||||
context: ExecutionContext
|
||||
) -> dict[str, Any]:
|
||||
"""执行节点逻辑"""
|
||||
pass
|
||||
```
|
||||
|
||||
#### 4. 数据实体 (entities.py)
|
||||
|
||||
主要数据结构：
|
||||
|
||||
```python
|
||||
class WorkflowDefinition:
|
||||
"""工作流定义"""
|
||||
uuid: str
|
||||
name: str
|
||||
nodes: list[NodeDefinition]
|
||||
edges: list[EdgeDefinition]
|
||||
settings: WorkflowSettings
|
||||
|
||||
class ExecutionContext:
|
||||
"""执行上下文"""
|
||||
execution_id: str
|
||||
workflow_id: str
|
||||
status: ExecutionStatus
|
||||
variables: dict
|
||||
node_states: dict[str, NodeState]
|
||||
history: list[ExecutionStep]
|
||||
```
|
||||
|
||||
### 前端组件
|
||||
|
||||
#### 1. WorkflowEditorComponent
|
||||
|
||||
主编辑器组件，基于 React Flow 实现：
|
||||
|
||||
- **画布交互**：拖拽、缩放、平移
|
||||
- **节点连接**：自动验证端口类型
|
||||
- **撤销/重做**：基于历史记录栈
|
||||
- **复制/粘贴**：支持多选复制
|
||||
|
||||
关键功能：
|
||||
|
||||
```tsx
|
||||
function WorkflowEditorInner() {
|
||||
const { nodes, edges, onNodesChange, onEdgesChange, onConnect } = useWorkflowStore();
|
||||
|
||||
// 拖放添加节点
|
||||
const onDrop = useCallback((event: React.DragEvent) => {
|
||||
const type = event.dataTransfer.getData('application/reactflow');
|
||||
const position = screenToFlowPosition({ x: event.clientX, y: event.clientY });
|
||||
addNode(type, position);
|
||||
}, []);
|
||||
|
||||
// 复制粘贴
|
||||
const handleCopy = useCallback(() => { ... }, []);
|
||||
const handlePaste = useCallback(() => { ... }, []);
|
||||
}
|
||||
```
|
||||
|
||||
#### 2. NodePalette
|
||||
|
||||
节点面板组件，展示可用节点类型：
|
||||
|
||||
```tsx
|
||||
function NodePalette() {
|
||||
// 按类别组织节点
|
||||
const categories = [
|
||||
{ id: 'trigger', name: '触发节点', icon: Zap },
|
||||
{ id: 'ai', name: 'AI 节点', icon: Brain },
|
||||
{ id: 'process', name: '处理节点', icon: Cpu },
|
||||
{ id: 'control', name: '控制节点', icon: GitBranch },
|
||||
{ id: 'action', name: '动作节点', icon: Send },
|
||||
{ id: 'integration', name: '集成节点', icon: Plug },
|
||||
];
|
||||
|
||||
// 拖拽开始
|
||||
const onDragStart = (event: React.DragEvent, nodeType: string) => {
|
||||
event.dataTransfer.setData('application/reactflow', nodeType);
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
#### 3. PropertyPanel
|
||||
|
||||
属性面板组件，动态渲染节点配置表单：
|
||||
|
||||
```tsx
|
||||
function PropertyPanel() {
|
||||
const { selectedNodeId, nodes, updateNodeData } = useWorkflowStore();
|
||||
|
||||
// 根据节点类型获取配置元数据
|
||||
const selectedNode = nodes.find(n => n.id === selectedNodeId);
|
||||
const nodeConfig = getNodeConfig(selectedNode?.data?.nodeType);
|
||||
|
||||
// 动态渲染配置字段
|
||||
return (
|
||||
<div>
|
||||
{nodeConfig?.fields.map(field => (
|
||||
<ConfigField key={field.name} field={field} />
|
||||
))}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
#### 4. WorkflowDebugger
|
||||
|
||||
调试器组件，支持实时调试：
|
||||
|
||||
```tsx
|
||||
function WorkflowDebugger({ workflowUuid, workflow }) {
|
||||
const [debugState, setDebugState] = useState<DebugState>('idle');
|
||||
const [executionId, setExecutionId] = useState<string>('');
|
||||
const [logs, setLogs] = useState<ExecutionLog[]>([]);
|
||||
|
||||
// 启动调试
|
||||
const startDebug = async () => {
|
||||
const result = await backendClient.post(
|
||||
`/api/v1/workflows/${workflowUuid}/debug/start`,
|
||||
{ context, variables, breakpoints }
|
||||
);
|
||||
setExecutionId(result.execution_id);
|
||||
};
|
||||
|
||||
// 轮询状态
|
||||
useEffect(() => {
|
||||
if (debugState === 'running') {
|
||||
const interval = setInterval(fetchState, 500);
|
||||
return () => clearInterval(interval);
|
||||
}
|
||||
}, [debugState]);
|
||||
}
|
||||
```
|
||||
|
||||
#### 5. useWorkflowStore
|
||||
|
||||
Zustand 状态管理：
|
||||
|
||||
```typescript
|
||||
interface WorkflowState {
|
||||
nodes: WorkflowNode[];
|
||||
edges: WorkflowEdge[];
|
||||
selectedNodeId: string | null;
|
||||
history: HistoryEntry[];
|
||||
historyIndex: number;
|
||||
isDirty: boolean;
|
||||
|
||||
// Actions
|
||||
addNode: (type: string, position: XYPosition) => void;
|
||||
updateNodeData: (nodeId: string, data: Partial<NodeData>) => void;
|
||||
deleteNode: (nodeId: string) => void;
|
||||
undo: () => void;
|
||||
redo: () => void;
|
||||
}
|
||||
|
||||
export const useWorkflowStore = create<WorkflowState>((set, get) => ({
|
||||
// ... state and actions
|
||||
}));
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 数据库表结构
|
||||
|
||||
### workflows 表
|
||||
|
||||
```sql
|
||||
CREATE TABLE workflows (
|
||||
uuid VARCHAR(255) PRIMARY KEY,
|
||||
name VARCHAR(255) NOT NULL,
|
||||
description TEXT,
|
||||
emoji VARCHAR(10) DEFAULT '🔄',
|
||||
version INTEGER DEFAULT 1,
|
||||
is_enabled BOOLEAN DEFAULT TRUE,
|
||||
definition JSON NOT NULL, -- 节点和边定义
|
||||
global_config JSON DEFAULT '{}', -- 全局配置
|
||||
extensions_preferences JSON, -- 插件和 MCP 配置
|
||||
created_at TIMESTAMP,
|
||||
updated_at TIMESTAMP
|
||||
);
|
||||
```
|
||||
|
||||
### workflow_versions 表
|
||||
|
||||
```sql
|
||||
CREATE TABLE workflow_versions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
workflow_uuid VARCHAR(255) NOT NULL,
|
||||
version INTEGER NOT NULL,
|
||||
definition JSON NOT NULL,
|
||||
global_config JSON DEFAULT '{}',
|
||||
created_at TIMESTAMP,
|
||||
created_by VARCHAR(255),
|
||||
UNIQUE(workflow_uuid, version)
|
||||
);
|
||||
```
|
||||
|
||||
### workflow_executions 表
|
||||
|
||||
```sql
|
||||
CREATE TABLE workflow_executions (
|
||||
uuid VARCHAR(255) PRIMARY KEY,
|
||||
workflow_uuid VARCHAR(255) NOT NULL,
|
||||
workflow_version INTEGER NOT NULL,
|
||||
status VARCHAR(20) NOT NULL, -- pending/running/completed/failed/cancelled
|
||||
trigger_type VARCHAR(50),
|
||||
trigger_data JSON,
|
||||
variables JSON,
|
||||
start_time TIMESTAMP,
|
||||
end_time TIMESTAMP,
|
||||
error TEXT,
|
||||
created_at TIMESTAMP
|
||||
);
|
||||
```
|
||||
|
||||
### workflow_node_executions 表
|
||||
|
||||
```sql
|
||||
CREATE TABLE workflow_node_executions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
execution_uuid VARCHAR(255) NOT NULL,
|
||||
node_id VARCHAR(100) NOT NULL,
|
||||
node_type VARCHAR(50) NOT NULL,
|
||||
status VARCHAR(20) NOT NULL,
|
||||
inputs JSON,
|
||||
outputs JSON,
|
||||
start_time TIMESTAMP,
|
||||
end_time TIMESTAMP,
|
||||
error TEXT,
|
||||
retry_count INTEGER DEFAULT 0
|
||||
);
|
||||
```
|
||||
|
||||
### workflow_triggers 表
|
||||
|
||||
```sql
|
||||
CREATE TABLE workflow_triggers (
|
||||
uuid VARCHAR(255) PRIMARY KEY,
|
||||
workflow_uuid VARCHAR(255) NOT NULL,
|
||||
type VARCHAR(50) NOT NULL, -- message/cron/event/webhook
|
||||
config JSON NOT NULL,
|
||||
is_enabled BOOLEAN DEFAULT TRUE,
|
||||
priority INTEGER DEFAULT 0,
|
||||
created_at TIMESTAMP,
|
||||
updated_at TIMESTAMP
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API 接口文档
|
||||
|
||||
### Workflow CRUD
|
||||
|
||||
| 方法 | 路径 | 描述 |
|
||||
|-----|------|------|
|
||||
| GET | `/api/v1/workflows` | 获取工作流列表 |
|
||||
| POST | `/api/v1/workflows` | 创建工作流 |
|
||||
| GET | `/api/v1/workflows/:uuid` | 获取单个工作流 |
|
||||
| PUT | `/api/v1/workflows/:uuid` | 更新工作流 |
|
||||
| DELETE | `/api/v1/workflows/:uuid` | 删除工作流 |
|
||||
| POST | `/api/v1/workflows/:uuid/copy` | 复制工作流 |
|
||||
|
||||
### 执行相关
|
||||
|
||||
| 方法 | 路径 | 描述 |
|
||||
|-----|------|------|
|
||||
| POST | `/api/v1/workflows/:uuid/execute` | 手动执行工作流 |
|
||||
| GET | `/api/v1/workflows/:uuid/executions` | 获取执行记录 |
|
||||
|
||||
### 版本管理
|
||||
|
||||
| 方法 | 路径 | 描述 |
|
||||
|-----|------|------|
|
||||
| GET | `/api/v1/workflows/:uuid/versions` | 获取版本列表 |
|
||||
| POST | `/api/v1/workflows/:uuid/rollback/:version` | 回滚到指定版本 |
|
||||
|
||||
### 调试 API
|
||||
|
||||
| 方法 | 路径 | 描述 |
|
||||
|-----|------|------|
|
||||
| POST | `/api/v1/workflows/:uuid/debug/start` | 启动调试 |
|
||||
| POST | `/api/v1/workflows/:uuid/debug/:exec_id/pause` | 暂停执行 |
|
||||
| POST | `/api/v1/workflows/:uuid/debug/:exec_id/resume` | 继续执行 |
|
||||
| POST | `/api/v1/workflows/:uuid/debug/:exec_id/stop` | 停止执行 |
|
||||
| POST | `/api/v1/workflows/:uuid/debug/:exec_id/step` | 单步执行 |
|
||||
| GET | `/api/v1/workflows/:uuid/debug/:exec_id/state` | 获取调试状态 |
|
||||
|
||||
### 节点类型
|
||||
|
||||
| 方法 | 路径 | 描述 |
|
||||
|-----|------|------|
|
||||
| GET | `/api/v1/workflows/_/node-types` | 获取所有节点类型 |
|
||||
| GET | `/api/v1/workflows/_/node-types/categories` | 按类别获取节点类型 |
|
||||
|
||||
---
|
||||
|
||||
## 如何添加新节点类型
|
||||
|
||||
### 步骤 1：创建节点类
|
||||
|
||||
在 `LangBot/src/langbot/pkg/workflow/nodes/` 下创建或修改文件：
|
||||
|
||||
```python
|
||||
from ..node import WorkflowNode, NodePort, NodeConfig, workflow_node
|
||||
from ..entities import ExecutionContext
|
||||
|
||||
@workflow_node('my_custom_node')
|
||||
class MyCustomNode(WorkflowNode):
|
||||
"""自定义节点"""
|
||||
|
||||
# 元数据
|
||||
type_name = 'my_custom_node'
|
||||
name = '我的自定义节点'
|
||||
description = '这是一个自定义节点'
|
||||
category = 'process' # trigger/process/control/action/integration
|
||||
icon = '🔧'
|
||||
|
||||
# 输入端口
|
||||
inputs = [
|
||||
NodePort(name='input', type='string', description='输入数据', required=True),
|
||||
]
|
||||
|
||||
# 输出端口
|
||||
outputs = [
|
||||
NodePort(name='output', type='string', description='输出数据'),
|
||||
]
|
||||
|
||||
# 配置字段
|
||||
config_schema = [
|
||||
NodeConfig(
|
||||
name='option',
|
||||
type='select',
|
||||
required=True,
|
||||
options=['选项A', '选项B'],
|
||||
description='选择一个选项'
|
||||
),
|
||||
NodeConfig(
|
||||
name='value',
|
||||
type='string',
|
||||
required=False,
|
||||
default='默认值',
|
||||
description='配置值'
|
||||
),
|
||||
]
|
||||
|
||||
async def execute(
|
||||
self,
|
||||
inputs: dict[str, Any],
|
||||
context: ExecutionContext
|
||||
) -> dict[str, Any]:
|
||||
"""执行节点逻辑"""
|
||||
input_data = inputs.get('input', '')
|
||||
option = self.get_config('option')
|
||||
value = self.get_config('value', '')
|
||||
|
||||
# 处理逻辑
|
||||
result = f"处理: {input_data} with {option} and {value}"
|
||||
|
||||
return {'output': result}
|
||||
```
|
||||
|
||||
### 步骤 2：注册节点
|
||||
|
||||
在 `LangBot/src/langbot/pkg/workflow/nodes/__init__.py` 中导入：
|
||||
|
||||
```python
|
||||
from .process import (
|
||||
CodeExecutorNode,
|
||||
HttpRequestNode,
|
||||
DataTransformNode,
|
||||
MyCustomNode, # 添加新节点
|
||||
)
|
||||
```
|
||||
|
||||
### 步骤 3：添加前端配置
|
||||
|
||||
在 `LangBot/web/src/app/home/workflows/components/workflow-editor/node-configs/` 目录下添加配置：
|
||||
|
||||
```typescript
|
||||
// process-configs.ts
|
||||
export const processNodeConfigs: NodeConfigMap = {
|
||||
// ... 其他配置
|
||||
|
||||
my_custom_node: {
|
||||
type: 'my_custom_node',
|
||||
label: 'workflows.nodes.myCustomNode',
|
||||
description: 'workflows.nodes.myCustomNodeDesc',
|
||||
icon: 'Wrench',
|
||||
category: 'process',
|
||||
fields: [
|
||||
{
|
||||
name: 'option',
|
||||
type: 'select',
|
||||
label: 'workflows.fields.option',
|
||||
required: true,
|
||||
options: [
|
||||
{ value: '选项A', label: '选项 A' },
|
||||
{ value: '选项B', label: '选项 B' },
|
||||
],
|
||||
},
|
||||
{
|
||||
name: 'value',
|
||||
type: 'string',
|
||||
label: 'workflows.fields.value',
|
||||
required: false,
|
||||
defaultValue: '默认值',
|
||||
},
|
||||
],
|
||||
},
|
||||
};
|
||||
```
|
||||
|
||||
### 步骤 4：添加国际化
|
||||
|
||||
在 `LangBot/web/src/i18n/locales/` 中添加翻译：
|
||||
|
||||
```typescript
|
||||
// zh-Hans.ts
|
||||
workflows: {
|
||||
nodes: {
|
||||
myCustomNode: '我的自定义节点',
|
||||
myCustomNodeDesc: '这是一个自定义节点',
|
||||
},
|
||||
fields: {
|
||||
option: '选项',
|
||||
value: '值',
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 调试功能实现
|
||||
|
||||
### 后端调试状态管理
|
||||
|
||||
```python
|
||||
class DebugExecutionState:
|
||||
"""调试执行状态"""
|
||||
|
||||
def __init__(self, execution_id: str, breakpoints: list[str] = None):
|
||||
self.execution_id = execution_id
|
||||
self.status: str = 'running'
|
||||
self.is_paused: bool = False
|
||||
self.is_stopped: bool = False
|
||||
self.breakpoints: set[str] = set(breakpoints or [])
|
||||
self.logs: list[ExecutionLog] = []
|
||||
self._pause_event = asyncio.Event()
|
||||
|
||||
def pause(self):
|
||||
"""暂停执行"""
|
||||
self.is_paused = True
|
||||
self._pause_event.clear()
|
||||
|
||||
def resume(self):
|
||||
"""继续执行"""
|
||||
self.is_paused = False
|
||||
self._pause_event.set()
|
||||
|
||||
async def wait_if_paused(self):
|
||||
"""如果暂停则等待"""
|
||||
if self.is_paused:
|
||||
await self._pause_event.wait()
|
||||
```
|
||||
|
||||
### 前端调试流程
|
||||
|
||||
1. **设置断点**：点击节点设置断点
|
||||
2. **启动调试**：调用 `/debug/start` 启动调试执行
|
||||
3. **轮询状态**：定期调用 `/debug/:id/state` 获取状态
|
||||
4. **控制执行**：调用 pause/resume/step/stop 控制执行
|
||||
5. **查看日志**：实时显示执行日志和节点状态
|
||||
|
||||
```typescript
|
||||
// 调试状态轮询
|
||||
const fetchDebugState = async () => {
|
||||
const state = await backendClient.get(
|
||||
`/api/v1/workflows/${workflowUuid}/debug/${executionId}/state`
|
||||
);
|
||||
|
||||
// 更新节点状态
|
||||
setNodeStates(state.node_states);
|
||||
|
||||
// 追加新日志
|
||||
if (state.new_logs.length > 0) {
|
||||
setLogs(prev => [...prev, ...state.new_logs]);
|
||||
}
|
||||
|
||||
// 检查完成状态
|
||||
if (state.status === 'completed' || state.status === 'error') {
|
||||
setDebugState('idle');
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 扩展阅读
|
||||
|
||||
- [Workflow 功能设计文档](../../../plans/langbot-workflow-design.md)
|
||||
- [用户使用指南](../user-guide/workflow-guide.md)
|
||||
- [API 认证文档](../API_KEY_AUTH.md)
|
||||
@@ -1,198 +0,0 @@
|
||||
{
|
||||
"openapi": "3.0.3",
|
||||
"info": {
|
||||
"title": "LangBot HTTP Bot Adapter",
|
||||
"version": "1.0.0",
|
||||
"description": "Server-to-server HTTP integration for a LangBot pipeline. Inbound messages are POSTed to the unified webhook route; replies are delivered to a configured callback URL (one POST per reply part). All requests are HMAC-SHA256 signed. See docs/platforms/http-bot.md."
|
||||
},
|
||||
"paths": {
|
||||
"/bots/{bot_uuid}": {
|
||||
"post": {
|
||||
"summary": "Push a message into the pipeline (fire-and-collect)",
|
||||
"description": "Returns 202 immediately. Replies arrive asynchronously on the configured callback URL. Reuse the same session_id within the aggregation window to merge multiple messages into one turn (N->1).",
|
||||
"parameters": [
|
||||
{ "$ref": "#/components/parameters/BotUuid" },
|
||||
{ "$ref": "#/components/parameters/Timestamp" },
|
||||
{ "$ref": "#/components/parameters/Signature" },
|
||||
{ "$ref": "#/components/parameters/Idempotency" }
|
||||
],
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": { "application/json": { "schema": { "$ref": "#/components/schemas/InboundMessage" } } }
|
||||
},
|
||||
"responses": {
|
||||
"202": {
|
||||
"description": "Accepted (queued for the pipeline)",
|
||||
"content": { "application/json": { "schema": { "$ref": "#/components/schemas/AcceptedResponse" } } }
|
||||
},
|
||||
"400": { "$ref": "#/components/responses/Error" },
|
||||
"401": { "$ref": "#/components/responses/Error" },
|
||||
"409": { "$ref": "#/components/responses/Error" },
|
||||
"413": { "$ref": "#/components/responses/Error" }
|
||||
}
|
||||
}
|
||||
},
|
||||
"/bots/{bot_uuid}/sync": {
|
||||
"post": {
|
||||
"summary": "Push a message and wait for the collapsed reply",
|
||||
"description": "Blocking convenience mode. Waits for is_final and returns all reply parts collapsed into one array. Lossy (no sequence/streaming). One in-flight sync per session_id.",
|
||||
"parameters": [
|
||||
{ "$ref": "#/components/parameters/BotUuid" },
|
||||
{ "$ref": "#/components/parameters/Timestamp" },
|
||||
{ "$ref": "#/components/parameters/Signature" }
|
||||
],
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": { "application/json": { "schema": { "$ref": "#/components/schemas/InboundMessage" } } }
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "The collapsed reply",
|
||||
"content": { "application/json": { "schema": { "$ref": "#/components/schemas/SyncResponse" } } }
|
||||
},
|
||||
"400": { "$ref": "#/components/responses/Error" },
|
||||
"401": { "$ref": "#/components/responses/Error" }
|
||||
}
|
||||
}
|
||||
},
|
||||
"/bots/{bot_uuid}/reset": {
|
||||
"post": {
|
||||
"summary": "Reset a session's conversation",
|
||||
"parameters": [
|
||||
{ "$ref": "#/components/parameters/BotUuid" },
|
||||
{ "$ref": "#/components/parameters/Timestamp" },
|
||||
{ "$ref": "#/components/parameters/Signature" }
|
||||
],
|
||||
"requestBody": {
|
||||
"required": true,
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"required": ["session_id"],
|
||||
"properties": {
|
||||
"session_id": { "type": "string" },
|
||||
"session_type": { "type": "string", "enum": ["person", "group"] }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": { "description": "Reset done" },
|
||||
"400": { "$ref": "#/components/responses/Error" },
|
||||
"401": { "$ref": "#/components/responses/Error" }
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"components": {
|
||||
"parameters": {
|
||||
"BotUuid": {
|
||||
"name": "bot_uuid", "in": "path", "required": true,
|
||||
"schema": { "type": "string", "format": "uuid" }
|
||||
},
|
||||
"Timestamp": {
|
||||
"name": "X-LB-Timestamp", "in": "header", "required": true,
|
||||
"description": "Unix seconds; rejected if more than +/-300s from server time.",
|
||||
"schema": { "type": "string" }
|
||||
},
|
||||
"Signature": {
|
||||
"name": "X-LB-Signature", "in": "header", "required": true,
|
||||
"description": "sha256=<hex> of HMAC-SHA256(secret, \"{timestamp}.\" + raw_body).",
|
||||
"schema": { "type": "string" }
|
||||
},
|
||||
"Idempotency": {
|
||||
"name": "X-LB-Idempotency-Key", "in": "header", "required": false,
|
||||
"description": "Dedup key; a repeat within the dedup window returns 409.",
|
||||
"schema": { "type": "string" }
|
||||
}
|
||||
},
|
||||
"schemas": {
|
||||
"Segment": {
|
||||
"type": "object",
|
||||
"required": ["type"],
|
||||
"properties": {
|
||||
"type": { "type": "string", "enum": ["Plain", "Image", "Voice", "File", "At", "Quote"] },
|
||||
"text": { "type": "string", "description": "For type=Plain." },
|
||||
"url": { "type": "string", "description": "For media types." },
|
||||
"base64": { "type": "string", "description": "For media types (data URI or raw base64)." }
|
||||
}
|
||||
},
|
||||
"InboundMessage": {
|
||||
"type": "object",
|
||||
"required": ["session_id", "message"],
|
||||
"properties": {
|
||||
"session_id": { "type": "string", "description": "Caller-defined; maps 1:1 to a LangBot session." },
|
||||
"session_type": { "type": "string", "enum": ["person", "group"], "default": "person" },
|
||||
"sender": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": { "type": "string" },
|
||||
"name": { "type": "string" },
|
||||
"group_name": { "type": "string", "description": "For session_type=group." }
|
||||
}
|
||||
},
|
||||
"message": { "type": "array", "items": { "$ref": "#/components/schemas/Segment" } }
|
||||
}
|
||||
},
|
||||
"AcceptedResponse": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"code": { "type": "integer", "example": 0 },
|
||||
"msg": { "type": "string", "example": "accepted" },
|
||||
"data": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"session_id": { "type": "string" },
|
||||
"accepted_message_id": { "type": "string", "example": "in_01H..." },
|
||||
"aggregating": { "type": "boolean" }
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"SyncResponse": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"code": { "type": "integer", "example": 0 },
|
||||
"msg": { "type": "string", "example": "ok" },
|
||||
"data": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"session_id": { "type": "string" },
|
||||
"reply_to": { "type": "string" },
|
||||
"message": { "type": "array", "items": { "$ref": "#/components/schemas/Segment" } }
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"Callback": {
|
||||
"type": "object",
|
||||
"description": "Delivered by LangBot to your callback_url, one POST per reply part. Signed with the outbound secret.",
|
||||
"properties": {
|
||||
"session_id": { "type": "string" },
|
||||
"reply_to": { "type": "string", "description": "The accepted_message_id this answers." },
|
||||
"sequence": { "type": "integer", "description": "1-based ordinal within the turn." },
|
||||
"is_final": { "type": "boolean", "description": "True on the last part of the turn." },
|
||||
"stream": { "type": "boolean" },
|
||||
"message": { "type": "array", "items": { "$ref": "#/components/schemas/Segment" } },
|
||||
"timestamp": { "type": "string", "format": "date-time" }
|
||||
}
|
||||
},
|
||||
"ErrorEnvelope": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"code": { "type": "integer", "example": 40101 },
|
||||
"msg": { "type": "string", "example": "invalid signature: signature_mismatch" },
|
||||
"data": { "nullable": true }
|
||||
}
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"Error": {
|
||||
"description": "Error envelope",
|
||||
"content": { "application/json": { "schema": { "$ref": "#/components/schemas/ErrorEnvelope" } } }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,256 +0,0 @@
|
||||
# HTTP Bot Adapter — Integration Guide
|
||||
|
||||
Integrate **any backend system** with a LangBot pipeline over plain HTTP. Push
|
||||
messages in via a signed webhook; receive replies on a callback URL. No
|
||||
long-lived connection, full support for message **aggregation** (many inbound
|
||||
messages merged into one turn) and **multi-part replies** (one turn → many
|
||||
outbound messages).
|
||||
|
||||
This is the right adapter for **server-to-server** integrations — ticketing
|
||||
systems, CRMs, internal tools, custom web backends. (For an in-browser,
|
||||
real-time chat widget, use the embeddable Web Page Bot instead.)
|
||||
|
||||
> **5-minute goal:** stand up a callback receiver, send a message, and watch a
|
||||
> multi-part reply arrive — using the reference client in
|
||||
> [`examples/http-bot/`](../../examples/http-bot/).
|
||||
|
||||
---
|
||||
|
||||
## 1. Mental model
|
||||
|
||||
```
|
||||
Your backend ──(1) POST signed message──► LangBot /bots/<bot_uuid>
|
||||
(pipeline runs: aggregate → think → reply)
|
||||
Your callback ◄─(2) POST signed reply(s)── LangBot one POST per reply part
|
||||
```
|
||||
|
||||
- **(1) Inbound** is *fire-and-collect*: LangBot answers `202 Accepted`
|
||||
immediately and does **not** return the pipeline result on that response.
|
||||
- **(2) Outbound** replies arrive later as separate signed POSTs to your
|
||||
`callback_url`. A single turn may produce **several** callbacks (e.g. a tool
|
||||
call narration followed by the final answer).
|
||||
- Everything is keyed by a **`session_id` you choose** (e.g. a ticket number).
|
||||
Each `session_id` maps to one isolated LangBot conversation.
|
||||
|
||||
---
|
||||
|
||||
## 2. Create the bot
|
||||
|
||||
1. In the LangBot dashboard, add a bot and choose the **HTTP Bot** platform.
|
||||
2. Fill in the config:
|
||||
|
||||
| Field | Required | Notes |
|
||||
|---|---|---|
|
||||
| **Inbound Signing Secret** | yes | Your backend signs inbound requests with this. |
|
||||
| **Outbound Callback URL** | yes | Where LangBot POSTs replies. **Config-only** — cannot be overridden per message (SSRF protection). |
|
||||
| **Outbound Signing Secret** | no | LangBot signs callbacks with this; defaults to the inbound secret. |
|
||||
| **Default Session Type** | no | `person` (default) or `group`. |
|
||||
| **Require Inbound Signature** | no | Keep `true` in production. |
|
||||
| **Callback Timeout / Max Retries** | no | Defaults: 15s, 3 retries. |
|
||||
|
||||
3. Bind the bot to a **pipeline** and **enable** it.
|
||||
4. Copy the **Inbound Webhook URL** shown in the config — it looks like
|
||||
`https://your-langbot/bots/<bot_uuid>`.
|
||||
|
||||
---
|
||||
|
||||
## 3. The signature scheme
|
||||
|
||||
Both directions use the same dependency-free HMAC-SHA256 scheme:
|
||||
|
||||
```
|
||||
signing_string = "{timestamp}." + raw_body_bytes
|
||||
signature = "sha256=" + hex(HMAC_SHA256(secret, signing_string))
|
||||
```
|
||||
|
||||
Sent as headers:
|
||||
|
||||
| Header | Meaning |
|
||||
|---|---|
|
||||
| `X-LB-Timestamp` | Unix seconds. Rejected if more than **±300s** from server time. |
|
||||
| `X-LB-Signature` | `sha256=<hex>` over `"{timestamp}." + body`. |
|
||||
| `X-LB-Idempotency-Key` | *(optional, inbound)* dedup key; retries with the same key return `409`. |
|
||||
|
||||
Verify outbound callbacks the same way, using the **outbound** secret (or the
|
||||
inbound secret if you left it blank).
|
||||
|
||||
A six-line reference implementation is in `examples/http-bot/client.py`
|
||||
(`sign()` / `verify()`); a Node/TS version is in `client.ts`.
|
||||
|
||||
---
|
||||
|
||||
## 4. Send your first message (curl)
|
||||
|
||||
```bash
|
||||
BOT="https://your-langbot/bots/<bot_uuid>"
|
||||
SECRET="your-inbound-secret"
|
||||
BODY='{"session_id":"ticket-10293","message":[{"type":"Plain","text":"Export keeps failing on the dashboard."}]}'
|
||||
TS=$(date +%s)
|
||||
SIG="sha256=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -r | cut -d' ' -f1)"
|
||||
|
||||
curl -sS -X POST "$BOT" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "X-LB-Timestamp: $TS" \
|
||||
-H "X-LB-Signature: $SIG" \
|
||||
-d "$BODY"
|
||||
# -> 202 {"code":0,"msg":"accepted","data":{"session_id":"ticket-10293","accepted_message_id":"in_...","aggregating":true}}
|
||||
```
|
||||
|
||||
The reply(s) will be POSTed to your configured callback URL shortly after.
|
||||
|
||||
---
|
||||
|
||||
## 5. Inbound request format
|
||||
|
||||
`POST /bots/{bot_uuid}`
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"session_id": "ticket-10293", // REQUIRED. Your stable id. Maps 1:1 to a LangBot session.
|
||||
"session_type": "person", // optional: "person" | "group"; default from config
|
||||
"sender": { // optional metadata, surfaced to the pipeline/plugins
|
||||
"id": "user-5567",
|
||||
"name": "Alice"
|
||||
},
|
||||
"message": [ // REQUIRED. A LangBot MessageChain (array of segments).
|
||||
{ "type": "Plain", "text": "Export keeps failing on the dashboard." },
|
||||
{ "type": "Image", "url": "https://example.com/screenshot.png" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Message segments.** Text uses `{"type":"Plain","text":"..."}`. Images use
|
||||
`{"type":"Image","url":"..."}` (or `base64`). Other supported types: `Voice`,
|
||||
`File`, `At`, `Quote`.
|
||||
|
||||
> Note: the callback URL is **not** accepted in the body — it is taken only from
|
||||
> bot config. This is deliberate (prevents an attacker who obtains the inbound
|
||||
> secret from redirecting replies to an arbitrary host).
|
||||
|
||||
### Aggregation (N → 1)
|
||||
|
||||
If your pipeline has **message aggregation** enabled, send several messages with
|
||||
the **same `session_id`** within the aggregation window and they are merged into
|
||||
**one** pipeline turn. No special flag — just reuse the `session_id`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Outbound callback format
|
||||
|
||||
LangBot POSTs each reply part to your `callback_url`:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"session_id": "ticket-10293", // echoes the inbound session
|
||||
"reply_to": "in_01H...", // the accepted_message_id this answers
|
||||
"sequence": 1, // 1-based ordinal within this turn
|
||||
"is_final": false, // true on the last part of the turn
|
||||
"stream": false, // true for streamed chunks
|
||||
"message": [ { "type": "Plain", "text": "Looking into it…" } ],
|
||||
"timestamp": "2026-06-22T09:00:01Z"
|
||||
}
|
||||
```
|
||||
|
||||
Your endpoint should return `2xx` quickly. Non-2xx / timeout → LangBot retries
|
||||
with exponential backoff (up to `callback_max_retries`).
|
||||
|
||||
### Multi-part replies (1 → M)
|
||||
|
||||
One turn may emit multiple callbacks, delivered **in `sequence` order** for a
|
||||
given session:
|
||||
|
||||
```
|
||||
seq=1 is_final=false "Checking your export logs…"
|
||||
seq=2 is_final=false "Found 2 failed exports."
|
||||
seq=3 is_final=true "Fixed — please try again."
|
||||
```
|
||||
|
||||
Stitch by `session_id` + `sequence`; the turn is complete when
|
||||
`is_final: true` arrives.
|
||||
|
||||
---
|
||||
|
||||
## 7. Reset a session
|
||||
|
||||
Start a fresh conversation for a `session_id` (drops history):
|
||||
|
||||
```
|
||||
POST /bots/{bot_uuid}/reset
|
||||
{ "session_id": "ticket-10293", "session_type": "person" }
|
||||
→ 200 { "code":0, "msg":"reset", "data": { "session_id":"ticket-10293", "removed": true } }
|
||||
```
|
||||
|
||||
Signed exactly like an inbound message.
|
||||
|
||||
---
|
||||
|
||||
## 8. Synchronous convenience mode
|
||||
|
||||
If you don't need streaming/multi-part and just want one reply back on the same
|
||||
HTTP call, POST to `/sync`. LangBot waits for the turn to finish and returns all
|
||||
parts **collapsed** into one array:
|
||||
|
||||
```
|
||||
POST /bots/{bot_uuid}/sync
|
||||
{ "session_id": "ticket-10293", "message": [ { "type":"Plain", "text":"hi" } ] }
|
||||
→ 200 { "code":0, "msg":"ok",
|
||||
"data": { "session_id":"ticket-10293", "reply_to":"in_...",
|
||||
"message": [ {"type":"Plain","text":"..."}, ... ] } }
|
||||
```
|
||||
|
||||
This is **lossy** (you lose `sequence` / streaming boundaries) and blocks up to
|
||||
`callback_timeout × 4` seconds. Prefer the callback model for anything
|
||||
real-time or multi-part. Only one in-flight `/sync` per `session_id`.
|
||||
|
||||
---
|
||||
|
||||
## 9. Error envelope
|
||||
|
||||
```jsonc
|
||||
{ "code": 40101, "msg": "invalid signature: signature_mismatch", "data": null }
|
||||
```
|
||||
|
||||
| HTTP | code | meaning |
|
||||
|---|---|---|
|
||||
| 202 | 0 | accepted |
|
||||
| 400 | 40001 | malformed body / missing `session_id` or `message` |
|
||||
| 401 | 40101 | bad/expired signature |
|
||||
| 409 | 40901 | duplicate idempotency key |
|
||||
| 413 | 41301 | message too large (>1 MiB) |
|
||||
| 500 | 50001 | internal error |
|
||||
|
||||
---
|
||||
|
||||
## 10. Try it end-to-end in 5 minutes
|
||||
|
||||
```bash
|
||||
cd examples/http-bot
|
||||
pip install flask requests
|
||||
|
||||
# Terminal 1 — your callback receiver (point the bot's callback_url here, e.g. via a tunnel):
|
||||
python client.py serve --port 8900 --secret SHARED_SECRET
|
||||
|
||||
# Terminal 2 — push a message:
|
||||
python client.py push \
|
||||
--url https://your-langbot/bots/<bot_uuid> \
|
||||
--secret SHARED_SECRET \
|
||||
--session ticket-1 \
|
||||
--text "hello"
|
||||
```
|
||||
|
||||
Watch Terminal 1 print each reply part (`[part ]` / `[FINAL]`) with its
|
||||
sequence number — that's 1→M working, signatures verified.
|
||||
|
||||
A machine-readable contract is in
|
||||
[`docs/http-bot-openapi.json`](../http-bot-openapi.json).
|
||||
|
||||
---
|
||||
|
||||
## 11. Security checklist
|
||||
|
||||
- Keep **Require Inbound Signature** on in production.
|
||||
- Use **HTTPS** callback URLs; the URL is config-only (no per-message override).
|
||||
- Treat the secrets like passwords; rotate via the dashboard.
|
||||
- The inbound route is unauthenticated at the framework level **by design** —
|
||||
security comes entirely from the HMAC signature, so never disable it on a
|
||||
public deployment.
|
||||
@@ -1,196 +0,0 @@
|
||||
# MCP Resources PR #2215 Review
|
||||
|
||||
> 更新日期: 2026-06-29
|
||||
> 分支: `mcp_resources`
|
||||
> PR: langbot-app/LangBot#2215
|
||||
> 主题: MCP Resources 在 LangBot 中的产品价值、AgentRunner 集成方式与后续架构方向
|
||||
|
||||
## 结论
|
||||
|
||||
PR #2215 对 LangBot 有明确价值：它补齐了 MCP 协议中 Resources 这一重要能力，让 MCP server 不再只暴露 tools，也可以暴露文档、代码片段、配置、日志、图片等上下文资源。管理端可以发现和预览资源，Agent 也可以通过当前实现按需列出和读取资源。
|
||||
|
||||
但当前 AgentRunner 层的接入方式更接近一个可用的第一阶段方案，而不是最终架构。现在 MCP Resources 被包装成两个 synthetic tools：
|
||||
|
||||
- `langbot_mcp_list_resources`
|
||||
- `langbot_mcp_read_resource`
|
||||
|
||||
这让模型可以通过 function calling 主动探索资源，落地成本低，也复用了已有 `ToolManager` / `LocalAgentRunner` 的工具调用链路。不过从 MCP 规范和主流实现来看，Resources 更适合作为一种一等上下文来源，而不是长期隐藏在工具列表里。
|
||||
|
||||
建议保留当前 synthetic tools 作为探索能力，同时把后续主线设计调整为：MCP Resources 是 pipeline / conversation / message 级别可选择、可固定、可审计的上下文输入。
|
||||
|
||||
## 当前实现判断
|
||||
|
||||
当前 AgentRunner 集成路径如下：
|
||||
|
||||
```text
|
||||
Pipeline 绑定 MCP server
|
||||
-> query.variables['_pipeline_bound_mcp_servers']
|
||||
-> Preproc 为 local-agent 加载工具
|
||||
-> ToolManager.get_all_tools()
|
||||
-> MCPLoader 注入 synthetic resource tools
|
||||
-> LocalAgentRunner 将工具 schema 传给模型
|
||||
-> 模型发起 list/read tool call
|
||||
-> ToolManager.execute_func_call()
|
||||
-> MCPLoader 调 MCP session.list_resources/read_resource
|
||||
-> tool result 回灌给模型
|
||||
```
|
||||
|
||||
这个路径的优点是：
|
||||
|
||||
- 复用现有工具调用机制，改动范围小。
|
||||
- Agent 可以按需探索资源，不需要每轮预先读取所有资源。
|
||||
- 可以沿用 pipeline 绑定的 MCP server 范围，避免越权读取未绑定 server。
|
||||
- 对已有 MCP tools 行为影响较小。
|
||||
|
||||
主要问题是：
|
||||
|
||||
- Resources 在语义上被降级成 tools，和 MCP 规范里的 resource primitive 不完全一致。
|
||||
- 模型必须先理解并主动调用 `list/read`，资源不会自然成为上下文。
|
||||
- pipeline 不能配置“默认携带某些资源”或“本轮附加某些资源”。
|
||||
- UI 资源 tab 目前是管理端预览能力，和 Agent 上下文选择没有打通。
|
||||
- 对 blob、图片、大文件、结构化资源的处理还比较粗糙。
|
||||
- 缺少 resource templates、订阅更新、缓存、chunk、token budget、trace 与审计策略。
|
||||
|
||||
## 主流项目做法
|
||||
|
||||
### MCP 官方规范
|
||||
|
||||
MCP Resources 是 server 暴露上下文数据的协议能力。规范没有要求 resources 必须以 tool call 形式给模型使用，而是把如何选择、过滤、读取和纳入上下文交给 Host application。
|
||||
|
||||
这意味着比较正统的集成方式是：LangBot 作为 Host，在 pipeline、会话或消息层决定哪些 resources 进入模型上下文。
|
||||
|
||||
参考: https://modelcontextprotocol.io/specification/2025-06-18/server/resources
|
||||
|
||||
### VS Code Copilot
|
||||
|
||||
VS Code 把 MCP Resources 做成 chat context 的一部分。用户可以通过 `Add Context > MCP Resources` 或命令浏览 MCP resources，并把选中的资源附加到一次 chat request。
|
||||
|
||||
这是目前最值得 LangBot 参考的产品形态：资源不是模型工具，而是用户和 Host 可控的上下文附件。
|
||||
|
||||
参考: https://code.visualstudio.com/docs/agent-customization/mcp-servers
|
||||
|
||||
### Anthropic SDK
|
||||
|
||||
Anthropic 的 client-side MCP helpers 提供资源读取和转换能力，例如把 MCP resource 转为 Claude message content 或 file。也就是说，应用先读取 resource，再显式放进模型消息。
|
||||
|
||||
这同样是 application-owned context injection，而不是把 resource 伪装成模型工具。
|
||||
|
||||
参考: https://platform.claude.com/docs/en/agents-and-tools/mcp-connector
|
||||
|
||||
### LangChain MCP Adapters
|
||||
|
||||
LangChain 把 MCP Resources 更像 data loader / document input 来处理，可以把资源加载成 `Blob`，再进入 LangChain 的文档、检索或上下文处理链路。
|
||||
|
||||
这说明 Resources 很适合作为知识源、文档源或上下文源，而不只是即时工具调用。
|
||||
|
||||
参考: https://docs.langchain.com/oss/python/langchain/mcp
|
||||
|
||||
### OpenAI Agents SDK
|
||||
|
||||
OpenAI Agents SDK 主路径仍偏向 MCP tools，但底层 MCP server API 已经有 `list_resources`、`list_resource_templates`、`read_resource` 等能力。当前形态说明 resources 是 client 能力，但并未默认变成 agent-visible tools。
|
||||
|
||||
参考: https://openai.github.io/openai-agents-python/mcp/
|
||||
|
||||
### Cline
|
||||
|
||||
Cline 会拉取 MCP tools、resources、resourceTemplates、prompts，并通过类似 `access_mcp_resource` 的内置访问方式让模型读取资源。这个方向和 LangBot 当前 synthetic tools 比较接近。
|
||||
|
||||
这种模式适合让 Agent 自主探索，但更像 Host 自定义的模型访问协议，不应成为唯一集成路径。
|
||||
|
||||
参考: https://github.com/cline/cline/blob/main/src/services/mcp/McpHub.ts
|
||||
|
||||
## 建议架构方向
|
||||
|
||||
### 1. 保留探索型工具
|
||||
|
||||
保留当前两个 synthetic tools：
|
||||
|
||||
- `langbot_mcp_list_resources`
|
||||
- `langbot_mcp_read_resource`
|
||||
|
||||
它们适合处理“用户没有显式选择资源，但 Agent 判断需要探索 MCP server 上下文”的场景。后续可以优化工具描述、返回格式、资源大小限制和错误信息。
|
||||
|
||||
### 2. 增加一等 Resource Context
|
||||
|
||||
新增一个 Host 层资源上下文概念，例如：
|
||||
|
||||
```text
|
||||
PipelineResourceBinding
|
||||
ConversationResourceAttachment
|
||||
MessageResourceAttachment
|
||||
```
|
||||
|
||||
Preproc 或独立的 `ResourceContextProvider` 在模型调用前读取这些资源，按 MIME 类型、大小、token budget 转为模型可消费的上下文。
|
||||
|
||||
### 3. 打通 UI 与 Agent 上下文
|
||||
|
||||
当前 MCP 详情页的 Resources tab 可以继续作为资源发现和预览入口。建议增加操作：
|
||||
|
||||
- 添加到本轮上下文
|
||||
- 固定到当前 pipeline
|
||||
- 固定到当前 bot / conversation
|
||||
- 查看资源读取历史和错误
|
||||
|
||||
这样 UI 资源管理能力才能真正影响 Agent 行为。
|
||||
|
||||
### 4. 支持 resource templates
|
||||
|
||||
MCP resource templates 允许 server 暴露参数化资源，例如：
|
||||
|
||||
```text
|
||||
repo://{owner}/{repo}/file/{path}
|
||||
log://{service}/{date}
|
||||
```
|
||||
|
||||
LangBot 后续应支持模板发现、参数填写、实例化和绑定。否则只能使用静态 resources，覆盖面会受限。
|
||||
|
||||
### 5. 增加资源处理策略
|
||||
|
||||
建议补齐：
|
||||
|
||||
- 文本资源 token budget 与截断策略。
|
||||
- 大文件 chunk 与摘要策略。
|
||||
- 图片/blob 的模型能力判断与 fallback。
|
||||
- MIME 类型白名单与安全限制。
|
||||
- 缓存与过期策略。
|
||||
- `resources/listChanged` 或订阅更新。
|
||||
- resource read trace，便于审计 Agent 读取了什么上下文。
|
||||
|
||||
## 推荐落地顺序
|
||||
|
||||
### Phase 1: 完成当前 PR 可用性
|
||||
|
||||
- 保留 synthetic tools。
|
||||
- 明确文档说明当前 Agent 集成是 tool-mediated。
|
||||
- 完善资源工具描述，降低模型误用概率。
|
||||
- 给 read/list 增加大小限制和更清晰的 MIME 处理。
|
||||
- 前端 Resources tab 与 Tools tab 分离，保持管理端清晰。
|
||||
|
||||
### Phase 2: 做 Host-owned context attachments
|
||||
|
||||
- 在 pipeline 或 conversation 层新增 resource attachment 配置。
|
||||
- Preproc 读取已绑定 resources，注入模型上下文。
|
||||
- UI 支持“添加到上下文 / 固定到 pipeline”。
|
||||
- 记录每轮实际注入的 resource URI 和 token 消耗。
|
||||
|
||||
### Phase 3: 做完整 MCP Resources 能力
|
||||
|
||||
- 支持 resource templates。
|
||||
- 支持资源订阅更新。
|
||||
- 支持 chunk、summary、RAG 化接入。
|
||||
- 为 DifyAgentRunner、LocalAgentRunner 等不同 runner 定义统一资源上下文接口。
|
||||
|
||||
## 最终建议
|
||||
|
||||
PR #2215 可以作为 MCP Resources 的第一阶段实现继续推进。它让 LangBot 快速拥有“资源发现、预览、按需读取”的闭环，也给 Agent 探索资源提供了可运行路径。
|
||||
|
||||
但在正式设计上，不建议把 “Resources == Tools” 固化为长期抽象。LangBot 更应该把 MCP Resources 定位为上下文来源，与 tools、prompts、knowledge base 并列：
|
||||
|
||||
```text
|
||||
Tools -> Agent 可以执行的动作
|
||||
Resources -> Host/用户/Agent 可以选择的上下文数据
|
||||
Prompts -> 可复用的任务模板
|
||||
Knowledge -> 可检索、可索引的长期知识
|
||||
```
|
||||
|
||||
这样既尊重 MCP 协议语义，也能让 LangBot 在 Agent 工作流、企业知识接入和多 MCP server 管理上走得更稳。
|
||||
@@ -1,425 +0,0 @@
|
||||
# Workflow 用户指南
|
||||
|
||||
本文档帮助您了解和使用 LangBot 的 Workflow（工作流）功能，通过可视化方式构建自动化的对话处理流程。
|
||||
|
||||
## 目录
|
||||
|
||||
- [功能介绍](#功能介绍)
|
||||
- [快速入门](#快速入门)
|
||||
- [节点类型说明](#节点类型说明)
|
||||
- [编辑器使用指南](#编辑器使用指南)
|
||||
- [调试功能](#调试功能)
|
||||
- [常见问题解答](#常见问题解答)
|
||||
|
||||
---
|
||||
|
||||
## 功能介绍
|
||||
|
||||
### 什么是 Workflow？
|
||||
|
||||
Workflow（工作流）是 LangBot 提供的可视化自动化编排系统。通过拖拽节点、连接边的方式，您可以：
|
||||
|
||||
- 📝 **构建复杂的对话流程**：使用条件分支、循环等控制节点
|
||||
- 🤖 **调用 AI 能力**：集成 LLM、知识库检索、参数提取
|
||||
- 🔗 **连接外部服务**：集成 Dify、n8n、Coze 等平台
|
||||
- ⚡ **自动化任务执行**：消息触发、定时触发、Webhook 触发
|
||||
|
||||
### Workflow vs Pipeline
|
||||
|
||||
| 对比项 | Pipeline | Workflow |
|
||||
|-------|----------|----------|
|
||||
| 配置方式 | 表单配置 | 可视化拖拽 |
|
||||
| 流程控制 | 线性执行 | 支持分支、循环、并行 |
|
||||
| 适用场景 | 简单对话 | 复杂流程 |
|
||||
| 学习曲线 | 低 | 中等 |
|
||||
|
||||
---
|
||||
|
||||
## 快速入门
|
||||
|
||||
### 第一步：创建 Workflow
|
||||
|
||||
1. 在侧边栏点击 **Workflow** 进入工作流列表
|
||||
2. 点击右上角 **创建工作流** 按钮
|
||||
3. 填写基本信息：
|
||||
- **名称**：给工作流起一个描述性的名字
|
||||
- **描述**：可选，说明工作流的用途
|
||||
- **图标**：选择一个 emoji 作为标识
|
||||
|
||||
### 第二步：添加节点
|
||||
|
||||
进入编辑器后，左侧是节点面板，中间是画布区域，右侧是属性面板。
|
||||
|
||||
1. **添加触发节点**：从左侧面板拖拽一个"消息触发"节点到画布
|
||||
2. **添加 AI 节点**：拖拽一个"LLM 调用"节点
|
||||
3. **添加回复节点**：拖拽一个"回复消息"节点
|
||||
|
||||
### 第三步：连接节点
|
||||
|
||||
1. 将鼠标悬停在触发节点的输出端口（右侧小圆点）
|
||||
2. 按住鼠标拖拽到 LLM 节点的输入端口（左侧小圆点）
|
||||
3. 同样方式连接 LLM 节点和回复节点
|
||||
|
||||
```
|
||||
[消息触发] ──▶ [LLM 调用] ──▶ [回复消息]
|
||||
```
|
||||
|
||||
### 第四步：配置节点
|
||||
|
||||
点击 LLM 调用节点，在右侧属性面板配置：
|
||||
|
||||
- **运行方式**：选择"本地 Agent"
|
||||
- **系统提示词**：描述 AI 的角色和行为
|
||||
- **模型**：选择要使用的 LLM 模型
|
||||
|
||||
点击回复消息节点配置：
|
||||
|
||||
- **消息内容**：设置为 `{{nodes.llm_call.outputs.response}}`（引用 LLM 输出）
|
||||
|
||||
### 第五步：保存并绑定
|
||||
|
||||
1. 点击工具栏的 **保存** 按钮
|
||||
2. 返回 Bot 配置页面
|
||||
3. 在 Bot 的绑定设置中选择 **Workflow**，然后选择刚创建的工作流
|
||||
|
||||
恭喜！您已经创建了第一个 Workflow。
|
||||
|
||||
---
|
||||
|
||||
## 节点类型说明
|
||||
|
||||
### 触发节点 (Trigger)
|
||||
|
||||
触发节点是工作流的入口，定义何时启动执行。
|
||||
|
||||
| 节点 | 说明 | 输出 |
|
||||
|-----|------|------|
|
||||
| 消息触发 | 收到消息时触发 | message, sender_id, platform |
|
||||
| 定时触发 | 按 Cron 表达式定时触发 | timestamp |
|
||||
| Webhook 触发 | 收到 HTTP 请求时触发 | request_body, headers |
|
||||
| 事件触发 | 系统事件触发 | event_type, event_data |
|
||||
|
||||
**消息触发配置示例**：
|
||||
|
||||
```yaml
|
||||
触发条件:
|
||||
- 关键词匹配: ["帮助", "help"]
|
||||
- 平台: ["wechat", "qq"]
|
||||
```
|
||||
|
||||
### AI 节点
|
||||
|
||||
AI 节点用于调用各种 AI 能力。
|
||||
|
||||
| 节点 | 说明 | 典型用途 |
|
||||
|-----|------|---------|
|
||||
| LLM 调用 | 调用大语言模型 | 生成回复、理解意图 |
|
||||
| 问题分类器 | 对用户问题分类 | 路由到不同处理分支 |
|
||||
| 参数提取器 | 从文本提取结构化数据 | 提取订单号、日期等 |
|
||||
| 知识库检索 | 查询知识库 | RAG 增强回复 |
|
||||
|
||||
**LLM 调用配置示例**：
|
||||
|
||||
```yaml
|
||||
运行方式: 本地 Agent
|
||||
模型: gpt-4
|
||||
系统提示词: |
|
||||
你是一个友好的客服助手。
|
||||
请根据用户的问题提供帮助。
|
||||
温度: 0.7
|
||||
最大 Token 数: 2000
|
||||
```
|
||||
|
||||
### 处理节点 (Process)
|
||||
|
||||
处理节点用于数据处理和外部调用。
|
||||
|
||||
| 节点 | 说明 | 典型用途 |
|
||||
|-----|------|---------|
|
||||
| 代码执行 | 执行 Python/JavaScript 代码 | 数据处理、格式转换 |
|
||||
| HTTP 请求 | 发送 HTTP 请求 | 调用外部 API |
|
||||
| 数据转换 | JSON/模板转换 | 数据格式化 |
|
||||
|
||||
**HTTP 请求配置示例**：
|
||||
|
||||
```yaml
|
||||
URL: https://api.example.com/data
|
||||
方法: POST
|
||||
请求头:
|
||||
Content-Type: application/json
|
||||
Authorization: Bearer {{variables.api_key}}
|
||||
请求体: |
|
||||
{"query": "{{message.content}}"}
|
||||
```
|
||||
|
||||
### 控制节点 (Control)
|
||||
|
||||
控制节点用于流程控制。
|
||||
|
||||
| 节点 | 说明 | 用途 |
|
||||
|-----|------|------|
|
||||
| 条件分支 | 二选一分支 | if-else 逻辑 |
|
||||
| 多路分支 | 多选一分支 | switch-case 逻辑 |
|
||||
| 循环 | 遍历数组 | 批量处理 |
|
||||
| 并行 | 同时执行多分支 | 并发处理 |
|
||||
| 等待 | 暂停执行 | 延时处理 |
|
||||
| 合并 | 合并多个分支 | 汇总结果 |
|
||||
|
||||
**条件分支配置示例**：
|
||||
|
||||
```yaml
|
||||
条件表达式: "{{nodes.classifier.outputs.category}}" == "complaint"
|
||||
真分支: 投诉处理
|
||||
假分支: 普通咨询
|
||||
```
|
||||
|
||||
### 动作节点 (Action)
|
||||
|
||||
动作节点执行具体操作。
|
||||
|
||||
| 节点 | 说明 | 用途 |
|
||||
|-----|------|------|
|
||||
| 发送消息 | 主动发送消息 | 通知、推送 |
|
||||
| 回复消息 | 回复当前消息 | 对话回复 |
|
||||
| 存储数据 | 保存数据到存储 | 持久化 |
|
||||
| 调用 Pipeline | 调用现有 Pipeline | 复用现有流程 |
|
||||
|
||||
**回复消息配置示例**：
|
||||
|
||||
```yaml
|
||||
消息内容: |
|
||||
感谢您的咨询！
|
||||
|
||||
{{nodes.llm_call.outputs.response}}
|
||||
|
||||
如有其他问题，随时联系我。
|
||||
```
|
||||
|
||||
### 集成节点 (Integration)
|
||||
|
||||
集成节点连接外部平台。
|
||||
|
||||
| 节点 | 说明 | 平台 |
|
||||
|-----|------|------|
|
||||
| Dify 工作流 | 调用 Dify 应用 | Dify |
|
||||
| Dify 知识库 | 查询 Dify 知识库 | Dify |
|
||||
| n8n 工作流 | 调用 n8n 流程 | n8n |
|
||||
| Langflow | 调用 Langflow 流程 | Langflow |
|
||||
| Coze Bot | 调用扣子 Bot | Coze |
|
||||
|
||||
**Dify 工作流配置示例**：
|
||||
|
||||
```yaml
|
||||
API 地址: https://api.dify.ai/v1
|
||||
API Key: sk-xxxxx
|
||||
应用类型: workflow
|
||||
同步对话历史: true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 编辑器使用指南
|
||||
|
||||
### 画布操作
|
||||
|
||||
| 操作 | 方式 |
|
||||
|-----|------|
|
||||
| 平移画布 | 按住鼠标中键/空格+左键 拖拽 |
|
||||
| 缩放画布 | 鼠标滚轮 / 工具栏按钮 |
|
||||
| 框选多个节点 | 按住 Shift + 拖拽框选 |
|
||||
| 适应视图 | 点击工具栏"适应"按钮 |
|
||||
|
||||
### 节点操作
|
||||
|
||||
| 操作 | 方式 |
|
||||
|-----|------|
|
||||
| 添加节点 | 从左侧面板拖拽到画布 |
|
||||
| 移动节点 | 点击节点拖拽 |
|
||||
| 删除节点 | 选中后按 Delete / 点击工具栏删除 |
|
||||
| 复制节点 | 选中后 Ctrl+C / 工具栏复制 |
|
||||
| 粘贴节点 | Ctrl+V / 工具栏粘贴 |
|
||||
|
||||
### 连接操作
|
||||
|
||||
| 操作 | 方式 |
|
||||
|-----|------|
|
||||
| 创建连接 | 从输出端口拖拽到输入端口 |
|
||||
| 删除连接 | 点击连接线后按 Delete |
|
||||
| 选中连接 | 点击连接线 |
|
||||
|
||||
### 快捷键
|
||||
|
||||
| 快捷键 | 功能 |
|
||||
|-------|------|
|
||||
| Ctrl + Z | 撤销 |
|
||||
| Ctrl + Shift + Z | 重做 |
|
||||
| Ctrl + C | 复制 |
|
||||
| Ctrl + V | 粘贴 |
|
||||
| Delete | 删除选中 |
|
||||
| Ctrl + S | 保存 |
|
||||
|
||||
### 工具栏功能
|
||||
|
||||
```
|
||||
[撤销] [重做] | [放大] [缩小] [适应] | [复制] [粘贴] [删除] | [保存] [调试]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 调试功能
|
||||
|
||||
### 启动调试
|
||||
|
||||
1. 点击工具栏的 **调试** 按钮
|
||||
2. 在调试面板中配置初始数据：
|
||||
- **输入消息**：模拟用户发送的消息
|
||||
- **会话 ID**：可选，用于测试会话变量
|
||||
- **变量**：设置初始变量值
|
||||
|
||||
3. 点击 **开始调试** 按钮
|
||||
|
||||
### 调试控制
|
||||
|
||||
| 按钮 | 功能 |
|
||||
|-----|------|
|
||||
| ▶️ 开始/继续 | 开始或继续执行 |
|
||||
| ⏸️ 暂停 | 暂停执行 |
|
||||
| ⏹️ 停止 | 停止执行 |
|
||||
| ⏭️ 单步 | 执行下一个节点 |
|
||||
|
||||
### 断点
|
||||
|
||||
- **设置断点**：点击节点上的断点图标
|
||||
- **断点触发**：执行到断点时自动暂停
|
||||
- **查看状态**：在暂停时查看节点的输入输出
|
||||
|
||||
### 执行日志
|
||||
|
||||
调试面板下方显示实时日志：
|
||||
|
||||
```
|
||||
[INFO] 2024-01-15 10:30:00 - Starting debug execution
|
||||
[INFO] 2024-01-15 10:30:00 - Executing node: message_trigger
|
||||
[DEBUG] 2024-01-15 10:30:00 - Node inputs: {"message": "你好"}
|
||||
[INFO] 2024-01-15 10:30:01 - Node completed in 50ms
|
||||
[INFO] 2024-01-15 10:30:01 - Executing node: llm_call
|
||||
...
|
||||
```
|
||||
|
||||
### 节点状态颜色
|
||||
|
||||
| 颜色 | 状态 |
|
||||
|-----|------|
|
||||
| 灰色 | 待执行 |
|
||||
| 蓝色 | 执行中 |
|
||||
| 绿色 | 已完成 |
|
||||
| 红色 | 失败 |
|
||||
| 黄色 | 已跳过 |
|
||||
|
||||
---
|
||||
|
||||
## 常见问题解答
|
||||
|
||||
### Q1：如何在节点间传递数据？
|
||||
|
||||
使用表达式语法引用其他节点的输出：
|
||||
|
||||
```
|
||||
{{nodes.节点ID.outputs.输出名称}}
|
||||
```
|
||||
|
||||
例如：
|
||||
- `{{nodes.llm_call.outputs.response}}` - 引用 LLM 节点的响应
|
||||
- `{{nodes.http_request.outputs.body}}` - 引用 HTTP 请求的响应体
|
||||
|
||||
### Q2：如何使用变量？
|
||||
|
||||
Workflow 支持三种变量类型：
|
||||
|
||||
1. **工作流变量**：`{{variables.变量名}}`
|
||||
2. **会话变量**：`{{conversation_variables.变量名}}`
|
||||
3. **消息上下文**：`{{message.content}}`、`{{message.sender_id}}`
|
||||
|
||||
### Q3：条件分支如何写条件表达式？
|
||||
|
||||
支持以下运算符：
|
||||
|
||||
- 比较：`==`, `!=`, `>`, `<`, `>=`, `<=`
|
||||
- 逻辑：`and`, `or`, `not`
|
||||
- 包含：`in`
|
||||
|
||||
示例：
|
||||
```python
|
||||
# 字符串比较
|
||||
"{{nodes.classifier.outputs.intent}}" == "purchase"
|
||||
|
||||
# 数值比较
|
||||
{{nodes.extractor.outputs.amount}} > 1000
|
||||
|
||||
# 包含检查
|
||||
"退款" in "{{message.content}}"
|
||||
```
|
||||
|
||||
### Q4：如何处理错误？
|
||||
|
||||
1. **节点级重试**：在节点配置中设置重试次数
|
||||
2. **全局错误处理**：在 Workflow 设置中配置错误处理策略
|
||||
3. **条件分支**：使用条件节点检查上一节点的状态
|
||||
|
||||
### Q5：如何查看执行历史？
|
||||
|
||||
1. 进入 Workflow 详情页
|
||||
2. 点击 **执行历史** 标签
|
||||
3. 查看每次执行的状态、耗时、输入输出
|
||||
|
||||
### Q6：Workflow 可以被多个 Bot 使用吗？
|
||||
|
||||
是的。一个 Workflow 可以被多个 Bot 绑定使用，但每个 Bot 只能绑定一个处理单元（Pipeline 或 Workflow）。
|
||||
|
||||
### Q7：如何复制现有的 Workflow？
|
||||
|
||||
在 Workflow 列表页，点击工作流卡片右上角的菜单，选择"复制"即可创建副本。
|
||||
|
||||
### Q8：支持版本回滚吗？
|
||||
|
||||
支持。每次保存都会创建新版本。在 Workflow 详情页可以查看版本历史并回滚到指定版本。
|
||||
|
||||
---
|
||||
|
||||
## 最佳实践
|
||||
|
||||
### 1. 合理命名
|
||||
|
||||
- 为节点和 Workflow 使用描述性名称
|
||||
- 使用统一的命名规范
|
||||
|
||||
### 2. 模块化设计
|
||||
|
||||
- 将复杂流程拆分为多个小 Workflow
|
||||
- 使用"调用 Pipeline"节点复用现有流程
|
||||
|
||||
### 3. 错误处理
|
||||
|
||||
- 为关键节点设置重试机制
|
||||
- 使用条件分支处理异常情况
|
||||
- 添加日志记录便于排查问题
|
||||
|
||||
### 4. 测试先行
|
||||
|
||||
- 使用调试功能充分测试
|
||||
- 准备多种测试场景
|
||||
- 检查边界情况
|
||||
|
||||
### 5. 性能优化
|
||||
|
||||
- 避免不必要的节点
|
||||
- 使用并行节点提高效率
|
||||
- 合理设置超时时间
|
||||
|
||||
---
|
||||
|
||||
## 更多资源
|
||||
|
||||
- [开发者文档](../development/workflow-system.md)
|
||||
- [设计文档](../../../plans/langbot-workflow-design.md)
|
||||
- [API 文档](../service-api-openapi.json)
|
||||
@@ -1,75 +0,0 @@
|
||||
# HTTP Bot Adapter — Reference Clients
|
||||
|
||||
> English | [中文](./README.zh.md)
|
||||
|
||||
Minimal, dependency-light clients for the LangBot **HTTP Bot** platform adapter.
|
||||
They show the whole loop: signing a request, pushing a message, and receiving
|
||||
multi-part replies on a callback endpoint.
|
||||
|
||||
Full guide: [docs.langbot.app — HTTP Bot](https://docs.langbot.app/en/usage/platforms/http-bot).
|
||||
Machine-readable contract: [`docs/http-bot-openapi.json`](../../docs/http-bot-openapi.json).
|
||||
|
||||
## Files
|
||||
|
||||
| File | What it is |
|
||||
|---|---|
|
||||
| `playground.py` | **Interactive browser debug console** — a single-file web app you open in a browser to chat with a running `http_bot` bot and watch signing / 202 / callbacks live. Zero extra deps. |
|
||||
| `client.py` | Python client + Flask callback receiver (`pip install flask requests`). |
|
||||
| `client.ts` | TypeScript/Node 18+ client + callback receiver, **zero deps** (`npx tsx client.ts`). |
|
||||
|
||||
All three implement the identical HMAC-SHA256 scheme
|
||||
(`sha256=hex(HMAC(secret, "{timestamp}." + body))`) — verified byte-for-byte
|
||||
against the adapter.
|
||||
|
||||
## Interactive playground (recommended first run)
|
||||
|
||||
A self-contained web console: type a message in your browser, it is signed and
|
||||
POSTed to a **running** `http_bot` bot, and the bot's replies stream back into
|
||||
the page — with a debug panel showing the signature, the `202` ack, and each
|
||||
callback's `sequence` / signature-verification.
|
||||
|
||||
```bash
|
||||
# From the LangBot repo root, with the backend already running:
|
||||
PUBLIC_IP=<your-host-ip> ./.venv/bin/python examples/http-bot/playground.py
|
||||
# then open http://<your-host-ip>:8920/
|
||||
```
|
||||
|
||||
On startup it reads the LangBot API key + the `http_bot` bot from
|
||||
`data/langbot.db`, and configures that bot (inbound/outbound secret +
|
||||
`callback_url`) to point back at itself via the LangBot API — the bot reloads
|
||||
live, no restart needed. Requirements: an enabled `http_bot` bot bound to a
|
||||
working pipeline, and port `8920` reachable from your browser.
|
||||
|
||||
Env knobs: `PUBLIC_IP` (default `127.0.0.1`), `PLAYGROUND_PORT` (default `8920`).
|
||||
|
||||
## Headless clients
|
||||
|
||||
```bash
|
||||
# Python — Terminal 1: callback receiver (your callback_url target)
|
||||
python client.py serve --port 8900 --secret SHARED_SECRET
|
||||
|
||||
# Python — Terminal 2: push a message
|
||||
python client.py push --url https://your-langbot/bots/<BOT_UUID> \
|
||||
--secret SHARED_SECRET --session ticket-1 --text "hello"
|
||||
|
||||
# blocking sync mode
|
||||
python client.py sync --url https://your-langbot/bots/<BOT_UUID> \
|
||||
--secret SHARED_SECRET --session ticket-1 --text "hello"
|
||||
|
||||
# reset a session
|
||||
python client.py reset --url https://your-langbot/bots/<BOT_UUID> \
|
||||
--secret SHARED_SECRET --session ticket-1
|
||||
```
|
||||
|
||||
```bash
|
||||
# TypeScript (Node 18+)
|
||||
npx tsx client.ts serve 8900 SHARED_SECRET
|
||||
npx tsx client.ts push https://your-langbot/bots/<BOT_UUID> SHARED_SECRET ticket-1 "hello"
|
||||
```
|
||||
|
||||
When the bot replies, the receiver prints each part with its `sequence` and an
|
||||
`[FINAL]` marker on the last one — that's the 1→M multi-reply model in action.
|
||||
|
||||
> The bot's `callback_url` must be reachable from LangBot. For local testing,
|
||||
> expose your receiver with a tunnel (cloudflared / ngrok) and set that URL in
|
||||
> the bot config.
|
||||
@@ -1,71 +0,0 @@
|
||||
# HTTP Bot 适配器 —— 参考客户端
|
||||
|
||||
> [English](./README.md) | 中文
|
||||
|
||||
面向 LangBot **HTTP Bot** 平台适配器的极简、低依赖客户端示例。
|
||||
它们完整展示了整条链路:对请求签名、推送一条消息、在回调端点接收
|
||||
1→M 的多段回复。
|
||||
|
||||
完整指南:[docs.langbot.app —— HTTP Bot](https://docs.langbot.app/zh/usage/platforms/http-bot)。
|
||||
机器可读的接口契约:[`docs/http-bot-openapi.json`](../../docs/http-bot-openapi.json)。
|
||||
|
||||
## 文件清单
|
||||
|
||||
| 文件 | 是什么 |
|
||||
|---|---|
|
||||
| `playground.py` | **浏览器交互式调试台** —— 单文件 Web 应用,在浏览器里和一个运行中的 `http_bot` bot 对话,实时观察签名 / 202 / 回调。零额外依赖。 |
|
||||
| `client.py` | Python 客户端 + Flask 回调接收端(`pip install flask requests`)。 |
|
||||
| `client.ts` | TypeScript/Node 18+ 客户端 + 回调接收端,**零依赖**(`npx tsx client.ts`)。 |
|
||||
|
||||
三者实现完全一致的 HMAC-SHA256 签名方案
|
||||
(`sha256=hex(HMAC(secret, "{timestamp}." + body))`)—— 已与适配器逐字节比对验证。
|
||||
|
||||
## 交互式 playground(推荐先跑这个)
|
||||
|
||||
一个自包含的 Web 控制台:在浏览器里输入消息,它会被签名并 POST 给一个
|
||||
**运行中**的 `http_bot` bot,bot 的回复会流式回到页面上 —— 调试面板会显示
|
||||
签名、`202` 确认,以及每条回调的 `sequence` / 签名验证结果。
|
||||
|
||||
```bash
|
||||
# 在 LangBot 仓库根目录、后端已启动的前提下:
|
||||
PUBLIC_IP=<你的主机IP> ./.venv/bin/python examples/http-bot/playground.py
|
||||
# 然后打开 http://<你的主机IP>:8920/
|
||||
```
|
||||
|
||||
启动时它会从 `data/langbot.db` 读取 LangBot API key 和 `http_bot` bot,
|
||||
并通过 LangBot API 把该 bot 配好(入站/出站密钥 + `callback_url`)指回自己 ——
|
||||
bot 会热加载,无需重启。前提:有一个已启用、绑定了可用 pipeline 的
|
||||
`http_bot` bot,且端口 `8920` 能从你的浏览器访问到。
|
||||
|
||||
可调环境变量:`PUBLIC_IP`(默认 `127.0.0.1`)、`PLAYGROUND_PORT`(默认 `8920`)。
|
||||
|
||||
## 无头客户端
|
||||
|
||||
```bash
|
||||
# Python —— 终端 1:回调接收端(你的 callback_url 指向它)
|
||||
python client.py serve --port 8900 --secret SHARED_SECRET
|
||||
|
||||
# Python —— 终端 2:推送一条消息
|
||||
python client.py push --url https://your-langbot/bots/<BOT_UUID> \
|
||||
--secret SHARED_SECRET --session ticket-1 --text "hello"
|
||||
|
||||
# 阻塞式同步模式
|
||||
python client.py sync --url https://your-langbot/bots/<BOT_UUID> \
|
||||
--secret SHARED_SECRET --session ticket-1 --text "hello"
|
||||
|
||||
# 重置一个会话
|
||||
python client.py reset --url https://your-langbot/bots/<BOT_UUID> \
|
||||
--secret SHARED_SECRET --session ticket-1
|
||||
```
|
||||
|
||||
```bash
|
||||
# TypeScript(Node 18+)
|
||||
npx tsx client.ts serve 8900 SHARED_SECRET
|
||||
npx tsx client.ts push https://your-langbot/bots/<BOT_UUID> SHARED_SECRET ticket-1 "hello"
|
||||
```
|
||||
|
||||
当 bot 回复时,接收端会逐条打印,带上各自的 `sequence`,并在最后一条标记
|
||||
`[FINAL]` —— 这就是 1→M 多段回复模型的实际效果。
|
||||
|
||||
> bot 的 `callback_url` 必须能从 LangBot 访问到。本地测试时,可用隧道
|
||||
> (cloudflared / ngrok)把你的接收端暴露出去,并把那个 URL 填进 bot 配置。
|
||||
@@ -1,167 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""LangBot HTTP Bot adapter — reference client (Python).
|
||||
|
||||
Two things in one file:
|
||||
|
||||
1. ``push()`` / ``push_sync()`` — send a message into a LangBot ``http_bot`` bot.
|
||||
2. A tiny Flask callback receiver that verifies signatures and prints replies,
|
||||
so you can watch N->1 aggregation and 1->M multi-reply working live.
|
||||
|
||||
Usage
|
||||
-----
|
||||
pip install flask requests
|
||||
|
||||
# Terminal 1 — start the callback receiver (this is your callback_url):
|
||||
python client.py serve --port 8900 --secret SHARED_SECRET
|
||||
|
||||
# Terminal 2 — push a message (async; reply lands on the receiver):
|
||||
python client.py push \
|
||||
--url https://your-langbot/bots/<BOT_UUID> \
|
||||
--secret SHARED_SECRET \
|
||||
--session ticket-10293 \
|
||||
--text "Export keeps failing on the dashboard."
|
||||
|
||||
# Or push and block for the collapsed reply (sync convenience mode):
|
||||
python client.py sync --url https://your-langbot/bots/<BOT_UUID> \
|
||||
--secret SHARED_SECRET --session ticket-10293 --text "hi"
|
||||
|
||||
The signing scheme is HMAC-SHA256 over ``"{timestamp}." + raw_body``; see
|
||||
``sign()`` below — it is intentionally tiny and easy to port.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
|
||||
HEADER_TIMESTAMP = 'X-LB-Timestamp'
|
||||
HEADER_SIGNATURE = 'X-LB-Signature'
|
||||
HEADER_IDEMPOTENCY = 'X-LB-Idempotency-Key'
|
||||
REPLAY_WINDOW = 300
|
||||
|
||||
|
||||
def sign(secret: str, body: bytes, timestamp: int | None = None) -> tuple[str, str]:
|
||||
"""Return (timestamp, signature) for *body*."""
|
||||
ts = str(timestamp if timestamp is not None else int(time.time()))
|
||||
mac = hmac.new(secret.encode(), f'{ts}.'.encode() + body, hashlib.sha256)
|
||||
return ts, 'sha256=' + mac.hexdigest()
|
||||
|
||||
|
||||
def verify(secret: str, body: bytes, timestamp: str | None, signature: str | None) -> bool:
|
||||
"""Verify an inbound signature (used by the callback receiver)."""
|
||||
if not timestamp or not signature:
|
||||
return False
|
||||
try:
|
||||
if abs(int(time.time()) - int(float(timestamp))) > REPLAY_WINDOW:
|
||||
return False
|
||||
except ValueError:
|
||||
return False
|
||||
_, expected = sign(secret, body, int(float(timestamp)))
|
||||
return hmac.compare_digest(expected, signature)
|
||||
|
||||
|
||||
def _post(url: str, secret: str, payload: dict, idempotency: bool = True):
|
||||
import requests
|
||||
|
||||
body = json.dumps(payload, ensure_ascii=False).encode()
|
||||
ts, sig = sign(secret, body)
|
||||
headers = {
|
||||
'Content-Type': 'application/json',
|
||||
HEADER_TIMESTAMP: ts,
|
||||
HEADER_SIGNATURE: sig,
|
||||
}
|
||||
if idempotency:
|
||||
headers[HEADER_IDEMPOTENCY] = uuid.uuid4().hex
|
||||
resp = requests.post(url, data=body, headers=headers, timeout=30)
|
||||
print(f'-> {resp.status_code} {resp.text}')
|
||||
return resp
|
||||
|
||||
|
||||
def push(url: str, secret: str, session: str, text: str, session_type: str = 'person'):
|
||||
"""Fire-and-collect: returns 202 immediately; reply arrives on your callback."""
|
||||
payload = {
|
||||
'session_id': session,
|
||||
'session_type': session_type,
|
||||
'message': [{'type': 'Plain', 'text': text}],
|
||||
}
|
||||
return _post(url.rstrip('/'), secret, payload)
|
||||
|
||||
|
||||
def push_sync(url: str, secret: str, session: str, text: str, session_type: str = 'person'):
|
||||
"""Blocking convenience: POST to /sync and get the collapsed reply back."""
|
||||
payload = {
|
||||
'session_id': session,
|
||||
'session_type': session_type,
|
||||
'message': [{'type': 'Plain', 'text': text}],
|
||||
}
|
||||
resp = _post(url.rstrip('/') + '/sync', secret, payload, idempotency=False)
|
||||
return resp
|
||||
|
||||
|
||||
def reset(url: str, secret: str, session: str, session_type: str = 'person'):
|
||||
"""Reset a session's conversation (next message starts fresh)."""
|
||||
payload = {'session_id': session, 'session_type': session_type}
|
||||
return _post(url.rstrip('/') + '/reset', secret, payload, idempotency=False)
|
||||
|
||||
|
||||
def serve(port: int, secret: str):
|
||||
"""Run a callback receiver that verifies signatures and prints replies."""
|
||||
from flask import Flask, request
|
||||
|
||||
app = Flask(__name__)
|
||||
|
||||
@app.route('/', methods=['POST'])
|
||||
def recv():
|
||||
raw = request.get_data()
|
||||
ok = verify(secret, raw, request.headers.get(HEADER_TIMESTAMP), request.headers.get(HEADER_SIGNATURE))
|
||||
if not ok:
|
||||
print('!! signature verification FAILED — rejecting')
|
||||
return {'error': 'bad signature'}, 401
|
||||
data = json.loads(raw)
|
||||
text_parts = [c.get('text', '') for c in data.get('message', []) if c.get('type') == 'Plain']
|
||||
marker = 'FINAL' if data.get('is_final') else 'part '
|
||||
print(
|
||||
f'[{marker}] session={data["session_id"]} seq={data["sequence"]} '
|
||||
f'reply_to={data.get("reply_to")}: {" ".join(text_parts)}'
|
||||
)
|
||||
return {'ok': True}
|
||||
|
||||
print(f'callback receiver listening on http://0.0.0.0:{port}/ (Ctrl-C to stop)')
|
||||
app.run(host='0.0.0.0', port=port)
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
p = argparse.ArgumentParser(description='LangBot HTTP Bot reference client')
|
||||
sub = p.add_subparsers(dest='cmd', required=True)
|
||||
|
||||
sp = sub.add_parser('serve', help='run the callback receiver')
|
||||
sp.add_argument('--port', type=int, default=8900)
|
||||
sp.add_argument('--secret', required=True)
|
||||
|
||||
for name in ('push', 'sync', 'reset'):
|
||||
c = sub.add_parser(name)
|
||||
c.add_argument('--url', required=True, help='https://host/bots/<BOT_UUID>')
|
||||
c.add_argument('--secret', required=True)
|
||||
c.add_argument('--session', required=True)
|
||||
c.add_argument('--session-type', default='person', choices=['person', 'group'])
|
||||
if name != 'reset':
|
||||
c.add_argument('--text', required=True)
|
||||
|
||||
args = p.parse_args(argv)
|
||||
if args.cmd == 'serve':
|
||||
serve(args.port, args.secret)
|
||||
elif args.cmd == 'push':
|
||||
push(args.url, args.secret, args.session, args.text, args.session_type)
|
||||
elif args.cmd == 'sync':
|
||||
push_sync(args.url, args.secret, args.session, args.text, args.session_type)
|
||||
elif args.cmd == 'reset':
|
||||
reset(args.url, args.secret, args.session, args.session_type)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
@@ -1,123 +0,0 @@
|
||||
/**
|
||||
* LangBot HTTP Bot adapter — reference client (TypeScript / Node 18+).
|
||||
*
|
||||
* Zero runtime dependencies (uses global `fetch`, `crypto`, and `http`).
|
||||
*
|
||||
* - `push()` : fire-and-collect; reply lands on your callback URL.
|
||||
* - `pushSync()` : POST /sync and await the collapsed reply.
|
||||
* - `reset()` : reset a session's conversation.
|
||||
* - `startReceiver()` : a callback server that verifies signatures and logs
|
||||
* replies, so you can watch N->1 and 1->M live.
|
||||
*
|
||||
* Run the demos:
|
||||
* npx tsx client.ts serve 8900 SHARED_SECRET
|
||||
* npx tsx client.ts push https://host/bots/<UUID> SHARED_SECRET ticket-1 "hello"
|
||||
* npx tsx client.ts sync https://host/bots/<UUID> SHARED_SECRET ticket-1 "hello"
|
||||
* npx tsx client.ts reset https://host/bots/<UUID> SHARED_SECRET ticket-1
|
||||
*/
|
||||
|
||||
import { createHmac, randomUUID, timingSafeEqual } from 'node:crypto';
|
||||
import { createServer } from 'node:http';
|
||||
|
||||
const HEADER_TIMESTAMP = 'X-LB-Timestamp';
|
||||
const HEADER_SIGNATURE = 'X-LB-Signature';
|
||||
const HEADER_IDEMPOTENCY = 'X-LB-Idempotency-Key';
|
||||
const REPLAY_WINDOW = 300;
|
||||
|
||||
/** Compute the `sha256=<hex>` signature over `"{ts}." + body`. */
|
||||
export function sign(secret: string, body: Buffer | string, timestamp?: number): [string, string] {
|
||||
const ts = String(timestamp ?? Math.floor(Date.now() / 1000));
|
||||
const buf = typeof body === 'string' ? Buffer.from(body) : body;
|
||||
const mac = createHmac('sha256', secret).update(Buffer.concat([Buffer.from(`${ts}.`), buf])).digest('hex');
|
||||
return [ts, `sha256=${mac}`];
|
||||
}
|
||||
|
||||
/** Verify an inbound signature (used by the callback receiver). */
|
||||
export function verify(secret: string, body: Buffer, timestamp?: string, signature?: string): boolean {
|
||||
if (!timestamp || !signature) return false;
|
||||
if (Math.abs(Math.floor(Date.now() / 1000) - Number(timestamp)) > REPLAY_WINDOW) return false;
|
||||
const [, expected] = sign(secret, body, Number(timestamp));
|
||||
const a = Buffer.from(expected);
|
||||
const b = Buffer.from(signature);
|
||||
return a.length === b.length && timingSafeEqual(a, b);
|
||||
}
|
||||
|
||||
interface Segment { type: string; text?: string; url?: string; [k: string]: unknown }
|
||||
|
||||
async function post(url: string, secret: string, payload: object, idempotency = true) {
|
||||
const body = Buffer.from(JSON.stringify(payload));
|
||||
const [ts, sig] = sign(secret, body);
|
||||
const headers: Record<string, string> = {
|
||||
'Content-Type': 'application/json',
|
||||
[HEADER_TIMESTAMP]: ts,
|
||||
[HEADER_SIGNATURE]: sig,
|
||||
};
|
||||
if (idempotency) headers[HEADER_IDEMPOTENCY] = randomUUID();
|
||||
const resp = await fetch(url, { method: 'POST', headers, body });
|
||||
const text = await resp.text();
|
||||
console.log(`-> ${resp.status} ${text}`);
|
||||
return { status: resp.status, text };
|
||||
}
|
||||
|
||||
/** Fire-and-collect: 202 now, reply later on your callback URL. */
|
||||
export function push(url: string, secret: string, session: string, text: string, sessionType = 'person') {
|
||||
return post(url.replace(/\/$/, ''), secret, {
|
||||
session_id: session,
|
||||
session_type: sessionType,
|
||||
message: [{ type: 'Plain', text }] as Segment[],
|
||||
});
|
||||
}
|
||||
|
||||
/** Blocking convenience: POST /sync, get the collapsed reply. */
|
||||
export function pushSync(url: string, secret: string, session: string, text: string, sessionType = 'person') {
|
||||
return post(`${url.replace(/\/$/, '')}/sync`, secret, {
|
||||
session_id: session,
|
||||
session_type: sessionType,
|
||||
message: [{ type: 'Plain', text }] as Segment[],
|
||||
}, false);
|
||||
}
|
||||
|
||||
/** Reset a session's conversation. */
|
||||
export function reset(url: string, secret: string, session: string, sessionType = 'person') {
|
||||
return post(`${url.replace(/\/$/, '')}/reset`, secret, { session_id: session, session_type: sessionType }, false);
|
||||
}
|
||||
|
||||
/** Run a callback receiver that verifies signatures and prints replies. */
|
||||
export function startReceiver(port: number, secret: string) {
|
||||
const server = createServer((req, res) => {
|
||||
if (req.method !== 'POST') { res.writeHead(405).end(); return; }
|
||||
const chunks: Buffer[] = [];
|
||||
req.on('data', (c) => chunks.push(c));
|
||||
req.on('end', () => {
|
||||
const raw = Buffer.concat(chunks);
|
||||
const ok = verify(secret, raw, req.headers[HEADER_TIMESTAMP.toLowerCase()] as string,
|
||||
req.headers[HEADER_SIGNATURE.toLowerCase()] as string);
|
||||
if (!ok) {
|
||||
console.log('!! signature verification FAILED — rejecting');
|
||||
res.writeHead(401, { 'Content-Type': 'application/json' }).end(JSON.stringify({ error: 'bad signature' }));
|
||||
return;
|
||||
}
|
||||
const data = JSON.parse(raw.toString());
|
||||
const parts = (data.message as Segment[]).filter((c) => c.type === 'Plain').map((c) => c.text).join(' ');
|
||||
const marker = data.is_final ? 'FINAL' : 'part ';
|
||||
console.log(`[${marker}] session=${data.session_id} seq=${data.sequence} reply_to=${data.reply_to}: ${parts}`);
|
||||
res.writeHead(200, { 'Content-Type': 'application/json' }).end(JSON.stringify({ ok: true }));
|
||||
});
|
||||
});
|
||||
server.listen(port, () => console.log(`callback receiver listening on http://0.0.0.0:${port}/ (Ctrl-C to stop)`));
|
||||
}
|
||||
|
||||
// --- CLI ---
|
||||
const [cmd, ...rest] = process.argv.slice(2);
|
||||
if (cmd === 'serve') {
|
||||
startReceiver(Number(rest[0] ?? 8900), rest[1] ?? 'SHARED_SECRET');
|
||||
} else if (cmd === 'push') {
|
||||
push(rest[0], rest[1], rest[2], rest[3]);
|
||||
} else if (cmd === 'sync') {
|
||||
pushSync(rest[0], rest[1], rest[2], rest[3]);
|
||||
} else if (cmd === 'reset') {
|
||||
reset(rest[0], rest[1], rest[2]);
|
||||
} else if (cmd) {
|
||||
console.error(`unknown command: ${cmd}`);
|
||||
process.exit(1);
|
||||
}
|
||||
@@ -1,349 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""LangBot HTTP Bot — interactive playground (public, browser-based).
|
||||
|
||||
This is a REAL end-to-end demo against the RUNNING LangBot instance on this
|
||||
host. It is NOT a mock and NOT an in-process import: every message you type in
|
||||
the browser is signed and POSTed to the live `http_bot` bot at
|
||||
http://127.0.0.1:5300/bots/<uuid>, and the bot's replies come back to this
|
||||
server's /callback endpoint over real HTTP, then stream to your browser via SSE.
|
||||
|
||||
What it does on startup:
|
||||
1. Reads the LangBot API key + the http_bot bot from data/langbot.db.
|
||||
2. Configures the bot via the LangBot API (PUT /api/v1/platform/bots/<uuid>):
|
||||
sets inbound_secret + outbound_secret + callback_url to point back here.
|
||||
(LangBot reloads the bot live — no server restart needed.)
|
||||
3. Serves a chat page on 0.0.0.0:<PORT> so you can open it from the internet.
|
||||
|
||||
Run: ./.venv/bin/python examples/http-bot/playground.py
|
||||
Then open: http://<this-host-public-ip>:<PORT>/
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import sys
|
||||
|
||||
REPO = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))
|
||||
sys.path.insert(0, os.path.join(REPO, 'src'))
|
||||
|
||||
from aiohttp import web # noqa: E402
|
||||
import aiohttp # noqa: E402
|
||||
|
||||
from langbot.pkg.platform.sources import http_bot_signing as sg # noqa: E402
|
||||
|
||||
# ---- config -----------------------------------------------------------------
|
||||
LANGBOT_BASE = 'http://127.0.0.1:5300'
|
||||
DB_PATH = os.path.join(REPO, 'data', 'langbot.db')
|
||||
PUBLIC_IP = os.environ.get('PUBLIC_IP', '127.0.0.1')
|
||||
PORT = int(os.environ.get('PLAYGROUND_PORT', '8920'))
|
||||
SECRET = 'playground-shared-secret'
|
||||
|
||||
# SSE subscribers: list of asyncio.Queue
|
||||
subscribers: list[asyncio.Queue] = []
|
||||
|
||||
|
||||
def db_lookup() -> tuple[str, str]:
|
||||
"""Return (api_key, http_bot_uuid) from the LangBot DB."""
|
||||
db = sqlite3.connect(DB_PATH)
|
||||
db.row_factory = sqlite3.Row
|
||||
api_key = db.execute('SELECT key FROM api_keys LIMIT 1').fetchone()['key']
|
||||
bot = db.execute("SELECT uuid FROM bots WHERE adapter='http_bot' LIMIT 1").fetchone()
|
||||
if not bot:
|
||||
raise SystemExit('No http_bot bot found. Create one in the WebUI first.')
|
||||
return api_key, bot['uuid']
|
||||
|
||||
|
||||
async def configure_bot(api_key: str, bot_uuid: str, callback_url: str):
|
||||
"""Point the live bot at this playground via the LangBot API.
|
||||
|
||||
update_bot() runs a raw SQL UPDATE with whatever keys we send, so we send a
|
||||
MINIMAL payload: only adapter_config (built from scratch, not read back —
|
||||
the GET masks secrets). LangBot reloads + reruns the bot live.
|
||||
"""
|
||||
cfg = {
|
||||
'inbound_secret': SECRET,
|
||||
'outbound_secret': SECRET,
|
||||
'callback_url': callback_url,
|
||||
'signature_required': True,
|
||||
'default_session_type': 'person',
|
||||
'callback_timeout': 15,
|
||||
'callback_max_retries': 3,
|
||||
}
|
||||
async with aiohttp.ClientSession() as s:
|
||||
async with s.put(
|
||||
f'{LANGBOT_BASE}/api/v1/platform/bots/{bot_uuid}',
|
||||
headers={'Authorization': f'Bearer {api_key}', 'Content-Type': 'application/json'},
|
||||
json={'adapter_config': cfg},
|
||||
) as r:
|
||||
txt = await r.text()
|
||||
print(f'[configure] PUT adapter_config -> {r.status} {txt[:200]}')
|
||||
return r.status < 400
|
||||
|
||||
|
||||
async def broadcast(event: dict):
|
||||
for q in list(subscribers):
|
||||
try:
|
||||
q.put_nowait(event)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
# ---- HTTP handlers ----------------------------------------------------------
|
||||
async def index(request: web.Request):
|
||||
return web.Response(text=PAGE, content_type='text/html')
|
||||
|
||||
|
||||
async def send(request: web.Request):
|
||||
"""Browser -> here -> signed POST -> live LangBot bot."""
|
||||
body_in = await request.json()
|
||||
session_id = body_in.get('session_id') or 'playground-1'
|
||||
text = body_in.get('text', '')
|
||||
bot_uuid = request.app['bot_uuid']
|
||||
|
||||
payload = {
|
||||
'session_id': session_id,
|
||||
'sender': {'id': 'browser-user', 'name': 'You'},
|
||||
'message': [{'type': 'Plain', 'text': text}],
|
||||
}
|
||||
raw = json.dumps(payload, ensure_ascii=False).encode()
|
||||
ts, sig = sg.sign(SECRET, raw)
|
||||
url = f'{LANGBOT_BASE}/bots/{bot_uuid}'
|
||||
|
||||
# echo what we send to the browser timeline
|
||||
await broadcast(
|
||||
{'dir': 'out', 'kind': 'request', 'session_id': session_id, 'text': text, 'url': url, 'sig': sig[:24] + '…'}
|
||||
)
|
||||
|
||||
async with aiohttp.ClientSession() as s:
|
||||
async with s.post(
|
||||
url,
|
||||
data=raw,
|
||||
headers={
|
||||
'Content-Type': 'application/json',
|
||||
sg.HEADER_TIMESTAMP: ts,
|
||||
sg.HEADER_SIGNATURE: sig,
|
||||
},
|
||||
) as r:
|
||||
status = r.status
|
||||
try:
|
||||
jr = await r.json()
|
||||
except Exception:
|
||||
jr = {'raw': await r.text()}
|
||||
await broadcast({'dir': 'in', 'kind': 'ack', 'status': status, 'data': jr})
|
||||
return web.json_response({'status': status, 'data': jr})
|
||||
|
||||
|
||||
async def callback(request: web.Request):
|
||||
"""Live LangBot bot -> here. Verify signature, stream to browser."""
|
||||
raw = await request.read()
|
||||
ok, why = sg.verify(SECRET, raw, request.headers.get(sg.HEADER_TIMESTAMP), request.headers.get(sg.HEADER_SIGNATURE))
|
||||
data = json.loads(raw)
|
||||
text = ' '.join(c.get('text', '') for c in data.get('message', []) if c.get('type') == 'Plain')
|
||||
await broadcast(
|
||||
{
|
||||
'dir': 'in',
|
||||
'kind': 'reply',
|
||||
'session_id': data.get('session_id'),
|
||||
'sequence': data.get('sequence'),
|
||||
'is_final': data.get('is_final'),
|
||||
'sig_ok': ok,
|
||||
'sig_why': why,
|
||||
'text': text,
|
||||
}
|
||||
)
|
||||
return web.json_response({'ok': True})
|
||||
|
||||
|
||||
async def events(request: web.Request):
|
||||
"""SSE stream to the browser."""
|
||||
resp = web.StreamResponse(
|
||||
headers={
|
||||
'Content-Type': 'text/event-stream',
|
||||
'Cache-Control': 'no-cache',
|
||||
'Connection': 'keep-alive',
|
||||
'Access-Control-Allow-Origin': '*',
|
||||
}
|
||||
)
|
||||
await resp.prepare(request)
|
||||
q: asyncio.Queue = asyncio.Queue()
|
||||
subscribers.append(q)
|
||||
try:
|
||||
await resp.write(b': connected\n\n')
|
||||
while True:
|
||||
try:
|
||||
ev = await asyncio.wait_for(q.get(), timeout=15)
|
||||
await resp.write(f'data: {json.dumps(ev, ensure_ascii=False)}\n\n'.encode())
|
||||
except asyncio.TimeoutError:
|
||||
await resp.write(b': ping\n\n')
|
||||
except (asyncio.CancelledError, ConnectionResetError):
|
||||
pass
|
||||
finally:
|
||||
if q in subscribers:
|
||||
subscribers.remove(q)
|
||||
return resp
|
||||
|
||||
|
||||
PAGE = r"""<!doctype html>
|
||||
<html lang="zh"><head><meta charset="utf-8"/>
|
||||
<meta name="viewport" content="width=device-width,initial-scale=1"/>
|
||||
<title>LangBot HTTP Bot · 调试台</title>
|
||||
<style>
|
||||
:root{
|
||||
--bg:#f7f8fa; --panel:#ffffff; --line:#e8eaed; --ink:#1f2329; --mut:#8a909a;
|
||||
--brand:#2563eb; --brand-soft:#eef3ff; --ok:#16a34a; --bad:#dc2626; --code:#f3f4f6;
|
||||
}
|
||||
*{box-sizing:border-box}
|
||||
html,body{height:100%}
|
||||
body{margin:0;background:var(--bg);color:var(--ink);
|
||||
font:14px/1.6 -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"PingFang SC","Microsoft YaHei",sans-serif}
|
||||
.top{height:52px;background:var(--panel);border-bottom:1px solid var(--line);
|
||||
display:flex;align-items:center;gap:10px;padding:0 18px}
|
||||
.logo{width:26px;height:26px;border-radius:7px;background:var(--brand);display:grid;place-items:center;color:#fff;font-weight:700;font-size:14px}
|
||||
.top b{font-size:15px} .top .ver{font-size:12px;color:var(--mut)}
|
||||
.dot{width:8px;height:8px;border-radius:50%;background:#cbd2dc;display:inline-block;margin-right:5px;vertical-align:middle}
|
||||
.dot.on{background:var(--ok)} .dot.off{background:var(--bad)}
|
||||
.conn{margin-left:auto;font-size:12px;color:var(--mut)}
|
||||
.wrap{max-width:1080px;margin:0 auto;padding:18px;display:grid;grid-template-columns:1fr 360px;gap:16px}
|
||||
@media(max-width:880px){.wrap{grid-template-columns:1fr}}
|
||||
.card{background:var(--panel);border:1px solid var(--line);border-radius:12px;display:flex;flex-direction:column;min-height:0}
|
||||
.card h3{margin:0;padding:12px 16px;font-size:13px;font-weight:600;color:#4b5563;border-bottom:1px solid var(--line);display:flex;align-items:center;gap:8px}
|
||||
.chat{height:62vh}
|
||||
.msgs{flex:1;overflow:auto;padding:16px;display:flex;flex-direction:column;gap:12px}
|
||||
.row{display:flex;flex-direction:column;gap:4px;max-width:82%}
|
||||
.row.me{align-self:flex-end;align-items:flex-end}
|
||||
.row.bot{align-self:flex-start}
|
||||
.bub{padding:9px 13px;border-radius:12px;white-space:pre-wrap;word-break:break-word}
|
||||
.me .bub{background:var(--brand);color:#fff;border-bottom-right-radius:3px}
|
||||
.bot .bub{background:#f1f3f6;color:var(--ink);border-bottom-left-radius:3px}
|
||||
.meta{font-size:11px;color:var(--mut)}
|
||||
.meta .ok{color:var(--ok)} .meta .bad{color:var(--bad)}
|
||||
.sys{align-self:center;font-size:12px;color:var(--mut);background:#f1f3f6;border-radius:8px;padding:4px 12px}
|
||||
.bar{display:flex;gap:8px;padding:12px;border-top:1px solid var(--line)}
|
||||
.bar input{flex:1;border:1px solid var(--line);border-radius:9px;padding:10px 12px;font-size:14px;outline:none}
|
||||
.bar input:focus{border-color:var(--brand);box-shadow:0 0 0 3px var(--brand-soft)}
|
||||
.bar button{background:var(--brand);color:#fff;border:0;border-radius:9px;padding:0 18px;font-size:14px;font-weight:500;cursor:pointer}
|
||||
.bar button:disabled{opacity:.5;cursor:default}
|
||||
.side{height:62vh}
|
||||
.kv{padding:12px 16px;border-bottom:1px solid var(--line);font-size:12px}
|
||||
.kv .k{color:var(--mut)} .kv .v{color:var(--ink);word-break:break-all}
|
||||
.kv code{background:var(--code);border-radius:5px;padding:1px 5px;font-size:11px}
|
||||
.sessrow{display:flex;align-items:center;gap:8px;padding:10px 16px;border-bottom:1px solid var(--line);font-size:12px}
|
||||
.sessrow input{flex:1;border:1px solid var(--line);border-radius:7px;padding:5px 8px;font-size:12px}
|
||||
.sessrow button{border:1px solid var(--line);background:#fff;border-radius:7px;padding:5px 9px;font-size:12px;cursor:pointer;color:#4b5563}
|
||||
.trace{flex:1;overflow:auto;padding:10px 12px;font:11px/1.55 ui-monospace,SFMono-Regular,Menlo,monospace}
|
||||
.ev{padding:6px 8px;border-radius:7px;margin-bottom:6px;border:1px solid var(--line)}
|
||||
.ev .t{font-weight:600;font-size:10px;letter-spacing:.3px;text-transform:uppercase}
|
||||
.ev.out{background:#f5f8ff;border-color:#dbe6ff}.ev.out .t{color:var(--brand)}
|
||||
.ev.ack{background:#f4f6f8}.ev.ack .t{color:#6b7280}
|
||||
.ev.reply{background:#f1faf3;border-color:#cdeed6}.ev.reply .t{color:var(--ok)}
|
||||
.ev pre{margin:3px 0 0;white-space:pre-wrap;word-break:break-all;color:#374151}
|
||||
</style></head>
|
||||
<body>
|
||||
<div class="top">
|
||||
<div class="logo">L</div>
|
||||
<b>HTTP Bot 调试台</b><span class="ver">examples/http-bot</span>
|
||||
<span class="conn"><span class="dot off" id="cdot"></span><span id="conn">连接中…</span></span>
|
||||
</div>
|
||||
<div class="wrap">
|
||||
<!-- chat -->
|
||||
<div class="card chat">
|
||||
<h3>对话 · 真实发往运行中的 http_bot</h3>
|
||||
<div class="msgs" id="msgs"></div>
|
||||
<div class="bar">
|
||||
<input id="msg" placeholder="输入消息,回车发送…" autofocus/>
|
||||
<button id="send">发送</button>
|
||||
</div>
|
||||
</div>
|
||||
<!-- debug -->
|
||||
<div class="card side">
|
||||
<h3>调试信息</h3>
|
||||
<div class="kv"><span class="k">入站地址</span><br><span class="v"><code id="endpoint">/bots/<uuid></code></span></div>
|
||||
<div class="kv"><span class="k">签名</span> <span class="v">HMAC-SHA256 · <code>X-LB-Signature</code></span></div>
|
||||
<div class="sessrow">
|
||||
<span class="k">会话</span>
|
||||
<input id="sid" value="playground-1"/>
|
||||
<button id="reset">新会话</button>
|
||||
</div>
|
||||
<div class="trace" id="trace"></div>
|
||||
</div>
|
||||
</div>
|
||||
<script>
|
||||
const $=s=>document.querySelector(s);
|
||||
const msgs=$('#msgs'),trace=$('#trace'),inp=$('#msg'),btn=$('#send'),
|
||||
conn=$('#conn'),cdot=$('#cdot'),sidIn=$('#sid');
|
||||
function el(c){const d=document.createElement('div');d.className=c;return d}
|
||||
function atBottom(n){n.scrollTop=n.scrollHeight}
|
||||
function bubble(side,text,metaHtml){
|
||||
const r=el('row '+side),b=el('bub');b.textContent=text;r.appendChild(b);
|
||||
if(metaHtml){const m=el('meta');m.innerHTML=metaHtml;r.appendChild(m)}
|
||||
msgs.appendChild(r);atBottom(msgs)}
|
||||
function sys(t){const d=el('sys');d.textContent=t;msgs.appendChild(d);atBottom(msgs)}
|
||||
function logEv(kind,title,obj){
|
||||
const e=el('ev '+kind),t=el('t');t.textContent=title;e.appendChild(t);
|
||||
if(obj!==undefined){const p=document.createElement('pre');
|
||||
p.textContent=typeof obj==='string'?obj:JSON.stringify(obj,null,2);e.appendChild(p)}
|
||||
trace.appendChild(e);atBottom(trace)}
|
||||
|
||||
const es=new EventSource('/events');
|
||||
es.onopen=()=>{conn.textContent='SSE 已连接';cdot.className='dot on'};
|
||||
es.onerror=()=>{conn.textContent='SSE 断开,重连…';cdot.className='dot off'};
|
||||
es.onmessage=e=>{const ev=JSON.parse(e.data);
|
||||
if(ev.kind==='request'){
|
||||
if(ev.endpoint)$('#endpoint').textContent=ev.url||ev.endpoint;
|
||||
logEv('out','出站 · 已签名 POST',{url:ev.url,session_id:ev.session_id,'X-LB-Signature':ev.sig});
|
||||
}else if(ev.kind==='ack'){
|
||||
const id=ev.data&&ev.data.data&&ev.data.data.accepted_message_id;
|
||||
sys(`LangBot 已接收 · HTTP ${ev.status}`);
|
||||
logEv('ack','入站确认 202',{status:ev.status,accepted_message_id:id||'-'});
|
||||
}else if(ev.kind==='reply'){
|
||||
const sig=ev.sig_ok?'<span class=ok>验签通过</span>':'<span class=bad>验签失败</span>';
|
||||
bubble('bot',ev.text,`seq=${ev.sequence} · ${ev.is_final?'<b>FINAL</b>':'中间段'} · ${sig}`);
|
||||
logEv('reply',`回调 · seq ${ev.sequence}${ev.is_final?' · FINAL':''}`,
|
||||
{session_id:ev.session_id,sequence:ev.sequence,is_final:ev.is_final,sig_ok:ev.sig_ok,text:ev.text});
|
||||
}};
|
||||
|
||||
async function send(){
|
||||
const t=inp.value.trim();if(!t)return;inp.value='';btn.disabled=true;
|
||||
bubble('me',t,'已签名 → POST /bots/<uuid>');
|
||||
try{await fetch('/send',{method:'POST',headers:{'Content-Type':'application/json'},
|
||||
body:JSON.stringify({session_id:sidIn.value.trim()||'playground-1',text:t})});}
|
||||
catch(e){sys('发送失败:'+e)}
|
||||
btn.disabled=false;inp.focus();}
|
||||
btn.onclick=send;inp.addEventListener('keydown',e=>{if(e.key==='Enter')send()});
|
||||
$('#reset').onclick=()=>{sidIn.value='playground-'+Math.random().toString(36).slice(2,7);
|
||||
sys('已切换到新会话 '+sidIn.value);};
|
||||
sys('调试台就绪 · 每条消息都会真实发往运行中的 http_bot,右侧可观察签名 / 202 / 回调全过程。');
|
||||
</script>
|
||||
</body></html>"""
|
||||
|
||||
|
||||
async def main():
|
||||
api_key, bot_uuid = db_lookup()
|
||||
callback_url = f'http://{PUBLIC_IP}:{PORT}/callback'
|
||||
print(f'[init] http_bot uuid = {bot_uuid}')
|
||||
print(f'[init] callback_url = {callback_url}')
|
||||
ok = await configure_bot(api_key, bot_uuid, callback_url)
|
||||
if not ok:
|
||||
print('[warn] bot config update failed; check the API key / payload shape')
|
||||
|
||||
app = web.Application()
|
||||
app['bot_uuid'] = bot_uuid
|
||||
app.router.add_get('/', index)
|
||||
app.router.add_post('/send', send)
|
||||
app.router.add_post('/callback', callback)
|
||||
app.router.add_get('/events', events)
|
||||
|
||||
runner = web.AppRunner(app)
|
||||
await runner.setup()
|
||||
site = web.TCPSite(runner, '0.0.0.0', PORT)
|
||||
await site.start()
|
||||
print(f'\n ▶ 打开: http://{PUBLIC_IP}:{PORT}/\n')
|
||||
while True:
|
||||
await asyncio.sleep(3600)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -1,48 +0,0 @@
|
||||
# Page Bot Adapter — Embed Demo
|
||||
|
||||
> English | [中文](./README.zh.md)
|
||||
|
||||
A single self-contained HTML page that demos the LangBot **Page Bot**
|
||||
(`web_page_bot`) embeddable chat widget — the one you drop onto any website with
|
||||
a single `<script>` tag.
|
||||
|
||||
Full guide: [docs.langbot.app — Page Bot](https://docs.langbot.app/en/usage/platforms/webpage).
|
||||
|
||||
## Files
|
||||
|
||||
| File | What it is |
|
||||
|---|---|
|
||||
| `index.html` | **Browser demo** — open it, point it at a running LangBot instance + a Page Bot you created, and it loads the live embed widget so you can chat with the bot exactly as a site visitor would. Zero deps, no build step. |
|
||||
|
||||
## How to use
|
||||
|
||||
1. In the LangBot WebUI, create a bot with the **Page Bot** (`页面机器人`)
|
||||
adapter and bind it to a working pipeline. Copy its **bot UUID** from the
|
||||
generated embed code.
|
||||
2. Open `index.html` in a browser. Any of these work:
|
||||
- double-click the file, or
|
||||
- serve the folder: `python3 -m http.server 8930` then open
|
||||
`http://localhost:8930/examples/web-page-bot/`.
|
||||
3. Fill in:
|
||||
- **LangBot base URL** — where your instance is reachable from the browser
|
||||
(e.g. `http://localhost:5300`, or your public address).
|
||||
- **Page Bot UUID** — from step 1.
|
||||
- **Widget title** — optional, sets the `data-title` attribute.
|
||||
4. Click **Load widget**. A floating chat bubble appears in the bottom-right
|
||||
corner — click it and chat.
|
||||
|
||||
The page also renders the exact `<script>` snippet you'd paste into your own
|
||||
site (before `</body>`), and updates it live as you edit the fields.
|
||||
|
||||
## What it demonstrates
|
||||
|
||||
- The embed contract: `<script data-title="…" src="<base>/api/v1/embed/<uuid>/widget.js"></script>`.
|
||||
- `widget.js` is served by LangBot pre-configured for that bot UUID — title,
|
||||
bubble icon, language and optional Cloudflare Turnstile protection all come
|
||||
from the bot's config, no page changes needed.
|
||||
- Messages travel over a WebSocket to the bot's bound pipeline; replies stream
|
||||
back into the bubble.
|
||||
|
||||
> The widget loads `widget.js` from your LangBot instance, so the **base URL
|
||||
> must be reachable from the browser** you open this page in. If LangBot runs on
|
||||
> a server, use its public address instead of `localhost`.
|
||||
@@ -1,44 +0,0 @@
|
||||
# 页面机器人适配器 —— 嵌入演示
|
||||
|
||||
> [English](./README.md) | 中文
|
||||
|
||||
一个自包含的单文件 HTML 页面，用于演示 LangBot **页面机器人**
|
||||
(`web_page_bot`) 的可嵌入聊天组件 —— 也就是你用一行 `<script>` 标签就能放到任意
|
||||
网站上的那个组件。
|
||||
|
||||
完整指南：[docs.langbot.app —— 页面机器人](https://docs.langbot.app/zh/usage/platforms/webpage)。
|
||||
|
||||
## 文件清单
|
||||
|
||||
| 文件 | 是什么 |
|
||||
|---|---|
|
||||
| `index.html` | **浏览器演示页** —— 打开它，填上一个运行中的 LangBot 实例地址 + 你创建的页面机器人，它就会加载真实的嵌入组件，让你像网站访客一样和机器人对话。零依赖，无需构建。 |
|
||||
|
||||
## 使用方法
|
||||
|
||||
1. 在 LangBot WebUI 中，用 **页面机器人**（`web_page_bot`）适配器创建一个机器人，
|
||||
并绑定一个可用的流水线。从生成的嵌入代码里复制它的 **机器人 UUID**。
|
||||
2. 在浏览器中打开 `index.html`，以下任一方式皆可：
|
||||
- 直接双击该文件；或
|
||||
- 起一个静态服务：`python3 -m http.server 8930`，然后打开
|
||||
`http://localhost:8930/examples/web-page-bot/`。
|
||||
3. 填写：
|
||||
- **LangBot base URL** —— 你的实例在该浏览器中可访问的地址
|
||||
（例如 `http://localhost:5300`，或你的公网地址）。
|
||||
- **页面机器人 UUID** —— 第 1 步里复制的。
|
||||
- **组件标题** —— 可选，对应 `data-title` 属性。
|
||||
4. 点击 **Load widget**。页面右下角会出现一个浮动聊天气泡 —— 点开即可对话。
|
||||
|
||||
页面还会实时渲染出你需要粘贴到自己网站（放在 `</body>` 前）的那段 `<script>`
|
||||
代码，并随着你编辑输入框同步更新。
|
||||
|
||||
## 它演示了什么
|
||||
|
||||
- 嵌入契约：`<script data-title="…" src="<base>/api/v1/embed/<uuid>/widget.js"></script>`。
|
||||
- `widget.js` 由 LangBot 针对该机器人 UUID 预配置后下发 —— 标题、气泡图标、语言
|
||||
以及可选的 Cloudflare Turnstile 防护，全部来自机器人配置，无需改动页面。
|
||||
- 消息通过 WebSocket 发往机器人绑定的流水线，回复流式回到气泡中。
|
||||
|
||||
> 组件会从你的 LangBot 实例加载 `widget.js`，因此 **base URL 必须能从你打开本页
|
||||
> 的浏览器访问到**。如果 LangBot 部署在服务器上，请用它的公网地址而非
|
||||
> `localhost`。
|
||||
@@ -1,205 +0,0 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>LangBot Page Bot · Embed Demo</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg: #f7f8fa; --panel: #ffffff; --line: #e8eaed; --ink: #1f2329;
|
||||
--mut: #8a909a; --brand: #2563eb; --brand-soft: #eef3ff;
|
||||
--ok: #16a34a; --bad: #dc2626; --code: #f3f4f6;
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
html, body { height: 100%; }
|
||||
body {
|
||||
margin: 0; background: var(--bg); color: var(--ink);
|
||||
font: 14px/1.6 -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto,
|
||||
"PingFang SC", "Microsoft YaHei", sans-serif;
|
||||
}
|
||||
.top {
|
||||
height: 52px; background: var(--panel); border-bottom: 1px solid var(--line);
|
||||
display: flex; align-items: center; gap: 10px; padding: 0 18px;
|
||||
}
|
||||
.logo {
|
||||
width: 26px; height: 26px; border-radius: 7px; background: var(--brand);
|
||||
display: grid; place-items: center; color: #fff; font-weight: 700; font-size: 14px;
|
||||
}
|
||||
.top b { font-size: 15px; }
|
||||
.top .ver { font-size: 12px; color: var(--mut); }
|
||||
.wrap { max-width: 760px; margin: 0 auto; padding: 28px 18px 80px; }
|
||||
.hero h1 { margin: 8px 0 6px; font-size: 22px; }
|
||||
.hero p { margin: 0 0 4px; color: var(--mut); }
|
||||
.card {
|
||||
background: var(--panel); border: 1px solid var(--line); border-radius: 12px;
|
||||
padding: 20px; margin-top: 20px;
|
||||
}
|
||||
.card h3 {
|
||||
margin: 0 0 14px; font-size: 14px; font-weight: 600; color: #4b5563;
|
||||
display: flex; align-items: center; gap: 8px;
|
||||
}
|
||||
.card h3 .num {
|
||||
width: 20px; height: 20px; border-radius: 50%; background: var(--brand-soft);
|
||||
color: var(--brand); display: grid; place-items: center; font-size: 12px; font-weight: 700;
|
||||
}
|
||||
.field { margin-bottom: 14px; }
|
||||
.field:last-child { margin-bottom: 0; }
|
||||
.field label { display: block; font-size: 12px; color: var(--mut); margin-bottom: 5px; }
|
||||
.field input {
|
||||
width: 100%; border: 1px solid var(--line); border-radius: 9px;
|
||||
padding: 10px 12px; font-size: 14px; outline: none; font-family: inherit;
|
||||
}
|
||||
.field input:focus { border-color: var(--brand); box-shadow: 0 0 0 3px var(--brand-soft); }
|
||||
.hint { font-size: 12px; color: var(--mut); margin-top: 5px; }
|
||||
.hint code { background: var(--code); border-radius: 5px; padding: 1px 5px; font-size: 11px; }
|
||||
.actions { display: flex; gap: 10px; margin-top: 18px; align-items: center; }
|
||||
button {
|
||||
border: 0; border-radius: 9px; padding: 10px 18px; font-size: 14px;
|
||||
font-weight: 500; cursor: pointer; font-family: inherit;
|
||||
}
|
||||
.btn-primary { background: var(--brand); color: #fff; }
|
||||
.btn-primary:disabled { opacity: .5; cursor: default; }
|
||||
.btn-ghost { background: #fff; border: 1px solid var(--line); color: #4b5563; }
|
||||
.status { font-size: 13px; color: var(--mut); }
|
||||
.status .ok { color: var(--ok); }
|
||||
.status .bad { color: var(--bad); }
|
||||
pre {
|
||||
background: #0f172a; color: #e2e8f0; border-radius: 10px; padding: 14px 16px;
|
||||
overflow: auto; font: 12px/1.6 ui-monospace, SFMono-Regular, Menlo, monospace;
|
||||
margin: 0;
|
||||
}
|
||||
.snippet-row { position: relative; }
|
||||
.snippet-row .copy {
|
||||
position: absolute; top: 10px; right: 10px; background: rgba(255,255,255,.12);
|
||||
color: #fff; border: 0; border-radius: 7px; padding: 5px 10px; font-size: 12px; cursor: pointer;
|
||||
}
|
||||
ul.steps { margin: 0; padding-left: 18px; color: #4b5563; }
|
||||
ul.steps li { margin-bottom: 6px; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="top">
|
||||
<div class="logo">L</div>
|
||||
<b>Page Bot · Embed Demo</b>
|
||||
<span class="ver">examples/web-page-bot</span>
|
||||
</div>
|
||||
|
||||
<div class="wrap">
|
||||
<div class="hero">
|
||||
<h1>Try the LangBot Page Bot widget</h1>
|
||||
<p>Point this page at a running LangBot instance and a <strong>Page Bot</strong> you created,</p>
|
||||
<p>then load the live embed widget below to chat with it — exactly as your site visitors would.</p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3><span class="num">1</span> Connect your Page Bot</h3>
|
||||
<div class="field">
|
||||
<label for="base">LangBot base URL</label>
|
||||
<input id="base" placeholder="http://localhost:5300" value="http://localhost:5300" />
|
||||
<div class="hint">The address where your LangBot instance is reachable from this browser. No trailing slash.</div>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label for="uuid">Page Bot UUID</label>
|
||||
<input id="uuid" placeholder="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" />
|
||||
<div class="hint">Create a bot with the <code>Page Bot</code> adapter in the WebUI, then copy its UUID from the embed code.</div>
|
||||
</div>
|
||||
<div class="field">
|
||||
<label for="title">Widget title (optional)</label>
|
||||
<input id="title" placeholder="LangBot" value="LangBot" />
|
||||
</div>
|
||||
<div class="actions">
|
||||
<button id="load" class="btn-primary">Load widget</button>
|
||||
<button id="unload" class="btn-ghost">Remove widget</button>
|
||||
<span class="status" id="status">Not loaded.</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3><span class="num">2</span> The embed snippet</h3>
|
||||
<p style="margin:0 0 12px;color:var(--mut)">This is exactly what you paste into your own site (before <code></body></code>). It updates as you edit the fields above.</p>
|
||||
<div class="snippet-row">
|
||||
<button class="copy" id="copy">Copy</button>
|
||||
<pre id="snippet"><script data-title="LangBot" src="http://localhost:5300/api/v1/embed/<bot-uuid>/widget.js"></script></pre>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3><span class="num">3</span> How it works</h3>
|
||||
<ul class="steps">
|
||||
<li>The <code><script></code> tag pulls <code>widget.js</code> from your LangBot instance, pre-configured for that bot UUID.</li>
|
||||
<li>A floating chat bubble appears in the bottom-right corner of the page.</li>
|
||||
<li>Messages travel over a WebSocket to the bot's bound pipeline; replies stream back into the bubble.</li>
|
||||
<li>Title, bubble icon, language and optional Cloudflare Turnstile protection are all set in the bot's config — no page changes needed.</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
var $ = function (s) { return document.querySelector(s); };
|
||||
var baseEl = $("#base"), uuidEl = $("#uuid"), titleEl = $("#title"),
|
||||
statusEl = $("#status"), snippetEl = $("#snippet");
|
||||
var WIDGET_ID = "langbot-embed-demo-script";
|
||||
|
||||
function clean(v) { return (v || "").trim().replace(/\/+$/, ""); }
|
||||
|
||||
function buildSrc() {
|
||||
var base = clean(baseEl.value) || "http://localhost:5300";
|
||||
var uuid = uuidEl.value.trim() || "<bot-uuid>";
|
||||
return base + "/api/v1/embed/" + uuid + "/widget.js";
|
||||
}
|
||||
|
||||
function refreshSnippet() {
|
||||
var title = titleEl.value.trim() || "LangBot";
|
||||
var src = buildSrc();
|
||||
snippetEl.textContent =
|
||||
'<script data-title="' + title + '" src="' + src + '"><\/script>';
|
||||
}
|
||||
|
||||
function setStatus(html) { statusEl.innerHTML = html; }
|
||||
|
||||
function removeWidget() {
|
||||
var old = document.getElementById(WIDGET_ID);
|
||||
if (old) old.remove();
|
||||
// The widget injects its own DOM (bubble + panel). Clear the common containers it creates.
|
||||
document.querySelectorAll('[id^="langbot-"]').forEach(function (n) {
|
||||
if (n.id !== WIDGET_ID) n.remove();
|
||||
});
|
||||
}
|
||||
|
||||
function loadWidget() {
|
||||
var uuid = uuidEl.value.trim();
|
||||
var uuidRe = /^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/i;
|
||||
if (!uuidRe.test(uuid)) {
|
||||
setStatus('<span class="bad">Enter a valid bot UUID first.</span>');
|
||||
return;
|
||||
}
|
||||
removeWidget();
|
||||
var s = document.createElement("script");
|
||||
s.id = WIDGET_ID;
|
||||
s.setAttribute("data-title", titleEl.value.trim() || "LangBot");
|
||||
s.src = buildSrc();
|
||||
s.onload = function () {
|
||||
setStatus('<span class="ok">Widget loaded — look bottom-right.</span>');
|
||||
};
|
||||
s.onerror = function () {
|
||||
setStatus('<span class="bad">Failed to load widget.js — check the base URL and that the bot is enabled.</span>');
|
||||
};
|
||||
document.body.appendChild(s);
|
||||
setStatus("Loading…");
|
||||
}
|
||||
|
||||
$("#load").onclick = loadWidget;
|
||||
$("#unload").onclick = function () {
|
||||
removeWidget();
|
||||
setStatus("Widget removed.");
|
||||
};
|
||||
$("#copy").onclick = function () {
|
||||
navigator.clipboard.writeText(snippetEl.textContent).then(function () {
|
||||
var b = $("#copy"); b.textContent = "Copied"; setTimeout(function () { b.textContent = "Copy"; }, 1200);
|
||||
});
|
||||
};
|
||||
[baseEl, uuidEl, titleEl].forEach(function (el) { el.addEventListener("input", refreshSnippet); });
|
||||
refreshSnippet();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
+6
-6
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "langbot"
|
||||
version = "4.10.4"
|
||||
version = "4.10.2"
|
||||
description = "Production-grade platform for building agentic IM bots"
|
||||
readme = "README.md"
|
||||
license-files = ["LICENSE"]
|
||||
@@ -8,7 +8,7 @@ requires-python = ">=3.11,<4.0"
|
||||
dependencies = [
|
||||
"aiocqhttp>=1.4.4",
|
||||
"aiofiles>=24.1.0",
|
||||
"aiohttp>=3.14.1",
|
||||
"aiohttp>=3.14.0",
|
||||
"aioshutil>=1.5",
|
||||
"aiosqlite>=0.21.0",
|
||||
"anthropic>=0.51.0",
|
||||
@@ -16,7 +16,7 @@ dependencies = [
|
||||
"async-lru>=2.0.5",
|
||||
"certifi>=2025.4.26",
|
||||
"colorlog~=6.6.0",
|
||||
"cryptography>=48.0.1",
|
||||
"cryptography>=46.0.7",
|
||||
"dashscope>=1.25.10",
|
||||
"dingtalk-stream>=0.24.0",
|
||||
"discord-py>=2.5.2",
|
||||
@@ -61,16 +61,16 @@ dependencies = [
|
||||
"beautifulsoup4>=4.12.3",
|
||||
"ebooklib>=0.18",
|
||||
"html2text>=2024.2.26",
|
||||
"langchain>=1.3.9",
|
||||
"langchain>=0.2.0",
|
||||
"langchain-core>=1.3.3",
|
||||
"langsmith>=0.8.18",
|
||||
"langsmith>=0.8.0",
|
||||
"python-multipart>=0.0.27",
|
||||
"Mako>=1.3.12",
|
||||
"langchain-text-splitters>=1.1.2",
|
||||
"chromadb>=1.0.0,<2.0.0",
|
||||
"qdrant-client (>=1.15.1,<2.0.0)",
|
||||
"pyseekdb==1.1.0.post3",
|
||||
"langbot-plugin @ file:///home/qinjunyan/code/projects/langbot/langbot-plugin-sdk",
|
||||
"langbot-plugin==0.4.5",
|
||||
"asyncpg>=0.30.0",
|
||||
"line-bot-sdk>=3.19.0",
|
||||
"matrix-nio>=0.25.2",
|
||||
|
||||
+1
-2
@@ -26,7 +26,7 @@ and LangBot's own Local Agent) working with the LangBot ecosystem.
|
||||
|
||||
## Quick start (for an AI agent)
|
||||
|
||||
1. Read this README, `AGENTS.md`, and `docs/user-guide.md` to understand the layout.
|
||||
1. Read this README, `AGENTS.md`, and `qa-agent-docs/` to understand the layout.
|
||||
2. Read `skills/.env` for shared local defaults. On a new machine, copy
|
||||
`skills/.env.example` to `skills/.env.local` (gitignored) and override
|
||||
machine-specific values there. Never commit secrets.
|
||||
@@ -48,7 +48,6 @@ bin/lbs env show # inspect resolved env defaults (redacted)
|
||||
bin/lbs env doctor # diagnose local environment readiness
|
||||
bin/lbs case list --ready
|
||||
bin/lbs test plan <case-id>
|
||||
bin/lbs suite plan langbot-debug-chat-load-gate
|
||||
```
|
||||
|
||||
## Maintenance rule
|
||||
|
||||
@@ -1,171 +0,0 @@
|
||||
# LangBot QA Skills User Guide
|
||||
|
||||
Use this guide as the first operational path after reading `README.md` and
|
||||
`AGENTS.md`.
|
||||
|
||||
## 1. Configure Local Inputs
|
||||
|
||||
Read `skills/.env`, then create `skills/.env.local` for machine-local values.
|
||||
Do not commit `.env.local`, browser profiles, reports, tokens, API keys, OAuth
|
||||
state, or provider credentials.
|
||||
|
||||
Minimum local fields for live browser QA:
|
||||
|
||||
```bash
|
||||
LANGBOT_REPO=/path/to/LangBot
|
||||
LANGBOT_WEB_REPO=/path/to/LangBot/web
|
||||
LANGBOT_BACKEND_URL=http://127.0.0.1:5300
|
||||
LANGBOT_FRONTEND_URL=http://127.0.0.1:3000
|
||||
LANGBOT_DEV_FRONTEND_URL=http://127.0.0.1:3000
|
||||
LANGBOT_BROWSER_PROFILE=/path/to/langbot-browser-profile
|
||||
LANGBOT_CHROMIUM_EXECUTABLE=/path/to/chromium-or-playwright-chrome
|
||||
LANGBOT_E2E_LOGIN_USER=qa-local@example.com
|
||||
```
|
||||
|
||||
`LANGBOT_E2E_LOGIN_USER` is a local QA account. The setup automation uses the
|
||||
LangBot recovery key from the active checkout to initialize or refresh that
|
||||
local account and write a browser `localStorage` token. It does not need the
|
||||
user's GitHub or Space credentials.
|
||||
|
||||
## 2. Check Readiness
|
||||
|
||||
From `skills/`:
|
||||
|
||||
```bash
|
||||
bin/lbs env show
|
||||
bin/lbs env doctor
|
||||
bin/lbs validate
|
||||
bin/lbs index --check
|
||||
```
|
||||
|
||||
`env doctor` should report reachable backend and frontend URLs before live
|
||||
browser cases are run. Missing Space provider credentials are not a LangBot
|
||||
product pass; classify them as `env_issue` and configure the local Space
|
||||
provider before measuring Debug Chat performance.
|
||||
|
||||
## 3. Start Services
|
||||
|
||||
Start the backend from `LANGBOT_REPO`:
|
||||
|
||||
```bash
|
||||
cd "$LANGBOT_REPO"
|
||||
uv run main.py
|
||||
```
|
||||
|
||||
Start the standalone frontend from `LANGBOT_WEB_REPO` and point it at the
|
||||
backend:
|
||||
|
||||
```bash
|
||||
cd "$LANGBOT_WEB_REPO"
|
||||
VITE_API_BASE_URL="$LANGBOT_BACKEND_URL" pnpm dev --host 0.0.0.0
|
||||
```
|
||||
|
||||
If `VITE_API_BASE_URL` is missing, browser tests can load the Vite page but send
|
||||
API requests to the frontend port, which produces false UI failures.
|
||||
|
||||
## 4. Prepare User-Path Fixtures
|
||||
|
||||
For local-agent Debug Chat cases and the user-path performance gate:
|
||||
|
||||
```bash
|
||||
node scripts/e2e/ensure-local-agent-pipeline.mjs --write-env
|
||||
```
|
||||
|
||||
The script:
|
||||
|
||||
- refreshes the local QA login and browser token;
|
||||
- marks the local wizard as skipped;
|
||||
- creates or updates a local QA pipeline;
|
||||
- scans Space LLM models, tests candidates, and switches to the first working
|
||||
Space model with tested fallback models;
|
||||
- writes `LANGBOT_PIPELINE_URL`, `LANGBOT_PIPELINE_NAME`, and local-agent
|
||||
pipeline/model variables into `skills/.env.local`;
|
||||
- returns `env_issue` when no Space model can be scanned or tested.
|
||||
|
||||
Useful model controls:
|
||||
|
||||
```bash
|
||||
LANGBOT_E2E_MODEL_TEST_LIMIT=8
|
||||
LANGBOT_E2E_MODEL_FALLBACK_COUNT=3
|
||||
LANGBOT_E2E_SKIP_MODEL_UUIDS=uuid-a,uuid-b
|
||||
LANGBOT_E2E_SKIP_MODEL_NAMES=model-a,model-b
|
||||
LANGBOT_E2E_SCAN_SPACE_MODELS=true
|
||||
```
|
||||
|
||||
The setup writes a current-runtime compatibility `max-round` value into the
|
||||
pipeline config because this backend still reads that field directly during
|
||||
message truncation. Do not treat it as a long-term QA contract.
|
||||
|
||||
## 5. Run Gates
|
||||
|
||||
Fast contract gate, no live service required:
|
||||
|
||||
```bash
|
||||
bin/lbs suite run langbot-performance-contract-gate --run-id langbot-contract-local
|
||||
```
|
||||
|
||||
Live backend gate:
|
||||
|
||||
```bash
|
||||
bin/lbs suite run langbot-live-backend-gate --run-id langbot-backend-local
|
||||
```
|
||||
|
||||
Browser-visible user-path performance gate:
|
||||
|
||||
```bash
|
||||
bin/lbs suite plan langbot-user-path-performance-gate
|
||||
bin/lbs suite run langbot-user-path-performance-gate --run-id langbot-user-path-local --include-manual-check
|
||||
```
|
||||
|
||||
Controlled Debug Chat message-path load gate (manual/non-required; run fake-provider cases serially when they share `LANGBOT_FAKE_PROVIDER_URL`):
|
||||
|
||||
```bash
|
||||
bin/lbs suite plan langbot-debug-chat-load-gate
|
||||
bin/lbs test run langbot-fake-provider-debug-chat-load --run-id langbot-fake-load-local
|
||||
bin/lbs test run langbot-fake-provider-debug-chat-slow-load --run-id langbot-fake-slow-local
|
||||
bin/lbs test run langbot-fake-provider-debug-chat-fault-recovery --run-id langbot-fake-fault-local
|
||||
bin/lbs test run langbot-space-debug-chat-concurrency-smoke --run-id langbot-space-smoke-local
|
||||
```
|
||||
|
||||
Cross-pipeline Debug Chat isolation is a separate manual regression gate because
|
||||
current releases may fail it due to product bug #2286:
|
||||
|
||||
```bash
|
||||
bin/lbs suite plan langbot-debug-chat-isolation-gate
|
||||
bin/lbs suite run langbot-debug-chat-isolation-gate --run-id langbot-debug-chat-isolation-local --include-manual-check
|
||||
```
|
||||
|
||||
Start with `langbot-fake-provider-debug-chat-load`. It launches a local
|
||||
OpenAI-compatible fake provider, creates the matching provider/model/pipeline,
|
||||
then sends concurrent WebSocket Debug Chat messages through the real backend.
|
||||
Use `langbot-fake-provider-debug-chat-slow-load` to measure the same path under
|
||||
deterministic streaming latency. Use
|
||||
`langbot-fake-provider-debug-chat-fault-recovery` to inject bounded provider
|
||||
HTTP failures and confirm later Debug Chat requests recover. Use the separate
|
||||
`langbot-debug-chat-isolation-gate` to verify that concurrent Debug Chat traffic
|
||||
on two pipelines does not leak assistant responses across pipeline boundaries;
|
||||
current releases may fail that gate because of #2286, so keep it out of the
|
||||
normal load gate until the product fix lands.
|
||||
Use `langbot-space-debug-chat-concurrency-smoke` only as a low-volume live
|
||||
provider smoke; it includes Space/model/network latency and should be compared
|
||||
against the fake-provider baseline before attributing failures to LangBot.
|
||||
|
||||
`manual_check` means the agent must confirm the declared preconditions for that
|
||||
run window. When setup automation is declared, run output may stop early with
|
||||
`env_issue`; fix that environment input before treating the product path as
|
||||
measured.
|
||||
|
||||
## 6. Read Results
|
||||
|
||||
Suite reports live under `skills/reports/`. Evidence lives under
|
||||
`skills/reports/evidence/<run-id>/`.
|
||||
|
||||
For performance cases, inspect:
|
||||
|
||||
- `metrics.json` for p50/p95/p99, error rate, and total duration;
|
||||
- `automation-result.json` for threshold decisions and artifacts;
|
||||
- `console.log` and `network.log` for frontend/API failures;
|
||||
- backend logs for provider, runner, WebSocket, or persistence failures.
|
||||
|
||||
Do not call a user-path performance result a LangBot overhead regression until
|
||||
provider/tool/network time has been separated or ruled out.
|
||||
@@ -48,18 +48,7 @@
|
||||
},
|
||||
"type": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"smoke",
|
||||
"regression",
|
||||
"feature",
|
||||
"provider",
|
||||
"exploratory",
|
||||
"contract",
|
||||
"performance",
|
||||
"reliability",
|
||||
"chaos",
|
||||
"security"
|
||||
]
|
||||
"enum": ["smoke", "regression", "feature", "provider", "exploratory"]
|
||||
},
|
||||
"priority": {
|
||||
"type": "string",
|
||||
@@ -113,11 +102,7 @@
|
||||
"backend_log",
|
||||
"frontend_log",
|
||||
"api_diagnostic",
|
||||
"filesystem",
|
||||
"metrics",
|
||||
"trace",
|
||||
"profile",
|
||||
"resource_log"
|
||||
"filesystem"
|
||||
]
|
||||
},
|
||||
"minItems": 1
|
||||
@@ -203,101 +188,9 @@
|
||||
"type": "string",
|
||||
"enum": ["person", "group"]
|
||||
},
|
||||
"automation_debug_chat_response_p95_ms": {
|
||||
"type": "string"
|
||||
},
|
||||
"automation_debug_chat_max_error_rate": {
|
||||
"type": "string"
|
||||
},
|
||||
"automation_debug_chat_load_requests": {
|
||||
"type": "string"
|
||||
},
|
||||
"automation_debug_chat_load_concurrency": {
|
||||
"type": "string"
|
||||
},
|
||||
"automation_debug_chat_load_timeout_ms": {
|
||||
"type": "string"
|
||||
},
|
||||
"automation_debug_chat_load_response_p95_ms": {
|
||||
"type": "string"
|
||||
},
|
||||
"automation_debug_chat_load_first_response_p95_ms": {
|
||||
"type": "string"
|
||||
},
|
||||
"automation_debug_chat_load_max_error_rate": {
|
||||
"type": "string"
|
||||
},
|
||||
"automation_debug_chat_load_min_error_rate": {
|
||||
"type": "string"
|
||||
},
|
||||
"automation_debug_chat_load_min_error_count": {
|
||||
"type": "string"
|
||||
},
|
||||
"automation_debug_chat_load_min_ok_count": {
|
||||
"type": "string"
|
||||
},
|
||||
"automation_debug_chat_load_min_provider_fault_count": {
|
||||
"type": "string"
|
||||
},
|
||||
"automation_debug_chat_load_expected_prefix": {
|
||||
"type": "string"
|
||||
},
|
||||
"automation_debug_chat_load_prompt_template": {
|
||||
"type": "string"
|
||||
},
|
||||
"automation_debug_chat_load_stream": {
|
||||
"type": "string",
|
||||
"enum": ["0", "1", "false", "true"]
|
||||
},
|
||||
"automation_debug_chat_load_reset": {
|
||||
"type": "string",
|
||||
"enum": ["0", "1", "false", "true"]
|
||||
},
|
||||
"automation_debug_chat_load_fail_on_final_mismatch": {
|
||||
"type": "string",
|
||||
"enum": ["0", "1", "false", "true"]
|
||||
},
|
||||
"automation_fake_provider_response_text": {
|
||||
"type": "string"
|
||||
},
|
||||
"automation_fake_provider_first_token_delay_ms": {
|
||||
"type": "string"
|
||||
},
|
||||
"automation_fake_provider_chunk_delay_ms": {
|
||||
"type": "string"
|
||||
},
|
||||
"automation_fake_provider_chunk_count": {
|
||||
"type": "string"
|
||||
},
|
||||
"automation_fake_provider_fail_first_n": {
|
||||
"type": "string"
|
||||
},
|
||||
"automation_fake_provider_fail_every_n": {
|
||||
"type": "string"
|
||||
},
|
||||
"automation_fake_provider_fault_status": {
|
||||
"type": "string"
|
||||
},
|
||||
"automation_fake_provider_fail_after_first_chunk": {
|
||||
"type": "string",
|
||||
"enum": ["0", "1", "false", "true"]
|
||||
},
|
||||
"automation_fake_provider_dynamic_response": {
|
||||
"type": "string",
|
||||
"enum": ["0", "1", "false", "true"]
|
||||
},
|
||||
"automation_filesystem_checks_json": {
|
||||
"type": "string"
|
||||
},
|
||||
"metrics_thresholds_json": {
|
||||
"type": "string"
|
||||
},
|
||||
"load_profile_json": {
|
||||
"type": "string"
|
||||
},
|
||||
"fault_model_json": {
|
||||
"type": "string"
|
||||
},
|
||||
"automation_pipeline_url_env": {
|
||||
"type": "string",
|
||||
"pattern": "^[A-Z][A-Z0-9_]*$"
|
||||
|
||||
@@ -18,17 +18,7 @@
|
||||
},
|
||||
"type": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"smoke",
|
||||
"regression",
|
||||
"release_gate",
|
||||
"exploratory",
|
||||
"contract",
|
||||
"performance",
|
||||
"reliability",
|
||||
"chaos",
|
||||
"security"
|
||||
]
|
||||
"enum": ["smoke", "regression", "release_gate", "exploratory"]
|
||||
},
|
||||
"priority": {
|
||||
"type": "string",
|
||||
|
||||
Executable → Regular
Executable → Regular
Executable → Regular
@@ -1,205 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { spawn } from "node:child_process";
|
||||
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
||||
import { dirname, resolve } from "node:path";
|
||||
import { env } from "node:process";
|
||||
import {
|
||||
appendLine,
|
||||
ensureEvidence,
|
||||
evidencePaths,
|
||||
loadEnvFiles,
|
||||
redact,
|
||||
writeResult,
|
||||
} from "./lib/langbot-e2e.mjs";
|
||||
|
||||
const caseId = "ensure-fake-provider-cross-pipelines";
|
||||
const DEFAULT_PIPELINE_A_NAME = "LangBot QA Fake Provider Debug Chat A";
|
||||
const DEFAULT_PIPELINE_B_NAME = "LangBot QA Fake Provider Debug Chat B";
|
||||
|
||||
await loadEnvFiles();
|
||||
const paths = evidencePaths(caseId);
|
||||
await ensureEvidence(paths);
|
||||
|
||||
const writeEnv = process.argv.includes("--write-env");
|
||||
const envLocalPath = resolve("skills/.env.local");
|
||||
const pipelineAName = env.LANGBOT_FAKE_PROVIDER_PIPELINE_A_NAME || DEFAULT_PIPELINE_A_NAME;
|
||||
const pipelineBName = env.LANGBOT_FAKE_PROVIDER_PIPELINE_B_NAME || DEFAULT_PIPELINE_B_NAME;
|
||||
|
||||
const result = {
|
||||
source: "setup_automation",
|
||||
case_id: caseId,
|
||||
run_id: paths.runId,
|
||||
status: "fail",
|
||||
reason: "",
|
||||
pipeline_a: {
|
||||
name: pipelineAName,
|
||||
id: "",
|
||||
url: "",
|
||||
},
|
||||
pipeline_b: {
|
||||
name: pipelineBName,
|
||||
id: "",
|
||||
url: "",
|
||||
},
|
||||
fake_provider: {
|
||||
url: "",
|
||||
base_url: "",
|
||||
pid: null,
|
||||
},
|
||||
wrote_env: false,
|
||||
evidence: {
|
||||
console_log: paths.consoleLog,
|
||||
automation_result_json: paths.automationResultJson,
|
||||
result_json: paths.resultJson,
|
||||
},
|
||||
evidence_collected: ["api_diagnostic", "filesystem"],
|
||||
};
|
||||
|
||||
try {
|
||||
console.error(`[langbot-qa] configuring cross-pipeline QA fixtures: pipeline_a=\"${pipelineAName}\", pipeline_b=\"${pipelineBName}\"`);
|
||||
console.error("[langbot-qa] run these fake-provider setup/probe commands serially when they share LANGBOT_FAKE_PROVIDER_URL.");
|
||||
if (pipelineAName === pipelineBName) {
|
||||
throw new Error("LANGBOT_FAKE_PROVIDER_PIPELINE_A_NAME and LANGBOT_FAKE_PROVIDER_PIPELINE_B_NAME must be different.");
|
||||
}
|
||||
|
||||
const setupA = await runPipelineSetup(pipelineAName, "A");
|
||||
const setupB = await runPipelineSetup(pipelineBName, "B");
|
||||
result.pipeline_a = {
|
||||
name: setupA.pipeline_name || pipelineAName,
|
||||
id: setupA.pipeline_id || "",
|
||||
url: setupA.pipeline_url || "",
|
||||
};
|
||||
result.pipeline_b = {
|
||||
name: setupB.pipeline_name || pipelineBName,
|
||||
id: setupB.pipeline_id || "",
|
||||
url: setupB.pipeline_url || "",
|
||||
};
|
||||
result.fake_provider = {
|
||||
url: setupB.fake_provider?.url || setupA.fake_provider?.url || "",
|
||||
base_url: setupB.fake_provider?.base_url || setupA.fake_provider?.base_url || "",
|
||||
pid: setupB.fake_provider?.pid ?? setupA.fake_provider?.pid ?? null,
|
||||
};
|
||||
|
||||
if (!result.pipeline_a.url || !result.pipeline_b.url || !result.fake_provider.url) {
|
||||
throw new Error("Cross-pipeline fake provider setup did not return both pipeline URLs and provider URL.");
|
||||
}
|
||||
|
||||
if (writeEnv) {
|
||||
await upsertEnvLocal(envLocalPath, {
|
||||
LANGBOT_FAKE_PROVIDER_URL: result.fake_provider.url,
|
||||
LANGBOT_FAKE_PROVIDER_BASE_URL: result.fake_provider.base_url,
|
||||
LANGBOT_FAKE_PROVIDER_PID: result.fake_provider.pid ? String(result.fake_provider.pid) : "",
|
||||
LANGBOT_FAKE_PROVIDER_PIPELINE_A_URL: result.pipeline_a.url,
|
||||
LANGBOT_FAKE_PROVIDER_PIPELINE_A_NAME: result.pipeline_a.name,
|
||||
LANGBOT_FAKE_PROVIDER_PIPELINE_B_URL: result.pipeline_b.url,
|
||||
LANGBOT_FAKE_PROVIDER_PIPELINE_B_NAME: result.pipeline_b.name,
|
||||
});
|
||||
result.wrote_env = true;
|
||||
}
|
||||
|
||||
result.status = "pass";
|
||||
result.reason = "Fake provider cross-pipeline fixtures are configured.";
|
||||
} catch (error) {
|
||||
result.status = looksLikeEnvIssue(error) ? "env_issue" : "fail";
|
||||
result.reason = safeReason(error.message);
|
||||
} finally {
|
||||
await writeResult(paths, result);
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
}
|
||||
|
||||
process.exit(result.status === "pass" ? 0 : result.status === "env_issue" ? 2 : 1);
|
||||
|
||||
function runPipelineSetup(pipelineName, label) {
|
||||
return new Promise((resolvePromise, rejectPromise) => {
|
||||
const child = spawn(process.execPath, ["scripts/e2e/ensure-fake-provider-pipeline.mjs"], {
|
||||
cwd: resolve("."),
|
||||
env: {
|
||||
...env,
|
||||
LANGBOT_FAKE_PROVIDER_PIPELINE_NAME: pipelineName,
|
||||
LANGBOT_FAKE_PROVIDER_FIRST_TOKEN_DELAY_MS: env.LANGBOT_FAKE_PROVIDER_FIRST_TOKEN_DELAY_MS || "25",
|
||||
LANGBOT_FAKE_PROVIDER_CHUNK_DELAY_MS: env.LANGBOT_FAKE_PROVIDER_CHUNK_DELAY_MS || "10",
|
||||
LANGBOT_FAKE_PROVIDER_CHUNK_COUNT: env.LANGBOT_FAKE_PROVIDER_CHUNK_COUNT || "0",
|
||||
LANGBOT_FAKE_PROVIDER_FAIL_FIRST_N: "0",
|
||||
LANGBOT_FAKE_PROVIDER_FAIL_EVERY_N: "0",
|
||||
LANGBOT_FAKE_PROVIDER_FAULT_STATUS: env.LANGBOT_FAKE_PROVIDER_FAULT_STATUS || "500",
|
||||
LANGBOT_FAKE_PROVIDER_FAIL_AFTER_FIRST_CHUNK: "false",
|
||||
LANGBOT_FAKE_PROVIDER_DYNAMIC_RESPONSE: "true",
|
||||
},
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
child.stdout.on("data", (chunk) => {
|
||||
const text = chunk.toString();
|
||||
stdout += text;
|
||||
appendLine(paths.consoleLog, `[setup ${label} stdout] ${text.trimEnd()}`).catch(() => {});
|
||||
});
|
||||
child.stderr.on("data", (chunk) => {
|
||||
const text = chunk.toString();
|
||||
stderr += text;
|
||||
appendLine(paths.consoleLog, `[setup ${label} stderr] ${text.trimEnd()}`).catch(() => {});
|
||||
});
|
||||
child.on("error", rejectPromise);
|
||||
child.on("close", (code) => {
|
||||
const parsed = parseJsonOutput(stdout);
|
||||
if (code !== 0 || parsed.status !== "pass") {
|
||||
rejectPromise(new Error(parsed.reason || stderr || `Fake provider pipeline setup ${label} exited with ${code}.`));
|
||||
return;
|
||||
}
|
||||
resolvePromise(parsed);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
function parseJsonOutput(text) {
|
||||
const trimmed = String(text || "").trim();
|
||||
if (!trimmed) return {};
|
||||
try {
|
||||
return JSON.parse(trimmed);
|
||||
} catch {
|
||||
const start = trimmed.indexOf("{");
|
||||
const end = trimmed.lastIndexOf("}");
|
||||
if (start >= 0 && end > start) {
|
||||
try {
|
||||
return JSON.parse(trimmed.slice(start, end + 1));
|
||||
} catch {
|
||||
return {};
|
||||
}
|
||||
}
|
||||
return {};
|
||||
}
|
||||
}
|
||||
|
||||
async function upsertEnvLocal(path, updates) {
|
||||
await mkdir(dirname(path), { recursive: true });
|
||||
let text = "";
|
||||
try {
|
||||
text = await readFile(path, "utf8");
|
||||
} catch {
|
||||
text = "";
|
||||
}
|
||||
const lines = text.split(/\r?\n/);
|
||||
const seen = new Set();
|
||||
const next = lines.map((line) => {
|
||||
const trimmed = line.trim();
|
||||
const match = trimmed.match(/^([A-Z][A-Z0-9_]*)=/);
|
||||
if (!match || updates[match[1]] === undefined) return line;
|
||||
seen.add(match[1]);
|
||||
return `${match[1]}=${updates[match[1]]}`;
|
||||
});
|
||||
for (const [key, value] of Object.entries(updates)) {
|
||||
if (!seen.has(key)) next.push(`${key}=${value}`);
|
||||
}
|
||||
await writeFile(path, `${next.join("\n").replace(/\n+$/, "")}\n`, "utf8");
|
||||
}
|
||||
|
||||
function looksLikeEnvIssue(error) {
|
||||
const message = String(error?.message || error || "");
|
||||
return /fetch failed|ECONNREFUSED|ENOTFOUND|LANGBOT_.*not configured|Could not read recovery_key|Backend did not respond/i.test(message);
|
||||
}
|
||||
|
||||
function safeReason(value) {
|
||||
return redact(String(value || "")).slice(0, 1000);
|
||||
}
|
||||
@@ -1,635 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { spawn } from "node:child_process";
|
||||
import { open, readFile, mkdir, writeFile } from "node:fs/promises";
|
||||
import { dirname, resolve } from "node:path";
|
||||
import { env } from "node:process";
|
||||
import {
|
||||
apiJson,
|
||||
ensureEvidence,
|
||||
evidencePaths,
|
||||
loadEnvFiles,
|
||||
redact,
|
||||
resetAndAuthLocalUser,
|
||||
writeResult,
|
||||
} from "./lib/langbot-e2e.mjs";
|
||||
|
||||
const RUNNER_ID = "local-agent";
|
||||
const DEFAULT_LOCAL_PASSWORD = "LangBotE2ELocalPass!2026";
|
||||
const DEFAULT_PIPELINE_NAME = "LangBot QA Fake Provider Debug Chat";
|
||||
const DEFAULT_PROVIDER_NAME = "LangBot QA Fake OpenAI Provider";
|
||||
const QA_RESOURCE_DESCRIPTION = "Managed by LangBot skills QA automation for controlled fake-provider Debug Chat tests. Safe to delete when local QA fixtures are no longer needed.";
|
||||
const DEFAULT_MODEL_NAME = "gpt-4o-mini";
|
||||
const DEFAULT_REQUESTER = "openai-chat-completions";
|
||||
|
||||
const caseId = "ensure-fake-provider-pipeline";
|
||||
|
||||
await loadEnvFiles();
|
||||
const paths = evidencePaths(caseId);
|
||||
await ensureEvidence(paths);
|
||||
|
||||
const writeEnv = process.argv.includes("--write-env");
|
||||
const frontendUrl = env.LANGBOT_FRONTEND_URL || "";
|
||||
const backendUrl = env.LANGBOT_BACKEND_URL || "";
|
||||
const envLocalPath = resolve("skills/.env.local");
|
||||
const repoRoot = resolve(env.LANGBOT_REPO || "..");
|
||||
const fakeStateDir = resolve(env.LANGBOT_FAKE_PROVIDER_STATE_DIR || resolve(repoRoot, ".qa/fake-provider"));
|
||||
const fakeStatePath = resolve(fakeStateDir, "state.json");
|
||||
const fakeStdoutPath = resolve(fakeStateDir, "fake-provider.stdout.log");
|
||||
const fakeStderrPath = resolve(fakeStateDir, "fake-provider.stderr.log");
|
||||
const pipelineName = env.LANGBOT_FAKE_PROVIDER_PIPELINE_NAME || DEFAULT_PIPELINE_NAME;
|
||||
const providerName = env.LANGBOT_FAKE_PROVIDER_NAME || DEFAULT_PROVIDER_NAME;
|
||||
const requester = env.LANGBOT_FAKE_PROVIDER_REQUESTER || DEFAULT_REQUESTER;
|
||||
const modelName = env.LANGBOT_FAKE_PROVIDER_MODEL_NAME || DEFAULT_MODEL_NAME;
|
||||
|
||||
const result = {
|
||||
source: "automation",
|
||||
case_id: caseId,
|
||||
run_id: paths.runId,
|
||||
status: "fail",
|
||||
reason: "",
|
||||
frontend_url: frontendUrl,
|
||||
backend_url: backendUrl,
|
||||
fake_provider: {
|
||||
url: "",
|
||||
base_url: "",
|
||||
pid: null,
|
||||
reused: false,
|
||||
config: {},
|
||||
state_file: fakeStatePath,
|
||||
stdout_log: fakeStdoutPath,
|
||||
stderr_log: fakeStderrPath,
|
||||
},
|
||||
provider: {
|
||||
uuid: "",
|
||||
name: providerName,
|
||||
requester,
|
||||
created: false,
|
||||
updated: false,
|
||||
},
|
||||
model: {
|
||||
uuid: "",
|
||||
name: modelName,
|
||||
created: false,
|
||||
updated: false,
|
||||
test_status: "not_run",
|
||||
test_reason: "",
|
||||
},
|
||||
pipeline_id: "",
|
||||
pipeline_name: pipelineName,
|
||||
pipeline_url: "",
|
||||
created: false,
|
||||
updated: false,
|
||||
wrote_env: false,
|
||||
evidence: {
|
||||
console_log: paths.consoleLog,
|
||||
network_log: paths.networkLog,
|
||||
automation_result_json: paths.automationResultJson,
|
||||
result_json: paths.resultJson,
|
||||
},
|
||||
evidence_collected: ["api_diagnostic", "network", "filesystem"],
|
||||
};
|
||||
|
||||
try {
|
||||
console.error(`[langbot-qa] configuring QA-owned fake-provider fixtures: provider=\"${providerName}\", pipeline=\"${pipelineName}\"`);
|
||||
console.error("[langbot-qa] this setup may create or update local QA provider/model/pipeline resources on the selected backend.");
|
||||
if (!backendUrl) {
|
||||
result.status = "env_issue";
|
||||
throw new Error("LANGBOT_BACKEND_URL is not configured.");
|
||||
}
|
||||
if (!frontendUrl) {
|
||||
result.status = "env_issue";
|
||||
throw new Error("LANGBOT_FRONTEND_URL is not configured.");
|
||||
}
|
||||
|
||||
const fakeProvider = await ensureFakeProvider();
|
||||
const setupConfig = await configureFakeProvider(fakeProvider.url, healthyFakeProviderConfig(), true);
|
||||
result.fake_provider = {
|
||||
...result.fake_provider,
|
||||
...fakeProvider,
|
||||
config: setupConfig.config || healthyFakeProviderConfig(),
|
||||
};
|
||||
|
||||
const user = env.LANGBOT_E2E_LOGIN_USER || "";
|
||||
const password = env.LANGBOT_E2E_LOGIN_PASSWORD || DEFAULT_LOCAL_PASSWORD;
|
||||
if (!user) {
|
||||
result.status = "env_issue";
|
||||
throw new Error("LANGBOT_E2E_LOGIN_USER is required so this setup can create/update the fake provider pipeline.");
|
||||
}
|
||||
|
||||
const auth = await resetAndAuthLocalUser({ backendUrl, user, password });
|
||||
const wizard = await skipWizard({ backendUrl, token: auth.token });
|
||||
if (wizard.status !== "pass") {
|
||||
result.status = "fail";
|
||||
throw new Error(wizard.reason || "Failed to mark the local QA wizard as skipped.");
|
||||
}
|
||||
|
||||
const provider = await ensureProvider({
|
||||
backendUrl,
|
||||
token: auth.token,
|
||||
name: providerName,
|
||||
requester,
|
||||
baseUrl: fakeProvider.base_url,
|
||||
});
|
||||
result.provider = provider;
|
||||
|
||||
const model = await ensureModel({
|
||||
backendUrl,
|
||||
token: auth.token,
|
||||
providerUuid: provider.uuid,
|
||||
name: modelName,
|
||||
});
|
||||
result.model = model;
|
||||
|
||||
const pipeline = await ensurePipeline({
|
||||
backendUrl,
|
||||
token: auth.token,
|
||||
name: pipelineName,
|
||||
modelUuid: model.uuid,
|
||||
});
|
||||
Object.assign(result, pipeline);
|
||||
result.pipeline_url = `${frontendUrl.replace(/\/$/, "")}/home/pipelines?id=${encodeURIComponent(pipeline.pipeline_id)}`;
|
||||
|
||||
const runConfig = await configureFakeProvider(fakeProvider.url, targetFakeProviderConfig(), true);
|
||||
result.fake_provider.config = runConfig.config || targetFakeProviderConfig();
|
||||
|
||||
if (writeEnv) {
|
||||
await upsertEnvLocal(envLocalPath, {
|
||||
LANGBOT_E2E_LOGIN_USER: user,
|
||||
LANGBOT_FAKE_PROVIDER_URL: fakeProvider.url,
|
||||
LANGBOT_FAKE_PROVIDER_BASE_URL: fakeProvider.base_url,
|
||||
LANGBOT_FAKE_PROVIDER_PID: fakeProvider.pid ? String(fakeProvider.pid) : "",
|
||||
LANGBOT_FAKE_PROVIDER_PROVIDER_UUID: provider.uuid,
|
||||
LANGBOT_FAKE_PROVIDER_MODEL_UUID: model.uuid,
|
||||
LANGBOT_FAKE_PROVIDER_PIPELINE_URL: result.pipeline_url,
|
||||
LANGBOT_FAKE_PROVIDER_PIPELINE_NAME: pipelineName,
|
||||
});
|
||||
result.wrote_env = true;
|
||||
}
|
||||
|
||||
result.status = "pass";
|
||||
result.reason = `Fake provider pipeline is configured with ${requester}/${modelName}.`;
|
||||
} catch (error) {
|
||||
result.status = result.status === "env_issue" ? "env_issue" : "fail";
|
||||
result.reason = result.reason || safeReason(error.message);
|
||||
} finally {
|
||||
await writeResult(paths, result);
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
}
|
||||
|
||||
process.exit(result.status === "pass" ? 0 : result.status === "env_issue" ? 2 : 1);
|
||||
|
||||
async function ensureFakeProvider() {
|
||||
const envUrl = normalizeProviderRootUrl(env.LANGBOT_FAKE_PROVIDER_URL || "");
|
||||
if (envUrl && await fakeProviderHealthy(envUrl) && await fakeProviderConfigurable(envUrl)) {
|
||||
return {
|
||||
url: envUrl,
|
||||
base_url: `${envUrl}/v1`,
|
||||
pid: null,
|
||||
reused: true,
|
||||
};
|
||||
}
|
||||
|
||||
const state = await readState(fakeStatePath);
|
||||
const stateUrl = normalizeProviderRootUrl(state.url || "");
|
||||
if (stateUrl && await fakeProviderHealthy(stateUrl)) {
|
||||
if (await fakeProviderConfigurable(stateUrl)) {
|
||||
return {
|
||||
url: stateUrl,
|
||||
base_url: state.base_url || `${stateUrl}/v1`,
|
||||
pid: Number.isInteger(state.pid) ? state.pid : null,
|
||||
reused: true,
|
||||
};
|
||||
}
|
||||
if (Number.isInteger(state.pid)) await stopProcess(state.pid);
|
||||
}
|
||||
|
||||
await mkdir(fakeStateDir, { recursive: true });
|
||||
await writeFile(fakeStatePath, `${JSON.stringify({ status: "starting", started_at: new Date().toISOString() }, null, 2)}\n`, "utf8");
|
||||
const stdout = await open(fakeStdoutPath, "a");
|
||||
const stderr = await open(fakeStderrPath, "a");
|
||||
const scriptPath = resolve("scripts/e2e/fake-openai-provider.mjs");
|
||||
const host = env.LANGBOT_FAKE_PROVIDER_HOST || "127.0.0.1";
|
||||
const port = env.LANGBOT_FAKE_PROVIDER_PORT || "0";
|
||||
const child = spawn(process.execPath, [
|
||||
scriptPath,
|
||||
`--host=${host}`,
|
||||
`--port=${port}`,
|
||||
`--state-file=${fakeStatePath}`,
|
||||
], {
|
||||
cwd: resolve("."),
|
||||
detached: true,
|
||||
env: {
|
||||
...env,
|
||||
LANGBOT_FAKE_PROVIDER_MODEL_NAME: modelName,
|
||||
},
|
||||
stdio: ["ignore", stdout.fd, stderr.fd],
|
||||
});
|
||||
child.unref();
|
||||
await stdout.close();
|
||||
await stderr.close();
|
||||
|
||||
const started = await waitForFakeProviderState(fakeStatePath, child.pid, 10_000);
|
||||
if (!started.url || !await fakeProviderHealthy(started.url) || !await fakeProviderConfigurable(started.url)) {
|
||||
throw new Error(`Fake provider did not become healthy. See ${fakeStderrPath}`);
|
||||
}
|
||||
|
||||
return {
|
||||
url: started.url,
|
||||
base_url: started.base_url || `${started.url}/v1`,
|
||||
pid: child.pid ?? started.pid ?? null,
|
||||
reused: false,
|
||||
};
|
||||
}
|
||||
|
||||
async function configureFakeProvider(rootUrl, config, resetRequestCount) {
|
||||
const response = await fetch(`${normalizeProviderRootUrl(rootUrl)}/__qa/config`, {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
config,
|
||||
reset_request_count: resetRequestCount,
|
||||
}),
|
||||
signal: AbortSignal.timeout(3000),
|
||||
});
|
||||
const json = await response.json().catch(() => ({}));
|
||||
if (!response.ok || json.ok !== true) {
|
||||
throw new Error(`Fake provider config failed with HTTP ${response.status}.`);
|
||||
}
|
||||
return json;
|
||||
}
|
||||
|
||||
async function fakeProviderHealthy(rootUrl) {
|
||||
try {
|
||||
const response = await fetch(`${rootUrl.replace(/\/$/, "")}/healthz`, {
|
||||
signal: AbortSignal.timeout(2000),
|
||||
});
|
||||
if (!response.ok) return false;
|
||||
const json = await response.json().catch(() => ({}));
|
||||
return json.ok === true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async function fakeProviderConfigurable(rootUrl) {
|
||||
try {
|
||||
const response = await fetch(`${rootUrl.replace(/\/$/, "")}/__qa/config`, {
|
||||
signal: AbortSignal.timeout(2000),
|
||||
});
|
||||
if (!response.ok) return false;
|
||||
const json = await response.json().catch(() => ({}));
|
||||
return json.ok === true && json.config && typeof json.config === "object";
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async function stopProcess(pid) {
|
||||
try {
|
||||
process.kill(pid, "SIGTERM");
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
await sleep(500);
|
||||
}
|
||||
|
||||
async function waitForFakeProviderState(path, expectedPid, timeoutMs) {
|
||||
const startedAt = Date.now();
|
||||
let lastState = {};
|
||||
while (Date.now() - startedAt < timeoutMs) {
|
||||
const state = await readState(path);
|
||||
if (state.url && (!expectedPid || state.pid === expectedPid)) return state;
|
||||
lastState = state;
|
||||
await sleep(150);
|
||||
}
|
||||
return lastState;
|
||||
}
|
||||
|
||||
async function readState(path) {
|
||||
try {
|
||||
return JSON.parse(await readFile(path, "utf8"));
|
||||
} catch {
|
||||
return {};
|
||||
}
|
||||
}
|
||||
|
||||
function normalizeProviderRootUrl(value) {
|
||||
const trimmed = String(value || "").trim().replace(/\/$/, "");
|
||||
return trimmed.endsWith("/v1") ? trimmed.slice(0, -3) : trimmed;
|
||||
}
|
||||
|
||||
function healthyFakeProviderConfig() {
|
||||
return {
|
||||
response_text: "OK",
|
||||
first_token_delay_ms: 25,
|
||||
chunk_delay_ms: 10,
|
||||
chunk_count: 0,
|
||||
fault_status: 500,
|
||||
fail_first_n: 0,
|
||||
fail_every_n: 0,
|
||||
fail_after_first_chunk: false,
|
||||
dynamic_response: true,
|
||||
};
|
||||
}
|
||||
|
||||
function targetFakeProviderConfig() {
|
||||
return {
|
||||
response_text: env.LANGBOT_FAKE_PROVIDER_RESPONSE_TEXT || "OK",
|
||||
first_token_delay_ms: nonNegativeInteger(env.LANGBOT_FAKE_PROVIDER_FIRST_TOKEN_DELAY_MS, 25),
|
||||
chunk_delay_ms: nonNegativeInteger(env.LANGBOT_FAKE_PROVIDER_CHUNK_DELAY_MS, 10),
|
||||
chunk_count: nonNegativeInteger(env.LANGBOT_FAKE_PROVIDER_CHUNK_COUNT, 0),
|
||||
fault_status: httpFaultStatus(env.LANGBOT_FAKE_PROVIDER_FAULT_STATUS, 500),
|
||||
fail_first_n: nonNegativeInteger(env.LANGBOT_FAKE_PROVIDER_FAIL_FIRST_N, 0),
|
||||
fail_every_n: nonNegativeInteger(env.LANGBOT_FAKE_PROVIDER_FAIL_EVERY_N, 0),
|
||||
fail_after_first_chunk: envBool(env.LANGBOT_FAKE_PROVIDER_FAIL_AFTER_FIRST_CHUNK, false),
|
||||
dynamic_response: envBool(env.LANGBOT_FAKE_PROVIDER_DYNAMIC_RESPONSE, true),
|
||||
};
|
||||
}
|
||||
|
||||
async function skipWizard({ backendUrl, token }) {
|
||||
const response = await apiJson(backendUrl, "/api/v1/system/wizard/completed", {
|
||||
method: "POST",
|
||||
token,
|
||||
body: { status: "skipped" },
|
||||
});
|
||||
const ok = response.status < 400 && response.json.code === 0;
|
||||
return {
|
||||
status: ok ? "pass" : "fail",
|
||||
http_status: response.status,
|
||||
code: response.json.code ?? null,
|
||||
reason: ok ? "Wizard marked skipped for local QA." : response.json.msg || "Wizard status update failed.",
|
||||
};
|
||||
}
|
||||
|
||||
async function ensureProvider({ backendUrl, token, name, requester, baseUrl }) {
|
||||
const list = await apiJson(backendUrl, "/api/v1/provider/providers", { token });
|
||||
if (isApiFailure(list)) {
|
||||
throw new Error(list.json.msg || "Failed to list providers.");
|
||||
}
|
||||
const providers = list.json.data?.providers || [];
|
||||
const existing = providers.find((provider) => (
|
||||
provider.name === name
|
||||
|| (provider.requester === requester && String(provider.base_url || "").replace(/\/$/, "") === baseUrl.replace(/\/$/, ""))
|
||||
));
|
||||
const body = {
|
||||
name,
|
||||
requester,
|
||||
base_url: baseUrl,
|
||||
api_keys: [env.LANGBOT_FAKE_PROVIDER_API_KEY || "langbot-fake-provider-key"],
|
||||
};
|
||||
|
||||
if (existing?.uuid) {
|
||||
const update = await apiJson(backendUrl, `/api/v1/provider/providers/${encodeURIComponent(existing.uuid)}`, {
|
||||
method: "PUT",
|
||||
token,
|
||||
body,
|
||||
});
|
||||
if (isApiFailure(update)) {
|
||||
throw new Error(update.json.msg || "Failed to update fake provider.");
|
||||
}
|
||||
return {
|
||||
uuid: existing.uuid,
|
||||
name,
|
||||
requester,
|
||||
created: false,
|
||||
updated: true,
|
||||
};
|
||||
}
|
||||
|
||||
const create = await apiJson(backendUrl, "/api/v1/provider/providers", {
|
||||
method: "POST",
|
||||
token,
|
||||
body,
|
||||
});
|
||||
const uuid = create.json.data?.uuid || "";
|
||||
if (isApiFailure(create) || !uuid) {
|
||||
throw new Error(create.json.msg || "Failed to create fake provider.");
|
||||
}
|
||||
return {
|
||||
uuid,
|
||||
name,
|
||||
requester,
|
||||
created: true,
|
||||
updated: false,
|
||||
};
|
||||
}
|
||||
|
||||
async function ensureModel({ backendUrl, token, providerUuid, name }) {
|
||||
const list = await apiJson(backendUrl, `/api/v1/provider/models/llm?provider_uuid=${encodeURIComponent(providerUuid)}`, { token });
|
||||
if (isApiFailure(list)) {
|
||||
throw new Error(list.json.msg || "Failed to list fake provider models.");
|
||||
}
|
||||
const models = list.json.data?.models || [];
|
||||
const existing = models.find((model) => model.name === name);
|
||||
const body = {
|
||||
name,
|
||||
provider_uuid: providerUuid,
|
||||
abilities: [],
|
||||
context_length: positiveInteger(env.LANGBOT_FAKE_PROVIDER_CONTEXT_LENGTH, 8192),
|
||||
extra_args: {},
|
||||
prefered_ranking: 0,
|
||||
};
|
||||
let modelUuid = existing?.uuid || "";
|
||||
let created = false;
|
||||
let updated = false;
|
||||
|
||||
if (modelUuid) {
|
||||
const update = await apiJson(backendUrl, `/api/v1/provider/models/llm/${encodeURIComponent(modelUuid)}`, {
|
||||
method: "PUT",
|
||||
token,
|
||||
body,
|
||||
});
|
||||
if (isApiFailure(update)) {
|
||||
throw new Error(update.json.msg || "Failed to update fake provider model.");
|
||||
}
|
||||
updated = true;
|
||||
} else {
|
||||
const create = await apiJson(backendUrl, "/api/v1/provider/models/llm", {
|
||||
method: "POST",
|
||||
token,
|
||||
body,
|
||||
});
|
||||
modelUuid = create.json.data?.uuid || "";
|
||||
if (isApiFailure(create) || !modelUuid) {
|
||||
throw new Error(create.json.msg || "Failed to create fake provider model.");
|
||||
}
|
||||
created = true;
|
||||
}
|
||||
|
||||
const test = await apiJson(backendUrl, `/api/v1/provider/models/llm/${encodeURIComponent(modelUuid)}/test`, {
|
||||
method: "POST",
|
||||
token,
|
||||
body: { extra_args: {} },
|
||||
});
|
||||
if (isApiFailure(test)) {
|
||||
throw new Error(safeReason(test.json.msg || test.json.message || "Fake provider model test failed."));
|
||||
}
|
||||
|
||||
return {
|
||||
uuid: modelUuid,
|
||||
name,
|
||||
created,
|
||||
updated,
|
||||
test_status: "pass",
|
||||
test_reason: "",
|
||||
};
|
||||
}
|
||||
|
||||
async function ensurePipeline({ backendUrl, token, name, modelUuid }) {
|
||||
const list = await apiJson(backendUrl, "/api/v1/pipelines", { token });
|
||||
if (isApiFailure(list)) {
|
||||
throw new Error(list.json.msg || "Failed to list pipelines.");
|
||||
}
|
||||
const pipelines = list.json.data?.pipelines || [];
|
||||
let pipeline = pipelines.find((item) => item.name === name) || null;
|
||||
let created = false;
|
||||
|
||||
if (!pipeline) {
|
||||
const create = await apiJson(backendUrl, "/api/v1/pipelines", {
|
||||
method: "POST",
|
||||
token,
|
||||
body: {
|
||||
name,
|
||||
description: QA_RESOURCE_DESCRIPTION,
|
||||
emoji: "QA",
|
||||
},
|
||||
});
|
||||
const pipelineId = create.json.data?.uuid || "";
|
||||
if (isApiFailure(create) || !pipelineId) {
|
||||
throw new Error(create.json.msg || "Failed to create fake provider pipeline.");
|
||||
}
|
||||
created = true;
|
||||
pipeline = { uuid: pipelineId };
|
||||
}
|
||||
|
||||
const loaded = await apiJson(backendUrl, `/api/v1/pipelines/${encodeURIComponent(pipeline.uuid)}`, { token });
|
||||
pipeline = loaded.json.data?.pipeline || null;
|
||||
if (isApiFailure(loaded) || !pipeline?.uuid) {
|
||||
throw new Error(loaded.json.msg || "Failed to load fake provider pipeline.");
|
||||
}
|
||||
|
||||
const config = pipeline.config && typeof pipeline.config === "object" ? pipeline.config : {};
|
||||
const ai = config.ai && typeof config.ai === "object" ? config.ai : {};
|
||||
const existingLocalAgentConfig = ai["local-agent"] && typeof ai["local-agent"] === "object"
|
||||
? ai["local-agent"]
|
||||
: {};
|
||||
const localAgentConfig = {
|
||||
timeout: 60,
|
||||
prompt: [{ role: "system", content: "You are a deterministic QA assistant. Reply exactly as instructed." }],
|
||||
"remove-think": false,
|
||||
"knowledge-bases": [],
|
||||
"box-session-id-template": "{launcher_type}_{launcher_id}",
|
||||
"retrieval-top-k": 5,
|
||||
"rerank-model": "",
|
||||
"rerank-top-k": 5,
|
||||
"max-tool-iterations": 20,
|
||||
"tool-execution-mode": "parallel",
|
||||
"max-tool-result-chars": 20000,
|
||||
"context-history-fetch-limit": 20,
|
||||
"context-window-tokens": 8192,
|
||||
"context-reserve-tokens": 1024,
|
||||
"context-keep-recent-tokens": 2048,
|
||||
"context-summary-tokens": 1024,
|
||||
...existingLocalAgentConfig,
|
||||
// Current backend truncation still reads this field directly.
|
||||
"max-round": positiveInteger(existingLocalAgentConfig["max-round"], 10),
|
||||
model: {
|
||||
primary: modelUuid,
|
||||
fallbacks: [],
|
||||
},
|
||||
};
|
||||
const updatedConfig = {
|
||||
...config,
|
||||
ai: {
|
||||
...ai,
|
||||
runner: {
|
||||
...(ai.runner && typeof ai.runner === "object" ? ai.runner : {}),
|
||||
id: RUNNER_ID,
|
||||
runner: RUNNER_ID,
|
||||
"expire-time": 0,
|
||||
},
|
||||
"local-agent": localAgentConfig,
|
||||
},
|
||||
};
|
||||
|
||||
const update = await apiJson(backendUrl, `/api/v1/pipelines/${encodeURIComponent(pipeline.uuid)}`, {
|
||||
method: "PUT",
|
||||
token,
|
||||
body: {
|
||||
name,
|
||||
description: QA_RESOURCE_DESCRIPTION,
|
||||
emoji: "QA",
|
||||
config: updatedConfig,
|
||||
},
|
||||
});
|
||||
if (isApiFailure(update)) {
|
||||
throw new Error(update.json.msg || "Failed to update fake provider pipeline.");
|
||||
}
|
||||
|
||||
return {
|
||||
pipeline_id: pipeline.uuid,
|
||||
pipeline_name: name,
|
||||
created,
|
||||
updated: true,
|
||||
};
|
||||
}
|
||||
|
||||
function isApiFailure(response) {
|
||||
return response.status >= 400 || (response.json.code !== undefined && response.json.code !== 0);
|
||||
}
|
||||
|
||||
function positiveInteger(value, fallback) {
|
||||
const parsed = Number(value);
|
||||
return Number.isInteger(parsed) && parsed > 0 ? parsed : fallback;
|
||||
}
|
||||
|
||||
function nonNegativeInteger(value, fallback) {
|
||||
const parsed = Number(value);
|
||||
return Number.isInteger(parsed) && parsed >= 0 ? parsed : fallback;
|
||||
}
|
||||
|
||||
function httpFaultStatus(value, fallback) {
|
||||
const parsed = Number(value);
|
||||
return Number.isInteger(parsed) && parsed >= 400 && parsed <= 599 ? parsed : fallback;
|
||||
}
|
||||
|
||||
function envBool(value, fallback) {
|
||||
if (value === undefined || value === "") return fallback;
|
||||
if (/^(1|true|yes|on)$/i.test(String(value))) return true;
|
||||
if (/^(0|false|no|off)$/i.test(String(value))) return false;
|
||||
return fallback;
|
||||
}
|
||||
|
||||
function sleep(ms) {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
function safeReason(value) {
|
||||
return redact(String(value || "")).slice(0, 1000);
|
||||
}
|
||||
|
||||
async function upsertEnvLocal(path, updates) {
|
||||
await mkdir(dirname(path), { recursive: true });
|
||||
let text = "";
|
||||
try {
|
||||
text = await readFile(path, "utf8");
|
||||
} catch {
|
||||
text = "";
|
||||
}
|
||||
const lines = text.split(/\r?\n/);
|
||||
const seen = new Set();
|
||||
const next = lines.map((line) => {
|
||||
const trimmed = line.trim();
|
||||
const equals = trimmed.indexOf("=");
|
||||
if (equals <= 0 || trimmed.startsWith("#")) return line;
|
||||
const key = trimmed.slice(0, equals).trim();
|
||||
if (!(key in updates)) return line;
|
||||
seen.add(key);
|
||||
return `${key}=${updates[key]}`;
|
||||
});
|
||||
for (const [key, value] of Object.entries(updates)) {
|
||||
if (!seen.has(key)) next.push(`${key}=${value}`);
|
||||
}
|
||||
await writeFile(path, `${next.filter((line, index) => line !== "" || index < next.length - 1).join("\n")}\n`, "utf8");
|
||||
}
|
||||
Executable → Regular
Executable → Regular
+14
-311
@@ -10,7 +10,6 @@ import {
|
||||
ensureEvidence,
|
||||
evidencePaths,
|
||||
loadEnvFiles,
|
||||
redact,
|
||||
resetAndAuthLocalUser,
|
||||
safeScreenshot,
|
||||
setBrowserToken,
|
||||
@@ -18,12 +17,9 @@ import {
|
||||
writeResult,
|
||||
} from "./lib/langbot-e2e.mjs";
|
||||
|
||||
const RUNNER_ID = "local-agent";
|
||||
const SPACE_PROVIDER_UUID = "00000000-0000-0000-0000-000000000000";
|
||||
const RUNNER_ID = "plugin:langbot/local-agent/default";
|
||||
const DEFAULT_PIPELINE_NAME = "Agent QA Local Agent Debug Chat";
|
||||
const DEFAULT_LOCAL_PASSWORD = "LangBotE2ELocalPass!2026";
|
||||
const DEFAULT_MODEL_TEST_LIMIT = 8;
|
||||
const DEFAULT_MODEL_FALLBACK_COUNT = 3;
|
||||
const caseId = "ensure-local-agent-pipeline";
|
||||
|
||||
await loadEnvFiles();
|
||||
@@ -49,18 +45,11 @@ const result = {
|
||||
pipeline_url: "",
|
||||
runner_id: RUNNER_ID,
|
||||
selected_model_id: "",
|
||||
selected_model_name: "",
|
||||
fallback_model_ids: [],
|
||||
model_count: 0,
|
||||
space_model_count: 0,
|
||||
scanned_space_model_count: 0,
|
||||
tested_model_count: 0,
|
||||
model_tests: [],
|
||||
created: false,
|
||||
updated: false,
|
||||
wrote_env: false,
|
||||
auth: null,
|
||||
wizard: null,
|
||||
browser_token_check: null,
|
||||
page_signal: "",
|
||||
evidence: {
|
||||
@@ -82,7 +71,6 @@ try {
|
||||
const user = env.LANGBOT_E2E_LOGIN_USER || "";
|
||||
const password = env.LANGBOT_E2E_LOGIN_PASSWORD || DEFAULT_LOCAL_PASSWORD;
|
||||
if (!user) {
|
||||
result.status = "env_issue";
|
||||
throw new Error("LANGBOT_E2E_LOGIN_USER is required so this setup can create/update the pipeline via backend API.");
|
||||
}
|
||||
|
||||
@@ -93,13 +81,6 @@ try {
|
||||
backend_token_check: auth.check,
|
||||
};
|
||||
|
||||
const wizard = await skipWizard({ backendUrl, token: auth.token });
|
||||
result.wizard = wizard;
|
||||
if (wizard.status !== "pass") {
|
||||
result.status = "fail";
|
||||
throw new Error(wizard.reason || "Failed to mark the local QA wizard as skipped.");
|
||||
}
|
||||
|
||||
const prepared = await ensureLocalAgentPipeline({
|
||||
backendUrl,
|
||||
token: auth.token,
|
||||
@@ -118,10 +99,6 @@ try {
|
||||
LANGBOT_PIPELINE_NAME: result.pipeline_name || pipelineName,
|
||||
LANGBOT_LOCAL_AGENT_PIPELINE_URL: result.pipeline_url,
|
||||
LANGBOT_LOCAL_AGENT_PIPELINE_NAME: result.pipeline_name || pipelineName,
|
||||
...(result.selected_model_id ? {
|
||||
LANGBOT_LOCAL_AGENT_MODEL_UUID: result.selected_model_id,
|
||||
LANGBOT_E2E_MODEL_UUID: result.selected_model_id,
|
||||
} : {}),
|
||||
});
|
||||
result.wrote_env = true;
|
||||
}
|
||||
@@ -150,21 +127,6 @@ try {
|
||||
|
||||
process.exit(result.status === "pass" ? 0 : result.status === "env_issue" ? 2 : 1);
|
||||
|
||||
async function skipWizard({ backendUrl, token }) {
|
||||
const response = await apiJson(backendUrl, "/api/v1/system/wizard/completed", {
|
||||
method: "POST",
|
||||
token,
|
||||
body: { status: "skipped" },
|
||||
});
|
||||
const ok = response.status < 400 && response.json.code === 0;
|
||||
return {
|
||||
status: ok ? "pass" : "fail",
|
||||
http_status: response.status,
|
||||
code: response.json.code ?? null,
|
||||
reason: ok ? "Wizard marked skipped for local QA." : response.json.msg || "Wizard status update failed.",
|
||||
};
|
||||
}
|
||||
|
||||
async function ensureLocalAgentPipeline({ backendUrl, token, pipelineName, runnerId }) {
|
||||
const [pipelineList, modelList] = await Promise.all([
|
||||
apiJson(backendUrl, "/api/v1/pipelines", { token }),
|
||||
@@ -187,19 +149,7 @@ async function ensureLocalAgentPipeline({ backendUrl, token, pipelineName, runne
|
||||
}
|
||||
|
||||
const models = modelList.json.data?.models || [];
|
||||
const skippedModelIds = new Set(
|
||||
String(env.LANGBOT_E2E_SKIP_MODEL_UUIDS || "")
|
||||
.split(",")
|
||||
.map((item) => item.trim())
|
||||
.filter(Boolean),
|
||||
);
|
||||
const skippedModelNames = new Set(
|
||||
String(env.LANGBOT_E2E_SKIP_MODEL_NAMES || "")
|
||||
.split(",")
|
||||
.map((item) => item.trim())
|
||||
.filter(Boolean),
|
||||
);
|
||||
const spaceModels = models.filter((model) => isSpaceModel(model) && !skippedModelIds.has(model.uuid));
|
||||
const selectedModel = models.find((model) => model.uuid) || null;
|
||||
const pipelines = pipelineList.json.data?.pipelines || [];
|
||||
let pipeline = pipelines.find((item) => item.name === pipelineName) || null;
|
||||
let created = false;
|
||||
@@ -220,7 +170,6 @@ async function ensureLocalAgentPipeline({ backendUrl, token, pipelineName, runne
|
||||
reason: createdResponse.json.msg || "Failed to create pipeline.",
|
||||
create_status: createdResponse.status,
|
||||
model_count: models.length,
|
||||
space_model_count: spaceModels.length,
|
||||
};
|
||||
}
|
||||
const pipelineId = createdResponse.json.data?.uuid || "";
|
||||
@@ -234,7 +183,6 @@ async function ensureLocalAgentPipeline({ backendUrl, token, pipelineName, runne
|
||||
status: "fail",
|
||||
reason: "Pipeline was not created or resolved.",
|
||||
model_count: models.length,
|
||||
space_model_count: spaceModels.length,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -246,37 +194,27 @@ async function ensureLocalAgentPipeline({ backendUrl, token, pipelineName, runne
|
||||
get_status: loaded.status,
|
||||
pipeline_id: pipeline.uuid,
|
||||
model_count: models.length,
|
||||
space_model_count: spaceModels.length,
|
||||
};
|
||||
}
|
||||
pipeline = loaded.json.data.pipeline;
|
||||
|
||||
const config = pipeline.config && typeof pipeline.config === "object" ? pipeline.config : {};
|
||||
const ai = config.ai && typeof config.ai === "object" ? config.ai : {};
|
||||
const rawExistingLocalAgentConfig = ai["local-agent"] && typeof ai["local-agent"] === "object"
|
||||
? ai["local-agent"]
|
||||
const runnerConfig = ai.runner_config && typeof ai.runner_config === "object" ? ai.runner_config : {};
|
||||
const rawExistingLocalAgentConfig = runnerConfig[runnerId] && typeof runnerConfig[runnerId] === "object"
|
||||
? runnerConfig[runnerId]
|
||||
: {};
|
||||
const existingLocalAgentConfig = rawExistingLocalAgentConfig;
|
||||
const existingModel = existingLocalAgentConfig.model && typeof existingLocalAgentConfig.model === "object"
|
||||
? existingLocalAgentConfig.model
|
||||
: {};
|
||||
const requestedModelId = env.LANGBOT_LOCAL_AGENT_MODEL_UUID || env.LANGBOT_E2E_MODEL_UUID || "";
|
||||
const selected = await selectWorkingSpaceModel({
|
||||
backendUrl,
|
||||
token,
|
||||
models,
|
||||
skippedModelIds,
|
||||
skippedModelNames,
|
||||
requestedModelId,
|
||||
existingModelId: existingModel.primary || "",
|
||||
});
|
||||
const selectedModelId = selected.selected_model_id || "";
|
||||
const selectedModelId = requestedModelId || existingModel.primary || selectedModel?.uuid || "";
|
||||
const localAgentConfig = {
|
||||
timeout: 300,
|
||||
prompt: [{ role: "system", content: "You are a helpful assistant." }],
|
||||
"remove-think": false,
|
||||
"knowledge-bases": [],
|
||||
"box-session-id-template": "{launcher_type}_{launcher_id}",
|
||||
"retrieval-top-k": 5,
|
||||
"rerank-model": "",
|
||||
"rerank-top-k": 5,
|
||||
@@ -289,11 +227,9 @@ async function ensureLocalAgentPipeline({ backendUrl, token, pipelineName, runne
|
||||
"context-keep-recent-tokens": 20000,
|
||||
"context-summary-tokens": 8000,
|
||||
...existingLocalAgentConfig,
|
||||
// Current backend truncation still reads this field directly.
|
||||
"max-round": positiveInteger(existingLocalAgentConfig["max-round"], 10),
|
||||
model: {
|
||||
primary: selectedModelId,
|
||||
fallbacks: selected.fallback_model_ids || [],
|
||||
fallbacks: requestedModelId ? [] : Array.isArray(existingModel.fallbacks) ? existingModel.fallbacks : [],
|
||||
},
|
||||
};
|
||||
const updatedConfig = {
|
||||
@@ -303,10 +239,12 @@ async function ensureLocalAgentPipeline({ backendUrl, token, pipelineName, runne
|
||||
runner: {
|
||||
...(ai.runner && typeof ai.runner === "object" ? ai.runner : {}),
|
||||
id: runnerId,
|
||||
runner: runnerId,
|
||||
"expire-time": 0,
|
||||
},
|
||||
"local-agent": localAgentConfig,
|
||||
runner_config: {
|
||||
...runnerConfig,
|
||||
[runnerId]: localAgentConfig,
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
@@ -327,31 +265,19 @@ async function ensureLocalAgentPipeline({ backendUrl, token, pipelineName, runne
|
||||
update_status: updateResponse.status,
|
||||
pipeline_id: pipeline.uuid,
|
||||
model_count: models.length,
|
||||
space_model_count: spaceModels.length,
|
||||
scanned_space_model_count: selected.scanned_space_model_count,
|
||||
tested_model_count: selected.tested_model_count,
|
||||
model_tests: selected.model_tests,
|
||||
selected_model_id: selectedModelId,
|
||||
selected_model_name: selected.selected_model_name,
|
||||
fallback_model_ids: selected.fallback_model_ids,
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
status: selectedModelId ? "pass" : "env_issue",
|
||||
reason: selectedModelId
|
||||
? `Local-agent pipeline is configured for Debug Chat with Space model ${selected.selected_model_name || selectedModelId} and ${selected.fallback_model_ids.length} fallback(s).`
|
||||
: selected.reason || "No working Space LLM model is configured in this LangBot instance.",
|
||||
? "Local-agent pipeline is configured for Debug Chat."
|
||||
: "Pipeline was created but no LLM model is configured in this LangBot instance.",
|
||||
pipeline_id: pipeline.uuid,
|
||||
pipeline_name: pipelineName,
|
||||
pipeline_name: pipeline.name,
|
||||
model_count: models.length,
|
||||
space_model_count: spaceModels.length,
|
||||
scanned_space_model_count: selected.scanned_space_model_count,
|
||||
tested_model_count: selected.tested_model_count,
|
||||
model_tests: selected.model_tests,
|
||||
selected_model_id: selectedModelId,
|
||||
selected_model_name: selected.selected_model_name,
|
||||
fallback_model_ids: selected.fallback_model_ids,
|
||||
created,
|
||||
updated: true,
|
||||
};
|
||||
@@ -361,229 +287,6 @@ function isApiFailure(response) {
|
||||
return response.status >= 400 || (response.json.code !== undefined && response.json.code !== 0);
|
||||
}
|
||||
|
||||
function isSpaceModel(model) {
|
||||
const provider = model?.provider && typeof model.provider === "object" ? model.provider : {};
|
||||
return model?.provider_uuid === SPACE_PROVIDER_UUID
|
||||
|| provider.uuid === SPACE_PROVIDER_UUID
|
||||
|| provider.requester === "space-chat-completions"
|
||||
|| provider.name === "LangBot Models";
|
||||
}
|
||||
|
||||
async function selectWorkingSpaceModel({
|
||||
backendUrl,
|
||||
token,
|
||||
models,
|
||||
skippedModelIds,
|
||||
skippedModelNames,
|
||||
requestedModelId,
|
||||
existingModelId,
|
||||
}) {
|
||||
const modelTests = [];
|
||||
const testLimit = positiveInteger(env.LANGBOT_E2E_MODEL_TEST_LIMIT, DEFAULT_MODEL_TEST_LIMIT);
|
||||
const fallbackCount = positiveInteger(env.LANGBOT_E2E_MODEL_FALLBACK_COUNT, DEFAULT_MODEL_FALLBACK_COUNT);
|
||||
const workingModels = [];
|
||||
const spaceModels = rankModels(models.filter((model) => (
|
||||
model.uuid
|
||||
&& isSpaceModel(model)
|
||||
&& !skippedModelIds.has(model.uuid)
|
||||
&& !skippedModelNames.has(model.name)
|
||||
)));
|
||||
const requestedModel = requestedModelId
|
||||
? spaceModels.find((model) => model.uuid === requestedModelId) || null
|
||||
: null;
|
||||
const existingModel = existingModelId
|
||||
? spaceModels.find((model) => model.uuid === existingModelId) || null
|
||||
: null;
|
||||
const candidates = uniqueCandidates([
|
||||
...(requestedModel ? [existingCandidate(requestedModel, "requested")] : []),
|
||||
...(existingModel ? [existingCandidate(existingModel, "existing-pipeline")] : []),
|
||||
...spaceModels.map((model) => existingCandidate(model, "configured-space")),
|
||||
]);
|
||||
|
||||
let scanResult = { status: "skipped", models: [], reason: "" };
|
||||
if (env.LANGBOT_E2E_SCAN_SPACE_MODELS !== "false") {
|
||||
scanResult = await scanSpaceModels({ backendUrl, token });
|
||||
if (scanResult.status === "pass") {
|
||||
const knownNames = new Set(spaceModels.map((model) => model.name));
|
||||
candidates.push(...scanResult.models
|
||||
.filter((model) => model.name && !knownNames.has(model.name) && !skippedModelNames.has(model.name))
|
||||
.map((model) => scannedCandidate(model)));
|
||||
}
|
||||
}
|
||||
|
||||
const unique = uniqueCandidates(candidates);
|
||||
for (const candidate of unique.slice(0, testLimit)) {
|
||||
const test = await ensureAndTestModel({ backendUrl, token, candidate });
|
||||
modelTests.push(test);
|
||||
if (test.status === "pass" && test.model_uuid) {
|
||||
workingModels.push(test);
|
||||
if (workingModels.length >= fallbackCount + 1) break;
|
||||
}
|
||||
}
|
||||
|
||||
if (workingModels.length > 0) {
|
||||
const [primary, ...fallbacks] = workingModels;
|
||||
return {
|
||||
status: "pass",
|
||||
reason: "",
|
||||
selected_model_id: primary.model_uuid,
|
||||
selected_model_name: primary.model_name,
|
||||
fallback_model_ids: fallbacks.map((model) => model.model_uuid),
|
||||
scanned_space_model_count: scanResult.models.length,
|
||||
tested_model_count: modelTests.length,
|
||||
model_tests: modelTests,
|
||||
};
|
||||
}
|
||||
|
||||
const baseReason = unique.length === 0
|
||||
? scanResult.reason || "No Space LLM model candidates are available."
|
||||
: `No working Space LLM model found after testing ${modelTests.length} candidate(s).`;
|
||||
return {
|
||||
status: "env_issue",
|
||||
reason: requestedModelId && !requestedModel
|
||||
? `Requested Space LLM model ${requestedModelId} is missing or skipped; ${baseReason}`
|
||||
: baseReason,
|
||||
selected_model_id: "",
|
||||
selected_model_name: "",
|
||||
fallback_model_ids: [],
|
||||
scanned_space_model_count: scanResult.models.length,
|
||||
tested_model_count: modelTests.length,
|
||||
model_tests: modelTests,
|
||||
};
|
||||
}
|
||||
|
||||
async function scanSpaceModels({ backendUrl, token }) {
|
||||
const response = await apiJson(
|
||||
backendUrl,
|
||||
`/api/v1/provider/providers/${encodeURIComponent(SPACE_PROVIDER_UUID)}/scan-models?type=llm`,
|
||||
{ token },
|
||||
);
|
||||
if (isApiFailure(response)) {
|
||||
return {
|
||||
status: "env_issue",
|
||||
models: [],
|
||||
reason: safeReason(response.json.msg || response.json.message || "Failed to scan Space LLM models."),
|
||||
};
|
||||
}
|
||||
return {
|
||||
status: "pass",
|
||||
models: response.json.data?.models || [],
|
||||
reason: "",
|
||||
};
|
||||
}
|
||||
|
||||
async function ensureAndTestModel({ backendUrl, token, candidate }) {
|
||||
let modelUuid = candidate.uuid || "";
|
||||
let created = false;
|
||||
if (!modelUuid) {
|
||||
const create = await apiJson(backendUrl, "/api/v1/provider/models/llm", {
|
||||
method: "POST",
|
||||
token,
|
||||
body: {
|
||||
name: candidate.name,
|
||||
provider_uuid: SPACE_PROVIDER_UUID,
|
||||
abilities: candidate.abilities || [],
|
||||
context_length: candidate.context_length ?? null,
|
||||
extra_args: {},
|
||||
prefered_ranking: positiveInteger(candidate.prefered_ranking, 0),
|
||||
},
|
||||
});
|
||||
modelUuid = create.json.data?.uuid || "";
|
||||
if (isApiFailure(create) || !modelUuid) {
|
||||
return modelTestResult(candidate, {
|
||||
status: "fail",
|
||||
reason: safeReason(create.json.msg || "Failed to create scanned Space model."),
|
||||
http_status: create.status,
|
||||
});
|
||||
}
|
||||
created = true;
|
||||
}
|
||||
|
||||
const test = await apiJson(backendUrl, `/api/v1/provider/models/llm/${encodeURIComponent(modelUuid)}/test`, {
|
||||
method: "POST",
|
||||
token,
|
||||
body: { extra_args: {} },
|
||||
});
|
||||
const passed = !isApiFailure(test);
|
||||
if (!passed && created) {
|
||||
await apiJson(backendUrl, `/api/v1/provider/models/llm/${encodeURIComponent(modelUuid)}`, {
|
||||
method: "DELETE",
|
||||
token,
|
||||
}).catch(() => {});
|
||||
}
|
||||
return modelTestResult(candidate, {
|
||||
status: passed ? "pass" : "fail",
|
||||
reason: passed ? "" : safeReason(test.json.msg || test.json.message || "Space model test failed."),
|
||||
http_status: test.status,
|
||||
model_uuid: modelUuid,
|
||||
created,
|
||||
});
|
||||
}
|
||||
|
||||
function modelTestResult(candidate, details) {
|
||||
return {
|
||||
source: candidate.source,
|
||||
model_uuid: details.model_uuid || candidate.uuid || "",
|
||||
model_name: candidate.name,
|
||||
status: details.status,
|
||||
reason: details.reason || "",
|
||||
http_status: details.http_status ?? null,
|
||||
created: Boolean(details.created),
|
||||
};
|
||||
}
|
||||
|
||||
function existingCandidate(model, source) {
|
||||
return {
|
||||
source,
|
||||
uuid: model.uuid,
|
||||
name: model.name,
|
||||
abilities: model.abilities || [],
|
||||
context_length: model.context_length,
|
||||
prefered_ranking: model.prefered_ranking,
|
||||
};
|
||||
}
|
||||
|
||||
function scannedCandidate(model) {
|
||||
return {
|
||||
source: "scanned-space",
|
||||
uuid: "",
|
||||
name: model.name || model.id,
|
||||
abilities: model.abilities || [],
|
||||
context_length: model.context_length,
|
||||
prefered_ranking: model.prefered_ranking,
|
||||
};
|
||||
}
|
||||
|
||||
function uniqueCandidates(candidates) {
|
||||
const seen = new Set();
|
||||
const result = [];
|
||||
for (const candidate of candidates) {
|
||||
const key = candidate.uuid ? `uuid:${candidate.uuid}` : `name:${candidate.name}`;
|
||||
if (!candidate.name || seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
result.push(candidate);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
function rankModels(models) {
|
||||
return [...models].sort((left, right) => {
|
||||
const leftRank = Number.isFinite(Number(left.prefered_ranking)) ? Number(left.prefered_ranking) : 9999;
|
||||
const rightRank = Number.isFinite(Number(right.prefered_ranking)) ? Number(right.prefered_ranking) : 9999;
|
||||
if (leftRank !== rightRank) return leftRank - rightRank;
|
||||
return String(left.name || "").localeCompare(String(right.name || ""));
|
||||
});
|
||||
}
|
||||
|
||||
function positiveInteger(value, fallback) {
|
||||
const parsed = Number(value);
|
||||
return Number.isInteger(parsed) && parsed > 0 ? parsed : fallback;
|
||||
}
|
||||
|
||||
function safeReason(value) {
|
||||
return redact(String(value || "")).slice(0, 1000);
|
||||
}
|
||||
|
||||
async function upsertEnvLocal(path, updates) {
|
||||
let text = "";
|
||||
try {
|
||||
|
||||
Executable → Regular
@@ -1,496 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { createServer } from "node:http";
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
import { dirname, resolve } from "node:path";
|
||||
import { env, exit } from "node:process";
|
||||
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
const host = args.host || env.LANGBOT_FAKE_PROVIDER_HOST || "127.0.0.1";
|
||||
const port = integer(args.port ?? env.LANGBOT_FAKE_PROVIDER_PORT, 0);
|
||||
const stateFile = args["state-file"] || env.LANGBOT_FAKE_PROVIDER_STATE_FILE || "";
|
||||
const modelName = env.LANGBOT_FAKE_PROVIDER_MODEL_NAME || "gpt-4o-mini";
|
||||
const config = {
|
||||
response_text: env.LANGBOT_FAKE_PROVIDER_RESPONSE_TEXT || "OK",
|
||||
first_token_delay_ms: integer(env.LANGBOT_FAKE_PROVIDER_FIRST_TOKEN_DELAY_MS, 25),
|
||||
chunk_delay_ms: integer(env.LANGBOT_FAKE_PROVIDER_CHUNK_DELAY_MS, 10),
|
||||
chunk_count: integer(env.LANGBOT_FAKE_PROVIDER_CHUNK_COUNT, 0),
|
||||
fault_status: integer(env.LANGBOT_FAKE_PROVIDER_FAULT_STATUS, 500),
|
||||
fail_first_n: integer(env.LANGBOT_FAKE_PROVIDER_FAIL_FIRST_N, 0),
|
||||
fail_every_n: integer(env.LANGBOT_FAKE_PROVIDER_FAIL_EVERY_N, 0),
|
||||
fail_after_first_chunk: bool(env.LANGBOT_FAKE_PROVIDER_FAIL_AFTER_FIRST_CHUNK, false),
|
||||
dynamic_response: !/^(0|false|no|off)$/i.test(env.LANGBOT_FAKE_PROVIDER_DYNAMIC_RESPONSE || ""),
|
||||
request_log_limit: integer(env.LANGBOT_FAKE_PROVIDER_REQUEST_LOG_LIMIT, 500),
|
||||
};
|
||||
|
||||
let requestCount = 0;
|
||||
const recentRequests = [];
|
||||
|
||||
const server = createServer(async (request, response) => {
|
||||
const startedAt = Date.now();
|
||||
const startedPerf = performance.now();
|
||||
let requestRecord = null;
|
||||
const url = new URL(request.url || "/", `http://${request.headers.host || `${host}:${port}`}`);
|
||||
try {
|
||||
if (request.method === "GET" && url.pathname === "/healthz") {
|
||||
sendJson(response, 200, {
|
||||
ok: true,
|
||||
model: modelName,
|
||||
config,
|
||||
request_count: requestCount,
|
||||
recent_request_count: recentRequests.length,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (request.method === "GET" && url.pathname === "/__qa/config") {
|
||||
sendJson(response, 200, {
|
||||
ok: true,
|
||||
model: modelName,
|
||||
config,
|
||||
request_count: requestCount,
|
||||
recent_requests: recentRequests,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (request.method === "POST" && url.pathname === "/__qa/config") {
|
||||
const body = await readJson(request);
|
||||
applyConfig(body.config && typeof body.config === "object" ? body.config : body);
|
||||
if (body.reset_request_count !== false) resetRequestState();
|
||||
sendJson(response, 200, {
|
||||
ok: true,
|
||||
model: modelName,
|
||||
config,
|
||||
request_count: requestCount,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (request.method === "POST" && url.pathname === "/__qa/reset") {
|
||||
resetRequestState();
|
||||
sendJson(response, 200, {
|
||||
ok: true,
|
||||
model: modelName,
|
||||
config,
|
||||
request_count: requestCount,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (request.method === "GET" && ["/models", "/v1/models"].includes(url.pathname)) {
|
||||
sendJson(response, 200, {
|
||||
object: "list",
|
||||
data: [
|
||||
{
|
||||
id: modelName,
|
||||
object: "model",
|
||||
created: 1,
|
||||
owned_by: "langbot-qa",
|
||||
type: "llm",
|
||||
},
|
||||
],
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (request.method === "POST" && ["/chat/completions", "/v1/chat/completions"].includes(url.pathname)) {
|
||||
requestCount += 1;
|
||||
const body = await readJson(request);
|
||||
const requestId = `chatcmpl-langbot-fake-${requestCount}`;
|
||||
const shouldFail = requestCount <= config.fail_first_n
|
||||
|| (config.fail_every_n > 0 && requestCount % config.fail_every_n === 0);
|
||||
const replyText = responseTextForBody(body);
|
||||
requestRecord = recordRequest({
|
||||
id: requestId,
|
||||
request_number: requestCount,
|
||||
path: url.pathname,
|
||||
stream: Boolean(body.stream),
|
||||
model: body.model || "",
|
||||
message_count: Array.isArray(body.messages) ? body.messages.length : 0,
|
||||
should_fail: shouldFail,
|
||||
status: "running",
|
||||
http_status: null,
|
||||
expected_text: replyText,
|
||||
response_text_preview: previewText(replyText),
|
||||
started_at: new Date(startedAt).toISOString(),
|
||||
started_epoch_ms: startedAt,
|
||||
configured_first_token_delay_ms: config.first_token_delay_ms,
|
||||
configured_chunk_delay_ms: config.chunk_delay_ms,
|
||||
configured_chunk_count: config.chunk_count,
|
||||
});
|
||||
|
||||
if (shouldFail) {
|
||||
await sleep(config.first_token_delay_ms);
|
||||
sendJson(response, config.fault_status, {
|
||||
error: {
|
||||
message: `LangBot fake provider injected HTTP ${config.fault_status}`,
|
||||
type: "fake_provider_fault",
|
||||
code: "fake_provider_fault",
|
||||
},
|
||||
});
|
||||
finishRequestRecord(requestRecord, startedPerf, {
|
||||
status: "http_fault",
|
||||
http_status: config.fault_status,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (body.stream) {
|
||||
await streamCompletion(response, {
|
||||
requestId,
|
||||
model: body.model || modelName,
|
||||
content: replyText,
|
||||
failAfterFirstChunk: config.fail_after_first_chunk,
|
||||
requestRecord,
|
||||
startedPerf,
|
||||
});
|
||||
} else {
|
||||
await sleep(config.first_token_delay_ms + config.chunk_delay_ms);
|
||||
sendJson(response, 200, completionPayload({
|
||||
requestId,
|
||||
model: body.model || modelName,
|
||||
content: replyText,
|
||||
}));
|
||||
markRequestTiming(requestRecord, "first_chunk", startedPerf);
|
||||
markRequestTiming(requestRecord, "first_content_chunk", startedPerf);
|
||||
requestRecord.content_chunk_count = 1;
|
||||
finishRequestRecord(requestRecord, startedPerf, {
|
||||
status: "ok",
|
||||
http_status: 200,
|
||||
});
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
sendJson(response, 404, {
|
||||
error: {
|
||||
message: `No fake provider route for ${request.method} ${url.pathname}`,
|
||||
type: "not_found",
|
||||
},
|
||||
});
|
||||
} catch (error) {
|
||||
if (requestRecord) {
|
||||
finishRequestRecord(requestRecord, startedPerf, {
|
||||
status: "fake_provider_error",
|
||||
http_status: 500,
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
});
|
||||
}
|
||||
sendJson(response, 500, {
|
||||
error: {
|
||||
message: error instanceof Error ? error.message : String(error),
|
||||
type: "fake_provider_error",
|
||||
},
|
||||
});
|
||||
} finally {
|
||||
const durationMs = Date.now() - startedAt;
|
||||
if (url.pathname !== "/healthz") {
|
||||
console.log(JSON.stringify({
|
||||
at: new Date().toISOString(),
|
||||
method: request.method,
|
||||
path: url.pathname,
|
||||
duration_ms: durationMs,
|
||||
}));
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
server.listen(port, host, async () => {
|
||||
const address = server.address();
|
||||
const selectedPort = typeof address === "object" && address ? address.port : port;
|
||||
const url = `http://${host}:${selectedPort}`;
|
||||
const state = {
|
||||
status: "ready",
|
||||
pid: process.pid,
|
||||
url,
|
||||
base_url: `${url}/v1`,
|
||||
model: modelName,
|
||||
started_at: new Date().toISOString(),
|
||||
};
|
||||
if (stateFile) {
|
||||
const path = resolve(stateFile);
|
||||
await mkdir(dirname(path), { recursive: true });
|
||||
await writeFile(path, `${JSON.stringify(state, null, 2)}\n`, "utf8");
|
||||
}
|
||||
console.log(JSON.stringify(state));
|
||||
});
|
||||
|
||||
server.on("error", (error) => {
|
||||
console.error(JSON.stringify({
|
||||
status: "error",
|
||||
reason: error instanceof Error ? error.message : String(error),
|
||||
}));
|
||||
exit(1);
|
||||
});
|
||||
|
||||
process.on("SIGTERM", () => {
|
||||
server.close(() => exit(0));
|
||||
});
|
||||
|
||||
function parseArgs(argv) {
|
||||
const result = {};
|
||||
for (const item of argv) {
|
||||
const match = item.match(/^--([^=]+)(?:=(.*))?$/);
|
||||
if (!match) continue;
|
||||
result[match[1]] = match[2] ?? "1";
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
function integer(value, fallback) {
|
||||
const parsed = Number.parseInt(String(value ?? ""), 10);
|
||||
return Number.isFinite(parsed) && parsed >= 0 ? parsed : fallback;
|
||||
}
|
||||
|
||||
function bool(value, fallback) {
|
||||
if (value === undefined || value === "") return fallback;
|
||||
if (/^(1|true|yes|on)$/i.test(String(value))) return true;
|
||||
if (/^(0|false|no|off)$/i.test(String(value))) return false;
|
||||
return fallback;
|
||||
}
|
||||
|
||||
function sleep(ms) {
|
||||
return new Promise((resolve) => setTimeout(resolve, Math.max(0, ms)));
|
||||
}
|
||||
|
||||
async function readJson(request) {
|
||||
let text = "";
|
||||
for await (const chunk of request) text += chunk.toString();
|
||||
if (!text) return {};
|
||||
return JSON.parse(text);
|
||||
}
|
||||
|
||||
function sendJson(response, status, payload) {
|
||||
const text = `${JSON.stringify(payload)}\n`;
|
||||
response.writeHead(status, {
|
||||
"content-type": "application/json",
|
||||
"content-length": Buffer.byteLength(text),
|
||||
});
|
||||
response.end(text);
|
||||
}
|
||||
|
||||
function completionPayload({ requestId, model, content }) {
|
||||
const completionTokens = tokenEstimate(content);
|
||||
return {
|
||||
id: requestId,
|
||||
object: "chat.completion",
|
||||
created: Math.floor(Date.now() / 1000),
|
||||
model,
|
||||
choices: [
|
||||
{
|
||||
index: 0,
|
||||
message: {
|
||||
role: "assistant",
|
||||
content,
|
||||
},
|
||||
finish_reason: "stop",
|
||||
},
|
||||
],
|
||||
usage: {
|
||||
prompt_tokens: 8,
|
||||
completion_tokens: completionTokens,
|
||||
total_tokens: 8 + completionTokens,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function streamCompletion(response, {
|
||||
requestId,
|
||||
model,
|
||||
content,
|
||||
failAfterFirstChunk: failMidStream,
|
||||
requestRecord,
|
||||
startedPerf,
|
||||
}) {
|
||||
response.writeHead(200, {
|
||||
"content-type": "text/event-stream; charset=utf-8",
|
||||
"cache-control": "no-cache",
|
||||
"connection": "keep-alive",
|
||||
});
|
||||
|
||||
await sleep(config.first_token_delay_ms);
|
||||
markRequestTiming(requestRecord, "first_chunk", startedPerf);
|
||||
writeSse(response, {
|
||||
id: requestId,
|
||||
object: "chat.completion.chunk",
|
||||
created: Math.floor(Date.now() / 1000),
|
||||
model,
|
||||
choices: [{ index: 0, delta: { role: "assistant" }, finish_reason: null }],
|
||||
});
|
||||
|
||||
const chunks = splitContent(content);
|
||||
for (let index = 0; index < chunks.length; index += 1) {
|
||||
await sleep(config.chunk_delay_ms);
|
||||
if (index === 0) markRequestTiming(requestRecord, "first_content_chunk", startedPerf);
|
||||
requestRecord.content_chunk_count = (requestRecord.content_chunk_count || 0) + 1;
|
||||
writeSse(response, {
|
||||
id: requestId,
|
||||
object: "chat.completion.chunk",
|
||||
created: Math.floor(Date.now() / 1000),
|
||||
model,
|
||||
choices: [{ index: 0, delta: { content: chunks[index] }, finish_reason: null }],
|
||||
});
|
||||
if (failMidStream && index === 0) {
|
||||
finishRequestRecord(requestRecord, startedPerf, {
|
||||
status: "mid_stream_disconnect",
|
||||
http_status: 200,
|
||||
});
|
||||
response.destroy(new Error("LangBot fake provider injected mid-stream disconnect"));
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
await sleep(config.chunk_delay_ms);
|
||||
const completionTokens = tokenEstimate(content);
|
||||
writeSse(response, {
|
||||
id: requestId,
|
||||
object: "chat.completion.chunk",
|
||||
created: Math.floor(Date.now() / 1000),
|
||||
model,
|
||||
choices: [{ index: 0, delta: {}, finish_reason: "stop" }],
|
||||
usage: {
|
||||
prompt_tokens: 8,
|
||||
completion_tokens: completionTokens,
|
||||
total_tokens: 8 + completionTokens,
|
||||
},
|
||||
});
|
||||
response.write("data: [DONE]\n\n");
|
||||
response.end();
|
||||
finishRequestRecord(requestRecord, startedPerf, {
|
||||
status: "ok",
|
||||
http_status: 200,
|
||||
});
|
||||
}
|
||||
|
||||
function writeSse(response, payload) {
|
||||
response.write(`data: ${JSON.stringify(payload)}\n\n`);
|
||||
}
|
||||
|
||||
function splitContent(content) {
|
||||
const text = String(content);
|
||||
const requested = config.chunk_count;
|
||||
if (requested <= 1 || text.length <= 1) return [text];
|
||||
const chunkSize = Math.max(1, Math.ceil(text.length / requested));
|
||||
const chunks = [];
|
||||
for (let index = 0; index < text.length; index += chunkSize) {
|
||||
chunks.push(text.slice(index, index + chunkSize));
|
||||
}
|
||||
return chunks;
|
||||
}
|
||||
|
||||
function tokenEstimate(content) {
|
||||
return Math.max(1, Math.ceil(String(content || "").length / 4));
|
||||
}
|
||||
|
||||
function responseTextForBody(body) {
|
||||
if (!config.dynamic_response) {
|
||||
return config.response_text;
|
||||
}
|
||||
const messages = Array.isArray(body.messages) ? body.messages : [];
|
||||
const lastUser = [...messages].reverse().find((message) => message?.role === "user");
|
||||
const text = flattenContent(lastUser?.content || "");
|
||||
const quoted = text.match(/["'“”](.{1,80}?)["'“”]/);
|
||||
if (quoted?.[1]) return quoted[1].trim();
|
||||
const exact = text.match(/(?:reply|回复|输出|return)\s+(?:exactly\s+)?([A-Za-z0-9_.:@-]{1,80})/i);
|
||||
if (exact?.[1]) return exact[1].trim().replace(/[。.!?]+$/, "");
|
||||
const only = text.match(/只回复\s*([A-Za-z0-9_.:@-]{1,80})/);
|
||||
if (only?.[1]) return only[1].trim().replace(/[。.!?]+$/, "");
|
||||
return config.response_text;
|
||||
}
|
||||
|
||||
function flattenContent(content) {
|
||||
if (typeof content === "string") return content;
|
||||
if (Array.isArray(content)) {
|
||||
return content
|
||||
.map((item) => {
|
||||
if (typeof item === "string") return item;
|
||||
if (item && typeof item === "object") return item.text || "";
|
||||
return "";
|
||||
})
|
||||
.join("\n");
|
||||
}
|
||||
return "";
|
||||
}
|
||||
|
||||
function recordRequest(entry) {
|
||||
const item = {
|
||||
...entry,
|
||||
at: new Date().toISOString(),
|
||||
finished_at: null,
|
||||
finished_epoch_ms: null,
|
||||
duration_ms: null,
|
||||
first_chunk_at: null,
|
||||
first_chunk_epoch_ms: null,
|
||||
first_chunk_ms: null,
|
||||
first_content_chunk_at: null,
|
||||
first_content_chunk_epoch_ms: null,
|
||||
first_content_chunk_ms: null,
|
||||
content_chunk_count: 0,
|
||||
};
|
||||
recentRequests.push(item);
|
||||
while (recentRequests.length > config.request_log_limit) recentRequests.shift();
|
||||
return item;
|
||||
}
|
||||
|
||||
function markRequestTiming(entry, key, startedPerf) {
|
||||
if (!entry || entry[`${key}_at`]) return;
|
||||
const now = Date.now();
|
||||
entry[`${key}_at`] = new Date(now).toISOString();
|
||||
entry[`${key}_epoch_ms`] = now;
|
||||
entry[`${key}_ms`] = rounded(performance.now() - startedPerf);
|
||||
}
|
||||
|
||||
function finishRequestRecord(entry, startedPerf, updates = {}) {
|
||||
if (!entry || entry.finished_at) return;
|
||||
const now = Date.now();
|
||||
Object.assign(entry, updates);
|
||||
entry.finished_at = new Date(now).toISOString();
|
||||
entry.finished_epoch_ms = now;
|
||||
entry.duration_ms = rounded(performance.now() - startedPerf);
|
||||
}
|
||||
|
||||
function rounded(value) {
|
||||
return Number(value.toFixed(3));
|
||||
}
|
||||
|
||||
function previewText(value) {
|
||||
return String(value || "").slice(0, 120);
|
||||
}
|
||||
|
||||
function resetRequestState() {
|
||||
requestCount = 0;
|
||||
recentRequests.length = 0;
|
||||
}
|
||||
|
||||
function applyConfig(updates) {
|
||||
if (!updates || typeof updates !== "object") return;
|
||||
assignString(updates, "response_text");
|
||||
assignNonNegativeInteger(updates, "first_token_delay_ms");
|
||||
assignNonNegativeInteger(updates, "chunk_delay_ms");
|
||||
assignNonNegativeInteger(updates, "chunk_count");
|
||||
assignNonNegativeInteger(updates, "fail_first_n");
|
||||
assignNonNegativeInteger(updates, "fail_every_n");
|
||||
assignNonNegativeInteger(updates, "request_log_limit");
|
||||
if (updates.fault_status !== undefined) {
|
||||
const parsed = Number.parseInt(String(updates.fault_status), 10);
|
||||
if (Number.isInteger(parsed) && parsed >= 400 && parsed <= 599) config.fault_status = parsed;
|
||||
}
|
||||
assignBoolean(updates, "fail_after_first_chunk");
|
||||
assignBoolean(updates, "dynamic_response");
|
||||
}
|
||||
|
||||
function assignString(updates, key) {
|
||||
if (updates[key] !== undefined) config[key] = String(updates[key]);
|
||||
}
|
||||
|
||||
function assignNonNegativeInteger(updates, key) {
|
||||
if (updates[key] === undefined) return;
|
||||
const parsed = Number.parseInt(String(updates[key]), 10);
|
||||
if (Number.isInteger(parsed) && parsed >= 0) config[key] = parsed;
|
||||
}
|
||||
|
||||
function assignBoolean(updates, key) {
|
||||
if (updates[key] === undefined) return;
|
||||
config[key] = bool(updates[key], config[key]);
|
||||
}
|
||||
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
+1
-2
@@ -72,7 +72,6 @@ export async function writeResult(paths, result) {
|
||||
}
|
||||
|
||||
export async function loadEnvFiles(paths = ["skills/.env", "skills/.env.local"]) {
|
||||
const processEnvKeys = new Set(Object.keys(env));
|
||||
for (const path of paths) {
|
||||
let text = "";
|
||||
try {
|
||||
@@ -87,7 +86,7 @@ export async function loadEnvFiles(paths = ["skills/.env", "skills/.env.local"])
|
||||
if (equals <= 0) continue;
|
||||
const key = trimmed.slice(0, equals).trim();
|
||||
const value = trimmed.slice(equals + 1).trim().replace(/^["']|["']$/g, "");
|
||||
if (!processEnvKeys.has(key)) env[key] = value;
|
||||
if (!(key in env)) env[key] = value;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Executable → Regular
Executable → Regular
@@ -54,7 +54,6 @@ const debugChatSessionType = env.LANGBOT_E2E_DEBUG_CHAT_SESSION_TYPE || "person"
|
||||
const pipelineConfigDiagnosticPath = resolve(paths.evidenceDir, "pipeline-config-diagnostic.json");
|
||||
const debugChatResetDiagnosticPath = resolve(paths.evidenceDir, "debug-chat-reset-diagnostic.json");
|
||||
const pipelineConfigRestoreDiagnosticPath = resolve(paths.evidenceDir, "pipeline-config-restore-diagnostic.json");
|
||||
const metricsPath = resolve(paths.evidenceDir, "metrics.json");
|
||||
const startedAt = new Date();
|
||||
|
||||
let browser;
|
||||
@@ -81,11 +80,10 @@ let result = {
|
||||
console_log: paths.consoleLog,
|
||||
network_log: paths.networkLog,
|
||||
screenshot: paths.screenshot,
|
||||
metrics_json: metricsPath,
|
||||
automation_result_json: paths.automationResultJson,
|
||||
result_json: paths.resultJson,
|
||||
},
|
||||
evidence_collected: ["ui", "screenshot", "console", "network", "metrics"],
|
||||
evidence_collected: ["ui", "screenshot", "console", "network"],
|
||||
};
|
||||
|
||||
function boolFromEnv(value, defaultValue) {
|
||||
@@ -105,29 +103,6 @@ function parseJsonEnv(key, fallback) {
|
||||
}
|
||||
}
|
||||
|
||||
function positiveNumberEnv(key, fallback) {
|
||||
const value = Number(env[key] || "");
|
||||
return Number.isFinite(value) && value >= 0 ? value : fallback;
|
||||
}
|
||||
|
||||
function percentile(values, percentileValue) {
|
||||
if (values.length === 0) return 0;
|
||||
const sorted = [...values].sort((a, b) => a - b);
|
||||
const index = Math.min(sorted.length - 1, Math.ceil((percentileValue / 100) * sorted.length) - 1);
|
||||
return Number(sorted[index].toFixed(3));
|
||||
}
|
||||
|
||||
function stats(values) {
|
||||
if (values.length === 0) return { min: 0, p50: 0, p95: 0, p99: 0, max: 0 };
|
||||
return {
|
||||
min: Number(Math.min(...values).toFixed(3)),
|
||||
p50: percentile(values, 50),
|
||||
p95: percentile(values, 95),
|
||||
p99: percentile(values, 99),
|
||||
max: Number(Math.max(...values).toFixed(3)),
|
||||
};
|
||||
}
|
||||
|
||||
function promptStepsFromEnv() {
|
||||
const rawSteps = parseJsonEnv("LANGBOT_E2E_PROMPTS_JSON", null);
|
||||
if (rawSteps === null) {
|
||||
@@ -683,7 +658,6 @@ try {
|
||||
} else {
|
||||
for (let index = 0; index < promptSteps.length; index += 1) {
|
||||
const step = promptSteps[index];
|
||||
const promptStartedAt = Date.now();
|
||||
const chatResult = await runDebugChatPrompt(page, {
|
||||
prompt: step.prompt,
|
||||
expectedText: step.expectedText,
|
||||
@@ -691,13 +665,11 @@ try {
|
||||
imagePath: index === 0 ? imagePath : "",
|
||||
failureSignals: failureSignals.length > 0 ? failureSignals : undefined,
|
||||
});
|
||||
const promptDurationMs = Date.now() - promptStartedAt;
|
||||
result.chat_results.push({
|
||||
index,
|
||||
expected_text: step.expectedText,
|
||||
status: chatResult.status,
|
||||
reason: chatResult.reason,
|
||||
response_duration_ms: promptDurationMs,
|
||||
min_expected_count: chatResult.min_expected_count,
|
||||
final_count: chatResult.final_count,
|
||||
before_assistant_expected_count: chatResult.before_assistant_expected_count,
|
||||
@@ -742,56 +714,6 @@ try {
|
||||
const finishedAt = new Date();
|
||||
result.finished_at = finishedAt.toISOString();
|
||||
result.finished_at_local = localIsoWithOffset(finishedAt);
|
||||
result.duration_ms = finishedAt.getTime() - startedAt.getTime();
|
||||
const responseDurations = result.chat_results
|
||||
.map((item) => item.response_duration_ms)
|
||||
.filter((value) => Number.isFinite(value));
|
||||
const passedPrompts = result.chat_results.filter((item) => item.status === "pass").length;
|
||||
const attemptedPrompts = result.chat_results.length;
|
||||
const errorRate = attemptedPrompts === 0 ? 1 : Number(((attemptedPrompts - passedPrompts) / attemptedPrompts).toFixed(4));
|
||||
const responseStats = stats(responseDurations);
|
||||
const responseP95BudgetMs = positiveNumberEnv(
|
||||
"LANGBOT_E2E_DEBUG_CHAT_RESPONSE_P95_MS",
|
||||
positiveNumberEnv("LANGBOT_DEBUG_CHAT_RESPONSE_P95_MS", safeResponseTimeoutMs),
|
||||
);
|
||||
const maxErrorRate = positiveNumberEnv("LANGBOT_E2E_DEBUG_CHAT_MAX_ERROR_RATE", 0);
|
||||
const metrics = {
|
||||
probe: caseId,
|
||||
url: result.url,
|
||||
prompt_count: result.prompt_count,
|
||||
attempted_prompt_count: attemptedPrompts,
|
||||
passed_prompt_count: passedPrompts,
|
||||
error_rate: errorRate,
|
||||
response_duration_ms: responseStats,
|
||||
total_duration_ms: result.duration_ms,
|
||||
chat_results: result.chat_results,
|
||||
};
|
||||
result.metrics_summary = {
|
||||
prompt_count: metrics.prompt_count,
|
||||
attempted_prompt_count: metrics.attempted_prompt_count,
|
||||
passed_prompt_count: metrics.passed_prompt_count,
|
||||
error_rate: metrics.error_rate,
|
||||
response_p50_ms: metrics.response_duration_ms.p50,
|
||||
response_p95_ms: metrics.response_duration_ms.p95,
|
||||
total_duration_ms: metrics.total_duration_ms,
|
||||
};
|
||||
result.thresholds_summary = {
|
||||
response_p95_ms: {
|
||||
actual: metrics.response_duration_ms.p95,
|
||||
max: responseP95BudgetMs,
|
||||
pass: attemptedPrompts > 0 && metrics.response_duration_ms.p95 <= responseP95BudgetMs,
|
||||
},
|
||||
error_rate: {
|
||||
actual: metrics.error_rate,
|
||||
max: maxErrorRate,
|
||||
pass: metrics.error_rate <= maxErrorRate,
|
||||
},
|
||||
};
|
||||
await writeFile(metricsPath, `${JSON.stringify(metrics, null, 2)}\n`, "utf8");
|
||||
if (result.status === "pass" && !Object.values(result.thresholds_summary).every((item) => item.pass)) {
|
||||
result.status = "fail";
|
||||
result.reason = "Debug Chat performance breached response latency or error-rate thresholds.";
|
||||
}
|
||||
const existingEvidence = {};
|
||||
for (const [key, value] of Object.entries(result.evidence)) {
|
||||
if (typeof value !== "string") continue;
|
||||
|
||||
Executable → Regular
@@ -130,7 +130,6 @@
|
||||
"references/local-agent-runner.md",
|
||||
"references/mcp-stdio-testing.md",
|
||||
"references/model-provider-testing.md",
|
||||
"references/performance-reliability-testing.md",
|
||||
"references/pipeline-debug-chat.md",
|
||||
"references/plugin-e2e-smoke.md",
|
||||
"references/sandbox-skill-authoring.md",
|
||||
@@ -151,16 +150,6 @@
|
||||
"agent-runner-release-preflight",
|
||||
"agent-runner-runtime-chaos",
|
||||
"dify-agent-debug-chat",
|
||||
"langbot-fake-provider-debug-chat-cross-pipeline-isolation",
|
||||
"langbot-fake-provider-debug-chat-fault-recovery",
|
||||
"langbot-fake-provider-debug-chat-load",
|
||||
"langbot-fake-provider-debug-chat-slow-load",
|
||||
"langbot-fault-taxonomy-contract",
|
||||
"langbot-live-backend-latency",
|
||||
"langbot-live-backend-log-health",
|
||||
"langbot-live-control-plane-api",
|
||||
"langbot-overhead-accounting-contract",
|
||||
"langbot-space-debug-chat-concurrency-smoke",
|
||||
"langrag-kb-retrieve",
|
||||
"langrag-parser-golden-e2e",
|
||||
"langrag-sentinel-kb-discover",
|
||||
@@ -176,7 +165,6 @@
|
||||
"mcp-stdio-register",
|
||||
"mcp-stdio-tool-call",
|
||||
"pipeline-debug-chat",
|
||||
"pipeline-debug-chat-performance",
|
||||
"plugin-e2e-smoke",
|
||||
"provider-deepseek",
|
||||
"qa-plugin-smoke-live-install",
|
||||
@@ -498,316 +486,6 @@
|
||||
"backend_log"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "langbot-fake-provider-debug-chat-cross-pipeline-isolation",
|
||||
"title": "LangBot Debug Chat fake-provider cross-pipeline isolation probe",
|
||||
"mode": "probe",
|
||||
"area": "reliability",
|
||||
"type": "reliability",
|
||||
"priority": "p1",
|
||||
"risk": "high",
|
||||
"ci_eligible": false,
|
||||
"tags": [
|
||||
"reliability",
|
||||
"debug-chat",
|
||||
"websocket",
|
||||
"fake-provider",
|
||||
"isolation",
|
||||
"concurrency",
|
||||
"metrics"
|
||||
],
|
||||
"automation": "skills/langbot-testing/probes/langbot-debug-chat-cross-pipeline-isolation.mjs",
|
||||
"setup_automation": [
|
||||
"node:scripts/e2e/ensure-fake-provider-cross-pipelines.mjs --write-env"
|
||||
],
|
||||
"setup_provides_env": [
|
||||
"LANGBOT_FAKE_PROVIDER_URL",
|
||||
"LANGBOT_FAKE_PROVIDER_BASE_URL",
|
||||
"LANGBOT_FAKE_PROVIDER_PID",
|
||||
"LANGBOT_FAKE_PROVIDER_PIPELINE_A_URL",
|
||||
"LANGBOT_FAKE_PROVIDER_PIPELINE_A_NAME",
|
||||
"LANGBOT_FAKE_PROVIDER_PIPELINE_B_URL",
|
||||
"LANGBOT_FAKE_PROVIDER_PIPELINE_B_NAME"
|
||||
],
|
||||
"evidence_required": [
|
||||
"metrics",
|
||||
"network",
|
||||
"api_diagnostic",
|
||||
"filesystem"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "langbot-fake-provider-debug-chat-fault-recovery",
|
||||
"title": "LangBot Debug Chat fake-provider fault recovery probe",
|
||||
"mode": "probe",
|
||||
"area": "reliability",
|
||||
"type": "chaos",
|
||||
"priority": "p1",
|
||||
"risk": "high",
|
||||
"ci_eligible": false,
|
||||
"tags": [
|
||||
"reliability",
|
||||
"chaos",
|
||||
"debug-chat",
|
||||
"websocket",
|
||||
"fake-provider",
|
||||
"fault-injection",
|
||||
"metrics"
|
||||
],
|
||||
"automation": "skills/langbot-testing/probes/langbot-debug-chat-concurrency.mjs",
|
||||
"setup_automation": [
|
||||
"node:scripts/e2e/ensure-fake-provider-pipeline.mjs --write-env"
|
||||
],
|
||||
"setup_provides_env": [
|
||||
"LANGBOT_FAKE_PROVIDER_URL",
|
||||
"LANGBOT_FAKE_PROVIDER_BASE_URL",
|
||||
"LANGBOT_FAKE_PROVIDER_PID",
|
||||
"LANGBOT_FAKE_PROVIDER_PROVIDER_UUID",
|
||||
"LANGBOT_FAKE_PROVIDER_MODEL_UUID",
|
||||
"LANGBOT_FAKE_PROVIDER_PIPELINE_URL",
|
||||
"LANGBOT_FAKE_PROVIDER_PIPELINE_NAME"
|
||||
],
|
||||
"evidence_required": [
|
||||
"metrics",
|
||||
"network",
|
||||
"api_diagnostic",
|
||||
"filesystem"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "langbot-fake-provider-debug-chat-load",
|
||||
"title": "LangBot Debug Chat controlled fake-provider load probe",
|
||||
"mode": "probe",
|
||||
"area": "performance",
|
||||
"type": "performance",
|
||||
"priority": "p1",
|
||||
"risk": "medium",
|
||||
"ci_eligible": false,
|
||||
"tags": [
|
||||
"performance",
|
||||
"debug-chat",
|
||||
"websocket",
|
||||
"fake-provider",
|
||||
"load",
|
||||
"metrics"
|
||||
],
|
||||
"automation": "skills/langbot-testing/probes/langbot-debug-chat-concurrency.mjs",
|
||||
"setup_automation": [
|
||||
"node:scripts/e2e/ensure-fake-provider-pipeline.mjs --write-env"
|
||||
],
|
||||
"setup_provides_env": [
|
||||
"LANGBOT_FAKE_PROVIDER_URL",
|
||||
"LANGBOT_FAKE_PROVIDER_BASE_URL",
|
||||
"LANGBOT_FAKE_PROVIDER_PID",
|
||||
"LANGBOT_FAKE_PROVIDER_PROVIDER_UUID",
|
||||
"LANGBOT_FAKE_PROVIDER_MODEL_UUID",
|
||||
"LANGBOT_FAKE_PROVIDER_PIPELINE_URL",
|
||||
"LANGBOT_FAKE_PROVIDER_PIPELINE_NAME"
|
||||
],
|
||||
"evidence_required": [
|
||||
"metrics",
|
||||
"network",
|
||||
"api_diagnostic",
|
||||
"filesystem"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "langbot-fake-provider-debug-chat-slow-load",
|
||||
"title": "LangBot Debug Chat slow fake-provider load probe",
|
||||
"mode": "probe",
|
||||
"area": "performance",
|
||||
"type": "performance",
|
||||
"priority": "p1",
|
||||
"risk": "medium",
|
||||
"ci_eligible": false,
|
||||
"tags": [
|
||||
"performance",
|
||||
"debug-chat",
|
||||
"websocket",
|
||||
"fake-provider",
|
||||
"slow-provider",
|
||||
"load",
|
||||
"metrics"
|
||||
],
|
||||
"automation": "skills/langbot-testing/probes/langbot-debug-chat-concurrency.mjs",
|
||||
"setup_automation": [
|
||||
"node:scripts/e2e/ensure-fake-provider-pipeline.mjs --write-env"
|
||||
],
|
||||
"setup_provides_env": [
|
||||
"LANGBOT_FAKE_PROVIDER_URL",
|
||||
"LANGBOT_FAKE_PROVIDER_BASE_URL",
|
||||
"LANGBOT_FAKE_PROVIDER_PID",
|
||||
"LANGBOT_FAKE_PROVIDER_PROVIDER_UUID",
|
||||
"LANGBOT_FAKE_PROVIDER_MODEL_UUID",
|
||||
"LANGBOT_FAKE_PROVIDER_PIPELINE_URL",
|
||||
"LANGBOT_FAKE_PROVIDER_PIPELINE_NAME"
|
||||
],
|
||||
"evidence_required": [
|
||||
"metrics",
|
||||
"network",
|
||||
"api_diagnostic",
|
||||
"filesystem"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "langbot-fault-taxonomy-contract",
|
||||
"title": "LangBot fault taxonomy and cleanup contract",
|
||||
"mode": "probe",
|
||||
"area": "reliability",
|
||||
"type": "chaos",
|
||||
"priority": "p1",
|
||||
"risk": "medium",
|
||||
"ci_eligible": true,
|
||||
"tags": [
|
||||
"reliability",
|
||||
"chaos",
|
||||
"contract",
|
||||
"synthetic"
|
||||
],
|
||||
"automation": "skills/langbot-testing/probes/langbot-fault-taxonomy-contract.mjs",
|
||||
"setup_automation": [],
|
||||
"setup_provides_env": [],
|
||||
"evidence_required": [
|
||||
"metrics",
|
||||
"filesystem"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "langbot-live-backend-latency",
|
||||
"title": "LangBot live backend basic latency probe",
|
||||
"mode": "probe",
|
||||
"area": "performance",
|
||||
"type": "performance",
|
||||
"priority": "p1",
|
||||
"risk": "medium",
|
||||
"ci_eligible": false,
|
||||
"tags": [
|
||||
"performance",
|
||||
"live-backend",
|
||||
"latency",
|
||||
"metrics"
|
||||
],
|
||||
"automation": "skills/langbot-testing/probes/langbot-live-backend-latency.mjs",
|
||||
"setup_automation": [],
|
||||
"setup_provides_env": [],
|
||||
"evidence_required": [
|
||||
"metrics",
|
||||
"network",
|
||||
"api_diagnostic",
|
||||
"filesystem"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "langbot-live-backend-log-health",
|
||||
"title": "LangBot live backend log health probe",
|
||||
"mode": "probe",
|
||||
"area": "reliability",
|
||||
"type": "reliability",
|
||||
"priority": "p1",
|
||||
"risk": "medium",
|
||||
"ci_eligible": false,
|
||||
"tags": [
|
||||
"reliability",
|
||||
"live-backend",
|
||||
"backend-log",
|
||||
"metrics"
|
||||
],
|
||||
"automation": "skills/langbot-testing/probes/langbot-live-backend-log-health.mjs",
|
||||
"setup_automation": [],
|
||||
"setup_provides_env": [],
|
||||
"evidence_required": [
|
||||
"metrics",
|
||||
"backend_log",
|
||||
"filesystem"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "langbot-live-control-plane-api",
|
||||
"title": "LangBot live control-plane API probe",
|
||||
"mode": "probe",
|
||||
"area": "performance",
|
||||
"type": "performance",
|
||||
"priority": "p1",
|
||||
"risk": "medium",
|
||||
"ci_eligible": false,
|
||||
"tags": [
|
||||
"performance",
|
||||
"reliability",
|
||||
"live-backend",
|
||||
"control-plane",
|
||||
"metrics"
|
||||
],
|
||||
"automation": "skills/langbot-testing/probes/langbot-live-control-plane-api.mjs",
|
||||
"setup_automation": [],
|
||||
"setup_provides_env": [],
|
||||
"evidence_required": [
|
||||
"metrics",
|
||||
"network",
|
||||
"api_diagnostic",
|
||||
"filesystem"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "langbot-overhead-accounting-contract",
|
||||
"title": "LangBot overhead accounting metrics contract",
|
||||
"mode": "probe",
|
||||
"area": "performance",
|
||||
"type": "performance",
|
||||
"priority": "p1",
|
||||
"risk": "medium",
|
||||
"ci_eligible": true,
|
||||
"tags": [
|
||||
"performance",
|
||||
"metrics",
|
||||
"contract",
|
||||
"synthetic"
|
||||
],
|
||||
"automation": "skills/langbot-testing/probes/langbot-overhead-accounting-contract.mjs",
|
||||
"setup_automation": [],
|
||||
"setup_provides_env": [],
|
||||
"evidence_required": [
|
||||
"metrics",
|
||||
"resource_log",
|
||||
"filesystem"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "langbot-space-debug-chat-concurrency-smoke",
|
||||
"title": "LangBot Debug Chat real Space-provider concurrency smoke",
|
||||
"mode": "probe",
|
||||
"area": "performance",
|
||||
"type": "performance",
|
||||
"priority": "p1",
|
||||
"risk": "high",
|
||||
"ci_eligible": false,
|
||||
"tags": [
|
||||
"performance",
|
||||
"debug-chat",
|
||||
"websocket",
|
||||
"space",
|
||||
"live-provider",
|
||||
"smoke",
|
||||
"metrics"
|
||||
],
|
||||
"automation": "skills/langbot-testing/probes/langbot-debug-chat-concurrency.mjs",
|
||||
"setup_automation": [
|
||||
"node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env"
|
||||
],
|
||||
"setup_provides_env": [
|
||||
"LANGBOT_PIPELINE_URL",
|
||||
"LANGBOT_PIPELINE_NAME",
|
||||
"LANGBOT_LOCAL_AGENT_PIPELINE_URL",
|
||||
"LANGBOT_LOCAL_AGENT_PIPELINE_NAME",
|
||||
"LANGBOT_LOCAL_AGENT_MODEL_UUID",
|
||||
"LANGBOT_E2E_MODEL_UUID"
|
||||
],
|
||||
"evidence_required": [
|
||||
"metrics",
|
||||
"network",
|
||||
"api_diagnostic",
|
||||
"filesystem"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "langrag-kb-retrieve",
|
||||
"title": "LangRAG knowledge base ingests and retrieves a sentinel document",
|
||||
@@ -1233,38 +911,6 @@
|
||||
"backend_log"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "pipeline-debug-chat-performance",
|
||||
"title": "Pipeline Debug Chat user-path performance probe",
|
||||
"mode": "agent-browser",
|
||||
"area": "pipeline",
|
||||
"type": "performance",
|
||||
"priority": "p1",
|
||||
"risk": "medium",
|
||||
"ci_eligible": false,
|
||||
"tags": [
|
||||
"performance",
|
||||
"pipeline",
|
||||
"debug-chat",
|
||||
"user-path",
|
||||
"metrics"
|
||||
],
|
||||
"automation": "scripts/e2e/pipeline-debug-chat.mjs",
|
||||
"setup_automation": [
|
||||
"node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env"
|
||||
],
|
||||
"setup_provides_env": [
|
||||
"LANGBOT_PIPELINE_URL",
|
||||
"LANGBOT_PIPELINE_NAME"
|
||||
],
|
||||
"evidence_required": [
|
||||
"ui",
|
||||
"screenshot",
|
||||
"console",
|
||||
"network",
|
||||
"metrics"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "plugin-e2e-smoke",
|
||||
"title": "Plugin system installs a local plugin and exposes tool/page APIs",
|
||||
@@ -1413,12 +1059,6 @@
|
||||
"suites": [
|
||||
"agent-runner-release-gate",
|
||||
"core-smoke",
|
||||
"langbot-debug-chat-isolation-gate",
|
||||
"langbot-debug-chat-load-gate",
|
||||
"langbot-live-backend-gate",
|
||||
"langbot-performance-contract-gate",
|
||||
"langbot-performance-reliability-gate",
|
||||
"langbot-user-path-performance-gate",
|
||||
"local-agent-gate"
|
||||
],
|
||||
"suite_summaries": [
|
||||
@@ -1481,113 +1121,6 @@
|
||||
"local-agent-basic-debug-chat"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "langbot-debug-chat-isolation-gate",
|
||||
"title": "LangBot Debug Chat isolation gate",
|
||||
"description": "Manual/non-required cross-pipeline Debug Chat isolation gate. Current releases may fail this gate because of product bug #2286; use it as regression evidence after the routing fix lands.",
|
||||
"type": "reliability",
|
||||
"priority": "p1",
|
||||
"tags": [
|
||||
"reliability",
|
||||
"debug-chat",
|
||||
"websocket",
|
||||
"isolation",
|
||||
"concurrency"
|
||||
],
|
||||
"cases": [
|
||||
"langbot-fake-provider-debug-chat-cross-pipeline-isolation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "langbot-debug-chat-load-gate",
|
||||
"title": "LangBot Debug Chat load gate",
|
||||
"description": "Manual/non-required message-path load checks for Pipeline Debug Chat: controlled fake-provider baseline, slow-provider and fault-recovery profiles, plus optional real Space-provider smoke. Cross-pipeline isolation is split into langbot-debug-chat-isolation-gate because current releases may fail it due to product bug #2286.",
|
||||
"type": "performance",
|
||||
"priority": "p1",
|
||||
"tags": [
|
||||
"performance",
|
||||
"debug-chat",
|
||||
"websocket",
|
||||
"load"
|
||||
],
|
||||
"cases": [
|
||||
"langbot-fake-provider-debug-chat-load",
|
||||
"langbot-fake-provider-debug-chat-slow-load",
|
||||
"langbot-fake-provider-debug-chat-fault-recovery",
|
||||
"langbot-space-debug-chat-concurrency-smoke"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "langbot-live-backend-gate",
|
||||
"title": "LangBot live backend reliability gate",
|
||||
"description": "Live backend control-plane responsiveness and runtime log health checks for a locally running LangBot instance.",
|
||||
"type": "reliability",
|
||||
"priority": "p1",
|
||||
"tags": [
|
||||
"performance",
|
||||
"reliability",
|
||||
"live-backend",
|
||||
"metrics"
|
||||
],
|
||||
"cases": [
|
||||
"langbot-live-backend-latency",
|
||||
"langbot-live-control-plane-api",
|
||||
"langbot-live-backend-log-health"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "langbot-performance-contract-gate",
|
||||
"title": "LangBot performance contract gate",
|
||||
"description": "Fast synthetic contract checks for performance metric accounting and non-destructive reliability fault taxonomy.",
|
||||
"type": "contract",
|
||||
"priority": "p1",
|
||||
"tags": [
|
||||
"performance",
|
||||
"reliability",
|
||||
"contract",
|
||||
"metrics"
|
||||
],
|
||||
"cases": [
|
||||
"langbot-overhead-accounting-contract",
|
||||
"langbot-fault-taxonomy-contract"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "langbot-performance-reliability-gate",
|
||||
"title": "LangBot performance and reliability starter gate",
|
||||
"description": "Starter gate for LangBot performance accounting, live backend control-plane latency, and non-destructive fault taxonomy checks.",
|
||||
"type": "reliability",
|
||||
"priority": "p1",
|
||||
"tags": [
|
||||
"performance",
|
||||
"reliability",
|
||||
"metrics",
|
||||
"chaos"
|
||||
],
|
||||
"cases": [
|
||||
"langbot-overhead-accounting-contract",
|
||||
"langbot-fault-taxonomy-contract",
|
||||
"langbot-live-backend-latency",
|
||||
"langbot-live-control-plane-api",
|
||||
"langbot-live-backend-log-health"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "langbot-user-path-performance-gate",
|
||||
"title": "LangBot user-path performance gate",
|
||||
"description": "Browser-visible performance checks for user-facing LangBot paths such as Pipeline Debug Chat.",
|
||||
"type": "performance",
|
||||
"priority": "p1",
|
||||
"tags": [
|
||||
"performance",
|
||||
"browser",
|
||||
"debug-chat",
|
||||
"user-path"
|
||||
],
|
||||
"cases": [
|
||||
"pipeline-debug-chat-performance"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "local-agent-gate",
|
||||
"title": "Local Agent runner regression gate",
|
||||
@@ -1732,7 +1265,6 @@
|
||||
"sandbox-native-tools-unavailable",
|
||||
"socks-proxy-without-socksio",
|
||||
"survey-widget-blocks-debug-chat",
|
||||
"telemetry-proxy-noise",
|
||||
"tool-name-collision-between-mcp-and-plugin",
|
||||
"uv-run-resyncs-local-sdk"
|
||||
],
|
||||
@@ -1917,14 +1449,6 @@
|
||||
"mcp-stdio-tool-call"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "telemetry-proxy-noise",
|
||||
"title": "Telemetry posting fails through the proxy while the target flow succeeds",
|
||||
"category": "env_issue",
|
||||
"related_cases": [
|
||||
"langbot-space-debug-chat-concurrency-smoke"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "tool-name-collision-between-mcp-and-plugin",
|
||||
"title": "MCP and plugin expose the same tool name",
|
||||
|
||||
@@ -26,23 +26,6 @@ LANGBOT_NO_PROXY=localhost,127.0.0.1,::1
|
||||
LANGBOT_PIPELINE_URL=
|
||||
LANGBOT_PIPELINE_NAME=
|
||||
|
||||
# Optional fake OpenAI-compatible provider controls for Debug Chat load tests.
|
||||
# Leave URL empty to let setup automation start a local provider and write the
|
||||
# selected URL to skills/.env.local.
|
||||
LANGBOT_FAKE_PROVIDER_URL=
|
||||
LANGBOT_FAKE_PROVIDER_HOST=127.0.0.1
|
||||
LANGBOT_FAKE_PROVIDER_PORT=
|
||||
LANGBOT_FAKE_PROVIDER_MODEL_NAME=gpt-4o-mini
|
||||
LANGBOT_FAKE_PROVIDER_RESPONSE_TEXT=OK
|
||||
LANGBOT_FAKE_PROVIDER_FIRST_TOKEN_DELAY_MS=25
|
||||
LANGBOT_FAKE_PROVIDER_CHUNK_DELAY_MS=10
|
||||
LANGBOT_FAKE_PROVIDER_CHUNK_COUNT=0
|
||||
LANGBOT_FAKE_PROVIDER_FAIL_FIRST_N=0
|
||||
LANGBOT_FAKE_PROVIDER_FAIL_EVERY_N=0
|
||||
LANGBOT_FAKE_PROVIDER_FAULT_STATUS=500
|
||||
LANGBOT_FAKE_PROVIDER_FAIL_AFTER_FIRST_CHUNK=false
|
||||
LANGBOT_FAKE_PROVIDER_DYNAMIC_RESPONSE=true
|
||||
|
||||
# Optional case-specific runner targets. Prefer these for runner-specific cases
|
||||
# so the automation cannot silently test the wrong runner.
|
||||
LANGBOT_LOCAL_AGENT_PIPELINE_URL=
|
||||
|
||||
@@ -53,7 +53,7 @@ Start the new frontend from the web repo:
|
||||
|
||||
```bash
|
||||
cd "$LANGBOT_WEB_REPO"
|
||||
VITE_API_BASE_URL="$LANGBOT_BACKEND_URL" pnpm dev --host 0.0.0.0
|
||||
npm run dev
|
||||
```
|
||||
|
||||
Healthy startup includes:
|
||||
@@ -68,10 +68,6 @@ Quick check:
|
||||
curl -I --max-time 3 "$LANGBOT_FRONTEND_URL"
|
||||
```
|
||||
|
||||
If `VITE_API_BASE_URL` is missing, Vite still serves the page but frontend API
|
||||
calls may go to the frontend port instead of the backend port. That produces
|
||||
false browser failures in login, wizard, pipeline, and Debug Chat cases.
|
||||
|
||||
## Completion Signal
|
||||
|
||||
Environment setup is not complete until the required frontend/backend URLs are reachable and the chosen browser-control path can open the WebUI.
|
||||
|
||||
@@ -42,38 +42,6 @@ MyPlugin/
|
||||
|
||||
Each component has a `.yaml` (metadata) and `.py` (implementation).
|
||||
|
||||
## README & i18n convention (enforced on the marketplace)
|
||||
|
||||
A plugin published to LangBot Space serves a localized README on its detail page.
|
||||
The resolver (`langbot-space` `PluginService.GetPluginREADME`) works like this:
|
||||
|
||||
- **Root `README.md` MUST be in English.** It is the default and the fallback —
|
||||
when no per-language README matches the viewer's locale, the page serves the
|
||||
root `README.md`. A non-English root README makes the English/default view show
|
||||
the wrong language.
|
||||
- **All other languages live under `readme/README_{lang}.md`** — e.g.
|
||||
`readme/README_zh_Hans.md`, `readme/README_ja_JP.md`. The 8 supported locales:
|
||||
`en_US, zh_Hans, zh_Hant, ja_JP, th_TH, vi_VN, es_ES, ru_RU`.
|
||||
- `manifest.yaml` `metadata.label` / `metadata.description` should carry the same
|
||||
8-locale i18n set (`repository` must be a real, alive URL).
|
||||
|
||||
```
|
||||
MyPlugin/
|
||||
├── manifest.yaml
|
||||
├── README.md # English (default + fallback) — REQUIRED, must be English
|
||||
└── readme/
|
||||
├── README_zh_Hans.md
|
||||
├── README_zh_Hant.md
|
||||
├── README_ja_JP.md
|
||||
├── README_th_TH.md
|
||||
├── README_vi_VN.md
|
||||
├── README_es_ES.md
|
||||
└── README_ru_RU.md
|
||||
```
|
||||
|
||||
`manifest.yaml` (incl. `repository`) is the source of truth — the marketplace
|
||||
syncs from it, so edit the package and re-publish rather than patching live data.
|
||||
|
||||
## Critical SDK Pitfalls
|
||||
|
||||
### 1. MessageChain is a RootModel — iterate directly
|
||||
@@ -471,7 +439,7 @@ async def on_msg(event_context: context.EventContext):
|
||||
if isinstance(component, platform_message.Plain):
|
||||
text_parts.append(component.text)
|
||||
text = "".join(text_parts).strip()
|
||||
|
||||
|
||||
if should_handle(text):
|
||||
event_context.prevent_default()
|
||||
event_context.prevent_postorder()
|
||||
|
||||
@@ -21,7 +21,6 @@ Use this skill when an agent needs to verify LangBot behavior through the WebUI
|
||||
- **Sandbox-backed skill authoring**: read `references/sandbox-skill-authoring.md`.
|
||||
- **LangRAG knowledge bases**: read `references/langrag-knowledge-base.md`.
|
||||
- **MCP stdio tool testing**: read `references/mcp-stdio-testing.md`.
|
||||
- **Performance, reliability, or chaos probes**: read `references/performance-reliability-testing.md`.
|
||||
- **Drive a live instance over MCP (not raw HTTP)**: use the `langbot-mcp-ops` skill — the instance exposes an MCP server at `http://<host>:5300/mcp` (reuses API keys). Useful for setting up bots/pipelines/models as test fixtures programmatically.
|
||||
- **Known failures and fixes**: read `references/troubleshooting.md`.
|
||||
- **Reusable test groups**: run `bin/lbs suite list` and `bin/lbs suite plan <suite-id>` before manually assembling a case set.
|
||||
@@ -37,8 +36,6 @@ Use this skill when an agent needs to verify LangBot behavior through the WebUI
|
||||
- Use an authenticated browser profile prepared by `langbot-env-setup`.
|
||||
- Do not expose API keys, OAuth secrets, tokens, or localStorage token values in output.
|
||||
- A WebUI test is not complete until the visible UI result is checked against backend logs or network behavior.
|
||||
- A performance result is not complete without `metrics` evidence and a clear split between LangBot overhead and external provider/tool/network time.
|
||||
- A chaos or reliability result is not complete until the fault scope, cleanup, and recovery checks are recorded.
|
||||
- For a suite, use `bin/lbs suite start <suite-id>` to create the suite evidence root, per-case directories, and `suite-start.json`/`suite-start.md` handoff files; use `bin/lbs test result <case-id>` to write final per-case `result.json`, then run `bin/lbs suite report <suite-id> --evidence-dir <dir>`.
|
||||
- Do not mark a case `pass` until `test result --evidence` covers every value in the case's `evidence_required`.
|
||||
- For runner-specific Debug Chat cases, use the case-specific pipeline env declared by `automation_pipeline_url_env` / `automation_pipeline_name_env`; do not silently reuse a generic `LANGBOT_PIPELINE_URL`.
|
||||
|
||||
-84
@@ -1,84 +0,0 @@
|
||||
id: langbot-fake-provider-debug-chat-cross-pipeline-isolation
|
||||
title: "LangBot Debug Chat fake-provider cross-pipeline isolation probe"
|
||||
mode: probe
|
||||
area: reliability
|
||||
type: reliability
|
||||
priority: p1
|
||||
risk: high
|
||||
ci_eligible: false
|
||||
tags:
|
||||
- reliability
|
||||
- debug-chat
|
||||
- websocket
|
||||
- fake-provider
|
||||
- isolation
|
||||
- concurrency
|
||||
- metrics
|
||||
skills:
|
||||
- langbot-env-setup
|
||||
- langbot-testing
|
||||
env:
|
||||
- LANGBOT_BACKEND_URL
|
||||
- LANGBOT_FRONTEND_URL
|
||||
- LANGBOT_E2E_LOGIN_USER
|
||||
automation: skills/langbot-testing/probes/langbot-debug-chat-cross-pipeline-isolation.mjs
|
||||
automation_env:
|
||||
- LANGBOT_BACKEND_URL
|
||||
- LANGBOT_E2E_LOGIN_USER
|
||||
- LANGBOT_FAKE_PROVIDER_URL
|
||||
- LANGBOT_FAKE_PROVIDER_PIPELINE_A_URL
|
||||
- LANGBOT_FAKE_PROVIDER_PIPELINE_A_NAME
|
||||
- LANGBOT_FAKE_PROVIDER_PIPELINE_B_URL
|
||||
- LANGBOT_FAKE_PROVIDER_PIPELINE_B_NAME
|
||||
automation_debug_chat_load_requests: "6"
|
||||
automation_debug_chat_load_concurrency: "4"
|
||||
automation_debug_chat_load_timeout_ms: "30000"
|
||||
automation_debug_chat_load_response_p95_ms: "5000"
|
||||
automation_debug_chat_load_max_error_rate: "0"
|
||||
automation_debug_chat_load_prompt_template: '请只回复 "{expected}"，不要解释，不要添加其他字符。'
|
||||
automation_debug_chat_load_stream: "true"
|
||||
automation_debug_chat_load_reset: "true"
|
||||
metrics_thresholds_json: '{"cross_pipeline_leak_count":{"max":0},"response_p95_ms":{"max":5000},"error_rate":{"max":0}}'
|
||||
load_profile_json: '{"requests_per_pipeline":6,"pipelines":2,"concurrency":4,"path":"Pipeline Debug Chat WebSocket","provider":"controlled fake OpenAI-compatible provider","metric":"cross-pipeline response isolation and send-to-final-assistant-response"}'
|
||||
setup_automation:
|
||||
- "node:scripts/e2e/ensure-fake-provider-cross-pipelines.mjs --write-env"
|
||||
setup_provides_env:
|
||||
- LANGBOT_FAKE_PROVIDER_URL
|
||||
- LANGBOT_FAKE_PROVIDER_BASE_URL
|
||||
- LANGBOT_FAKE_PROVIDER_PID
|
||||
- LANGBOT_FAKE_PROVIDER_PIPELINE_A_URL
|
||||
- LANGBOT_FAKE_PROVIDER_PIPELINE_A_NAME
|
||||
- LANGBOT_FAKE_PROVIDER_PIPELINE_B_URL
|
||||
- LANGBOT_FAKE_PROVIDER_PIPELINE_B_NAME
|
||||
steps:
|
||||
- "Start or reuse the local fake OpenAI-compatible provider."
|
||||
- "Create or update two local-agent pipelines that both point at the controlled fake provider."
|
||||
- "Reset both Debug Chat sessions and the fake-provider request log."
|
||||
- "Open concurrent WebSocket Debug Chat connections to both pipelines and send unique pipeline-scoped response tokens."
|
||||
checks:
|
||||
- "automation-result.json status is pass only when every request receives its own expected token and cross_pipeline_leak_count is zero."
|
||||
- "metrics_summary includes by_pipeline status counts, fake-provider request count, and LangBot/provider timing estimates."
|
||||
- "samples.json contains per-request pipeline labels so any leak can be attributed to the receiving pipeline."
|
||||
evidence_required:
|
||||
- metrics
|
||||
- network
|
||||
- api_diagnostic
|
||||
- filesystem
|
||||
diagnostics:
|
||||
- "This probe targets Debug Chat isolation under concurrent traffic from two pipelines."
|
||||
- "It is designed to expose regressions where global pipeline state causes one pipeline's assistant response to be delivered to another pipeline's Debug Chat session."
|
||||
- "Same-pipeline foreign responses are tolerated because Debug Chat intentionally broadcasts within the same pipeline/session; cross-pipeline tokens are never tolerated."
|
||||
- "Known product bug: current releases may fail this probe because Debug Chat replies can read singleton WebSocket proxy pipeline state after another pipeline overwrites it. See https://github.com/langbot-app/LangBot/issues/2286."
|
||||
expected_failures:
|
||||
- "https://github.com/langbot-app/LangBot/issues/2286"
|
||||
success_patterns:
|
||||
- "Debug Chat cross-pipeline isolation probe passed"
|
||||
failure_patterns:
|
||||
- "cross_pipeline_leak"
|
||||
- "Timed out after"
|
||||
- "WebSocket connection error"
|
||||
- "Final assistant response did not include"
|
||||
troubleshooting:
|
||||
- backend-not-listening
|
||||
- debug-chat-history-contaminates-automation
|
||||
- local-agent-model-route-unavailable
|
||||
-95
@@ -1,95 +0,0 @@
|
||||
id: langbot-fake-provider-debug-chat-fault-recovery
|
||||
title: "LangBot Debug Chat fake-provider fault recovery probe"
|
||||
mode: probe
|
||||
area: reliability
|
||||
type: chaos
|
||||
priority: p1
|
||||
risk: high
|
||||
ci_eligible: false
|
||||
tags:
|
||||
- reliability
|
||||
- chaos
|
||||
- debug-chat
|
||||
- websocket
|
||||
- fake-provider
|
||||
- fault-injection
|
||||
- metrics
|
||||
skills:
|
||||
- langbot-env-setup
|
||||
- langbot-testing
|
||||
env:
|
||||
- LANGBOT_BACKEND_URL
|
||||
- LANGBOT_FRONTEND_URL
|
||||
- LANGBOT_E2E_LOGIN_USER
|
||||
automation: skills/langbot-testing/probes/langbot-debug-chat-concurrency.mjs
|
||||
automation_env:
|
||||
- LANGBOT_BACKEND_URL
|
||||
- LANGBOT_E2E_LOGIN_USER
|
||||
- LANGBOT_FAKE_PROVIDER_PIPELINE_URL
|
||||
- LANGBOT_FAKE_PROVIDER_PIPELINE_NAME
|
||||
automation_pipeline_url_env: LANGBOT_FAKE_PROVIDER_PIPELINE_URL
|
||||
automation_pipeline_name_env: LANGBOT_FAKE_PROVIDER_PIPELINE_NAME
|
||||
automation_debug_chat_load_requests: "6"
|
||||
automation_debug_chat_load_concurrency: "1"
|
||||
automation_debug_chat_load_timeout_ms: "15000"
|
||||
automation_debug_chat_load_response_p95_ms: "5000"
|
||||
automation_debug_chat_load_max_error_rate: "0"
|
||||
automation_debug_chat_load_min_ok_count: "6"
|
||||
automation_debug_chat_load_min_provider_fault_count: "2"
|
||||
automation_debug_chat_load_expected_prefix: "FAULTQA"
|
||||
automation_debug_chat_load_prompt_template: '请只回复 "{expected}"，不要解释，不要添加其他字符。'
|
||||
automation_debug_chat_load_stream: "true"
|
||||
automation_debug_chat_load_reset: "true"
|
||||
automation_debug_chat_load_fail_on_final_mismatch: "true"
|
||||
automation_fake_provider_first_token_delay_ms: "25"
|
||||
automation_fake_provider_chunk_delay_ms: "10"
|
||||
automation_fake_provider_chunk_count: "0"
|
||||
automation_fake_provider_fail_first_n: "2"
|
||||
automation_fake_provider_fail_every_n: "0"
|
||||
automation_fake_provider_fault_status: "503"
|
||||
metrics_thresholds_json: '{"response_p95_ms":{"max":5000},"error_rate":{"max":0},"ok_count_min":{"min":6},"fake_provider_fault_count_min":{"min":2}}'
|
||||
fault_model_json: '{"provider_fault":"HTTP 503 for first 2 fake-provider chat completions after reset","expected_behavior":"LangBot retries or otherwise recovers from bounded provider failures so every Debug Chat request receives its expected response without backend crash."}'
|
||||
load_profile_json: '{"requests":6,"concurrency":1,"path":"Pipeline Debug Chat WebSocket","provider":"controlled fake OpenAI-compatible provider","classification":"fault-recovery-not-throughput-benchmark"}'
|
||||
setup_automation:
|
||||
- "node:scripts/e2e/ensure-fake-provider-pipeline.mjs --write-env"
|
||||
setup_provides_env:
|
||||
- LANGBOT_FAKE_PROVIDER_URL
|
||||
- LANGBOT_FAKE_PROVIDER_BASE_URL
|
||||
- LANGBOT_FAKE_PROVIDER_PID
|
||||
- LANGBOT_FAKE_PROVIDER_PROVIDER_UUID
|
||||
- LANGBOT_FAKE_PROVIDER_MODEL_UUID
|
||||
- LANGBOT_FAKE_PROVIDER_PIPELINE_URL
|
||||
- LANGBOT_FAKE_PROVIDER_PIPELINE_NAME
|
||||
steps:
|
||||
- "Configure the local fake provider to return HTTP 503 for the first two chat completions after reset."
|
||||
- "Create or update the LangBot provider, model, and local-agent pipeline that points at the fake provider."
|
||||
- "Reset the target Debug Chat session and fake-provider request counter."
|
||||
- "Send a sequential Debug Chat batch and verify later requests recover after the injected provider faults."
|
||||
checks:
|
||||
- "automation-result.json status is pass when the fake provider records at least two injected faults, every Debug Chat request succeeds, and total user-visible error rate stays at zero."
|
||||
- "metrics_summary includes fake_provider_fault_count and status_counts for the same run window."
|
||||
- "backend logs show request handling for the same run window without unexpected Traceback or task-leak findings."
|
||||
evidence_required:
|
||||
- metrics
|
||||
- network
|
||||
- api_diagnostic
|
||||
- filesystem
|
||||
diagnostics:
|
||||
- "This is a fault-recovery probe, not a throughput benchmark."
|
||||
- "Provider faults may be retried inside the provider/requester path; judge this case by fake_provider_fault_count plus user-visible success/error metrics."
|
||||
- "The profile uses concurrency 1 because Debug Chat broadcasts assistant responses to every connection in a session, and failed responses do not carry the unique success token needed for concurrent attribution."
|
||||
success_patterns:
|
||||
- "Debug Chat WebSocket concurrency probe passed"
|
||||
- "Streaming completed"
|
||||
failure_patterns:
|
||||
- "fake_provider_fault"
|
||||
- "HTTP 503"
|
||||
- "Timed out after"
|
||||
- "All models failed during streaming setup"
|
||||
expected_failures:
|
||||
- "fake_provider_fault"
|
||||
- "HTTP 503"
|
||||
troubleshooting:
|
||||
- backend-not-listening
|
||||
- debug-chat-history-contaminates-automation
|
||||
- local-agent-model-route-unavailable
|
||||
@@ -1,81 +0,0 @@
|
||||
id: langbot-fake-provider-debug-chat-load
|
||||
title: "LangBot Debug Chat controlled fake-provider load probe"
|
||||
mode: probe
|
||||
area: performance
|
||||
type: performance
|
||||
priority: p1
|
||||
risk: medium
|
||||
ci_eligible: false
|
||||
tags:
|
||||
- performance
|
||||
- debug-chat
|
||||
- websocket
|
||||
- fake-provider
|
||||
- load
|
||||
- metrics
|
||||
skills:
|
||||
- langbot-env-setup
|
||||
- langbot-testing
|
||||
env:
|
||||
- LANGBOT_BACKEND_URL
|
||||
- LANGBOT_FRONTEND_URL
|
||||
- LANGBOT_E2E_LOGIN_USER
|
||||
automation: skills/langbot-testing/probes/langbot-debug-chat-concurrency.mjs
|
||||
automation_env:
|
||||
- LANGBOT_BACKEND_URL
|
||||
- LANGBOT_E2E_LOGIN_USER
|
||||
- LANGBOT_FAKE_PROVIDER_PIPELINE_URL
|
||||
- LANGBOT_FAKE_PROVIDER_PIPELINE_NAME
|
||||
automation_pipeline_url_env: LANGBOT_FAKE_PROVIDER_PIPELINE_URL
|
||||
automation_pipeline_name_env: LANGBOT_FAKE_PROVIDER_PIPELINE_NAME
|
||||
automation_debug_chat_load_requests: "12"
|
||||
automation_debug_chat_load_concurrency: "4"
|
||||
automation_debug_chat_load_timeout_ms: "30000"
|
||||
automation_debug_chat_load_response_p95_ms: "5000"
|
||||
automation_debug_chat_load_first_response_p95_ms: "3000"
|
||||
automation_debug_chat_load_max_error_rate: "0"
|
||||
automation_debug_chat_load_expected_prefix: "FAKEQA"
|
||||
automation_debug_chat_load_prompt_template: '请只回复 "{expected}"，不要解释，不要添加其他字符。'
|
||||
automation_debug_chat_load_stream: "true"
|
||||
automation_debug_chat_load_reset: "true"
|
||||
metrics_thresholds_json: '{"response_p95_ms":{"max":5000},"first_response_p95_ms":{"max":3000},"error_rate":{"max":0}}'
|
||||
load_profile_json: '{"requests":12,"concurrency":4,"path":"Pipeline Debug Chat WebSocket","provider":"controlled fake OpenAI-compatible provider","metric":"send-to-final-assistant-response"}'
|
||||
setup_automation:
|
||||
- "node:scripts/e2e/ensure-fake-provider-pipeline.mjs --write-env"
|
||||
setup_provides_env:
|
||||
- LANGBOT_FAKE_PROVIDER_URL
|
||||
- LANGBOT_FAKE_PROVIDER_BASE_URL
|
||||
- LANGBOT_FAKE_PROVIDER_PID
|
||||
- LANGBOT_FAKE_PROVIDER_PROVIDER_UUID
|
||||
- LANGBOT_FAKE_PROVIDER_MODEL_UUID
|
||||
- LANGBOT_FAKE_PROVIDER_PIPELINE_URL
|
||||
- LANGBOT_FAKE_PROVIDER_PIPELINE_NAME
|
||||
steps:
|
||||
- "Start or reuse the local fake OpenAI-compatible provider."
|
||||
- "Create or update the LangBot provider, model, and local-agent pipeline that points at the fake provider."
|
||||
- "Reset the target Debug Chat session."
|
||||
- "Open concurrent WebSocket Debug Chat connections and send unique deterministic prompts through the real backend pipeline."
|
||||
checks:
|
||||
- "automation-result.json status is pass when every request receives its own expected assistant response."
|
||||
- "metrics_summary includes request count, concurrency, p50/p95 response latency, first response latency, throughput, and error rate."
|
||||
- "thresholds_summary shows response_p95_ms, first_response_p95_ms, and error_rate pass."
|
||||
evidence_required:
|
||||
- metrics
|
||||
- network
|
||||
- api_diagnostic
|
||||
- filesystem
|
||||
diagnostics:
|
||||
- "This probe removes external model latency from the measurement; it still exercises the live LangBot backend, provider requester, local-agent runner, pipeline, and Debug Chat WebSocket adapter."
|
||||
- "Use this as the repeatable message-path baseline before comparing against Space or another real provider."
|
||||
success_patterns:
|
||||
- "Debug Chat WebSocket concurrency probe passed"
|
||||
- "Streaming completed"
|
||||
failure_patterns:
|
||||
- "WebSocket connection error"
|
||||
- "Timed out after"
|
||||
- "Final assistant response did not include"
|
||||
- "All models failed during streaming setup"
|
||||
troubleshooting:
|
||||
- backend-not-listening
|
||||
- debug-chat-history-contaminates-automation
|
||||
- local-agent-model-route-unavailable
|
||||
@@ -1,88 +0,0 @@
|
||||
id: langbot-fake-provider-debug-chat-slow-load
|
||||
title: "LangBot Debug Chat slow fake-provider load probe"
|
||||
mode: probe
|
||||
area: performance
|
||||
type: performance
|
||||
priority: p1
|
||||
risk: medium
|
||||
ci_eligible: false
|
||||
tags:
|
||||
- performance
|
||||
- debug-chat
|
||||
- websocket
|
||||
- fake-provider
|
||||
- slow-provider
|
||||
- load
|
||||
- metrics
|
||||
skills:
|
||||
- langbot-env-setup
|
||||
- langbot-testing
|
||||
env:
|
||||
- LANGBOT_BACKEND_URL
|
||||
- LANGBOT_FRONTEND_URL
|
||||
- LANGBOT_E2E_LOGIN_USER
|
||||
automation: skills/langbot-testing/probes/langbot-debug-chat-concurrency.mjs
|
||||
automation_env:
|
||||
- LANGBOT_BACKEND_URL
|
||||
- LANGBOT_E2E_LOGIN_USER
|
||||
- LANGBOT_FAKE_PROVIDER_PIPELINE_URL
|
||||
- LANGBOT_FAKE_PROVIDER_PIPELINE_NAME
|
||||
automation_pipeline_url_env: LANGBOT_FAKE_PROVIDER_PIPELINE_URL
|
||||
automation_pipeline_name_env: LANGBOT_FAKE_PROVIDER_PIPELINE_NAME
|
||||
automation_debug_chat_load_requests: "8"
|
||||
automation_debug_chat_load_concurrency: "4"
|
||||
automation_debug_chat_load_timeout_ms: "45000"
|
||||
automation_debug_chat_load_response_p95_ms: "10000"
|
||||
automation_debug_chat_load_first_response_p95_ms: "7000"
|
||||
automation_debug_chat_load_max_error_rate: "0"
|
||||
automation_debug_chat_load_expected_prefix: "SLOWQA"
|
||||
automation_debug_chat_load_prompt_template: '请只回复 "{expected}"，不要解释，不要添加其他字符。'
|
||||
automation_debug_chat_load_stream: "true"
|
||||
automation_debug_chat_load_reset: "true"
|
||||
automation_fake_provider_first_token_delay_ms: "1000"
|
||||
automation_fake_provider_chunk_delay_ms: "250"
|
||||
automation_fake_provider_chunk_count: "4"
|
||||
automation_fake_provider_fail_first_n: "0"
|
||||
automation_fake_provider_fail_every_n: "0"
|
||||
automation_fake_provider_fault_status: "500"
|
||||
metrics_thresholds_json: '{"response_p95_ms":{"max":10000},"first_response_p95_ms":{"max":7000},"error_rate":{"max":0}}'
|
||||
load_profile_json: '{"requests":8,"concurrency":4,"path":"Pipeline Debug Chat WebSocket","provider":"controlled slow fake OpenAI-compatible provider","metric":"send-to-final-assistant-response","provider_profile":{"first_token_delay_ms":1000,"chunk_delay_ms":250,"chunk_count":4}}'
|
||||
setup_automation:
|
||||
- "node:scripts/e2e/ensure-fake-provider-pipeline.mjs --write-env"
|
||||
setup_provides_env:
|
||||
- LANGBOT_FAKE_PROVIDER_URL
|
||||
- LANGBOT_FAKE_PROVIDER_BASE_URL
|
||||
- LANGBOT_FAKE_PROVIDER_PID
|
||||
- LANGBOT_FAKE_PROVIDER_PROVIDER_UUID
|
||||
- LANGBOT_FAKE_PROVIDER_MODEL_UUID
|
||||
- LANGBOT_FAKE_PROVIDER_PIPELINE_URL
|
||||
- LANGBOT_FAKE_PROVIDER_PIPELINE_NAME
|
||||
steps:
|
||||
- "Configure the local fake provider with deterministic slow streaming latency."
|
||||
- "Create or update the LangBot provider, model, and local-agent pipeline that points at the fake provider."
|
||||
- "Reset the target Debug Chat session."
|
||||
- "Open concurrent WebSocket Debug Chat connections and send unique deterministic prompts through the real backend pipeline."
|
||||
checks:
|
||||
- "automation-result.json status is pass when every request receives its own expected assistant response."
|
||||
- "metrics_summary shows zero errors under the slow-provider profile."
|
||||
- "thresholds_summary shows response_p95_ms, first_response_p95_ms, and error_rate pass."
|
||||
evidence_required:
|
||||
- metrics
|
||||
- network
|
||||
- api_diagnostic
|
||||
- filesystem
|
||||
diagnostics:
|
||||
- "This probe keeps the model deterministic while injecting provider latency, so it catches backend timeout, streaming, and WebSocket backpressure issues without Space variability."
|
||||
- "Compare with langbot-fake-provider-debug-chat-load to separate fixed LangBot overhead from provider-latency amplification."
|
||||
success_patterns:
|
||||
- "Debug Chat WebSocket concurrency probe passed"
|
||||
- "Streaming completed"
|
||||
failure_patterns:
|
||||
- "WebSocket connection error"
|
||||
- "Timed out after"
|
||||
- "Final assistant response did not include"
|
||||
- "All models failed during streaming setup"
|
||||
troubleshooting:
|
||||
- backend-not-listening
|
||||
- debug-chat-history-contaminates-automation
|
||||
- local-agent-model-route-unavailable
|
||||
@@ -1,35 +0,0 @@
|
||||
id: langbot-fault-taxonomy-contract
|
||||
title: "LangBot fault taxonomy and cleanup contract"
|
||||
mode: probe
|
||||
area: reliability
|
||||
type: chaos
|
||||
priority: p1
|
||||
risk: medium
|
||||
ci_eligible: true
|
||||
tags:
|
||||
- reliability
|
||||
- chaos
|
||||
- contract
|
||||
- synthetic
|
||||
skills:
|
||||
- langbot-testing
|
||||
automation: skills/langbot-testing/probes/langbot-fault-taxonomy-contract.mjs
|
||||
fault_model_json: '{"kind":"taxonomy-contract","destructive":false,"scenarios":["provider-timeout","plugin-runtime-disconnect","mcp-stdio-server-exit","operator-missing-login","transient-marketplace-timeout"]}'
|
||||
steps:
|
||||
- "Run `rtk bin/lbs test run langbot-fault-taxonomy-contract --dry-run` first; remove `--dry-run` after checking the evidence directory."
|
||||
- "Automation validates that representative fault scenarios declare target, injected fault, expected status, recovery check, and cleanup."
|
||||
- "Review metrics.json, fault-model.json, and automation-result.json under LBS_EVIDENCE_DIR."
|
||||
checks:
|
||||
- "automation-result.json status is pass."
|
||||
- "Every scenario has an expected status in pass, fail, blocked, env_issue, or flaky."
|
||||
- "Every scenario declares a cleanup action and recovery check."
|
||||
evidence_required:
|
||||
- metrics
|
||||
- filesystem
|
||||
diagnostics:
|
||||
- "This is a non-destructive taxonomy contract probe; it does not inject real runtime faults."
|
||||
- "Use it as a gate before adding live chaos cases that kill runtimes, route traffic through a proxy, or disrupt a backend dependency."
|
||||
success_patterns:
|
||||
- "Fault taxonomy contract declares status"
|
||||
failure_patterns:
|
||||
- "missing required scenario fields"
|
||||
@@ -1,42 +0,0 @@
|
||||
id: langbot-live-backend-latency
|
||||
title: "LangBot live backend basic latency probe"
|
||||
mode: probe
|
||||
area: performance
|
||||
type: performance
|
||||
priority: p1
|
||||
risk: medium
|
||||
ci_eligible: false
|
||||
tags:
|
||||
- performance
|
||||
- live-backend
|
||||
- latency
|
||||
- metrics
|
||||
skills:
|
||||
- langbot-testing
|
||||
env:
|
||||
- LANGBOT_BACKEND_URL
|
||||
automation: skills/langbot-testing/probes/langbot-live-backend-latency.mjs
|
||||
metrics_thresholds_json: '{"backend_p95_ms":{"max":1000},"error_rate":{"max":0}}'
|
||||
load_profile_json: '{"requests":12,"concurrency":2,"endpoints":["/healthz"]}'
|
||||
steps:
|
||||
- "Confirm the selected LangBot backend is the intended test target."
|
||||
- "Run `rtk bin/lbs test run langbot-live-backend-latency --dry-run` first; remove `--dry-run` after checking LANGBOT_BACKEND_URL and evidence directory."
|
||||
- "Automation sends a small request batch to LANGBOT_BACKEND_URL/healthz and records latency, status counts, and network errors."
|
||||
checks:
|
||||
- "automation-result.json status is pass when the backend responds and p95/error-rate thresholds pass."
|
||||
- "automation-result.json status is env_issue when the backend is not reachable."
|
||||
- "metrics.json and network.log are written under LBS_EVIDENCE_DIR."
|
||||
evidence_required:
|
||||
- metrics
|
||||
- network
|
||||
- api_diagnostic
|
||||
- filesystem
|
||||
diagnostics:
|
||||
- "This probe measures backend health endpoint reachability latency only; it does not cover model/provider, browser, Debug Chat, RAG, or plugin runtime latency."
|
||||
success_patterns:
|
||||
- "Live backend latency probe passed"
|
||||
failure_patterns:
|
||||
- "Backend did not respond"
|
||||
- "breached latency or error-rate thresholds"
|
||||
troubleshooting:
|
||||
- socks-proxy-without-socksio
|
||||
@@ -1,45 +0,0 @@
|
||||
id: langbot-live-backend-log-health
|
||||
title: "LangBot live backend log health probe"
|
||||
mode: probe
|
||||
area: reliability
|
||||
type: reliability
|
||||
priority: p1
|
||||
risk: medium
|
||||
ci_eligible: false
|
||||
tags:
|
||||
- reliability
|
||||
- live-backend
|
||||
- backend-log
|
||||
- metrics
|
||||
skills:
|
||||
- langbot-testing
|
||||
env:
|
||||
- LANGBOT_BACKEND_URL
|
||||
automation: skills/langbot-testing/probes/langbot-live-backend-log-health.mjs
|
||||
metrics_thresholds_json: '{"fail_count":{"max":0}}'
|
||||
load_profile_json: '{"lookback_seconds":300,"log_source":"LANGBOT_BACKEND_LOG or latest LANGBOT_REPO/data/logs/langbot-*.log"}'
|
||||
steps:
|
||||
- "Confirm the selected LangBot backend log belongs to the intended test target."
|
||||
- "Run `rtk bin/lbs test run langbot-live-backend-log-health --dry-run` first; remove `--dry-run` after checking evidence directory and log source."
|
||||
- "Automation scans the recent backend log window for fail-severity runtime findings such as Traceback, ImportError, ERROR, unclosed sessions, and unawaited coroutines."
|
||||
checks:
|
||||
- "automation-result.json status is pass only when fail_count is 0."
|
||||
- "metrics_summary includes scanned_line_count, fail_count, warning_count, and finding_count."
|
||||
- "findings.json and scanned-backend.log are written under LBS_EVIDENCE_DIR."
|
||||
evidence_required:
|
||||
- metrics
|
||||
- backend_log
|
||||
- filesystem
|
||||
diagnostics:
|
||||
- "Set LANGBOT_BACKEND_LOG to an explicit log path when the latest log file is not the run target."
|
||||
- "Set LANGBOT_BACKEND_LOG_SINCE or LANGBOT_BACKEND_LOG_LOOKBACK_SECONDS to control the scan window."
|
||||
- "This probe measures runtime log health; it does not prove user-facing Debug Chat, plugin, model, or RAG behavior."
|
||||
success_patterns:
|
||||
- "Live backend log health passed"
|
||||
failure_patterns:
|
||||
- "Traceback"
|
||||
- "ImportError"
|
||||
- "ERROR"
|
||||
- "unclosed"
|
||||
troubleshooting:
|
||||
- socks-proxy-without-socksio
|
||||
@@ -1,44 +0,0 @@
|
||||
id: langbot-live-control-plane-api
|
||||
title: "LangBot live control-plane API probe"
|
||||
mode: probe
|
||||
area: performance
|
||||
type: performance
|
||||
priority: p1
|
||||
risk: medium
|
||||
ci_eligible: false
|
||||
tags:
|
||||
- performance
|
||||
- reliability
|
||||
- live-backend
|
||||
- control-plane
|
||||
- metrics
|
||||
skills:
|
||||
- langbot-testing
|
||||
env:
|
||||
- LANGBOT_BACKEND_URL
|
||||
automation: skills/langbot-testing/probes/langbot-live-control-plane-api.mjs
|
||||
metrics_thresholds_json: '{"error_rate":{"max":0},"response_shape_failures":{"max":0},"healthz_p95_ms":{"max":500},"system_info_p95_ms":{"max":1000}}'
|
||||
load_profile_json: '{"requests":20,"concurrency":4,"endpoints":["/healthz","/api/v1/system/info"],"auth_required":false}'
|
||||
steps:
|
||||
- "Confirm the selected LangBot backend is the intended test target."
|
||||
- "Run `rtk bin/lbs test run langbot-live-control-plane-api --dry-run` first; remove `--dry-run` after checking LANGBOT_BACKEND_URL and evidence directory."
|
||||
- "Automation sends a small request batch to /healthz and /api/v1/system/info, then validates status code, JSON shape, and latency budgets."
|
||||
checks:
|
||||
- "automation-result.json status is pass when every control-plane request returns HTTP 200, JSON code 0, and required response fields."
|
||||
- "metrics_summary includes per-endpoint p50/p95 latency, error rate, status counts, and response_shape_failures."
|
||||
- "thresholds_summary shows error_rate, response_shape_failures, healthz_p95_ms, and system_info_p95_ms all pass."
|
||||
evidence_required:
|
||||
- metrics
|
||||
- network
|
||||
- api_diagnostic
|
||||
- filesystem
|
||||
diagnostics:
|
||||
- "This probe measures unauthenticated backend control-plane readiness; it does not cover authenticated UI flows, Debug Chat, model calls, plugins, or RAG."
|
||||
- "A system_info shape failure usually means the API contract or startup state changed and should be investigated before treating latency as healthy."
|
||||
success_patterns:
|
||||
- "Live control-plane API probe passed"
|
||||
failure_patterns:
|
||||
- "Backend did not respond"
|
||||
- "breached shape, latency, or error-rate thresholds"
|
||||
troubleshooting:
|
||||
- socks-proxy-without-socksio
|
||||
@@ -1,37 +0,0 @@
|
||||
id: langbot-overhead-accounting-contract
|
||||
title: "LangBot overhead accounting metrics contract"
|
||||
mode: probe
|
||||
area: performance
|
||||
type: performance
|
||||
priority: p1
|
||||
risk: medium
|
||||
ci_eligible: true
|
||||
tags:
|
||||
- performance
|
||||
- metrics
|
||||
- contract
|
||||
- synthetic
|
||||
skills:
|
||||
- langbot-testing
|
||||
automation: skills/langbot-testing/probes/langbot-overhead-accounting-contract.mjs
|
||||
metrics_thresholds_json: '{"sample_count":{"min":50},"langbot_overhead_p95_ms":{"max":25},"accounting_gap_max_ms":{"max":0.001}}'
|
||||
load_profile_json: '{"kind":"synthetic-overhead-accounting","samples":80,"external_latency_segments":["provider","external_tool","network"]}'
|
||||
steps:
|
||||
- "Run `rtk bin/lbs test run langbot-overhead-accounting-contract --dry-run` first; remove `--dry-run` after checking the evidence directory."
|
||||
- "Automation generates deterministic message-path latency samples and separates LangBot overhead from provider/tool/network latency."
|
||||
- "Review metrics.json, thresholds.json, resource-log.json, and automation-result.json under LBS_EVIDENCE_DIR."
|
||||
checks:
|
||||
- "automation-result.json status is pass."
|
||||
- "metrics_summary includes sample_count, langbot_overhead_p95_ms, e2e_latency_p95_ms, external_latency_p95_ms, and accounting_gap_max_ms."
|
||||
- "thresholds_summary shows sample_count, langbot_overhead_p95_ms, and accounting_gap_max_ms all pass."
|
||||
evidence_required:
|
||||
- metrics
|
||||
- resource_log
|
||||
- filesystem
|
||||
diagnostics:
|
||||
- "This is a synthetic contract probe for the QA harness; it is not live product performance."
|
||||
- "Use it to verify that reports can carry overhead accounting metrics before running live backend or browser performance probes."
|
||||
success_patterns:
|
||||
- "Overhead accounting contract passed"
|
||||
failure_patterns:
|
||||
- "breached one or more thresholds"
|
||||
@@ -1,84 +0,0 @@
|
||||
id: langbot-space-debug-chat-concurrency-smoke
|
||||
title: "LangBot Debug Chat real Space-provider concurrency smoke"
|
||||
mode: probe
|
||||
area: performance
|
||||
type: performance
|
||||
priority: p1
|
||||
risk: high
|
||||
ci_eligible: false
|
||||
tags:
|
||||
- performance
|
||||
- debug-chat
|
||||
- websocket
|
||||
- space
|
||||
- live-provider
|
||||
- smoke
|
||||
- metrics
|
||||
skills:
|
||||
- langbot-env-setup
|
||||
- langbot-testing
|
||||
env:
|
||||
- LANGBOT_BACKEND_URL
|
||||
- LANGBOT_FRONTEND_URL
|
||||
- LANGBOT_E2E_LOGIN_USER
|
||||
automation: skills/langbot-testing/probes/langbot-debug-chat-concurrency.mjs
|
||||
automation_env:
|
||||
- LANGBOT_BACKEND_URL
|
||||
- LANGBOT_E2E_LOGIN_USER
|
||||
- LANGBOT_LOCAL_AGENT_PIPELINE_URL
|
||||
- LANGBOT_LOCAL_AGENT_PIPELINE_NAME
|
||||
automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL
|
||||
automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME
|
||||
automation_debug_chat_load_requests: "3"
|
||||
automation_debug_chat_load_concurrency: "2"
|
||||
automation_debug_chat_load_timeout_ms: "120000"
|
||||
automation_debug_chat_load_response_p95_ms: "120000"
|
||||
automation_debug_chat_load_max_error_rate: "0"
|
||||
automation_debug_chat_load_expected_prefix: "SPACEQA"
|
||||
automation_debug_chat_load_prompt_template: '请只回复 "{expected}"，不要解释，不要添加其他字符。'
|
||||
automation_debug_chat_load_stream: "true"
|
||||
automation_debug_chat_load_reset: "true"
|
||||
metrics_thresholds_json: '{"response_p95_ms":{"max":120000},"error_rate":{"max":0}}'
|
||||
load_profile_json: '{"requests":3,"concurrency":2,"path":"Pipeline Debug Chat WebSocket","provider":"LangBot Space model route","metric":"send-to-final-assistant-response","classification":"smoke-not-benchmark"}'
|
||||
setup_automation:
|
||||
- "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env"
|
||||
setup_provides_env:
|
||||
- LANGBOT_PIPELINE_URL
|
||||
- LANGBOT_PIPELINE_NAME
|
||||
- LANGBOT_LOCAL_AGENT_PIPELINE_URL
|
||||
- LANGBOT_LOCAL_AGENT_PIPELINE_NAME
|
||||
- LANGBOT_LOCAL_AGENT_MODEL_UUID
|
||||
- LANGBOT_E2E_MODEL_UUID
|
||||
preconditions:
|
||||
- "The selected local LangBot instance is safe for a low-volume real Space model smoke run."
|
||||
- "Treat Space/provider/network failures as environment or dependency findings until fake-provider baseline evidence separates LangBot overhead."
|
||||
steps:
|
||||
- "Prepare a local-agent pipeline with a tested Space model and fallback models."
|
||||
- "Reset the target Debug Chat session."
|
||||
- "Open a small number of concurrent WebSocket Debug Chat connections and send unique deterministic prompts through the live Space provider path."
|
||||
checks:
|
||||
- "automation-result.json status is pass when every request receives its own expected assistant response."
|
||||
- "metrics_summary includes request count, concurrency, p95 response latency, throughput, and error rate."
|
||||
- "The report classifies the result as a live-provider smoke, not a stable LangBot overhead benchmark."
|
||||
evidence_required:
|
||||
- metrics
|
||||
- network
|
||||
- api_diagnostic
|
||||
- filesystem
|
||||
diagnostics:
|
||||
- "This probe measures real user-path latency through Space and includes provider latency, model behavior, and network effects."
|
||||
- "Compare with langbot-fake-provider-debug-chat-load before attributing slow or failed runs to LangBot itself."
|
||||
success_patterns:
|
||||
- "Debug Chat WebSocket concurrency probe passed"
|
||||
- "Streaming completed"
|
||||
failure_patterns:
|
||||
- "invalid api key"
|
||||
- "WebSocket connection error"
|
||||
- "Timed out after"
|
||||
- "Final assistant response did not include"
|
||||
- "All models failed during streaming setup"
|
||||
troubleshooting:
|
||||
- local-agent-model-route-unavailable
|
||||
- marketplace-network-flaky
|
||||
- proxy-env-mismatch
|
||||
- telemetry-proxy-noise
|
||||
@@ -1,80 +0,0 @@
|
||||
id: pipeline-debug-chat-performance
|
||||
title: "Pipeline Debug Chat user-path performance probe"
|
||||
mode: agent-browser
|
||||
area: pipeline
|
||||
type: performance
|
||||
priority: p1
|
||||
risk: medium
|
||||
ci_eligible: false
|
||||
tags:
|
||||
- performance
|
||||
- pipeline
|
||||
- debug-chat
|
||||
- user-path
|
||||
- metrics
|
||||
skills:
|
||||
- langbot-env-setup
|
||||
- langbot-testing
|
||||
env:
|
||||
- LANGBOT_FRONTEND_URL
|
||||
- LANGBOT_BACKEND_URL
|
||||
env_any:
|
||||
- LANGBOT_PIPELINE_URL|LANGBOT_PIPELINE_NAME
|
||||
automation: scripts/e2e/pipeline-debug-chat.mjs
|
||||
automation_env:
|
||||
- LANGBOT_FRONTEND_URL
|
||||
- LANGBOT_BACKEND_URL
|
||||
- LANGBOT_BROWSER_PROFILE
|
||||
- LANGBOT_CHROMIUM_EXECUTABLE
|
||||
- LANGBOT_E2E_PROMPT
|
||||
- LANGBOT_E2E_EXPECTED_TEXT
|
||||
- LANGBOT_E2E_RESPONSE_TIMEOUT_MS
|
||||
automation_env_any:
|
||||
- LANGBOT_PIPELINE_URL|LANGBOT_PIPELINE_NAME
|
||||
automation_prompt: "请只回复 OK，用于性能测试。"
|
||||
automation_expected_text: "OK"
|
||||
automation_response_timeout_ms: "120000"
|
||||
automation_reset_debug_chat: "true"
|
||||
automation_debug_chat_response_p95_ms: "120000"
|
||||
automation_debug_chat_max_error_rate: "0"
|
||||
metrics_thresholds_json: '{"response_p95_ms":{"max":120000},"error_rate":{"max":0}}'
|
||||
load_profile_json: '{"prompts":1,"browser":true,"path":"Pipeline Debug Chat","metric":"send-to-visible-completion"}'
|
||||
setup_automation:
|
||||
- "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env"
|
||||
setup_provides_env:
|
||||
- LANGBOT_PIPELINE_URL
|
||||
- LANGBOT_PIPELINE_NAME
|
||||
preconditions:
|
||||
- "LANGBOT_PIPELINE_URL or LANGBOT_PIPELINE_NAME points to the pipeline intended for this Debug Chat performance run."
|
||||
- "The target pipeline is safe to reset Debug Chat history for this run."
|
||||
- "The target pipeline has a known-good runner/model; provider latency should be interpreted separately from LangBot overhead."
|
||||
steps:
|
||||
- "Open LANGBOT_FRONTEND_URL with the prepared browser profile."
|
||||
- "Open the target pipeline and select Debug Chat."
|
||||
- "Reset Debug Chat history through the backend API when configured."
|
||||
- "Send the deterministic prompt and wait for the expected assistant response."
|
||||
checks:
|
||||
- "automation-result.json status is pass when the expected assistant response appears."
|
||||
- "metrics_summary includes response_p50_ms, response_p95_ms, error_rate, and total_duration_ms."
|
||||
- "thresholds_summary shows response_p95_ms and error_rate pass."
|
||||
evidence_required:
|
||||
- ui
|
||||
- screenshot
|
||||
- console
|
||||
- network
|
||||
- metrics
|
||||
diagnostics:
|
||||
- "This case measures browser-visible send-to-completion latency; it does not split provider latency from LangBot overhead."
|
||||
- "Use backend logs and provider diagnostics to explain slow runs before calling them LangBot regressions."
|
||||
success_patterns:
|
||||
- "Processing request from person_websocket"
|
||||
- "Streaming completed"
|
||||
failure_patterns:
|
||||
- "Action invoke_llm_stream call timed out"
|
||||
- "Task exception was never retrieved"
|
||||
- "All models failed during streaming setup"
|
||||
troubleshooting:
|
||||
- debug-chat-history-contaminates-automation
|
||||
- local-agent-model-route-unavailable
|
||||
- plugin-runtime-timeout
|
||||
- proxy-env-mismatch
|
||||
@@ -0,0 +1,58 @@
|
||||
id: skill-discovery-via-mcp-gateway
|
||||
title: "External harness discovers LangBot skills via langbot_list_assets (all-tool model)"
|
||||
mode: agent-browser
|
||||
area: sandbox
|
||||
type: regression
|
||||
priority: p2
|
||||
risk: medium
|
||||
ci_eligible: false
|
||||
tags:
|
||||
- skills
|
||||
- mcp-gateway
|
||||
- acp-agent-runner
|
||||
- all-tool-model
|
||||
- tools
|
||||
skills:
|
||||
- langbot-env-setup
|
||||
- langbot-testing
|
||||
env:
|
||||
- LANGBOT_FRONTEND_URL
|
||||
- LANGBOT_BACKEND_URL
|
||||
- LANGBOT_ACP_AGENT_RUNNER_PIPELINE_URL
|
||||
- LANGBOT_ACP_AGENT_RUNNER_PIPELINE_NAME
|
||||
preconditions:
|
||||
- "An external-harness runner pipeline (e.g. ACP remote claude-code) is configured with langbot-assets-enabled=true so the LangBot MCP gateway is exposed to the harness."
|
||||
- "The remote harness (claude-code) is reachable and responsive (claude -p returns within the runner timeout)."
|
||||
- "At least one pipeline-visible skill exists in the Box skill store (otherwise the count is 0, which is still a valid pass for the discovery surface)."
|
||||
automation: scripts/e2e/pipeline-debug-chat.mjs
|
||||
automation_env:
|
||||
- LANGBOT_FRONTEND_URL
|
||||
- LANGBOT_BROWSER_PROFILE
|
||||
- LANGBOT_CHROMIUM_EXECUTABLE
|
||||
- LANGBOT_ACP_AGENT_RUNNER_PIPELINE_URL
|
||||
- LANGBOT_ACP_AGENT_RUNNER_PIPELINE_NAME
|
||||
automation_pipeline_url_env: LANGBOT_ACP_AGENT_RUNNER_PIPELINE_URL
|
||||
automation_pipeline_name_env: LANGBOT_ACP_AGENT_RUNNER_PIPELINE_NAME
|
||||
automation_prompt: "You have LangBot tools available via an MCP server (tools prefixed langbot_). Call langbot_list_assets with asset_types = [\"skills\",\"tools\"]. Then reply with one single line: the literal token PROBEDONE, a space, the number of skills you found, a space, and the number of tools you found."
|
||||
automation_expected_text: "PROBEDONE"
|
||||
automation_response_timeout_ms: "540000"
|
||||
steps:
|
||||
- "Open LANGBOT_FRONTEND_URL and navigate to the external-harness (ACP) pipeline."
|
||||
- "Open Debug Chat with langbot-assets-enabled on the runner."
|
||||
- "Send the automation_prompt asking the harness to call langbot_list_assets with asset_types [skills, tools]."
|
||||
- "Capture the final reply, backend logs, and the MCP gateway call trace."
|
||||
checks:
|
||||
- "UI: final reply contains PROBEDONE followed by a skill count and a tool count."
|
||||
- "Logs: backend shows the harness invoked langbot_list_assets and the response included a 'skills' asset class (this is the all-tool-model discovery surface added on this branch)."
|
||||
- "Behavior parity: a local-agent runner reaches the same skills via use_funcs / activate; the external harness reaches them via langbot_list_assets + langbot_call_tool."
|
||||
evidence_required:
|
||||
- ui
|
||||
- screenshot
|
||||
- backend_log
|
||||
expected_failures:
|
||||
- "runner.timeout when the remote claude-code harness is unauthenticated or slow to start — this is an environment issue, not a discovery-surface regression."
|
||||
diagnostics:
|
||||
- "If runner.timeout: ssh into the harness host and confirm `claude -p 'hi'` returns quickly; the ACP runner cannot complete until the harness responds."
|
||||
- "Activated-skill OPERATE on docker+shared-fs is tracked separately by issue #2271 and is out of scope for this discovery case."
|
||||
troubleshooting:
|
||||
- sandbox-native-tools-unavailable
|
||||
@@ -1,3 +1 @@
|
||||
dist/*
|
||||
!dist/
|
||||
!dist/qa-plugin-smoke-0.1.0.lbpkg
|
||||
dist/
|
||||
|
||||
Vendored
BIN
Binary file not shown.
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
Executable → Regular
@@ -1,837 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import crypto from "node:crypto";
|
||||
import net from "node:net";
|
||||
import tls from "node:tls";
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
import { join, resolve } from "node:path";
|
||||
import { env, exit } from "node:process";
|
||||
import {
|
||||
apiJson,
|
||||
appendLine,
|
||||
ensureEvidence,
|
||||
evidencePaths,
|
||||
loadEnvFiles,
|
||||
localIsoWithOffset,
|
||||
redact,
|
||||
resetAndAuthLocalUser,
|
||||
writeResult,
|
||||
} from "../../../scripts/e2e/lib/langbot-e2e.mjs";
|
||||
import {
|
||||
buildProviderTimingMetrics,
|
||||
summarizeFakeProviderState,
|
||||
} from "./lib/fake-provider-timing.mjs";
|
||||
|
||||
const DEFAULT_LOCAL_PASSWORD = "LangBotE2ELocalPass!2026";
|
||||
|
||||
await loadEnvFiles();
|
||||
const caseId = env.LBS_CASE_ID || "langbot-debug-chat-concurrency";
|
||||
const paths = evidencePaths(caseId);
|
||||
await ensureEvidence(paths);
|
||||
|
||||
const startedAt = new Date();
|
||||
const metricsPath = resolve(paths.evidenceDir, "metrics.json");
|
||||
const samplesPath = resolve(paths.evidenceDir, "samples.json");
|
||||
const fakeProviderStatePath = resolve(paths.evidenceDir, "fake-provider-state.json");
|
||||
const resetDiagnosticPath = resolve(paths.evidenceDir, "debug-chat-reset-diagnostic.json");
|
||||
const backendUrl = env.LANGBOT_BACKEND_URL || "";
|
||||
const fakeProviderUrl = env.LANGBOT_FAKE_PROVIDER_URL || "";
|
||||
const pipelineUrl = env.LANGBOT_E2E_PIPELINE_URL || env.LANGBOT_PIPELINE_URL || "";
|
||||
const pipelineName = env.LANGBOT_E2E_PIPELINE_NAME || env.LANGBOT_PIPELINE_NAME || "";
|
||||
const sessionType = env.LANGBOT_DEBUG_CHAT_LOAD_SESSION_TYPE || env.LANGBOT_E2E_DEBUG_CHAT_SESSION_TYPE || "person";
|
||||
const totalRequests = positiveInteger(env.LANGBOT_DEBUG_CHAT_LOAD_REQUESTS, defaultRequests(caseId));
|
||||
const concurrency = Math.min(totalRequests, positiveInteger(env.LANGBOT_DEBUG_CHAT_LOAD_CONCURRENCY, defaultConcurrency(caseId)));
|
||||
const timeoutMs = positiveInteger(env.LANGBOT_DEBUG_CHAT_LOAD_TIMEOUT_MS, defaultTimeout(caseId));
|
||||
const expectedPrefix = env.LANGBOT_DEBUG_CHAT_LOAD_EXPECTED_PREFIX || "LBQA";
|
||||
const promptTemplate = env.LANGBOT_DEBUG_CHAT_LOAD_PROMPT_TEMPLATE
|
||||
|| "请只回复 \"{expected}\"，不要解释，不要添加其他字符。";
|
||||
const stream = bool(env.LANGBOT_DEBUG_CHAT_LOAD_STREAM, true);
|
||||
const resetBeforeRun = bool(env.LANGBOT_DEBUG_CHAT_LOAD_RESET, true);
|
||||
const responseP95BudgetMs = positiveNumber(env.LANGBOT_DEBUG_CHAT_LOAD_RESPONSE_P95_MS, defaultP95Budget(caseId));
|
||||
const firstResponseP95BudgetMs = positiveNumber(env.LANGBOT_DEBUG_CHAT_LOAD_FIRST_RESPONSE_P95_MS, 0);
|
||||
const maxErrorRate = positiveNumber(env.LANGBOT_DEBUG_CHAT_LOAD_MAX_ERROR_RATE, 0);
|
||||
const minErrorRate = positiveNumber(env.LANGBOT_DEBUG_CHAT_LOAD_MIN_ERROR_RATE, 0);
|
||||
const minErrorCount = nonNegativeInteger(env.LANGBOT_DEBUG_CHAT_LOAD_MIN_ERROR_COUNT, 0);
|
||||
const minOkCount = nonNegativeInteger(env.LANGBOT_DEBUG_CHAT_LOAD_MIN_OK_COUNT, 0);
|
||||
const minProviderFaultCount = nonNegativeInteger(env.LANGBOT_DEBUG_CHAT_LOAD_MIN_PROVIDER_FAULT_COUNT, 0);
|
||||
const failOnFinalMismatch = bool(env.LANGBOT_DEBUG_CHAT_LOAD_FAIL_ON_FINAL_MISMATCH, false);
|
||||
const failureSignals = textList(env.LANGBOT_E2E_FAILURE_SIGNALS || env.LANGBOT_DEBUG_CHAT_LOAD_FAILURE_SIGNALS || "");
|
||||
|
||||
const result = {
|
||||
source: "automation",
|
||||
case_id: caseId,
|
||||
run_id: paths.runId,
|
||||
status: "fail",
|
||||
reason: "",
|
||||
started_at: startedAt.toISOString(),
|
||||
started_at_local: localIsoWithOffset(startedAt),
|
||||
finished_at: "",
|
||||
finished_at_local: "",
|
||||
duration_ms: 0,
|
||||
backend_url: backendUrl,
|
||||
pipeline_url: pipelineUrl,
|
||||
pipeline_name: pipelineName,
|
||||
pipeline_id: "",
|
||||
session_type: sessionType,
|
||||
load_profile: {
|
||||
requests: totalRequests,
|
||||
concurrency,
|
||||
timeout_ms: timeoutMs,
|
||||
stream,
|
||||
reset_before_run: resetBeforeRun,
|
||||
fail_on_final_mismatch: failOnFinalMismatch,
|
||||
},
|
||||
evidence: {
|
||||
network_log: paths.networkLog,
|
||||
metrics_json: metricsPath,
|
||||
samples_json: samplesPath,
|
||||
fake_provider_state_json: fakeProviderStatePath,
|
||||
debug_chat_reset_diagnostic_json: resetDiagnosticPath,
|
||||
automation_result_json: paths.automationResultJson,
|
||||
result_json: paths.resultJson,
|
||||
},
|
||||
evidence_collected: ["metrics", "network", "api_diagnostic", "filesystem"],
|
||||
};
|
||||
|
||||
try {
|
||||
if (!backendUrl) {
|
||||
result.status = "env_issue";
|
||||
throw new Error("LANGBOT_BACKEND_URL is not configured.");
|
||||
}
|
||||
if (!["person", "group"].includes(sessionType)) {
|
||||
throw new Error(`LANGBOT_DEBUG_CHAT_LOAD_SESSION_TYPE must be person or group, got ${sessionType}.`);
|
||||
}
|
||||
const backendReady = await backendReachable(backendUrl);
|
||||
if (!backendReady) {
|
||||
result.status = "env_issue";
|
||||
throw new Error(`Backend did not respond at ${backendUrl}.`);
|
||||
}
|
||||
|
||||
const user = env.LANGBOT_E2E_LOGIN_USER || "";
|
||||
const password = env.LANGBOT_E2E_LOGIN_PASSWORD || DEFAULT_LOCAL_PASSWORD;
|
||||
if (!user) {
|
||||
result.status = "env_issue";
|
||||
throw new Error("LANGBOT_E2E_LOGIN_USER is required so this probe can resolve/reset the Debug Chat session.");
|
||||
}
|
||||
const auth = await resetAndAuthLocalUser({ backendUrl, user, password });
|
||||
|
||||
const pipeline = await resolvePipeline({ backendUrl, token: auth.token, pipelineUrl, pipelineName });
|
||||
result.pipeline_id = pipeline.id;
|
||||
result.pipeline_name = pipeline.name || pipelineName;
|
||||
if (!result.pipeline_url && env.LANGBOT_FRONTEND_URL) {
|
||||
result.pipeline_url = `${env.LANGBOT_FRONTEND_URL.replace(/\/$/, "")}/home/pipelines?id=${encodeURIComponent(pipeline.id)}`;
|
||||
}
|
||||
|
||||
if (resetBeforeRun) {
|
||||
const reset = await apiJson(backendUrl, `/api/v1/pipelines/${encodeURIComponent(pipeline.id)}/ws/reset/${encodeURIComponent(sessionType)}`, {
|
||||
method: "POST",
|
||||
token: auth.token,
|
||||
});
|
||||
const resetDiagnostic = {
|
||||
status: isApiFailure(reset) ? "fail" : "ready",
|
||||
http_status: reset.status,
|
||||
code: reset.json.code ?? null,
|
||||
reason: isApiFailure(reset) ? reset.json.msg || "Debug Chat reset failed." : "Debug Chat session reset.",
|
||||
};
|
||||
await writeFile(resetDiagnosticPath, `${JSON.stringify(resetDiagnostic, null, 2)}\n`, "utf8");
|
||||
if (resetDiagnostic.status === "fail") {
|
||||
throw new Error(resetDiagnostic.reason);
|
||||
}
|
||||
}
|
||||
|
||||
const wsUrl = websocketUrl(backendUrl, pipeline.id, sessionType);
|
||||
const loadStartedAt = performance.now();
|
||||
const samples = await runLoad({
|
||||
wsUrl,
|
||||
totalRequests,
|
||||
concurrency,
|
||||
timeoutMs,
|
||||
promptTemplate,
|
||||
expectedPrefix,
|
||||
stream,
|
||||
failOnFinalMismatch,
|
||||
failureSignals,
|
||||
});
|
||||
const loadDurationMs = performance.now() - loadStartedAt;
|
||||
const fakeProviderState = await readFakeProviderState(fakeProviderUrl);
|
||||
if (fakeProviderState) {
|
||||
await writeFile(fakeProviderStatePath, `${JSON.stringify(fakeProviderState, null, 2)}\n`, "utf8");
|
||||
}
|
||||
const metrics = buildMetrics({
|
||||
samples,
|
||||
totalRequests,
|
||||
concurrency,
|
||||
timeoutMs,
|
||||
loadDurationMs,
|
||||
backendUrl,
|
||||
pipelineId: pipeline.id,
|
||||
sessionType,
|
||||
fakeProviderState,
|
||||
});
|
||||
const thresholds = buildThresholds(metrics);
|
||||
const passed = Object.values(thresholds).every((item) => item.pass);
|
||||
result.status = passed ? "pass" : "fail";
|
||||
result.reason = passed
|
||||
? "Debug Chat WebSocket concurrency probe passed all thresholds."
|
||||
: "Debug Chat WebSocket concurrency probe breached latency or error-rate thresholds.";
|
||||
result.metrics_summary = {
|
||||
requests: metrics.total_requests,
|
||||
concurrency: metrics.concurrency,
|
||||
ok_count: metrics.ok_count,
|
||||
error_count: metrics.error_count,
|
||||
timeout_count: metrics.timeout_count,
|
||||
error_rate: metrics.error_rate,
|
||||
response_p50_ms: metrics.response_duration_ms.p50,
|
||||
response_p95_ms: metrics.response_duration_ms.p95,
|
||||
first_assistant_event_p95_ms: metrics.first_assistant_event_ms.p95,
|
||||
first_assistant_content_p95_ms: metrics.first_assistant_content_ms.p95,
|
||||
first_response_p95_ms: metrics.first_response_ms.p95,
|
||||
throughput_rps: metrics.throughput_rps,
|
||||
status_counts: metrics.status_counts,
|
||||
fake_provider_request_count: metrics.fake_provider?.request_count ?? null,
|
||||
fake_provider_fault_count: metrics.fake_provider?.fault_count ?? null,
|
||||
fake_provider_duration_p95_ms: metrics.provider_timing?.provider_duration_ms.p95 ?? null,
|
||||
langbot_overhead_estimate_p95_ms: metrics.provider_timing?.langbot_overhead_estimate_ms.p95 ?? null,
|
||||
send_to_provider_start_p95_ms: metrics.provider_timing?.send_to_provider_start_ms.p95 ?? null,
|
||||
provider_finish_to_ws_final_p95_ms: metrics.provider_timing?.provider_finish_to_ws_final_ms.p95 ?? null,
|
||||
provider_timing_matched_request_count: metrics.provider_timing?.matched_request_count ?? null,
|
||||
};
|
||||
result.thresholds_summary = thresholds;
|
||||
result.artifacts = {
|
||||
metrics_json: metricsPath,
|
||||
samples_json: samplesPath,
|
||||
fake_provider_state_json: fakeProviderState ? fakeProviderStatePath : "",
|
||||
network_log: paths.networkLog,
|
||||
automation_result_json: paths.automationResultJson,
|
||||
result_json: paths.resultJson,
|
||||
};
|
||||
|
||||
await writeFile(metricsPath, `${JSON.stringify({ ...metrics, thresholds }, null, 2)}\n`, "utf8");
|
||||
await writeFile(samplesPath, `${JSON.stringify(samples, null, 2)}\n`, "utf8");
|
||||
} catch (error) {
|
||||
if (!["env_issue", "blocked"].includes(result.status)) {
|
||||
result.status = looksLikeEnvIssue(error) ? "env_issue" : "fail";
|
||||
}
|
||||
result.reason = result.reason || safeReason(error.message);
|
||||
} finally {
|
||||
const finishedAt = new Date();
|
||||
result.finished_at = finishedAt.toISOString();
|
||||
result.finished_at_local = localIsoWithOffset(finishedAt);
|
||||
result.duration_ms = finishedAt.getTime() - startedAt.getTime();
|
||||
await mkdir(paths.evidenceDir, { recursive: true });
|
||||
await writeResult(paths, result);
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
}
|
||||
|
||||
exit(result.status === "pass" ? 0 : result.status === "env_issue" || result.status === "blocked" ? 2 : 1);
|
||||
|
||||
function defaultRequests(id) {
|
||||
return id.includes("space") ? 3 : 12;
|
||||
}
|
||||
|
||||
function defaultConcurrency(id) {
|
||||
return id.includes("space") ? 1 : 4;
|
||||
}
|
||||
|
||||
function defaultTimeout(id) {
|
||||
return id.includes("space") ? 120_000 : 30_000;
|
||||
}
|
||||
|
||||
function defaultP95Budget(id) {
|
||||
return id.includes("space") ? 120_000 : 5_000;
|
||||
}
|
||||
|
||||
function positiveInteger(value, fallback) {
|
||||
const parsed = Number.parseInt(String(value || ""), 10);
|
||||
return Number.isInteger(parsed) && parsed > 0 ? parsed : fallback;
|
||||
}
|
||||
|
||||
function nonNegativeInteger(value, fallback) {
|
||||
const parsed = Number.parseInt(String(value ?? ""), 10);
|
||||
return Number.isInteger(parsed) && parsed >= 0 ? parsed : fallback;
|
||||
}
|
||||
|
||||
function positiveNumber(value, fallback) {
|
||||
const parsed = Number(value || "");
|
||||
return Number.isFinite(parsed) && parsed >= 0 ? parsed : fallback;
|
||||
}
|
||||
|
||||
function bool(value, fallback) {
|
||||
if (value === undefined || value === "") return fallback;
|
||||
if (/^(1|true|yes|on)$/i.test(String(value))) return true;
|
||||
if (/^(0|false|no|off)$/i.test(String(value))) return false;
|
||||
return fallback;
|
||||
}
|
||||
|
||||
function textList(value) {
|
||||
return String(value || "")
|
||||
.split(/\r?\n|,/)
|
||||
.map((item) => item.trim())
|
||||
.filter(Boolean);
|
||||
}
|
||||
|
||||
async function backendReachable(baseUrl) {
|
||||
try {
|
||||
const response = await fetch(`${baseUrl.replace(/\/$/, "")}/healthz`, {
|
||||
signal: AbortSignal.timeout(3000),
|
||||
});
|
||||
return response.status < 500;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async function readFakeProviderState(rootUrl) {
|
||||
if (!rootUrl) return null;
|
||||
try {
|
||||
const response = await fetch(`${normalizeProviderRootUrl(rootUrl)}/__qa/config`, {
|
||||
signal: AbortSignal.timeout(3000),
|
||||
});
|
||||
const json = await response.json().catch(() => ({}));
|
||||
return {
|
||||
status: response.ok && json.ok === true ? "loaded" : "unavailable",
|
||||
url: normalizeProviderRootUrl(rootUrl),
|
||||
http_status: response.status,
|
||||
model: json.model || "",
|
||||
config: json.config || {},
|
||||
request_count: Number.isFinite(json.request_count) ? json.request_count : null,
|
||||
recent_requests: Array.isArray(json.recent_requests) ? json.recent_requests : [],
|
||||
};
|
||||
} catch (error) {
|
||||
return {
|
||||
status: "unavailable",
|
||||
url: normalizeProviderRootUrl(rootUrl),
|
||||
reason: safeReason(error.message),
|
||||
request_count: null,
|
||||
recent_requests: [],
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
function normalizeProviderRootUrl(value) {
|
||||
const trimmed = String(value || "").trim().replace(/\/$/, "");
|
||||
return trimmed.endsWith("/v1") ? trimmed.slice(0, -3) : trimmed;
|
||||
}
|
||||
|
||||
function pipelineIdFromUrl(url) {
|
||||
if (!url) return "";
|
||||
try {
|
||||
const parsed = new URL(url);
|
||||
return parsed.searchParams.get("id") || "";
|
||||
} catch {
|
||||
return "";
|
||||
}
|
||||
}
|
||||
|
||||
async function resolvePipeline({ backendUrl, token, pipelineUrl, pipelineName }) {
|
||||
const idFromUrl = pipelineIdFromUrl(pipelineUrl);
|
||||
if (idFromUrl) {
|
||||
const response = await apiJson(backendUrl, `/api/v1/pipelines/${encodeURIComponent(idFromUrl)}`, { token });
|
||||
const pipeline = response.json.data?.pipeline;
|
||||
if (isApiFailure(response) || !pipeline?.uuid) {
|
||||
throw new Error(response.json.msg || `Could not load pipeline ${idFromUrl}.`);
|
||||
}
|
||||
return { id: pipeline.uuid, name: pipeline.name || "" };
|
||||
}
|
||||
if (!pipelineName) {
|
||||
throw new Error("Set LANGBOT_E2E_PIPELINE_URL or LANGBOT_E2E_PIPELINE_NAME before running this probe.");
|
||||
}
|
||||
const response = await apiJson(backendUrl, "/api/v1/pipelines", { token });
|
||||
if (isApiFailure(response)) {
|
||||
throw new Error(response.json.msg || "Failed to list pipelines.");
|
||||
}
|
||||
const pipeline = (response.json.data?.pipelines || []).find((item) => item.name === pipelineName);
|
||||
if (!pipeline?.uuid) {
|
||||
throw new Error(`Could not find pipeline named ${pipelineName}.`);
|
||||
}
|
||||
return { id: pipeline.uuid, name: pipeline.name || pipelineName };
|
||||
}
|
||||
|
||||
function isApiFailure(response) {
|
||||
return response.status >= 400 || (response.json.code !== undefined && response.json.code !== 0);
|
||||
}
|
||||
|
||||
function websocketUrl(baseUrl, pipelineId, sessionType) {
|
||||
const parsed = new URL(baseUrl);
|
||||
parsed.protocol = parsed.protocol === "https:" ? "wss:" : "ws:";
|
||||
parsed.pathname = `/api/v1/pipelines/${encodeURIComponent(pipelineId)}/ws/connect`;
|
||||
parsed.search = `?session_type=${encodeURIComponent(sessionType)}`;
|
||||
return parsed.toString();
|
||||
}
|
||||
|
||||
async function runLoad(options) {
|
||||
const samples = [];
|
||||
let nextIndex = 0;
|
||||
const workers = Array.from({ length: options.concurrency }, async () => {
|
||||
while (nextIndex < options.totalRequests) {
|
||||
const index = nextIndex;
|
||||
nextIndex += 1;
|
||||
const sample = await runSingleRequest({ ...options, index });
|
||||
samples.push(sample);
|
||||
}
|
||||
});
|
||||
await Promise.all(workers);
|
||||
return samples.sort((left, right) => left.index - right.index);
|
||||
}
|
||||
|
||||
function expectedForIndex(prefix, index) {
|
||||
return `${prefix}-${String(index + 1).padStart(4, "0")}`;
|
||||
}
|
||||
|
||||
function promptForIndex(template, expected) {
|
||||
return template.replaceAll("{expected}", expected);
|
||||
}
|
||||
|
||||
function runSingleRequest({
|
||||
wsUrl,
|
||||
index,
|
||||
timeoutMs,
|
||||
promptTemplate,
|
||||
expectedPrefix,
|
||||
stream,
|
||||
failOnFinalMismatch,
|
||||
failureSignals,
|
||||
}) {
|
||||
return new Promise((resolve) => {
|
||||
const expected = expectedForIndex(expectedPrefix, index);
|
||||
const prompt = promptForIndex(promptTemplate, expected);
|
||||
const sample = {
|
||||
index,
|
||||
status: "running",
|
||||
ok: false,
|
||||
expected_text: expected,
|
||||
prompt,
|
||||
response_text: "",
|
||||
started_at: new Date().toISOString(),
|
||||
started_epoch_ms: Date.now(),
|
||||
connected_at: null,
|
||||
connected_epoch_ms: null,
|
||||
sent_at: null,
|
||||
sent_epoch_ms: null,
|
||||
first_assistant_event_at: null,
|
||||
first_assistant_event_epoch_ms: null,
|
||||
first_assistant_event_ms: null,
|
||||
first_assistant_content_at: null,
|
||||
first_assistant_content_epoch_ms: null,
|
||||
first_assistant_content_ms: null,
|
||||
first_response_at: null,
|
||||
first_response_epoch_ms: null,
|
||||
connected_ms: null,
|
||||
first_response_ms: null,
|
||||
response_duration_ms: null,
|
||||
finished_at: null,
|
||||
finished_epoch_ms: null,
|
||||
event_count: 0,
|
||||
foreign_response_count: 0,
|
||||
last_foreign_response_text: "",
|
||||
error: "",
|
||||
close_code: null,
|
||||
close_reason: "",
|
||||
};
|
||||
let closed = false;
|
||||
let connectedAt = 0;
|
||||
let sentAt = 0;
|
||||
const startedAt = performance.now();
|
||||
let client = null;
|
||||
const timer = setTimeout(() => {
|
||||
finish("timeout", `Timed out after ${timeoutMs} ms.`);
|
||||
}, timeoutMs);
|
||||
|
||||
client = openRawWebSocket(wsUrl, {
|
||||
onOpen() {
|
||||
connectedAt = performance.now();
|
||||
const now = Date.now();
|
||||
sample.connected_at = new Date(now).toISOString();
|
||||
sample.connected_epoch_ms = now;
|
||||
sample.connected_ms = rounded(connectedAt - startedAt);
|
||||
},
|
||||
onMessage(text) {
|
||||
sample.event_count += 1;
|
||||
let data;
|
||||
try {
|
||||
data = JSON.parse(String(text || ""));
|
||||
} catch (error) {
|
||||
finish("error", `Invalid WebSocket JSON: ${error.message}`);
|
||||
return;
|
||||
}
|
||||
appendLine(paths.networkLog, JSON.stringify({
|
||||
request_index: index,
|
||||
type: data.type,
|
||||
session_type: data.session_type || "",
|
||||
role: data.data?.role || "",
|
||||
is_final: data.data?.is_final ?? null,
|
||||
content_preview: redact(String(data.data?.content || data.message || "").slice(0, 200)),
|
||||
})).catch(() => {});
|
||||
|
||||
if (data.type === "connected") {
|
||||
sentAt = performance.now();
|
||||
const now = Date.now();
|
||||
sample.sent_at = new Date(now).toISOString();
|
||||
sample.sent_epoch_ms = now;
|
||||
client.send(JSON.stringify({
|
||||
type: "message",
|
||||
message: [{ type: "Plain", text: prompt }],
|
||||
stream,
|
||||
}));
|
||||
return;
|
||||
}
|
||||
if (data.type === "error") {
|
||||
finish("error", data.message || "WebSocket error message.");
|
||||
return;
|
||||
}
|
||||
if (data.type !== "response" || data.data?.role !== "assistant") return;
|
||||
|
||||
const content = String(data.data.content || "");
|
||||
markFirstAssistantEvent(sample, sentAt);
|
||||
if (content) sample.response_text = content;
|
||||
if (content) markFirstAssistantContent(sample, sentAt);
|
||||
if (content.includes(expected) && sample.first_response_ms === null && sentAt > 0) {
|
||||
const now = Date.now();
|
||||
sample.first_response_at = new Date(now).toISOString();
|
||||
sample.first_response_epoch_ms = now;
|
||||
sample.first_response_ms = rounded(performance.now() - sentAt);
|
||||
}
|
||||
if (data.data.is_final === true) {
|
||||
const ok = sample.response_text.includes(expected);
|
||||
if (ok) {
|
||||
if (sample.first_response_ms === null && sentAt > 0) {
|
||||
sample.first_response_ms = rounded(performance.now() - sentAt);
|
||||
}
|
||||
finish("pass", "");
|
||||
} else if (matchesFailureSignal(sample.response_text, failureSignals)) {
|
||||
finish("app_error", `Assistant final response matched a failure signal: ${sample.response_text}`);
|
||||
} else if (failOnFinalMismatch && !containsLoadToken(sample.response_text, expectedPrefix)) {
|
||||
finish("mismatch", `Final assistant response did not include ${expected}: ${sample.response_text}`);
|
||||
} else {
|
||||
sample.foreign_response_count += 1;
|
||||
sample.last_foreign_response_text = sample.response_text;
|
||||
}
|
||||
}
|
||||
},
|
||||
onError(error) {
|
||||
finish("connection_error", `WebSocket connection error: ${error.message}`);
|
||||
},
|
||||
onClose(event) {
|
||||
sample.close_code = event.code;
|
||||
sample.close_reason = event.reason || "";
|
||||
if (!closed) finish("closed", `WebSocket closed before final assistant response: ${event.code}`);
|
||||
},
|
||||
});
|
||||
|
||||
function finish(status, reason) {
|
||||
if (closed) return;
|
||||
closed = true;
|
||||
clearTimeout(timer);
|
||||
sample.status = status;
|
||||
sample.ok = status === "pass";
|
||||
sample.error = status === "timeout" && sample.foreign_response_count > 0
|
||||
? `${reason || ""} Saw ${sample.foreign_response_count} foreign assistant response(s); last=${sample.last_foreign_response_text}`
|
||||
: reason || "";
|
||||
if (sentAt > 0) sample.response_duration_ms = rounded(performance.now() - sentAt);
|
||||
else sample.response_duration_ms = rounded(performance.now() - startedAt);
|
||||
const now = Date.now();
|
||||
sample.finished_at = new Date(now).toISOString();
|
||||
sample.finished_epoch_ms = now;
|
||||
try {
|
||||
client?.close();
|
||||
} catch {
|
||||
// Closing a failed socket should not hide the sample result.
|
||||
}
|
||||
resolve(sample);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
function markFirstAssistantEvent(sample, sentAt) {
|
||||
if (sample.first_assistant_event_ms !== null || sentAt <= 0) return;
|
||||
const now = Date.now();
|
||||
sample.first_assistant_event_at = new Date(now).toISOString();
|
||||
sample.first_assistant_event_epoch_ms = now;
|
||||
sample.first_assistant_event_ms = rounded(performance.now() - sentAt);
|
||||
}
|
||||
|
||||
function markFirstAssistantContent(sample, sentAt) {
|
||||
if (sample.first_assistant_content_ms !== null || sentAt <= 0) return;
|
||||
const now = Date.now();
|
||||
sample.first_assistant_content_at = new Date(now).toISOString();
|
||||
sample.first_assistant_content_epoch_ms = now;
|
||||
sample.first_assistant_content_ms = rounded(performance.now() - sentAt);
|
||||
}
|
||||
|
||||
function containsLoadToken(text, prefix) {
|
||||
const escaped = String(prefix).replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
||||
return new RegExp(`${escaped}-\\d{4}`).test(String(text || ""));
|
||||
}
|
||||
|
||||
function matchesFailureSignal(text, signals) {
|
||||
const lower = String(text || "").toLowerCase();
|
||||
return signals.some((signal) => lower.includes(signal.toLowerCase()));
|
||||
}
|
||||
|
||||
function openRawWebSocket(wsUrl, handlers) {
|
||||
const parsed = new URL(wsUrl);
|
||||
const secure = parsed.protocol === "wss:";
|
||||
const port = Number(parsed.port || (secure ? 443 : 80));
|
||||
const host = parsed.hostname;
|
||||
const path = `${parsed.pathname}${parsed.search}`;
|
||||
const key = crypto.randomBytes(16).toString("base64");
|
||||
const socket = secure
|
||||
? tls.connect({ host, port, servername: host })
|
||||
: net.connect({ host, port });
|
||||
let opened = false;
|
||||
let closed = false;
|
||||
let buffer = Buffer.alloc(0);
|
||||
|
||||
socket.setNoDelay(true);
|
||||
socket.on("connect", () => {
|
||||
const originProtocol = secure ? "https" : "http";
|
||||
const request = [
|
||||
`GET ${path} HTTP/1.1`,
|
||||
`Host: ${parsed.host}`,
|
||||
"Upgrade: websocket",
|
||||
"Connection: Upgrade",
|
||||
`Sec-WebSocket-Key: ${key}`,
|
||||
"Sec-WebSocket-Version: 13",
|
||||
`Origin: ${originProtocol}://${parsed.host}`,
|
||||
"",
|
||||
"",
|
||||
].join("\r\n");
|
||||
socket.write(request);
|
||||
});
|
||||
socket.on("data", (chunk) => {
|
||||
buffer = Buffer.concat([buffer, chunk]);
|
||||
if (!opened) {
|
||||
const headerEnd = buffer.indexOf("\r\n\r\n");
|
||||
if (headerEnd === -1) return;
|
||||
const headerText = buffer.slice(0, headerEnd).toString("utf8");
|
||||
buffer = buffer.slice(headerEnd + 4);
|
||||
if (!/^HTTP\/1\.1 101\b/i.test(headerText)) {
|
||||
handlers.onError(new Error(`Handshake failed: ${headerText.split("\r\n")[0] || "missing status"}`));
|
||||
socket.destroy();
|
||||
return;
|
||||
}
|
||||
opened = true;
|
||||
handlers.onOpen();
|
||||
}
|
||||
processFrames();
|
||||
});
|
||||
socket.on("error", (error) => {
|
||||
if (!closed) handlers.onError(error);
|
||||
});
|
||||
socket.on("close", () => {
|
||||
if (closed) return;
|
||||
closed = true;
|
||||
handlers.onClose({ code: null, reason: "" });
|
||||
});
|
||||
|
||||
function processFrames() {
|
||||
while (true) {
|
||||
const frame = readFrame(buffer);
|
||||
if (!frame) return;
|
||||
buffer = buffer.slice(frame.consumed);
|
||||
if (frame.opcode === 0x1) {
|
||||
handlers.onMessage(frame.payload.toString("utf8"));
|
||||
} else if (frame.opcode === 0x8) {
|
||||
const code = frame.payload.length >= 2 ? frame.payload.readUInt16BE(0) : null;
|
||||
const reason = frame.payload.length > 2 ? frame.payload.slice(2).toString("utf8") : "";
|
||||
closed = true;
|
||||
handlers.onClose({ code, reason });
|
||||
socket.end();
|
||||
return;
|
||||
} else if (frame.opcode === 0x9) {
|
||||
writeFrame(socket, 0xA, frame.payload);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
send(text) {
|
||||
if (closed || !opened) return;
|
||||
writeFrame(socket, 0x1, Buffer.from(text, "utf8"));
|
||||
},
|
||||
close() {
|
||||
if (closed) return;
|
||||
closed = true;
|
||||
if (!socket.destroyed) {
|
||||
if (opened) writeFrame(socket, 0x8, Buffer.alloc(0));
|
||||
setTimeout(() => socket.end(), 50).unref();
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function readFrame(buffer) {
|
||||
if (buffer.length < 2) return null;
|
||||
const first = buffer[0];
|
||||
const second = buffer[1];
|
||||
const opcode = first & 0x0f;
|
||||
const masked = Boolean(second & 0x80);
|
||||
let length = second & 0x7f;
|
||||
let offset = 2;
|
||||
if (length === 126) {
|
||||
if (buffer.length < offset + 2) return null;
|
||||
length = buffer.readUInt16BE(offset);
|
||||
offset += 2;
|
||||
} else if (length === 127) {
|
||||
if (buffer.length < offset + 8) return null;
|
||||
const high = buffer.readUInt32BE(offset);
|
||||
const low = buffer.readUInt32BE(offset + 4);
|
||||
length = high * 2 ** 32 + low;
|
||||
offset += 8;
|
||||
}
|
||||
let mask = null;
|
||||
if (masked) {
|
||||
if (buffer.length < offset + 4) return null;
|
||||
mask = buffer.slice(offset, offset + 4);
|
||||
offset += 4;
|
||||
}
|
||||
if (buffer.length < offset + length) return null;
|
||||
let payload = buffer.slice(offset, offset + length);
|
||||
if (mask) {
|
||||
payload = Buffer.from(payload);
|
||||
for (let index = 0; index < payload.length; index += 1) {
|
||||
payload[index] ^= mask[index % 4];
|
||||
}
|
||||
}
|
||||
return {
|
||||
opcode,
|
||||
payload,
|
||||
consumed: offset + length,
|
||||
};
|
||||
}
|
||||
|
||||
function writeFrame(socket, opcode, payload) {
|
||||
const body = Buffer.isBuffer(payload) ? payload : Buffer.from(payload || "");
|
||||
const mask = crypto.randomBytes(4);
|
||||
const headerLength = body.length < 126 ? 2 : body.length <= 0xffff ? 4 : 10;
|
||||
const header = Buffer.alloc(headerLength);
|
||||
header[0] = 0x80 | opcode;
|
||||
if (body.length < 126) {
|
||||
header[1] = 0x80 | body.length;
|
||||
} else if (body.length <= 0xffff) {
|
||||
header[1] = 0x80 | 126;
|
||||
header.writeUInt16BE(body.length, 2);
|
||||
} else {
|
||||
header[1] = 0x80 | 127;
|
||||
header.writeUInt32BE(Math.floor(body.length / 2 ** 32), 2);
|
||||
header.writeUInt32BE(body.length >>> 0, 6);
|
||||
}
|
||||
const masked = Buffer.from(body);
|
||||
for (let index = 0; index < masked.length; index += 1) {
|
||||
masked[index] ^= mask[index % 4];
|
||||
}
|
||||
socket.write(Buffer.concat([header, mask, masked]));
|
||||
}
|
||||
|
||||
function rounded(value) {
|
||||
return Number(value.toFixed(3));
|
||||
}
|
||||
|
||||
function percentile(values, percentileValue) {
|
||||
if (values.length === 0) return 0;
|
||||
const sorted = [...values].sort((a, b) => a - b);
|
||||
const index = Math.min(sorted.length - 1, Math.ceil((percentileValue / 100) * sorted.length) - 1);
|
||||
return rounded(sorted[index]);
|
||||
}
|
||||
|
||||
function stats(values) {
|
||||
if (values.length === 0) return { min: 0, p50: 0, p95: 0, p99: 0, max: 0 };
|
||||
return {
|
||||
min: rounded(Math.min(...values)),
|
||||
p50: percentile(values, 50),
|
||||
p95: percentile(values, 95),
|
||||
p99: percentile(values, 99),
|
||||
max: rounded(Math.max(...values)),
|
||||
};
|
||||
}
|
||||
|
||||
function buildMetrics({ samples, totalRequests, concurrency, timeoutMs, loadDurationMs, backendUrl, pipelineId, sessionType, fakeProviderState }) {
|
||||
const okSamples = samples.filter((sample) => sample.ok);
|
||||
const statusCounts = {};
|
||||
for (const sample of samples) {
|
||||
statusCounts[sample.status] = (statusCounts[sample.status] || 0) + 1;
|
||||
}
|
||||
const errorCount = samples.length - okSamples.length;
|
||||
return {
|
||||
probe: caseId,
|
||||
backend_url: backendUrl,
|
||||
pipeline_id: pipelineId,
|
||||
session_type: sessionType,
|
||||
total_requests: totalRequests,
|
||||
completed_requests: samples.length,
|
||||
concurrency,
|
||||
timeout_ms: timeoutMs,
|
||||
ok_count: okSamples.length,
|
||||
error_count: errorCount,
|
||||
timeout_count: samples.filter((sample) => sample.status === "timeout").length,
|
||||
error_rate: samples.length === 0 ? 1 : rounded(errorCount / samples.length),
|
||||
load_duration_ms: rounded(loadDurationMs),
|
||||
throughput_rps: loadDurationMs <= 0 ? 0 : rounded(okSamples.length / (loadDurationMs / 1000)),
|
||||
status_counts: statusCounts,
|
||||
connected_ms: stats(samples.map((sample) => sample.connected_ms).filter(Number.isFinite)),
|
||||
first_assistant_event_ms: stats(samples.map((sample) => sample.first_assistant_event_ms).filter(Number.isFinite)),
|
||||
first_assistant_content_ms: stats(samples.map((sample) => sample.first_assistant_content_ms).filter(Number.isFinite)),
|
||||
first_response_ms: stats(okSamples.map((sample) => sample.first_response_ms).filter(Number.isFinite)),
|
||||
response_duration_ms: stats(okSamples.map((sample) => sample.response_duration_ms).filter(Number.isFinite)),
|
||||
fake_provider: summarizeFakeProviderState(fakeProviderState),
|
||||
provider_timing: buildProviderTimingMetrics(samples, fakeProviderState),
|
||||
samples,
|
||||
};
|
||||
}
|
||||
|
||||
function buildThresholds(metrics) {
|
||||
const thresholds = {
|
||||
error_rate: { actual: metrics.error_rate, max: maxErrorRate, pass: metrics.error_rate <= maxErrorRate },
|
||||
response_p95_ms: {
|
||||
actual: metrics.response_duration_ms.p95,
|
||||
max: responseP95BudgetMs,
|
||||
pass: metrics.ok_count > 0 && metrics.response_duration_ms.p95 <= responseP95BudgetMs,
|
||||
},
|
||||
};
|
||||
if (minErrorRate > 0) {
|
||||
thresholds.error_rate_min = {
|
||||
actual: metrics.error_rate,
|
||||
min: minErrorRate,
|
||||
pass: metrics.error_rate >= minErrorRate,
|
||||
};
|
||||
}
|
||||
if (minErrorCount > 0) {
|
||||
thresholds.error_count_min = {
|
||||
actual: metrics.error_count,
|
||||
min: minErrorCount,
|
||||
pass: metrics.error_count >= minErrorCount,
|
||||
};
|
||||
}
|
||||
if (minOkCount > 0) {
|
||||
thresholds.ok_count_min = {
|
||||
actual: metrics.ok_count,
|
||||
min: minOkCount,
|
||||
pass: metrics.ok_count >= minOkCount,
|
||||
};
|
||||
}
|
||||
if (minProviderFaultCount > 0) {
|
||||
const actual = metrics.fake_provider?.fault_count ?? 0;
|
||||
thresholds.fake_provider_fault_count_min = {
|
||||
actual,
|
||||
min: minProviderFaultCount,
|
||||
pass: actual >= minProviderFaultCount,
|
||||
};
|
||||
}
|
||||
if (firstResponseP95BudgetMs > 0) {
|
||||
thresholds.first_response_p95_ms = {
|
||||
actual: metrics.first_response_ms.p95,
|
||||
max: firstResponseP95BudgetMs,
|
||||
pass: metrics.ok_count > 0 && metrics.first_response_ms.p95 <= firstResponseP95BudgetMs,
|
||||
};
|
||||
}
|
||||
return thresholds;
|
||||
}
|
||||
|
||||
function looksLikeEnvIssue(error) {
|
||||
const message = String(error?.message || error || "");
|
||||
return /fetch failed|ECONNREFUSED|ENOTFOUND|LANGBOT_.*not configured|Could not read recovery_key|Backend did not respond/i.test(message);
|
||||
}
|
||||
|
||||
function safeReason(value) {
|
||||
return redact(String(value || "")).slice(0, 1000);
|
||||
}
|
||||
@@ -1,861 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import crypto from "node:crypto";
|
||||
import net from "node:net";
|
||||
import tls from "node:tls";
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
import { resolve } from "node:path";
|
||||
import { env, exit } from "node:process";
|
||||
import {
|
||||
apiJson,
|
||||
appendLine,
|
||||
ensureEvidence,
|
||||
evidencePaths,
|
||||
loadEnvFiles,
|
||||
localIsoWithOffset,
|
||||
redact,
|
||||
resetAndAuthLocalUser,
|
||||
writeResult,
|
||||
} from "../../../scripts/e2e/lib/langbot-e2e.mjs";
|
||||
import {
|
||||
buildProviderTimingMetrics,
|
||||
summarizeFakeProviderState,
|
||||
} from "./lib/fake-provider-timing.mjs";
|
||||
|
||||
const DEFAULT_LOCAL_PASSWORD = "LangBotE2ELocalPass!2026";
|
||||
|
||||
await loadEnvFiles();
|
||||
const caseId = env.LBS_CASE_ID || "langbot-debug-chat-cross-pipeline-isolation";
|
||||
const paths = evidencePaths(caseId);
|
||||
await ensureEvidence(paths);
|
||||
|
||||
const startedAt = new Date();
|
||||
const metricsPath = resolve(paths.evidenceDir, "metrics.json");
|
||||
const samplesPath = resolve(paths.evidenceDir, "samples.json");
|
||||
const fakeProviderStatePath = resolve(paths.evidenceDir, "fake-provider-state.json");
|
||||
const resetDiagnosticPath = resolve(paths.evidenceDir, "debug-chat-reset-diagnostic.json");
|
||||
const backendUrl = env.LANGBOT_BACKEND_URL || "";
|
||||
const fakeProviderUrl = env.LANGBOT_FAKE_PROVIDER_URL || "";
|
||||
const sessionType = env.LANGBOT_DEBUG_CHAT_LOAD_SESSION_TYPE || env.LANGBOT_E2E_DEBUG_CHAT_SESSION_TYPE || "person";
|
||||
const requestsPerPipeline = positiveInteger(env.LANGBOT_DEBUG_CHAT_LOAD_REQUESTS, 6);
|
||||
const concurrency = Math.min(requestsPerPipeline * 2, positiveInteger(env.LANGBOT_DEBUG_CHAT_LOAD_CONCURRENCY, 4));
|
||||
const timeoutMs = positiveInteger(env.LANGBOT_DEBUG_CHAT_LOAD_TIMEOUT_MS, 30_000);
|
||||
const stream = bool(env.LANGBOT_DEBUG_CHAT_LOAD_STREAM, true);
|
||||
const resetBeforeRun = bool(env.LANGBOT_DEBUG_CHAT_LOAD_RESET, true);
|
||||
const responseP95BudgetMs = positiveNumber(env.LANGBOT_DEBUG_CHAT_LOAD_RESPONSE_P95_MS, 5_000);
|
||||
const maxErrorRate = positiveNumber(env.LANGBOT_DEBUG_CHAT_LOAD_MAX_ERROR_RATE, 0);
|
||||
const promptTemplate = env.LANGBOT_DEBUG_CHAT_LOAD_PROMPT_TEMPLATE
|
||||
|| "请只回复 \"{expected}\"，不要解释，不要添加其他字符。";
|
||||
const failureSignals = textList(env.LANGBOT_E2E_FAILURE_SIGNALS || env.LANGBOT_DEBUG_CHAT_LOAD_FAILURE_SIGNALS || "");
|
||||
|
||||
const pipelineTargets = [
|
||||
{
|
||||
label: "A",
|
||||
expectedPrefix: "PIPEA",
|
||||
otherPrefix: "PIPEB",
|
||||
url: env.LANGBOT_FAKE_PROVIDER_PIPELINE_A_URL || "",
|
||||
name: env.LANGBOT_FAKE_PROVIDER_PIPELINE_A_NAME || "",
|
||||
},
|
||||
{
|
||||
label: "B",
|
||||
expectedPrefix: "PIPEB",
|
||||
otherPrefix: "PIPEA",
|
||||
url: env.LANGBOT_FAKE_PROVIDER_PIPELINE_B_URL || "",
|
||||
name: env.LANGBOT_FAKE_PROVIDER_PIPELINE_B_NAME || "",
|
||||
},
|
||||
];
|
||||
|
||||
const result = {
|
||||
source: "automation",
|
||||
case_id: caseId,
|
||||
run_id: paths.runId,
|
||||
status: "fail",
|
||||
reason: "",
|
||||
started_at: startedAt.toISOString(),
|
||||
started_at_local: localIsoWithOffset(startedAt),
|
||||
finished_at: "",
|
||||
finished_at_local: "",
|
||||
duration_ms: 0,
|
||||
backend_url: backendUrl,
|
||||
session_type: sessionType,
|
||||
pipelines: [],
|
||||
load_profile: {
|
||||
requests_per_pipeline: requestsPerPipeline,
|
||||
total_requests: requestsPerPipeline * 2,
|
||||
concurrency,
|
||||
timeout_ms: timeoutMs,
|
||||
stream,
|
||||
reset_before_run: resetBeforeRun,
|
||||
},
|
||||
evidence: {
|
||||
network_log: paths.networkLog,
|
||||
metrics_json: metricsPath,
|
||||
samples_json: samplesPath,
|
||||
fake_provider_state_json: fakeProviderStatePath,
|
||||
debug_chat_reset_diagnostic_json: resetDiagnosticPath,
|
||||
automation_result_json: paths.automationResultJson,
|
||||
result_json: paths.resultJson,
|
||||
},
|
||||
evidence_collected: ["metrics", "network", "api_diagnostic", "filesystem"],
|
||||
};
|
||||
|
||||
try {
|
||||
if (!backendUrl) {
|
||||
result.status = "env_issue";
|
||||
throw new Error("LANGBOT_BACKEND_URL is not configured.");
|
||||
}
|
||||
if (!["person", "group"].includes(sessionType)) {
|
||||
throw new Error(`LANGBOT_DEBUG_CHAT_LOAD_SESSION_TYPE must be person or group, got ${sessionType}.`);
|
||||
}
|
||||
for (const target of pipelineTargets) {
|
||||
if (!target.url && !target.name) {
|
||||
result.status = "env_issue";
|
||||
throw new Error(`Set LANGBOT_FAKE_PROVIDER_PIPELINE_${target.label}_URL or LANGBOT_FAKE_PROVIDER_PIPELINE_${target.label}_NAME.`);
|
||||
}
|
||||
}
|
||||
|
||||
const backendReady = await backendReachable(backendUrl);
|
||||
if (!backendReady) {
|
||||
result.status = "env_issue";
|
||||
throw new Error(`Backend did not respond at ${backendUrl}.`);
|
||||
}
|
||||
|
||||
const user = env.LANGBOT_E2E_LOGIN_USER || "";
|
||||
const password = env.LANGBOT_E2E_LOGIN_PASSWORD || DEFAULT_LOCAL_PASSWORD;
|
||||
if (!user) {
|
||||
result.status = "env_issue";
|
||||
throw new Error("LANGBOT_E2E_LOGIN_USER is required so this probe can resolve/reset Debug Chat sessions.");
|
||||
}
|
||||
const auth = await resetAndAuthLocalUser({ backendUrl, user, password });
|
||||
const pipelines = [];
|
||||
for (const target of pipelineTargets) {
|
||||
const pipeline = await resolvePipeline({
|
||||
backendUrl,
|
||||
token: auth.token,
|
||||
pipelineUrl: target.url,
|
||||
pipelineName: target.name,
|
||||
});
|
||||
pipelines.push({
|
||||
...target,
|
||||
id: pipeline.id,
|
||||
name: pipeline.name || target.name,
|
||||
wsUrl: websocketUrl(backendUrl, pipeline.id, sessionType),
|
||||
});
|
||||
}
|
||||
result.pipelines = pipelines.map((pipeline) => ({
|
||||
label: pipeline.label,
|
||||
id: pipeline.id,
|
||||
name: pipeline.name,
|
||||
url: pipeline.url,
|
||||
}));
|
||||
|
||||
if (resetBeforeRun) {
|
||||
const resetDiagnostics = [];
|
||||
for (const pipeline of pipelines) {
|
||||
const reset = await apiJson(backendUrl, `/api/v1/pipelines/${encodeURIComponent(pipeline.id)}/ws/reset/${encodeURIComponent(sessionType)}`, {
|
||||
method: "POST",
|
||||
token: auth.token,
|
||||
});
|
||||
resetDiagnostics.push({
|
||||
pipeline_label: pipeline.label,
|
||||
pipeline_id: pipeline.id,
|
||||
status: isApiFailure(reset) ? "fail" : "ready",
|
||||
http_status: reset.status,
|
||||
code: reset.json.code ?? null,
|
||||
reason: isApiFailure(reset) ? reset.json.msg || "Debug Chat reset failed." : "Debug Chat session reset.",
|
||||
});
|
||||
}
|
||||
await writeFile(resetDiagnosticPath, `${JSON.stringify(resetDiagnostics, null, 2)}\n`, "utf8");
|
||||
const failedReset = resetDiagnostics.find((item) => item.status === "fail");
|
||||
if (failedReset) throw new Error(failedReset.reason);
|
||||
}
|
||||
await resetFakeProvider(fakeProviderUrl);
|
||||
|
||||
const jobs = [];
|
||||
for (let index = 0; index < requestsPerPipeline; index += 1) {
|
||||
for (const pipeline of pipelines) {
|
||||
jobs.push({ ...pipeline, index });
|
||||
}
|
||||
}
|
||||
|
||||
const loadStartedAt = performance.now();
|
||||
const samples = await runLoad({
|
||||
jobs,
|
||||
concurrency,
|
||||
timeoutMs,
|
||||
promptTemplate,
|
||||
stream,
|
||||
failureSignals,
|
||||
});
|
||||
const loadDurationMs = performance.now() - loadStartedAt;
|
||||
const fakeProviderState = await readFakeProviderState(fakeProviderUrl);
|
||||
if (fakeProviderState) {
|
||||
await writeFile(fakeProviderStatePath, `${JSON.stringify(fakeProviderState, null, 2)}\n`, "utf8");
|
||||
}
|
||||
const metrics = buildMetrics({
|
||||
samples,
|
||||
requestsPerPipeline,
|
||||
concurrency,
|
||||
timeoutMs,
|
||||
loadDurationMs,
|
||||
backendUrl,
|
||||
sessionType,
|
||||
fakeProviderState,
|
||||
});
|
||||
const thresholds = buildThresholds(metrics);
|
||||
const passed = Object.values(thresholds).every((item) => item.pass);
|
||||
result.status = passed ? "pass" : "fail";
|
||||
result.reason = passed
|
||||
? "Debug Chat cross-pipeline isolation probe passed all thresholds."
|
||||
: "Debug Chat cross-pipeline isolation probe found leaks, errors, or latency threshold breaches.";
|
||||
result.metrics_summary = {
|
||||
requests_per_pipeline: metrics.requests_per_pipeline,
|
||||
total_requests: metrics.total_requests,
|
||||
concurrency: metrics.concurrency,
|
||||
ok_count: metrics.ok_count,
|
||||
error_count: metrics.error_count,
|
||||
cross_pipeline_leak_count: metrics.cross_pipeline_leak_count,
|
||||
timeout_count: metrics.timeout_count,
|
||||
error_rate: metrics.error_rate,
|
||||
response_p95_ms: metrics.response_duration_ms.p95,
|
||||
first_response_p95_ms: metrics.first_response_ms.p95,
|
||||
throughput_rps: metrics.throughput_rps,
|
||||
status_counts: metrics.status_counts,
|
||||
by_pipeline: metrics.by_pipeline,
|
||||
fake_provider_request_count: metrics.fake_provider?.request_count ?? null,
|
||||
fake_provider_duration_p95_ms: metrics.provider_timing?.provider_duration_ms.p95 ?? null,
|
||||
langbot_overhead_estimate_p95_ms: metrics.provider_timing?.langbot_overhead_estimate_ms.p95 ?? null,
|
||||
send_to_provider_start_p95_ms: metrics.provider_timing?.send_to_provider_start_ms.p95 ?? null,
|
||||
provider_finish_to_ws_final_p95_ms: metrics.provider_timing?.provider_finish_to_ws_final_ms.p95 ?? null,
|
||||
};
|
||||
result.thresholds_summary = thresholds;
|
||||
result.artifacts = {
|
||||
metrics_json: metricsPath,
|
||||
samples_json: samplesPath,
|
||||
fake_provider_state_json: fakeProviderState ? fakeProviderStatePath : "",
|
||||
network_log: paths.networkLog,
|
||||
automation_result_json: paths.automationResultJson,
|
||||
result_json: paths.resultJson,
|
||||
};
|
||||
|
||||
await writeFile(metricsPath, `${JSON.stringify({ ...metrics, thresholds }, null, 2)}\n`, "utf8");
|
||||
await writeFile(samplesPath, `${JSON.stringify(samples, null, 2)}\n`, "utf8");
|
||||
} catch (error) {
|
||||
if (!["env_issue", "blocked"].includes(result.status)) {
|
||||
result.status = looksLikeEnvIssue(error) ? "env_issue" : "fail";
|
||||
}
|
||||
result.reason = result.reason || safeReason(error.message);
|
||||
} finally {
|
||||
const finishedAt = new Date();
|
||||
result.finished_at = finishedAt.toISOString();
|
||||
result.finished_at_local = localIsoWithOffset(finishedAt);
|
||||
result.duration_ms = finishedAt.getTime() - startedAt.getTime();
|
||||
await mkdir(paths.evidenceDir, { recursive: true });
|
||||
await writeResult(paths, result);
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
}
|
||||
|
||||
exit(result.status === "pass" ? 0 : result.status === "env_issue" || result.status === "blocked" ? 2 : 1);
|
||||
|
||||
async function backendReachable(baseUrl) {
|
||||
try {
|
||||
const response = await fetch(`${baseUrl.replace(/\/$/, "")}/healthz`, {
|
||||
signal: AbortSignal.timeout(3000),
|
||||
});
|
||||
return response.status < 500;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async function resetFakeProvider(rootUrl) {
|
||||
if (!rootUrl) return;
|
||||
try {
|
||||
await fetch(`${normalizeProviderRootUrl(rootUrl)}/__qa/reset`, {
|
||||
method: "POST",
|
||||
signal: AbortSignal.timeout(3000),
|
||||
});
|
||||
} catch {
|
||||
// Missing fake-provider diagnostics should not hide the isolation result.
|
||||
}
|
||||
}
|
||||
|
||||
async function readFakeProviderState(rootUrl) {
|
||||
if (!rootUrl) return null;
|
||||
try {
|
||||
const response = await fetch(`${normalizeProviderRootUrl(rootUrl)}/__qa/config`, {
|
||||
signal: AbortSignal.timeout(3000),
|
||||
});
|
||||
const json = await response.json().catch(() => ({}));
|
||||
return {
|
||||
status: response.ok && json.ok === true ? "loaded" : "unavailable",
|
||||
url: normalizeProviderRootUrl(rootUrl),
|
||||
http_status: response.status,
|
||||
model: json.model || "",
|
||||
config: json.config || {},
|
||||
request_count: Number.isFinite(json.request_count) ? json.request_count : null,
|
||||
recent_requests: Array.isArray(json.recent_requests) ? json.recent_requests : [],
|
||||
};
|
||||
} catch (error) {
|
||||
return {
|
||||
status: "unavailable",
|
||||
url: normalizeProviderRootUrl(rootUrl),
|
||||
reason: safeReason(error.message),
|
||||
request_count: null,
|
||||
recent_requests: [],
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
function normalizeProviderRootUrl(value) {
|
||||
const trimmed = String(value || "").trim().replace(/\/$/, "");
|
||||
return trimmed.endsWith("/v1") ? trimmed.slice(0, -3) : trimmed;
|
||||
}
|
||||
|
||||
function pipelineIdFromUrl(url) {
|
||||
if (!url) return "";
|
||||
try {
|
||||
const parsed = new URL(url);
|
||||
return parsed.searchParams.get("id") || "";
|
||||
} catch {
|
||||
return "";
|
||||
}
|
||||
}
|
||||
|
||||
async function resolvePipeline({ backendUrl, token, pipelineUrl, pipelineName }) {
|
||||
const idFromUrl = pipelineIdFromUrl(pipelineUrl);
|
||||
if (idFromUrl) {
|
||||
const response = await apiJson(backendUrl, `/api/v1/pipelines/${encodeURIComponent(idFromUrl)}`, { token });
|
||||
const pipeline = response.json.data?.pipeline;
|
||||
if (isApiFailure(response) || !pipeline?.uuid) {
|
||||
throw new Error(response.json.msg || `Could not load pipeline ${idFromUrl}.`);
|
||||
}
|
||||
return { id: pipeline.uuid, name: pipeline.name || "" };
|
||||
}
|
||||
if (!pipelineName) {
|
||||
throw new Error("Set pipeline URL or name before running this probe.");
|
||||
}
|
||||
const response = await apiJson(backendUrl, "/api/v1/pipelines", { token });
|
||||
if (isApiFailure(response)) {
|
||||
throw new Error(response.json.msg || "Failed to list pipelines.");
|
||||
}
|
||||
const pipeline = (response.json.data?.pipelines || []).find((item) => item.name === pipelineName);
|
||||
if (!pipeline?.uuid) {
|
||||
throw new Error(`Could not find pipeline named ${pipelineName}.`);
|
||||
}
|
||||
return { id: pipeline.uuid, name: pipeline.name || pipelineName };
|
||||
}
|
||||
|
||||
function isApiFailure(response) {
|
||||
return response.status >= 400 || (response.json.code !== undefined && response.json.code !== 0);
|
||||
}
|
||||
|
||||
function websocketUrl(baseUrl, pipelineId, sessionTypeValue) {
|
||||
const parsed = new URL(baseUrl);
|
||||
parsed.protocol = parsed.protocol === "https:" ? "wss:" : "ws:";
|
||||
parsed.pathname = `/api/v1/pipelines/${encodeURIComponent(pipelineId)}/ws/connect`;
|
||||
parsed.search = `?session_type=${encodeURIComponent(sessionTypeValue)}`;
|
||||
return parsed.toString();
|
||||
}
|
||||
|
||||
async function runLoad(options) {
|
||||
const samples = [];
|
||||
const queue = [...options.jobs];
|
||||
const workers = Array.from({ length: options.concurrency }, async () => {
|
||||
while (queue.length > 0) {
|
||||
const job = queue.shift();
|
||||
if (!job) continue;
|
||||
const sample = await runSingleRequest({ ...options, job });
|
||||
samples.push(sample);
|
||||
}
|
||||
});
|
||||
await Promise.all(workers);
|
||||
return samples.sort((left, right) => (
|
||||
left.pipeline_label.localeCompare(right.pipeline_label) || left.index - right.index
|
||||
));
|
||||
}
|
||||
|
||||
function expectedForIndex(prefix, index) {
|
||||
return `${prefix}-${String(index + 1).padStart(4, "0")}`;
|
||||
}
|
||||
|
||||
function promptForIndex(template, expected) {
|
||||
return template.replaceAll("{expected}", expected);
|
||||
}
|
||||
|
||||
function runSingleRequest({
|
||||
job,
|
||||
timeoutMs,
|
||||
promptTemplate,
|
||||
stream,
|
||||
failureSignals,
|
||||
}) {
|
||||
return new Promise((resolvePromise) => {
|
||||
const expected = expectedForIndex(job.expectedPrefix, job.index);
|
||||
const prompt = promptForIndex(promptTemplate, expected);
|
||||
const sample = {
|
||||
index: job.index,
|
||||
pipeline_label: job.label,
|
||||
pipeline_id: job.id,
|
||||
pipeline_name: job.name,
|
||||
status: "running",
|
||||
ok: false,
|
||||
expected_text: expected,
|
||||
expected_prefix: job.expectedPrefix,
|
||||
other_prefix: job.otherPrefix,
|
||||
prompt,
|
||||
response_text: "",
|
||||
started_at: new Date().toISOString(),
|
||||
started_epoch_ms: Date.now(),
|
||||
connected_at: null,
|
||||
connected_epoch_ms: null,
|
||||
sent_at: null,
|
||||
sent_epoch_ms: null,
|
||||
first_assistant_event_at: null,
|
||||
first_assistant_event_epoch_ms: null,
|
||||
first_assistant_event_ms: null,
|
||||
first_assistant_content_at: null,
|
||||
first_assistant_content_epoch_ms: null,
|
||||
first_assistant_content_ms: null,
|
||||
first_response_at: null,
|
||||
first_response_epoch_ms: null,
|
||||
connected_ms: null,
|
||||
first_response_ms: null,
|
||||
response_duration_ms: null,
|
||||
finished_at: null,
|
||||
finished_epoch_ms: null,
|
||||
event_count: 0,
|
||||
same_pipeline_foreign_response_count: 0,
|
||||
cross_pipeline_leak_count: 0,
|
||||
last_foreign_response_text: "",
|
||||
error: "",
|
||||
close_code: null,
|
||||
close_reason: "",
|
||||
};
|
||||
let closed = false;
|
||||
let connectedAt = 0;
|
||||
let sentAt = 0;
|
||||
const startedPerf = performance.now();
|
||||
let client = null;
|
||||
const timer = setTimeout(() => {
|
||||
finish("timeout", `Timed out after ${timeoutMs} ms.`);
|
||||
}, timeoutMs);
|
||||
|
||||
client = openRawWebSocket(job.wsUrl, {
|
||||
onOpen() {
|
||||
connectedAt = performance.now();
|
||||
const now = Date.now();
|
||||
sample.connected_at = new Date(now).toISOString();
|
||||
sample.connected_epoch_ms = now;
|
||||
sample.connected_ms = rounded(connectedAt - startedPerf);
|
||||
},
|
||||
onMessage(text) {
|
||||
sample.event_count += 1;
|
||||
let data;
|
||||
try {
|
||||
data = JSON.parse(String(text || ""));
|
||||
} catch (error) {
|
||||
finish("error", `Invalid WebSocket JSON: ${error.message}`);
|
||||
return;
|
||||
}
|
||||
appendLine(paths.networkLog, JSON.stringify({
|
||||
pipeline_label: job.label,
|
||||
request_index: job.index,
|
||||
type: data.type,
|
||||
session_type: data.session_type || "",
|
||||
role: data.data?.role || "",
|
||||
is_final: data.data?.is_final ?? null,
|
||||
content_preview: redact(String(data.data?.content || data.message || "").slice(0, 200)),
|
||||
})).catch(() => {});
|
||||
|
||||
if (data.type === "connected") {
|
||||
sentAt = performance.now();
|
||||
const now = Date.now();
|
||||
sample.sent_at = new Date(now).toISOString();
|
||||
sample.sent_epoch_ms = now;
|
||||
client.send(JSON.stringify({
|
||||
type: "message",
|
||||
message: [{ type: "Plain", text: prompt }],
|
||||
stream,
|
||||
}));
|
||||
return;
|
||||
}
|
||||
if (data.type === "error") {
|
||||
finish("error", data.message || "WebSocket error message.");
|
||||
return;
|
||||
}
|
||||
if (data.type !== "response" || data.data?.role !== "assistant") return;
|
||||
|
||||
const content = String(data.data.content || "");
|
||||
markFirstAssistantEvent(sample, sentAt);
|
||||
if (content) sample.response_text = content;
|
||||
if (content) markFirstAssistantContent(sample, sentAt);
|
||||
if (containsPipelineToken(content, job.otherPrefix)) {
|
||||
sample.cross_pipeline_leak_count += 1;
|
||||
finish("cross_pipeline_leak", `Pipeline ${job.label} received response from ${job.otherPrefix}: ${content}`);
|
||||
return;
|
||||
}
|
||||
if (content.includes(expected) && sample.first_response_ms === null && sentAt > 0) {
|
||||
const now = Date.now();
|
||||
sample.first_response_at = new Date(now).toISOString();
|
||||
sample.first_response_epoch_ms = now;
|
||||
sample.first_response_ms = rounded(performance.now() - sentAt);
|
||||
}
|
||||
if (data.data.is_final === true) {
|
||||
const ok = sample.response_text.includes(expected);
|
||||
if (ok) {
|
||||
if (sample.first_response_ms === null && sentAt > 0) {
|
||||
const now = Date.now();
|
||||
sample.first_response_at = new Date(now).toISOString();
|
||||
sample.first_response_epoch_ms = now;
|
||||
sample.first_response_ms = rounded(performance.now() - sentAt);
|
||||
}
|
||||
finish("pass", "");
|
||||
} else if (matchesFailureSignal(sample.response_text, failureSignals)) {
|
||||
finish("app_error", `Assistant final response matched a failure signal: ${sample.response_text}`);
|
||||
} else if (containsPipelineToken(sample.response_text, job.expectedPrefix)) {
|
||||
sample.same_pipeline_foreign_response_count += 1;
|
||||
sample.last_foreign_response_text = sample.response_text;
|
||||
} else {
|
||||
finish("mismatch", `Final assistant response did not include ${expected}: ${sample.response_text}`);
|
||||
}
|
||||
}
|
||||
},
|
||||
onError(error) {
|
||||
finish("connection_error", `WebSocket connection error: ${error.message}`);
|
||||
},
|
||||
onClose(event) {
|
||||
sample.close_code = event.code;
|
||||
sample.close_reason = event.reason || "";
|
||||
if (!closed) finish("closed", `WebSocket closed before final assistant response: ${event.code}`);
|
||||
},
|
||||
});
|
||||
|
||||
function finish(status, reason) {
|
||||
if (closed) return;
|
||||
closed = true;
|
||||
clearTimeout(timer);
|
||||
sample.status = status;
|
||||
sample.ok = status === "pass";
|
||||
sample.error = status === "timeout" && sample.same_pipeline_foreign_response_count > 0
|
||||
? `${reason || ""} Saw ${sample.same_pipeline_foreign_response_count} same-pipeline foreign assistant response(s); last=${sample.last_foreign_response_text}`
|
||||
: reason || "";
|
||||
if (sentAt > 0) sample.response_duration_ms = rounded(performance.now() - sentAt);
|
||||
else sample.response_duration_ms = rounded(performance.now() - startedPerf);
|
||||
const now = Date.now();
|
||||
sample.finished_at = new Date(now).toISOString();
|
||||
sample.finished_epoch_ms = now;
|
||||
try {
|
||||
client?.close();
|
||||
} catch {
|
||||
// Closing a failed socket should not hide the sample result.
|
||||
}
|
||||
resolvePromise(sample);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
function markFirstAssistantEvent(sample, sentAt) {
|
||||
if (sample.first_assistant_event_ms !== null || sentAt <= 0) return;
|
||||
const now = Date.now();
|
||||
sample.first_assistant_event_at = new Date(now).toISOString();
|
||||
sample.first_assistant_event_epoch_ms = now;
|
||||
sample.first_assistant_event_ms = rounded(performance.now() - sentAt);
|
||||
}
|
||||
|
||||
function markFirstAssistantContent(sample, sentAt) {
|
||||
if (sample.first_assistant_content_ms !== null || sentAt <= 0) return;
|
||||
const now = Date.now();
|
||||
sample.first_assistant_content_at = new Date(now).toISOString();
|
||||
sample.first_assistant_content_epoch_ms = now;
|
||||
sample.first_assistant_content_ms = rounded(performance.now() - sentAt);
|
||||
}
|
||||
|
||||
function containsPipelineToken(text, prefix) {
|
||||
const escaped = String(prefix).replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
||||
return new RegExp(`${escaped}-\\d{4}`).test(String(text || ""));
|
||||
}
|
||||
|
||||
function matchesFailureSignal(text, signals) {
|
||||
const lower = String(text || "").toLowerCase();
|
||||
return signals.some((signal) => lower.includes(signal.toLowerCase()));
|
||||
}
|
||||
|
||||
function openRawWebSocket(wsUrl, handlers) {
|
||||
const parsed = new URL(wsUrl);
|
||||
const secure = parsed.protocol === "wss:";
|
||||
const port = Number(parsed.port || (secure ? 443 : 80));
|
||||
const host = parsed.hostname;
|
||||
const path = `${parsed.pathname}${parsed.search}`;
|
||||
const key = crypto.randomBytes(16).toString("base64");
|
||||
const socket = secure
|
||||
? tls.connect({ host, port, servername: host })
|
||||
: net.connect({ host, port });
|
||||
let opened = false;
|
||||
let closed = false;
|
||||
let buffer = Buffer.alloc(0);
|
||||
|
||||
socket.setNoDelay(true);
|
||||
socket.on("connect", () => {
|
||||
const originProtocol = secure ? "https" : "http";
|
||||
const request = [
|
||||
`GET ${path} HTTP/1.1`,
|
||||
`Host: ${parsed.host}`,
|
||||
"Upgrade: websocket",
|
||||
"Connection: Upgrade",
|
||||
`Sec-WebSocket-Key: ${key}`,
|
||||
"Sec-WebSocket-Version: 13",
|
||||
`Origin: ${originProtocol}://${parsed.host}`,
|
||||
"",
|
||||
"",
|
||||
].join("\r\n");
|
||||
socket.write(request);
|
||||
});
|
||||
socket.on("data", (chunk) => {
|
||||
buffer = Buffer.concat([buffer, chunk]);
|
||||
if (!opened) {
|
||||
const headerEnd = buffer.indexOf("\r\n\r\n");
|
||||
if (headerEnd === -1) return;
|
||||
const headerText = buffer.slice(0, headerEnd).toString("utf8");
|
||||
buffer = buffer.slice(headerEnd + 4);
|
||||
if (!/^HTTP\/1\.1 101\b/i.test(headerText)) {
|
||||
handlers.onError(new Error(`Handshake failed: ${headerText.split("\r\n")[0] || "missing status"}`));
|
||||
socket.destroy();
|
||||
return;
|
||||
}
|
||||
opened = true;
|
||||
handlers.onOpen();
|
||||
}
|
||||
processFrames();
|
||||
});
|
||||
socket.on("error", (error) => {
|
||||
if (!closed) handlers.onError(error);
|
||||
});
|
||||
socket.on("close", () => {
|
||||
if (closed) return;
|
||||
closed = true;
|
||||
handlers.onClose({ code: null, reason: "" });
|
||||
});
|
||||
|
||||
function processFrames() {
|
||||
while (true) {
|
||||
const frame = readFrame(buffer);
|
||||
if (!frame) return;
|
||||
buffer = buffer.slice(frame.consumed);
|
||||
if (frame.opcode === 0x1) {
|
||||
handlers.onMessage(frame.payload.toString("utf8"));
|
||||
} else if (frame.opcode === 0x8) {
|
||||
const code = frame.payload.length >= 2 ? frame.payload.readUInt16BE(0) : null;
|
||||
const reason = frame.payload.length > 2 ? frame.payload.slice(2).toString("utf8") : "";
|
||||
closed = true;
|
||||
handlers.onClose({ code, reason });
|
||||
socket.end();
|
||||
return;
|
||||
} else if (frame.opcode === 0x9) {
|
||||
writeFrame(socket, 0xA, frame.payload);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
send(text) {
|
||||
if (closed || !opened) return;
|
||||
writeFrame(socket, 0x1, Buffer.from(text, "utf8"));
|
||||
},
|
||||
close() {
|
||||
if (closed) return;
|
||||
closed = true;
|
||||
if (!socket.destroyed) {
|
||||
if (opened) writeFrame(socket, 0x8, Buffer.alloc(0));
|
||||
setTimeout(() => socket.end(), 50).unref();
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function readFrame(buffer) {
|
||||
if (buffer.length < 2) return null;
|
||||
const first = buffer[0];
|
||||
const second = buffer[1];
|
||||
const opcode = first & 0x0f;
|
||||
const masked = Boolean(second & 0x80);
|
||||
let length = second & 0x7f;
|
||||
let offset = 2;
|
||||
if (length === 126) {
|
||||
if (buffer.length < offset + 2) return null;
|
||||
length = buffer.readUInt16BE(offset);
|
||||
offset += 2;
|
||||
} else if (length === 127) {
|
||||
if (buffer.length < offset + 8) return null;
|
||||
const high = buffer.readUInt32BE(offset);
|
||||
const low = buffer.readUInt32BE(offset + 4);
|
||||
length = high * 2 ** 32 + low;
|
||||
offset += 8;
|
||||
}
|
||||
let mask = null;
|
||||
if (masked) {
|
||||
if (buffer.length < offset + 4) return null;
|
||||
mask = buffer.slice(offset, offset + 4);
|
||||
offset += 4;
|
||||
}
|
||||
if (buffer.length < offset + length) return null;
|
||||
let payload = buffer.slice(offset, offset + length);
|
||||
if (mask) {
|
||||
payload = Buffer.from(payload);
|
||||
for (let index = 0; index < payload.length; index += 1) {
|
||||
payload[index] ^= mask[index % 4];
|
||||
}
|
||||
}
|
||||
return {
|
||||
opcode,
|
||||
payload,
|
||||
consumed: offset + length,
|
||||
};
|
||||
}
|
||||
|
||||
function writeFrame(socket, opcode, payload) {
|
||||
const body = Buffer.isBuffer(payload) ? payload : Buffer.from(payload || "");
|
||||
const mask = crypto.randomBytes(4);
|
||||
const headerLength = body.length < 126 ? 2 : body.length <= 0xffff ? 4 : 10;
|
||||
const header = Buffer.alloc(headerLength);
|
||||
header[0] = 0x80 | opcode;
|
||||
if (body.length < 126) {
|
||||
header[1] = 0x80 | body.length;
|
||||
} else if (body.length <= 0xffff) {
|
||||
header[1] = 0x80 | 126;
|
||||
header.writeUInt16BE(body.length, 2);
|
||||
} else {
|
||||
header[1] = 0x80 | 127;
|
||||
header.writeUInt32BE(Math.floor(body.length / 2 ** 32), 2);
|
||||
header.writeUInt32BE(body.length >>> 0, 6);
|
||||
}
|
||||
const masked = Buffer.from(body);
|
||||
for (let index = 0; index < masked.length; index += 1) {
|
||||
masked[index] ^= mask[index % 4];
|
||||
}
|
||||
socket.write(Buffer.concat([header, mask, masked]));
|
||||
}
|
||||
|
||||
function buildMetrics({ samples, requestsPerPipeline, concurrency, timeoutMs, loadDurationMs, backendUrl, sessionType, fakeProviderState }) {
|
||||
const okSamples = samples.filter((sample) => sample.ok);
|
||||
const statusCounts = {};
|
||||
const byPipeline = {};
|
||||
for (const sample of samples) {
|
||||
statusCounts[sample.status] = (statusCounts[sample.status] || 0) + 1;
|
||||
if (!byPipeline[sample.pipeline_label]) {
|
||||
byPipeline[sample.pipeline_label] = {
|
||||
ok_count: 0,
|
||||
error_count: 0,
|
||||
cross_pipeline_leak_count: 0,
|
||||
timeout_count: 0,
|
||||
};
|
||||
}
|
||||
if (sample.ok) byPipeline[sample.pipeline_label].ok_count += 1;
|
||||
else byPipeline[sample.pipeline_label].error_count += 1;
|
||||
byPipeline[sample.pipeline_label].cross_pipeline_leak_count += sample.cross_pipeline_leak_count || 0;
|
||||
if (sample.status === "timeout") byPipeline[sample.pipeline_label].timeout_count += 1;
|
||||
}
|
||||
const errorCount = samples.length - okSamples.length;
|
||||
return {
|
||||
probe: caseId,
|
||||
backend_url: backendUrl,
|
||||
session_type: sessionType,
|
||||
requests_per_pipeline: requestsPerPipeline,
|
||||
total_requests: requestsPerPipeline * 2,
|
||||
completed_requests: samples.length,
|
||||
concurrency,
|
||||
timeout_ms: timeoutMs,
|
||||
ok_count: okSamples.length,
|
||||
error_count: errorCount,
|
||||
timeout_count: samples.filter((sample) => sample.status === "timeout").length,
|
||||
cross_pipeline_leak_count: samples.reduce((count, sample) => count + (sample.cross_pipeline_leak_count || 0), 0),
|
||||
error_rate: samples.length === 0 ? 1 : rounded(errorCount / samples.length),
|
||||
load_duration_ms: rounded(loadDurationMs),
|
||||
throughput_rps: loadDurationMs <= 0 ? 0 : rounded(okSamples.length / (loadDurationMs / 1000)),
|
||||
status_counts: statusCounts,
|
||||
by_pipeline: byPipeline,
|
||||
connected_ms: stats(samples.map((sample) => sample.connected_ms).filter(Number.isFinite)),
|
||||
first_assistant_event_ms: stats(samples.map((sample) => sample.first_assistant_event_ms).filter(Number.isFinite)),
|
||||
first_assistant_content_ms: stats(samples.map((sample) => sample.first_assistant_content_ms).filter(Number.isFinite)),
|
||||
first_response_ms: stats(okSamples.map((sample) => sample.first_response_ms).filter(Number.isFinite)),
|
||||
response_duration_ms: stats(okSamples.map((sample) => sample.response_duration_ms).filter(Number.isFinite)),
|
||||
fake_provider: summarizeFakeProviderState(fakeProviderState),
|
||||
provider_timing: buildProviderTimingMetrics(samples, fakeProviderState),
|
||||
samples,
|
||||
};
|
||||
}
|
||||
|
||||
function buildThresholds(metrics) {
|
||||
return {
|
||||
cross_pipeline_leak_count: {
|
||||
actual: metrics.cross_pipeline_leak_count,
|
||||
max: 0,
|
||||
pass: metrics.cross_pipeline_leak_count === 0,
|
||||
},
|
||||
error_rate: {
|
||||
actual: metrics.error_rate,
|
||||
max: maxErrorRate,
|
||||
pass: metrics.error_rate <= maxErrorRate,
|
||||
},
|
||||
response_p95_ms: {
|
||||
actual: metrics.response_duration_ms.p95,
|
||||
max: responseP95BudgetMs,
|
||||
pass: metrics.ok_count > 0 && metrics.response_duration_ms.p95 <= responseP95BudgetMs,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function positiveInteger(value, fallback) {
|
||||
const parsed = Number.parseInt(String(value || ""), 10);
|
||||
return Number.isInteger(parsed) && parsed > 0 ? parsed : fallback;
|
||||
}
|
||||
|
||||
function positiveNumber(value, fallback) {
|
||||
const parsed = Number(value || "");
|
||||
return Number.isFinite(parsed) && parsed >= 0 ? parsed : fallback;
|
||||
}
|
||||
|
||||
function bool(value, fallback) {
|
||||
if (value === undefined || value === "") return fallback;
|
||||
if (/^(1|true|yes|on)$/i.test(String(value))) return true;
|
||||
if (/^(0|false|no|off)$/i.test(String(value))) return false;
|
||||
return fallback;
|
||||
}
|
||||
|
||||
function textList(value) {
|
||||
return String(value || "")
|
||||
.split(/\r?\n|,/)
|
||||
.map((item) => item.trim())
|
||||
.filter(Boolean);
|
||||
}
|
||||
|
||||
function rounded(value) {
|
||||
return Number(value.toFixed(3));
|
||||
}
|
||||
|
||||
function percentile(values, percentileValue) {
|
||||
if (values.length === 0) return 0;
|
||||
const sorted = [...values].sort((a, b) => a - b);
|
||||
const index = Math.min(sorted.length - 1, Math.ceil((percentileValue / 100) * sorted.length) - 1);
|
||||
return rounded(sorted[index]);
|
||||
}
|
||||
|
||||
function stats(values) {
|
||||
if (values.length === 0) return { min: 0, p50: 0, p95: 0, p99: 0, max: 0 };
|
||||
return {
|
||||
min: rounded(Math.min(...values)),
|
||||
p50: percentile(values, 50),
|
||||
p95: percentile(values, 95),
|
||||
p99: percentile(values, 99),
|
||||
max: rounded(Math.max(...values)),
|
||||
};
|
||||
}
|
||||
|
||||
function looksLikeEnvIssue(error) {
|
||||
const message = String(error?.message || error || "");
|
||||
return /fetch failed|ECONNREFUSED|ENOTFOUND|LANGBOT_.*not configured|Could not read recovery_key|Backend did not respond/i.test(message);
|
||||
}
|
||||
|
||||
function safeReason(value) {
|
||||
return redact(String(value || "")).slice(0, 1000);
|
||||
}
|
||||
@@ -1,159 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
import { join, resolve } from "node:path";
|
||||
import { env, exit } from "node:process";
|
||||
|
||||
function pad(value, size = 2) {
|
||||
return String(value).padStart(size, "0");
|
||||
}
|
||||
|
||||
function localIsoWithOffset(date = new Date()) {
|
||||
const offsetMinutes = -date.getTimezoneOffset();
|
||||
const sign = offsetMinutes >= 0 ? "+" : "-";
|
||||
const absolute = Math.abs(offsetMinutes);
|
||||
return [
|
||||
`${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())}`,
|
||||
`T${pad(date.getHours())}:${pad(date.getMinutes())}:${pad(date.getSeconds())}.${pad(date.getMilliseconds(), 3)}`,
|
||||
`${sign}${pad(Math.floor(absolute / 60))}:${pad(absolute % 60)}`,
|
||||
].join("");
|
||||
}
|
||||
|
||||
function timestampSlug(date = new Date()) {
|
||||
return date.toISOString().replace(/\.\d{3}Z$/, "Z").replace(/[^0-9A-Za-z]+/g, "-").replace(/^-|-$/g, "");
|
||||
}
|
||||
|
||||
const scenarios = [
|
||||
{
|
||||
id: "provider-timeout",
|
||||
target: "provider",
|
||||
injected_fault: "fake provider request exceeds the configured timeout",
|
||||
expected_status: "env_issue",
|
||||
recovery_check: "provider route is reachable or the case remains outside product pass/fail",
|
||||
cleanup: "stop fake provider or reset proxy route",
|
||||
},
|
||||
{
|
||||
id: "plugin-runtime-disconnect",
|
||||
target: "plugin-runtime",
|
||||
injected_fault: "runtime control channel disconnects during an action",
|
||||
expected_status: "fail",
|
||||
recovery_check: "runtime reconnects and a deterministic plugin action succeeds",
|
||||
cleanup: "restart the local plugin runtime process",
|
||||
},
|
||||
{
|
||||
id: "mcp-stdio-server-exit",
|
||||
target: "mcp",
|
||||
injected_fault: "stdio server exits mid-call",
|
||||
expected_status: "fail",
|
||||
recovery_check: "server can be registered again and exposes the expected tool",
|
||||
cleanup: "remove temporary MCP server registration",
|
||||
},
|
||||
{
|
||||
id: "operator-missing-login",
|
||||
target: "webui",
|
||||
injected_fault: "browser profile is not authenticated",
|
||||
expected_status: "blocked",
|
||||
recovery_check: "authenticated profile can open the same WebUI origin",
|
||||
cleanup: "no product cleanup; refresh local login state",
|
||||
},
|
||||
{
|
||||
id: "transient-marketplace-timeout",
|
||||
target: "marketplace",
|
||||
injected_fault: "marketplace request times out once and then succeeds",
|
||||
expected_status: "flaky",
|
||||
recovery_check: "rerun passes with the same product revision and no code change",
|
||||
cleanup: "clear retry-only evidence and keep the run classified as flaky",
|
||||
},
|
||||
];
|
||||
|
||||
function validateScenario(scenario) {
|
||||
const missing = ["id", "target", "injected_fault", "expected_status", "recovery_check", "cleanup"]
|
||||
.filter((key) => !scenario[key]);
|
||||
const allowedStatuses = new Set(["pass", "fail", "blocked", "env_issue", "flaky"]);
|
||||
return {
|
||||
id: scenario.id,
|
||||
pass: missing.length === 0 && allowedStatuses.has(scenario.expected_status),
|
||||
missing,
|
||||
expected_status: scenario.expected_status,
|
||||
};
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const root = resolve(env.LBS_ROOT || process.cwd());
|
||||
const caseId = "langbot-fault-taxonomy-contract";
|
||||
const runId = env.LBS_RUN_ID || `${timestampSlug()}-${caseId}`;
|
||||
const evidenceDir = resolve(env.LBS_EVIDENCE_DIR || join(root, "reports", "evidence", runId));
|
||||
await mkdir(evidenceDir, { recursive: true });
|
||||
|
||||
const startedAt = new Date();
|
||||
const validations = scenarios.map(validateScenario);
|
||||
const statusCounts = {};
|
||||
for (const scenario of scenarios) {
|
||||
statusCounts[scenario.expected_status] = (statusCounts[scenario.expected_status] || 0) + 1;
|
||||
}
|
||||
const metrics = {
|
||||
probe: caseId,
|
||||
scenario_count: scenarios.length,
|
||||
status_counts: statusCounts,
|
||||
scenarios,
|
||||
validations,
|
||||
};
|
||||
const thresholds = {
|
||||
scenario_count: { actual: scenarios.length, min: 5, pass: scenarios.length >= 5 },
|
||||
invalid_scenario_count: {
|
||||
actual: validations.filter((item) => !item.pass).length,
|
||||
max: 0,
|
||||
pass: validations.every((item) => item.pass),
|
||||
},
|
||||
cleanup_declared_count: {
|
||||
actual: scenarios.filter((item) => item.cleanup).length,
|
||||
min: scenarios.length,
|
||||
pass: scenarios.every((item) => item.cleanup),
|
||||
},
|
||||
};
|
||||
const status = Object.values(thresholds).every((item) => item.pass) ? "pass" : "fail";
|
||||
const metricsPath = join(evidenceDir, "metrics.json");
|
||||
const faultModelPath = join(evidenceDir, "fault-model.json");
|
||||
const automationResultPath = join(evidenceDir, "automation-result.json");
|
||||
const resultPath = join(evidenceDir, "result.json");
|
||||
|
||||
await writeFile(metricsPath, `${JSON.stringify(metrics, null, 2)}\n`, "utf8");
|
||||
await writeFile(faultModelPath, `${JSON.stringify({ scenarios }, null, 2)}\n`, "utf8");
|
||||
|
||||
const finishedAt = new Date();
|
||||
const result = {
|
||||
source: "automation",
|
||||
case_id: caseId,
|
||||
run_id: runId,
|
||||
status,
|
||||
reason: status === "pass"
|
||||
? "Fault taxonomy contract declares status, recovery, and cleanup for every scenario."
|
||||
: "Fault taxonomy contract is missing required scenario fields.",
|
||||
started_at: startedAt.toISOString(),
|
||||
started_at_local: localIsoWithOffset(startedAt),
|
||||
finished_at: finishedAt.toISOString(),
|
||||
finished_at_local: localIsoWithOffset(finishedAt),
|
||||
duration_ms: finishedAt.getTime() - startedAt.getTime(),
|
||||
metrics_summary: {
|
||||
scenario_count: metrics.scenario_count,
|
||||
status_counts: metrics.status_counts,
|
||||
invalid_scenario_count: thresholds.invalid_scenario_count.actual,
|
||||
},
|
||||
thresholds_summary: thresholds,
|
||||
artifacts: {
|
||||
metrics_json: metricsPath,
|
||||
fault_model_json: faultModelPath,
|
||||
automation_result_json: automationResultPath,
|
||||
result_json: resultPath,
|
||||
},
|
||||
evidence_collected: ["metrics", "filesystem"],
|
||||
};
|
||||
|
||||
const resultText = `${JSON.stringify(result, null, 2)}\n`;
|
||||
await writeFile(automationResultPath, resultText, "utf8");
|
||||
await writeFile(resultPath, resultText, "utf8");
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
exit(status === "pass" ? 0 : 1);
|
||||
}
|
||||
|
||||
await main();
|
||||
@@ -1,212 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
import { join, resolve } from "node:path";
|
||||
import { env, exit } from "node:process";
|
||||
|
||||
function pad(value, size = 2) {
|
||||
return String(value).padStart(size, "0");
|
||||
}
|
||||
|
||||
function localIsoWithOffset(date = new Date()) {
|
||||
const offsetMinutes = -date.getTimezoneOffset();
|
||||
const sign = offsetMinutes >= 0 ? "+" : "-";
|
||||
const absolute = Math.abs(offsetMinutes);
|
||||
return [
|
||||
`${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())}`,
|
||||
`T${pad(date.getHours())}:${pad(date.getMinutes())}:${pad(date.getSeconds())}.${pad(date.getMilliseconds(), 3)}`,
|
||||
`${sign}${pad(Math.floor(absolute / 60))}:${pad(absolute % 60)}`,
|
||||
].join("");
|
||||
}
|
||||
|
||||
function timestampSlug(date = new Date()) {
|
||||
return date.toISOString().replace(/\.\d{3}Z$/, "Z").replace(/[^0-9A-Za-z]+/g, "-").replace(/^-|-$/g, "");
|
||||
}
|
||||
|
||||
function percentile(values, percentileValue) {
|
||||
if (values.length === 0) return 0;
|
||||
const sorted = [...values].sort((a, b) => a - b);
|
||||
const index = Math.min(sorted.length - 1, Math.ceil((percentileValue / 100) * sorted.length) - 1);
|
||||
return Number(sorted[index].toFixed(3));
|
||||
}
|
||||
|
||||
function stats(values) {
|
||||
if (values.length === 0) return { min: 0, p50: 0, p95: 0, p99: 0, max: 0 };
|
||||
return {
|
||||
min: Number(Math.min(...values).toFixed(3)),
|
||||
p50: percentile(values, 50),
|
||||
p95: percentile(values, 95),
|
||||
p99: percentile(values, 99),
|
||||
max: Number(Math.max(...values).toFixed(3)),
|
||||
};
|
||||
}
|
||||
|
||||
function parseJsonList(value, fallback) {
|
||||
if (!value) return fallback;
|
||||
try {
|
||||
const parsed = JSON.parse(value);
|
||||
return Array.isArray(parsed) && parsed.every((item) => typeof item === "string") ? parsed : fallback;
|
||||
} catch {
|
||||
return fallback;
|
||||
}
|
||||
}
|
||||
|
||||
function joinUrl(baseUrl, path) {
|
||||
const base = baseUrl.replace(/\/+$/, "");
|
||||
const suffix = path.startsWith("/") ? path : `/${path}`;
|
||||
return `${base}${suffix}`;
|
||||
}
|
||||
|
||||
async function fetchOnce(url, timeoutMs) {
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(() => controller.abort(), timeoutMs);
|
||||
const started = performance.now();
|
||||
try {
|
||||
const response = await fetch(url, { method: "GET", signal: controller.signal });
|
||||
await response.arrayBuffer();
|
||||
const latencyMs = performance.now() - started;
|
||||
return {
|
||||
url,
|
||||
ok: response.status < 500,
|
||||
status: response.status,
|
||||
latency_ms: Number(latencyMs.toFixed(3)),
|
||||
error: "",
|
||||
};
|
||||
} catch (error) {
|
||||
const latencyMs = performance.now() - started;
|
||||
return {
|
||||
url,
|
||||
ok: false,
|
||||
status: 0,
|
||||
latency_ms: Number(latencyMs.toFixed(3)),
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
};
|
||||
} finally {
|
||||
clearTimeout(timeout);
|
||||
}
|
||||
}
|
||||
|
||||
async function runBatches(urls, totalRequests, concurrency, timeoutMs) {
|
||||
const queue = Array.from({ length: totalRequests }, (_, index) => urls[index % urls.length]);
|
||||
const results = [];
|
||||
while (queue.length > 0) {
|
||||
const batch = queue.splice(0, concurrency);
|
||||
results.push(...await Promise.all(batch.map((url) => fetchOnce(url, timeoutMs))));
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const root = resolve(env.LBS_ROOT || process.cwd());
|
||||
const caseId = "langbot-live-backend-latency";
|
||||
const runId = env.LBS_RUN_ID || `${timestampSlug()}-${caseId}`;
|
||||
const evidenceDir = resolve(env.LBS_EVIDENCE_DIR || join(root, "reports", "evidence", runId));
|
||||
await mkdir(evidenceDir, { recursive: true });
|
||||
|
||||
const startedAt = new Date();
|
||||
const backendUrl = env.LANGBOT_BACKEND_URL || "";
|
||||
const endpoints = parseJsonList(env.LANGBOT_PERF_ENDPOINTS_JSON, ["/healthz"]);
|
||||
const totalRequests = Number(env.LANGBOT_PERF_REQUESTS || "12");
|
||||
const concurrency = Number(env.LANGBOT_PERF_CONCURRENCY || "2");
|
||||
const timeoutMs = Number(env.LANGBOT_PERF_TIMEOUT_MS || "5000");
|
||||
const p95BudgetMs = Number(env.LANGBOT_PERF_BACKEND_P95_MS || "1000");
|
||||
const maxErrorRate = Number(env.LANGBOT_PERF_MAX_ERROR_RATE || "0");
|
||||
const metricsPath = join(evidenceDir, "metrics.json");
|
||||
const networkLogPath = join(evidenceDir, "network.log");
|
||||
const automationResultPath = join(evidenceDir, "automation-result.json");
|
||||
const resultPath = join(evidenceDir, "result.json");
|
||||
|
||||
let status = "fail";
|
||||
let reason = "";
|
||||
let results = [];
|
||||
if (!backendUrl) {
|
||||
status = "env_issue";
|
||||
reason = "LANGBOT_BACKEND_URL is not configured.";
|
||||
} else {
|
||||
const urls = endpoints.map((path) => joinUrl(backendUrl, path));
|
||||
results = await runBatches(urls, totalRequests, concurrency, timeoutMs);
|
||||
const okCount = results.filter((item) => item.ok).length;
|
||||
const errorCount = results.length - okCount;
|
||||
const errorRate = results.length === 0 ? 1 : errorCount / results.length;
|
||||
const latencies = results.filter((item) => item.ok).map((item) => item.latency_ms);
|
||||
const latencyStats = stats(latencies);
|
||||
const allConnectionFailures = results.length > 0 && results.every((item) => item.status === 0);
|
||||
if (allConnectionFailures) {
|
||||
status = "env_issue";
|
||||
reason = `Backend did not respond at ${backendUrl}.`;
|
||||
} else if (latencyStats.p95 <= p95BudgetMs && errorRate <= maxErrorRate) {
|
||||
status = "pass";
|
||||
reason = "Live backend latency probe passed all thresholds.";
|
||||
} else {
|
||||
status = "fail";
|
||||
reason = "Live backend latency probe breached latency or error-rate thresholds.";
|
||||
}
|
||||
}
|
||||
|
||||
const statusCounts = {};
|
||||
for (const item of results) {
|
||||
const key = item.status === 0 ? "network_error" : String(item.status);
|
||||
statusCounts[key] = (statusCounts[key] || 0) + 1;
|
||||
}
|
||||
const okResults = results.filter((item) => item.ok);
|
||||
const metrics = {
|
||||
probe: caseId,
|
||||
backend_url: backendUrl,
|
||||
endpoints,
|
||||
total_requests: totalRequests,
|
||||
concurrency,
|
||||
timeout_ms: timeoutMs,
|
||||
ok_count: okResults.length,
|
||||
error_count: results.length - okResults.length,
|
||||
error_rate: results.length === 0 ? 1 : Number(((results.length - okResults.length) / results.length).toFixed(4)),
|
||||
latency_ms: stats(okResults.map((item) => item.latency_ms)),
|
||||
status_counts: statusCounts,
|
||||
};
|
||||
const thresholds = {
|
||||
backend_p95_ms: { actual: metrics.latency_ms.p95, max: p95BudgetMs, pass: metrics.latency_ms.p95 <= p95BudgetMs },
|
||||
error_rate: { actual: metrics.error_rate, max: maxErrorRate, pass: metrics.error_rate <= maxErrorRate },
|
||||
};
|
||||
|
||||
await writeFile(metricsPath, `${JSON.stringify({ ...metrics, samples: results }, null, 2)}\n`, "utf8");
|
||||
await writeFile(networkLogPath, results.map((item) => JSON.stringify(item)).join("\n") + (results.length > 0 ? "\n" : ""), "utf8");
|
||||
|
||||
const finishedAt = new Date();
|
||||
const result = {
|
||||
source: "automation",
|
||||
case_id: caseId,
|
||||
run_id: runId,
|
||||
status,
|
||||
reason,
|
||||
started_at: startedAt.toISOString(),
|
||||
started_at_local: localIsoWithOffset(startedAt),
|
||||
finished_at: finishedAt.toISOString(),
|
||||
finished_at_local: localIsoWithOffset(finishedAt),
|
||||
duration_ms: finishedAt.getTime() - startedAt.getTime(),
|
||||
url: backendUrl,
|
||||
metrics_summary: {
|
||||
requests: metrics.total_requests,
|
||||
concurrency: metrics.concurrency,
|
||||
ok_count: metrics.ok_count,
|
||||
error_rate: metrics.error_rate,
|
||||
latency_p50_ms: metrics.latency_ms.p50,
|
||||
latency_p95_ms: metrics.latency_ms.p95,
|
||||
status_counts: metrics.status_counts,
|
||||
},
|
||||
thresholds_summary: thresholds,
|
||||
artifacts: {
|
||||
metrics_json: metricsPath,
|
||||
network_log: networkLogPath,
|
||||
automation_result_json: automationResultPath,
|
||||
result_json: resultPath,
|
||||
},
|
||||
evidence_collected: ["metrics", "network", "api_diagnostic", "filesystem"],
|
||||
};
|
||||
|
||||
const resultText = `${JSON.stringify(result, null, 2)}\n`;
|
||||
await writeFile(automationResultPath, resultText, "utf8");
|
||||
await writeFile(resultPath, resultText, "utf8");
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
exit(status === "pass" ? 0 : status === "env_issue" ? 2 : 1);
|
||||
}
|
||||
|
||||
await main();
|
||||
@@ -1,205 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { existsSync, readdirSync, statSync } from "node:fs";
|
||||
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
||||
import { join, resolve } from "node:path";
|
||||
import { env, exit } from "node:process";
|
||||
|
||||
function pad(value, size = 2) {
|
||||
return String(value).padStart(size, "0");
|
||||
}
|
||||
|
||||
function localIsoWithOffset(date = new Date()) {
|
||||
const offsetMinutes = -date.getTimezoneOffset();
|
||||
const sign = offsetMinutes >= 0 ? "+" : "-";
|
||||
const absolute = Math.abs(offsetMinutes);
|
||||
return [
|
||||
`${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())}`,
|
||||
`T${pad(date.getHours())}:${pad(date.getMinutes())}:${pad(date.getSeconds())}.${pad(date.getMilliseconds(), 3)}`,
|
||||
`${sign}${pad(Math.floor(absolute / 60))}:${pad(absolute % 60)}`,
|
||||
].join("");
|
||||
}
|
||||
|
||||
function timestampSlug(date = new Date()) {
|
||||
return date.toISOString().replace(/\.\d{3}Z$/, "Z").replace(/[^0-9A-Za-z]+/g, "-").replace(/^-|-$/g, "");
|
||||
}
|
||||
|
||||
function repoRootFromEnv(root) {
|
||||
return env.LANGBOT_REPO ? resolve(env.LANGBOT_REPO) : resolve(root, "..");
|
||||
}
|
||||
|
||||
function latestBackendLog(root) {
|
||||
const explicit = env.LANGBOT_BACKEND_LOG;
|
||||
if (explicit) return resolve(explicit);
|
||||
|
||||
const logsDir = join(repoRootFromEnv(root), "data", "logs");
|
||||
if (!existsSync(logsDir)) return "";
|
||||
const candidates = readdirSync(logsDir)
|
||||
.filter((name) => /^langbot-.*\.log$/.test(name))
|
||||
.map((name) => join(logsDir, name))
|
||||
.filter((path) => {
|
||||
try {
|
||||
return statSync(path).isFile();
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
})
|
||||
.sort((left, right) => statSync(right).mtimeMs - statSync(left).mtimeMs);
|
||||
return candidates[0] || "";
|
||||
}
|
||||
|
||||
function parseSince(startedAt) {
|
||||
if (env.LANGBOT_BACKEND_LOG_SINCE) return new Date(env.LANGBOT_BACKEND_LOG_SINCE);
|
||||
const lookbackSeconds = Number(env.LANGBOT_BACKEND_LOG_LOOKBACK_SECONDS || "300");
|
||||
return new Date(startedAt.getTime() - lookbackSeconds * 1000);
|
||||
}
|
||||
|
||||
function parseTimestamp(line, year) {
|
||||
const localMatch = line.match(/^\[(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})\.(\d{3})\]/);
|
||||
if (localMatch) {
|
||||
const [, month, day, hour, minute, second, millisecond] = localMatch;
|
||||
return new Date(`${year}-${month}-${day}T${hour}:${minute}:${second}.${millisecond}+08:00`);
|
||||
}
|
||||
|
||||
const accessMatch = line.match(/^\[(\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2}) ([+-]\d{4})\]/);
|
||||
if (accessMatch) {
|
||||
const [, fullYear, month, day, hour, minute, second, offset] = accessMatch;
|
||||
const normalizedOffset = `${offset.slice(0, 3)}:${offset.slice(3)}`;
|
||||
return new Date(`${fullYear}-${month}-${day}T${hour}:${minute}:${second}${normalizedOffset}`);
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
function findingForLine(line, number) {
|
||||
const rules = [
|
||||
{ severity: "fail", kind: "python_traceback", pattern: /\bTraceback(?: \(most recent call last\))?/i },
|
||||
{ severity: "fail", kind: "unretrieved_task_exception", pattern: /Task exception was never retrieved/i },
|
||||
{ severity: "fail", kind: "unawaited_coroutine", pattern: /RuntimeWarning:\s+coroutine .* was never awaited/i },
|
||||
{ severity: "fail", kind: "unclosed_client_session", pattern: /Unclosed client session/i },
|
||||
{ severity: "fail", kind: "unclosed_connector", pattern: /Unclosed connector/i },
|
||||
{ severity: "fail", kind: "import_error", pattern: /\bImportError\b/i },
|
||||
{ severity: "fail", kind: "error_log", pattern: /\b(?:ERROR|CRITICAL)\b/ },
|
||||
{ severity: "warning", kind: "warning_log", pattern: /\bWARNING\b/ },
|
||||
];
|
||||
|
||||
for (const rule of rules) {
|
||||
if (rule.pattern.test(line)) {
|
||||
return {
|
||||
severity: rule.severity,
|
||||
kind: rule.kind,
|
||||
line: number,
|
||||
excerpt: line,
|
||||
};
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function scanLines(text, since, year) {
|
||||
const findings = [];
|
||||
const scanned = [];
|
||||
let includeContinuation = false;
|
||||
const lines = text.split(/\r?\n/);
|
||||
for (const [index, line] of lines.entries()) {
|
||||
const number = index + 1;
|
||||
const timestamp = parseTimestamp(line, year);
|
||||
if (timestamp) includeContinuation = timestamp >= since;
|
||||
if (!includeContinuation) continue;
|
||||
scanned.push({ number, text: line });
|
||||
const finding = findingForLine(line, number);
|
||||
if (finding) findings.push(finding);
|
||||
}
|
||||
return { findings, scanned, total_lines: lines.length };
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const root = resolve(env.LBS_ROOT || process.cwd());
|
||||
const caseId = "langbot-live-backend-log-health";
|
||||
const runId = env.LBS_RUN_ID || `${timestampSlug()}-${caseId}`;
|
||||
const evidenceDir = resolve(env.LBS_EVIDENCE_DIR || join(root, "reports", "evidence", runId));
|
||||
await mkdir(evidenceDir, { recursive: true });
|
||||
|
||||
const startedAt = new Date();
|
||||
const since = parseSince(startedAt);
|
||||
const logPath = latestBackendLog(root);
|
||||
const metricsPath = join(evidenceDir, "metrics.json");
|
||||
const findingsPath = join(evidenceDir, "findings.json");
|
||||
const scannedLogPath = join(evidenceDir, "scanned-backend.log");
|
||||
const automationResultPath = join(evidenceDir, "automation-result.json");
|
||||
const resultPath = join(evidenceDir, "result.json");
|
||||
|
||||
let status = "fail";
|
||||
let reason = "";
|
||||
let scan = { findings: [], scanned: [], total_lines: 0 };
|
||||
if (!logPath || !existsSync(logPath)) {
|
||||
status = "env_issue";
|
||||
reason = "No LangBot backend log file was found. Set LANGBOT_BACKEND_LOG or LANGBOT_REPO.";
|
||||
} else {
|
||||
const text = await readFile(logPath, "utf8");
|
||||
scan = scanLines(text, since, startedAt.getFullYear());
|
||||
const failCount = scan.findings.filter((item) => item.severity === "fail").length;
|
||||
status = failCount === 0 ? "pass" : "fail";
|
||||
reason = status === "pass"
|
||||
? "Live backend log health passed; no fail-severity findings in the scanned window."
|
||||
: "Live backend log health found fail-severity backend log findings.";
|
||||
}
|
||||
|
||||
const warningCount = scan.findings.filter((item) => item.severity === "warning").length;
|
||||
const failCount = scan.findings.filter((item) => item.severity === "fail").length;
|
||||
const metrics = {
|
||||
probe: caseId,
|
||||
backend_log: logPath,
|
||||
since: since.toISOString(),
|
||||
scanned_line_count: scan.scanned.length,
|
||||
total_line_count: scan.total_lines,
|
||||
fail_count: failCount,
|
||||
warning_count: warningCount,
|
||||
finding_count: scan.findings.length,
|
||||
};
|
||||
const thresholds = {
|
||||
fail_count: { actual: failCount, max: 0, pass: failCount === 0 },
|
||||
};
|
||||
|
||||
await writeFile(metricsPath, `${JSON.stringify(metrics, null, 2)}\n`, "utf8");
|
||||
await writeFile(findingsPath, `${JSON.stringify(scan.findings, null, 2)}\n`, "utf8");
|
||||
await writeFile(scannedLogPath, scan.scanned.map((item) => `${item.number}: ${item.text}`).join("\n") + (scan.scanned.length > 0 ? "\n" : ""), "utf8");
|
||||
|
||||
const finishedAt = new Date();
|
||||
const result = {
|
||||
source: "automation",
|
||||
case_id: caseId,
|
||||
run_id: runId,
|
||||
status,
|
||||
reason,
|
||||
started_at: startedAt.toISOString(),
|
||||
started_at_local: localIsoWithOffset(startedAt),
|
||||
finished_at: finishedAt.toISOString(),
|
||||
finished_at_local: localIsoWithOffset(finishedAt),
|
||||
duration_ms: finishedAt.getTime() - startedAt.getTime(),
|
||||
url: logPath,
|
||||
metrics_summary: {
|
||||
scanned_line_count: metrics.scanned_line_count,
|
||||
fail_count: metrics.fail_count,
|
||||
warning_count: metrics.warning_count,
|
||||
finding_count: metrics.finding_count,
|
||||
},
|
||||
thresholds_summary: thresholds,
|
||||
artifacts: {
|
||||
metrics_json: metricsPath,
|
||||
findings_json: findingsPath,
|
||||
scanned_backend_log: scannedLogPath,
|
||||
automation_result_json: automationResultPath,
|
||||
result_json: resultPath,
|
||||
},
|
||||
evidence_collected: ["metrics", "backend_log", "filesystem"],
|
||||
};
|
||||
|
||||
const resultText = `${JSON.stringify(result, null, 2)}\n`;
|
||||
await writeFile(automationResultPath, resultText, "utf8");
|
||||
await writeFile(resultPath, resultText, "utf8");
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
exit(status === "pass" ? 0 : status === "env_issue" ? 2 : 1);
|
||||
}
|
||||
|
||||
await main();
|
||||
@@ -1,311 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
import { join, resolve } from "node:path";
|
||||
import { env, exit } from "node:process";
|
||||
|
||||
function pad(value, size = 2) {
|
||||
return String(value).padStart(size, "0");
|
||||
}
|
||||
|
||||
function localIsoWithOffset(date = new Date()) {
|
||||
const offsetMinutes = -date.getTimezoneOffset();
|
||||
const sign = offsetMinutes >= 0 ? "+" : "-";
|
||||
const absolute = Math.abs(offsetMinutes);
|
||||
return [
|
||||
`${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())}`,
|
||||
`T${pad(date.getHours())}:${pad(date.getMinutes())}:${pad(date.getSeconds())}.${pad(date.getMilliseconds(), 3)}`,
|
||||
`${sign}${pad(Math.floor(absolute / 60))}:${pad(absolute % 60)}`,
|
||||
].join("");
|
||||
}
|
||||
|
||||
function timestampSlug(date = new Date()) {
|
||||
return date.toISOString().replace(/\.\d{3}Z$/, "Z").replace(/[^0-9A-Za-z]+/g, "-").replace(/^-|-$/g, "");
|
||||
}
|
||||
|
||||
function percentile(values, percentileValue) {
|
||||
if (values.length === 0) return 0;
|
||||
const sorted = [...values].sort((a, b) => a - b);
|
||||
const index = Math.min(sorted.length - 1, Math.ceil((percentileValue / 100) * sorted.length) - 1);
|
||||
return Number(sorted[index].toFixed(3));
|
||||
}
|
||||
|
||||
function stats(values) {
|
||||
if (values.length === 0) return { min: 0, p50: 0, p95: 0, p99: 0, max: 0 };
|
||||
return {
|
||||
min: Number(Math.min(...values).toFixed(3)),
|
||||
p50: percentile(values, 50),
|
||||
p95: percentile(values, 95),
|
||||
p99: percentile(values, 99),
|
||||
max: Number(Math.max(...values).toFixed(3)),
|
||||
};
|
||||
}
|
||||
|
||||
function joinUrl(baseUrl, path) {
|
||||
const base = baseUrl.replace(/\/+$/, "");
|
||||
const suffix = path.startsWith("/") ? path : `/${path}`;
|
||||
return `${base}${suffix}`;
|
||||
}
|
||||
|
||||
function parseJsonObject(value, fallback) {
|
||||
if (!value) return fallback;
|
||||
try {
|
||||
const parsed = JSON.parse(value);
|
||||
return parsed && typeof parsed === "object" && !Array.isArray(parsed) ? parsed : fallback;
|
||||
} catch {
|
||||
return fallback;
|
||||
}
|
||||
}
|
||||
|
||||
function controlPlaneEndpoints() {
|
||||
return [
|
||||
{
|
||||
id: "healthz",
|
||||
path: "/healthz",
|
||||
expected_status: 200,
|
||||
expected_code: 0,
|
||||
p95_budget_ms: Number(env.LANGBOT_PERF_HEALTHZ_P95_MS || "500"),
|
||||
required_data_fields: [],
|
||||
},
|
||||
{
|
||||
id: "system_info",
|
||||
path: "/api/v1/system/info",
|
||||
expected_status: 200,
|
||||
expected_code: 0,
|
||||
p95_budget_ms: Number(env.LANGBOT_PERF_SYSTEM_INFO_P95_MS || "1000"),
|
||||
required_data_fields: ["version", "edition", "enable_marketplace"],
|
||||
},
|
||||
];
|
||||
}
|
||||
|
||||
async function fetchEndpoint(backendUrl, endpoint, timeoutMs) {
|
||||
const url = joinUrl(backendUrl, endpoint.path);
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(() => controller.abort(), timeoutMs);
|
||||
const started = performance.now();
|
||||
let bodyText = "";
|
||||
let json = null;
|
||||
let jsonValid = false;
|
||||
let error = "";
|
||||
|
||||
try {
|
||||
const response = await fetch(url, {
|
||||
method: "GET",
|
||||
headers: { "accept": "application/json" },
|
||||
signal: controller.signal,
|
||||
});
|
||||
bodyText = await response.text();
|
||||
try {
|
||||
json = bodyText ? JSON.parse(bodyText) : null;
|
||||
jsonValid = json !== null;
|
||||
} catch (parseError) {
|
||||
error = parseError instanceof Error ? parseError.message : String(parseError);
|
||||
}
|
||||
|
||||
const data = json && typeof json === "object" && json.data && typeof json.data === "object" ? json.data : {};
|
||||
const missingFields = endpoint.required_data_fields.filter((field) => !(field in data));
|
||||
const statusOk = response.status === endpoint.expected_status;
|
||||
const codeOk = !json || typeof json !== "object" ? false : json.code === endpoint.expected_code;
|
||||
const shapeOk = jsonValid && missingFields.length === 0;
|
||||
const latencyMs = performance.now() - started;
|
||||
return {
|
||||
endpoint_id: endpoint.id,
|
||||
path: endpoint.path,
|
||||
url,
|
||||
status: response.status,
|
||||
ok: statusOk && codeOk && shapeOk,
|
||||
status_ok: statusOk,
|
||||
code_ok: codeOk,
|
||||
json_valid: jsonValid,
|
||||
missing_fields: missingFields,
|
||||
response_code: json && typeof json === "object" ? json.code : null,
|
||||
latency_ms: Number(latencyMs.toFixed(3)),
|
||||
error,
|
||||
};
|
||||
} catch (fetchError) {
|
||||
const latencyMs = performance.now() - started;
|
||||
return {
|
||||
endpoint_id: endpoint.id,
|
||||
path: endpoint.path,
|
||||
url,
|
||||
status: 0,
|
||||
ok: false,
|
||||
status_ok: false,
|
||||
code_ok: false,
|
||||
json_valid: false,
|
||||
missing_fields: endpoint.required_data_fields,
|
||||
response_code: null,
|
||||
latency_ms: Number(latencyMs.toFixed(3)),
|
||||
error: fetchError instanceof Error ? fetchError.message : String(fetchError),
|
||||
};
|
||||
} finally {
|
||||
clearTimeout(timeout);
|
||||
}
|
||||
}
|
||||
|
||||
async function runBatches(backendUrl, endpoints, totalRequests, concurrency, timeoutMs) {
|
||||
const queue = Array.from({ length: totalRequests }, (_, index) => endpoints[index % endpoints.length]);
|
||||
const results = [];
|
||||
while (queue.length > 0) {
|
||||
const batch = queue.splice(0, concurrency);
|
||||
results.push(...await Promise.all(batch.map((endpoint) => fetchEndpoint(backendUrl, endpoint, timeoutMs))));
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
function endpointMetrics(endpoints, results) {
|
||||
return Object.fromEntries(endpoints.map((endpoint) => {
|
||||
const samples = results.filter((item) => item.endpoint_id === endpoint.id);
|
||||
const okSamples = samples.filter((item) => item.ok);
|
||||
return [
|
||||
endpoint.id,
|
||||
{
|
||||
path: endpoint.path,
|
||||
requests: samples.length,
|
||||
ok_count: okSamples.length,
|
||||
error_rate: samples.length === 0 ? 1 : Number(((samples.length - okSamples.length) / samples.length).toFixed(4)),
|
||||
latency_ms: stats(okSamples.map((item) => item.latency_ms)),
|
||||
p95_budget_ms: endpoint.p95_budget_ms,
|
||||
},
|
||||
];
|
||||
}));
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const root = resolve(env.LBS_ROOT || process.cwd());
|
||||
const caseId = "langbot-live-control-plane-api";
|
||||
const runId = env.LBS_RUN_ID || `${timestampSlug()}-${caseId}`;
|
||||
const evidenceDir = resolve(env.LBS_EVIDENCE_DIR || join(root, "reports", "evidence", runId));
|
||||
await mkdir(evidenceDir, { recursive: true });
|
||||
|
||||
const startedAt = new Date();
|
||||
const backendUrl = env.LANGBOT_BACKEND_URL || "";
|
||||
const endpoints = controlPlaneEndpoints();
|
||||
const configuredBudgets = parseJsonObject(env.LANGBOT_CONTROL_PLANE_P95_BUDGETS_JSON, {});
|
||||
for (const endpoint of endpoints) {
|
||||
const budget = configuredBudgets[endpoint.id];
|
||||
if (typeof budget === "number" && Number.isFinite(budget)) endpoint.p95_budget_ms = budget;
|
||||
}
|
||||
const totalRequests = Number(env.LANGBOT_CONTROL_PLANE_REQUESTS || "20");
|
||||
const concurrency = Number(env.LANGBOT_CONTROL_PLANE_CONCURRENCY || "4");
|
||||
const timeoutMs = Number(env.LANGBOT_CONTROL_PLANE_TIMEOUT_MS || "5000");
|
||||
const maxErrorRate = Number(env.LANGBOT_CONTROL_PLANE_MAX_ERROR_RATE || "0");
|
||||
const metricsPath = join(evidenceDir, "metrics.json");
|
||||
const endpointsPath = join(evidenceDir, "endpoints.json");
|
||||
const networkLogPath = join(evidenceDir, "network.log");
|
||||
const automationResultPath = join(evidenceDir, "automation-result.json");
|
||||
const resultPath = join(evidenceDir, "result.json");
|
||||
|
||||
let status = "fail";
|
||||
let reason = "";
|
||||
let results = [];
|
||||
if (!backendUrl) {
|
||||
status = "env_issue";
|
||||
reason = "LANGBOT_BACKEND_URL is not configured.";
|
||||
} else {
|
||||
results = await runBatches(backendUrl, endpoints, totalRequests, concurrency, timeoutMs);
|
||||
const allConnectionFailures = results.length > 0 && results.every((item) => item.status === 0);
|
||||
if (allConnectionFailures) {
|
||||
status = "env_issue";
|
||||
reason = `Backend did not respond at ${backendUrl}.`;
|
||||
}
|
||||
}
|
||||
|
||||
const okResults = results.filter((item) => item.ok);
|
||||
const statusCounts = {};
|
||||
for (const item of results) {
|
||||
const key = item.status === 0 ? "network_error" : String(item.status);
|
||||
statusCounts[key] = (statusCounts[key] || 0) + 1;
|
||||
}
|
||||
const perEndpoint = endpointMetrics(endpoints, results);
|
||||
const responseShapeFailures = results.filter((item) => !item.json_valid || item.missing_fields.length > 0 || !item.code_ok).length;
|
||||
const errorRate = results.length === 0 ? 1 : Number(((results.length - okResults.length) / results.length).toFixed(4));
|
||||
const thresholds = {
|
||||
error_rate: { actual: errorRate, max: maxErrorRate, pass: errorRate <= maxErrorRate },
|
||||
response_shape_failures: { actual: responseShapeFailures, max: 0, pass: responseShapeFailures === 0 },
|
||||
};
|
||||
for (const endpoint of endpoints) {
|
||||
const actual = perEndpoint[endpoint.id].latency_ms.p95;
|
||||
thresholds[`${endpoint.id}_p95_ms`] = {
|
||||
actual,
|
||||
max: endpoint.p95_budget_ms,
|
||||
pass: actual <= endpoint.p95_budget_ms,
|
||||
};
|
||||
}
|
||||
|
||||
if (status !== "env_issue") {
|
||||
const passed = Object.values(thresholds).every((item) => item.pass);
|
||||
status = passed ? "pass" : "fail";
|
||||
reason = passed
|
||||
? "Live control-plane API probe passed all thresholds."
|
||||
: "Live control-plane API probe breached shape, latency, or error-rate thresholds.";
|
||||
}
|
||||
|
||||
const metrics = {
|
||||
probe: caseId,
|
||||
backend_url: backendUrl,
|
||||
total_requests: totalRequests,
|
||||
concurrency,
|
||||
timeout_ms: timeoutMs,
|
||||
ok_count: okResults.length,
|
||||
error_count: results.length - okResults.length,
|
||||
error_rate: errorRate,
|
||||
status_counts: statusCounts,
|
||||
response_shape_failures: responseShapeFailures,
|
||||
endpoints: perEndpoint,
|
||||
};
|
||||
|
||||
await writeFile(metricsPath, `${JSON.stringify({ ...metrics, samples: results }, null, 2)}\n`, "utf8");
|
||||
await writeFile(endpointsPath, `${JSON.stringify(endpoints, null, 2)}\n`, "utf8");
|
||||
await writeFile(networkLogPath, results.map((item) => JSON.stringify(item)).join("\n") + (results.length > 0 ? "\n" : ""), "utf8");
|
||||
|
||||
const finishedAt = new Date();
|
||||
const result = {
|
||||
source: "automation",
|
||||
case_id: caseId,
|
||||
run_id: runId,
|
||||
status,
|
||||
reason,
|
||||
started_at: startedAt.toISOString(),
|
||||
started_at_local: localIsoWithOffset(startedAt),
|
||||
finished_at: finishedAt.toISOString(),
|
||||
finished_at_local: localIsoWithOffset(finishedAt),
|
||||
duration_ms: finishedAt.getTime() - startedAt.getTime(),
|
||||
url: backendUrl,
|
||||
metrics_summary: {
|
||||
requests: metrics.total_requests,
|
||||
concurrency: metrics.concurrency,
|
||||
ok_count: metrics.ok_count,
|
||||
error_rate: metrics.error_rate,
|
||||
response_shape_failures: metrics.response_shape_failures,
|
||||
endpoints: Object.fromEntries(Object.entries(metrics.endpoints).map(([id, value]) => [
|
||||
id,
|
||||
{
|
||||
path: value.path,
|
||||
ok_count: value.ok_count,
|
||||
error_rate: value.error_rate,
|
||||
latency_p50_ms: value.latency_ms.p50,
|
||||
latency_p95_ms: value.latency_ms.p95,
|
||||
},
|
||||
])),
|
||||
status_counts: metrics.status_counts,
|
||||
},
|
||||
thresholds_summary: thresholds,
|
||||
artifacts: {
|
||||
metrics_json: metricsPath,
|
||||
endpoints_json: endpointsPath,
|
||||
network_log: networkLogPath,
|
||||
automation_result_json: automationResultPath,
|
||||
result_json: resultPath,
|
||||
},
|
||||
evidence_collected: ["metrics", "network", "api_diagnostic", "filesystem"],
|
||||
};
|
||||
|
||||
const resultText = `${JSON.stringify(result, null, 2)}\n`;
|
||||
await writeFile(automationResultPath, resultText, "utf8");
|
||||
await writeFile(resultPath, resultText, "utf8");
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
exit(status === "pass" ? 0 : status === "env_issue" ? 2 : 1);
|
||||
}
|
||||
|
||||
await main();
|
||||
@@ -1,162 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
import { join, resolve } from "node:path";
|
||||
import { env, exit } from "node:process";
|
||||
|
||||
function pad(value, size = 2) {
|
||||
return String(value).padStart(size, "0");
|
||||
}
|
||||
|
||||
function localIsoWithOffset(date = new Date()) {
|
||||
const offsetMinutes = -date.getTimezoneOffset();
|
||||
const sign = offsetMinutes >= 0 ? "+" : "-";
|
||||
const absolute = Math.abs(offsetMinutes);
|
||||
return [
|
||||
`${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())}`,
|
||||
`T${pad(date.getHours())}:${pad(date.getMinutes())}:${pad(date.getSeconds())}.${pad(date.getMilliseconds(), 3)}`,
|
||||
`${sign}${pad(Math.floor(absolute / 60))}:${pad(absolute % 60)}`,
|
||||
].join("");
|
||||
}
|
||||
|
||||
function timestampSlug(date = new Date()) {
|
||||
return date.toISOString().replace(/\.\d{3}Z$/, "Z").replace(/[^0-9A-Za-z]+/g, "-").replace(/^-|-$/g, "");
|
||||
}
|
||||
|
||||
function percentile(values, percentileValue) {
|
||||
if (values.length === 0) return 0;
|
||||
const sorted = [...values].sort((a, b) => a - b);
|
||||
const index = Math.min(sorted.length - 1, Math.ceil((percentileValue / 100) * sorted.length) - 1);
|
||||
return Number(sorted[index].toFixed(3));
|
||||
}
|
||||
|
||||
function stats(values) {
|
||||
return {
|
||||
min: Number(Math.min(...values).toFixed(3)),
|
||||
p50: percentile(values, 50),
|
||||
p95: percentile(values, 95),
|
||||
p99: percentile(values, 99),
|
||||
max: Number(Math.max(...values).toFixed(3)),
|
||||
};
|
||||
}
|
||||
|
||||
function threshold(actual, limit, operator) {
|
||||
const pass = operator === "<=" ? actual <= limit : actual >= limit;
|
||||
return { actual, [operator === "<=" ? "max" : "min"]: limit, pass };
|
||||
}
|
||||
|
||||
function makeSample(index) {
|
||||
const ingress = 1 + (index % 5) * 0.22;
|
||||
const pipeline = 2.8 + (index % 7) * 0.31;
|
||||
const persistence = 1.1 + (index % 4) * 0.2;
|
||||
const pluginIpc = 1.9 + (index % 6) * 0.27;
|
||||
const rag = index % 3 === 0 ? 4.4 : 0.8 + (index % 5) * 0.18;
|
||||
const streaming = 1.5 + (index % 8) * 0.24;
|
||||
const provider = 80 + (index % 13) * 11;
|
||||
const externalTool = index % 4 === 0 ? 25 + (index % 9) * 3 : 0;
|
||||
const network = 8 + (index % 10) * 1.7;
|
||||
const overhead = ingress + pipeline + persistence + pluginIpc + rag + streaming;
|
||||
const external = provider + externalTool + network;
|
||||
const total = overhead + external;
|
||||
return {
|
||||
index,
|
||||
segments_ms: {
|
||||
ingress,
|
||||
pipeline,
|
||||
persistence,
|
||||
plugin_ipc: pluginIpc,
|
||||
rag,
|
||||
streaming,
|
||||
provider,
|
||||
external_tool: externalTool,
|
||||
network,
|
||||
},
|
||||
langbot_overhead_ms: Number(overhead.toFixed(3)),
|
||||
external_latency_ms: Number(external.toFixed(3)),
|
||||
e2e_latency_ms: Number(total.toFixed(3)),
|
||||
accounting_gap_ms: Number((total - external - overhead).toFixed(6)),
|
||||
};
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const root = resolve(env.LBS_ROOT || process.cwd());
|
||||
const caseId = "langbot-overhead-accounting-contract";
|
||||
const runId = env.LBS_RUN_ID || `${timestampSlug()}-${caseId}`;
|
||||
const evidenceDir = resolve(env.LBS_EVIDENCE_DIR || join(root, "reports", "evidence", runId));
|
||||
await mkdir(evidenceDir, { recursive: true });
|
||||
|
||||
const startedAt = new Date();
|
||||
const sampleCount = Number(env.LANGBOT_PERF_CONTRACT_SAMPLES || "80");
|
||||
const overheadP95BudgetMs = Number(env.LANGBOT_PERF_OVERHEAD_P95_MS || "25");
|
||||
const samples = Array.from({ length: sampleCount }, (_, index) => makeSample(index));
|
||||
const overheads = samples.map((sample) => sample.langbot_overhead_ms);
|
||||
const e2e = samples.map((sample) => sample.e2e_latency_ms);
|
||||
const external = samples.map((sample) => sample.external_latency_ms);
|
||||
const gaps = samples.map((sample) => Math.abs(sample.accounting_gap_ms));
|
||||
const memory = process.memoryUsage();
|
||||
|
||||
const metrics = {
|
||||
probe: caseId,
|
||||
sample_count: sampleCount,
|
||||
langbot_overhead_ms: stats(overheads),
|
||||
e2e_latency_ms: stats(e2e),
|
||||
external_latency_ms: stats(external),
|
||||
accounting_gap_max_ms: Number(Math.max(...gaps).toFixed(6)),
|
||||
samples,
|
||||
};
|
||||
const thresholds = {
|
||||
sample_count: threshold(sampleCount, 50, ">="),
|
||||
langbot_overhead_p95_ms: threshold(metrics.langbot_overhead_ms.p95, overheadP95BudgetMs, "<="),
|
||||
accounting_gap_max_ms: threshold(metrics.accounting_gap_max_ms, 0.001, "<="),
|
||||
};
|
||||
const status = Object.values(thresholds).every((item) => item.pass) ? "pass" : "fail";
|
||||
const metricsPath = join(evidenceDir, "metrics.json");
|
||||
const thresholdsPath = join(evidenceDir, "thresholds.json");
|
||||
const resourceLogPath = join(evidenceDir, "resource-log.json");
|
||||
const automationResultPath = join(evidenceDir, "automation-result.json");
|
||||
const resultPath = join(evidenceDir, "result.json");
|
||||
|
||||
await writeFile(metricsPath, `${JSON.stringify(metrics, null, 2)}\n`, "utf8");
|
||||
await writeFile(thresholdsPath, `${JSON.stringify(thresholds, null, 2)}\n`, "utf8");
|
||||
await writeFile(resourceLogPath, `${JSON.stringify({ memory, pid: process.pid }, null, 2)}\n`, "utf8");
|
||||
|
||||
const finishedAt = new Date();
|
||||
const result = {
|
||||
source: "automation",
|
||||
case_id: caseId,
|
||||
run_id: runId,
|
||||
status,
|
||||
reason: status === "pass"
|
||||
? "Overhead accounting contract passed all thresholds."
|
||||
: "Overhead accounting contract breached one or more thresholds.",
|
||||
started_at: startedAt.toISOString(),
|
||||
started_at_local: localIsoWithOffset(startedAt),
|
||||
finished_at: finishedAt.toISOString(),
|
||||
finished_at_local: localIsoWithOffset(finishedAt),
|
||||
duration_ms: finishedAt.getTime() - startedAt.getTime(),
|
||||
metrics_summary: {
|
||||
sample_count: metrics.sample_count,
|
||||
langbot_overhead_p95_ms: metrics.langbot_overhead_ms.p95,
|
||||
e2e_latency_p95_ms: metrics.e2e_latency_ms.p95,
|
||||
external_latency_p95_ms: metrics.external_latency_ms.p95,
|
||||
accounting_gap_max_ms: metrics.accounting_gap_max_ms,
|
||||
},
|
||||
thresholds_summary: thresholds,
|
||||
artifacts: {
|
||||
metrics_json: metricsPath,
|
||||
thresholds_json: thresholdsPath,
|
||||
resource_log_json: resourceLogPath,
|
||||
automation_result_json: automationResultPath,
|
||||
result_json: resultPath,
|
||||
},
|
||||
evidence_collected: ["metrics", "resource_log", "filesystem"],
|
||||
};
|
||||
|
||||
const resultText = `${JSON.stringify(result, null, 2)}\n`;
|
||||
await writeFile(automationResultPath, resultText, "utf8");
|
||||
await writeFile(resultPath, resultText, "utf8");
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
exit(status === "pass" ? 0 : 1);
|
||||
}
|
||||
|
||||
await main();
|
||||
@@ -1,134 +0,0 @@
|
||||
export function summarizeFakeProviderState(state) {
|
||||
if (!state) return null;
|
||||
const recentRequests = Array.isArray(state.recent_requests) ? state.recent_requests : [];
|
||||
const chatRequests = recentRequests.filter((request) => String(request?.path || "").includes("/chat/completions"));
|
||||
const successfulRequests = chatRequests.filter((request) => request?.status === "ok");
|
||||
const faultRequests = chatRequests.filter((request) => (
|
||||
request?.should_fail === true
|
||||
|| request?.status === "http_fault"
|
||||
|| (Number.isFinite(request?.http_status) && request.http_status >= 400)
|
||||
));
|
||||
|
||||
return {
|
||||
status: state.status || "unknown",
|
||||
url: state.url || "",
|
||||
request_count: Number.isFinite(state.request_count) ? state.request_count : recentRequests.length,
|
||||
recent_request_count: recentRequests.length,
|
||||
chat_request_count: chatRequests.length,
|
||||
fault_count: faultRequests.length,
|
||||
streamed_request_count: chatRequests.filter((request) => request?.stream === true).length,
|
||||
duration_ms: stats(chatRequests.map((request) => numberOrNull(request?.duration_ms)).filter(Number.isFinite)),
|
||||
successful_duration_ms: stats(successfulRequests.map((request) => numberOrNull(request?.duration_ms)).filter(Number.isFinite)),
|
||||
first_chunk_ms: stats(successfulRequests.map((request) => numberOrNull(request?.first_chunk_ms)).filter(Number.isFinite)),
|
||||
first_content_chunk_ms: stats(successfulRequests.map((request) => numberOrNull(request?.first_content_chunk_ms)).filter(Number.isFinite)),
|
||||
content_chunk_count: stats(successfulRequests.map((request) => numberOrNull(request?.content_chunk_count)).filter(Number.isFinite)),
|
||||
config: state.config || {},
|
||||
};
|
||||
}
|
||||
|
||||
export function buildProviderTimingMetrics(samples, state) {
|
||||
const recentRequests = Array.isArray(state?.recent_requests) ? state.recent_requests : [];
|
||||
const byExpectedText = new Map();
|
||||
for (const request of recentRequests) {
|
||||
const expected = String(request?.expected_text || "");
|
||||
if (!expected) continue;
|
||||
if (!byExpectedText.has(expected)) byExpectedText.set(expected, []);
|
||||
byExpectedText.get(expected).push(request);
|
||||
}
|
||||
|
||||
const segments = [];
|
||||
const missingExpectedText = [];
|
||||
for (const sample of samples) {
|
||||
const expected = String(sample?.expected_text || "");
|
||||
if (!expected) continue;
|
||||
const request = (byExpectedText.get(expected) || []).shift();
|
||||
if (!request) {
|
||||
missingExpectedText.push(expected);
|
||||
continue;
|
||||
}
|
||||
const segment = buildTimingSegment(sample, request);
|
||||
if (segment) segments.push(segment);
|
||||
}
|
||||
|
||||
const values = (key) => segments.map((segment) => numberOrNull(segment[key])).filter(Number.isFinite);
|
||||
return {
|
||||
matched_request_count: segments.length,
|
||||
missing_provider_match_count: missingExpectedText.length,
|
||||
missing_expected_text: missingExpectedText.slice(0, 20),
|
||||
send_to_provider_start_ms: stats(values("send_to_provider_start_ms")),
|
||||
provider_duration_ms: stats(values("provider_duration_ms")),
|
||||
provider_finish_to_ws_final_ms: stats(values("provider_finish_to_ws_final_ms")),
|
||||
langbot_overhead_estimate_ms: stats(values("langbot_overhead_estimate_ms")),
|
||||
e2e_minus_provider_ms: stats(values("e2e_minus_provider_ms")),
|
||||
provider_first_content_to_ws_first_content_ms: stats(values("provider_first_content_to_ws_first_content_ms")),
|
||||
segments,
|
||||
};
|
||||
}
|
||||
|
||||
function buildTimingSegment(sample, request) {
|
||||
const sentEpochMs = numberOrNull(sample.sent_epoch_ms);
|
||||
const finishedEpochMs = numberOrNull(sample.finished_epoch_ms);
|
||||
const providerStartedEpochMs = numberOrNull(request.started_epoch_ms);
|
||||
const providerFinishedEpochMs = numberOrNull(request.finished_epoch_ms);
|
||||
const providerFirstContentEpochMs = numberOrNull(request.first_content_chunk_epoch_ms);
|
||||
const wsFirstContentEpochMs = numberOrNull(sample.first_assistant_content_epoch_ms);
|
||||
const responseDurationMs = numberOrNull(sample.response_duration_ms);
|
||||
const providerDurationMs = numberOrNull(request.duration_ms);
|
||||
|
||||
const sendToProviderStartMs = finiteDelta(providerStartedEpochMs, sentEpochMs);
|
||||
const providerFinishToWsFinalMs = finiteDelta(finishedEpochMs, providerFinishedEpochMs);
|
||||
const e2eMinusProviderMs = Number.isFinite(responseDurationMs) && Number.isFinite(providerDurationMs)
|
||||
? rounded(responseDurationMs - providerDurationMs)
|
||||
: null;
|
||||
const overheadEstimateMs = Number.isFinite(sendToProviderStartMs) && Number.isFinite(providerFinishToWsFinalMs)
|
||||
? rounded(sendToProviderStartMs + providerFinishToWsFinalMs)
|
||||
: e2eMinusProviderMs;
|
||||
|
||||
return {
|
||||
sample_index: sample.index,
|
||||
pipeline_label: sample.pipeline_label || "",
|
||||
expected_text: sample.expected_text || "",
|
||||
provider_request_id: request.id || "",
|
||||
provider_request_number: request.request_number ?? null,
|
||||
response_duration_ms: responseDurationMs,
|
||||
provider_duration_ms: providerDurationMs,
|
||||
send_to_provider_start_ms: sendToProviderStartMs,
|
||||
provider_finish_to_ws_final_ms: providerFinishToWsFinalMs,
|
||||
langbot_overhead_estimate_ms: overheadEstimateMs,
|
||||
e2e_minus_provider_ms: e2eMinusProviderMs,
|
||||
provider_first_content_to_ws_first_content_ms: finiteDelta(wsFirstContentEpochMs, providerFirstContentEpochMs),
|
||||
provider_status: request.status || "",
|
||||
provider_http_status: request.http_status ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
function finiteDelta(left, right) {
|
||||
return Number.isFinite(left) && Number.isFinite(right) ? rounded(left - right) : null;
|
||||
}
|
||||
|
||||
export function stats(values) {
|
||||
if (values.length === 0) return { min: 0, p50: 0, p95: 0, p99: 0, max: 0 };
|
||||
return {
|
||||
min: rounded(Math.min(...values)),
|
||||
p50: percentile(values, 50),
|
||||
p95: percentile(values, 95),
|
||||
p99: percentile(values, 99),
|
||||
max: rounded(Math.max(...values)),
|
||||
};
|
||||
}
|
||||
|
||||
export function percentile(values, percentileValue) {
|
||||
if (values.length === 0) return 0;
|
||||
const sorted = [...values].sort((a, b) => a - b);
|
||||
const index = Math.min(sorted.length - 1, Math.ceil((percentileValue / 100) * sorted.length) - 1);
|
||||
return rounded(sorted[index]);
|
||||
}
|
||||
|
||||
export function rounded(value) {
|
||||
return Number(value.toFixed(3));
|
||||
}
|
||||
|
||||
function numberOrNull(value) {
|
||||
const number = Number(value);
|
||||
return Number.isFinite(number) ? number : null;
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user