diff --git a/AGENTS.md b/AGENTS.md index d5dbe0274..86eee323f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -133,6 +133,27 @@ When writing a migration, follow these rules: > **Legacy migration system (deprecated — do not extend).** The old 3.x migration system under `src/langbot/pkg/persistence/migrations/` (`DBMigration` subclasses in `dbmXXX_*.py`, run from `pkg/persistence/mgr.py`) is **frozen**. Do **not** add new `dbmXXX_*.py` files. The chain is capped at `required_database_version = 25` (`pkg/utils/constants.py`); those files only exist to upgrade pre-existing 3.x databases up to the Alembic baseline and are kept read-only. All new schema changes go through Alembic. +## Agent-Facing Surfaces (MCP + Skills) + +LangBot is built to be **agent-friendly**. Three surfaces let AI agents work +with LangBot, and they MUST be kept in lockstep with the HTTP API: + +1. **MCP server** — `src/langbot/pkg/api/mcp/` exposes a curated subset of the + API as MCP tools at `/mcp` (API-key authenticated, including the + `api.global_api_key` from config.yaml). `server.py` defines the tools (they + call the service layer directly); `mount.py` is the ASGI dispatcher. +2. **In-repo skills** — `skills/` is the **single source of truth** for agent + skills (plugin/core/deploy/e2e/MCP-ops). Docs and the landing page link here + rather than embedding their own copies. +3. **API-key auth** — `api.global_api_key` (config.yaml) authenticates the API + and MCP without a login session; see `docs/API_KEY_AUTH.md`. + +> **Maintenance rule (important).** When you add, remove, or change an HTTP API +> endpoint that should be agent-accessible, you MUST update **both** the matching +> MCP tool in `src/langbot/pkg/api/mcp/server.py` **and** the relevant skill under +> `skills/` (especially `skills/skills/langbot-mcp-ops`). The API, the MCP tool +> surface, and the skills are one system — drift between them is a bug. + ## Some Principles - Keep it simple, stupid. diff --git a/README.md b/README.md index 7027fe1a5..5bdb582f1 100644 --- a/README.md +++ b/README.md @@ -148,6 +148,19 @@ docker compose up -d --- +## Built for AI Agents 🤖 + +LangBot is **agent-friendly by design** — your coding agents (Claude Code, Codex, Copilot, Cursor, …) can operate, extend, and deploy LangBot with first-class support: + +- **MCP Server** — LangBot exposes a built-in [Model Context Protocol](https://modelcontextprotocol.io/) endpoint at `/mcp`, mirroring the HTTP API so an agent can manage bots, pipelines, plugins, and models programmatically. Authenticate with the same API key (set a global key in `config.yaml` or use a per-user key) — no login flow required. Configure it in the Web panel's **API & MCP** tab. +- **In-repo Skills** — The [`skills/`](skills/) directory is the **single source of truth** for working with LangBot: plugin development, core development, end-to-end testing, deployment, and operating the LangBot / LangBot Space MCP servers. Point your agent at this directory and it knows how to build. +- **AGENTS.md** — Every repo ships an [`AGENTS.md`](AGENTS.md) (symlinked to `CLAUDE.md`) describing architecture, conventions, and the rule that API changes must keep the MCP server and skills in sync. +- **`llms.txt`** — Machine-readable project context for LLMs is published on the website. + +> **Cloud / Marketplace:** [LangBot Space](https://space.langbot.app) also exposes an MCP server so agents can search and inspect the plugin / MCP / skill marketplace, authenticated with a Personal Access Token. + +--- + ## Live Demo **Try it now:** https://demo.langbot.dev/ diff --git a/README_CN.md b/README_CN.md index 436231f9c..18378c767 100644 --- a/README_CN.md +++ b/README_CN.md @@ -174,6 +174,19 @@ docker compose up -d --- +## 为 AI Agent 而生 🤖 + +LangBot **从设计上就对 Agent 友好** —— 你的编码 Agent（Claude Code、Codex、Copilot、Cursor 等）可以一等公民般地操作、扩展和部署 LangBot： + +- **MCP Server** —— LangBot 内置 [Model Context Protocol](https://modelcontextprotocol.io/) 端点 `/mcp`，与 HTTP API 对齐，Agent 可编程式管理机器人、流水线、插件和模型。使用同一套 API Key 鉴权（可在 `config.yaml` 配置全局 Key，或使用用户 Key），无需登录流程。在 Web 面板的 **API 与 MCP** 标签页中配置。 +- **仓库内 Skills** —— [`skills/`](skills/) 目录是使用 LangBot 的**唯一事实来源**：插件开发、核心开发、端到端测试、部署，以及操作 LangBot / LangBot Space MCP Server。把 Agent 指向这个目录，它就知道如何动手。 +- **AGENTS.md** —— 每个仓库都提供 [`AGENTS.md`](AGENTS.md)（软链到 `CLAUDE.md`），描述架构、规范，以及「API 变更必须同步更新 MCP Server 和 skills」的约定。 +- **`llms.txt`** —— 面向 LLM 的机器可读项目上下文已发布在官网。 + +> **云端 / 市场：** [LangBot Space](https://space.langbot.app) 同样开放 MCP Server，Agent 可搜索和查看插件 / MCP / Skill 市场，使用 Personal Access Token 鉴权。 + +--- + ## 社区 [![Discord](https://img.shields.io/discord/1335141740050649118?logo=discord&label=Discord)](https://discord.gg/wdNEHETs87) diff --git a/README_ES.md b/README_ES.md index e42aa4455..ee805fe53 100644 --- a/README_ES.md +++ b/README_ES.md @@ -155,6 +155,17 @@ docker compose up -d *Nota: Entorno de demostración público. No ingrese información confidencial.* +## Diseñado para Agentes de IA 🤖 + +LangBot es **agent-friendly por diseño** —— tus agentes de codificación (Claude Code, Codex, Copilot, Cursor, …) pueden operar, extender y desplegar LangBot con soporte de primera clase: + +- **Servidor MCP** —— LangBot expone un endpoint integrado de [Model Context Protocol](https://modelcontextprotocol.io/) en `/mcp`, replicando la API HTTP para que un agente gestione bots, pipelines, plugins y modelos de forma programática. Autentícate con la misma API key (configura una clave global en `config.yaml` o usa una clave por usuario) —— sin flujo de login. Configúralo en la pestaña **API & MCP** del panel web. +- **Skills en el repositorio** —— El directorio [`skills/`](skills/) es la **única fuente de verdad** para trabajar con LangBot: desarrollo de plugins, desarrollo del core, pruebas end-to-end, despliegue y operación de los servidores MCP de LangBot / LangBot Space. Apunta tu agente a este directorio y sabrá cómo construir. +- **AGENTS.md** —— Cada repo incluye un [`AGENTS.md`](AGENTS.md) (enlazado simbólicamente a `CLAUDE.md`) que describe la arquitectura, las convenciones y la regla de que los cambios en la API deben mantener sincronizados el servidor MCP y los skills. +- **`llms.txt`** —— El contexto del proyecto legible por máquina para LLMs está publicado en el sitio web. + +> **Nube / Marketplace:** [LangBot Space](https://space.langbot.app) también expone un servidor MCP para que los agentes busquen e inspeccionen el marketplace de plugins / MCP / skills, autenticados con un Personal Access Token. + --- ## Comunidad diff --git a/README_FR.md b/README_FR.md index 311017991..7a02083d3 100644 --- a/README_FR.md +++ b/README_FR.md @@ -155,6 +155,17 @@ docker compose up -d *Note : Environnement de démonstration public. Ne saisissez pas d'informations sensibles.* +## Conçu pour les agents IA 🤖 + +LangBot est **agent-friendly par conception** —— vos agents de codage (Claude Code, Codex, Copilot, Cursor, …) peuvent exploiter, étendre et déployer LangBot avec un support de premier ordre : + +- **Serveur MCP** —— LangBot expose un endpoint [Model Context Protocol](https://modelcontextprotocol.io/) intégré sur `/mcp`, reflétant l'API HTTP pour qu'un agent gère bots, pipelines, plugins et modèles de façon programmatique. Authentifiez-vous avec la même clé API (définissez une clé globale dans `config.yaml` ou utilisez une clé par utilisateur) —— sans flux de connexion. Configurez-le dans l'onglet **API & MCP** du panneau web. +- **Skills dans le dépôt** —— Le répertoire [`skills/`](skills/) est la **source unique de vérité** pour travailler avec LangBot : développement de plugins, développement du cœur, tests de bout en bout, déploiement et exploitation des serveurs MCP de LangBot / LangBot Space. Pointez votre agent vers ce répertoire et il saura construire. +- **AGENTS.md** —— Chaque dépôt fournit un [`AGENTS.md`](AGENTS.md) (lien symbolique vers `CLAUDE.md`) décrivant l'architecture, les conventions et la règle selon laquelle les changements d'API doivent garder le serveur MCP et les skills synchronisés. +- **`llms.txt`** —— Le contexte projet lisible par machine pour les LLM est publié sur le site web. + +> **Cloud / Marketplace :** [LangBot Space](https://space.langbot.app) expose également un serveur MCP pour que les agents recherchent et inspectent le marketplace de plugins / MCP / skills, authentifiés avec un Personal Access Token. + --- ## Communauté diff --git a/README_JP.md b/README_JP.md index 090dddda1..4d98df928 100644 --- a/README_JP.md +++ b/README_JP.md @@ -155,6 +155,17 @@ docker compose up -d *注意: 公開デモ環境です。機密情報を入力しないでください。* +## AI エージェントのために 🤖 + +LangBot は **設計段階からエージェントフレンドリー** です。お使いのコーディングエージェント（Claude Code、Codex、Copilot、Cursor など）が、ファーストクラスのサポートで LangBot を操作・拡張・デプロイできます： + +- **MCP サーバー** —— LangBot は組み込みの [Model Context Protocol](https://modelcontextprotocol.io/) エンドポイント `/mcp` を公開し、HTTP API とミラーリングされているため、エージェントがボット・パイプライン・プラグイン・モデルをプログラム的に管理できます。同じ API キーで認証（`config.yaml` でグローバルキーを設定、またはユーザーキーを使用）—— ログインフロー不要。Web パネルの **API & MCP** タブで設定します。 +- **リポジトリ内 Skills** —— [`skills/`](skills/) ディレクトリは LangBot を扱うための**唯一の信頼できる情報源**です：プラグイン開発、コア開発、E2E テスト、デプロイ、LangBot / LangBot Space MCP サーバーの操作。エージェントをこのディレクトリに向ければ、構築方法を理解します。 +- **AGENTS.md** —— すべてのリポジトリに [`AGENTS.md`](AGENTS.md)（`CLAUDE.md` へのシンボリックリンク）があり、アーキテクチャ・規約、そして「API 変更時は MCP サーバーと skills を同期する」というルールを記述しています。 +- **`llms.txt`** —— LLM 向けの機械可読なプロジェクトコンテキストを公式サイトで公開しています。 + +> **クラウド / マーケット：** [LangBot Space](https://space.langbot.app) も MCP サーバーを公開しており、エージェントが Personal Access Token で認証してプラグイン / MCP / Skill マーケットを検索・確認できます。 + --- ## コミュニティ diff --git a/README_KO.md b/README_KO.md index 85e4ae4fb..b15309c71 100644 --- a/README_KO.md +++ b/README_KO.md @@ -155,6 +155,17 @@ docker compose up -d *참고: 공개 데모 환경입니다. 민감한 정보를 입력하지 마세요.* +## AI 에이전트를 위한 설계 🤖 + +LangBot은 **설계 단계부터 에이전트 친화적**입니다 —— 코딩 에이전트(Claude Code, Codex, Copilot, Cursor 등)가 일급 지원으로 LangBot을 운영·확장·배포할 수 있습니다: + +- **MCP 서버** —— LangBot은 내장 [Model Context Protocol](https://modelcontextprotocol.io/) 엔드포인트 `/mcp`를 제공하며, HTTP API와 동일하게 미러링되어 에이전트가 봇·파이프라인·플러그인·모델을 프로그래밍 방식으로 관리할 수 있습니다. 동일한 API 키로 인증하며(`config.yaml`에 전역 키 설정 또는 사용자 키 사용) 로그인 절차가 필요 없습니다. 웹 패널의 **API & MCP** 탭에서 설정합니다. +- **저장소 내 Skills** —— [`skills/`](skills/) 디렉터리는 LangBot 작업의 **단일 진실 공급원**입니다: 플러그인 개발, 코어 개발, E2E 테스트, 배포, LangBot / LangBot Space MCP 서버 운영. 에이전트를 이 디렉터리로 안내하면 빌드 방법을 알게 됩니다. +- **AGENTS.md** —— 모든 저장소에는 [`AGENTS.md`](AGENTS.md)(`CLAUDE.md`로 심볼릭 링크)가 있으며 아키텍처, 규약, 그리고 API 변경 시 MCP 서버와 skills를 동기화해야 한다는 규칙을 설명합니다. +- **`llms.txt`** —— LLM을 위한 기계 판독 가능한 프로젝트 컨텍스트가 웹사이트에 게시되어 있습니다. + +> **클라우드 / 마켓플레이스:** [LangBot Space](https://space.langbot.app)도 MCP 서버를 제공하여 에이전트가 Personal Access Token으로 인증해 플러그인 / MCP / Skill 마켓플레이스를 검색하고 조회할 수 있습니다. + --- ## 커뮤니티 diff --git a/README_RU.md b/README_RU.md index 3fc382ce2..c9ec941ea 100644 --- a/README_RU.md +++ b/README_RU.md @@ -155,6 +155,17 @@ docker compose up -d *Примечание: Публичная демо-среда. Не вводите конфиденциальную информацию.* +## Создано для ИИ-агентов 🤖 + +LangBot **дружелюбен к агентам по своей архитектуре** —— ваши кодинг-агенты (Claude Code, Codex, Copilot, Cursor и др.) могут управлять, расширять и развёртывать LangBot с первоклассной поддержкой: + +- **MCP-сервер** —— LangBot предоставляет встроенную конечную точку [Model Context Protocol](https://modelcontextprotocol.io/) по адресу `/mcp`, зеркалирующую HTTP API, чтобы агент мог программно управлять ботами, пайплайнами, плагинами и моделями. Аутентификация той же API-ключом (задайте глобальный ключ в `config.yaml` или используйте пользовательский ключ) —— без процедуры входа. Настраивается на вкладке **API & MCP** веб-панели. +- **Skills в репозитории** —— Каталог [`skills/`](skills/) является **единственным источником истины** для работы с LangBot: разработка плагинов, разработка ядра, сквозное тестирование, развёртывание и работа с MCP-серверами LangBot / LangBot Space. Направьте агента в этот каталог, и он будет знать, как собирать. +- **AGENTS.md** —— Каждый репозиторий содержит [`AGENTS.md`](AGENTS.md) (символическая ссылка на `CLAUDE.md`), описывающий архитектуру, соглашения и правило: изменения API должны синхронизировать MCP-сервер и skills. +- **`llms.txt`** —— Машиночитаемый контекст проекта для LLM опубликован на сайте. + +> **Облако / Маркетплейс:** [LangBot Space](https://space.langbot.app) также предоставляет MCP-сервер, чтобы агенты могли искать и просматривать маркетплейс плагинов / MCP / skills, аутентифицируясь с помощью Personal Access Token. + --- ## Сообщество diff --git a/README_TW.md b/README_TW.md index 606469fc6..26c66d32e 100644 --- a/README_TW.md +++ b/README_TW.md @@ -171,6 +171,17 @@ docker compose up -d *注意：公開演示環境，請不要在其中填入任何敏感資訊。* +## 為 AI Agent 而生 🤖 + +LangBot **從設計上就對 Agent 友善** —— 你的編碼 Agent（Claude Code、Codex、Copilot、Cursor 等）可以一等公民般地操作、擴充和部署 LangBot： + +- **MCP Server** —— LangBot 內建 [Model Context Protocol](https://modelcontextprotocol.io/) 端點 `/mcp`，與 HTTP API 對齊，Agent 可程式化管理機器人、流水線、外掛和模型。使用同一套 API Key 鑑權（可在 `config.yaml` 設定全域 Key，或使用使用者 Key），無需登入流程。在 Web 面板的 **API 與 MCP** 分頁中設定。 +- **倉庫內 Skills** —— [`skills/`](skills/) 目錄是使用 LangBot 的**唯一事實來源**：外掛開發、核心開發、端到端測試、部署，以及操作 LangBot / LangBot Space MCP Server。把 Agent 指向這個目錄，它就知道如何動手。 +- **AGENTS.md** —— 每個倉庫都提供 [`AGENTS.md`](AGENTS.md)（軟連結到 `CLAUDE.md`），描述架構、規範，以及「API 變更必須同步更新 MCP Server 和 skills」的約定。 +- **`llms.txt`** —— 面向 LLM 的機器可讀專案上下文已發布在官網。 + +> **雲端 / 市集：** [LangBot Space](https://space.langbot.app) 同樣開放 MCP Server，Agent 可搜尋和檢視外掛 / MCP / Skill 市集，使用 Personal Access Token 鑑權。 + --- ## 社群 diff --git a/README_VI.md b/README_VI.md index 1d021224e..d3a07145b 100644 --- a/README_VI.md +++ b/README_VI.md @@ -155,6 +155,17 @@ docker compose up -d *Lưu ý: Môi trường demo công khai. Không nhập thông tin nhạy cảm.* +## Được xây dựng cho AI Agent 🤖 + +LangBot **thân thiện với agent ngay từ thiết kế** —— các coding agent của bạn (Claude Code, Codex, Copilot, Cursor, …) có thể vận hành, mở rộng và triển khai LangBot với sự hỗ trợ hạng nhất: + +- **MCP Server** —— LangBot cung cấp endpoint [Model Context Protocol](https://modelcontextprotocol.io/) tích hợp tại `/mcp`, phản chiếu HTTP API để agent quản lý bot, pipeline, plugin và model theo cách lập trình. Xác thực bằng cùng một API key (đặt key toàn cục trong `config.yaml` hoặc dùng key theo người dùng) —— không cần luồng đăng nhập. Cấu hình tại tab **API & MCP** trong bảng điều khiển Web. +- **Skills trong repo** —— Thư mục [`skills/`](skills/) là **nguồn sự thật duy nhất** để làm việc với LangBot: phát triển plugin, phát triển core, kiểm thử end-to-end, triển khai và vận hành MCP Server của LangBot / LangBot Space. Trỏ agent của bạn vào thư mục này và nó sẽ biết cách xây dựng. +- **AGENTS.md** —— Mỗi repo đều có [`AGENTS.md`](AGENTS.md) (liên kết tượng trưng tới `CLAUDE.md`) mô tả kiến trúc, quy ước và quy tắc rằng thay đổi API phải giữ MCP Server và skills đồng bộ. +- **`llms.txt`** —— Ngữ cảnh dự án có thể đọc bằng máy dành cho LLM được công bố trên website. + +> **Cloud / Marketplace:** [LangBot Space](https://space.langbot.app) cũng cung cấp MCP Server để agent tìm kiếm và kiểm tra marketplace plugin / MCP / skill, xác thực bằng Personal Access Token. + --- ## Cộng đồng diff --git a/docs/API_KEY_AUTH.md b/docs/API_KEY_AUTH.md index 3aa0d363e..ad3bb6693 100644 --- a/docs/API_KEY_AUTH.md +++ b/docs/API_KEY_AUTH.md @@ -10,6 +10,38 @@ API keys can be managed through the web interface: 2. Click the "API Keys" button at the bottom of the sidebar 3. Create, view, copy, or delete API keys as needed +## Global API Key (config.yaml) + +In addition to web-UI-created keys (stored in the database, prefixed `lbk_`), +LangBot supports a **global API key** defined directly in `data/config.yaml`. +This is useful for automated deployments, infrastructure-as-code, and AI agents +that need API/MCP access **without a login session and without creating a +database record first**. + +```yaml +api: + port: 5300 + # ... + global_api_key: 'your-strong-secret-here' # leave empty to disable +``` + +Behavior: + +- When `api.global_api_key` is a non-empty string, that exact value is accepted + anywhere a normal API key is accepted — the `X-API-Key` header or + `Authorization: Bearer ` — across the HTTP service API **and the MCP + server**. +- The global key does **not** require the `lbk_` prefix; use any sufficiently + strong secret. +- Leave it empty (`''`, the default) to disable it entirely; only database-backed + `lbk_` keys will then be accepted. +- Existing installs are unaffected until you add the key — config completion only + backfills top-level keys, and the lookup is defensive when the field is absent. + +> **Security:** the global key is stored in plaintext in `config.yaml`. Only +> enable it on trusted/internal deployments, keep the file permissions tight, +> always serve over HTTPS, and rotate the value if it may have leaked. + ## Using API Keys ### Authentication Headers diff --git a/skills/.gitignore b/skills/.gitignore new file mode 100644 index 000000000..daeb4d92b --- /dev/null +++ b/skills/.gitignore @@ -0,0 +1,9 @@ +node_modules/ +coverage/ +.tap/ +__pycache__/ +*.pyc +skills/.env.local +reports/ +skills/*/reports/ +.browser/ diff --git a/skills/AGENTS.md b/skills/AGENTS.md new file mode 100644 index 000000000..ad47ee220 --- /dev/null +++ b/skills/AGENTS.md @@ -0,0 +1,67 @@ +# Agent Workflow + +This repository stores reusable LangBot agent-testing assets. Keep changes structured so the next agent does not need to rediscover paths. + +## First Steps + +1. Read `skills/.env` before using local URLs, paths, browser profiles, or proxy defaults. If present, `skills/.env.local` overrides it for this machine and must not be committed. On a new machine, copy `skills/.env.example` to `skills/.env.local` first. +2. Pick the smallest relevant skill: + - `langbot-env-setup` for environment, browser, OAuth, proxy, and startup. + - `langbot-testing` for WebUI, provider, pipeline, cases, and troubleshooting. + - `langbot-skills-maintenance` for adding, deduplicating, or auditing this skills repository. +3. Prefer existing cases and troubleshooting entries before exploring from scratch. + +## Editing Rules + +- UI/browser testing is the primary QA path. API/curl checks are diagnostic only and cannot make a UI case pass by themselves. +- Put skills under `skills//`. +- Keep `SKILL.md` concise; move detailed workflows to `references/`. +- Put reusable test paths in `cases/*.yaml`. +- New or edited cases must include `priority`, `risk`, `ci_eligible`, and `evidence_required` so agents can select the right test set without rereading every file. +- Use `env_any` / `automation_env_any` for one-of machine inputs, such as `LANGBOT_PIPELINE_URL|LANGBOT_PIPELINE_NAME`; do not list those alternatives as separate all-required env keys. +- Put reusable groups of cases in `suites/*.yaml` rather than hardcoding test sets in docs or CLI code. +- Put growing failure knowledge in `troubleshooting/*.yaml`. +- Do not hardcode local ports in testing docs; use `skills/.env` variables and machine-local `skills/.env.local` overrides. +- Do not store secrets, API keys, OAuth tokens, or localStorage token values. + +## Required Checks + +After structural changes, run: + +```bash +bin/lbs validate +``` + +After changing skills, cases, or troubleshooting assets, run: + +```bash +bin/lbs index +``` + +Use `bin/lbs env show` to inspect defaults and `bin/lbs env doctor` when diagnosing local environment readiness. Env output is redacted by default; do not work around that by printing raw secrets. +Use `bin/lbs fixture check` before fixture-heavy cases such as MCP, RAG, multimodal, or plugin smoke tests. +Use `bin/lbs case list --ready` for cases that have no missing machine inputs and no manual preconditions. Use `bin/lbs case list --machine-ready` when you want to keep `manual-check` candidates and confirm their preconditions yourself. + +Before executing a saved QA path, generate the agent-facing plan: + +```bash +bin/lbs test plan +``` + +Read the plan readiness sections before running the browser path. Missing env, +automation env, or fixture readiness means the case is not ready to execute and +should be marked `blocked` or fixed first. +`manual_check` means machine inputs are present but the agent must verify the +declared `preconditions` or `setup` items before executing the UI path. Do not +turn a `manual_check` case into `pass` until those items were checked in the +same run. + +Before executing a group of saved QA paths, generate the suite plan: + +```bash +bin/lbs suite plan +``` + +Use `bin/lbs suite start ` to create a shared suite run id, suite evidence root, per-case evidence directories, and `suite-start.json`/`suite-start.md` handoff files. Then run `bin/lbs suite report --evidence-dir ` to aggregate case results. +Automation scripts write `automation-result.json`; write the final per-case `result.json` with `bin/lbs test result --result --reason --evidence-dir --evidence ` after collecting the required evidence. A `pass` result must include all required evidence. +For runner-specific Debug Chat cases, prefer case-specific pipeline env keys such as `LANGBOT_LOCAL_AGENT_PIPELINE_URL` over the generic `LANGBOT_PIPELINE_URL`; otherwise an agent can accidentally test the wrong runner. diff --git a/skills/README.md b/skills/README.md new file mode 100644 index 000000000..d21ee4a16 --- /dev/null +++ b/skills/README.md @@ -0,0 +1,54 @@ +# LangBot Skills + +This directory is the **single source of truth** for LangBot's agent skills — +reusable, on-demand instruction packs for AI agents (Claude Code, Codex, Cursor, +and LangBot's own Local Agent) working with the LangBot ecosystem. + +> These skills were consolidated here from the former `langbot-app/langbot-skills` +> repository (now archived). Documentation and the landing page link here; do not +> re-copy skill content elsewhere — link to this directory instead. + +## Skill catalog + +| Skill | What it covers | +| --- | --- | +| [`langbot-dev`](skills/langbot-dev) | Core backend + web frontend development (Quart, Vite, API, migrations, MCP server) | +| [`langbot-plugin-dev`](skills/langbot-plugin-dev) | Plugin SDK / component development, debugging, WebSocket testing | +| [`langbot-deploy`](skills/langbot-deploy) | Docker / Compose / Kubernetes deployment, config.yaml, Box runtime, global API key | +| [`langbot-testing`](skills/langbot-testing) | WebUI / e2e QA harness, cases, fixtures, troubleshooting (the `bin/lbs` CLI) | +| [`langbot-env-setup`](skills/langbot-env-setup) | Local dev/test environment, browser access, OAuth, proxy, startup | +| [`langbot-mcp-ops`](skills/langbot-mcp-ops) | Operating a LangBot instance through its MCP server (`/mcp`) | +| [`langbot-space-ops`](skills/langbot-space-ops) | Browsing the LangBot Space marketplaces through the Space MCP server | +| [`langbot-eba-adapter-dev`](skills/langbot-eba-adapter-dev) | Building platform adapters for the Event-Based Agents architecture | +| [`langbot-skills-maintenance`](skills/langbot-skills-maintenance) | Adding, deduplicating, and auditing skills in this directory | + +`skills.index.json` is the machine-readable index (regenerate with `bin/lbs index`). + +## Quick start (for an AI agent) + +1. Read this README, `AGENTS.md`, and `qa-agent-docs/` to understand the layout. +2. Read `skills/.env` for shared local defaults. On a new machine, copy + `skills/.env.example` to `skills/.env.local` (gitignored) and override + machine-specific values there. Never commit secrets. +3. Pick the smallest relevant skill from the catalog above and follow its + `SKILL.md`. + +## The `lbs` CLI + +The testing assets ship with a small CLI (`bin/lbs`, Node ≥ 22.6): + +```bash +bin/lbs validate # validate skills/cases/troubleshooting structure +bin/lbs index # regenerate skills.index.json +bin/lbs env show # inspect resolved env defaults (redacted) +bin/lbs env doctor # diagnose local environment readiness +bin/lbs case list --ready +bin/lbs test plan +``` + +## Maintenance rule + +When the LangBot / LangBot Space **API or MCP server changes**, the +corresponding skill here MUST be updated in the same change. The MCP tool +surface, the API, and these skills are kept in lockstep — see each repo's +`AGENTS.md`. diff --git a/skills/package.json b/skills/package.json new file mode 100644 index 000000000..10fbb6b6b --- /dev/null +++ b/skills/package.json @@ -0,0 +1,22 @@ +{ + "private": true, + "type": "module", + "bin": { + "lbs": "./bin/lbs" + }, + "scripts": { + "lbs": "node src/lbs.ts", + "test": "node test/lbs-cli.test.ts", + "validate": "node src/lbs.ts validate", + "index": "node src/lbs.ts index", + "index:check": "node src/lbs.ts index --check", + "check:syntax": "find src test scripts -type f \\( -name '*.ts' -o -name '*.mjs' \\) -print0 | xargs -0 -n1 node --check", + "check": "npm run check:syntax && npm run validate && npm test" + }, + "engines": { + "node": ">=22.6" + }, + "devDependencies": { + "playwright": "^1.60.0" + } +} diff --git a/skills/qa-agent-docs/qa-agent/00-technology-options.md b/skills/qa-agent-docs/qa-agent/00-technology-options.md new file mode 100644 index 000000000..0a021f4fb --- /dev/null +++ b/skills/qa-agent-docs/qa-agent/00-technology-options.md @@ -0,0 +1,117 @@ +# LangBot Agent Testing 技术选型 + +## 状态 + +这是技术选型背景文档，不是当前路线图。当前黑盒 E2E QA 的实施顺序见： + +```text +docs/qa-agent/04-black-box-e2e-roadmap.md +``` + +## 目标 + +`langbot-skills` 的目标不是替代测试框架，而是沉淀 agent 可复用的测试资产，让开发者 clone 仓库后，可以让 Codex、Claude Code、Computer Use 或 Playwright MCP 复用已有路径完成 LangBot 功能验证。 + +核心原则： + +- Skill 负责路由和少量规则。 +- Reference 负责可读流程和背景知识。 +- Case 负责结构化测试路径。 +- Troubleshooting 负责结构化故障资产。 +- `lbs` 负责结构校验、索引、资产创建和未来的运行/报告能力。 +- UI/browser 是产品 QA 的主路径；API/curl 只用于诊断。 + +## 浏览器控制层 + +不同开发者可用的浏览器控制能力不同，所以浏览器层必须可替换。 + +| 方案 | 适用场景 | 优点 | 代价 | +|---|---|---|---| +| Codex / Claude Computer Use | agent 可以直接控制可见浏览器 | 登录和交互路径最自然，通常不需要额外 MCP 浏览器桥接 | 依赖具体 agent 工具能力 | +| Playwright MCP | 没有 Computer Use，但有 MCP 浏览器工具 | 稳定、可脚本化、适合回归路径 | OAuth 登录通常需要额外 visible profile | +| 直接 Playwright 脚本 | 测试路径非常稳定，适合 CI | 可重复性强 | 需要维护脚本和 selector | +| 商业 AI QA 平台 | 团队希望外包测试运行平台 | 报告和 PR 集成完整 | 成本和平台绑定 | + +## 当前推荐 + +先采用分层降级： + +```text +有 Computer Use？ + 是 -> 使用 Computer Use 控制浏览器 + 否 -> 使用 Playwright MCP + +需要 GitHub OAuth？ + 是 -> 使用持久浏览器 profile，让用户手动完成登录 + 否 -> 直接使用已有登录态或测试账号状态 +``` + +具体选择逻辑沉淀在： + +```text +skills/langbot-env-setup/references/browser-access-selection.md +``` + +测试原则固定在： + +```text +docs/qa-agent/03-agent-browser-qa-principles.md +``` + +## 环境变量层 + +测试文档不应写死端口。共享默认值放在： + +```text +skills/.env +``` + +关键变量： + +```text +LANGBOT_FRONTEND_URL +LANGBOT_BACKEND_URL +LANGBOT_DEV_FRONTEND_URL +LANGBOT_REPO +LANGBOT_WEB_REPO +LANGBOT_BROWSER_PROFILE +``` + +Agent 执行测试前应先读取 `skills/.env`，再用用户提供的当前环境或已启动服务覆盖默认值。 + +## 测试资产层 + +测试资产分两类： + +```text +skills// + references/ # Markdown 流程说明 + cases/ # 结构化测试用例 + troubleshooting/ # 结构化故障记录 +``` + +当前已实现： + +- `SKILL.md` 路由 +- `references/*.md` +- `lbs case new/list/show` +- `lbs trouble show/search` +- `lbs test plan` +- `lbs test report` +- `lbs list / validate / index` + +下一步重点： + +- 日志守卫规则补充 +- 报告产物管理 + +## 关键判断 + +不要强制所有内容只能通过 CLI 修改。更好的模式是： + +- 新增 case/troubleshooting：优先使用 `lbs` +- 大段流程说明：允许直接编辑 Markdown +- 结构性变更后：必须运行 `lbs validate` +- 任何生成索引的变更后：运行 `lbs index` + +这样既能沉淀结构化资产，又不会在 schema 未稳定时拖慢迭代。 diff --git a/skills/qa-agent-docs/qa-agent/01-qa-agent-harness-plan.md b/skills/qa-agent-docs/qa-agent/01-qa-agent-harness-plan.md new file mode 100644 index 000000000..06b52ee7b --- /dev/null +++ b/skills/qa-agent-docs/qa-agent/01-qa-agent-harness-plan.md @@ -0,0 +1,231 @@ +# LangBot Skills 测试资产库规划 + +## 状态 + +这是早期测试资产库规划文档，保留用于解释 `langbot-skills` 的分层来源。 + +当前路线已经收敛为黑盒 E2E QA：开发者用 agent 通过浏览器测试 LangBot， +稳定路径沉淀为 case，失败知识沉淀为 troubleshooting。`lbs test report` 和 +日志守卫已有 MVP，后续重点是报告证据、case 元数据和少量稳定路径自动化。当前优先级见： + +```text +docs/qa-agent/04-black-box-e2e-roadmap.md +``` + +本文中关于 `case list/show`、`trouble show/search`、`test plan` 的“计划实现” +内容已经部分过时，因为这些能力已经落地。 + +## 目标 + +让开发者 clone `langbot-skills` 后，可以把测试意图交给 agent，由 agent 复用已有环境配置、测试路径和故障知识完成 LangBot 功能验证。 + +典型场景： + +- 冒烟测试：验证 pipeline Debug Chat、provider、常见页面是否正常。 +- Provider 测试：添加 DeepSeek/OpenAI/Claude 等供应商并验证模型可用。 +- 新 feature 测试：探索新 UI 路径，并在稳定后沉淀成 case/reference。 +- 回归测试：复用旧路径，避免每个窗口重新探索登录、模型配置、pipeline 调试。 +- 故障沉淀：把 runtime 超时、代理不一致、WebSocket 问题记录为可搜索资产。 + +核心方向见 `03-agent-browser-qa-principles.md`：agent 必须以浏览器/UI 为主路径，API/curl 只能作为诊断手段。 + +## 当前仓库结构 + +```text +skills/ + .env # 共享默认变量 + langbot-env-setup/ # 环境准备、浏览器控制路径、代理、登录态 + langbot-testing/ # WebUI / provider / pipeline 测试入口 + langbot-plugin-dev/ # 插件开发测试 + langbot-eba-adapter-dev/ # 平台适配器开发测试 +src/ + lbs.ts # CLI 源码 +bin/ + lbs # CLI 入口 +docs/ + qa-agent/ # 规划文档，历史目录名保留 +``` + +## 设计分层 + +### 1. Skill 层 + +`SKILL.md` 只做触发和路由，不承载大段流程。 + +例子： + +```text +langbot-env-setup -> 选择 Computer Use / Playwright MCP / OAuth profile / proxy +langbot-testing -> 选择 WebUI / pipeline / provider / troubleshooting +``` + +### 2. Reference 层 + +Markdown 记录人和 agent 都能读的流程说明。 + +适合内容： + +- 如何选择浏览器控制方式 +- 如何启动/检查服务 +- 如何执行 pipeline Debug Chat +- 如何处理 OAuth 登录态 + +### 3. Case 层 + +使用 YAML 记录可重复测试路径。 + +建议结构： + +```text +skills/langbot-testing/cases/ + pipeline-debug-chat.yaml + provider-deepseek.yaml +``` + +建议格式： + +```yaml +id: pipeline-debug-chat +title: Pipeline Debug Chat returns a bot response +mode: agent-browser +area: pipeline +type: smoke +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL +steps: + - Open LANGBOT_FRONTEND_URL + - Navigate to Pipelines + - Open target pipeline + - Select Debug Chat + - Send deterministic prompt +checks: + - "UI: User message appears" + - "UI: Bot message appears" + - "Console: No unexpected frontend errors" + - "Logs: Backend log includes Conversation(0) Streaming completed" +diagnostics: + - "Use API/curl only after the UI path is attempted, to distinguish frontend display failure from backend/runtime failure." +troubleshooting: + - plugin-runtime-timeout + - proxy-env-mismatch +``` + +### 4. Troubleshooting 层 + +故障资产会逐渐变大，适合结构化记录。 + +历史 Markdown 入口保留在： + +```text +skills/langbot-testing/references/troubleshooting.md +``` + +当前 canonical 结构化故障资产在： + +```text +skills/langbot-testing/troubleshooting/ + plugin-runtime-timeout.yaml + proxy-env-mismatch.yaml +``` + +### 5. CLI 层 + +`lbs` 是统一入口，不再引入独立 `qa` 命令。 + +已实现或当前可用： + +```bash +bin/lbs list +bin/lbs validate +bin/lbs index +bin/lbs new-skill +bin/lbs new-ref +bin/lbs case new pipeline-debug-chat --title "Pipeline Debug Chat" +bin/lbs case list +bin/lbs case show pipeline-debug-chat +bin/lbs trouble list +bin/lbs trouble show plugin-runtime-timeout +bin/lbs trouble search runtime +bin/lbs trouble add --title ... --symptom ... --cause ... --fix ... +bin/lbs test plan pipeline-debug-chat +bin/lbs test start pipeline-debug-chat +bin/lbs test run pipeline-debug-chat --dry-run +bin/lbs test report pipeline-debug-chat +bin/lbs test report pipeline-debug-chat --backend-log /path/to/backend.log +``` + +## 测试库位置 + +不要使用隐藏 `.qa/` 作为主测试库。测试资产应该和 skill 放在一起，便于触发和维护： + +```text +skills/langbot-testing/ + references/ + cases/ + troubleshooting/ + reports/ # 可选，本地运行产物可按需忽略或输出到外部目录 +``` + +如果未来需要项目本地测试库，可以允许 `lbs` 支持 `--workspace` 或项目根目录配置，但 canonical 资产仍保存在 `langbot-skills`。 + +## 阶段规划 + +### 阶段一：环境和测试路径沉淀 + +状态：基本完成，持续维护。 + +- `skills/.env` 管共享默认变量。 +- `langbot-env-setup` 拆出 Computer Use、Playwright MCP、OAuth profile、proxy、service startup。 +- `langbot-testing` 记录 WebUI、pipeline、provider 测试路径。 +- `lbs validate/index` 维护结构。 + +完成标准： + +- agent 可以从 `skills/.env` 和 references 中找到当前测试入口。 +- pipeline Debug Chat 这类路径不再需要从头探索。 + +### 阶段二：结构化 case/troubleshooting + +状态：主体已完成，继续补齐元数据和资产质量。 + +目标： + +- `lbs case new/list/show` +- `lbs trouble show/search` +- case id 去重、字段校验、索引生成 + +完成标准： + +- 冒烟测试路径可以用结构化 case 表示。 +- 下一个 agent 窗口可以直接读取 case 执行。 + +### 阶段三：计划和报告 + +状态：已有 MVP，继续完善。 + +目标： + +- `lbs test plan ` +- agent 按 plan 使用浏览器执行 UI QA +- `lbs test report` +- 日志守卫集成 +- 报告产物和 evidence 约定 + +完成标准： + +- agent 可以按 case plan 执行浏览器测试。 +- 结果报告包含 UI 结果、后端日志、console 错误和 troubleshooting 建议。 + +## 执行规则 + +- agent 可以直接编辑 Markdown reference。 +- 新增结构化 case/troubleshooting 时，优先使用 `lbs`。 +- 每次结构变更后运行 `bin/lbs validate`。 +- 每次索引相关变更后运行 `bin/lbs index`。 +- 测试文档不写死端口，使用 `skills/.env` 中的 URL 变量。 +- 测试 case 的 `mode` 固定为 `agent-browser`。 +- API/curl 只能写入 `diagnostics`，不能替代 UI 步骤和 UI 检查。 diff --git a/skills/qa-agent-docs/qa-agent/02-log-guard-plan.md b/skills/qa-agent-docs/qa-agent/02-log-guard-plan.md new file mode 100644 index 000000000..a166b1378 --- /dev/null +++ b/skills/qa-agent-docs/qa-agent/02-log-guard-plan.md @@ -0,0 +1,161 @@ +# 日志守卫规划 + +## 状态 + +这是当前活跃设计，已有第一版文件扫描 MVP。实现边界需要和黑盒 E2E 路线保持一致： + +- 日志守卫服务于 `lbs test report`。 +- 它不替代浏览器/UI 判断。 +- 它不发展成独立后端 API 测试框架。 +- 第一版默认扫描 `LANGBOT_REPO/data/logs/` 下最新的 `langbot-*.log`，也可扫描 agent + 显式提供的 backend/frontend/console 日志文件。 + +当前总体路线见： + +```text +docs/qa-agent/04-black-box-e2e-roadmap.md +``` + +## 目标 + +日志守卫是 `lbs test report` 的一部分，用来在 agent 执行测试期间捕获 UI 断言之外的运行时问题。 + +当前命令方向已收敛为 `lbs test plan` / `lbs test report`。日志守卫服务于 agent-browser QA，不是独立的后端 API 测试入口。 + +LangBot 是异步且集成度高的系统，有些问题不会直接表现为页面失败： + +- 后台任务异常 +- 未等待的协程 +- Provider 流式调用失败 +- 插件 runtime 超时 +- 平台发送失败 +- 数据库连接问题 +- 敏感信息泄露 + +日志守卫负责把这些信号结构化地放进测试报告，并关联到 troubleshooting 资产。 + +## 输入 + +日志守卫应从环境和运行上下文读取配置： + +- `skills/.env` 中的 `LANGBOT_BACKEND_URL` +- `skills/.env` 中的 `LANGBOT_REPO`，用于自动发现 LangBot 后端日志 +- `lbs test plan` / report 记录的 case id +- LangBot 后端进程输出 +- 前端 dev server 输出 +- 浏览器 console/network 错误 +- case 声明的 success/failure patterns 和 expected failures + +## MVP 范围 + +- 读取一个或多个日志流或日志文件。 +- 检测错误模式。 +- 支持按 case id 或 pattern 白名单。 +- 输出 JSON/Markdown 摘要。 +- 发现非预期错误时让测试报告标记失败；未来如果有自动执行器，再返回非零退出码。 + +## 错误分类 + +### 永远非预期 + +除非 case 明确声明，否则应失败： + +- `Traceback` +- `Task exception was never retrieved` +- `RuntimeWarning: coroutine .* was never awaited` +- `Unclosed client session` +- `Unclosed connector` +- `KeyError` +- `TypeError` +- `AttributeError` +- 密钥、token、secret 明文泄露 + +### Case 预期错误 + +只有当前 case 声明时允许： + +- 无效 provider key +- Provider 认证失败 +- 无效 webhook payload +- 插件测试故意抛错 +- 超时测试 +- 限流测试 + +### 仅警告 + +报告但默认不失败： + +- 可恢复重试 +- 恢复的超时 +- 废弃配置 +- 慢请求 +- 版本检查失败 + +## 与 Troubleshooting 集成 + +日志守卫不只输出错误文本，还应尽量匹配已知 troubleshooting id。 + +例子： + +```text +Action list_plugins call timed out +Action list_agent_runners call timed out +Action invoke_llm_stream call timed out +``` + +可映射到： + +```text +plugin-runtime-timeout +``` + +```text +uppercase proxy points to one host, lowercase proxy points to another +``` + +可映射到： + +```text +proxy-env-mismatch +``` + +## 未来命令 + +```bash +bin/lbs test plan pipeline-debug-chat +bin/lbs test start pipeline-debug-chat +bin/lbs test run pipeline-debug-chat --dry-run +bin/lbs test report pipeline-debug-chat +bin/lbs test report --output report.md +bin/lbs test report pipeline-debug-chat --backend-log /path/to/backend.log --console-log /path/to/console.log +bin/lbs test report pipeline-debug-chat --since "2026-05-21T10:30:00+08:00" +bin/lbs test report pipeline-debug-chat --tail-lines 2000 +bin/lbs test report pipeline-debug-chat --since "2026-05-21T10:30:00+08:00" --tail-lines 2000 +bin/lbs test report pipeline-debug-chat --no-auto-log +``` + +运行报告应包含： + +- case id +- URL 和环境变量摘要，不能包含 secrets +- 浏览器可见结果 +- 后端日志摘要 +- console/network 错误 +- 匹配到的 troubleshooting id +- 通过/失败结论 + +## MVP 完成标准 + +- 可以自动扫描最新 LangBot 后端日志，也可以扫描前端日志和 console 日志文件。 +- 可以用 `--since` 或 `--tail-lines` 把扫描范围限制到本次测试窗口。 +- 可以检测明显 Python/运行时错误和 secret 泄露风险。 +- 可以识别 case 声明的 success/failure patterns。 +- 可以识别 troubleshooting pattern，包括 `plugin-runtime-timeout` 和 `proxy-env-mismatch`。 +- 支持 case 级白名单。 +- 输出机器可读摘要。 +- 至少一个 `langbot-testing` case 使用它。 + +当前 MVP 已覆盖自动发现 LangBot 后端日志、文件扫描、`--since`/`--tail-lines` 扫描窗口、 +基础错误检测、case success/failure signal、troubleshooting 匹配、secret 脱敏和 `--json` +输出。仍待继续完善的是 live log 采集、更多规则、case 级 expected failure 的资产化和真实 +E2E report 样例。 diff --git a/skills/qa-agent-docs/qa-agent/03-agent-browser-qa-principles.md b/skills/qa-agent-docs/qa-agent/03-agent-browser-qa-principles.md new file mode 100644 index 000000000..15a18c21a --- /dev/null +++ b/skills/qa-agent-docs/qa-agent/03-agent-browser-qa-principles.md @@ -0,0 +1,57 @@ +# Agent Browser QA Principles + +This document fixes the direction of LangBot agent testing so the project does not drift into a backend API smoke-test framework. + +## Primary Goal + +`langbot-skills` should help an agent behave like a QA engineer using the product, not like a backend curl script. + +The primary path is: + +```text +developer intent -> lbs test plan -> agent controls browser -> UI result + console + logs -> report/assets +``` + +## Rules + +1. Browser/UI interaction is the source of truth for product QA cases. +2. A backend API or curl response is never enough to mark a UI case passed. +3. API/curl/log checks are allowed as diagnostics after a UI path is attempted or when debugging environment readiness. +4. A case passes only when the user-visible UI result is correct. +5. The agent should inspect browser console/network output when available. +6. If screenshot or vision capability is available, the agent should check for blank pages, overlap, hidden actions, broken layout, and error toasts. +7. If no visual model is available, use DOM/accessibility snapshots and console output instead. +8. New stable UI paths should be added as `cases/*.yaml`. +9. New recurring failure modes should be added as `troubleshooting/*.yaml`. +10. Secrets, tokens, API keys, and localStorage token values must never be printed. + +## Command Semantics + +`lbs` manages assets and produces plans. It does not replace the agent's browser-control ability. + +```bash +bin/lbs test plan pipeline-debug-chat +``` + +This command outputs: + +- environment variables to use +- required skills +- browser steps +- UI/console/visual/log checks +- diagnostic options +- related troubleshooting patterns +- report template + +The active agent then executes the plan with Computer Use, Playwright MCP, or another available browser-control tool. + +## Diagnostics + +Diagnostics can include: + +- `bin/lbs env doctor` +- browser console/network inspection +- backend logs +- targeted API/curl checks + +Diagnostics answer "where did it fail?" They do not replace "did the user-visible UI work?" diff --git a/skills/qa-agent-docs/qa-agent/04-black-box-e2e-roadmap.md b/skills/qa-agent-docs/qa-agent/04-black-box-e2e-roadmap.md new file mode 100644 index 000000000..4d8652519 --- /dev/null +++ b/skills/qa-agent-docs/qa-agent/04-black-box-e2e-roadmap.md @@ -0,0 +1,299 @@ +# 黑盒 E2E QA 路线图 + +## 定位 + +LangBot 有大量外部依赖：模型供应商、plugin runtime、浏览器登录态、 +marketplace/network、RAG engine、sandbox backend、平台适配器等。单测仍然有价值， +但这个 QA 方向当前不优先解决 LangBot core 的单测覆盖率问题，因为重 mock 往往不能 +真实代表产品路径。 + +`langbot-skills` 当前目标是让黑盒 E2E 测试变得可执行、可沉淀、可复用： + +```text +开发者测试意图 +-> 复用或新增 case +-> agent 通过浏览器执行 +-> UI + console + network + log 证据 +-> report +-> 反哺 case / troubleshooting +``` + +这是面向开发者的 QA 资产库。开发者可以让 agent 测一个 feature；如果路径稳定， +就把路径正规化为 case，让下一个开发者或 QA agent 继续复用。 + +## 非目标 + +- 这一阶段不优先建设 LangBot core 单测覆盖率。 +- 不把 API/curl 作为 WebUI 行为的通过标准。 +- 不要求每个 case 都能进 CI。 +- 不在 report 和日志守卫有用之前急着做完整 browser runner。 +- 不把外部 provider、OAuth、marketplace 抖动直接判成产品失败，除非证据明确。 + +## 当前状态 + +仓库已经具备第一层基础设施： + +- `skills/.env` 和 `skills/.env.local` 管理测试环境； +- `langbot-env-setup`、`langbot-testing`、`langbot-plugin-dev` 等 skill； +- `skills/langbot-testing/cases` 下的结构化 case； +- `skills/langbot-testing/troubleshooting` 下的结构化故障资产； +- RAG、多模态、plugin、MCP 等 fixture； +- `bin/lbs validate`、`bin/lbs index`、`bin/lbs case`、`bin/lbs trouble`、 + `bin/lbs test plan`、`bin/lbs test start`、`bin/lbs test report`。 + +所以当前已经不是“先把路径写进 Markdown”的阶段，而是进入“让每次运行有证据、 +有报告、能沉淀”的阶段。 + +## 测试模型 + +UI case 只有在用户可见行为正确时才能通过。辅助证据必须解释同一次运行。 + +通过一个 UI case 的最低证据： + +- 用户可见的成功信号，例如 bot 回复、provider 保存成功、文件上传完成、plugin 页面渲染； +- 没有意外 browser console error； +- 相关时间窗口内没有意外后端/runtime 错误； +- 有截图、DOM snapshot 或同等视觉/结构证据，如果当前 agent 能获取； +- API/curl 只在解释同一条 UI 路径时作为诊断证据。 + +失败报告需要保留足够信息，让开发者能复现或分流： + +- case id 和实际测试 URL； +- 使用的 browser path； +- 最后可见 UI 状态； +- console/network 症状； +- 相关后端/前端日志； +- 匹配到的 troubleshooting id； +- 这是产品失败、环境问题、外部依赖抖动，还是证据不足。 + +## 结果词汇 + +统一使用这些结果： + +- `pass`：UI 行为正确，辅助证据干净。 +- `fail`：UI 行为错误，或同一次运行的 console/log 出现意外产品错误。 +- `blocked`：缺登录、缺 provider credentials、服务未启动等原因导致目标路径没有跑起来。 +- `env_issue`：失败在目标行为之外，例如 proxy、OAuth、provider quota、marketplace outage、 + 本地服务启动问题。 +- `flaky`：同一环境下结果不稳定，进入门禁前需要先稳定。 + +做 merge/release 判断时，`env_issue` 和 `blocked` 不能算产品通过。 + +## 路线图 + +### Phase 0：对齐文档 + +目标：明确当前黑盒 E2E 方向。 + +交付物： + +- `docs/qa-agent/README.md` 文档状态导航； +- 本路线图； +- 给旧规划文档加状态说明。 + +完成标准： + +- 新贡献者不用通读所有旧文档，也能知道当前重点。 + +### Phase 1：Test Report MVP + +状态：已有第一版。 + +目标：让每次 agent browser 测试都有一致报告格式，即使 browser 执行还没自动化。 + +建议命令： + +```bash +bin/lbs test start +bin/lbs test report --output reports/-.md +``` + +MVP 行为： + +- 读取 case 和关联 troubleshooting； +- 生成 Markdown report 模板； +- 生成 run handoff，固定本次测试的 start timestamp 和推荐 report command； +- 写入脱敏后的环境摘要； +- 提供 `pass/fail/blocked/env_issue/flaky` 结果选项； +- 包含 UI result、console errors、network symptoms、logs、screenshots、 + diagnostics、matched troubleshooting、assets to update 等 section； +- 支持 `--json`，输出机器可读报告。 + +第一版已经是 report generator，不急着做自动判定。先把 evidence 收集格式统一起来， +再做自动化更稳。 + +完成标准： + +- agent 可以先跑 `lbs test start `，用它给出的时间窗口执行浏览器路径， + 然后按固定格式填写 report，不需要每次重新发明报告结构。 + +### Phase 2：日志守卫 MVP + +状态：已有第一版文件扫描。 + +目标：捕获 UI 不一定明显展示的 runtime 问题。 + +日志守卫应集成进 `lbs test report`，不要发展成独立后端 API 测试框架。 + +建议命令形态： + +```bash +bin/lbs test report \ + --backend-log /path/to/backend.log \ + --frontend-log /path/to/frontend.log \ + --console-log /path/to/console.log \ + --evidence-dir reports/evidence/ \ + --since "2026-05-21T10:30:00+08:00" \ + --tail-lines 2000 \ + --output reports/-.md +``` + +MVP 行为： + +- 默认从 `LANGBOT_REPO/data/logs/` 扫描最新 `langbot-*.log`； +- 支持 agent 显式提供 backend、frontend、console 日志文件； +- 支持读取 evidence 目录下的 `automation-result.json`，把浏览器自动化脚本结论纳入报告； +- 支持 `lbs test result` 为人工/agent browser 运行写入标准 `result.json`，供 suite 聚合； +- 支持 `--since` 和 `--tail-lines`，避免历史日志污染本次报告； +- 检测默认非预期模式，例如 `Traceback`、未 await coroutine、unclosed client/connector、 + `KeyError`、`TypeError`、`AttributeError`、明显 secret 泄露； +- 匹配 case 声明的 `success_patterns` 和 `failure_patterns`； +- 匹配已知 troubleshooting，先支持 `plugin-runtime-timeout` 和 `proxy-env-mismatch`； +- 只有 case 明确声明时，才允许 expected failure； +- 将发现分类为 fail、warning、matched troubleshooting、ignored expected issue； +- 永远不打印 secret 值。 + +完成标准： + +- 至少 `pipeline-debug-chat` 能生成包含日志摘要和 troubleshooting 匹配结果的 report。 + +### Phase 3：Case 元数据加固 + +状态：已有第一版。 + +目标：让 case 更容易选择、执行和晋级。 + +字段逐步补充，保持向后兼容： + +```yaml +priority: p0 | p1 | p2 +risk: low | medium | high +ci_eligible: false +preconditions: + - "Authenticated browser profile is available." +setup: + - "Start LangBot backend and frontend." +cleanup: + - "Remove temporary provider, plugin, or knowledge base if created." +expected_failures: [] +success_patterns: + - "Conversation(0) Streaming completed" +failure_patterns: + - "Action invoke_llm_stream call timed out" +evidence: + required: + - ui + - console + - backend_log +``` + +当前实现采用扁平字段 `evidence_required`，避免轻量 YAML 解析器在 case 文件里承载嵌套结构。 +`bin/lbs validate` 会校验 `priority`、`risk`、`ci_eligible`、`evidence_required`、 +`automation` 脚本路径、case 关联 skill 和 troubleshooting 交叉引用。`bin/lbs case list` +支持 `--json`、`--type`、`--area`、`--tag`、`--priority`、`--risk`、`--automation`、`--ci` +、`--ready` 和 `--machine-ready` 过滤，方便 agent 快速选择测试集。 +`env_any` 和 `automation_env_any` 用于表达 URL-or-name 这类 one-of 输入，避免把可替代变量误判为全部必填。 + +当前也有 `skills//suites/*.yaml` 和 `bin/lbs suite plan `，用于组织常跑测试集， +例如 `core-smoke`、`local-agent-gate` 和 +`agent-runner-release-gate`。发布门禁使用 `agent-runner-release-preflight` +先分类配置 blockers 和 runtime env issues，再运行较重的浏览器 Debug Chat case。 +依赖 fixture 的 case 可以在浏览器执行前先跑 `bin/lbs fixture check`，检查 +`fixtures/fixtures.json` 登记的 deterministic 文件、plugin 包和本地测试 server 是否存在。 +`bin/lbs suite start ` 会生成 suite run id、suite evidence root、per-case evidence 目录、 +`suite-start.json`/`suite-start.md` handoff 文件和 per-case evidence 命令； +浏览器自动化脚本会写入 `automation-result.json`，供 `bin/lbs test report` 展示原始自动化结论； +`bin/lbs test result ` 会在人工/agent browser case 完成后写入最终 `result.json`； +`bin/lbs suite report --evidence-dir ` 会聚合各 case 的 `result.json`，并且 +不会把缺少 required evidence 的 `pass` 当作 suite 通过。 +Runner 专用 Debug Chat case 通过 `automation_pipeline_url_env` 和 +`automation_pipeline_name_env` 绑定专用 pipeline 变量，避免 local-agent、Codex 或 +Claude Code case 误用通用 `LANGBOT_PIPELINE_URL` 后产生假阳性。 +Debug Chat case 还可以通过 `automation_stream_output` 固定流式或非流式发送路径。 +多模态 Debug Chat case 可以通过 `automation_image_base64_fixture` 复用 deterministic 图片 fixture。 +`test plan` 和 `suite plan` 会输出 readiness，让 agent 在执行浏览器前就看到缺失的 env、 +自动化变量、fixture，以及需要人工确认的 `manual_check` 前置条件。 + +完成标准： + +- `lbs case list` 或后续 filter 能回答“smoke 跑哪些”、“哪些适合 CI”、 + “哪些需要真实 provider credentials”。 + +### Phase 4：开发者沉淀流程 + +目标：开发者让 agent 测新 feature 后，稳定路径不会丢在聊天记录里。 + +流程： + +1. 开发者要求 agent 通过浏览器测试某个 feature。 +2. agent 先按 UI 主路径探索。 +3. agent 用 `lbs test start` 固定运行窗口，再用 `lbs test report` 写报告。 +4. 如果路径稳定，agent 新增或更新 case。 +5. 如果出现可复用故障，agent 新增或更新 troubleshooting。 +6. agent 跑 `bin/lbs validate` 和 `bin/lbs index`。 + +完成标准： + +- feature QA 的结果能进入资产库，而不是只留在一次对话里。 + +### Phase 5：选择性浏览器自动化 + +状态：已有第一版 `test run` 入口和两个 Playwright 脚本。 + +目标：只自动化少量稳定、值得重复跑的黑盒路径。 + +建议顺序： + +1. `webui-login-state` +2. `pipeline-debug-chat` +3. `local-agent-basic-debug-chat` +4. `local-agent-rag-debug-chat` +5. 一个基于 deterministic fixture 的 plugin 或 MCP smoke path + +执行策略： + +- 继续把 Computer Use 或 Playwright MCP 作为默认交互路径； +- 只给稳定、确定性的路径补直接 Playwright script； +- 保存 screenshots、console logs、trace/video； +- flaky 或强依赖真实 credentials 的 provider case 暂时不要进 CI。 + +当前已经绑定： + +- `webui-login-state` -> `scripts/e2e/webui-login-state.mjs` +- `pipeline-debug-chat` -> `scripts/e2e/pipeline-debug-chat.mjs` + +第一版自动化先产出 `reports/evidence//` 下的 console、network、screenshot 和 +result JSON。真实执行后仍要用 `lbs test report --since ... --console-log ...` 做日志守卫和 +最终报告。开发期间可以先用 `bin/lbs test run --dry-run` 检查命令和 evidence 路径。 +Debug Chat 类脚本应复用 `scripts/e2e/lib/debug-chat.mjs`，避免重复实现 visible response leaf +判断和已知失败信号分类。 + +完成标准： + +- 小规模 smoke subset 可以不靠人工决定每一步点击；更大的资产库仍然服务于人工/agent + 驱动的探索式 E2E。 + +## 下一批动工切片 + +在做 browser runner 之前，继续做这些： + +1. 等 LangBot 当前开发状态稳定后，用一次真实 `pipeline-debug-chat` 跑通 + `test start -> test run -> test report -> test result -> suite report`，产出 sample report。 +2. 只给 smoke/local-agent 首批 case 补必要元数据。 +3. 继续补日志守卫规则，尤其是 WebSocket、plugin runtime、provider streaming、前端 + chunk/rendering failure。 +4. 约定 report 产物目录、截图和 console/network 导出的命名方式。 +5. 再评估是否开始给 `webui-login-state` 和 `pipeline-debug-chat` 做直接 Playwright + 自动化。 + +这样 infra 会立刻有用，同时保留后续自动化 browser execution 的空间。 diff --git a/skills/qa-agent-docs/qa-agent/README.md b/skills/qa-agent-docs/qa-agent/README.md new file mode 100644 index 000000000..e8443450d --- /dev/null +++ b/skills/qa-agent-docs/qa-agent/README.md @@ -0,0 +1,46 @@ +# LangBot QA Agent 文档导航 + +这个目录记录 `langbot-skills` 当前的 QA 方向和后续建设顺序。 + +## 当前判断 + +当前重点是 LangBot 的黑盒 E2E QA，不是 LangBot core 的单测覆盖率建设。 + +`langbot-skills` 要帮助开发者和 QA agent 做接近人工测试的 WebUI 验证： + +- 打开真实 LangBot WebUI； +- 按用户路径点击和输入； +- 检查用户可见的 UI 结果； +- 查看 console、network、截图、后端和前端日志； +- 输出可复用的测试报告； +- 把稳定 feature 路径沉淀为 case； +- 把重复故障沉淀为 troubleshooting。 + +API 和 curl 只做诊断。它们可以解释失败原因，但不能让一个 UI case 通过。 + +## 文档状态 + +| 文档 | 状态 | 用途 | +| --- | --- | --- | +| `04-black-box-e2e-roadmap.md` | 当前主路线图 | 决定下一步建设什么。 | +| `03-agent-browser-qa-principles.md` | 当前原则文档 | 定义 browser-first QA 的通过标准。 | +| `02-log-guard-plan.md` | 当前活跃设计 | 设计 `lbs test report` 里的日志守卫。 | +| `../user-guide.md` | 当前使用手册 | 开发者日常使用。 | +| `00-technology-options.md` | 背景文档 | 选择 Computer Use、Playwright MCP 或未来直接 Playwright。 | +| `01-qa-agent-harness-plan.md` | 历史规划，部分过时 | 解释最初分层和目录设计；使用前先看状态说明。 | + +## 已过时的点 + +`01-qa-agent-harness-plan.md` 还保留早期规划状态。现在结构化 cases、 +结构化 troubleshooting、`validate`、`index`、`lbs test plan` 都已经落地。 + +已经补上第一版 `lbs test start`、`lbs test run`、`lbs test report` 和日志守卫文件扫描。 +`webui-login-state`、`pipeline-debug-chat` 已经绑定直接 Playwright 自动化脚本。后续重点是： + +- 报告 evidence 字段继续打磨； +- case success/failure signal 和日志守卫规则继续补充； +- 报告产物和 evidence 约定； +- 等 LangBot 当前开发状态稳定后跑真实 sample report。 + +不要再把旧阶段列表当成当前 source of truth。后续排序以 +`04-black-box-e2e-roadmap.md` 为准。 diff --git a/skills/qa-agent-docs/user-guide.md b/skills/qa-agent-docs/user-guide.md new file mode 100644 index 000000000..f19beeb97 --- /dev/null +++ b/skills/qa-agent-docs/user-guide.md @@ -0,0 +1,521 @@ +# LangBot Skills 用户使用手册 + +## 这个仓库解决什么 + +`langbot-skills` 是给 agent 使用的 LangBot 测试资产库。开发者 clone 后，可以让 Codex、Claude Code、Computer Use 或 Playwright MCP 复用已有环境配置、测试路径和故障知识，像 QA 一样操作 LangBot WebUI。 + +核心目标： + +- 不让下一个 agent 窗口从头探索登录、模型配置、pipeline 调试。 +- 把稳定 UI 测试路径沉淀为 case。 +- 把常见故障沉淀为 troubleshooting。 +- 让 agent 优先通过浏览器点击验证产品行为。 +- API/curl/log 只作为诊断手段，不作为 UI case 通过标准。 + +## 快速开始 + +1. Clone 仓库。 + +2. 检查本地默认变量： + +```bash +bin/lbs env show +``` + +默认变量在： + +```text +skills/.env +``` + +本机专用覆盖写到： + +```text +skills/.env.local +``` + +它会覆盖 `skills/.env` 中的同名变量，并且不应该提交。 +`skills/.env` 是共享默认值，不应写入本机绝对路径、浏览器 profile、provider key 或其他凭据。 +新机器建议从模板开始： + +```bash +cp skills/.env.example skills/.env.local +``` + +常用变量： + +```text +LANGBOT_FRONTEND_URL +LANGBOT_BACKEND_URL +LANGBOT_DEV_FRONTEND_URL +LANGBOT_REPO +LANGBOT_WEB_REPO +LANGBOT_BROWSER_PROFILE +``` + +3. 检查环境是否就绪： + +```bash +bin/lbs env doctor +bin/lbs fixture check +``` + +`env doctor` 会检查 URL、路径、代理变量等。代理变量是可选项；只有大小写代理变量互相冲突时才会报错。失败不一定代表仓库坏了，通常说明本地 LangBot 没启动、代理不一致或浏览器 profile 不存在。 +`fixture check` 会检查仓库内测试 fixture 是否存在，例如 MCP stdio server、RAG 文档、多模态图片、qa-plugin-smoke 包和 QA AgentRunner 包。它也会校验 `.lbpkg` 是 zip 包，并检查 QA AgentRunner fixture 的入口文件未漂移。 + +4. 查看已有测试 case： + +```bash +bin/lbs case list +bin/lbs case list --json --priority p0 --automation +bin/lbs case list --ready +bin/lbs case list --machine-ready +bin/lbs suite list +bin/lbs suite plan core-smoke +bin/lbs suite plan agent-runner-release-gate +bin/lbs suite start core-smoke +bin/lbs suite start core-smoke --run-id core-smoke-local --evidence-dir reports/evidence/core-smoke-local +``` + +`case list` 支持按 `--type`、`--area`、`--tag`、`--priority`、`--risk`、`--automation` +、`--ci`、`--ready` 和 `--machine-ready` 过滤。`--ready` 只显示没有缺机器输入且没有人工前置条件的 case； +`--machine-ready` 过滤掉缺机器输入的 case，但保留 `manual-check`，表示执行前还要确认前置条件。需要交给 agent 自动选择测试集时，优先使用 `--json`， +其中包含 `priority`、`risk`、`ci_eligible`、`automation`、`evidence_required` 以及 +env/automation/fixture/manual readiness。 +Case metadata 中的 `env` 和 `automation_env` 表示全部必填；URL 或 name 这类二选一输入会放在 +`env_any` 或 `automation_env_any`，readiness 只要求组合里至少一个变量有值。 + +如果要跑一组已沉淀的测试路径，优先使用 suite。Suite 位于 `skills//suites/*.yaml`， +只负责组织 case，不改变 UI/browser 作为通过标准的原则。 +`suite plan` 会聚合 readiness：缺环境变量、缺自动化变量、缺 fixture 或需要 +`manual_check` 时，会在执行前标出受影响的 case。`manual_check` 不是产品通过， +它表示机器配置已满足但 agent 必须先确认 case 里的 `preconditions` 或 `setup`。 +Runner externalization 发布判断使用 `agent-runner-release-gate`。先跑 +`agent-runner-release-preflight`，把缺 pipeline、runner id 错误、插件未安装这类 +`blocked`，以及 provider key、Box、插件运行时这类 `env_issue` 分开，再执行较重的 +浏览器 Debug Chat case。 + +5. 生成 agent 执行计划： + +```bash +bin/lbs test plan pipeline-debug-chat +``` + +然后把计划交给当前 agent 执行。agent 应使用 Computer Use、Playwright MCP 或其他浏览器控制能力去操作 UI。 +`test plan` 中的 Environment、Automation Readiness、Fixture Readiness 和 Manual +Readiness 是执行前门禁；如果 readiness 缺失，应先补配置或将本次 case 标记为 +`blocked`。如果状态是 `manual_check`，先确认 `preconditions` 和 `setup`，再开始 UI +执行。不要把后续 curl/API 诊断当成 UI case 通过。 + +## 推荐使用方式 + +### 冒烟测试 + +你可以直接对 agent 说： + +```text +帮我跑一下 LangBot 新前端 smoke test。 +``` + +agent 应该： + +- 读 `skills/.env` +- 优先查看 `bin/lbs suite plan core-smoke`，或查找 `type: smoke` 的 cases +- 生成 test plan +- 用浏览器执行 UI 操作 +- 检查 console、截图、后端日志 +- 输出简短 QA 报告 + +### Runner Externalization 发布门禁 + +你可以直接对 agent 说： + +```text +按 agent-runner release gate 跑完整矩阵，先做 preflight，再跑浏览器 case，并把 blocked/env_issue/fail 分开。 +``` + +agent 应该先查看 `skills/langbot-testing/references/agent-runner-release-gate.md`， +再执行： + +```bash +bin/lbs test recommend +bin/lbs suite plan agent-runner-release-gate +bin/lbs test run agent-runner-release-preflight --dry-run +bin/lbs suite start agent-runner-release-gate --run-id agent-runner-release-local --evidence-dir reports/evidence/agent-runner-release-local +``` + +`test recommend` 输出的 run 命令默认带 `--dry-run`；确认 readiness 和 `manual_check` 前置条件后，再去掉 `--dry-run` 执行。 + +完成所有 case 后，用： + +```bash +bin/lbs suite report agent-runner-release-gate --evidence-dir reports/evidence/agent-runner-release-local +``` + +没有最终 `result.json`、缺 required evidence、或把 `blocked`/`env_issue` 当成 pass， +都不能算发布门禁通过。 + +### 新 Feature 测试 + +你可以说： + +```text +我改了 provider 页面，帮我测一下 DeepSeek provider 添加、测试、绑定 pipeline 是否正常。 +``` + +agent 应该： + +- 查找相关 case 和 reference +- 如果没有稳定路径，先探索 UI +- 用浏览器执行真实交互 +- 失败时用日志/API 辅助定位 +- 稳定后新增或更新 case/reference +- 新故障沉淀为 troubleshooting + +### 定点排错 + +你可以说： + +```text +Debug Chat 点了没回复，帮我查是前端问题还是后端问题。 +``` + +agent 应该： + +- 先通过 UI 复现 +- 看 console/network +- 看后端日志 +- 必要时用 API/curl 做诊断 +- 匹配 troubleshooting +- 给出修复建议或直接修复 + +## 重要原则 + +这些原则固定在： + +```text +docs/qa-agent/03-agent-browser-qa-principles.md +``` + +简化版： + +- UI/browser 是测试主路径。 +- API/curl/log 只做诊断。 +- 后端接口成功不等于 UI case 通过。 +- case 通过必须以用户可见 UI 结果为准。 +- 有视觉能力时应检查截图。 +- 没有视觉能力时用 DOM/accessibility snapshot 和 console。 +- 不要打印 token、API key、OAuth secret 或 localStorage token 值。 + +## 规划文档 + +如果要判断下一步建设什么，先看： + +```text +docs/qa-agent/README.md +docs/qa-agent/04-black-box-e2e-roadmap.md +``` + +`01-qa-agent-harness-plan.md` 是早期规划，部分内容已经被当前实现和路线图替代。 + +## 常用命令 + +### 环境 + +```bash +bin/lbs env show +bin/lbs env show --json +bin/lbs env doctor +bin/lbs fixture list +bin/lbs fixture check +bin/lbs fixture check --json +``` + +`env show` 和 `env doctor` 默认会对 token、API key、password、secret 以及 URL basic auth +做脱敏。不要把 `.env.local` 里的原始凭据复制进测试报告。 + +### Skill 和索引 + +```bash +bin/lbs list +bin/lbs validate +bin/lbs index --check +bin/lbs index +``` + +### Case + +```bash +bin/lbs case list +bin/lbs case list --type smoke +bin/lbs case list --json --priority p1 --tag local-agent +bin/lbs case list --ready +bin/lbs case list --machine-ready +bin/lbs case show pipeline-debug-chat +bin/lbs case new my-feature --title "My Feature Works" +``` + +### Suite + +```bash +bin/lbs suite list +bin/lbs suite list --json --priority p1 +bin/lbs suite show local-agent-gate +bin/lbs suite plan core-smoke +bin/lbs suite plan local-agent-gate --json +bin/lbs suite start core-smoke +bin/lbs suite start core-smoke --run-id core-smoke-local --evidence-dir reports/evidence/core-smoke-local +bin/lbs suite run core-smoke --dry-run --json +bin/lbs suite run core-smoke --run-id core-smoke-local --evidence-dir reports/evidence/core-smoke-local +bin/lbs suite start core-smoke --json +bin/lbs suite report core-smoke --evidence-dir reports/evidence/ +bin/lbs suite report core-smoke --evidence-dir reports/evidence/ --json +bin/lbs suite new my-feature-gate --title "My Feature Gate" +``` + +`suite start` 不直接控制浏览器。它生成统一 run id、suite evidence root、每个 case 的 evidence +目录、`suite-start.json`/`suite-start.md` handoff 文件，以及每个 case 的 `test run`、`test report` +和 `test result` 命令模板。需要固定路径时，使用 `--run-id` 和 `--evidence-dir`。 +`suite run --dry-run --json` 只预览 planned/skipped case，不创建 evidence，也不执行 automation。 +`suite run` 会顺序执行 suite 中已有 automation、机器 readiness 已满足且不需要 `manual_check` 的 case，并在最后聚合 `suite report`。 +缺 env、automation env 或 fixture 的 case 默认会跳过；确实要强制执行时，加 `--include-not-ready`。 +确认前置条件后，才用 `--include-manual-check` 执行这类 case。 +所有 case 执行完并写入最终 `result.json` 后， +`suite report` 会读取各 case evidence 目录并汇总为 `pass`、`fail`、`blocked`、`env_issue`、 +`flaky`、`incomplete` 等状态。`pass` 必须声明已经收集 case 的全部 required evidence； +否则 suite 会保持 `incomplete`，避免把缺证据的运行误判成通过。 + +### Test Plan + +```bash +bin/lbs test plan pipeline-debug-chat +bin/lbs test plan pipeline-debug-chat --json +``` + +### Test Start + +```bash +bin/lbs test start pipeline-debug-chat +bin/lbs test start pipeline-debug-chat --json +``` + +`test start` 用于 agent 开始一次浏览器测试前记录 run id、开始时间和推荐 report 命令。 +它会把 `--since ""` 写进后续报告命令，减少历史日志污染本次判断。 +如果 case 绑定了自动化脚本，输出里也会包含 `test run` 命令和 evidence 目录。 + +### Test Automation + +```bash +bin/lbs test run webui-login-state --dry-run +bin/lbs test run pipeline-debug-chat --dry-run +bin/lbs test run webui-login-state --run-id login-smoke --output reports/evidence/login-smoke +bin/lbs test run pipeline-debug-chat --run-id pipeline-smoke --output reports/evidence/pipeline-smoke +``` + +查看当前所有带自动化脚本的 case： + +```bash +bin/lbs case list --automation +bin/lbs case list --json --automation +``` + +当前自动化覆盖包括登录态、通用 Pipeline Debug Chat、local-agent runner 的基础回复、 +PromptPreProcessing、RAG、plugin tool、MCP stdio tool、非流式、多模态和 RAG+多模态路径。 +不要在文档里手工维护静态 case 清单；以 `case list --automation` 和 suite 定义为准。 + +自动化脚本位于 `scripts/e2e/`。它们会保存： + +- `console.log` +- `network.log` +- `screenshot.png` +- `automation-result.json` + +新增 Debug Chat 类自动化时，优先复用 `scripts/e2e/lib/debug-chat.mjs` 中的 pipeline 打开、 +prompt 发送、visible response leaf 判断和失败信号分类，不要在新脚本里复制 DOM 扫描逻辑。 + +脚本需要本地安装 Playwright 后才能真正执行： + +```bash +npm install +npx playwright install chromium +``` + +`pipeline-debug-chat` 通用自动化建议配置 `LANGBOT_PIPELINE_URL`。如果没有 direct URL， +脚本会尝试通过 `LANGBOT_PIPELINE_NAME` 从 Pipelines 页面进入目标 pipeline。两者都没有时， +该自动化会返回 `blocked`，不会伪造通过。 + +Runner 专用 case 不应复用通用 pipeline 变量。Local Agent、Codex AgentRunner 和 +Claude Code AgentRunner 这类 case 会通过 `automation_pipeline_url_env` / +`automation_pipeline_name_env` 映射到 case-specific env，例如 +`LANGBOT_LOCAL_AGENT_PIPELINE_URL`。这些 case 如果缺少专用变量，会返回 `blocked`， +不会退回到 `LANGBOT_PIPELINE_URL`，避免跑错 pipeline 后产生假阳性。 +如果 case 声明了 `setup_automation`，只有 `bin/lbs test run ` 的真实执行路径会先运行这些 setup； +`test plan`、`suite plan`、`case list` 和 `--dry-run` 只展示它们，不会修改本地环境。 +setup 可以是 `case:` 或仓库内 `node:scripts/... --flag`，每一步证据会写到主 evidence 目录下的 +`setup/` 子目录。setup 失败时主 automation 不会继续执行；setup 写入 `.env.local` 后，主 automation +会重新读取环境。用 `setup_provides_env` 声明 setup 会生成的变量，可以让 readiness 正确显示机器可准备状态。 +如果 Debug Chat case 需要固定流式/非流式路径，可以在 case 中设置 +`automation_stream_output: "1"` 或 `"0"`，脚本会在发送消息前切换 Debug Chat 的 Stream 控件。 +如果 case 需要上传图片，可以设置 `automation_image_base64_fixture` 指向仓库内的 base64 PNG fixture， +脚本会在 evidence 目录写出临时 PNG 并通过 Debug Chat 上传控件发送。 +`bin/lbs test plan --json` 和 `bin/lbs suite plan --json` +都会显示这些专用变量是否已配置。 + +### Test Report 和日志守卫 + +```bash +bin/lbs test report pipeline-debug-chat +bin/lbs test report pipeline-debug-chat --output reports/pipeline-debug-chat.md +bin/lbs test report pipeline-debug-chat \ + --backend-log /path/to/backend.log \ + --frontend-log /path/to/frontend.log \ + --console-log /path/to/console.log +bin/lbs test report pipeline-debug-chat --evidence-dir reports/evidence/pipeline-smoke +bin/lbs test report pipeline-debug-chat --backend-log /path/to/backend.log --json +bin/lbs test report pipeline-debug-chat --since "2026-05-21T10:30:00+08:00" +bin/lbs test report pipeline-debug-chat --tail-lines 2000 +bin/lbs test report pipeline-debug-chat --since "2026-05-21T10:30:00+08:00" --tail-lines 2000 +``` + +`test report` 会生成报告模板，并默认从 `LANGBOT_REPO/data/logs/` 自动选择最新的 +`langbot-*.log` 作为 LangBot 后端日志扫描。也可以用 `--backend-log` 覆盖，或用 +`--no-auto-log` 只生成模板。 + +如果提供 `--evidence-dir`，或 `--console-log` 指向 `reports/evidence//console.log`， +报告会优先读取同目录的 `automation-result.json`，并展示自动化脚本的 `status`、`reason`、 +起止时间和目标 URL。 + +日志守卫会扫描常见错误、secret 泄露风险、case 声明的 success/failure patterns，以及已知 +troubleshooting pattern。它不控制浏览器，也不替代 UI 通过判断。`success_patterns` +命中会作为通过证据写入 `success_signals`；声明了 success pattern 但本次扫描窗口没有命中， +会给 warning；`failure_patterns` 命中会让本次日志守卫 fail。 + +建议在执行浏览器 case 前记录当前时间，然后在报告阶段使用 `--since`。如果只想快速看 +最近日志，可以使用 `--tail-lines`。 + +### Runtime Log Guard + +如果还没有进入某个具体 UI case，只是想观察 LangBot 后端日志，可以直接使用 `log` +命令。它和 `test report` 使用同一套扫描器、secret 脱敏、troubleshooting pattern 和 +case success/failure pattern。 + +```bash +bin/lbs log scan --tail-lines 300 +bin/lbs log scan --case pipeline-debug-chat --since "2026-05-21T10:30:00+08:00" +bin/lbs log scan --backend-log /path/to/langbot.log --json +bin/lbs log scan --failure-pattern "runner.tool_loop_error|Action invoke_llm_stream call timed out" --strict +``` + +`log scan` 默认从 `LANGBOT_REPO/data/logs/` 自动选择最新的 `langbot-*.log`。传入 +`--case ` 后，会额外应用该 case 声明的 `success_patterns`、`failure_patterns` +和 related troubleshooting。默认用于观察，返回码保持 0；加 `--strict` 后，`fail` 或 +`env_issue` 会返回非 0，适合脚本门禁。 + +运行期观察可以用 `watch`： + +```bash +bin/lbs log watch --case pipeline-debug-chat +bin/lbs log watch --backend-log /path/to/langbot.log --interval-ms 500 +bin/lbs log watch --duration-ms 30000 --strict --json +``` + +`log watch` 默认从启动时的文件末尾开始，只观察新追加的日志；加 `--from-start` 可从文件开头扫。 +它会实时打印新命中的 findings 和 success signals。为了避免当前历史日志噪声影响观察，默认不因 +异常返回非 0；加 `--strict` 后，退出时如果看到 `fail` 或 `env_issue` 会返回非 0。 + +给一次 QA 运行包日志窗口时，用 `guard start/stop`： + +```bash +bin/lbs log guard start --run-id local-debug --case pipeline-debug-chat +# 执行浏览器或手工测试 +bin/lbs log guard stop --run-id local-debug +``` + +`start` 会在 `reports/log-guards/.json` 记录起始时间、case 和当前后端日志路径； +`stop` 会用 start/stop 时间作为扫描窗口，生成 `reports/log-guards/.md`，并默认按 +strict guard 返回码处理。临时只想收集报告、不想让命令失败，可以加 `--no-strict`。 + +当前 LangBot core 日志还不是完全结构化日志，runtime guard 主要依赖时间窗口和文本 pattern。 +已支持 ISO 时间戳和 LangBot 当前的 `[MM-DD HH:mm:ss.SSS]` 前缀；没有时间戳的连续行会跟随上一条 +带时间戳的日志块。如果后续 core 能提供稳定 request id、conversation id、plugin action id 或 +JSON log field，guard 可以从“时间窗口 + 文本匹配”升级为更精确的关联分析。 + +### Test Result + +```bash +bin/lbs test result pipeline-debug-chat \ + --result pass \ + --reason "Debug Chat returned OK and the report log guard was clean." \ + --evidence-dir reports/evidence/pipeline-smoke \ + --started-at "2026-05-21T10:30:00+08:00" \ + --evidence ui,screenshot,console,backend_log +``` + +`test result` 用于把一次人工/agent browser 运行的最终判断写成标准 `result.json`， +供 `suite report` 聚合。它不会替代 UI 测试：如果写 `--result pass`，`--evidence` +必须覆盖该 case 的 `evidence_required`，否则命令会失败。自动化脚本写 +`automation-result.json`；如果 case 还要求 backend log、API diagnostic 或 filesystem +evidence，agent 需要在报告和诊断完成后再用 `test result` 写最终 `result.json`。 + +### Troubleshooting + +```bash +bin/lbs trouble list langbot-testing +bin/lbs trouble show plugin-runtime-timeout +bin/lbs trouble search runtime +bin/lbs trouble add langbot-testing --title "..." --symptom "..." --cause "..." --fix "..." +``` + +## 目录说明 + +```text +skills/ + .env # 共享默认变量 + langbot-env-setup/ # 环境、浏览器、登录态、代理 + langbot-testing/ # WebUI / provider / pipeline 测试 +schemas/ # 结构化资产 schema +src/ # lbs TypeScript 源码 +bin/ # lbs 入口 +docs/ # 设计文档和用户手册 +AGENTS.md # agent 维护协议 +``` + +## 添加一个新测试路径 + +1. 先让 agent 通过浏览器探索并执行路径。 +2. 稳定后创建 case： + +```bash +bin/lbs case new provider-xxx --title "Provider XXX can be configured" --area provider --type provider +``` + +3. 编辑生成的 `cases/*.yaml`，补充真实步骤、检查点和 troubleshooting。 + +4. 校验： + +```bash +bin/lbs validate +bin/lbs index --check +bin/lbs index +``` + +## 添加一个新故障 + +```bash +bin/lbs trouble add langbot-testing \ + --title "Plugin runtime actions time out" \ + --symptom "Debug Chat shows Agent runner temporarily unavailable" \ + --cause "Old plugin runtime survived backend restart" \ + --fix "Stop old runtime processes and restart LangBot" +``` + +然后编辑生成的 YAML，补充 `patterns`、`related_cases` 和验证方式。 + +## 当前边界 + +- `lbs test plan` 只生成测试计划，不直接控制浏览器。 +- `lbs test report` 生成报告，默认扫描最新 LangBot 后端日志；也可扫描显式提供的 + backend/frontend/console 日志文件。 +- 真正的 UI 操作由当前 agent 的浏览器能力执行。 +- `env doctor` 是 readiness check，不是产品测试。 +- `curl/API` 是诊断工具，不是主要测试路径。 diff --git a/skills/schemas/README.md b/skills/schemas/README.md new file mode 100644 index 000000000..0b06d9edd --- /dev/null +++ b/skills/schemas/README.md @@ -0,0 +1,59 @@ +# Schemas + +这个目录存放 LangBot skills 结构化资产的 JSON Schema。 + +它们不是测试脚本，也不会执行浏览器动作。它们的作用是定义 agent 和维护者后续新增资产时应该遵守的文件结构。 + +## 文件说明 + +- `skills//fixtures/fixtures.json` + 不是 JSON Schema，但由 `bin/lbs validate` 校验。 + 它登记 deterministic fixture 文件、类型和关联 case，供 `bin/lbs fixture check` 做 readiness 检查。 + +- `case.schema.json` + 约束 `skills//cases/*.yaml` 的格式。 + Case 描述 agent-browser 或 probe QA 路径，包括前置条件、步骤、检查点、诊断手段和关联故障。 + +- `suite.schema.json` + 约束 `skills//suites/*.yaml` 的格式。 + Suite 只组织 case 集合，用于 smoke、regression 或 release gate 等测试入口。 + +- `troubleshooting.schema.json` + 约束 `skills//troubleshooting/*.yaml` 的格式。 + Troubleshooting 条目描述症状、日志/错误模式、可能原因、修复步骤和验证信号。 + +- `skill-index.schema.json` + 约束生成文件 `skills.index.json` 的格式。 + 这个索引用于让 agent 快速发现已有 skills、references、cases、suites 和 troubleshooting。 + +- `reports/evidence//result.json` + 不是 catalog schema，而是执行期最终裁定产物，由 `bin/lbs test result` 写入。 + `suite report` 读取其中的 `status`、`reason`、起止时间和 `evidence_collected`， + 并用 `evidence_missing` 防止缺证据的 `pass` 被当作完整通过。 + +- `reports/evidence//automation-result.json` + 不是 catalog schema，而是浏览器自动化脚本的原始运行结论，供 `bin/lbs test report` + 展示和推断日志扫描窗口。 + +## 为什么需要 schemas + +Schemas 是基础设施护栏： + +- 防止 case、suite 和 troubleshooting 随着增长变得格式混乱 +- 让 `bin/lbs validate` 能发现缺字段和错误结构 +- 为未来编辑器提示和 CI 校验留接口 +- 帮助 agent 新增资产时知道应该写哪些字段 + +## 当前校验方式 + +`bin/lbs validate` 做轻量、schema 对齐的校验，不引入额外依赖。它会检查必填字段、 +枚举值、boolean 字段、重复列表项、automation 脚本存在性，以及 case、suite、skill、 +troubleshooting 之间的交叉引用。这里的 schema 仍是格式契约；如果未来引入正式 JSON +Schema validator，应继续保持这些本地交叉引用检查。 + +Case 里的 `env` / `automation_env` 表示所有列出的变量都需要配置。遇到二选一输入时， +使用 `env_any` / `automation_env_any`，每一项写成 `LANGBOT_PIPELINE_URL|LANGBOT_PIPELINE_NAME` +这类 one-of 组合，避免 agent 因为只配置了 URL 或 name 其中之一而误判未就绪。 +`setup` 和 `preconditions` 是人工确认项，会让 readiness 进入 `manual_check`； +`setup_automation` 是 `test run` 可以自动执行的准备步骤，配合 `setup_provides_env` +声明它会生成的机器变量。 diff --git a/skills/schemas/case.schema.json b/skills/schemas/case.schema.json new file mode 100644 index 000000000..f6365c062 --- /dev/null +++ b/skills/schemas/case.schema.json @@ -0,0 +1,219 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://langbot.app/schemas/langbot-skills/case.schema.json", + "title": "LangBot Skill Test Case", + "type": "object", + "required": [ + "id", + "title", + "mode", + "area", + "type", + "priority", + "risk", + "ci_eligible", + "tags", + "skills", + "steps", + "checks", + "evidence_required" + ], + "allOf": [ + { + "if": { + "properties": { + "mode": { "const": "agent-browser" } + } + }, + "then": { + "required": ["env"] + } + } + ], + "additionalProperties": true, + "properties": { + "id": { + "type": "string", + "pattern": "^[a-z0-9][a-z0-9_-]*$" + }, + "title": { + "type": "string" + }, + "mode": { + "type": "string", + "enum": ["agent-browser", "probe"] + }, + "area": { + "type": "string" + }, + "type": { + "type": "string", + "enum": ["smoke", "regression", "feature", "provider", "exploratory"] + }, + "priority": { + "type": "string", + "enum": ["p0", "p1", "p2"] + }, + "risk": { + "type": "string", + "enum": ["low", "medium", "high"] + }, + "ci_eligible": { + "type": "boolean" + }, + "tags": { + "type": "array", + "items": { "type": "string" } + }, + "skills": { + "type": "array", + "items": { "type": "string" } + }, + "env": { + "type": "array", + "items": { "type": "string" } + }, + "env_any": { + "type": "array", + "items": { + "type": "string", + "pattern": "^[A-Z][A-Z0-9_]*(\\|[A-Z][A-Z0-9_]*)+$" + } + }, + "steps": { + "type": "array", + "items": { "type": "string" }, + "minItems": 1 + }, + "checks": { + "type": "array", + "items": { "type": "string" }, + "minItems": 1 + }, + "evidence_required": { + "type": "array", + "items": { + "type": "string", + "enum": [ + "ui", + "screenshot", + "console", + "network", + "backend_log", + "frontend_log", + "api_diagnostic", + "filesystem" + ] + }, + "minItems": 1 + }, + "preconditions": { + "type": "array", + "items": { "type": "string" } + }, + "setup": { + "type": "array", + "items": { "type": "string" } + }, + "setup_automation": { + "type": "array", + "items": { + "type": "string", + "pattern": "^(?:case:[a-z0-9][a-z0-9_-]*|node:scripts/[A-Za-z0-9_./-]+\\.(?:mjs|js|ts)(?:\\s+--[A-Za-z0-9][A-Za-z0-9_-]*(?:=[A-Za-z0-9_./:@-]+)?)*)$" + } + }, + "setup_provides_env": { + "type": "array", + "items": { + "type": "string", + "pattern": "^[A-Z][A-Z0-9_]*$" + } + }, + "cleanup": { + "type": "array", + "items": { "type": "string" } + }, + "diagnostics": { + "type": "array", + "items": { "type": "string" } + }, + "automation": { + "type": "string" + }, + "automation_env": { + "type": "array", + "items": { "type": "string" } + }, + "automation_env_any": { + "type": "array", + "items": { + "type": "string", + "pattern": "^[A-Z][A-Z0-9_]*(\\|[A-Z][A-Z0-9_]*)+$" + } + }, + "automation_prompt": { + "type": "string" + }, + "automation_prompts_json": { + "type": "string" + }, + "automation_expected_text": { + "type": "string" + }, + "automation_response_timeout_ms": { + "type": "string" + }, + "automation_stream_output": { + "type": "string", + "enum": ["0", "1", "false", "true"] + }, + "automation_image_base64_fixture": { + "type": "string" + }, + "automation_runner_config_patch_json": { + "type": "string" + }, + "automation_restore_runner_config": { + "type": "string", + "enum": ["0", "1", "false", "true"] + }, + "automation_expected_runner_id": { + "type": "string" + }, + "automation_reset_debug_chat": { + "type": "string", + "enum": ["0", "1", "false", "true"] + }, + "automation_debug_chat_session_type": { + "type": "string", + "enum": ["person", "group"] + }, + "automation_filesystem_checks_json": { + "type": "string" + }, + "automation_pipeline_url_env": { + "type": "string", + "pattern": "^[A-Z][A-Z0-9_]*$" + }, + "automation_pipeline_name_env": { + "type": "string", + "pattern": "^[A-Z][A-Z0-9_]*$" + }, + "success_patterns": { + "type": "array", + "items": { "type": "string" } + }, + "failure_patterns": { + "type": "array", + "items": { "type": "string" } + }, + "expected_failures": { + "type": "array", + "items": { "type": "string" } + }, + "troubleshooting": { + "type": "array", + "items": { "type": "string" } + } + } +} diff --git a/skills/schemas/skill-index.schema.json b/skills/schemas/skill-index.schema.json new file mode 100644 index 000000000..1a080c55e --- /dev/null +++ b/skills/schemas/skill-index.schema.json @@ -0,0 +1,147 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://langbot.app/schemas/langbot-skills/skill-index.schema.json", + "title": "LangBot Skills Index", + "type": "object", + "required": ["generated_by", "skills"], + "additionalProperties": false, + "properties": { + "generated_by": { + "type": "string" + }, + "skills": { + "type": "array", + "items": { + "type": "object", + "required": [ + "directory", + "name", + "description", + "references", + "cases", + "case_summaries", + "suites", + "suite_summaries", + "fixtures", + "troubleshooting", + "troubleshooting_summaries" + ], + "additionalProperties": false, + "properties": { + "directory": { "type": "string" }, + "name": { "type": "string" }, + "description": { "type": "string" }, + "references": { + "type": "array", + "items": { "type": "string" } + }, + "cases": { + "type": "array", + "items": { "type": "string" } + }, + "case_summaries": { + "type": "array", + "items": { + "type": "object", + "required": ["id", "title", "mode", "area", "type", "priority", "risk", "ci_eligible", "tags", "automation", "setup_automation", "setup_provides_env", "evidence_required"], + "additionalProperties": false, + "properties": { + "id": { "type": "string" }, + "title": { "type": "string" }, + "mode": { "type": "string", "enum": ["agent-browser", "probe"] }, + "area": { "type": "string" }, + "type": { "type": "string" }, + "priority": { "type": "string" }, + "risk": { "type": "string" }, + "ci_eligible": { "type": "boolean" }, + "tags": { + "type": "array", + "items": { "type": "string" } + }, + "automation": { "type": "string" }, + "setup_automation": { + "type": "array", + "items": { "type": "string" } + }, + "setup_provides_env": { + "type": "array", + "items": { "type": "string" } + }, + "evidence_required": { + "type": "array", + "items": { "type": "string" } + } + } + } + }, + "suites": { + "type": "array", + "items": { "type": "string" } + }, + "suite_summaries": { + "type": "array", + "items": { + "type": "object", + "required": ["id", "title", "description", "type", "priority", "tags", "cases"], + "additionalProperties": false, + "properties": { + "id": { "type": "string" }, + "title": { "type": "string" }, + "description": { "type": "string" }, + "type": { "type": "string" }, + "priority": { "type": "string" }, + "tags": { + "type": "array", + "items": { "type": "string" } + }, + "cases": { + "type": "array", + "items": { "type": "string" } + } + } + } + }, + "fixtures": { + "type": "array", + "items": { + "type": "object", + "required": ["id", "title", "kind", "path", "related_cases"], + "additionalProperties": false, + "properties": { + "id": { "type": "string" }, + "title": { "type": "string" }, + "kind": { "type": "string" }, + "path": { "type": "string" }, + "related_cases": { + "type": "array", + "items": { "type": "string" } + } + } + } + }, + "troubleshooting": { + "type": "array", + "items": { "type": "string" } + }, + "troubleshooting_summaries": { + "type": "array", + "items": { + "type": "object", + "required": ["id", "title", "category", "related_cases"], + "additionalProperties": false, + "properties": { + "id": { "type": "string" }, + "title": { "type": "string" }, + "category": { "type": "string" }, + "related_cases": { + "type": "array", + "items": { "type": "string" } + } + } + } + } + } + } + } + } +} diff --git a/skills/schemas/suite.schema.json b/skills/schemas/suite.schema.json new file mode 100644 index 000000000..3da1a3e85 --- /dev/null +++ b/skills/schemas/suite.schema.json @@ -0,0 +1,38 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://langbot.app/schemas/langbot-skills/suite.schema.json", + "title": "LangBot Skill Test Suite", + "type": "object", + "required": ["id", "title", "description", "type", "priority", "tags", "cases"], + "additionalProperties": true, + "properties": { + "id": { + "type": "string", + "pattern": "^[a-z0-9][a-z0-9_-]*$" + }, + "title": { + "type": "string" + }, + "description": { + "type": "string" + }, + "type": { + "type": "string", + "enum": ["smoke", "regression", "release_gate", "exploratory"] + }, + "priority": { + "type": "string", + "enum": ["p0", "p1", "p2"] + }, + "tags": { + "type": "array", + "items": { "type": "string" }, + "minItems": 1 + }, + "cases": { + "type": "array", + "items": { "type": "string" }, + "minItems": 1 + } + } +} diff --git a/skills/schemas/troubleshooting.schema.json b/skills/schemas/troubleshooting.schema.json new file mode 100644 index 000000000..1005f5e04 --- /dev/null +++ b/skills/schemas/troubleshooting.schema.json @@ -0,0 +1,51 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://langbot.app/schemas/langbot-skills/troubleshooting.schema.json", + "title": "LangBot Skill Troubleshooting Entry", + "type": "object", + "required": ["id", "title", "symptoms", "patterns", "likely_causes", "fix_steps", "verification"], + "additionalProperties": true, + "properties": { + "id": { + "type": "string", + "pattern": "^[a-z0-9][a-z0-9_-]*$" + }, + "title": { + "type": "string" + }, + "date": { + "type": "string" + }, + "category": { + "type": "string", + "enum": ["product", "env_issue", "external_dependency", "blocked", "flaky"] + }, + "symptoms": { + "type": "array", + "items": { "type": "string" }, + "minItems": 1 + }, + "patterns": { + "type": "array", + "items": { "type": "string" }, + "minItems": 1 + }, + "likely_causes": { + "type": "array", + "items": { "type": "string" }, + "minItems": 1 + }, + "fix_steps": { + "type": "array", + "items": { "type": "string" }, + "minItems": 1 + }, + "verification": { + "type": "string" + }, + "related_cases": { + "type": "array", + "items": { "type": "string" } + } + } +} diff --git a/skills/scripts/e2e/agent-runner-release-preflight.mjs b/skills/scripts/e2e/agent-runner-release-preflight.mjs new file mode 100644 index 000000000..6ac69f2f1 --- /dev/null +++ b/skills/scripts/e2e/agent-runner-release-preflight.mjs @@ -0,0 +1,476 @@ +#!/usr/bin/env node + +import { existsSync, readFileSync } from "node:fs"; +import { writeFile } from "node:fs/promises"; +import { resolve } from "node:path"; +import { env } from "node:process"; +import { + createBrowser, + ensureEvidence, + evidencePaths, + exitCode, + localIsoWithOffset, + safeScreenshot, + writeResult, +} from "./lib/langbot-e2e.mjs"; + +function loadEnvDefaults(path) { + if (!existsSync(path)) return; + for (const rawLine of readFileSync(path, "utf8").split(/\r?\n/)) { + const line = rawLine.trim(); + if (!line || line.startsWith("#")) continue; + const sep = line.indexOf("="); + if (sep === -1) continue; + const key = line.slice(0, sep).trim(); + if (env[key]) continue; + env[key] = line.slice(sep + 1).trim().replace(/^["']|["']$/g, ""); + } +} + +function boolFromEnv(value, defaultValue) { + if (value === undefined || value === "") return defaultValue; + if (/^(0|false|no|off)$/i.test(value)) return false; + if (/^(1|true|yes|on)$/i.test(value)) return true; + return defaultValue; +} + +function firstEnv(...keys) { + for (const key of keys) { + if (env[key]) return env[key]; + } + return ""; +} + +function redactMessage(text) { + return String(text ?? "") + .replace(/\bbearer\s+[A-Za-z0-9._~+/=-]{8,}/gi, "Bearer [redacted]") + .replace(/\bsk-[A-Za-z0-9_-]{6,}\b/g, "[redacted]") + .replace(/(api[_-]?key|authorization|credential|jwt|oauth|password|secret|token)\s*[:=]\s*["']?[^"',\s]+/gi, "$1=[redacted]"); +} + +function isEnvironmentError(message) { + return /Playwright is not installed|LANGBOT_FRONTEND_URL|LANGBOT_BACKEND_URL|ERR_CONNECTION_REFUSED|ECONNREFUSED|net::ERR_|fetch failed|timed out/i + .test(message); +} + +loadEnvDefaults("skills/.env"); +loadEnvDefaults("skills/.env.local"); + +const caseId = env.LBS_CASE_ID || "agent-runner-release-preflight"; +const paths = evidencePaths(caseId); +await ensureEvidence(paths); + +const backendUrl = env.LANGBOT_BACKEND_URL || ""; +const frontendUrl = env.LANGBOT_FRONTEND_URL || backendUrl; +const testModels = boolFromEnv(env.LANGBOT_PREFLIGHT_TEST_MODELS, true); +const requireVision = boolFromEnv(env.LANGBOT_PREFLIGHT_REQUIRE_VISION, true); +const diagnosticPath = resolve(paths.evidenceDir, "api-diagnostic.json"); +const startedAt = new Date(); + +const targets = [ + { + id: "local-agent", + expected_runner_id: "plugin:langbot/local-agent/default", + pipeline_url: firstEnv("LANGBOT_LOCAL_AGENT_PIPELINE_URL"), + pipeline_name: firstEnv("LANGBOT_LOCAL_AGENT_PIPELINE_NAME"), + require_func_call_model: true, + require_vision_model: requireVision, + require_langbot_mcp: false, + }, + { + id: "acp-agent-runner", + expected_runner_id: "plugin:langbot/acp-agent-runner/default", + pipeline_url: firstEnv("LANGBOT_ACP_AGENT_RUNNER_PIPELINE_URL", "LANGBOT_AGENT_RUNNER_PIPELINE_URL"), + pipeline_name: firstEnv("LANGBOT_ACP_AGENT_RUNNER_PIPELINE_NAME", "LANGBOT_AGENT_RUNNER_PIPELINE_NAME"), + require_func_call_model: false, + require_vision_model: false, + }, +]; + +let browser; +const result = { + source: "automation", + case_id: caseId, + run_id: paths.runId, + started_at: startedAt.toISOString(), + started_at_local: localIsoWithOffset(startedAt), + finished_at: "", + finished_at_local: "", + status: "fail", + reason: "", + frontend_url: frontendUrl, + backend_url: backendUrl, + test_models: testModels, + require_vision_model: requireVision, + evidence: { + console_log: paths.consoleLog, + network_log: paths.networkLog, + screenshot: paths.screenshot, + api_diagnostic_json: diagnosticPath, + automation_result_json: paths.automationResultJson, + result_json: paths.resultJson, + }, + evidence_collected: ["ui", "screenshot", "console", "network", "api_diagnostic"], +}; + +async function run() { + if (!backendUrl || !frontendUrl) { + result.status = "env_issue"; + result.reason = "LANGBOT_FRONTEND_URL and LANGBOT_BACKEND_URL must be configured."; + return; + } + + browser = await createBrowser(paths); + const { page } = browser; + await page.goto(frontendUrl, { waitUntil: "domcontentloaded" }); + await page.waitForLoadState("networkidle", { timeout: 10_000 }).catch(() => {}); + + const diagnostic = await page.evaluate(async ({ backendUrl, targets, testModels }) => { + const blockers = []; + const envIssues = []; + const warnings = []; + const checks = []; + + const addCheck = (name, status, detail = {}) => { + checks.push({ name, status, ...detail }); + if (status === "blocked") blockers.push({ name, ...detail }); + if (status === "env_issue") envIssues.push({ name, ...detail }); + }; + const safeMessage = (value) => String(value ?? "") + .replace(/\bbearer\s+[A-Za-z0-9._~+/=-]{8,}/gi, "Bearer [redacted]") + .replace(/\bsk-[A-Za-z0-9_-]{6,}\b/g, "[redacted]") + .replace(/(api[_-]?key|authorization|credential|jwt|oauth|password|secret|token)\s*[:=]\s*["']?[^"',\s]+/gi, "$1=[redacted]"); + + const token = localStorage.getItem("token"); + if (!token) { + addCheck("browser-auth", "blocked", { reason: "Browser profile has no localStorage token." }); + return { authenticated: false, blockers, env_issues: envIssues, warnings, checks }; + } + + const headers = { + Authorization: `Bearer ${token}`, + "Content-Type": "application/json", + }; + const getJson = async (path) => { + const response = await fetch(`${backendUrl}${path}`, { headers }); + return { + status: response.status, + json: await response.json().catch(() => ({})), + }; + }; + const postJson = async (path, body) => { + const response = await fetch(`${backendUrl}${path}`, { + method: "POST", + headers, + body: JSON.stringify(body), + }); + return { + status: response.status, + json: await response.json().catch(() => ({})), + }; + }; + + const tokenCheck = await getJson("/api/v1/user/check-token"); + addCheck( + "browser-auth", + tokenCheck.status < 400 && (tokenCheck.json.code ?? 0) === 0 ? "pass" : "blocked", + { http_status: tokenCheck.status, code: tokenCheck.json.code ?? null, reason: safeMessage(tokenCheck.json.msg || "") }, + ); + + const systemInfo = await getJson("/api/v1/system/info"); + addCheck( + "backend-system-info", + systemInfo.status < 400 ? "pass" : "env_issue", + { + http_status: systemInfo.status, + version: systemInfo.json.data?.version || systemInfo.json.data?.system?.version || "", + }, + ); + + const pluginSystem = await getJson("/api/v1/system/status/plugin-system"); + addCheck( + "plugin-system", + pluginSystem.status < 400 && (pluginSystem.json.code ?? 0) === 0 ? "pass" : "env_issue", + { + http_status: pluginSystem.status, + code: pluginSystem.json.code ?? null, + status: pluginSystem.json.data?.status || pluginSystem.json.data?.state || "", + reason: safeMessage(pluginSystem.json.msg || ""), + }, + ); + + const boxStatus = await getJson("/api/v1/box/status"); + addCheck( + "box-runtime", + boxStatus.status < 400 && (boxStatus.json.code ?? 0) === 0 ? "pass" : "env_issue", + { + http_status: boxStatus.status, + code: boxStatus.json.code ?? null, + status: boxStatus.json.data?.status || "", + backend: boxStatus.json.data?.backend || "", + reason: safeMessage(boxStatus.json.msg || ""), + }, + ); + + const plugins = await getJson("/api/v1/plugins"); + const installedPluginIds = (plugins.json.data?.plugins || []) + .map((plugin) => { + const metadata = plugin.manifest?.manifest?.metadata || plugin.manifest?.metadata || plugin.metadata || {}; + return metadata.author && metadata.name ? `${metadata.author}/${metadata.name}` : ""; + }) + .filter(Boolean); + const requiredPlugins = ["langbot/local-agent", "langbot/acp-agent-runner", "qa/plugin-smoke"]; + const pluginPresence = Object.fromEntries(requiredPlugins.map((id) => [id, installedPluginIds.includes(id)])); + for (const [id, present] of Object.entries(pluginPresence)) { + addCheck(`plugin:${id}`, present ? "pass" : "blocked", { plugin_id: id, reason: present ? "" : "Required plugin is not listed by /api/v1/plugins." }); + } + + const tools = await getJson("/api/v1/tools"); + const toolNames = (tools.json.data?.tools || []) + .map((tool) => tool.name || tool.tool_name || tool.function?.name || "") + .filter(Boolean) + .sort(); + addCheck( + "tool:qa_plugin_echo", + toolNames.includes("qa_plugin_echo") ? "pass" : "blocked", + { reason: toolNames.includes("qa_plugin_echo") ? "" : "qa-plugin-smoke tool qa_plugin_echo is not exposed through /api/v1/tools." }, + ); + if (!toolNames.includes("qa_mcp_echo")) { + warnings.push({ + name: "tool:qa_mcp_echo", + reason: "qa_mcp_echo is not currently exposed. This is acceptable before mcp-stdio-register, but mcp-stdio-tool-call must run after registration.", + }); + } + + const modelResponse = await getJson("/api/v1/provider/models/llm"); + const models = (modelResponse.json.data?.models || []).map((model) => ({ + uuid: model.uuid, + name: model.name, + abilities: Array.isArray(model.abilities) ? model.abilities : [], + provider_uuid: model.provider_uuid || model.provider?.uuid || "", + provider_name: model.provider_name || model.provider?.name || "", + requester: model.requester || model.provider?.requester || "", + })); + addCheck( + "llm-model-list", + modelResponse.status < 400 && (modelResponse.json.code ?? 0) === 0 ? "pass" : "env_issue", + { http_status: modelResponse.status, model_count: models.length, reason: safeMessage(modelResponse.json.msg || "") }, + ); + const modelById = new Map(models.map((model) => [model.uuid, model])); + + const pipelineList = await getJson("/api/v1/pipelines"); + const pipelines = pipelineList.json.data?.pipelines || []; + addCheck( + "pipeline-list", + pipelineList.status < 400 && (pipelineList.json.code ?? 0) === 0 ? "pass" : "blocked", + { http_status: pipelineList.status, pipeline_count: pipelines.length, reason: safeMessage(pipelineList.json.msg || "") }, + ); + + const resolvedPipelines = []; + const modelTested = new Set(); + for (const target of targets) { + let pipelineId = ""; + let matchedBy = ""; + if (target.pipeline_url) { + try { + pipelineId = new URL(target.pipeline_url).searchParams.get("id") || ""; + matchedBy = pipelineId ? "url" : ""; + } catch { + pipelineId = ""; + } + } + if (!pipelineId && target.pipeline_name) { + const match = pipelines.find((pipeline) => pipeline.name === target.pipeline_name); + if (match) { + pipelineId = match.uuid; + matchedBy = "name"; + } + } + if (!pipelineId) { + addCheck(`pipeline:${target.id}`, "blocked", { + target: target.id, + reason: "Required pipeline env is missing or could not resolve to a pipeline id.", + }); + continue; + } + + const response = await getJson(`/api/v1/pipelines/${encodeURIComponent(pipelineId)}`); + const pipeline = response.json.data?.pipeline; + if (response.status >= 400 || !pipeline) { + addCheck(`pipeline:${target.id}`, "blocked", { + target: target.id, + pipeline_id: pipelineId, + http_status: response.status, + reason: safeMessage(response.json.msg || "Could not load pipeline."), + }); + continue; + } + + const config = pipeline.config || {}; + const aiConfig = config.ai && typeof config.ai === "object" ? config.ai : {}; + const runner = aiConfig.runner && typeof aiConfig.runner === "object" ? aiConfig.runner : {}; + const runnerId = runner.id || runner.runner || ""; + const runnerConfigs = aiConfig.runner_config && typeof aiConfig.runner_config === "object" ? aiConfig.runner_config : {}; + const runnerConfig = runnerConfigs[runnerId] && typeof runnerConfigs[runnerId] === "object" ? runnerConfigs[runnerId] : {}; + const pipelineSummary = { + target: target.id, + pipeline_id: pipelineId, + pipeline_name: pipeline.name, + matched_by: matchedBy, + runner_id: runnerId, + expected_runner_id: target.expected_runner_id, + runner_config_keys: Object.keys(runnerConfig).sort(), + }; + resolvedPipelines.push(pipelineSummary); + + addCheck( + `pipeline:${target.id}:runner`, + runnerId === target.expected_runner_id ? "pass" : "blocked", + { + ...pipelineSummary, + reason: runnerId === target.expected_runner_id ? "" : `Expected ${target.expected_runner_id}, got ${runnerId || ""}.`, + }, + ); + + if (target.require_func_call_model || target.require_vision_model || (testModels && target.id === "local-agent")) { + const modelConfig = runnerConfig.model; + const primaryModelId = typeof modelConfig === "string" + ? modelConfig + : modelConfig && typeof modelConfig === "object" + ? modelConfig.primary || "" + : ""; + if (!primaryModelId) { + addCheck(`pipeline:${target.id}:primary-model`, "blocked", { + ...pipelineSummary, + reason: "Local-agent runner config has no primary model.", + }); + continue; + } + const model = modelById.get(primaryModelId); + if (!model) { + addCheck(`pipeline:${target.id}:primary-model`, "blocked", { + ...pipelineSummary, + model_uuid: primaryModelId, + reason: "Primary model is not listed by /api/v1/provider/models/llm.", + }); + continue; + } + addCheck(`pipeline:${target.id}:primary-model`, "pass", { + ...pipelineSummary, + model: { + uuid: model.uuid, + name: model.name, + abilities: model.abilities, + provider_name: model.provider_name, + requester: model.requester, + }, + }); + if (target.require_func_call_model) { + addCheck( + `pipeline:${target.id}:func-call-model`, + model.abilities.includes("func_call") ? "pass" : "env_issue", + { + model_uuid: model.uuid, + model_name: model.name, + abilities: model.abilities, + reason: model.abilities.includes("func_call") ? "" : "Release gate includes tool-call cases; the local-agent primary model must advertise func_call.", + }, + ); + } + if (target.require_vision_model) { + addCheck( + `pipeline:${target.id}:vision-model`, + model.abilities.includes("vision") ? "pass" : "env_issue", + { + model_uuid: model.uuid, + model_name: model.name, + abilities: model.abilities, + reason: model.abilities.includes("vision") ? "" : "Release gate includes multimodal cases; the local-agent primary model must advertise vision.", + }, + ); + } + if (testModels && !modelTested.has(model.uuid)) { + modelTested.add(model.uuid); + const modelTest = await postJson(`/api/v1/provider/models/llm/${encodeURIComponent(model.uuid)}/test`, { extra_args: {} }); + const passed = modelTest.status < 400 && (modelTest.json.code ?? 0) === 0; + addCheck( + `model-test:${model.name}`, + passed ? "pass" : "env_issue", + { + model_uuid: model.uuid, + model_name: model.name, + http_status: modelTest.status, + code: modelTest.json.code ?? null, + reason: passed ? "" : safeMessage(modelTest.json.msg || modelTest.json.message || "Model test failed."), + }, + ); + } + } + } + + return { + authenticated: true, + blockers, + env_issues: envIssues, + warnings, + checks, + resolved_pipelines: resolvedPipelines, + tools: { + required: ["qa_plugin_echo"], + optional_before_register: ["qa_mcp_echo"], + present: toolNames.filter((name) => ["qa_plugin_echo", "qa_mcp_echo"].includes(name)), + }, + models, + }; + }, { backendUrl, targets, testModels }); + + diagnostic.blockers = (diagnostic.blockers || []).map((item) => ({ ...item, reason: redactMessage(item.reason || "") })); + diagnostic.env_issues = (diagnostic.env_issues || []).map((item) => ({ ...item, reason: redactMessage(item.reason || "") })); + await writeFile(diagnosticPath, `${JSON.stringify(diagnostic, null, 2)}\n`, "utf8"); + await safeScreenshot(page, paths.screenshot); + + const blockers = diagnostic.blockers || []; + const envIssues = diagnostic.env_issues || []; + if (blockers.length > 0) { + result.status = "blocked"; + result.reason = `Preflight blocked: ${blockers.map((item) => item.name).join(", ")}`; + } else if (envIssues.length > 0) { + result.status = "env_issue"; + result.reason = `Preflight environment issue: ${envIssues.map((item) => item.name).join(", ")}`; + } else { + result.status = "pass"; + result.reason = "Release gate preflight passed: auth, plugin runtime, required pipelines, runner ids, tools, and local-agent model checks are ready."; + } + result.check_count = Array.isArray(diagnostic.checks) ? diagnostic.checks.length : 0; + result.warning_count = Array.isArray(diagnostic.warnings) ? diagnostic.warnings.length : 0; +} + +try { + await run(); +} catch (error) { + const message = redactMessage(error instanceof Error ? error.message : String(error)); + result.status = isEnvironmentError(message) ? "env_issue" : "fail"; + result.reason = message; + await writeFile(diagnosticPath, `${JSON.stringify({ + authenticated: false, + blockers: [], + env_issues: result.status === "env_issue" ? [{ name: "preflight-runtime", reason: message }] : [], + warnings: [], + checks: [ + { + name: "preflight-runtime", + status: result.status, + reason: message, + }, + ], + }, null, 2)}\n`, "utf8").catch(() => {}); +} finally { + if (browser) await browser.close().catch(() => {}); + const finishedAt = new Date(); + result.finished_at = finishedAt.toISOString(); + result.finished_at_local = localIsoWithOffset(finishedAt); + await writeResult(paths, result); + console.log(JSON.stringify(result, null, 2)); +} + +process.exit(exitCode(result.status)); diff --git a/skills/scripts/e2e/ensure-acp-agent-runner-pipeline.mjs b/skills/scripts/e2e/ensure-acp-agent-runner-pipeline.mjs new file mode 100644 index 000000000..1278fcd1b --- /dev/null +++ b/skills/scripts/e2e/ensure-acp-agent-runner-pipeline.mjs @@ -0,0 +1,263 @@ +#!/usr/bin/env node + +import { readFile, writeFile } from "node:fs/promises"; +import { resolve } from "node:path"; +import { env } from "node:process"; +import { + apiJson, + ensureEvidence, + evidencePaths, + loadEnvFiles, + resetAndAuthLocalUser, + writeResult, +} from "./lib/langbot-e2e.mjs"; + +const RUNNER_ID = "plugin:langbot/acp-agent-runner/default"; +const DEFAULT_PIPELINE_NAME = "Agent QA ACP Claude Debug Chat"; +const DEFAULT_LOCAL_PASSWORD = "LangBotE2ELocalPass!2026"; +const caseId = "ensure-acp-agent-runner-pipeline"; + +await loadEnvFiles(); +const paths = evidencePaths(caseId); +await ensureEvidence(paths); + +const writeEnv = process.argv.includes("--write-env"); +const frontendUrl = env.LANGBOT_FRONTEND_URL || ""; +const backendUrl = env.LANGBOT_BACKEND_URL || ""; +const pipelineName = env.LANGBOT_E2E_CREATE_PIPELINE_NAME || env.LANGBOT_ACP_AGENT_RUNNER_PIPELINE_NAME || DEFAULT_PIPELINE_NAME; +const sshTarget = env.LANGBOT_ACP_AGENT_RUNNER_SSH_TARGET || "yhh@101.34.71.12"; +const sshConnectTimeout = env.LANGBOT_ACP_AGENT_RUNNER_SSH_CONNECT_TIMEOUT || "8"; +const sshPort = env.LANGBOT_ACP_AGENT_RUNNER_SSH_PORT || "22"; +const sshIdentityFile = env.LANGBOT_ACP_AGENT_RUNNER_SSH_IDENTITY_FILE || ""; +const sshExtraOptions = env.LANGBOT_ACP_AGENT_RUNNER_SSH_EXTRA_OPTIONS || ""; +const remoteWorkspace = env.LANGBOT_ACP_AGENT_RUNNER_REMOTE_WORKSPACE || "/home/yhh/langbot-e2e/acp-workspace"; +const envLocalPath = resolve("skills/.env.local"); + +const result = { + source: "automation", + case_id: caseId, + run_id: paths.runId, + status: "fail", + reason: "", + frontend_url: frontendUrl, + backend_url: backendUrl, + pipeline_name: pipelineName, + pipeline_id: "", + pipeline_url: "", + runner_id: RUNNER_ID, + ssh_target: sshTarget, + ssh_port: sshPort, + remote_workspace: remoteWorkspace, + wrote_env: false, + auth: null, + evidence: { + automation_result_json: paths.automationResultJson, + result_json: paths.resultJson, + }, + evidence_collected: ["api_diagnostic"], +}; + +try { + if (!frontendUrl) throw new Error("LANGBOT_FRONTEND_URL is not configured."); + if (!backendUrl) throw new Error("LANGBOT_BACKEND_URL is not configured."); + + const user = env.LANGBOT_E2E_LOGIN_USER || ""; + const password = env.LANGBOT_E2E_LOGIN_PASSWORD || DEFAULT_LOCAL_PASSWORD; + if (!user) { + throw new Error("LANGBOT_E2E_LOGIN_USER is required so this setup can create/update the pipeline via backend API."); + } + + const auth = await resetAndAuthLocalUser({ backendUrl, user, password }); + result.auth = { + source: "local_recovery_login", + user, + backend_token_check: auth.check, + }; + + const runnerConfig = { + provider: "claude-code", + location: "remote-ssh", + workspace: remoteWorkspace, + "ssh-target": sshTarget, + "ssh-port": Number.parseInt(sshPort, 10), + "ssh-identity-file": sshIdentityFile, + "ssh-connect-timeout": Number.parseInt(sshConnectTimeout, 10), + "ssh-extra-options": sshExtraOptions, + "langbot-assets-enabled": true, + "mcp-bridge-request-timeout": 90, + "reuse-session": false, + "create-session-if-missing": true, + "append-run-scope-prompt": true, + "startup-timeout": 30, + "initialize-timeout": 120, + timeout: 300, + }; + + const prepared = await ensurePipeline({ + backendUrl, + token: auth.token, + pipelineName, + runnerId: RUNNER_ID, + runnerConfig, + }); + Object.assign(result, prepared); + if (result.pipeline_id) { + result.pipeline_url = `${frontendUrl.replace(/\/$/, "")}/home/pipelines?id=${encodeURIComponent(result.pipeline_id)}`; + } + + if (writeEnv && result.pipeline_id) { + await upsertEnvLocal(envLocalPath, { + LANGBOT_E2E_LOGIN_USER: user, + LANGBOT_ACP_AGENT_RUNNER_SSH_TARGET: sshTarget, + LANGBOT_ACP_AGENT_RUNNER_SSH_PORT: sshPort, + LANGBOT_ACP_AGENT_RUNNER_SSH_IDENTITY_FILE: sshIdentityFile, + LANGBOT_ACP_AGENT_RUNNER_SSH_EXTRA_OPTIONS: sshExtraOptions, + LANGBOT_ACP_AGENT_RUNNER_REMOTE_WORKSPACE: remoteWorkspace, + LANGBOT_ACP_AGENT_RUNNER_PIPELINE_URL: result.pipeline_url, + LANGBOT_ACP_AGENT_RUNNER_PIPELINE_NAME: result.pipeline_name || pipelineName, + }); + result.wrote_env = true; + } +} catch (error) { + result.reason = result.reason || error.message; +} finally { + await writeResult(paths, result); + console.log(JSON.stringify(result, null, 2)); +} + +process.exit(result.status === "pass" ? 0 : result.status === "env_issue" ? 2 : 1); + +async function ensurePipeline({ backendUrl, token, pipelineName, runnerId, runnerConfig }) { + const pipelineList = await apiJson(backendUrl, "/api/v1/pipelines", { token }); + if (isApiFailure(pipelineList)) { + return { + status: "fail", + reason: pipelineList.json.msg || "Failed to list pipelines.", + list_status: pipelineList.status, + }; + } + + const pipelines = pipelineList.json.data?.pipelines || []; + let pipeline = pipelines.find((item) => item.name === pipelineName) || null; + let created = false; + + if (!pipeline) { + const createdResponse = await apiJson(backendUrl, "/api/v1/pipelines", { + method: "POST", + token, + body: { + name: pipelineName, + description: "Local QA pipeline for real ACP Claude AgentRunner Debug Chat smoke tests.", + emoji: "QA", + }, + }); + if (isApiFailure(createdResponse)) { + return { + status: "fail", + reason: createdResponse.json.msg || "Failed to create pipeline.", + create_status: createdResponse.status, + }; + } + const pipelineId = createdResponse.json.data?.uuid || ""; + const loaded = await apiJson(backendUrl, `/api/v1/pipelines/${encodeURIComponent(pipelineId)}`, { token }); + pipeline = loaded.json.data?.pipeline || null; + created = true; + } + + if (!pipeline?.uuid) { + return { + status: "fail", + reason: "Pipeline was not created or resolved.", + }; + } + + const loaded = await apiJson(backendUrl, `/api/v1/pipelines/${encodeURIComponent(pipeline.uuid)}`, { token }); + if (isApiFailure(loaded) || !loaded.json.data?.pipeline) { + return { + status: "fail", + reason: loaded.json.msg || "Failed to load pipeline.", + get_status: loaded.status, + pipeline_id: pipeline.uuid, + }; + } + pipeline = loaded.json.data.pipeline; + + const config = pipeline.config && typeof pipeline.config === "object" ? pipeline.config : {}; + const ai = config.ai && typeof config.ai === "object" ? config.ai : {}; + const runnerConfigs = ai.runner_config && typeof ai.runner_config === "object" ? ai.runner_config : {}; + const updatedConfig = { + ...config, + ai: { + ...ai, + runner: { + ...(ai.runner && typeof ai.runner === "object" ? ai.runner : {}), + id: runnerId, + "expire-time": 0, + }, + runner_config: { + ...runnerConfigs, + [runnerId]: runnerConfig, + }, + }, + }; + + const updateResponse = await apiJson(backendUrl, `/api/v1/pipelines/${encodeURIComponent(pipeline.uuid)}`, { + method: "PUT", + token, + body: { + name: pipelineName, + description: "Local QA pipeline for real ACP Claude AgentRunner Debug Chat smoke tests.", + emoji: "QA", + config: updatedConfig, + }, + }); + if (isApiFailure(updateResponse)) { + return { + status: "fail", + reason: updateResponse.json.msg || "Failed to update pipeline.", + update_status: updateResponse.status, + pipeline_id: pipeline.uuid, + }; + } + + return { + status: "pass", + reason: created ? "ACP AgentRunner pipeline created and configured." : "ACP AgentRunner pipeline updated.", + pipeline_id: pipeline.uuid, + pipeline_name: pipelineName, + created, + updated: true, + }; +} + +function isApiFailure(response) { + return response.status >= 400 || (response.json && response.json.code !== undefined && response.json.code !== 0); +} + +async function upsertEnvLocal(path, values) { + let text = ""; + try { + text = await readFile(path, "utf8"); + } catch { + text = ""; + } + const lines = text.split(/\r?\n/); + const keys = new Set(Object.keys(values)); + const output = []; + for (const line of lines) { + const match = line.match(/^([A-Z][A-Z0-9_]*)=/); + if (match && keys.has(match[1])) { + output.push(`${match[1]}=${values[match[1]]}`); + keys.delete(match[1]); + } else if (line !== "" || output.length > 0) { + output.push(line); + } + } + if (keys.size > 0 && output.length > 0 && output[output.length - 1] !== "") { + output.push(""); + } + for (const key of keys) { + output.push(`${key}=${values[key]}`); + } + await writeFile(path, `${output.join("\n").replace(/\n+$/, "")}\n`, "utf8"); +} diff --git a/skills/scripts/e2e/ensure-langrag-sentinel-kb.mjs b/skills/scripts/e2e/ensure-langrag-sentinel-kb.mjs new file mode 100644 index 000000000..b2cf9e7b2 --- /dev/null +++ b/skills/scripts/e2e/ensure-langrag-sentinel-kb.mjs @@ -0,0 +1,293 @@ +#!/usr/bin/env node + +import { readFile, writeFile } from "node:fs/promises"; +import { resolve } from "node:path"; +import { env } from "node:process"; +import { + apiJson, + ensureEvidence, + evidencePaths, + loadEnvFiles, + resetAndAuthLocalUser, + writeResult, +} from "./lib/langbot-e2e.mjs"; + +const caseId = env.LBS_CASE_ID || "ensure-langrag-sentinel-kb"; + +await loadEnvFiles(); +const paths = evidencePaths(caseId); +await ensureEvidence(paths); + +const backendUrl = env.LANGBOT_BACKEND_URL || ""; +const user = env.LANGBOT_E2E_LOGIN_USER || ""; +const password = env.LANGBOT_E2E_LOGIN_PASSWORD || "LangBotE2ELocalPass!2026"; +const expectedText = env.LANGBOT_E2E_EXPECTED_TEXT || "azalea-cobalt-7421"; +const query = env.LANGBOT_E2E_RETRIEVE_QUERY || "What is the local agent runner retrieval sentinel?"; +const writeEnv = process.argv.includes("--write-env"); +const checkOnly = process.argv.includes("--check-only"); +const envLocalPath = resolve("skills/.env.local"); +const kbName = env.LANGBOT_E2E_RAG_KB_NAME || "qa-local-agent-rag"; +const sentinelPath = resolve(env.LANGBOT_E2E_RAG_SENTINEL_DOC || "skills/langbot-testing/fixtures/rag/sentinel-doc.txt"); +const waitMs = Number(env.LANGBOT_E2E_RAG_WAIT_MS || 180_000); + +const result = { + source: "automation", + case_id: caseId, + run_id: paths.runId, + status: "fail", + reason: "", + backend_url: backendUrl, + expected_text: expectedText, + query, + kb_uuid: "", + kb_name: "", + kb_created: false, + uploaded_file_id: "", + store_task_id: "", + embedding_model_uuid: "", + engine_plugin_id: "", + checked_bases: [], + file_statuses: [], + wrote_env: false, + evidence: { + automation_result_json: paths.automationResultJson, + result_json: paths.resultJson, + }, + evidence_collected: ["api_diagnostic"], +}; + +try { + if (!backendUrl) throw new Error("LANGBOT_BACKEND_URL is not configured."); + if (!user) throw new Error("LANGBOT_E2E_LOGIN_USER is required."); + + const auth = await resetAndAuthLocalUser({ backendUrl, user, password }); + const basesResponse = await apiJson(backendUrl, "/api/v1/knowledge/bases", { token: auth.token }); + if (basesResponse.status >= 400 || basesResponse.json.code !== 0) { + throw new Error(basesResponse.json.msg || `Failed to list knowledge bases: HTTP ${basesResponse.status}.`); + } + + let bases = basesResponse.json.data?.bases || []; + await findSentinelBase(backendUrl, auth.token, bases, result); + + if (!result.kb_uuid && !checkOnly) { + const targetBase = bases.find((base) => { + const uuid = base.uuid || base.id || ""; + return (base.name || "") === kbName && !hasRetrieveFailure(result.checked_bases, uuid); + }); + result.kb_uuid = targetBase?.uuid || targetBase?.id || ""; + result.kb_name = targetBase?.name || kbName; + + if (!result.kb_uuid) { + const setup = await createKnowledgeBase(backendUrl, auth.token, kbName); + result.kb_uuid = setup.kbUuid; + result.kb_name = kbName; + result.kb_created = true; + result.embedding_model_uuid = setup.embeddingModelUuid; + result.engine_plugin_id = setup.enginePluginId; + } + + const upload = await uploadDocument(backendUrl, auth.token, sentinelPath); + result.uploaded_file_id = upload.fileId; + + const store = await apiJson(backendUrl, `/api/v1/knowledge/bases/${encodeURIComponent(result.kb_uuid)}/files`, { + method: "POST", + token: auth.token, + body: { file_id: upload.fileId }, + }); + if (store.status >= 400 || store.json.code !== 0) { + throw new Error(store.json.msg || `Failed to store file in knowledge base: HTTP ${store.status}.`); + } + result.store_task_id = store.json.data?.task_id || ""; + + const ready = await waitForSentinel(backendUrl, auth.token, result.kb_uuid, query, expectedText, waitMs); + result.file_statuses = ready.fileStatuses; + if (ready.matched) { + result.checked_bases.push(ready.checked); + } + } + + if (!result.kb_uuid) { + result.status = "env_issue"; + result.reason = checkOnly + ? `No existing knowledge base retrieved expected sentinel: ${expectedText}` + : `Could not create or verify LangRAG sentinel knowledge base: ${expectedText}`; + } else { + if (writeEnv) { + await upsertEnvLocal(envLocalPath, { + LANGBOT_LOCAL_AGENT_RAG_KB_UUID: result.kb_uuid, + }); + result.wrote_env = true; + } + result.status = "pass"; + result.reason = `Found LangRAG sentinel knowledge base: ${result.kb_uuid}`; + } +} catch (error) { + result.status = /not configured|required|No existing knowledge base/.test(error.message) ? "env_issue" : "fail"; + result.reason = error.message; +} finally { + await writeResult(paths, result); + console.log(JSON.stringify(result, null, 2)); +} + +process.exit(result.status === "pass" ? 0 : result.status === "env_issue" ? 2 : 1); + +async function findSentinelBase(backendUrl, token, bases, result) { + for (const base of bases) { + const uuid = base.uuid || base.id || ""; + if (!uuid) continue; + const checked = await retrieveSentinel(backendUrl, token, uuid, base.name || "", result.query, result.expected_text); + result.checked_bases.push(checked); + if (checked.matched) { + result.kb_uuid = uuid; + result.kb_name = checked.name; + return; + } + } +} + +async function createKnowledgeBase(backendUrl, token, name) { + const enginesResponse = await apiJson(backendUrl, "/api/v1/knowledge/engines", { token }); + if (enginesResponse.status >= 400 || enginesResponse.json.code !== 0) { + throw new Error(enginesResponse.json.msg || `Failed to list knowledge engines: HTTP ${enginesResponse.status}.`); + } + const engines = enginesResponse.json.data?.engines || []; + const engine = engines.find((item) => item.plugin_id === "langbot-team/LangRAG") + || engines.find((item) => JSON.stringify(item.name || item.label || "").includes("LangRAG")); + const enginePluginId = engine?.plugin_id || ""; + if (!enginePluginId) throw new Error("LangRAG knowledge engine is not installed."); + + const embeddingModelUuid = await pickEmbeddingModel(backendUrl, token); + const create = await apiJson(backendUrl, "/api/v1/knowledge/bases", { + method: "POST", + token, + body: { + name, + description: "Automated LangBot agent-runner RAG sentinel knowledge base.", + knowledge_engine_plugin_id: enginePluginId, + creation_settings: { + embedding_model_uuid: embeddingModelUuid, + index_type: "chunk", + chunk_size: 512, + overlap: 50, + }, + retrieval_settings: { + top_k: 5, + search_type: "vector", + query_rewrite: "off", + rerank: "off", + context_window: 0, + }, + }, + }); + const kbUuid = create.json.data?.uuid || ""; + if (create.status >= 400 || create.json.code !== 0 || !kbUuid) { + throw new Error(create.json.msg || `Failed to create knowledge base: HTTP ${create.status}.`); + } + return { kbUuid, embeddingModelUuid, enginePluginId }; +} + +async function pickEmbeddingModel(backendUrl, token) { + const configured = env.LANGBOT_LOCAL_AGENT_RAG_EMBEDDING_MODEL_UUID || env.LANGBOT_RAG_EMBEDDING_MODEL_UUID || ""; + if (configured) return configured; + + const modelsResponse = await apiJson(backendUrl, "/api/v1/provider/models/embedding", { token }); + if (modelsResponse.status >= 400 || modelsResponse.json.code !== 0) { + throw new Error(modelsResponse.json.msg || `Failed to list embedding models: HTTP ${modelsResponse.status}.`); + } + const models = modelsResponse.json.data?.models || []; + const preferred = models.find((model) => /chroma|MiniLM/i.test(model.name || "")) + || models.find((model) => /text-embedding-3-small/i.test(model.name || "")) + || [...models].sort((a, b) => (a.prefered_ranking ?? 9999) - (b.prefered_ranking ?? 9999))[0]; + const uuid = preferred?.uuid || ""; + if (!uuid) throw new Error("No embedding model is configured."); + return uuid; +} + +async function uploadDocument(backendUrl, token, path) { + const bytes = await readFile(path); + const form = new FormData(); + form.append("file", new Blob([bytes], { type: "text/plain" }), "sentinel-doc.txt"); + const response = await fetch(`${backendUrl.replace(/\/$/, "")}/api/v1/files/documents`, { + method: "POST", + headers: { + Authorization: `Bearer ${token}`, + }, + body: form, + }); + const json = await response.json().catch(() => ({})); + const fileId = json.data?.file_id || ""; + if (response.status >= 400 || json.code !== 0 || !fileId) { + throw new Error(json.msg || `Failed to upload sentinel document: HTTP ${response.status}.`); + } + return { fileId }; +} + +async function waitForSentinel(backendUrl, token, kbUuid, query, expectedText, timeoutMs) { + const started = Date.now(); + let fileStatuses = []; + let lastChecked = null; + while (Date.now() - started < timeoutMs) { + const files = await apiJson(backendUrl, `/api/v1/knowledge/bases/${encodeURIComponent(kbUuid)}/files`, { token }); + fileStatuses = files.json.data?.files || fileStatuses; + lastChecked = await retrieveSentinel(backendUrl, token, kbUuid, kbName, query, expectedText); + if (lastChecked.matched) { + return { matched: true, fileStatuses, checked: lastChecked }; + } + if (fileStatuses.some((item) => item.status === "failed")) break; + await sleep(2_000); + } + result.reason = lastChecked?.msg + || `LangRAG sentinel was not retrievable within ${timeoutMs}ms; file statuses: ${JSON.stringify(fileStatuses)}`; + result.kb_uuid = ""; + return { matched: false, fileStatuses, checked: lastChecked }; +} + +async function retrieveSentinel(backendUrl, token, uuid, name, query, expectedText) { + const retrieve = await apiJson(backendUrl, `/api/v1/knowledge/bases/${encodeURIComponent(uuid)}/retrieve`, { + method: "POST", + token, + body: { query }, + }); + const text = JSON.stringify(retrieve.json.data?.results || []); + return { + uuid, + name, + http_status: retrieve.status, + code: retrieve.json.code ?? null, + msg: retrieve.json.msg || "", + matched: text.includes(expectedText), + }; +} + +function sleep(ms) { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +function hasRetrieveFailure(checkedBases, uuid) { + const checked = checkedBases.find((item) => item.uuid === uuid); + return checked && (checked.http_status >= 500 || (typeof checked.code === "number" && checked.code < 0)); +} + +async function upsertEnvLocal(path, values) { + let text = ""; + try { + text = await readFile(path, "utf8"); + } catch { + text = ""; + } + const lines = text.split(/\r?\n/); + const keys = new Set(Object.keys(values)); + const output = []; + for (const line of lines) { + const match = line.match(/^([A-Z][A-Z0-9_]*)=/); + if (match && keys.has(match[1])) { + output.push(`${match[1]}=${values[match[1]]}`); + keys.delete(match[1]); + } else if (line !== "" || output.length > 0) { + output.push(line); + } + } + if (keys.size > 0 && output.length > 0 && output[output.length - 1] !== "") output.push(""); + for (const key of keys) output.push(`${key}=${values[key]}`); + await writeFile(path, `${output.join("\n").replace(/\n+$/, "")}\n`, "utf8"); +} diff --git a/skills/scripts/e2e/ensure-local-agent-pipeline.mjs b/skills/scripts/e2e/ensure-local-agent-pipeline.mjs new file mode 100644 index 000000000..0962c6bf5 --- /dev/null +++ b/skills/scripts/e2e/ensure-local-agent-pipeline.mjs @@ -0,0 +1,312 @@ +#!/usr/bin/env node + +import { readFile, writeFile } from "node:fs/promises"; +import { resolve } from "node:path"; +import { env } from "node:process"; +import { + apiJson, + bodyText, + createBrowser, + ensureEvidence, + evidencePaths, + loadEnvFiles, + resetAndAuthLocalUser, + safeScreenshot, + setBrowserToken, + verifyBrowserToken, + writeResult, +} from "./lib/langbot-e2e.mjs"; + +const RUNNER_ID = "plugin:langbot/local-agent/default"; +const DEFAULT_PIPELINE_NAME = "Agent QA Local Agent Debug Chat"; +const DEFAULT_LOCAL_PASSWORD = "LangBotE2ELocalPass!2026"; +const caseId = "ensure-local-agent-pipeline"; + +await loadEnvFiles(); +const paths = evidencePaths(caseId); +await ensureEvidence(paths); + +const writeEnv = process.argv.includes("--write-env"); +const pipelineName = env.LANGBOT_E2E_CREATE_PIPELINE_NAME || env.LANGBOT_LOCAL_AGENT_PIPELINE_NAME || DEFAULT_PIPELINE_NAME; +const frontendUrl = env.LANGBOT_FRONTEND_URL || ""; +const backendUrl = env.LANGBOT_BACKEND_URL || ""; +const envLocalPath = resolve("skills/.env.local"); + +const result = { + source: "automation", + case_id: caseId, + run_id: paths.runId, + status: "fail", + reason: "", + frontend_url: frontendUrl, + backend_url: backendUrl, + pipeline_name: pipelineName, + pipeline_id: "", + pipeline_url: "", + runner_id: RUNNER_ID, + selected_model_id: "", + model_count: 0, + created: false, + updated: false, + wrote_env: false, + auth: null, + browser_token_check: null, + page_signal: "", + evidence: { + console_log: paths.consoleLog, + network_log: paths.networkLog, + screenshot: paths.screenshot, + automation_result_json: paths.automationResultJson, + result_json: paths.resultJson, + }, + evidence_collected: ["api_diagnostic", "console", "network", "screenshot"], +}; + +let browser; + +try { + if (!frontendUrl) throw new Error("LANGBOT_FRONTEND_URL is not configured."); + if (!backendUrl) throw new Error("LANGBOT_BACKEND_URL is not configured."); + + const user = env.LANGBOT_E2E_LOGIN_USER || ""; + const password = env.LANGBOT_E2E_LOGIN_PASSWORD || DEFAULT_LOCAL_PASSWORD; + if (!user) { + throw new Error("LANGBOT_E2E_LOGIN_USER is required so this setup can create/update the pipeline via backend API."); + } + + const auth = await resetAndAuthLocalUser({ backendUrl, user, password }); + result.auth = { + source: "local_recovery_login", + user, + backend_token_check: auth.check, + }; + + const prepared = await ensureLocalAgentPipeline({ + backendUrl, + token: auth.token, + pipelineName, + runnerId: RUNNER_ID, + }); + Object.assign(result, prepared); + if (result.pipeline_id) { + result.pipeline_url = `${frontendUrl.replace(/\/$/, "")}/home/pipelines?id=${encodeURIComponent(result.pipeline_id)}`; + } + + if (writeEnv && result.pipeline_id) { + await upsertEnvLocal(envLocalPath, { + LANGBOT_E2E_LOGIN_USER: user, + LANGBOT_PIPELINE_URL: result.pipeline_url, + LANGBOT_PIPELINE_NAME: result.pipeline_name || pipelineName, + LANGBOT_LOCAL_AGENT_PIPELINE_URL: result.pipeline_url, + LANGBOT_LOCAL_AGENT_PIPELINE_NAME: result.pipeline_name || pipelineName, + }); + result.wrote_env = true; + } + + browser = await createBrowser(paths); + const { page } = browser; + await setBrowserToken(page, frontendUrl, auth.token); + const browserCheck = await verifyBrowserToken(page, backendUrl); + result.browser_token_check = browserCheck; + if (!browserCheck.authenticated) { + throw new Error(browserCheck.reason || "Browser token check failed after setup."); + } + await page.goto(result.pipeline_url || frontendUrl, { waitUntil: "domcontentloaded" }); + await page.waitForLoadState("networkidle", { timeout: 10_000 }).catch(() => {}); + const text = await bodyText(page); + result.page_signal = ["Pipelines", "流水线", pipelineName].find((signal) => text.includes(signal)) || ""; +} catch (error) { + result.status = result.status === "env_issue" ? "env_issue" : "fail"; + result.reason = result.reason || error.message; +} finally { + if (browser?.page) await safeScreenshot(browser.page, paths.screenshot); + if (browser) await browser.close().catch(() => {}); + await writeResult(paths, result); + console.log(JSON.stringify(result, null, 2)); +} + +process.exit(result.status === "pass" ? 0 : result.status === "env_issue" ? 2 : 1); + +async function ensureLocalAgentPipeline({ backendUrl, token, pipelineName, runnerId }) { + const [pipelineList, modelList] = await Promise.all([ + apiJson(backendUrl, "/api/v1/pipelines", { token }), + apiJson(backendUrl, "/api/v1/provider/models/llm", { token }), + ]); + + if (isApiFailure(pipelineList)) { + return { + status: "fail", + reason: pipelineList.json.msg || "Failed to list pipelines.", + list_status: pipelineList.status, + }; + } + if (isApiFailure(modelList)) { + return { + status: "fail", + reason: modelList.json.msg || "Failed to list LLM models.", + model_status: modelList.status, + }; + } + + const models = modelList.json.data?.models || []; + const selectedModel = models.find((model) => model.uuid) || null; + const pipelines = pipelineList.json.data?.pipelines || []; + let pipeline = pipelines.find((item) => item.name === pipelineName) || null; + let created = false; + + if (!pipeline) { + const createdResponse = await apiJson(backendUrl, "/api/v1/pipelines", { + method: "POST", + token, + body: { + name: pipelineName, + description: "Local QA pipeline for AgentRunner Debug Chat smoke tests.", + emoji: "QA", + }, + }); + if (isApiFailure(createdResponse)) { + return { + status: "fail", + reason: createdResponse.json.msg || "Failed to create pipeline.", + create_status: createdResponse.status, + model_count: models.length, + }; + } + const pipelineId = createdResponse.json.data?.uuid || ""; + const loaded = await apiJson(backendUrl, `/api/v1/pipelines/${encodeURIComponent(pipelineId)}`, { token }); + pipeline = loaded.json.data?.pipeline || null; + created = true; + } + + if (!pipeline?.uuid) { + return { + status: "fail", + reason: "Pipeline was not created or resolved.", + model_count: models.length, + }; + } + + const loaded = await apiJson(backendUrl, `/api/v1/pipelines/${encodeURIComponent(pipeline.uuid)}`, { token }); + if (isApiFailure(loaded) || !loaded.json.data?.pipeline) { + return { + status: "fail", + reason: loaded.json.msg || "Failed to load pipeline.", + get_status: loaded.status, + pipeline_id: pipeline.uuid, + model_count: models.length, + }; + } + pipeline = loaded.json.data.pipeline; + + const config = pipeline.config && typeof pipeline.config === "object" ? pipeline.config : {}; + const ai = config.ai && typeof config.ai === "object" ? config.ai : {}; + const runnerConfig = ai.runner_config && typeof ai.runner_config === "object" ? ai.runner_config : {}; + const rawExistingLocalAgentConfig = runnerConfig[runnerId] && typeof runnerConfig[runnerId] === "object" + ? runnerConfig[runnerId] + : {}; + const existingLocalAgentConfig = rawExistingLocalAgentConfig; + const existingModel = existingLocalAgentConfig.model && typeof existingLocalAgentConfig.model === "object" + ? existingLocalAgentConfig.model + : {}; + const requestedModelId = env.LANGBOT_LOCAL_AGENT_MODEL_UUID || env.LANGBOT_E2E_MODEL_UUID || ""; + const selectedModelId = requestedModelId || existingModel.primary || selectedModel?.uuid || ""; + const localAgentConfig = { + timeout: 300, + prompt: [{ role: "system", content: "You are a helpful assistant." }], + "remove-think": false, + "knowledge-bases": [], + "retrieval-top-k": 5, + "rerank-model": "", + "rerank-top-k": 5, + "max-tool-iterations": 20, + "tool-execution-mode": "parallel", + "max-tool-result-chars": 20000, + "context-history-fetch-limit": 50, + "context-window-tokens": 200000, + "context-reserve-tokens": 16384, + "context-keep-recent-tokens": 20000, + "context-summary-tokens": 8000, + ...existingLocalAgentConfig, + model: { + primary: selectedModelId, + fallbacks: requestedModelId ? [] : Array.isArray(existingModel.fallbacks) ? existingModel.fallbacks : [], + }, + }; + const updatedConfig = { + ...config, + ai: { + ...ai, + runner: { + ...(ai.runner && typeof ai.runner === "object" ? ai.runner : {}), + id: runnerId, + "expire-time": 0, + }, + runner_config: { + ...runnerConfig, + [runnerId]: localAgentConfig, + }, + }, + }; + + const updateResponse = await apiJson(backendUrl, `/api/v1/pipelines/${encodeURIComponent(pipeline.uuid)}`, { + method: "PUT", + token, + body: { + name: pipelineName, + description: "Local QA pipeline for AgentRunner Debug Chat smoke tests.", + emoji: "QA", + config: updatedConfig, + }, + }); + if (isApiFailure(updateResponse)) { + return { + status: "fail", + reason: updateResponse.json.msg || "Failed to update pipeline config.", + update_status: updateResponse.status, + pipeline_id: pipeline.uuid, + model_count: models.length, + selected_model_id: selectedModelId, + }; + } + + return { + status: selectedModelId ? "pass" : "env_issue", + reason: selectedModelId + ? "Local-agent pipeline is configured for Debug Chat." + : "Pipeline was created but no LLM model is configured in this LangBot instance.", + pipeline_id: pipeline.uuid, + pipeline_name: pipeline.name, + model_count: models.length, + selected_model_id: selectedModelId, + created, + updated: true, + }; +} + +function isApiFailure(response) { + return response.status >= 400 || (response.json.code !== undefined && response.json.code !== 0); +} + +async function upsertEnvLocal(path, updates) { + let text = ""; + try { + text = await readFile(path, "utf8"); + } catch { + text = ""; + } + const lines = text.split(/\r?\n/); + const seen = new Set(); + const next = lines.map((line) => { + const trimmed = line.trim(); + const equals = trimmed.indexOf("="); + if (equals <= 0 || trimmed.startsWith("#")) return line; + const key = trimmed.slice(0, equals).trim(); + if (!(key in updates)) return line; + seen.add(key); + return `${key}=${updates[key]}`; + }); + for (const [key, value] of Object.entries(updates)) { + if (!seen.has(key)) next.push(`${key}=${value}`); + } + await writeFile(path, `${next.filter((line, index) => line !== "" || index < next.length - 1).join("\n")}\n`, "utf8"); +} diff --git a/skills/scripts/e2e/ensure-qa-agent-runner-pipeline.mjs b/skills/scripts/e2e/ensure-qa-agent-runner-pipeline.mjs new file mode 100644 index 000000000..abc43241a --- /dev/null +++ b/skills/scripts/e2e/ensure-qa-agent-runner-pipeline.mjs @@ -0,0 +1,230 @@ +#!/usr/bin/env node + +import { readFile, writeFile } from "node:fs/promises"; +import { resolve } from "node:path"; +import { env } from "node:process"; +import { + apiJson, + ensureEvidence, + evidencePaths, + loadEnvFiles, + resetAndAuthLocalUser, + writeResult, +} from "./lib/langbot-e2e.mjs"; + +const RUNNER_ID = "plugin:qa/agent-runner/default"; +const DEFAULT_PIPELINE_NAME = "Agent QA Deterministic Runner Debug Chat"; +const DEFAULT_LOCAL_PASSWORD = "LangBotE2ELocalPass!2026"; +const caseId = "ensure-qa-agent-runner-pipeline"; + +await loadEnvFiles(); +const paths = evidencePaths(caseId); +await ensureEvidence(paths); + +const writeEnv = process.argv.includes("--write-env"); +const frontendUrl = env.LANGBOT_FRONTEND_URL || ""; +const backendUrl = env.LANGBOT_BACKEND_URL || ""; +const pipelineName = env.LANGBOT_E2E_CREATE_PIPELINE_NAME || env.LANGBOT_QA_AGENT_RUNNER_PIPELINE_NAME || DEFAULT_PIPELINE_NAME; +const envLocalPath = resolve("skills/.env.local"); + +const result = { + source: "automation", + case_id: caseId, + run_id: paths.runId, + status: "fail", + reason: "", + frontend_url: frontendUrl, + backend_url: backendUrl, + pipeline_name: pipelineName, + pipeline_id: "", + pipeline_url: "", + runner_id: RUNNER_ID, + wrote_env: false, + auth: null, + evidence: { + automation_result_json: paths.automationResultJson, + result_json: paths.resultJson, + }, + evidence_collected: ["api_diagnostic"], +}; + +try { + if (!frontendUrl) throw new Error("LANGBOT_FRONTEND_URL is not configured."); + if (!backendUrl) throw new Error("LANGBOT_BACKEND_URL is not configured."); + + const user = env.LANGBOT_E2E_LOGIN_USER || ""; + const password = env.LANGBOT_E2E_LOGIN_PASSWORD || DEFAULT_LOCAL_PASSWORD; + if (!user) { + throw new Error("LANGBOT_E2E_LOGIN_USER is required so this setup can create/update the pipeline via backend API."); + } + + const auth = await resetAndAuthLocalUser({ backendUrl, user, password }); + result.auth = { + source: "local_recovery_login", + user, + backend_token_check: auth.check, + }; + + const prepared = await ensurePipeline({ + backendUrl, + token: auth.token, + pipelineName, + runnerId: RUNNER_ID, + runnerConfig: {}, + }); + Object.assign(result, prepared); + if (result.pipeline_id) { + result.pipeline_url = `${frontendUrl.replace(/\/$/, "")}/home/pipelines?id=${encodeURIComponent(result.pipeline_id)}`; + } + + if (writeEnv && result.pipeline_id) { + await upsertEnvLocal(envLocalPath, { + LANGBOT_E2E_LOGIN_USER: user, + LANGBOT_QA_AGENT_RUNNER_PIPELINE_URL: result.pipeline_url, + LANGBOT_QA_AGENT_RUNNER_PIPELINE_NAME: result.pipeline_name || pipelineName, + }); + result.wrote_env = true; + } +} catch (error) { + result.reason = result.reason || error.message; +} finally { + await writeResult(paths, result); + console.log(JSON.stringify(result, null, 2)); +} + +process.exit(result.status === "pass" ? 0 : result.status === "env_issue" ? 2 : 1); + +async function ensurePipeline({ backendUrl, token, pipelineName, runnerId, runnerConfig }) { + const pipelineList = await apiJson(backendUrl, "/api/v1/pipelines", { token }); + if (isApiFailure(pipelineList)) { + return { + status: "fail", + reason: pipelineList.json.msg || "Failed to list pipelines.", + list_status: pipelineList.status, + }; + } + + const pipelines = pipelineList.json.data?.pipelines || []; + let pipeline = pipelines.find((item) => item.name === pipelineName) || null; + let created = false; + + if (!pipeline) { + const createdResponse = await apiJson(backendUrl, "/api/v1/pipelines", { + method: "POST", + token, + body: { + name: pipelineName, + description: "Local QA pipeline for deterministic QA AgentRunner Debug Chat smoke tests.", + emoji: "QA", + }, + }); + if (isApiFailure(createdResponse)) { + return { + status: "fail", + reason: createdResponse.json.msg || "Failed to create pipeline.", + create_status: createdResponse.status, + }; + } + const pipelineId = createdResponse.json.data?.uuid || ""; + const loaded = await apiJson(backendUrl, `/api/v1/pipelines/${encodeURIComponent(pipelineId)}`, { token }); + pipeline = loaded.json.data?.pipeline || null; + created = true; + } + + if (!pipeline?.uuid) { + return { + status: "fail", + reason: "Pipeline was not created or resolved.", + }; + } + + const loaded = await apiJson(backendUrl, `/api/v1/pipelines/${encodeURIComponent(pipeline.uuid)}`, { token }); + if (isApiFailure(loaded) || !loaded.json.data?.pipeline) { + return { + status: "fail", + reason: loaded.json.msg || "Failed to load pipeline.", + get_status: loaded.status, + pipeline_id: pipeline.uuid, + }; + } + pipeline = loaded.json.data.pipeline; + + const config = pipeline.config && typeof pipeline.config === "object" ? pipeline.config : {}; + const ai = config.ai && typeof config.ai === "object" ? config.ai : {}; + const runnerConfigs = ai.runner_config && typeof ai.runner_config === "object" ? ai.runner_config : {}; + const updatedConfig = { + ...config, + ai: { + ...ai, + runner: { + ...(ai.runner && typeof ai.runner === "object" ? ai.runner : {}), + id: runnerId, + "expire-time": 0, + }, + runner_config: { + ...runnerConfigs, + [runnerId]: runnerConfig, + }, + }, + }; + + const updateResponse = await apiJson(backendUrl, `/api/v1/pipelines/${encodeURIComponent(pipeline.uuid)}`, { + method: "PUT", + token, + body: { + name: pipelineName, + description: "Local QA pipeline for deterministic QA AgentRunner Debug Chat smoke tests.", + emoji: "QA", + config: updatedConfig, + }, + }); + if (isApiFailure(updateResponse)) { + return { + status: "fail", + reason: updateResponse.json.msg || "Failed to update pipeline.", + update_status: updateResponse.status, + pipeline_id: pipeline.uuid, + }; + } + + return { + status: "pass", + reason: created ? "QA AgentRunner pipeline created and configured." : "QA AgentRunner pipeline updated.", + pipeline_id: pipeline.uuid, + pipeline_name: pipelineName, + created, + updated: true, + }; +} + +function isApiFailure(response) { + return response.status >= 400 || (response.json && response.json.code !== undefined && response.json.code !== 0); +} + +async function upsertEnvLocal(path, values) { + let text = ""; + try { + text = await readFile(path, "utf8"); + } catch { + text = ""; + } + const lines = text.split(/\r?\n/); + const keys = new Set(Object.keys(values)); + const output = []; + for (const line of lines) { + const match = line.match(/^([A-Z][A-Z0-9_]*)=/); + if (match && keys.has(match[1])) { + output.push(`${match[1]}=${values[match[1]]}`); + keys.delete(match[1]); + } else if (line !== "" || output.length > 0) { + output.push(line); + } + } + if (keys.size > 0 && output.length > 0 && output[output.length - 1] !== "") { + output.push(""); + } + for (const key of keys) { + output.push(`${key}=${values[key]}`); + } + await writeFile(path, `${output.join("\n").replace(/\n+$/, "")}\n`, "utf8"); +} diff --git a/skills/scripts/e2e/install-qa-plugin-smoke.mjs b/skills/scripts/e2e/install-qa-plugin-smoke.mjs new file mode 100644 index 000000000..73b89af69 --- /dev/null +++ b/skills/scripts/e2e/install-qa-plugin-smoke.mjs @@ -0,0 +1,198 @@ +#!/usr/bin/env node + +import { readFile } from "node:fs/promises"; +import { resolve } from "node:path"; +import { env } from "node:process"; +import { + apiJson, + ensureEvidence, + evidencePaths, + loadEnvFiles, + resetAndAuthLocalUser, + writeResult, +} from "./lib/langbot-e2e.mjs"; + +const caseId = env.LBS_CASE_ID || "install-qa-plugin-smoke"; +const paths = evidencePaths(caseId); +await loadEnvFiles(); +await ensureEvidence(paths); + +const backendUrl = env.LANGBOT_BACKEND_URL || ""; +const user = env.LANGBOT_E2E_LOGIN_USER || ""; +const password = env.LANGBOT_E2E_LOGIN_PASSWORD || "LangBotE2ELocalPass!2026"; +const packagePath = resolve( + env.LANGBOT_E2E_PLUGIN_PACKAGE + || env.LANGBOT_QA_PLUGIN_SMOKE_PACKAGE + || "skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/dist/qa-plugin-smoke-0.1.0.lbpkg", +); +const expectedPluginId = env.LANGBOT_E2E_EXPECTED_PLUGIN_ID || "qa/plugin-smoke"; +const expectedTool = env.LANGBOT_E2E_EXPECTED_TOOL || (expectedPluginId === "qa/plugin-smoke" ? "qa_plugin_echo" : ""); +const expectedRunnerId = env.LANGBOT_E2E_EXPECTED_RUNNER_ID || ""; + +const result = { + source: "automation", + case_id: caseId, + run_id: paths.runId, + status: "fail", + reason: "", + backend_url: backendUrl, + package_path: packagePath, + package_preview: null, + task_id: null, + task: null, + plugin_present_before: false, + plugin_present_after: false, + tool_names: [], + runner_ids: [], + evidence: { + automation_result_json: paths.automationResultJson, + result_json: paths.resultJson, + }, + evidence_collected: ["api_diagnostic", "filesystem"], +}; + +try { + if (!backendUrl) throw new Error("LANGBOT_BACKEND_URL is not configured."); + if (!user) throw new Error("LANGBOT_E2E_LOGIN_USER is required."); + const bytes = await readFile(packagePath); + + const auth = await resetAndAuthLocalUser({ backendUrl, user, password }); + result.package_preview = await previewPackage(backendUrl, auth.token, bytes, packagePath); + const metadata = result.package_preview.metadata || {}; + if (`${metadata.author}/${metadata.name}` !== expectedPluginId) { + throw new Error(`Fixture package metadata is ${metadata.author}/${metadata.name}, expected ${expectedPluginId}.`); + } + result.plugin_present_before = await hasPlugin(backendUrl, auth.token); + + if (!result.plugin_present_before) { + const form = new FormData(); + form.set("file", new Blob([bytes]), packagePath.split("/").pop()); + const response = await fetch(`${backendUrl.replace(/\/$/, "")}/api/v1/plugins/install/local`, { + method: "POST", + headers: { Authorization: `Bearer ${auth.token}` }, + body: form, + }); + const json = await response.json().catch(() => ({})); + if (response.status >= 400 || json.code !== 0) { + throw new Error(json.msg || `Plugin install request failed with HTTP ${response.status}.`); + } + result.task_id = json.data?.task_id ?? null; + if (!result.task_id) throw new Error("Plugin install response did not include task_id."); + result.task = await waitForTask(backendUrl, auth.token, result.task_id); + if (!isTaskComplete(result.task)) { + throw new Error(`Plugin install task did not complete successfully: ${JSON.stringify(result.task)}`); + } + } + + await sleep(1000); + result.plugin_present_after = await hasPlugin(backendUrl, auth.token); + if (!result.plugin_present_after) throw new Error(`${expectedPluginId} is not listed by /api/v1/plugins after install.`); + if (expectedTool) { + result.tool_names = await listToolNames(backendUrl, auth.token); + if (!result.tool_names.includes(expectedTool)) { + throw new Error(`${expectedTool} is not listed by /api/v1/tools after install.`); + } + } + if (expectedRunnerId) { + result.runner_ids = await listRunnerIds(backendUrl, auth.token); + if (!result.runner_ids.includes(expectedRunnerId)) { + throw new Error(`${expectedRunnerId} is not listed by /api/v1/pipelines/_/metadata after install.`); + } + } + + result.status = "pass"; + result.reason = `${expectedPluginId} is installed.`; +} catch (error) { + result.status = "fail"; + result.reason = error.message; +} finally { + await writeResult(paths, result); + console.log(JSON.stringify(result, null, 2)); +} + +process.exit(result.status === "pass" ? 0 : 1); + +async function hasPlugin(backendUrl, token) { + const response = await apiJson(backendUrl, "/api/v1/plugins", { token }); + const plugins = response.json.data?.plugins || []; + return plugins.some((plugin) => { + const metadata = plugin.manifest?.manifest?.metadata || plugin.manifest?.metadata || plugin.metadata || {}; + return `${metadata.author}/${metadata.name}` === expectedPluginId; + }); +} + +async function previewPackage(backendUrl, token, bytes, packagePath) { + const form = new FormData(); + form.set("file", new Blob([bytes]), packagePath.split("/").pop()); + const response = await fetch(`${backendUrl.replace(/\/$/, "")}/api/v1/plugins/install/local/preview`, { + method: "POST", + headers: { Authorization: `Bearer ${token}` }, + body: form, + }); + const json = await response.json().catch(() => ({})); + if (response.status >= 400 || json.code !== 0) { + throw new Error(json.msg || `Plugin package preview failed with HTTP ${response.status}.`); + } + return { + metadata: json.data?.metadata || {}, + component_types: json.data?.component_types || [], + file_count: json.data?.file_count ?? null, + }; +} + +async function listToolNames(backendUrl, token) { + const response = await apiJson(backendUrl, "/api/v1/tools", { token }); + return (response.json.data?.tools || []) + .map((tool) => tool.name || tool.tool_name || tool.function?.name || "") + .filter(Boolean) + .sort(); +} + +async function listRunnerIds(backendUrl, token) { + const response = await apiJson(backendUrl, "/api/v1/pipelines/_/metadata", { token }); + const configs = response.json.data?.configs || []; + return configs + .flatMap((section) => section.stages || []) + .flatMap((stage) => stage.config || []) + .filter((item) => item.name === "id") + .flatMap((item) => item.options || []) + .map((option) => option.name || option.value || option.id || "") + .filter(Boolean) + .sort(); +} + +async function waitForTask(backendUrl, token, taskId) { + const deadline = Date.now() + Number(env.LANGBOT_PLUGIN_INSTALL_TIMEOUT_MS || 120000); + let last = null; + while (Date.now() < deadline) { + const response = await apiJson(backendUrl, `/api/v1/system/tasks/${encodeURIComponent(taskId)}`, { token }); + last = response.json.data || response.json; + if (isTaskComplete(last) || isTaskFailed(last)) return last; + await sleep(1000); + } + return last; +} + +function isTaskComplete(task) { + const status = String(task?.status || task?.state || "").toLowerCase(); + const runtimeStatus = String(task?.runtime?.status || task?.runtime?.state || "").toLowerCase(); + return ["done", "completed", "success", "succeeded", "finished"].includes(status) + || ["done", "completed", "success", "succeeded", "finished"].includes(runtimeStatus) + || task?.done === true + || task?.completed === true + || (task?.runtime?.done === true && !task?.runtime?.exception); +} + +function isTaskFailed(task) { + const status = String(task?.status || task?.state || "").toLowerCase(); + const runtimeStatus = String(task?.runtime?.status || task?.runtime?.state || "").toLowerCase(); + return ["failed", "error", "cancelled", "canceled"].includes(status) + || ["failed", "error", "cancelled", "canceled"].includes(runtimeStatus) + || task?.failed === true + || Boolean(task?.error) + || Boolean(task?.runtime?.exception); +} + +function sleep(ms) { + return new Promise((resolve) => setTimeout(resolve, ms)); +} diff --git a/skills/scripts/e2e/langrag-kb-retrieve.mjs b/skills/scripts/e2e/langrag-kb-retrieve.mjs new file mode 100644 index 000000000..a2b489a60 --- /dev/null +++ b/skills/scripts/e2e/langrag-kb-retrieve.mjs @@ -0,0 +1,134 @@ +#!/usr/bin/env node + +import { + bodyText, + createBrowser, + ensureEvidence, + evidencePaths, + exitCode, + isLoginUrl, + localIsoWithOffset, + safeScreenshot, + writeResult, +} from "./lib/langbot-e2e.mjs"; + +const caseId = process.env.LBS_CASE_ID || "langrag-kb-retrieve"; +const paths = evidencePaths(caseId); +await ensureEvidence(paths); + +const startedAt = new Date(); +const frontendUrl = process.env.LANGBOT_FRONTEND_URL || ""; +const backendUrl = process.env.LANGBOT_BACKEND_URL || ""; +const kbUuid = process.env.LANGBOT_LOCAL_AGENT_RAG_KB_UUID || process.env.LANGBOT_RAG_KB_UUID || ""; +const query = process.env.LANGBOT_E2E_RETRIEVE_QUERY || "What is the local agent runner retrieval sentinel?"; +const expectedText = process.env.LANGBOT_E2E_EXPECTED_TEXT || "azalea-cobalt-7421"; + +let browser; +const result = { + source: "automation", + case_id: caseId, + run_id: paths.runId, + started_at: startedAt.toISOString(), + started_at_local: localIsoWithOffset(startedAt), + finished_at: "", + finished_at_local: "", + status: "fail", + reason: "", + url: "", + kb_uuid: kbUuid, + query, + expected_text: expectedText, + evidence: { + console_log: paths.consoleLog, + network_log: paths.networkLog, + screenshot: paths.screenshot, + automation_result_json: paths.automationResultJson, + result_json: paths.resultJson, + }, + evidence_collected: ["ui", "screenshot", "console", "network", "api_diagnostic"], +}; + +try { + if (!frontendUrl) throw new Error("LANGBOT_FRONTEND_URL is not configured."); + if (!backendUrl) throw new Error("LANGBOT_BACKEND_URL is not configured."); + if (!kbUuid) throw new Error("LANGBOT_LOCAL_AGENT_RAG_KB_UUID or LANGBOT_RAG_KB_UUID is required."); + + browser = await createBrowser(paths); + const { page } = browser; + await page.goto(`${frontendUrl.replace(/\/$/, "")}/home/knowledge`, { waitUntil: "domcontentloaded" }); + await page.waitForLoadState("networkidle", { timeout: 10_000 }).catch(() => {}); + result.url = page.url(); + + const text = await bodyText(page); + if (isLoginUrl(page.url()) || /登录|Login|Sign in/i.test(text)) { + result.status = "blocked"; + result.reason = "Browser profile is not authenticated for LANGBOT_FRONTEND_URL."; + } else if (!/Knowledge|知识库|qa-local-agent-rag/i.test(text)) { + result.status = "fail"; + result.reason = "Knowledge page opened, but no Knowledge UI signal or QA KB name was visible."; + } else { + const retrieve = await page.evaluate(async ({ backendUrl, kbUuid, query }) => { + const token = localStorage.getItem("token"); + if (!token) { + return { status: "blocked", authenticated: false, reason: "Browser profile has no localStorage token." }; + } + const response = await fetch(`${backendUrl}/api/v1/knowledge/bases/${encodeURIComponent(kbUuid)}/retrieve`, { + method: "POST", + headers: { + Authorization: `Bearer ${token}`, + "Content-Type": "application/json", + }, + body: JSON.stringify({ query }), + }); + const json = await response.json().catch(() => ({})); + return { + status: response.status >= 400 ? "fail" : "ready", + authenticated: true, + http_status: response.status, + code: json.code ?? null, + msg: json.msg || "", + results: json.data?.results || [], + }; + }, { backendUrl, kbUuid, query }); + + result.retrieve = { + ...retrieve, + results: Array.isArray(retrieve.results) + ? retrieve.results.map((item) => ({ + score: item.score ?? item.distance ?? null, + text: String(item.text || item.content || "").slice(0, 500), + metadata: item.metadata || {}, + })) + : [], + }; + + const resultText = JSON.stringify(result.retrieve.results || []); + if (retrieve.status === "blocked") { + result.status = "blocked"; + result.reason = retrieve.reason || "Retrieve API blocked."; + } else if (retrieve.status === "fail") { + result.status = "fail"; + result.reason = retrieve.msg || "Retrieve API failed."; + } else if (!resultText.includes(expectedText)) { + result.status = "fail"; + result.reason = `Retrieve results did not contain expected text: ${expectedText}`; + } else { + result.status = "pass"; + result.reason = `Knowledge retrieve returned expected sentinel: ${expectedText}`; + } + } + + await safeScreenshot(page, paths.screenshot); +} catch (error) { + result.status = /Playwright is not installed|not configured|required/.test(error.message) ? "env_issue" : "fail"; + result.reason = error.message; +} finally { + if (browser) await browser.close().catch(() => {}); + const finishedAt = new Date(); + result.finished_at = finishedAt.toISOString(); + result.finished_at_local = localIsoWithOffset(finishedAt); + await writeResult(paths, result); + console.log(JSON.stringify(result, null, 2)); +} + +process.exit(exitCode(result.status)); diff --git a/skills/scripts/e2e/lib/debug-chat.mjs b/skills/scripts/e2e/lib/debug-chat.mjs new file mode 100644 index 000000000..7d44d00c9 --- /dev/null +++ b/skills/scripts/e2e/lib/debug-chat.mjs @@ -0,0 +1,416 @@ +import { + bodyText, + clickFirstVisible, + countOccurrences, + gotoFrontend, + isLoginUrl, +} from "./langbot-e2e.mjs"; + +export const DEBUG_CHAT_FAILURE_SIGNALS = [ + "Agent runner temporarily unavailable", + "All models failed during streaming setup", + "调用超时", + "超时", +]; + +export function minExpectedOccurrences(beforeText, expectedText, prompt) { + const beforeCount = countOccurrences(beforeText, expectedText); + return beforeCount + (String(prompt).includes(expectedText) ? 2 : 1); +} + +export function latestExpectedLeafMatches(latestExpectedLeaf, prompt) { + return Boolean(latestExpectedLeaf) + && latestExpectedLeaf !== prompt + && !String(latestExpectedLeaf).includes(prompt); +} + +export function findNewFailureSignal(beforeText, afterText, failureSignals = DEBUG_CHAT_FAILURE_SIGNALS) { + return failureSignals.find((signal) => countOccurrences(afterText, signal) > countOccurrences(beforeText, signal)) || ""; +} + +function findFailureSignalInText(text, failureSignals = DEBUG_CHAT_FAILURE_SIGNALS) { + return failureSignals.find((signal) => String(text || "").includes(signal)) || ""; +} + +function countExpectedInMessages(messages, expectedText) { + return messages + .filter((message) => message.role === "assistant") + .reduce((count, message) => count + countOccurrences(message.text, expectedText), 0); +} + +function debugChatInput(page) { + return page + .locator('input[placeholder*="message"], input[placeholder*="消息"], textarea[placeholder*="message"], textarea[placeholder*="消息"]') + .last(); +} + +async function clickDebugChatTab(page) { + const tabByRole = page.getByRole("tab", { name: /Debug Chat|调试聊天|调试对话|Debug|调试/i }).first(); + if (await tabByRole.isVisible({ timeout: 3_000 }).catch(() => false)) { + await tabByRole.click(); + return true; + } + + const tabBySelector = page.locator('[role="tab"]').filter({ hasText: /Debug Chat|调试聊天|调试对话|Debug|调试/i }).first(); + if (await tabBySelector.isVisible({ timeout: 2_000 }).catch(() => false)) { + await tabBySelector.click(); + return true; + } + + return Boolean(await clickFirstVisible(page, ["Debug Chat", "调试聊天", "调试对话"], 2_000)); +} + +async function waitForDebugChatReady(page, timeout = 20_000) { + const input = debugChatInput(page); + const visible = await input.isVisible({ timeout }).catch(() => false); + if (!visible) { + return { + ready: false, + reason: "Debug Chat tab was clicked, but the Debug Chat input did not become visible.", + }; + } + + const enabled = await input.isEnabled({ timeout }).catch(() => false); + if (!enabled) { + return { + ready: false, + reason: "Debug Chat input is visible but disabled; WebSocket may not be connected.", + }; + } + + return { ready: true, reason: "" }; +} + +export function classifyDebugChatResult({ + beforeText, + afterText, + expectedText, + prompt, + latestExpectedLeaf, + latestFailureLeaf, + beforeMessages = null, + afterMessages = null, + latestAssistantText = "", + failureSignals = DEBUG_CHAT_FAILURE_SIGNALS, +}) { + const minExpectedCount = minExpectedOccurrences(beforeText, expectedText, prompt); + const finalCount = countOccurrences(afterText, expectedText); + const failureText = findNewFailureSignal(beforeText, afterText, failureSignals); + const promptContainsExpected = String(prompt).includes(expectedText); + const hasMessageEvidence = Array.isArray(beforeMessages) && Array.isArray(afterMessages); + const beforeAssistantExpectedCount = hasMessageEvidence + ? countExpectedInMessages(beforeMessages, expectedText) + : null; + const afterAssistantExpectedCount = hasMessageEvidence + ? countExpectedInMessages(afterMessages, expectedText) + : null; + const assistantExpectedIncreased = hasMessageEvidence + ? afterAssistantExpectedCount > beforeAssistantExpectedCount + : false; + + if (hasMessageEvidence) { + const latestAssistantFailure = findFailureSignalInText(latestAssistantText, failureSignals); + if (latestAssistantFailure) { + return { + status: "fail", + reason: `Debug Chat displayed a known failure signal in the latest assistant message: ${latestAssistantFailure}`, + min_expected_count: minExpectedCount, + final_count: finalCount, + failure_signal: latestAssistantFailure, + before_assistant_expected_count: beforeAssistantExpectedCount, + after_assistant_expected_count: afterAssistantExpectedCount, + }; + } + if (assistantExpectedIncreased && String(latestAssistantText).includes(expectedText)) { + return { + status: "pass", + reason: `Expected text appeared in a new assistant message: ${expectedText}`, + min_expected_count: minExpectedCount, + final_count: finalCount, + before_assistant_expected_count: beforeAssistantExpectedCount, + after_assistant_expected_count: afterAssistantExpectedCount, + }; + } + if (failureText) { + return { + status: "fail", + reason: `Debug Chat displayed a known failure signal: ${failureText}`, + min_expected_count: minExpectedCount, + final_count: finalCount, + failure_signal: failureText, + before_assistant_expected_count: beforeAssistantExpectedCount, + after_assistant_expected_count: afterAssistantExpectedCount, + }; + } + return { + status: "fail", + reason: `Expected text did not appear in a new assistant message. Expected assistant occurrences to increase above ${beforeAssistantExpectedCount}, saw ${afterAssistantExpectedCount}.`, + min_expected_count: minExpectedCount, + final_count: finalCount, + before_assistant_expected_count: beforeAssistantExpectedCount, + after_assistant_expected_count: afterAssistantExpectedCount, + }; + } + if (failureText) { + return { + status: "fail", + reason: `Debug Chat displayed a known failure signal: ${failureText}`, + min_expected_count: minExpectedCount, + final_count: finalCount, + failure_signal: failureText, + before_assistant_expected_count: beforeAssistantExpectedCount, + after_assistant_expected_count: afterAssistantExpectedCount, + }; + } + if (latestExpectedLeafMatches(latestExpectedLeaf, prompt) && finalCount >= minExpectedCount) { + return { + status: "pass", + reason: `Expected text appeared in the latest visible response leaf: ${expectedText}`, + min_expected_count: minExpectedCount, + final_count: finalCount, + }; + } + if (!promptContainsExpected && finalCount >= minExpectedCount) { + return { + status: "pass", + reason: `Expected text appeared enough times for user prompt plus bot response: ${expectedText}`, + min_expected_count: minExpectedCount, + final_count: finalCount, + }; + } + return { + status: "fail", + reason: `Bot response did not appear. Expected ${minExpectedCount} occurrences of ${expectedText}, saw ${finalCount}.`, + min_expected_count: minExpectedCount, + final_count: finalCount, + }; +} + +export async function openPipelineDebugChat(page, { pipelineUrl, pipelineName, envHint = "LANGBOT_PIPELINE_URL or LANGBOT_PIPELINE_NAME" }) { + if (pipelineUrl) { + await page.goto(pipelineUrl, { waitUntil: "domcontentloaded" }); + await page.waitForLoadState("networkidle", { timeout: 10_000 }).catch(() => {}); + } else { + if (!pipelineName) { + return { + opened: false, + status: "blocked", + reason: `Set ${envHint} before running pipeline-debug-chat automation.`, + }; + } + await gotoFrontend(page); + if (isLoginUrl(page.url())) { + return { + opened: false, + status: "blocked", + reason: "Browser profile is not authenticated for LANGBOT_FRONTEND_URL.", + }; + } + const clickedPipelines = await clickFirstVisible(page, ["Pipelines", "流水线"], 4_000); + if (!clickedPipelines) { + return { opened: false, status: "fail", reason: "Could not find Pipelines navigation." }; + } + await page.waitForLoadState("networkidle", { timeout: 10_000 }).catch(() => {}); + const clickedPipeline = await clickFirstVisible(page, [pipelineName], 5_000); + if (!clickedPipeline) { + return { opened: false, status: "blocked", reason: `Could not find pipeline named ${pipelineName}.` }; + } + } + + if (isLoginUrl(page.url())) { + return { + opened: false, + status: "blocked", + reason: "Browser profile is not authenticated for LANGBOT_FRONTEND_URL.", + }; + } + + const clickedDebug = await clickDebugChatTab(page); + if (!clickedDebug) { + return { opened: false, status: "fail", reason: "Could not find the Debug Chat tab." }; + } + await page.waitForLoadState("networkidle", { timeout: 10_000 }).catch(() => {}); + const ready = await waitForDebugChatReady(page); + if (!ready.ready) { + return { opened: false, status: "fail", reason: ready.reason }; + } + return { opened: true }; +} + +export async function latestVisibleLeafText(page, needles) { + return await page.evaluate((items) => { + const isVisible = (element) => { + const style = window.getComputedStyle(element); + const rect = element.getBoundingClientRect(); + return style.visibility !== "hidden" + && style.display !== "none" + && rect.width > 0 + && rect.height > 0; + }; + const leaves = []; + for (const element of document.body.querySelectorAll("*")) { + if (!isVisible(element)) continue; + const text = element.innerText?.trim(); + if (!text || text.length > 4000) continue; + const visibleChildHasText = Array.from(element.children).some((child) => ( + isVisible(child) && child.innerText?.trim() + )); + if (visibleChildHasText) continue; + if (!items.some((needle) => text.includes(needle))) continue; + leaves.push(text); + } + return leaves.at(-1) || ""; + }, needles); +} + +export async function visibleDebugChatMessages(page) { + return await page.evaluate(() => { + const isVisible = (element) => { + const style = window.getComputedStyle(element); + const rect = element.getBoundingClientRect(); + return style.visibility !== "hidden" + && style.display !== "none" + && rect.width > 0 + && rect.height > 0; + }; + const classText = (element) => String(element.getAttribute("class") || ""); + return Array.from(document.querySelectorAll("div.max-w-3xl")) + .filter((element) => isVisible(element)) + .map((element) => { + const row = element.parentElement; + const text = element.innerText?.trim() || ""; + const isUser = classText(element).includes("user-message-bubble") + || classText(row).includes("justify-end"); + return { + role: isUser ? "user" : "assistant", + text, + }; + }) + .filter((message) => message.text); + }); +} + +export async function waitForExpectedDebugChatText(page, { expectedText, minExpectedCount, timeoutMs }) { + await page.waitForFunction( + ({ expected, min }) => { + return document.body.innerText.split(expected).length - 1 >= min; + }, + { expected: expectedText, min: minExpectedCount }, + { timeout: timeoutMs }, + ).catch(() => {}); +} + +export async function waitForDebugChatTextStable(page, { timeoutMs = 5_000, quietMs = 750 } = {}) { + const startedAt = Date.now(); + let lastText = await bodyText(page); + let stableSince = Date.now(); + while (Date.now() - startedAt < timeoutMs) { + await page.waitForTimeout(250); + const currentText = await bodyText(page); + if (currentText !== lastText) { + lastText = currentText; + stableSince = Date.now(); + continue; + } + if (Date.now() - stableSince >= quietMs) return; + } +} + +export async function attachDebugChatImage(page, imagePath) { + if (!imagePath) return { status: "not_required", reason: "" }; + const input = page.locator('input[type="file"][accept*="image"], input[type="file"]').first(); + if (!await input.count()) { + return { status: "fail", reason: "Could not find a Debug Chat image upload input." }; + } + await input.setInputFiles(imagePath); + await page.locator("img").last().waitFor({ state: "visible", timeout: 10_000 }).catch(() => {}); + return { status: "ready", reason: `Attached image fixture: ${imagePath}` }; +} + +export async function sendDebugChatPrompt(page, prompt, imagePath = "") { + const imageResult = await attachDebugChatImage(page, imagePath); + if (imageResult.status === "fail") return imageResult; + + const input = debugChatInput(page); + const inputVisible = await input.isVisible({ timeout: 5_000 }).catch(() => false); + const inputEnabled = inputVisible && await input.isEnabled({ timeout: 10_000 }).catch(() => false); + if (!inputVisible || !inputEnabled) return false; + await input.fill(prompt).catch(async () => { + await input.click(); + await input.pressSequentially(prompt); + }); + const clickedSend = await clickFirstVisible(page, ["Send", "发送", "提交"], 1_500); + if (!clickedSend) await page.keyboard.press("Enter"); + await page.getByText(prompt, { exact: false }).last().waitFor({ state: "visible", timeout: 10_000 }).catch(() => {}); + return true; +} + +export async function runDebugChatPrompt(page, { prompt, expectedText, responseTimeoutMs, imagePath = "", failureSignals = DEBUG_CHAT_FAILURE_SIGNALS }) { + const beforeText = await bodyText(page); + const beforeMessages = await visibleDebugChatMessages(page); + const minExpectedCount = minExpectedOccurrences(beforeText, expectedText, prompt); + const sent = await sendDebugChatPrompt(page, prompt, imagePath); + if (sent !== true) { + if (sent && typeof sent === "object" && typeof sent.reason === "string") return sent; + return { status: "fail", reason: "Could not find a Debug Chat text input." }; + } + + await waitForExpectedDebugChatText(page, { + expectedText, + minExpectedCount, + prompt, + timeoutMs: responseTimeoutMs, + }); + await waitForDebugChatTextStable(page); + + const afterText = await bodyText(page); + const afterMessages = await visibleDebugChatMessages(page); + const latestAssistantText = afterMessages.filter((message) => message.role === "assistant").at(-1)?.text || ""; + const latestExpectedLeaf = await latestVisibleLeafText(page, [expectedText]); + const failureText = findNewFailureSignal(beforeText, afterText, failureSignals); + const latestFailureLeaf = failureText ? await latestVisibleLeafText(page, [failureText]) : ""; + + return classifyDebugChatResult({ + beforeText, + afterText, + expectedText, + prompt, + latestExpectedLeaf, + latestFailureLeaf, + beforeMessages, + afterMessages, + latestAssistantText, + failureSignals, + }); +} + +export async function setDebugChatStreamOutput(page, desired) { + if (desired === null || desired === undefined) return { status: "not_required", reason: "" }; + + const streamSwitch = page.locator('[role="switch"]').first(); + if (!await streamSwitch.isVisible({ timeout: 5_000 }).catch(() => false)) { + return { status: "blocked", reason: "Debug Chat stream switch was not visible." }; + } + if (!await streamSwitch.isEnabled({ timeout: 10_000 }).catch(() => false)) { + return { status: "blocked", reason: "Debug Chat stream switch was visible but disabled." }; + } + + const checked = (await streamSwitch.getAttribute("aria-checked").catch(() => null)) === "true"; + if (checked !== desired) { + await streamSwitch.click(); + await page.waitForFunction( + ({ selector, expected }) => document.querySelector(selector)?.getAttribute("aria-checked") === String(expected), + { selector: '[role="switch"]', expected: desired }, + { timeout: 5_000 }, + ).catch(() => {}); + } + + const finalChecked = (await streamSwitch.getAttribute("aria-checked").catch(() => null)) === "true"; + if (finalChecked !== desired) { + return { + status: "fail", + reason: `Debug Chat stream switch did not reach requested state: ${desired ? "on" : "off"}.`, + }; + } + return { status: "ready", reason: `Debug Chat stream switch is ${desired ? "on" : "off"}.` }; +} diff --git a/skills/scripts/e2e/lib/langbot-e2e.mjs b/skills/scripts/e2e/lib/langbot-e2e.mjs new file mode 100644 index 000000000..fc7a52e4f --- /dev/null +++ b/skills/scripts/e2e/lib/langbot-e2e.mjs @@ -0,0 +1,341 @@ +import { appendFile, mkdir, readFile, stat, writeFile } from "node:fs/promises"; +import { join, resolve } from "node:path"; +import { env } from "node:process"; + +const secretRe = /(?:authorization|bearer|token|secret|password|api[_-]?key|jwt|oauth)\s*[:=]\s*["']?[^"',\s]+/gi; + +export function redact(text) { + return String(text ?? "") + .replace(secretRe, (match) => match.replace(/[:=]\s*["']?.*$/, "=[redacted]")) + .replace(/\bbearer\s+[A-Za-z0-9._~+/=-]{8,}/gi, "Bearer [redacted]") + .replace(/\bsk-[A-Za-z0-9_-]{6,}\b/g, "[redacted]"); +} + +export function timestampSlug(date = new Date()) { + return date.toISOString().replace(/\.\d{3}Z$/, "Z").replace(/[^0-9A-Za-z]+/g, "-").replace(/^-|-$/g, ""); +} + +export function localIsoWithOffset(date = new Date()) { + const offsetMinutes = -date.getTimezoneOffset(); + const sign = offsetMinutes >= 0 ? "+" : "-"; + const absolute = Math.abs(offsetMinutes); + const pad = (value) => String(value).padStart(2, "0"); + const yyyy = date.getFullYear(); + const mm = pad(date.getMonth() + 1); + const dd = pad(date.getDate()); + const hh = pad(date.getHours()); + const mi = pad(date.getMinutes()); + const ss = pad(date.getSeconds()); + const ms = String(date.getMilliseconds()).padStart(3, "0"); + return `${yyyy}-${mm}-${dd}T${hh}:${mi}:${ss}.${ms}${sign}${pad(Math.floor(absolute / 60))}:${pad(absolute % 60)}`; +} + +export function evidencePaths(caseId) { + const runId = env.LBS_RUN_ID || `${timestampSlug()}-${caseId}`; + const evidenceDir = resolve(env.LBS_EVIDENCE_DIR || join("reports", "evidence", runId)); + return { + runId, + evidenceDir, + consoleLog: join(evidenceDir, "console.log"), + networkLog: join(evidenceDir, "network.log"), + screenshot: join(evidenceDir, "screenshot.png"), + automationResultJson: join(evidenceDir, "automation-result.json"), + resultJson: join(evidenceDir, "result.json"), + }; +} + +export async function ensureEvidence(paths) { + await mkdir(paths.evidenceDir, { recursive: true }); + await appendFile(paths.consoleLog, "", "utf8"); + await appendFile(paths.networkLog, "", "utf8"); +} + +export async function pathExists(path) { + try { + await stat(path); + return true; + } catch { + return false; + } +} + +export async function appendLine(path, line) { + await appendFile(path, `[${localIsoWithOffset()}] ${redact(line)}\n`, "utf8"); +} + +export async function writeResult(paths, result) { + const text = `${JSON.stringify(result, null, 2)}\n`; + if (paths.automationResultJson) await writeFile(paths.automationResultJson, text, "utf8"); + if (paths.resultJson && paths.resultJson !== paths.automationResultJson) { + await writeFile(paths.resultJson, text, "utf8"); + } +} + +export async function loadEnvFiles(paths = ["skills/.env", "skills/.env.local"]) { + for (const path of paths) { + let text = ""; + try { + text = await readFile(path, "utf8"); + } catch { + continue; + } + for (const line of text.split(/\r?\n/)) { + const trimmed = line.trim(); + if (!trimmed || trimmed.startsWith("#")) continue; + const equals = trimmed.indexOf("="); + if (equals <= 0) continue; + const key = trimmed.slice(0, equals).trim(); + const value = trimmed.slice(equals + 1).trim().replace(/^["']|["']$/g, ""); + if (!(key in env)) env[key] = value; + } + } +} + +export async function readRecoveryKey(repo = env.LANGBOT_REPO || "../LangBot") { + const configPath = resolve(repo, "data/config.yaml"); + const config = await readFile(configPath, "utf8"); + const match = config.match(/^\s*recovery_key:\s*['"]?([^'"\s#]+)['"]?\s*$/m); + return match?.[1] || ""; +} + +export async function apiJson(backendUrl, path, { method = "GET", token = "", body } = {}) { + const headers = { "Content-Type": "application/json" }; + if (token) headers.Authorization = `Bearer ${token}`; + const response = await fetch(`${backendUrl.replace(/\/$/, "")}${path}`, { + method, + headers, + body: body === undefined ? undefined : JSON.stringify(body), + }); + return { + status: response.status, + json: await response.json().catch(() => ({})), + }; +} + +export async function checkBackendToken(backendUrl, token) { + if (!token) { + return { authenticated: false, http_status: 0, code: null, reason: "No token." }; + } + const response = await apiJson(backendUrl, "/api/v1/user/check-token", { token }); + const code = response.json.code ?? null; + const authenticated = response.status < 400 && code === 0; + return { + authenticated, + http_status: response.status, + code, + reason: authenticated ? "Token accepted by backend." : response.json.msg || "Backend rejected token.", + }; +} + +export async function resetAndAuthLocalUser({ backendUrl, user, password, recoveryKey = "" }) { + const key = recoveryKey || await readRecoveryKey(); + if (!key) throw new Error("Could not read recovery_key from LangBot config."); + + const reset = await apiJson(backendUrl, "/api/v1/user/reset-password", { + method: "POST", + body: { + user, + recovery_key: key, + new_password: password, + }, + }); + if (reset.status >= 400 || reset.json.code !== 0) { + throw new Error(reset.json.msg || `Password reset failed with HTTP ${reset.status}.`); + } + + const auth = await apiJson(backendUrl, "/api/v1/user/auth", { + method: "POST", + body: { user, password }, + }); + const token = auth.json.data?.token || ""; + if (auth.status >= 400 || auth.json.code !== 0 || !token) { + throw new Error(auth.json.msg || `Auth failed with HTTP ${auth.status}.`); + } + + const check = await checkBackendToken(backendUrl, token); + if (!check.authenticated) { + throw new Error(check.reason || "Authenticated token failed backend token check."); + } + + return { token, check }; +} + +export async function setBrowserToken(page, frontendUrl, token) { + await page.addInitScript((value) => { + localStorage.setItem("token", value); + }, token); + await page.goto(frontendUrl, { waitUntil: "domcontentloaded" }); + await page.evaluate((value) => localStorage.setItem("token", value), token); +} + +export async function verifyBrowserToken(page, backendUrl) { + return await page.evaluate(async (baseUrl) => { + const token = localStorage.getItem("token"); + if (!token) { + return { authenticated: false, http_status: 0, code: null, reason: "No localStorage token." }; + } + try { + const response = await fetch(`${baseUrl.replace(/\/$/, "")}/api/v1/user/check-token`, { + headers: { + Authorization: `Bearer ${token}`, + "Content-Type": "application/json", + }, + }); + const json = await response.json().catch(() => ({})); + const code = json.code ?? null; + const authenticated = response.status < 400 && code === 0; + return { + authenticated, + http_status: response.status, + code, + reason: authenticated ? "Token accepted by backend." : json.msg || "Backend rejected token.", + }; + } catch (error) { + return { + authenticated: false, + http_status: 0, + code: null, + reason: error.message, + }; + } + }, backendUrl); +} + +export function exitCode(status) { + if (status === "pass") return 0; + if (status === "blocked" || status === "env_issue") return 2; + return 1; +} + +export async function loadPlaywright() { + try { + return await import("playwright"); + } catch { + throw new Error( + "Playwright is not installed. Install it in this repo with `npm install --save-dev playwright`, then run `npx playwright install chromium`.", + ); + } +} + +export async function createBrowser(paths) { + const { chromium } = await loadPlaywright(); + const headed = env.LBS_HEADED === "1"; + const launchOptions = { + headless: !headed, + }; + if (env.LANGBOT_CHROMIUM_EXECUTABLE && await pathExists(env.LANGBOT_CHROMIUM_EXECUTABLE)) { + launchOptions.executablePath = env.LANGBOT_CHROMIUM_EXECUTABLE; + } + + let browser; + let context; + if (env.LANGBOT_BROWSER_PROFILE) { + context = await chromium.launchPersistentContext(resolve(env.LANGBOT_BROWSER_PROFILE), { + ...launchOptions, + viewport: { width: 1440, height: 960 }, + }); + } else { + browser = await chromium.launch(launchOptions); + context = await browser.newContext({ viewport: { width: 1440, height: 960 } }); + } + const page = context.pages()[0] || await context.newPage(); + + page.on("console", (message) => { + appendLine(paths.consoleLog, `[${message.type()}] ${message.text()}`).catch(() => {}); + }); + page.on("pageerror", (error) => { + appendLine(paths.consoleLog, `[pageerror] ${error.message}`).catch(() => {}); + }); + page.on("requestfailed", (request) => { + appendLine(paths.networkLog, `[requestfailed] ${request.method()} ${request.url()} ${request.failure()?.errorText ?? ""}`).catch(() => {}); + }); + page.on("response", (response) => { + if (response.status() < 400) return; + appendLine(paths.networkLog, `[response] ${response.status()} ${response.url()}`).catch(() => {}); + }); + + return { + page, + context, + async close() { + await context.close(); + if (browser) await browser.close(); + }, + }; +} + +export async function safeScreenshot(page, path) { + try { + await page.screenshot({ path, fullPage: true }); + } catch { + // Screenshot evidence is useful, but a screenshot failure should not hide the real test result. + } +} + +export async function gotoFrontend(page) { + const frontendUrl = env.LANGBOT_FRONTEND_URL; + if (!frontendUrl) { + throw new Error("LANGBOT_FRONTEND_URL is not configured."); + } + await page.goto(frontendUrl, { waitUntil: "domcontentloaded" }); + await page.waitForLoadState("networkidle", { timeout: 10_000 }).catch(() => {}); +} + +export function isLoginUrl(url) { + return /\/login(?:[/?#]|$)/.test(url); +} + +export async function bodyText(page) { + return await page.locator("body").innerText({ timeout: 5_000 }).catch(() => ""); +} + +export function countOccurrences(haystack, needle) { + if (!needle) return 0; + return String(haystack).split(needle).length - 1; +} + +export async function clickFirstVisible(page, labels, timeout = 2_000) { + for (const label of labels) { + const roleButton = page.getByRole("button", { name: label }).first(); + if (await roleButton.isVisible({ timeout }).catch(() => false)) { + await roleButton.click(); + return label; + } + + const roleLink = page.getByRole("link", { name: label }).first(); + if (await roleLink.isVisible({ timeout }).catch(() => false)) { + await roleLink.click(); + return label; + } + + const text = page.getByText(label, { exact: false }).first(); + if (await text.isVisible({ timeout }).catch(() => false)) { + await text.click(); + return label; + } + } + return null; +} + +export async function fillFirstTextInput(page, value) { + const candidates = [ + page.getByRole("textbox").last(), + page.locator("textarea").last(), + page.locator("[contenteditable=true]").last(), + page.locator("input[type=text]").last(), + ]; + + for (const locator of candidates) { + if (!await locator.isVisible({ timeout: 2_000 }).catch(() => false)) continue; + await locator.fill(value).catch(async () => { + await locator.click(); + await locator.pressSequentially(value); + }); + return true; + } + return false; +} + +export async function waitForVisibleText(page, text, timeout = 20_000) { + await page.getByText(text, { exact: false }).last().waitFor({ state: "visible", timeout }); +} diff --git a/skills/scripts/e2e/local-agent-steering-debug-chat.mjs b/skills/scripts/e2e/local-agent-steering-debug-chat.mjs new file mode 100644 index 000000000..dae2e265c --- /dev/null +++ b/skills/scripts/e2e/local-agent-steering-debug-chat.mjs @@ -0,0 +1,565 @@ +#!/usr/bin/env node + +import { writeFile } from "node:fs/promises"; +import { env } from "node:process"; +import { + DEBUG_CHAT_FAILURE_SIGNALS, + openPipelineDebugChat, + setDebugChatStreamOutput, + visibleDebugChatMessages, + waitForDebugChatTextStable, +} from "./lib/debug-chat.mjs"; +import { + createBrowser, + ensureEvidence, + evidencePaths, + exitCode, + localIsoWithOffset, + loadEnvFiles, + pathExists, + safeScreenshot, + writeResult, +} from "./lib/langbot-e2e.mjs"; + +await loadEnvFiles(); + +const caseId = env.LBS_CASE_ID || "local-agent-steering-debug-chat"; +const paths = evidencePaths(caseId); +await ensureEvidence(paths); + +const backendUrl = (env.LANGBOT_BACKEND_URL || "").replace(/\/$/, ""); +const pipelineUrl = env.LANGBOT_E2E_PIPELINE_URL || env.LANGBOT_LOCAL_AGENT_PIPELINE_URL || env.LANGBOT_PIPELINE_URL || ""; +const pipelineName = env.LANGBOT_E2E_PIPELINE_NAME || env.LANGBOT_LOCAL_AGENT_PIPELINE_NAME || env.LANGBOT_PIPELINE_NAME || ""; +const expectedRunnerId = env.LANGBOT_E2E_EXPECTED_RUNNER_ID || "plugin:langbot/local-agent/default"; +const expectedText = env.LANGBOT_E2E_EXPECTED_TEXT || "qa_steering_sentinel_6194"; +const responseTimeoutMs = positiveInt(env.LANGBOT_E2E_RESPONSE_TIMEOUT_MS, 240000); +const followupDelayMs = 1000; +const followupEnabledTimeoutMs = 1500; +const firstPrompt = env.LANGBOT_E2E_PROMPT || [ + "You are running the LangBot steering E2E test.", + "First call the qa_plugin_sleep tool with seconds=8 and text=steering-e2e-anchor.", + "Do not answer before the tool result is available.", + "After the tool returns, answer the latest user follow-up.", + "If no follow-up was injected, reply only STEERING_NO_FOLLOWUP.", +].join(" "); +const followupPrompt = [ + "This is a steering follow-up sent while the first tool call is still active.", + `Return only ${expectedText}.`, +].join(" "); + +const pipelineConfigDiagnosticPath = `${paths.evidenceDir}/pipeline-config-diagnostic.json`; +const debugChatResetDiagnosticPath = `${paths.evidenceDir}/debug-chat-reset-diagnostic.json`; +const toolDiagnosticPath = `${paths.evidenceDir}/tool-diagnostic.json`; + +let browser; +const result = { + source: "automation", + case_id: caseId, + run_id: paths.runId, + status: "fail", + reason: "", + started_at: new Date().toISOString(), + started_at_local: localIsoWithOffset(new Date()), + url: "", + backend_url: backendUrl, + pipeline_url: pipelineUrl, + pipeline_name: pipelineName, + expected_runner_id: expectedRunnerId, + first_prompt: firstPrompt, + followup_prompt: followupPrompt, + expected_text: expectedText, + followup_delay_ms: followupDelayMs, + followup_enabled_timeout_ms: followupEnabledTimeoutMs, + response_timeout_ms: responseTimeoutMs, + pipeline_config: null, + debug_chat_reset: null, + tool_diagnostic: null, + steering: null, + evidence: { + console_log: paths.consoleLog, + network_log: paths.networkLog, + screenshot: paths.screenshot, + automation_result_json: paths.automationResultJson, + result_json: paths.resultJson, + }, + evidence_collected: ["ui", "console", "network", "screenshot"], +}; + +try { + if (!backendUrl) { + result.status = "env_issue"; + result.reason = "LANGBOT_BACKEND_URL is required."; + throw new Error(result.reason); + } + + browser = await createBrowser(paths); + const { page } = browser; + + const openResult = await openPipelineDebugChat(page, { + pipelineUrl, + pipelineName, + envHint: "case-specific pipeline env mapped to LANGBOT_E2E_PIPELINE_URL or LANGBOT_E2E_PIPELINE_NAME", + }); + result.url = page.url(); + if (!openResult.opened) { + result.status = openResult.status; + result.reason = openResult.reason; + } else { + const pipelineDiagnostic = await inspectPipeline(page, { + backendUrl, + pipelineUrl, + pipelineName, + expectedRunnerId, + }); + await writeFile(pipelineConfigDiagnosticPath, `${JSON.stringify(pipelineDiagnostic, null, 2)}\n`, "utf8"); + result.evidence.pipeline_config_diagnostic_json = pipelineConfigDiagnosticPath; + result.pipeline_config = pipelineDiagnostic; + if (!result.evidence_collected.includes("api_diagnostic")) result.evidence_collected.push("api_diagnostic"); + + const toolDiagnostic = await inspectToolNames(page, { backendUrl }); + await writeFile(toolDiagnosticPath, `${JSON.stringify(toolDiagnostic, null, 2)}\n`, "utf8"); + result.evidence.tool_diagnostic_json = toolDiagnosticPath; + result.tool_diagnostic = toolDiagnostic; + + if (pipelineDiagnostic.status === "fail" || pipelineDiagnostic.status === "blocked") { + result.status = pipelineDiagnostic.status; + result.reason = pipelineDiagnostic.reason || "Pipeline diagnostic failed."; + } else if (toolDiagnostic.status === "fail" || toolDiagnostic.status === "blocked") { + result.status = toolDiagnostic.status; + result.reason = toolDiagnostic.reason || "Tool diagnostic failed."; + } else if (!toolDiagnostic.tool_names.includes("qa_plugin_sleep")) { + result.status = "blocked"; + result.reason = "qa_plugin_sleep is not exposed by /api/v1/tools; rebuild/reinstall qa-plugin-smoke before running steering E2E."; + } else { + const resetDiagnostic = await resetPipelineDebugChat(page, { + backendUrl, + pipelineId: pipelineDiagnostic.pipeline_id, + sessionType: "person", + }); + await writeFile(debugChatResetDiagnosticPath, `${JSON.stringify(resetDiagnostic, null, 2)}\n`, "utf8"); + result.evidence.debug_chat_reset_diagnostic_json = debugChatResetDiagnosticPath; + result.debug_chat_reset = resetDiagnostic; + + if (resetDiagnostic.status === "fail" || resetDiagnostic.status === "blocked") { + result.status = resetDiagnostic.status; + result.reason = resetDiagnostic.reason || "Debug Chat reset failed."; + } else { + await page.waitForTimeout(1000); + const reopenResult = await openPipelineDebugChat(page, { + pipelineUrl, + pipelineName, + envHint: "case-specific pipeline env mapped to LANGBOT_E2E_PIPELINE_URL or LANGBOT_E2E_PIPELINE_NAME", + }); + result.url = page.url(); + if (!reopenResult.opened) { + result.status = reopenResult.status; + result.reason = reopenResult.reason; + } else { + const streamResult = await setDebugChatStreamOutput(page, true); + if (streamResult.status === "blocked" || streamResult.status === "fail") { + result.status = streamResult.status; + result.reason = streamResult.reason; + } else { + result.steering = await runSteeringProbe(page); + result.status = result.steering.status; + result.reason = result.steering.reason; + } + } + } + } + } +} catch (error) { + if (!["env_issue", "blocked", "fail", "pass"].includes(result.status) || !result.reason) { + result.status = /Playwright is not installed|LANGBOT_FRONTEND_URL/.test(error.message) ? "env_issue" : "fail"; + } + result.reason = result.reason || error.message; +} finally { + if (browser?.page) await safeScreenshot(browser.page, paths.screenshot); + if (browser) await browser.close().catch(() => {}); + const finishedAt = new Date(); + result.finished_at = finishedAt.toISOString(); + result.finished_at_local = localIsoWithOffset(finishedAt); + const existingEvidence = {}; + for (const [key, value] of Object.entries(result.evidence)) { + if (typeof value !== "string") continue; + const isResultFile = value === paths.automationResultJson || value === paths.resultJson; + if (isResultFile || await pathExists(value)) existingEvidence[key] = value; + } + result.evidence = existingEvidence; + await writeResult(paths, result); + console.log(JSON.stringify(result, null, 2)); +} + +process.exit(exitCode(result.status)); + +async function runSteeringProbe(page) { + const beforeMessages = await visibleDebugChatMessages(page); + const beforeAssistantCount = countRole(beforeMessages, "assistant"); + const beforeUserCount = countRole(beforeMessages, "user"); + const firstStartedAt = Date.now(); + const firstSend = await sendPrompt(page, firstPrompt, { enabledTimeoutMs: 5000 }); + if (!firstSend.sent) { + return { + status: "fail", + reason: firstSend.reason || "Could not send first Debug Chat prompt.", + first_send: firstSend, + before_assistant_count: beforeAssistantCount, + before_user_count: beforeUserCount, + }; + } + + await page.waitForTimeout(followupDelayMs); + const preFollowupMessages = await visibleDebugChatMessages(page); + const preFollowupAssistantCount = countRole(preFollowupMessages, "assistant"); + const followupStartedAt = Date.now(); + const followupSend = await sendPrompt(page, followupPrompt, { enabledTimeoutMs: followupEnabledTimeoutMs }); + const followupSentAt = Date.now(); + if (!followupSend.sent) { + return { + status: "fail", + reason: followupSend.reason || "Could not send steering follow-up while the first run was active.", + first_send: firstSend, + followup_send: followupSend, + first_to_followup_attempt_ms: followupStartedAt - firstStartedAt, + followup_send_latency_ms: followupSentAt - followupStartedAt, + before_assistant_count: beforeAssistantCount, + pre_followup_assistant_count: preFollowupAssistantCount, + before_user_count: beforeUserCount, + }; + } + + const waitResult = await waitForLatestAssistantContaining(page, { + expectedText, + beforeAssistantCount, + timeoutMs: responseTimeoutMs, + }); + await waitForDebugChatTextStable(page); + const afterMessages = await visibleDebugChatMessages(page); + const afterAssistantCount = countRole(afterMessages, "assistant"); + const afterUserCount = countRole(afterMessages, "user"); + const latestAssistantText = latestRoleText(afterMessages, "assistant"); + const failureSignal = findFailureSignal(latestAssistantText) || findFailureSignal(messagesText(afterMessages)); + const newAssistantCount = afterAssistantCount - beforeAssistantCount; + const newUserCount = afterUserCount - beforeUserCount; + + const base = { + first_send: firstSend, + followup_send: followupSend, + first_to_followup_attempt_ms: followupStartedAt - firstStartedAt, + followup_send_latency_ms: followupSentAt - followupStartedAt, + before_assistant_count: beforeAssistantCount, + pre_followup_assistant_count: preFollowupAssistantCount, + after_assistant_count: afterAssistantCount, + new_assistant_count: newAssistantCount, + before_user_count: beforeUserCount, + after_user_count: afterUserCount, + new_user_count: newUserCount, + latest_assistant_text: latestAssistantText, + assistant_containing_expected_seen: waitResult.seen, + failure_signal: failureSignal, + }; + + if (failureSignal) { + return { + ...base, + status: "fail", + reason: `Debug Chat displayed a known failure signal: ${failureSignal}`, + }; + } + if (!waitResult.seen) { + return { + ...base, + status: "fail", + reason: `No new assistant message contained steering sentinel ${expectedText}.`, + }; + } + if (!latestAssistantText.includes(expectedText)) { + return { + ...base, + status: "fail", + reason: `Latest assistant message did not contain steering sentinel ${expectedText}.`, + }; + } + if (newUserCount < 2) { + return { + ...base, + status: "fail", + reason: `Expected two new user messages, saw ${newUserCount}.`, + }; + } + if (newAssistantCount !== 1) { + return { + ...base, + status: "fail", + reason: `Expected one assistant response for one claimed steering run, saw ${newAssistantCount}. More than one usually means the follow-up became a separate run.`, + }; + } + if (latestAssistantText.includes("STEERING_NO_FOLLOWUP")) { + return { + ...base, + status: "fail", + reason: "Runner answered the no-follow-up branch, so steering was not injected.", + }; + } + + return { + ...base, + status: "pass", + reason: `Follow-up sentinel ${expectedText} appeared in the only new assistant response after two user messages.`, + }; +} + +function debugChatInput(page) { + return page + .locator('input[placeholder*="message"], input[placeholder*="消息"], textarea[placeholder*="message"], textarea[placeholder*="消息"]') + .last(); +} + +async function sendPrompt(page, prompt, { enabledTimeoutMs }) { + const input = debugChatInput(page); + const inputVisible = await input.isVisible({ timeout: 5000 }).catch(() => false); + if (!inputVisible) return { sent: false, reason: "Debug Chat input is not visible." }; + const inputEnabled = await input.isEnabled({ timeout: enabledTimeoutMs }).catch(() => false); + if (!inputEnabled) return { sent: false, reason: `Debug Chat input was not enabled within ${enabledTimeoutMs}ms.` }; + + await input.fill(prompt).catch(async () => { + await input.click(); + await input.pressSequentially(prompt); + }); + await input.press("Enter"); + await page.getByText(prompt, { exact: false }).last().waitFor({ state: "visible", timeout: 10000 }).catch(() => {}); + return { + sent: true, + submitted_by: "keyboard_enter", + }; +} + +async function waitForLatestAssistantContaining(page, { expectedText, beforeAssistantCount, timeoutMs }) { + const deadline = Date.now() + timeoutMs; + let lastMessages = []; + let latestAssistantText = ""; + while (Date.now() < deadline) { + const messages = await visibleDebugChatMessages(page); + lastMessages = messages; + latestAssistantText = latestRoleText(messages, "assistant"); + if (countRole(messages, "assistant") > beforeAssistantCount && latestAssistantText.includes(expectedText)) { + return { + seen: true, + latest_assistant_text: latestAssistantText, + messages, + }; + } + const failureSignal = findFailureSignal(latestAssistantText); + if (failureSignal) { + return { + seen: false, + latest_assistant_text: latestAssistantText, + messages, + failure_signal: failureSignal, + }; + } + await page.waitForTimeout(500); + } + return { + seen: false, + latest_assistant_text: latestAssistantText, + messages: lastMessages, + }; +} + +async function inspectPipeline(page, { backendUrl, pipelineUrl, pipelineName, expectedRunnerId }) { + const pipelineIdFromUrl = pipelineIdFromUrlValue(pipelineUrl); + return await page.evaluate(async ({ backendUrl, pipelineIdFromUrl, pipelineName, expectedRunnerId }) => { + const token = localStorage.getItem("token"); + if (!token) { + return { + status: "blocked", + authenticated: false, + reason: "Browser profile has no localStorage token.", + }; + } + const getJson = async (path) => { + const response = await fetch(`${backendUrl}${path}`, { + headers: { + Authorization: `Bearer ${token}`, + "Content-Type": "application/json", + }, + }); + return { + status: response.status, + json: await response.json().catch(() => ({})), + }; + }; + + let pipelineId = pipelineIdFromUrl; + let matchedBy = pipelineId ? "url" : ""; + if (!pipelineId) { + if (!pipelineName) { + return { + status: "blocked", + authenticated: true, + pipeline_resolved: false, + reason: "Set LANGBOT_LOCAL_AGENT_PIPELINE_URL or LANGBOT_LOCAL_AGENT_PIPELINE_NAME.", + }; + } + const list = await getJson("/api/v1/pipelines"); + const pipelines = list.json.data?.pipelines || []; + const match = pipelines.find((pipeline) => pipeline.name === pipelineName); + if (!match) { + return { + status: "blocked", + authenticated: true, + pipeline_resolved: false, + list_status: list.status, + reason: `Could not find pipeline named ${pipelineName}.`, + }; + } + pipelineId = match.uuid; + matchedBy = "name"; + } + + const loaded = await getJson(`/api/v1/pipelines/${encodeURIComponent(pipelineId)}`); + const pipeline = loaded.json.data?.pipeline; + if (loaded.status >= 400 || !pipeline) { + return { + status: "fail", + authenticated: true, + pipeline_resolved: false, + pipeline_id: pipelineId, + get_status: loaded.status, + reason: loaded.json.msg || "Could not load pipeline.", + }; + } + const config = pipeline.config || {}; + const runner = config.ai?.runner || {}; + const runnerId = runner.id || runner.runner || ""; + if (!runnerId) { + return { + status: "blocked", + authenticated: true, + pipeline_resolved: true, + pipeline_id: pipelineId, + pipeline_name: pipeline.name, + matched_by: matchedBy, + reason: "Pipeline has no ai.runner.id or legacy ai.runner.runner.", + }; + } + if (expectedRunnerId && runnerId !== expectedRunnerId) { + return { + status: "blocked", + authenticated: true, + pipeline_resolved: true, + pipeline_id: pipelineId, + pipeline_name: pipeline.name, + matched_by: matchedBy, + runner_id: runnerId, + expected_runner_id: expectedRunnerId, + reason: `Pipeline runner mismatch: expected ${expectedRunnerId}, got ${runnerId}.`, + }; + } + return { + status: "ready", + authenticated: true, + pipeline_resolved: true, + pipeline_id: pipelineId, + pipeline_name: pipeline.name, + matched_by: matchedBy, + runner_id: runnerId, + expected_runner_id: expectedRunnerId || "", + }; + }, { backendUrl, pipelineIdFromUrl, pipelineName, expectedRunnerId }); +} + +async function inspectToolNames(page, { backendUrl }) { + return await page.evaluate(async ({ backendUrl }) => { + const token = localStorage.getItem("token"); + if (!token) { + return { + status: "blocked", + authenticated: false, + tool_names: [], + reason: "Browser profile has no localStorage token.", + }; + } + const response = await fetch(`${backendUrl}/api/v1/tools`, { + headers: { + Authorization: `Bearer ${token}`, + "Content-Type": "application/json", + }, + }); + const json = await response.json().catch(() => ({})); + const toolNames = (json.data?.tools || []) + .map((tool) => tool.name || tool.tool_name || tool.function?.name || "") + .filter(Boolean) + .sort(); + return { + status: response.status >= 400 ? "fail" : "ready", + authenticated: true, + http_status: response.status, + code: json.code ?? null, + tool_names: toolNames, + reason: response.status >= 400 ? json.msg || "Could not list tools." : "Tool list loaded.", + }; + }, { backendUrl }); +} + +async function resetPipelineDebugChat(page, { backendUrl, pipelineId, sessionType }) { + return await page.evaluate(async ({ backendUrl, pipelineId, sessionType }) => { + const token = localStorage.getItem("token"); + if (!token) { + return { + status: "blocked", + authenticated: false, + pipeline_id: pipelineId, + session_type: sessionType, + reason: "Browser profile has no localStorage token.", + }; + } + const response = await fetch( + `${backendUrl}/api/v1/pipelines/${encodeURIComponent(pipelineId)}/ws/reset/${encodeURIComponent(sessionType)}`, + { + method: "POST", + headers: { + Authorization: `Bearer ${token}`, + "Content-Type": "application/json", + }, + }, + ); + const json = await response.json().catch(() => ({})); + return { + status: response.status >= 400 ? "fail" : "ready", + authenticated: true, + pipeline_id: pipelineId, + session_type: sessionType, + reset_status: response.status, + reset_code: json.code ?? null, + reason: response.status >= 400 ? json.msg || "Debug Chat reset failed." : "Debug Chat session reset.", + }; + }, { backendUrl, pipelineId, sessionType }); +} + +function pipelineIdFromUrlValue(value) { + const match = String(value || "").match(/\/pipelines?\/([^/?#]+)/i); + return match ? decodeURIComponent(match[1]) : ""; +} + +function countRole(messages, role) { + return messages.filter((message) => message.role === role).length; +} + +function latestRoleText(messages, role) { + return messages.filter((message) => message.role === role).at(-1)?.text || ""; +} + +function messagesText(messages) { + return messages.map((message) => message.text).join("\n"); +} + +function findFailureSignal(text) { + return DEBUG_CHAT_FAILURE_SIGNALS.find((signal) => String(text || "").includes(signal)) || ""; +} + +function positiveInt(value, fallback) { + const parsed = Number.parseInt(String(value || ""), 10); + return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback; +} diff --git a/skills/scripts/e2e/mcp-stdio-fixture.mjs b/skills/scripts/e2e/mcp-stdio-fixture.mjs new file mode 100755 index 000000000..894101301 --- /dev/null +++ b/skills/scripts/e2e/mcp-stdio-fixture.mjs @@ -0,0 +1,185 @@ +#!/usr/bin/env node + +import { spawn } from "node:child_process"; +import { existsSync, readFileSync } from "node:fs"; +import { resolve } from "node:path"; +import { env } from "node:process"; +import { + ensureEvidence, + evidencePaths, + exitCode, + localIsoWithOffset, + writeResult, +} from "./lib/langbot-e2e.mjs"; + +function loadEnvDefaults(path) { + if (!existsSync(path)) return; + for (const rawLine of readFileSync(path, "utf8").split(/\r?\n/)) { + const line = rawLine.trim(); + if (!line || line.startsWith("#")) continue; + const sep = line.indexOf("="); + if (sep === -1) continue; + const key = line.slice(0, sep).trim(); + if (env[key]) continue; + env[key] = line.slice(sep + 1).trim().replace(/^["']|["']$/g, ""); + } +} + +loadEnvDefaults("skills/.env"); +loadEnvDefaults("skills/.env.local"); + +const caseId = env.LBS_CASE_ID || "mcp-stdio-fixture-direct"; +const paths = evidencePaths(caseId); +await ensureEvidence(paths); + +const startedAt = new Date(); +const fixturePath = resolve(env.LANGBOT_MCP_FIXTURE_PATH || "skills/langbot-testing/fixtures/mcp/qa_mcp_echo_server.py"); +const langbotRepo = env.LANGBOT_REPO ? resolve(env.LANGBOT_REPO) : ""; +const uvCandidates = [ + env.LANGBOT_MCP_FIXTURE_UV, + "uv", +].filter(Boolean); +const uv = uvCandidates.find((candidate) => candidate === "uv" || existsSync(candidate)); +const pythonCandidates = [ + env.LANGBOT_MCP_FIXTURE_PYTHON, + langbotRepo ? `${langbotRepo}/.venv/bin/python` : "", + "python3", +].filter(Boolean); +const python = pythonCandidates.find((candidate) => candidate === "python3" || existsSync(candidate)); +const command = langbotRepo && uv + ? { executable: uv, args: ["run", "python", fixturePath], cwd: langbotRepo, mode: "uv" } + : python + ? { executable: python, args: [fixturePath], cwd: resolve("."), mode: "python" } + : null; +const expectedText = "qa_mcp_echo:mcp-stdio-fixture-ok"; + +const result = { + source: "automation", + case_id: caseId, + run_id: paths.runId, + started_at: startedAt.toISOString(), + started_at_local: localIsoWithOffset(startedAt), + finished_at: "", + finished_at_local: "", + status: "fail", + reason: "", + fixture_path: fixturePath, + command, + expected_text: expectedText, + evidence: { + automation_result_json: paths.automationResultJson, + result_json: paths.resultJson, + }, +}; + +function parseJsonLines(buffer) { + return buffer + .split(/\r?\n/) + .map((line) => line.trim()) + .filter(Boolean) + .map((line) => { + try { + return JSON.parse(line); + } catch { + return null; + } + }) + .filter(Boolean); +} + +async function request(child, id, method, params) { + child.stdin.write(`${JSON.stringify({ jsonrpc: "2.0", id, method, params })}\n`); +} + +async function run() { + if (!command) { + result.status = "env_issue"; + result.reason = "No uv or Python interpreter found. Set LANGBOT_REPO, LANGBOT_MCP_FIXTURE_UV, or LANGBOT_MCP_FIXTURE_PYTHON."; + return; + } + if (!existsSync(fixturePath)) { + result.status = "env_issue"; + result.reason = `MCP fixture not found: ${fixturePath}`; + return; + } + + const child = spawn(command.executable, command.args, { + cwd: command.cwd, + stdio: ["pipe", "pipe", "pipe"], + }); + let stdout = ""; + let stderr = ""; + child.stdout.setEncoding("utf8"); + child.stderr.setEncoding("utf8"); + child.stdout.on("data", (chunk) => { + stdout += chunk; + }); + child.stderr.on("data", (chunk) => { + stderr += chunk; + }); + + const timeout = setTimeout(() => child.kill("SIGTERM"), 10_000); + try { + await new Promise((resolveReady) => setTimeout(resolveReady, 100)); + await request(child, 1, "initialize", { + protocolVersion: "2024-11-05", + capabilities: {}, + clientInfo: { name: "langbot-skills", version: "0" }, + }); + await new Promise((resolveReady) => setTimeout(resolveReady, 200)); + child.stdin.write(`${JSON.stringify({ jsonrpc: "2.0", method: "notifications/initialized", params: {} })}\n`); + await request(child, 2, "tools/list", {}); + await request(child, 3, "tools/call", { + name: "qa_mcp_echo", + arguments: { text: "mcp-stdio-fixture-ok" }, + }); + await new Promise((resolveDone) => setTimeout(resolveDone, 1500)); + } finally { + clearTimeout(timeout); + child.kill("SIGTERM"); + } + + const messages = parseJsonLines(stdout); + if (/No module named ['"]mcp['"]|ModuleNotFoundError/i.test(stderr)) { + result.status = "env_issue"; + result.reason = `Python environment cannot import mcp. Set LANGBOT_MCP_FIXTURE_PYTHON to a LangBot venv Python. stderr=${stderr.trim()}`; + return; + } + const listResult = messages.find((message) => message.id === 2)?.result; + const callResult = messages.find((message) => message.id === 3)?.result; + const toolNames = Array.isArray(listResult?.tools) + ? listResult.tools.map((tool) => tool.name) + : []; + const callText = Array.isArray(callResult?.content) + ? callResult.content.map((item) => item.text || "").join("\n") + : ""; + + if (!toolNames.includes("qa_mcp_echo")) { + result.status = "fail"; + result.reason = `MCP fixture did not list qa_mcp_echo. stderr=${stderr.trim()}`; + return; + } + if (!callText.includes(expectedText)) { + result.status = "fail"; + result.reason = `MCP fixture call did not return ${expectedText}. stderr=${stderr.trim()}`; + return; + } + + result.status = "pass"; + result.reason = "MCP stdio fixture listed qa_mcp_echo and returned the deterministic tool result without a model provider."; +} + +try { + await run(); +} catch (error) { + result.status = "fail"; + result.reason = error instanceof Error ? error.message : String(error); +} finally { + const finishedAt = new Date(); + result.finished_at = finishedAt.toISOString(); + result.finished_at_local = localIsoWithOffset(finishedAt); + await writeResult(paths, result); + console.log(JSON.stringify(result, null, 2)); +} + +process.exit(exitCode(result.status)); diff --git a/skills/scripts/e2e/mcp-stdio-register.mjs b/skills/scripts/e2e/mcp-stdio-register.mjs new file mode 100644 index 000000000..e2e31a9ad --- /dev/null +++ b/skills/scripts/e2e/mcp-stdio-register.mjs @@ -0,0 +1,234 @@ +#!/usr/bin/env node + +import { existsSync, readFileSync } from "node:fs"; +import { writeFile } from "node:fs/promises"; +import { resolve } from "node:path"; +import { env } from "node:process"; +import { + createBrowser, + ensureEvidence, + evidencePaths, + exitCode, + localIsoWithOffset, + safeScreenshot, + writeResult, +} from "./lib/langbot-e2e.mjs"; + +function loadEnvDefaults(path) { + if (!existsSync(path)) return; + for (const rawLine of readFileSync(path, "utf8").split(/\r?\n/)) { + const line = rawLine.trim(); + if (!line || line.startsWith("#")) continue; + const sep = line.indexOf("="); + if (sep === -1) continue; + const key = line.slice(0, sep).trim(); + if (env[key]) continue; + env[key] = line.slice(sep + 1).trim().replace(/^["']|["']$/g, ""); + } +} + +loadEnvDefaults("skills/.env"); +loadEnvDefaults("skills/.env.local"); + +const caseId = env.LBS_CASE_ID || "mcp-stdio-register"; +const paths = evidencePaths(caseId); +await ensureEvidence(paths); + +const startedAt = new Date(); +const serverName = env.LANGBOT_MCP_SERVER_NAME || "qa-local-stdio"; +const expectedTool = env.LANGBOT_MCP_EXPECTED_TOOL || "qa_mcp_echo"; +const fixturePath = resolve(env.LANGBOT_MCP_FIXTURE_PATH || "skills/langbot-testing/fixtures/mcp/qa_mcp_echo_server.py"); +const fixtureCommand = env.LANGBOT_MCP_FIXTURE_COMMAND || "python"; +const fixtureArgs = env.LANGBOT_MCP_FIXTURE_ARGS + ? JSON.parse(env.LANGBOT_MCP_FIXTURE_ARGS) + : [fixturePath]; +const startupTimeoutSec = Number(env.LANGBOT_MCP_STARTUP_TIMEOUT_SEC || "300"); +const readyTimeoutMs = Number(env.LANGBOT_MCP_READY_TIMEOUT_MS || "360000"); +const backendUrl = env.LANGBOT_BACKEND_URL || ""; +const apiDiagnosticPath = resolve(paths.evidenceDir, "api-diagnostic.json"); + +let browser; +const result = { + source: "automation", + case_id: caseId, + run_id: paths.runId, + started_at: startedAt.toISOString(), + started_at_local: localIsoWithOffset(startedAt), + finished_at: "", + finished_at_local: "", + status: "fail", + reason: "", + server_name: serverName, + fixture_path: fixturePath, + expected_tool: expectedTool, + evidence: { + console_log: paths.consoleLog, + network_log: paths.networkLog, + screenshot: paths.screenshot, + api_diagnostic_json: apiDiagnosticPath, + automation_result_json: paths.automationResultJson, + result_json: paths.resultJson, + }, + evidence_collected: ["api_diagnostic"], +}; + +async function run() { + if (!backendUrl) { + result.status = "env_issue"; + result.reason = "LANGBOT_BACKEND_URL is not configured."; + return; + } + if (!existsSync(fixturePath)) { + result.status = "env_issue"; + result.reason = `MCP fixture not found: ${fixturePath}`; + return; + } + + browser = await createBrowser(paths); + const { page } = browser; + await page.goto(env.LANGBOT_FRONTEND_URL, { waitUntil: "domcontentloaded" }); + await page.waitForLoadState("networkidle", { timeout: 10_000 }).catch(() => {}); + + const diagnostic = await page.evaluate(async ({ + backendUrl, + serverName, + expectedTool, + fixturePath, + fixtureCommand, + fixtureArgs, + startupTimeoutSec, + readyTimeoutMs, + }) => { + const token = localStorage.getItem("token"); + if (!token) { + return { + authenticated: false, + save_status: 0, + save_code: null, + save_msg: "Browser profile has no localStorage token.", + tool_names: [], + has_expected_tool: false, + runtime_status: null, + runtime_tool_names: [], + runtime_error: "", + }; + } + + const headers = { + Authorization: `Bearer ${token}`, + "Content-Type": "application/json", + }; + const serverConfig = { + name: serverName, + mode: "stdio", + enable: true, + extra_args: { + command: fixtureCommand, + args: fixtureArgs, + env: {}, + box: { + startup_timeout_sec: startupTimeoutSec, + }, + }, + }; + const getJson = async (path) => { + const response = await fetch(`${backendUrl}${path}`, { headers }); + return { + status: response.status, + json: await response.json().catch(() => ({})), + }; + }; + const sendJson = async (method, path, body) => { + const response = await fetch(`${backendUrl}${path}`, { + method, + headers, + body: JSON.stringify(body), + }); + return { + status: response.status, + json: await response.json().catch(() => ({})), + }; + }; + + const serverPath = `/api/v1/mcp/servers/${encodeURIComponent(serverName)}`; + const beforeServer = await getJson(serverPath); + const save = beforeServer.status === 404 + ? await sendJson("POST", "/api/v1/mcp/servers", serverConfig) + : await sendJson("PUT", serverPath, serverConfig); + + const deadline = Date.now() + readyTimeoutMs; + let lastTools = []; + let lastRuntime = null; + while (Date.now() < deadline) { + await new Promise((resolveReady) => setTimeout(resolveReady, 500)); + const tools = await getJson("/api/v1/tools"); + const server = await getJson(serverPath); + lastTools = (tools.json.data?.tools || []) + .map((tool) => tool.name || tool.tool_name || tool.function?.name || "") + .filter(Boolean) + .sort(); + lastRuntime = server.json.data?.server?.runtime_info || null; + if (lastTools.includes(expectedTool)) break; + } + + return { + authenticated: true, + before_status: beforeServer.status, + save_status: save.status, + save_code: save.json.code ?? null, + save_msg: save.json.msg || "", + tool_names: lastTools, + has_expected_tool: lastTools.includes(expectedTool), + runtime_status: lastRuntime?.status || null, + runtime_tool_names: (lastRuntime?.tools || []) + .map((tool) => tool.name || tool.tool_name || "") + .filter(Boolean) + .sort(), + runtime_tool_count: lastRuntime?.tool_count ?? null, + runtime_error: lastRuntime?.error_message || "", + }; + }, { backendUrl, serverName, expectedTool, fixturePath, fixtureCommand, fixtureArgs, startupTimeoutSec, readyTimeoutMs }); + + await writeFile(apiDiagnosticPath, `${JSON.stringify(diagnostic, null, 2)}\n`, "utf8"); + await safeScreenshot(page, paths.screenshot); + + if (!diagnostic.authenticated) { + result.status = "blocked"; + result.reason = "Browser profile is not authenticated for LangBot; cannot update MCP server."; + return; + } + if (diagnostic.save_status >= 400 || diagnostic.save_code !== 0) { + result.status = "fail"; + result.reason = `Failed to save MCP server ${serverName}: ${diagnostic.save_status} ${diagnostic.save_msg}`; + return; + } + if (diagnostic.runtime_status !== "connected") { + result.status = "fail"; + result.reason = `MCP server ${serverName} is not connected after save: ${diagnostic.runtime_status || "missing runtime"}. ${diagnostic.runtime_error}`; + return; + } + if (!diagnostic.has_expected_tool || !diagnostic.runtime_tool_names.includes(expectedTool)) { + result.status = "fail"; + result.reason = `MCP server ${serverName} did not expose ${expectedTool}. See ${apiDiagnosticPath}.`; + return; + } + + result.status = "pass"; + result.reason = `MCP server ${serverName} is connected and exposes ${expectedTool} through LangBot /api/v1/tools.`; +} + +try { + await run(); +} catch (error) { + result.status = /Playwright is not installed|LANGBOT_FRONTEND_URL/.test(error.message) ? "env_issue" : "fail"; + result.reason = error instanceof Error ? error.message : String(error); +} finally { + if (browser) await browser.close().catch(() => {}); + const finishedAt = new Date(); + result.finished_at = finishedAt.toISOString(); + result.finished_at_local = localIsoWithOffset(finishedAt); + await writeResult(paths, result); + console.log(JSON.stringify(result, null, 2)); +} + +process.exit(exitCode(result.status)); diff --git a/skills/scripts/e2e/pipeline-debug-chat.mjs b/skills/scripts/e2e/pipeline-debug-chat.mjs new file mode 100755 index 000000000..87fe9ae79 --- /dev/null +++ b/skills/scripts/e2e/pipeline-debug-chat.mjs @@ -0,0 +1,728 @@ +#!/usr/bin/env node + +import { spawn } from "node:child_process"; +import { readFile, writeFile } from "node:fs/promises"; +import { resolve } from "node:path"; +import { env } from "node:process"; +import { + openPipelineDebugChat, + runDebugChatPrompt, + setDebugChatStreamOutput, +} from "./lib/debug-chat.mjs"; +import { + createBrowser, + ensureEvidence, + evidencePaths, + exitCode, + localIsoWithOffset, + pathExists, + safeScreenshot, + writeResult, +} from "./lib/langbot-e2e.mjs"; + +const caseId = env.LBS_CASE_ID || "pipeline-debug-chat"; +const paths = evidencePaths(caseId); +await ensureEvidence(paths); + +const expectedText = env.LANGBOT_E2E_EXPECTED_TEXT || "OK"; +const prompt = env.LANGBOT_E2E_PROMPT || `请只回复 ${expectedText}，用于前端调试测试。`; +const responseTimeoutMs = Number.parseInt(env.LANGBOT_E2E_RESPONSE_TIMEOUT_MS || "120000", 10); +const safeResponseTimeoutMs = Number.isFinite(responseTimeoutMs) && responseTimeoutMs > 0 ? responseTimeoutMs : 120000; +const streamOutput = /^(0|false)$/i.test(env.LANGBOT_E2E_STREAM_OUTPUT || "") + ? false + : /^(1|true)$/i.test(env.LANGBOT_E2E_STREAM_OUTPUT || "") + ? true + : null; +const failureSignals = (env.LANGBOT_E2E_FAILURE_SIGNALS || "") + .split(/\r?\n/) + .map((item) => item.trim()) + .filter(Boolean); +const imageBase64Path = env.LANGBOT_E2E_IMAGE_BASE64_PATH || ""; +const imagePathEnv = env.LANGBOT_E2E_IMAGE_PATH || ""; +const backendUrl = env.LANGBOT_BACKEND_URL || ""; +const pipelineRequired = env.LANGBOT_E2E_PIPELINE_REQUIRED === "1"; +const pipelineUrl = pipelineRequired + ? env.LANGBOT_E2E_PIPELINE_URL + : (env.LANGBOT_E2E_PIPELINE_URL || env.LANGBOT_PIPELINE_URL); +const pipelineName = pipelineRequired + ? env.LANGBOT_E2E_PIPELINE_NAME + : (env.LANGBOT_E2E_PIPELINE_NAME || env.LANGBOT_PIPELINE_NAME); +const expectedRunnerId = env.LANGBOT_E2E_EXPECTED_RUNNER_ID || ""; +const resetDebugChat = boolFromEnv(env.LANGBOT_E2E_RESET_DEBUG_CHAT, false); +const restoreRunnerConfig = boolFromEnv(env.LANGBOT_E2E_RESTORE_RUNNER_CONFIG, true); +const debugChatSessionType = env.LANGBOT_E2E_DEBUG_CHAT_SESSION_TYPE || "person"; +const pipelineConfigDiagnosticPath = resolve(paths.evidenceDir, "pipeline-config-diagnostic.json"); +const debugChatResetDiagnosticPath = resolve(paths.evidenceDir, "debug-chat-reset-diagnostic.json"); +const pipelineConfigRestoreDiagnosticPath = resolve(paths.evidenceDir, "pipeline-config-restore-diagnostic.json"); +const startedAt = new Date(); + +let browser; +let restorePlan = null; +let result = { + source: "automation", + case_id: caseId, + run_id: paths.runId, + started_at: startedAt.toISOString(), + started_at_local: localIsoWithOffset(startedAt), + finished_at: "", + finished_at_local: "", + status: "fail", + reason: "", + url: "", + prompt, + expected_text: expectedText, + response_timeout_ms: safeResponseTimeoutMs, + stream_output: streamOutput, + image_fixture: imageBase64Path || imagePathEnv, + prompt_count: 1, + chat_results: [], + evidence: { + console_log: paths.consoleLog, + network_log: paths.networkLog, + screenshot: paths.screenshot, + automation_result_json: paths.automationResultJson, + result_json: paths.resultJson, + }, + evidence_collected: ["ui", "screenshot", "console", "network"], +}; + +function boolFromEnv(value, defaultValue) { + if (value === undefined || value === "") return defaultValue; + if (/^(0|false|no|off)$/i.test(value)) return false; + if (/^(1|true|yes|on)$/i.test(value)) return true; + return defaultValue; +} + +function parseJsonEnv(key, fallback) { + const raw = env[key]; + if (!raw) return fallback; + try { + return JSON.parse(raw); + } catch (error) { + throw new Error(`${key} must be valid JSON: ${error.message}`); + } +} + +function promptStepsFromEnv() { + const rawSteps = parseJsonEnv("LANGBOT_E2E_PROMPTS_JSON", null); + if (rawSteps === null) { + return [{ prompt, expectedText, responseTimeoutMs: safeResponseTimeoutMs }]; + } + if (!Array.isArray(rawSteps) || rawSteps.length === 0) { + throw new Error("LANGBOT_E2E_PROMPTS_JSON must be a non-empty JSON array."); + } + return rawSteps.map((item, index) => { + if (typeof item === "string") { + return { prompt: item, expectedText, responseTimeoutMs: safeResponseTimeoutMs }; + } + if (!item || typeof item !== "object" || typeof item.prompt !== "string" || !item.prompt) { + throw new Error(`LANGBOT_E2E_PROMPTS_JSON[${index}] must be a string or an object with a prompt string.`); + } + const stepTimeout = Number.parseInt(String(item.response_timeout_ms || item.responseTimeoutMs || safeResponseTimeoutMs), 10); + return { + prompt: item.prompt, + expectedText: String(item.expected_text || item.expectedText || expectedText), + responseTimeoutMs: Number.isFinite(stepTimeout) && stepTimeout > 0 ? stepTimeout : safeResponseTimeoutMs, + }; + }); +} + +function expandEnvRefs(value) { + return String(value || "").replace(/\$\{([A-Z][A-Z0-9_]*)\}|\$([A-Z][A-Z0-9_]*)/g, (_match, braced, bare) => { + return env[braced || bare] || ""; + }); +} + +function textList(value) { + if (value === undefined || value === null || value === "") return []; + return Array.isArray(value) ? value.map(String) : [String(value)]; +} + +function runArgv(argv, { cwd = "", timeoutMs = 30_000 } = {}) { + return new Promise((resolveRun) => { + if (!Array.isArray(argv) || argv.length === 0 || !argv.every((item) => typeof item === "string" && item)) { + resolveRun({ + status: "fail", + reason: "Filesystem command check requires a non-empty argv string array.", + exit_code: null, + stdout: "", + stderr: "", + }); + return; + } + + const child = spawn(argv[0], argv.slice(1), { + cwd: cwd ? resolve(cwd) : undefined, + env, + shell: false, + }); + let stdout = ""; + let stderr = ""; + let timedOut = false; + const timer = setTimeout(() => { + timedOut = true; + child.kill("SIGTERM"); + }, timeoutMs); + + child.stdout.on("data", (chunk) => { + stdout += chunk.toString(); + }); + child.stderr.on("data", (chunk) => { + stderr += chunk.toString(); + }); + child.on("error", (error) => { + clearTimeout(timer); + resolveRun({ + status: "fail", + reason: error.message, + exit_code: null, + stdout, + stderr, + }); + }); + child.on("close", (code) => { + clearTimeout(timer); + resolveRun({ + status: timedOut ? "fail" : "pass", + reason: timedOut ? `Command timed out after ${timeoutMs} ms.` : "", + exit_code: code, + stdout, + stderr, + }); + }); + }); +} + +async function runFilesystemChecks(checks) { + if (!Array.isArray(checks) || checks.length === 0) { + return { status: "not_required", checks: [] }; + } + const results = []; + for (let index = 0; index < checks.length; index += 1) { + const check = checks[index]; + if (!check || typeof check !== "object") { + results.push({ index, status: "fail", reason: "Filesystem check must be an object." }); + continue; + } + const contains = textList(check.contains); + const notContains = textList(check.not_contains || check.notContains); + const expectedExitCode = Number.isInteger(check.exit_code) + ? check.exit_code + : Number.isInteger(check.expected_exit_code) + ? check.expected_exit_code + : 0; + const expectedStdout = textList(check.stdout_contains || check.expected_stdout || check.expectedStdout); + + if (check.path) { + const path = resolve(expandEnvRefs(check.path)); + let text = ""; + try { + text = await readFile(path, "utf8"); + } catch (error) { + results.push({ index, status: "fail", type: "file", path, reason: error.message }); + continue; + } + const missing = contains.filter((needle) => !text.includes(needle)); + const forbidden = notContains.filter((needle) => text.includes(needle)); + results.push({ + index, + status: missing.length || forbidden.length ? "fail" : "pass", + type: "file", + path, + missing, + forbidden, + reason: missing.length + ? `Missing expected text: ${missing.join(", ")}` + : forbidden.length + ? `Found forbidden text: ${forbidden.join(", ")}` + : "", + }); + continue; + } + + if (check.argv) { + const cwd = check.cwd ? expandEnvRefs(check.cwd) : ""; + const timeoutMs = Number.parseInt(String(check.timeout_ms || check.timeoutMs || "30000"), 10); + const run = await runArgv(check.argv.map(expandEnvRefs), { + cwd, + timeoutMs: Number.isFinite(timeoutMs) && timeoutMs > 0 ? timeoutMs : 30_000, + }); + const missingStdout = expectedStdout.filter((needle) => !run.stdout.includes(needle)); + const exitMatches = run.exit_code === expectedExitCode; + results.push({ + index, + status: run.status === "pass" && exitMatches && missingStdout.length === 0 ? "pass" : "fail", + type: "command", + argv: check.argv, + cwd, + exit_code: run.exit_code, + expected_exit_code: expectedExitCode, + missing_stdout: missingStdout, + stdout_preview: run.stdout.slice(0, 2000), + stderr_preview: run.stderr.slice(0, 2000), + reason: run.reason + || (!exitMatches ? `Expected exit code ${expectedExitCode}, saw ${run.exit_code}.` : "") + || (missingStdout.length ? `Missing stdout text: ${missingStdout.join(", ")}` : ""), + }); + continue; + } + + results.push({ index, status: "fail", reason: "Filesystem check requires either path or argv." }); + } + const failed = results.filter((item) => item.status !== "pass"); + return { + status: failed.length ? "fail" : "pass", + checks: results, + reason: failed.length ? `Filesystem checks failed: ${failed.map((item) => item.index).join(", ")}` : "", + }; +} + +function pipelineIdFromUrl(url) { + if (!url) return ""; + try { + const parsed = new URL(url); + return parsed.searchParams.get("id") || ""; + } catch { + return ""; + } +} + +function sanitizePipelineDiagnostic(diagnostic) { + const { restore_config: _restoreConfig, ...safe } = diagnostic || {}; + return safe; +} + +async function prepareImageFixture(paths) { + if (imagePathEnv) return resolve(imagePathEnv); + if (!imageBase64Path) return ""; + const source = resolve(imageBase64Path); + const target = resolve(paths.evidenceDir, "image-fixture.png"); + const encoded = await readFile(source, "utf8"); + await writeFile(target, Buffer.from(encoded.replace(/\s+/g, ""), "base64")); + return target; +} + +async function inspectAndPatchPipelineConfig(page, { + backendUrl, + pipelineUrl, + pipelineName, + runnerConfigPatch, + expectedRunnerId, +}) { + const pipelineIdFromUrlValue = pipelineIdFromUrl(pipelineUrl) || pipelineIdFromUrl(page.url()); + return await page.evaluate(async ({ + backendUrl, + pipelineIdFromUrlValue, + pipelineName, + runnerConfigPatch, + expectedRunnerId, + }) => { + const token = localStorage.getItem("token"); + if (!token) { + return { + status: "blocked", + authenticated: false, + reason: "Browser profile has no localStorage token.", + }; + } + + const headers = { + Authorization: `Bearer ${token}`, + "Content-Type": "application/json", + }; + const getJson = async (path) => { + const response = await fetch(`${backendUrl}${path}`, { headers }); + return { + status: response.status, + json: await response.json().catch(() => ({})), + }; + }; + const putJson = async (path, body) => { + const response = await fetch(`${backendUrl}${path}`, { + method: "PUT", + headers, + body: JSON.stringify(body), + }); + return { + status: response.status, + json: await response.json().catch(() => ({})), + }; + }; + + let pipelineId = pipelineIdFromUrlValue || ""; + let matchedBy = pipelineId ? "url" : ""; + if (!pipelineId && pipelineName) { + const list = await getJson("/api/v1/pipelines"); + const pipelines = list.json.data?.pipelines || []; + const match = pipelines.find((pipeline) => pipeline.name === pipelineName); + if (!match) { + return { + status: "blocked", + authenticated: true, + pipeline_resolved: false, + list_status: list.status, + reason: `Could not find pipeline named ${pipelineName}.`, + }; + } + pipelineId = match.uuid; + matchedBy = "name"; + } + + if (!pipelineId) { + return { + status: "blocked", + authenticated: true, + pipeline_resolved: false, + reason: "Could not resolve pipeline id from URL or pipeline name.", + }; + } + + const before = await getJson(`/api/v1/pipelines/${encodeURIComponent(pipelineId)}`); + const pipeline = before.json.data?.pipeline; + if (before.status >= 400 || !pipeline) { + return { + status: "fail", + authenticated: true, + pipeline_resolved: false, + pipeline_id: pipelineId, + get_status: before.status, + reason: before.json.msg || "Could not load pipeline.", + }; + } + + const config = JSON.parse(JSON.stringify(pipeline.config || {})); + const aiConfig = config.ai && typeof config.ai === "object" ? config.ai : {}; + const runner = aiConfig.runner && typeof aiConfig.runner === "object" ? aiConfig.runner : {}; + const runnerId = runner.id || runner.runner || ""; + if (!runnerId) { + return { + status: "blocked", + authenticated: true, + pipeline_resolved: true, + pipeline_id: pipelineId, + pipeline_name: pipeline.name, + matched_by: matchedBy, + reason: "Pipeline has no ai.runner.id or legacy ai.runner.runner.", + }; + } + if (expectedRunnerId && runnerId !== expectedRunnerId) { + return { + status: "blocked", + authenticated: true, + pipeline_resolved: true, + pipeline_id: pipelineId, + pipeline_name: pipeline.name, + matched_by: matchedBy, + runner_id: runnerId, + expected_runner_id: expectedRunnerId, + reason: `Pipeline runner mismatch: expected ${expectedRunnerId}, got ${runnerId}.`, + }; + } + + const runnerConfigs = aiConfig.runner_config && typeof aiConfig.runner_config === "object" + ? aiConfig.runner_config + : {}; + const currentRunnerConfig = runnerConfigs[runnerId] && typeof runnerConfigs[runnerId] === "object" + ? runnerConfigs[runnerId] + : {}; + const patchKeys = Object.keys(runnerConfigPatch || {}); + const baseDiagnostic = { + status: "ready", + authenticated: true, + pipeline_resolved: true, + pipeline_id: pipelineId, + pipeline_name: pipeline.name, + matched_by: matchedBy, + runner_id: runnerId, + expected_runner_id: expectedRunnerId || "", + patch_keys: patchKeys, + runner_config_before_keys: Object.keys(currentRunnerConfig), + patched: patchKeys.length > 0, + }; + + if (patchKeys.length === 0) { + return baseDiagnostic; + } + + const updatedRunnerConfig = { + ...currentRunnerConfig, + ...runnerConfigPatch, + }; + const updatedConfig = { + ...config, + ai: { + ...aiConfig, + runner: { + ...runner, + id: runnerId, + }, + runner_config: { + ...runnerConfigs, + [runnerId]: updatedRunnerConfig, + }, + }, + }; + + const update = await putJson(`/api/v1/pipelines/${encodeURIComponent(pipelineId)}`, { + config: updatedConfig, + }); + if (update.status >= 400) { + return { + ...baseDiagnostic, + status: "fail", + put_status: update.status, + put_code: update.json.code ?? null, + reason: update.json.msg || "Pipeline config update failed.", + }; + } + + return { + ...baseDiagnostic, + put_status: update.status, + put_code: update.json.code ?? null, + runner_config_after_keys: Object.keys(updatedRunnerConfig), + restore_config: config, + }; + }, { + backendUrl, + pipelineIdFromUrlValue, + pipelineName, + runnerConfigPatch, + expectedRunnerId, + }); +} + +async function restorePipelineConfig(page, { backendUrl, pipelineId, config }) { + return await page.evaluate(async ({ backendUrl, pipelineId, config }) => { + const token = localStorage.getItem("token"); + if (!token) { + return { + status: "blocked", + authenticated: false, + pipeline_id: pipelineId, + reason: "Browser profile has no localStorage token.", + }; + } + const response = await fetch(`${backendUrl}/api/v1/pipelines/${encodeURIComponent(pipelineId)}`, { + method: "PUT", + headers: { + Authorization: `Bearer ${token}`, + "Content-Type": "application/json", + }, + body: JSON.stringify({ config }), + }); + const json = await response.json().catch(() => ({})); + return { + status: response.status >= 400 ? "fail" : "ready", + authenticated: true, + pipeline_id: pipelineId, + put_status: response.status, + put_code: json.code ?? null, + reason: response.status >= 400 ? json.msg || "Pipeline config restore failed." : "Pipeline config restored.", + }; + }, { backendUrl, pipelineId, config }); +} + +async function resetPipelineDebugChat(page, { backendUrl, pipelineId, sessionType }) { + return await page.evaluate(async ({ backendUrl, pipelineId, sessionType }) => { + const token = localStorage.getItem("token"); + if (!token) { + return { + status: "blocked", + authenticated: false, + pipeline_id: pipelineId, + session_type: sessionType, + reason: "Browser profile has no localStorage token.", + }; + } + const response = await fetch( + `${backendUrl}/api/v1/pipelines/${encodeURIComponent(pipelineId)}/ws/reset/${encodeURIComponent(sessionType)}`, + { + method: "POST", + headers: { + Authorization: `Bearer ${token}`, + "Content-Type": "application/json", + }, + }, + ); + const json = await response.json().catch(() => ({})); + return { + status: response.status >= 400 ? "fail" : "ready", + authenticated: true, + pipeline_id: pipelineId, + session_type: sessionType, + reset_status: response.status, + reset_code: json.code ?? null, + reason: response.status >= 400 ? json.msg || "Debug Chat reset failed." : "Debug Chat session reset.", + }; + }, { backendUrl, pipelineId, sessionType }); +} + +try { + browser = await createBrowser(paths); + const { page } = browser; + const imagePath = await prepareImageFixture(paths); + const promptSteps = promptStepsFromEnv(); + const filesystemChecks = parseJsonEnv("LANGBOT_E2E_FILESYSTEM_CHECKS_JSON", []); + const runnerConfigPatch = parseJsonEnv("LANGBOT_E2E_RUNNER_CONFIG_PATCH_JSON", {}); + const runnerPatchKeys = Object.keys(runnerConfigPatch); + if (runnerPatchKeys.length > 0 || resetDebugChat || expectedRunnerId) { + if (!backendUrl) { + result.status = "env_issue"; + result.reason = "LANGBOT_BACKEND_URL is required for runner config patch, runner assertion, or Debug Chat reset."; + throw new Error(result.reason); + } + } + result.prompt_count = promptSteps.length; + result.prompt = promptSteps.length === 1 ? promptSteps[0].prompt : `${promptSteps.length} prompts`; + result.expected_text = promptSteps.at(-1)?.expectedText || expectedText; + + const openResult = await openPipelineDebugChat(page, { + pipelineUrl, + pipelineName, + envHint: pipelineRequired + ? "case-specific pipeline env mapped to LANGBOT_E2E_PIPELINE_URL or LANGBOT_E2E_PIPELINE_NAME" + : "LANGBOT_PIPELINE_URL or LANGBOT_PIPELINE_NAME", + }); + result.url = page.url(); + + if (!openResult.opened) { + result.status = openResult.status; + result.reason = openResult.reason; + } else { + result.status = "running"; + result.reason = ""; + if (runnerPatchKeys.length > 0 || resetDebugChat || expectedRunnerId) { + const pipelineDiagnostic = await inspectAndPatchPipelineConfig(page, { + backendUrl, + pipelineUrl, + pipelineName, + runnerConfigPatch, + expectedRunnerId, + }); + const safeDiagnostic = sanitizePipelineDiagnostic(pipelineDiagnostic); + await writeFile(pipelineConfigDiagnosticPath, `${JSON.stringify(safeDiagnostic, null, 2)}\n`, "utf8"); + result.evidence.pipeline_config_diagnostic_json = pipelineConfigDiagnosticPath; + result.pipeline_config = safeDiagnostic; + if (!result.evidence_collected.includes("api_diagnostic")) result.evidence_collected.push("api_diagnostic"); + + if (pipelineDiagnostic.status === "fail" || pipelineDiagnostic.status === "blocked") { + result.status = pipelineDiagnostic.status; + result.reason = pipelineDiagnostic.reason || "Pipeline config preparation failed."; + } else { + if (pipelineDiagnostic.restore_config && restoreRunnerConfig) { + restorePlan = { + backendUrl, + pipelineId: pipelineDiagnostic.pipeline_id, + config: pipelineDiagnostic.restore_config, + }; + } + if (resetDebugChat) { + const resetDiagnostic = await resetPipelineDebugChat(page, { + backendUrl, + pipelineId: pipelineDiagnostic.pipeline_id, + sessionType: debugChatSessionType, + }); + await writeFile(debugChatResetDiagnosticPath, `${JSON.stringify(resetDiagnostic, null, 2)}\n`, "utf8"); + result.evidence.debug_chat_reset_diagnostic_json = debugChatResetDiagnosticPath; + result.debug_chat_reset = resetDiagnostic; + if (resetDiagnostic.status === "fail" || resetDiagnostic.status === "blocked") { + result.status = resetDiagnostic.status; + result.reason = resetDiagnostic.reason || "Debug Chat reset failed."; + } else { + await page.waitForTimeout(1000); + const reopenResult = await openPipelineDebugChat(page, { + pipelineUrl, + pipelineName, + envHint: pipelineRequired + ? "case-specific pipeline env mapped to LANGBOT_E2E_PIPELINE_URL or LANGBOT_E2E_PIPELINE_NAME" + : "LANGBOT_PIPELINE_URL or LANGBOT_PIPELINE_NAME", + }); + result.url = page.url(); + if (!reopenResult.opened) { + result.status = reopenResult.status; + result.reason = reopenResult.reason; + } + } + } + } + } + + if (result.status === "fail" || result.status === "blocked" || result.status === "env_issue") { + // Preparation already determined the outcome. + } else { + const streamResult = await setDebugChatStreamOutput(page, streamOutput); + if (streamResult.status === "blocked" || streamResult.status === "fail") { + result.status = streamResult.status; + result.reason = streamResult.reason; + } else { + for (let index = 0; index < promptSteps.length; index += 1) { + const step = promptSteps[index]; + const chatResult = await runDebugChatPrompt(page, { + prompt: step.prompt, + expectedText: step.expectedText, + responseTimeoutMs: step.responseTimeoutMs, + imagePath: index === 0 ? imagePath : "", + failureSignals: failureSignals.length > 0 ? failureSignals : undefined, + }); + result.chat_results.push({ + index, + expected_text: step.expectedText, + status: chatResult.status, + reason: chatResult.reason, + min_expected_count: chatResult.min_expected_count, + final_count: chatResult.final_count, + before_assistant_expected_count: chatResult.before_assistant_expected_count, + after_assistant_expected_count: chatResult.after_assistant_expected_count, + failure_signal: chatResult.failure_signal || "", + }); + result.status = chatResult.status; + result.reason = `Prompt ${index + 1}/${promptSteps.length}: ${chatResult.reason}`; + if (chatResult.status !== "pass") break; + } + } + } + + if (result.status === "pass" && filesystemChecks.length > 0) { + const filesystemResult = await runFilesystemChecks(filesystemChecks); + result.filesystem_checks = filesystemResult; + if (!result.evidence_collected.includes("filesystem")) result.evidence_collected.push("filesystem"); + if (filesystemResult.status === "fail") { + result.status = "fail"; + result.reason = filesystemResult.reason || "Filesystem checks failed."; + } + } + } +} catch (error) { + if (!["env_issue", "blocked", "fail", "pass"].includes(result.status) || !result.reason) { + result.status = /Playwright is not installed|LANGBOT_FRONTEND_URL/.test(error.message) ? "env_issue" : "fail"; + } + result.reason = result.reason || error.message; +} finally { + if (browser?.page) await safeScreenshot(browser.page, paths.screenshot); + if (browser?.page && restorePlan) { + const restoreDiagnostic = await restorePipelineConfig(browser.page, restorePlan).catch((error) => ({ + status: "fail", + pipeline_id: restorePlan.pipelineId, + reason: error.message, + })); + await writeFile(pipelineConfigRestoreDiagnosticPath, `${JSON.stringify(restoreDiagnostic, null, 2)}\n`, "utf8"); + result.evidence.pipeline_config_restore_diagnostic_json = pipelineConfigRestoreDiagnosticPath; + result.pipeline_config_restore = restoreDiagnostic; + } + if (browser) await browser.close().catch(() => {}); + const finishedAt = new Date(); + result.finished_at = finishedAt.toISOString(); + result.finished_at_local = localIsoWithOffset(finishedAt); + const existingEvidence = {}; + for (const [key, value] of Object.entries(result.evidence)) { + if (typeof value !== "string") continue; + const isResultFile = value === paths.automationResultJson || value === paths.resultJson; + if (isResultFile || await pathExists(value)) existingEvidence[key] = value; + } + result.evidence = existingEvidence; + await writeResult(paths, result); + console.log(JSON.stringify(result, null, 2)); +} + +process.exit(exitCode(result.status)); diff --git a/skills/scripts/e2e/refresh-local-login.mjs b/skills/scripts/e2e/refresh-local-login.mjs new file mode 100644 index 000000000..66b4f34c6 --- /dev/null +++ b/skills/scripts/e2e/refresh-local-login.mjs @@ -0,0 +1,84 @@ +#!/usr/bin/env node + +import { env } from "node:process"; +import { + bodyText, + createBrowser, + ensureEvidence, + evidencePaths, + loadEnvFiles, + resetAndAuthLocalUser, + safeScreenshot, + setBrowserToken, + verifyBrowserToken, + writeResult, +} from "./lib/langbot-e2e.mjs"; + +const caseId = "refresh-local-login"; +const paths = evidencePaths(caseId); +await loadEnvFiles(); +await ensureEvidence(paths); + +const result = { + source: "automation", + case_id: caseId, + status: "fail", + reason: "", + user: env.LANGBOT_E2E_LOGIN_USER || "", + frontend_url: env.LANGBOT_FRONTEND_URL || "", + backend_url: env.LANGBOT_BACKEND_URL || "", + backend_token_check: null, + browser_token_check: null, + evidence: { + console_log: paths.consoleLog, + network_log: paths.networkLog, + screenshot: paths.screenshot, + automation_result_json: paths.automationResultJson, + result_json: paths.resultJson, + }, + evidence_collected: ["ui", "screenshot", "console", "api_diagnostic"], +}; + +let browser; + +try { + const backendUrl = env.LANGBOT_BACKEND_URL; + const frontendUrl = env.LANGBOT_FRONTEND_URL; + const user = env.LANGBOT_E2E_LOGIN_USER; + const password = env.LANGBOT_E2E_LOGIN_PASSWORD || "LangBotE2ELocalPass!2026"; + if (!backendUrl) throw new Error("LANGBOT_BACKEND_URL is not configured."); + if (!frontendUrl) throw new Error("LANGBOT_FRONTEND_URL is not configured."); + if (!user) throw new Error("LANGBOT_E2E_LOGIN_USER is required."); + + const auth = await resetAndAuthLocalUser({ backendUrl, user, password }); + result.backend_token_check = auth.check; + + browser = await createBrowser(paths); + const { page } = browser; + await setBrowserToken(page, frontendUrl, auth.token); + const browserCheck = await verifyBrowserToken(page, backendUrl); + result.browser_token_check = browserCheck; + if (!browserCheck.authenticated) { + throw new Error(browserCheck.reason || "Browser token check failed."); + } + + await page.goto(`${frontendUrl.replace(/\/$/, "")}/home/monitoring`, { waitUntil: "domcontentloaded" }); + await page.waitForLoadState("networkidle", { timeout: 10_000 }).catch(() => {}); + const text = await bodyText(page); + if (!text.includes("Dashboard") && !text.includes("Pipelines") && !text.includes("流水线")) { + throw new Error("Token was written, but authenticated navigation was not visible."); + } + + result.status = "pass"; + result.reason = "Browser profile localStorage token refreshed."; +} catch (error) { + result.status = "fail"; + result.reason = error.message; +} finally { + if (browser?.page) await safeScreenshot(browser.page, paths.screenshot); + if (browser) await browser.close().catch(() => {}); + await writeResult(paths, result); + console.log(JSON.stringify(result, null, 2)); +} + +process.exit(result.status === "pass" ? 0 : 1); diff --git a/skills/scripts/e2e/webui-login-state.mjs b/skills/scripts/e2e/webui-login-state.mjs new file mode 100755 index 000000000..fbbb53f33 --- /dev/null +++ b/skills/scripts/e2e/webui-login-state.mjs @@ -0,0 +1,107 @@ +#!/usr/bin/env node + +import { + bodyText, + createBrowser, + ensureEvidence, + evidencePaths, + exitCode, + gotoFrontend, + isLoginUrl, + loadEnvFiles, + localIsoWithOffset, + safeScreenshot, + verifyBrowserToken, + writeResult, +} from "./lib/langbot-e2e.mjs"; + +const caseId = "webui-login-state"; +await loadEnvFiles(); +const paths = evidencePaths(caseId); +await ensureEvidence(paths); + +const startedAt = new Date(); +let browser; +let result = { + source: "automation", + case_id: caseId, + run_id: paths.runId, + started_at: startedAt.toISOString(), + started_at_local: localIsoWithOffset(startedAt), + finished_at: "", + finished_at_local: "", + status: "fail", + reason: "", + url: "", + auth: null, + evidence: { + console_log: paths.consoleLog, + network_log: paths.networkLog, + screenshot: paths.screenshot, + automation_result_json: paths.automationResultJson, + result_json: paths.resultJson, + }, + evidence_collected: ["ui", "screenshot", "console"], +}; + +try { + browser = await createBrowser(paths); + const { page } = browser; + await gotoFrontend(page); + result.url = page.url(); + + const backendUrl = process.env.LANGBOT_BACKEND_URL || ""; + if (!backendUrl) { + result.status = "env_issue"; + result.reason = "LANGBOT_BACKEND_URL is not configured."; + await safeScreenshot(page, paths.screenshot); + throw new Error(result.reason); + } + + const auth = await verifyBrowserToken(page, backendUrl); + result.auth = auth; + const text = await bodyText(page); + const navigationSignals = [ + "Dashboard", + "Bots", + "Pipelines", + "Knowledge", + "Plugins", + "首页", + "机器人", + "流水线", + "知识库", + "插件", + ]; + const matchedSignal = navigationSignals.find((signal) => text.includes(signal)); + + if (!auth.authenticated) { + result.status = "blocked"; + result.reason = auth.reason || "Browser profile token was not accepted by backend."; + } else if (isLoginUrl(page.url()) || /登录|Login|Sign in/i.test(text)) { + result.status = "fail"; + result.reason = "Backend accepted the token, but the WebUI still showed the login page."; + } else if (!matchedSignal) { + result.status = "fail"; + result.reason = "Opened WebUI, but no known LangBot navigation signal was visible."; + } else { + result.status = "pass"; + result.reason = `Authenticated navigation signal visible: ${matchedSignal}`; + } + + await safeScreenshot(page, paths.screenshot); +} catch (error) { + if (!["env_issue", "blocked", "fail", "pass"].includes(result.status) || !result.reason) { + result.status = /Playwright is not installed|LANGBOT_FRONTEND_URL/.test(error.message) ? "env_issue" : "fail"; + result.reason = error.message; + } +} finally { + if (browser) await browser.close().catch(() => {}); + const finishedAt = new Date(); + result.finished_at = finishedAt.toISOString(); + result.finished_at_local = localIsoWithOffset(finishedAt); + await writeResult(paths, result); + console.log(JSON.stringify(result, null, 2)); +} + +process.exit(exitCode(result.status)); diff --git a/skills/skills.index.json b/skills/skills.index.json new file mode 100644 index 000000000..d56a84822 --- /dev/null +++ b/skills/skills.index.json @@ -0,0 +1,1472 @@ +{ + "generated_by": "lbs", + "skills": [ + { + "directory": "langbot-deploy", + "name": "langbot-deploy", + "description": "Deploy and configure a LangBot instance — Docker / Docker Compose, Kubernetes, the config.yaml model, the Box sandbox runtime, the plugin runtime, and the global API key. Use when installing, deploying, upgrading, or configuring LangBot in production or self-hosted environments. Triggers on \"deploy langbot\", \"langbot docker\", \"langbot compose\", \"langbot kubernetes\", \"langbot config.yaml\", \"langbot box runtime\", \"langbot global api key\".", + "references": [], + "cases": [], + "case_summaries": [], + "suites": [], + "suite_summaries": [], + "fixtures": [], + "troubleshooting": [], + "troubleshooting_summaries": [] + }, + { + "directory": "langbot-dev", + "name": "langbot-dev", + "description": "Develop, build, and debug the LangBot core backend and web frontend. Use when working inside the LangBot repository — backend (Python/Quart, src/langbot/pkg), the Vite/React web UI, HTTP API controllers/services, Alembic migrations, or the MCP server. Covers the dev environment (uv, pnpm), repo layout, the API auth model (user token / API key / global key), adding API endpoints, and the rule that API changes must update the MCP server and skills. Triggers on \"langbot backend\", \"langbot dev\", \"langbot api\", \"add langbot endpoint\", \"langbot migration\".", + "references": [], + "cases": [], + "case_summaries": [], + "suites": [], + "suite_summaries": [], + "fixtures": [], + "troubleshooting": [], + "troubleshooting_summaries": [] + }, + { + "directory": "langbot-eba-adapter-dev", + "name": "langbot-eba-adapter-dev", + "description": "Build, refactor, and test LangBot platform adapters for the Event-Based Agents architecture. Use when adding or migrating Telegram, Discord, or other messaging platform adapters to the EBA adapter layout, validating unified event/message conversion, writing live adapter probes, or using standalone plugin runtime plus Computer Use for end-to-end platform testing.", + "references": [], + "cases": [], + "case_summaries": [], + "suites": [], + "suite_summaries": [], + "fixtures": [], + "troubleshooting": [], + "troubleshooting_summaries": [] + }, + { + "directory": "langbot-env-setup", + "name": "langbot-env-setup", + "description": "Prepare a local LangBot development and testing environment for an AI agent. Use when setting up WSL or Linux development, shared local URL variables, proxy variables, backend/frontend startup, Playwright MCP browser access, GitHub OAuth browser login, persisted Chrome profiles, or future Codex computer-use environment paths.", + "references": [ + "references/browser-access-selection.md", + "references/computer-use.md", + "references/oauth-browser-profile.md", + "references/playwright-mcp.md", + "references/proxy.md", + "references/service-startup.md", + "references/wsl-notes.md" + ], + "cases": [], + "case_summaries": [], + "suites": [], + "suite_summaries": [], + "fixtures": [], + "troubleshooting": [], + "troubleshooting_summaries": [] + }, + { + "directory": "langbot-mcp-ops", + "name": "langbot-mcp-ops", + "description": "Operate a LangBot instance through its built-in MCP (Model Context Protocol) server. Use when an AI agent needs to manage LangBot — list/create/update/delete bots, pipelines, models, knowledge bases, MCP servers, and skills — over MCP instead of raw HTTP. Covers the /mcp endpoint, API-key auth (web-UI lbk_ keys and the config.yaml global key), the tool surface, and client configuration. Triggers on \"langbot mcp\", \"manage langbot via mcp\", \"langbot /mcp\", \"langbot mcp server\".", + "references": [], + "cases": [], + "case_summaries": [], + "suites": [], + "suite_summaries": [], + "fixtures": [], + "troubleshooting": [], + "troubleshooting_summaries": [] + }, + { + "directory": "langbot-plugin-dev", + "name": "langbot-plugin-dev", + "description": "Develop, debug, and test LangBot plugins. Use when creating new LangBot plugins, fixing plugin bugs, setting up a LangBot test environment, or testing plugins via WebSocket. Covers plugin component architecture (EventListener, Command, Tool), the plugin SDK API (invoke_llm, get_llm_models, send_message, plugin storage), common pitfalls, and automated WebSocket-based testing. Triggers on \"langbot plugin\", \"lbp\", \"GroupChatSummary\", \"plugin debug\", \"langbot test\".", + "references": [ + "references/test-env-setup.md" + ], + "cases": [], + "case_summaries": [], + "suites": [], + "suite_summaries": [], + "fixtures": [], + "troubleshooting": [], + "troubleshooting_summaries": [] + }, + { + "directory": "langbot-skills-maintenance", + "name": "langbot-skills-maintenance", + "description": "Maintain the langbot-skills repository with low duplication. Use when adding, editing, or auditing LangBot skills, references, cases, troubleshooting entries, indexes, or periodic entropy-control checks for this skills repository.", + "references": [ + "references/curation-workflow.md" + ], + "cases": [], + "case_summaries": [], + "suites": [], + "suite_summaries": [], + "fixtures": [], + "troubleshooting": [], + "troubleshooting_summaries": [] + }, + { + "directory": "langbot-space-ops", + "name": "langbot-space-ops", + "description": "Browse and search the LangBot Space marketplaces (plugins, MCP servers, skills) through the Space MCP server. Use when an AI agent needs to discover LangBot extensions on space.langbot.app over MCP. Covers the /mcp endpoint, Personal Access Token (PAT) auth, the tool surface, and client configuration. Triggers on \"langbot space mcp\", \"search langbot plugins\", \"langbot marketplace mcp\", \"space.langbot.app mcp\".", + "references": [], + "cases": [], + "case_summaries": [], + "suites": [], + "suite_summaries": [], + "fixtures": [], + "troubleshooting": [], + "troubleshooting_summaries": [] + }, + { + "directory": "langbot-testing", + "name": "langbot-testing", + "description": "Test LangBot WebUI and core product flows with an automated browser and backend logs. Use when validating the configured LangBot frontend, pipeline Debug Chat, model provider setup and test buttons, bot and knowledge-base UI flows, or troubleshooting failed LangBot end-to-end tests.", + "references": [ + "references/agent-runner-qa-workflow.md", + "references/agent-runner-release-gate.md", + "references/dify-agent-runner.md", + "references/langrag-knowledge-base.md", + "references/local-agent-runner-coverage.md", + "references/local-agent-runner.md", + "references/mcp-stdio-testing.md", + "references/model-provider-testing.md", + "references/pipeline-debug-chat.md", + "references/plugin-e2e-smoke.md", + "references/sandbox-skill-authoring.md", + "references/troubleshooting.md", + "references/web-ui-testing.md" + ], + "cases": [ + "acp-agent-runner-debug-chat", + "agent-runner-async-db-readiness", + "agent-runner-behavior-matrix", + "agent-runner-fixture-contract", + "agent-runner-ledger-concurrency", + "agent-runner-ledger-contention", + "agent-runner-ledger-invariants", + "agent-runner-ledger-stress", + "agent-runner-live-install", + "agent-runner-qa-debug-chat", + "agent-runner-release-preflight", + "agent-runner-runtime-chaos", + "dify-agent-debug-chat", + "langrag-kb-retrieve", + "langrag-parser-golden-e2e", + "langrag-sentinel-kb-discover", + "local-agent-basic-debug-chat", + "local-agent-context-compaction-debug-chat", + "local-agent-effective-prompt-debug-chat", + "local-agent-multimodal-debug-chat", + "local-agent-nonstreaming-debug-chat", + "local-agent-plugin-tool-call-debug-chat", + "local-agent-rag-debug-chat", + "local-agent-rag-multimodal-debug-chat", + "local-agent-steering-debug-chat", + "mcp-stdio-register", + "mcp-stdio-tool-call", + "pipeline-debug-chat", + "plugin-e2e-smoke", + "provider-deepseek", + "qa-plugin-smoke-live-install", + "sandbox-skill-authoring-e2e", + "sandbox-skill-authoring-edit-existing-e2e", + "webui-login-state" + ], + "case_summaries": [ + { + "id": "acp-agent-runner-debug-chat", + "title": "ACP AgentRunner can answer through Debug Chat using real remote Claude", + "mode": "agent-browser", + "area": "pipeline", + "type": "regression", + "priority": "p2", + "risk": "high", + "ci_eligible": false, + "tags": [ + "agent-runner", + "acp", + "claude", + "external-runner", + "pipeline" + ], + "automation": "scripts/e2e/pipeline-debug-chat.mjs", + "setup_automation": [ + "node:scripts/e2e/ensure-acp-agent-runner-pipeline.mjs --write-env" + ], + "setup_provides_env": [ + "LANGBOT_ACP_AGENT_RUNNER_PIPELINE_URL", + "LANGBOT_ACP_AGENT_RUNNER_PIPELINE_NAME" + ], + "evidence_required": [ + "ui", + "console", + "backend_log" + ] + }, + { + "id": "agent-runner-async-db-readiness", + "title": "AgentRunner async DB readiness probe", + "mode": "probe", + "area": "release", + "type": "smoke", + "priority": "p0", + "risk": "high", + "ci_eligible": true, + "tags": [ + "agent-runner", + "probe", + "async-db", + "aiosqlite" + ], + "automation": "skills/langbot-testing/probes/agent-runner-async-db-readiness.mjs", + "setup_automation": [], + "setup_provides_env": [], + "evidence_required": [ + "filesystem" + ] + }, + { + "id": "agent-runner-behavior-matrix", + "title": "AgentRunner deterministic behavior matrix probe", + "mode": "probe", + "area": "release", + "type": "regression", + "priority": "p0", + "risk": "high", + "ci_eligible": true, + "tags": [ + "agent-runner", + "probe", + "deterministic-runner", + "protocol" + ], + "automation": "skills/langbot-testing/probes/agent-runner-behavior-matrix.mjs", + "setup_automation": [], + "setup_provides_env": [], + "evidence_required": [ + "filesystem" + ] + }, + { + "id": "agent-runner-fixture-contract", + "title": "QA AgentRunner fixture contract probe", + "mode": "probe", + "area": "release", + "type": "regression", + "priority": "p0", + "risk": "high", + "ci_eligible": true, + "tags": [ + "agent-runner", + "probe", + "fixture", + "deterministic-runner" + ], + "automation": "skills/langbot-testing/probes/agent-runner-fixture-contract.mjs", + "setup_automation": [], + "setup_provides_env": [], + "evidence_required": [ + "filesystem" + ] + }, + { + "id": "agent-runner-ledger-concurrency", + "title": "AgentRunner run ledger concurrency and auth pytest probe", + "mode": "probe", + "area": "release", + "type": "regression", + "priority": "p0", + "risk": "high", + "ci_eligible": true, + "tags": [ + "agent-runner", + "probe", + "pytest", + "run-ledger", + "concurrency" + ], + "automation": "skills/langbot-testing/probes/agent-runner-ledger-concurrency.mjs", + "setup_automation": [], + "setup_provides_env": [], + "evidence_required": [ + "filesystem" + ] + }, + { + "id": "agent-runner-ledger-contention", + "title": "AgentRunner ledger SQLite contention probe", + "mode": "probe", + "area": "release", + "type": "regression", + "priority": "p1", + "risk": "high", + "ci_eligible": true, + "tags": [ + "agent-runner", + "probe", + "stress", + "ledger", + "concurrency" + ], + "automation": "skills/langbot-testing/probes/agent-runner-ledger-contention.mjs", + "setup_automation": [], + "setup_provides_env": [], + "evidence_required": [ + "filesystem" + ] + }, + { + "id": "agent-runner-ledger-invariants", + "title": "AgentRunner ledger schema and status invariants probe", + "mode": "probe", + "area": "release", + "type": "regression", + "priority": "p0", + "risk": "high", + "ci_eligible": true, + "tags": [ + "agent-runner", + "probe", + "ledger", + "invariant" + ], + "automation": "skills/langbot-testing/probes/agent-runner-ledger-invariants.mjs", + "setup_automation": [], + "setup_provides_env": [], + "evidence_required": [ + "filesystem" + ] + }, + { + "id": "agent-runner-ledger-stress", + "title": "AgentRunner ledger lightweight stress probe", + "mode": "probe", + "area": "release", + "type": "regression", + "priority": "p1", + "risk": "high", + "ci_eligible": true, + "tags": [ + "agent-runner", + "probe", + "stress", + "ledger" + ], + "automation": "skills/langbot-testing/probes/agent-runner-ledger-stress.mjs", + "setup_automation": [], + "setup_provides_env": [], + "evidence_required": [ + "filesystem" + ] + }, + { + "id": "agent-runner-live-install", + "title": "QA AgentRunner package installs and registers in LangBot", + "mode": "probe", + "area": "release", + "type": "regression", + "priority": "p0", + "risk": "high", + "ci_eligible": false, + "tags": [ + "agent-runner", + "plugin", + "local-install", + "fixture" + ], + "automation": "scripts/e2e/install-qa-plugin-smoke.mjs", + "setup_automation": [], + "setup_provides_env": [], + "evidence_required": [ + "api_diagnostic", + "filesystem" + ] + }, + { + "id": "agent-runner-qa-debug-chat", + "title": "QA AgentRunner returns deterministic output through Debug Chat", + "mode": "agent-browser", + "area": "pipeline", + "type": "regression", + "priority": "p0", + "risk": "high", + "ci_eligible": false, + "tags": [ + "agent-runner", + "pipeline", + "debug-chat", + "fixture" + ], + "automation": "scripts/e2e/pipeline-debug-chat.mjs", + "setup_automation": [ + "case:agent-runner-live-install", + "node:scripts/e2e/ensure-qa-agent-runner-pipeline.mjs --write-env" + ], + "setup_provides_env": [ + "LANGBOT_QA_AGENT_RUNNER_PIPELINE_URL", + "LANGBOT_QA_AGENT_RUNNER_PIPELINE_NAME" + ], + "evidence_required": [ + "ui", + "screenshot", + "console", + "network", + "api_diagnostic" + ] + }, + { + "id": "agent-runner-release-preflight", + "title": "Agent runner release gate preflight validates environment readiness", + "mode": "agent-browser", + "area": "release", + "type": "smoke", + "priority": "p0", + "risk": "high", + "ci_eligible": false, + "tags": [ + "agent-runner", + "release-gate", + "preflight", + "environment" + ], + "automation": "scripts/e2e/agent-runner-release-preflight.mjs", + "setup_automation": [], + "setup_provides_env": [], + "evidence_required": [ + "ui", + "screenshot", + "console", + "network", + "api_diagnostic" + ] + }, + { + "id": "agent-runner-runtime-chaos", + "title": "AgentRunner SDK runtime chaos pytest probe", + "mode": "probe", + "area": "release", + "type": "regression", + "priority": "p0", + "risk": "high", + "ci_eligible": true, + "tags": [ + "agent-runner", + "probe", + "pytest", + "runtime", + "sdk" + ], + "automation": "skills/langbot-testing/probes/agent-runner-runtime-chaos.mjs", + "setup_automation": [], + "setup_provides_env": [], + "evidence_required": [ + "filesystem" + ] + }, + { + "id": "dify-agent-debug-chat", + "title": "Dify AgentRunner returns a response through Pipeline Debug Chat", + "mode": "agent-browser", + "area": "pipeline", + "type": "provider", + "priority": "p2", + "risk": "medium", + "ci_eligible": false, + "tags": [ + "dify", + "agent-runner", + "pipeline" + ], + "automation": "", + "setup_automation": [], + "setup_provides_env": [], + "evidence_required": [ + "ui", + "console", + "backend_log" + ] + }, + { + "id": "langrag-kb-retrieve", + "title": "LangRAG knowledge base ingests and retrieves a sentinel document", + "mode": "agent-browser", + "area": "knowledge", + "type": "feature", + "priority": "p1", + "risk": "medium", + "ci_eligible": false, + "tags": [ + "langrag", + "knowledge", + "rag" + ], + "automation": "scripts/e2e/langrag-kb-retrieve.mjs", + "setup_automation": [], + "setup_provides_env": [], + "evidence_required": [ + "ui", + "screenshot", + "console", + "backend_log" + ] + }, + { + "id": "langrag-parser-golden-e2e", + "title": "LangRAG and GeneralParsers retrieve a structured golden document", + "mode": "agent-browser", + "area": "knowledge", + "type": "regression", + "priority": "p1", + "risk": "high", + "ci_eligible": false, + "tags": [ + "golden", + "e2e", + "langrag", + "parser", + "knowledge" + ], + "automation": "", + "setup_automation": [], + "setup_provides_env": [], + "evidence_required": [ + "ui", + "screenshot", + "console", + "backend_log" + ] + }, + { + "id": "langrag-sentinel-kb-discover", + "title": "Existing LangRAG sentinel knowledge base is discoverable", + "mode": "probe", + "area": "knowledge", + "type": "regression", + "priority": "p1", + "risk": "medium", + "ci_eligible": false, + "tags": [ + "langrag", + "knowledge", + "rag", + "fixture" + ], + "automation": "scripts/e2e/ensure-langrag-sentinel-kb.mjs", + "setup_automation": [], + "setup_provides_env": [], + "evidence_required": [ + "api_diagnostic" + ] + }, + { + "id": "local-agent-basic-debug-chat", + "title": "Local Agent Debug Chat returns a deterministic streaming response", + "mode": "agent-browser", + "area": "pipeline", + "type": "smoke", + "priority": "p0", + "risk": "medium", + "ci_eligible": false, + "tags": [ + "local-agent", + "pipeline", + "streaming" + ], + "automation": "scripts/e2e/pipeline-debug-chat.mjs", + "setup_automation": [ + "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env" + ], + "setup_provides_env": [ + "LANGBOT_LOCAL_AGENT_PIPELINE_URL", + "LANGBOT_LOCAL_AGENT_PIPELINE_NAME" + ], + "evidence_required": [ + "ui", + "screenshot", + "console", + "backend_log" + ] + }, + { + "id": "local-agent-context-compaction-debug-chat", + "title": "Local Agent compacts long Debug Chat history and preserves older facts", + "mode": "agent-browser", + "area": "pipeline", + "type": "regression", + "priority": "p1", + "risk": "high", + "ci_eligible": false, + "tags": [ + "local-agent", + "pipeline", + "context", + "compaction" + ], + "automation": "scripts/e2e/pipeline-debug-chat.mjs", + "setup_automation": [ + "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env" + ], + "setup_provides_env": [ + "LANGBOT_LOCAL_AGENT_PIPELINE_URL", + "LANGBOT_LOCAL_AGENT_PIPELINE_NAME" + ], + "evidence_required": [ + "ui", + "screenshot", + "console", + "backend_log", + "api_diagnostic" + ] + }, + { + "id": "local-agent-effective-prompt-debug-chat", + "title": "Local Agent consumes host effective prompt after PromptPreProcessing", + "mode": "agent-browser", + "area": "pipeline", + "type": "regression", + "priority": "p1", + "risk": "high", + "ci_eligible": false, + "tags": [ + "local-agent", + "prompt", + "plugin", + "pipeline" + ], + "automation": "scripts/e2e/pipeline-debug-chat.mjs", + "setup_automation": [ + "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env", + "case:qa-plugin-smoke-live-install" + ], + "setup_provides_env": [ + "LANGBOT_LOCAL_AGENT_PIPELINE_URL", + "LANGBOT_LOCAL_AGENT_PIPELINE_NAME" + ], + "evidence_required": [ + "ui", + "console", + "backend_log" + ] + }, + { + "id": "local-agent-multimodal-debug-chat", + "title": "Local Agent Debug Chat preserves uploaded image input", + "mode": "agent-browser", + "area": "pipeline", + "type": "regression", + "priority": "p2", + "risk": "medium", + "ci_eligible": false, + "tags": [ + "local-agent", + "multimodal", + "pipeline" + ], + "automation": "scripts/e2e/pipeline-debug-chat.mjs", + "setup_automation": [ + "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env" + ], + "setup_provides_env": [ + "LANGBOT_LOCAL_AGENT_PIPELINE_URL", + "LANGBOT_LOCAL_AGENT_PIPELINE_NAME" + ], + "evidence_required": [ + "ui", + "screenshot", + "console", + "network", + "backend_log" + ] + }, + { + "id": "local-agent-nonstreaming-debug-chat", + "title": "Local Agent Debug Chat returns a deterministic non-streaming response", + "mode": "agent-browser", + "area": "pipeline", + "type": "regression", + "priority": "p1", + "risk": "medium", + "ci_eligible": false, + "tags": [ + "local-agent", + "pipeline", + "non-streaming" + ], + "automation": "scripts/e2e/pipeline-debug-chat.mjs", + "setup_automation": [ + "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env" + ], + "setup_provides_env": [ + "LANGBOT_LOCAL_AGENT_PIPELINE_URL", + "LANGBOT_LOCAL_AGENT_PIPELINE_NAME" + ], + "evidence_required": [ + "ui", + "console", + "backend_log" + ] + }, + { + "id": "local-agent-plugin-tool-call-debug-chat", + "title": "Local Agent can call a plugin-provided tool", + "mode": "agent-browser", + "area": "pipeline", + "type": "regression", + "priority": "p1", + "risk": "high", + "ci_eligible": false, + "tags": [ + "local-agent", + "plugin", + "tools", + "pipeline" + ], + "automation": "scripts/e2e/pipeline-debug-chat.mjs", + "setup_automation": [ + "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env", + "case:qa-plugin-smoke-live-install" + ], + "setup_provides_env": [ + "LANGBOT_LOCAL_AGENT_PIPELINE_URL", + "LANGBOT_LOCAL_AGENT_PIPELINE_NAME" + ], + "evidence_required": [ + "ui", + "console", + "backend_log", + "api_diagnostic" + ] + }, + { + "id": "local-agent-rag-debug-chat", + "title": "Local Agent Debug Chat answers from a LangRAG knowledge base", + "mode": "agent-browser", + "area": "pipeline", + "type": "regression", + "priority": "p1", + "risk": "high", + "ci_eligible": false, + "tags": [ + "local-agent", + "langrag", + "pipeline" + ], + "automation": "scripts/e2e/pipeline-debug-chat.mjs", + "setup_automation": [ + "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env", + "node:scripts/e2e/ensure-langrag-sentinel-kb.mjs --write-env" + ], + "setup_provides_env": [ + "LANGBOT_LOCAL_AGENT_PIPELINE_URL", + "LANGBOT_LOCAL_AGENT_PIPELINE_NAME", + "LANGBOT_LOCAL_AGENT_RAG_KB_UUID" + ], + "evidence_required": [ + "ui", + "screenshot", + "console", + "backend_log", + "api_diagnostic" + ] + }, + { + "id": "local-agent-rag-multimodal-debug-chat", + "title": "Local Agent preserves image input while using RAG context", + "mode": "agent-browser", + "area": "pipeline", + "type": "regression", + "priority": "p2", + "risk": "high", + "ci_eligible": false, + "tags": [ + "local-agent", + "rag", + "multimodal", + "pipeline" + ], + "automation": "scripts/e2e/pipeline-debug-chat.mjs", + "setup_automation": [ + "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env", + "node:scripts/e2e/ensure-langrag-sentinel-kb.mjs --write-env" + ], + "setup_provides_env": [ + "LANGBOT_LOCAL_AGENT_PIPELINE_URL", + "LANGBOT_LOCAL_AGENT_PIPELINE_NAME", + "LANGBOT_LOCAL_AGENT_RAG_KB_UUID" + ], + "evidence_required": [ + "ui", + "screenshot", + "console", + "backend_log" + ] + }, + { + "id": "local-agent-steering-debug-chat", + "title": "Local Agent injects a follow-up into an active run", + "mode": "agent-browser", + "area": "pipeline", + "type": "regression", + "priority": "p1", + "risk": "high", + "ci_eligible": false, + "tags": [ + "local-agent", + "pipeline", + "steering", + "tools" + ], + "automation": "scripts/e2e/local-agent-steering-debug-chat.mjs", + "setup_automation": [ + "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env", + "case:qa-plugin-smoke-live-install" + ], + "setup_provides_env": [ + "LANGBOT_LOCAL_AGENT_PIPELINE_URL", + "LANGBOT_LOCAL_AGENT_PIPELINE_NAME" + ], + "evidence_required": [ + "ui", + "screenshot", + "console", + "backend_log", + "api_diagnostic" + ] + }, + { + "id": "mcp-stdio-register", + "title": "MCP stdio fixture is registered and exposes qa_mcp_echo", + "mode": "agent-browser", + "area": "mcp", + "type": "smoke", + "priority": "p1", + "risk": "medium", + "ci_eligible": false, + "tags": [ + "mcp", + "tools", + "fixture", + "preflight" + ], + "automation": "scripts/e2e/mcp-stdio-register.mjs", + "setup_automation": [], + "setup_provides_env": [], + "evidence_required": [ + "screenshot", + "console", + "network", + "api_diagnostic" + ] + }, + { + "id": "mcp-stdio-tool-call", + "title": "MCP stdio server exposes a tool that Local Agent can call", + "mode": "agent-browser", + "area": "mcp", + "type": "feature", + "priority": "p1", + "risk": "high", + "ci_eligible": false, + "tags": [ + "mcp", + "tools", + "local-agent" + ], + "automation": "scripts/e2e/pipeline-debug-chat.mjs", + "setup_automation": [ + "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env", + "case:mcp-stdio-register" + ], + "setup_provides_env": [ + "LANGBOT_LOCAL_AGENT_PIPELINE_URL", + "LANGBOT_LOCAL_AGENT_PIPELINE_NAME" + ], + "evidence_required": [ + "ui", + "console", + "backend_log", + "api_diagnostic" + ] + }, + { + "id": "pipeline-debug-chat", + "title": "Pipeline Debug Chat returns a bot response", + "mode": "agent-browser", + "area": "pipeline", + "type": "smoke", + "priority": "p0", + "risk": "medium", + "ci_eligible": false, + "tags": [ + "smoke", + "pipeline" + ], + "automation": "scripts/e2e/pipeline-debug-chat.mjs", + "setup_automation": [], + "setup_provides_env": [], + "evidence_required": [ + "ui", + "screenshot", + "console", + "backend_log" + ] + }, + { + "id": "plugin-e2e-smoke", + "title": "Plugin system installs a local plugin and exposes tool/page APIs", + "mode": "agent-browser", + "area": "plugin", + "type": "smoke", + "priority": "p1", + "risk": "medium", + "ci_eligible": false, + "tags": [ + "plugin", + "runtime", + "local-install", + "e2e" + ], + "automation": "", + "setup_automation": [], + "setup_provides_env": [], + "evidence_required": [ + "ui", + "console", + "backend_log", + "api_diagnostic" + ] + }, + { + "id": "provider-deepseek", + "title": "DeepSeek provider can be configured and used", + "mode": "agent-browser", + "area": "provider", + "type": "provider", + "priority": "p2", + "risk": "medium", + "ci_eligible": false, + "tags": [ + "provider", + "model" + ], + "automation": "", + "setup_automation": [], + "setup_provides_env": [], + "evidence_required": [ + "ui", + "console", + "backend_log" + ] + }, + { + "id": "qa-plugin-smoke-live-install", + "title": "QA plugin smoke package installs and exposes tools", + "mode": "probe", + "area": "plugin", + "type": "regression", + "priority": "p1", + "risk": "medium", + "ci_eligible": false, + "tags": [ + "plugin", + "local-install", + "fixture" + ], + "automation": "scripts/e2e/install-qa-plugin-smoke.mjs", + "setup_automation": [], + "setup_provides_env": [], + "evidence_required": [ + "api_diagnostic", + "filesystem" + ] + }, + { + "id": "sandbox-skill-authoring-e2e", + "title": "Local Agent creates, registers, activates, and uses a sandbox skill", + "mode": "agent-browser", + "area": "sandbox", + "type": "regression", + "priority": "p2", + "risk": "high", + "ci_eligible": false, + "tags": [ + "sandbox", + "skills", + "local-agent", + "tools", + "e2b", + "nsjail" + ], + "automation": "", + "setup_automation": [], + "setup_provides_env": [], + "evidence_required": [ + "ui", + "backend_log", + "api_diagnostic", + "filesystem" + ] + }, + { + "id": "sandbox-skill-authoring-edit-existing-e2e", + "title": "Local Agent modifies an activated sandbox skill package", + "mode": "agent-browser", + "area": "sandbox", + "type": "regression", + "priority": "p2", + "risk": "high", + "ci_eligible": false, + "tags": [ + "sandbox", + "skills", + "local-agent", + "tools", + "edit" + ], + "automation": "scripts/e2e/pipeline-debug-chat.mjs", + "setup_automation": [], + "setup_provides_env": [], + "evidence_required": [ + "ui", + "backend_log", + "api_diagnostic", + "filesystem" + ] + }, + { + "id": "webui-login-state", + "title": "Configured frontend opens with authenticated LangBot WebUI state", + "mode": "agent-browser", + "area": "auth", + "type": "smoke", + "priority": "p0", + "risk": "low", + "ci_eligible": false, + "tags": [ + "smoke", + "auth" + ], + "automation": "scripts/e2e/webui-login-state.mjs", + "setup_automation": [], + "setup_provides_env": [], + "evidence_required": [ + "ui", + "screenshot", + "console" + ] + } + ], + "suites": [ + "agent-runner-release-gate", + "core-smoke", + "local-agent-gate" + ], + "suite_summaries": [ + { + "id": "agent-runner-release-gate", + "title": "Agent runner externalization release gate", + "description": "Release gate for runner externalization: pytest probes, environment preflight, fixture registration, local-agent capability paths, and ACP external harness execution.", + "type": "release_gate", + "priority": "p0", + "tags": [ + "agent-runner", + "release-gate", + "local-agent", + "external-runner" + ], + "cases": [ + "agent-runner-fixture-contract", + "agent-runner-behavior-matrix", + "agent-runner-ledger-invariants", + "agent-runner-async-db-readiness", + "agent-runner-ledger-stress", + "agent-runner-ledger-contention", + "agent-runner-ledger-concurrency", + "agent-runner-runtime-chaos", + "agent-runner-live-install", + "agent-runner-qa-debug-chat", + "agent-runner-release-preflight", + "webui-login-state", + "pipeline-debug-chat", + "qa-plugin-smoke-live-install", + "plugin-e2e-smoke", + "langrag-kb-retrieve", + "local-agent-basic-debug-chat", + "local-agent-effective-prompt-debug-chat", + "local-agent-context-compaction-debug-chat", + "local-agent-rag-debug-chat", + "local-agent-plugin-tool-call-debug-chat", + "mcp-stdio-register", + "mcp-stdio-tool-call", + "local-agent-nonstreaming-debug-chat", + "local-agent-multimodal-debug-chat", + "local-agent-rag-multimodal-debug-chat", + "acp-agent-runner-debug-chat" + ] + }, + { + "id": "core-smoke", + "title": "Core browser smoke suite", + "description": "Fast browser-first checks for login state, Pipeline Debug Chat, and the basic local-agent runner path.", + "type": "smoke", + "priority": "p0", + "tags": [ + "smoke", + "browser", + "p0" + ], + "cases": [ + "webui-login-state", + "pipeline-debug-chat", + "local-agent-basic-debug-chat" + ] + }, + { + "id": "local-agent-gate", + "title": "Local Agent runner regression gate", + "description": "High-signal local-agent runner checks covering prompt bridge, RAG, context compaction, plugin tools, MCP tools, non-streaming, and multimodal paths.", + "type": "regression", + "priority": "p1", + "tags": [ + "local-agent", + "agent-runner", + "regression" + ], + "cases": [ + "local-agent-basic-debug-chat", + "qa-plugin-smoke-live-install", + "local-agent-effective-prompt-debug-chat", + "local-agent-context-compaction-debug-chat", + "local-agent-rag-debug-chat", + "local-agent-plugin-tool-call-debug-chat", + "local-agent-steering-debug-chat", + "mcp-stdio-tool-call", + "local-agent-nonstreaming-debug-chat", + "local-agent-multimodal-debug-chat", + "local-agent-rag-multimodal-debug-chat" + ] + } + ], + "fixtures": [ + { + "id": "qa-agent-runner-behaviors", + "title": "Deterministic AgentRunner behavior matrix", + "kind": "json", + "path": "fixtures/agent-runner/qa-runner-behaviors.json", + "related_cases": [ + "agent-runner-behavior-matrix", + "agent-runner-ledger-invariants", + "agent-runner-runtime-chaos" + ] + }, + { + "id": "qa-agent-runner-source", + "title": "QA deterministic AgentRunner fixture source", + "kind": "plugin_source", + "path": "fixtures/plugins/qa-agent-runner/manifest.yaml", + "related_cases": [ + "agent-runner-fixture-contract", + "agent-runner-behavior-matrix", + "agent-runner-live-install", + "agent-runner-qa-debug-chat" + ] + }, + { + "id": "qa-agent-runner-package", + "title": "QA deterministic AgentRunner prebuilt package", + "kind": "plugin_package", + "path": "fixtures/plugins/qa-agent-runner/dist/qa-agent-runner-0.1.0.lbpkg", + "related_cases": [ + "agent-runner-fixture-contract", + "agent-runner-live-install", + "agent-runner-qa-debug-chat" + ] + }, + { + "id": "mcp-stdio-echo-server", + "title": "MCP stdio qa_mcp_echo server", + "kind": "python", + "path": "fixtures/mcp/qa_mcp_echo_server.py", + "related_cases": [ + "mcp-stdio-tool-call" + ] + }, + { + "id": "rag-sentinel-doc", + "title": "LangRAG sentinel text document", + "kind": "text", + "path": "fixtures/rag/sentinel-doc.txt", + "related_cases": [ + "langrag-kb-retrieve", + "local-agent-rag-debug-chat" + ] + }, + { + "id": "rag-parser-golden-html", + "title": "LangRAG parser golden HTML document", + "kind": "html", + "path": "fixtures/rag/parser-golden.html", + "related_cases": [ + "langrag-parser-golden-e2e" + ] + }, + { + "id": "multimodal-red-square", + "title": "64x64 red-square image fixture", + "kind": "base64_png", + "path": "fixtures/multimodal/red-square.png.base64", + "related_cases": [ + "local-agent-multimodal-debug-chat", + "local-agent-rag-multimodal-debug-chat" + ] + }, + { + "id": "qa-plugin-smoke-source", + "title": "QA plugin smoke fixture source", + "kind": "plugin_source", + "path": "fixtures/plugins/qa-plugin-smoke/manifest.yaml", + "related_cases": [ + "qa-plugin-smoke-live-install", + "plugin-e2e-smoke", + "local-agent-effective-prompt-debug-chat", + "local-agent-plugin-tool-call-debug-chat", + "local-agent-steering-debug-chat" + ] + }, + { + "id": "qa-plugin-smoke-package", + "title": "QA plugin smoke prebuilt package", + "kind": "plugin_package", + "path": "fixtures/plugins/qa-plugin-smoke/dist/qa-plugin-smoke-0.1.0.lbpkg", + "related_cases": [ + "qa-plugin-smoke-live-install", + "plugin-e2e-smoke" + ] + } + ], + "troubleshooting": [ + "agent-runner-actor-context-fields", + "aiosqlite-connect-hangs", + "ambiguous-runner-default-label", + "backend-not-listening", + "box-session-conflict-logical-metadata", + "debug-chat-history-contaminates-automation", + "dynamic-form-missing-config-id", + "e2b-extra-mount-sync-missing", + "local-agent-model-route-unavailable", + "marketplace-network-flaky", + "mcp-stdio-args-not-applied", + "nsjail-cli-compatibility", + "pipeline-form-controlled-warning", + "plugin-dependency-install-offline", + "plugin-runtime-timeout", + "provider-image-parse-error", + "proxy-env-mismatch", + "sandbox-native-tools-unavailable", + "socks-proxy-without-socksio", + "survey-widget-blocks-debug-chat", + "tool-name-collision-between-mcp-and-plugin", + "uv-run-resyncs-local-sdk" + ], + "troubleshooting_summaries": [ + { + "id": "agent-runner-actor-context-fields", + "title": "AgentRunner reads old actor.type and actor.id fields", + "category": "product", + "related_cases": [ + "dify-agent-debug-chat", + "pipeline-debug-chat" + ] + }, + { + "id": "aiosqlite-connect-hangs", + "title": "aiosqlite connect hangs before ledger pytest starts", + "category": "env_issue", + "related_cases": [ + "agent-runner-ledger-concurrency", + "agent-runner-ledger-invariants" + ] + }, + { + "id": "ambiguous-runner-default-label", + "title": "AgentRunner selector shows multiple Default or 默认 options", + "category": "product", + "related_cases": [ + "dify-agent-debug-chat", + "local-agent-rag-debug-chat" + ] + }, + { + "id": "backend-not-listening", + "title": "LangBot backend URL has no listening service", + "category": "product", + "related_cases": [ + "pipeline-debug-chat", + "webui-login-state", + "local-agent-basic-debug-chat" + ] + }, + { + "id": "box-session-conflict-logical-metadata", + "title": "BoxSessionConflictError after a successful first exec", + "category": "product", + "related_cases": [ + "sandbox-skill-authoring-e2e" + ] + }, + { + "id": "debug-chat-history-contaminates-automation", + "title": "Old Debug Chat messages contaminate automation assertions", + "category": "product", + "related_cases": [ + "pipeline-debug-chat", + "local-agent-plugin-tool-call-debug-chat", + "mcp-stdio-tool-call" + ] + }, + { + "id": "dynamic-form-missing-config-id", + "title": "Dynamic form fields have no stable key", + "category": "product", + "related_cases": [ + "langrag-kb-retrieve", + "local-agent-rag-debug-chat" + ] + }, + { + "id": "e2b-extra-mount-sync-missing", + "title": "Activated skill files are missing or not written back on E2B", + "category": "product", + "related_cases": [ + "sandbox-skill-authoring-e2e" + ] + }, + { + "id": "local-agent-model-route-unavailable", + "title": "Local Agent model route is unavailable for the requested run shape", + "category": "env_issue", + "related_cases": [ + "local-agent-basic-debug-chat", + "local-agent-plugin-tool-call-debug-chat", + "mcp-stdio-tool-call", + "local-agent-multimodal-debug-chat", + "local-agent-nonstreaming-debug-chat" + ] + }, + { + "id": "marketplace-network-flaky", + "title": "Marketplace requests are flaky but plugin data may still load", + "category": "product", + "related_cases": [ + "langrag-kb-retrieve" + ] + }, + { + "id": "mcp-stdio-args-not-applied", + "title": "MCP Stdio test runs only the command without arguments", + "category": "product", + "related_cases": [ + "mcp-stdio-tool-call" + ] + }, + { + "id": "nsjail-cli-compatibility", + "title": "Installed nsjail CLI rejects generated sandbox arguments", + "category": "product", + "related_cases": [ + "sandbox-skill-authoring-e2e" + ] + }, + { + "id": "pipeline-form-controlled-warning", + "title": "Pipeline form input switches from uncontrolled to controlled", + "category": "product", + "related_cases": [ + "pipeline-debug-chat", + "local-agent-rag-debug-chat" + ] + }, + { + "id": "plugin-dependency-install-offline", + "title": "Local plugin dependency install fails in an offline or proxy-limited runtime", + "category": "product", + "related_cases": [ + "langrag-parser-golden-e2e" + ] + }, + { + "id": "plugin-runtime-timeout", + "title": "Plugin runtime actions time out", + "category": "product", + "related_cases": [ + "pipeline-debug-chat", + "provider-deepseek", + "langrag-parser-golden-e2e" + ] + }, + { + "id": "provider-image-parse-error", + "title": "Model provider rejects the uploaded image before runner behavior is exercised", + "category": "product", + "related_cases": [ + "local-agent-multimodal-debug-chat" + ] + }, + { + "id": "proxy-env-mismatch", + "title": "Uppercase and lowercase proxy variables differ", + "category": "product", + "related_cases": [ + "pipeline-debug-chat", + "provider-deepseek", + "webui-login-state" + ] + }, + { + "id": "sandbox-native-tools-unavailable", + "title": "Native sandbox tools are unavailable even though a backend is configured", + "category": "product", + "related_cases": [ + "sandbox-skill-authoring-e2e" + ] + }, + { + "id": "socks-proxy-without-socksio", + "title": "Python HTTP clients fail when ALL_PROXY uses SOCKS without socksio", + "category": "product", + "related_cases": [ + "sandbox-skill-authoring-e2e", + "provider-deepseek" + ] + }, + { + "id": "survey-widget-blocks-debug-chat", + "title": "Survey widget blocks Debug Chat controls", + "category": "product", + "related_cases": [ + "pipeline-debug-chat", + "local-agent-rag-debug-chat", + "mcp-stdio-tool-call" + ] + }, + { + "id": "tool-name-collision-between-mcp-and-plugin", + "title": "MCP and plugin expose the same tool name", + "category": "product", + "related_cases": [ + "mcp-stdio-tool-call", + "local-agent-plugin-tool-call-debug-chat" + ] + }, + { + "id": "uv-run-resyncs-local-sdk", + "title": "uv run resyncs the locked SDK instead of the local editable SDK", + "category": "product", + "related_cases": [ + "plugin-e2e-smoke" + ] + } + ] + } + ] +} diff --git a/skills/skills/.env b/skills/skills/.env new file mode 100644 index 000000000..4e5aae69f --- /dev/null +++ b/skills/skills/.env @@ -0,0 +1,46 @@ +# Shared defaults for LangBot skills. +# Agents should read this file first, then load machine-local overrides from +# skills/.env.local. Do not put workstation-specific absolute paths or secrets +# in this committed file. + +# The UI URL that testing skills should open. +# Default to the standalone Vite frontend. Set this to the backend WebUI URL +# instead if your LangBot checkout serves the frontend from the backend. +LANGBOT_FRONTEND_URL=http://127.0.0.1:3000 + +# LangBot API/backend URL. +LANGBOT_BACKEND_URL=http://127.0.0.1:5300 + +# Common standalone frontend dev URL. This is a candidate, not the default. +LANGBOT_DEV_FRONTEND_URL=http://127.0.0.1:3000 + +# Local repository paths. Copy skills/.env.example to skills/.env.local and set +# these for your checkout. +LANGBOT_REPO= +LANGBOT_WEB_REPO= +LANGBOT_RAG_PLUGIN_REPO= +LANGBOT_PARSER_PLUGIN_REPO= + +# Browser profile and Playwright/Chromium paths. +LANGBOT_BROWSER_PROFILE= +LANGBOT_CHROMIUM_EXECUTABLE= + +# Optional local proxy defaults. Do not store secrets here. +LANGBOT_PROXY_HTTP= +LANGBOT_PROXY_SOCKS= +LANGBOT_NO_PROXY=localhost,127.0.0.1,::1 + +# Optional case-specific pipeline targets. Put machine-local values in +# skills/.env.local so runner-specific cases do not accidentally reuse the +# generic LANGBOT_PIPELINE_URL. +# LANGBOT_PIPELINE_URL=http://127.0.0.1:3000/home/pipelines?id= +# LANGBOT_PIPELINE_NAME=Generic QA Pipeline +# LANGBOT_LOCAL_AGENT_PIPELINE_URL=http://127.0.0.1:3000/home/pipelines?id= +# LANGBOT_LOCAL_AGENT_PIPELINE_NAME=Local Agent QA Pipeline +# LANGBOT_ACP_AGENT_RUNNER_PIPELINE_URL=http://127.0.0.1:3000/home/pipelines?id= +# LANGBOT_ACP_AGENT_RUNNER_PIPELINE_NAME=ACP AgentRunner QA Pipeline +# LANGBOT_ACP_AGENT_RUNNER_SSH_TARGET=yhh@101.34.71.12 +# LANGBOT_ACP_AGENT_RUNNER_SSH_PORT=22 +# LANGBOT_ACP_AGENT_RUNNER_SSH_IDENTITY_FILE= +# LANGBOT_ACP_AGENT_RUNNER_SSH_EXTRA_OPTIONS= +# LANGBOT_ACP_AGENT_RUNNER_REMOTE_WORKSPACE=/home/yhh/langbot-e2e/acp-workspace diff --git a/skills/skills/.env.example b/skills/skills/.env.example new file mode 100644 index 000000000..a8f5ebf09 --- /dev/null +++ b/skills/skills/.env.example @@ -0,0 +1,36 @@ +# Copy this file to skills/.env.local and adjust it for your machine. +# Do not put API keys, OAuth tokens, browser localStorage tokens, or provider +# credentials in committed files. + +# LangBot WebUI and backend endpoints. +LANGBOT_FRONTEND_URL=http://127.0.0.1:3000 +LANGBOT_BACKEND_URL=http://127.0.0.1:5300 +LANGBOT_DEV_FRONTEND_URL=http://127.0.0.1:3000 + +# Local repository paths. +LANGBOT_REPO=/path/to/LangBot +LANGBOT_WEB_REPO=/path/to/LangBot/web +LANGBOT_RAG_PLUGIN_REPO=/path/to/langbot-rag +LANGBOT_PARSER_PLUGIN_REPO=/path/to/langbot-parser + +# Browser profile and Playwright/Chromium paths. +LANGBOT_BROWSER_PROFILE=/path/to/langbot-playwright-profile +LANGBOT_CHROMIUM_EXECUTABLE=/path/to/ms-playwright/chromium/chrome + +# Optional local proxy defaults. Leave blank if not needed. +LANGBOT_PROXY_HTTP= +LANGBOT_PROXY_SOCKS= +LANGBOT_NO_PROXY=localhost,127.0.0.1,::1 + +# Optional generic pipeline target for generic Debug Chat smoke tests. +LANGBOT_PIPELINE_URL= +LANGBOT_PIPELINE_NAME= + +# Optional case-specific runner targets. Prefer these for runner-specific cases +# so the automation cannot silently test the wrong runner. +LANGBOT_LOCAL_AGENT_PIPELINE_URL= +LANGBOT_LOCAL_AGENT_PIPELINE_NAME= +LANGBOT_CODEX_AGENT_PIPELINE_URL= +LANGBOT_CODEX_AGENT_PIPELINE_NAME= +LANGBOT_CLAUDE_CODE_AGENT_PIPELINE_URL= +LANGBOT_CLAUDE_CODE_AGENT_PIPELINE_NAME= diff --git a/skills/skills/langbot-deploy/SKILL.md b/skills/skills/langbot-deploy/SKILL.md new file mode 100644 index 000000000..3a314fbfa --- /dev/null +++ b/skills/skills/langbot-deploy/SKILL.md @@ -0,0 +1,84 @@ +--- +name: langbot-deploy +description: Deploy and configure a LangBot instance — Docker / Docker Compose, Kubernetes, the config.yaml model, the Box sandbox runtime, the plugin runtime, and the global API key. Use when installing, deploying, upgrading, or configuring LangBot in production or self-hosted environments. Triggers on "deploy langbot", "langbot docker", "langbot compose", "langbot kubernetes", "langbot config.yaml", "langbot box runtime", "langbot global api key". +--- + +# LangBot Deployment & Configuration + +Covers running LangBot in production. For development see `langbot-dev`. + +## Docker Compose (recommended) + +```bash +git clone https://github.com/langbot-app/LangBot +cd LangBot/docker + +# Full stack (sandbox/Box + stdio MCP hosting + skill add/edit enabled) +docker compose --profile all up + +# Basic (no Box runtime) +docker compose up +``` + +The `all` / `box` profile starts three services: + +- `langbot` — main app, serves API + UI on `:5300`. +- `langbot_plugin_runtime` — plugin runtime (control `:5400`, debug `:5401`). +- `langbot_box` — Box sandbox runtime (`:5410`). Uses the host Docker socket to + spawn sandbox containers, so the **Box root host path and in-container path + must be identical** (`BOX__LOCAL__HOST_ROOT=${LANGBOT_BOX_ROOT:-${PWD}/data/box}`). + +With Box off, the dashboard/skills list stays visible (read-only) but sandbox +tools, skill add/edit, and stdio MCP are disabled. Set `box.enabled: false` +(or `BOX__ENABLED=false`) to match. + +## Kubernetes + +See `docker/kubernetes.yaml` and the deployment guide at +https://docs.langbot.app. `docker/deploy-k8s-test.sh` is a test helper. + +## config.yaml (generated at `data/config.yaml` on first run) + +Top-level sections: `api`, `system`, `command`, `concurrency`, `proxy`, +`database`, `vdb`, `storage`, `plugin`, `monitoring`, `box`, `space`. + +Key settings: + +| Key | Meaning | +| --- | --- | +| `api.port` | HTTP API + UI port (default 5300) | +| `api.global_api_key` | **Global API key** for the HTTP API + MCP server. Non-empty = accepted with no login/DB record; no `lbk_` prefix required. Empty = disabled. Plaintext — trusted/internal only, serve over HTTPS. | +| `plugin.runtime_ws_url` | Standalone plugin runtime WS URL (e.g. `ws://langbot_plugin_runtime:5400/control/ws`) | +| `box.enabled` | Master switch for the Box sandbox runtime | +| `box.backend` | `local` (Docker/nsjail autopick) / `docker` / `nsjail` / `e2b`; env override `BOX__BACKEND` | +| `box.runtime.endpoint` | External Box runtime URL (e.g. `ws://127.0.0.1:5410`); empty = local auto-managed | + +Many keys have `ENV__SUBKEY` overrides (e.g. `BOX__BACKEND`, `BOX__ENABLED`). + +## Runtimes & flags + +- LangBot started directly spawns the plugin runtime over **stdio**. +- In containers it connects to a standalone runtime over **WebSocket**; start + with `--standalone-runtime`. +- Box has a parallel `--standalone-box` flag; the Docker box host is + `langbot_box:5410`. + +## Global API key — enabling for agents/automation + +```yaml +# data/config.yaml +api: + port: 5300 + global_api_key: 'a-strong-secret' # empty disables it +``` + +This key authenticates both the HTTP API and the MCP server (`/mcp`) without a +login session. See `langbot-mcp-ops` for using it, and `docs/API_KEY_AUTH.md`. + +## Pitfalls + +- "No supported sandbox backend (Docker / nsjail / E2B)" with Docker running + usually means the user isn't in the `docker` group → + `sudo usermod -aG docker ` and restart in a new shell. +- Box root host/container path mismatch breaks sandbox container creation. +- Don't commit a non-empty `api.global_api_key` to version control. diff --git a/skills/skills/langbot-dev/SKILL.md b/skills/skills/langbot-dev/SKILL.md new file mode 100644 index 000000000..01ee06fd1 --- /dev/null +++ b/skills/skills/langbot-dev/SKILL.md @@ -0,0 +1,116 @@ +--- +name: langbot-dev +description: Develop, build, and debug the LangBot core backend and web frontend. Use when working inside the LangBot repository — backend (Python/Quart, src/langbot/pkg), the Vite/React web UI, HTTP API controllers/services, Alembic migrations, or the MCP server. Covers the dev environment (uv, pnpm), repo layout, the API auth model (user token / API key / global key), adding API endpoints, and the rule that API changes must update the MCP server and skills. Triggers on "langbot backend", "langbot dev", "langbot api", "add langbot endpoint", "langbot migration". +--- + +# LangBot Core Development + +This skill covers developing the LangBot core (the main repo), distinct from +plugin development (see `langbot-plugin-dev`) and deployment (`langbot-deploy`). + +## Stack + +- **Backend**: Python `>=3.11,<4.0`, deps via `uv`. Framework: **Quart** (async + Flask). Serves the HTTP API + pre-built web UI on `http://127.0.0.1:5300`. +- **Frontend** (`web/`): **Vite + React Router 7 + shadcn/ui + Tailwind**, + managed by `pnpm`. Dev server on `:3000`. (NOT Next.js — `dev` script is `vite`.) + +## Dev environment + +```bash +# Backend +pip install uv +uv sync --dev +uv run main.py # API + UI on http://127.0.0.1:5300 + +# Frontend (separate terminal) +cd web +cp .env.example .env +pnpm install +pnpm dev # http://127.0.0.1:3000 (reads VITE_API_BASE_URL) + +# Lint/format hooks (CI runs the same checks) +uv run pre-commit install +``` + +First run generates `data/config.yaml`; DB defaults to SQLite (PostgreSQL +supported). Migrations run automatically on startup. + +## Repo layout (key paths) + +``` +src/langbot/ +├── __main__.py # entrypoint, CLI flags (--standalone-runtime/-box/--debug) +├── pkg/ +│ ├── api/ +│ │ ├── http/ # Quart controllers + services +│ │ │ ├── controller/groups/ # route groups (@group.group_class) +│ │ │ └── service/ # business logic (called by controllers AND MCP) +│ │ └── mcp/ # MCP server (server.py = tools, mount.py = ASGI dispatch) +│ ├── core/ # app bootstrap, stages, task manager +│ ├── platform/ provider/ pipeline/ plugin/ box/ skill/ rag/ vector/ +│ ├── command/ persistence/ storage/ config/ entity/ telemetry/ +│ └── templates/config.yaml # config template (top-level: api, system, plugin, box, space...) +├── web/ # Vite SPA +└── docker/ # compose deployment +``` + +## HTTP API auth model + +Route auth is declared per-route via `AuthType` in +`pkg/api/http/controller/group.py`: + +- `NONE` — public. +- `USER_TOKEN` — web UI JWT (`Authorization: Bearer `). +- `API_KEY` — `X-API-Key` or `Authorization: Bearer `. +- `USER_TOKEN_OR_API_KEY` — either. + +API keys are verified by `apikey_service.verify_api_key()`, which accepts: +1. the **global key** from `config.yaml` `api.global_api_key` (no DB, no login, + no `lbk_` prefix required), then +2. **web-UI keys** (DB-stored, `lbk_` prefix). + +Route groups self-register via `@group.group_class(name, path)` and are +discovered by `importutil.import_modules_in_pkg`. + +## Adding an API endpoint + +1. Add/extend a controller in `pkg/api/http/controller/groups/` and the matching + service method in `pkg/api/http/service/`. +2. Pick the right `AuthType`. +3. **If the endpoint should be agent-accessible, add/adjust the matching MCP tool + in `pkg/api/mcp/server.py` and update the `langbot-mcp-ops` skill.** API and + MCP surface must stay aligned (see `AGENTS.md`). +4. Update `docs/service-api-openapi.json` if you maintain the OpenAPI overview. + +## Database migrations (Alembic) + +Single migration set supports SQLite + PostgreSQL. Files in +`src/langbot/pkg/persistence/alembic/versions/`. + +```bash +# From project root (needs data/config.yaml) +uv run python -m langbot.pkg.persistence.alembic_runner autogenerate "description" +``` + +## Standards + +- All code comments/docstrings in **English**; user-facing strings need **i18n** + (`en_US` + `zh_Hans` minimum, `ja_JP` where present). +- Consider toC and toB compatibility + security. +- Commit format: `(): ` (feat/fix/docs/refactor/...). + +## Tests + +```bash +uv run pytest tests/unit_tests -q # unit tests +uv run pytest tests/unit_tests/api -q # API service tests +uv run python tests/manual/mcp_smoke.py # MCP server e2e smoke +``` + +## See also + +- `langbot-plugin-dev` — plugin SDK / runtime development. +- `langbot-testing` — WebUI/e2e QA harness (`bin/lbs`). +- `langbot-deploy` — Docker/compose deployment + config. +- `langbot-mcp-ops` — operating the LangBot MCP server. diff --git a/skills/skills/langbot-eba-adapter-dev/SKILL.md b/skills/skills/langbot-eba-adapter-dev/SKILL.md new file mode 100644 index 000000000..b0fd180cd --- /dev/null +++ b/skills/skills/langbot-eba-adapter-dev/SKILL.md @@ -0,0 +1,301 @@ +--- +name: langbot-eba-adapter-dev +description: Build, refactor, and test LangBot platform adapters for the Event-Based Agents architecture. Use when adding or migrating Telegram, Discord, or other messaging platform adapters to the EBA adapter layout, validating unified event/message conversion, writing live adapter probes, or using standalone plugin runtime plus Computer Use for end-to-end platform testing. +--- + +# LangBot EBA Adapter Development + +Use this skill when implementing or reviewing a LangBot platform adapter under the Event-Based Agents architecture. + +## Controlling a running instance via MCP + +Beyond writing code, you can **drive a live LangBot instance over MCP** — no raw +HTTP needed. Two MCP servers exist (both reuse existing API keys; see `AGENTS.md`): + +- **LangBot instance** — `http://:5300/mcp` (auth: web-UI `lbk_` key or the + `api.global_api_key` from `config.yaml`). Manage bots, pipelines, models, + knowledge bases, and skills. See the **`langbot-mcp-ops`** skill. +- **LangBot Space marketplace** — `https://space.langbot.app/mcp` (auth: Personal + Access Token). Search plugins / MCP servers / skills. See the + **`langbot-space-ops`** skill. + +> Any change to an agent-accessible HTTP API endpoint must keep the matching MCP +> tool and these skills in sync. + +## Core Rule + +Do not let platform-native event or message shapes leak into LangBot's common path. Each adapter must convert incoming SDK objects into unified EBA entities before dispatch: + +- Events: `langbot_plugin.api.entities.builtin.platform.events` +- Message chains: `langbot_plugin.api.entities.builtin.platform.message.MessageChain` +- Users/groups/members: `langbot_plugin.api.entities.builtin.platform.entities` +- Raw platform objects may remain only in `source_platform_object` for debugging or platform-specific escape hatches. + +## Start Here + +1. Read the EBA design docs in `LangBot/docs/event-based-agents/`. +2. Read the architecture-level acceptance checklist before writing or validating code: + - `LangBot/docs/event-based-agents/adapters/acceptance-checklist.md` +3. Read the current reference adapter before writing code. Prefer Telegram first: + - `LangBot/src/langbot/pkg/platform/adapters/telegram/` + - `LangBot/docs/event-based-agents/adapters/telegram.md` +4. Read the legacy source adapter for the target platform: + - `LangBot/src/langbot/pkg/platform/sources/.py` + - `LangBot/src/langbot/pkg/platform/sources/.yaml` +5. Inspect SDK entity definitions in `langbot-plugin-sdk/src/langbot_plugin/api/entities/builtin/platform/`. +6. Search before assuming APIs. Platform SDKs change often. + +## Adapter Layout + +Create one directory per adapter: + +```text +LangBot/src/langbot/pkg/platform/adapters// +├── __init__.py +├── adapter.py +├── api_impl.py +├── event_converter.py +├── manifest.yaml +├── message_converter.py +├── platform_api.py +├── types.py +└── .svg +``` + +Add optional helpers such as `voice.py` only when the platform has a real domain-specific surface. + +Ensure `pyproject.toml` package data includes adapter assets: + +```toml +package-data = { "langbot" = ["templates/**", "pkg/platform/sources/*", "pkg/platform/adapters/**", ...] } +``` + +## Implementation Checklist + +- `manifest.yaml` declares `metadata.name`, config schema, supported events, common APIs, and platform-specific APIs. +- `adapter.py` creates the platform client, subscribes to native events, filters self/bot loops where appropriate, calls `event_converter.target2yiri(...)`, then dispatches the EBA event. +- `event_converter.py` maps native events to EBA event classes such as `MessageReceivedEvent`, `MessageEditedEvent`, `MessageDeletedEvent`, `MessageReactionEvent`, `MemberJoinedEvent`, `BotInvitedToGroupEvent`, and `PlatformSpecificEvent`. +- `message_converter.py` maps native messages to `MessageChain`, and maps `MessageChain` back to the platform send format. +- `api_impl.py` implements common EBA APIs: send, reply, edit, delete, forward, user/group/member lookup, moderation, upload/file URL, leave group. +- `platform_api.py` keeps platform-specific calls behind `call_platform_api(action, params)`. +- Unsupported common APIs must raise explicit SDK platform errors such as `NotSupportedError`; do not silently no-op. +- Destructive APIs such as kick, ban, leave, delete, or moderation must be gated in live tests and documented. + +## Conversion Contract + +For message events, the common shape should look like this regardless of platform: + +```python +platform_events.MessageReceivedEvent( + type="message.received", + adapter_name="", + message_id=, + message_chain=platform_message.MessageChain([...]), + sender=platform_entities.User(...), + chat_type=platform_entities.ChatType.PRIVATE or ChatType.GROUP, + chat_id=, + group=platform_entities.UserGroup(...) or None, + source_platform_object=, +) +``` + +Message content should use common components: + +- `Source` for original message id/time when available. +- `Plain` for text. +- `At` / `AtAll` for mentions. +- `Image`, `Voice`, `File` for media. +- `Forward` only when the platform can represent or emulate it safely. + +If a platform event cannot cleanly map to a common event, emit `PlatformSpecificEvent` with a compact `action` and structured `data`. + +## Unit Tests + +Add focused tests under `LangBot/tests/unit_tests/platform/test__eba_adapter.py`. + +Cover at least: + +- Manifest supported events match adapter `supported_events()`. +- Manifest supported APIs match adapter `supported_apis()`. +- Platform API map matches manifest actions. +- Dispatcher chooses the most specific EBA listener. +- Message converter maps every supported common component both directions where possible: + - `Source` + - `Plain` + - `At` + - `AtAll` + - `Image` + - `Voice` + - `File` + - `Quote` + - `Face` + - `Forward` + - `Unknown` + - mixed chains preserving order +- Event converter maps message received/edited/deleted/reaction, raw uncached gateway events, member events, and bot join/leave events. +- Send/reply methods pass correct platform kwargs and return `MessageResult`. + +Run the existing reference adapter tests too: + +```bash +cd LangBot +uv run pytest tests/unit_tests/platform/test__eba_adapter.py tests/unit_tests/platform/test_telegram_eba_adapter.py +uv run python -m py_compile tests/e2e/live__eba_probe.py +git diff --check +``` + +## Live Test Workflow + +Direct adapter live probes are useful diagnostics, but they are not sufficient acceptance evidence for EBA. Treat `tests/e2e/live__eba_probe.py` as an auxiliary tool only. The final adapter record must distinguish: + +- `plugin-e2e-ui`: real SDK plugin through standalone runtime, LangBot core, adapter, and a real/simulator UI action. This can mark an inbound UI item complete. +- `plugin-e2e-protocol`: real SDK plugin through standalone runtime, LangBot core, adapter, and a protocol-boundary injected event. This is useful evidence but must not be claimed as UI coverage. +- `plugin-e2e-outbound`: real SDK plugin calls an API and the bot output is visible in the real/simulator UI. This can mark send/API coverage complete. +- `adapter-live`: direct adapter probe connected to a real/simulator endpoint. This is auxiliary only. +- `unit`: mocked conversion/API-shape coverage. This is auxiliary only. +- `not-supported`: platform protocol or SDK has no equivalent. Must include the reason. +- `blocked`: intended capability could not be verified. This is not complete. + +Write a live probe in `LangBot/tests/e2e/live__eba_probe.py`. It should: + +1. Read token/client ids from environment variables or CLI args. +2. Start the adapter directly. +3. Register an EBA listener and write JSONL evidence to `LangBot/data/temp/`. +4. Wait for a real user/platform event instead of fabricating the entrypoint. +5. Exercise common APIs and `call_platform_api` actions. +6. Observe returned gateway events for edit/delete/reaction/member/bot lifecycle where available. +7. Print a summary containing passed, failed, skipped, and observed event types. +8. Redact or avoid printing secrets. +9. Keep destructive operations behind flags and run them last. + +Use Computer Use when the user asks for real platform end-to-end coverage. Actually send messages/click reactions in the platform UI or otherwise trigger real user-side events; do not replace that with unit tests. + +For media/component acceptance, keep the direction and trigger source explicit: + +- Real inbound media only counts when a human-side platform UI or simulator UI sends the image/file/voice to the bot and the plugin JSONL records the corresponding common component. +- Bot outbound media only proves `send_message`/adapter send conversion. It does not prove inbound conversion. +- Protocol-boundary injection, such as sending a OneBot event directly into a reverse WebSocket adapter, is useful and should be labelled `plugin-e2e-protocol`, but it must not be reported as UI-level end-to-end media upload. +- If the UI cannot send or upload the media, record the item as `blocked` with the exact client/simulator limitation. + +## Standalone Runtime + Plugin Test + +When validating the whole LangBot EBA path, test with the SDK standalone runtime and a real test plugin. This is the required acceptance path; direct adapter calls do not prove the EBA architecture path. + +The required path is: + +```text +Real platform / simulator UI + -> platform SDK native event + -> adapter event converter + -> unified EBA event/entity/message types + -> LangBot core event dispatch + -> standalone SDK runtime + -> real test plugin listener + -> plugin calls platform APIs through SDK + -> LangBot core API dispatch + -> adapter API implementation + -> real platform / simulator UI +``` + +Typical shape: + +```bash +# Terminal 1, SDK repo +cd langbot-plugin-sdk +uv run python -m langbot_plugin.cli.__init__ rt \ + --debug-only \ + --ws-control-port 5400 \ + --ws-debug-port 5401 \ + --skip-deps-check + +# Terminal 2, LangBot repo +cd LangBot +export PYTHONPATH=/absolute/path/to/langbot-plugin-sdk/src:${PYTHONPATH:-} +uv run main.py --standalone-runtime + +# Terminal 3, plugin directory +export DEBUG_RUNTIME_WS_URL=ws://127.0.0.1:5401/plugin/ws +export EBA_PROBE_LOG=/absolute/path/to/LangBot/data/temp/_eba_plugin_probe.jsonl +export EBA_PROBE_API=1 +export EBA_PROBE_COMPONENT_SWEEP=1 +export EBA_PROBE_PLATFORM_API=1 +uv --project /absolute/path/to/langbot-plugin-sdk run python -m langbot_plugin.cli.__init__ run +``` + +Use an EBA probe plugin that subscribes to all relevant EBA event classes and runs SDK API calls after the first `MessageReceived`. + +The plugin evidence should be JSONL and include: + +- event class and `event.type` +- adapter name +- chat type and chat ID +- sender/user/group IDs with secrets redacted +- `bot_uuid` and `adapter_name`, proving LangBot filled common routing fields before plugin dispatch +- received `message_chain` component list +- API action name, input summary, result or error +- unsupported or blocked reason when an item is skipped + +For full adapter acceptance, enable both probe sweeps: + +- `EBA_PROBE_COMPONENT_SWEEP=1` sends the required outbound message components through `send_message`. +- `EBA_PROBE_PLATFORM_API=1` calls common safe APIs plus selected `call_platform_api` actions for the adapter. + +The SDK must support `plugin.call_platform_api(bot_uuid, action, params)` for platform-specific acceptance. If the SDK cannot call a platform-specific action from the plugin, the adapter cannot be fully accepted even if direct adapter probes pass. + +## Required EBA Acceptance Coverage + +Before marking an adapter migrated, fill out an adapter record against `LangBot/docs/event-based-agents/adapters/acceptance-checklist.md`. + +At minimum, the record must cover these categories: + +- Message receive component tests through `plugin-e2e-ui`: `Source`, `Plain`, `At`, `AtAll`, `Image`, `Voice`, `File`, `Quote`, `Face`, `Forward`, `Unknown`, and mixed chains where the platform supports them. Protocol-only receive evidence must be labelled `plugin-e2e-protocol`. +- Message send component tests through `plugin-e2e-outbound`: `Plain`, `At`, `AtAll`, `Image`, `Voice`, `File`, `Quote`, `Face`, `Forward`, and mixed chains where the platform supports them. +- Every event declared in `manifest.yaml -> spec.supported_events`. +- Every common API declared in `manifest.yaml -> spec.supported_apis.required` and `optional`. +- Every action declared in `manifest.yaml -> spec.platform_specific_apis`. +- Compatibility tests for manifest declarations, legacy message listener fallback, EBA listener specificity, bot self-message filtering, and `source_platform_object` reply/debug behavior. + +Do not declare an event or API in the manifest unless it has an implementation path and an acceptance entry. If a platform or simulator lacks a capability, document it as `not-supported` or `blocked` rather than silently omitting the test. + +## Common Pitfalls + +- `get_bots()` may return bot dictionaries, not UUID strings. Probe plugins should select an enabled dict and pass `bot["uuid"]` to `get_bot_info()` and `send_message()`. +- Make sure the probe subscribes to every event you claim to verify. Missing `MessageDeleted` subscription can make a working adapter look untested. +- Some platforms emit both cached and raw gateway events, producing duplicate evidence for delete/reaction. Count this explicitly; do not treat duplicates as failure unless semantics differ. +- Self-message filtering is platform-specific. Filter bot-originated `message.received` loops, but do not accidentally filter edit/delete events needed for bot-owned API probes. +- Reaction events may be filtered for bot self reactions. To test user reaction add/remove, use real UI interaction or a real user token path if permitted. +- File uploads usually happen as message attachments. A standalone `upload_file` API may need to be `NotSupportedError`. +- Live probes should not leak bot tokens through command output, logs, docs, or final answers. +- Discord requires privileged intents for message content and members. Missing intents can look like converter bugs. +- Telegram Bot API exposes only limited member lists; document capability gaps. +- Do not mark moderation APIs verified unless they ran against a disposable target member/bot. +- If `leave_group` is tested, run it last because the test bot will be removed from the server/group. +- Restore local LangBot DB/test state after live runs if you enabled temporary bots or changed plugin settings. + +## Documentation Record + +Add or update `LangBot/docs/event-based-agents/adapters/.md` in the same style as Telegram: + +- Status and adapter directory. +- Configuration table matching manifest fields. +- Supported EBA event list. +- Common API table with support and limitations. +- `call_platform_api` action list. +- Receive component table with evidence level per component. +- Send component table with evidence level per component. +- Event table with evidence level per event. +- Common API table with evidence level per API. +- Platform-specific API table with evidence level per action. +- Live test record with exact date, endpoint/simulator, standalone runtime command, test plugin path/name, JSONL evidence path, channel/group type, observed events, APIs exercised, destructive operations, and skipped items. + +Be honest. Put untested or skipped APIs in the document with the reason. Do not imply full parity when a platform cannot provide the same information density. + +## Before Finishing + +- Run unit tests and compile the live probe. +- Run the standalone runtime plugin E2E path for every required acceptance item that the platform supports. +- Run `git diff --check`. +- Summarize live JSONL evidence by event type. +- Stop all long-running runtimes and probes. +- Confirm no secrets are staged. +- Leave unrelated untracked files alone. diff --git a/skills/skills/langbot-env-setup/SKILL.md b/skills/skills/langbot-env-setup/SKILL.md new file mode 100644 index 000000000..887a46713 --- /dev/null +++ b/skills/skills/langbot-env-setup/SKILL.md @@ -0,0 +1,28 @@ +--- +name: langbot-env-setup +description: Prepare a local LangBot development and testing environment for an AI agent. Use when setting up WSL or Linux development, shared local URL variables, proxy variables, backend/frontend startup, Playwright MCP browser access, GitHub OAuth browser login, persisted Chrome profiles, or future Codex computer-use environment paths. +--- + +# LangBot Environment Setup + +Use this skill when a task needs LangBot to be in a testable state before product testing or development verification. + +## Routing + +- **Shared local variables**: read `../.env` before using URL, path, browser profile, or proxy defaults. +- **Always start here**: read `references/browser-access-selection.md` to choose the browser-control path. +- **LangBot service checks and startup**: read `references/service-startup.md`. +- **Computer Use available**: read `references/computer-use.md`. This path usually needs less browser/MCP setup. +- **No Computer Use, browser automation required**: read `references/playwright-mcp.md`. +- **GitHub OAuth or persisted login profile**: read `references/oauth-browser-profile.md`. +- **WSL-specific notes**: read `references/wsl-notes.md` only when running under WSL. +- **Proxy setup**: read `references/proxy.md` when external login, model provider tests, or package downloads time out. +- **Headless-only automation**: use only after a profile already contains a valid LangBot login. Do not ask the agent to enter GitHub credentials or 2FA. + +## Rules + +- Never handle the user's GitHub password, passkey, recovery code, or 2FA secret. +- For OAuth login, open a visible browser and let the user complete the credential steps. +- Reuse a fixed browser profile path so the agent can later access the logged-in LangBot session. +- Keep environment-specific paths and commands in `references/`, not in this file. +- Treat environment setup as complete only after the target LangBot services are reachable and the browser profile can access the WebUI. diff --git a/skills/skills/langbot-env-setup/references/browser-access-selection.md b/skills/skills/langbot-env-setup/references/browser-access-selection.md new file mode 100644 index 000000000..d1f98217e --- /dev/null +++ b/skills/skills/langbot-env-setup/references/browser-access-selection.md @@ -0,0 +1,15 @@ +# Browser Access Selection + +Choose the lightest browser-control path that can complete the task. + +## Decision Order + +1. If Codex Computer Use, Claude Computer Use, or another visible browser-control tool is available, use `computer-use.md`. +2. If no computer-control tool is available but Playwright MCP is available, use `playwright-mcp.md`. +3. If the browser session must survive restarts or OAuth login is required, also use `oauth-browser-profile.md`. +4. If running under WSL, add `wsl-notes.md`. +5. If external sites or model providers time out, add `proxy.md`. + +## Principle + +Computer Use and Playwright MCP are alternative browser-control paths. Both still need LangBot services to be reachable, so service checks stay in `service-startup.md`. diff --git a/skills/skills/langbot-env-setup/references/computer-use.md b/skills/skills/langbot-env-setup/references/computer-use.md new file mode 100644 index 000000000..f0bd8b182 --- /dev/null +++ b/skills/skills/langbot-env-setup/references/computer-use.md @@ -0,0 +1,20 @@ +# Computer Use Browser Path + +Use this path when Codex Computer Use, Claude Computer Use, or another agent-visible browser-control capability is available. + +## Why This Path Is Simpler + +Computer Use can interact with a visible browser directly, so it usually does not need Playwright MCP configuration or a separate MCP browser bridge. + +## Workflow + +1. Verify LangBot backend/frontend with `service-startup.md`. +2. Open the WebUI in the controlled browser. +3. If login is needed, let the user complete GitHub OAuth. Never handle credentials or 2FA. +4. Keep the browser/profile available for later testing. +5. Hand off to `langbot-testing` after the page shows the logged-in WebUI. + +## Still Required + +- Proxy may still be needed for GitHub OAuth or model provider tests. Use `proxy.md`. +- Persisted profile details may still matter if the computer-control browser is restarted. Use `oauth-browser-profile.md` if login state must survive. diff --git a/skills/skills/langbot-env-setup/references/oauth-browser-profile.md b/skills/skills/langbot-env-setup/references/oauth-browser-profile.md new file mode 100644 index 000000000..b3ac8a91e --- /dev/null +++ b/skills/skills/langbot-env-setup/references/oauth-browser-profile.md @@ -0,0 +1,62 @@ +# OAuth Browser Profile + +Use this reference when LangBot or LangBot Space needs GitHub OAuth login and the agent must reuse the authenticated browser state later. + +Read `skills/.env` first for `LANGBOT_BACKEND_URL`, `LANGBOT_FRONTEND_URL`, `LANGBOT_BROWSER_PROFILE`, `LANGBOT_CHROMIUM_EXECUTABLE`, and proxy defaults. + +## Rules + +- Never handle the user's GitHub password, passkey, recovery code, or 2FA secret. +- Open a visible browser and let the user complete credential steps. +- Reuse a fixed browser profile path. +- Do not print token values. It is acceptable to report localStorage key names. + +## Manual Visible Login Flow + +1. Verify LangBot backend is reachable with `service-startup.md`. +2. Launch a visible Chromium window with the persistent profile: + +```bash +setsid "$LANGBOT_CHROMIUM_EXECUTABLE" \ + --no-sandbox \ + --ozone-platform=x11 \ + --user-data-dir="$LANGBOT_BROWSER_PROFILE" \ + --proxy-server="$LANGBOT_PROXY_SOCKS" \ + --proxy-bypass-list="$LANGBOT_NO_PROXY" \ + "$LANGBOT_BACKEND_URL/login" \ + >/tmp/langbot-visible-chrome.log 2>&1 < /dev/null & +``` + +3. The user completes: + +```text +Login with Space -> Login with GitHub -> GitHub credentials / 2FA -> authorize +``` + +4. The agent can then reuse the same profile for automated checks. + +## Expected Successful State + +After login, LangBot should redirect away from `/login`, for example to a `/home/...` URL on the selected origin. + +Expected visible signals: + +```text +LangBot +Dashboard +Home +Bots +Pipelines +Knowledge +Extensions +``` + +Expected localStorage key names: + +```text +token +userEmail +langbot_language +``` + +If the user logged in on one origin but `LANGBOT_FRONTEND_URL` still shows `/login`, copy only the auth state needed between origins. Do not print token values. diff --git a/skills/skills/langbot-env-setup/references/playwright-mcp.md b/skills/skills/langbot-env-setup/references/playwright-mcp.md new file mode 100644 index 000000000..3e460e279 --- /dev/null +++ b/skills/skills/langbot-env-setup/references/playwright-mcp.md @@ -0,0 +1,30 @@ +# Playwright MCP Browser Path + +Use this path when the agent needs browser automation but no Computer Use browser-control path is available. + +## Known Paths + +- Persistent browser profile: `LANGBOT_BROWSER_PROFILE` from `skills/.env.local` +- Chromium executable: `LANGBOT_CHROMIUM_EXECUTABLE` from `skills/.env.local` +- Codex MCP config: `$CODEX_HOME/config.toml` or the config path used by the active agent. + +## MCP Config + +Keep the profile path fixed so the agent can reuse authenticated state. + +```toml +[mcp_servers.playwright] +command = "npx" +args = ["-y", "@playwright/mcp@latest", "--no-sandbox", "--executable-path", "", "--proxy-server", "", "--proxy-bypass", "localhost,127.0.0.1", "--user-data-dir", ""] +``` + +After changing MCP config, restart Codex so the MCP server is relaunched with the new args. + +## Visible Login + +For OAuth login, Playwright MCP's headless browser is not enough. Launch a visible browser with the same profile and let the user complete login. Use `oauth-browser-profile.md`. + +## Common Failures + +- MCP still uses old args after editing config: restart Codex or kill old `playwright-mcp` processes and restart the session. +- Browser is headless during OAuth: use the visible login command from `oauth-browser-profile.md`. diff --git a/skills/skills/langbot-env-setup/references/proxy.md b/skills/skills/langbot-env-setup/references/proxy.md new file mode 100644 index 000000000..14b276464 --- /dev/null +++ b/skills/skills/langbot-env-setup/references/proxy.md @@ -0,0 +1,30 @@ +# Proxy Setup + +Use this reference when GitHub OAuth, package installation, model provider tests, or external API calls time out. + +Read defaults from `skills/.env` first. + +## Standard Local Proxy + +```bash +export HTTP_PROXY="$LANGBOT_PROXY_HTTP" +export HTTPS_PROXY="$LANGBOT_PROXY_HTTP" +export ALL_PROXY="$LANGBOT_PROXY_SOCKS" +export http_proxy="$LANGBOT_PROXY_HTTP" +export https_proxy="$LANGBOT_PROXY_HTTP" +export all_proxy="$LANGBOT_PROXY_SOCKS" +export NO_PROXY="$LANGBOT_NO_PROXY" +export no_proxy="$LANGBOT_NO_PROXY" +``` + +## Rule + +Keep uppercase and lowercase proxy variables consistent. Different libraries read different names. + +## Checks + +```bash +env | rg -i '^(http|https|all|no)_?proxy=' +curl -I --max-time 8 --proxy "$LANGBOT_PROXY_SOCKS" https://github.com +curl -I --max-time 3 "$LANGBOT_BACKEND_URL" +``` diff --git a/skills/skills/langbot-env-setup/references/service-startup.md b/skills/skills/langbot-env-setup/references/service-startup.md new file mode 100644 index 000000000..4f7b3ec27 --- /dev/null +++ b/skills/skills/langbot-env-setup/references/service-startup.md @@ -0,0 +1,73 @@ +# Service Startup + +Use this reference for LangBot backend/frontend readiness checks regardless of OS or browser-control method. Read `skills/.env` first and override those defaults with user-provided values or detected running services. + +## Variables + +- `LANGBOT_REPO` +- `LANGBOT_WEB_REPO` +- `LANGBOT_BACKEND_URL` +- `LANGBOT_FRONTEND_URL` +- `LANGBOT_DEV_FRONTEND_URL` + +## Backend + +Start LangBot from the backend repo: + +```bash +cd "$LANGBOT_REPO" +uv run main.py +``` + +Healthy startup includes: + +```text +Running on http://0.0.0.0: +Connected to plugin runtime. +Plugin langbot/local-agent initialized +``` + +Quick check: + +```bash +curl -I --max-time 3 "$LANGBOT_BACKEND_URL/login" +``` + +If `bin/lbs env doctor` reports that `LANGBOT_BACKEND_URL` has no TCP listener, +the backend is not running at the configured host and port. A reachable +standalone frontend on `LANGBOT_FRONTEND_URL` does not prove backend readiness. + +Prefer a visible terminal session while debugging backend startup. Detached +background startup methods can hide early process exits in local agent runs; if +you use one, immediately verify both the process and the listener: + +```bash +ps -eo pid,cmd | rg 'main.py|uv run main|langbot' +ss -ltnp | rg ':5300' +curl -I --max-time 3 "$LANGBOT_BACKEND_URL/login" +``` + +## Frontend + +Start the new frontend from the web repo: + +```bash +cd "$LANGBOT_WEB_REPO" +npm run dev +``` + +Healthy startup includes: + +```text +Local: +``` + +Quick check: + +```bash +curl -I --max-time 3 "$LANGBOT_FRONTEND_URL" +``` + +## Completion Signal + +Environment setup is not complete until the required frontend/backend URLs are reachable and the chosen browser-control path can open the WebUI. diff --git a/skills/skills/langbot-env-setup/references/wsl-notes.md b/skills/skills/langbot-env-setup/references/wsl-notes.md new file mode 100644 index 000000000..6b12b020c --- /dev/null +++ b/skills/skills/langbot-env-setup/references/wsl-notes.md @@ -0,0 +1,36 @@ +# WSL Notes + +Use this reference only for WSL-specific details. Do not put generic LangBot startup or browser-login steps here. + +## Network + +GitHub login and model provider calls may require proxy access from WSL. + +Working proxy form: + +```bash +socks5://127.0.0.1:7890 +``` + +Bypass local LangBot: + +```bash +localhost,127.0.0.1 +``` + +Quick checks: + +```bash +curl -I --max-time 8 --proxy socks5h://127.0.0.1:7890 https://github.com +curl -I --max-time 3 "$LANGBOT_BACKEND_URL" +``` + +## Visible Browser + +If OAuth requires a visible browser, WSL must have a usable display path. If a visible Chromium launch fails, check the local WSL GUI/X11 setup before changing LangBot config. + +## Common Failures + +- `ERR_NETWORK_CHANGED` or GitHub timeout: browser is not using the SOCKS proxy. +- LangBot connection refused: backend is not running or not reachable from WSL. +- User cannot type credentials: browser is headless or not visible to the user. diff --git a/skills/skills/langbot-mcp-ops/SKILL.md b/skills/skills/langbot-mcp-ops/SKILL.md new file mode 100644 index 000000000..94f25a3a4 --- /dev/null +++ b/skills/skills/langbot-mcp-ops/SKILL.md @@ -0,0 +1,99 @@ +--- +name: langbot-mcp-ops +description: Operate a LangBot instance through its built-in MCP (Model Context Protocol) server. Use when an AI agent needs to manage LangBot — list/create/update/delete bots, pipelines, models, knowledge bases, MCP servers, and skills — over MCP instead of raw HTTP. Covers the /mcp endpoint, API-key auth (web-UI lbk_ keys and the config.yaml global key), the tool surface, and client configuration. Triggers on "langbot mcp", "manage langbot via mcp", "langbot /mcp", "langbot mcp server". +--- + +# LangBot MCP Operations + +LangBot exposes an **MCP server** so AI agents can manage an instance +programmatically. It mirrors a curated subset of the HTTP service API. + +## Endpoint + +``` +http://:5300/mcp +``` + +Transport: **streamable HTTP** (stateless, JSON responses). Same host/port as +the web UI and HTTP API. + +## Authentication + +Reuses the same API keys as the HTTP API. Send either header: + +``` +X-API-Key: +# or +Authorization: Bearer +``` + +Two kinds of key are accepted: + +1. **Web-UI key** — created in the web UI (sidebar → API Keys), prefixed `lbk_`, + stored in the database. +2. **Global API key** — set in `data/config.yaml` under `api.global_api_key`. + Requires no login session and no DB record; does not need the `lbk_` prefix. + Leave empty to disable. See the `langbot-deploy` skill for config details. + +Requests without a valid key get `401 Unauthorized`. + +## Client configuration + +```json +{ + "mcpServers": { + "langbot": { + "url": "http://:5300/mcp", + "headers": { "X-API-Key": "" } + } + } +} +``` + +## Tool surface + +The tools wrap the LangBot service layer. Current tools (v1): + +| Tool | Purpose | +| --- | --- | +| `get_system_info` | Version, edition, instance id | +| `list_bots` / `get_bot` / `create_bot` / `update_bot` / `delete_bot` | Manage messaging-platform bots (secrets redacted on read) | +| `list_pipelines` / `get_pipeline` / `create_pipeline` / `update_pipeline` / `delete_pipeline` | Manage pipelines | +| `list_llm_models` / `get_llm_model` / `list_embedding_models` / `list_model_providers` | Inspect models & providers | +| `list_knowledge_bases` / `get_knowledge_base` / `retrieve_knowledge_base` | RAG knowledge bases (incl. semantic search) | +| `list_mcp_servers` | External MCP servers LangBot connects to (as a client) | +| `list_skills` / `get_skill` | Installed skills | + +Mutating tools (`create_*`, `update_*`) take a JSON object matching the same +shape as the corresponding HTTP API request body. Discover resources with the +`list_*` / `get_*` tools before mutating; identifiers are UUIDs. + +## How to use + +1. Get an API key (web UI key, or set `api.global_api_key` in config.yaml). +2. Point your MCP client at `http://:5300/mcp` with the key header. +3. Call `get_system_info` to confirm connectivity. +4. Use `list_*` tools to discover, then `get_*` / `create_*` / `update_*` / + `delete_*` as needed. + +## Implementation & maintenance (for LangBot developers) + +- Server: `src/langbot/pkg/api/mcp/server.py` (FastMCP). Tools call the service + layer directly, so the MCP surface stays aligned with the API. +- Mount: `src/langbot/pkg/api/mcp/mount.py` — an ASGI dispatcher fronting Quart, + authenticating `/mcp` requests, running the streamable-HTTP session manager. +- Smoke test: `tests/manual/mcp_smoke.py`. + +> When you add, remove, or change an HTTP API endpoint that should be +> agent-accessible, update the corresponding MCP tool **and** this skill. The +> MCP tool surface and the API must stay aligned (see `AGENTS.md`). + +## Pitfalls + +- `/mcp` is the **server** LangBot exposes. The `/api/v1/mcp` routes are the + **client** side (managing external MCP servers LangBot connects to). Don't + confuse them. +- A `401` means the key is wrong, missing, or (for the global key) + `api.global_api_key` is empty in config.yaml. +- The global key is plaintext in config.yaml — only enable it on trusted/internal + deployments and serve over HTTPS. diff --git a/skills/skills/langbot-plugin-dev/SKILL.md b/skills/skills/langbot-plugin-dev/SKILL.md new file mode 100644 index 000000000..247e973c2 --- /dev/null +++ b/skills/skills/langbot-plugin-dev/SKILL.md @@ -0,0 +1,452 @@ +--- +name: langbot-plugin-dev +description: Develop, debug, and test LangBot plugins. Use when creating new LangBot plugins, fixing plugin bugs, setting up a LangBot test environment, or testing plugins via WebSocket. Covers plugin component architecture (EventListener, Command, Tool), the plugin SDK API (invoke_llm, get_llm_models, send_message, plugin storage), common pitfalls, and automated WebSocket-based testing. Triggers on "langbot plugin", "lbp", "GroupChatSummary", "plugin debug", "langbot test". +--- + +# LangBot Plugin Development & Debugging + +## Controlling a running instance via MCP + +Beyond writing code, you can **drive a live LangBot instance over MCP** — no raw +HTTP needed. Two MCP servers exist (both reuse existing API keys; see `AGENTS.md`): + +- **LangBot instance** — `http://:5300/mcp` (auth: web-UI `lbk_` key or the + `api.global_api_key` from `config.yaml`). Manage bots, pipelines, models, + knowledge bases, and skills. See the **`langbot-mcp-ops`** skill. +- **LangBot Space marketplace** — `https://space.langbot.app/mcp` (auth: Personal + Access Token). Search plugins / MCP servers / skills. See the + **`langbot-space-ops`** skill. + +> Any change to an agent-accessible HTTP API endpoint must keep the matching MCP +> tool and these skills in sync. + +## Plugin Architecture + +A LangBot plugin consists of: + +``` +MyPlugin/ +├── manifest.yaml # Plugin metadata, config schema +├── main.py # BasePlugin subclass (entry point, shared state) +├── components/ +│ ├── event_listener/ # Hook pipeline events +│ │ ├── collector.yaml +│ │ └── collector.py +│ ├── commands/ # !command handlers +│ │ ├── mycommand.yaml +│ │ └── mycommand.py +│ └── tools/ # LLM function-call tools +│ ├── mytool.yaml +│ └── mytool.py +``` + +Each component has a `.yaml` (metadata) and `.py` (implementation). + +## Critical SDK Pitfalls + +### 1. MessageChain is a RootModel — iterate directly + +```python +# ❌ WRONG — MessageChain has no .components attribute +for component in event.message_chain.components: + +# ✅ CORRECT — MessageChain is a Pydantic RootModel, iterate directly +for component in event.message_chain: +``` + +### 2. Message.content must be `list[ContentElement]` or `str`, not a single ContentElement + +```python +from langbot_plugin.api.entities.builtin.provider import message as provider_message + +# ❌ WRONG — single ContentElement +Message(role="user", content=ContentElement.from_text("hello")) + +# ✅ CORRECT — list of ContentElement +Message(role="user", content=[ContentElement.from_text("hello")]) + +# ✅ ALSO CORRECT — plain string +Message(role="user", content="hello") +``` + +### 3. invoke_llm does NOT accept timeout + +```python +# ❌ WRONG +await self.invoke_llm(llm_model_uuid=uuid, messages=msgs, timeout=60) + +# ✅ CORRECT +await self.invoke_llm(llm_model_uuid=uuid, messages=msgs) +``` + +### 4. invoke_llm response.content can be str OR list + +```python +response = await self.invoke_llm(...) +if response.content: + if isinstance(response.content, str): + return response.content + elif isinstance(response.content, list): + parts = [e.text for e in response.content if hasattr(e, "text") and e.text] + return "\n".join(parts) +``` + +### 5. get_llm_models() returns UUIDs + +```python +# Returns list[str] of model UUIDs +models = await self.get_llm_models() +model_uuid = models[0] # First available model UUID +``` + +**Known bug (v4.9.3):** The host handler may return `list[dict]` instead of `list[str]`. If you hit `TypeError: unhashable type: 'dict'` in `invoke_llm`, the fix is in `LangBot/src/langbot/pkg/plugin/handler.py` — change `'llm_models': llm_models` to `'llm_models': [m['uuid'] for m in llm_models]`. + +### 6. invoke_llm parameter is `llm_model_uuid`, NOT `model_uuid` + +```python +# ❌ WRONG — will throw "got an unexpected keyword argument" +await self.invoke_llm(messages=msgs, model_uuid=uuid) + +# ✅ CORRECT +await self.invoke_llm(messages=msgs, llm_model_uuid=uuid) +``` + +### 7. prevent_default() alone does NOT block LLM response + +To fully prevent the default LLM pipeline from responding when your EventListener handles the message, you must call **both**: + +```python +event_context.prevent_default() # Block default behavior +event_context.prevent_postorder() # Block subsequent plugins/pipeline +``` + +Using only `prevent_default()` still allows the LLM to generate a response. + +### 8. get_plugin_storage / set_plugin_storage may throw KeyError: 'owner' + +This is a version mismatch between the SDK and host. Wrap storage calls in try/except: + +```python +try: + data = await self.get_plugin_storage("my_key") +except Exception: + data = None # Fallback gracefully +``` + +### 9. Component YAML must have full structure, not just name/description + +```yaml +# ❌ WRONG — will silently fail to register the component +name: translator +description: + en_US: 'Does stuff' + +# ✅ CORRECT — full component YAML +apiVersion: v1 +kind: EventListener +metadata: + name: translator + label: + en_US: Translator +spec: +execution: + python: + path: translator.py + attr: Translator +``` + +### 10. BasePlugin import path + +```python +# ❌ WRONG +from langbot_plugin.api.definition.base_plugin import BasePlugin + +# ✅ CORRECT +from langbot_plugin.api.definition.plugin import BasePlugin +``` + +## Pipeline Events + +Events the EventListener can hook (from most general to most specific): + +| Event | When | +|---|---| +| `GroupMessageReceived` | **Any** group message arrives (before trigger rules) | +| `PersonMessageReceived` | **Any** private message arrives | +| `GroupNormalMessageReceived` | Group message passes trigger rules, going to LLM | +| `PersonNormalMessageReceived` | Private message going to LLM | +| `GroupCommandSent` | Group message matched as command | +| `PersonCommandSent` | Private message matched as command | +| `NormalMessageResponded` | LLM generated a response | +| `PromptPreProcessing` | About to build LLM context | + +**Key insight:** `*MessageReceived` fires for ALL messages regardless of trigger rules. `*NormalMessageReceived` only fires for messages that match the pipeline's trigger rules (e.g., @bot, prefix, random%). Use `*MessageReceived` for message collection/logging. + +## EventContext API + +```python +@self.handler(events.GroupMessageReceived) +async def on_msg(event_context: context.EventContext): + event = event_context.event + event.launcher_id # Group ID + event.sender_id # Sender ID + event.message_chain # MessageChain (iterate directly) + + # Reply to the current conversation + await event_context.reply(MessageChain([Plain(text="hello")])) + + # Block default pipeline behavior + event_context.prevent_default() + + # Block subsequent plugins + event_context.prevent_postorder() +``` + +## Setting Up a Test Environment + +### Deploy via Docker (GitOps + Portainer) + +See `references/test-env-setup.md` for full deployment steps. + +Quick summary: +1. Create `docker-compose.yaml` in `server-deploy` repo +2. Deploy via Portainer git repository method +3. Set up admin account via `/api/v1/user/init` POST +4. Configure LLM provider and model via API +5. Copy plugin to `data/plugins/` directory + +### WebSocket Testing + +LangBot's WebUI chat uses WebSocket. Connect to test message flow: + +``` +ws://:/api/v1/pipelines//ws/connect?session_type=group +``` + +- `session_type=group` for group chat simulation +- `session_type=person` for private chat (always triggers pipeline) + +**Requires Origin header** to pass CORS: +```javascript +const ws = new WebSocket(url, { + headers: { Origin: 'https://your-langbot-domain' } +}); +``` + +Send messages: +```json +{"type": "message", "message": [{"type": "Plain", "text": "hello"}]} +``` + +Receive: +- `{"type": "connected", ...}` — connection established +- `{"type": "user_message", "data": {...}}` — echo of sent message +- `{"type": "response", "data": {"content": "...", "is_final": true/false}}` — bot reply (streamed) + +### Group Trigger Rules + +Group messages only enter the pipeline if trigger rules are met: + +```json +{ + "group-respond-rules": { + "at": true, // Respond when @bot + "prefix": ["ai"], // Respond to messages starting with "ai" + "random": 0.0, // Probability of responding to any message (0.0-1.0) + "regexp": [] // Regex patterns + } +} +``` + +For testing, set `random: 1.0` via PUT `/api/v1/pipelines/` to respond to all messages. + +**Important:** EventListener hooks like `GroupMessageReceived` fire regardless of trigger rules. Only the LLM processing (`GroupNormalMessageReceived` and beyond) requires trigger rules. + +### Plugin Hot-Reload + +There is **no hot-reload**. After changing plugin files: + +```bash +docker restart +# Wait ~5 seconds for plugin to re-mount +``` + +The main LangBot container does NOT need restart for plugin changes — only the runtime container. + +## API Quick Reference + +### Admin Setup + +```bash +# Initialize admin account (first time only) +curl -X POST $BASE/api/v1/user/init \ + -H "Content-Type: application/json" \ + -d '{"user":"admin@test.com","password":"test123"}' + +# Login +curl -X POST $BASE/api/v1/user/auth \ + -H "Content-Type: application/json" \ + -d '{"user":"admin@test.com","password":"test123"}' +# Returns: {"data":{"token":"eyJ..."}} +``` + +### Provider & Model Setup + +```bash +# Create provider +curl -X POST $BASE/api/v1/provider/providers \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"name":"MyProvider","requester":"new-api-chat-completions","base_url":"https://api.example.com/v1","api_keys":["sk-xxx"]}' + +# Create LLM model +curl -X POST $BASE/api/v1/provider/models/llm \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"name":"gpt-4o-mini","provider_uuid":"","abilities":["chat","tool-use"]}' + +# List models +curl $BASE/api/v1/provider/models/llm -H "Authorization: Bearer $TOKEN" +``` + +### Pipeline Config + +```bash +# Get pipeline +curl $BASE/api/v1/pipelines -H "Authorization: Bearer $TOKEN" + +# Update pipeline (e.g., set model, modify trigger rules) +curl -X PUT $BASE/api/v1/pipelines/ \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '' +``` + +## Plugin Config Types + +Supported `type` values in `manifest.yaml` `spec.config`: + +| Type | Description | Value | +|---|---|---| +| `string` | Text input | string | +| `int` / `integer` | Number input | int | +| `float` | Decimal input | float | +| `bool` / `boolean` | Toggle | bool | +| `select` | Dropdown (needs `options`) | string | +| `prompt-editor` | Multi-line prompt editor | string | +| `llm-model-selector` | LLM model picker UI | UUID string | +| `bot-selector` | Bot picker UI | UUID string | + +Example — let users choose which model the plugin uses: + +```yaml +spec: + config: + - name: model + type: llm-model-selector + label: + en_US: 'LLM Model' + zh_Hans: 'LLM 模型' + description: + en_US: 'Select the LLM model. Falls back to first available if not set.' + zh_Hans: '选择 LLM 模型。未设置时使用第一个可用模型。' + required: false +``` + +Read config in plugin code: + +```python +model_uuid = self.get_config().get("model") +``` + +## Container Restart Timing + +After plugin file changes, **only the runtime container needs restart**: + +```bash +docker restart langbot-test-runtime +# Wait ~15 seconds before testing +``` + +**When to restart both (runtime first, then host):** +- Added/removed Command or Tool components (host caches component lists) +- Changed `manifest.yaml` structure + +```bash +docker restart langbot-test-runtime +sleep 8 +docker restart langbot-test +sleep 8 +``` + +**⚠️ Do NOT restart both simultaneously** — the host may connect before plugins are mounted, causing 502 errors or missing plugin registrations. + +## Debugging Checklist + +When a plugin doesn't work: + +1. **Check runtime logs**: `docker logs ` — look for mount/init errors +2. **Check host logs**: `docker logs ` — look for pipeline processing errors +3. **Verify plugin loaded**: `GET /api/v1/plugins` — should list your plugin +4. **Test person mode first**: `session_type=person` always triggers pipeline, isolating trigger rule issues +5. **Check trigger rules**: Group mode requires @bot, prefix match, or random% to enter pipeline +6. **Verify model configured**: Pipeline's `config.ai.local-agent.model.primary` must point to a valid model UUID with working API keys + +## Publishing Plugins + +After testing, publish via `lbp publish`: + +```bash +cd /path/to/MyPlugin +lbp publish +``` + +This builds `.lbpkg` and uploads to Space marketplace as a draft. Then go to https://space.langbot.app/market to upload screenshots and submit for review. + +**Prerequisite:** Must be logged in via `lbp login --token lbpat_xxx` (PAT from Space profile page). + +## Reference: EventListener-Only Plugin Pattern + +For plugins that react to messages without commands or tools (e.g., auto-summarize URLs, collect messages, translate): + +``` +MyPlugin/ +├── manifest.yaml # Only EventListener in spec.components +├── main.py # BasePlugin with shared logic (fetch, LLM calls) +├── components/ +│ └── event_listener/ +│ ├── detector.yaml +│ └── detector.py +└── requirements.txt +``` + +**manifest.yaml** — only declare EventListener: +```yaml +spec: + components: + EventListener: + fromDirs: + - path: components/event_listener/ +``` + +**detector.py** — hook `*MessageReceived`, extract text, process, reply: +```python +@self.handler(events.PersonMessageReceived) +async def on_msg(event_context: context.EventContext): + event = event_context.event + text_parts = [] + for component in event.message_chain: + if isinstance(component, platform_message.Plain): + text_parts.append(component.text) + text = "".join(text_parts).strip() + + if should_handle(text): + event_context.prevent_default() + event_context.prevent_postorder() + result = await self.plugin.process(text) + await event_context.reply(platform_message.MessageChain([ + platform_message.Plain(text=result) + ])) +``` + +**Key:** Access shared plugin logic via `self.plugin` (the BasePlugin instance). diff --git a/skills/skills/langbot-plugin-dev/references/test-env-setup.md b/skills/skills/langbot-plugin-dev/references/test-env-setup.md new file mode 100644 index 000000000..c4fa20394 --- /dev/null +++ b/skills/skills/langbot-plugin-dev/references/test-env-setup.md @@ -0,0 +1,116 @@ +# Test Environment Setup + +## Docker Compose (GitOps) + +Create in `server-deploy` repo under `servers//langbot-test/docker-compose.yaml`: + +```yaml +version: "3" +services: + langbot_plugin_runtime: + image: rockchin/langbot:latest + container_name: langbot-test-runtime + volumes: + - /opt/docker-data/langbot-test/data/plugins:/app/data/plugins + ports: + - "5411:5401" + restart: on-failure + environment: + - TZ=Asia/Shanghai + command: ["uv", "run", "--no-sync", "-m", "langbot_plugin.cli.__init__", "rt"] + networks: + - langbot_test_network + + langbot: + image: rockchin/langbot:latest + container_name: langbot-test + volumes: + - /opt/docker-data/langbot-test/data:/app/data + ports: + - "5310:5300" + restart: on-failure + depends_on: + - langbot_plugin_runtime + environment: + - TZ=Asia/Shanghai + networks: + - langbot_test_network + +networks: + langbot_test_network: + driver: bridge +``` + +## Post-Deploy Configuration + +After first start, LangBot auto-generates `data/config.yaml`. You need to update `plugin.runtime_ws_url` to match the runtime container name: + +```bash +# On the host, edit config +sed -i 's|ws://localhost:5400/control/ws|ws://langbot-test-runtime:5400/control/ws|' \ + /opt/docker-data/langbot-test/data/config.yaml +docker restart langbot-test +``` + +## Installing a Plugin + +Copy plugin directory to `data/plugins/` on the host: + +```bash +scp -r MyPlugin/ user@host:/opt/docker-data/langbot-test/data/plugins/MyPlugin/ +docker restart langbot-test-runtime # Runtime picks up new plugins on restart +``` + +## Caddy Reverse Proxy (Optional) + +If testing externally, add to Caddyfile on the same host: + +``` +langbot-test.example.com { + reverse_proxy langbot-test:5300 +} +``` + +Then reload: `docker exec caddy caddy reload --config /etc/caddy/Caddyfile` + +The WebSocket endpoint works through Caddy without special config. + +## WebSocket Test Script (Node.js) + +```javascript +const WebSocket = require('ws'); + +const PIPELINE_UUID = ''; +const BASE = 'wss://langbot-test.example.com'; +const URL = `${BASE}/api/v1/pipelines/${PIPELINE_UUID}/ws/connect?session_type=group`; + +const ws = new WebSocket(URL, { + headers: { Origin: BASE } +}); + +const send = (text) => { + ws.send(JSON.stringify({ + type: 'message', + message: [{ type: 'Plain', text }] + })); + console.log('[SENT]', text); +}; + +ws.on('message', (data) => { + const msg = JSON.parse(data.toString()); + if (msg.type === 'connected') { + console.log('Connected!'); + // Send test messages + send('Message 1'); + setTimeout(() => send('Message 2'), 500); + setTimeout(() => send('!summary'), 2000); + } else if (msg.type === 'response' && msg.data?.is_final) { + console.log('[BOT]', msg.data.content); + } +}); + +ws.on('error', (e) => console.error('Error:', e.message)); +setTimeout(() => { ws.close(); process.exit(); }, 60000); +``` + +Requires: `npm install ws` diff --git a/skills/skills/langbot-skills-maintenance/SKILL.md b/skills/skills/langbot-skills-maintenance/SKILL.md new file mode 100644 index 000000000..4ba8fe692 --- /dev/null +++ b/skills/skills/langbot-skills-maintenance/SKILL.md @@ -0,0 +1,40 @@ +--- +name: langbot-skills-maintenance +description: Maintain the langbot-skills repository with low duplication. Use when adding, editing, or auditing LangBot skills, references, cases, troubleshooting entries, indexes, or periodic entropy-control checks for this skills repository. +--- + +# LangBot Skills Maintenance + +Use this skill before changing reusable assets in this repository. + +## Workflow + +1. Read `AGENTS.md`, `skills/.env`, and the relevant existing skill files. +2. Classify the change: + - `SKILL.md` for routing and concise operating rules. + - `references/*.md` for canonical detailed workflows. + - `cases/*.yaml` for executable test-plan skeletons. + - `suites/*.yaml` for reusable groups of case ids. + - `fixtures/fixtures.json` for deterministic fixture readiness metadata. + - `reports/evidence//automation-result.json` as automation output and `reports/evidence//result.json` as final judgment output; neither is a catalog asset to commit. + - `troubleshooting/*.yaml` for one reusable failure mode. +3. Search existing assets before adding new files: + - `rg "" skills` + - `bin/lbs case list` + - `bin/lbs suite list` + - `bin/lbs fixture list` +4. Put detail in one canonical place and link to it from cases or routing bullets. +5. Run the checks in `AGENTS.md` after edits. + +## Entropy Rules + +- Prefer extending an existing reference or troubleshooting entry when the root cause is the same. +- Keep cases short: setup, action, evidence, pass/fail checks. Do not paste long prompts or debug transcripts when a reference exists. +- Put machine-checkable inputs in `env`, `automation_env`, or fixtures; put operator-confirmed assumptions in `preconditions` so `test plan` can surface `manual_check`. +- Keep suites short: title, intent, tags, and ordered case ids. Do not duplicate case steps inside a suite. +- Keep fixture manifests factual: id, title, path, kind, and related case ids. Do not encode environment-specific absolute paths. +- Keep troubleshooting entries narrow: symptoms, patterns, likely causes, fixes, related assets. +- Do not hardcode local ports, browser profile paths, secrets, tokens, or provider keys. +- Use `bin/lbs index --check` to verify the committed index is current without writing it; run `bin/lbs index` when the index needs regeneration. + +For periodic repository audits, read `references/curation-workflow.md`. diff --git a/skills/skills/langbot-skills-maintenance/references/curation-workflow.md b/skills/skills/langbot-skills-maintenance/references/curation-workflow.md new file mode 100644 index 000000000..c40f25d1d --- /dev/null +++ b/skills/skills/langbot-skills-maintenance/references/curation-workflow.md @@ -0,0 +1,70 @@ +# Curation Workflow + +Use this checklist when the repository starts accumulating repeated cases, copied steps, or overlapping troubleshooting entries. + +## Audit Pass + +1. Inspect the current surface: + - `bin/lbs case list` + - `bin/lbs case list --json --priority p0 --automation` + - `bin/lbs case list --ready` + - `bin/lbs case list --machine-ready` + - `bin/lbs suite list` + - `bin/lbs fixture list` + - `rg "sandbox|provider|pipeline|plugin|knowledge|mcp" skills` + - `rg "If .* fails|Known Pitfalls|Debug Chat|/api/v1" skills` +2. Group nearby assets by intent, not by file path: + - user-facing scenario + - backend or provider dependency + - failure signature + - pass/fail evidence +3. Pick one canonical owner: + - stable procedures belong in `references/` + - deterministic files and packages belong in `fixtures/` plus `fixtures/fixtures.json` + - repeated failure signatures belong in `troubleshooting/` + - runnable QA paths belong in `cases/` + - reusable groups of QA paths belong in `suites/` + - skill entry points belong in `SKILL.md` + +## Merge Or Split + +Merge when two files share the same trigger, root cause, and fix. Keep the stronger id and move missing patterns into it. + +Split when a file mixes unrelated failure modes or requires different fixes. Each troubleshooting id should map to one diagnosis path. + +Move repeated step lists out of cases and into a reference when more than one case would need the same prompt, UI path, or log interpretation. + +Add or update a suite when developers repeatedly run the same ordered group of cases. Do not copy case steps into suites; use `bin/lbs suite plan ` to expand the group. +Use `bin/lbs suite start ` and `bin/lbs suite report --evidence-dir ` when validating that a suite is operational end to end. + +Add or update `fixtures/fixtures.json` when a case depends on a deterministic file, plugin package, or local test server. The manifest should use repo-relative paths under the owning skill and should not contain machine-local absolute paths. + +When adding Debug Chat Playwright automation, reuse `scripts/e2e/lib/debug-chat.mjs` for navigation, prompt send, response leaf matching, and known failure classification. Keep case-specific prompts and expected sentinels in case YAML automation fields when possible. + +## Case Review + +For every changed case: + +1. Ensure `steps` describe what to execute, not every command in the underlying implementation. +2. Ensure `checks` contain observable UI, log, network, or filesystem evidence. +3. Ensure `diagnostics` are fallback investigation hints, not pass criteria. +4. Ensure `priority`, `risk`, `ci_eligible`, and `evidence_required` match the actual repeatability and evidence burden. +5. Put must-have env vars in `env` / `automation_env`; put one-of choices such as URL-or-name in `env_any` / `automation_env_any`. +6. Ensure linked `skills` and `troubleshooting` ids exist. +7. Run: + + ```bash + bin/lbs validate + bin/lbs index --check + bin/lbs index + bin/lbs test plan + ``` + +## Final Gate + +Before handing off: + +- `git diff --stat` should show a focused change set. +- `skills.index.json` should be regenerated only by `bin/lbs index`. +- No new asset should contain local credentials, OAuth tokens, API keys, or copied localStorage values. +- The final note should say which checks ran and which cases or troubleshooting ids changed. diff --git a/skills/skills/langbot-space-ops/SKILL.md b/skills/skills/langbot-space-ops/SKILL.md new file mode 100644 index 000000000..196205a70 --- /dev/null +++ b/skills/skills/langbot-space-ops/SKILL.md @@ -0,0 +1,79 @@ +--- +name: langbot-space-ops +description: Browse and search the LangBot Space marketplaces (plugins, MCP servers, skills) through the Space MCP server. Use when an AI agent needs to discover LangBot extensions on space.langbot.app over MCP. Covers the /mcp endpoint, Personal Access Token (PAT) auth, the tool surface, and client configuration. Triggers on "langbot space mcp", "search langbot plugins", "langbot marketplace mcp", "space.langbot.app mcp". +--- + +# LangBot Space MCP Operations + +LangBot Space (space.langbot.app) exposes an **MCP server** so user-facing AI +agents can browse and search the marketplaces (plugins, MCP servers, skills). + +## Endpoint + +``` +https://space.langbot.app/mcp +``` + +Transport: **streamable HTTP** (stateless, JSON responses). For a self-hosted +Space instance: `http://:8383/mcp`. + +## Authentication + +Reuses the existing **Personal Access Token (PAT)** — the same token the `lbp` +CLI uses. Create one in your Space account (Profile → Personal Access Tokens), +then send it as a Bearer token: + +``` +Authorization: Bearer lbpat_...uests without a valid PAT get `401 Unauthorized`. + +## Client configuration + +```json +{ + "mcpServers": { + "langbot-space": { + "url": "https://space.langbot.app/mcp", + "headers": { "Authorization": "Bearer " } + } + } +} +``` + +## Tool surface + +| Tool | Purpose | +| --- | --- | +| `list_plugins` / `search_plugins` / `get_plugin` | Plugin marketplace | +| `list_mcp_servers` / `search_mcp_servers` / `get_mcp_server` | MCP-server marketplace | +| `list_skills` / `search_skills` / `get_skill` | Skill marketplace | + +`list_*` and `search_*` are paged (`page`, `page_size`). `get_*` takes +`author` + `name`. The tool surface mirrors the REST endpoints under +`/api/v1/marketplace/*` and is read/browse only. + +## How to use + +1. Create a PAT in your Space account settings. +2. Point your MCP client at `https://space.langbot.app/mcp` with the Bearer PAT. +3. Use `search_plugins` / `search_mcp_servers` / `search_skills` to find items, + then `get_*` for details (e.g. to obtain author/name for installation in + LangBot itself). + +## Implementation & maintenance (for Space developers) + +- Server: `internal/controller/mcp/server.go` (official Go MCP SDK + `github.com/modelcontextprotocol/go-sdk`). Tools call the service layer + (`PluginService`, `MCPService`, `SkillService`) directly. +- Mount: `internal/controller/api.go` at `/mcp` and `/mcp/*any`. +- Auth: PAT via `AccountService.ValidatePersonalAccessToken`. +- Docs: `docs/MCP_SERVER.md`. + +> When you add, remove, or change a marketplace API endpoint that should be +> agent-accessible, update the corresponding MCP tool **and** this skill. The +> MCP tool surface and the API must stay aligned (see `AGENTS.md`). + +## Pitfalls + +- The PAT prefix is `lbpat_` (Space), distinct from LangBot's `lbk_` API keys. +- This server is read/browse only; it does not publish or modify marketplace + items. Use the web UI or REST API (with appropriate auth) for that. diff --git a/skills/skills/langbot-testing/SKILL.md b/skills/skills/langbot-testing/SKILL.md new file mode 100644 index 000000000..e9db1980f --- /dev/null +++ b/skills/skills/langbot-testing/SKILL.md @@ -0,0 +1,41 @@ +--- +name: langbot-testing +description: Test LangBot WebUI and core product flows with an automated browser and backend logs. Use when validating the configured LangBot frontend, pipeline Debug Chat, model provider setup and test buttons, bot and knowledge-base UI flows, or troubleshooting failed LangBot end-to-end tests. +--- + +# LangBot Testing + +Use this skill when an agent needs to verify LangBot behavior through the WebUI instead of only reading code. + +## Routing + +- **General WebUI testing**: read `references/web-ui-testing.md`. +- **Pipeline Debug Chat**: read `references/pipeline-debug-chat.md`. +- **Dify AgentRunner**: read `references/dify-agent-runner.md`. +- **Model provider setup or test button**: read `references/model-provider-testing.md`. +- **Plugin install/runtime/tool/page smoke**: read `references/plugin-e2e-smoke.md`. +- **Local Agent Runner**: read `references/local-agent-runner.md`. +- **Local Agent Runner path coverage**: read `references/local-agent-runner-coverage.md`. +- **Diff-aware AgentRunner QA after code changes**: read `references/agent-runner-qa-workflow.md`. +- **Agent Runner release gate**: read `references/agent-runner-release-gate.md`. +- **Sandbox-backed skill authoring**: read `references/sandbox-skill-authoring.md`. +- **LangRAG knowledge bases**: read `references/langrag-knowledge-base.md`. +- **MCP stdio tool testing**: read `references/mcp-stdio-testing.md`. +- **Drive a live instance over MCP (not raw HTTP)**: use the `langbot-mcp-ops` skill — the instance exposes an MCP server at `http://:5300/mcp` (reuses API keys). Useful for setting up bots/pipelines/models as test fixtures programmatically. +- **Known failures and fixes**: read `references/troubleshooting.md`. +- **Reusable test groups**: run `bin/lbs suite list` and `bin/lbs suite plan ` before manually assembling a case set. + +## Rules + +- Read `../.env` first and use `LANGBOT_FRONTEND_URL` and `LANGBOT_BACKEND_URL` instead of hardcoded ports. +- If a standalone frontend dev server is running, `LANGBOT_FRONTEND_URL` may point to `LANGBOT_DEV_FRONTEND_URL`; otherwise it may point to the backend WebUI. +- Confirm the backend and frontend are actually running before testing. +- Run `bin/lbs fixture check` before fixture-heavy MCP, RAG, multimodal, or plugin smoke tests. +- For runner externalization release checks, run `bin/lbs test run agent-runner-release-preflight` before the full `agent-runner-release-gate` suite so configuration blockers are separated from product failures. +- Read `Manual Readiness` in `bin/lbs test plan `; `manual_check` means the declared preconditions or setup still need operator confirmation for this run. +- Use an authenticated browser profile prepared by `langbot-env-setup`. +- Do not expose API keys, OAuth secrets, tokens, or localStorage token values in output. +- A WebUI test is not complete until the visible UI result is checked against backend logs or network behavior. +- For a suite, use `bin/lbs suite start ` to create the suite evidence root, per-case directories, and `suite-start.json`/`suite-start.md` handoff files; use `bin/lbs test result ` to write final per-case `result.json`, then run `bin/lbs suite report --evidence-dir `. +- Do not mark a case `pass` until `test result --evidence` covers every value in the case's `evidence_required`. +- For runner-specific Debug Chat cases, use the case-specific pipeline env declared by `automation_pipeline_url_env` / `automation_pipeline_name_env`; do not silently reuse a generic `LANGBOT_PIPELINE_URL`. diff --git a/skills/skills/langbot-testing/cases/acp-agent-runner-debug-chat.yaml b/skills/skills/langbot-testing/cases/acp-agent-runner-debug-chat.yaml new file mode 100644 index 000000000..ae10cea66 --- /dev/null +++ b/skills/skills/langbot-testing/cases/acp-agent-runner-debug-chat.yaml @@ -0,0 +1,79 @@ +id: acp-agent-runner-debug-chat +title: "ACP AgentRunner can answer through Debug Chat using real remote Claude" +mode: agent-browser +area: pipeline +type: regression +priority: p2 +risk: high +ci_eligible: false +tags: + - agent-runner + - acp + - claude + - external-runner + - pipeline +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL +env_any: + - LANGBOT_ACP_AGENT_RUNNER_PIPELINE_URL|LANGBOT_ACP_AGENT_RUNNER_PIPELINE_NAME +automation: scripts/e2e/pipeline-debug-chat.mjs +automation_env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_BROWSER_PROFILE + - LANGBOT_CHROMIUM_EXECUTABLE + - LANGBOT_ACP_AGENT_RUNNER_PIPELINE_URL + - LANGBOT_ACP_AGENT_RUNNER_PIPELINE_NAME + - LANGBOT_E2E_PROMPT + - LANGBOT_E2E_EXPECTED_TEXT + - LANGBOT_E2E_EXPECTED_RUNNER_ID + - LANGBOT_E2E_RESPONSE_TIMEOUT_MS +automation_pipeline_url_env: LANGBOT_ACP_AGENT_RUNNER_PIPELINE_URL +automation_pipeline_name_env: LANGBOT_ACP_AGENT_RUNNER_PIPELINE_NAME +automation_expected_runner_id: "plugin:langbot/acp-agent-runner/default" +automation_prompt: "Use the injected LangBot MCP server tool langbot_get_current_event once. If the MCP call succeeds, reply with exactly ACP_AGENT_RUNNER_E2E_OK." +automation_expected_text: "ACP_AGENT_RUNNER_E2E_OK" +automation_response_timeout_ms: "300000" +setup_automation: + - "node:scripts/e2e/ensure-acp-agent-runner-pipeline.mjs --write-env" +setup_provides_env: + - LANGBOT_ACP_AGENT_RUNNER_PIPELINE_URL + - LANGBOT_ACP_AGENT_RUNNER_PIPELINE_NAME +preconditions: + - "The remote machine has a working Claude Code login and can run npx -y @agentclientprotocol/claude-agent-acp." + - "LangBot can non-interactively SSH to the remote machine; the runner opens the MCP reverse tunnel automatically." +steps: + - "Open LANGBOT_FRONTEND_URL." + - "Open the ACP AgentRunner QA pipeline." + - "Confirm the pipeline AI runner is plugin:langbot/acp-agent-runner/default." + - "Open Debug Chat." + - "Ask the real remote Claude ACP agent to call langbot_get_current_event and return ACP_AGENT_RUNNER_E2E_OK exactly." +checks: + - "UI: Debug Chat shows the user prompt." + - "UI: Debug Chat shows a Bot response containing ACP_AGENT_RUNNER_E2E_OK." + - "Logs: Backend logs include Processing request from person_websocket and Streaming completed for this run." + - "Logs: No acp runner request error appears for this run." + - "Console: No unexpected frontend errors appear during Debug Chat." +evidence_required: + - ui + - console + - backend_log +diagnostics: + - "Use scripts/e2e/ensure-acp-agent-runner-pipeline.mjs --write-env to create/update the pipeline." + - "For remote Claude on 101, verify ssh yhh@101.34.71.12 can run without password prompts; no separate ssh -R process is required." +success_patterns: + - "ACP_AGENT_RUNNER_E2E_OK" + - "Processing request from person_websocket" + - "Streaming completed" +failure_patterns: + - "acp.command_not_found" + - "acp.process_exited" + - "Agent runner plugin:langbot/acp-agent-runner/default execution failed" +troubleshooting: + - backend-not-listening + - plugin-runtime-timeout + - proxy-env-mismatch diff --git a/skills/skills/langbot-testing/cases/agent-runner-async-db-readiness.yaml b/skills/skills/langbot-testing/cases/agent-runner-async-db-readiness.yaml new file mode 100644 index 000000000..3324a1c8e --- /dev/null +++ b/skills/skills/langbot-testing/cases/agent-runner-async-db-readiness.yaml @@ -0,0 +1,34 @@ +id: agent-runner-async-db-readiness +title: "AgentRunner async DB readiness probe" +mode: probe +area: release +type: smoke +priority: p0 +risk: high +ci_eligible: true +tags: + - agent-runner + - probe + - async-db + - aiosqlite +skills: + - langbot-testing +env: +automation: skills/langbot-testing/probes/agent-runner-async-db-readiness.mjs +steps: + - "Run `rtk bin/lbs test run agent-runner-async-db-readiness --dry-run` first; remove `--dry-run` after checking the planned evidence directory." + - "Automation checks whether a direct aiosqlite in-memory connection can create a table within the readiness timeout." +checks: + - "automation-result.json status is pass or env_issue." + - "pass means async SQLite tests are worth running." + - "env_issue means async SQLite-dependent Host pytest probes should be classified as environment-limited until fixed." +evidence_required: + - filesystem +diagnostics: + - "If this probe returns env_issue, run agent-runner-ledger-invariants for fast ledger coverage and skip async ledger pytest as a release blocker in this environment." +success_patterns: + - "AIOSQLITE_READY" +failure_patterns: + - "aiosqlite readiness timed out" +troubleshooting: + - aiosqlite-connect-hangs diff --git a/skills/skills/langbot-testing/cases/agent-runner-behavior-matrix.yaml b/skills/skills/langbot-testing/cases/agent-runner-behavior-matrix.yaml new file mode 100644 index 000000000..944bca68a --- /dev/null +++ b/skills/skills/langbot-testing/cases/agent-runner-behavior-matrix.yaml @@ -0,0 +1,34 @@ +id: agent-runner-behavior-matrix +title: "AgentRunner deterministic behavior matrix probe" +mode: probe +area: release +type: regression +priority: p0 +risk: high +ci_eligible: true +tags: + - agent-runner + - probe + - deterministic-runner + - protocol +skills: + - langbot-testing +env: +automation: skills/langbot-testing/probes/agent-runner-behavior-matrix.mjs +steps: + - "Run `rtk bin/lbs test run agent-runner-behavior-matrix --dry-run` first; remove `--dry-run` after checking the planned evidence directory." + - "Automation reads fixtures/agent-runner/qa-runner-behaviors.json and validates each result sequence through the Host AgentResultNormalizer." +checks: + - "automation-result.json status is pass." + - "probe-stdout.log contains QA_RUNNER_BEHAVIOR_MATRIX_OK." + - "The matrix covers ok, stream_ok, empty_output, malformed_result, and controlled_failure." +evidence_required: + - filesystem +diagnostics: + - "fail means the deterministic behavior fixture and Host result normalization disagree." +success_patterns: + - "QA_RUNNER_BEHAVIOR_MATRIX_OK" +failure_patterns: + - "AssertionError" + - "RunnerProtocolError" + - "behavior matrix exited" diff --git a/skills/skills/langbot-testing/cases/agent-runner-fixture-contract.yaml b/skills/skills/langbot-testing/cases/agent-runner-fixture-contract.yaml new file mode 100644 index 000000000..d8f0e04bc --- /dev/null +++ b/skills/skills/langbot-testing/cases/agent-runner-fixture-contract.yaml @@ -0,0 +1,35 @@ +id: agent-runner-fixture-contract +title: "QA AgentRunner fixture contract probe" +mode: probe +area: release +type: regression +priority: p0 +risk: high +ci_eligible: true +tags: + - agent-runner + - probe + - fixture + - deterministic-runner +skills: + - langbot-testing +env: +automation: skills/langbot-testing/probes/agent-runner-fixture-contract.mjs +steps: + - "Run `rtk bin/lbs test run agent-runner-fixture-contract --dry-run` first; remove `--dry-run` after checking the planned evidence directory." + - "Automation imports the QA AgentRunner fixture source and executes normal, streaming, and controlled-failure paths with SDK entities." +checks: + - "automation-result.json status is pass." + - "probe-stdout.log contains QA_AGENT_RUNNER_FIXTURE_CONTRACT_OK." + - "Normal input returns QA_AGENT_RUNNER_OK:." + - "Streaming input emits message.delta chunks and completes." + - "Failure input returns QA_AGENT_RUNNER_CONTROLLED_FAILURE." +evidence_required: + - filesystem +diagnostics: + - "This validates the deterministic fixture source contract. It does not prove the plugin package is installed in a live LangBot instance." +success_patterns: + - "QA_AGENT_RUNNER_FIXTURE_CONTRACT_OK" +failure_patterns: + - "AssertionError" + - "fixture contract exited" diff --git a/skills/skills/langbot-testing/cases/agent-runner-ledger-concurrency.yaml b/skills/skills/langbot-testing/cases/agent-runner-ledger-concurrency.yaml new file mode 100644 index 000000000..3b57c5435 --- /dev/null +++ b/skills/skills/langbot-testing/cases/agent-runner-ledger-concurrency.yaml @@ -0,0 +1,41 @@ +id: agent-runner-ledger-concurrency +title: "AgentRunner run ledger concurrency and auth pytest probe" +mode: probe +area: release +type: regression +priority: p0 +risk: high +ci_eligible: true +tags: + - agent-runner + - probe + - pytest + - run-ledger + - concurrency +skills: + - langbot-testing +env: +automation: skills/langbot-testing/probes/agent-runner-ledger-concurrency.mjs +preconditions: + - "This Host pytest probe can be slow in the current multi-repo dev environment; keep it in the release gate, but do not treat a timeout as a browser E2E failure without checking pytest logs." +steps: + - "Run `rtk bin/lbs test run agent-runner-ledger-concurrency --dry-run` first; remove `--dry-run` only after `agent-runner-async-db-readiness` is pass." + - "Automation resolves LANGBOT_REPO, defaulting to ../LangBot when the env var is unset." + - "Automation runs selected high-value tests from test_run_ledger_store.py and test_run_ledger_api_auth.py." +checks: + - "automation-result.json status is pass." + - "pytest exit status is 0 for selected run ledger claim, lease, status, token, ownership, and active-claim tests." + - "pytest-stdout.log and pytest-stderr.log are written under LBS_EVIDENCE_DIR." +evidence_required: + - filesystem +diagnostics: + - "env_issue means LANGBOT_REPO/default ../LangBot did not resolve, rtk/uv was unavailable, or the expected test files are missing." + - "fail means one of the selected LangBot run ledger pytest targets failed or timed out." +success_patterns: + - "pytest passed" +failure_patterns: + - "pytest exited with status" + - "pytest timed out" + - "Failed to start pytest command" +troubleshooting: + - aiosqlite-connect-hangs diff --git a/skills/skills/langbot-testing/cases/agent-runner-ledger-contention.yaml b/skills/skills/langbot-testing/cases/agent-runner-ledger-contention.yaml new file mode 100644 index 000000000..e90abd8b7 --- /dev/null +++ b/skills/skills/langbot-testing/cases/agent-runner-ledger-contention.yaml @@ -0,0 +1,34 @@ +id: agent-runner-ledger-contention +title: "AgentRunner ledger SQLite contention probe" +mode: probe +area: release +type: regression +priority: p1 +risk: high +ci_eligible: true +tags: + - agent-runner + - probe + - stress + - ledger + - concurrency +skills: + - langbot-testing +env: +automation: skills/langbot-testing/probes/agent-runner-ledger-contention.mjs +steps: + - "Run `rtk bin/lbs test run agent-runner-ledger-contention --dry-run` first; remove `--dry-run` after checking the planned evidence directory." + - "Automation creates 120 queued runs in a file-backed SQLite database and uses eight worker threads to claim runs under write contention." +checks: + - "automation-result.json status is pass." + - "probe-stdout.log contains LEDGER_CONTENTION_OK." + - "Every run is claimed once, reaches completed status, and has dispatch_attempts = 1." +evidence_required: + - filesystem +diagnostics: + - "This probe catches obvious exactly-once claim regressions under local SQLite contention; it does not replace async Host pytest or PostgreSQL concurrency checks." +success_patterns: + - "LEDGER_CONTENTION_OK" +failure_patterns: + - "AssertionError" + - "ledger contention exited" diff --git a/skills/skills/langbot-testing/cases/agent-runner-ledger-invariants.yaml b/skills/skills/langbot-testing/cases/agent-runner-ledger-invariants.yaml new file mode 100644 index 000000000..578cd8a31 --- /dev/null +++ b/skills/skills/langbot-testing/cases/agent-runner-ledger-invariants.yaml @@ -0,0 +1,35 @@ +id: agent-runner-ledger-invariants +title: "AgentRunner ledger schema and status invariants probe" +mode: probe +area: release +type: regression +priority: p0 +risk: high +ci_eligible: true +tags: + - agent-runner + - probe + - ledger + - invariant +skills: + - langbot-testing +env: +automation: skills/langbot-testing/probes/agent-runner-ledger-invariants.mjs +steps: + - "Run `rtk bin/lbs test run agent-runner-ledger-invariants --dry-run` first; remove `--dry-run` after checking the planned evidence directory." + - "Automation resolves LANGBOT_REPO, defaulting to ../LangBot, and imports the sibling SDK from LANGBOT_PLUGIN_SDK_REPO or ../langbot-plugin-sdk/src." + - "Automation checks run status sets, terminal status validation, ledger table/index DDL, and a minimal synchronous insert/read path." +checks: + - "automation-result.json status is pass." + - "probe-stdout.log contains LEDGER_INVARIANTS_OK." + - "The probe does not require aiosqlite or browser UI." +evidence_required: + - filesystem +diagnostics: + - "env_issue means LANGBOT_REPO/default ../LangBot did not resolve or Python dependencies are unavailable." + - "fail means a ledger schema/status invariant changed and the release gate needs review." +success_patterns: + - "LEDGER_INVARIANTS_OK" +failure_patterns: + - "AssertionError" + - "ledger invariant probe exited" diff --git a/skills/skills/langbot-testing/cases/agent-runner-ledger-stress.yaml b/skills/skills/langbot-testing/cases/agent-runner-ledger-stress.yaml new file mode 100644 index 000000000..acc527943 --- /dev/null +++ b/skills/skills/langbot-testing/cases/agent-runner-ledger-stress.yaml @@ -0,0 +1,33 @@ +id: agent-runner-ledger-stress +title: "AgentRunner ledger lightweight stress probe" +mode: probe +area: release +type: regression +priority: p1 +risk: high +ci_eligible: true +tags: + - agent-runner + - probe + - stress + - ledger +skills: + - langbot-testing +env: +automation: skills/langbot-testing/probes/agent-runner-ledger-stress.mjs +steps: + - "Run `rtk bin/lbs test run agent-runner-ledger-stress --dry-run` first; remove `--dry-run` after checking the planned evidence directory." + - "Automation creates 100 queued runs in synchronous SQLite and simulates five runtimes claiming them in priority order." +checks: + - "automation-result.json status is pass." + - "probe-stdout.log contains LEDGER_STRESS_OK." + - "Every run is claimed once and reaches a terminal completed status." +evidence_required: + - filesystem +diagnostics: + - "This probe is a fast deterministic stress baseline; it does not replace PostgreSQL/async concurrency tests." +success_patterns: + - "LEDGER_STRESS_OK" +failure_patterns: + - "AssertionError" + - "ledger stress exited" diff --git a/skills/skills/langbot-testing/cases/agent-runner-live-install.yaml b/skills/skills/langbot-testing/cases/agent-runner-live-install.yaml new file mode 100644 index 000000000..f1bcaaf72 --- /dev/null +++ b/skills/skills/langbot-testing/cases/agent-runner-live-install.yaml @@ -0,0 +1,46 @@ +id: agent-runner-live-install +title: "QA AgentRunner package installs and registers in LangBot" +mode: probe +area: release +type: regression +priority: p0 +risk: high +ci_eligible: false +tags: + - agent-runner + - plugin + - local-install + - fixture +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_BACKEND_URL + - LANGBOT_REPO + - LANGBOT_E2E_LOGIN_USER +automation: scripts/e2e/install-qa-plugin-smoke.mjs +automation_plugin_package: "skills/langbot-testing/fixtures/plugins/qa-agent-runner/dist/qa-agent-runner-0.1.0.lbpkg" +automation_expected_plugin_id: "qa/agent-runner" +automation_expected_tool: "" +automation_expected_runner_id: "plugin:qa/agent-runner/default" +steps: + - "Run `rtk bin/lbs test run agent-runner-live-install --dry-run` first; remove `--dry-run` only after readiness points at a local test LangBot instance." + - "Automation authenticates the local test user, uploads the QA AgentRunner .lbpkg package, waits for the install task, and reads pipeline metadata." +checks: + - "automation-result.json status is pass." + - "/api/v1/plugins lists qa/agent-runner after install." + - "/api/v1/pipelines/_/metadata lists plugin:qa/agent-runner/default as an available runner." +evidence_required: + - api_diagnostic + - filesystem +diagnostics: + - "This proves the deterministic package installs and registers a runner. It does not prove Debug Chat execution; use a later browser case for that." + - "If installation fails during dependencies, inspect plugin runtime logs and plugin-dependency-install-offline." +success_patterns: + - "qa/agent-runner is installed." +failure_patterns: + - "Plugin install task did not complete successfully" + - "plugin:qa/agent-runner/default is not listed" +troubleshooting: + - plugin-runtime-timeout + - plugin-dependency-install-offline diff --git a/skills/skills/langbot-testing/cases/agent-runner-qa-debug-chat.yaml b/skills/skills/langbot-testing/cases/agent-runner-qa-debug-chat.yaml new file mode 100644 index 000000000..1d86b5ec7 --- /dev/null +++ b/skills/skills/langbot-testing/cases/agent-runner-qa-debug-chat.yaml @@ -0,0 +1,70 @@ +id: agent-runner-qa-debug-chat +title: "QA AgentRunner returns deterministic output through Debug Chat" +mode: agent-browser +area: pipeline +type: regression +priority: p0 +risk: high +ci_eligible: false +tags: + - agent-runner + - pipeline + - debug-chat + - fixture +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_QA_AGENT_RUNNER_PIPELINE_URL + - LANGBOT_QA_AGENT_RUNNER_PIPELINE_NAME +automation: scripts/e2e/pipeline-debug-chat.mjs +automation_env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_BROWSER_PROFILE + - LANGBOT_CHROMIUM_EXECUTABLE + - LANGBOT_QA_AGENT_RUNNER_PIPELINE_URL + - LANGBOT_QA_AGENT_RUNNER_PIPELINE_NAME +automation_pipeline_url_env: LANGBOT_QA_AGENT_RUNNER_PIPELINE_URL +automation_pipeline_name_env: LANGBOT_QA_AGENT_RUNNER_PIPELINE_NAME +automation_expected_runner_id: "plugin:qa/agent-runner/default" +automation_prompt: "hello-live" +automation_expected_text: "QA_AGENT_RUNNER_OK:hello-live" +automation_response_timeout_ms: "120000" +automation_reset_debug_chat: "1" +setup_automation: + - "case:agent-runner-live-install" + - "node:scripts/e2e/ensure-qa-agent-runner-pipeline.mjs --write-env" +setup_provides_env: + - LANGBOT_QA_AGENT_RUNNER_PIPELINE_URL + - LANGBOT_QA_AGENT_RUNNER_PIPELINE_NAME +steps: + - "Open LANGBOT_FRONTEND_URL." + - "Open the pipeline from LANGBOT_QA_AGENT_RUNNER_PIPELINE_URL or LANGBOT_QA_AGENT_RUNNER_PIPELINE_NAME." + - "Confirm the pipeline AI runner is plugin:qa/agent-runner/default." + - "Open Debug Chat." + - "Send: hello-live." +checks: + - "UI: The user message appears in Debug Chat." + - "UI: A Bot message appears and contains QA_AGENT_RUNNER_OK:hello-live." + - "API diagnostic: pipeline config uses plugin:qa/agent-runner/default." + - "Console: No unexpected frontend runtime errors appear during the send/receive path." +evidence_required: + - ui + - screenshot + - console + - network + - api_diagnostic +diagnostics: + - "This is the deterministic live execution proof that sits after fixture contract and live install." + - "If the runner id mismatch is reported, rerun ensure-qa-agent-runner-pipeline.mjs --write-env." +success_patterns: + - "QA_AGENT_RUNNER_OK:hello-live" +failure_patterns: + - "plugin:qa/agent-runner/default execution failed" + - "Action invoke_llm_stream call timed out" + - "Agent runner temporarily unavailable" +troubleshooting: + - plugin-runtime-timeout diff --git a/skills/skills/langbot-testing/cases/agent-runner-release-preflight.yaml b/skills/skills/langbot-testing/cases/agent-runner-release-preflight.yaml new file mode 100644 index 000000000..b70000591 --- /dev/null +++ b/skills/skills/langbot-testing/cases/agent-runner-release-preflight.yaml @@ -0,0 +1,74 @@ +id: agent-runner-release-preflight +title: "Agent runner release gate preflight validates environment readiness" +mode: agent-browser +area: release +type: smoke +priority: p0 +risk: high +ci_eligible: false +tags: + - agent-runner + - release-gate + - preflight + - environment +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL +env_any: + - LANGBOT_LOCAL_AGENT_PIPELINE_URL|LANGBOT_LOCAL_AGENT_PIPELINE_NAME + - LANGBOT_ACP_AGENT_RUNNER_PIPELINE_URL|LANGBOT_ACP_AGENT_RUNNER_PIPELINE_NAME +automation: scripts/e2e/agent-runner-release-preflight.mjs +automation_env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_BROWSER_PROFILE + - LANGBOT_CHROMIUM_EXECUTABLE +automation_env_any: + - LANGBOT_LOCAL_AGENT_PIPELINE_URL|LANGBOT_LOCAL_AGENT_PIPELINE_NAME + - LANGBOT_ACP_AGENT_RUNNER_PIPELINE_URL|LANGBOT_ACP_AGENT_RUNNER_PIPELINE_NAME +preconditions: + - "LANGBOT_LOCAL_AGENT_PIPELINE_URL or LANGBOT_LOCAL_AGENT_PIPELINE_NAME points to the local-agent release pipeline." + - "LANGBOT_ACP_AGENT_RUNNER_PIPELINE_URL or LANGBOT_ACP_AGENT_RUNNER_PIPELINE_NAME points to the ACP AgentRunner release pipeline." + - "The active browser profile is authenticated for the same LangBot backend." + - "By default the preflight performs a cheap model test for the local-agent primary model; set LANGBOT_PREFLIGHT_TEST_MODELS=0 only when deliberately classifying model credentials outside this run." +steps: + - "Open LANGBOT_FRONTEND_URL with the configured browser profile." + - "Use the browser token to call LangBot backend readiness APIs without printing token values." + - "Check plugin runtime status, Box status, required runner plugins, qa-plugin-smoke, and qa_plugin_echo." + - "Resolve the local-agent and ACP AgentRunner QA pipelines from their case-specific env vars." + - "Assert each pipeline uses the expected runner id." + - "Assert the external runner pipeline uses the expected runner id." + - "Assert the local-agent primary model advertises func_call and vision for the full release gate." + - "Run the local-agent primary model test endpoint unless LANGBOT_PREFLIGHT_TEST_MODELS=0." +checks: + - "API diagnostic: api-diagnostic.json has no blockers and no env_issues." + - "API diagnostic: required pipelines resolve to plugin:langbot/local-agent/default and plugin:langbot/acp-agent-runner/default." + - "API diagnostic: qa_plugin_echo is exposed by /api/v1/tools." + - "API diagnostic: local-agent model check catches invalid credentials or missing func_call/vision before release E2E starts." + - "Secret safety: token values, api keys, and provider secrets are not printed." +evidence_required: + - ui + - screenshot + - console + - network + - api_diagnostic +diagnostics: + - "blocked means the test instance is not configured for the full release gate: missing pipeline, wrong runner id, or missing plugin." + - "env_issue means the runtime or upstream dependency is not usable: backend unavailable, plugin runtime down, Box down, or the local-agent model cannot pass a model test." + - "If qa_mcp_echo is absent here, continue to mcp-stdio-register before mcp-stdio-tool-call; qa_mcp_echo is not required before registration." + - "If the model check fails with invalid api key, switch the local-agent release pipeline to a known-good func_call model before diagnosing runner code." +success_patterns: + - "Release gate preflight passed" +failure_patterns: + - "Preflight blocked" + - "Preflight environment issue" + - "invalid api key" + - "runner.llm_error" +troubleshooting: + - backend-not-listening + - plugin-runtime-timeout + - local-agent-model-route-unavailable + - proxy-env-mismatch diff --git a/skills/skills/langbot-testing/cases/agent-runner-runtime-chaos.yaml b/skills/skills/langbot-testing/cases/agent-runner-runtime-chaos.yaml new file mode 100644 index 000000000..d7aeca392 --- /dev/null +++ b/skills/skills/langbot-testing/cases/agent-runner-runtime-chaos.yaml @@ -0,0 +1,38 @@ +id: agent-runner-runtime-chaos +title: "AgentRunner SDK runtime chaos pytest probe" +mode: probe +area: release +type: regression +priority: p0 +risk: high +ci_eligible: true +tags: + - agent-runner + - probe + - pytest + - runtime + - sdk +skills: + - langbot-testing +env: +automation: skills/langbot-testing/probes/agent-runner-runtime-chaos.mjs +steps: + - "Run `rtk bin/lbs test run agent-runner-runtime-chaos --dry-run` first; remove `--dry-run` after checking the SDK repo target and evidence directory." + - "Automation resolves LANGBOT_PLUGIN_SDK_REPO, defaulting to ../langbot-plugin-sdk when the env var is unset." + - "Automation runs the existing SDK pytest files tests/runtime/plugin/test_mgr_agent_runner.py and tests/runtime/test_pull_api_handlers.py." +checks: + - "automation-result.json status is pass." + - "pytest exit status is 0 for the existing AgentRunner runtime and pull API handler tests." + - "pytest-stdout.log and pytest-stderr.log are written under LBS_EVIDENCE_DIR." +evidence_required: + - filesystem +diagnostics: + - "This probe does not open the WebUI; it runs SDK pytest targets directly." + - "env_issue means LANGBOT_PLUGIN_SDK_REPO/default ../langbot-plugin-sdk did not resolve, rtk/uv was unavailable, or the expected test files are missing." + - "fail means the existing SDK runtime pytest target failed or timed out." +success_patterns: + - "pytest passed" +failure_patterns: + - "pytest exited with status" + - "pytest timed out" + - "Failed to start pytest command" diff --git a/skills/skills/langbot-testing/cases/dify-agent-debug-chat.yaml b/skills/skills/langbot-testing/cases/dify-agent-debug-chat.yaml new file mode 100644 index 000000000..42b0f62bf --- /dev/null +++ b/skills/skills/langbot-testing/cases/dify-agent-debug-chat.yaml @@ -0,0 +1,51 @@ +id: dify-agent-debug-chat +title: "Dify AgentRunner returns a response through Pipeline Debug Chat" +mode: agent-browser +area: pipeline +type: provider +priority: p2 +risk: medium +ci_eligible: false +tags: + - dify + - agent-runner + - pipeline +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL +preconditions: + - "A Dify app Service API key is available from the active secret source and must not be printed in reports." + - "The target pipeline is safe to modify for Dify runner configuration." +steps: + - "Ensure a Dify app Service API key is available from the active secret source." + - "Open LANGBOT_FRONTEND_URL." + - "Navigate to Pipelines and open the target pipeline." + - "Open Configuration > AI." + - "Select runner Dify." + - "Set Base URL, App Type, API Key, Base Prompt, and Timeout according to references/dify-agent-runner.md." + - "Save the pipeline." + - "Open Debug Chat." + - "Send a prompt asking the bot to reply exactly with a unique sentinel, for example LANGBOT_DIFY_OK_." +checks: + - "UI: The runner selector and runner config identify Dify, not a generic Default label." + - "UI: Debug Chat shows a Bot message containing the sentinel." + - "History/log validation: The sentinel is present in an assistant/bot message, not only in the echoed User message." + - "Logs: Backend logs show Dify /chat-messages returned HTTP 200 and Conversation(0) Streaming completed." + - "Console: No unexpected frontend errors appear during runner configuration or Debug Chat." + - "Secret safety: No Dify API key, JWT, or browser token is printed in reports." +evidence_required: + - ui + - console + - backend_log +diagnostics: + - "Use direct Dify streaming API only to distinguish invalid Dify credentials from LangBot runner failures." + - "If direct Dify blocking mode fails for an Agent Chat app, retry streaming before treating credentials as invalid." + - "Use GET /api/v1/pipelines/{uuid} only to confirm saved runner_config." +troubleshooting: + - agent-runner-actor-context-fields + - ambiguous-runner-default-label + - plugin-runtime-timeout + - proxy-env-mismatch diff --git a/skills/skills/langbot-testing/cases/langrag-kb-retrieve.yaml b/skills/skills/langbot-testing/cases/langrag-kb-retrieve.yaml new file mode 100644 index 000000000..d7dbd884e --- /dev/null +++ b/skills/skills/langbot-testing/cases/langrag-kb-retrieve.yaml @@ -0,0 +1,57 @@ +id: langrag-kb-retrieve +title: "LangRAG knowledge base ingests and retrieves a sentinel document" +mode: agent-browser +area: knowledge +type: feature +priority: p1 +risk: medium +ci_eligible: false +tags: + - langrag + - knowledge + - rag +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL +automation: scripts/e2e/langrag-kb-retrieve.mjs +automation_env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_BROWSER_PROFILE + - LANGBOT_CHROMIUM_EXECUTABLE +automation_env_any: + - LANGBOT_LOCAL_AGENT_RAG_KB_UUID|LANGBOT_RAG_KB_UUID +automation_expected_text: "azalea-cobalt-7421" +preconditions: + - "LangRAG is installed and initialized in the active LangBot instance." + - "A working embedding model is available, preferably chroma-all-MiniLM-L6-v2 for local repeatability." + - "LANGBOT_LOCAL_AGENT_RAG_KB_UUID points to a LangRAG knowledge base containing azalea-cobalt-7421." +steps: + - "Open LANGBOT_FRONTEND_URL." + - "Navigate to Knowledge." + - "Create a knowledge base with engine LangRAG." + - "Select a working embedding model, preferably local Chroma embedding model chroma-all-MiniLM-L6-v2." + - "Upload skills/langbot-testing/fixtures/rag/sentinel-doc.txt." + - "Wait until the document row status is Completed." + - "Open Retrieve Test and query: What is the local agent runner retrieval sentinel?" +checks: + - "UI: The knowledge base appears in the Knowledge sidebar." + - "UI: The uploaded document status becomes Completed." + - "UI: Retrieve Test shows the uploaded document content." + - "UI: Retrieve Test result contains azalea-cobalt-7421." + - "Console: No unexpected frontend errors appear during creation, upload, or retrieve." +evidence_required: + - ui + - screenshot + - console + - backend_log +diagnostics: + - "If no LangRAG engine is available, check /api/v1/knowledge/engines and install langbot-team/LangRAG." + - "If the embedding selector does not show a local Chroma model, confirm the model exists under embedding_models, not llm_models." +troubleshooting: + - marketplace-network-flaky + - dynamic-form-missing-config-id + - pipeline-form-controlled-warning diff --git a/skills/skills/langbot-testing/cases/langrag-parser-golden-e2e.yaml b/skills/skills/langbot-testing/cases/langrag-parser-golden-e2e.yaml new file mode 100644 index 000000000..c1c7da35f --- /dev/null +++ b/skills/skills/langbot-testing/cases/langrag-parser-golden-e2e.yaml @@ -0,0 +1,69 @@ +id: langrag-parser-golden-e2e +title: "LangRAG and GeneralParsers retrieve a structured golden document" +mode: agent-browser +area: knowledge +type: regression +priority: p1 +risk: high +ci_eligible: false +tags: + - golden + - e2e + - langrag + - parser + - knowledge +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_REPO + - LANGBOT_WEB_REPO + - LANGBOT_RAG_PLUGIN_REPO + - LANGBOT_PARSER_PLUGIN_REPO +preconditions: + - "LANGBOT_REPO and plugin repo env values point to the worktrees intended for this golden parser/RAG run." + - "The active LangBot environment can build and install local LangRAG and GeneralParsers plugin packages." + - "A working embedding model is available before the Knowledge UI path starts." +steps: + - "Use LANGBOT_REPO as the active LangBot worktree and confirm it is the current master worktree for this run." + - "Start or verify the LangBot backend and frontend from LANGBOT_REPO and LANGBOT_WEB_REPO." + - "Build local plugin zips from LANGBOT_RAG_PLUGIN_REPO and LANGBOT_PARSER_PLUGIN_REPO using LANGBOT_REPO/.venv/bin/lbp build." + - "Open LANGBOT_FRONTEND_URL." + - "Navigate to Plugins and install or update the local LangRAG zip and GeneralParsers zip." + - "Wait until both plugins are initialized: langbot-team/LangRAG and langbot-team/GeneralParsers." + - "Navigate to Knowledge." + - "Create a knowledge base with engine LangRAG and a working embedding model, preferably chroma-all-MiniLM-L6-v2." + - "Keep index type Chunk for this golden case." + - "Upload skills/langbot-testing/fixtures/rag/parser-golden.html." + - "When the upload UI asks for a parser, select GeneralParsers for text/html." + - "Wait until the document row status is Completed." + - "Open Retrieve Test and query: What is the parser-rag golden sentinel and which parser/engine pair is documented? Return the exact sentinel and pair." +checks: + - "UI: Plugins shows langbot-team/LangRAG initialized." + - "UI: Plugins shows langbot-team/GeneralParsers initialized." + - "UI: The upload flow selects GeneralParsers for the HTML fixture, or the parser selector clearly defaults to GeneralParsers." + - "UI: The uploaded parser-golden.html document status becomes Completed." + - "UI: Retrieve Test result contains aurora-parser-rag-9137." + - "UI: Retrieve Test result contains GeneralParsers and LangRAG." + - "UI: Retrieve Test result preserves the table as Markdown, including '| Parser field | Golden value |'." + - "Console: No unexpected frontend errors appear during plugin install, KB creation, upload, or retrieve." + - "Logs: Backend/plugin logs show GeneralParsers parsed parser-golden.html and LangRAG used pre-parsed external parser content." +evidence_required: + - ui + - screenshot + - console + - backend_log +diagnostics: + - "If LangRAG is missing, check /api/v1/knowledge/engines for plugin id langbot-team/LangRAG." + - "If GeneralParsers is missing, check /api/v1/knowledge/parsers?mime_type=text/html for plugin id langbot-team/GeneralParsers." + - "If GeneralParsers local install fails on PyMuPDF, confirm the active LangBot master venv can import fitz and retry Upload Local." + - "If the document completes but table pipes are missing, inspect logs for LangRAG fallback to the internal FileParser instead of external parser content." + - "If embedding selection is empty, confirm the test embedding model exists under embedding_models, not llm_models." +troubleshooting: + - plugin-runtime-timeout + - plugin-dependency-install-offline + - marketplace-network-flaky + - dynamic-form-missing-config-id + - proxy-env-mismatch diff --git a/skills/skills/langbot-testing/cases/langrag-sentinel-kb-discover.yaml b/skills/skills/langbot-testing/cases/langrag-sentinel-kb-discover.yaml new file mode 100644 index 000000000..1e1c246ff --- /dev/null +++ b/skills/skills/langbot-testing/cases/langrag-sentinel-kb-discover.yaml @@ -0,0 +1,38 @@ +id: langrag-sentinel-kb-discover +title: "Existing LangRAG sentinel knowledge base is discoverable" +mode: probe +area: knowledge +type: regression +priority: p1 +risk: medium +ci_eligible: false +tags: + - langrag + - knowledge + - rag + - fixture +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_BACKEND_URL + - LANGBOT_REPO + - LANGBOT_E2E_LOGIN_USER +automation: scripts/e2e/ensure-langrag-sentinel-kb.mjs +automation_expected_text: "azalea-cobalt-7421" +steps: + - "Run `rtk bin/lbs test run langrag-sentinel-kb-discover --dry-run` first; remove `--dry-run` only after readiness points at a local test LangBot instance." + - "Automation authenticates the local test user, lists knowledge bases, retrieves against each one, and looks for azalea-cobalt-7421." +checks: + - "automation-result.json status is pass when an existing KB retrieves the sentinel." + - "When run with `node:scripts/e2e/ensure-langrag-sentinel-kb.mjs --write-env`, LANGBOT_LOCAL_AGENT_RAG_KB_UUID is written to skills/.env.local." +evidence_required: + - api_diagnostic +diagnostics: + - "This case does not create a knowledge base. It only discovers a KB already prepared by langrag-kb-retrieve or by an equivalent local setup." +success_patterns: + - "Found LangRAG sentinel knowledge base" +failure_patterns: + - "No existing knowledge base retrieved expected sentinel" +troubleshooting: + - marketplace-network-flaky diff --git a/skills/skills/langbot-testing/cases/local-agent-basic-debug-chat.yaml b/skills/skills/langbot-testing/cases/local-agent-basic-debug-chat.yaml new file mode 100644 index 000000000..0b1cc1352 --- /dev/null +++ b/skills/skills/langbot-testing/cases/local-agent-basic-debug-chat.yaml @@ -0,0 +1,71 @@ +id: local-agent-basic-debug-chat +title: "Local Agent Debug Chat returns a deterministic streaming response" +mode: agent-browser +area: pipeline +type: smoke +priority: p0 +risk: medium +ci_eligible: false +tags: + - local-agent + - pipeline + - streaming +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation: scripts/e2e/pipeline-debug-chat.mjs +automation_env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BROWSER_PROFILE + - LANGBOT_CHROMIUM_EXECUTABLE + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL +automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation_prompt: "请只回复 OK，用于前端调试测试。" +automation_expected_text: "OK" +setup_automation: + - "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env" +setup_provides_env: + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +steps: + - "Open LANGBOT_FRONTEND_URL." + - "Navigate to Pipelines and open the target local-agent pipeline." + - "Open Configuration > AI." + - "Use runner Default or the pluginized langbot/local-agent runner." + - "Select a model that is known to answer Debug Chat in the current environment." + - "Clear Knowledge Bases unless this run intentionally combines RAG with the smoke path." + - "Save the pipeline." + - "Open Debug Chat." + - "Ensure the stream switch is enabled when the UI exposes it." + - "Send: 请只回复 OK，用于前端调试测试。" +checks: + - "UI: The user message appears in Debug Chat." + - "UI: A Bot message appears and contains OK." + - "Console: No unexpected frontend runtime errors appear during the send/receive path." + - "Logs: Backend logs show the debug chat request completed on the streaming path instead of timing out in plugin/runtime calls." +evidence_required: + - ui + - screenshot + - console + - backend_log +diagnostics: + - "Provider errors such as model_not_found or no available channel mean the selected model is unavailable; switch to a known-good model before diagnosing local-agent." + - "Use GET /api/v1/pipelines/{uuid} only to confirm the saved runner and model config." +success_patterns: + - "Processing request from person_websocket" + - "Streaming completed" +failure_patterns: + - "Action invoke_llm_stream call timed out" + - "Task exception was never retrieved" + - "survey widget blocks debug chat" +troubleshooting: + - plugin-runtime-timeout + - proxy-env-mismatch + - survey-widget-blocks-debug-chat diff --git a/skills/skills/langbot-testing/cases/local-agent-context-compaction-debug-chat.yaml b/skills/skills/langbot-testing/cases/local-agent-context-compaction-debug-chat.yaml new file mode 100644 index 000000000..7c0d6a32a --- /dev/null +++ b/skills/skills/langbot-testing/cases/local-agent-context-compaction-debug-chat.yaml @@ -0,0 +1,87 @@ +id: local-agent-context-compaction-debug-chat +title: "Local Agent compacts long Debug Chat history and preserves older facts" +mode: agent-browser +area: pipeline +type: regression +priority: p1 +risk: high +ci_eligible: false +tags: + - local-agent + - pipeline + - context + - compaction +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation: scripts/e2e/pipeline-debug-chat.mjs +automation_env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_BROWSER_PROFILE + - LANGBOT_CHROMIUM_EXECUTABLE + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL +automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation_expected_runner_id: "plugin:langbot/local-agent/default" +automation_runner_config_patch_json: '{"context-window-tokens":225,"context-reserve-tokens":50,"context-keep-recent-tokens":30,"context-summary-tokens":105,"knowledge-bases":[]}' +automation_restore_runner_config: "1" +automation_reset_debug_chat: "1" +automation_debug_chat_session_type: "person" +automation_expected_text: "qa_compaction_sentinel_7391" +automation_response_timeout_ms: "180000" +automation_prompts_json: '[{"prompt":"请记住这个用于 local-agent context compaction 回归测试的暗号：qa_compaction_sentinel_7391。请只回复 MEMORY_SET。","expected_text":"MEMORY_SET","response_timeout_ms":"180000"},{"prompt":"下面这轮只用于制造长历史压力，内容没有业务含义。请忽略填充内容，最后只回复 CONTEXT_PRESSURE_READY。填充片段 A001 context padding for local-agent compaction. A002 context padding for local-agent compaction. A003 context padding for local-agent compaction. A004 context padding for local-agent compaction. A005 context padding for local-agent compaction. A006 context padding for local-agent compaction. A007 context padding for local-agent compaction. A008 context padding for local-agent compaction. A009 context padding for local-agent compaction. A010 context padding for local-agent compaction. A011 context padding for local-agent compaction. A012 context padding for local-agent compaction. A013 context padding for local-agent compaction. A014 context padding for local-agent compaction. A015 context padding for local-agent compaction. A016 context padding for local-agent compaction. A017 context padding for local-agent compaction. A018 context padding for local-agent compaction. A019 context padding for local-agent compaction. A020 context padding for local-agent compaction. A021 context padding for local-agent compaction. A022 context padding for local-agent compaction. A023 context padding for local-agent compaction. A024 context padding for local-agent compaction. A025 context padding for local-agent compaction. A026 context padding for local-agent compaction. A027 context padding for local-agent compaction. A028 context padding for local-agent compaction. A029 context padding for local-agent compaction. A030 context padding for local-agent compaction. A031 context padding for local-agent compaction. A032 context padding for local-agent compaction. A033 context padding for local-agent compaction. A034 context padding for local-agent compaction. A035 context padding for local-agent compaction. A036 context padding for local-agent compaction. A037 context padding for local-agent compaction. A038 context padding for local-agent compaction. A039 context padding for local-agent compaction. A040 context padding for local-agent compaction.","expected_text":"CONTEXT_PRESSURE_READY","response_timeout_ms":"180000"},{"prompt":"刚才第一轮我要求你记住的测试暗号是什么？请只回复暗号本身，不要解释。","expected_text":"qa_compaction_sentinel_7391","response_timeout_ms":"180000"}]' +setup_automation: + - "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env" +setup_provides_env: + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +preconditions: + - "The selected model route can follow short deterministic instructions across multiple Debug Chat turns." +steps: + - "Open the target local-agent pipeline through LANGBOT_FRONTEND_URL." + - "Use the authenticated browser token only inside automation to GET and PUT /api/v1/pipelines/{uuid}." + - "Assert the saved runner is plugin:langbot/local-agent/default." + - "Temporarily set context-window-tokens, context-reserve-tokens, context-keep-recent-tokens, and context-summary-tokens to force compaction, and clear knowledge-bases so RAG does not answer the memory question." + - "Reset the person Debug Chat session for the target pipeline." + - "Send the sentinel memory prompt and wait for MEMORY_SET." + - "Send the long padding prompt and wait for CONTEXT_PRESSURE_READY." + - "Ask for the original sentinel and wait for qa_compaction_sentinel_7391." + - "Restore the original runner config." +checks: + - "UI: All three user messages appear in Debug Chat." + - "UI: The final Bot message contains qa_compaction_sentinel_7391." + - "API diagnostic: pipeline-config-diagnostic.json shows patched=true and patch_keys include the four token context compaction fields plus knowledge-bases." + - "API diagnostic: pipeline-config-restore-diagnostic.json shows the original runner config was restored." + - "Logs: Backend completes the multi-turn Debug Chat path without runner timeout or model setup failure." + - "Console: No unexpected frontend runtime errors appear during the run." +evidence_required: + - ui + - screenshot + - console + - backend_log + - api_diagnostic +diagnostics: + - "If the final sentinel is missing, inspect whether pipeline-config-diagnostic.json targeted ai.runner_config[runnerId], cleared knowledge-bases, and whether the backend log shows the local-agent runner loading the small context settings." + - "If the model ignores deterministic replies, rerun with a known-good model route before diagnosing ContextAssembler." + - "If restore fails, use pipeline-config-restore-diagnostic.json and GET /api/v1/pipelines/{uuid} to confirm the current saved config before retrying." +success_patterns: + - "Processing request from person_websocket" + - "Streaming completed" +failure_patterns: + - "Action invoke_llm_stream call timed out" + - "All models failed during streaming setup" + - "Task exception was never retrieved" + - "survey widget blocks debug chat" +troubleshooting: + - local-agent-model-route-unavailable + - plugin-runtime-timeout + - proxy-env-mismatch + - survey-widget-blocks-debug-chat + - debug-chat-history-contaminates-automation diff --git a/skills/skills/langbot-testing/cases/local-agent-effective-prompt-debug-chat.yaml b/skills/skills/langbot-testing/cases/local-agent-effective-prompt-debug-chat.yaml new file mode 100644 index 000000000..d19d4a109 --- /dev/null +++ b/skills/skills/langbot-testing/cases/local-agent-effective-prompt-debug-chat.yaml @@ -0,0 +1,67 @@ +id: local-agent-effective-prompt-debug-chat +title: "Local Agent consumes host effective prompt after PromptPreProcessing" +mode: agent-browser +area: pipeline +type: regression +priority: p1 +risk: high +ci_eligible: false +tags: + - local-agent + - prompt + - plugin + - pipeline +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation: scripts/e2e/pipeline-debug-chat.mjs +automation_env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BROWSER_PROFILE + - LANGBOT_CHROMIUM_EXECUTABLE + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL +automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation_prompt: "qa-effective-prompt" +automation_expected_text: "PROMPT_PREPROCESS_OK" +automation_response_timeout_ms: "180000" +setup_automation: + - "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env" + - "case:qa-plugin-smoke-live-install" +setup_provides_env: + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +preconditions: + - "The target pipeline is safe to modify and can bind the fixture plugin through Extensions." +steps: + - "Install or enable the bundled qa-plugin-smoke fixture plugin." + - "Confirm the fixture plugin is bound to the target pipeline through Extensions." + - "Open the target local-agent pipeline." + - "Open Configuration > AI." + - "Use runner Default or the pluginized langbot/local-agent runner." + - "Select a model that is known to follow system prompts in Debug Chat." + - "Save the pipeline." + - "Open Debug Chat." + - "Send: qa-effective-prompt" +checks: + - "UI: A Bot message appears and contains PROMPT_PREPROCESS_OK." + - "Logs: PromptPreProcessing runs for the fixture plugin before the local-agent runner invokes the model." + - "Logs: Backend completes the debug-chat request without plugin/runtime timeout." + - "Console: No unexpected frontend runtime errors appear during configuration or chat." +evidence_required: + - ui + - console + - backend_log +diagnostics: + - "If the bot does not return PROMPT_PREPROCESS_OK, verify the fixture plugin is installed, enabled, and bound to the pipeline before diagnosing ctx.adapter.extra.prompt." + - "If the plugin event runs but the answer ignores the sentinel, inspect whether the runner is using ctx.adapter.extra.prompt instead of static runner config prompt." +troubleshooting: + - plugin-runtime-timeout + - proxy-env-mismatch + - survey-widget-blocks-debug-chat diff --git a/skills/skills/langbot-testing/cases/local-agent-multimodal-debug-chat.yaml b/skills/skills/langbot-testing/cases/local-agent-multimodal-debug-chat.yaml new file mode 100644 index 000000000..298b59dca --- /dev/null +++ b/skills/skills/langbot-testing/cases/local-agent-multimodal-debug-chat.yaml @@ -0,0 +1,71 @@ +id: local-agent-multimodal-debug-chat +title: "Local Agent Debug Chat preserves uploaded image input" +mode: agent-browser +area: pipeline +type: regression +priority: p2 +risk: medium +ci_eligible: false +tags: + - local-agent + - multimodal + - pipeline +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation: scripts/e2e/pipeline-debug-chat.mjs +automation_env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BROWSER_PROFILE + - LANGBOT_CHROMIUM_EXECUTABLE + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL +automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation_prompt: "I attached an image. Reply only IMAGE_OK if you received the image." +automation_expected_text: "IMAGE_OK" +automation_image_base64_fixture: "skills/langbot-testing/fixtures/multimodal/red-square.png.base64" +setup_automation: + - "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env" +setup_provides_env: + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +preconditions: + - "The selected model route accepts image input, or the case is intentionally checking graceful provider rejection." +steps: + - "Prepare a small PNG file for upload. The bundled fixture base64 is at skills/langbot-testing/fixtures/multimodal/red-square.png.base64 if a temporary file is needed." + - "Open LANGBOT_FRONTEND_URL." + - "Navigate to Pipelines and open the target local-agent pipeline." + - "Open Configuration > AI." + - "Use runner Default or the pluginized langbot/local-agent runner." + - "Select a model that supports image input in the current environment, or use a known model that at least accepts the uploaded image payload." + - "Save the pipeline." + - "Open Debug Chat." + - "Attach the PNG through the image/file upload control. Prefer the bundled 64x64 red-square fixture; 1x1 images may be rejected by some model providers before runner behavior is exercised." + - "Confirm the user compose area or sent message shows the image attachment." + - "Send: I attached an image. Reply only IMAGE_OK if you received the image." +checks: + - "UI: The sent User message shows an image attachment, not just text." + - "UI: The Bot message contains IMAGE_OK." + - "Network or logs: The browser sends an image upload request, or backend logs show the local-agent input contains an image." + - "Console: No unexpected frontend runtime errors appear during upload or Debug Chat." +evidence_required: + - ui + - screenshot + - console + - network + - backend_log +diagnostics: + - "If the model cannot process image input, repeat with a multimodal-capable model before diagnosing local-agent." + - "For RAG plus multimodal coverage, keep a KB bound and verify the image remains visible while the answer uses the KB sentinel." +troubleshooting: + - local-agent-model-route-unavailable + - plugin-runtime-timeout + - proxy-env-mismatch + - provider-image-parse-error + - survey-widget-blocks-debug-chat diff --git a/skills/skills/langbot-testing/cases/local-agent-nonstreaming-debug-chat.yaml b/skills/skills/langbot-testing/cases/local-agent-nonstreaming-debug-chat.yaml new file mode 100644 index 000000000..910032d79 --- /dev/null +++ b/skills/skills/langbot-testing/cases/local-agent-nonstreaming-debug-chat.yaml @@ -0,0 +1,64 @@ +id: local-agent-nonstreaming-debug-chat +title: "Local Agent Debug Chat returns a deterministic non-streaming response" +mode: agent-browser +area: pipeline +type: regression +priority: p1 +risk: medium +ci_eligible: false +tags: + - local-agent + - pipeline + - non-streaming +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation: scripts/e2e/pipeline-debug-chat.mjs +automation_env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BROWSER_PROFILE + - LANGBOT_CHROMIUM_EXECUTABLE + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL +automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation_prompt: "Reply only NONSTREAM_OK." +automation_expected_text: "NONSTREAM_OK" +automation_stream_output: "0" +setup_automation: + - "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env" +setup_provides_env: + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +steps: + - "Open LANGBOT_FRONTEND_URL." + - "Navigate to Pipelines and open the target local-agent pipeline." + - "Open Configuration > AI." + - "Use runner Default or the pluginized langbot/local-agent runner." + - "Select a model that is known to answer Debug Chat in the current environment." + - "Save the pipeline." + - "Open Debug Chat." + - "Disable the stream switch when the UI exposes it." + - "Send: Reply only NONSTREAM_OK." +checks: + - "UI: The user message appears in Debug Chat." + - "UI: A Bot message appears and contains NONSTREAM_OK." + - "Logs: Backend completes the request as a normal response rather than only relying on the streaming-completed path." + - "Console: No unexpected frontend runtime errors appear during the send/receive path." +evidence_required: + - ui + - console + - backend_log +diagnostics: + - "If the UI still streams after the switch is disabled, inspect the adapter streaming capability and runner config before diagnosing the model." + - "Use GET /api/v1/pipelines/{uuid} only to confirm the saved runner and model config." +troubleshooting: + - local-agent-model-route-unavailable + - plugin-runtime-timeout + - proxy-env-mismatch + - survey-widget-blocks-debug-chat diff --git a/skills/skills/langbot-testing/cases/local-agent-plugin-tool-call-debug-chat.yaml b/skills/skills/langbot-testing/cases/local-agent-plugin-tool-call-debug-chat.yaml new file mode 100644 index 000000000..1c37e2fb1 --- /dev/null +++ b/skills/skills/langbot-testing/cases/local-agent-plugin-tool-call-debug-chat.yaml @@ -0,0 +1,68 @@ +id: local-agent-plugin-tool-call-debug-chat +title: "Local Agent can call a plugin-provided tool" +mode: agent-browser +area: pipeline +type: regression +priority: p1 +risk: high +ci_eligible: false +tags: + - local-agent + - plugin + - tools + - pipeline +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation: scripts/e2e/pipeline-debug-chat.mjs +automation_env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BROWSER_PROFILE + - LANGBOT_CHROMIUM_EXECUTABLE + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL +automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation_prompt: "Call the qa_plugin_echo tool with exactly this text: plugin-tool-ok-local-agent. Return only the tool result." +automation_expected_text: "qa-plugin-smoke:plugin-tool-ok-local-agent" +automation_response_timeout_ms: "180000" +setup_automation: + - "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env" + - "case:qa-plugin-smoke-live-install" +setup_provides_env: + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +preconditions: + - "The selected model route supports function/tool calling." +steps: + - "Install or enable the bundled qa-plugin-smoke fixture plugin." + - "Confirm /api/v1/tools or the plugin detail shows qa_plugin_echo." + - "Confirm the fixture plugin is bound to the target pipeline through Extensions, or that all plugins are enabled." + - "Open the target local-agent pipeline." + - "Use runner Default or the pluginized langbot/local-agent runner." + - "Select a model with function-calling ability that is known to work with tools in the current environment." + - "Open Debug Chat." + - "Send: Call the qa_plugin_echo tool with exactly this text: plugin-tool-ok-local-agent. Return only the tool result." +checks: + - "UI: Debug Chat bot response contains qa-plugin-smoke:plugin-tool-ok-local-agent." + - "Logs: Backend logs show the plugin tool call was executed, not only listed." + - "Console: No unexpected frontend errors appear during Debug Chat." +evidence_required: + - ui + - console + - backend_log + - api_diagnostic +diagnostics: + - "If qa_plugin_echo is not listed, rebuild and reinstall the qa-plugin-smoke fixture plugin." + - "If the selected model returns model_not_found or no available channel only when tools are provided, switch to a known-good function-calling model before diagnosing plugin tools or local-agent." +troubleshooting: + - local-agent-model-route-unavailable + - tool-name-collision-between-mcp-and-plugin + - plugin-runtime-timeout + - proxy-env-mismatch + - survey-widget-blocks-debug-chat diff --git a/skills/skills/langbot-testing/cases/local-agent-rag-debug-chat.yaml b/skills/skills/langbot-testing/cases/local-agent-rag-debug-chat.yaml new file mode 100644 index 000000000..b7e715b16 --- /dev/null +++ b/skills/skills/langbot-testing/cases/local-agent-rag-debug-chat.yaml @@ -0,0 +1,76 @@ +id: local-agent-rag-debug-chat +title: "Local Agent Debug Chat answers from a LangRAG knowledge base" +mode: agent-browser +area: pipeline +type: regression +priority: p1 +risk: high +ci_eligible: false +tags: + - local-agent + - langrag + - pipeline +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation: scripts/e2e/pipeline-debug-chat.mjs +automation_env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_BROWSER_PROFILE + - LANGBOT_CHROMIUM_EXECUTABLE + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME + - LANGBOT_LOCAL_AGENT_RAG_KB_UUID +automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL +automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation_expected_runner_id: "plugin:langbot/local-agent/default" +automation_runner_config_patch_json: '{"knowledge-bases":["${LANGBOT_LOCAL_AGENT_RAG_KB_UUID}"]}' +automation_restore_runner_config: "1" +automation_reset_debug_chat: "1" +automation_prompt: "Using the knowledge base, what is the local agent runner retrieval sentinel? Return only the sentinel." +automation_expected_text: "azalea-cobalt-7421" +setup_automation: + - "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env" + - "node:scripts/e2e/ensure-langrag-sentinel-kb.mjs --write-env" +setup_provides_env: + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME + - LANGBOT_LOCAL_AGENT_RAG_KB_UUID +preconditions: + - "The target pipeline already has a text-capable model route that is available for this run." +steps: + - "Ensure case langrag-kb-retrieve has produced a knowledge base containing sentinel azalea-cobalt-7421." + - "Open LANGBOT_FRONTEND_URL." + - "Navigate to Pipelines and open the target pipeline." + - "Open Configuration > AI." + - "Use runner Default and add the LangRAG knowledge base to Knowledge Bases." + - "Save the pipeline." + - "Open Debug Chat." + - "Send: Using the knowledge base, what is the local agent runner retrieval sentinel? Return only the sentinel." +checks: + - "UI: The AI config shows the selected knowledge base under Knowledge Bases." + - "UI: The pipeline saves successfully." + - "UI: Debug Chat shows a Bot message containing azalea-cobalt-7421." + - "Console: No unexpected frontend errors appear during configuration or chat." + - "Logs: Backend logs show the debug chat request completed instead of plugin/runtime timeout." + - "API diagnostic: pipeline-config-diagnostic.json shows knowledge-bases and model were temporarily patched." + - "API diagnostic: pipeline-config-restore-diagnostic.json shows the original runner config was restored." +evidence_required: + - ui + - screenshot + - console + - backend_log + - api_diagnostic +diagnostics: + - "Use GET /api/v1/pipelines/{uuid} only to confirm the saved runner_config contains the knowledge base uuid." + - "If the bot ignores the knowledge base, rerun Retrieve Test before debugging the runner." +troubleshooting: + - plugin-runtime-timeout + - proxy-env-mismatch + - survey-widget-blocks-debug-chat diff --git a/skills/skills/langbot-testing/cases/local-agent-rag-multimodal-debug-chat.yaml b/skills/skills/langbot-testing/cases/local-agent-rag-multimodal-debug-chat.yaml new file mode 100644 index 000000000..4e0759363 --- /dev/null +++ b/skills/skills/langbot-testing/cases/local-agent-rag-multimodal-debug-chat.yaml @@ -0,0 +1,74 @@ +id: local-agent-rag-multimodal-debug-chat +title: "Local Agent preserves image input while using RAG context" +mode: agent-browser +area: pipeline +type: regression +priority: p2 +risk: high +ci_eligible: false +tags: + - local-agent + - rag + - multimodal + - pipeline +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation: scripts/e2e/pipeline-debug-chat.mjs +automation_env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_BROWSER_PROFILE + - LANGBOT_CHROMIUM_EXECUTABLE + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME + - LANGBOT_LOCAL_AGENT_RAG_KB_UUID +automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL +automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation_runner_config_patch_json: '{"knowledge-bases":["${LANGBOT_LOCAL_AGENT_RAG_KB_UUID}"]}' +automation_restore_runner_config: "1" +automation_reset_debug_chat: "1" +automation_prompt: "I attached an image. Using the knowledge base, what is the local agent runner retrieval sentinel? Return only the sentinel." +automation_expected_text: "azalea-cobalt-7421" +automation_image_base64_fixture: "skills/langbot-testing/fixtures/multimodal/red-square.png.base64" +setup_automation: + - "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env" + - "node:scripts/e2e/ensure-langrag-sentinel-kb.mjs --write-env" +setup_provides_env: + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME + - LANGBOT_LOCAL_AGENT_RAG_KB_UUID +preconditions: + - "The selected model route accepts image input, or the case is intentionally checking graceful provider rejection." +steps: + - "Ensure case langrag-kb-retrieve has produced a knowledge base containing sentinel azalea-cobalt-7421." + - "Prepare a PNG file for upload. Prefer the bundled fixture at skills/langbot-testing/fixtures/multimodal/red-square.png.base64." + - "Open LANGBOT_FRONTEND_URL." + - "Navigate to Pipelines and open the target local-agent pipeline." + - "Open Debug Chat." + - "Attach the PNG through the image/file upload control." + - "Send: I attached an image. Use the knowledge base and reply only the unique sentinel from the knowledge base." +checks: + - "UI: The sent User message shows an image attachment." + - "UI: The Bot message contains the expected KB sentinel." + - "Logs: The same backend request line contains [Image], and the request completes without runner.llm_error." + - "Console: No unexpected frontend runtime errors appear during upload or Debug Chat." +evidence_required: + - ui + - screenshot + - console + - backend_log +diagnostics: + - "This case does not require the model to identify image content; provider vision quality is not the local-agent runner contract." + - "If the provider rejects the image, retest with the 64x64 red-square fixture and a known multimodal-capable model route." + - "If the image is visible but the sentinel is absent, first verify the KB is bound and retrievable in local-agent-rag-debug-chat." +troubleshooting: + - plugin-runtime-timeout + - proxy-env-mismatch + - provider-image-parse-error + - survey-widget-blocks-debug-chat diff --git a/skills/skills/langbot-testing/cases/local-agent-steering-debug-chat.yaml b/skills/skills/langbot-testing/cases/local-agent-steering-debug-chat.yaml new file mode 100644 index 000000000..973822f2b --- /dev/null +++ b/skills/skills/langbot-testing/cases/local-agent-steering-debug-chat.yaml @@ -0,0 +1,87 @@ +id: local-agent-steering-debug-chat +title: "Local Agent injects a follow-up into an active run" +mode: agent-browser +area: pipeline +type: regression +priority: p1 +risk: high +ci_eligible: false +tags: + - local-agent + - pipeline + - steering + - tools +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation: scripts/e2e/local-agent-steering-debug-chat.mjs +automation_env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_BROWSER_PROFILE + - LANGBOT_CHROMIUM_EXECUTABLE + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL +automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation_expected_runner_id: "plugin:langbot/local-agent/default" +automation_reset_debug_chat: "1" +automation_stream_output: "1" +automation_prompt: "You are running the LangBot steering E2E test. First call the qa_plugin_sleep tool with seconds=8 and text=steering-e2e-anchor. Do not answer before the tool result is available. After the tool returns, answer the latest user follow-up. If no follow-up was injected, reply only STEERING_NO_FOLLOWUP." +automation_expected_text: "qa_steering_sentinel_6194" +automation_response_timeout_ms: "240000" +setup_automation: + - "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env" + - "case:qa-plugin-smoke-live-install" +setup_provides_env: + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +preconditions: + - "The selected model route supports function/tool calling and can call qa_plugin_sleep." +steps: + - "Open the target local-agent pipeline." + - "Assert the saved runner is plugin:langbot/local-agent/default." + - "Reset the person Debug Chat session for the target pipeline." + - "Assert /api/v1/tools contains qa_plugin_sleep." + - "Send the first prompt that instructs the model to call qa_plugin_sleep for 8 seconds." + - "After 1 second, send a second Debug Chat message containing qa_steering_sentinel_6194 while the first run is still active." + - "Wait for the final assistant response." +checks: + - "UI: Debug Chat shows two new user messages but only one new assistant response." + - "UI: The only new assistant response contains qa_steering_sentinel_6194." + - "API diagnostic: pipeline-config-diagnostic.json confirms the local-agent runner id." + - "API diagnostic: tool-diagnostic.json confirms qa_plugin_sleep is exposed." + - "Logs: Backend handles the follow-up without starting an independent second assistant response." + - "Console: No unexpected frontend runtime errors appear during Debug Chat." +evidence_required: + - ui + - screenshot + - console + - backend_log + - api_diagnostic +diagnostics: + - "If qa_plugin_sleep is missing, rebuild and reinstall or resync the qa-plugin-smoke fixture plugin, then restart the backend." + - "If the UI cannot send the follow-up because the input is disabled during a run, the WebUI path does not yet support steering." + - "If two assistant responses appear, the follow-up likely started a separate run instead of being claimed as steering." + - "If the model replies STEERING_NO_FOLLOWUP, inspect Host steering claim logs and local-agent steering_pull calls." +success_patterns: + - "Processing request from person_websocket" + - "Steering" + - "Streaming completed" +failure_patterns: + - "STEERING_NO_FOLLOWUP" + - "Action invoke_llm_stream call timed out" + - "All models failed during streaming setup" + - "Task exception was never retrieved" + - "survey widget blocks debug chat" +troubleshooting: + - local-agent-model-route-unavailable + - plugin-runtime-timeout + - proxy-env-mismatch + - survey-widget-blocks-debug-chat + - debug-chat-history-contaminates-automation diff --git a/skills/skills/langbot-testing/cases/mcp-stdio-register.yaml b/skills/skills/langbot-testing/cases/mcp-stdio-register.yaml new file mode 100644 index 000000000..6c9857ba9 --- /dev/null +++ b/skills/skills/langbot-testing/cases/mcp-stdio-register.yaml @@ -0,0 +1,59 @@ +id: mcp-stdio-register +title: "MCP stdio fixture is registered and exposes qa_mcp_echo" +mode: agent-browser +area: mcp +type: smoke +priority: p1 +risk: medium +ci_eligible: false +tags: + - mcp + - tools + - fixture + - preflight +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL +automation: scripts/e2e/mcp-stdio-register.mjs +automation_env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_BROWSER_PROFILE + - LANGBOT_CHROMIUM_EXECUTABLE +preconditions: + - "The active LangBot instance is safe to create or update the qa-local-stdio MCP server." + - "box.local.allowed_mount_roots allows the bundled MCP fixture path when Box runs stdio MCP servers." + - "The bundled dependency-free fixture skills/langbot-testing/fixtures/mcp/qa_mcp_echo_server.py exists." +steps: + - "Open LANGBOT_FRONTEND_URL with the configured browser profile." + - "Use the browser token to upsert MCP server qa-local-stdio in stdio mode." + - "Point the server command to python with the bundled qa_mcp_echo_server.py fixture path." + - "Poll /api/v1/tools and /api/v1/mcp/servers/qa-local-stdio until qa_mcp_echo is visible." +checks: + - "API diagnostic: qa-local-stdio runtime_status is connected." + - "API diagnostic: runtime_tool_names includes qa_mcp_echo." + - "API diagnostic: /api/v1/tools includes qa_mcp_echo." + - "Console and network logs contain no unexpected frontend/runtime failures." +evidence_required: + - screenshot + - console + - network + - api_diagnostic +diagnostics: + - "If the server does not connect, run node scripts/e2e/mcp-stdio-fixture.mjs to validate the fixture outside LangBot first." + - "If /api/v1/tools keeps showing an old MCP name, rerun this case after restarting LangBot so the server registry is fresh." + - "If Box reports an allowed mount root error, update the local LangBot data config rather than changing shared test assets." +success_patterns: + - "MCP server qa-local-stdio is connected and exposes qa_mcp_echo" +failure_patterns: + - "MCP fixture not found" + - "Browser profile has no localStorage token" + - "did not expose qa_mcp_echo" +troubleshooting: + - mcp-stdio-args-not-applied + - backend-not-listening + - plugin-runtime-timeout + - proxy-env-mismatch diff --git a/skills/skills/langbot-testing/cases/mcp-stdio-tool-call.yaml b/skills/skills/langbot-testing/cases/mcp-stdio-tool-call.yaml new file mode 100644 index 000000000..593eecba3 --- /dev/null +++ b/skills/skills/langbot-testing/cases/mcp-stdio-tool-call.yaml @@ -0,0 +1,88 @@ +id: mcp-stdio-tool-call +title: "MCP stdio server exposes a tool that Local Agent can call" +mode: agent-browser +area: mcp +type: feature +priority: p1 +risk: high +ci_eligible: false +tags: + - mcp + - tools + - local-agent +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation: scripts/e2e/pipeline-debug-chat.mjs +automation_env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BROWSER_PROFILE + - LANGBOT_CHROMIUM_EXECUTABLE + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL +automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation_prompt: "Call the qa_mcp_echo MCP tool with exactly this text: mcp-ok-local-agent. Return only the tool result." +automation_expected_text: "qa_mcp_echo:mcp-ok-local-agent" +automation_response_timeout_ms: "180000" +setup_automation: + - "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env" + - "case:mcp-stdio-register" +setup_provides_env: + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +failure_patterns: + - "qa-plugin-smoke:mcp-ok-local-agent" + - "qa_echo:mcp-ok-local-agent" + - "Agent runner temporarily unavailable" + - "runner.llm_error" + - "model_not_found" + - "no available channel for model" +preconditions: + - "box.local.allowed_mount_roots includes the bundled MCP fixture directory when LangBot runs stdio MCP servers through Box." + - "The selected model route supports function/tool calling." +steps: + - "Open LANGBOT_FRONTEND_URL." + - "Navigate to MCP Servers." + - "Create a Stdio MCP server named qa-local-stdio." + - "Set command to python." + - "Add one argument: the absolute path to skills/langbot-testing/fixtures/mcp/qa_mcp_echo_server.py." + - "Click Test and wait for the test task to finish." + - "Submit the server." + - "Confirm the server detail page shows Tools: 1 and qa_mcp_echo." + - "Open the target local-agent pipeline." + - "Use runner Default or the pluginized langbot/local-agent runner." + - "Select a model with function-calling ability that is known to work with tools in the current environment." + - "Open Debug Chat." + - "Send: Call the qa_mcp_echo tool with exactly this text: mcp-ok-local-agent. Return only the tool result." +checks: + - "UI: The MCP server is connected after submit." + - "UI: The MCP detail page shows qa_mcp_echo." + - "API diagnostic: /api/v1/tools contains qa_mcp_echo." + - "UI: Debug Chat bot response contains qa_mcp_echo:mcp-ok-local-agent." + - "Logs: Backend logs show the MCP tool call was executed, not only listed." + - "Console: No unexpected frontend errors appear during MCP form use or Debug Chat." +evidence_required: + - ui + - console + - backend_log + - api_diagnostic +diagnostics: + - "Run node scripts/e2e/mcp-stdio-fixture.mjs to verify the bundled stdio fixture can list and call qa_mcp_echo without involving a model provider." + - "Run node scripts/e2e/mcp-stdio-register.mjs to upsert qa-local-stdio in LangBot and verify /api/v1/tools exposes qa_mcp_echo." + - "If backend logs show host_path is outside allowed_mount_roots, add the fixture directory to box.local.allowed_mount_roots in the local LangBot data config." + - "If backend logs show uv: not found, refresh qa-local-stdio with command python instead of uv." + - "If /api/v1/tools still shows qa_echo instead of qa_mcp_echo, refresh qa-local-stdio with the register diagnostic or restart the backend." + - "If Debug Chat cannot click Send, close the survey widget or other overlays before retrying." + - "If the model returns model_not_found or no available channel only when tools are provided, switch to a known-good function-calling model before diagnosing MCP or local-agent." +troubleshooting: + - mcp-stdio-args-not-applied + - tool-name-collision-between-mcp-and-plugin + - local-agent-model-route-unavailable + - survey-widget-blocks-debug-chat + - plugin-runtime-timeout diff --git a/skills/skills/langbot-testing/cases/pipeline-debug-chat.yaml b/skills/skills/langbot-testing/cases/pipeline-debug-chat.yaml new file mode 100644 index 000000000..1f0c16f4e --- /dev/null +++ b/skills/skills/langbot-testing/cases/pipeline-debug-chat.yaml @@ -0,0 +1,64 @@ +id: pipeline-debug-chat +title: "Pipeline Debug Chat returns a bot response" +mode: agent-browser +area: pipeline +type: smoke +priority: p0 +risk: medium +ci_eligible: false +tags: + - smoke + - pipeline +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL +env_any: + - LANGBOT_PIPELINE_URL|LANGBOT_PIPELINE_NAME +automation: scripts/e2e/pipeline-debug-chat.mjs +automation_env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BROWSER_PROFILE + - LANGBOT_CHROMIUM_EXECUTABLE + - LANGBOT_E2E_PROMPT + - LANGBOT_E2E_EXPECTED_TEXT +automation_env_any: + - LANGBOT_PIPELINE_URL|LANGBOT_PIPELINE_NAME +automation_prompt: "请只回复 OK，用于前端调试测试。" +automation_expected_text: "OK" +preconditions: + - "LANGBOT_PIPELINE_URL or LANGBOT_PIPELINE_NAME points to the pipeline intended for this generic Debug Chat smoke run." + - "The target pipeline is safe to save or already configured with a known-good runner/model." +steps: + - "Open LANGBOT_FRONTEND_URL." + - "Navigate to Pipelines." + - "Open the target pipeline." + - "Select Debug Chat." + - "Send a short deterministic prompt, such as 请只回复 OK，用于前端调试测试。" +checks: + - "UI: The page shows a User message containing the prompt." + - "UI: The page shows a Bot message containing the expected response, such as OK." + - "Console: No unexpected frontend errors appear during the interaction." + - "Visual: If screenshot/vision is available, the chat panel is not blank, overlapped, or visually broken." + - "Logs: Backend logs include Processing request from person_websocket and Streaming completed." +evidence_required: + - ui + - screenshot + - console + - backend_log +diagnostics: + - "If the UI does not show a Bot response, inspect console/network and backend logs before using API/curl diagnostics." + - "Use API/curl only to distinguish frontend display failure from backend/runtime failure." +success_patterns: + - "Processing request from person_websocket" + - "Streaming completed" +failure_patterns: + - "Action invoke_llm_stream call timed out" + - "Task exception was never retrieved" +troubleshooting: + - debug-chat-history-contaminates-automation + - local-agent-model-route-unavailable + - plugin-runtime-timeout + - proxy-env-mismatch diff --git a/skills/skills/langbot-testing/cases/plugin-e2e-smoke.yaml b/skills/skills/langbot-testing/cases/plugin-e2e-smoke.yaml new file mode 100644 index 000000000..87dd27e3f --- /dev/null +++ b/skills/skills/langbot-testing/cases/plugin-e2e-smoke.yaml @@ -0,0 +1,57 @@ +id: plugin-e2e-smoke +title: "Plugin system installs a local plugin and exposes tool/page APIs" +mode: agent-browser +area: plugin +type: smoke +priority: p1 +risk: medium +ci_eligible: false +tags: + - plugin + - runtime + - local-install + - e2e +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_REPO +preconditions: + - "LangBot is started from the target worktree with the SDK under test installed." + - "The active instance is a local test instance where installing or updating the qa-plugin-smoke fixture is acceptable." +steps: + - "Ensure LangBot was started from the target worktree with the SDK under test installed, and use uv run --no-sync so uv does not resync langbot-plugin from the lockfile." + - "Build the fixture plugin at skills/langbot-testing/fixtures/plugins/qa-plugin-smoke using the same local SDK under test." + - "Open LANGBOT_FRONTEND_URL." + - "Log in or initialize a local test account if the clean test instance is not initialized." + - "Navigate to Plugins." + - "Install a local plugin package using the generated qa-plugin-smoke zip file." + - "Wait for the plugin installation task to finish." + - "Confirm the installed plugin list shows QA Plugin Smoke with initialized or running status." + - "Open the plugin detail and confirm the qa_echo Tool, qa_plugin_echo Tool, Prompt Probe EventListener, and Smoke Page components are visible." + - "Open the Smoke Page from the plugin extension page entry if the UI exposes it." + - "Use the page or API diagnostic to call endpoint /ping and confirm the response sentinel qa-plugin-smoke-page." +checks: + - "UI: The plugin installation task finishes successfully and the plugin card appears." + - "UI: Plugin detail shows qa_echo, qa_plugin_echo, Prompt Probe, and Smoke Page components." + - "UI: The plugin extension page renders without a blank iframe or runtime error when reachable from the sidebar." + - "API diagnostic: /api/v1/plugins contains qa/plugin-smoke with status initialized." + - "API diagnostic: /api/v1/tools contains qa_echo and qa_plugin_echo from qa/plugin-smoke." + - "API diagnostic: /api/v1/plugins/qa/plugin-smoke/page-api with page_id smoke and endpoint /ping returns qa-plugin-smoke-page." + - "Logs: Backend logs show Connected to plugin runtime and no plugin action timeout during install/list/page-api." + - "Console: No unexpected frontend errors appear during plugin install or detail navigation." +evidence_required: + - ui + - console + - backend_log + - api_diagnostic +diagnostics: + - "Before trusting results, confirm Python imports langbot_plugin from the local SDK source path, not site-packages from PyPI." + - "If uv run changes the installed SDK back to the lockfile package, reinstall the local SDK and rerun commands with uv run --no-sync." + - "If local install stalls, inspect /api/v1/system/tasks/ and backend logs." +troubleshooting: + - plugin-runtime-timeout + - marketplace-network-flaky + - uv-run-resyncs-local-sdk diff --git a/skills/skills/langbot-testing/cases/provider-deepseek.yaml b/skills/skills/langbot-testing/cases/provider-deepseek.yaml new file mode 100644 index 000000000..eeb0e7d8e --- /dev/null +++ b/skills/skills/langbot-testing/cases/provider-deepseek.yaml @@ -0,0 +1,42 @@ +id: provider-deepseek +title: "DeepSeek provider can be configured and used" +mode: agent-browser +area: provider +type: provider +priority: p2 +risk: medium +ci_eligible: false +tags: + - provider + - model +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL +preconditions: + - "A DeepSeek-compatible API key and model route are available from the active secret source and must not be printed in reports." + - "A test pipeline is available for confirming the saved provider can answer Debug Chat." +steps: + - "Open LANGBOT_FRONTEND_URL." + - "Navigate to Models." + - "Add or edit a DeepSeek provider." + - "Fill the required base URL, API key, and model fields from the active secret source." + - "Run the provider or model test action in the UI." + - "Run a small Pipeline Debug Chat prompt to confirm the provider is usable." +checks: + - "UI: The provider/model test action succeeds in the page." + - "UI: A Pipeline Debug Chat prompt returns a Bot response after provider setup." + - "Console: No unexpected frontend errors appear while saving or testing the provider." + - "Secret safety: No API key, token, or secret is printed in logs or reports." +evidence_required: + - ui + - console + - backend_log +diagnostics: + - "If provider testing fails, inspect backend logs and provider error messages to distinguish invalid credentials from UI wiring issues." + - "Use API/curl only as a diagnostic aid after reproducing through the UI." +troubleshooting: + - proxy-env-mismatch + - plugin-runtime-timeout diff --git a/skills/skills/langbot-testing/cases/qa-plugin-smoke-live-install.yaml b/skills/skills/langbot-testing/cases/qa-plugin-smoke-live-install.yaml new file mode 100644 index 000000000..f0ec3c336 --- /dev/null +++ b/skills/skills/langbot-testing/cases/qa-plugin-smoke-live-install.yaml @@ -0,0 +1,44 @@ +id: qa-plugin-smoke-live-install +title: "QA plugin smoke package installs and exposes tools" +mode: probe +area: plugin +type: regression +priority: p1 +risk: medium +ci_eligible: false +tags: + - plugin + - local-install + - fixture +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_BACKEND_URL + - LANGBOT_REPO + - LANGBOT_E2E_LOGIN_USER +automation: scripts/e2e/install-qa-plugin-smoke.mjs +automation_plugin_package: "skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/dist/qa-plugin-smoke-0.1.0.lbpkg" +automation_expected_plugin_id: "qa/plugin-smoke" +automation_expected_tool: "qa_plugin_echo" +steps: + - "Run `rtk bin/lbs test run qa-plugin-smoke-live-install --dry-run` first; remove `--dry-run` only after readiness points at a local test LangBot instance." + - "Automation authenticates the local test user, uploads the QA plugin smoke .lbpkg package when missing, waits for install, and checks tool exposure." +checks: + - "automation-result.json status is pass." + - "/api/v1/plugins lists qa/plugin-smoke after install." + - "/api/v1/tools lists qa_plugin_echo after install." +evidence_required: + - api_diagnostic + - filesystem +diagnostics: + - "This prepares prompt/tool/steering cases that depend on qa-plugin-smoke. It does not prove the model can call the tools." + - "If install fails during dependencies, inspect plugin runtime logs and plugin-dependency-install-offline." +success_patterns: + - "qa/plugin-smoke is installed." +failure_patterns: + - "Plugin install task did not complete successfully" + - "qa_plugin_echo is not listed" +troubleshooting: + - plugin-runtime-timeout + - plugin-dependency-install-offline diff --git a/skills/skills/langbot-testing/cases/sandbox-skill-authoring-e2e.yaml b/skills/skills/langbot-testing/cases/sandbox-skill-authoring-e2e.yaml new file mode 100644 index 000000000..91bb97fd3 --- /dev/null +++ b/skills/skills/langbot-testing/cases/sandbox-skill-authoring-e2e.yaml @@ -0,0 +1,54 @@ +id: sandbox-skill-authoring-e2e +title: "Local Agent creates, registers, activates, and uses a sandbox skill" +mode: agent-browser +area: sandbox +type: regression +priority: p2 +risk: high +ci_eligible: false +tags: + - sandbox + - skills + - local-agent + - tools + - e2b + - nsjail +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +preconditions: + - "LANGBOT_LOCAL_AGENT_PIPELINE_URL or LANGBOT_LOCAL_AGENT_PIPELINE_NAME points to the local-agent pipeline under test." + - "LangBot is started with the sandbox backend intended for this run, such as e2b or nsjail." + - "The selected model route supports tool/function calling strongly enough to invoke sandbox tools." +steps: + - "Start LangBot with the target sandbox backend and confirm the Box status UI or LANGBOT_BACKEND_URL /api/v1/box/status reports the expected backend." + - "Open the target Local Agent pipeline in Debug Chat with a function-calling model." + - "Run the canonical prompt pattern in references/sandbox-skill-authoring.md with a unique skill name." + - "Capture the visible final response, backend logs, and the registered skill-store files." +checks: + - "UI: Debug Chat final assistant response contains E2E_OK:." + - "Logs: The model called exec, register_skill, activate, then exec again from the activated skill path." + - "Logs: The selected backend name is the expected one, such as e2b or nsjail." + - "Skill store: The registered package and activated writeback match references/sandbox-skill-authoring.md." + - "Box status: recent_error_count is 0 after the run." +evidence_required: + - ui + - backend_log + - api_diagnostic + - filesystem +diagnostics: + - "If Debug Chat fails before tool execution, first validate the model provider and function-calling support." + - "If /workspace/.skills/ is missing only on E2B, check extra mount sync behavior." + - "If nsjail starts but every command fails before execve, check nsjail CLI compatibility and chroot mount targets." + - "API or raw provider probes are diagnostic only; they do not make this UI case pass by themselves." +troubleshooting: + - sandbox-native-tools-unavailable + - e2b-extra-mount-sync-missing + - box-session-conflict-logical-metadata + - nsjail-cli-compatibility + - socks-proxy-without-socksio diff --git a/skills/skills/langbot-testing/cases/sandbox-skill-authoring-edit-existing-e2e.yaml b/skills/skills/langbot-testing/cases/sandbox-skill-authoring-edit-existing-e2e.yaml new file mode 100644 index 000000000..af4c383a3 --- /dev/null +++ b/skills/skills/langbot-testing/cases/sandbox-skill-authoring-edit-existing-e2e.yaml @@ -0,0 +1,71 @@ +id: sandbox-skill-authoring-edit-existing-e2e +title: "Local Agent modifies an activated sandbox skill package" +mode: agent-browser +area: sandbox +type: regression +priority: p2 +risk: high +ci_eligible: false +tags: + - sandbox + - skills + - local-agent + - tools + - edit +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_REPO + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation: scripts/e2e/pipeline-debug-chat.mjs +automation_env: + - LANGBOT_REPO + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_BROWSER_PROFILE + - LANGBOT_CHROMIUM_EXECUTABLE + - LANGBOT_LOCAL_AGENT_PIPELINE_URL + - LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL +automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME +automation_expected_text: "UPDATED_MARKER_EDIT_REGRESSION" +automation_prompts_json: '[{"prompt":"Skill name is lb-skill-edit-regression. Use exactly one exec tool call with workdir=/workspace. In that single command, remove old /workspace/lb-skill-edit-regression, create SKILL.md, scripts/use.py, and data/input.json. The JSON must contain numbers [1,2,3,4], factors [2,3,4], and marker CREATE_MARKER_EDIT_REGRESSION. The Python script must read data/input.json relative to __file__ and print exactly SANDBOX_SKILL_CREATE_OK sum=10 product=24 marker=CREATE_MARKER_EDIT_REGRESSION. Run python3 /workspace/lb-skill-edit-regression/scripts/use.py in the same command. After the exec succeeds, final answer exactly CREATE_OK:lb-skill-edit-regression.","expected_text":"CREATE_MARKER_EDIT_REGRESSION","response_timeout_ms":"180000"},{"prompt":"Continue with the same skill name lb-skill-edit-regression. Strictly call register_skill with path=/workspace/lb-skill-edit-regression and name=lb-skill-edit-regression, then call activate with skill_name=lb-skill-edit-regression. Do not call exec, read, write, edit, grep, or ls in this step. After both tools succeed, final answer exactly REGISTER_ACTIVATE_OK:lb-skill-edit-regression.","expected_text":"REGISTER_ACTIVATE_OK:lb-skill-edit-regression","response_timeout_ms":"180000"},{"prompt":"Continue with the same activated skill lb-skill-edit-regression. Use exactly one exec tool call with workdir=/workspace/.skills/lb-skill-edit-regression. In that single exec command, run exactly this shell logic: set -e; overwrite SKILL.md with a heading and UPDATED_MARKER_EDIT_REGRESSION; overwrite data/input.json with numbers [5,6], factors [7,8], and marker UPDATED_MARKER_EDIT_REGRESSION; overwrite scripts/use.py so it reads data/input.json relative to __file__ and prints exactly SANDBOX_SKILL_MODIFIED_OK sum=11 product=56 marker=UPDATED_MARKER_EDIT_REGRESSION; run python3 scripts/use.py. Do not write exit(1). Do not call find, ls, read, grep as separate tools. If the single exec exits successfully and prints SANDBOX_SKILL_MODIFIED_OK, do not call more tools. Final answer exactly E2E_OK:lb-skill-edit-regression:SANDBOX_SKILL_MODIFIED_OK.","expected_text":"UPDATED_MARKER_EDIT_REGRESSION","response_timeout_ms":"240000"}]' +automation_filesystem_checks_json: '[{"path":"${LANGBOT_REPO}/data/box/skills/lb-skill-edit-regression/SKILL.md","contains":"UPDATED_MARKER_EDIT_REGRESSION"},{"path":"${LANGBOT_REPO}/data/box/skills/lb-skill-edit-regression/data/input.json","contains":["UPDATED_MARKER_EDIT_REGRESSION","\"numbers\": [5, 6]","\"factors\": [7, 8]"]},{"path":"${LANGBOT_REPO}/data/box/skills/lb-skill-edit-regression/scripts/use.py","contains":["SANDBOX_SKILL_MODIFIED_OK"],"not_contains":["SANDBOX_SKILL_CREATE_OK","exit(1)"]},{"argv":["python3","scripts/use.py"],"cwd":"${LANGBOT_REPO}/data/box/skills/lb-skill-edit-regression","stdout_contains":"SANDBOX_SKILL_MODIFIED_OK sum=11 product=56 marker=UPDATED_MARKER_EDIT_REGRESSION","exit_code":0}]' +automation_stream_output: "0" +automation_reset_debug_chat: "1" +automation_response_timeout_ms: "420000" +preconditions: + - "LANGBOT_LOCAL_AGENT_PIPELINE_URL or LANGBOT_LOCAL_AGENT_PIPELINE_NAME points to the local-agent pipeline under test." + - "LangBot is started with the sandbox backend intended for this run, such as e2b or nsjail." + - "The selected model route supports tool/function calling strongly enough to invoke sandbox tools." +steps: + - "Start LangBot with the target sandbox backend and confirm native sandbox tools are available." + - "Open the target Local Agent pipeline in Debug Chat with a function-calling model." + - "Run the automation prompt or the equivalent prompt pattern from references/sandbox-skill-authoring.md." + - "Capture the visible final response, backend logs, and the registered skill-store files." +checks: + - "UI: A Bot message, not only the user prompt, contains UPDATED_MARKER_EDIT_REGRESSION." + - "Logs: The model called exec, register_skill, activate, then a second exec whose workdir is /workspace/.skills/lb-skill-edit-regression." + - "Logs: The second exec stdout contains SANDBOX_SKILL_MODIFIED_OK and GREP_ALL_OK or equivalent grep success." + - "Automation: Filesystem checks pass for the registered skill-store files under LANGBOT_REPO/data/box/skills/lb-skill-edit-regression." + - "Skill store: SKILL.md and data/input.json contain UPDATED_MARKER_EDIT_REGRESSION." + - "Skill store: scripts/use.py is the modified script and prints SANDBOX_SKILL_MODIFIED_OK sum=11 product=56 marker=UPDATED_MARKER_EDIT_REGRESSION." +evidence_required: + - ui + - backend_log + - api_diagnostic + - filesystem +diagnostics: + - "If the model stops after activation and only reruns the original script, treat this as a failed edit-existing E2E even if create/register/activate succeeded." + - "If the expected marker appears only in the user prompt, treat the run as failed and inspect automation-result.json assistant counts." + - "If the browser opens /login or console shows 401, refresh the authenticated browser profile before diagnosing the runner." +troubleshooting: + - sandbox-native-tools-unavailable + - e2b-extra-mount-sync-missing + - box-session-conflict-logical-metadata + - nsjail-cli-compatibility + - socks-proxy-without-socksio diff --git a/skills/skills/langbot-testing/cases/webui-login-state.yaml b/skills/skills/langbot-testing/cases/webui-login-state.yaml new file mode 100644 index 000000000..50119b156 --- /dev/null +++ b/skills/skills/langbot-testing/cases/webui-login-state.yaml @@ -0,0 +1,41 @@ +id: webui-login-state +title: "Configured frontend opens with authenticated LangBot WebUI state" +mode: agent-browser +area: auth +type: smoke +priority: p0 +risk: low +ci_eligible: false +tags: + - smoke + - auth +skills: + - langbot-env-setup + - langbot-testing +env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BACKEND_URL + - LANGBOT_BROWSER_PROFILE +automation: scripts/e2e/webui-login-state.mjs +automation_env: + - LANGBOT_FRONTEND_URL + - LANGBOT_BROWSER_PROFILE + - LANGBOT_CHROMIUM_EXECUTABLE +steps: + - "Open LANGBOT_FRONTEND_URL with the configured browser-control path." + - "Confirm the page is not stuck on /login." + - "Check that the sidebar or dashboard shows a logged-in user." + - "If login is missing, use langbot-env-setup OAuth browser profile guidance." +checks: + - "UI: The WebUI shows LangBot navigation such as Dashboard, Bots, Pipelines, or Knowledge." + - "UI: The page is not stuck on /login after the configured profile is loaded." + - "Console: No unexpected frontend errors appear during initial load." + - "Secret safety: The agent does not print token values while checking auth state." +evidence_required: + - ui + - screenshot + - console +diagnostics: + - "If the page stays on /login, inspect only localStorage key names and origin mismatch; do not print token values." +troubleshooting: + - proxy-env-mismatch diff --git a/skills/skills/langbot-testing/fixtures/agent-runner/qa-runner-behaviors.json b/skills/skills/langbot-testing/fixtures/agent-runner/qa-runner-behaviors.json new file mode 100644 index 000000000..b2f153321 --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/agent-runner/qa-runner-behaviors.json @@ -0,0 +1,38 @@ +{ + "id": "qa-agent-runner-behaviors", + "behaviors": [ + { + "name": "ok", + "results": [ + {"type": "message.completed", "data": {"message": {"role": "assistant", "content": "QA_RUNNER_OK"}}}, + {"type": "run.completed", "data": {"finish_reason": "stop"}} + ] + }, + { + "name": "stream_ok", + "results": [ + {"type": "message.delta", "data": {"chunk": {"role": "assistant", "content": "QA_"}}}, + {"type": "message.delta", "data": {"chunk": {"role": "assistant", "content": "RUNNER_STREAM_OK"}}}, + {"type": "run.completed", "data": {"finish_reason": "stop"}} + ] + }, + { + "name": "empty_output", + "results": [ + {"type": "run.completed", "data": {"finish_reason": "stop"}} + ] + }, + { + "name": "malformed_result", + "results": [ + {"type": "message.completed", "data": {}} + ] + }, + { + "name": "controlled_failure", + "results": [ + {"type": "run.failed", "data": {"error": "QA runner controlled failure", "code": "qa.failure", "retryable": false}} + ] + } + ] +} diff --git a/skills/skills/langbot-testing/fixtures/fixtures.json b/skills/skills/langbot-testing/fixtures/fixtures.json new file mode 100644 index 000000000..70badac14 --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/fixtures.json @@ -0,0 +1,93 @@ +[ + { + "id": "qa-agent-runner-behaviors", + "title": "Deterministic AgentRunner behavior matrix", + "kind": "json", + "path": "fixtures/agent-runner/qa-runner-behaviors.json", + "related_cases": [ + "agent-runner-behavior-matrix", + "agent-runner-ledger-invariants", + "agent-runner-runtime-chaos" + ], + "checks": ["exists"] + }, + { + "id": "qa-agent-runner-source", + "title": "QA deterministic AgentRunner fixture source", + "kind": "plugin_source", + "path": "fixtures/plugins/qa-agent-runner/manifest.yaml", + "related_cases": [ + "agent-runner-fixture-contract", + "agent-runner-behavior-matrix", + "agent-runner-live-install", + "agent-runner-qa-debug-chat" + ], + "checks": ["exists", "qa_agent_runner_source"] + }, + { + "id": "qa-agent-runner-package", + "title": "QA deterministic AgentRunner prebuilt package", + "kind": "plugin_package", + "path": "fixtures/plugins/qa-agent-runner/dist/qa-agent-runner-0.1.0.lbpkg", + "related_cases": [ + "agent-runner-fixture-contract", + "agent-runner-live-install", + "agent-runner-qa-debug-chat" + ], + "checks": ["exists", "zip_package"] + }, + { + "id": "mcp-stdio-echo-server", + "title": "MCP stdio qa_mcp_echo server", + "kind": "python", + "path": "fixtures/mcp/qa_mcp_echo_server.py", + "related_cases": ["mcp-stdio-tool-call"], + "checks": ["exists", "direct_mcp_script", "langbot_registration_script"] + }, + { + "id": "rag-sentinel-doc", + "title": "LangRAG sentinel text document", + "kind": "text", + "path": "fixtures/rag/sentinel-doc.txt", + "related_cases": ["langrag-kb-retrieve", "local-agent-rag-debug-chat"], + "checks": ["exists"] + }, + { + "id": "rag-parser-golden-html", + "title": "LangRAG parser golden HTML document", + "kind": "html", + "path": "fixtures/rag/parser-golden.html", + "related_cases": ["langrag-parser-golden-e2e"], + "checks": ["exists"] + }, + { + "id": "multimodal-red-square", + "title": "64x64 red-square image fixture", + "kind": "base64_png", + "path": "fixtures/multimodal/red-square.png.base64", + "related_cases": ["local-agent-multimodal-debug-chat", "local-agent-rag-multimodal-debug-chat"], + "checks": ["exists"] + }, + { + "id": "qa-plugin-smoke-source", + "title": "QA plugin smoke fixture source", + "kind": "plugin_source", + "path": "fixtures/plugins/qa-plugin-smoke/manifest.yaml", + "related_cases": [ + "qa-plugin-smoke-live-install", + "plugin-e2e-smoke", + "local-agent-effective-prompt-debug-chat", + "local-agent-plugin-tool-call-debug-chat", + "local-agent-steering-debug-chat" + ], + "checks": ["exists"] + }, + { + "id": "qa-plugin-smoke-package", + "title": "QA plugin smoke prebuilt package", + "kind": "plugin_package", + "path": "fixtures/plugins/qa-plugin-smoke/dist/qa-plugin-smoke-0.1.0.lbpkg", + "related_cases": ["qa-plugin-smoke-live-install", "plugin-e2e-smoke"], + "checks": ["exists", "zip_package"] + } +] diff --git a/skills/skills/langbot-testing/fixtures/mcp/qa_mcp_echo_server.py b/skills/skills/langbot-testing/fixtures/mcp/qa_mcp_echo_server.py new file mode 100644 index 000000000..44e04b8e8 --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/mcp/qa_mcp_echo_server.py @@ -0,0 +1,123 @@ +from __future__ import annotations + +import json +import sys +import typing + +SERVER_INFO = {"name": "langbot-qa-stdio", "version": "0.1.0"} +TOOL_NAME = "qa_mcp_echo" + + +def _write_message(message: dict[str, typing.Any]) -> None: + sys.stdout.write( + json.dumps(message, ensure_ascii=False, separators=(",", ":")) + "\n" + ) + sys.stdout.flush() + + +def _result( + message_id: typing.Any, result: dict[str, typing.Any] +) -> dict[str, typing.Any]: + return {"jsonrpc": "2.0", "id": message_id, "result": result} + + +def _error(message_id: typing.Any, code: int, message: str) -> dict[str, typing.Any]: + return { + "jsonrpc": "2.0", + "id": message_id, + "error": { + "code": code, + "message": message, + }, + } + + +def _tool_schema() -> dict[str, typing.Any]: + return { + "name": TOOL_NAME, + "description": "Return a deterministic QA echo string.", + "inputSchema": { + "type": "object", + "properties": { + "text": { + "type": "string", + "description": "Text to include in the QA echo response.", + }, + }, + "required": ["text"], + "additionalProperties": False, + }, + } + + +def handle_message(message: dict[str, typing.Any]) -> dict[str, typing.Any] | None: + message_id = message.get("id") + method = str(message.get("method") or "") + params = message.get("params") or {} + if not isinstance(params, dict): + params = {} + + if message_id is None: + return None + + if method == "initialize": + return _result( + message_id, + { + "protocolVersion": str(params.get("protocolVersion") or "2024-11-05"), + "capabilities": {"tools": {"listChanged": False}}, + "serverInfo": SERVER_INFO, + }, + ) + + if method == "ping": + return _result(message_id, {}) + + if method == "tools/list": + return _result(message_id, {"tools": [_tool_schema()]}) + + if method == "tools/call": + name = str(params.get("name") or "") + arguments = params.get("arguments") or {} + if not isinstance(arguments, dict): + arguments = {} + if name != TOOL_NAME: + return _error(message_id, -32602, f"Unknown tool: {name}") + text = str(arguments.get("text") or "") + return _result( + message_id, + { + "content": [ + { + "type": "text", + "text": f"{TOOL_NAME}:{text}", + } + ], + "isError": False, + }, + ) + + return _error(message_id, -32601, f"Method not found: {method}") + + +def main() -> int: + for raw_line in sys.stdin: + line = raw_line.strip() + if not line: + continue + try: + message = json.loads(line) + except json.JSONDecodeError as exc: + _write_message(_error(None, -32700, f"Parse error: {exc}")) + continue + if not isinstance(message, dict): + _write_message(_error(None, -32600, "Invalid request")) + continue + response = handle_message(message) + if response is not None: + _write_message(response) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/skills/skills/langbot-testing/fixtures/multimodal/red-pixel.png.base64 b/skills/skills/langbot-testing/fixtures/multimodal/red-pixel.png.base64 new file mode 100644 index 000000000..c8e6472e9 --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/multimodal/red-pixel.png.base64 @@ -0,0 +1 @@ +iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAADUlEQVR4nGP4z8DwHwAFAAH/e+m+7wAAAABJRU5ErkJggg== diff --git a/skills/skills/langbot-testing/fixtures/multimodal/red-square.png.base64 b/skills/skills/langbot-testing/fixtures/multimodal/red-square.png.base64 new file mode 100644 index 000000000..808a8341a --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/multimodal/red-square.png.base64 @@ -0,0 +1 @@ +iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAIAAAAlC+aJAAAAT0lEQVR42u3PQQkAAAgEsIty/TMZxgi+hcEKLNO+FgEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQGBywIJs8EAKp/R7QAAAABJRU5ErkJggg== diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-agent-runner/README.md b/skills/skills/langbot-testing/fixtures/plugins/qa-agent-runner/README.md new file mode 100644 index 000000000..cbde4cc52 --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-agent-runner/README.md @@ -0,0 +1,15 @@ +# QA AgentRunner Fixture + +Deterministic AgentRunner plugin source used by `langbot-skills` probes and future browser release-gate cases. + +Runner id after installation should be: + +```text +plugin:qa/agent-runner/default +``` + +Expected behavior: + +- normal input returns `QA_AGENT_RUNNER_OK:` +- input containing `stream` emits streaming chunks then completes +- input containing `fail` returns `QA_AGENT_RUNNER_CONTROLLED_FAILURE` diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-agent-runner/assets/icon.svg b/skills/skills/langbot-testing/fixtures/plugins/qa-agent-runner/assets/icon.svg new file mode 100644 index 000000000..2a0e899bb --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-agent-runner/assets/icon.svg @@ -0,0 +1,5 @@ + + + + + diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-agent-runner/components/agent_runner/default.py b/skills/skills/langbot-testing/fixtures/plugins/qa-agent-runner/components/agent_runner/default.py new file mode 100644 index 000000000..7d2fd37dc --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-agent-runner/components/agent_runner/default.py @@ -0,0 +1,39 @@ +from __future__ import annotations + +import typing + +from langbot_plugin.api.definition.components.agent_runner.runner import AgentRunner +from langbot_plugin.api.entities.builtin.agent_runner import AgentRunContext, AgentRunResult +from langbot_plugin.api.entities.builtin.provider.message import Message, MessageChunk + + +class DefaultAgentRunner(AgentRunner): + async def run( + self, + ctx: AgentRunContext, + ) -> typing.AsyncGenerator[AgentRunResult, None]: + text = (ctx.input.to_text() or "").strip() + if "fail" in text.lower(): + yield AgentRunResult.run_failed( + ctx.run_id, + error="QA_AGENT_RUNNER_CONTROLLED_FAILURE", + code="qa.controlled_failure", + retryable=False, + ) + return + + content = f"QA_AGENT_RUNNER_OK:{text or 'empty'}" + if "stream" in text.lower(): + for chunk in ("QA_", "AGENT_", f"RUNNER_OK:{text}"): + yield AgentRunResult.message_delta( + ctx.run_id, + MessageChunk(role="assistant", content=chunk), + ) + yield AgentRunResult.run_completed(ctx.run_id, finish_reason="stop") + return + + yield AgentRunResult.run_completed( + ctx.run_id, + Message(role="assistant", content=content), + finish_reason="stop", + ) diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-agent-runner/components/agent_runner/default.yaml b/skills/skills/langbot-testing/fixtures/plugins/qa-agent-runner/components/agent_runner/default.yaml new file mode 100644 index 000000000..a2fd64d98 --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-agent-runner/components/agent_runner/default.yaml @@ -0,0 +1,30 @@ +apiVersion: langbot/v1 +kind: AgentRunner +metadata: + name: default + label: + en_US: QA Deterministic Runner + zh_Hans: QA 确定性 Runner + description: + en_US: Deterministic runner fixture that returns stable QA sentinel output. + zh_Hans: 返回稳定 QA 哨兵输出的确定性 runner 夹具。 +spec: + capabilities: + streaming: true + tool_calling: false + knowledge_retrieval: false + multimodal_input: false + skill_authoring: false + interrupt: false + permissions: + models: [] + tools: [] + knowledge_bases: [] + history: [] + events: [] + storage: [] + config: [] +execution: + python: + path: default.py + attr: DefaultAgentRunner diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-agent-runner/dist/qa-agent-runner-0.1.0.lbpkg b/skills/skills/langbot-testing/fixtures/plugins/qa-agent-runner/dist/qa-agent-runner-0.1.0.lbpkg new file mode 100644 index 000000000..aba6ead2d Binary files /dev/null and b/skills/skills/langbot-testing/fixtures/plugins/qa-agent-runner/dist/qa-agent-runner-0.1.0.lbpkg differ diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-agent-runner/main.py b/skills/skills/langbot-testing/fixtures/plugins/qa-agent-runner/main.py new file mode 100644 index 000000000..6e1c392d2 --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-agent-runner/main.py @@ -0,0 +1,8 @@ +from __future__ import annotations + +from langbot_plugin.api.definition.plugin import BasePlugin + + +class QAAgentRunnerPlugin(BasePlugin): + async def initialize(self) -> None: + self.ready_marker = "qa-agent-runner-ready" diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-agent-runner/manifest.yaml b/skills/skills/langbot-testing/fixtures/plugins/qa-agent-runner/manifest.yaml new file mode 100644 index 000000000..a7fd08cac --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-agent-runner/manifest.yaml @@ -0,0 +1,25 @@ +apiVersion: langbot/v1 +kind: Plugin +metadata: + author: qa + name: agent-runner + repository: https://example.invalid/langbot/qa-agent-runner + version: 0.1.0 + description: + en_US: Deterministic AgentRunner fixture for LangBot QA. + zh_Hans: LangBot QA 使用的确定性 AgentRunner 夹具。 + label: + en_US: QA AgentRunner + zh_Hans: QA AgentRunner + icon: assets/icon.svg +spec: + config: [] + components: + AgentRunner: + fromDirs: + - path: components/agent_runner/ + maxDepth: 1 +execution: + python: + path: main.py + attr: QAAgentRunnerPlugin diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/.gitignore b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/.gitignore new file mode 100644 index 000000000..849ddff3b --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/.gitignore @@ -0,0 +1 @@ +dist/ diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/README.md b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/README.md new file mode 100644 index 000000000..e276f5774 --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/README.md @@ -0,0 +1,9 @@ +# QA Plugin Smoke + +Local fixture plugin for LangBot plugin E2E smoke testing. + +Tools: + +- `qa_echo(text)` returns `qa-plugin-smoke:`. +- `qa_plugin_echo(text)` returns `qa-plugin-smoke:`. +- `qa_plugin_sleep(seconds, text)` waits up to 15 seconds and returns `qa-plugin-smoke:sleep::`. diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/assets/icon.svg b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/assets/icon.svg new file mode 100644 index 000000000..440e9c82c --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/assets/icon.svg @@ -0,0 +1,6 @@ + + + + + + diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/events/prompt_probe.py b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/events/prompt_probe.py new file mode 100644 index 000000000..c2b06d0e3 --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/events/prompt_probe.py @@ -0,0 +1,79 @@ +from __future__ import annotations + +from typing import Any + +from langbot_plugin.api.definition.components.common.event_listener import EventListener +from langbot_plugin.api.entities import context, events +from langbot_plugin.api.entities.builtin.provider.message import Message + + +def _content_text(content: Any) -> str: + if isinstance(content, str): + return content + if not isinstance(content, list): + return "" + + parts: list[str] = [] + for item in content: + text = item.get("text") if isinstance(item, dict) else getattr(item, "text", None) + if text: + parts.append(str(text)) + return "".join(parts) + + +def _message_text(message: Any) -> str: + if message is None: + return "" + return _content_text(getattr(message, "content", None)) + + +def _message_chain_text(message_chain: Any) -> str: + if message_chain is None: + return "" + try: + return str(message_chain) + except Exception: + return "" + + +async def _current_user_text(event_context: context.EventContext) -> str: + try: + query_var_text = await event_context.get_query_var("user_message_text") + except Exception: + query_var_text = None + if query_var_text: + return str(query_var_text) + + query = getattr(event_context.event, "query", None) + if query is not None: + message_chain_text = _message_chain_text(getattr(query, "message_chain", None)) + if message_chain_text: + return message_chain_text + + user_message_text = _message_text(getattr(query, "user_message", None)) + if user_message_text: + return user_message_text + + return "\n".join( + text for text in (_message_text(message) for message in event_context.event.prompt) if text + ) + + +class PromptProbeEventListener(EventListener): + async def initialize(self) -> None: + await super().initialize() + + @self.handler(events.PromptPreProcessing) + async def on_prompt_pre_processing(event_context: context.EventContext) -> None: + if "qa-effective-prompt" not in await _current_user_text(event_context): + return + + event_context.event.default_prompt.append( + Message( + role="system", + content=( + "QA prompt probe: if the current user message contains " + "qa-effective-prompt, reply only PROMPT_PREPROCESS_OK." + ), + ) + ) diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/events/prompt_probe.yaml b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/events/prompt_probe.yaml new file mode 100644 index 000000000..1e1cf0f9b --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/events/prompt_probe.yaml @@ -0,0 +1,12 @@ +apiVersion: v1 +kind: EventListener +metadata: + name: prompt_probe + label: + en_US: Prompt Probe + zh_Hans: Prompt 探针 +spec: +execution: + python: + path: prompt_probe.py + attr: PromptProbeEventListener diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/pages/smoke/i18n/en_US.json b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/pages/smoke/i18n/en_US.json new file mode 100644 index 000000000..0967ef424 --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/pages/smoke/i18n/en_US.json @@ -0,0 +1 @@ +{} diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/pages/smoke/i18n/zh_Hans.json b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/pages/smoke/i18n/zh_Hans.json new file mode 100644 index 000000000..0967ef424 --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/pages/smoke/i18n/zh_Hans.json @@ -0,0 +1 @@ +{} diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/pages/smoke/index.html b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/pages/smoke/index.html new file mode 100644 index 000000000..f7a9ad289 --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/pages/smoke/index.html @@ -0,0 +1,43 @@ + + + + + + Smoke Page + + + +
+

Smoke Page

+

qa-plugin-smoke-page

+

waiting

+
+ + + + diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/pages/smoke/smoke.py b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/pages/smoke/smoke.py new file mode 100644 index 000000000..4e6d2625e --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/pages/smoke/smoke.py @@ -0,0 +1,15 @@ +from __future__ import annotations + +from langbot_plugin.api.definition.components.page import Page, PageRequest, PageResponse + + +class SmokePage(Page): + async def handle_api(self, request: PageRequest) -> PageResponse: + return PageResponse.ok( + { + "sentinel": "qa-plugin-smoke-page", + "endpoint": request.endpoint, + "method": request.method, + "body": request.body, + } + ) diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/pages/smoke/smoke.yaml b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/pages/smoke/smoke.yaml new file mode 100644 index 000000000..83c9eb772 --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/pages/smoke/smoke.yaml @@ -0,0 +1,16 @@ +apiVersion: v1 +kind: Page +metadata: + name: smoke + label: + en_US: Smoke Page + zh_Hans: 冒烟测试页 + description: + en_US: Page component for plugin e2e smoke testing. + zh_Hans: 插件端到端冒烟测试页面组件。 +spec: + path: index.html +execution: + python: + path: smoke.py + attr: SmokePage diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/tools/qa_echo.py b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/tools/qa_echo.py new file mode 100644 index 000000000..69e0fb2d1 --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/tools/qa_echo.py @@ -0,0 +1,17 @@ +from __future__ import annotations + +from typing import Any + +from langbot_plugin.api.definition.components.tool.tool import Tool +from langbot_plugin.api.entities.builtin.provider import session as provider_session + + +class QAEchoTool(Tool): + async def call( + self, + params: dict[str, Any], + session: provider_session.Session, + query_id: int, + ) -> str: + text = str(params.get("text", "")) + return f"qa-plugin-smoke:{text}" diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/tools/qa_echo.yaml b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/tools/qa_echo.yaml new file mode 100644 index 000000000..277594743 --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/tools/qa_echo.yaml @@ -0,0 +1,24 @@ +apiVersion: v1 +kind: Tool +metadata: + name: qa_echo + label: + en_US: QA Echo + zh_Hans: QA 回显 + description: + en_US: Echoes deterministic text for plugin smoke testing. + zh_Hans: 为插件冒烟测试回显确定性文本。 +spec: + parameters: + type: object + properties: + text: + type: string + description: Text to echo. + required: + - text + llm_prompt: Echo deterministic text for plugin smoke testing. +execution: + python: + path: qa_echo.py + attr: QAEchoTool diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/tools/qa_plugin_echo.py b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/tools/qa_plugin_echo.py new file mode 100644 index 000000000..343218046 --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/tools/qa_plugin_echo.py @@ -0,0 +1,17 @@ +from __future__ import annotations + +from typing import Any + +from langbot_plugin.api.definition.components.tool.tool import Tool +from langbot_plugin.api.entities.builtin.provider import session as provider_session + + +class QAPluginEchoTool(Tool): + async def call( + self, + params: dict[str, Any], + session: provider_session.Session, + query_id: int, + ) -> str: + text = str(params.get("text", "")) + return f"qa-plugin-smoke:{text}" diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/tools/qa_plugin_echo.yaml b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/tools/qa_plugin_echo.yaml new file mode 100644 index 000000000..73ea93e10 --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/tools/qa_plugin_echo.yaml @@ -0,0 +1,24 @@ +apiVersion: v1 +kind: Tool +metadata: + name: qa_plugin_echo + label: + en_US: QA Plugin Echo + zh_Hans: QA 插件回显 + description: + en_US: Echoes deterministic text with a plugin-specific tool name. + zh_Hans: 使用插件专属工具名回显确定性文本。 +spec: + parameters: + type: object + properties: + text: + type: string + description: Text to echo. + required: + - text + llm_prompt: Echo deterministic text with qa-plugin-smoke prefix for plugin tool testing. +execution: + python: + path: qa_plugin_echo.py + attr: QAPluginEchoTool diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/tools/qa_plugin_sleep.py b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/tools/qa_plugin_sleep.py new file mode 100644 index 000000000..1d0d97dd7 --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/tools/qa_plugin_sleep.py @@ -0,0 +1,26 @@ +from __future__ import annotations + +import asyncio +from typing import Any + +from langbot_plugin.api.definition.components.tool.tool import Tool +from langbot_plugin.api.entities.builtin.provider import session as provider_session + + +class QAPluginSleepTool(Tool): + async def call( + self, + params: dict[str, Any], + session: provider_session.Session, + query_id: int, + ) -> str: + raw_seconds = params.get("seconds", 0) + try: + seconds = float(raw_seconds) + except (TypeError, ValueError): + seconds = 0.0 + seconds = max(0.0, min(seconds, 15.0)) + text = str(params.get("text", "")) + await asyncio.sleep(seconds) + seconds_label = str(int(seconds)) if seconds.is_integer() else str(seconds) + return f"qa-plugin-smoke:sleep:{seconds_label}:{text}" diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/tools/qa_plugin_sleep.yaml b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/tools/qa_plugin_sleep.yaml new file mode 100644 index 000000000..6bc9bc9dc --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/tools/qa_plugin_sleep.yaml @@ -0,0 +1,28 @@ +apiVersion: v1 +kind: Tool +metadata: + name: qa_plugin_sleep + label: + en_US: QA Plugin Sleep + zh_Hans: QA 插件延迟 + description: + en_US: Sleeps for a bounded number of seconds and returns deterministic text. + zh_Hans: 延迟指定秒数后返回确定性文本。 +spec: + parameters: + type: object + properties: + seconds: + type: number + description: Seconds to sleep, clamped to the 0-15 range. + text: + type: string + description: Text to include in the deterministic result. + required: + - seconds + - text + llm_prompt: Sleep for a bounded duration and return qa-plugin-smoke sleep text for steering tests. +execution: + python: + path: qa_plugin_sleep.py + attr: QAPluginSleepTool diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/main.py b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/main.py new file mode 100644 index 000000000..e2a50d3e6 --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/main.py @@ -0,0 +1,8 @@ +from __future__ import annotations + +from langbot_plugin.api.definition.plugin import BasePlugin + + +class PluginSmoke(BasePlugin): + async def initialize(self) -> None: + self.ready_marker = "qa-plugin-smoke-ready" diff --git a/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/manifest.yaml b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/manifest.yaml new file mode 100644 index 000000000..b5a0f55c4 --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/manifest.yaml @@ -0,0 +1,33 @@ +apiVersion: v1 +kind: Plugin +metadata: + author: qa + name: plugin-smoke + repository: https://example.invalid/langbot/qa-plugin-smoke + version: 0.1.0 + description: + en_US: Local fixture plugin for LangBot plugin e2e smoke tests. + zh_Hans: LangBot 插件端到端冒烟测试用本地夹具插件。 + label: + en_US: QA Plugin Smoke + zh_Hans: QA 插件冒烟测试 + icon: assets/icon.svg +spec: + config: [] + components: + EventListener: + fromDirs: + - path: components/events/ + maxDepth: 1 + Tool: + fromDirs: + - path: components/tools/ + maxDepth: 1 + Page: + fromDirs: + - path: components/pages/ + maxDepth: 2 +execution: + python: + path: main.py + attr: PluginSmoke diff --git a/skills/skills/langbot-testing/fixtures/rag/parser-golden.html b/skills/skills/langbot-testing/fixtures/rag/parser-golden.html new file mode 100644 index 000000000..a684f9307 --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/rag/parser-golden.html @@ -0,0 +1,32 @@ + + + + + LangRAG GeneralParsers Golden Fixture + + +
+
+

Parser Golden Case

+

The parser-rag golden sentinel is aurora-parser-rag-9137.

+
+ + + + + + + + + + + + + +
Parser fieldGolden value
parser_pluginGeneralParsers
knowledge_engineLangRAG
+
+

Retrieve tests should return aurora-parser-rag-9137 with GeneralParsers and LangRAG.

+
+
+ + diff --git a/skills/skills/langbot-testing/fixtures/rag/sentinel-doc.txt b/skills/skills/langbot-testing/fixtures/rag/sentinel-doc.txt new file mode 100644 index 000000000..310d6c1a2 --- /dev/null +++ b/skills/skills/langbot-testing/fixtures/rag/sentinel-doc.txt @@ -0,0 +1,6 @@ +LangBot QA test document. + +The local agent runner retrieval sentinel is azalea-cobalt-7421. +When asked about the sentinel, answer with exactly azalea-cobalt-7421. + +This document is intentionally small so LangRAG ingestion and retrieval can be tested quickly. diff --git a/skills/skills/langbot-testing/probes/agent-runner-async-db-readiness.mjs b/skills/skills/langbot-testing/probes/agent-runner-async-db-readiness.mjs new file mode 100644 index 000000000..3bf1facf0 --- /dev/null +++ b/skills/skills/langbot-testing/probes/agent-runner-async-db-readiness.mjs @@ -0,0 +1,153 @@ +#!/usr/bin/env node + +import { spawn } from "node:child_process"; +import { existsSync } from "node:fs"; +import { mkdir, writeFile } from "node:fs/promises"; +import { join, resolve } from "node:path"; +import { env } from "node:process"; + +function timestampSlug(date = new Date()) { + return date.toISOString().replace(/\.\d{3}Z$/, "Z").replace(/[^0-9A-Za-z]+/g, "-").replace(/^-|-$/g, ""); +} + +function localIsoWithOffset(date = new Date()) { + const offsetMinutes = -date.getTimezoneOffset(); + const sign = offsetMinutes >= 0 ? "+" : "-"; + const absolute = Math.abs(offsetMinutes); + const pad = (value) => String(value).padStart(2, "0"); + return [ + `${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())}`, + `T${pad(date.getHours())}:${pad(date.getMinutes())}:${pad(date.getSeconds())}.${String(date.getMilliseconds()).padStart(3, "0")}`, + `${sign}${pad(Math.floor(absolute / 60))}:${pad(absolute % 60)}`, + ].join(""); +} + +function run(command, timeoutMs, childEnv) { + return new Promise((resolveDone) => { + const child = spawn(command.executable, command.args, { + cwd: command.cwd, + detached: true, + env: childEnv, + stdio: ["ignore", "pipe", "pipe"], + }); + let stdout = ""; + let stderr = ""; + let timedOut = false; + const timeout = setTimeout(() => { + timedOut = true; + try { + process.kill(-child.pid, "SIGTERM"); + } catch { + child.kill("SIGTERM"); + } + }, timeoutMs); + child.stdout.setEncoding("utf8"); + child.stderr.setEncoding("utf8"); + child.stdout.on("data", (chunk) => { + stdout += chunk; + }); + child.stderr.on("data", (chunk) => { + stderr += chunk; + }); + child.on("error", (error) => { + clearTimeout(timeout); + resolveDone({ stdout, stderr, error, timedOut, status: null, signal: null }); + }); + child.on("close", (status, signal) => { + clearTimeout(timeout); + resolveDone({ stdout, stderr, error: null, timedOut, status, signal }); + }); + }); +} + +const script = ` +import asyncio +import aiosqlite + +async def main(): + async with aiosqlite.connect(':memory:') as db: + await db.execute('create table t(id integer primary key)') + await db.commit() + print('AIOSQLITE_READY') + +asyncio.run(main()) +`; + +async function main() { + const root = resolve(env.LBS_ROOT || process.cwd()); + const caseId = "agent-runner-async-db-readiness"; + const runId = env.LBS_RUN_ID || `${timestampSlug()}-${caseId}`; + const evidenceDir = resolve(env.LBS_EVIDENCE_DIR || join(root, "reports", "evidence", runId)); + await mkdir(evidenceDir, { recursive: true }); + const startedAt = new Date(); + const langbotRepo = resolve(root, env.LANGBOT_REPO || "../LangBot"); + const stdoutLog = join(evidenceDir, "probe-stdout.log"); + const stderrLog = join(evidenceDir, "probe-stderr.log"); + const automationResultJson = join(evidenceDir, "automation-result.json"); + const resultJson = join(evidenceDir, "result.json"); + const timeoutMs = Number(env.LANGBOT_ASYNC_DB_READINESS_TIMEOUT_MS || "5000"); + const command = { executable: "rtk", args: ["uv", "run", "python", "-c", script], cwd: langbotRepo }; + const result = { + source: "automation", + probe: "aiosqlite-readiness", + case_id: caseId, + run_id: runId, + started_at: startedAt.toISOString(), + started_at_local: localIsoWithOffset(startedAt), + finished_at: "", + finished_at_local: "", + duration_ms: 0, + status: "fail", + reason: "", + repo_path: langbotRepo, + command, + timeout_ms: timeoutMs, + exit_status: null, + signal: null, + evidence: { stdout_log: stdoutLog, stderr_log: stderrLog, automation_result_json: automationResultJson, result_json: resultJson }, + evidence_collected: ["filesystem"], + }; + try { + if (!existsSync(langbotRepo)) { + result.status = "env_issue"; + result.reason = `LANGBOT_REPO/default ../LangBot did not resolve: ${langbotRepo}`; + } else { + const proc = await run(command, timeoutMs, { + ...process.env, + UV_CACHE_DIR: env.UV_CACHE_DIR || join(evidenceDir, ".uv-cache"), + }); + await writeFile(stdoutLog, proc.stdout, "utf8"); + await writeFile(stderrLog, proc.stderr, "utf8"); + result.exit_status = proc.status; + result.signal = proc.signal; + if (proc.error) { + result.status = "env_issue"; + result.reason = proc.error.message; + } else if (proc.timedOut) { + result.status = "env_issue"; + result.reason = `aiosqlite readiness timed out after ${timeoutMs}ms`; + } else if (proc.status === 0 && proc.stdout.includes("AIOSQLITE_READY")) { + result.status = "pass"; + result.reason = "aiosqlite readiness passed"; + } else { + result.status = "env_issue"; + result.reason = `aiosqlite readiness exited with status ${proc.status}`; + } + } + } catch (error) { + result.status = "env_issue"; + result.reason = error instanceof Error ? error.message : String(error); + } finally { + const finishedAt = new Date(); + result.finished_at = finishedAt.toISOString(); + result.finished_at_local = localIsoWithOffset(finishedAt); + result.duration_ms = finishedAt.getTime() - startedAt.getTime(); + const resultText = `${JSON.stringify(result, null, 2)}\n`; + await writeFile(automationResultJson, resultText, "utf8"); + await writeFile(resultJson, resultText, "utf8"); + console.log(JSON.stringify(result, null, 2)); + } + process.exit(result.status === "pass" ? 0 : result.status === "env_issue" ? 2 : 1); +} + +await main(); diff --git a/skills/skills/langbot-testing/probes/agent-runner-behavior-matrix.mjs b/skills/skills/langbot-testing/probes/agent-runner-behavior-matrix.mjs new file mode 100644 index 000000000..bc7ded87b --- /dev/null +++ b/skills/skills/langbot-testing/probes/agent-runner-behavior-matrix.mjs @@ -0,0 +1,220 @@ +#!/usr/bin/env node + +import { spawn } from "node:child_process"; +import { existsSync } from "node:fs"; +import { mkdir, writeFile } from "node:fs/promises"; +import { delimiter, join, resolve } from "node:path"; +import { env } from "node:process"; + +function timestampSlug(date = new Date()) { + return date.toISOString().replace(/\.\d{3}Z$/, "Z").replace(/[^0-9A-Za-z]+/g, "-").replace(/^-|-$/g, ""); +} + +function localIsoWithOffset(date = new Date()) { + const offsetMinutes = -date.getTimezoneOffset(); + const sign = offsetMinutes >= 0 ? "+" : "-"; + const absolute = Math.abs(offsetMinutes); + const pad = (value) => String(value).padStart(2, "0"); + return [ + `${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())}`, + `T${pad(date.getHours())}:${pad(date.getMinutes())}:${pad(date.getSeconds())}.${String(date.getMilliseconds()).padStart(3, "0")}`, + `${sign}${pad(Math.floor(absolute / 60))}:${pad(absolute % 60)}`, + ].join(""); +} + +function run(command, timeoutMs, childEnv) { + return new Promise((resolveDone) => { + const child = spawn(command.executable, command.args, { + cwd: command.cwd, + detached: true, + env: childEnv, + stdio: ["ignore", "pipe", "pipe"], + }); + let stdout = ""; + let stderr = ""; + let timedOut = false; + const timeout = setTimeout(() => { + timedOut = true; + try { + process.kill(-child.pid, "SIGTERM"); + } catch { + child.kill("SIGTERM"); + } + }, timeoutMs); + child.stdout.setEncoding("utf8"); + child.stderr.setEncoding("utf8"); + child.stdout.on("data", (chunk) => { + stdout += chunk; + }); + child.stderr.on("data", (chunk) => { + stderr += chunk; + }); + child.on("error", (error) => { + clearTimeout(timeout); + resolveDone({ stdout, stderr, error, timedOut, status: null, signal: null }); + }); + child.on("close", (status, signal) => { + clearTimeout(timeout); + resolveDone({ stdout, stderr, error: null, timedOut, status, signal }); + }); + }); +} + +const script = String.raw` +import asyncio +import json +import sys +from pathlib import Path + +from langbot.pkg.agent.runner.descriptor import AgentRunnerDescriptor +from langbot.pkg.agent.runner.errors import RunnerExecutionError, RunnerProtocolError +from langbot.pkg.agent.runner.result_normalizer import AgentResultNormalizer + +class Logger: + def debug(self, *_args, **_kwargs): pass + def info(self, *_args, **_kwargs): pass + def warning(self, *_args, **_kwargs): pass + def error(self, *_args, **_kwargs): pass + +class App: + logger = Logger() + +def descriptor(): + return AgentRunnerDescriptor( + id='plugin:qa/agent-runner/default', + source='plugin', + label={'en_US': 'QA AgentRunner'}, + plugin_author='qa', + plugin_name='agent-runner', + runner_name='default', + capabilities={'streaming': True}, + ) + +async def consume_behavior(normalizer, desc, behavior): + messages = [] + chunks = [] + failures = [] + protocol_errors = [] + for result in behavior['results']: + try: + normalized = await normalizer.normalize(result, desc) + except RunnerExecutionError as exc: + failures.append(str(exc)) + continue + except RunnerProtocolError as exc: + protocol_errors.append(str(exc)) + continue + if normalized is None: + continue + content = getattr(normalized, 'content', '') + if normalized.__class__.__name__ == 'MessageChunk': + chunks.append(content) + else: + messages.append(content) + return { + 'name': behavior['name'], + 'messages': messages, + 'chunks': chunks, + 'failures': failures, + 'protocol_errors': protocol_errors, + } + +async def main(path): + data = json.loads(Path(path).read_text()) + normalizer = AgentResultNormalizer(App()) + desc = descriptor() + observed = [await consume_behavior(normalizer, desc, item) for item in data['behaviors']] + by_name = {item['name']: item for item in observed} + assert by_name['ok']['messages'] == ['QA_RUNNER_OK'], by_name['ok'] + assert ''.join(by_name['stream_ok']['chunks']) == 'QA_RUNNER_STREAM_OK', by_name['stream_ok'] + assert by_name['empty_output']['messages'] == [] and by_name['empty_output']['chunks'] == [], by_name['empty_output'] + assert by_name['malformed_result']['messages'] == [] and by_name['malformed_result']['chunks'] == [], by_name['malformed_result'] + assert by_name['controlled_failure']['failures'], by_name['controlled_failure'] + print('QA_RUNNER_BEHAVIOR_MATRIX_OK behaviors=%d' % len(observed)) + +asyncio.run(main(sys.argv[1])) +`; + +async function main() { + const root = resolve(env.LBS_ROOT || process.cwd()); + const caseId = "agent-runner-behavior-matrix"; + const runId = env.LBS_RUN_ID || `${timestampSlug()}-${caseId}`; + const evidenceDir = resolve(env.LBS_EVIDENCE_DIR || join(root, "reports", "evidence", runId)); + await mkdir(evidenceDir, { recursive: true }); + const startedAt = new Date(); + const langbotRepo = resolve(root, env.LANGBOT_REPO || "../LangBot"); + const sdkSrc = resolve(root, env.LANGBOT_PLUGIN_SDK_REPO || "../langbot-plugin-sdk/src"); + const fixturePath = resolve(root, "skills/langbot-testing/fixtures/agent-runner/qa-runner-behaviors.json"); + const stdoutLog = join(evidenceDir, "probe-stdout.log"); + const stderrLog = join(evidenceDir, "probe-stderr.log"); + const automationResultJson = join(evidenceDir, "automation-result.json"); + const resultJson = join(evidenceDir, "result.json"); + const timeoutMs = Number(env.LANGBOT_AGENT_RUNNER_PROBE_TIMEOUT_MS || "30000"); + const command = { executable: "rtk", args: ["uv", "run", "python", "-c", script, fixturePath], cwd: langbotRepo }; + const result = { + source: "automation", + probe: "agent-runner-behavior-matrix", + case_id: caseId, + run_id: runId, + started_at: startedAt.toISOString(), + started_at_local: localIsoWithOffset(startedAt), + finished_at: "", + finished_at_local: "", + duration_ms: 0, + status: "fail", + reason: "", + fixture_path: fixturePath, + repo_path: langbotRepo, + python_paths: [sdkSrc], + command, + timeout_ms: timeoutMs, + exit_status: null, + signal: null, + evidence: { stdout_log: stdoutLog, stderr_log: stderrLog, automation_result_json: automationResultJson, result_json: resultJson }, + evidence_collected: ["filesystem"], + }; + try { + if (!existsSync(langbotRepo) || !existsSync(fixturePath)) { + result.status = "env_issue"; + result.reason = `missing repo or fixture: ${langbotRepo} ${fixturePath}`; + } else { + const proc = await run(command, timeoutMs, { + ...process.env, + PYTHONPATH: [sdkSrc, process.env.PYTHONPATH].filter(Boolean).join(delimiter), + UV_CACHE_DIR: env.UV_CACHE_DIR || join(evidenceDir, ".uv-cache"), + }); + await writeFile(stdoutLog, proc.stdout, "utf8"); + await writeFile(stderrLog, proc.stderr, "utf8"); + result.exit_status = proc.status; + result.signal = proc.signal; + if (proc.error) { + result.status = "env_issue"; + result.reason = proc.error.message; + } else if (proc.timedOut) { + result.status = "fail"; + result.reason = `behavior matrix timed out after ${timeoutMs}ms`; + } else if (proc.status === 0 && proc.stdout.includes("QA_RUNNER_BEHAVIOR_MATRIX_OK")) { + result.status = "pass"; + result.reason = "behavior matrix passed"; + } else { + result.status = "fail"; + result.reason = `behavior matrix exited with status ${proc.status}`; + } + } + } catch (error) { + result.status = "fail"; + result.reason = error instanceof Error ? error.message : String(error); + } finally { + const finishedAt = new Date(); + result.finished_at = finishedAt.toISOString(); + result.finished_at_local = localIsoWithOffset(finishedAt); + result.duration_ms = finishedAt.getTime() - startedAt.getTime(); + const resultText = `${JSON.stringify(result, null, 2)}\n`; + await writeFile(automationResultJson, resultText, "utf8"); + await writeFile(resultJson, resultText, "utf8"); + console.log(JSON.stringify(result, null, 2)); + } + process.exit(result.status === "pass" ? 0 : result.status === "env_issue" ? 2 : 1); +} + +await main(); diff --git a/skills/skills/langbot-testing/probes/agent-runner-fixture-contract.mjs b/skills/skills/langbot-testing/probes/agent-runner-fixture-contract.mjs new file mode 100644 index 000000000..00c447f71 --- /dev/null +++ b/skills/skills/langbot-testing/probes/agent-runner-fixture-contract.mjs @@ -0,0 +1,206 @@ +#!/usr/bin/env node + +import { spawn } from "node:child_process"; +import { existsSync } from "node:fs"; +import { mkdir, writeFile } from "node:fs/promises"; +import { delimiter, join, resolve } from "node:path"; +import { env } from "node:process"; + +function timestampSlug(date = new Date()) { + return date.toISOString().replace(/\.\d{3}Z$/, "Z").replace(/[^0-9A-Za-z]+/g, "-").replace(/^-|-$/g, ""); +} + +function localIsoWithOffset(date = new Date()) { + const offsetMinutes = -date.getTimezoneOffset(); + const sign = offsetMinutes >= 0 ? "+" : "-"; + const absolute = Math.abs(offsetMinutes); + const pad = (value) => String(value).padStart(2, "0"); + return [ + `${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())}`, + `T${pad(date.getHours())}:${pad(date.getMinutes())}:${pad(date.getSeconds())}.${String(date.getMilliseconds()).padStart(3, "0")}`, + `${sign}${pad(Math.floor(absolute / 60))}:${pad(absolute % 60)}`, + ].join(""); +} + +function run(command, timeoutMs, childEnv) { + return new Promise((resolveDone) => { + const child = spawn(command.executable, command.args, { + cwd: command.cwd, + detached: true, + env: childEnv, + stdio: ["ignore", "pipe", "pipe"], + }); + let stdout = ""; + let stderr = ""; + let timedOut = false; + const timeout = setTimeout(() => { + timedOut = true; + try { + process.kill(-child.pid, "SIGTERM"); + } catch { + child.kill("SIGTERM"); + } + }, timeoutMs); + child.stdout.setEncoding("utf8"); + child.stderr.setEncoding("utf8"); + child.stdout.on("data", (chunk) => { + stdout += chunk; + }); + child.stderr.on("data", (chunk) => { + stderr += chunk; + }); + child.on("error", (error) => { + clearTimeout(timeout); + resolveDone({ stdout, stderr, error, timedOut, status: null, signal: null }); + }); + child.on("close", (status, signal) => { + clearTimeout(timeout); + resolveDone({ stdout, stderr, error: null, timedOut, status, signal }); + }); + }); +} + +const script = String.raw` +import asyncio +import importlib.util +import sys +from pathlib import Path + +from langbot_plugin.api.entities.builtin.agent_runner.context import AgentRunContext +from langbot_plugin.api.entities.builtin.agent_runner.delivery import DeliveryContext +from langbot_plugin.api.entities.builtin.agent_runner.event import AgentEventContext +from langbot_plugin.api.entities.builtin.agent_runner.input import AgentInput +from langbot_plugin.api.entities.builtin.agent_runner.resources import AgentResources +from langbot_plugin.api.entities.builtin.agent_runner.runtime import AgentRuntimeContext +from langbot_plugin.api.entities.builtin.agent_runner.trigger import AgentTrigger + +fixture = Path(sys.argv[1]) +runner_py = fixture / "components" / "agent_runner" / "default.py" +manifest = fixture / "manifest.yaml" +runner_yaml = fixture / "components" / "agent_runner" / "default.yaml" +assert manifest.exists(), manifest +assert runner_yaml.exists(), runner_yaml +spec = importlib.util.spec_from_file_location("qa_agent_runner_fixture", runner_py) +module = importlib.util.module_from_spec(spec) +assert spec and spec.loader +spec.loader.exec_module(module) + +def context(run_id, text): + return AgentRunContext( + run_id=run_id, + trigger=AgentTrigger(type="message.received", source="webui"), + event=AgentEventContext(event_id=f"evt-{run_id}", event_type="message.received", source="webui"), + input=AgentInput(text=text), + delivery=DeliveryContext(surface="debug_chat"), + resources=AgentResources(), + runtime=AgentRuntimeContext(langbot_version="qa"), + ) + +async def collect(text): + runner = module.DefaultAgentRunner() + results = [] + async for result in runner.run(context(f"run-{len(text)}", text)): + results.append(result) + return results + +async def main(): + normal = await collect("hello") + assert len(normal) == 1, normal + assert normal[0].type.value == "run.completed" + assert normal[0].data["message"]["content"] == "QA_AGENT_RUNNER_OK:hello" + + stream = await collect("stream hello") + assert [item.type.value for item in stream] == ["message.delta", "message.delta", "message.delta", "run.completed"] + assert "".join(item.data["chunk"]["content"] for item in stream[:3]) == "QA_AGENT_RUNNER_OK:stream hello" + + failed = await collect("please fail") + assert len(failed) == 1 + assert failed[0].type.value == "run.failed" + assert failed[0].data["error"] == "QA_AGENT_RUNNER_CONTROLLED_FAILURE" + print("QA_AGENT_RUNNER_FIXTURE_CONTRACT_OK") + +asyncio.run(main()) +`; + +async function main() { + const root = resolve(env.LBS_ROOT || process.cwd()); + const caseId = "agent-runner-fixture-contract"; + const runId = env.LBS_RUN_ID || `${timestampSlug()}-${caseId}`; + const evidenceDir = resolve(env.LBS_EVIDENCE_DIR || join(root, "reports", "evidence", runId)); + await mkdir(evidenceDir, { recursive: true }); + const startedAt = new Date(); + const sdkRepo = resolve(root, env.LANGBOT_PLUGIN_SDK_REPO || "../langbot-plugin-sdk"); + const sdkSrc = resolve(sdkRepo, "src"); + const fixturePath = resolve(root, "skills/langbot-testing/fixtures/plugins/qa-agent-runner"); + const stdoutLog = join(evidenceDir, "probe-stdout.log"); + const stderrLog = join(evidenceDir, "probe-stderr.log"); + const automationResultJson = join(evidenceDir, "automation-result.json"); + const resultJson = join(evidenceDir, "result.json"); + const timeoutMs = Number(env.LANGBOT_AGENT_RUNNER_PROBE_TIMEOUT_MS || "30000"); + const command = { executable: "rtk", args: ["uv", "run", "python", "-c", script, fixturePath], cwd: sdkRepo }; + const result = { + source: "automation", + probe: "agent-runner-fixture-contract", + case_id: caseId, + run_id: runId, + started_at: startedAt.toISOString(), + started_at_local: localIsoWithOffset(startedAt), + finished_at: "", + finished_at_local: "", + duration_ms: 0, + status: "fail", + reason: "", + repo_path: sdkRepo, + fixture_path: fixturePath, + command, + timeout_ms: timeoutMs, + exit_status: null, + signal: null, + evidence: { stdout_log: stdoutLog, stderr_log: stderrLog, automation_result_json: automationResultJson, result_json: resultJson }, + evidence_collected: ["filesystem"], + }; + try { + if (!existsSync(sdkRepo) || !existsSync(fixturePath)) { + result.status = "env_issue"; + result.reason = `SDK repo or fixture path missing: ${sdkRepo} ${fixturePath}`; + } else { + const proc = await run(command, timeoutMs, { + ...process.env, + PYTHONPATH: [sdkSrc, process.env.PYTHONPATH].filter(Boolean).join(delimiter), + UV_CACHE_DIR: env.UV_CACHE_DIR || join(evidenceDir, ".uv-cache"), + }); + await writeFile(stdoutLog, proc.stdout, "utf8"); + await writeFile(stderrLog, proc.stderr, "utf8"); + result.exit_status = proc.status; + result.signal = proc.signal; + if (proc.error) { + result.status = "env_issue"; + result.reason = proc.error.message; + } else if (proc.timedOut) { + result.status = "fail"; + result.reason = `fixture contract probe timed out after ${timeoutMs}ms`; + } else if (proc.status === 0 && proc.stdout.includes("QA_AGENT_RUNNER_FIXTURE_CONTRACT_OK")) { + result.status = "pass"; + result.reason = "QA AgentRunner fixture contract passed"; + } else { + result.status = "fail"; + result.reason = `fixture contract exited with status ${proc.status}`; + } + } + } catch (error) { + result.status = "fail"; + result.reason = error instanceof Error ? error.message : String(error); + } finally { + const finishedAt = new Date(); + result.finished_at = finishedAt.toISOString(); + result.finished_at_local = localIsoWithOffset(finishedAt); + result.duration_ms = finishedAt.getTime() - startedAt.getTime(); + const resultText = `${JSON.stringify(result, null, 2)}\n`; + await writeFile(automationResultJson, resultText, "utf8"); + await writeFile(resultJson, resultText, "utf8"); + console.log(JSON.stringify(result, null, 2)); + } + process.exit(result.status === "pass" ? 0 : result.status === "env_issue" ? 2 : 1); +} + +await main(); diff --git a/skills/skills/langbot-testing/probes/agent-runner-ledger-concurrency.mjs b/skills/skills/langbot-testing/probes/agent-runner-ledger-concurrency.mjs new file mode 100644 index 000000000..f6c98389a --- /dev/null +++ b/skills/skills/langbot-testing/probes/agent-runner-ledger-concurrency.mjs @@ -0,0 +1,20 @@ +#!/usr/bin/env node + +import { runPytestProbe } from "./pytest-probe.mjs"; + +await runPytestProbe({ + caseId: "agent-runner-ledger-concurrency", + repoEnvKey: "LANGBOT_REPO", + defaultRepo: "../LangBot", + pythonPathEnvKeys: ["LANGBOT_PLUGIN_SDK_REPO"], + defaultPythonPaths: ["../langbot-plugin-sdk/src"], + description: "LangBot AgentRunner run ledger claim, lease, authorization, and runtime-admin pytest probe.", + testTargets: [ + "tests/unit_tests/agent/test_run_ledger_store.py::test_create_queued_run_claim_renew_release", + "tests/unit_tests/agent/test_run_ledger_store.py::test_expired_claim_can_be_reclaimed", + "tests/unit_tests/agent/test_run_ledger_api_auth.py::test_runtime_admin_can_register_list_and_claim_with_own_run_session", + "tests/unit_tests/agent/test_run_ledger_api_auth.py::test_run_append_result_basic_path", + "tests/unit_tests/agent/test_run_ledger_api_auth.py::test_run_finalize_basic_path", + "tests/unit_tests/agent/test_run_ledger_api_auth.py::test_run_claim_renew_and_release_actions", + ], +}); diff --git a/skills/skills/langbot-testing/probes/agent-runner-ledger-contention.mjs b/skills/skills/langbot-testing/probes/agent-runner-ledger-contention.mjs new file mode 100644 index 000000000..ec4670011 --- /dev/null +++ b/skills/skills/langbot-testing/probes/agent-runner-ledger-contention.mjs @@ -0,0 +1,230 @@ +#!/usr/bin/env node + +import { spawn } from "node:child_process"; +import { existsSync } from "node:fs"; +import { mkdir, writeFile } from "node:fs/promises"; +import { delimiter, join, resolve } from "node:path"; +import { env } from "node:process"; + +function timestampSlug(date = new Date()) { + return date.toISOString().replace(/\.\d{3}Z$/, "Z").replace(/[^0-9A-Za-z]+/g, "-").replace(/^-|-$/g, ""); +} + +function localIsoWithOffset(date = new Date()) { + const offsetMinutes = -date.getTimezoneOffset(); + const sign = offsetMinutes >= 0 ? "+" : "-"; + const absolute = Math.abs(offsetMinutes); + const pad = (value) => String(value).padStart(2, "0"); + return [ + `${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())}`, + `T${pad(date.getHours())}:${pad(date.getMinutes())}:${pad(date.getSeconds())}.${String(date.getMilliseconds()).padStart(3, "0")}`, + `${sign}${pad(Math.floor(absolute / 60))}:${pad(absolute % 60)}`, + ].join(""); +} + +function run(command, timeoutMs, childEnv) { + return new Promise((resolveDone) => { + const child = spawn(command.executable, command.args, { + cwd: command.cwd, + detached: true, + env: childEnv, + stdio: ["ignore", "pipe", "pipe"], + }); + let stdout = ""; + let stderr = ""; + let timedOut = false; + const timeout = setTimeout(() => { + timedOut = true; + try { + process.kill(-child.pid, "SIGTERM"); + } catch { + child.kill("SIGTERM"); + } + }, timeoutMs); + child.stdout.setEncoding("utf8"); + child.stderr.setEncoding("utf8"); + child.stdout.on("data", (chunk) => { + stdout += chunk; + }); + child.stderr.on("data", (chunk) => { + stderr += chunk; + }); + child.on("error", (error) => { + clearTimeout(timeout); + resolveDone({ stdout, stderr, error, timedOut, status: null, signal: null }); + }); + child.on("close", (status, signal) => { + clearTimeout(timeout); + resolveDone({ stdout, stderr, error: null, timedOut, status, signal }); + }); + }); +} + +const script = String.raw` +import concurrent.futures +import sqlite3 +import sys +import time +from pathlib import Path + +from sqlalchemy import create_engine + +from langbot.pkg.entity.persistence.agent_run import AgentRun, AgentRunEvent, AgentRuntime + +db_path = Path(sys.argv[1]) +run_count = 120 +worker_count = 8 +engine = create_engine(f"sqlite:///{db_path}") +for table in (AgentRun.__table__, AgentRunEvent.__table__, AgentRuntime.__table__): + table.create(engine) + +with engine.begin() as conn: + conn.execute(AgentRun.__table__.insert(), [ + { + "run_id": f"run-{i:03d}", + "event_id": f"evt-{i:03d}", + "binding_id": "binding-contention", + "runner_id": "plugin:qa/agent-runner/default", + "status": "queued", + "queue_name": "default", + "priority": run_count - i, + } + for i in range(run_count) + ]) + +def worker(worker_id): + claimed = [] + conn = sqlite3.connect(db_path, timeout=10, isolation_level=None) + conn.execute("pragma busy_timeout=10000") + try: + while True: + try: + conn.execute("begin immediate") + row = conn.execute( + "select run_id from agent_run where status = 'queued' " + "order by priority desc, id asc limit 1" + ).fetchone() + if row is None: + conn.execute("commit") + return claimed + run_id = row[0] + updated = conn.execute( + "update agent_run " + "set status = 'completed', claimed_by_runtime_id = ?, dispatch_attempts = coalesce(dispatch_attempts, 0) + 1 " + "where run_id = ? and status = 'queued'", + (f"worker-{worker_id}", run_id), + ).rowcount + conn.execute("commit") + if updated == 1: + claimed.append(run_id) + except sqlite3.OperationalError as exc: + try: + conn.execute("rollback") + except sqlite3.OperationalError: + pass + if "locked" not in str(exc).lower(): + raise + time.sleep(0.01) + finally: + conn.close() + +with concurrent.futures.ThreadPoolExecutor(max_workers=worker_count) as pool: + claims = [run_id for worker_claims in pool.map(worker, range(worker_count)) for run_id in worker_claims] + +conn = sqlite3.connect(db_path) +rows = conn.execute( + "select run_id, status, dispatch_attempts, claimed_by_runtime_id from agent_run" +).fetchall() +conn.close() + +assert len(claims) == run_count, len(claims) +assert len(set(claims)) == run_count, "duplicate claims detected" +assert all(row[1] == "completed" for row in rows), rows[:5] +assert all(row[2] == 1 for row in rows), rows[:5] +assert all(row[3] for row in rows), rows[:5] +print(f"LEDGER_CONTENTION_OK runs={run_count} workers={worker_count}") +`; + +async function main() { + const root = resolve(env.LBS_ROOT || process.cwd()); + const caseId = "agent-runner-ledger-contention"; + const runId = env.LBS_RUN_ID || `${timestampSlug()}-${caseId}`; + const evidenceDir = resolve(env.LBS_EVIDENCE_DIR || join(root, "reports", "evidence", runId)); + await mkdir(evidenceDir, { recursive: true }); + const startedAt = new Date(); + const langbotRepo = resolve(root, env.LANGBOT_REPO || "../LangBot"); + const sdkSrc = resolve(root, env.LANGBOT_PLUGIN_SDK_REPO || "../langbot-plugin-sdk/src"); + const dbPath = join(evidenceDir, "ledger-contention.sqlite3"); + const stdoutLog = join(evidenceDir, "probe-stdout.log"); + const stderrLog = join(evidenceDir, "probe-stderr.log"); + const automationResultJson = join(evidenceDir, "automation-result.json"); + const resultJson = join(evidenceDir, "result.json"); + const timeoutMs = Number(env.LANGBOT_AGENT_RUNNER_PROBE_TIMEOUT_MS || "30000"); + const command = { executable: "rtk", args: ["uv", "run", "python", "-c", script, dbPath], cwd: langbotRepo }; + const result = { + source: "automation", + probe: "agent-runner-ledger-contention", + case_id: caseId, + run_id: runId, + started_at: startedAt.toISOString(), + started_at_local: localIsoWithOffset(startedAt), + finished_at: "", + finished_at_local: "", + duration_ms: 0, + status: "fail", + reason: "", + repo_path: langbotRepo, + python_paths: [sdkSrc], + database_path: dbPath, + command, + timeout_ms: timeoutMs, + exit_status: null, + signal: null, + evidence: { stdout_log: stdoutLog, stderr_log: stderrLog, database: dbPath, automation_result_json: automationResultJson, result_json: resultJson }, + evidence_collected: ["filesystem"], + }; + try { + if (!existsSync(langbotRepo)) { + result.status = "env_issue"; + result.reason = `LANGBOT_REPO/default ../LangBot did not resolve: ${langbotRepo}`; + } else { + const proc = await run(command, timeoutMs, { + ...process.env, + PYTHONPATH: [sdkSrc, process.env.PYTHONPATH].filter(Boolean).join(delimiter), + UV_CACHE_DIR: env.UV_CACHE_DIR || join(evidenceDir, ".uv-cache"), + }); + await writeFile(stdoutLog, proc.stdout, "utf8"); + await writeFile(stderrLog, proc.stderr, "utf8"); + result.exit_status = proc.status; + result.signal = proc.signal; + if (proc.error) { + result.status = "env_issue"; + result.reason = proc.error.message; + } else if (proc.timedOut) { + result.status = "fail"; + result.reason = `ledger contention timed out after ${timeoutMs}ms`; + } else if (proc.status === 0 && proc.stdout.includes("LEDGER_CONTENTION_OK")) { + result.status = "pass"; + result.reason = "ledger contention probe passed"; + } else { + result.status = "fail"; + result.reason = `ledger contention exited with status ${proc.status}`; + } + } + } catch (error) { + result.status = "fail"; + result.reason = error instanceof Error ? error.message : String(error); + } finally { + const finishedAt = new Date(); + result.finished_at = finishedAt.toISOString(); + result.finished_at_local = localIsoWithOffset(finishedAt); + result.duration_ms = finishedAt.getTime() - startedAt.getTime(); + const resultText = `${JSON.stringify(result, null, 2)}\n`; + await writeFile(automationResultJson, resultText, "utf8"); + await writeFile(resultJson, resultText, "utf8"); + console.log(JSON.stringify(result, null, 2)); + } + process.exit(result.status === "pass" ? 0 : result.status === "env_issue" ? 2 : 1); +} + +await main(); diff --git a/skills/skills/langbot-testing/probes/agent-runner-ledger-invariants.mjs b/skills/skills/langbot-testing/probes/agent-runner-ledger-invariants.mjs new file mode 100644 index 000000000..1b5416d10 --- /dev/null +++ b/skills/skills/langbot-testing/probes/agent-runner-ledger-invariants.mjs @@ -0,0 +1,211 @@ +#!/usr/bin/env node + +import { spawn } from "node:child_process"; +import { existsSync } from "node:fs"; +import { mkdir, writeFile } from "node:fs/promises"; +import { delimiter, join, resolve } from "node:path"; +import { env } from "node:process"; + +function timestampSlug(date = new Date()) { + return date.toISOString().replace(/\.\d{3}Z$/, "Z").replace(/[^0-9A-Za-z]+/g, "-").replace(/^-|-$/g, ""); +} + +function localIsoWithOffset(date = new Date()) { + const offsetMinutes = -date.getTimezoneOffset(); + const sign = offsetMinutes >= 0 ? "+" : "-"; + const absolute = Math.abs(offsetMinutes); + const pad = (value) => String(value).padStart(2, "0"); + return [ + `${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())}`, + `T${pad(date.getHours())}:${pad(date.getMinutes())}:${pad(date.getSeconds())}.${String(date.getMilliseconds()).padStart(3, "0")}`, + `${sign}${pad(Math.floor(absolute / 60))}:${pad(absolute % 60)}`, + ].join(""); +} + +function resolveFromRoot(root, value) { + return resolve(root, value); +} + +function runProcess(command, timeoutMs, childEnv) { + return new Promise((resolveDone) => { + const child = spawn(command.executable, command.args, { + cwd: command.cwd, + detached: true, + env: childEnv, + stdio: ["ignore", "pipe", "pipe"], + }); + let stdout = ""; + let stderr = ""; + let timedOut = false; + const timeout = setTimeout(() => { + timedOut = true; + try { + process.kill(-child.pid, "SIGTERM"); + } catch { + child.kill("SIGTERM"); + } + }, timeoutMs); + child.stdout.setEncoding("utf8"); + child.stderr.setEncoding("utf8"); + child.stdout.on("data", (chunk) => { + stdout += chunk; + }); + child.stderr.on("data", (chunk) => { + stderr += chunk; + }); + child.on("error", (error) => { + clearTimeout(timeout); + resolveDone({ stdout, stderr, error, timedOut, status: null, signal: null }); + }); + child.on("close", (status, signal) => { + clearTimeout(timeout); + resolveDone({ stdout, stderr, error: null, timedOut, status, signal }); + }); + }); +} + +const probeScript = String.raw` +import sqlite3 +from sqlalchemy import create_engine, inspect + +from langbot.pkg.agent.runner.run_ledger_store import TERMINAL_STATUSES +from langbot.pkg.entity.persistence.agent_run import AgentRun, AgentRunEvent, AgentRuntime + +expected_statuses = {'created', 'queued', 'claimed', 'running', 'completed', 'failed', 'cancelled', 'timeout'} +expected_terminal = {'completed', 'failed', 'cancelled', 'timeout'} +assert TERMINAL_STATUSES == expected_terminal, TERMINAL_STATUSES + +engine = create_engine('sqlite:///:memory:') +for table in (AgentRun.__table__, AgentRunEvent.__table__, AgentRuntime.__table__): + table.create(engine) + +inspector = inspect(engine) +assert set(inspector.get_table_names()) == {'agent_run', 'agent_run_event', 'agent_runtime'} +agent_run_indexes = {index['name']: tuple(index['column_names']) for index in inspector.get_indexes('agent_run')} +for name in ( + 'ix_agent_run_scope_status', + 'ix_agent_run_runner_status', + 'ix_agent_run_queue_claim', + 'ix_agent_run_run_id', + 'ix_agent_run_claim_token', +): + assert name in agent_run_indexes, agent_run_indexes + +event_uniques = { + unique['name']: tuple(unique['column_names']) + for unique in inspector.get_unique_constraints('agent_run_event') +} +assert event_uniques['uq_agent_run_event_run_sequence'] == ('run_id', 'sequence') + +with engine.begin() as conn: + conn.execute(AgentRun.__table__.insert().values( + run_id='run-sync', + event_id='evt-sync', + binding_id='binding-sync', + runner_id='plugin:test/runner/default', + status='queued', + queue_name='default', + priority=10, + )) + row = conn.execute(AgentRun.__table__.select().where(AgentRun.__table__.c.run_id == 'run-sync')).mappings().one() + assert row['status'] == 'queued' + conn.execute(AgentRunEvent.__table__.insert().values( + run_id='run-sync', + sequence=1, + type='message.completed', + data_json='{}', + source='runner', + )) + +print('LEDGER_INVARIANTS_OK tables=3 statuses=8') +`; + +async function main() { + const root = resolve(env.LBS_ROOT || process.cwd()); + const caseId = "agent-runner-ledger-invariants"; + const runId = env.LBS_RUN_ID || `${timestampSlug()}-${caseId}`; + const evidenceDir = resolve(env.LBS_EVIDENCE_DIR || join(root, "reports", "evidence", runId)); + await mkdir(evidenceDir, { recursive: true }); + const startedAt = new Date(); + const langbotRepo = resolveFromRoot(root, env.LANGBOT_REPO || "../LangBot"); + const sdkSrc = resolveFromRoot(root, env.LANGBOT_PLUGIN_SDK_REPO || "../langbot-plugin-sdk/src"); + const stdoutLog = join(evidenceDir, "probe-stdout.log"); + const stderrLog = join(evidenceDir, "probe-stderr.log"); + const automationResultJson = join(evidenceDir, "automation-result.json"); + const resultJson = join(evidenceDir, "result.json"); + const command = { executable: "rtk", args: ["uv", "run", "python", "-c", probeScript], cwd: langbotRepo }; + const timeoutMs = Number(env.LANGBOT_AGENT_RUNNER_PROBE_TIMEOUT_MS || "30000"); + const result = { + source: "automation", + probe: "python-sync", + case_id: caseId, + run_id: runId, + started_at: startedAt.toISOString(), + started_at_local: localIsoWithOffset(startedAt), + finished_at: "", + finished_at_local: "", + duration_ms: 0, + status: "fail", + reason: "", + repo_path: langbotRepo, + python_paths: [sdkSrc], + command, + timeout_ms: timeoutMs, + exit_status: null, + signal: null, + evidence: { + stdout_log: stdoutLog, + stderr_log: stderrLog, + automation_result_json: automationResultJson, + result_json: resultJson, + }, + evidence_collected: ["filesystem"], + }; + + try { + if (!existsSync(langbotRepo)) { + result.status = "env_issue"; + result.reason = `LANGBOT_REPO/default ../LangBot did not resolve: ${langbotRepo}`; + } else { + const childEnv = { + ...process.env, + PYTHONPATH: [sdkSrc, process.env.PYTHONPATH].filter(Boolean).join(delimiter), + UV_CACHE_DIR: env.UV_CACHE_DIR || join(evidenceDir, ".uv-cache"), + }; + await mkdir(childEnv.UV_CACHE_DIR, { recursive: true }); + const proc = await runProcess(command, timeoutMs, childEnv); + result.exit_status = proc.status; + result.signal = proc.signal; + await writeFile(stdoutLog, proc.stdout, "utf8"); + await writeFile(stderrLog, proc.stderr, "utf8"); + if (proc.error) { + result.status = "env_issue"; + result.reason = proc.error.message; + } else if (proc.timedOut) { + result.status = "fail"; + result.reason = `ledger invariant probe timed out after ${timeoutMs}ms`; + } else if (proc.status === 0) { + result.status = "pass"; + result.reason = "ledger invariant probe passed"; + } else { + result.status = "fail"; + result.reason = `ledger invariant probe exited with status ${proc.status}`; + } + } + } catch (error) { + result.status = "fail"; + result.reason = error instanceof Error ? error.message : String(error); + } finally { + const finishedAt = new Date(); + result.finished_at = finishedAt.toISOString(); + result.finished_at_local = localIsoWithOffset(finishedAt); + result.duration_ms = finishedAt.getTime() - startedAt.getTime(); + const resultText = `${JSON.stringify(result, null, 2)}\n`; + await writeFile(automationResultJson, resultText, "utf8"); + await writeFile(resultJson, resultText, "utf8"); + console.log(JSON.stringify(result, null, 2)); + } + process.exit(result.status === "pass" ? 0 : result.status === "env_issue" ? 2 : 1); +} + +await main(); diff --git a/skills/skills/langbot-testing/probes/agent-runner-ledger-stress.mjs b/skills/skills/langbot-testing/probes/agent-runner-ledger-stress.mjs new file mode 100644 index 000000000..3575358a2 --- /dev/null +++ b/skills/skills/langbot-testing/probes/agent-runner-ledger-stress.mjs @@ -0,0 +1,195 @@ +#!/usr/bin/env node + +import { spawn } from "node:child_process"; +import { existsSync } from "node:fs"; +import { mkdir, writeFile } from "node:fs/promises"; +import { delimiter, join, resolve } from "node:path"; +import { env } from "node:process"; + +function timestampSlug(date = new Date()) { + return date.toISOString().replace(/\.\d{3}Z$/, "Z").replace(/[^0-9A-Za-z]+/g, "-").replace(/^-|-$/g, ""); +} + +function localIsoWithOffset(date = new Date()) { + const offsetMinutes = -date.getTimezoneOffset(); + const sign = offsetMinutes >= 0 ? "+" : "-"; + const absolute = Math.abs(offsetMinutes); + const pad = (value) => String(value).padStart(2, "0"); + return [ + `${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())}`, + `T${pad(date.getHours())}:${pad(date.getMinutes())}:${pad(date.getSeconds())}.${String(date.getMilliseconds()).padStart(3, "0")}`, + `${sign}${pad(Math.floor(absolute / 60))}:${pad(absolute % 60)}`, + ].join(""); +} + +function run(command, timeoutMs, childEnv) { + return new Promise((resolveDone) => { + const child = spawn(command.executable, command.args, { + cwd: command.cwd, + detached: true, + env: childEnv, + stdio: ["ignore", "pipe", "pipe"], + }); + let stdout = ""; + let stderr = ""; + let timedOut = false; + const timeout = setTimeout(() => { + timedOut = true; + try { + process.kill(-child.pid, "SIGTERM"); + } catch { + child.kill("SIGTERM"); + } + }, timeoutMs); + child.stdout.setEncoding("utf8"); + child.stderr.setEncoding("utf8"); + child.stdout.on("data", (chunk) => { + stdout += chunk; + }); + child.stderr.on("data", (chunk) => { + stderr += chunk; + }); + child.on("error", (error) => { + clearTimeout(timeout); + resolveDone({ stdout, stderr, error, timedOut, status: null, signal: null }); + }); + child.on("close", (status, signal) => { + clearTimeout(timeout); + resolveDone({ stdout, stderr, error: null, timedOut, status, signal }); + }); + }); +} + +const script = String.raw` +from sqlalchemy import create_engine, select, update + +from langbot.pkg.entity.persistence.agent_run import AgentRun, AgentRunEvent, AgentRuntime + +run_count = 100 +runtime_count = 5 +engine = create_engine('sqlite:///:memory:') +for table in (AgentRun.__table__, AgentRunEvent.__table__, AgentRuntime.__table__): + table.create(engine) + +claimed = [] +with engine.begin() as conn: + conn.execute(AgentRun.__table__.insert(), [ + { + 'run_id': f'run-{i:03d}', + 'event_id': f'evt-{i:03d}', + 'binding_id': 'binding-stress', + 'runner_id': 'plugin:qa/agent-runner/default', + 'status': 'queued', + 'queue_name': 'default', + 'priority': run_count - i, + } + for i in range(run_count) + ]) + while True: + row = conn.execute( + select(AgentRun.__table__) + .where(AgentRun.__table__.c.status == 'queued') + .order_by(AgentRun.__table__.c.priority.desc(), AgentRun.__table__.c.id.asc()) + .limit(1) + ).mappings().first() + if row is None: + break + runtime_id = f'runtime-{len(claimed) % runtime_count}' + conn.execute( + update(AgentRun.__table__) + .where(AgentRun.__table__.c.run_id == row['run_id']) + .values(status='completed', claimed_by_runtime_id=runtime_id, dispatch_attempts=1) + ) + claimed.append(row['run_id']) + rows = conn.execute(select(AgentRun.__table__)).mappings().all() + +assert len(claimed) == run_count +assert len(set(claimed)) == run_count +assert all(row['status'] == 'completed' for row in rows) +assert all(row['dispatch_attempts'] == 1 for row in rows) +assert claimed[0] == 'run-000' +assert claimed[-1] == 'run-099' +print(f'LEDGER_STRESS_OK runs={run_count} runtimes={runtime_count}') +`; + +async function main() { + const root = resolve(env.LBS_ROOT || process.cwd()); + const caseId = "agent-runner-ledger-stress"; + const runId = env.LBS_RUN_ID || `${timestampSlug()}-${caseId}`; + const evidenceDir = resolve(env.LBS_EVIDENCE_DIR || join(root, "reports", "evidence", runId)); + await mkdir(evidenceDir, { recursive: true }); + const startedAt = new Date(); + const langbotRepo = resolve(root, env.LANGBOT_REPO || "../LangBot"); + const sdkSrc = resolve(root, env.LANGBOT_PLUGIN_SDK_REPO || "../langbot-plugin-sdk/src"); + const stdoutLog = join(evidenceDir, "probe-stdout.log"); + const stderrLog = join(evidenceDir, "probe-stderr.log"); + const automationResultJson = join(evidenceDir, "automation-result.json"); + const resultJson = join(evidenceDir, "result.json"); + const timeoutMs = Number(env.LANGBOT_AGENT_RUNNER_PROBE_TIMEOUT_MS || "30000"); + const command = { executable: "rtk", args: ["uv", "run", "python", "-c", script], cwd: langbotRepo }; + const result = { + source: "automation", + probe: "agent-runner-ledger-stress", + case_id: caseId, + run_id: runId, + started_at: startedAt.toISOString(), + started_at_local: localIsoWithOffset(startedAt), + finished_at: "", + finished_at_local: "", + duration_ms: 0, + status: "fail", + reason: "", + repo_path: langbotRepo, + python_paths: [sdkSrc], + command, + timeout_ms: timeoutMs, + exit_status: null, + signal: null, + evidence: { stdout_log: stdoutLog, stderr_log: stderrLog, automation_result_json: automationResultJson, result_json: resultJson }, + evidence_collected: ["filesystem"], + }; + try { + if (!existsSync(langbotRepo)) { + result.status = "env_issue"; + result.reason = `LANGBOT_REPO/default ../LangBot did not resolve: ${langbotRepo}`; + } else { + const proc = await run(command, timeoutMs, { + ...process.env, + PYTHONPATH: [sdkSrc, process.env.PYTHONPATH].filter(Boolean).join(delimiter), + UV_CACHE_DIR: env.UV_CACHE_DIR || join(evidenceDir, ".uv-cache"), + }); + await writeFile(stdoutLog, proc.stdout, "utf8"); + await writeFile(stderrLog, proc.stderr, "utf8"); + result.exit_status = proc.status; + result.signal = proc.signal; + if (proc.error) { + result.status = "env_issue"; + result.reason = proc.error.message; + } else if (proc.timedOut) { + result.status = "fail"; + result.reason = `ledger stress timed out after ${timeoutMs}ms`; + } else if (proc.status === 0 && proc.stdout.includes("LEDGER_STRESS_OK")) { + result.status = "pass"; + result.reason = "ledger stress probe passed"; + } else { + result.status = "fail"; + result.reason = `ledger stress exited with status ${proc.status}`; + } + } + } catch (error) { + result.status = "fail"; + result.reason = error instanceof Error ? error.message : String(error); + } finally { + const finishedAt = new Date(); + result.finished_at = finishedAt.toISOString(); + result.finished_at_local = localIsoWithOffset(finishedAt); + result.duration_ms = finishedAt.getTime() - startedAt.getTime(); + const resultText = `${JSON.stringify(result, null, 2)}\n`; + await writeFile(automationResultJson, resultText, "utf8"); + await writeFile(resultJson, resultText, "utf8"); + console.log(JSON.stringify(result, null, 2)); + } + process.exit(result.status === "pass" ? 0 : result.status === "env_issue" ? 2 : 1); +} + +await main(); diff --git a/skills/skills/langbot-testing/probes/agent-runner-runtime-chaos.mjs b/skills/skills/langbot-testing/probes/agent-runner-runtime-chaos.mjs new file mode 100644 index 000000000..07ee5f988 --- /dev/null +++ b/skills/skills/langbot-testing/probes/agent-runner-runtime-chaos.mjs @@ -0,0 +1,14 @@ +#!/usr/bin/env node + +import { runPytestProbe } from "./pytest-probe.mjs"; + +await runPytestProbe({ + caseId: "agent-runner-runtime-chaos", + repoEnvKey: "LANGBOT_PLUGIN_SDK_REPO", + defaultRepo: "../langbot-plugin-sdk", + description: "LangBot plugin SDK AgentRunner runtime failure, timeout, forwarding, and pull API pytest probe.", + testTargets: [ + "tests/runtime/plugin/test_mgr_agent_runner.py", + "tests/runtime/test_pull_api_handlers.py", + ], +}); diff --git a/skills/skills/langbot-testing/probes/pytest-probe.mjs b/skills/skills/langbot-testing/probes/pytest-probe.mjs new file mode 100644 index 000000000..db98361e4 --- /dev/null +++ b/skills/skills/langbot-testing/probes/pytest-probe.mjs @@ -0,0 +1,222 @@ +import { spawn } from "node:child_process"; +import { existsSync, readFileSync } from "node:fs"; +import { mkdir, writeFile } from "node:fs/promises"; +import { basename, delimiter, join, resolve } from "node:path"; +import { env } from "node:process"; + +function loadEnvDefaults(root) { + for (const path of [join(root, "skills/.env"), join(root, "skills/.env.local")]) { + if (!existsSync(path)) continue; + for (const rawLine of readFileSync(path, "utf8").split(/\r?\n/)) { + const line = rawLine.trim(); + if (!line || line.startsWith("#")) continue; + const sep = line.indexOf("="); + if (sep === -1) continue; + const key = line.slice(0, sep).trim(); + if (env[key]) continue; + env[key] = line.slice(sep + 1).trim().replace(/^["']|["']$/g, ""); + } + } +} + +function timestampSlug(date = new Date()) { + return date.toISOString().replace(/\.\d{3}Z$/, "Z").replace(/[^0-9A-Za-z]+/g, "-").replace(/^-|-$/g, ""); +} + +function localIsoWithOffset(date = new Date()) { + const offsetMinutes = -date.getTimezoneOffset(); + const sign = offsetMinutes >= 0 ? "+" : "-"; + const absolute = Math.abs(offsetMinutes); + const pad = (value) => String(value).padStart(2, "0"); + return [ + `${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())}`, + `T${pad(date.getHours())}:${pad(date.getMinutes())}:${pad(date.getSeconds())}.${String(date.getMilliseconds()).padStart(3, "0")}`, + `${sign}${pad(Math.floor(absolute / 60))}:${pad(absolute % 60)}`, + ].join(""); +} + +function resolveFromRoot(root, value) { + if (!value) return ""; + return resolve(root, value); +} + +function truncate(text, maxLength = 12000) { + if (text.length <= maxLength) return text; + return `${text.slice(0, maxLength)}\n...[truncated ${text.length - maxLength} chars]`; +} + +function exitCode(status) { + if (status === "pass") return 0; + if (status === "blocked" || status === "env_issue") return 2; + return 1; +} + +async function runProcess(command, timeoutMs, childEnv) { + return await new Promise((resolveDone) => { + const child = spawn(command.executable, command.args, { + cwd: command.cwd, + env: childEnv, + detached: true, + stdio: ["ignore", "pipe", "pipe"], + }); + let stdout = ""; + let stderr = ""; + let timedOut = false; + const timeout = setTimeout(() => { + timedOut = true; + try { + process.kill(-child.pid, "SIGTERM"); + } catch { + child.kill("SIGTERM"); + } + setTimeout(() => { + try { + process.kill(-child.pid, "SIGKILL"); + } catch { + child.kill("SIGKILL"); + } + }, 5000).unref(); + }, timeoutMs); + + child.stdout.setEncoding("utf8"); + child.stderr.setEncoding("utf8"); + child.stdout.on("data", (chunk) => { + stdout += chunk; + }); + child.stderr.on("data", (chunk) => { + stderr += chunk; + }); + child.on("error", (error) => { + clearTimeout(timeout); + resolveDone({ stdout, stderr, error, timedOut, status: null, signal: null }); + }); + child.on("close", (status, signal) => { + clearTimeout(timeout); + resolveDone({ stdout, stderr, error: null, timedOut, status, signal }); + }); + }); +} + +export async function runPytestProbe({ + caseId, + repoEnvKey, + defaultRepo, + pythonPathEnvKeys = [], + defaultPythonPaths = [], + testTargets, + description, + timeoutMs, +}) { + const root = resolve(env.LBS_ROOT || process.cwd()); + loadEnvDefaults(root); + const resolvedTimeoutMs = Number(timeoutMs || env.LANGBOT_AGENT_RUNNER_PROBE_TIMEOUT_MS || "180000"); + + const runId = env.LBS_RUN_ID || `${timestampSlug()}-${caseId}`; + const evidenceDir = resolve(env.LBS_EVIDENCE_DIR || join(root, "reports", "evidence", runId)); + await mkdir(evidenceDir, { recursive: true }); + const uvCacheDir = env.UV_CACHE_DIR || join(evidenceDir, ".uv-cache"); + await mkdir(uvCacheDir, { recursive: true }); + + const startedAt = new Date(); + const repoPath = resolveFromRoot(root, env[repoEnvKey] || defaultRepo); + const pythonPaths = [ + ...pythonPathEnvKeys.map((key) => env[key]).filter(Boolean), + ...defaultPythonPaths, + ].map((value) => resolveFromRoot(root, value)); + const automationResultJson = join(evidenceDir, "automation-result.json"); + const stdoutLog = join(evidenceDir, "pytest-stdout.log"); + const stderrLog = join(evidenceDir, "pytest-stderr.log"); + const resultJson = join(evidenceDir, "result.json"); + const command = { + executable: "rtk", + args: ["uv", "run", "pytest", "-q", ...testTargets], + cwd: repoPath, + }; + const result = { + source: "automation", + probe: "pytest", + case_id: caseId, + run_id: runId, + description, + started_at: startedAt.toISOString(), + started_at_local: localIsoWithOffset(startedAt), + finished_at: "", + finished_at_local: "", + duration_ms: 0, + status: "fail", + reason: "", + repo_env_key: repoEnvKey, + repo_path: repoPath, + python_paths: pythonPaths, + test_targets: testTargets, + command, + timeout_ms: resolvedTimeoutMs, + uv_cache_dir: uvCacheDir, + exit_status: null, + signal: null, + evidence: { + pytest_stdout_log: stdoutLog, + pytest_stderr_log: stderrLog, + automation_result_json: automationResultJson, + result_json: resultJson, + }, + evidence_collected: ["filesystem"], + }; + + await writeFile(stdoutLog, "", "utf8"); + await writeFile(stderrLog, "", "utf8"); + + try { + if (!existsSync(repoPath)) { + result.status = "env_issue"; + result.reason = `${repoEnvKey || "repo"} did not resolve to an existing directory: ${repoPath}`; + } else { + const missingTargets = testTargets.filter((target) => !existsSync(join(repoPath, target.split("::")[0]))); + if (missingTargets.length > 0) { + result.status = "env_issue"; + result.reason = `pytest target file(s) not found in ${basename(repoPath)}: ${missingTargets.join(", ")}`; + } else { + const childEnv = { ...process.env, UV_CACHE_DIR: uvCacheDir }; + if (pythonPaths.length > 0) { + childEnv.PYTHONPATH = [pythonPaths.join(delimiter), childEnv.PYTHONPATH].filter(Boolean).join(delimiter); + } + const proc = await runProcess(command, resolvedTimeoutMs, childEnv); + result.exit_status = proc.status; + result.signal = proc.signal; + await writeFile(stdoutLog, truncate(proc.stdout), "utf8"); + await writeFile(stderrLog, truncate(proc.stderr), "utf8"); + + if (proc.error) { + result.status = "env_issue"; + result.reason = `Failed to start pytest command '${command.executable}': ${proc.error.message}`; + } else if (proc.timedOut) { + result.status = "fail"; + result.reason = `pytest timed out after ${resolvedTimeoutMs}ms. See ${stdoutLog} and ${stderrLog}.`; + } else if (proc.status === 0) { + result.status = "pass"; + result.reason = `pytest passed for ${testTargets.join(", ")}.`; + } else if (/command not found|no such file or directory|executable file not found/i.test(`${proc.stdout}\n${proc.stderr}`)) { + result.status = "env_issue"; + result.reason = `pytest command could not run in ${repoPath}. See ${stdoutLog} and ${stderrLog}.`; + } else { + result.status = "fail"; + result.reason = `pytest exited with status ${proc.status}. See ${stdoutLog} and ${stderrLog}.`; + } + } + } + } catch (error) { + result.status = "fail"; + result.reason = error instanceof Error ? error.message : String(error); + } finally { + const finishedAt = new Date(); + result.finished_at = finishedAt.toISOString(); + result.finished_at_local = localIsoWithOffset(finishedAt); + result.duration_ms = finishedAt.getTime() - startedAt.getTime(); + const resultText = `${JSON.stringify(result, null, 2)}\n`; + await writeFile(automationResultJson, resultText, "utf8"); + await writeFile(resultJson, resultText, "utf8"); + console.log(JSON.stringify(result, null, 2)); + } + + process.exit(exitCode(result.status)); +} diff --git a/skills/skills/langbot-testing/references/agent-runner-qa-workflow.md b/skills/skills/langbot-testing/references/agent-runner-qa-workflow.md new file mode 100644 index 000000000..76e62fa44 --- /dev/null +++ b/skills/skills/langbot-testing/references/agent-runner-qa-workflow.md @@ -0,0 +1,112 @@ +# AgentRunner QA Workflow + +Use this workflow when an agent finishes AgentRunner-related code and enters a +test phase. + +## Order + +1. Inspect changed repositories with `rtk git status --short` and + `rtk git diff --stat`. + Also run `rtk git diff --name-only` and, when untracked files are present, + `rtk git ls-files --others --exclude-standard` so new probes, fixtures, and + cases are not missed. +2. Choose the smallest relevant checks: + - Fast contract probes first. + - Targeted repo tests second. + - Browser cases only after contract probes are green or clearly classified. +3. Start from saved assets: + - `rtk bin/lbs test recommend` + - `rtk bin/lbs suite plan agent-runner-release-gate` + - `rtk bin/lbs case list --tag agent-runner` + - `rtk bin/lbs trouble search agent-runner` +4. Run probes before browser cases: + - `rtk bin/lbs test run agent-runner-fixture-contract --dry-run` + - `rtk bin/lbs test run agent-runner-live-install --dry-run` when a local LangBot + backend is available and installing the QA fixture is acceptable. + - `rtk bin/lbs test run agent-runner-qa-debug-chat --dry-run` when WebUI live + execution needs deterministic coverage without a model provider. This + case runs its setup automation first: install the QA AgentRunner fixture, + create/update the QA pipeline, write the case-specific pipeline env, then + execute Debug Chat. + - `rtk bin/lbs test run agent-runner-ledger-invariants --dry-run` + - `rtk bin/lbs test run agent-runner-ledger-contention --dry-run` + - `rtk bin/lbs test run agent-runner-runtime-chaos --dry-run` + - `rtk bin/lbs test run agent-runner-ledger-concurrency --dry-run` when async DB + readiness is known good. +5. Run browser release paths only after environment preflight: + - `rtk bin/lbs test run agent-runner-release-preflight --dry-run` + - `rtk bin/lbs suite start agent-runner-release-gate --run-id ` + +Remove `--dry-run` only after readiness and `manual_check` preconditions are +confirmed for the current test instance. + +## Diff Triage + +Use changed paths to choose checks. Start with the most specific row that +matches the diff, then add adjacent rows only when shared contracts changed. +For the common case, run `rtk bin/lbs test recommend` first and use this table +only to review or adjust the generated list. + +| Changed Path or Area | First Checks | Add Browser Case When | +| --- | --- | --- | +| `LangBot/src/langbot/pkg/agent/runner/*`, `tests/unit_tests/agent/test_result_normalizer.py`, protocol/result/context/resource builders | `rtk bin/lbs test run agent-runner-fixture-contract --dry-run`; `rtk bin/lbs test run agent-runner-behavior-matrix --dry-run`; targeted LangBot unit tests for touched files | Result shape, user-visible runner output, or Debug Chat delivery changed: add `pipeline-debug-chat` or `local-agent-basic-debug-chat`. | +| `LangBot/src/langbot/pkg/entity/persistence/agent_run.py`, `run_journal.py`, run ledger store/API/auth tests, claim/lease/status code | `rtk bin/lbs test run agent-runner-ledger-invariants --dry-run`; `rtk bin/lbs test run agent-runner-ledger-stress --dry-run`; `rtk bin/lbs test run agent-runner-ledger-contention --dry-run`; `rtk bin/lbs test run agent-runner-async-db-readiness --dry-run` before `rtk bin/lbs test run agent-runner-ledger-concurrency --dry-run` | Debug Chat run lifecycle, resume, or visible completion changed: add `local-agent-basic-debug-chat`. | +| `langbot-plugin-sdk/src/langbot_plugin/api/entities/builtin/agent_runner/*`, `api/proxies/agent_run_api.py`, runtime pull handlers, plugin manager/runtime IO | `rtk bin/lbs test run agent-runner-runtime-chaos --dry-run`; `rtk bin/lbs test run agent-runner-behavior-matrix --dry-run`; targeted SDK pytest | Runtime delivery or tool-call surface changed: add `agent-runner-release-preflight`, then `local-agent-basic-debug-chat`. | +| `langbot-agent-runner/*/components/agent_runner/*`, external runner daemon/client code, ACP/Codex/Claude runner command wrappers | Repo-local targeted tests; `rtk bin/lbs test run agent-runner-runtime-chaos --dry-run`; `rtk bin/lbs test run agent-runner-release-preflight --dry-run` | ACP or external coding runner behavior changed: add `acp-agent-runner-debug-chat`. | +| Prompt preprocessing, effective prompt, pipeline AI config, runner binding/default runner migration | `rtk bin/lbs test run agent-runner-behavior-matrix --dry-run`; targeted LangBot pipeline/agent tests | The runner reads host-provided prompt or saved runner config: add `local-agent-effective-prompt-debug-chat`. | +| Context window, transcript, history/event state, compaction, checkpoint/steering | `rtk bin/lbs test run agent-runner-behavior-matrix --dry-run`; targeted LangBot agent state/context tests | Multi-turn memory, compaction, or steering behavior changed: add `local-agent-context-compaction-debug-chat` and, for steering-specific changes, `local-agent-steering-debug-chat`. | +| Plugin tool authorization, host tool listing, MCP tool bridge, function-call conversion | `rtk bin/lbs test run agent-runner-behavior-matrix --dry-run`; targeted plugin/MCP/tool tests | Tool execution is user-visible: add `local-agent-plugin-tool-call-debug-chat`; for MCP-specific changes add `mcp-stdio-register` then `mcp-stdio-tool-call`. | +| RAG context injection, knowledge base retrieval, resource packaging | Targeted LangBot RAG/resource tests; `rtk bin/lbs test run agent-runner-behavior-matrix --dry-run` when runner input shape changed | Runner answer should include retrieved context: add `langrag-kb-retrieve` and `local-agent-rag-debug-chat`. | +| Streaming/non-streaming adapter, provider message conversion, image or multimodal payloads | `rtk bin/lbs test run agent-runner-behavior-matrix --dry-run`; targeted provider/pipeline tests | User-visible transport changed: add `local-agent-basic-debug-chat`, `local-agent-nonstreaming-debug-chat`, or `local-agent-multimodal-debug-chat` matching the diff. | +| Only `langbot-skills` cases, probes, fixtures, references, schemas, or CLI planning code | `rtk bin/lbs validate`; relevant `rtk bin/lbs test plan ` or `rtk bin/lbs test run --dry-run` if supported | Do not run browser cases unless the edited asset itself changes the browser path being validated. | + +If a diff crosses more than two rows or touches protocol, ledger, SDK runtime, +and browser-visible runner behavior together, stop trying to hand-pick a tiny +set and use `agent-runner-release-gate`. + +## Recommended Minimal Sets + +- Protocol or normalization only: `agent-runner-fixture-contract`, + `agent-runner-behavior-matrix`, plus targeted repo unit tests. +- Ledger persistence only: `agent-runner-ledger-invariants`, + `agent-runner-ledger-stress`, `agent-runner-ledger-contention`, and + `agent-runner-async-db-readiness`; run `agent-runner-ledger-concurrency` only + when async DB readiness passes. +- SDK runtime only: `agent-runner-runtime-chaos` plus targeted SDK pytest. +- Local-agent user path: `agent-runner-release-preflight` then + `local-agent-basic-debug-chat`; add the specific RAG/tool/MCP/context/ + multimodal case only when the diff touches that contract. +- External ACP runner path: `agent-runner-release-preflight` then + `acp-agent-runner-debug-chat`. + +## Asset Rules + +- If a stable product path is missing, add one `cases/*.yaml` file. +- If a non-UI invariant or stress check is missing, add one `mode: probe` case + and one script under `probes/`. +- If the failure repeats, add one `troubleshooting/*.yaml` entry. +- Keep probe scripts deterministic. Prefer stdlib, existing repo tests, and + local fixtures over real provider calls. +- Do not mark a browser case passed from API/curl/probe evidence alone. + +## Result Classification + +- `pass`: declared checks and required evidence passed. +- `fail`: product or contract behavior is wrong. +- `env_issue`: the target could not run because a runtime dependency failed, + such as `aiosqlite.connect()` hanging before Host ledger tests start. +- `blocked`: required pipeline, plugin, credential, or fixture is missing. +- `flaky`: rerun needed because evidence shows transient instability. + +## Maintenance Gate + +After adding or editing QA assets: + +```bash +rtk bin/lbs validate +rtk bin/lbs index +rtk npm test +``` + +Commit only reusable assets and code. Do not commit `reports/evidence/*` +run output. diff --git a/skills/skills/langbot-testing/references/agent-runner-release-gate.md b/skills/skills/langbot-testing/references/agent-runner-release-gate.md new file mode 100644 index 000000000..89a75fcfa --- /dev/null +++ b/skills/skills/langbot-testing/references/agent-runner-release-gate.md @@ -0,0 +1,172 @@ +# Agent Runner Release Gate + +Use this reference when judging whether runner externalization is release-ready. The goal is not to enumerate every possible prompt. The gate covers product abilities and trust boundaries with deterministic normal-path cases, then leaves rare negative branches to unit and contract tests. + +## Coverage Strategy + +Treat the release gate as five layers: + +| Layer | Purpose | Primary Assets | +| --- | --- | --- | +| Contract gate | Protocol, SDK, auth, stores, and plugin handler behavior without a browser. | `agent-runner-fixture-contract`, `agent-runner-behavior-matrix`, `agent-runner-ledger-invariants`, `agent-runner-ledger-stress`, `agent-runner-ledger-contention`, `agent-runner-runtime-chaos`, `agent-runner-ledger-concurrency`, plus unit and contract tests in touched repos. | +| Environment preflight | Prove the selected live instance is configured for the full gate before expensive browser cases start. | `agent-runner-live-install`, `agent-runner-qa-debug-chat`, `agent-runner-release-preflight`. | +| Fixture gate | Prove deterministic plugin, RAG, multimodal, and MCP fixtures are installed and registered. | `plugin-e2e-smoke`, `langrag-kb-retrieve`, `mcp-stdio-register`. | +| Local-agent capability gate | Prove normal user-facing local-agent paths through WebUI Debug Chat. | `local-agent-gate` cases. | +| External harness gate | Prove ACP can execute an external coding agent through the WebUI Debug Chat path. | `acp-agent-runner-debug-chat`. | + +Run the full release gate with: + +```bash +rtk bin/lbs suite run agent-runner-release-gate --dry-run --json +rtk bin/lbs suite plan agent-runner-release-gate +rtk bin/lbs suite start agent-runner-release-gate --run-id agent-runner-release- +``` + +Confirm readiness and `manual_check` preconditions before removing `--dry-run` +or running the generated per-case commands. Then finish with: + +```bash +rtk bin/lbs suite report agent-runner-release-gate --evidence-dir reports/evidence/agent-runner-release- +``` + +For a quick early blocker check, run: + +```bash +rtk bin/lbs test run agent-runner-release-preflight --dry-run +``` + +For the code-level AgentRunner probes, run: + +```bash +rtk bin/lbs test run agent-runner-behavior-matrix --dry-run +rtk bin/lbs test run agent-runner-fixture-contract --dry-run +rtk bin/lbs test run agent-runner-ledger-invariants --dry-run +rtk bin/lbs test run agent-runner-ledger-stress --dry-run +rtk bin/lbs test run agent-runner-ledger-contention --dry-run +rtk bin/lbs test run agent-runner-async-db-readiness --dry-run +rtk bin/lbs test run agent-runner-ledger-concurrency --dry-run +rtk bin/lbs test run agent-runner-runtime-chaos --dry-run +rtk bin/lbs test run agent-runner-live-install --dry-run +rtk bin/lbs test run agent-runner-qa-debug-chat --dry-run +``` + +`agent-runner-behavior-matrix` executes the deterministic behavior fixture at +`fixtures/agent-runner/qa-runner-behaviors.json` through Host result +normalization. It covers normal completed output, streaming output, empty output, +malformed payloads, and controlled failure output without a model provider. + +`agent-runner-fixture-contract` imports the source fixture at +`fixtures/plugins/qa-agent-runner` and executes normal, streaming, and +controlled-failure paths with SDK protocol entities. It proves the deterministic +QA runner source is usable before a live installation/browser case uses it. +`bin/lbs fixture check` also verifies the matching +`fixtures/plugins/qa-agent-runner/dist/qa-agent-runner-0.1.0.lbpkg` package is +present and is a zip package. + +`agent-runner-live-install` uploads that package to a local LangBot backend and +checks that `qa/agent-runner` is installed and +`plugin:qa/agent-runner/default` appears in pipeline runner metadata. It is an +API integration gate, not a Debug Chat execution proof. + +`agent-runner-qa-debug-chat` is the deterministic live execution proof. It uses +a pipeline created by `scripts/e2e/ensure-qa-agent-runner-pipeline.mjs` and +expects Debug Chat to return `QA_AGENT_RUNNER_OK:` through +`plugin:qa/agent-runner/default`. + +`agent-runner-ledger-invariants` is the fast Host ledger probe. It uses +synchronous SQLite and checks run status sets, terminal status validation, +ledger table/index DDL, and a minimal insert/read path without a browser or +`aiosqlite`. + +`agent-runner-ledger-stress` is a fast deterministic stress baseline. It uses +synchronous SQLite to create 100 queued runs and simulates five runtimes claiming +each run exactly once. It does not replace async/PostgreSQL concurrency tests, +but catches schema and ordering regressions quickly. + +`agent-runner-ledger-contention` is a local write-contention probe. It uses a +file-backed SQLite database, 120 queued runs, and eight worker threads to verify +that each run is claimed exactly once under concurrent writers. It still does +not replace async/PostgreSQL concurrency tests. + +`agent-runner-async-db-readiness` checks whether direct `aiosqlite` startup is +healthy before running async Host ledger pytest probes. + +`agent-runner-ledger-concurrency` is the async Host pytest probe. It exercises +selected run ledger store/API auth tests from `LANGBOT_REPO` or `../LangBot`. +If it times out before any test result and a direct `aiosqlite.connect()` script +also hangs, classify the run with troubleshooting id +`aiosqlite-connect-hangs` instead of treating it as a browser E2E failure. + +`agent-runner-runtime-chaos` runs SDK AgentRunner runtime and pull API handler +tests from `LANGBOT_PLUGIN_SDK_REPO` or `../langbot-plugin-sdk`. +Each probe writes `automation-result.json` and probe logs under +`LBS_EVIDENCE_DIR`. + +## Normal-Path Matrix + +| Product Path | Case Coverage | What It Proves | +| --- | --- | --- | +| Authenticated WebUI session | `webui-login-state`, `agent-runner-release-preflight` | The browser profile can operate the same backend that later cases use. | +| Generic Pipeline Debug Chat | `pipeline-debug-chat` | The WebUI Debug Chat path itself works before runner-specific failures are diagnosed. | +| Deterministic QA runner install | `agent-runner-live-install` | A local `.lbpkg` AgentRunner package can install and register a runner. | +| Deterministic QA runner Debug Chat | `agent-runner-qa-debug-chat` | The installed QA runner executes through WebUI Debug Chat without a model provider. | +| Required runner plugins | `agent-runner-release-preflight` | `langbot/local-agent` and `langbot/acp-agent-runner` are visible to the host. | +| Required QA plugin tool | `plugin-e2e-smoke`, `agent-runner-release-preflight` | The deterministic `qa_plugin_echo` tool is exposed before tool-loop cases start. | +| Knowledge base fixture | `langrag-kb-retrieve`, `local-agent-rag-debug-chat` | LangRAG data is queryable and the runner inserts retrieved context. | +| Effective prompt bridge | `local-agent-effective-prompt-debug-chat` | Host prompt preprocessing reaches the runner. | +| History and compaction | `local-agent-context-compaction-debug-chat` | Runner-owned history budgeting keeps recoverable older context. | +| Streaming LLM | `local-agent-basic-debug-chat` | The default streaming path returns a visible answer. | +| Non-streaming LLM | `local-agent-nonstreaming-debug-chat` | The non-streaming adapter path returns a visible answer. | +| Plugin tool loop | `local-agent-plugin-tool-call-debug-chat` | Function-call capable models can call host plugin tools through authorization. | +| MCP registration | `mcp-stdio-register` | The deterministic stdio MCP server is registered and exposes `qa_mcp_echo`. | +| MCP tool loop | `mcp-stdio-tool-call` | Local-agent can call the registered MCP tool through the same tool loop. | +| Multimodal input | `local-agent-multimodal-debug-chat` | Image upload and structured input reach the runner. | +| Multimodal plus RAG | `local-agent-rag-multimodal-debug-chat` | RAG still works when structured image input is present. | +| ACP external harness execution | `acp-agent-runner-debug-chat` | ACP executes the configured coding agent and returns visible Debug Chat output. | + +## Status Taxonomy + +Use the same final result categories for every case: + +- `pass`: the visible UI behavior and required evidence match the case checks. +- `fail`: the configured product path is reachable, but LangBot or the runner behaves incorrectly. +- `blocked`: the test instance is not configured for this gate, for example missing pipeline, wrong runner id, missing required plugin, or unreachable ACP agent runtime. +- `env_issue`: the runtime dependency is unhealthy, for example backend down, plugin runtime down, Box down, provider route unavailable, invalid API key, or missing model ability in the selected route. +- `flaky`: the path can pass but the run hit a transient network, marketplace, upstream provider, or timing problem that needs a rerun and evidence. + +Do not count `blocked` or `env_issue` as product pass. They are useful release signals because they prevent false confidence. + +## PR Gate + +Before a browser release run, also keep the code-level gate green in the repos touched by the branch: + +```bash +# langbot-agent-runner +rtk uv run pytest -q + +# langbot-plugin-sdk +rtk uv run pytest -q + +# langbot-skills saved AgentRunner probes +rtk bin/lbs test run agent-runner-behavior-matrix --dry-run +rtk bin/lbs test run agent-runner-ledger-invariants --dry-run +rtk bin/lbs test run agent-runner-ledger-stress --dry-run +rtk bin/lbs test run agent-runner-async-db-readiness --dry-run +rtk bin/lbs test run agent-runner-ledger-concurrency --dry-run +rtk bin/lbs test run agent-runner-runtime-chaos --dry-run + +# LangBot, target the touched package first, then broaden if shared behavior changed +rtk uv run pytest -q +``` + +These tests cover field-level protocol conformance, SDK proxy behavior, auth failures, and negative branches that should not depend on a live provider. The browser gate then proves the normal user paths still compose correctly. + +## Release Decision + +For runner externalization, a release candidate is acceptable only when: + +- PR contract tests are green in every touched repo. +- `agent-runner-release-preflight` has no blockers and no environment issues. +- Every case in `agent-runner-release-gate` has a final `result.json`. +- No `pass` result is missing required evidence. +- Any skipped case is explicitly classified as `blocked` or `env_issue` with a concrete owner and follow-up. diff --git a/skills/skills/langbot-testing/references/dify-agent-runner.md b/skills/skills/langbot-testing/references/dify-agent-runner.md new file mode 100644 index 000000000..e57695950 --- /dev/null +++ b/skills/skills/langbot-testing/references/dify-agent-runner.md @@ -0,0 +1,48 @@ +# Dify AgentRunner + +Use this reference when validating `langbot/dify-agent` through LangBot WebUI. + +## Prepare Dify + +- Use a Dify Service API key from the target Dify app. Do not print the key in reports. +- For Dify Agent Chat apps, configure LangBot `app-type` as `agent`. +- Dify Agent Chat Service API may reject direct `blocking` mode with `Agent Chat App does not support blocking mode`; use streaming for direct diagnostics. + +## LangBot Configuration + +1. Open `LANGBOT_FRONTEND_URL`. +2. Navigate to `Pipelines` and open the target pipeline. +3. Open `Configuration > AI`. +4. Select runner `Dify`. +5. Configure: + - `Base URL`: usually `https://api.dify.ai/v1` + - `App Type`: `Agent` for Dify Agent Chat apps + - `API Key`: Dify Service API key + - `Base Prompt`: short neutral prompt unless the case needs a specific prompt + - `Timeout`: at least `60` when testing through proxies +6. Save before using Debug Chat. + +## Debug Chat Check + +Send a prompt with a unique sentinel: + +```text +Reply exactly with LANGBOT_DIFY_ and nothing else. +``` + +Pass only when: + +- UI shows a `Bot` message containing the sentinel. +- WebSocket history or DOM inspection confirms the sentinel is in an assistant/bot message, not only in the user message. +- Backend logs show the request completed, for example `HTTP Request: POST https://api.dify.ai/v1/chat-messages "HTTP/1.1 200 OK"` and `Conversation(0) Streaming completed`. + +## Diagnostics + +- `GET /api/v1/pipelines/{uuid}` can confirm the saved runner id is `plugin:langbot/dify-agent/default` and runner config contains `app-type`, `base-url`, and `api-key`. +- Direct Dify streaming API calls are useful only to distinguish invalid Dify credentials from LangBot runner issues. +- If Debug Chat returns `Agent runner execution failed`, inspect backend logs before changing UI settings. + +## Known Failure Signatures + +- `AttributeError: 'ActorContext' object has no attribute 'type'`: runner code is reading old actor fields; see troubleshooting `agent-runner-actor-context-fields`. +- Multiple runner options display as `默认`: component labels are ambiguous; see troubleshooting `ambiguous-runner-default-label`. diff --git a/skills/skills/langbot-testing/references/langrag-knowledge-base.md b/skills/skills/langbot-testing/references/langrag-knowledge-base.md new file mode 100644 index 000000000..00e0fb816 --- /dev/null +++ b/skills/skills/langbot-testing/references/langrag-knowledge-base.md @@ -0,0 +1,72 @@ +# LangRAG Knowledge Base + +Use this reference when validating LangRAG creation, document ingestion, retrieval, and local-agent RAG behavior. + +## Setup + +1. Install `langbot-team/LangRAG` from Marketplace if `/api/v1/knowledge/engines` has no LangRAG engine. +2. Confirm `LANGBOT_BACKEND_URL/api/v1/knowledge/engines` contains plugin id `langbot-team/LangRAG`. +3. Prefer local Chroma embedding for offline/free tests: + - Provider requester: `chroma-embedding` + - Embedding model name: `chroma-all-MiniLM-L6-v2` + +Important: a Chroma embedding entry must exist under `embedding_models`. A model accidentally created as an LLM model will appear in the wrong model selector and will not satisfy LangRAG's embedding-model field. + +## Parser Golden Case + +Use `cases/langrag-parser-golden-e2e.yaml` when validating the LangRAG + GeneralParsers integration on the current master worktree. + +Fixture: + +```text +fixtures/rag/parser-golden.html +``` + +Golden intent: + +- Start LangBot from `LANGBOT_REPO`, which should point at the master worktree for this run. +- Build and install/update local `LANGBOT_RAG_PLUGIN_REPO` and `LANGBOT_PARSER_PLUGIN_REPO`. +- Upload the HTML fixture and select GeneralParsers when the parser chooser is shown. +- Confirm retrieval returns `aurora-parser-rag-9137`, `GeneralParsers`, `LangRAG`, and the Markdown table header `| Parser field | Golden value |`. +- Confirm logs show LangRAG used external pre-parsed content instead of the internal fallback parser. + +Local install pitfall: + +- If GeneralParsers fails while installing `PyMuPDF>=1.24.0`, read `troubleshooting/plugin-dependency-install-offline.yaml`. +- The golden case can continue after the active LangBot master venv can `import fitz` and `python -m pip install --dry-run 'PyMuPDF>=1.24.0'` reports the requirement is satisfied. + +## Browser Flow + +1. Open `LANGBOT_FRONTEND_URL`. +2. Navigate to `Knowledge`. +3. Create a knowledge base. +4. Select engine `LangRAG`. +5. Select embedding model `chroma-all-MiniLM-L6-v2` or another known working embedding model. +6. Keep the index type as `Chunk` for smoke/regression tests. +7. Upload a small sentinel document. +8. Wait until the document row status is `Completed`. +9. Open `Retrieve Test` and query for the sentinel. + +Recommended fixture: + +```text +fixtures/rag/sentinel-doc.txt +``` + +## Pass Criteria + +- The created knowledge base appears in the sidebar. +- The uploaded document reaches `Completed`. +- Retrieve Test returns the uploaded document with the sentinel text. +- Browser console has no unexpected errors. + +## Local-Agent RAG Check + +After retrieval passes: + +1. Open the target pipeline. +2. In `Configuration > AI`, add the knowledge base to `Knowledge Bases`. +3. Save. +4. Open `Debug Chat`. +5. Ask for the sentinel. +6. Confirm the bot response contains the exact sentinel. diff --git a/skills/skills/langbot-testing/references/local-agent-runner-coverage.md b/skills/skills/langbot-testing/references/local-agent-runner-coverage.md new file mode 100644 index 000000000..04495f8d5 --- /dev/null +++ b/skills/skills/langbot-testing/references/local-agent-runner-coverage.md @@ -0,0 +1,74 @@ +# Local Agent Runner Coverage + +Use this matrix when judging whether the external `langbot/local-agent` plugin still behaves like the old built-in local-agent runner. + +The QA target is end-to-end behavior. UI cases prove the host, SDK, plugin runtime, and WebUI work together. Unit or component tests are still needed for negative branches that are hard to trigger reliably through a live provider. + +## Code Path Basis + +- `LangBot/src/langbot/pkg/agent/runner/context_builder.py` builds the Protocol v1 context from the event envelope: `ctx.input.text`, `ctx.input.contents`, attachments, state, resources, and runtime metadata. +- `LangBot/src/langbot/pkg/agent/runner/pipeline_adapter.py` adapts Pipeline-only fields into `ctx.adapter.extra.prompt`, `ctx.adapter.extra.params`, and optional `ctx.bootstrap.messages`. +- `LangBot/src/langbot/pkg/agent/runner/resource_builder.py` authorizes models, fallback models, rerank models, tools, and knowledge bases for the current run. +- `LangBot/src/langbot/pkg/plugin/handler.py` validates run-scoped model/tool/rerank access and calls the host model provider or tool manager with the current query. +- `langbot-local-agent/components/agent_runner/default.py` selects streaming or non-streaming execution, retrieves RAG context, builds messages, invokes models with fallback, and runs tool loops. +- `langbot-local-agent/pkg/messages.py` prefers the host effective prompt from `ctx.adapter.extra.prompt`, uses `ctx.bootstrap.messages` only as a small bootstrap window, and preserves structured/multimodal input while inserting RAG context. + +TODO: Treat `ctx.adapter.extra.prompt` as a temporary Pipeline bridge for old +local-agent behavior parity. It is not the final answer for how user plugins or +host hooks should influence agent behavior after Pipeline is replaced. + +## Minimum UI Gate + +These browser cases are the minimum gate for a local-agent migration check: + +| Case | Path Covered | Expected Behavior | +| --- | --- | --- | +| `local-agent-basic-debug-chat` | Streaming LLM invocation with effective host context | Bot returns deterministic `OK`; backend logs streaming completion. | +| `local-agent-effective-prompt-debug-chat` | PromptPreProcessing and host effective prompt handoff through `ctx.adapter.extra.prompt` | Bot returns `PROMPT_PREPROCESS_OK` from the fixture prompt probe. | +| `local-agent-context-compaction-debug-chat` | Runner-owned context budgeting and old-history compaction | Automation temporarily shrinks the runner context window, sends multi-turn Debug Chat history, and the bot still recovers the older sentinel. | +| `local-agent-rag-debug-chat` | Knowledge-base authorization, retrieval, and RAG prompt insertion | Bot returns the KB sentinel, not a generic answer. | +| `mcp-stdio-tool-call` | MCP stdio discovery, tool detail, model function calling, and tool execution | Bot returns `qa_mcp_echo:` and backend logs the MCP tool call. | +| `local-agent-plugin-tool-call-debug-chat` | Plugin tool discovery, tool detail, model function calling, and tool execution | Bot returns `qa-plugin-smoke:` and backend logs the plugin tool call. | +| `local-agent-steering-debug-chat` | Host steering claim, runner pull at turn boundary, and follow-up injection during an active tool loop | Two user messages produce one assistant response containing the steering sentinel. | +| `local-agent-multimodal-debug-chat` | Image upload, structured input contents, and multimodal runner consumption | UI shows uploaded image and bot returns `IMAGE_OK`; backend receives an image input. | +| `local-agent-rag-multimodal-debug-chat` | RAG insertion while structured image input is present | UI shows uploaded image, bot returns the KB sentinel, and backend logs the same request with `[Image]`. | +| `local-agent-nonstreaming-debug-chat` | Host non-streaming adapter path and runner non-streaming invocation | Bot returns `NONSTREAM_OK`; backend completes without the streaming-completed path. | + +## Full Coverage Matrix + +| Area | How To Cover | Pass Signal | +| --- | --- | --- | +| Effective prompt | Use the `qa-plugin-smoke` prompt probe and send `qa-effective-prompt`. | The answer follows `query.prompt.messages` and returns `PROMPT_PREPROCESS_OK`; plugin-local fallback config prompt is not used when host prompt exists. | +| Current text input | Send a deterministic text-only Debug Chat prompt. | `ctx.input.text` becomes the user text and the bot answers the text request. | +| Structured input contents | Upload an image with text in Debug Chat. | User message shows the image; backend log or request payload contains image content; model can acknowledge it. | +| Multimodal plus RAG | Run `local-agent-rag-multimodal-debug-chat`. | RAG sentinel is still retrievable and the image is not dropped from the user message; exact image-preservation inside the model message is covered by unit tests. | +| History and context compaction | Run `local-agent-context-compaction-debug-chat` with a small temporary `context-window-tokens` budget. | The runner compacts older history into `` and the final answer still recovers the older sentinel from the compacted context. | +| Streaming model invocation | Enable Debug Chat streaming and ask for `OK`. | UI receives incremental bot output and backend logs streaming completion. | +| Non-streaming model invocation | Disable Debug Chat streaming or use a non-streaming adapter path. | UI receives a final bot message and backend logs a normal response completion. | +| Model fallback before first chunk | Configure a failing primary and working fallback, preferably with a controlled test provider. | First model failure does not fail the run; fallback model produces the final answer. | +| Failure after streaming commit | Use a controlled provider that emits one chunk and then fails. | Runner reports a terminal run failure and does not fallback after partial output. | +| No authorized model | Clear model config or configure a model not in run resources. | Runner returns `runner.no_model` instead of calling an unauthorized model. | +| MCP tool call | Use `qa-local-stdio` and `qa_mcp_echo`. | Bot returns the exact `qa_mcp_echo:` result; `/api/v1/tools` contains `qa_mcp_echo`. | +| Plugin tool call | Install a fixture plugin exposing a deterministic tool and bind it to the pipeline. | Runner lists the plugin tool and can call it through the same tool loop as MCP tools. | +| Run steering | Use `local-agent-steering-debug-chat` with the fixture `qa_plugin_sleep` tool. | A follow-up sent while the sleep tool keeps the run active is claimed into the same run: two user messages, one assistant response, sentinel included. | +| Tool errors | Make the model request an unauthorized tool or invalid arguments in a controlled unit/component test. | Tool result contains an error message and the run does not bypass authorization. | +| Tool iteration limit | Use a controlled model/tool fixture that repeatedly requests more tool calls. | Runner stops with `runner.tool_loop_limit` at the configured limit. | +| Knowledge retrieval | Bind a KB containing a unique sentinel. | Bot returns the sentinel and backend logs LangRAG retrieval. | +| Legacy `knowledge-base` config | Load a pipeline config using the old single-KB field. | Runner still retrieves from the KB. | +| Rerank | Configure `rerank-model` and `rerank-top-k` with a working rerank provider. | Retrieval order follows rerank output; unauthorized or failing rerank falls back to original retrieval order. | +| Remove-think | Enable output `remove-think` on a model that emits think tags. | Final visible output omits think content on both streaming and non-streaming paths. | +| Model extra args | Configure provider/model extra args and run Debug Chat. | Host merges persisted model extra args before provider invocation. | +| Query-aware tools | Call a tool that needs the current Query/session context. | Tool receives the active query and behaves the same as it did under the built-in runner. | +| Params filtering | Add public and secret-like variables before the run. | Public params are visible to the runner; `_internal`, token, key, password, and credential fields are filtered. | +| Actor/session context | Run through Debug Chat and at least one platform adapter path. | `conversation`, `actor`, `subject`, and state scopes contain stable IDs for the current launcher and sender. | + +## Reporting Rules + +When reporting a local-agent QA result, separate these categories: + +- `Passed by UI`: path was verified through browser-visible behavior and backend/network evidence. +- `Covered by unit/component tests`: path is deterministic in tests but not practical as a live UI case. +- `Not covered`: path still needs a fixture or provider setup. +- `Environment issue`: provider channel, proxy, OAuth, or external marketplace/network problem outside the runner path. + +Do not mark the whole runner healthy based only on a single text Debug Chat response. diff --git a/skills/skills/langbot-testing/references/local-agent-runner.md b/skills/skills/langbot-testing/references/local-agent-runner.md new file mode 100644 index 000000000..63a2b7e94 --- /dev/null +++ b/skills/skills/langbot-testing/references/local-agent-runner.md @@ -0,0 +1,75 @@ +# Local Agent Runner + +Use this reference when validating the pluginized `langbot/local-agent` runner through the WebUI. + +The goal is behavior parity with the old built-in local-agent runner. The code does not need to be identical, but the visible behavior should match: effective prompt, current input, history, model selection and fallback, tool calling, knowledge retrieval, multimodal input, streaming and non-streaming output all have to reach the runner through the host and SDK. + +For path-by-path coverage, read [Local Agent Runner Coverage](local-agent-runner-coverage.md). + +## Main Surface + +- Open `LANGBOT_FRONTEND_URL`. +- Navigate to `Pipelines`. +- Open the target pipeline. +- In `Configuration > AI`, select runner `Default`. +- Configure: + - `Model`: an LLM model that is known to answer Debug Chat. + - `Knowledge Bases`: only when validating RAG behavior. + - `Rerank Model`: leave `None` unless the case explicitly tests reranking. +- Save the pipeline before using Debug Chat. + +## Debug Chat Checks + +Use `Debug Chat` as the primary local-agent validation path. + +For a basic runner check, send a deterministic prompt such as: + +```text +请只回复 OK，用于前端调试测试。 +``` + +For a RAG check, bind a knowledge base containing a unique sentinel and ask for that sentinel. + +For a tool check, ensure the target tool is visible in `/api/v1/tools`, then ask the runner to call it with deterministic input. +Avoid simultaneous fixtures with the same visible tool name. The current MCP fixture uses `qa_mcp_echo` and the plugin fixture uses `qa_plugin_echo` for unambiguous runner checks. If a run returns `qa-plugin-smoke:` during an MCP case, it exercised a plugin tool or stale registration, not the MCP tool. +If the direct MCP fixture passes but `/api/v1/tools` still shows the old MCP name, run `node scripts/e2e/mcp-stdio-register.mjs` to refresh `qa-local-stdio` before rerunning Debug Chat. + +For a multimodal check, upload a small image and ask for a deterministic acknowledgement. Prefer the bundled 64x64 red-square fixture over a 1x1 image because some model providers reject tiny images before the runner path is exercised. + +For a non-streaming check, disable the Debug Chat stream switch before sending the prompt. + +## Timeout And Tool Regression Checks + +When validating runner timeout or SDK deadline changes, confirm `Configuration > AI` renders the runner timeout field and that the saved value is the one used by the run context. The default local-agent timeout is expected to be `300` seconds unless the pipeline overrides it. + +Pair a basic Debug Chat run with a deterministic plugin tool call, for example `qa_plugin_echo`, then correlate the browser response with backend logs. A healthy run shows the tool call started and completed, and does not emit `runner.timeout`, `Action ... timed out`, `All models failed`, `Traceback`, or unexpected `ERROR` lines for the same request. + +## Minimum Regression Gate + +Run these cases before saying the pluginized local-agent behavior is healthy: + +- `local-agent-basic-debug-chat`: basic streaming model invocation. +- `local-agent-effective-prompt-debug-chat`: host effective prompt after PromptPreProcessing reaches the runner. +- `local-agent-rag-debug-chat`: LangRAG retrieval reaches the runner and affects the answer. +- `mcp-stdio-tool-call`: MCP tool discovery and local-agent tool loop. +- `local-agent-plugin-tool-call-debug-chat`: plugin tool discovery and local-agent tool loop. +- `local-agent-multimodal-debug-chat`: uploaded image reaches `ctx.input.contents`. +- `local-agent-rag-multimodal-debug-chat`: RAG retrieval still works when the same user message carries an image. +- `local-agent-nonstreaming-debug-chat`: runner works when the host adapter cannot or should not stream. + +## Pass Criteria + +- The UI shows the user message and a bot response. +- Console has no unexpected React/runtime errors. +- Backend logs show the debug-chat request completed rather than timing out in plugin/runtime calls. +- When testing RAG or tools, the answer contains the expected sentinel or tool result, not a generic explanation. +- Provider errors such as `model_not_found` or `no available channel` are environment/model availability failures. They do not prove MCP, RAG, or local-agent runner failure unless the same model works outside the tested runner path. +- A model that works for basic streaming may still fail for tool-call, multimodal, or non-streaming request shapes. Treat `runner.llm_error` and `runner.tool_loop_error` with `model_not_found`, `invalid api key`, or upstream saturation as environment/model-route failures until retested with a known-good model for that exact shape. + +## Diagnostic API + +API checks are diagnostic only: + +- `GET /api/v1/pipelines/{uuid}` confirms saved runner config. +- `GET /api/v1/tools` confirms available MCP/plugin tools. +- `GET /api/v1/knowledge/bases` confirms available knowledge bases. diff --git a/skills/skills/langbot-testing/references/mcp-stdio-testing.md b/skills/skills/langbot-testing/references/mcp-stdio-testing.md new file mode 100644 index 000000000..c1930b81a --- /dev/null +++ b/skills/skills/langbot-testing/references/mcp-stdio-testing.md @@ -0,0 +1,111 @@ +# MCP Stdio Testing + +Use this reference when validating MCP server creation, tool discovery, and local-agent tool calls. + +## Minimal Fixture + +Use the bundled test server: + +```text +fixtures/mcp/qa_mcp_echo_server.py +``` + +It exposes one tool: + +```text +qa_mcp_echo(text: str) -> str +``` + +Expected tool result: + +```text +qa_mcp_echo: +``` + +Older versions of this fixture used the visible name `qa_echo`, which collides +with the `qa-plugin-smoke` plugin tool. This fixture now uses the unique +`qa_mcp_echo` name. If a run still returns the plugin sentinel: + +```text +qa-plugin-smoke: +``` + +that proves the run used the plugin tool or a stale MCP registration, not the +current MCP fixture. Refresh the MCP server/tool registration before treating +the result as MCP coverage. + +## Browser Flow + +1. Open `LANGBOT_FRONTEND_URL`. +2. Navigate to `MCP Servers`. +3. Create a new MCP server. +4. Set mode to `Stdio`. +5. Fill the command and each argument separately: + - Command: `python` + - Arg 1: absolute path to `fixtures/mcp/qa_mcp_echo_server.py` +6. Click `Test`. +7. Submit the server. +8. Confirm the server page shows `Tools: 1` and `qa_mcp_echo`. + +Do not paste `python ...` into the command field as one string. LangBot stores `command` and `args` separately. + +## Tool Discovery Checks + +- UI: MCP detail page shows status connected and `qa_mcp_echo`. +- API diagnostic: `GET /api/v1/mcp/servers` shows `runtime_info.status=connected`. +- API diagnostic: `GET /api/v1/tools` contains `qa_mcp_echo`. + +## Provider-Independent Fixture Check + +Use this diagnostic before blaming Local Agent or a model provider: + +```bash +node scripts/e2e/mcp-stdio-fixture.mjs +``` + +It launches the bundled stdio server directly, lists tools over MCP, and calls +`qa_mcp_echo` without invoking a LangBot model. A pass proves the fixture and MCP +stdio framing work; it does not prove the provider-backed Local Agent tool loop. + +## LangBot Runtime Registration Check + +Use this diagnostic when the direct fixture passes but LangBot still lists an old +tool name or the saved MCP server may be stale: + +```bash +node scripts/e2e/mcp-stdio-register.mjs +``` + +It upserts `qa-local-stdio` through the authenticated WebUI session, points it at +the bundled `qa_mcp_echo_server.py`, then checks `/api/v1/tools` and the MCP +runtime info. A pass proves LangBot has refreshed the saved server and exposes +`qa_mcp_echo` before any model provider is involved. + +## Local-Agent Tool Call Check + +1. Open the target pipeline. +2. Confirm `Extensions` allows the MCP server, or that all MCP servers are enabled. +3. Use runner `Default` or the pluginized `langbot/local-agent` runner. +4. Select a model with function-calling ability that is known to work with tools in the current environment. +5. Open `Debug Chat`. +6. Ask: + +```text +Call the qa_mcp_echo tool with exactly this text: mcp-ok-local-agent. Return only the tool result. +``` + +Pass when the bot response contains: + +```text +qa_mcp_echo:mcp-ok-local-agent +``` + +Do not count this case as passed when the bot returns: + +```text +qa-plugin-smoke:mcp-ok-local-agent +``` + +That proves a plugin tool was called, not the MCP server. + +If the provider returns `model_not_found` or `no available channel` only when tools are supplied, switch to a known-good function-calling model before diagnosing MCP or local-agent. That failure means the selected model route is unavailable for the requested tool-call shape. diff --git a/skills/skills/langbot-testing/references/model-provider-testing.md b/skills/skills/langbot-testing/references/model-provider-testing.md new file mode 100644 index 000000000..1431d549e --- /dev/null +++ b/skills/skills/langbot-testing/references/model-provider-testing.md @@ -0,0 +1,25 @@ +# Model Provider Testing + +## Goal + +Verify that a model provider can be added, configured with a key, and tested from the WebUI. + +## Rules + +- Never print the API key or token value. +- Prefer using a test key supplied through the user's environment or secret manager. +- After saving a provider, use the WebUI test button when available. +- Confirm the provider is usable by running a small pipeline Debug Chat test, not only by checking that the form saved. + +## DeepSeek Flow + +1. Open `LANGBOT_FRONTEND_URL` from `skills/.env` or the active user-provided environment. +2. Go to `Models`. +3. Add or edit a DeepSeek provider. +4. Fill the required base URL, API key, and model fields according to the current UI. +5. Click the provider/model test button. +6. If the UI test succeeds, verify with a pipeline Debug Chat message. + +## Completion Signal + +Report the provider name, model name, UI test result, and pipeline Debug Chat result. Do not include secrets. diff --git a/skills/skills/langbot-testing/references/pipeline-debug-chat.md b/skills/skills/langbot-testing/references/pipeline-debug-chat.md new file mode 100644 index 000000000..7059c1bcd --- /dev/null +++ b/skills/skills/langbot-testing/references/pipeline-debug-chat.md @@ -0,0 +1,53 @@ +# Pipeline Debug Chat + +## Goal + +Verify that a pipeline can receive a private debug message and return a bot response through the configured frontend. + +## Path + +1. Open `LANGBOT_FRONTEND_URL` from `skills/.env` or the active user-provided environment. +2. Navigate to `Pipelines`. +3. Open the target pipeline. +4. Select `Debug Chat`. +5. Send a short deterministic prompt, for example: + + ```text + 请只回复 OK，用于前端调试测试。 + ``` + +## Success Criteria + +The UI should show: + +- A `User` message containing the prompt. +- A `Bot` message containing the expected response, for example `OK`. + +When the prompt itself contains a sentinel token, do not treat `document.body` containing that token as success. Confirm the token appears in a `Bot`/assistant message, WebSocket history entry, or backend completion log. + +For `scripts/e2e/pipeline-debug-chat.mjs`, inspect +`automation-result.json` when a sentinel is present in the prompt. A pass should +show the expected text in a new assistant message; the +`after_assistant_expected_count` value must increase beyond +`before_assistant_expected_count`. If only the user prompt contains the +sentinel, the run is a failure even when the page body contains enough total +occurrences. + +The backend log should include: + +```text +Processing request from person_websocket... +Streaming completed +``` + +## Failure Criteria + +Treat the test as failed if: + +- Only the user message appears. +- The page shows `Agent runner temporarily unavailable`. +- Backend logs contain `All models failed during streaming setup`. +- Backend logs contain `Action invoke_llm_stream call timed out`. +- Backend logs contain `Action list_plugins call timed out`. + +When failures match these signatures, read `troubleshooting.md`. diff --git a/skills/skills/langbot-testing/references/plugin-e2e-smoke.md b/skills/skills/langbot-testing/references/plugin-e2e-smoke.md new file mode 100644 index 000000000..c77cf36f7 --- /dev/null +++ b/skills/skills/langbot-testing/references/plugin-e2e-smoke.md @@ -0,0 +1,66 @@ +# Plugin E2E Smoke + +Use this reference to validate LangBot plugin behavior with a browser-first flow and API/log diagnostics. + +## Fixture + +Use the bundled local plugin source: + +```text +fixtures/plugins/qa-plugin-smoke +``` + +It registers: + +- Plugin id: `qa/plugin-smoke` +- Tool: `qa_echo(text: string)` returning `qa-plugin-smoke:` +- Tool: `qa_plugin_echo(text: string)` returning `qa-plugin-smoke:` +- Tool: `qa_plugin_sleep(seconds: number, text: string)` returning `qa-plugin-smoke:sleep::` after a bounded delay +- Page: `smoke`, with an HTML asset and a backend page API sentinel `qa-plugin-smoke-page` + +## SDK Under Test + +When validating a local SDK build, install it into the LangBot worktree virtualenv: + +```bash +cd "$LANGBOT_REPO" +uv pip install -e /absolute/path/to/langbot-plugin-sdk-test-build +uv run --no-sync python -c "import langbot_plugin, pathlib; print(pathlib.Path(langbot_plugin.__file__).resolve())" +``` + +The printed path must point into the local SDK source tree. Use `uv run --no-sync` for LangBot startup and tests; plain `uv run` may sync the lockfile and restore the PyPI package. + +## Build The Fixture + +From the fixture directory, build with the same SDK that LangBot will run: + +```bash +cd skills/langbot-testing/fixtures/plugins/qa-plugin-smoke +"$LANGBOT_REPO/.venv/bin/lbp" build +``` + +The generated zip under `dist/` is the file to upload from the WebUI. + +## Browser Flow + +1. Start or verify backend and frontend. +2. Open `LANGBOT_FRONTEND_URL`. +3. Initialize or log in to the test instance. +4. Navigate to `Plugins`. +5. Choose local plugin install and upload the generated `qa-plugin-smoke` zip. +6. Wait for the install task to finish. +7. Confirm the plugin list/detail shows `QA Plugin Smoke`, `qa_echo`, and `Smoke Page`. +8. Open the plugin extension page if it appears in the sidebar and verify it renders the sentinel text. + +## Diagnostic Checks + +Use API checks only to confirm what the UI exercised: + +- `GET /api/v1/plugins` contains `qa/plugin-smoke` with initialized status. +- `GET /api/v1/tools` contains `qa_echo`, `qa_plugin_echo`, and `qa_plugin_sleep`. +- `POST /api/v1/plugins/qa/plugin-smoke/page-api` with `page_id=smoke`, `endpoint=/ping`, `method=GET` returns `qa-plugin-smoke-page`. +- Backend logs include `Connected to plugin runtime` and no `Action ... call timed out` entries. + +## Cleanup + +Delete `qa/plugin-smoke` through the WebUI or `DELETE /api/v1/plugins/qa/plugin-smoke?delete_data=true` after recording results. diff --git a/skills/skills/langbot-testing/references/sandbox-skill-authoring.md b/skills/skills/langbot-testing/references/sandbox-skill-authoring.md new file mode 100644 index 000000000..db9b82647 --- /dev/null +++ b/skills/skills/langbot-testing/references/sandbox-skill-authoring.md @@ -0,0 +1,136 @@ +# Sandbox Skill Authoring + +## Goal + +Verify that Local Agent can use sandbox tools to create, register, activate, and use a LangBot skill package through the same path a user would exercise in Debug Chat. + +This flow applies to Docker, nsjail, and E2B backends. API calls are useful diagnostics, but the primary pass/fail signal is the model-driven Debug Chat tool sequence. + +## Preconditions + +1. Read `../.env` and use `LANGBOT_FRONTEND_URL` and `LANGBOT_BACKEND_URL`. +2. Start LangBot with the intended backend: + - `BOX_BACKEND=e2b` when validating E2B. + - `BOX_BACKEND=nsjail` when validating nsjail. + - `BOX_BACKEND=local` or `docker` when validating local container fallback. +3. Confirm `/api/v1/box/status` reports `available: true` and the expected backend name. +4. Confirm Debug Chat uses a model with function-calling ability. +5. Confirm backend logs say native sandbox tools are available. + +Do not store sandbox provider keys, JWTs, OAuth tokens, or localStorage values in the case or notes. + +## Debug Chat Prompt Pattern + +Use a unique skill name per run, for example: + +```text +lb-sandbox-agent-e2e- +``` + +Send a prompt that requires the model to do all of the following: + +1. Use `exec` to create a multi-file skill under `/workspace/`. +2. Include at least: + - `SKILL.md` + - `scripts/use.py` + - `data/input.json` +3. In the same first `exec`, run the script and verify a deterministic marker such as: + + ```text + SANDBOX_COMPLEX_SKILL_OK sum=10 product=24 + ``` + +4. Call `register_skill` with `path=/workspace/`. +5. Call `activate` with `skill_name=`. +6. Call `exec` with `workdir=/workspace/.skills/` and run: + + ```bash + python3 scripts/use.py && echo SANDBOX_ACTIVATED_WRITEBACK_OK > activated_writeback.txt && cat activated_writeback.txt + ``` + +7. Require the final answer to contain only an explicit success marker: + + ```text + E2E_OK: + ``` + +Keep the test script robust to working-directory changes. Prefer resolving data paths from `__file__`: + +```python +from pathlib import Path +data_path = Path(__file__).resolve().parent.parent / "data" / "input.json" +``` + +## Success Criteria + +The UI should show an assistant final response containing `E2E_OK:`. + +Backend logs should show: + +- `exec tool invoked` +- `register_skill` +- `activate` +- a second `exec` whose workdir is `/workspace/.skills/` +- `backend=e2b`, `backend=nsjail`, or the expected local backend + +After the run, verify the skill store through the UI or API: + +- Skill root lists `SKILL.md`, `scripts`, `data`, and `activated_writeback.txt`. +- `scripts/use.py` is readable. +- `data/input.json` is readable. +- `activated_writeback.txt` contains `SANDBOX_ACTIVATED_WRITEBACK_OK`. +- `/api/v1/box/errors` is empty. + +## Existing Skill Edit Variant + +The base `sandbox-skill-authoring-e2e` case only proves create, register, +activate, and use. To prove that an already activated skill can be modified, +run `sandbox-skill-authoring-edit-existing-e2e` or use its prompt pattern. + +The edit variant must include these additional checks: + +- The second `exec` uses `workdir=/workspace/.skills/`. +- The second `exec` overwrites `SKILL.md`, `data/input.json`, and + `scripts/use.py` under the activated skill path. +- The modified script prints a deterministic marker such as: + + ```text + SANDBOX_SKILL_MODIFIED_OK sum=11 product=56 marker= + ``` + +- `grep -q SKILL.md scripts/use.py data/input.json` succeeds + in the same activated-path command. +- Filesystem evidence under the Box-managed skill store shows the updated + marker, not only the original create marker. + +If the model stops after activation and only reruns the original script, treat +that run as a failed edit-existing E2E even when create/register/activate +succeeded. + +## Diagnostic Checks + +Use these only after the model-driven Debug Chat flow fails: + +- `/api/v1/box/status` to confirm backend selection and recent errors. +- `/api/v1/box/sessions` to check leaked or conflicting sessions. +- Direct backend probes to separate provider credentials from LangBot integration. +- Filesystem inspection under the configured Box-managed skill store. + +For E2B raw HTTP diagnostics, include a valid template id such as `base`; a missing template can produce schema validation errors that are unrelated to authentication. + +## Known Pitfalls + +- When Box is available, skills may be owned by the Box runtime and stored in Box-managed skill storage. Do not assume `data/skills` is the active source of truth. +- Public E2B does not provide local bind mounts. Main workspace and activated skill extra mounts must be synchronized into and back out of the E2B sandbox. +- Session metadata should keep LangBot logical paths such as `/workspace`; storing provider-internal paths can make later requests look incompatible. +- nsjail versions differ. Some expose only `--disable_clone_new*` flags and use `--bindmount` instead of `--rw_bind`. +- On WSL, cgroup v2 may exist but not be writable. The backend should warn and fall back to rlimits rather than fail the sandbox. +- If `ALL_PROXY` uses a SOCKS URL and `socksio` is not installed, some Python HTTP clients can fail during startup. Prefer consistent HTTP proxy variables unless SOCKS support is installed. + +## Related Troubleshooting + +- `sandbox-native-tools-unavailable` +- `e2b-extra-mount-sync-missing` +- `box-session-conflict-logical-metadata` +- `nsjail-cli-compatibility` +- `socks-proxy-without-socksio` diff --git a/skills/skills/langbot-testing/references/troubleshooting.md b/skills/skills/langbot-testing/references/troubleshooting.md new file mode 100644 index 000000000..286990c59 --- /dev/null +++ b/skills/skills/langbot-testing/references/troubleshooting.md @@ -0,0 +1,91 @@ +# Troubleshooting + +Troubleshooting entries are now managed as structured YAML under `../troubleshooting/`. Use `bin/lbs trouble add langbot-testing ...` to add new entries when a new failure mode is confirmed. + +This Markdown file is a human-readable legacy summary. Prefer the YAML entries for automation. + +## plugin-runtime-timeout: Plugin runtime actions time out + +Structured entry: `../troubleshooting/plugin-runtime-timeout.yaml` + +Date: 2026-05-16 + +### Symptom + +The WebUI can send a Debug Chat message, but the bot response is missing or says `Agent runner temporarily unavailable`. Backend logs may include `Action list_plugins call timed out`, `Action list_agent_runners call timed out`, or `Action invoke_llm_stream call timed out`. + +### Likely Cause + +An old `langbot_plugin` runtime process survived a backend restart, or multiple runtime processes are active at once. The backend is running, but plugin actions do not get a valid response. + +### Fix + +Stop the LangBot backend and any orphaned `langbot_plugin.cli` runtime processes, confirm the configured backend URL is free/reachable as appropriate, then start LangBot again. A healthy startup logs `Connected to plugin runtime`, mounts `langbot/local-agent`, and initializes the default agent runner. + +### Verification + +Run a pipeline Debug Chat prompt. The UI should show a Bot response, and backend logs should include `Streaming completed`. + +## proxy-env-mismatch: Uppercase and lowercase proxy variables differ + +Structured entry: `../troubleshooting/proxy-env-mismatch.yaml` + +Date: 2026-05-16 + +### Symptom + +External model calls time out even though the proxy appears to be configured. Environment inspection shows uppercase proxy variables using `127.0.0.1:7890` while lowercase variables still point to an old WSL gateway such as `172.30.144.1:7890`. + +### Likely Cause + +Different libraries read different proxy variable names. If lowercase and uppercase values differ, model/provider calls can use the wrong proxy. + +### Fix + +Start LangBot with consistent `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, `http_proxy`, `https_proxy`, `all_proxy`, `NO_PROXY`, and `no_proxy` values. + +### Verification + +Backend startup should no longer show old proxy addresses, and a pipeline Debug Chat should complete with a Bot response. + +## mcp-stdio-args-not-applied: MCP Stdio test runs only the command without arguments + +Structured entry: `../troubleshooting/mcp-stdio-args-not-applied.yaml` + +The MCP form may appear filled correctly, but the backend logs show bare `uv` help text or `Connection closed`. Confirm command and args are split correctly, then check whether the form test handler reads the latest stdio args. + +## survey-widget-blocks-debug-chat: Survey widget blocks Debug Chat controls + +Structured entry: `../troubleshooting/survey-widget-blocks-debug-chat.yaml` + +If Playwright cannot click Debug Chat `Send` because a fixed bottom-right element intercepts pointer events, close or minimize the survey widget before continuing the browser test. + +## dynamic-form-missing-config-id: Dynamic form fields have no stable key + +Structured entry: `../troubleshooting/dynamic-form-missing-config-id.yaml` + +Dynamic plugin/runner schemas can trigger React unique-key warnings when schema items lack `id`. Prefer backend-generated stable ids and keep frontend fallback keys. + +## pipeline-form-controlled-warning: Pipeline form input switches from uncontrolled to controlled + +Structured entry: `../troubleshooting/pipeline-form-controlled-warning.yaml` + +Pipeline edit forms should initialize string fields to empty strings and coalesce loaded nullable values before rendering inputs. + +## marketplace-network-flaky: Marketplace requests are flaky but plugin data may still load + +Structured entry: `../troubleshooting/marketplace-network-flaky.yaml` + +Marketplace icon/tag/recommendation requests can fail while plugin cards are already visible. Retry first, and use backend component endpoints only to confirm installation results. + +## agent-runner-actor-context-fields: AgentRunner reads old actor fields + +Structured entry: `../troubleshooting/agent-runner-actor-context-fields.yaml` + +If Debug Chat fails and logs include `ActorContext` missing `type` or `id`, update runner code to use protocol v1 fields `actor_type` and `actor_id`. + +## ambiguous-runner-default-label: Runner selector shows multiple Default options + +Structured entry: `../troubleshooting/ambiguous-runner-default-label.yaml` + +Keep `metadata.name: default` for stable component ids, but set user-facing `metadata.label` to provider-specific names such as `Dify` or `本地 Agent`. diff --git a/skills/skills/langbot-testing/references/web-ui-testing.md b/skills/skills/langbot-testing/references/web-ui-testing.md new file mode 100644 index 000000000..8a2421f13 --- /dev/null +++ b/skills/skills/langbot-testing/references/web-ui-testing.md @@ -0,0 +1,37 @@ +# Web UI Testing + +## Baseline + +- Read shared defaults from `skills/.env`. +- Open `LANGBOT_FRONTEND_URL`. +- Use `LANGBOT_BACKEND_URL` for backend/API/log checks. +- Use Playwright MCP or another browser automation tool with a persisted authenticated profile. + +## Workflow + +1. Start or verify the backend. +2. Start or verify the selected frontend. +3. Open `LANGBOT_FRONTEND_URL`. +4. Confirm the sidebar shows the logged-in user instead of the login page. +5. Navigate through the target flow with role/text selectors where possible. +6. Check browser console errors, visible UI state, and backend logs. + +## Browser Vs API Boundary + +Use browser automation as the acceptance path for WebUI cases. API or curl checks are useful for readiness, saved config inspection, and log correlation, but they do not cover login state, form rendering, frontend save behavior, websocket streaming, or console regressions. + +For a UI case, curl can support the report but cannot make the case pass by itself. A passing report should include the visible browser result and any backend/API diagnostics that explain the same run. + +## Authentication Notes + +If the user logged in on one origin but `LANGBOT_FRONTEND_URL` still shows `/login`, copy only the auth state needed for the selected origin. Do not print token values. + +## Completion Signal + +Report: + +- URL tested. +- User action performed. +- Visible result. +- Backend or network confirmation. +- Any console/backend errors that remain. diff --git a/skills/skills/langbot-testing/suites/agent-runner-release-gate.yaml b/skills/skills/langbot-testing/suites/agent-runner-release-gate.yaml new file mode 100644 index 000000000..a2c150669 --- /dev/null +++ b/skills/skills/langbot-testing/suites/agent-runner-release-gate.yaml @@ -0,0 +1,38 @@ +id: agent-runner-release-gate +title: "Agent runner externalization release gate" +description: "Release gate for runner externalization: pytest probes, environment preflight, fixture registration, local-agent capability paths, and ACP external harness execution." +type: release_gate +priority: p0 +tags: + - agent-runner + - release-gate + - local-agent + - external-runner +cases: + - agent-runner-fixture-contract + - agent-runner-behavior-matrix + - agent-runner-ledger-invariants + - agent-runner-async-db-readiness + - agent-runner-ledger-stress + - agent-runner-ledger-contention + - agent-runner-ledger-concurrency + - agent-runner-runtime-chaos + - agent-runner-live-install + - agent-runner-qa-debug-chat + - agent-runner-release-preflight + - webui-login-state + - pipeline-debug-chat + - qa-plugin-smoke-live-install + - plugin-e2e-smoke + - langrag-kb-retrieve + - local-agent-basic-debug-chat + - local-agent-effective-prompt-debug-chat + - local-agent-context-compaction-debug-chat + - local-agent-rag-debug-chat + - local-agent-plugin-tool-call-debug-chat + - mcp-stdio-register + - mcp-stdio-tool-call + - local-agent-nonstreaming-debug-chat + - local-agent-multimodal-debug-chat + - local-agent-rag-multimodal-debug-chat + - acp-agent-runner-debug-chat diff --git a/skills/skills/langbot-testing/suites/core-smoke.yaml b/skills/skills/langbot-testing/suites/core-smoke.yaml new file mode 100644 index 000000000..d0cfe9127 --- /dev/null +++ b/skills/skills/langbot-testing/suites/core-smoke.yaml @@ -0,0 +1,13 @@ +id: core-smoke +title: "Core browser smoke suite" +description: "Fast browser-first checks for login state, Pipeline Debug Chat, and the basic local-agent runner path." +type: smoke +priority: p0 +tags: + - smoke + - browser + - p0 +cases: + - webui-login-state + - pipeline-debug-chat + - local-agent-basic-debug-chat diff --git a/skills/skills/langbot-testing/suites/local-agent-gate.yaml b/skills/skills/langbot-testing/suites/local-agent-gate.yaml new file mode 100644 index 000000000..42eee0228 --- /dev/null +++ b/skills/skills/langbot-testing/suites/local-agent-gate.yaml @@ -0,0 +1,21 @@ +id: local-agent-gate +title: "Local Agent runner regression gate" +description: "High-signal local-agent runner checks covering prompt bridge, RAG, context compaction, plugin tools, MCP tools, non-streaming, and multimodal paths." +type: regression +priority: p1 +tags: + - local-agent + - agent-runner + - regression +cases: + - local-agent-basic-debug-chat + - qa-plugin-smoke-live-install + - local-agent-effective-prompt-debug-chat + - local-agent-context-compaction-debug-chat + - local-agent-rag-debug-chat + - local-agent-plugin-tool-call-debug-chat + - local-agent-steering-debug-chat + - mcp-stdio-tool-call + - local-agent-nonstreaming-debug-chat + - local-agent-multimodal-debug-chat + - local-agent-rag-multimodal-debug-chat diff --git a/skills/skills/langbot-testing/troubleshooting/agent-runner-actor-context-fields.yaml b/skills/skills/langbot-testing/troubleshooting/agent-runner-actor-context-fields.yaml new file mode 100644 index 000000000..48c980197 --- /dev/null +++ b/skills/skills/langbot-testing/troubleshooting/agent-runner-actor-context-fields.yaml @@ -0,0 +1,22 @@ +id: agent-runner-actor-context-fields +title: "AgentRunner reads old actor.type and actor.id fields" +date: 2026-05-17 +symptoms: + - "Pipeline Debug Chat shows Agent runner execution failed." + - "Backend logs show an AttributeError from an AgentRunner plugin." +patterns: + - "AttributeError: 'ActorContext' object has no attribute 'type'" + - "AttributeError: 'ActorContext' object has no attribute 'id'" + - "return f\"{actor.type}_{actor.id}\"" +likely_causes: + - "The plugin was written against old actor field names." + - "Protocol v1 ActorContext uses actor_type and actor_id." +fix_steps: + - "Update runner code to read actor.actor_type and actor.actor_id." + - "Keep getattr fallback to type/id only if compatibility with older host data is required." + - "Restart LangBot or the plugin runtime so the updated plugin code is loaded." + - "Add a regression test that builds AgentRunContext with ActorContext(actor_type=..., actor_id=...)." +verification: "Run dify-agent-debug-chat or another AgentRunner Debug Chat and confirm the assistant/bot message contains the expected sentinel while backend logs show Streaming completed." +related_cases: + - dify-agent-debug-chat + - pipeline-debug-chat diff --git a/skills/skills/langbot-testing/troubleshooting/aiosqlite-connect-hangs.yaml b/skills/skills/langbot-testing/troubleshooting/aiosqlite-connect-hangs.yaml new file mode 100644 index 000000000..796a6dbac --- /dev/null +++ b/skills/skills/langbot-testing/troubleshooting/aiosqlite-connect-hangs.yaml @@ -0,0 +1,23 @@ +id: aiosqlite-connect-hangs +title: "aiosqlite connect hangs before ledger pytest starts" +category: env_issue +symptoms: + - "AgentRunner ledger pytest probe times out after collecting tests but before reporting a test result." + - "pytest stdout stops at a line like tests/unit_tests/agent/test_run_ledger_store.py." + - "A direct aiosqlite.connect(':memory:') script prints its first line and then hangs." +patterns: + - "pytest timed out after" + - "tests/unit_tests/agent/test_run_ledger_store.py" + - "aiosqlite.connect" +likely_causes: + - "The current execution environment cannot complete aiosqlite connection startup." + - "The failure is below LangBot ledger logic when direct aiosqlite connection also hangs." + - "Threading and synchronous sqlite can still work, so this is narrower than a general Python or SQLite outage." +fix_steps: + - "Run `rtk bin/lbs test run agent-runner-ledger-invariants` first for fast ledger schema/status coverage that does not use aiosqlite." + - "Classify `agent-runner-ledger-concurrency` timeout as env_issue when direct aiosqlite.connect also hangs." + - "Retest the async pytest target in an environment where aiosqlite connect completes." +verification: "A direct aiosqlite.connect(':memory:') script completes, then `rtk bin/lbs test run agent-runner-ledger-concurrency` no longer times out." +related_cases: + - agent-runner-ledger-concurrency + - agent-runner-ledger-invariants diff --git a/skills/skills/langbot-testing/troubleshooting/ambiguous-runner-default-label.yaml b/skills/skills/langbot-testing/troubleshooting/ambiguous-runner-default-label.yaml new file mode 100644 index 000000000..74e03df3d --- /dev/null +++ b/skills/skills/langbot-testing/troubleshooting/ambiguous-runner-default-label.yaml @@ -0,0 +1,22 @@ +id: ambiguous-runner-default-label +title: "AgentRunner selector shows multiple Default or 默认 options" +date: 2026-05-17 +symptoms: + - "The Pipeline AI runner selector shows multiple options named Default or 默认." + - "Users must inspect plugin ids to distinguish Dify, Local Agent, or other runners." +patterns: + - "metadata.name: default" + - "label.zh_Hans: 默认" + - "label.en_US: Default" +likely_causes: + - "AgentRunner component ids are commonly named default, but the user-facing metadata.label was also left generic." + - "The frontend displays metadata.label as the primary option label." +fix_steps: + - "Keep metadata.name as default if the plugin component id is intended to remain stable." + - "Change metadata.label to the provider or runner family name, such as Dify, Local Agent, Coze, DashScope, Langflow, n8n, or Tbox." + - "Restart LangBot or plugin runtime to refresh runner metadata cache." + - "Check /api/v1/pipelines/_/metadata and the WebUI runner selector." +verification: "Runner options are distinguishable by visible label while their ids remain stable, such as plugin:langbot/dify-agent/default." +related_cases: + - dify-agent-debug-chat + - local-agent-rag-debug-chat diff --git a/skills/skills/langbot-testing/troubleshooting/backend-not-listening.yaml b/skills/skills/langbot-testing/troubleshooting/backend-not-listening.yaml new file mode 100644 index 000000000..37df32df0 --- /dev/null +++ b/skills/skills/langbot-testing/troubleshooting/backend-not-listening.yaml @@ -0,0 +1,29 @@ +id: backend-not-listening +title: "LangBot backend URL has no listening service" +date: 2026-05-30 +symptoms: + - "The frontend URL opens, but backend-dependent WebUI cases are blocked." + - "env doctor fails only for LANGBOT_BACKEND_URL." + - "curl to the backend port fails before receiving an HTTP status." +patterns: + - "LANGBOT_BACKEND_URL" + - "no HTTP service reachable" + - "Failed to connect to 127.0.0.1 port 5300" + - "TypeError: fetch failed" + - "Couldn't connect to server" +likely_causes: + - "The LangBot backend process is not running." + - "The frontend dev server is running by itself, so the browser opens but API calls cannot work." + - "The backend was started in a disposable foreground session and stopped when that session ended." + - "A detached startup command exited early and no process remains bound to the configured port." +fix_steps: + - "Run bin/lbs env doctor and confirm whether LANGBOT_BACKEND_URL reports no listener." + - "Start the backend from LANGBOT_REPO with uv run main.py." + - "Wait for Running on http://0.0.0.0:5300 and Connected to plugin runtime." + - "Verify ss -ltnp shows the configured backend port." + - "Re-run bin/lbs env doctor before starting browser QA." +verification: "LANGBOT_BACKEND_URL returns an HTTP status, env doctor passes the backend check, and backend logs show Connected to plugin runtime." +related_cases: + - pipeline-debug-chat + - webui-login-state + - local-agent-basic-debug-chat diff --git a/skills/skills/langbot-testing/troubleshooting/box-session-conflict-logical-metadata.yaml b/skills/skills/langbot-testing/troubleshooting/box-session-conflict-logical-metadata.yaml new file mode 100644 index 000000000..7443fda1f --- /dev/null +++ b/skills/skills/langbot-testing/troubleshooting/box-session-conflict-logical-metadata.yaml @@ -0,0 +1,23 @@ +id: box-session-conflict-logical-metadata +title: "BoxSessionConflictError after a successful first exec" +date: 2026-05-18 +symptoms: + - "The first sandbox exec succeeds, but the next exec in the same Debug Chat fails." + - "The model repeatedly retries commands and eventually reports a session conflict." + - "Skill registration or activation cannot continue after the first command." +patterns: + - "BoxSessionConflictError" + - "already exists with image=host" + - "already exists with mount_path=/home/user/workspace" +likely_causes: + - "Backend session metadata stored provider-specific values instead of LangBot logical spec values." + - "E2B stored adapted /home/user/workspace as mount_path, while later specs still use /workspace." + - "nsjail stored image=host, while later specs still use the configured logical sandbox image." +fix_steps: + - "Store logical spec.mount_path in BoxSessionInfo for E2B session metadata." + - "Store spec.image in nsjail BoxSessionInfo even though nsjail executes against host-mounted system paths." + - "Keep provider-specific path rewriting inside the backend command execution layer." + - "Restart LangBot to clear stale in-memory sessions after applying the fix." +verification: "Run two or more exec calls in the same Debug Chat session. The second call should reuse the session instead of raising BoxSessionConflictError." +related_cases: + - sandbox-skill-authoring-e2e diff --git a/skills/skills/langbot-testing/troubleshooting/debug-chat-history-contaminates-automation.yaml b/skills/skills/langbot-testing/troubleshooting/debug-chat-history-contaminates-automation.yaml new file mode 100644 index 000000000..3b0156c94 --- /dev/null +++ b/skills/skills/langbot-testing/troubleshooting/debug-chat-history-contaminates-automation.yaml @@ -0,0 +1,25 @@ +id: debug-chat-history-contaminates-automation +title: "Old Debug Chat messages contaminate automation assertions" +date: 2026-05-21 +symptoms: + - "A browser automation run reports Agent runner temporarily unavailable even though the latest bot response contains the expected text." + - "The screenshot shows an old failure bubble above the latest successful exchange." + - "The automation scans the whole document body instead of comparing only new messages or new failure counts." +patterns: + - "Agent runner temporarily unavailable" + - "Expected text appeared enough times" + - "Debug Chat displayed a known failure signal" +likely_causes: + - "Debug Chat history persists across runs and old failed messages remain visible." + - "The automation checks failure text anywhere in document.body after the run." + - "The automation does not compare failure counts before and after sending the prompt." +fix_steps: + - "Before sending the prompt, record counts for expected text and known failure text." + - "After sending the prompt, require the expected text count to increase enough for the user prompt plus bot response." + - "Treat known failure text as a new failure only if its count increased during this run." + - "For manual review, inspect the newest user/bot message pair rather than the whole chat transcript." +verification: "A latest successful bot response is not failed by old failure bubbles already present in the Debug Chat history." +related_cases: + - pipeline-debug-chat + - local-agent-plugin-tool-call-debug-chat + - mcp-stdio-tool-call diff --git a/skills/skills/langbot-testing/troubleshooting/dynamic-form-missing-config-id.yaml b/skills/skills/langbot-testing/troubleshooting/dynamic-form-missing-config-id.yaml new file mode 100644 index 000000000..6504b9a20 --- /dev/null +++ b/skills/skills/langbot-testing/troubleshooting/dynamic-form-missing-config-id.yaml @@ -0,0 +1,21 @@ +id: dynamic-form-missing-config-id +title: "Dynamic form fields have no stable key" +date: 2026-05-17 +symptoms: + - "Browser console shows a React warning about children in a list needing a unique key." + - "The warning appears when rendering plugin or runner dynamic form schemas." +patterns: + - "Each child in a list should have a unique key prop" + - "DynamicFormComponent" + - "config schema" +likely_causes: + - "A plugin component schema item has name but no id." + - "The frontend maps dynamic config items using config.id as the only key." +fix_steps: + - "Prefer adding stable ids to backend metadata derived from component id and field name." + - "Keep a frontend fallback key using config.id, config.name, then list index for defensive rendering." + - "Reload the affected form and inspect console warnings." +verification: "The dynamic form renders runner/plugin fields without React unique key warnings." +related_cases: + - langrag-kb-retrieve + - local-agent-rag-debug-chat diff --git a/skills/skills/langbot-testing/troubleshooting/e2b-extra-mount-sync-missing.yaml b/skills/skills/langbot-testing/troubleshooting/e2b-extra-mount-sync-missing.yaml new file mode 100644 index 000000000..c739750c3 --- /dev/null +++ b/skills/skills/langbot-testing/troubleshooting/e2b-extra-mount-sync-missing.yaml @@ -0,0 +1,23 @@ +id: e2b-extra-mount-sync-missing +title: "Activated skill files are missing or not written back on E2B" +date: 2026-05-18 +symptoms: + - "register_skill succeeds but the activated skill cannot be used from /workspace/.skills/." + - "A file written inside /workspace/.skills/ does not appear in the Box-managed skill store." + - "The same activate-and-exec flow works on Docker or nsjail but fails on E2B." +patterns: + - "No such file or directory: /workspace/.skills" + - "Skill file not found after activated skill exec" + - "E2B command runs under /home/user/workspace while LangBot paths use /workspace" +likely_causes: + - "Public E2B does not provide host bind mounts, so LangBot must upload and download mounted directories." + - "Only the main workspace mount was synced, while activated skills are passed as extra_mounts." + - "Writeback from writable extra_mounts was not downloaded after command completion." +fix_steps: + - "Sync the main host_path and each extra_mount into the adapted E2B path before command execution." + - "After command execution, sync writable host_path and writable extra_mounts back to host storage." + - "Rewrite logical /workspace command paths to E2B's internal writable path only at execution time." + - "Keep session metadata mount_path as LangBot's logical path, not the E2B-internal path." +verification: "Run sandbox-skill-authoring-e2e with BOX_BACKEND=e2b and confirm activated_writeback.txt is readable from the registered skill package after the second exec." +related_cases: + - sandbox-skill-authoring-e2e diff --git a/skills/skills/langbot-testing/troubleshooting/local-agent-model-route-unavailable.yaml b/skills/skills/langbot-testing/troubleshooting/local-agent-model-route-unavailable.yaml new file mode 100644 index 000000000..779654492 --- /dev/null +++ b/skills/skills/langbot-testing/troubleshooting/local-agent-model-route-unavailable.yaml @@ -0,0 +1,35 @@ +id: local-agent-model-route-unavailable +title: "Local Agent model route is unavailable for the requested run shape" +category: env_issue +date: 2026-05-21 +symptoms: + - "Debug Chat shows Agent runner temporarily unavailable after the user message is sent." + - "Basic streaming prompts may work, but tool-call, non-streaming, or multimodal prompts fail with the same runner and pipeline." + - "The failure happens after the local-agent runner starts, not during plugin discovery." +patterns: + - "runner.llm_error" + - "runner.tool_loop_error" + - "model_not_found" + - "no available channel for model" + - "invalid api key" + - "当前分组上游负载已饱和" + - "All models failed during streaming setup" +likely_causes: + - "The selected model route is unavailable in the current LangBot Space or upstream group." + - "The selected model works for plain chat but is not available for tool-call, multimodal, or non-streaming request shapes." + - "The provider credential or quota is invalid for the non-streaming path." + - "The selected model has function-call or vision metadata, but the upstream distributor cannot currently serve that model." +fix_steps: + - "First rerun a basic local-agent Debug Chat prompt on the same pipeline to confirm the runner host path still works." + - "Switch to a known-good model for the exact request shape being tested: plain chat, tool call, multimodal, or non-streaming." + - "When tool-call cases fail, retest with a model that is known to support function calling in the active environment." + - "When multimodal cases fail, retest with a model route that is known to accept image content." + - "When non-streaming fails with invalid api key, verify the provider credential used by the non-streaming requester path." + - "Do not classify this as an MCP, RAG, or local-agent runner implementation failure until the same model route works for the requested request shape outside the failing case." +verification: "The same case produces the expected bot-visible sentinel or tool result, and backend logs show request completion without runner.llm_error, runner.tool_loop_error, or All models failed." +related_cases: + - local-agent-basic-debug-chat + - local-agent-plugin-tool-call-debug-chat + - mcp-stdio-tool-call + - local-agent-multimodal-debug-chat + - local-agent-nonstreaming-debug-chat diff --git a/skills/skills/langbot-testing/troubleshooting/marketplace-network-flaky.yaml b/skills/skills/langbot-testing/troubleshooting/marketplace-network-flaky.yaml new file mode 100644 index 000000000..649368613 --- /dev/null +++ b/skills/skills/langbot-testing/troubleshooting/marketplace-network-flaky.yaml @@ -0,0 +1,20 @@ +id: marketplace-network-flaky +title: "Marketplace requests are flaky but plugin data may still load" +date: 2026-05-17 +symptoms: + - "Marketplace page shows failed plugin list toasts or console network errors." + - "Some plugin cards still render and can be installed." +patterns: + - "Failed to get plugin list" + - "ERR_NETWORK_CHANGED" + - "https://space.langbot.app/api/v1/marketplace" +likely_causes: + - "Transient network/proxy instability while loading marketplace search, tags, icons, releases, or recommendation lists." + - "Non-critical marketplace resource requests fail after the main plugin list has already rendered." +fix_steps: + - "Retry the Marketplace page before treating the flow as blocked." + - "If the target plugin card is visible, continue through the UI install flow." + - "Use /api/v1/plugins and /api/v1/knowledge/engines only to confirm install result." +verification: "The target plugin appears in Installed Plugins or the relevant component endpoint, such as /api/v1/knowledge/engines for LangRAG." +related_cases: + - langrag-kb-retrieve diff --git a/skills/skills/langbot-testing/troubleshooting/mcp-stdio-args-not-applied.yaml b/skills/skills/langbot-testing/troubleshooting/mcp-stdio-args-not-applied.yaml new file mode 100644 index 000000000..c079d70e5 --- /dev/null +++ b/skills/skills/langbot-testing/troubleshooting/mcp-stdio-args-not-applied.yaml @@ -0,0 +1,36 @@ +id: mcp-stdio-args-not-applied +title: "MCP Stdio test runs only the command without arguments" +date: 2026-05-17 +symptoms: + - "The MCP server Test button fails even though the same command works in a terminal." + - "Backend logs show uv help text or another bare command output instead of starting the MCP server." + - "Backend logs show uv: not found after the fixture path is accepted by Box." + - "Backend logs show host_path is outside allowed_mount_roots before the stdio process starts." + - "The task exception is Connection failed, please check URL or Connection closed." + - "Backend startup logs say qa_mcp_echo_server.py cannot be opened from the LangBot repo root." +patterns: + - "An extremely fast Python package manager." + - "Usage: uv [OPTIONS] " + - "/bin/sh: 1: uv: not found" + - "host_path is outside allowed_mount_roots" + - "McpError: Connection closed" + - "mcp-test-_" + - "can't open file '.../LangBot/qa_mcp_echo_server.py'" +likely_causes: + - "Box refuses to mount the bundled fixture directory because it is not listed in box.local.allowed_mount_roots." + - "The saved MCP server command uses uv, but uv is not available inside the Box runtime PATH." + - "The WebUI did not pass the current stdio args to the test request." + - "The MCP form exposed a stale imperative test handler from an old render." + - "The user entered the whole command line into the command field instead of splitting command and args." + - "An old qa-local-stdio database record still points at a temporary LangBot root script instead of the bundled skills fixture." +fix_steps: + - "Confirm the fixture directory is listed in box.local.allowed_mount_roots for the local LangBot data config." + - "Confirm command is only the executable, normally python for this fixture." + - "Confirm args are separate entries; for this fixture use the server script path as the single arg." + - "Confirm the fixture script is the current dependency-free qa_mcp_echo_server.py." + - "For qa-local-stdio, set the server script path to the absolute skills fixture path: skills/langbot-testing/fixtures/mcp/qa_mcp_echo_server.py." + - "If the UI still sends empty args, fix the MCP form test handler so it reads the latest state." + - "Retest with the bundled qa_mcp_echo_server.py fixture." +verification: "The MCP test task completes without exception, the saved server shows connected, and /api/v1/tools contains qa_mcp_echo." +related_cases: + - mcp-stdio-tool-call diff --git a/skills/skills/langbot-testing/troubleshooting/nsjail-cli-compatibility.yaml b/skills/skills/langbot-testing/troubleshooting/nsjail-cli-compatibility.yaml new file mode 100644 index 000000000..97f752cef --- /dev/null +++ b/skills/skills/langbot-testing/troubleshooting/nsjail-cli-compatibility.yaml @@ -0,0 +1,29 @@ +id: nsjail-cli-compatibility +title: "Installed nsjail CLI rejects generated sandbox arguments" +date: 2026-05-18 +symptoms: + - "Box status reports nsjail as available, but every command exits before the user command runs." + - "Direct nsjail --help works, but LangBot exec fails." + - "On WSL, cgroup v2 warnings appear before falling back to rlimits." +patterns: + - "nsjail: unrecognized option '--clone_newuser'" + - "nsjail: unrecognized option '--clone_newnet'" + - "nsjail: unrecognized option '--rw_bind'" + - "execve('sh') failed: No such file or directory" + - "Failed to mount mandatory point: '/workspace'" + - "createCgroup() ... failed" +likely_causes: + - "The installed nsjail version enables clone namespaces by default and exposes only disable flags." + - "The installed nsjail uses --bindmount for read-write bind mounts, not --rw_bind." + - "A bare chroot at / cannot create mount targets for /workspace." + - "The command uses sh instead of /bin/sh inside the chroot." + - "cgroup v2 exists but is not writable by the LangBot user." +fix_steps: + - "Generate no positive --clone_new* flags; use --disable_clone_newnet only when network is requested." + - "Use --bindmount for read-write mounts and --bindmount_ro for read-only mounts." + - "Create a per-session chroot root and pre-create mount target directories such as /workspace, /tmp, /home, and activated skill paths." + - "Execute commands through /bin/sh -lc." + - "Check cgroup v2 writability before using cgroup flags; warn and use rlimits when not writable." +verification: "Run sandbox-skill-authoring-e2e with BOX_BACKEND=nsjail. The model should complete exec, register_skill, activate, and activated skill exec without nsjail argument errors." +related_cases: + - sandbox-skill-authoring-e2e diff --git a/skills/skills/langbot-testing/troubleshooting/pipeline-form-controlled-warning.yaml b/skills/skills/langbot-testing/troubleshooting/pipeline-form-controlled-warning.yaml new file mode 100644 index 000000000..92f788a87 --- /dev/null +++ b/skills/skills/langbot-testing/troubleshooting/pipeline-form-controlled-warning.yaml @@ -0,0 +1,20 @@ +id: pipeline-form-controlled-warning +title: "Pipeline form input switches from uncontrolled to controlled" +date: 2026-05-17 +symptoms: + - "Browser console shows a React uncontrolled input to controlled input warning." + - "The warning appears after opening a pipeline edit page or loading pipeline data." +patterns: + - "A component is changing an uncontrolled input to be controlled" + - "PipelineFormComponent" +likely_causes: + - "React Hook Form default values omit string fields that later receive loaded values." + - "Input value receives undefined before the backend response arrives." +fix_steps: + - "Initialize pipeline basic fields such as name and description to empty strings." + - "When loading backend data, coalesce nullable string values to empty strings." + - "Pass value={field.value ?? ''} for text inputs that can render before data loads." +verification: "Opening the pipeline edit page and AI config produces no uncontrolled/controlled input warning." +related_cases: + - pipeline-debug-chat + - local-agent-rag-debug-chat diff --git a/skills/skills/langbot-testing/troubleshooting/plugin-dependency-install-offline.yaml b/skills/skills/langbot-testing/troubleshooting/plugin-dependency-install-offline.yaml new file mode 100644 index 000000000..f5a484df8 --- /dev/null +++ b/skills/skills/langbot-testing/troubleshooting/plugin-dependency-install-offline.yaml @@ -0,0 +1,24 @@ +id: plugin-dependency-install-offline +title: "Local plugin dependency install fails in an offline or proxy-limited runtime" +date: 2026-05-17 +symptoms: + - "Installing a local .lbpkg starts but fails during Installing Dependencies." + - "The plugin files may appear under data/plugins, but the plugin is not initialized or listed as active." +patterns: + - "Failed to install dependency: PyMuPDF>=1.24.0" + - "ERROR: Could not find a version that satisfies the requirement PyMuPDF" + - "ERROR: No matching distribution found for PyMuPDF" + - "Installing langbot-team-GeneralParsers" +likely_causes: + - "The LangBot plugin runtime is started without a working network path to PyPI or the configured package index." + - "The backend was started with proxy variables unset to avoid SOCKS-related HTTP client errors, so plugin dependency installation cannot download uncached packages." + - "The dependency is available in another local venv or cache, but not in the active LangBot master venv used by the plugin runtime." +fix_steps: + - "Confirm the active runtime venv: use the same LangBot master .venv that starts the backend." + - "Before retrying local plugin install, verify required dependencies with python -m pip install --dry-run ''." + - "If the environment is offline, install from an existing local wheel/cache or another trusted local venv into the active LangBot master venv." + - "Retry the UI Upload Local install and wait for Plugin installed successfully." + - "For GeneralParsers, confirm import fitz works in the active LangBot master venv before retrying." +verification: "Installed Plugins shows the local plugin initialized, and backend logs include Plugin langbot-team/GeneralParsers initialized." +related_cases: + - langrag-parser-golden-e2e diff --git a/skills/skills/langbot-testing/troubleshooting/plugin-runtime-timeout.yaml b/skills/skills/langbot-testing/troubleshooting/plugin-runtime-timeout.yaml new file mode 100644 index 000000000..3b0f33f36 --- /dev/null +++ b/skills/skills/langbot-testing/troubleshooting/plugin-runtime-timeout.yaml @@ -0,0 +1,31 @@ +id: plugin-runtime-timeout +title: "Plugin runtime actions time out" +date: 2026-05-16 +symptoms: + - "Debug Chat can send a user message, but no useful Bot response appears." + - "The page may show Agent runner temporarily unavailable." + - "Installed Plugins may show Plugin System Connection Error." + - "Knowledge sidebar or plugin sidebar loading may hang or time out." +patterns: + - "Action list_plugins call timed out" + - "Action list_agent_runners call timed out" + - "Action invoke_llm_stream call timed out" + - "All models failed during streaming setup" + - "Failed to fetch plugins for sidebar" + - "Failed to fetch knowledge bases for sidebar" +likely_causes: + - "An old langbot_plugin runtime process survived a backend restart." + - "Multiple plugin runtime processes are active and the backend is waiting on the wrong one." + - "Plugin runtime got a control connection but never finished plugin discovery or launch." + - "A disposable test plugin in data/plugins is blocking runtime startup." +fix_steps: + - "Stop the LangBot backend." + - "Stop orphaned langbot_plugin.cli runtime processes." + - "Confirm the configured backend URL is free or reachable as appropriate." + - "Start LangBot again and wait for Connected to plugin runtime plus langbot/local-agent initialization." + - "For disposable test data only, move suspected fixture plugins out of data/plugins and restart to isolate plugin discovery failures." +verification: "Open Installed Plugins and confirm the plugin list loads, then run pipeline-debug-chat and confirm the UI shows a Bot response while backend logs include Streaming completed." +related_cases: + - pipeline-debug-chat + - provider-deepseek + - langrag-parser-golden-e2e diff --git a/skills/skills/langbot-testing/troubleshooting/provider-image-parse-error.yaml b/skills/skills/langbot-testing/troubleshooting/provider-image-parse-error.yaml new file mode 100644 index 000000000..4ddbcfef7 --- /dev/null +++ b/skills/skills/langbot-testing/troubleshooting/provider-image-parse-error.yaml @@ -0,0 +1,23 @@ +id: provider-image-parse-error +title: "Model provider rejects the uploaded image before runner behavior is exercised" +date: 2026-05-17 +symptoms: + - "Debug Chat shows Agent runner temporarily unavailable after an image upload." + - "Backend logs include image_parse_error or unsupported image from the model provider." + - "The upload endpoint succeeds, but the LLM request fails before a useful model response is returned." +patterns: + - "image_parse_error" + - "unsupported image" + - "You uploaded an unsupported image" + - "runner.llm_error" +likely_causes: + - "The fixture image is too small or otherwise rejected by the active model provider." + - "The selected model or provider route does not support image input even though the WebUI upload succeeded." + - "The provider reports image validation errors as rate-limit or requester errors." +fix_steps: + - "Retest with a normal-sized PNG such as the bundled 64x64 red-square fixture." + - "Switch to a model route known to accept image input before diagnosing local-agent multimodal handling." + - "Use backend logs to separate provider image parsing failure from runner context or message-building failure." +verification: "The same Debug Chat image prompt reaches the model without image_parse_error; if semantic vision is unreliable, treat UI image visibility plus message-building tests as the runner coverage signal." +related_cases: + - local-agent-multimodal-debug-chat diff --git a/skills/skills/langbot-testing/troubleshooting/proxy-env-mismatch.yaml b/skills/skills/langbot-testing/troubleshooting/proxy-env-mismatch.yaml new file mode 100644 index 000000000..b9b398edd --- /dev/null +++ b/skills/skills/langbot-testing/troubleshooting/proxy-env-mismatch.yaml @@ -0,0 +1,22 @@ +id: proxy-env-mismatch +title: "Uppercase and lowercase proxy variables differ" +date: 2026-05-16 +symptoms: + - "External model calls time out even though a proxy appears to be configured." + - "GitHub OAuth or provider tests fail intermittently from the agent environment." +patterns: + - "HTTP_PROXY and http_proxy point to different hosts" + - "HTTPS_PROXY and https_proxy point to different hosts" + - "ALL_PROXY and all_proxy point to different hosts" + - "old WSL gateway proxy remains in lowercase variables" +likely_causes: + - "Different HTTP clients read different proxy environment variable names." + - "The shell inherited stale lowercase proxy values from an older session." +fix_steps: + - "Set HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, http_proxy, https_proxy, all_proxy, NO_PROXY, and no_proxy consistently." + - "Restart LangBot with the corrected proxy environment." +verification: "Run env | rg -i '^(http|https|all|no)_?proxy=' and confirm uppercase/lowercase pairs are consistent, then run pipeline-debug-chat." +related_cases: + - pipeline-debug-chat + - provider-deepseek + - webui-login-state diff --git a/skills/skills/langbot-testing/troubleshooting/sandbox-native-tools-unavailable.yaml b/skills/skills/langbot-testing/troubleshooting/sandbox-native-tools-unavailable.yaml new file mode 100644 index 000000000..6c6106036 --- /dev/null +++ b/skills/skills/langbot-testing/troubleshooting/sandbox-native-tools-unavailable.yaml @@ -0,0 +1,24 @@ +id: sandbox-native-tools-unavailable +title: "Native sandbox tools are unavailable even though a backend is configured" +date: 2026-05-18 +symptoms: + - "Backend logs show Native sandbox tools (exec/read/write/edit/glob/grep) are NOT available." + - "The Box runtime later reports that E2B, nsjail, or Docker is configured." + - "Debug Chat does not expose exec, register_skill, or activate as usable tools." +patterns: + - "Native sandbox tools ... are NOT available" + - "No sandbox backend (Docker/nsjail/E2B) is ready" + - "box.backend is set but /api/v1/box/status backend is unavailable" +likely_causes: + - "Backend status was checked before the Box runtime selected or reselected a backend." + - "The configured backend failed availability checks because credentials, binary, Docker daemon, or proxy settings were not ready." + - "The runtime cached an unavailable backend state after INIT config changed the backend." +fix_steps: + - "Check /api/v1/box/status and verify backend.name and backend.available." + - "Restart LangBot after fixing backend credentials, binary installation, Docker daemon, or proxy settings." + - "Ensure the Box runtime reselects a backend when get_backend_info is called and the cached backend is empty." + - "For E2B, verify the key without printing it and confirm any required template setting." + - "For nsjail, run nsjail --help and confirm the binary is on PATH for the LangBot process." +verification: "Run sandbox-skill-authoring-e2e. Logs should show Native sandbox tools are available and /api/v1/box/status should report available=true with the expected backend." +related_cases: + - sandbox-skill-authoring-e2e diff --git a/skills/skills/langbot-testing/troubleshooting/socks-proxy-without-socksio.yaml b/skills/skills/langbot-testing/troubleshooting/socks-proxy-without-socksio.yaml new file mode 100644 index 000000000..b3e6d7b93 --- /dev/null +++ b/skills/skills/langbot-testing/troubleshooting/socks-proxy-without-socksio.yaml @@ -0,0 +1,24 @@ +id: socks-proxy-without-socksio +title: "Python HTTP clients fail when ALL_PROXY uses SOCKS without socksio" +date: 2026-05-18 +symptoms: + - "LangBot startup fails while importing or initializing a provider client." + - "The failure happens before sandbox testing begins." + - "Clearing ALL_PROXY lets LangBot start, but external HTTP calls may then need another proxy path." +patterns: + - "Using SOCKS proxy, but the 'socksio' package is not installed" + - "httpx using SOCKS proxy" + - "ALL_PROXY=socks5://" +likely_causes: + - "The process inherited ALL_PROXY or all_proxy with a SOCKS URL." + - "A dependency uses httpx without the socks extra installed." + - "HTTP_PROXY and HTTPS_PROXY would have been sufficient, but ALL_PROXY took precedence for that client." +fix_steps: + - "Start LangBot with ALL_PROXY and all_proxy unset when SOCKS support is not installed." + - "Keep HTTP_PROXY and HTTPS_PROXY set consistently if an HTTP proxy is available." + - "Alternatively install the dependency's SOCKS support in the runtime environment." + - "Keep NO_PROXY/no_proxy covering localhost, 127.0.0.1, and ::1." +verification: "LangBot starts cleanly, model provider calls still use the intended proxy path, and sandbox-skill-authoring-e2e can reach the model provider." +related_cases: + - sandbox-skill-authoring-e2e + - provider-deepseek diff --git a/skills/skills/langbot-testing/troubleshooting/survey-widget-blocks-debug-chat.yaml b/skills/skills/langbot-testing/troubleshooting/survey-widget-blocks-debug-chat.yaml new file mode 100644 index 000000000..f70ce6334 --- /dev/null +++ b/skills/skills/langbot-testing/troubleshooting/survey-widget-blocks-debug-chat.yaml @@ -0,0 +1,22 @@ +id: survey-widget-blocks-debug-chat +title: "Survey widget blocks Debug Chat controls" +date: 2026-05-17 +symptoms: + - "Debug Chat input is filled but the Send button cannot be clicked." + - "Playwright reports that a fixed bottom-right widget intercepts pointer events." +patterns: + - "subtree intercepts pointer events" + - "Help us improve" + - "fixed bottom-6 right-6" +likely_causes: + - "The survey widget overlaps the Debug Chat action area at the current viewport size." + - "The test viewport is narrow enough that the floating widget covers the Send button." +fix_steps: + - "Close or minimize the survey widget before interacting with Debug Chat." + - "Alternatively, resize the browser wider or use keyboard submission if the UI supports it." + - "Continue the browser test after confirming the widget is gone." +verification: "The Send button can be clicked and Debug Chat shows both the User message and Bot response." +related_cases: + - pipeline-debug-chat + - local-agent-rag-debug-chat + - mcp-stdio-tool-call diff --git a/skills/skills/langbot-testing/troubleshooting/tool-name-collision-between-mcp-and-plugin.yaml b/skills/skills/langbot-testing/troubleshooting/tool-name-collision-between-mcp-and-plugin.yaml new file mode 100644 index 000000000..ab40b0406 --- /dev/null +++ b/skills/skills/langbot-testing/troubleshooting/tool-name-collision-between-mcp-and-plugin.yaml @@ -0,0 +1,25 @@ +id: tool-name-collision-between-mcp-and-plugin +title: "MCP and plugin expose the same tool name" +date: 2026-05-21 +symptoms: + - "An MCP tool-call case completes, but the returned sentinel has a plugin prefix instead of the MCP prefix." + - "The prompt asks for the MCP fixture tool, but the bot returns qa-plugin-smoke:." + - "GET /api/v1/tools or the runner tool list contains multiple tools with the same visible name." +patterns: + - "qa-plugin-smoke:mcp-ok-local-agent" + - "multiple tools with the same visible name" + - "duplicate tool name" + - "qa/plugin-smoke" +likely_causes: + - "A stale MCP registration still exposes the old qa_echo fixture name while qa-plugin-smoke is installed." + - "The model selects the plugin tool because both tools have the same call name or similar descriptions." + - "The test prompt names an ambiguous tool, so the browser result cannot prove which provider executed the tool unless the returned sentinel is checked." +fix_steps: + - "Use unique tool names across simultaneous test fixtures. The current MCP fixture should expose qa_mcp_echo, not qa_echo." + - "When testing MCP, require the bot response to contain the MCP-specific sentinel qa_mcp_echo:, not qa-plugin-smoke:." + - "When testing plugin tools, prefer qa_plugin_echo or another plugin-only tool name that cannot be confused with the MCP fixture." + - "Refresh the MCP server registration or restart the backend if /api/v1/tools still shows the old qa_echo MCP fixture." +verification: "The MCP case passes only when the bot-visible response contains the MCP fixture sentinel and the plugin-only sentinel is absent for the same input." +related_cases: + - mcp-stdio-tool-call + - local-agent-plugin-tool-call-debug-chat diff --git a/skills/skills/langbot-testing/troubleshooting/uv-run-resyncs-local-sdk.yaml b/skills/skills/langbot-testing/troubleshooting/uv-run-resyncs-local-sdk.yaml new file mode 100644 index 000000000..e05a2f04b --- /dev/null +++ b/skills/skills/langbot-testing/troubleshooting/uv-run-resyncs-local-sdk.yaml @@ -0,0 +1,22 @@ +id: uv-run-resyncs-local-sdk +title: "uv run resyncs the locked SDK instead of the local editable SDK" +date: 2026-05-17 +symptoms: + - "LangBot was expected to use a local langbot-plugin SDK checkout, but import inspection points to .venv/site-packages." + - "uv output shows langbot-plugin was uninstalled and installed again immediately before a test or startup command." + - "A test appears to pass but is actually using the PyPI SDK from uv.lock." +patterns: + - "uv run python -c import langbot_plugin prints .venv/lib/python*/site-packages/langbot_plugin" + - "Uninstalled 1 package" + - "Installed 1 package" + - "langbot-plugin==0.3.11 without file:/// local source" +likely_causes: + - "Plain uv run synchronizes the environment to uv.lock before executing the command." + - "The editable local SDK was installed with uv pip install -e, but the next uv run reverted it." +fix_steps: + - "Install the local SDK into the LangBot worktree with uv pip install -e /absolute/path/to/langbot-plugin-sdk-test-build." + - "Run validation and startup commands with uv run --no-sync." + - "Before trusting results, print pathlib.Path(langbot_plugin.__file__).resolve() and confirm it points to the local SDK source tree." +verification: "uv run --no-sync python -c import inspection prints the local SDK source path, and plugin-e2e-smoke passes with that same process environment." +related_cases: + - plugin-e2e-smoke diff --git a/skills/src/cli.ts b/skills/src/cli.ts new file mode 100644 index 000000000..8ff85d4d6 --- /dev/null +++ b/skills/src/cli.ts @@ -0,0 +1,120 @@ +import { dirname, resolve } from "node:path"; +import { cwd, exit } from "node:process"; +import { existsSync } from "node:fs"; +import type { CommandContext } from "./types.ts"; + +export function usage(): never { + console.log(`Usage: + bin/lbs [--root ] list + bin/lbs [--root ] validate + bin/lbs [--root ] index [--check] + bin/lbs [--root ] new-skill [--description ] + bin/lbs [--root ] new-ref + + bin/lbs [--root ] env show [--json] + bin/lbs [--root ] env doctor + + bin/lbs [--root ] fixture list [skill] [--json] + bin/lbs [--root ] fixture check [skill] [--json] + + bin/lbs [--root ] log scan [--json] [--output ] [--backend-log ] [--frontend-log ] [--console-log ] [--case ] [--success-pattern ] [--failure-pattern ] [--expected-failure ] [--since ] [--until ] [--tail-lines ] [--no-auto-log] [--strict] + bin/lbs [--root ] log watch [--json] [--backend-log ] [--case ] [--success-pattern ] [--failure-pattern ] [--expected-failure ] [--interval-ms ] [--duration-ms ] [--from-start] [--strict] + bin/lbs [--root ] log guard start [--run-id ] [--output-dir ] [--backend-log ] [--case ] [--json] + bin/lbs [--root ] log guard stop --run-id [--output-dir ] [--session ] [--output ] [--case ] [--backend-log ] [--since ] [--until ] [--json] [--no-strict] + + bin/lbs [--root ] case new --title [--skill langbot-testing] [--mode agent-browser|probe] [--area ] [--type smoke] + bin/lbs [--root ] case list [skill] [--json] [--type ] [--area ] [--tag ] [--priority p0|p1|p2] [--risk low|medium|high] [--automation] [--ci] [--ready] [--machine-ready] + bin/lbs [--root ] case show [skill] + + bin/lbs [--root ] suite new --title [--skill langbot-testing] [--description ] [--type smoke] [--priority p2] + bin/lbs [--root ] suite list [skill] [--json] [--type ] [--priority p0|p1|p2] + bin/lbs [--root ] suite show [skill] + bin/lbs [--root ] suite plan [skill] [--json] + bin/lbs [--root ] suite start [skill] [--run-id ] [--evidence-dir ] [--output ] [--json] + bin/lbs [--root ] suite run [skill] [--run-id ] [--evidence-dir ] [--output ] [--headed] [--dry-run] [--include-manual-check] [--include-not-ready] [--json] + bin/lbs [--root ] suite report [skill] [--run-id ] [--evidence-dir ] [--output ] [--json] + + bin/lbs [--root ] test plan [skill] [--json] + bin/lbs [--root ] test recommend [--file ] [--json] + bin/lbs [--root ] test start [skill] [--output ] [--json] + bin/lbs [--root ] test run [skill] [--output ] [--run-id ] [--headed] [--dry-run] [--json] + bin/lbs [--root ] test report [skill] [--output ] [--json] [--backend-log ] [--frontend-log ] [--console-log ] [--evidence-dir ] [--since ] [--until ] [--tail-lines ] [--no-auto-log] + bin/lbs [--root ] test result [skill] --result --reason --evidence-dir [--evidence ui,console,backend_log] [--started-at ] [--finished-at ] [--run-id ] [--url ] [--browser-path ] [--report ] [--notes ] [--json] + + bin/lbs [--root ] trouble list [skill] + bin/lbs [--root ] trouble show [skill] + bin/lbs [--root ] trouble search + bin/lbs [--root ] trouble add --title --symptom --cause --fix [--id ] [--verify ] +`); + exit(2); +} + +export function fail(message: string): never { + console.error(`ERROR: ${message}`); + exit(1); +} + +export function repoRoot(start: string): string { + let current = resolve(start); + while (true) { + // The skills assets root is identified by skills.index.json (present at the + // root of this assets tree). Check it first so that when the tree lives + // inside a larger repo (e.g. LangBot/skills/), we stop at the assets root + // and not at the outer repo's .git/README.md. + if (existsSync(`${current}/skills.index.json`) && existsSync(`${current}/bin/lbs`)) { + return current; + } + if (existsSync(`${current}/.git`) && existsSync(`${current}/README.md`)) { + return current; + } + const parent = dirname(current); + if (parent === current) return resolve(start); + current = parent; + } +} + +export function parseGlobalArgs(rawArgs: string[]): CommandContext { + let root = repoRoot(cwd()); + const args = [...rawArgs]; + + for (let i = 0; i < args.length; ) { + if (args[i] === "--root") { + const value = args[i + 1]; + if (!value) fail("--root requires a path"); + root = resolve(value); + args.splice(i, 2); + continue; + } + i += 1; + } + + return { root, args }; +} + +export function parseOptions(args: string[]): { positional: string[]; options: Record } { + const positional: string[] = []; + const options: Record = {}; + + for (let i = 0; i < args.length; i += 1) { + const arg = args[i]; + if (arg.startsWith("--")) { + const key = arg.slice(2); + const value = args[i + 1]; + if (!value || value.startsWith("--")) { + options[key] = true; + } else { + options[key] = value; + i += 1; + } + } else { + positional.push(arg); + } + } + + return { positional, options }; +} + +export function optionString(options: Record, key: string): string | undefined { + const value = options[key]; + return typeof value === "string" ? value : undefined; +} diff --git a/skills/src/commands/case.ts b/skills/src/commands/case.ts new file mode 100644 index 000000000..eeec83728 --- /dev/null +++ b/skills/src/commands/case.ts @@ -0,0 +1,180 @@ +import { existsSync, mkdirSync, writeFileSync } from "node:fs"; +import { join, resolve } from "node:path"; +import type { CommandContext } from "../types.ts"; +import { parseOptions, optionString, usage, fail } from "../cli.ts"; +import { caseModeValues } from "../constants.ts"; +import { boolValue, findStructuredItem, getSkill, listValue, loadStructuredItems, scalar, yamlList, yamlQuote } from "../fs.ts"; +import { caseAutomationReadiness, caseEnvReadiness, caseFixtureReadiness, caseManualReadiness, runtimeEnv } from "../readiness.ts"; +import { setupAutomationEntries } from "../setup-automation.ts"; + +function casePath(root: string, skillName: string, id: string): string { + const skill = getSkill(root, skillName); + const dir = join(skill.path, "cases"); + mkdirSync(dir, { recursive: true }); + return join(dir, `${id}.yaml`); +} + +export function commandCaseNew(ctx: CommandContext): number { + const { positional, options } = parseOptions(ctx.args.slice(2)); + const id = positional[0]; + const title = optionString(options, "title"); + if (!id || !title) usage(); + + const skill = optionString(options, "skill") ?? "langbot-testing"; + const path = casePath(ctx.root, skill, id); + if (existsSync(path)) fail(`case already exists: ${path}`); + + const area = optionString(options, "area") ?? "general"; + const type = optionString(options, "type") ?? "smoke"; + const mode = optionString(options, "mode") ?? "agent-browser"; + if (!caseModeValues.includes(mode)) fail(`--mode must be one of ${caseModeValues.join(", ")}`); + const isProbe = mode === "probe"; + + const text = + `id: ${id}\n` + + `title: ${yamlQuote(title)}\n` + + `mode: ${mode}\n` + + `area: ${area}\n` + + `type: ${type}\n` + + "priority: p2\n" + + "risk: medium\n" + + "ci_eligible: false\n" + + "tags:\n" + + yamlList([type]) + + "\nskills:\n" + + yamlList(["langbot-env-setup", skill]) + + "\nenv:\n" + + yamlList(isProbe ? [] : ["LANGBOT_FRONTEND_URL", "LANGBOT_BACKEND_URL"]) + + "\nsteps:\n" + + yamlList([isProbe ? "Describe the probe command, script, or diagnostic to run." : "Describe the user-visible action to perform."]) + + "\nchecks:\n" + + yamlList(isProbe + ? [ + "Probe: Describe the expected success signal.", + "Evidence: Required logs, API diagnostics, or filesystem artifacts are written.", + ] + : [ + "UI: Describe the user-visible success signal.", + "Console: No unexpected frontend errors.", + "Logs: Relevant backend processing completed when applicable.", + ]) + + "\nevidence_required:\n" + + yamlList(isProbe ? ["api_diagnostic"] : ["ui", "console"]) + + "\ndiagnostics:\n" + + yamlList([isProbe + ? "Use logs, API, or filesystem diagnostics to explain probe failures." + : "Use API/curl/logs only to distinguish frontend failure from backend/runtime failure."]) + + "\ntroubleshooting:\n" + + yamlList([]) + + "\n"; + + writeFileSync(path, text, "utf8"); + console.log(path); + return 0; +} + +function caseRow(item: ReturnType[number], root: string): Record { + const automation = scalar(item.fields, "automation"); + const env = runtimeEnv(root); + const id = scalar(item.fields, "id"); + const row = { + skill: item.skill, + id, + title: scalar(item.fields, "title"), + mode: scalar(item.fields, "mode"), + area: scalar(item.fields, "area"), + type: scalar(item.fields, "type"), + priority: scalar(item.fields, "priority"), + risk: scalar(item.fields, "risk"), + ci_eligible: boolValue(item.fields, "ci_eligible") ?? false, + tags: listValue(item.fields, "tags"), + env: listValue(item.fields, "env"), + env_any: listValue(item.fields, "env_any"), + preconditions: listValue(item.fields, "preconditions"), + setup: listValue(item.fields, "setup"), + setup_automation: setupAutomationEntries(item), + setup_provides_env: listValue(item.fields, "setup_provides_env"), + cleanup: listValue(item.fields, "cleanup"), + evidence_required: listValue(item.fields, "evidence_required"), + automation, + automation_exists: automation ? existsSync(resolve(root, automation)) : false, + env_readiness: caseEnvReadiness(item, env), + automation_readiness: caseAutomationReadiness(item, env), + fixture_readiness: caseFixtureReadiness(root, id), + manual_readiness: caseManualReadiness(item), + }; + return { + ...row, + readiness: readinessLabel(row), + }; +} + +function hasTag(row: Record, tag: string): boolean { + const tags = row.tags; + return Array.isArray(tags) && tags.includes(tag); +} + +function hasMissingReadiness(row: Record): boolean { + for (const key of ["env_readiness", "automation_readiness", "fixture_readiness"]) { + const value = row[key] as Record | undefined; + if (value?.status === "missing") return true; + } + return false; +} + +function hasManualCheck(row: Record): boolean { + const manual = row.manual_readiness as Record | undefined; + return manual?.status === "manual_check"; +} + +function readinessLabel(row: Record): string { + if (hasMissingReadiness(row)) return "not-ready"; + return hasManualCheck(row) ? "manual-check" : "ready"; +} + +export function commandCaseList(ctx: CommandContext): number { + const { positional, options } = parseOptions(ctx.args.slice(2)); + const skill = positional[0]; + const rows = loadStructuredItems(ctx.root, "cases", skill) + .map((item) => caseRow(item, ctx.root)) + .filter((row) => !optionString(options, "type") || row.type === optionString(options, "type")) + .filter((row) => !optionString(options, "area") || row.area === optionString(options, "area")) + .filter((row) => !optionString(options, "priority") || row.priority === optionString(options, "priority")) + .filter((row) => !optionString(options, "risk") || row.risk === optionString(options, "risk")) + .filter((row) => !optionString(options, "tag") || hasTag(row, optionString(options, "tag") ?? "")) + .filter((row) => options.automation !== true || Boolean(row.automation)) + .filter((row) => options.ci !== true || row.ci_eligible === true) + .filter((row) => options["machine-ready"] !== true || !hasMissingReadiness(row)) + .filter((row) => options.ready !== true || (!hasMissingReadiness(row) && !hasManualCheck(row))); + + if (options.json === true) { + console.log(JSON.stringify(rows, null, 2)); + return 0; + } + + for (const row of rows) { + console.log([ + row.skill, + row.id, + row.type, + row.area, + row.priority, + row.risk, + row.ci_eligible ? "ci" : "manual", + row.automation ? "automated" : "manual-path", + row.readiness, + row.title, + ].join("\t")); + } + return 0; +} + +export function commandCaseShow(ctx: CommandContext): number { + const positional = ctx.args.slice(2); + if (positional.length < 1 || positional.length > 2) usage(); + const item = positional.length === 1 + ? findStructuredItem(ctx.root, "cases", positional[0]) + : findStructuredItem(ctx.root, "cases", positional[0], positional[1]); + console.log(item.raw.trimEnd()); + return 0; +} diff --git a/skills/src/commands/env.ts b/skills/src/commands/env.ts new file mode 100644 index 000000000..d5d1eeaf4 --- /dev/null +++ b/skills/src/commands/env.ts @@ -0,0 +1,140 @@ +import { existsSync } from "node:fs"; +import { Socket } from "node:net"; +import type { CommandContext } from "../types.ts"; +import { parseOptions } from "../cli.ts"; +import { loadEnv } from "../fs.ts"; +import { requiredEnvKeys } from "../constants.ts"; +import { redactEnvValue } from "../readiness.ts"; + +export function commandEnvShow(ctx: CommandContext): number { + const { options } = parseOptions(ctx.args.slice(2)); + const env = loadEnv(ctx.root); + const outputEnv = Object.fromEntries( + Object.entries(env).map(([key, value]) => [key, redactEnvValue(key, value)]), + ); + if (options.json === true) { + console.log(JSON.stringify(outputEnv, null, 2)); + return 0; + } + for (const key of Object.keys(outputEnv).sort()) { + console.log(`${key}=${outputEnv[key]}`); + } + return 0; +} + +async function checkUrl(label: string, url: string): Promise<{ ok: boolean; message: string }> { + if (!url) return { ok: false, message: `${label}: missing` }; + const displayUrl = redactEnvValue(label, url); + try { + const response = await fetch(url, { method: "HEAD", signal: AbortSignal.timeout(2500) }); + return { ok: response.ok || response.status < 500, message: `${label}: ${displayUrl} -> HTTP ${response.status}` }; + } catch (error) { + return { ok: false, message: `${label}: ${displayUrl} -> ${String(error).replace(/\s+/g, " ")}` }; + } +} + +function endpoint(url: string): { host: string; port: number } | null { + try { + const parsed = new URL(url); + if (parsed.protocol !== "http:" && parsed.protocol !== "https:") return null; + const port = parsed.port ? Number(parsed.port) : parsed.protocol === "https:" ? 443 : 80; + return { host: parsed.hostname, port }; + } catch { + return null; + } +} + +async function checkTcpListener(url: string): Promise<{ ok: boolean; message: string } | null> { + const target = endpoint(url); + if (!target) return null; + + return await new Promise((resolve) => { + const socket = new Socket(); + let settled = false; + const finish = (ok: boolean, detail: string) => { + if (settled) return; + settled = true; + socket.destroy(); + resolve({ + ok, + message: `${target.host}:${target.port} ${detail}`, + }); + }; + + socket.setTimeout(1500); + socket.once("connect", () => finish(true, "is listening")); + socket.once("timeout", () => finish(false, "did not accept TCP connection before timeout")); + socket.once("error", (error) => finish(false, `is not listening (${error.message})`)); + socket.connect(target.port, target.host); + }); +} + +function startupHint(label: string, env: Record): string | null { + if (label === "LANGBOT_BACKEND_URL" && env.LANGBOT_REPO) { + return `start backend: cd ${env.LANGBOT_REPO} && uv run main.py`; + } + if (label === "LANGBOT_FRONTEND_URL" && env.LANGBOT_WEB_REPO) { + return `start frontend: cd ${env.LANGBOT_WEB_REPO} && pnpm dev`; + } + return null; +} + +function compareProxyPair(env: Record, upper: string, lower: string): string | null { + const upperValue = process.env[upper] ?? env[upper] ?? ""; + const lowerValue = process.env[lower] ?? env[lower] ?? ""; + if (upperValue && lowerValue && upperValue !== lowerValue) { + return `${upper}/${lower}: mismatch (${redactEnvValue(upper, upperValue)} vs ${redactEnvValue(lower, lowerValue)})`; + } + return null; +} + +export async function commandEnvDoctor(ctx: CommandContext): Promise { + const env = loadEnv(ctx.root); + const failures: string[] = []; + const warnings: string[] = []; + + for (const key of requiredEnvKeys) { + if (!env[key]) failures.push(`missing ${key}`); + } + + for (const [label, path] of [ + ["LANGBOT_REPO", env.LANGBOT_REPO], + ["LANGBOT_WEB_REPO", env.LANGBOT_WEB_REPO], + ["LANGBOT_CHROMIUM_EXECUTABLE", env.LANGBOT_CHROMIUM_EXECUTABLE], + ]) { + if (!path || !existsSync(path)) failures.push(`${label}: path does not exist (${path || "missing"})`); + } + + if (env.LANGBOT_BROWSER_PROFILE && !existsSync(env.LANGBOT_BROWSER_PROFILE)) { + warnings.push(`LANGBOT_BROWSER_PROFILE: path does not exist yet (${env.LANGBOT_BROWSER_PROFILE})`); + } + + for (const mismatch of [ + compareProxyPair(env, "HTTP_PROXY", "http_proxy"), + compareProxyPair(env, "HTTPS_PROXY", "https_proxy"), + compareProxyPair(env, "ALL_PROXY", "all_proxy"), + compareProxyPair(env, "NO_PROXY", "no_proxy"), + ]) { + if (mismatch) failures.push(mismatch); + } + + for (const [label, result] of await Promise.all([ + checkUrl("LANGBOT_BACKEND_URL", env.LANGBOT_BACKEND_URL).then((result) => ["LANGBOT_BACKEND_URL", result] as const), + checkUrl("LANGBOT_FRONTEND_URL", env.LANGBOT_FRONTEND_URL).then((result) => ["LANGBOT_FRONTEND_URL", result] as const), + ])) { + if (result.ok) console.log(`OK: ${result.message}`); + else { + failures.push(result.message); + const tcp = await checkTcpListener(env[label]); + if (tcp && !tcp.ok) failures.push(`${label}: no HTTP service reachable because ${tcp.message}`); + const hint = startupHint(label, env); + if (hint) warnings.push(`${label}: ${hint}`); + } + } + + for (const warning of warnings) console.log(`WARN: ${warning}`); + for (const failure of failures) console.log(`FAIL: ${failure}`); + if (failures.length > 0) return 1; + console.log("OK: environment looks usable"); + return 0; +} diff --git a/skills/src/commands/fixture.ts b/skills/src/commands/fixture.ts new file mode 100644 index 000000000..6c5bf72aa --- /dev/null +++ b/skills/src/commands/fixture.ts @@ -0,0 +1,132 @@ +import type { CommandContext } from "../types.ts"; +import { parseOptions } from "../cli.ts"; +import { loadFixtureItems } from "../fixtures.ts"; +import { dirname, join } from "node:path"; +import { existsSync, readFileSync } from "node:fs"; + +function fixtureRows(root: string, skill: string | undefined): ReturnType { + return loadFixtureItems(root, skill); +} + +function qaAgentRunnerSourceFindings(item: ReturnType["items"][number]) { + if (!item.checks.includes("qa_agent_runner_source") || !item.exists) return []; + const root = dirname(item.absolute_path); + const required = [ + "main.py", + "components/agent_runner/default.yaml", + "components/agent_runner/default.py", + "assets/icon.svg", + ]; + const missing = required + .filter((path) => !existsSync(join(root, path))) + .map((path) => ({ + severity: "fail", + kind: "fixture_check_missing_file", + id: item.id, + path: `${item.path.replace(/\/[^/]+$/, "")}/${path}`, + })); + if (missing.length > 0) return missing; + + const manifest = readFileSync(item.absolute_path, "utf8"); + const runnerYaml = readFileSync(join(root, "components/agent_runner/default.yaml"), "utf8"); + const runnerPy = readFileSync(join(root, "components/agent_runner/default.py"), "utf8"); + const requiredText = [ + [manifest, "AgentRunner", "manifest.yaml"], + [manifest, "QAAgentRunnerPlugin", "manifest.yaml"], + [runnerYaml, "kind: AgentRunner", "components/agent_runner/default.yaml"], + [runnerYaml, "DefaultAgentRunner", "components/agent_runner/default.yaml"], + [runnerPy, "QA_AGENT_RUNNER_OK", "components/agent_runner/default.py"], + [runnerPy, "QA_AGENT_RUNNER_CONTROLLED_FAILURE", "components/agent_runner/default.py"], + ]; + return requiredText + .filter(([text, needle]) => !text.includes(needle)) + .map(([, needle, relativePath]) => ({ + severity: "fail", + kind: "fixture_check_missing_text", + id: item.id, + path: `${item.path.replace(/\/[^/]+$/, "")}/${relativePath}`, + detail: `missing ${needle}`, + })); +} + +function zipPackageFindings(item: ReturnType["items"][number]) { + if (!item.checks.includes("zip_package") || !item.exists) return []; + const header = readFileSync(item.absolute_path).subarray(0, 4).toString("binary"); + if (header === "PK\u0003\u0004" || header === "PK\u0005\u0006") return []; + return [{ + severity: "fail", + kind: "fixture_check_invalid_zip", + id: item.id, + path: item.path, + }]; +} + +export function commandFixtureList(ctx: CommandContext): number { + const { positional, options } = parseOptions(ctx.args.slice(2)); + const skill = positional[0]; + const result = fixtureRows(ctx.root, skill); + + if (options.json === true) { + console.log(JSON.stringify(result.items, null, 2)); + return result.errors.length > 0 ? 1 : 0; + } + + for (const item of result.items) { + console.log([ + item.skill, + item.id, + item.kind, + item.exists ? "present" : "missing", + item.path, + item.title, + ].join("\t")); + } + for (const error of result.errors) console.error(`ERROR: ${error}`); + return result.errors.length > 0 ? 1 : 0; +} + +export function commandFixtureCheck(ctx: CommandContext): number { + const { positional, options } = parseOptions(ctx.args.slice(2)); + const skill = positional[0]; + const result = fixtureRows(ctx.root, skill); + const findings = [ + ...result.errors.map((error) => ({ severity: "fail", kind: "invalid_manifest", detail: error })), + ...result.items + .filter((item) => !item.exists) + .map((item) => ({ + severity: "fail", + kind: "missing_fixture", + id: item.id, + path: item.path, + absolute_path: item.absolute_path, + })), + ...result.items.flatMap(qaAgentRunnerSourceFindings), + ...result.items.flatMap(zipPackageFindings), + ]; + const report = { + status: findings.some((finding) => finding.severity === "fail") ? "fail" : "pass", + fixture_count: result.items.length, + findings, + fixtures: result.items, + }; + + if (options.json === true) { + console.log(JSON.stringify(report, null, 2)); + } else { + console.log(`# Fixture Check`); + console.log(""); + console.log(`status: ${report.status}`); + console.log(`fixture_count: ${report.fixture_count}`); + console.log(""); + console.log("## Fixtures"); + for (const item of result.items) { + console.log(`- ${item.id}: ${item.exists ? "present" : "missing"} (${item.path})`); + } + console.log(""); + console.log("## Findings"); + if (findings.length === 0) console.log("- None."); + else for (const finding of findings) console.log(`- [${finding.severity}] ${finding.kind}: ${"detail" in finding ? finding.detail : finding.id}`); + } + + return report.status === "pass" ? 0 : 1; +} diff --git a/skills/src/commands/log.ts b/skills/src/commands/log.ts new file mode 100644 index 000000000..aad9eaa66 --- /dev/null +++ b/skills/src/commands/log.ts @@ -0,0 +1,427 @@ +import { existsSync, mkdirSync, readFileSync, statSync, writeFileSync } from "node:fs"; +import { setTimeout as delay } from "node:timers/promises"; +import { dirname, join, resolve } from "node:path"; +import type { CommandContext } from "../types.ts"; +import { optionString, parseOptions, usage } from "../cli.ts"; +import { findStructuredItem, loadEnv } from "../fs.ts"; +import { + latestLangBotLogPath, + logPatternContextFromStructuredItem, + renderLogFinding, + renderLogSuccessSignal, + scanLogSources, + scanLogText, + strictLogGuardExitCode, + type LogFinding, + type LogGuardPatternContext, + type LogGuardResult, + type LogSuccessSignal, +} from "../log-guard.ts"; + +type LogGuardSession = { + source: "log-guard-session"; + run_id: string; + started_at: string; + started_at_local: string; + backend_log: string; + case_id: string; + case_skill: string; +}; + +type WatchSummary = { + mode: "watch"; + status: string; + path: string; + started_at_local: string; + finished_at_local: string; + bytes_read: number; + findings: LogFinding[]; + success_signals: LogSuccessSignal[]; +}; + +function pad2(value: number): string { + return String(value).padStart(2, "0"); +} + +function pad3(value: number): string { + return String(value).padStart(3, "0"); +} + +function localIsoWithOffset(date: Date): string { + const offsetMinutes = -date.getTimezoneOffset(); + const sign = offsetMinutes >= 0 ? "+" : "-"; + const absoluteOffset = Math.abs(offsetMinutes); + return [ + `${date.getFullYear()}-${pad2(date.getMonth() + 1)}-${pad2(date.getDate())}`, + `T${pad2(date.getHours())}:${pad2(date.getMinutes())}:${pad2(date.getSeconds())}.${pad3(date.getMilliseconds())}`, + `${sign}${pad2(Math.floor(absoluteOffset / 60))}:${pad2(absoluteOffset % 60)}`, + ].join(""); +} + +function timestampSlug(localIso: string): string { + return localIso + .replace(/T/, "-") + .replace(/[.:+]/g, "-") + .replace(/[^A-Za-z0-9_-]+/g, "-") + .replace(/-+/g, "-") + .replace(/^-|-$/g, ""); +} + +function writeOrPrint(content: string, output: string | undefined): void { + if (!output) { + console.log(content.trimEnd()); + return; + } + const path = resolve(output); + mkdirSync(dirname(path), { recursive: true }); + writeFileSync(path, content, "utf8"); + console.log(path); +} + +function positiveIntegerOption(options: Record, key: string, fallback: number): number { + const raw = optionString(options, key); + if (!raw) return fallback; + const parsed = Number.parseInt(raw, 10); + if (!/^\d+$/.test(raw) || parsed <= 0) return fallback; + return parsed; +} + +function splitPatternList(value: string | undefined): string[] { + if (!value) return []; + return value + .split(/\s*\|\s*|\s*,\s*/) + .map((item) => item.trim()) + .filter(Boolean); +} + +function patternContextFromOptions(root: string, options: Record): LogGuardPatternContext { + const caseId = optionString(options, "case"); + const base = caseId ? logPatternContextFromStructuredItem(findStructuredItem(root, "cases", caseId)) : {}; + return { + successPatterns: [ + ...(base.successPatterns ?? []), + ...splitPatternList(optionString(options, "success-pattern")), + ], + failurePatterns: [ + ...(base.failurePatterns ?? []), + ...splitPatternList(optionString(options, "failure-pattern")), + ], + expectedFailures: [ + ...(base.expectedFailures ?? []), + ...splitPatternList(optionString(options, "expected-failure")), + ], + relatedTroubleshootingIds: base.relatedTroubleshootingIds ?? [], + }; +} + +function latestOrExplicitBackendLog(root: string, options: Record): string { + const explicit = optionString(options, "backend-log"); + if (explicit) return resolve(explicit); + const auto = latestLangBotLogPath(loadEnv(root)); + return auto ? resolve(auto) : ""; +} + +function renderSources(result: LogGuardResult): string[] { + const lines: string[] = []; + if (result.sources.length === 0) { + lines.push("- sources: no log files provided; use --backend-log or configure LANGBOT_REPO."); + return lines; + } + lines.push("- sources:"); + for (const source of result.sources) { + const origin = source.auto_detected ? ", auto" : ""; + const total = source.total_line_count === undefined ? "" : `/${source.total_line_count}`; + const range = source.start_line === undefined || source.end_line === undefined + ? "" + : `, lines ${source.start_line}-${source.end_line}`; + const timestamped = source.timestamped_line_count === undefined ? "" : `, ${source.timestamped_line_count} timestamped`; + lines.push(` - ${source.source}: ${source.path} (${source.status}${origin}, ${source.line_count}${total} lines${range}${timestamped})`); + } + return lines; +} + +function renderLogGuardMarkdown(title: string, result: LogGuardResult, extra: string[] = []): string { + const lines: string[] = []; + lines.push(`# ${title}`); + lines.push(""); + lines.push(`Generated: ${new Date().toISOString()}`); + lines.push(`Status: ${result.status}`); + lines.push(`Scan mode: ${result.scan.mode}`); + if (result.scan.since) lines.push(`Since: ${result.scan.since}`); + if (result.scan.until) lines.push(`Until: ${result.scan.until}`); + if (result.scan.tail_lines !== undefined) lines.push(`Tail lines: ${result.scan.tail_lines}`); + if (extra.length > 0) { + lines.push(""); + lines.push("## Context"); + for (const item of extra) lines.push(`- ${item}`); + } + lines.push(""); + lines.push("## Sources"); + lines.push(...renderSources(result)); + if (result.scan.warnings.length > 0) { + lines.push(""); + lines.push("## Scan Warnings"); + for (const warning of result.scan.warnings) lines.push(`- ${warning}`); + } + lines.push(""); + lines.push("## Findings"); + if (result.findings.length === 0) lines.push("- None."); + else for (const finding of result.findings) lines.push(renderLogFinding(finding)); + lines.push(""); + lines.push("## Success Signals"); + if (result.success_signals.length === 0) lines.push("- None."); + else for (const signal of result.success_signals) lines.push(renderLogSuccessSignal(signal)); + lines.push(""); + return `${lines.join("\n").trimEnd()}\n`; +} + +function statusFromEvents(findings: LogFinding[], successSignals: LogSuccessSignal[]): string { + if (findings.some((finding) => finding.severity === "fail" || finding.severity === "missing_input")) return "fail"; + if (findings.some((finding) => finding.severity === "matched_troubleshooting" && finding.related_to_case !== false)) return "fail"; + if (findings.some((finding) => finding.severity === "env_issue")) return "env_issue"; + if (findings.some((finding) => finding.severity === "warning")) return "warning"; + if (successSignals.length > 0) return "pass"; + return "no_activity"; +} + +function strictSummaryExitCode(status: string): number { + return status === "fail" || status === "env_issue" ? 1 : 0; +} + +function sessionDir(options: Record): string { + return optionString(options, "output-dir") ?? join("reports", "log-guards"); +} + +function sessionPath(options: Record, runId: string): string { + return join(sessionDir(options), `${runId}.json`); +} + +function readSession(options: Record): LogGuardSession | undefined { + const runId = optionString(options, "run-id"); + const explicitSession = optionString(options, "session"); + const path = explicitSession ? resolve(explicitSession) : runId ? resolve(sessionPath(options, runId)) : ""; + if (!path || !existsSync(path)) return undefined; + return JSON.parse(readFileSync(path, "utf8")) as LogGuardSession; +} + +export function commandLogScan(ctx: CommandContext): number { + const { positional, options } = parseOptions(ctx.args.slice(2)); + if (positional.length > 0) usage(); + + const result = scanLogSources(ctx.root, options, patternContextFromOptions(ctx.root, options)); + const output = optionString(options, "output"); + const content = options.json === true + ? `${JSON.stringify(result, null, 2)}\n` + : renderLogGuardMarkdown("Log Guard Scan", result, [ + optionString(options, "case") ? `case: ${optionString(options, "case")}` : "case: none", + options.strict === true ? "strict: yes" : "strict: no", + ]); + writeOrPrint(content, output); + return options.strict === true ? strictLogGuardExitCode(result) : 0; +} + +export async function commandLogWatch(ctx: CommandContext): Promise { + const { positional, options } = parseOptions(ctx.args.slice(2)); + if (positional.length > 0) usage(); + + const path = latestOrExplicitBackendLog(ctx.root, options); + if (!path) { + console.error("ERROR: no backend log found; pass --backend-log or configure LANGBOT_REPO."); + return 1; + } + if (!existsSync(path)) { + console.error(`ERROR: backend log does not exist: ${path}`); + return 1; + } + + const context = patternContextFromOptions(ctx.root, options); + const intervalMs = positiveIntegerOption(options, "interval-ms", 1000); + const durationMs = optionString(options, "duration-ms") + ? positiveIntegerOption(options, "duration-ms", 0) + : 0; + const startedAtLocal = localIsoWithOffset(new Date()); + const findings: LogFinding[] = []; + const successSignals: LogSuccessSignal[] = []; + let bytesRead = 0; + let offset = options["from-start"] === true ? 0 : statSync(path).size; + let baseLineNumber = options["from-start"] === true + ? 0 + : readFileSync(path).subarray(0, offset).toString("utf8").split(/\r?\n/).length - 1; + let carry = ""; + + if (options.json !== true) { + console.log(`# Log Guard Watch`); + console.log(`Path: ${path}`); + console.log(`Started: ${startedAtLocal}`); + console.log(`Mode: ${options["from-start"] === true ? "from-start" : "new-lines"}`); + } + + const startedMs = Date.now(); + let stopRequested = false; + const stop = (): void => { + stopRequested = true; + }; + process.once("SIGINT", stop); + process.once("SIGTERM", stop); + + const poll = (): void => { + const buffer = readFileSync(path); + if (buffer.length < offset) { + offset = 0; + baseLineNumber = 0; + carry = ""; + } + if (buffer.length === offset) return; + + const chunk = buffer.subarray(offset).toString("utf8"); + offset = buffer.length; + bytesRead += Buffer.byteLength(chunk); + const text = `${carry}${chunk}`; + const hasCompleteLine = /\r?\n$/.test(text); + const lastNewline = Math.max(text.lastIndexOf("\n"), text.lastIndexOf("\r")); + if (!hasCompleteLine && lastNewline === -1) { + carry = text; + return; + } + + const complete = hasCompleteLine ? text : text.slice(0, lastNewline + 1); + carry = hasCompleteLine ? "" : text.slice(lastNewline + 1); + if (!complete) return; + + const result = scanLogText(ctx.root, "backend", path, complete, {}, context, baseLineNumber, false); + baseLineNumber += complete.split(/\r?\n/).length - 1; + findings.push(...result.findings); + successSignals.push(...result.success_signals); + + if (options.json !== true) { + for (const finding of result.findings) console.log(renderLogFinding(finding)); + for (const signal of result.success_signals) console.log(renderLogSuccessSignal(signal)); + } + }; + + try { + do { + poll(); + if (stopRequested) break; + if (durationMs > 0 && Date.now() - startedMs >= durationMs) break; + await delay(Math.min(intervalMs, durationMs > 0 ? Math.max(1, durationMs - (Date.now() - startedMs)) : intervalMs)); + } while (!stopRequested); + + if (carry) { + const result = scanLogText(ctx.root, "backend", path, carry, {}, context, baseLineNumber, false); + findings.push(...result.findings); + successSignals.push(...result.success_signals); + if (options.json !== true) { + for (const finding of result.findings) console.log(renderLogFinding(finding)); + for (const signal of result.success_signals) console.log(renderLogSuccessSignal(signal)); + } + } + } finally { + process.off("SIGINT", stop); + process.off("SIGTERM", stop); + } + + const summary: WatchSummary = { + mode: "watch", + status: statusFromEvents(findings, successSignals), + path, + started_at_local: startedAtLocal, + finished_at_local: localIsoWithOffset(new Date()), + bytes_read: bytesRead, + findings, + success_signals: successSignals, + }; + + if (options.json === true) { + console.log(JSON.stringify(summary, null, 2)); + } else { + console.log(`Status: ${summary.status}`); + console.log(`Bytes read: ${summary.bytes_read}`); + } + return options.strict === true ? strictSummaryExitCode(summary.status) : 0; +} + +export function commandLogGuard(ctx: CommandContext): number { + const sub = ctx.args[2]; + if (sub === "start") return commandLogGuardStart(ctx); + if (sub === "stop") return commandLogGuardStop(ctx); + usage(); +} + +function commandLogGuardStart(ctx: CommandContext): number { + const { positional, options } = parseOptions(ctx.args.slice(3)); + if (positional.length > 0) usage(); + + const now = new Date(); + const startedAtLocal = localIsoWithOffset(now); + const runId = optionString(options, "run-id") ?? `log-guard-${timestampSlug(startedAtLocal)}`; + const caseId = optionString(options, "case") ?? ""; + const caseItem = caseId ? findStructuredItem(ctx.root, "cases", caseId) : undefined; + const session: LogGuardSession = { + source: "log-guard-session", + run_id: runId, + started_at: now.toISOString(), + started_at_local: startedAtLocal, + backend_log: latestOrExplicitBackendLog(ctx.root, options), + case_id: caseId, + case_skill: caseItem?.skill ?? "", + }; + const path = resolve(sessionPath(options, runId)); + mkdirSync(dirname(path), { recursive: true }); + writeFileSync(path, `${JSON.stringify(session, null, 2)}\n`, "utf8"); + + const result = { + ...session, + path, + stop_command: `bin/lbs log guard stop --run-id ${runId} --output-dir ${sessionDir(options)}`, + }; + if (options.json === true) console.log(JSON.stringify(result, null, 2)); + else { + console.log(`# Log Guard Session`); + console.log(`Run: ${runId}`); + console.log(`Started: ${startedAtLocal}`); + console.log(`Session: ${path}`); + if (session.backend_log) console.log(`Backend log: ${session.backend_log}`); + if (session.case_id) console.log(`Case: ${session.case_id}`); + console.log(`Stop: ${result.stop_command}`); + } + return 0; +} + +function commandLogGuardStop(ctx: CommandContext): number { + const { positional, options } = parseOptions(ctx.args.slice(3)); + if (positional.length > 0) usage(); + + const session = readSession(options); + if (!session) { + console.error("ERROR: log guard session not found; pass --run-id with --output-dir or --session."); + return 1; + } + + const now = new Date(); + const scanOptions: Record = { + ...options, + since: optionString(options, "since") ?? session.started_at_local, + until: optionString(options, "until") ?? localIsoWithOffset(now), + }; + if (session.backend_log && typeof scanOptions["backend-log"] !== "string") { + scanOptions["backend-log"] = session.backend_log; + } + + const caseId = optionString(options, "case") ?? session.case_id; + const context = caseId + ? logPatternContextFromStructuredItem(findStructuredItem(ctx.root, "cases", caseId)) + : patternContextFromOptions(ctx.root, options); + const result = scanLogSources(ctx.root, scanOptions, context); + const output = optionString(options, "output") ?? join(sessionDir(options), `${session.run_id}.md`); + const content = options.json === true + ? `${JSON.stringify({ session, result }, null, 2)}\n` + : renderLogGuardMarkdown("Log Guard Report", result, [ + `run_id: ${session.run_id}`, + `started: ${session.started_at_local}`, + `finished: ${scanOptions.until}`, + caseId ? `case: ${caseId}` : "case: none", + ]); + writeOrPrint(content, options.json === true ? optionString(options, "output") : output); + return options["no-strict"] === true ? 0 : strictLogGuardExitCode(result); +} diff --git a/skills/src/commands/skill.ts b/skills/src/commands/skill.ts new file mode 100644 index 000000000..7efbc0d79 --- /dev/null +++ b/skills/src/commands/skill.ts @@ -0,0 +1,128 @@ +import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import type { CommandContext } from "../types.ts"; +import { fail, optionString, parseOptions, usage } from "../cli.ts"; +import { loadFixtureItems } from "../fixtures.ts"; +import { boolValue, getSkill, globMarkdownRefs, listValue, loadSkills, loadStructuredItems, scalar, skillsRoot } from "../fs.ts"; + +export function commandList(ctx: CommandContext): number { + for (const skill of loadSkills(ctx.root)) { + console.log(`${skill.directory}\t${skill.name}\t${skill.description}`); + } + return 0; +} + +function buildIndexData(root: string): Record { + const caseSummary = (item: ReturnType[number]) => ({ + id: scalar(item.fields, "id"), + title: scalar(item.fields, "title"), + mode: scalar(item.fields, "mode"), + area: scalar(item.fields, "area"), + type: scalar(item.fields, "type"), + priority: scalar(item.fields, "priority"), + risk: scalar(item.fields, "risk"), + ci_eligible: boolValue(item.fields, "ci_eligible") ?? false, + tags: listValue(item.fields, "tags"), + automation: scalar(item.fields, "automation"), + setup_automation: listValue(item.fields, "setup_automation"), + setup_provides_env: listValue(item.fields, "setup_provides_env"), + evidence_required: listValue(item.fields, "evidence_required"), + }); + const troubleshootingSummary = (item: ReturnType[number]) => ({ + id: scalar(item.fields, "id"), + title: scalar(item.fields, "title"), + category: scalar(item.fields, "category") || "product", + related_cases: listValue(item.fields, "related_cases"), + }); + const suiteSummary = (item: ReturnType[number]) => ({ + id: scalar(item.fields, "id"), + title: scalar(item.fields, "title"), + description: scalar(item.fields, "description"), + type: scalar(item.fields, "type"), + priority: scalar(item.fields, "priority"), + tags: listValue(item.fields, "tags"), + cases: listValue(item.fields, "cases"), + }); + return { + generated_by: "lbs", + skills: loadSkills(root).map((skill) => ({ + directory: skill.directory, + name: skill.name, + description: skill.description, + references: globMarkdownRefs(skill.path), + cases: loadStructuredItems(root, "cases", skill.directory).map((item) => scalar(item.fields, "id")), + case_summaries: loadStructuredItems(root, "cases", skill.directory).map(caseSummary), + suites: loadStructuredItems(root, "suites", skill.directory).map((item) => scalar(item.fields, "id")), + suite_summaries: loadStructuredItems(root, "suites", skill.directory).map(suiteSummary), + fixtures: loadFixtureItems(root, skill.directory).items.map((item) => ({ + id: item.id, + title: item.title, + kind: item.kind, + path: item.path, + related_cases: item.related_cases, + })), + troubleshooting: loadStructuredItems(root, "troubleshooting", skill.directory).map((item) => scalar(item.fields, "id")), + troubleshooting_summaries: loadStructuredItems(root, "troubleshooting", skill.directory).map(troubleshootingSummary), + })), + }; +} + +export function commandIndex(ctx: CommandContext): number { + const { options } = parseOptions(ctx.args.slice(1)); + const data = buildIndexData(ctx.root); + const out = join(ctx.root, "skills.index.json"); + const content = `${JSON.stringify(data, null, 2)}\n`; + if (options.check === true) { + if (!existsSync(out)) { + console.error(`ERROR: missing index: ${out}`); + return 1; + } + if (readFileSync(out, "utf8") !== content) { + console.error(`ERROR: index is stale: ${out}`); + return 1; + } + console.log(`OK ${out}`); + return 0; + } + writeFileSync(out, content, "utf8"); + console.log(out); + return 0; +} + +export function commandNewSkill(ctx: CommandContext): number { + const { positional, options } = parseOptions(ctx.args.slice(1)); + const name = positional[0]; + if (!name) usage(); + + const skillDir = join(skillsRoot(ctx.root), name); + const skillMd = join(skillDir, "SKILL.md"); + if (existsSync(skillMd)) fail(`skill already exists: ${skillDir}`); + + mkdirSync(skillDir, { recursive: true }); + const description = optionString(options, "description") ?? `Use when working with ${name}.`; + const text = + `---\nname: ${name}\ndescription: ${description}\n---\n\n` + + `# ${name}\n\n` + + "Add concise routing and workflow instructions here.\n"; + writeFileSync(skillMd, text, "utf8"); + console.log(skillMd); + return 0; +} + +export function commandNewRef(ctx: CommandContext): number { + const skill = ctx.args[1]; + const rawName = ctx.args[2]; + if (!skill || !rawName) usage(); + + const skillDir = getSkill(ctx.root, skill).path; + const refsDir = join(skillDir, "references"); + mkdirSync(refsDir, { recursive: true }); + const name = rawName.endsWith(".md") ? rawName : `${rawName}.md`; + const refPath = join(refsDir, name); + if (existsSync(refPath)) fail(`reference already exists: ${refPath}`); + + const title = name.replace(/\.md$/, "").replace(/-/g, " ").replace(/\b\w/g, (char) => char.toUpperCase()); + writeFileSync(refPath, `# ${title}\n\nAdd concise reusable instructions here.\n`, "utf8"); + console.log(refPath); + return 0; +} diff --git a/skills/src/commands/suite.ts b/skills/src/commands/suite.ts new file mode 100644 index 000000000..403156100 --- /dev/null +++ b/skills/src/commands/suite.ts @@ -0,0 +1,704 @@ +import { existsSync, mkdirSync, readFileSync, statSync, writeFileSync } from "node:fs"; +import { dirname, join, resolve } from "node:path"; +import { spawnSync } from "node:child_process"; +import { execPath } from "node:process"; +import type { CommandContext, StructuredItem } from "../types.ts"; +import { fail, optionString, parseOptions, usage } from "../cli.ts"; +import { findStructuredItem, getSkill, listValue, loadStructuredItems, scalar, yamlList, yamlQuote } from "../fs.ts"; +import { caseAutomationReadiness, caseEnvReadiness, caseFixtureReadiness, caseManualReadiness, runtimeEnv } from "../readiness.ts"; +import { lbsScriptPath, setupAutomationEntries } from "../setup-automation.ts"; + +function suitePath(root: string, skillName: string, id: string): string { + const skill = getSkill(root, skillName); + const dir = join(skill.path, "suites"); + mkdirSync(dir, { recursive: true }); + return join(dir, `${id}.yaml`); +} + +function caseItemById(root: string, id: string): StructuredItem { + return findStructuredItem(root, "cases", id); +} + +function suiteCaseSummary(root: string, id: string): Record { + const item = caseItemById(root, id); + const env = runtimeEnv(root); + const caseId = scalar(item.fields, "id"); + return { + skill: item.skill, + id: caseId, + title: scalar(item.fields, "title"), + mode: scalar(item.fields, "mode"), + area: scalar(item.fields, "area"), + type: scalar(item.fields, "type"), + priority: scalar(item.fields, "priority"), + risk: scalar(item.fields, "risk"), + tags: listValue(item.fields, "tags"), + preconditions: listValue(item.fields, "preconditions"), + setup: listValue(item.fields, "setup"), + setup_automation: setupAutomationEntries(item), + setup_provides_env: listValue(item.fields, "setup_provides_env"), + automation: scalar(item.fields, "automation"), + evidence_required: listValue(item.fields, "evidence_required"), + env_readiness: caseEnvReadiness(item, env), + automation_readiness: caseAutomationReadiness(item, env), + fixture_readiness: caseFixtureReadiness(root, caseId), + manual_readiness: caseManualReadiness(item), + }; +} + +function suiteSummary(item: StructuredItem): Record { + return { + skill: item.skill, + id: scalar(item.fields, "id"), + title: scalar(item.fields, "title"), + description: scalar(item.fields, "description"), + type: scalar(item.fields, "type"), + priority: scalar(item.fields, "priority"), + tags: listValue(item.fields, "tags"), + cases: listValue(item.fields, "cases"), + }; +} + +function findSuite(root: string, args: string[]): StructuredItem { + if (args.length < 1 || args.length > 2) usage(); + return args.length === 1 + ? findStructuredItem(root, "suites", args[0]) + : findStructuredItem(root, "suites", args[0], args[1]); +} + +function pad2(value: number): string { + return String(value).padStart(2, "0"); +} + +function pad3(value: number): string { + return String(value).padStart(3, "0"); +} + +function localIsoWithOffset(date: Date): string { + const offsetMinutes = -date.getTimezoneOffset(); + const sign = offsetMinutes >= 0 ? "+" : "-"; + const absoluteOffset = Math.abs(offsetMinutes); + return [ + `${date.getFullYear()}-${pad2(date.getMonth() + 1)}-${pad2(date.getDate())}`, + `T${pad2(date.getHours())}:${pad2(date.getMinutes())}:${pad2(date.getSeconds())}.${pad3(date.getMilliseconds())}`, + `${sign}${pad2(Math.floor(absoluteOffset / 60))}:${pad2(absoluteOffset % 60)}`, + ].join(""); +} + +function timestampSlug(localIso: string): string { + return localIso + .replace(/T/, "-") + .replace(/[.:+]/g, "-") + .replace(/[^A-Za-z0-9_-]+/g, "-") + .replace(/-+/g, "-") + .replace(/^-|-$/g, ""); +} + +function writeOrPrint(content: string, output: string | undefined): void { + if (!output) { + console.log(content.trimEnd()); + return; + } + const path = resolve(output); + mkdirSync(dirname(path), { recursive: true }); + writeFileSync(path, content, "utf8"); + console.log(path); +} + +function suiteCases(root: string, item: StructuredItem): Record[] { + return listValue(item.fields, "cases").map((id) => suiteCaseSummary(root, id)); +} + +function statusOf(caseItem: Record, key: string): string { + const value = caseItem[key] as Record | undefined; + return typeof value?.status === "string" ? value.status : "not_required"; +} + +function readinessSummary(cases: Array>): Record { + const missingEnv = cases.filter((item) => statusOf(item, "env_readiness") === "missing").map((item) => item.id); + const missingAutomation = cases.filter((item) => statusOf(item, "automation_readiness") === "missing").map((item) => item.id); + const missingFixture = cases.filter((item) => statusOf(item, "fixture_readiness") === "missing").map((item) => item.id); + const manualCheck = cases.filter((item) => statusOf(item, "manual_readiness") === "manual_check").map((item) => item.id); + const missingCount = missingEnv.length + missingAutomation.length + missingFixture.length; + return { + status: missingCount > 0 ? "missing" : manualCheck.length > 0 ? "manual_check" : "ready", + missing_env_cases: missingEnv, + missing_automation_env_cases: missingAutomation, + missing_fixture_cases: missingFixture, + manual_check_cases: manualCheck, + }; +} + +function hasProbeCases(cases: Array>): boolean { + return cases.some((caseItem) => caseItem.mode === "probe"); +} + +function suiteReportGuidance(cases: Array>): string { + return hasProbeCases(cases) + ? "Run each case according to its mode; probe cases may collect non-UI evidence, while agent-browser cases still require browser/UI execution." + : "Run each case through browser/UI first; use test report with the evidence directory and backend log window after execution."; +} + +function suiteResultPolicy(cases: Array>): string[] { + if (hasProbeCases(cases)) { + return [ + "A suite is not pass unless every case has a result and required evidence for the same run window.", + "agent-browser cases require UI/browser results; probe cases are judged by their declared checks and required evidence.", + "blocked and env_issue are not product pass; report them separately.", + ]; + } + + return [ + "A suite is not pass unless every case has a UI/browser result and required evidence for the same run window.", + "blocked and env_issue are not product pass; report them separately.", + ]; +} + +function suiteEvidencePolicy(cases: Array>): string[] { + if (hasProbeCases(cases)) { + return [ + "Run each case according to its mode. Agent-browser cases use browser/UI; probe cases use their declared probe steps or automation.", + "Use each case evidence_dir for screenshots, console.log, network.log, automation-result.json, result.json, and any probe artifacts.", + "After case execution and report review, run each result_command_template with the final status and collected evidence.", + "After per-case result.json files exist, run the suite report command to aggregate them.", + "blocked and env_issue are not product pass; they must be reported separately from pass.", + ]; + } + + return [ + "Run each case through browser/UI. API/curl/log diagnostics cannot make a UI case pass by themselves.", + "Use each case evidence_dir for screenshots, console.log, network.log, automation-result.json, and final result.json.", + "After case execution and report review, run each result_command_template with the final status and collected evidence.", + "After per-case result.json files exist, run the suite report command to aggregate them.", + "blocked and env_issue are not product pass; they must be reported separately from pass.", + ]; +} + +function buildSuitePlan(root: string, item: StructuredItem): Record { + const suite = suiteSummary(item); + const cases = suiteCases(root, item); + return { + ...suite, + cases, + readiness: readinessSummary(cases), + commands: cases.map((caseItem) => ({ + id: caseItem.id, + plan: `bin/lbs test plan ${caseItem.id}`, + start: `bin/lbs test start ${caseItem.id}`, + automation: caseItem.automation ? `bin/lbs test run ${caseItem.id} --dry-run` : "", + })), + report_guidance: suiteReportGuidance(cases), + }; +} + +export function commandSuiteNew(ctx: CommandContext): number { + const { positional, options } = parseOptions(ctx.args.slice(2)); + const id = positional[0]; + const title = optionString(options, "title"); + if (!id || !title) usage(); + + const skill = optionString(options, "skill") ?? "langbot-testing"; + const path = suitePath(ctx.root, skill, id); + if (existsSync(path)) fail(`suite already exists: ${path}`); + + const text = + `id: ${id}\n` + + `title: ${yamlQuote(title)}\n` + + `description: ${yamlQuote(optionString(options, "description") ?? "Describe when to run this suite.")}\n` + + `type: ${optionString(options, "type") ?? "smoke"}\n` + + `priority: ${optionString(options, "priority") ?? "p2"}\n` + + "tags:\n" + + yamlList([optionString(options, "type") ?? "smoke"]) + + "\ncases:\n" + + yamlList(["webui-login-state"]) + + "\n"; + + writeFileSync(path, text, "utf8"); + console.log(path); + return 0; +} + +export function commandSuiteList(ctx: CommandContext): number { + const { positional, options } = parseOptions(ctx.args.slice(2)); + const skill = positional[0]; + const rows = loadStructuredItems(ctx.root, "suites", skill) + .map(suiteSummary) + .filter((row) => !optionString(options, "type") || row.type === optionString(options, "type")) + .filter((row) => !optionString(options, "priority") || row.priority === optionString(options, "priority")); + + if (options.json === true) { + console.log(JSON.stringify(rows, null, 2)); + return 0; + } + + for (const row of rows) { + console.log([ + row.skill, + row.id, + row.type, + row.priority, + Array.isArray(row.cases) ? row.cases.length : 0, + row.title, + ].join("\t")); + } + return 0; +} + +export function commandSuiteShow(ctx: CommandContext): number { + const item = findSuite(ctx.root, ctx.args.slice(2)); + console.log(item.raw.trimEnd()); + return 0; +} + +export function commandSuitePlan(ctx: CommandContext): number { + const { positional: args, options } = parseOptions(ctx.args.slice(2)); + const item = findSuite(ctx.root, args); + const plan = buildSuitePlan(ctx.root, item); + const suite = suiteSummary(item); + const cases = suiteCases(ctx.root, item); + + if (options.json === true) { + console.log(JSON.stringify(plan, null, 2)); + return 0; + } + + console.log(`# Suite Plan: ${suite.id}`); + console.log(""); + console.log(`Title: ${suite.title}`); + console.log(`Type: ${suite.type}`); + console.log(`Priority: ${suite.priority}`); + console.log(`Description: ${suite.description}`); + console.log(""); + const readiness = readinessSummary(cases); + console.log("## Readiness"); + console.log(`Status: ${readiness.status}`); + for (const [key, value] of Object.entries(readiness)) { + if (key === "status" || !Array.isArray(value) || value.length === 0) continue; + console.log(`- ${key}: ${value.join(", ")}`); + } + console.log(""); + console.log("## Cases"); + for (const [index, caseItem] of cases.entries()) { + console.log(`${index + 1}. ${caseItem.id} [${caseItem.priority}/${caseItem.risk}] ${caseItem.title}`); + console.log(` - plan: bin/lbs test plan ${caseItem.id}`); + console.log(` - start: bin/lbs test start ${caseItem.id}`); + if (caseItem.automation) console.log(` - automation dry-run: bin/lbs test run ${caseItem.id} --dry-run`); + console.log(` - evidence: ${Array.isArray(caseItem.evidence_required) ? caseItem.evidence_required.join(", ") : ""}`); + const envReadiness = caseItem.env_readiness as Record; + const automationReadiness = caseItem.automation_readiness as Record; + const fixtureReadiness = caseItem.fixture_readiness as Record; + const manualReadiness = caseItem.manual_readiness as Record; + const missing: string[] = []; + if (Array.isArray(envReadiness.missing) && envReadiness.missing.length > 0) missing.push(`env=${envReadiness.missing.join(",")}`); + if (Array.isArray(automationReadiness.missing) && automationReadiness.missing.length > 0) missing.push(`automation_env=${automationReadiness.missing.join(",")}`); + if (Array.isArray(fixtureReadiness.missing) && fixtureReadiness.missing.length > 0) missing.push(`fixture=${fixtureReadiness.missing.join(",")}`); + const manualLabel = manualReadiness.status === "manual_check" ? " manual_check" : ""; + console.log(` - readiness: ${missing.length === 0 ? `ready${manualLabel}` : `missing ${missing.join(" ")}`}`); + const preconditions = caseItem.preconditions; + if (Array.isArray(preconditions) && preconditions.length > 0) console.log(` - preconditions: ${preconditions.length}`); + const setupAutomation = caseItem.setup_automation; + if (Array.isArray(setupAutomation) && setupAutomation.length > 0) console.log(` - setup automation: ${setupAutomation.length}`); + } + console.log(""); + console.log("## Result Policy"); + for (const policy of suiteResultPolicy(cases)) console.log(`- ${policy}`); + return 0; +} + +function suiteStartPath(root: string, path: string): string { + return resolve(root, path); +} + +function ensureDirectory(root: string, path: string, label: string): void { + const resolvedPath = suiteStartPath(root, path); + if (existsSync(resolvedPath) && !statSync(resolvedPath).isDirectory()) { + fail(`${label} exists and is not a directory: ${resolvedPath}`); + } + mkdirSync(resolvedPath, { recursive: true }); +} + +function buildSuiteStart( + root: string, + item: StructuredItem, + args: string[], + options: Record, +): Record { + const now = new Date(); + const startedAtLocal = localIsoWithOffset(now); + const suite = suiteSummary(item); + const suiteId = String(suite.id); + const runId = optionString(options, "run-id") ?? `${timestampSlug(startedAtLocal)}-${suiteId}`; + const evidenceRoot = optionString(options, "evidence-dir") ?? join("reports", "evidence", runId); + const reportPath = join("reports", `${runId}.md`); + const manifestPath = join(evidenceRoot, "suite-start.json"); + const handoffPath = join(evidenceRoot, "suite-start.md"); + const cases = suiteCases(root, item).map((caseItem) => { + const caseId = String(caseItem.id); + const caseRunId = `${runId}-${caseId}`; + const evidenceDir = join(evidenceRoot, caseId); + const consoleLog = join(evidenceDir, "console.log"); + const caseReportPath = join("reports", `${caseRunId}.md`); + return { + ...caseItem, + run_id: caseRunId, + evidence_dir: evidenceDir, + plan_command: `bin/lbs test plan ${caseId}`, + start_command: `bin/lbs test start ${caseId}`, + automation_command: caseItem.automation + ? `bin/lbs test run ${caseId} --run-id ${caseRunId} --output ${evidenceDir}` + : "", + report_command: caseItem.automation + ? `bin/lbs test report ${caseId} --since "${startedAtLocal}" --console-log ${consoleLog} --evidence-dir ${evidenceDir} --output ${caseReportPath}` + : `bin/lbs test report ${caseId} --since "${startedAtLocal}" --evidence-dir ${evidenceDir} --output ${caseReportPath}`, + result_command_template: `bin/lbs test result ${caseId} --result --reason "" --evidence-dir ${evidenceDir} --run-id ${caseRunId} --started-at "${startedAtLocal}" --evidence ${Array.isArray(caseItem.evidence_required) ? caseItem.evidence_required.join(",") : ""}`, + }; + }); + + const locator = args.join(" "); + return { + run_id: runId, + started_at: now.toISOString(), + started_at_local: startedAtLocal, + suite, + evidence_root: evidenceRoot, + manifest_path: manifestPath, + handoff_path: handoffPath, + cases, + suite_report_path: reportPath, + plan_command: `bin/lbs suite plan ${locator}`, + report_command: `bin/lbs suite report ${locator} --run-id ${runId} --evidence-dir ${evidenceRoot} --output ${reportPath}`, + evidence_policy: suiteEvidencePolicy(cases), + }; +} + +function writeSuiteStartArtifacts(root: string, start: Record, rendered: string): void { + const evidenceRoot = String(start.evidence_root || ""); + if (!evidenceRoot) return; + + ensureDirectory(root, evidenceRoot, "suite evidence directory"); + for (const caseItem of start.cases as Array>) { + const evidenceDir = String(caseItem.evidence_dir || ""); + if (evidenceDir) ensureDirectory(root, evidenceDir, "case evidence directory"); + } + + const manifestPath = String(start.manifest_path || ""); + if (manifestPath) { + const path = suiteStartPath(root, manifestPath); + mkdirSync(dirname(path), { recursive: true }); + writeFileSync(path, `${JSON.stringify(start, null, 2)}\n`, "utf8"); + } + + const handoffPath = String(start.handoff_path || ""); + if (handoffPath) { + const path = suiteStartPath(root, handoffPath); + mkdirSync(dirname(path), { recursive: true }); + writeFileSync(path, rendered, "utf8"); + } +} + +function renderSuiteStart(start: Record): string { + const suite = start.suite as Record; + const cases = start.cases as Array>; + const lines: string[] = []; + lines.push(`# Suite Start: ${suite.id}`); + lines.push(""); + lines.push(`Run: ${start.run_id}`); + lines.push(`Started: ${start.started_at_local}`); + lines.push(`Title: ${suite.title}`); + lines.push(`Evidence root: ${start.evidence_root}`); + lines.push(""); + lines.push("## Commands"); + lines.push(`- plan: ${start.plan_command}`); + lines.push(`- report: ${start.report_command}`); + lines.push(""); + lines.push("## Cases"); + for (const [index, caseItem] of cases.entries()) { + lines.push(`${index + 1}. ${caseItem.id} [${caseItem.priority}/${caseItem.risk}] ${caseItem.title}`); + lines.push(` - evidence_dir: ${caseItem.evidence_dir}`); + lines.push(` - plan: ${caseItem.plan_command}`); + if (caseItem.automation_command) lines.push(` - automation: ${caseItem.automation_command}`); + else lines.push(` - manual start: ${caseItem.start_command}`); + lines.push(` - report: ${caseItem.report_command}`); + lines.push(` - result template: ${caseItem.result_command_template}`); + } + lines.push(""); + lines.push("## Evidence Policy"); + for (const item of start.evidence_policy as string[]) lines.push(`- ${item}`); + return `${lines.join("\n").trimEnd()}\n`; +} + +export function commandSuiteStart(ctx: CommandContext): number { + const { positional: args, options } = parseOptions(ctx.args.slice(2)); + const item = findSuite(ctx.root, args); + const start = buildSuiteStart(ctx.root, item, args, options); + const rendered = renderSuiteStart(start); + writeSuiteStartArtifacts(ctx.root, start, rendered); + const content = options.json === true ? `${JSON.stringify(start, null, 2)}\n` : rendered; + writeOrPrint(content, optionString(options, "output")); + return 0; +} + +function suiteRunCaseArgs(root: string, caseItem: Record, headed: boolean): string[] { + const args = [ + lbsScriptPath(), + "--root", + root, + "test", + "run", + String(caseItem.id), + "--run-id", + String(caseItem.run_id), + "--output", + String(caseItem.evidence_dir), + ]; + if (headed) args.push("--headed"); + return args; +} + +function suiteReportExitCode(status: string): number { + if (status === "pass") return 0; + if (status === "blocked" || status === "env_issue" || status === "flaky") return 2; + return 1; +} + +function outputTail(value: string | Buffer | null | undefined): string { + return String(value ?? "").trim().slice(-4000); +} + +function executionProblemStatus(executions: Array>): string { + const statuses = executions.map((item) => String(item.status)); + if (statuses.includes("nonzero")) return "fail"; + if (statuses.includes("skipped")) return "incomplete"; + return ""; +} + +function missingReadinessReason(caseItem: Record): string { + const labels: Array<[string, string]> = [ + ["env", "env_readiness"], + ["automation_env", "automation_readiness"], + ["fixture", "fixture_readiness"], + ]; + const missing = labels.flatMap(([label, key]) => { + const value = caseItem[key] as Record | undefined; + if (value?.status !== "missing") return []; + const names = Array.isArray(value.missing) ? value.missing.filter((item): item is string => typeof item === "string") : []; + return [`${label}=${names.length > 0 ? names.join(",") : "missing"}`]; + }); + return missing.length > 0 + ? `case readiness missing (${missing.join(" ")}); rerun with --include-not-ready after fixing or intentionally accepting readiness gaps` + : ""; +} + +export function commandSuiteRun(ctx: CommandContext): number { + const { positional: args, options } = parseOptions(ctx.args.slice(2)); + const item = findSuite(ctx.root, args); + const start = buildSuiteStart(ctx.root, item, args, options); + const renderedStart = renderSuiteStart(start); + const dryRun = options["dry-run"] === true; + if (!dryRun) writeSuiteStartArtifacts(ctx.root, start, renderedStart); + + const executions = []; + for (const caseItem of start.cases as Array>) { + if (statusOf(caseItem, "manual_readiness") === "manual_check" && options["include-manual-check"] !== true) { + executions.push({ id: caseItem.id, status: "skipped", reason: "case requires manual_check; rerun with --include-manual-check after confirming preconditions" }); + continue; + } + const missingReadiness = missingReadinessReason(caseItem); + if (missingReadiness && options["include-not-ready"] !== true) { + executions.push({ id: caseItem.id, status: "skipped", reason: missingReadiness }); + continue; + } + if (!caseItem.automation) { + executions.push({ id: caseItem.id, status: "skipped", reason: "case has no automation" }); + continue; + } + const runArgs = suiteRunCaseArgs(ctx.root, caseItem, options.headed === true); + if (dryRun) { + executions.push({ id: caseItem.id, status: "planned", reason: "dry-run; case automation not executed", command: [execPath, ...runArgs].join(" ") }); + continue; + } + if (options.json !== true) console.log(`Suite case: ${caseItem.id}`); + const result = spawnSync(execPath, runArgs, { + cwd: ctx.root, + encoding: "utf8", + stdio: options.json === true ? "pipe" : "inherit", + }); + const status = result.error ? 1 : result.status ?? 1; + executions.push({ + id: caseItem.id, + status: status === 0 ? "ok" : "nonzero", + exit_status: status, + reason: result.error?.message || "", + stdout: outputTail(result.stdout), + stderr: outputTail(result.stderr), + }); + } + + const report = buildSuiteReport(ctx.root, item, { + ...options, + "run-id": String(start.run_id), + "evidence-dir": String(start.evidence_root), + }, executions); + const payload = { + run_id: start.run_id, + evidence_root: start.evidence_root, + executions, + report, + }; + const content = options.json === true + ? `${JSON.stringify(payload, null, 2)}\n` + : renderSuiteReport(report); + writeOrPrint(content, optionString(options, "output") ?? (options.json === true || dryRun ? undefined : String(start.suite_report_path || ""))); + return dryRun ? 0 : suiteReportExitCode(String(report.status)); +} + +function arrayField(data: Record, key: string): string[] { + const value = data[key]; + return Array.isArray(value) ? value.filter((item): item is string => typeof item === "string") : []; +} + +function readCaseResult(evidenceDir: string, caseId: string, expectedRunId: string, requiredEvidence: string[]): Record { + const resultPath = join(evidenceDir, "result.json"); + if (!existsSync(resultPath)) { + return { status: "missing", path: resultPath, reason: "result.json not found" }; + } + try { + const parsed = JSON.parse(readFileSync(resultPath, "utf8")) as Record; + if (parsed.case_id !== caseId) { + return { + status: "invalid", + path: resultPath, + reason: `result.json case_id mismatch: expected ${caseId}, got ${String(parsed.case_id ?? "missing")}`, + }; + } + if (expectedRunId && parsed.run_id !== expectedRunId) { + return { + status: "invalid", + path: resultPath, + reason: `result.json run_id mismatch: expected ${expectedRunId}, got ${String(parsed.run_id ?? "missing")}`, + }; + } + const collected = arrayField(parsed, "evidence_collected"); + const missing = requiredEvidence.filter((item) => !collected.includes(item)); + return { + status: typeof parsed.status === "string" ? parsed.status : "invalid", + path: resultPath, + reason: typeof parsed.reason === "string" ? parsed.reason : "", + started_at_local: typeof parsed.started_at_local === "string" ? parsed.started_at_local : "", + finished_at_local: typeof parsed.finished_at_local === "string" ? parsed.finished_at_local : "", + url: typeof parsed.url === "string" ? parsed.url : "", + evidence_collected: collected, + evidence_required: requiredEvidence, + evidence_missing: missing, + evidence_status: missing.length === 0 ? "complete" : "incomplete", + }; + } catch (error) { + return { status: "invalid", path: resultPath, reason: String(error) }; + } +} + +function suiteStatus(caseResults: Array>): string { + const statuses = caseResults.map((item) => String(item.status)); + if (statuses.length === 0) return "not_run"; + if (statuses.includes("fail") || statuses.includes("invalid")) return "fail"; + if (statuses.includes("missing")) return "incomplete"; + if (caseResults.some((item) => item.status === "pass" && item.evidence_status !== "complete")) return "incomplete"; + if (statuses.every((status) => status === "pass")) return "pass"; + if (statuses.includes("blocked")) return "blocked"; + if (statuses.includes("env_issue")) return "env_issue"; + if (statuses.includes("flaky")) return "flaky"; + return "unknown"; +} + +function buildSuiteReport( + root: string, + item: StructuredItem, + options: Record, + executions: Array> = [], +): Record { + const suite = suiteSummary(item); + const runId = optionString(options, "run-id") ?? ""; + const evidenceRoot = optionString(options, "evidence-dir") ?? (runId ? join("reports", "evidence", runId) : ""); + const cases = suiteCases(root, item).map((caseItem) => { + const caseId = String(caseItem.id); + const expectedCaseRunId = runId ? `${runId}-${caseId}` : ""; + const evidenceDir = evidenceRoot ? join(evidenceRoot, caseId) : ""; + const requiredEvidence = Array.isArray(caseItem.evidence_required) ? caseItem.evidence_required : []; + const result = evidenceDir + ? readCaseResult(evidenceDir, caseId, expectedCaseRunId, requiredEvidence) + : { status: "missing", path: "", reason: "Set --evidence-dir or --run-id to locate case result.json files" }; + return { + ...caseItem, + evidence_dir: evidenceDir, + result, + }; + }); + const counts: Record = {}; + for (const item of cases) { + const status = String((item.result as Record).status); + counts[status] = (counts[status] ?? 0) + 1; + } + + const resultStatus = suiteStatus(cases.map((item) => item.result as Record)); + const executionStatus = executionProblemStatus(executions); + return { + generated_at: new Date().toISOString(), + run_id: runId, + suite, + evidence_root: evidenceRoot, + status: executionStatus || resultStatus, + counts, + cases, + execution_status: executionStatus || "ok", + decision_policy: [ + "pass requires every case result to be pass.", + "suite run pass also requires every attempted execution to finish ok.", + "blocked and env_issue are not product pass.", + "pass results missing required evidence keep the suite incomplete.", + "result.json must match the expected case_id and suite case run_id.", + "missing or invalid result.json means the suite is incomplete or failed to collect evidence.", + ], + }; +} + +function renderSuiteReport(report: Record): string { + const suite = report.suite as Record; + const cases = report.cases as Array>; + const counts = report.counts as Record; + const lines: string[] = []; + lines.push(`# Suite Report: ${suite.id}`); + lines.push(""); + lines.push(`Generated: ${report.generated_at}`); + if (report.run_id) lines.push(`Run: ${report.run_id}`); + lines.push(`Title: ${suite.title}`); + lines.push(`Status: ${report.status}`); + lines.push(`Evidence root: ${report.evidence_root || "not provided"}`); + lines.push(""); + lines.push("## Counts"); + for (const key of Object.keys(counts).sort()) lines.push(`- ${key}: ${counts[key]}`); + if (Object.keys(counts).length === 0) lines.push("- None."); + lines.push(""); + lines.push("## Cases"); + for (const caseItem of cases) { + const result = caseItem.result as Record; + lines.push(`- ${caseItem.id}: ${result.status} - ${result.reason || "no reason"}`); + if (Array.isArray(result.evidence_missing) && result.evidence_missing.length > 0) { + lines.push(` evidence_missing: ${result.evidence_missing.join(", ")}`); + } + if (caseItem.evidence_dir) lines.push(` evidence_dir: ${caseItem.evidence_dir}`); + if (result.path) lines.push(` result_json: ${result.path}`); + } + lines.push(""); + lines.push("## Decision Policy"); + for (const item of report.decision_policy as string[]) lines.push(`- ${item}`); + return `${lines.join("\n").trimEnd()}\n`; +} + +export function commandSuiteReport(ctx: CommandContext): number { + const { positional: args, options } = parseOptions(ctx.args.slice(2)); + const item = findSuite(ctx.root, args); + const report = buildSuiteReport(ctx.root, item, options); + const content = options.json === true ? `${JSON.stringify(report, null, 2)}\n` : renderSuiteReport(report); + writeOrPrint(content, optionString(options, "output")); + return 0; +} diff --git a/skills/src/commands/test.ts b/skills/src/commands/test.ts new file mode 100644 index 000000000..2cce7a1e5 --- /dev/null +++ b/skills/src/commands/test.ts @@ -0,0 +1,1424 @@ +import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { dirname, join, resolve } from "node:path"; +import { spawnSync } from "node:child_process"; +import { env as processEnv, execPath } from "node:process"; +import type { CommandContext, StructuredItem } from "../types.ts"; +import { parseOptions, usage } from "../cli.ts"; +import { caseEvidenceValues, testResultStatusValues } from "../constants.ts"; +import { boolValue, findStructuredItem, listValue, loadEnv, loadStructuredItems, scalar } from "../fs.ts"; +import { splitEnvAnyGroup } from "../env-groups.ts"; +import { + readAutomationResultEvidence, + renderLogFinding, + renderLogSuccessSignal, + scanStructuredLogSources, + type AutomationResultEvidence, + type LogFinding, + type LogGuardResult, + type LogSuccessSignal, +} from "../log-guard.ts"; +import { + automationEnvDefaults, + caseAutomationReadiness, + caseEnvReadiness, + caseFixtureReadiness, + caseManualReadiness, + redactEnvValue, + resolvedAutomationEnvOverrides, + runtimeEnv, + type AutomationReadiness, + type EnvReadiness, + type FixtureReadiness, + type ManualReadiness, +} from "../readiness.ts"; +import { + lbsScriptPath, + parseSetupAutomationEntry, + setupAutomationEntries, + setupAutomationEvidenceName, + setupAutomationScriptPath, +} from "../setup-automation.ts"; + +type TroubleshootingSummary = { + id: string; + title: string; + patterns: string[]; + verification: string; +}; + +type TestPlan = { + id: string; + title: string; + mode: string; + principle: string; + env: Record; + env_readiness: EnvReadiness; + automation_readiness: AutomationReadiness; + fixture_readiness: FixtureReadiness; + manual_readiness: ManualReadiness; + required_skills: string[]; + preconditions: string[]; + setup: string[]; + setup_automation: string[]; + setup_provides_env: string[]; + cleanup: string[]; + steps: string[]; + checks: string[]; + diagnostics: string[]; + visual_checks: string[]; + evidence_required: string[]; + success_patterns: string[]; + failure_patterns: string[]; + troubleshooting: TroubleshootingSummary[]; + report_template: Record; +}; + +type TestStart = { + run_id: string; + started_at: string; + started_at_local: string; + case: Record; + environment: Record; + required_skills: string[]; + preconditions: string[]; + setup: string[]; + setup_automation: string[]; + setup_provides_env: string[]; + cleanup: string[]; + steps: string[]; + checks: string[]; + success_patterns: string[]; + failure_patterns: string[]; + evidence_required: string[]; + automation?: { + script: string; + command: string; + evidence_dir: string; + }; + recommended_report_path: string; + plan_command: string; + report_command: string; + result_command_template: string; + evidence_checklist: string[]; +}; + +type TestAutomationRun = { + run_id: string; + started_at: string; + started_at_local: string; + case: Record; + setup_automation: SetupAutomation[]; + automation: { + script: string; + script_path: string; + exists: boolean; + required_env: string[]; + evidence_dir: string; + console_log: string; + network_log: string; + screenshot: string; + automation_result_json: string; + result_json: string; + command: string; + report_command: string; + env_defaults: Record; + env_aliases: Array<{ + target: string; + source: string; + configured: boolean; + }>; + pipeline_env_required: boolean; + }; +}; + +type SetupAutomation = { + entry: string; + kind: "case" | "node"; + target: string; + args: string[]; + command: string; + dry_run_command: string; + evidence_dir: string; + exists: boolean; +}; + +type TestResultRecord = { + source: "final"; + case_id: string; + run_id: string; + written_at: string; + written_at_local: string; + started_at: string; + started_at_local: string; + finished_at: string; + finished_at_local: string; + status: string; + reason: string; + url: string; + browser_path: string; + evidence_dir: string; + evidence_collected: string[]; + evidence_required: string[]; + evidence_missing: string[]; + evidence_status: "complete" | "incomplete"; + report_path: string; + notes: string; +}; + +type ManualEvidenceTemplate = { + result: string; + [key: string]: string; +}; + +type TestReport = { + generated_at: string; + case: Record; + result_options: string[]; + automation_result: AutomationResultEvidence; + manual_evidence: ManualEvidenceTemplate; + environment: Record; + required_skills: string[]; + steps: string[]; + checks: string[]; + diagnostics: string[]; + evidence_required: string[]; + success_patterns: string[]; + failure_patterns: string[]; + expected_failures: string[]; + troubleshooting: TroubleshootingSummary[]; + log_guard: LogGuardResult; +}; + +type TestRecommendation = { + id: string; + reason: string; +}; + +type TestRecommendReport = { + generated_at: string; + changed_files: string[]; + recommendations: TestRecommendation[]; + commands: string[]; + notes: string[]; +}; + +function relatedTroubleshooting(root: string, item: StructuredItem): StructuredItem[] { + return listValue(item.fields, "troubleshooting") + .map((id) => { + try { + return findStructuredItem(root, "troubleshooting", id); + } catch { + return null; + } + }) + .filter((entry): entry is StructuredItem => entry !== null); +} + +function findCase(root: string, args: string[]): StructuredItem { + if (args.length < 1 || args.length > 2) usage(); + + return args.length === 1 + ? findStructuredItem(root, "cases", args[0]) + : findStructuredItem(root, "cases", args[0], args[1]); +} + +function caseSummary(item: StructuredItem): Record { + return { + skill: item.skill, + id: scalar(item.fields, "id"), + title: scalar(item.fields, "title"), + mode: scalar(item.fields, "mode"), + area: scalar(item.fields, "area"), + type: scalar(item.fields, "type"), + priority: scalar(item.fields, "priority"), + risk: scalar(item.fields, "risk"), + ci_eligible: boolValue(item.fields, "ci_eligible") ?? false, + tags: listValue(item.fields, "tags"), + }; +} + +function caseMode(item: StructuredItem): string { + return scalar(item.fields, "mode") || "agent-browser"; +} + +function isProbeMode(mode: string): boolean { + return mode === "probe"; +} + +function modePrinciple(mode: string): string { + return isProbeMode(mode) + ? "Run the declared probe steps and collect the required evidence. Browser/UI interaction is not required unless the case steps explicitly call for it." + : "Use browser/UI interaction as the primary QA path. API/curl/log checks are diagnostic only and cannot make a UI case pass by themselves."; +} + +function stepHeading(mode: string): string { + return isProbeMode(mode) ? "Probe Steps" : "Browser Steps"; +} + +function visualChecks(mode: string): string[] { + if (isProbeMode(mode)) return []; + return [ + "If the active agent has screenshot/vision capability, capture before/after screenshots.", + "Look for blank pages, overlapping text, hidden primary actions, error toasts, or broken layout.", + "If no visual model is available, use DOM/accessibility snapshots and console output instead.", + ]; +} + +function reportTemplate(mode: string): Record { + if (isProbeMode(mode)) { + return { + result: "pass | fail | blocked | env_issue | flaky", + target_tested: "Probe target, endpoint, file, command, or service actually checked", + execution_path: "automation script | shell command | direct API | other", + probe_result: "What the probe observed", + logs_or_artifacts: "Log, filesystem, API, or other artifact paths collected", + diagnostics: "Extra diagnostics used, if any", + matched_troubleshooting: "Troubleshooting ids matched, if any", + assets_to_update: "New case/reference/troubleshooting entries to add", + }; + } + + return { + result: "pass | fail | blocked | env_issue | flaky", + url_tested: "LANGBOT_FRONTEND_URL actually opened", + browser_path: "Computer Use | Playwright MCP | other", + ui_result: "What the user-visible UI showed", + console_errors: "Unexpected browser console errors, if any", + backend_logs: "Relevant backend log lines, if checked", + screenshots: "Screenshot paths or skipped reason", + diagnostics: "API/curl/log diagnostics used, if any", + matched_troubleshooting: "Troubleshooting ids matched, if any", + assets_to_update: "New case/reference/troubleshooting entries to add", + }; +} + +function evidenceChecklist(mode: string): string[] { + if (isProbeMode(mode)) { + return [ + "Execute the declared probe steps or automation script.", + "Store required logs, API diagnostics, filesystem artifacts, or other evidence in the evidence directory.", + "After execution, run the report command to scan logs from the start timestamp.", + "Write a final result.json with the result command only after required evidence has been collected.", + "Mark the final result as pass, fail, blocked, env_issue, or flaky in the generated report.", + ]; + } + + return [ + "Open the configured LangBot WebUI and execute the browser steps.", + "Capture screenshot paths when screenshot/vision tooling is available.", + "Record unexpected console errors and failed network requests without pasting secrets.", + "After browser execution, run the report command to scan logs from the start timestamp.", + "Write a final result.json with the result command only after required evidence has been collected.", + "Mark the final result as pass, fail, blocked, env_issue, or flaky in the generated report.", + ]; +} + +function manualEvidenceTemplate(mode: string): ManualEvidenceTemplate { + if (isProbeMode(mode)) { + return { + result: "pass | fail | blocked | env_issue | flaky", + target_tested: "TODO: probe target, endpoint, file, command, or service actually checked", + execution_path: "TODO: automation script | shell command | direct API | other", + probe_result: "TODO: observed probe result", + logs_or_artifacts: "TODO: evidence paths or skipped reason", + diagnostics: "TODO: additional diagnostics used, if any", + matched_troubleshooting: "TODO: troubleshooting ids matched, if any", + assets_to_update: "TODO: case/reference/troubleshooting updates to make", + }; + } + + return { + result: "pass | fail | blocked | env_issue | flaky", + url_tested: "LANGBOT_FRONTEND_URL actually opened", + browser_path: "Computer Use | Playwright MCP | direct Playwright | other", + ui_result: "TODO: user-visible result", + console_errors: "TODO: unexpected browser console errors or none", + network_symptoms: "TODO: failed requests, websocket issues, or none", + backend_logs: "TODO: relevant backend log lines or skipped reason", + frontend_logs: "TODO: relevant frontend dev-server log lines or skipped reason", + screenshots: "TODO: screenshot paths or skipped reason", + diagnostics: "TODO: API/curl/log diagnostics used, if any", + matched_troubleshooting: "TODO: troubleshooting ids matched, if any", + assets_to_update: "TODO: case/reference/troubleshooting updates to make", + }; +} + +function envSummary(item: StructuredItem, env: Record): Record { + const keys = [ + ...listValue(item.fields, "env"), + ...listValue(item.fields, "env_any").flatMap(splitEnvAnyGroup), + ]; + return Object.fromEntries(Array.from(new Set(keys)).map((key) => [key, redactEnvValue(key, env[key] ?? "")])); +} + +function buildPlan(root: string, item: StructuredItem): TestPlan { + const env = runtimeEnv(root); + const troubles = relatedTroubleshooting(root, item); + const id = scalar(item.fields, "id"); + const mode = caseMode(item); + return { + id, + title: scalar(item.fields, "title"), + mode, + principle: modePrinciple(mode), + env: envSummary(item, env), + env_readiness: caseEnvReadiness(item, env), + automation_readiness: caseAutomationReadiness(item, env), + fixture_readiness: caseFixtureReadiness(root, id), + manual_readiness: caseManualReadiness(item), + required_skills: listValue(item.fields, "skills"), + preconditions: listValue(item.fields, "preconditions"), + setup: listValue(item.fields, "setup"), + setup_automation: setupAutomationEntries(item), + setup_provides_env: listValue(item.fields, "setup_provides_env"), + cleanup: listValue(item.fields, "cleanup"), + steps: listValue(item.fields, "steps"), + checks: listValue(item.fields, "checks"), + diagnostics: listValue(item.fields, "diagnostics"), + visual_checks: visualChecks(mode), + evidence_required: listValue(item.fields, "evidence_required"), + success_patterns: listValue(item.fields, "success_patterns"), + failure_patterns: listValue(item.fields, "failure_patterns"), + troubleshooting: troubles.map((entry) => ({ + id: scalar(entry.fields, "id"), + title: scalar(entry.fields, "title"), + patterns: listValue(entry.fields, "patterns"), + verification: scalar(entry.fields, "verification"), + })), + report_template: reportTemplate(mode), + }; +} + +export function commandTestPlan(ctx: CommandContext): number { + const { positional: args, options } = parseOptions(ctx.args.slice(2)); + const item = findCase(ctx.root, args); + const plan = buildPlan(ctx.root, item); + + if (options.json === true) { + console.log(JSON.stringify(plan, null, 2)); + return 0; + } + + console.log(`# Test Plan: ${plan.id}`); + console.log(""); + console.log(`Title: ${plan.title}`); + console.log(`Mode: ${plan.mode}`); + console.log(""); + console.log("## Principle"); + console.log(plan.principle); + console.log(""); + console.log("## Environment"); + for (const [key, value] of Object.entries(plan.env)) console.log(`- ${key}=${value}`); + if (plan.env_readiness.missing.length > 0) console.log(`- missing: ${plan.env_readiness.missing.join(", ")}`); + console.log(""); + console.log("## Automation Readiness"); + console.log(`- status: ${plan.automation_readiness.status}`); + if (plan.automation_readiness.script) console.log(`- script: ${plan.automation_readiness.script}`); + if (plan.automation_readiness.pipeline_env_required) console.log("- pipeline env: case-specific required"); + if (plan.automation_readiness.missing.length > 0) console.log(`- missing: ${plan.automation_readiness.missing.join(", ")}`); + if (plan.automation_readiness.defaulted.length > 0) console.log(`- case defaults: ${plan.automation_readiness.defaulted.join(", ")}`); + for (const alias of plan.automation_readiness.env_aliases) { + console.log(`- alias: ${alias.target} <- ${alias.source} (${alias.configured ? "configured" : "missing"})`); + } + console.log(""); + console.log("## Fixture Readiness"); + console.log(`- status: ${plan.fixture_readiness.status}`); + for (const fixture of plan.fixture_readiness.required) { + console.log(`- ${fixture.id}: ${fixture.exists ? "present" : "missing"} (${fixture.path})`); + } + console.log(""); + console.log("## Manual Readiness"); + console.log(`- status: ${plan.manual_readiness.status}`); + if (plan.preconditions.length === 0) console.log("- preconditions: none declared"); + for (const precondition of plan.preconditions) console.log(`- precondition: ${precondition}`); + if (plan.setup.length > 0) for (const item of plan.setup) console.log(`- setup: ${item}`); + if (plan.setup_automation.length > 0) { + for (const item of plan.setup_automation) console.log(`- setup automation: ${item}`); + } + if (plan.setup_provides_env.length > 0) console.log(`- setup provides env: ${plan.setup_provides_env.join(", ")}`); + if (plan.cleanup.length > 0) for (const item of plan.cleanup) console.log(`- cleanup: ${item}`); + console.log(""); + console.log("## Required Skills"); + for (const skill of plan.required_skills) console.log(`- ${skill}`); + console.log(""); + console.log(`## ${stepHeading(plan.mode)}`); + for (const [index, step] of plan.steps.entries()) console.log(`${index + 1}. ${step}`); + console.log(""); + console.log("## Checks"); + for (const check of plan.checks) console.log(`- ${check}`); + console.log(""); + console.log("## Diagnostics"); + if (plan.diagnostics.length === 0) console.log("- Optional: use API/curl/logs only to diagnose failures."); + for (const diagnostic of plan.diagnostics) console.log(`- ${diagnostic}`); + console.log(""); + if (plan.visual_checks.length > 0) { + console.log("## Visual Checks"); + for (const check of plan.visual_checks) console.log(`- ${check}`); + console.log(""); + } + console.log("## Required Evidence"); + if (plan.evidence_required.length === 0) console.log("- None declared."); + for (const evidence of plan.evidence_required) console.log(`- ${evidence}`); + console.log(""); + console.log("## Success Signals"); + if (plan.success_patterns.length === 0) console.log("- None declared."); + for (const pattern of plan.success_patterns) console.log(`- ${pattern}`); + console.log(""); + console.log("## Failure Signals"); + if (plan.failure_patterns.length === 0) console.log("- None declared."); + for (const pattern of plan.failure_patterns) console.log(`- ${pattern}`); + console.log(""); + console.log("## Troubleshooting"); + for (const entry of plan.troubleshooting) { + console.log(`- ${entry.id}: ${entry.title}`); + for (const pattern of entry.patterns) console.log(` pattern: ${pattern}`); + } + console.log(""); + console.log("## Report Template"); + for (const [key, value] of Object.entries(plan.report_template)) console.log(`- ${key}: ${value}`); + return 0; +} + +function normalizeChangedPath(path: string): string { + return path.replace(/\\/g, "/").replace(/^\.\//, ""); +} + +function isChangedFilePath(path: string): boolean { + return Boolean(path) && !path.endsWith("/") && !path.startsWith("--- ") && !path.startsWith("+++ "); +} + +function existingCaseIds(root: string): Set { + return new Set(loadStructuredItems(root, "cases").map((item) => scalar(item.fields, "id"))); +} + +function addRecommendation( + output: TestRecommendation[], + existing: Set, + id: string, + reason: string, +): void { + if (!existing.has(id) || output.some((item) => item.id === id)) return; + output.push({ id, reason }); +} + +function changedFilesFromGit(repo: string, prefix: string): string[] { + if (!existsSync(repo)) return []; + const argsList = [ + ["diff", "--name-only", "HEAD"], + ["status", "--short"], + ]; + const files: string[] = []; + for (const args of argsList) { + const result = spawnSync("git", args, { + cwd: repo, + encoding: "utf8", + }); + if (result.status !== 0) continue; + for (const raw of result.stdout.split(/\r?\n/)) { + if (!raw.trim()) continue; + const file = args[0] === "status" + ? raw.slice(3).trim().split(/\s+->\s+/).pop() ?? "" + : raw.trim(); + if (isChangedFilePath(file)) files.push(`${prefix}/${normalizeChangedPath(file)}`); + } + } + return files; +} + +function repoCandidates(root: string, env: Record): Array<{ path: string; prefix: string }> { + return [ + { path: env.LANGBOT_REPO || resolve(root, "../LangBot"), prefix: "LangBot" }, + { path: env.LANGBOT_PLUGIN_SDK_REPO || resolve(root, "../langbot-plugin-sdk"), prefix: "langbot-plugin-sdk" }, + { path: env.LANGBOT_AGENT_RUNNER_REPO || resolve(root, "../langbot-agent-runner"), prefix: "langbot-agent-runner" }, + { path: env.LANGBOT_LOCAL_AGENT_REPO || resolve(root, "../langbot-local-agent"), prefix: "langbot-local-agent" }, + ]; +} + +function repeatedOptionValues(args: string[], key: string): string[] { + const values: string[] = []; + for (let i = 0; i < args.length; i += 1) { + if (args[i] !== `--${key}`) continue; + const value = args[i + 1]; + if (value && !value.startsWith("--")) values.push(value); + } + return values; +} + +function changedFiles(root: string, explicitFiles: string[]): string[] { + const explicit = explicitFiles.map(normalizeChangedPath); + if (explicit.length > 0) return Array.from(new Set(explicit)); + + const env = runtimeEnv(root); + const files = repoCandidates(root, env).flatMap((repo) => changedFilesFromGit(repo.path, repo.prefix)); + return Array.from(new Set(files)).sort(); +} + +function buildRecommendations(root: string, files: string[]): TestRecommendation[] { + const existing = existingCaseIds(root); + const recommendations: TestRecommendation[] = []; + const text = files.map(normalizeChangedPath); + const has = (pattern: RegExp) => text.some((file) => pattern.test(file)); + + if (has(/(^|\/)(result_normalizer|orchestrator|descriptor|errors)\.py$/) || has(/agent_runner\/result\.py$/)) { + addRecommendation(recommendations, existing, "agent-runner-fixture-contract", "Deterministic AgentRunner fixture contract should still execute."); + addRecommendation(recommendations, existing, "agent-runner-behavior-matrix", "AgentRunner result/orchestration contract changed."); + } + if (has(/fixtures\/plugins\/qa-agent-runner|components\/agent_runner|manifest\.ya?ml$/)) { + addRecommendation(recommendations, existing, "agent-runner-fixture-contract", "AgentRunner fixture or runner manifest changed."); + addRecommendation(recommendations, existing, "agent-runner-live-install", "AgentRunner plugin package should still install and register."); + addRecommendation(recommendations, existing, "agent-runner-qa-debug-chat", "Installed QA AgentRunner should still execute through Debug Chat."); + } + if (has(/fixtures\/plugins\/qa-plugin-smoke|qa_plugin_|qa-plugin-smoke/i)) { + addRecommendation(recommendations, existing, "qa-plugin-smoke-live-install", "QA plugin smoke fixture should install and expose tools."); + } + if (has(/(run_ledger|agent_run\.py|run_ledger\.py|alembic.*agent_run|test_run_ledger)/i)) { + addRecommendation(recommendations, existing, "agent-runner-ledger-invariants", "Run ledger schema/status code changed."); + addRecommendation(recommendations, existing, "agent-runner-ledger-stress", "Run ledger queue/claim behavior changed."); + addRecommendation(recommendations, existing, "agent-runner-ledger-contention", "Run ledger claim behavior changed; check local write contention."); + addRecommendation(recommendations, existing, "agent-runner-async-db-readiness", "Async DB readiness gates ledger concurrency probes."); + addRecommendation(recommendations, existing, "agent-runner-ledger-concurrency", "Run ledger concurrency/auth tests are relevant."); + } + if (has(/(plugin\/handler|agent_run_api|history_event_api|state_api|pull_api|runtime\/plugin|test_mgr_agent_runner|test_pull_api_handlers)/)) { + addRecommendation(recommendations, existing, "agent-runner-runtime-chaos", "SDK/runtime or Host action handling changed."); + } + if (has(/(LangBot\/web\/|^web\/|control-plane|frontend|\/page|\/pages)/)) { + addRecommendation(recommendations, existing, "agent-runner-release-preflight", "UI/control-plane surface changed; preflight catches wrong live target."); + addRecommendation(recommendations, existing, "webui-login-state", "Browser session must still reach LangBot WebUI."); + } + if (has(/(local-agent|context|compaction|rag|tool|mcp|multimodal)/i)) { + addRecommendation(recommendations, existing, "qa-plugin-smoke-live-install", "Tool-loop checks depend on the QA plugin smoke fixture."); + addRecommendation(recommendations, existing, "local-agent-basic-debug-chat", "Local-agent user path may be affected."); + addRecommendation(recommendations, existing, "local-agent-plugin-tool-call-debug-chat", "Tool-loop changes need browser evidence."); + } + if (has(/(^|\/)(acp|claude|codex)(\/|-)|langbot-agent-runner\//i)) { + addRecommendation(recommendations, existing, "acp-agent-runner-debug-chat", "External AgentRunner path may be affected."); + } + if (recommendations.length === 0) { + addRecommendation(recommendations, existing, "agent-runner-release-preflight", "No narrow AgentRunner rule matched; start with preflight if this branch touches runner behavior."); + } + return recommendations; +} + +function buildRecommendReport(root: string, explicitFiles: string[]): TestRecommendReport { + const files = changedFiles(root, explicitFiles); + const recommendations = buildRecommendations(root, files); + return { + generated_at: new Date().toISOString(), + changed_files: files, + recommendations, + commands: recommendations.flatMap((item) => [ + `bin/lbs test plan ${item.id}`, + `bin/lbs test run ${item.id} --dry-run`, + ]), + notes: [ + "Run probe cases before browser cases.", + "Remove --dry-run only after readiness and manual_check preconditions are confirmed.", + "Treat blocked/env_issue separately from product fail.", + "Browser cases still need required UI evidence before pass.", + ], + }; +} + +function renderRecommendReport(report: TestRecommendReport): string { + const lines: string[] = []; + lines.push("# Test Recommendations"); + lines.push(""); + lines.push(`Generated: ${report.generated_at}`); + lines.push(""); + lines.push("## Changed Files"); + if (report.changed_files.length === 0) lines.push("- None detected. Pass --file to recommend from explicit paths."); + else for (const file of report.changed_files) lines.push(`- ${file}`); + lines.push(""); + lines.push("## Recommended Cases"); + if (report.recommendations.length === 0) lines.push("- None."); + for (const item of report.recommendations) { + lines.push(`- ${item.id}: ${item.reason}`); + } + lines.push(""); + lines.push("## Commands"); + for (const command of report.commands) lines.push(`- ${command}`); + lines.push(""); + lines.push("## Notes"); + for (const note of report.notes) lines.push(`- ${note}`); + return `${lines.join("\n").trimEnd()}\n`; +} + +export function commandTestRecommend(ctx: CommandContext): number { + const { options } = parseOptions(ctx.args.slice(2)); + const report = buildRecommendReport(ctx.root, repeatedOptionValues(ctx.args.slice(2), "file")); + if (options.json === true) console.log(JSON.stringify(report, null, 2)); + else console.log(renderRecommendReport(report).trimEnd()); + return 0; +} + +function pad2(value: number): string { + return String(value).padStart(2, "0"); +} + +function pad3(value: number): string { + return String(value).padStart(3, "0"); +} + +function localIsoWithOffset(date: Date): string { + const offsetMinutes = -date.getTimezoneOffset(); + const sign = offsetMinutes >= 0 ? "+" : "-"; + const absoluteOffset = Math.abs(offsetMinutes); + return [ + `${date.getFullYear()}-${pad2(date.getMonth() + 1)}-${pad2(date.getDate())}`, + `T${pad2(date.getHours())}:${pad2(date.getMinutes())}:${pad2(date.getSeconds())}.${pad3(date.getMilliseconds())}`, + `${sign}${pad2(Math.floor(absoluteOffset / 60))}:${pad2(absoluteOffset % 60)}`, + ].join(""); +} + +function timestampSlug(localIso: string): string { + return localIso + .replace(/T/, "-") + .replace(/[.:+]/g, "-") + .replace(/[^A-Za-z0-9_-]+/g, "-") + .replace(/-+/g, "-") + .replace(/^-|-$/g, ""); +} + +function caseLocator(args: string[]): string { + return args.join(" "); +} + +function automationScript(item: StructuredItem): string { + return scalar(item.fields, "automation"); +} + +function setupCaseExists(root: string, target: string): boolean { + return loadStructuredItems(root, "cases").some((item) => scalar(item.fields, "id") === target); +} + +function setupAutomation(root: string, item: StructuredItem, runId: string, evidenceRoot: string): SetupAutomation[] { + return setupAutomationEntries(item).map((entry, index) => { + const spec = parseSetupAutomationEntry(entry); + const evidenceDir = join("setup", setupAutomationEvidenceName(index, spec)); + const fullEvidenceDir = join(evidenceRoot, evidenceDir); + const command = spec.kind === "case" + ? `bin/lbs test run ${spec.target} --run-id ${runId}-${spec.target} --output ${fullEvidenceDir}` + : `node ${spec.target}${spec.args.length > 0 ? ` ${spec.args.join(" ")}` : ""}`; + const dryRunCommand = spec.kind === "case" ? `${command} --dry-run` : ""; + return { + entry, + kind: spec.kind, + target: spec.target, + args: spec.args, + command, + dry_run_command: dryRunCommand, + evidence_dir: fullEvidenceDir, + exists: spec.kind === "case" ? setupCaseExists(root, spec.target) : existsSync(setupAutomationScriptPath(root, spec)), + }; + }); +} + +function isoFromDateInput(input: string): string { + const parsed = Date.parse(input); + return Number.isNaN(parsed) ? "" : new Date(parsed).toISOString(); +} + +function commaList(value: string | undefined): string[] { + if (!value) return []; + return value + .split(",") + .map((item) => item.trim()) + .filter(Boolean); +} + +function buildTestResult( + root: string, + item: StructuredItem, + options: Record, +): { result?: TestResultRecord; errors: string[] } { + const errors: string[] = []; + const status = typeof options.result === "string" ? options.result : ""; + const reason = typeof options.reason === "string" ? options.reason : ""; + const evidenceDir = typeof options["evidence-dir"] === "string" ? options["evidence-dir"] : ""; + const now = new Date(); + const writtenAtLocal = localIsoWithOffset(now); + const startedAtLocal = typeof options["started-at"] === "string" ? options["started-at"] : writtenAtLocal; + const finishedAtLocal = typeof options["finished-at"] === "string" ? options["finished-at"] : writtenAtLocal; + const startedAt = isoFromDateInput(startedAtLocal); + const finishedAt = isoFromDateInput(finishedAtLocal); + const evidenceCollected = commaList(typeof options.evidence === "string" ? options.evidence : undefined); + const evidenceRequired = listValue(item.fields, "evidence_required"); + const evidenceMissing = evidenceRequired.filter((value) => !evidenceCollected.includes(value)); + + if (!status) errors.push("--result is required"); + else if (!testResultStatusValues.includes(status)) { + errors.push(`--result must be one of ${testResultStatusValues.join(", ")}`); + } + if (!reason) errors.push("--reason is required"); + if (!evidenceDir) errors.push("--evidence-dir is required"); + if (!startedAt) errors.push(`--started-at is not a valid date/time: ${startedAtLocal}`); + if (!finishedAt) errors.push(`--finished-at is not a valid date/time: ${finishedAtLocal}`); + + const allowedEvidence = new Set(caseEvidenceValues); + for (const value of evidenceCollected) { + if (!allowedEvidence.has(value)) errors.push(`--evidence contains unsupported value '${value}'`); + } + if (status === "pass" && evidenceMissing.length > 0) { + errors.push(`pass result is missing required evidence: ${evidenceMissing.join(", ")}`); + } + + if (errors.length > 0) return { errors }; + + const resolvedEvidenceDir = resolve(evidenceDir); + return { + errors, + result: { + source: "final", + case_id: scalar(item.fields, "id"), + run_id: typeof options["run-id"] === "string" ? options["run-id"] : resolvedEvidenceDir.split(/[\\/]/).pop() ?? "", + written_at: now.toISOString(), + written_at_local: writtenAtLocal, + started_at: startedAt, + started_at_local: startedAtLocal, + finished_at: finishedAt, + finished_at_local: finishedAtLocal, + status, + reason, + url: typeof options.url === "string" ? options.url : "", + browser_path: typeof options["browser-path"] === "string" ? options["browser-path"] : "", + evidence_dir: evidenceDir, + evidence_collected: evidenceCollected, + evidence_required: evidenceRequired, + evidence_missing: evidenceMissing, + evidence_status: evidenceMissing.length === 0 ? "complete" : "incomplete", + report_path: typeof options.report === "string" ? options.report : "", + notes: typeof options.notes === "string" ? options.notes : "", + }, + }; +} + +function renderTestResult(result: TestResultRecord): string { + const lines: string[] = []; + lines.push(`# Test Result: ${result.case_id}`); + lines.push(""); + lines.push(`Run: ${result.run_id}`); + lines.push(`Status: ${result.status}`); + lines.push(`Reason: ${result.reason}`); + lines.push(`Evidence dir: ${result.evidence_dir}`); + lines.push(`Evidence status: ${result.evidence_status}`); + if (result.evidence_missing.length > 0) lines.push(`Evidence missing: ${result.evidence_missing.join(", ")}`); + if (result.url) lines.push(`URL: ${result.url}`); + if (result.browser_path) lines.push(`Browser path: ${result.browser_path}`); + if (result.report_path) lines.push(`Report: ${result.report_path}`); + lines.push(""); + lines.push("## Evidence Collected"); + if (result.evidence_collected.length === 0) lines.push("- None declared."); + else for (const value of result.evidence_collected) lines.push(`- ${value}`); + return `${lines.join("\n").trimEnd()}\n`; +} + +function buildStart(root: string, item: StructuredItem, args: string[]): TestStart { + const now = new Date(); + const startedAtLocal = localIsoWithOffset(now); + const id = scalar(item.fields, "id"); + const mode = caseMode(item); + const runId = `${timestampSlug(startedAtLocal)}-${id}`; + const recommendedReportPath = join("reports", `${runId}.md`); + const evidenceDir = join("reports", "evidence", runId); + const locator = caseLocator(args); + const script = automationScript(item); + const automationCommand = script + ? `bin/lbs test run ${locator} --run-id ${runId} --output ${evidenceDir}` + : undefined; + const consoleLog = join(evidenceDir, "console.log"); + const reportCommand = script + ? `bin/lbs test report ${locator} --since "${startedAtLocal}" --console-log ${consoleLog} --evidence-dir ${evidenceDir} --output ${recommendedReportPath}` + : `bin/lbs test report ${locator} --since "${startedAtLocal}" --evidence-dir ${evidenceDir} --output ${recommendedReportPath}`; + const resultCommandTemplate = `bin/lbs test result ${locator} --result --reason "" --evidence-dir ${evidenceDir} --started-at "${startedAtLocal}" --evidence ${listValue(item.fields, "evidence_required").join(",")}`; + + return { + run_id: runId, + started_at: now.toISOString(), + started_at_local: startedAtLocal, + case: caseSummary(item), + environment: envSummary(item, { ...loadEnv(root), ...processEnv }), + required_skills: listValue(item.fields, "skills"), + preconditions: listValue(item.fields, "preconditions"), + setup: listValue(item.fields, "setup"), + setup_automation: setupAutomationEntries(item), + setup_provides_env: listValue(item.fields, "setup_provides_env"), + cleanup: listValue(item.fields, "cleanup"), + steps: listValue(item.fields, "steps"), + checks: listValue(item.fields, "checks"), + evidence_required: listValue(item.fields, "evidence_required"), + success_patterns: listValue(item.fields, "success_patterns"), + failure_patterns: listValue(item.fields, "failure_patterns"), + automation: script + ? { + script, + command: automationCommand ?? "", + evidence_dir: evidenceDir, + } + : undefined, + recommended_report_path: recommendedReportPath, + plan_command: `bin/lbs test plan ${locator}`, + report_command: reportCommand, + result_command_template: resultCommandTemplate, + evidence_checklist: evidenceChecklist(mode), + }; +} + +function renderMarkdownStart(start: TestStart): string { + const lines: string[] = []; + const reportCase = start.case; + + lines.push(`# Test Start: ${reportCase.id}`); + lines.push(""); + lines.push(`Run: ${start.run_id}`); + lines.push(`Started: ${start.started_at_local}`); + lines.push(`Title: ${reportCase.title}`); + lines.push(`Skill: ${reportCase.skill}`); + lines.push(""); + lines.push("## Commands"); + lines.push(`- plan: ${start.plan_command}`); + if (start.automation) lines.push(`- automation: ${start.automation.command}`); + lines.push(`- report: ${start.report_command}`); + lines.push(`- result template: ${start.result_command_template}`); + lines.push(""); + lines.push("## Evidence Checklist"); + for (const item of start.evidence_checklist) lines.push(`- ${item}`); + lines.push(""); + lines.push(...renderLines("Required Evidence", start.evidence_required)); + lines.push(""); + lines.push(...renderLines("Preconditions", start.preconditions)); + lines.push(...renderLines("Setup", start.setup)); + lines.push(...renderLines("Setup Automation", start.setup_automation)); + lines.push(...renderLines("Setup Provides Env", start.setup_provides_env)); + lines.push(...renderLines("Cleanup", start.cleanup)); + lines.push(`## ${stepHeading(String(reportCase.mode || "agent-browser"))}`); + for (const [index, step] of start.steps.entries()) lines.push(`${index + 1}. ${step}`); + lines.push(""); + lines.push(...renderLines("Checks", start.checks)); + lines.push(...renderLines("Success Signals", start.success_patterns)); + lines.push(...renderLines("Failure Signals", start.failure_patterns)); + lines.push("## Environment"); + for (const [key, value] of Object.entries(start.environment)) lines.push(`- ${key}=${value}`); + lines.push(""); + + return `${lines.join("\n").trimEnd()}\n`; +} + +export function commandTestStart(ctx: CommandContext): number { + const { positional: args, options } = parseOptions(ctx.args.slice(2)); + const item = findCase(ctx.root, args); + const start = buildStart(ctx.root, item, args); + const output = typeof options.output === "string" ? options.output : undefined; + const content = options.json === true ? `${JSON.stringify(start, null, 2)}\n` : renderMarkdownStart(start); + + writeOrPrint(content, output); + return 0; +} + +export function commandTestResult(ctx: CommandContext): number { + const { positional: args, options } = parseOptions(ctx.args.slice(2)); + const item = findCase(ctx.root, args); + const { result, errors } = buildTestResult(ctx.root, item, options); + if (!result) { + for (const error of errors) console.error(`ERROR: ${error}`); + return 1; + } + + const resultPath = join(result.evidence_dir, "result.json"); + mkdirSync(dirname(resultPath), { recursive: true }); + writeFileSync(resultPath, `${JSON.stringify(result, null, 2)}\n`, "utf8"); + + if (options.json === true) console.log(JSON.stringify(result, null, 2)); + else console.log(renderTestResult(result).trimEnd()); + return 0; +} + +function buildAutomationRun( + root: string, + item: StructuredItem, + args: string[], + options: Record, +): TestAutomationRun { + const now = new Date(); + const startedAtLocal = localIsoWithOffset(now); + const id = scalar(item.fields, "id"); + const sourceEnv = runtimeEnv(root); + const runId = typeof options["run-id"] === "string" ? options["run-id"] : `${timestampSlug(startedAtLocal)}-${id}`; + const script = automationScript(item); + const scriptPath = script ? resolve(root, script) : ""; + const evidenceDir = typeof options.output === "string" + ? options.output + : join("reports", "evidence", runId); + const locator = caseLocator(args); + const consoleLog = join(evidenceDir, "console.log"); + const reportPath = join("reports", `${runId}.md`); + const reportCommand = `bin/lbs test report ${locator} --since "${startedAtLocal}" --console-log ${consoleLog} --evidence-dir ${evidenceDir} --output ${reportPath}`; + const runCommand = [ + "bin/lbs", + "test", + "run", + locator, + "--run-id", + runId, + "--output", + evidenceDir, + ].join(" "); + + return { + run_id: runId, + started_at: now.toISOString(), + started_at_local: startedAtLocal, + case: caseSummary(item), + setup_automation: setupAutomation(root, item, runId, evidenceDir), + automation: { + script, + script_path: scriptPath, + exists: scriptPath ? existsSync(scriptPath) : false, + required_env: [...listValue(item.fields, "automation_env"), ...listValue(item.fields, "automation_env_any")], + evidence_dir: evidenceDir, + console_log: consoleLog, + network_log: join(evidenceDir, "network.log"), + screenshot: join(evidenceDir, "screenshot.png"), + automation_result_json: join(evidenceDir, "automation-result.json"), + result_json: join(evidenceDir, "result.json"), + command: runCommand, + report_command: reportCommand, + env_defaults: automationEnvDefaults(item, sourceEnv), + env_aliases: caseAutomationReadiness(item, sourceEnv).env_aliases, + pipeline_env_required: caseAutomationReadiness(item, sourceEnv).pipeline_env_required, + }, + }; +} + +function renderAutomationRun(run: TestAutomationRun): string { + const lines: string[] = []; + lines.push(`# Test Automation: ${run.case.id}`); + lines.push(""); + lines.push(`Run: ${run.run_id}`); + lines.push(`Started: ${run.started_at_local}`); + lines.push(`Script: ${run.automation.script || "None declared."}`); + lines.push(`Script path: ${run.automation.script_path || "None declared."}`); + lines.push(`Script exists: ${run.automation.exists ? "yes" : "no"}`); + lines.push(""); + lines.push("## Setup Automation"); + if (run.setup_automation.length === 0) lines.push("- None declared."); + for (const setup of run.setup_automation) { + lines.push(`- ${setup.entry}`); + lines.push(` command: ${setup.command}`); + if (setup.dry_run_command) lines.push(` dry_run_command: ${setup.dry_run_command}`); + lines.push(` evidence_dir: ${setup.evidence_dir}`); + lines.push(` exists: ${setup.exists ? "yes" : "no"}`); + } + lines.push(""); + lines.push("## Commands"); + lines.push(`- run: ${run.automation.command}`); + lines.push(`- report: ${run.automation.report_command}`); + lines.push(""); + lines.push("## Evidence Files"); + lines.push(`- console_log: ${run.automation.console_log}`); + lines.push(`- network_log: ${run.automation.network_log}`); + lines.push(`- screenshot: ${run.automation.screenshot}`); + lines.push(`- automation_result_json: ${run.automation.automation_result_json}`); + lines.push(`- result_json: ${run.automation.result_json}`); + lines.push(""); + lines.push(...renderLines("Required Env", run.automation.required_env)); + lines.push("## Automation Env Defaults"); + const defaults = Object.entries(run.automation.env_defaults); + if (defaults.length === 0) lines.push("- None declared."); + for (const [key, value] of defaults) lines.push(`- ${key}=${redactEnvValue(key, value)}`); + lines.push("## Automation Env Aliases"); + if (run.automation.env_aliases.length === 0) lines.push("- None declared."); + for (const alias of run.automation.env_aliases) { + lines.push(`- ${alias.target} <- ${alias.source} (${alias.configured ? "configured" : "missing"})`); + } + if (run.automation.pipeline_env_required) lines.push("- Pipeline env is case-specific; global LANGBOT_PIPELINE_URL fallback is disabled."); + return `${lines.join("\n").trimEnd()}\n`; +} + +function automationEnv( + root: string, + item: StructuredItem, + run: TestAutomationRun, + evidenceDir: string, + options: Record, +): Record { + const baseEnv = runtimeEnv(root); + const envDefaults = automationEnvDefaults(item, baseEnv); + return { + ...processEnv, + ...envDefaults, + ...baseEnv, + ...resolvedAutomationEnvOverrides(item, baseEnv), + ...Object.fromEntries( + Object.keys(envDefaults) + .filter((key) => baseEnv[key] !== undefined) + .map((key) => [key, baseEnv[key]]), + ), + LBS_ROOT: root, + LBS_CASE_ID: String(run.case.id), + LBS_RUN_ID: run.run_id, + LBS_STARTED_AT: run.started_at, + LBS_STARTED_AT_LOCAL: run.started_at_local, + LBS_EVIDENCE_DIR: resolve(evidenceDir), + LBS_HEADED: options.headed === true ? "1" : processEnv.LBS_HEADED, + }; +} + +function readSetupResult(setup: SetupAutomation): { status?: string; reason?: string } { + try { + return JSON.parse(readFileSync(join(setup.evidence_dir, "automation-result.json"), "utf8")); + } catch { + return {}; + } +} + +function writeSetupFailureResult(run: TestAutomationRun, setup: SetupAutomation, exitStatus: number | null): void { + const now = new Date(); + const setupResult = readSetupResult(setup); + const status = setupResult.status && setupResult.status !== "pass" + ? setupResult.status + : exitStatus === 2 ? "env_issue" : "fail"; + const result = { + source: "setup_automation", + case_id: run.case.id, + run_id: run.run_id, + status, + reason: setupResult.reason || `Setup automation failed: ${setup.entry}`, + failed_setup: setup, + exit_status: exitStatus, + started_at: run.started_at, + started_at_local: run.started_at_local, + finished_at: now.toISOString(), + finished_at_local: localIsoWithOffset(now), + evidence_collected: ["api_diagnostic"], + }; + writeFileSync(join(run.automation.evidence_dir, "automation-result.json"), `${JSON.stringify(result, null, 2)}\n`, "utf8"); + writeFileSync(join(run.automation.evidence_dir, "result.json"), `${JSON.stringify(result, null, 2)}\n`, "utf8"); +} + +function executionTail(value: string | Buffer | null | undefined): string { + return String(value ?? "").trim().slice(-4000); +} + +function runSetupAutomation( + ctx: CommandContext, + item: StructuredItem, + run: TestAutomationRun, + setup: SetupAutomation, + options: Record, +): { status: number; execution: Record } { + if (!setup.exists) { + if (options.json !== true) console.error(`ERROR: setup automation target not found: ${setup.entry}`); + writeSetupFailureResult(run, setup, 1); + return { + status: 1, + execution: { entry: setup.entry, status: "nonzero", exit_status: 1, reason: "setup automation target not found" }, + }; + } + mkdirSync(setup.evidence_dir, { recursive: true }); + if (options.json !== true) { + console.log(`Setup: ${setup.entry}`); + console.log(`Setup evidence: ${setup.evidence_dir}`); + } + const env = automationEnv(ctx.root, item, run, setup.evidence_dir, options); + const args = setup.kind === "case" + ? [ + lbsScriptPath(), + "--root", + ctx.root, + "test", + "run", + setup.target, + "--run-id", + `${run.run_id}-${setup.target}`, + "--output", + setup.evidence_dir, + ...(options.headed === true ? ["--headed"] : []), + ] + : [setupAutomationScriptPath(ctx.root, parseSetupAutomationEntry(setup.entry)), ...setup.args]; + const result = spawnSync(execPath, args, { + cwd: ctx.root, + env, + encoding: "utf8", + stdio: options.json === true ? "pipe" : "inherit", + }); + if (result.error) { + if (options.json !== true) console.error(`ERROR: failed to run setup automation: ${result.error.message}`); + writeSetupFailureResult(run, setup, 1); + return { + status: 1, + execution: { + entry: setup.entry, + status: "nonzero", + exit_status: 1, + reason: result.error.message, + stdout: executionTail(result.stdout), + stderr: executionTail(result.stderr), + }, + }; + } + const status = result.status ?? 1; + if (status !== 0) writeSetupFailureResult(run, setup, status); + return { + status, + execution: { + entry: setup.entry, + status: status === 0 ? "ok" : "nonzero", + exit_status: status, + stdout: executionTail(result.stdout), + stderr: executionTail(result.stderr), + }, + }; +} + +export function commandTestRun(ctx: CommandContext): number { + const { positional: args, options } = parseOptions(ctx.args.slice(2)); + const item = findCase(ctx.root, args); + const run = buildAutomationRun(ctx.root, item, args, options); + const output = typeof options.plan_output === "string" ? options.plan_output : undefined; + + if (options["dry-run"] === true) { + const content = options.json === true ? `${JSON.stringify(run, null, 2)}\n` : renderAutomationRun(run); + writeOrPrint(content, output); + return 0; + } + + if (!run.automation.script) { + console.error(`ERROR: case has no automation script: ${run.case.id}`); + return 1; + } + if (!run.automation.exists) { + console.error(`ERROR: automation script not found: ${run.automation.script_path}`); + return 1; + } + + mkdirSync(run.automation.evidence_dir, { recursive: true }); + if (options.json !== true) { + console.log(`Run: ${run.run_id}`); + console.log(`Evidence: ${run.automation.evidence_dir}`); + console.log(`Report command: ${run.automation.report_command}`); + } + + const setupExecutions: Array> = []; + for (const setup of run.setup_automation) { + const { status, execution } = runSetupAutomation(ctx, item, run, setup, options); + setupExecutions.push(execution); + if (status !== 0) { + if (options.json === true) { + console.log(JSON.stringify({ + run, + setup_executions: setupExecutions, + automation_execution: null, + exit_status: status, + }, null, 2)); + } + return status; + } + } + + const env = automationEnv(ctx.root, item, run, run.automation.evidence_dir, options); + const result = spawnSync(execPath, [run.automation.script_path], { + cwd: ctx.root, + env, + encoding: "utf8", + stdio: options.json === true ? "pipe" : "inherit", + }); + + if (result.error) { + if (options.json !== true) console.error(`ERROR: failed to run automation: ${result.error.message}`); + if (options.json === true) { + console.log(JSON.stringify({ + run, + setup_executions: setupExecutions, + automation_execution: { + status: "nonzero", + exit_status: 1, + reason: result.error.message, + stdout: executionTail(result.stdout), + stderr: executionTail(result.stderr), + }, + exit_status: 1, + }, null, 2)); + } + return 1; + } + const status = result.status ?? 1; + if (options.json === true) { + console.log(JSON.stringify({ + run, + setup_executions: setupExecutions, + automation_execution: { + status: status === 0 ? "ok" : "nonzero", + exit_status: status, + stdout: executionTail(result.stdout), + stderr: executionTail(result.stderr), + }, + exit_status: status, + }, null, 2)); + } + return status; +} + + +function buildReport(root: string, item: StructuredItem, options: Record): TestReport { + const env = loadEnv(root); + const mode = caseMode(item); + const related = relatedTroubleshooting(root, item).map((entry) => ({ + id: scalar(entry.fields, "id"), + title: scalar(entry.fields, "title"), + patterns: listValue(entry.fields, "patterns"), + verification: scalar(entry.fields, "verification"), + })); + + return { + generated_at: new Date().toISOString(), + case: caseSummary(item), + result_options: ["pass", "fail", "blocked", "env_issue", "flaky"], + automation_result: readAutomationResultEvidence(options), + manual_evidence: manualEvidenceTemplate(mode), + environment: envSummary(item, env), + required_skills: listValue(item.fields, "skills"), + steps: listValue(item.fields, "steps"), + checks: listValue(item.fields, "checks"), + diagnostics: listValue(item.fields, "diagnostics"), + evidence_required: listValue(item.fields, "evidence_required"), + success_patterns: listValue(item.fields, "success_patterns"), + failure_patterns: listValue(item.fields, "failure_patterns"), + expected_failures: listValue(item.fields, "expected_failures"), + troubleshooting: related, + log_guard: scanStructuredLogSources(root, item, options), + }; +} + +function renderLines(title: string, values: string[]): string[] { + const lines = [`## ${title}`]; + if (values.length === 0) lines.push("- None declared."); + else for (const value of values) lines.push(`- ${value}`); + lines.push(""); + return lines; +} + +function renderFinding(finding: LogFinding): string { + return renderLogFinding(finding); +} + +function renderSuccessSignal(signal: LogSuccessSignal): string { + return renderLogSuccessSignal(signal); +} + +function renderMarkdownReport(report: TestReport): string { + const reportCase = report.case; + const evidence = report.manual_evidence; + const environment = report.environment; + const logGuard = report.log_guard; + const troubleshooting = report.troubleshooting; + const lines: string[] = []; + + lines.push(`# Test Report: ${reportCase.id}`); + lines.push(""); + lines.push(`Generated: ${report.generated_at}`); + lines.push(`Title: ${reportCase.title}`); + lines.push(`Skill: ${reportCase.skill}`); + lines.push(`Mode: ${reportCase.mode}`); + lines.push(`Area: ${reportCase.area}`); + lines.push(`Type: ${reportCase.type}`); + lines.push(""); + lines.push("## Result"); + lines.push(`- result: ${evidence.result}`); + for (const [key, value] of Object.entries(evidence)) { + if (key !== "result") lines.push(`- ${key}: ${value}`); + } + lines.push(""); + lines.push("## Automation Result"); + lines.push(`- status: ${report.automation_result.status}`); + if (report.automation_result.path) lines.push(`- path: ${report.automation_result.path}`); + if (report.automation_result.result) lines.push(`- result: ${report.automation_result.result}`); + if (report.automation_result.reason) lines.push(`- reason: ${report.automation_result.reason}`); + if (report.automation_result.started_at_local) lines.push(`- started_at_local: ${report.automation_result.started_at_local}`); + if (report.automation_result.finished_at_local) lines.push(`- finished_at_local: ${report.automation_result.finished_at_local}`); + if (report.automation_result.url) lines.push(`- url: ${report.automation_result.url}`); + if (report.automation_result.expected_text) lines.push(`- expected_text: ${report.automation_result.expected_text}`); + lines.push(""); + lines.push("## Environment"); + for (const [key, value] of Object.entries(environment)) lines.push(`- ${key}=${value}`); + lines.push(""); + lines.push(`## ${stepHeading(String(reportCase.mode || "agent-browser"))}`); + for (const [index, step] of report.steps.entries()) lines.push(`${index + 1}. ${step}`); + lines.push(""); + lines.push(...renderLines("Checks", report.checks)); + lines.push(...renderLines("Diagnostics", report.diagnostics)); + lines.push(...renderLines("Required Evidence", report.evidence_required)); + lines.push(...renderLines("Success Signals", report.success_patterns)); + lines.push(...renderLines("Failure Signals", report.failure_patterns)); + lines.push(...renderLines("Expected Failures", report.expected_failures)); + lines.push("## Log Guard"); + lines.push(`- status: ${logGuard.status}`); + lines.push(`- scan_mode: ${logGuard.scan.mode}`); + if (logGuard.scan.since) lines.push(`- since: ${logGuard.scan.since}`); + if (logGuard.scan.until) lines.push(`- until: ${logGuard.scan.until}`); + if (logGuard.scan.tail_lines !== undefined) lines.push(`- tail_lines: ${logGuard.scan.tail_lines}`); + if (logGuard.scan.warnings.length > 0) { + lines.push("- scan_warnings:"); + for (const warning of logGuard.scan.warnings) lines.push(` - ${warning}`); + } + if (logGuard.sources.length === 0) { + lines.push("- sources: no log files provided; run with --backend-log, --frontend-log, or --console-log to scan logs."); + } else { + lines.push("- sources:"); + for (const source of logGuard.sources) { + const origin = source.auto_detected ? ", auto" : ""; + const total = source.total_line_count === undefined ? "" : `/${source.total_line_count}`; + const range = source.start_line === undefined || source.end_line === undefined + ? "" + : `, lines ${source.start_line}-${source.end_line}`; + const timestamped = source.timestamped_line_count === undefined ? "" : `, ${source.timestamped_line_count} timestamped`; + lines.push(` - ${source.source}: ${source.path} (${source.status}${origin}, ${source.line_count}${total} lines${range}${timestamped})`); + } + } + lines.push("- findings:"); + if (logGuard.findings.length === 0) lines.push(" - None."); + else for (const finding of logGuard.findings) lines.push(` ${renderFinding(finding)}`); + lines.push("- success_signals:"); + if (logGuard.success_signals.length === 0) lines.push(" - None."); + else for (const signal of logGuard.success_signals) lines.push(` ${renderSuccessSignal(signal)}`); + lines.push(""); + lines.push("## Related Troubleshooting"); + if (troubleshooting.length === 0) lines.push("- None declared."); + for (const entry of troubleshooting) { + lines.push(`- ${entry.id}: ${entry.title}`); + if (entry.patterns.length > 0) lines.push(` patterns: ${entry.patterns.join(" | ")}`); + if (entry.verification) lines.push(` verification: ${entry.verification}`); + } + lines.push(""); + lines.push("## Decision Notes"); + if (isProbeMode(String(reportCase.mode))) { + lines.push("- Probe results should be judged from the declared checks and required evidence for the same run."); + } else { + lines.push("- API/curl diagnostics can explain the run, but cannot make this UI case pass by themselves."); + } + lines.push("- Do not paste API keys, OAuth secrets, tokens, or localStorage token values into this report."); + lines.push(""); + + return `${lines.join("\n").trimEnd()}\n`; +} + +function writeOrPrint(content: string, output: string | undefined): void { + if (!output) { + console.log(content.trimEnd()); + return; + } + const path = resolve(output); + mkdirSync(dirname(path), { recursive: true }); + writeFileSync(path, content, "utf8"); + console.log(path); +} + +export function commandTestReport(ctx: CommandContext): number { + const { positional: args, options } = parseOptions(ctx.args.slice(2)); + const item = findCase(ctx.root, args); + const report = buildReport(ctx.root, item, options); + const output = typeof options.output === "string" ? options.output : undefined; + const content = options.json === true ? `${JSON.stringify(report, null, 2)}\n` : renderMarkdownReport(report); + + writeOrPrint(content, output); + return 0; +} diff --git a/skills/src/commands/trouble.ts b/skills/src/commands/trouble.ts new file mode 100644 index 000000000..6429ab4f7 --- /dev/null +++ b/skills/src/commands/trouble.ts @@ -0,0 +1,95 @@ +import { appendFileSync, existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import type { CommandContext } from "../types.ts"; +import { fail, optionString, parseOptions, usage } from "../cli.ts"; +import { findStructuredItem, getSkill, loadStructuredItems, scalar, slugify, todayIso, yamlList, yamlQuote } from "../fs.ts"; + +function troubleshootingYamlPath(root: string, skillName: string, id: string): string { + const skill = getSkill(root, skillName); + const dir = join(skill.path, "troubleshooting"); + mkdirSync(dir, { recursive: true }); + return join(dir, `${id}.yaml`); +} + +function legacyTroubleshootingPath(root: string, skillName: string): string { + const skill = getSkill(root, skillName); + const refsDir = join(skill.path, "references"); + mkdirSync(refsDir, { recursive: true }); + const path = join(refsDir, "troubleshooting.md"); + if (!existsSync(path)) writeFileSync(path, "# Troubleshooting\n\n", "utf8"); + return path; +} + +export function commandTroubleList(ctx: CommandContext): number { + const skill = ctx.args[2]; + const yamlItems = loadStructuredItems(ctx.root, "troubleshooting", skill); + for (const item of yamlItems) { + console.log(`${item.skill}\t${scalar(item.fields, "id")}\t${scalar(item.fields, "title")}`); + } + + if (skill && yamlItems.length === 0) { + const legacyPath = legacyTroubleshootingPath(ctx.root, skill); + const text = readFileSync(legacyPath, "utf8"); + const headings = Array.from(text.matchAll(/^##\s+(.+)$/gm)).map((match) => match[1]); + for (const heading of headings) console.log(`${skill}\tlegacy\t${heading}`); + } + return 0; +} + +export function commandTroubleShow(ctx: CommandContext): number { + const positional = ctx.args.slice(2); + if (positional.length < 1 || positional.length > 2) usage(); + const item = positional.length === 1 + ? findStructuredItem(ctx.root, "troubleshooting", positional[0]) + : findStructuredItem(ctx.root, "troubleshooting", positional[0], positional[1]); + console.log(item.raw.trimEnd()); + return 0; +} + +export function commandTroubleSearch(ctx: CommandContext): number { + const query = ctx.args[2]?.toLowerCase(); + if (!query) usage(); + const items = loadStructuredItems(ctx.root, "troubleshooting").filter((item) => item.raw.toLowerCase().includes(query)); + for (const item of items) { + console.log(`${item.skill}\t${scalar(item.fields, "id")}\t${scalar(item.fields, "title")}`); + } + return 0; +} + +export function commandTroubleAdd(ctx: CommandContext): number { + const skill = ctx.args[2]; + if (!skill) usage(); + const { options } = parseOptions(ctx.args.slice(3)); + for (const key of ["title", "symptom", "cause", "fix"]) { + if (!optionString(options, key)) fail(`--${key} is required`); + } + + const title = optionString(options, "title") ?? ""; + const symptom = optionString(options, "symptom") ?? ""; + const id = optionString(options, "id") ?? slugify(title); + const path = troubleshootingYamlPath(ctx.root, skill, id); + if (existsSync(path)) fail(`troubleshooting entry already exists: ${path}`); + + const text = + `id: ${id}\n` + + `title: ${yamlQuote(title)}\n` + + `date: ${todayIso()}\n` + + "symptoms:\n" + + yamlList([symptom]) + + "\npatterns:\n" + + yamlList([symptom]) + + "\nlikely_causes:\n" + + yamlList([optionString(options, "cause") ?? ""]) + + "\nfix_steps:\n" + + yamlList([optionString(options, "fix") ?? ""]) + + "\nverification: " + + yamlQuote(optionString(options, "verify") ?? "Add the command, UI signal, or log line that proves the fix worked.") + + "\nrelated_cases:\n" + + yamlList([]) + + "\n"; + + writeFileSync(path, text, "utf8"); + appendFileSync(legacyTroubleshootingPath(ctx.root, skill), `\n## ${id}: ${title}\n\nSee \`../troubleshooting/${id}.yaml\`.\n`, "utf8"); + console.log(path); + return 0; +} diff --git a/skills/src/commands/validate.ts b/skills/src/commands/validate.ts new file mode 100644 index 000000000..8b15d6344 --- /dev/null +++ b/skills/src/commands/validate.ts @@ -0,0 +1,394 @@ +import { existsSync, readFileSync } from "node:fs"; +import { join } from "node:path"; +import { stderr } from "node:process"; +import type { Skill, StructuredItem } from "../types.ts"; +import { loadFixtureItems } from "../fixtures.ts"; +import { + caseEvidenceValues, + caseModeValues, + casePriorityValues, + caseRequiredLists, + caseRequiredStrings, + caseRiskValues, + caseTypeValues, + requiredEnvKeys, + suiteRequiredLists, + suiteRequiredStrings, + suiteTypeValues, + troubleRequiredLists, + troubleRequiredStrings, + troubleshootingCategoryValues, +} from "../constants.ts"; +import { boolValue, envExamplePath, envPath, listValue, loadSkills, loadStructuredItems, parseEnvFile, scalar } from "../fs.ts"; +import { envKeyPattern, isEnvAnyGroup, splitEnvAnyGroup } from "../env-groups.ts"; +import { parseSetupAutomationEntry, validateSetupAutomationEntry } from "../setup-automation.ts"; + +const refRe = /(?:\]\(|`)(references\/[A-Za-z0-9_.\-/]+\.md)(?:\)|`)/g; + +function validateStructuredItem(item: StructuredItem, requiredStrings: string[], requiredLists: string[]): string[] { + const errors: string[] = []; + const listKeys = item.path.includes("/cases/") && scalar(item.fields, "mode") === "probe" + ? requiredLists.filter((key) => key !== "env") + : requiredLists; + for (const key of requiredStrings) { + if (!scalar(item.fields, key)) errors.push(`${item.path}: missing '${key}'`); + } + for (const key of listKeys) { + if (listValue(item.fields, key).length === 0) errors.push(`${item.path}: missing '${key}' entries`); + } + const id = scalar(item.fields, "id"); + if (id && !/^[a-z0-9][a-z0-9_-]*$/.test(id)) { + errors.push(`${item.path}: id must use lowercase letters, digits, dashes, or underscores`); + } + return errors; +} + +function validateEnum(item: StructuredItem, key: string, values: string[]): string[] { + const value = scalar(item.fields, key); + if (!value) return []; + return values.includes(value) ? [] : [`${item.path}: '${key}' must be one of ${values.join(", ")}`]; +} + +function validateListEnum(item: StructuredItem, key: string, values: string[]): string[] { + const allowed = new Set(values); + return listValue(item.fields, key) + .filter((value) => !allowed.has(value)) + .map((value) => `${item.path}: '${key}' contains unsupported value '${value}'`); +} + +function validateDuplicateListValues(item: StructuredItem, keys: string[]): string[] { + const errors: string[] = []; + for (const key of keys) { + const seen = new Set(); + for (const value of listValue(item.fields, key)) { + if (seen.has(value)) errors.push(`${item.path}: '${key}' contains duplicate value '${value}'`); + seen.add(value); + } + } + return errors; +} + +function validateEnvKeyList(item: StructuredItem, key: string): string[] { + return listValue(item.fields, key) + .filter((value) => !envKeyPattern.test(value)) + .map((value) => `${item.path}: '${key}' contains invalid env key '${value}'`); +} + +function validateEnvKeyScalar(item: StructuredItem, key: string): string[] { + const value = scalar(item.fields, key); + if (!value) return []; + return envKeyPattern.test(value) + ? [] + : [`${item.path}: '${key}' contains invalid env key '${value}'`]; +} + +function validateJsonScalar(item: StructuredItem, key: string): string[] { + const value = scalar(item.fields, key); + if (!value) return []; + try { + JSON.parse(value); + return []; + } catch (error) { + return [`${item.path}: '${key}' must be valid JSON: ${(error as Error).message}`]; + } +} + +function validateEnvAnyList(item: StructuredItem, key: string): string[] { + return listValue(item.fields, key) + .filter((value) => !isEnvAnyGroup(value)) + .map((value) => `${item.path}: '${key}' contains invalid env any-group '${value}'`); +} + +function validateCaseItem(root: string, item: StructuredItem, skillNames: Set, troubleIds: Set, caseIds: Set): string[] { + const errors = [ + ...validateEnum(item, "mode", caseModeValues), + ...validateEnum(item, "type", caseTypeValues), + ...validateEnum(item, "priority", casePriorityValues), + ...validateEnum(item, "risk", caseRiskValues), + ...validateListEnum(item, "evidence_required", caseEvidenceValues), + ...validateDuplicateListValues(item, [ + "tags", + "skills", + "env", + "env_any", + "automation_env", + "automation_env_any", + "setup_automation", + "setup_provides_env", + "evidence_required", + "troubleshooting", + ]), + ...validateEnvKeyList(item, "env"), + ...validateEnvAnyList(item, "env_any"), + ...validateEnvKeyList(item, "automation_env"), + ...validateEnvAnyList(item, "automation_env_any"), + ...validateEnvKeyList(item, "setup_provides_env"), + ...validateEnvKeyScalar(item, "automation_pipeline_url_env"), + ...validateEnvKeyScalar(item, "automation_pipeline_name_env"), + ...validateJsonScalar(item, "automation_filesystem_checks_json"), + ...listValue(item.fields, "setup_automation").flatMap((entry) => ( + validateSetupAutomationEntry(root, entry, caseIds).map((error) => `${item.path}: ${error}`) + )), + ]; + + if (boolValue(item.fields, "ci_eligible") === undefined) { + errors.push(`${item.path}: missing or invalid boolean 'ci_eligible'`); + } + + for (const skill of listValue(item.fields, "skills")) { + if (!skillNames.has(skill)) errors.push(`${item.path}: references unknown skill '${skill}'`); + } + + for (const id of listValue(item.fields, "troubleshooting")) { + if (!troubleIds.has(id)) errors.push(`${item.path}: references unknown troubleshooting '${id}'`); + } + + const automation = scalar(item.fields, "automation"); + if (!automation && listValue(item.fields, "automation_env").length > 0) { + errors.push(`${item.path}: 'automation_env' requires 'automation'`); + } + if (!automation && listValue(item.fields, "automation_env_any").length > 0) { + errors.push(`${item.path}: 'automation_env_any' requires 'automation'`); + } + if (!automation && (scalar(item.fields, "automation_pipeline_url_env") || scalar(item.fields, "automation_pipeline_name_env"))) { + errors.push(`${item.path}: automation pipeline env aliases require 'automation'`); + } + if (listValue(item.fields, "setup_provides_env").length > 0 && listValue(item.fields, "setup_automation").length === 0) { + errors.push(`${item.path}: 'setup_provides_env' requires 'setup_automation'`); + } + for (const key of ["automation_pipeline_url_env", "automation_pipeline_name_env"]) { + const value = scalar(item.fields, key); + if (!value) continue; + const declared = new Set([ + ...listValue(item.fields, "env"), + ...listValue(item.fields, "env_any").flatMap(splitEnvAnyGroup), + ...listValue(item.fields, "automation_env"), + ...listValue(item.fields, "automation_env_any").flatMap(splitEnvAnyGroup), + ]); + if (!declared.has(value)) { + errors.push(`${item.path}: '${key}' value '${value}' must be listed in env, env_any, automation_env, or automation_env_any`); + } + } + if (automation && !existsSync(join(root, automation))) { + errors.push(`${item.path}: automation script does not exist: ${automation}`); + } + for (const entry of listValue(item.fields, "setup_automation")) { + const spec = parseSetupAutomationEntry(entry); + if (spec.kind === "case" && spec.target === scalar(item.fields, "id")) { + errors.push(`${item.path}: setup_automation cannot reference the same case '${spec.target}'`); + } + } + + const timeout = scalar(item.fields, "automation_response_timeout_ms"); + if (timeout && (!/^\d+$/.test(timeout) || Number.parseInt(timeout, 10) <= 0)) { + errors.push(`${item.path}: 'automation_response_timeout_ms' must be a positive integer string`); + } + const streamOutput = scalar(item.fields, "automation_stream_output"); + if (streamOutput && !["0", "1", "false", "true"].includes(streamOutput)) { + errors.push(`${item.path}: 'automation_stream_output' must be one of 0, 1, false, or true`); + } + const imageBase64Fixture = scalar(item.fields, "automation_image_base64_fixture"); + if (imageBase64Fixture && !existsSync(join(root, imageBase64Fixture))) { + errors.push(`${item.path}: automation image fixture does not exist: ${imageBase64Fixture}`); + } + + return errors; +} + +function validateSetupAutomationCycles(caseItems: StructuredItem[]): string[] { + const byId = new Map(caseItems.map((item) => [scalar(item.fields, "id"), item])); + const visiting = new Set(); + const visited = new Set(); + const errors: string[] = []; + + function visit(id: string, path: string[]): void { + if (visited.has(id)) return; + if (visiting.has(id)) { + const cycle = [...path.slice(path.indexOf(id)), id].join(" -> "); + const item = byId.get(id); + errors.push(`${item?.path ?? id}: setup_automation case cycle detected: ${cycle}`); + return; + } + const item = byId.get(id); + if (!item) return; + visiting.add(id); + for (const entry of listValue(item.fields, "setup_automation")) { + const spec = parseSetupAutomationEntry(entry); + if (spec.kind === "case") visit(spec.target, [...path, spec.target]); + } + visiting.delete(id); + visited.add(id); + } + + for (const id of byId.keys()) visit(id, [id]); + return errors; +} + +function validateTroubleshootingItem(item: StructuredItem, caseIds: Set): string[] { + const errors = [ + ...validateEnum(item, "category", troubleshootingCategoryValues), + ...validateDuplicateListValues(item, ["symptoms", "patterns", "likely_causes", "fix_steps", "related_cases"]), + ]; + for (const id of listValue(item.fields, "related_cases")) { + if (!caseIds.has(id)) errors.push(`${item.path}: references unknown case '${id}'`); + } + return errors; +} + +function validateFixtures(root: string, caseIds: Set): string[] { + const { items, errors } = loadFixtureItems(root); + const result = [...errors]; + const seen = new Map(); + for (const item of items) { + if (!/^[a-z0-9][a-z0-9_-]*$/.test(item.id)) { + result.push(`${item.manifest_path}: fixture id '${item.id}' must use lowercase letters, digits, dashes, or underscores`); + } + if (seen.has(item.id)) { + result.push(`${item.manifest_path}: duplicate fixture id '${item.id}' also used by ${seen.get(item.id)}`); + } else { + seen.set(item.id, item.manifest_path); + } + if (!item.exists) result.push(`${item.manifest_path}: fixture path does not exist: ${item.path}`); + for (const caseId of item.related_cases) { + if (!caseIds.has(caseId)) result.push(`${item.manifest_path}: fixture '${item.id}' references unknown case '${caseId}'`); + } + } + return result; +} + +function validateSuiteItem(item: StructuredItem, caseIds: Set): string[] { + const errors = [ + ...validateEnum(item, "type", suiteTypeValues), + ...validateEnum(item, "priority", casePriorityValues), + ...validateDuplicateListValues(item, ["tags", "cases"]), + ]; + for (const id of listValue(item.fields, "cases")) { + if (!caseIds.has(id)) errors.push(`${item.path}: references unknown case '${id}'`); + } + return errors; +} + +function validateDuplicateIds(items: StructuredItem[], label: string): string[] { + const errors: string[] = []; + const seen = new Map(); + for (const item of items) { + const id = scalar(item.fields, "id"); + if (!id) continue; + const key = `${item.skill}:${id}`; + if (seen.has(key)) errors.push(`${item.path}: duplicate ${label} id '${id}' also used by ${seen.get(key)}`); + else seen.set(key, item.path); + } + return errors; +} + +function validateGlobalDuplicateIds(items: StructuredItem[], label: string): string[] { + const errors: string[] = []; + const seen = new Map(); + for (const item of items) { + const id = scalar(item.fields, "id"); + if (!id) continue; + if (seen.has(id)) errors.push(`${item.path}: duplicate global ${label} id '${id}' also used by ${seen.get(id)}`); + else seen.set(id, item.path); + } + return errors; +} + +function validateEnv(root: string): string[] { + const path = envPath(root); + const examplePath = envExamplePath(root); + const errors: string[] = []; + if (!existsSync(path)) return [`${path}: missing shared env file`]; + const env = parseEnvFile(path); + for (const key of requiredEnvKeys) { + if (!(key in env)) errors.push(`${path}: missing ${key}`); + } + if (!existsSync(examplePath)) { + errors.push(`${examplePath}: missing env template`); + } else { + const example = parseEnvFile(examplePath); + for (const key of requiredEnvKeys) { + if (!(key in example)) errors.push(`${examplePath}: missing template key ${key}`); + } + } + return errors; +} + +function validateSchemas(root: string): string[] { + const errors: string[] = []; + for (const name of ["case.schema.json", "suite.schema.json", "troubleshooting.schema.json", "skill-index.schema.json"]) { + const path = join(root, "schemas", name); + if (!existsSync(path)) { + errors.push(`${path}: missing schema`); + continue; + } + try { + JSON.parse(readFileSync(path, "utf8")); + } catch (error) { + errors.push(`${path}: invalid JSON schema (${String(error)})`); + } + } + return errors; +} + +function validateSkill(skill: Skill): string[] { + const errors: string[] = []; + if (!skill.name) errors.push(`${skill.path}: missing frontmatter name`); + if (!skill.description) errors.push(`${skill.path}: missing frontmatter description`); + if (skill.name && skill.name !== skill.directory) { + errors.push(`${skill.path}: name '${skill.name}' does not match directory '${skill.directory}'`); + } + + const refs = new Set(); + for (const match of skill.body.matchAll(refRe)) refs.add(match[1]); + for (const ref of Array.from(refs).sort()) { + if (!existsSync(join(skill.path, ref))) { + errors.push(`${skill.path}: referenced file does not exist: ${ref}`); + } + } + + const legacyTroubleshooting = join(skill.path, "references", "troubleshooting.md"); + if (existsSync(legacyTroubleshooting)) { + const text = readFileSync(legacyTroubleshooting, "utf8"); + if (text.includes("\n## ") && !text.includes("### Symptom")) { + errors.push(`${legacyTroubleshooting}: troubleshooting entries should include '### Symptom'`); + } + } + + return errors; +} + +export function commandValidate(root: string): number { + const skills = loadSkills(root); + const caseItems = loadStructuredItems(root, "cases"); + const suiteItems = loadStructuredItems(root, "suites"); + const troubleItems = loadStructuredItems(root, "troubleshooting"); + const skillNames = new Set(skills.map((skill) => skill.name)); + const caseIds = new Set(caseItems.map((item) => scalar(item.fields, "id")).filter(Boolean)); + const troubleIds = new Set(troubleItems.map((item) => scalar(item.fields, "id")).filter(Boolean)); + const errors = [ + ...validateEnv(root), + ...validateSchemas(root), + ...skills.flatMap(validateSkill), + ...caseItems.flatMap((item) => validateStructuredItem(item, caseRequiredStrings, caseRequiredLists)), + ...caseItems.flatMap((item) => validateCaseItem(root, item, skillNames, troubleIds, caseIds)), + ...validateSetupAutomationCycles(caseItems), + ...suiteItems.flatMap((item) => validateStructuredItem(item, suiteRequiredStrings, suiteRequiredLists)), + ...suiteItems.flatMap((item) => validateSuiteItem(item, caseIds)), + ...troubleItems.flatMap((item) => validateStructuredItem(item, troubleRequiredStrings, troubleRequiredLists)), + ...troubleItems.flatMap((item) => validateTroubleshootingItem(item, caseIds)), + ...validateFixtures(root, caseIds), + ...validateDuplicateIds(caseItems, "case"), + ...validateDuplicateIds(suiteItems, "suite"), + ...validateDuplicateIds(troubleItems, "troubleshooting"), + ...validateGlobalDuplicateIds(caseItems, "case"), + ...validateGlobalDuplicateIds(suiteItems, "suite"), + ...validateGlobalDuplicateIds(troubleItems, "troubleshooting"), + ]; + + if (errors.length > 0) { + for (const error of errors) stderr.write(`ERROR: ${error}\n`); + return 1; + } + console.log("OK"); + return 0; +} diff --git a/skills/src/constants.ts b/skills/src/constants.ts new file mode 100644 index 000000000..015a9bd39 --- /dev/null +++ b/skills/src/constants.ts @@ -0,0 +1,34 @@ +export const requiredEnvKeys = [ + "LANGBOT_FRONTEND_URL", + "LANGBOT_BACKEND_URL", + "LANGBOT_DEV_FRONTEND_URL", + "LANGBOT_REPO", + "LANGBOT_WEB_REPO", + "LANGBOT_BROWSER_PROFILE", + "LANGBOT_CHROMIUM_EXECUTABLE", +]; + +export const caseModeValues = ["agent-browser", "probe"]; +export const caseTypeValues = ["smoke", "regression", "feature", "provider", "exploratory"]; +export const casePriorityValues = ["p0", "p1", "p2"]; +export const caseRiskValues = ["low", "medium", "high"]; +export const caseEvidenceValues = [ + "ui", + "screenshot", + "console", + "network", + "backend_log", + "frontend_log", + "api_diagnostic", + "filesystem", +]; +export const testResultStatusValues = ["pass", "fail", "blocked", "env_issue", "flaky"]; +export const troubleshootingCategoryValues = ["product", "env_issue", "external_dependency", "blocked", "flaky"]; +export const suiteTypeValues = ["smoke", "regression", "release_gate", "exploratory"]; +export const suiteRequiredStrings = ["id", "title", "description", "type", "priority"]; +export const suiteRequiredLists = ["tags", "cases"]; + +export const caseRequiredStrings = ["id", "title", "mode", "area", "type", "priority", "risk"]; +export const caseRequiredLists = ["tags", "skills", "env", "steps", "checks", "evidence_required"]; +export const troubleRequiredStrings = ["id", "title", "verification"]; +export const troubleRequiredLists = ["symptoms", "patterns", "likely_causes", "fix_steps"]; diff --git a/skills/src/env-groups.ts b/skills/src/env-groups.ts new file mode 100644 index 000000000..069193600 --- /dev/null +++ b/skills/src/env-groups.ts @@ -0,0 +1,10 @@ +export const envKeyPattern = /^[A-Z][A-Z0-9_]*$/; + +export function splitEnvAnyGroup(value: string): string[] { + return value.split("|").map((item) => item.trim()).filter(Boolean); +} + +export function isEnvAnyGroup(value: string): boolean { + const keys = splitEnvAnyGroup(value); + return keys.length >= 2 && new Set(keys).size === keys.length && keys.every((key) => envKeyPattern.test(key)); +} diff --git a/skills/src/fixtures.ts b/skills/src/fixtures.ts new file mode 100644 index 000000000..698ad3d8b --- /dev/null +++ b/skills/src/fixtures.ts @@ -0,0 +1,86 @@ +import { existsSync, readFileSync } from "node:fs"; +import { join } from "node:path"; +import { loadSkills } from "./fs.ts"; + +export type FixtureItem = { + skill: string; + manifest_path: string; + id: string; + title: string; + path: string; + kind: string; + related_cases: string[]; + checks: string[]; + absolute_path: string; + exists: boolean; +}; + +export type FixtureLoadResult = { + items: FixtureItem[]; + errors: string[]; +}; + +function stringField(data: Record, key: string): string { + const value = data[key]; + return typeof value === "string" ? value : ""; +} + +function stringList(data: Record, key: string): string[] { + const value = data[key]; + return Array.isArray(value) ? value.filter((item): item is string => typeof item === "string") : []; +} + +export function loadFixtureItems(root: string, skillFilter?: string): FixtureLoadResult { + const items: FixtureItem[] = []; + const errors: string[] = []; + const skills = loadSkills(root).filter((skill) => !skillFilter || skill.directory === skillFilter || skill.name === skillFilter); + + for (const skill of skills) { + const manifestPath = join(skill.path, "fixtures", "fixtures.json"); + if (!existsSync(manifestPath)) continue; + + let parsed: unknown; + try { + parsed = JSON.parse(readFileSync(manifestPath, "utf8")); + } catch (error) { + errors.push(`${manifestPath}: invalid fixture manifest JSON (${String(error)})`); + continue; + } + + if (!Array.isArray(parsed)) { + errors.push(`${manifestPath}: fixture manifest must be a JSON array`); + continue; + } + + for (const [index, entry] of parsed.entries()) { + if (!entry || typeof entry !== "object" || Array.isArray(entry)) { + errors.push(`${manifestPath}: fixture entry ${index} must be an object`); + continue; + } + const data = entry as Record; + const id = stringField(data, "id"); + const title = stringField(data, "title"); + const path = stringField(data, "path"); + const kind = stringField(data, "kind") || "file"; + if (!id || !title || !path) { + errors.push(`${manifestPath}: fixture entry ${index} must include id, title, and path`); + continue; + } + const absolutePath = join(skill.path, path); + items.push({ + skill: skill.directory, + manifest_path: manifestPath, + id, + title, + path, + kind, + related_cases: stringList(data, "related_cases"), + checks: stringList(data, "checks"), + absolute_path: absolutePath, + exists: existsSync(absolutePath), + }); + } + } + + return { items, errors }; +} diff --git a/skills/src/fs.ts b/skills/src/fs.ts new file mode 100644 index 000000000..238e253ff --- /dev/null +++ b/skills/src/fs.ts @@ -0,0 +1,226 @@ +import { existsSync, readdirSync, readFileSync, statSync } from "node:fs"; +import { join } from "node:path"; +import type { ParsedYaml, Skill, StructuredItem, StructuredItemKind } from "./types.ts"; +import { fail } from "./cli.ts"; + +const frontmatterRe = /^---\n([\s\S]*?)\n---\n/; + +export function statIsDirectory(path: string): boolean { + try { + return statSync(path).isDirectory(); + } catch { + return false; + } +} + +export function skillsRoot(root: string): string { + const nested = join(root, "skills"); + return existsSync(nested) && statIsDirectory(nested) ? nested : root; +} + +export function envPath(root: string): string { + return join(skillsRoot(root), ".env"); +} + +export function envLocalPath(root: string): string { + return join(skillsRoot(root), ".env.local"); +} + +export function envExamplePath(root: string): string { + return join(skillsRoot(root), ".env.example"); +} + +export function loadEnv(root: string): Record { + return { + ...parseEnvFile(envPath(root)), + ...parseEnvFile(envLocalPath(root)), + }; +} + +export function listDirectories(root: string): string[] { + return readdirSync(root) + .filter((name) => !name.startsWith(".")) + .filter((name) => statIsDirectory(join(root, name))) + .sort(); +} + +export function parseFrontmatter(text: string): { meta: Record; body: string } { + const match = text.match(frontmatterRe); + if (!match) return { meta: {}, body: text }; + + const meta: Record = {}; + for (const line of match[1].split("\n")) { + const sep = line.indexOf(":"); + if (sep === -1) continue; + const key = line.slice(0, sep).trim(); + const value = line.slice(sep + 1).trim().replace(/^["']|["']$/g, ""); + meta[key] = value; + } + + return { meta, body: text.slice(match[0].length) }; +} + +export function loadSkills(root: string): Skill[] { + const skills: Skill[] = []; + const base = skillsRoot(root); + for (const directory of listDirectories(base)) { + const skillPath = join(base, directory); + const skillMd = join(skillPath, "SKILL.md"); + if (!existsSync(skillMd)) continue; + const text = readFileSync(skillMd, "utf8"); + const { meta, body } = parseFrontmatter(text); + skills.push({ + path: skillPath, + directory, + name: meta.name ?? "", + description: meta.description ?? "", + body, + }); + } + return skills; +} + +export function getSkill(root: string, skillName: string): Skill { + const skill = loadSkills(root).find((item) => item.directory === skillName || item.name === skillName); + if (!skill) fail(`unknown skill: ${skillName}`); + return skill; +} + +export function parseEnvFile(path: string): Record { + if (!existsSync(path)) return {}; + const env: Record = {}; + for (const rawLine of readFileSync(path, "utf8").split(/\r?\n/)) { + const line = rawLine.trim(); + if (!line || line.startsWith("#")) continue; + const sep = line.indexOf("="); + if (sep === -1) continue; + const key = line.slice(0, sep).trim(); + const value = line.slice(sep + 1).trim().replace(/^["']|["']$/g, ""); + env[key] = value; + } + return env; +} + +export function globMarkdownRefs(skillPath: string): string[] { + const refsDir = join(skillPath, "references"); + if (!existsSync(refsDir)) return []; + return readdirSync(refsDir) + .filter((name) => name.endsWith(".md")) + .sort() + .map((name) => join("references", name)); +} + +export function globYamlFiles(dir: string): string[] { + if (!existsSync(dir)) return []; + return readdirSync(dir) + .filter((name) => name.endsWith(".yaml") || name.endsWith(".yml")) + .sort() + .map((name) => join(dir, name)); +} + +function unquote(value: string): string { + return value.trim().replace(/^["']|["']$/g, ""); +} + +function parseScalarValue(value: string): string | boolean { + const trimmed = value.trim(); + if (/^["'].*["']$/.test(trimmed)) return unquote(trimmed); + if (trimmed === "true") return true; + if (trimmed === "false") return false; + return trimmed; +} + +export function parseYamlLite(text: string): ParsedYaml { + const fields: ParsedYaml = {}; + let currentList: string | null = null; + + for (const rawLine of text.split(/\r?\n/)) { + const line = rawLine.replace(/\s+$/, ""); + if (!line.trim() || line.trim().startsWith("#")) continue; + + const pair = line.match(/^([A-Za-z0-9_]+):\s*(.*)$/); + if (pair) { + const key = pair[1]; + const value = pair[2]; + if (value === "") { + fields[key] = []; + currentList = key; + } else { + fields[key] = parseScalarValue(value); + currentList = null; + } + continue; + } + + const item = line.match(/^\s*-\s*(.*)$/); + if (item && currentList) { + const existing = fields[currentList]; + if (Array.isArray(existing)) existing.push(unquote(item[1])); + } + } + + return fields; +} + +export function scalar(fields: ParsedYaml, key: string): string { + const value = fields[key]; + return typeof value === "string" ? value : ""; +} + +export function boolValue(fields: ParsedYaml, key: string): boolean | undefined { + const value = fields[key]; + return typeof value === "boolean" ? value : undefined; +} + +export function listValue(fields: ParsedYaml, key: string): string[] { + const value = fields[key]; + return Array.isArray(value) ? value : []; +} + +export function yamlQuote(value: string): string { + return JSON.stringify(value); +} + +export function yamlList(values: string[]): string { + return values.map((value) => ` - ${yamlQuote(value)}`).join("\n"); +} + +export function loadStructuredItems(root: string, kind: StructuredItemKind, skillFilter?: string): StructuredItem[] { + const skills = skillFilter ? [getSkill(root, skillFilter)] : loadSkills(root); + const items: StructuredItem[] = []; + for (const skill of skills) { + for (const path of globYamlFiles(join(skill.path, kind))) { + const raw = readFileSync(path, "utf8"); + items.push({ path, skill: skill.directory, fields: parseYamlLite(raw), raw }); + } + } + return items.sort((a, b) => `${a.skill}:${scalar(a.fields, "id")}`.localeCompare(`${b.skill}:${scalar(b.fields, "id")}`)); +} + +export function findStructuredItem( + root: string, + kind: StructuredItemKind, + skillOrId: string, + maybeId?: string, +): StructuredItem { + const skillFilter = maybeId ? skillOrId : undefined; + const id = maybeId ?? skillOrId; + const matches = loadStructuredItems(root, kind, skillFilter).filter((item) => scalar(item.fields, "id") === id); + if (matches.length === 0) fail(`unknown ${kind.slice(0, -1)}: ${id}`); + if (matches.length > 1) { + fail(`ambiguous ${kind.slice(0, -1)} '${id}', specify skill: ${matches.map((item) => item.skill).join(", ")}`); + } + return matches[0]; +} + +export function slugify(input: string): string { + return input + .trim() + .toLowerCase() + .replace(/[^a-z0-9\u4e00-\u9fa5]+/g, "-") + .replace(/^-+|-+$/g, ""); +} + +export function todayIso(): string { + return new Date().toISOString().slice(0, 10); +} diff --git a/skills/src/lbs.ts b/skills/src/lbs.ts new file mode 100644 index 000000000..8329a5803 --- /dev/null +++ b/skills/src/lbs.ts @@ -0,0 +1,84 @@ +#!/usr/bin/env node + +import { argv, exit } from "node:process"; +import { parseGlobalArgs, usage } from "./cli.ts"; +import { commandCaseList, commandCaseNew, commandCaseShow } from "./commands/case.ts"; +import { commandEnvDoctor, commandEnvShow } from "./commands/env.ts"; +import { commandFixtureCheck, commandFixtureList } from "./commands/fixture.ts"; +import { commandIndex, commandList, commandNewRef, commandNewSkill } from "./commands/skill.ts"; +import { commandLogGuard, commandLogScan, commandLogWatch } from "./commands/log.ts"; +import { commandSuiteList, commandSuiteNew, commandSuitePlan, commandSuiteReport, commandSuiteRun, commandSuiteShow, commandSuiteStart } from "./commands/suite.ts"; +import { commandTestPlan, commandTestRecommend, commandTestReport, commandTestResult, commandTestRun, commandTestStart } from "./commands/test.ts"; +import { commandTroubleAdd, commandTroubleList, commandTroubleSearch, commandTroubleShow } from "./commands/trouble.ts"; +import { commandValidate } from "./commands/validate.ts"; + +async function main(): Promise { + const ctx = parseGlobalArgs(argv.slice(2)); + const command = ctx.args[0]; + if (!command) usage(); + + if (command === "list") return commandList(ctx); + if (command === "validate") return commandValidate(ctx.root); + if (command === "index") return commandIndex(ctx); + if (command === "new-skill") return commandNewSkill(ctx); + if (command === "new-ref") return commandNewRef(ctx); + + if (command === "env") { + const sub = ctx.args[1]; + if (sub === "show") return commandEnvShow(ctx); + if (sub === "doctor") return await commandEnvDoctor(ctx); + } + + if (command === "fixture") { + const sub = ctx.args[1]; + if (sub === "list") return commandFixtureList(ctx); + if (sub === "check") return commandFixtureCheck(ctx); + } + + if (command === "log") { + const sub = ctx.args[1]; + if (sub === "scan") return commandLogScan(ctx); + if (sub === "watch") return await commandLogWatch(ctx); + if (sub === "guard") return commandLogGuard(ctx); + } + + if (command === "case") { + const sub = ctx.args[1]; + if (sub === "new") return commandCaseNew(ctx); + if (sub === "list") return commandCaseList(ctx); + if (sub === "show") return commandCaseShow(ctx); + } + + if (command === "suite") { + const sub = ctx.args[1]; + if (sub === "new") return commandSuiteNew(ctx); + if (sub === "list") return commandSuiteList(ctx); + if (sub === "show") return commandSuiteShow(ctx); + if (sub === "plan") return commandSuitePlan(ctx); + if (sub === "start") return commandSuiteStart(ctx); + if (sub === "run") return commandSuiteRun(ctx); + if (sub === "report") return commandSuiteReport(ctx); + } + + if (command === "test") { + const sub = ctx.args[1]; + if (sub === "plan") return commandTestPlan(ctx); + if (sub === "recommend") return commandTestRecommend(ctx); + if (sub === "start") return commandTestStart(ctx); + if (sub === "run") return commandTestRun(ctx); + if (sub === "report") return commandTestReport(ctx); + if (sub === "result") return commandTestResult(ctx); + } + + if (command === "trouble") { + const sub = ctx.args[1]; + if (sub === "list") return commandTroubleList(ctx); + if (sub === "show") return commandTroubleShow(ctx); + if (sub === "search") return commandTroubleSearch(ctx); + if (sub === "add") return commandTroubleAdd(ctx); + } + + usage(); +} + +exit(await main()); diff --git a/skills/src/log-guard.ts b/skills/src/log-guard.ts new file mode 100644 index 000000000..253cb229e --- /dev/null +++ b/skills/src/log-guard.ts @@ -0,0 +1,805 @@ +import { existsSync, readdirSync, readFileSync, statSync } from "node:fs"; +import { dirname, join, resolve } from "node:path"; +import type { StructuredItem } from "./types.ts"; +import { listValue, loadEnv, loadStructuredItems, scalar } from "./fs.ts"; + +export type LogSourceName = "backend" | "frontend" | "console"; +export type FindingSeverity = + | "fail" + | "warning" + | "matched_troubleshooting" + | "env_issue" + | "ignored_expected_issue" + | "missing_input"; + +export type LogFinding = { + source: LogSourceName; + path: string; + severity: FindingSeverity; + kind: string; + pattern: string; + line?: number; + excerpt?: string; + troubleshooting_id?: string; + troubleshooting_title?: string; + related_to_case?: boolean; +}; + +export type LogLine = { + number: number; + text: string; +}; + +export type LogSuccessSignal = { + source: LogSourceName; + path: string; + kind: "case_success_pattern"; + pattern: string; + line?: number; + excerpt?: string; +}; + +export type LogScanMode = + | "whole-file" + | "since" + | "until" + | "since+until" + | "tail-lines" + | "since+tail-lines" + | "until+tail-lines" + | "since+until+tail-lines"; + +export type LogScanConfig = { + mode: LogScanMode; + since?: string; + since_epoch_ms?: number; + until?: string; + until_epoch_ms?: number; + tail_lines?: number; + warnings: string[]; +}; + +export type LogSourceSummary = { + source: LogSourceName; + path: string; + status: "scanned" | "missing" | "auto_not_found"; + line_count: number; + total_line_count?: number; + start_line?: number; + end_line?: number; + timestamped_line_count?: number; + auto_detected?: boolean; +}; + +export type LogGuardPatternContext = { + successPatterns?: string[]; + failurePatterns?: string[]; + expectedFailures?: string[]; + relatedTroubleshootingIds?: string[]; +}; + +export type LogGuardResult = { + status: string; + scan: LogScanConfig; + sources: LogSourceSummary[]; + success_signals: LogSuccessSignal[]; + findings: LogFinding[]; +}; + +export type AutomationResultEvidence = { + status: "not_provided" | "missing" | "invalid" | "loaded"; + path?: string; + result?: string; + reason?: string; + started_at?: string; + started_at_local?: string; + finished_at?: string; + finished_at_local?: string; + url?: string; + prompt?: string; + expected_text?: string; +}; + +type MutableScanState = { + findings: LogFinding[]; + successSignals: LogSuccessSignal[]; + seenFindings: Set; + seenSuccessSignals: Set; +}; + +const secretAssignmentRe = /\b(api[_-]?key|authorization|credential|jwt|oauth|password|secret|token)\s*[:=]\s*["']?([^"',\s]+)/gi; +const bearerSecretRe = /\bbearer\s+[A-Za-z0-9._~+/=-]{8,}/i; +const openAiStyleSecretRe = /\bsk-[A-Za-z0-9_-]{6,}\b/i; + +const unexpectedPatterns: Array<{ + kind: string; + pattern: string; + regex: RegExp; + severity: FindingSeverity; + sources?: LogSourceName[]; +}> = [ + { kind: "python_traceback", pattern: "Traceback", regex: /\bTraceback(?: \(most recent call last\))?/i, severity: "fail" }, + { + kind: "unretrieved_task_exception", + pattern: "Task exception was never retrieved", + regex: /Task exception was never retrieved/i, + severity: "fail", + }, + { + kind: "unawaited_coroutine", + pattern: "RuntimeWarning: coroutine .* was never awaited", + regex: /RuntimeWarning:\s+coroutine .* was never awaited/i, + severity: "fail", + }, + { + kind: "unclosed_client_session", + pattern: "Unclosed client session", + regex: /Unclosed client session/i, + severity: "fail", + }, + { kind: "unclosed_connector", pattern: "Unclosed connector", regex: /Unclosed connector/i, severity: "fail" }, + { kind: "key_error", pattern: "KeyError", regex: /(^|[^A-Za-z])KeyError(?:\b|:)/, severity: "fail" }, + { kind: "type_error", pattern: "TypeError", regex: /(^|[^A-Za-z])TypeError(?:\b|:)/, severity: "fail" }, + { + kind: "attribute_error", + pattern: "AttributeError", + regex: /(^|[^A-Za-z])AttributeError(?:\b|:)/, + severity: "fail", + }, + { + kind: "frontend_uncaught_error", + pattern: "Uncaught frontend error", + regex: /\bUncaught (?:[A-Za-z]*Error|Exception)|Unhandled(?: promise rejection|Rejection)/i, + severity: "fail", + sources: ["console", "frontend"], + }, + { + kind: "http_5xx", + pattern: "HTTP 5xx resource failure", + regex: /Failed to load resource: the server responded with a status of 5\d\d|HTTP\/\d(?:\.\d)?\s+5\d\d/i, + severity: "fail", + }, + { kind: "error_log", pattern: "ERROR or CRITICAL log line", regex: /\b(?:ERROR|CRITICAL)\b/, severity: "warning" }, +]; + +export function logPatternContextFromStructuredItem(item: StructuredItem): LogGuardPatternContext { + return { + successPatterns: listValue(item.fields, "success_patterns"), + failurePatterns: listValue(item.fields, "failure_patterns"), + expectedFailures: listValue(item.fields, "expected_failures"), + relatedTroubleshootingIds: listValue(item.fields, "troubleshooting"), + }; +} + +export function scanStructuredLogSources( + root: string, + item: StructuredItem, + options: Record, +): LogGuardResult { + return scanLogSources(root, options, logPatternContextFromStructuredItem(item)); +} + +export function scanLogSources( + root: string, + options: Record, + context: LogGuardPatternContext = {}, +): LogGuardResult { + const env = loadEnv(root); + const scan = parseScanConfig(optionsWithEvidenceWindow(options)); + const configuredSources: Array<{ source: LogSourceName; option: string }> = [ + { source: "backend", option: "backend-log" }, + { source: "frontend", option: "frontend-log" }, + { source: "console", option: "console-log" }, + ]; + const sources: LogSourceSummary[] = []; + const state: MutableScanState = { + findings: [], + successSignals: [], + seenFindings: new Set(), + seenSuccessSignals: new Set(), + }; + + for (const warning of scan.warnings) { + addFinding(state, { + source: "backend", + path: "log-scan-options", + severity: "missing_input", + kind: "invalid_log_scan_option", + pattern: warning, + }); + } + + for (const configured of configuredSources) { + const explicitPath = options[configured.option]; + const autoPath = configured.source === "backend" && options["no-auto-log"] !== true + ? latestLangBotLogPath(env) + : null; + const rawPath = typeof explicitPath === "string" ? explicitPath : autoPath; + const autoDetected = typeof explicitPath !== "string" && rawPath === autoPath; + if (!rawPath) { + if (configured.source === "backend" && options["no-auto-log"] !== true) { + const logsDir = env.LANGBOT_REPO ? join(env.LANGBOT_REPO, "data", "logs") : "LANGBOT_REPO/data/logs"; + sources.push({ source: "backend", path: join(logsDir, "langbot-*.log"), status: "auto_not_found", line_count: 0, auto_detected: true }); + } + continue; + } + + const path = resolve(rawPath); + if (!existsSync(path)) { + sources.push({ source: configured.source, path, status: "missing", line_count: 0 }); + if (!autoDetected) { + addFinding(state, { + source: configured.source, + path, + severity: "missing_input", + kind: "missing_log_file", + pattern: `${configured.option} path does not exist`, + }); + } + continue; + } + + const text = readFileSync(path, "utf8"); + scanLogTextIntoState(root, configured.source, path, text, scan, context, sources, state); + } + + finalizeMissingSuccessSignal(context, sources, state); + return buildLogGuardResult(scan, sources, state); +} + +export function scanLogText( + root: string, + source: LogSourceName, + path: string, + text: string, + options: Record = {}, + context: LogGuardPatternContext = {}, + baseLineNumber = 0, + includeMissingSuccessSignal = true, +): LogGuardResult { + const scan = parseScanConfig(options); + const sources: LogSourceSummary[] = []; + const state: MutableScanState = { + findings: [], + successSignals: [], + seenFindings: new Set(), + seenSuccessSignals: new Set(), + }; + + scanLogTextIntoState(root, source, resolve(path), text, scan, context, sources, state, baseLineNumber); + if (includeMissingSuccessSignal) finalizeMissingSuccessSignal(context, sources, state); + return buildLogGuardResult(scan, sources, state); +} + +function scanLogTextIntoState( + root: string, + source: LogSourceName, + path: string, + text: string, + scan: LogScanConfig, + context: LogGuardPatternContext, + sources: LogSourceSummary[], + state: MutableScanState, + baseLineNumber = 0, +): void { + const allLines = text.split(/\r?\n/).map((line, index) => ({ number: baseLineNumber + index + 1, text: line })); + const selected = selectLinesForScan(allLines, scan); + sources.push({ + source, + path, + status: "scanned", + line_count: selected.lines.length, + total_line_count: allLines.length, + start_line: selected.lines[0]?.number, + end_line: selected.lines[selected.lines.length - 1]?.number, + timestamped_line_count: selected.timestampedLineCount, + }); + + scanUnexpectedPatterns(state, source, path, selected.lines, context.expectedFailures ?? []); + scanCaseDeclaredPatterns( + state, + source, + path, + selected.lines, + context.successPatterns ?? [], + context.failurePatterns ?? [], + context.expectedFailures ?? [], + ); + scanTroubleshootingPatterns( + state, + source, + path, + selected.lines, + loadStructuredItems(root, "troubleshooting"), + new Set(context.relatedTroubleshootingIds ?? []), + context.expectedFailures ?? [], + ); +} + +function buildLogGuardResult(scan: LogScanConfig, sources: LogSourceSummary[], state: MutableScanState): LogGuardResult { + const scannedCount = sources.filter((source) => source.status === "scanned").length; + const status = scannedCount === 0 && state.findings.length === 0 + ? "not_run" + : state.findings.some((finding) => finding.severity === "fail" || finding.severity === "missing_input") + ? "fail" + : state.findings.some((finding) => finding.severity === "matched_troubleshooting" && finding.related_to_case !== false) + ? "fail" + : state.findings.some((finding) => finding.severity === "env_issue") + ? "env_issue" + : state.findings.some((finding) => finding.severity === "warning") + ? "warning" + : "pass"; + + return { status, scan, sources, success_signals: state.successSignals, findings: state.findings }; +} + +function finalizeMissingSuccessSignal( + context: LogGuardPatternContext, + sources: LogSourceSummary[], + state: MutableScanState, +): void { + const scannedCount = sources.filter((source) => source.status === "scanned").length; + const successPatterns = context.successPatterns ?? []; + if (scannedCount > 0 && successPatterns.length > 0 && state.successSignals.length === 0) { + addFinding(state, { + source: "backend", + path: "case-success-patterns", + severity: "warning", + kind: "missing_success_signal", + pattern: successPatterns.join(" | "), + excerpt: "No declared success_patterns matched the scanned log window.", + }); + } +} + +function shouldTreatAssignmentValueAsSecret(value: string): boolean { + const normalized = value.trim().replace(/^["']|["']$/g, ""); + const lower = normalized.toLowerCase(); + if (!normalized) return false; + if (["error", "invalid", "missing", "none", "null", "undefined", "redacted", "[redacted]"].includes(lower)) { + return false; + } + if (/^(error|invalid|missing|none|null|undefined)\b/i.test(normalized)) return false; + if (/^(your-|<|\$\{|example-|placeholder)/i.test(normalized)) return false; + if (openAiStyleSecretRe.test(normalized)) return true; + return normalized.length >= 8 && /[A-Za-z0-9]/.test(normalized); +} + +function redactSecretAssignments(text: string): string { + return text.replace(secretAssignmentRe, (match, key: string, value: string) => { + if (!shouldTreatAssignmentValueAsSecret(value)) return match; + return match.replace(value, "[redacted]"); + }); +} + +export function redactSecrets(text: string): string { + return redactSecretAssignments(text + .replace(/(\bauthorization\s*[:=]\s*bearer\s+)[A-Za-z0-9._~+/=-]+/gi, "$1[redacted]") + .replace(/\bbearer\s+[A-Za-z0-9._~+/=-]{8,}/gi, "Bearer [redacted]") + .replace(/\bsk-[A-Za-z0-9_-]{6,}\b/g, "[redacted]")); +} + +function hasSecretLeak(line: string): boolean { + secretAssignmentRe.lastIndex = 0; + const hasSecretAssignment = Array.from(line.matchAll(secretAssignmentRe)) + .some((match) => shouldTreatAssignmentValueAsSecret(match[2] ?? "")); + return hasSecretAssignment || bearerSecretRe.test(line) || openAiStyleSecretRe.test(line); +} + +function findingKey(finding: LogFinding): string { + return [ + finding.source, + finding.path, + finding.kind, + finding.pattern, + finding.line ?? "", + finding.troubleshooting_id ?? "", + ].join("\0"); +} + +function addFinding(state: MutableScanState, finding: LogFinding): void { + const key = findingKey(finding); + if (state.seenFindings.has(key)) return; + state.seenFindings.add(key); + state.findings.push(finding); +} + +function successSignalKey(signal: LogSuccessSignal): string { + return [signal.source, signal.path, signal.pattern, signal.line ?? ""].join("\0"); +} + +function addSuccessSignal(state: MutableScanState, signal: LogSuccessSignal): void { + const key = successSignalKey(signal); + if (state.seenSuccessSignals.has(key)) return; + state.seenSuccessSignals.add(key); + state.successSignals.push(signal); +} + +function isExpectedFinding(finding: LogFinding, expectedFailures: string[]): boolean { + if (finding.kind === "secret_leak" || finding.severity === "missing_input") return false; + const haystack = [ + finding.kind, + finding.pattern, + finding.troubleshooting_id ?? "", + finding.troubleshooting_title ?? "", + finding.excerpt ?? "", + ].join("\n").toLowerCase(); + return expectedFailures.some((item) => item && haystack.includes(item.toLowerCase())); +} + +function withExpectedSeverity(finding: LogFinding, expectedFailures: string[]): LogFinding { + if (!isExpectedFinding(finding, expectedFailures)) return finding; + return { ...finding, severity: "ignored_expected_issue" }; +} + +function scanUnexpectedPatterns( + state: MutableScanState, + source: LogSourceName, + path: string, + lines: LogLine[], + expectedFailures: string[], +): void { + for (const line of lines) { + for (const pattern of unexpectedPatterns) { + if (pattern.sources && !pattern.sources.includes(source)) continue; + if (!pattern.regex.test(line.text)) continue; + addFinding(state, withExpectedSeverity({ + source, + path, + severity: pattern.severity, + kind: pattern.kind, + pattern: pattern.pattern, + line: line.number, + excerpt: redactSecrets(line.text.trim()), + }, expectedFailures)); + } + + if (hasSecretLeak(line.text)) { + addFinding(state, { + source, + path, + severity: "fail", + kind: "secret_leak", + pattern: "secret-like value in logs", + line: line.number, + excerpt: redactSecrets(line.text.trim()), + }); + } + } +} + +function scanTroubleshootingPatterns( + state: MutableScanState, + source: LogSourceName, + path: string, + lines: LogLine[], + troubles: StructuredItem[], + relatedIds: Set, + expectedFailures: string[], +): void { + for (const entry of troubles) { + const id = scalar(entry.fields, "id"); + const title = scalar(entry.fields, "title"); + const category = scalar(entry.fields, "category"); + for (const pattern of listValue(entry.fields, "patterns")) { + const needle = pattern.toLowerCase(); + if (!needle) continue; + let matchesForPattern = 0; + for (const line of lines) { + if (!line.text.toLowerCase().includes(needle)) continue; + if (id === "plugin-runtime-timeout" && isModelRouteUnavailableText(line.text)) continue; + addFinding(state, withExpectedSeverity({ + source, + path, + severity: category === "env_issue" ? "env_issue" : "matched_troubleshooting", + kind: "troubleshooting_pattern", + pattern, + line: line.number, + excerpt: redactSecrets(line.text.trim()), + troubleshooting_id: id, + troubleshooting_title: title, + related_to_case: relatedIds.has(id), + }, expectedFailures)); + matchesForPattern += 1; + if (matchesForPattern >= 3) break; + } + } + } +} + +function isModelRouteUnavailableText(text: string): boolean { + return /model_not_found|no available channel for model|invalid api key|当前分组上游负载已饱和/i.test(text); +} + +function scanCaseDeclaredPatterns( + state: MutableScanState, + source: LogSourceName, + path: string, + lines: LogLine[], + successPatterns: string[], + failurePatterns: string[], + expectedFailures: string[], +): void { + for (const pattern of successPatterns) { + const needle = pattern.toLowerCase(); + if (!needle) continue; + let matchesForPattern = 0; + for (const line of lines) { + if (!line.text.toLowerCase().includes(needle)) continue; + addSuccessSignal(state, { + source, + path, + kind: "case_success_pattern", + pattern, + line: line.number, + excerpt: redactSecrets(line.text.trim()), + }); + matchesForPattern += 1; + if (matchesForPattern >= 3) break; + } + } + + for (const pattern of failurePatterns) { + const needle = pattern.toLowerCase(); + if (!needle) continue; + let matchesForPattern = 0; + for (const line of lines) { + if (!line.text.toLowerCase().includes(needle)) continue; + addFinding(state, withExpectedSeverity({ + source, + path, + severity: "fail", + kind: "case_failure_pattern", + pattern, + line: line.number, + excerpt: redactSecrets(line.text.trim()), + }, expectedFailures)); + matchesForPattern += 1; + if (matchesForPattern >= 3) break; + } + } +} + +function optionsWithEvidenceWindow(options: Record): Record { + if (typeof options.since === "string" && typeof options.until === "string") { + return options; + } + + const evidenceDir = evidenceDirFromOptions(options); + if (!evidenceDir) return options; + + const resultPath = automationResultPath(evidenceDir); + if (!existsSync(resultPath)) return options; + + try { + const result = JSON.parse(readFileSync(resultPath, "utf8")) as Record; + const enriched = { ...options }; + const startedAt = stringField(result, "started_at_local") ?? stringField(result, "started_at"); + const finishedAt = stringField(result, "finished_at_local") ?? stringField(result, "finished_at"); + + if (typeof enriched.since !== "string" && startedAt) { + enriched.since = startedAt; + } + if (typeof enriched.until !== "string" && finishedAt) { + enriched.until = finishedAt; + } + return enriched; + } catch { + return options; + } +} + +function stringField(data: Record, key: string): string | undefined { + const value = data[key]; + return typeof value === "string" && value.trim() ? value : undefined; +} + +function evidenceDirFromOptions(options: Record): string | undefined { + const explicit = typeof options["evidence-dir"] === "string" ? options["evidence-dir"] : undefined; + if (explicit) return resolve(explicit); + const consoleLog = typeof options["console-log"] === "string" ? options["console-log"] : undefined; + return consoleLog ? dirname(resolve(consoleLog)) : undefined; +} + +function automationResultPath(evidenceDir: string): string { + const primary = join(evidenceDir, "automation-result.json"); + if (existsSync(primary)) return primary; + return join(evidenceDir, "result.json"); +} + +export function readAutomationResultEvidence(options: Record): AutomationResultEvidence { + const evidenceDir = evidenceDirFromOptions(options); + if (!evidenceDir) return { status: "not_provided" }; + + const resultPath = automationResultPath(evidenceDir); + if (!existsSync(resultPath)) return { status: "missing", path: resultPath }; + + try { + const result = JSON.parse(readFileSync(resultPath, "utf8")) as Record; + if (result.source === "final") { + return { + status: "not_provided", + path: resultPath, + reason: "only final result.json is present; automation-result.json was not found", + }; + } + return { + status: "loaded", + path: resultPath, + result: stringField(result, "status"), + reason: stringField(result, "reason"), + started_at: stringField(result, "started_at"), + started_at_local: stringField(result, "started_at_local"), + finished_at: stringField(result, "finished_at"), + finished_at_local: stringField(result, "finished_at_local"), + url: stringField(result, "url"), + prompt: redactSecrets(stringField(result, "prompt") ?? ""), + expected_text: stringField(result, "expected_text"), + }; + } catch (error) { + return { status: "invalid", path: resultPath, reason: String(error) }; + } +} + +export function latestLangBotLogPath(env: Record): string | null { + const repo = env.LANGBOT_REPO; + if (!repo) return null; + const logsDir = join(repo, "data", "logs"); + if (!existsSync(logsDir)) return null; + + const candidates = readdirSync(logsDir) + .filter((name) => /^langbot-.*\.log$/.test(name)) + .map((name) => join(logsDir, name)) + .filter((path) => { + try { + return statSync(path).isFile(); + } catch { + return false; + } + }) + .sort((a, b) => statSync(b).mtimeMs - statSync(a).mtimeMs); + + return candidates[0] ?? null; +} + +export function parseScanConfig(options: Record): LogScanConfig { + const warnings: string[] = []; + const sinceInput = typeof options.since === "string" ? options.since : undefined; + const sinceMs = sinceInput ? Date.parse(sinceInput) : undefined; + const untilInput = typeof options.until === "string" ? options.until : undefined; + const untilMs = untilInput ? Date.parse(untilInput) : undefined; + const tailInput = typeof options["tail-lines"] === "string" ? options["tail-lines"] : undefined; + let tailLines: number | undefined; + + if (sinceInput && Number.isNaN(sinceMs)) { + warnings.push(`--since is not a valid date/time: ${sinceInput}`); + } + if (untilInput && Number.isNaN(untilMs)) { + warnings.push(`--until is not a valid date/time: ${untilInput}`); + } + + if (tailInput) { + const parsed = Number.parseInt(tailInput, 10); + if (!/^\d+$/.test(tailInput) || parsed <= 0) { + warnings.push(`--tail-lines must be a positive integer: ${tailInput}`); + } else { + tailLines = parsed; + } + } + + const hasSince = sinceInput !== undefined && sinceMs !== undefined && !Number.isNaN(sinceMs); + const hasUntil = untilInput !== undefined && untilMs !== undefined && !Number.isNaN(untilMs); + const hasTail = tailLines !== undefined; + let mode: LogScanMode = "whole-file"; + if (hasSince && hasUntil && hasTail) { + mode = "since+until+tail-lines"; + } else if (hasSince && hasUntil) { + mode = "since+until"; + } else if (hasSince && hasTail) { + mode = "since+tail-lines"; + } else if (hasUntil && hasTail) { + mode = "until+tail-lines"; + } else if (hasSince) { + mode = "since"; + } else if (hasUntil) { + mode = "until"; + } else if (hasTail) { + mode = "tail-lines"; + } + + return { + mode, + since: hasSince ? sinceInput : undefined, + since_epoch_ms: hasSince ? sinceMs : undefined, + until: hasUntil ? untilInput : undefined, + until_epoch_ms: hasUntil ? untilMs : undefined, + tail_lines: tailLines, + warnings, + }; +} + +function selectLinesForScan(lines: LogLine[], scan: LogScanConfig): { lines: LogLine[]; timestampedLineCount: number } { + let selected = lines; + let timestampedLineCount = 0; + + if (scan.since_epoch_ms !== undefined || scan.until_epoch_ms !== undefined) { + const offsetMinutes = + timezoneOffsetMinutes(scan.since) + ?? timezoneOffsetMinutes(scan.until) + ?? -new Date(scan.since_epoch_ms ?? scan.until_epoch_ms ?? Date.now()).getTimezoneOffset(); + const yearHint = new Date(scan.since_epoch_ms ?? scan.until_epoch_ms ?? Date.now()).getUTCFullYear(); + let includeCurrentBlock = false; + const filtered = lines.filter((line) => { + const timestamp = parseLogLineTimestampMs(line.text, yearHint, offsetMinutes); + if (timestamp !== null) { + timestampedLineCount += 1; + includeCurrentBlock = + (scan.since_epoch_ms === undefined || timestamp >= scan.since_epoch_ms) + && (scan.until_epoch_ms === undefined || timestamp <= scan.until_epoch_ms); + return includeCurrentBlock; + } + return includeCurrentBlock; + }); + selected = timestampedLineCount === 0 ? lines : filtered; + } else { + timestampedLineCount = lines.reduce((count, line) => ( + parseLogLineTimestampMs(line.text, new Date().getFullYear(), -new Date().getTimezoneOffset()) === null + ? count + : count + 1 + ), 0); + } + + if (scan.tail_lines !== undefined && selected.length > scan.tail_lines) { + selected = selected.slice(-scan.tail_lines); + } + + return { lines: selected, timestampedLineCount }; +} + +function timezoneOffsetMinutes(input: string | undefined): number | null { + if (!input) return null; + if (/[zZ]$/.test(input)) return 0; + const match = input.match(/([+-])(\d{2}):?(\d{2})$/); + if (!match) return null; + const sign = match[1] === "-" ? -1 : 1; + return sign * (Number.parseInt(match[2], 10) * 60 + Number.parseInt(match[3], 10)); +} + +function parseLogLineTimestampMs(line: string, yearHint: number, offsetMinutes: number): number | null { + const fullIso = line.match(/^\[?(\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}:\d{2}(?:\.\d{1,3})?(?:Z|[+-]\d{2}:?\d{2})?)\]?/); + if (fullIso) { + const timestamp = Date.parse(fullIso[1].replace(" ", "T")); + return Number.isNaN(timestamp) ? null : timestamp; + } + + const langBot = line.match(/^\[(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})\.(\d{3})\]/); + if (!langBot) return null; + const [, month, day, hour, minute, second, millisecond] = langBot; + return Date.UTC( + yearHint, + Number.parseInt(month, 10) - 1, + Number.parseInt(day, 10), + Number.parseInt(hour, 10), + Number.parseInt(minute, 10), + Number.parseInt(second, 10), + Number.parseInt(millisecond, 10), + ) - offsetMinutes * 60 * 1000; +} + +export function renderLogFinding(finding: LogFinding): string { + const location = finding.line ? `${finding.source}:${finding.line}` : finding.source; + const trouble = finding.troubleshooting_id ? ` (${finding.troubleshooting_id})` : ""; + const related = finding.related_to_case === true ? ", related" : ""; + const excerpt = finding.excerpt ? ` - ${finding.excerpt}` : ""; + return `- [${finding.severity}] ${location}: ${finding.kind}${trouble}${related}; pattern: ${finding.pattern}${excerpt}`; +} + +export function renderLogSuccessSignal(signal: LogSuccessSignal): string { + const location = signal.line ? `${signal.source}:${signal.line}` : signal.source; + const excerpt = signal.excerpt ? ` - ${signal.excerpt}` : ""; + return `- ${location}: ${signal.pattern}${excerpt}`; +} + +export function strictLogGuardExitCode(result: LogGuardResult): number { + return result.status === "fail" || result.status === "env_issue" ? 1 : 0; +} diff --git a/skills/src/readiness.ts b/skills/src/readiness.ts new file mode 100644 index 000000000..d72a09104 --- /dev/null +++ b/skills/src/readiness.ts @@ -0,0 +1,251 @@ +import { env as processEnv } from "node:process"; +import type { StructuredItem } from "./types.ts"; +import { loadFixtureItems } from "./fixtures.ts"; +import { listValue, loadEnv, scalar } from "./fs.ts"; +import { splitEnvAnyGroup } from "./env-groups.ts"; + +type EnvSource = Record; + +export type EnvReadiness = { + status: "ready" | "missing" | "not_required"; + required: string[]; + configured: string[]; + missing: string[]; + values: Record; +}; + +export type AutomationReadiness = EnvReadiness & { + script: string; + defaulted: string[]; + pipeline_env_required: boolean; + env_aliases: Array<{ + target: string; + source: string; + configured: boolean; + }>; +}; + +export type ManualReadiness = { + status: "manual_check" | "not_required"; + preconditions: string[]; + setup: string[]; + cleanup: string[]; +}; + +export type FixtureReadiness = { + status: "ready" | "missing" | "not_required"; + required: Array<{ + id: string; + kind: string; + path: string; + exists: boolean; + }>; + missing: string[]; +}; + +const secretKeyRe = /(?:api[_-]?key|authorization|bearer|credential|jwt|oauth|password|secret|token)/i; + +export function redactEnvValue(key: string, value: string): string { + if (!value) return ""; + if (secretKeyRe.test(key)) return "[redacted]"; + return value.replace(/(https?:\/\/)([^:@/\s]+):([^@/\s]+)@/i, "$1[redacted]@"); +} + +export function runtimeEnv(root: string): Record { + const result: Record = { ...loadEnv(root) }; + for (const [key, value] of Object.entries(processEnv)) { + if (typeof value === "string") result[key] = value; + } + return result; +} + +function envReadiness( + keys: string[], + env: EnvSource, + defaults: Record = {}, + anyGroups: string[] = [], + providedBySetup: Set = new Set(), +): EnvReadiness { + const required = [...keys]; + const configured = required.filter((key) => Boolean(env[key]) || Boolean(defaults[key]) || providedBySetup.has(key)); + const missing = required.filter((key) => !env[key] && !defaults[key] && !providedBySetup.has(key)); + const values: Record = Object.fromEntries( + required.map((key) => [key, redactEnvValue(key, env[key] ?? defaults[key] ?? setupProvidedValue(key, providedBySetup))]), + ); + + for (const group of anyGroups) { + const keysInGroup = splitEnvAnyGroup(group); + required.push(group); + const configuredKeys = keysInGroup.filter((key) => Boolean(env[key]) || Boolean(defaults[key]) || providedBySetup.has(key)); + if (configuredKeys.length === 0) missing.push(group); + else configured.push(...configuredKeys); + for (const key of keysInGroup) { + values[key] = redactEnvValue(key, env[key] ?? defaults[key] ?? setupProvidedValue(key, providedBySetup)); + } + } + + return { + status: required.length === 0 ? "not_required" : missing.length === 0 ? "ready" : "missing", + required, + configured: Array.from(new Set(configured)), + missing, + values, + }; +} + +function setupProvidedValue(key: string, providedBySetup: Set): string { + return providedBySetup.has(key) ? "[provided by setup_automation]" : ""; +} + +export function setupProvidedEnv(item: StructuredItem): Set { + return new Set(listValue(item.fields, "setup_provides_env")); +} + +export function automationEnvDefaults(item: StructuredItem, env: EnvSource = processEnv): Record { + const mapping: Array<[string, string]> = [ + ["automation_prompt", "LANGBOT_E2E_PROMPT"], + ["automation_prompts_json", "LANGBOT_E2E_PROMPTS_JSON"], + ["automation_expected_text", "LANGBOT_E2E_EXPECTED_TEXT"], + ["automation_response_timeout_ms", "LANGBOT_E2E_RESPONSE_TIMEOUT_MS"], + ["automation_stream_output", "LANGBOT_E2E_STREAM_OUTPUT"], + ["automation_image_base64_fixture", "LANGBOT_E2E_IMAGE_BASE64_PATH"], + ["automation_runner_config_patch_json", "LANGBOT_E2E_RUNNER_CONFIG_PATCH_JSON"], + ["automation_restore_runner_config", "LANGBOT_E2E_RESTORE_RUNNER_CONFIG"], + ["automation_expected_runner_id", "LANGBOT_E2E_EXPECTED_RUNNER_ID"], + ["automation_reset_debug_chat", "LANGBOT_E2E_RESET_DEBUG_CHAT"], + ["automation_debug_chat_session_type", "LANGBOT_E2E_DEBUG_CHAT_SESSION_TYPE"], + ["automation_filesystem_checks_json", "LANGBOT_E2E_FILESYSTEM_CHECKS_JSON"], + ["automation_plugin_package", "LANGBOT_E2E_PLUGIN_PACKAGE"], + ["automation_expected_plugin_id", "LANGBOT_E2E_EXPECTED_PLUGIN_ID"], + ["automation_expected_tool", "LANGBOT_E2E_EXPECTED_TOOL"], + ]; + const defaults: Record = {}; + for (const [field, envKey] of mapping) { + const value = scalar(item.fields, field); + if (value) defaults[envKey] = expandEnvRefs(value, env); + } + const failurePatterns = listValue(item.fields, "failure_patterns"); + if (failurePatterns.length > 0) defaults.LANGBOT_E2E_FAILURE_SIGNALS = failurePatterns.join("\n"); + return defaults; +} + +function expandEnvRefs(value: string, env: EnvSource): string { + return value.replace(/\$\{([A-Z][A-Z0-9_]*)\}|\$([A-Z][A-Z0-9_]*)/g, (_match, braced, bare) => { + return env[braced || bare] || ""; + }); +} + +export function caseEnvReadiness(item: StructuredItem, env: EnvSource): EnvReadiness { + const aliasSources = new Set(automationEnvAliases(item, env).map((alias) => alias.source)); + const provided = setupProvidedEnv(item); + return envReadiness( + listValue(item.fields, "env").filter((key) => !aliasSources.has(key)), + env, + {}, + listValue(item.fields, "env_any"), + provided, + ); +} + +function automationEnvAliases(item: StructuredItem, env: EnvSource): Array<{ + target: string; + source: string; + configured: boolean; +}> { + const provided = setupProvidedEnv(item); + const mapping: Array<[string, string]> = [ + ["automation_pipeline_url_env", "LANGBOT_E2E_PIPELINE_URL"], + ["automation_pipeline_name_env", "LANGBOT_E2E_PIPELINE_NAME"], + ]; + return mapping + .map(([field, target]) => { + const source = scalar(item.fields, field); + return source ? { target, source, configured: Boolean(env[source]) || provided.has(source) } : null; + }) + .filter((item): item is { target: string; source: string; configured: boolean } => item !== null); +} + +export function automationPipelineEnvRequired(item: StructuredItem): boolean { + return Boolean(scalar(item.fields, "automation_pipeline_url_env") || scalar(item.fields, "automation_pipeline_name_env")); +} + +export function caseAutomationReadiness(item: StructuredItem, env: EnvSource): AutomationReadiness { + const script = scalar(item.fields, "automation"); + const aliases = automationEnvAliases(item, env); + const aliasSources = new Set(aliases.map((alias) => alias.source)); + const defaults = automationEnvDefaults(item, env); + const provided = setupProvidedEnv(item); + const requiredKeys = listValue(item.fields, "automation_env").filter((key) => !aliasSources.has(key)); + const readiness = envReadiness(requiredKeys, env, defaults, listValue(item.fields, "automation_env_any"), provided); + const defaulted = requiredKeys.filter((key) => !env[key] && Boolean(defaults[key])); + const aliasConfigured = aliases.some((alias) => alias.configured); + const aliasMissing = automationPipelineEnvRequired(item) && !aliasConfigured + ? [aliases.map((alias) => alias.source).join("|")] + : []; + const missing = [...readiness.missing, ...aliasMissing].filter(Boolean); + const configured = [ + ...readiness.configured, + ...aliases.filter((alias) => alias.configured).map((alias) => alias.source), + ]; + const values = { + ...readiness.values, + ...Object.fromEntries(aliases.map((alias) => [ + alias.source, + redactEnvValue(alias.source, env[alias.source] ?? setupProvidedValue(alias.source, provided)), + ])), + }; + return { + ...readiness, + status: script ? missing.length === 0 ? "ready" : "missing" : "not_required", + script, + defaulted, + required: [...readiness.required, ...aliases.map((alias) => alias.source)], + configured, + missing, + values, + pipeline_env_required: automationPipelineEnvRequired(item), + env_aliases: aliases, + }; +} + +export function resolvedAutomationEnvOverrides(item: StructuredItem, env: EnvSource): Record { + const overrides: Record = {}; + for (const alias of automationEnvAliases(item, env)) { + const value = env[alias.source]; + if (value) overrides[alias.target] = value; + } + for (const [key, value] of Object.entries(automationEnvDefaults(item, env))) { + overrides[key] = expandEnvRefs(value, env); + } + if (automationPipelineEnvRequired(item)) overrides.LANGBOT_E2E_PIPELINE_REQUIRED = "1"; + return overrides; +} + +export function caseManualReadiness(item: StructuredItem): ManualReadiness { + const preconditions = listValue(item.fields, "preconditions"); + const setup = listValue(item.fields, "setup"); + const cleanup = listValue(item.fields, "cleanup"); + return { + status: preconditions.length > 0 || setup.length > 0 ? "manual_check" : "not_required", + preconditions, + setup, + cleanup, + }; +} + +export function caseFixtureReadiness(root: string, caseId: string): FixtureReadiness { + const fixtures = loadFixtureItems(root).items + .filter((item) => item.related_cases.includes(caseId)) + .map((item) => ({ + id: item.id, + kind: item.kind, + path: item.path, + exists: item.exists, + })); + const missing = fixtures.filter((item) => !item.exists).map((item) => item.id); + return { + status: fixtures.length === 0 ? "not_required" : missing.length === 0 ? "ready" : "missing", + required: fixtures, + missing, + }; +} diff --git a/skills/src/setup-automation.ts b/skills/src/setup-automation.ts new file mode 100644 index 000000000..808c04a21 --- /dev/null +++ b/skills/src/setup-automation.ts @@ -0,0 +1,90 @@ +import { existsSync } from "node:fs"; +import { basename, dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { listValue } from "./fs.ts"; +import type { StructuredItem } from "./types.ts"; + +export type SetupAutomationSpec = { + entry: string; + kind: "case" | "node"; + target: string; + args: string[]; +}; + +export function setupAutomationEntries(item: StructuredItem): string[] { + return listValue(item.fields, "setup_automation"); +} + +export function parseSetupAutomationEntry(entry: string): SetupAutomationSpec { + const trimmed = entry.trim(); + if (trimmed.startsWith("case:")) { + return { + entry, + kind: "case", + target: trimmed.slice("case:".length).trim(), + args: [], + }; + } + if (trimmed.startsWith("node:")) { + const words = trimmed.slice("node:".length).trim().split(/\s+/).filter(Boolean); + return { + entry, + kind: "node", + target: words[0] ?? "", + args: words.slice(1), + }; + } + return { + entry, + kind: "case", + target: "", + args: [], + }; +} + +export function validateSetupAutomationEntry(root: string, entry: string, caseIds: Set): string[] { + const spec = parseSetupAutomationEntry(entry); + const errors: string[] = []; + if (!entry.startsWith("case:") && !entry.startsWith("node:")) { + return [`setup_automation entry must start with 'case:' or 'node:': ${entry}`]; + } + if (!spec.target) errors.push(`setup_automation entry is missing a target: ${entry}`); + if (spec.kind === "case") { + if (spec.args.length > 0) errors.push(`setup_automation case entries cannot include args: ${entry}`); + if (spec.target && !/^[a-z0-9][a-z0-9_-]*$/.test(spec.target)) { + errors.push(`setup_automation case target must be a case id: ${entry}`); + } else if (spec.target && !caseIds.has(spec.target)) { + errors.push(`setup_automation references unknown case '${spec.target}'`); + } + } + if (spec.kind === "node") { + if (spec.target.startsWith("/") || spec.target.includes("..") || !spec.target.startsWith("scripts/")) { + errors.push(`setup_automation node target must be a repository scripts/ path: ${entry}`); + } + if (spec.target && !/\.(mjs|js|ts)$/.test(spec.target)) { + errors.push(`setup_automation node target must be a Node script: ${entry}`); + } + if (spec.target && !existsSync(join(root, spec.target))) { + errors.push(`setup_automation node script does not exist: ${spec.target}`); + } + for (const arg of spec.args) { + if (!/^--[A-Za-z0-9][A-Za-z0-9_-]*(?:=[A-Za-z0-9_./:@-]+)?$/.test(arg)) { + errors.push(`setup_automation node arg must be a simple --flag or --key=value: ${entry}`); + } + } + } + return errors; +} + +export function setupAutomationEvidenceName(index: number, spec: SetupAutomationSpec): string { + const target = spec.kind === "case" ? spec.target : basename(spec.target).replace(/\.[^.]+$/, ""); + return `${String(index + 1).padStart(2, "0")}-${target.replace(/[^A-Za-z0-9_-]+/g, "-")}`; +} + +export function setupAutomationScriptPath(root: string, spec: SetupAutomationSpec): string { + return spec.kind === "node" && spec.target ? resolve(root, spec.target) : ""; +} + +export function lbsScriptPath(): string { + return resolve(dirname(fileURLToPath(import.meta.url)), "lbs.ts"); +} diff --git a/skills/src/types.ts b/skills/src/types.ts new file mode 100644 index 000000000..ae9264249 --- /dev/null +++ b/skills/src/types.ts @@ -0,0 +1,25 @@ +export type Skill = { + path: string; + directory: string; + name: string; + description: string; + body: string; +}; + +export type CommandContext = { + root: string; + args: string[]; +}; + +export type ParsedYamlValue = string | boolean | string[]; + +export type ParsedYaml = Record; + +export type StructuredItem = { + path: string; + skill: string; + fields: ParsedYaml; + raw: string; +}; + +export type StructuredItemKind = "cases" | "suites" | "troubleshooting"; diff --git a/skills/test/lbs-cli.test.ts b/skills/test/lbs-cli.test.ts new file mode 100644 index 000000000..73ac9d1ed --- /dev/null +++ b/skills/test/lbs-cli.test.ts @@ -0,0 +1,3162 @@ +import assert from "node:assert/strict"; +import { test } from "node:test"; +import { appendFileSync, existsSync, mkdtempSync, mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { spawnSync } from "node:child_process"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { CommandContext } from "../src/types.ts"; +import { commandCaseList, commandCaseNew, commandCaseShow } from "../src/commands/case.ts"; +import { commandEnvDoctor, commandEnvShow } from "../src/commands/env.ts"; +import { commandFixtureCheck, commandFixtureList } from "../src/commands/fixture.ts"; +import { commandLogGuard, commandLogScan, commandLogWatch } from "../src/commands/log.ts"; +import { commandSuiteList, commandSuiteNew, commandSuitePlan, commandSuiteReport, commandSuiteRun, commandSuiteShow, commandSuiteStart } from "../src/commands/suite.ts"; +import { commandTestPlan, commandTestRecommend, commandTestReport, commandTestResult, commandTestRun, commandTestStart } from "../src/commands/test.ts"; +import { commandTroubleSearch } from "../src/commands/trouble.ts"; +import { commandValidate } from "../src/commands/validate.ts"; +import { commandIndex } from "../src/commands/skill.ts"; +import { loadEnv } from "../src/fs.ts"; +import { + classifyDebugChatResult, + findNewFailureSignal, + minExpectedOccurrences, +} from "../scripts/e2e/lib/debug-chat.mjs"; + +const root = process.cwd(); + +function ctx(args: string[]): CommandContext { + return { root, args }; +} + +function capture(fn: () => number): { code: number; output: string } { + const originalLog = console.log; + const lines: string[] = []; + console.log = (...args: unknown[]) => { + lines.push(args.map(String).join(" ")); + }; + try { + const code = fn(); + return { code, output: lines.join("\n") }; + } finally { + console.log = originalLog; + } +} + +function captureAll(fn: () => number): { code: number; output: string; error: string } { + const originalLog = console.log; + const originalWrite = process.stderr.write; + const lines: string[] = []; + const errors: string[] = []; + console.log = (...args: unknown[]) => { + lines.push(args.map(String).join(" ")); + }; + process.stderr.write = ((chunk: string | Uint8Array) => { + errors.push(String(chunk)); + return true; + }) as typeof process.stderr.write; + try { + const code = fn(); + return { code, output: lines.join("\n"), error: errors.join("") }; + } finally { + console.log = originalLog; + process.stderr.write = originalWrite; + } +} + +function suiteResult(caseId: string, runId: string, status = "pass", evidence = ["ui", "screenshot", "console", "backend_log"]): string { + return JSON.stringify({ + source: "final", + case_id: caseId, + run_id: `${runId}-${caseId}`, + status, + reason: `${caseId} ${status}`, + started_at_local: "2026-05-21T10:30:00.000+08:00", + finished_at_local: "2026-05-21T10:31:00.000+08:00", + evidence_collected: evidence, + }); +} + +async function captureAsync(fn: () => Promise): Promise<{ code: number; output: string }> { + const originalLog = console.log; + const lines: string[] = []; + console.log = (...args: unknown[]) => { + lines.push(args.map(String).join(" ")); + }; + try { + const code = await fn(); + return { code, output: lines.join("\n") }; + } finally { + console.log = originalLog; + } +} + +test("validate accepts the repository assets", () => { + const result = capture(() => commandValidate(root)); + assert.equal(result.code, 0); + assert.match(result.output, /^OK/m); +}); + +test("validate allows blank shared env values but requires declared keys", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-validate-env-template-")); + try { + const schemasDir = join(tmp, "schemas"); + const skillsDir = join(tmp, "skills"); + const testingDir = join(skillsDir, "langbot-testing"); + mkdirSync(schemasDir, { recursive: true }); + mkdirSync(testingDir, { recursive: true }); + for (const schemaName of ["case.schema.json", "suite.schema.json", "troubleshooting.schema.json", "skill-index.schema.json"]) { + writeFileSync(join(schemasDir, schemaName), "{}"); + } + writeFileSync(join(testingDir, "SKILL.md"), "---\nname: langbot-testing\ndescription: Testing.\n---\n\n# Testing\n"); + const envText = [ + "LANGBOT_FRONTEND_URL=http://127.0.0.1:3000", + "LANGBOT_BACKEND_URL=http://127.0.0.1:5300", + "LANGBOT_DEV_FRONTEND_URL=http://127.0.0.1:3000", + "LANGBOT_REPO=", + "LANGBOT_WEB_REPO=", + "LANGBOT_BROWSER_PROFILE=", + "LANGBOT_CHROMIUM_EXECUTABLE=", + ].join("\n"); + writeFileSync(join(skillsDir, ".env"), envText); + writeFileSync(join(skillsDir, ".env.example"), envText); + + const result = capture(() => commandValidate(tmp)); + + assert.equal(result.code, 0); + assert.match(result.output, /^OK/m); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("index includes case summaries for agent discovery", () => { + const result = capture(() => commandIndex({ root, args: ["index"] })); + assert.equal(result.code, 0); + const index = JSON.parse(readFileSync(join(root, "skills.index.json"), "utf8")); + const testing = index.skills.find((skill: { name: string }) => skill.name === "langbot-testing"); + assert.ok(testing); + assert.ok(testing.case_summaries.some((item: { id: string; priority: string; evidence_required: string[] }) => ( + item.id === "pipeline-debug-chat" && item.priority === "p0" && item.evidence_required.includes("backend_log") + ))); + assert.ok(testing.case_summaries.some((item: { id: string; setup_automation: string[]; setup_provides_env: string[] }) => ( + item.id === "agent-runner-qa-debug-chat" && + item.setup_automation.includes("case:agent-runner-live-install") && + item.setup_provides_env.includes("LANGBOT_QA_AGENT_RUNNER_PIPELINE_URL") + ))); + assert.ok(testing.suite_summaries.some((item: { id: string; cases: string[] }) => ( + item.id === "core-smoke" && item.cases.includes("pipeline-debug-chat") + ))); + assert.ok(testing.fixtures.some((item: { id: string; related_cases: string[] }) => ( + item.id === "mcp-stdio-echo-server" && item.related_cases.includes("mcp-stdio-tool-call") + ))); +}); + +test("index check detects stale index without writing", () => { + const path = join(root, "skills.index.json"); + const current = capture(() => commandIndex({ root, args: ["index"] })); + assert.equal(current.code, 0); + + const fresh = readFileSync(path, "utf8"); + try { + const ok = capture(() => commandIndex({ root, args: ["index", "--check"] })); + assert.equal(ok.code, 0); + assert.match(ok.output, /^OK /); + + writeFileSync(path, "{}\n"); + const stale = captureAll(() => commandIndex({ root, args: ["index", "--check"] })); + assert.equal(stale.code, 1); + assert.match(stale.error, /index is stale/); + assert.equal(readFileSync(path, "utf8"), "{}\n"); + } finally { + writeFileSync(path, fresh); + } +}); + +test("case list exposes seeded QA cases", () => { + const result = capture(() => commandCaseList(ctx(["case", "list"]))); + assert.equal(result.code, 0); + assert.match(result.output, /pipeline-debug-chat/); + assert.match(result.output, /provider-deepseek/); + assert.match(result.output, /webui-login-state/); +}); + +test("case list JSON filters by reusable agent-selection metadata", () => { + const result = capture(() => commandCaseList(ctx([ + "case", + "list", + "--json", + "--priority", + "p0", + "--automation", + ]))); + assert.equal(result.code, 0); + const rows = JSON.parse(result.output); + assert.ok(rows.length >= 2); + assert.ok(rows.every((row: { priority: string }) => row.priority === "p0")); + assert.ok(rows.every((row: { automation: string }) => row.automation)); + assert.ok(rows.some((row: { id: string; evidence_required: string[]; readiness: string }) => ( + row.id === "pipeline-debug-chat" && row.evidence_required.includes("backend_log") && row.readiness + ))); +}); + +test("case list distinguishes machine readiness from manual precondition checks", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-case-manual-readiness-")); + try { + const skillDir = join(tmp, "skills", "langbot-testing"); + mkdirSync(join(skillDir, "cases"), { recursive: true }); + writeFileSync( + join(skillDir, "SKILL.md"), + "---\nname: langbot-testing\ndescription: Testing.\n---\n\n# Testing\n", + ); + writeFileSync(join(tmp, "skills", ".env"), "LANGBOT_FRONTEND_URL=http://127.0.0.1:3000\n"); + writeFileSync( + join(skillDir, "cases", "manual-case.yaml"), + [ + "id: manual-case", + "title: Manual Case", + "mode: agent-browser", + "area: pipeline", + "type: smoke", + "priority: p2", + "risk: medium", + "ci_eligible: false", + "tags:", + " - smoke", + "skills:", + " - langbot-testing", + "env:", + " - LANGBOT_FRONTEND_URL", + "preconditions:", + " - Confirm the target pipeline is safe to modify.", + "steps:", + " - Open the page.", + "checks:", + " - UI: Page opens.", + "evidence_required:", + " - ui", + ].join("\n"), + ); + + const machineReady = capture(() => commandCaseList({ root: tmp, args: ["case", "list", "--machine-ready"] })); + assert.equal(machineReady.code, 0); + assert.match(machineReady.output, /manual-case/); + assert.match(machineReady.output, /manual-check/); + + const ready = capture(() => commandCaseList({ root: tmp, args: ["case", "list", "--ready"] })); + assert.equal(ready.code, 0); + assert.doesNotMatch(ready.output, /manual-case/); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("case show prints structured agent-browser case", () => { + const result = capture(() => commandCaseShow(ctx(["case", "show", "pipeline-debug-chat"]))); + assert.equal(result.code, 0); + assert.match(result.output, /^id: pipeline-debug-chat/m); + assert.match(result.output, /^mode: agent-browser/m); + assert.match(result.output, /^checks:/m); +}); + +test("case new writes required selection metadata", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-case-new-")); + try { + const skillDir = join(tmp, "skills", "langbot-testing"); + mkdirSync(skillDir, { recursive: true }); + writeFileSync( + join(skillDir, "SKILL.md"), + "---\nname: langbot-testing\ndescription: Testing.\n---\n\n# Testing\n", + ); + + const result = capture(() => commandCaseNew({ + root: tmp, + args: ["case", "new", "new-case", "--title", "New Case"], + })); + + assert.equal(result.code, 0); + const text = readFileSync(join(skillDir, "cases", "new-case.yaml"), "utf8"); + assert.match(text, /^priority: p2/m); + assert.match(text, /^risk: medium/m); + assert.match(text, /^ci_eligible: false/m); + assert.match(text, /^evidence_required:/m); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("suite list and plan expose reusable case groups", () => { + const list = capture(() => commandSuiteList(ctx(["suite", "list", "--json", "--priority", "p0"]))); + assert.equal(list.code, 0); + const suites = JSON.parse(list.output); + assert.ok(suites.some((suite: { id: string; cases: string[] }) => ( + suite.id === "core-smoke" && suite.cases.includes("webui-login-state") + ))); + + const plan = capture(() => commandSuitePlan(ctx(["suite", "plan", "core-smoke", "--json"]))); + assert.equal(plan.code, 0); + const suitePlan = JSON.parse(plan.output); + assert.equal(suitePlan.id, "core-smoke"); + assert.ok(suitePlan.cases.some((item: { id: string; evidence_required: string[] }) => ( + item.id === "pipeline-debug-chat" && item.evidence_required.includes("backend_log") + ))); + assert.ok(suitePlan.commands.some((item: { id: string; automation: string }) => ( + item.id === "pipeline-debug-chat" && item.automation.includes("test run") + ))); + + const localAgent = capture(() => commandSuitePlan(ctx(["suite", "plan", "local-agent-gate", "--json"]))); + assert.equal(localAgent.code, 0); + const localAgentPlan = JSON.parse(localAgent.output); + assert.ok(["ready", "missing", "manual_check"].includes(localAgentPlan.readiness.status)); + const basic = localAgentPlan.cases.find((item: { id: string }) => item.id === "local-agent-basic-debug-chat"); + assert.equal(basic.automation_readiness.pipeline_env_required, true); +}); + +test("suite show prints structured suite YAML", () => { + const result = capture(() => commandSuiteShow(ctx(["suite", "show", "local-agent-gate"]))); + assert.equal(result.code, 0); + assert.match(result.output, /^id: local-agent-gate/m); + assert.match(result.output, /^cases:/m); + assert.match(result.output, /local-agent-effective-prompt-debug-chat/); +}); + +test("suite start creates a run handoff with per-case evidence commands", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-suite-start-")); + try { + const evidenceRoot = join(tmp, "evidence"); + const result = capture(() => commandSuiteStart(ctx([ + "suite", + "start", + "core-smoke", + "--run-id", + "core-smoke-local", + "--evidence-dir", + evidenceRoot, + "--json", + ]))); + assert.equal(result.code, 0); + const start = JSON.parse(result.output); + assert.equal(start.suite.id, "core-smoke"); + assert.equal(start.run_id, "core-smoke-local"); + assert.equal(start.evidence_root, evidenceRoot); + assert.equal(start.manifest_path, join(evidenceRoot, "suite-start.json")); + assert.equal(start.handoff_path, join(evidenceRoot, "suite-start.md")); + assert.match(start.report_command, /bin\/lbs suite report core-smoke/); + assert.ok(existsSync(join(evidenceRoot, "suite-start.json"))); + assert.ok(existsSync(join(evidenceRoot, "suite-start.md"))); + const pipeline = start.cases.find((item: { id: string }) => item.id === "pipeline-debug-chat"); + assert.ok(pipeline); + assert.ok(existsSync(join(evidenceRoot, "pipeline-debug-chat"))); + assert.match(pipeline.automation_command, /bin\/lbs test run pipeline-debug-chat/); + assert.match(pipeline.report_command, /--evidence-dir .+pipeline-debug-chat/); + assert.match(pipeline.result_command_template, /bin\/lbs test result pipeline-debug-chat/); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("suite report aggregates case result JSON files", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-suite-report-")); + try { + const evidenceRoot = join(tmp, "suite-evidence"); + const runId = "suite-report-run"; + for (const [caseId, status] of [ + ["webui-login-state", "pass"], + ["pipeline-debug-chat", "pass"], + ["local-agent-basic-debug-chat", "env_issue"], + ]) { + const dir = join(evidenceRoot, caseId); + mkdirSync(dir, { recursive: true }); + writeFileSync( + join(dir, "result.json"), + suiteResult(caseId, runId, status), + ); + } + + const result = capture(() => commandSuiteReport(ctx([ + "suite", + "report", + "core-smoke", + "--run-id", + runId, + "--evidence-dir", + evidenceRoot, + "--json", + ]))); + + assert.equal(result.code, 0); + const report = JSON.parse(result.output); + assert.equal(report.status, "env_issue"); + assert.equal(report.counts.pass, 2); + assert.equal(report.counts.env_issue, 1); + assert.ok(report.cases.some((item: { id: string; result: { status: string } }) => ( + item.id === "local-agent-basic-debug-chat" && item.result.status === "env_issue" + ))); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("suite report treats pass without required evidence as incomplete", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-suite-report-evidence-")); + try { + const evidenceRoot = join(tmp, "suite-evidence"); + const runId = "suite-report-evidence"; + for (const caseId of ["webui-login-state", "pipeline-debug-chat", "local-agent-basic-debug-chat"]) { + const dir = join(evidenceRoot, caseId); + mkdirSync(dir, { recursive: true }); + writeFileSync( + join(dir, "result.json"), + suiteResult(caseId, runId, "pass", ["ui"]), + ); + } + + const result = capture(() => commandSuiteReport(ctx([ + "suite", + "report", + "core-smoke", + "--run-id", + runId, + "--evidence-dir", + evidenceRoot, + "--json", + ]))); + + assert.equal(result.code, 0); + const report = JSON.parse(result.output); + assert.equal(report.status, "incomplete"); + assert.ok(report.cases.some((item: { id: string; result: { evidence_missing: string[] } }) => ( + item.id === "pipeline-debug-chat" && item.result.evidence_missing.includes("backend_log") + ))); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("suite report marks missing case evidence as incomplete", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-suite-report-missing-")); + try { + const evidenceRoot = join(tmp, "suite-evidence"); + const runId = "suite-report-missing"; + mkdirSync(join(evidenceRoot, "webui-login-state"), { recursive: true }); + writeFileSync(join(evidenceRoot, "webui-login-state", "result.json"), suiteResult("webui-login-state", runId, "pass")); + + const result = capture(() => commandSuiteReport(ctx([ + "suite", + "report", + "core-smoke", + "--run-id", + runId, + "--evidence-dir", + evidenceRoot, + "--json", + ]))); + + assert.equal(result.code, 0); + const report = JSON.parse(result.output); + assert.equal(report.status, "incomplete"); + assert.ok(report.cases.some((item: { id: string; result: { status: string } }) => ( + item.id === "pipeline-debug-chat" && item.result.status === "missing" + ))); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("suite report rejects result files from the wrong case or run", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-suite-report-mismatch-")); + try { + const evidenceRoot = join(tmp, "suite-evidence"); + const runId = "suite-report-mismatch"; + for (const caseId of ["webui-login-state", "pipeline-debug-chat", "local-agent-basic-debug-chat"]) { + const dir = join(evidenceRoot, caseId); + mkdirSync(dir, { recursive: true }); + writeFileSync(join(dir, "result.json"), suiteResult(caseId, runId, "pass")); + } + writeFileSync(join(evidenceRoot, "pipeline-debug-chat", "result.json"), suiteResult("webui-login-state", runId, "pass")); + writeFileSync(join(evidenceRoot, "local-agent-basic-debug-chat", "result.json"), suiteResult("local-agent-basic-debug-chat", "old-run", "pass")); + + const result = capture(() => commandSuiteReport(ctx([ + "suite", + "report", + "core-smoke", + "--run-id", + runId, + "--evidence-dir", + evidenceRoot, + "--json", + ]))); + + assert.equal(result.code, 0); + const report = JSON.parse(result.output); + assert.equal(report.status, "fail"); + assert.ok(report.cases.some((item: { id: string; result: { status: string; reason: string } }) => ( + item.id === "pipeline-debug-chat" && item.result.status === "invalid" && item.result.reason.includes("case_id mismatch") + ))); + assert.ok(report.cases.some((item: { id: string; result: { status: string; reason: string } }) => ( + item.id === "local-agent-basic-debug-chat" && item.result.status === "invalid" && item.result.reason.includes("run_id mismatch") + ))); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("suite run executes automated cases and aggregates a verdict", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-suite-run-")); + try { + const skillDir = join(tmp, "skills", "langbot-testing"); + const casesDir = join(skillDir, "cases"); + const suitesDir = join(skillDir, "suites"); + const scriptsDir = join(tmp, "scripts"); + mkdirSync(casesDir, { recursive: true }); + mkdirSync(suitesDir, { recursive: true }); + mkdirSync(scriptsDir, { recursive: true }); + writeFileSync(join(skillDir, "SKILL.md"), "---\nname: langbot-testing\ndescription: Testing.\n---\n\n# Testing\n"); + writeFileSync(join(tmp, "skills", ".env"), ""); + writeFileSync( + join(casesDir, "one.yaml"), + [ + "id: one", + "title: One", + "mode: probe", + "area: qa", + "type: smoke", + "priority: p2", + "risk: low", + "ci_eligible: true", + "automation: scripts/pass.mjs", + "evidence_required:", + " - filesystem", + ].join("\n"), + ); + writeFileSync( + join(casesDir, "two.yaml"), + [ + "id: two", + "title: Two", + "mode: probe", + "area: qa", + "type: smoke", + "priority: p2", + "risk: low", + "ci_eligible: true", + "automation: scripts/pass.mjs", + "evidence_required:", + " - filesystem", + ].join("\n"), + ); + writeFileSync( + join(suitesDir, "mini.yaml"), + [ + "id: mini", + "title: Mini", + "description: Mini suite.", + "type: smoke", + "priority: p2", + "tags:", + " - qa", + "cases:", + " - one", + " - two", + ].join("\n"), + ); + writeFileSync( + join(scriptsDir, "pass.mjs"), + [ + "import { mkdirSync, writeFileSync } from 'node:fs';", + "import { join } from 'node:path';", + "mkdirSync(process.env.LBS_EVIDENCE_DIR, { recursive: true });", + "writeFileSync(join(process.env.LBS_EVIDENCE_DIR, 'result.json'), JSON.stringify({", + " case_id: process.env.LBS_CASE_ID,", + " run_id: process.env.LBS_RUN_ID,", + " status: 'pass',", + " reason: `${process.env.LBS_CASE_ID} pass`,", + " evidence_collected: ['filesystem']", + "}));", + "writeFileSync(join(process.env.LBS_EVIDENCE_DIR, 'automation-result.json'), JSON.stringify({ status: 'pass' }));", + ].join("\n"), + ); + + const result = capture(() => commandSuiteRun({ + root: tmp, + args: ["suite", "run", "mini", "--run-id", "mini-run", "--evidence-dir", join(tmp, "evidence"), "--json"], + })); + + assert.equal(result.code, 0); + const payload = JSON.parse(result.output); + assert.equal(payload.report.status, "pass"); + assert.equal(payload.report.counts.pass, 2); + assert.deepEqual(payload.executions.map((item: { status: string }) => item.status), ["ok", "ok"]); + assert.ok(existsSync(join(tmp, "evidence", "one", "result.json"))); + assert.ok(existsSync(join(tmp, "evidence", "two", "result.json"))); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("suite run JSON captures failed case output", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-suite-run-fail-")); + try { + const skillDir = join(tmp, "skills", "langbot-testing"); + const casesDir = join(skillDir, "cases"); + const suitesDir = join(skillDir, "suites"); + const scriptsDir = join(tmp, "scripts"); + mkdirSync(casesDir, { recursive: true }); + mkdirSync(suitesDir, { recursive: true }); + mkdirSync(scriptsDir, { recursive: true }); + writeFileSync(join(skillDir, "SKILL.md"), "---\nname: langbot-testing\ndescription: Testing.\n---\n\n# Testing\n"); + writeFileSync(join(tmp, "skills", ".env"), ""); + writeFileSync( + join(casesDir, "fail-case.yaml"), + [ + "id: fail-case", + "title: Fail Case", + "mode: probe", + "area: qa", + "type: smoke", + "priority: p2", + "risk: low", + "ci_eligible: true", + "automation: scripts/fail.mjs", + ].join("\n"), + ); + writeFileSync( + join(suitesDir, "mini.yaml"), + [ + "id: mini", + "title: Mini", + "description: Mini suite.", + "type: smoke", + "priority: p2", + "tags:", + " - qa", + "cases:", + " - fail-case", + ].join("\n"), + ); + writeFileSync(join(scriptsDir, "fail.mjs"), "console.error('child failure detail'); process.exit(1);\n"); + + const result = capture(() => commandSuiteRun({ + root: tmp, + args: ["suite", "run", "mini", "--run-id", "mini-run", "--evidence-dir", join(tmp, "evidence"), "--json"], + })); + + assert.equal(result.code, 1); + const payload = JSON.parse(result.output); + assert.equal(payload.executions[0].status, "nonzero"); + assert.match(payload.executions[0].stderr, /child failure detail/); + assert.equal(payload.report.status, "fail"); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("suite run failure cannot be masked by stale pass result", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-suite-run-stale-pass-")); + try { + const skillDir = join(tmp, "skills", "langbot-testing"); + const casesDir = join(skillDir, "cases"); + const suitesDir = join(skillDir, "suites"); + const scriptsDir = join(tmp, "scripts"); + const evidenceDir = join(tmp, "evidence"); + mkdirSync(casesDir, { recursive: true }); + mkdirSync(suitesDir, { recursive: true }); + mkdirSync(scriptsDir, { recursive: true }); + mkdirSync(join(evidenceDir, "fail-case"), { recursive: true }); + writeFileSync(join(skillDir, "SKILL.md"), "---\nname: langbot-testing\ndescription: Testing.\n---\n\n# Testing\n"); + writeFileSync(join(tmp, "skills", ".env"), ""); + writeFileSync( + join(casesDir, "fail-case.yaml"), + [ + "id: fail-case", + "title: Fail Case", + "mode: probe", + "area: qa", + "type: smoke", + "priority: p2", + "risk: low", + "ci_eligible: true", + "automation: scripts/fail.mjs", + "evidence_required:", + " - filesystem", + ].join("\n"), + ); + writeFileSync( + join(suitesDir, "mini.yaml"), + [ + "id: mini", + "title: Mini", + "description: Mini suite.", + "type: smoke", + "priority: p2", + "tags:", + " - qa", + "cases:", + " - fail-case", + ].join("\n"), + ); + writeFileSync(join(scriptsDir, "fail.mjs"), "process.exit(1);\n"); + writeFileSync(join(evidenceDir, "fail-case", "result.json"), JSON.stringify({ + case_id: "fail-case", + run_id: "stale-run-fail-case", + status: "pass", + evidence_collected: ["filesystem"], + })); + + const result = capture(() => commandSuiteRun({ + root: tmp, + args: ["suite", "run", "mini", "--run-id", "stale-run", "--evidence-dir", evidenceDir, "--json"], + })); + + assert.equal(result.code, 1); + const payload = JSON.parse(result.output); + assert.equal(payload.executions[0].status, "nonzero"); + assert.equal(payload.report.status, "fail"); + assert.equal(payload.report.execution_status, "fail"); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("suite run dry-run plans automation without creating evidence", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-suite-run-dry-")); + try { + const skillDir = join(tmp, "skills", "langbot-testing"); + const casesDir = join(skillDir, "cases"); + const suitesDir = join(skillDir, "suites"); + const scriptsDir = join(tmp, "scripts"); + mkdirSync(casesDir, { recursive: true }); + mkdirSync(suitesDir, { recursive: true }); + mkdirSync(scriptsDir, { recursive: true }); + writeFileSync(join(skillDir, "SKILL.md"), "---\nname: langbot-testing\ndescription: Testing.\n---\n\n# Testing\n"); + writeFileSync(join(tmp, "skills", ".env"), ""); + writeFileSync( + join(casesDir, "dry-case.yaml"), + [ + "id: dry-case", + "title: Dry Case", + "mode: probe", + "area: qa", + "type: smoke", + "priority: p2", + "risk: low", + "ci_eligible: true", + "automation: scripts/fail-if-run.mjs", + ].join("\n"), + ); + writeFileSync( + join(suitesDir, "dry-suite.yaml"), + [ + "id: dry-suite", + "title: Dry Suite", + "description: Dry run suite.", + "type: smoke", + "priority: p2", + "tags:", + " - qa", + "cases:", + " - dry-case", + ].join("\n"), + ); + writeFileSync(join(scriptsDir, "fail-if-run.mjs"), "process.exit(9);\n"); + + const evidenceDir = join(tmp, "evidence"); + const result = capture(() => commandSuiteRun({ + root: tmp, + args: ["suite", "run", "dry-suite", "--run-id", "dry-run", "--evidence-dir", evidenceDir, "--dry-run", "--json"], + })); + + assert.equal(result.code, 0); + const payload = JSON.parse(result.output); + assert.equal(payload.executions[0].status, "planned"); + assert.match(payload.executions[0].command, /test run dry-case/); + assert.equal(payload.report.status, "incomplete"); + assert.equal(existsSync(evidenceDir), false); + assert.equal(existsSync(join(tmp, "reports", "dry-run.md")), false); + + const markdown = capture(() => commandSuiteRun({ + root: tmp, + args: ["suite", "run", "dry-suite", "--run-id", "dry-run-markdown", "--evidence-dir", join(tmp, "evidence-md"), "--dry-run"], + })); + assert.equal(markdown.code, 0); + assert.match(markdown.output, /# Suite Report: dry-suite/); + assert.equal(existsSync(join(tmp, "reports", "dry-run-markdown.md")), false); + assert.equal(existsSync(join(tmp, "evidence-md")), false); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("suite run skips manual-check cases unless explicitly included", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-suite-run-manual-")); + try { + const skillDir = join(tmp, "skills", "langbot-testing"); + const casesDir = join(skillDir, "cases"); + const suitesDir = join(skillDir, "suites"); + const scriptsDir = join(tmp, "scripts"); + mkdirSync(casesDir, { recursive: true }); + mkdirSync(suitesDir, { recursive: true }); + mkdirSync(scriptsDir, { recursive: true }); + writeFileSync(join(skillDir, "SKILL.md"), "---\nname: langbot-testing\ndescription: Testing.\n---\n\n# Testing\n"); + writeFileSync(join(tmp, "skills", ".env"), ""); + writeFileSync( + join(casesDir, "manual-case.yaml"), + [ + "id: manual-case", + "title: Manual Case", + "mode: probe", + "area: qa", + "type: smoke", + "priority: p2", + "risk: low", + "ci_eligible: true", + "preconditions:", + " - Confirm this case is safe to run.", + "automation: scripts/pass.mjs", + "evidence_required:", + " - filesystem", + ].join("\n"), + ); + writeFileSync( + join(suitesDir, "manual-suite.yaml"), + [ + "id: manual-suite", + "title: Manual Suite", + "description: Manual check suite.", + "type: smoke", + "priority: p2", + "tags:", + " - qa", + "cases:", + " - manual-case", + ].join("\n"), + ); + writeFileSync( + join(scriptsDir, "pass.mjs"), + [ + "import { mkdirSync, writeFileSync } from 'node:fs';", + "import { join } from 'node:path';", + "mkdirSync(process.env.LBS_EVIDENCE_DIR, { recursive: true });", + "writeFileSync(join(process.env.LBS_EVIDENCE_DIR, 'result.json'), JSON.stringify({ case_id: process.env.LBS_CASE_ID, run_id: process.env.LBS_RUN_ID, status: 'pass', evidence_collected: ['filesystem'] }));", + ].join("\n"), + ); + + const skipped = capture(() => commandSuiteRun({ + root: tmp, + args: ["suite", "run", "manual-suite", "--run-id", "manual-run", "--evidence-dir", join(tmp, "evidence"), "--json"], + })); + assert.equal(skipped.code, 1); + const skippedPayload = JSON.parse(skipped.output); + assert.equal(skippedPayload.executions[0].status, "skipped"); + assert.match(skippedPayload.executions[0].reason, /manual_check/); + assert.equal(existsSync(join(tmp, "evidence", "manual-case", "result.json")), false); + + const included = capture(() => commandSuiteRun({ + root: tmp, + args: ["suite", "run", "manual-suite", "--run-id", "manual-run-included", "--evidence-dir", join(tmp, "evidence-included"), "--include-manual-check", "--json"], + })); + assert.equal(included.code, 0); + const includedPayload = JSON.parse(included.output); + assert.equal(includedPayload.executions[0].status, "ok"); + assert.equal(includedPayload.report.status, "pass"); + assert.ok(existsSync(join(tmp, "evidence-included", "manual-case", "result.json"))); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("suite run skips cases with missing machine readiness unless explicitly included", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-suite-run-readiness-")); + try { + const skillDir = join(tmp, "skills", "langbot-testing"); + const casesDir = join(skillDir, "cases"); + const suitesDir = join(skillDir, "suites"); + const fixturesDir = join(skillDir, "fixtures"); + const scriptsDir = join(tmp, "scripts"); + mkdirSync(casesDir, { recursive: true }); + mkdirSync(suitesDir, { recursive: true }); + mkdirSync(fixturesDir, { recursive: true }); + mkdirSync(scriptsDir, { recursive: true }); + writeFileSync(join(skillDir, "SKILL.md"), "---\nname: langbot-testing\ndescription: Testing.\n---\n\n# Testing\n"); + writeFileSync(join(tmp, "skills", ".env"), ""); + writeFileSync( + join(casesDir, "not-ready-case.yaml"), + [ + "id: not-ready-case", + "title: Not Ready Case", + "mode: probe", + "area: qa", + "type: smoke", + "priority: p2", + "risk: low", + "ci_eligible: true", + "env:", + " - LBS_TEST_SUITE_RUN_MISSING_ENV", + "automation_env:", + " - LBS_TEST_SUITE_RUN_MISSING_AUTOMATION_ENV", + "automation: scripts/pass.mjs", + "evidence_required:", + " - filesystem", + ].join("\n"), + ); + writeFileSync( + join(fixturesDir, "fixtures.json"), + `${JSON.stringify([{ + id: "missing-fixture", + title: "Missing fixture", + kind: "file", + path: "fixtures/missing.txt", + related_cases: ["not-ready-case"], + checks: ["exists"], + }], null, 2)}\n`, + ); + writeFileSync( + join(suitesDir, "readiness-suite.yaml"), + [ + "id: readiness-suite", + "title: Readiness Suite", + "description: Readiness suite.", + "type: smoke", + "priority: p2", + "tags:", + " - qa", + "cases:", + " - not-ready-case", + ].join("\n"), + ); + writeFileSync( + join(scriptsDir, "pass.mjs"), + [ + "import { mkdirSync, writeFileSync } from 'node:fs';", + "import { join } from 'node:path';", + "mkdirSync(process.env.LBS_EVIDENCE_DIR, { recursive: true });", + "writeFileSync(join(process.env.LBS_EVIDENCE_DIR, 'result.json'), JSON.stringify({ case_id: process.env.LBS_CASE_ID, run_id: process.env.LBS_RUN_ID, status: 'pass', evidence_collected: ['filesystem'] }));", + ].join("\n"), + ); + + const skipped = capture(() => commandSuiteRun({ + root: tmp, + args: ["suite", "run", "readiness-suite", "--run-id", "readiness-run", "--evidence-dir", join(tmp, "evidence"), "--json"], + })); + assert.equal(skipped.code, 1); + const skippedPayload = JSON.parse(skipped.output); + assert.equal(skippedPayload.executions[0].status, "skipped"); + assert.match(skippedPayload.executions[0].reason, /readiness missing/); + assert.match(skippedPayload.executions[0].reason, /LBS_TEST_SUITE_RUN_MISSING_ENV/); + assert.match(skippedPayload.executions[0].reason, /LBS_TEST_SUITE_RUN_MISSING_AUTOMATION_ENV/); + assert.match(skippedPayload.executions[0].reason, /missing-fixture/); + assert.equal(existsSync(join(tmp, "evidence", "not-ready-case", "result.json")), false); + + const included = capture(() => commandSuiteRun({ + root: tmp, + args: ["suite", "run", "readiness-suite", "--run-id", "readiness-run-included", "--evidence-dir", join(tmp, "evidence-included"), "--include-not-ready", "--json"], + })); + assert.equal(included.code, 0); + const includedPayload = JSON.parse(included.output); + assert.equal(includedPayload.executions[0].status, "ok"); + assert.equal(includedPayload.report.status, "pass"); + assert.ok(existsSync(join(tmp, "evidence-included", "not-ready-case", "result.json"))); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("suite new writes a reusable suite skeleton", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-suite-new-")); + try { + const skillDir = join(tmp, "skills", "langbot-testing"); + mkdirSync(skillDir, { recursive: true }); + writeFileSync( + join(skillDir, "SKILL.md"), + "---\nname: langbot-testing\ndescription: Testing.\n---\n\n# Testing\n", + ); + + const result = capture(() => commandSuiteNew({ + root: tmp, + args: ["suite", "new", "new-suite", "--title", "New Suite"], + })); + + assert.equal(result.code, 0); + const text = readFileSync(join(skillDir, "suites", "new-suite.yaml"), "utf8"); + assert.match(text, /^description:/m); + assert.match(text, /^priority: p2/m); + assert.match(text, /^cases:/m); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("fixture list and check expose reusable fixture readiness", () => { + const list = capture(() => commandFixtureList(ctx(["fixture", "list", "langbot-testing", "--json"]))); + assert.equal(list.code, 0); + const fixtures = JSON.parse(list.output); + assert.ok(fixtures.some((item: { id: string; exists: boolean }) => ( + item.id === "mcp-stdio-echo-server" && item.exists === true + ))); + + const check = capture(() => commandFixtureCheck(ctx(["fixture", "check", "langbot-testing", "--json"]))); + assert.equal(check.code, 0); + const report = JSON.parse(check.output); + assert.equal(report.status, "pass"); + assert.ok(report.fixtures.some((item: { id: string }) => item.id === "qa-plugin-smoke-package")); +}); + +test("fixture check reports missing manifest paths", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-fixture-check-")); + try { + const skillDir = join(tmp, "skills", "langbot-testing"); + mkdirSync(join(skillDir, "fixtures"), { recursive: true }); + writeFileSync( + join(skillDir, "SKILL.md"), + "---\nname: langbot-testing\ndescription: Testing.\n---\n\n# Testing\n", + ); + writeFileSync( + join(skillDir, "fixtures", "fixtures.json"), + JSON.stringify([{ id: "missing-fixture", title: "Missing Fixture", path: "fixtures/missing.txt" }]), + ); + + const result = capture(() => commandFixtureCheck({ root: tmp, args: ["fixture", "check", "langbot-testing", "--json"] })); + + assert.equal(result.code, 1); + const report = JSON.parse(result.output); + assert.equal(report.status, "fail"); + assert.ok(report.findings.some((finding: { id?: string }) => finding.id === "missing-fixture")); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("fixture check verifies QA AgentRunner source shape", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-fixture-check-")); + try { + const skillDir = join(tmp, "skills", "langbot-testing"); + const fixtureDir = join(skillDir, "fixtures", "plugins", "qa-agent-runner"); + mkdirSync(join(fixtureDir, "components", "agent_runner"), { recursive: true }); + writeFileSync( + join(skillDir, "SKILL.md"), + "---\nname: langbot-testing\ndescription: Testing.\n---\n\n# Testing\n", + ); + writeFileSync( + join(skillDir, "fixtures", "fixtures.json"), + JSON.stringify([{ + id: "qa-agent-runner-source", + title: "QA AgentRunner", + path: "fixtures/plugins/qa-agent-runner/manifest.yaml", + checks: ["exists", "qa_agent_runner_source"], + }]), + ); + writeFileSync(join(fixtureDir, "manifest.yaml"), "spec:\n components:\n AgentRunner: {}\nexecution:\n python:\n attr: QAAgentRunnerPlugin\n"); + + const result = capture(() => commandFixtureCheck({ root: tmp, args: ["fixture", "check", "langbot-testing", "--json"] })); + + assert.equal(result.code, 1); + const report = JSON.parse(result.output); + assert.ok(report.findings.some((finding: { kind?: string; path?: string }) => ( + finding.kind === "fixture_check_missing_file" + && finding.path?.endsWith("components/agent_runner/default.py") + ))); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("fixture check accepts complete QA AgentRunner source shape", () => { + const result = capture(() => commandFixtureCheck(ctx(["fixture", "check", "langbot-testing", "--json"]))); + assert.equal(result.code, 0); + const report = JSON.parse(result.output); + assert.ok(report.fixtures.some((item: { id: string; checks: string[] }) => ( + item.id === "qa-agent-runner-source" && item.checks.includes("qa_agent_runner_source") + ))); +}); + +test("fixture check rejects invalid plugin package files", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-fixture-check-")); + try { + const skillDir = join(tmp, "skills", "langbot-testing"); + mkdirSync(join(skillDir, "fixtures"), { recursive: true }); + writeFileSync( + join(skillDir, "SKILL.md"), + "---\nname: langbot-testing\ndescription: Testing.\n---\n\n# Testing\n", + ); + writeFileSync(join(skillDir, "fixtures", "bad.lbpkg"), "not a zip"); + writeFileSync( + join(skillDir, "fixtures", "fixtures.json"), + JSON.stringify([{ + id: "bad-package", + title: "Bad Package", + path: "fixtures/bad.lbpkg", + checks: ["exists", "zip_package"], + }]), + ); + + const result = capture(() => commandFixtureCheck({ root: tmp, args: ["fixture", "check", "langbot-testing", "--json"] })); + + assert.equal(result.code, 1); + const report = JSON.parse(result.output); + assert.ok(report.findings.some((finding: { kind?: string; id?: string }) => ( + finding.kind === "fixture_check_invalid_zip" && finding.id === "bad-package" + ))); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("debug chat classifier prefers latest response leaf over body counts", () => { + const result = classifyDebugChatResult({ + beforeText: "OK from an older chat", + afterText: "OK from an older chat\nUser: say OK\nBot: OK", + expectedText: "OK", + prompt: "say OK", + latestExpectedLeaf: "Bot: OK", + latestFailureLeaf: "", + }); + + assert.equal(result.status, "pass"); + assert.match(result.reason, /latest visible response leaf/); +}); + +test("debug chat classifier distinguishes new failure signals from old history", () => { + assert.equal( + findNewFailureSignal("Agent runner temporarily unavailable", "Agent runner temporarily unavailable"), + "", + ); + assert.equal( + findNewFailureSignal("", "Agent runner temporarily unavailable"), + "Agent runner temporarily unavailable", + ); + + const result = classifyDebugChatResult({ + beforeText: "", + afterText: "Agent runner temporarily unavailable", + expectedText: "OK", + prompt: "say OK", + latestExpectedLeaf: "", + latestFailureLeaf: "Agent runner temporarily unavailable", + }); + + assert.equal(result.status, "fail"); + assert.match(result.reason, /known failure signal/); + + const custom = classifyDebugChatResult({ + beforeText: "", + afterText: "Bot: qa-plugin-smoke:mcp-ok-local-agent", + expectedText: "qa_mcp_echo:mcp-ok-local-agent", + prompt: "call mcp", + latestExpectedLeaf: "", + latestFailureLeaf: "Bot: qa-plugin-smoke:mcp-ok-local-agent", + failureSignals: ["qa-plugin-smoke:mcp-ok-local-agent"], + }); + + assert.equal(custom.status, "fail"); + assert.equal(custom.failure_signal, "qa-plugin-smoke:mcp-ok-local-agent"); +}); + +test("debug chat classifier lets new failure signals override stale expected history", () => { + const result = classifyDebugChatResult({ + beforeText: "Bot: qa_mcp_echo:mcp-ok-local-agent", + afterText: [ + "Bot: qa_mcp_echo:mcp-ok-local-agent", + "User: Call qa_mcp_echo", + "Bot: Agent runner temporarily unavailable.", + ].join("\n"), + expectedText: "qa_mcp_echo:mcp-ok-local-agent", + prompt: "Call qa_mcp_echo", + latestExpectedLeaf: "Bot: qa_mcp_echo:mcp-ok-local-agent", + latestFailureLeaf: "Bot: Agent runner temporarily unavailable.", + }); + + assert.equal(result.status, "fail"); + assert.equal(result.failure_signal, "Agent runner temporarily unavailable"); +}); + +test("debug chat classifier does not pass on stale expected history without a new occurrence", () => { + const result = classifyDebugChatResult({ + beforeText: "Bot: qa_mcp_echo:mcp-ok-local-agent", + afterText: [ + "Bot: qa_mcp_echo:mcp-ok-local-agent", + "User: Call qa_mcp_echo", + ].join("\n"), + expectedText: "qa_mcp_echo:mcp-ok-local-agent", + prompt: "Call qa_mcp_echo", + latestExpectedLeaf: "Bot: qa_mcp_echo:mcp-ok-local-agent", + latestFailureLeaf: "", + }); + + assert.equal(result.status, "fail"); + assert.equal(result.final_count, 1); + assert.equal(result.min_expected_count, 2); +}); + +test("debug chat classifier accounts for prompt echo occurrences", () => { + assert.equal(minExpectedOccurrences("", "OK", "say OK"), 2); + const result = classifyDebugChatResult({ + beforeText: "", + afterText: "User: say OK", + expectedText: "OK", + prompt: "say OK", + latestExpectedLeaf: "User: say OK", + latestFailureLeaf: "", + }); + + assert.equal(result.status, "fail"); + assert.equal(result.min_expected_count, 2); + assert.equal(result.final_count, 1); +}); + +test("debug chat classifier requires new assistant evidence when message bubbles are available", () => { + const prompt = "If all steps succeed, final answer must be E2E_OK:skill"; + const result = classifyDebugChatResult({ + beforeText: "", + afterText: `User: ${prompt}`, + expectedText: "E2E_OK:skill", + prompt, + latestExpectedLeaf: prompt, + latestFailureLeaf: "", + beforeMessages: [], + afterMessages: [{ role: "user", text: prompt }], + latestAssistantText: "", + }); + + assert.equal(result.status, "fail"); + assert.match(result.reason, /new assistant message/); + assert.equal(result.before_assistant_expected_count, 0); + assert.equal(result.after_assistant_expected_count, 0); +}); + +test("debug chat classifier passes when expected text appears in a new assistant message", () => { + const prompt = "Return only E2E_OK:skill"; + const result = classifyDebugChatResult({ + beforeText: "", + afterText: `User: ${prompt}\nBot: E2E_OK:skill`, + expectedText: "E2E_OK:skill", + prompt, + latestExpectedLeaf: "E2E_OK:skill", + latestFailureLeaf: "", + beforeMessages: [], + afterMessages: [ + { role: "user", text: prompt }, + { role: "assistant", text: "E2E_OK:skill" }, + ], + latestAssistantText: "E2E_OK:skill", + }); + + assert.equal(result.status, "pass"); + assert.match(result.reason, /new assistant message/); + assert.equal(result.before_assistant_expected_count, 0); + assert.equal(result.after_assistant_expected_count, 1); +}); + +test("debug chat classifier allows a recovered failure when latest assistant is successful", () => { + const expectedText = "E2E_OK:skill"; + const result = classifyDebugChatResult({ + beforeText: "", + afterText: [ + "Bot: Agent runner temporarily unavailable", + `Bot: recovered and completed ${expectedText}`, + ].join("\n"), + expectedText, + prompt: "Modify the existing skill", + latestExpectedLeaf: `recovered and completed ${expectedText}`, + latestFailureLeaf: "Agent runner temporarily unavailable", + beforeMessages: [], + afterMessages: [ + { role: "assistant", text: "Agent runner temporarily unavailable" }, + { role: "assistant", text: `recovered and completed ${expectedText}` }, + ], + latestAssistantText: `recovered and completed ${expectedText}`, + }); + + assert.equal(result.status, "pass"); + assert.equal(result.before_assistant_expected_count, 0); + assert.equal(result.after_assistant_expected_count, 1); +}); + +test("env doctor explains a missing backend listener with a startup hint", async () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-env-doctor-")); + try { + const skillsDir = join(tmp, "skills"); + const repoDir = join(tmp, "LangBot"); + const webDir = join(repoDir, "web"); + const browserProfile = join(tmp, "browser-profile"); + const chromium = join(tmp, "chromium"); + mkdirSync(skillsDir, { recursive: true }); + mkdirSync(webDir, { recursive: true }); + mkdirSync(browserProfile, { recursive: true }); + writeFileSync(chromium, ""); + writeFileSync( + join(skillsDir, ".env"), + [ + "LANGBOT_BACKEND_URL=http://127.0.0.1:59998", + "LANGBOT_FRONTEND_URL=http://127.0.0.1:59998", + "LANGBOT_DEV_FRONTEND_URL=http://127.0.0.1:59998", + `LANGBOT_REPO=${repoDir}`, + `LANGBOT_WEB_REPO=${webDir}`, + `LANGBOT_BROWSER_PROFILE=${browserProfile}`, + `LANGBOT_CHROMIUM_EXECUTABLE=${chromium}`, + "LANGBOT_PROXY_HTTP=http://127.0.0.1:7890", + "LANGBOT_PROXY_SOCKS=socks5://127.0.0.1:7890", + "LANGBOT_NO_PROXY=localhost,127.0.0.1,::1", + ].join("\n"), + ); + + const result = await captureAsync(() => commandEnvDoctor({ root: tmp, args: ["env", "doctor"] })); + + assert.equal(result.code, 1); + assert.match(result.output, /FAIL: LANGBOT_BACKEND_URL: no HTTP service reachable because 127\.0\.0\.1:59998 is not listening/); + assert.match(result.output, new RegExp(`WARN: LANGBOT_BACKEND_URL: start backend: cd ${repoDir.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")} && uv run main.py`)); + assert.match(result.output, new RegExp(`WARN: LANGBOT_FRONTEND_URL: start frontend: cd ${webDir.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")} && pnpm dev`)); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("env doctor does not require proxy variables", async () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-env-doctor-no-proxy-")); + try { + const skillsDir = join(tmp, "skills"); + const repoDir = join(tmp, "LangBot"); + const webDir = join(repoDir, "web"); + const browserProfile = join(tmp, "browser-profile"); + const chromium = join(tmp, "chromium"); + mkdirSync(skillsDir, { recursive: true }); + mkdirSync(webDir, { recursive: true }); + mkdirSync(browserProfile, { recursive: true }); + writeFileSync(chromium, ""); + writeFileSync( + join(skillsDir, ".env"), + [ + "LANGBOT_BACKEND_URL=http://127.0.0.1:59997", + "LANGBOT_FRONTEND_URL=http://127.0.0.1:59997", + "LANGBOT_DEV_FRONTEND_URL=http://127.0.0.1:59997", + `LANGBOT_REPO=${repoDir}`, + `LANGBOT_WEB_REPO=${webDir}`, + `LANGBOT_BROWSER_PROFILE=${browserProfile}`, + `LANGBOT_CHROMIUM_EXECUTABLE=${chromium}`, + ].join("\n"), + ); + + const result = await captureAsync(() => commandEnvDoctor({ root: tmp, args: ["env", "doctor"] })); + + assert.equal(result.code, 1); + assert.doesNotMatch(result.output, /missing LANGBOT_PROXY|missing LANGBOT_NO_PROXY/); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("env show redacts secret-like values by default", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-env-show-redact-")); + try { + mkdirSync(join(tmp, "skills"), { recursive: true }); + writeFileSync( + join(tmp, "skills", ".env"), + [ + "LANGBOT_FRONTEND_URL=http://127.0.0.1:3000", + "LANGBOT_API_KEY=sk-test-secret", + "LANGBOT_PROXY_HTTP=http://user:pass@127.0.0.1:7890", + ].join("\n"), + ); + + const text = capture(() => commandEnvShow({ root: tmp, args: ["env", "show"] })); + assert.equal(text.code, 0); + assert.match(text.output, /LANGBOT_API_KEY=\[redacted\]/); + assert.match(text.output, /LANGBOT_PROXY_HTTP=http:\/\/\[redacted\]@127\.0\.0\.1:7890/); + assert.doesNotMatch(text.output, /sk-test-secret|user:pass/); + + const json = capture(() => commandEnvShow({ root: tmp, args: ["env", "show", "--json"] })); + assert.equal(json.code, 0); + const parsed = JSON.parse(json.output); + assert.equal(parsed.LANGBOT_API_KEY, "[redacted]"); + assert.equal(parsed.LANGBOT_PROXY_HTTP, "http://[redacted]@127.0.0.1:7890"); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("test plan renders agent-browser QA guidance", () => { + const result = capture(() => commandTestPlan(ctx(["test", "plan", "pipeline-debug-chat"]))); + assert.equal(result.code, 0); + assert.match(result.output, /Mode: agent-browser/); + assert.match(result.output, /Use browser\/UI interaction as the primary QA path/); + assert.match(result.output, /API\/curl\/log checks are diagnostic only/); + assert.match(result.output, /## Browser Steps/); + assert.match(result.output, /## Success Signals/); + assert.match(result.output, /## Required Evidence/); + assert.match(result.output, /## Automation Readiness/); + assert.match(result.output, /## Fixture Readiness/); + assert.match(result.output, /## Manual Readiness/); + assert.match(result.output, /backend_log/); +}); + +test("test plan JSON is parseable and includes troubleshooting patterns", () => { + const result = capture(() => commandTestPlan(ctx(["test", "plan", "pipeline-debug-chat", "--json"]))); + assert.equal(result.code, 0); + const plan = JSON.parse(result.output); + assert.equal(plan.id, "pipeline-debug-chat"); + assert.equal(plan.mode, "agent-browser"); + assert.ok(["ready", "missing"].includes(plan.automation_readiness.status)); + assert.ok(plan.automation_readiness.defaulted.includes("LANGBOT_E2E_PROMPT")); + assert.ok(plan.automation_readiness.defaulted.includes("LANGBOT_E2E_EXPECTED_TEXT")); + assert.equal(plan.manual_readiness.status, "manual_check"); + assert.ok(plan.success_patterns.includes("Streaming completed")); + assert.ok(plan.troubleshooting.some((entry: { id: string }) => entry.id === "plugin-runtime-timeout")); +}); + +test("test plan JSON exposes missing case-specific pipeline readiness", () => { + const result = capture(() => commandTestPlan(ctx(["test", "plan", "local-agent-basic-debug-chat", "--json"]))); + assert.equal(result.code, 0); + const plan = JSON.parse(result.output); + assert.equal(plan.env_readiness.status, "ready"); + assert.ok(["ready", "missing"].includes(plan.automation_readiness.status)); + assert.ok(plan.automation_readiness.pipeline_env_required); + assert.ok( + plan.automation_readiness.missing.includes("LANGBOT_LOCAL_AGENT_PIPELINE_URL|LANGBOT_LOCAL_AGENT_PIPELINE_NAME") + || plan.automation_readiness.configured.some((key: string) => key.startsWith("LANGBOT_LOCAL_AGENT_PIPELINE_")), + ); +}); + +test("generic pipeline readiness accepts either URL or name target", () => { + const originalUrl = process.env.LANGBOT_PIPELINE_URL; + const originalName = process.env.LANGBOT_PIPELINE_NAME; + try { + process.env.LANGBOT_PIPELINE_URL = "http://127.0.0.1:3000/home/pipelines?id=only-url"; + process.env.LANGBOT_PIPELINE_NAME = ""; + + const ready = capture(() => commandTestPlan(ctx(["test", "plan", "pipeline-debug-chat", "--json"]))); + assert.equal(ready.code, 0); + const plan = JSON.parse(ready.output); + assert.equal(plan.env_readiness.status, "ready"); + assert.equal(plan.automation_readiness.status, "ready"); + assert.ok(plan.automation_readiness.required.includes("LANGBOT_PIPELINE_URL|LANGBOT_PIPELINE_NAME")); + + process.env.LANGBOT_PIPELINE_URL = ""; + process.env.LANGBOT_PIPELINE_NAME = ""; + + const missing = capture(() => commandTestPlan(ctx(["test", "plan", "pipeline-debug-chat", "--json"]))); + assert.equal(missing.code, 0); + const missingPlan = JSON.parse(missing.output); + assert.equal(missingPlan.env_readiness.status, "missing"); + assert.ok(missingPlan.env_readiness.missing.includes("LANGBOT_PIPELINE_URL|LANGBOT_PIPELINE_NAME")); + assert.equal(missingPlan.automation_readiness.status, "missing"); + assert.ok(missingPlan.automation_readiness.missing.includes("LANGBOT_PIPELINE_URL|LANGBOT_PIPELINE_NAME")); + } finally { + if (originalUrl === undefined) delete process.env.LANGBOT_PIPELINE_URL; + else process.env.LANGBOT_PIPELINE_URL = originalUrl; + if (originalName === undefined) delete process.env.LANGBOT_PIPELINE_NAME; + else process.env.LANGBOT_PIPELINE_NAME = originalName; + } +}); + +test("test recommend maps AgentRunner ledger changes to focused probes", () => { + const result = capture(() => commandTestRecommend(ctx([ + "test", + "recommend", + "--file", + "LangBot/src/langbot/pkg/agent/runner/run_ledger_store.py", + "--file", + "LangBot/tests/unit_tests/agent/test_run_ledger_store.py", + "--json", + ]))); + assert.equal(result.code, 0); + const report = JSON.parse(result.output); + const ids = report.recommendations.map((item: { id: string }) => item.id); + assert.ok(ids.includes("agent-runner-ledger-invariants")); + assert.ok(ids.includes("agent-runner-ledger-stress")); + assert.ok(ids.includes("agent-runner-ledger-contention")); + assert.ok(ids.includes("agent-runner-async-db-readiness")); + assert.ok(ids.includes("agent-runner-ledger-concurrency")); + assert.ok(report.commands.every((command: string) => !command.startsWith("bin/lbs test run ") || command.endsWith(" --dry-run"))); + assert.ok(report.notes.some((note: string) => note.includes("Remove --dry-run"))); +}); + +test("test recommend maps AgentRunner result changes to fixture contract", () => { + const result = capture(() => commandTestRecommend(ctx([ + "test", + "recommend", + "--file", + "langbot-plugin-sdk/src/langbot_plugin/api/entities/builtin/agent_runner/result.py", + "--json", + ]))); + assert.equal(result.code, 0); + const report = JSON.parse(result.output); + const ids = report.recommendations.map((item: { id: string }) => item.id); + assert.ok(ids.includes("agent-runner-fixture-contract")); + assert.ok(ids.includes("agent-runner-behavior-matrix")); + assert.ok(!ids.includes("agent-runner-ledger-invariants")); +}); + +test("test recommend maps QA AgentRunner fixture changes to live install", () => { + const result = capture(() => commandTestRecommend(ctx([ + "test", + "recommend", + "--file", + "langbot-skills/skills/langbot-testing/fixtures/plugins/qa-agent-runner/components/agent_runner/default.py", + "--json", + ]))); + assert.equal(result.code, 0); + const report = JSON.parse(result.output); + const ids = report.recommendations.map((item: { id: string }) => item.id); + assert.ok(ids.includes("agent-runner-fixture-contract")); + assert.ok(ids.includes("agent-runner-live-install")); + assert.ok(ids.includes("agent-runner-qa-debug-chat")); +}); + +test("test recommend maps QA plugin smoke fixture changes to live install", () => { + const result = capture(() => commandTestRecommend(ctx([ + "test", + "recommend", + "--file", + "langbot-skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/main.py", + "--json", + ]))); + assert.equal(result.code, 0); + const report = JSON.parse(result.output); + const ids = report.recommendations.map((item: { id: string }) => item.id); + assert.ok(ids.includes("qa-plugin-smoke-live-install")); +}); + +test("test recommend keeps git status paths intact", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-recommend-git-")); + const originalRepos = { + LANGBOT_REPO: process.env.LANGBOT_REPO, + LANGBOT_PLUGIN_SDK_REPO: process.env.LANGBOT_PLUGIN_SDK_REPO, + LANGBOT_AGENT_RUNNER_REPO: process.env.LANGBOT_AGENT_RUNNER_REPO, + LANGBOT_LOCAL_AGENT_REPO: process.env.LANGBOT_LOCAL_AGENT_REPO, + }; + try { + const repo = join(tmp, "LangBot"); + mkdirSync(join(repo, "src", "langbot", "pkg", "agent", "runner"), { recursive: true }); + spawnSync("git", ["init"], { cwd: repo }); + spawnSync("git", ["config", "user.email", "qa@example.test"], { cwd: repo }); + spawnSync("git", ["config", "user.name", "QA"], { cwd: repo }); + writeFileSync(join(repo, "README.md"), "test\n"); + writeFileSync(join(repo, "src", "langbot", "pkg", "agent", "runner", "run_ledger_store.py"), "# test\n"); + spawnSync("git", ["add", "README.md", "src/langbot/pkg/agent/runner/run_ledger_store.py"], { cwd: repo }); + spawnSync("git", ["commit", "-m", "init"], { cwd: repo }); + writeFileSync(join(repo, "src", "langbot", "pkg", "agent", "runner", "run_ledger_store.py"), "# changed\n"); + + process.env.LANGBOT_REPO = repo; + process.env.LANGBOT_PLUGIN_SDK_REPO = join(tmp, "missing-sdk"); + process.env.LANGBOT_AGENT_RUNNER_REPO = join(tmp, "missing-runner"); + process.env.LANGBOT_LOCAL_AGENT_REPO = join(tmp, "missing-local"); + const result = capture(() => commandTestRecommend({ root, args: ["test", "recommend", "--json"] })); + + assert.equal(result.code, 0); + const report = JSON.parse(result.output); + assert.ok(report.changed_files.includes("LangBot/src/langbot/pkg/agent/runner/run_ledger_store.py")); + assert.ok(!report.changed_files.some((file: string) => file.includes("LangBot/rc/"))); + } finally { + for (const [key, value] of Object.entries(originalRepos)) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("test start creates a run handoff with a bounded report command", () => { + const result = capture(() => commandTestStart(ctx(["test", "start", "pipeline-debug-chat"]))); + assert.equal(result.code, 0); + assert.match(result.output, /^# Test Start: pipeline-debug-chat/m); + assert.match(result.output, /bin\/lbs test plan pipeline-debug-chat/); + assert.match(result.output, /bin\/lbs test run pipeline-debug-chat --run-id .+ --output reports\/evidence\/.+pipeline-debug-chat/); + assert.match(result.output, /bin\/lbs test report pipeline-debug-chat --since ".+" --console-log reports\/evidence\/.+\/console\.log --evidence-dir reports\/evidence\/.+ --output reports\/.+pipeline-debug-chat\.md/); + assert.match(result.output, /Streaming completed/); +}); + +test("test start JSON is parseable for agent orchestration", () => { + const result = capture(() => commandTestStart(ctx(["test", "start", "pipeline-debug-chat", "--json"]))); + assert.equal(result.code, 0); + const start = JSON.parse(result.output); + assert.equal(start.case.id, "pipeline-debug-chat"); + assert.match(start.run_id, /pipeline-debug-chat$/); + assert.match(start.started_at_local, /\d{4}-\d{2}-\d{2}T/); + assert.match(start.report_command, /--since/); + assert.match(start.result_command_template, /bin\/lbs test result pipeline-debug-chat/); + assert.match(start.automation.command, /bin\/lbs test run pipeline-debug-chat/); + assert.ok(start.success_patterns.includes("Streaming completed")); + assert.ok(start.evidence_required.includes("backend_log")); +}); + +test("test result writes a suite-readable result.json and enforces pass evidence", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-test-result-")); + try { + const evidenceDir = join(tmp, "pipeline-run"); + const ok = capture(() => commandTestResult(ctx([ + "test", + "result", + "pipeline-debug-chat", + "--result", + "pass", + "--reason", + "Debug Chat returned OK and logs were clean.", + "--evidence-dir", + evidenceDir, + "--started-at", + "2026-05-21T10:30:00.000+08:00", + "--evidence", + "ui,screenshot,console,backend_log", + "--json", + ]))); + + assert.equal(ok.code, 0); + const record = JSON.parse(ok.output); + assert.equal(record.source, "final"); + assert.equal(record.status, "pass"); + assert.equal(record.evidence_status, "complete"); + assert.deepEqual(record.evidence_missing, []); + assert.equal(JSON.parse(readFileSync(join(evidenceDir, "result.json"), "utf8")).case_id, "pipeline-debug-chat"); + + const missing = captureAll(() => commandTestResult(ctx([ + "test", + "result", + "pipeline-debug-chat", + "--result", + "pass", + "--reason", + "Missing backend evidence should not be accepted as pass.", + "--evidence-dir", + join(tmp, "missing-evidence"), + "--evidence", + "ui", + ]))); + assert.equal(missing.code, 1); + assert.match(missing.error, /missing required evidence/); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("test run dry-run exposes case automation script and evidence paths", () => { + const result = capture(() => commandTestRun(ctx([ + "test", + "run", + "pipeline-debug-chat", + "--run-id", + "run-123", + "--output", + "reports/evidence/run-123", + "--dry-run", + ]))); + assert.equal(result.code, 0); + assert.match(result.output, /^# Test Automation: pipeline-debug-chat/m); + assert.match(result.output, /scripts\/e2e\/pipeline-debug-chat\.mjs/); + assert.match(result.output, /console_log: reports\/evidence\/run-123\/console\.log/); + assert.match(result.output, /automation_result_json: reports\/evidence\/run-123\/automation-result\.json/); + assert.match(result.output, /result_json: reports\/evidence\/run-123\/result\.json/); + assert.match(result.output, /LANGBOT_PIPELINE_URL/); +}); + +test("test run dry-run JSON is parseable for automation orchestration", () => { + const result = capture(() => commandTestRun(ctx([ + "test", + "run", + "webui-login-state", + "--run-id", + "login-run", + "--dry-run", + "--json", + ]))); + assert.equal(result.code, 0); + const run = JSON.parse(result.output); + assert.equal(run.case.id, "webui-login-state"); + assert.equal(run.run_id, "login-run"); + assert.equal(run.automation.script, "scripts/e2e/webui-login-state.mjs"); + assert.equal(run.automation.exists, true); + assert.match(run.automation.automation_result_json, /automation-result\.json$/); + assert.match(run.automation.result_json, /result\.json$/); + assert.match(run.automation.report_command, /--console-log/); +}); + +test("test run JSON executes automation unless dry-run is explicit", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-run-json-exec-")); + try { + const skillDir = join(tmp, "skills", "langbot-testing"); + const casesDir = join(skillDir, "cases"); + const scriptsDir = join(tmp, "scripts"); + mkdirSync(casesDir, { recursive: true }); + mkdirSync(scriptsDir, { recursive: true }); + writeFileSync(join(skillDir, "SKILL.md"), "---\nname: langbot-testing\ndescription: Testing.\n---\n\n# Testing\n"); + writeFileSync(join(tmp, "skills", ".env"), ""); + writeFileSync( + join(casesDir, "json-exec.yaml"), + [ + "id: json-exec", + "title: JSON Exec", + "mode: probe", + "area: qa", + "type: smoke", + "priority: p2", + "risk: low", + "ci_eligible: true", + "automation: scripts/write-marker.mjs", + ].join("\n"), + ); + writeFileSync( + join(scriptsDir, "write-marker.mjs"), + [ + "import { writeFileSync } from 'node:fs';", + "import { join } from 'node:path';", + "writeFileSync(join(process.env.LBS_ROOT, 'json-ran.txt'), 'yes');", + ].join("\n"), + ); + + const result = capture(() => commandTestRun({ + root: tmp, + args: ["test", "run", "json-exec", "--run-id", "json-run", "--json"], + })); + + assert.equal(result.code, 0); + assert.equal(readFileSync(join(tmp, "json-ran.txt"), "utf8"), "yes"); + const payload = JSON.parse(result.output); + assert.equal(payload.exit_status, 0); + assert.equal(payload.automation_execution.status, "ok"); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("test run lets explicit environment override automation defaults", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-run-env-override-")); + const originalPatch = process.env.LANGBOT_E2E_RUNNER_CONFIG_PATCH_JSON; + try { + const skillDir = join(tmp, "skills", "langbot-testing"); + const casesDir = join(skillDir, "cases"); + const scriptsDir = join(tmp, "scripts"); + mkdirSync(casesDir, { recursive: true }); + mkdirSync(scriptsDir, { recursive: true }); + writeFileSync(join(skillDir, "SKILL.md"), "---\nname: langbot-testing\ndescription: Testing.\n---\n\n# Testing\n"); + writeFileSync(join(tmp, "skills", ".env"), ""); + writeFileSync( + join(casesDir, "env-override.yaml"), + [ + "id: env-override", + "title: Env Override", + "mode: agent-browser", + "area: pipeline", + "type: smoke", + "priority: p2", + "risk: low", + "ci_eligible: false", + "automation: scripts/write-env.mjs", + "automation_runner_config_patch_json: '{\"source\":\"default\"}'", + ].join("\n"), + ); + writeFileSync( + join(scriptsDir, "write-env.mjs"), + [ + "import { writeFileSync } from 'node:fs';", + "import { join } from 'node:path';", + "writeFileSync(join(process.env.LBS_ROOT, 'env-out.json'), JSON.stringify({", + " patch: process.env.LANGBOT_E2E_RUNNER_CONFIG_PATCH_JSON,", + "}));", + ].join("\n"), + ); + + process.env.LANGBOT_E2E_RUNNER_CONFIG_PATCH_JSON = '{"source":"explicit"}'; + const result = capture(() => commandTestRun({ + root: tmp, + args: ["test", "run", "env-override", "--run-id", "env-run"], + })); + + assert.equal(result.code, 0); + const observed = JSON.parse(readFileSync(join(tmp, "env-out.json"), "utf8")); + assert.equal(observed.patch, '{"source":"explicit"}'); + } finally { + if (originalPatch === undefined) delete process.env.LANGBOT_E2E_RUNNER_CONFIG_PATCH_JSON; + else process.env.LANGBOT_E2E_RUNNER_CONFIG_PATCH_JSON = originalPatch; + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("test run expands env references in automation defaults", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-run-env-expand-")); + try { + const skillDir = join(tmp, "skills", "langbot-testing"); + const casesDir = join(skillDir, "cases"); + const scriptsDir = join(tmp, "scripts"); + mkdirSync(casesDir, { recursive: true }); + mkdirSync(scriptsDir, { recursive: true }); + writeFileSync(join(skillDir, "SKILL.md"), "---\nname: langbot-testing\ndescription: Testing.\n---\n\n# Testing\n"); + writeFileSync(join(tmp, "skills", ".env"), "QA_KB_UUID=kb-from-env\nQA_MODEL_UUID=model-from-env\n"); + writeFileSync( + join(casesDir, "env-expand.yaml"), + [ + "id: env-expand", + "title: Env Expand", + "mode: agent-browser", + "area: pipeline", + "type: smoke", + "priority: p2", + "risk: low", + "ci_eligible: false", + "automation: scripts/write-expanded-env.mjs", + "automation_runner_config_patch_json: '{\"knowledge-bases\":[\"${QA_KB_UUID}\"],\"model\":{\"primary\":\"${QA_MODEL_UUID}\"}}'", + ].join("\n"), + ); + writeFileSync( + join(scriptsDir, "write-expanded-env.mjs"), + [ + "import { writeFileSync } from 'node:fs';", + "import { join } from 'node:path';", + "writeFileSync(join(process.env.LBS_ROOT, 'expanded-env-out.json'), JSON.stringify({", + " patch: process.env.LANGBOT_E2E_RUNNER_CONFIG_PATCH_JSON,", + "}));", + ].join("\n"), + ); + + const dryRun = capture(() => commandTestRun({ + root: tmp, + args: ["test", "run", "env-expand", "--run-id", "env-expand-dry", "--dry-run", "--json"], + })); + assert.equal(dryRun.code, 0); + const plan = JSON.parse(dryRun.output); + assert.equal( + plan.automation.env_defaults.LANGBOT_E2E_RUNNER_CONFIG_PATCH_JSON, + '{"knowledge-bases":["kb-from-env"],"model":{"primary":"model-from-env"}}', + ); + + const run = capture(() => commandTestRun({ + root: tmp, + args: ["test", "run", "env-expand", "--run-id", "env-expand-run"], + })); + assert.equal(run.code, 0); + const observed = JSON.parse(readFileSync(join(tmp, "expanded-env-out.json"), "utf8")); + assert.equal(observed.patch, '{"knowledge-bases":["kb-from-env"],"model":{"primary":"model-from-env"}}'); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("test run setup automation isolates evidence and reloads env", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-run-setup-automation-")); + try { + const skillDir = join(tmp, "skills", "langbot-testing"); + const casesDir = join(skillDir, "cases"); + const scriptsDir = join(tmp, "scripts"); + mkdirSync(casesDir, { recursive: true }); + mkdirSync(scriptsDir, { recursive: true }); + writeFileSync(join(skillDir, "SKILL.md"), "---\nname: langbot-testing\ndescription: Testing.\n---\n\n# Testing\n"); + writeFileSync(join(tmp, "skills", ".env"), "SETUP_VALUE=\n"); + writeFileSync( + join(casesDir, "setup-main.yaml"), + [ + "id: setup-main", + "title: Setup Main", + "mode: agent-browser", + "area: pipeline", + "type: smoke", + "priority: p2", + "risk: low", + "ci_eligible: false", + "env:", + " - SETUP_VALUE", + "setup_automation:", + " - \"node:scripts/write-setup-env.mjs --write-env\"", + "setup_provides_env:", + " - SETUP_VALUE", + "automation: scripts/read-setup-env.mjs", + ].join("\n"), + ); + writeFileSync( + join(casesDir, "setup-env-issue.yaml"), + [ + "id: setup-env-issue", + "title: Setup Env Issue", + "mode: agent-browser", + "area: pipeline", + "type: smoke", + "priority: p2", + "risk: low", + "ci_eligible: false", + "setup_automation:", + " - \"node:scripts/write-setup-env-issue.mjs\"", + "automation: scripts/read-setup-env.mjs", + ].join("\n"), + ); + writeFileSync( + join(casesDir, "setup-fail-after-pass.yaml"), + [ + "id: setup-fail-after-pass", + "title: Setup Fail After Pass", + "mode: agent-browser", + "area: pipeline", + "type: smoke", + "priority: p2", + "risk: low", + "ci_eligible: false", + "setup_automation:", + " - \"node:scripts/write-setup-pass-then-fail.mjs\"", + "automation: scripts/read-setup-env.mjs", + ].join("\n"), + ); + writeFileSync( + join(scriptsDir, "write-setup-env.mjs"), + [ + "import { mkdirSync, writeFileSync } from 'node:fs';", + "import { dirname, join } from 'node:path';", + "const local = join(process.env.LBS_ROOT, 'skills', '.env.local');", + "writeFileSync(local, 'SETUP_VALUE=from-setup\\n');", + "mkdirSync(process.env.LBS_EVIDENCE_DIR, { recursive: true });", + "writeFileSync(join(process.env.LBS_EVIDENCE_DIR, 'automation-result.json'), JSON.stringify({ status: 'pass', stage: 'setup' }));", + "writeFileSync(join(process.env.LBS_EVIDENCE_DIR, 'result.json'), JSON.stringify({ status: 'pass', stage: 'setup' }));", + "writeFileSync(join(dirname(process.env.LBS_EVIDENCE_DIR), 'setup-ran.txt'), 'yes');", + ].join("\n"), + ); + writeFileSync( + join(scriptsDir, "read-setup-env.mjs"), + [ + "import { mkdirSync, writeFileSync } from 'node:fs';", + "import { join } from 'node:path';", + "mkdirSync(process.env.LBS_EVIDENCE_DIR, { recursive: true });", + "writeFileSync(join(process.env.LBS_ROOT, 'main-observed.json'), JSON.stringify({ value: process.env.SETUP_VALUE }));", + "writeFileSync(join(process.env.LBS_EVIDENCE_DIR, 'automation-result.json'), JSON.stringify({ status: 'pass', stage: 'main' }));", + "writeFileSync(join(process.env.LBS_EVIDENCE_DIR, 'result.json'), JSON.stringify({ status: 'pass', stage: 'main' }));", + "if (process.env.SETUP_VALUE !== 'from-setup') process.exit(1);", + ].join("\n"), + ); + writeFileSync( + join(scriptsDir, "write-setup-env-issue.mjs"), + [ + "import { mkdirSync, writeFileSync } from 'node:fs';", + "import { join } from 'node:path';", + "mkdirSync(process.env.LBS_EVIDENCE_DIR, { recursive: true });", + "writeFileSync(join(process.env.LBS_EVIDENCE_DIR, 'automation-result.json'), JSON.stringify({ status: 'env_issue', reason: 'setup env missing' }));", + "process.exit(2);", + ].join("\n"), + ); + writeFileSync( + join(scriptsDir, "write-setup-pass-then-fail.mjs"), + [ + "import { mkdirSync, writeFileSync } from 'node:fs';", + "import { join } from 'node:path';", + "mkdirSync(process.env.LBS_EVIDENCE_DIR, { recursive: true });", + "writeFileSync(join(process.env.LBS_EVIDENCE_DIR, 'automation-result.json'), JSON.stringify({ status: 'pass', reason: 'stale pass before crash' }));", + "process.exit(1);", + ].join("\n"), + ); + + const dryRun = capture(() => commandTestRun({ + root: tmp, + args: ["test", "run", "setup-main", "--run-id", "setup-run", "--output", join(tmp, "evidence"), "--dry-run", "--json"], + })); + assert.equal(dryRun.code, 0); + const plan = JSON.parse(dryRun.output); + assert.equal(plan.setup_automation.length, 1); + assert.match(plan.setup_automation[0].evidence_dir, /setup\/01-write-setup-env$/); + assert.match(plan.setup_automation[0].command, /^node scripts\/write-setup-env\.mjs --write-env$/); + assert.equal(plan.setup_automation[0].dry_run_command, ""); + assert.equal(existsSync(join(tmp, "skills", ".env.local")), false); + + const run = capture(() => commandTestRun({ + root: tmp, + args: ["test", "run", "setup-main", "--run-id", "setup-run", "--output", join(tmp, "evidence")], + })); + assert.equal(run.code, 0); + const observed = JSON.parse(readFileSync(join(tmp, "main-observed.json"), "utf8")); + assert.equal(observed.value, "from-setup"); + const setupResult = JSON.parse(readFileSync(join(tmp, "evidence", "setup", "01-write-setup-env", "automation-result.json"), "utf8")); + const mainResult = JSON.parse(readFileSync(join(tmp, "evidence", "automation-result.json"), "utf8")); + assert.equal(setupResult.stage, "setup"); + assert.equal(mainResult.stage, "main"); + + const envIssue = capture(() => commandTestRun({ + root: tmp, + args: ["test", "run", "setup-env-issue", "--run-id", "setup-env-issue-run", "--output", join(tmp, "evidence-env-issue")], + })); + assert.equal(envIssue.code, 2); + const parentResult = JSON.parse(readFileSync(join(tmp, "evidence-env-issue", "automation-result.json"), "utf8")); + assert.equal(parentResult.status, "env_issue"); + assert.equal(parentResult.reason, "setup env missing"); + + const failAfterPass = capture(() => commandTestRun({ + root: tmp, + args: ["test", "run", "setup-fail-after-pass", "--run-id", "setup-fail-after-pass-run", "--output", join(tmp, "evidence-fail-after-pass")], + })); + assert.equal(failAfterPass.code, 1); + const failAfterPassResult = JSON.parse(readFileSync(join(tmp, "evidence-fail-after-pass", "automation-result.json"), "utf8")); + assert.equal(failAfterPassResult.status, "fail"); + assert.equal(failAfterPassResult.reason, "stale pass before crash"); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("test run setup automation can execute another case outside this source repo", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-run-setup-case-")); + try { + const skillDir = join(tmp, "skills", "langbot-testing"); + const casesDir = join(skillDir, "cases"); + const scriptsDir = join(tmp, "scripts"); + mkdirSync(casesDir, { recursive: true }); + mkdirSync(scriptsDir, { recursive: true }); + writeFileSync(join(skillDir, "SKILL.md"), "---\nname: langbot-testing\ndescription: Testing.\n---\n\n# Testing\n"); + writeFileSync(join(tmp, "skills", ".env"), "SETUP_VALUE=\n"); + writeFileSync( + join(casesDir, "setup-child.yaml"), + [ + "id: setup-child", + "title: Setup Child", + "mode: probe", + "area: qa", + "type: smoke", + "priority: p2", + "risk: low", + "ci_eligible: true", + "automation: scripts/write-child-env.mjs", + ].join("\n"), + ); + writeFileSync( + join(casesDir, "setup-parent.yaml"), + [ + "id: setup-parent", + "title: Setup Parent", + "mode: probe", + "area: qa", + "type: smoke", + "priority: p2", + "risk: low", + "ci_eligible: true", + "setup_automation:", + " - \"case:setup-child\"", + "setup_provides_env:", + " - SETUP_VALUE", + "automation: scripts/read-child-env.mjs", + ].join("\n"), + ); + writeFileSync( + join(scriptsDir, "write-child-env.mjs"), + [ + "import { mkdirSync, writeFileSync } from 'node:fs';", + "import { join } from 'node:path';", + "writeFileSync(join(process.env.LBS_ROOT, 'skills', '.env.local'), 'SETUP_VALUE=from-child\\n');", + "mkdirSync(process.env.LBS_EVIDENCE_DIR, { recursive: true });", + "writeFileSync(join(process.env.LBS_EVIDENCE_DIR, 'automation-result.json'), JSON.stringify({ status: 'pass' }));", + "writeFileSync(join(process.env.LBS_EVIDENCE_DIR, 'result.json'), JSON.stringify({ status: 'pass' }));", + ].join("\n"), + ); + writeFileSync( + join(scriptsDir, "read-child-env.mjs"), + [ + "import { mkdirSync, writeFileSync } from 'node:fs';", + "import { join } from 'node:path';", + "mkdirSync(process.env.LBS_EVIDENCE_DIR, { recursive: true });", + "writeFileSync(join(process.env.LBS_EVIDENCE_DIR, 'automation-result.json'), JSON.stringify({ status: 'pass', value: process.env.SETUP_VALUE }));", + "writeFileSync(join(process.env.LBS_EVIDENCE_DIR, 'result.json'), JSON.stringify({ status: 'pass', value: process.env.SETUP_VALUE }));", + "if (process.env.SETUP_VALUE !== 'from-child') process.exit(1);", + ].join("\n"), + ); + + const run = capture(() => commandTestRun({ + root: tmp, + args: ["test", "run", "setup-parent", "--run-id", "setup-parent-run", "--output", join(tmp, "evidence")], + })); + + assert.equal(run.code, 0); + assert.ok(existsSync(join(tmp, "evidence", "setup", "01-setup-child", "result.json"))); + const result = JSON.parse(readFileSync(join(tmp, "evidence", "automation-result.json"), "utf8")); + assert.equal(result.value, "from-child"); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("test run automation inherits parent process environment", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-run-env-inherit-")); + try { + const skillDir = join(tmp, "skills", "langbot-testing"); + const casesDir = join(skillDir, "cases"); + const scriptsDir = join(tmp, "scripts"); + mkdirSync(casesDir, { recursive: true }); + mkdirSync(scriptsDir, { recursive: true }); + writeFileSync(join(skillDir, "SKILL.md"), "---\nname: langbot-testing\ndescription: Testing.\n---\n\n# Testing\n"); + writeFileSync(join(tmp, "skills", ".env"), ""); + writeFileSync( + join(casesDir, "env-inherit.yaml"), + [ + "id: env-inherit", + "title: Env Inherit", + "mode: probe", + "area: qa", + "type: smoke", + "priority: p2", + "risk: low", + "ci_eligible: true", + "automation: scripts/read-path.mjs", + ].join("\n"), + ); + writeFileSync( + join(scriptsDir, "read-path.mjs"), + [ + "import { mkdirSync, writeFileSync } from 'node:fs';", + "import { join } from 'node:path';", + "mkdirSync(process.env.LBS_EVIDENCE_DIR, { recursive: true });", + "writeFileSync(join(process.env.LBS_EVIDENCE_DIR, 'automation-result.json'), JSON.stringify({ status: process.env.PATH ? 'pass' : 'fail' }));", + "process.exit(process.env.PATH ? 0 : 1);", + ].join("\n"), + ); + + const run = capture(() => commandTestRun({ + root: tmp, + args: ["test", "run", "env-inherit", "--run-id", "env-inherit-run", "--output", join(tmp, "evidence")], + })); + + assert.equal(run.code, 0); + const result = JSON.parse(readFileSync(join(tmp, "evidence", "automation-result.json"), "utf8")); + assert.equal(result.status, "pass"); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("test run dry-run marks missing setup case targets", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-run-setup-missing-case-")); + try { + const skillDir = join(tmp, "skills", "langbot-testing"); + const casesDir = join(skillDir, "cases"); + const scriptsDir = join(tmp, "scripts"); + mkdirSync(casesDir, { recursive: true }); + mkdirSync(scriptsDir, { recursive: true }); + writeFileSync(join(skillDir, "SKILL.md"), "---\nname: langbot-testing\ndescription: Testing.\n---\n\n# Testing\n"); + writeFileSync(join(tmp, "skills", ".env"), ""); + writeFileSync( + join(casesDir, "setup-parent.yaml"), + [ + "id: setup-parent", + "title: Setup Parent", + "mode: probe", + "area: qa", + "type: smoke", + "priority: p2", + "risk: low", + "ci_eligible: true", + "setup_automation:", + " - \"case:missing-child\"", + "automation: scripts/pass.mjs", + ].join("\n"), + ); + writeFileSync(join(scriptsDir, "pass.mjs"), "process.exit(0);\n"); + + const result = capture(() => commandTestRun({ + root: tmp, + args: ["test", "run", "setup-parent", "--dry-run", "--json"], + })); + + assert.equal(result.code, 0); + const run = JSON.parse(result.output); + assert.equal(run.setup_automation[0].entry, "case:missing-child"); + assert.doesNotMatch(run.setup_automation[0].command, /--dry-run/); + assert.match(run.setup_automation[0].dry_run_command, /--dry-run/); + assert.equal(run.setup_automation[0].exists, false); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("local-agent effective prompt case has runnable automation defaults", () => { + const result = capture(() => commandTestRun(ctx([ + "test", + "run", + "local-agent-effective-prompt-debug-chat", + "--run-id", + "effective-run", + "--dry-run", + "--json", + ]))); + assert.equal(result.code, 0); + const run = JSON.parse(result.output); + assert.equal(run.automation.script, "scripts/e2e/pipeline-debug-chat.mjs"); + assert.equal(run.automation.env_defaults.LANGBOT_E2E_PROMPT, "qa-effective-prompt"); + assert.equal(run.automation.env_defaults.LANGBOT_E2E_EXPECTED_TEXT, "PROMPT_PREPROCESS_OK"); + assert.equal(run.automation.env_defaults.LANGBOT_E2E_RESPONSE_TIMEOUT_MS, "180000"); + assert.equal(run.automation.pipeline_env_required, true); + assert.ok(run.automation.env_aliases.some((alias: { target: string; source: string }) => ( + alias.target === "LANGBOT_E2E_PIPELINE_URL" && alias.source === "LANGBOT_LOCAL_AGENT_PIPELINE_URL" + ))); +}); + +test("local-agent basic case can setup the local-agent pipeline env", () => { + const result = capture(() => commandTestRun(ctx([ + "test", + "run", + "local-agent-basic-debug-chat", + "--dry-run", + "--json", + ]))); + assert.equal(result.code, 0); + const run = JSON.parse(result.output); + assert.deepEqual(run.setup_automation.map((item: { entry: string }) => item.entry), [ + "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env", + ]); + + const planResult = capture(() => commandTestPlan(ctx(["test", "plan", "local-agent-basic-debug-chat", "--json"]))); + assert.equal(planResult.code, 0); + const plan = JSON.parse(planResult.output); + assert.deepEqual(plan.setup_provides_env, [ + "LANGBOT_LOCAL_AGENT_PIPELINE_URL", + "LANGBOT_LOCAL_AGENT_PIPELINE_NAME", + ]); + assert.equal(plan.automation_readiness.status, "ready"); +}); + +test("local-agent nonstreaming case disables stream output through automation defaults", () => { + const result = capture(() => commandTestRun(ctx([ + "test", + "run", + "local-agent-nonstreaming-debug-chat", + "--dry-run", + "--json", + ]))); + assert.equal(result.code, 0); + const run = JSON.parse(result.output); + assert.equal(run.automation.script, "scripts/e2e/pipeline-debug-chat.mjs"); + assert.equal(run.automation.env_defaults.LANGBOT_E2E_PROMPT, "Reply only NONSTREAM_OK."); + assert.equal(run.automation.env_defaults.LANGBOT_E2E_EXPECTED_TEXT, "NONSTREAM_OK"); + assert.equal(run.automation.env_defaults.LANGBOT_E2E_STREAM_OUTPUT, "0"); + assert.equal(run.automation.pipeline_env_required, true); +}); + +test("local-agent multimodal case exposes image fixture automation defaults", () => { + const result = capture(() => commandTestRun(ctx([ + "test", + "run", + "local-agent-multimodal-debug-chat", + "--dry-run", + "--json", + ]))); + assert.equal(result.code, 0); + const run = JSON.parse(result.output); + assert.equal(run.automation.script, "scripts/e2e/pipeline-debug-chat.mjs"); + assert.equal(run.automation.env_defaults.LANGBOT_E2E_EXPECTED_TEXT, "IMAGE_OK"); + assert.match(run.automation.env_defaults.LANGBOT_E2E_IMAGE_BASE64_PATH, /red-square\.png\.base64$/); + assert.equal(run.automation.pipeline_env_required, true); +}); + +test("MCP stdio case passes case-specific failure signals to automation defaults", () => { + const result = capture(() => commandTestRun(ctx([ + "test", + "run", + "mcp-stdio-tool-call", + "--dry-run", + "--json", + ]))); + assert.equal(result.code, 0); + const run = JSON.parse(result.output); + assert.match(run.automation.env_defaults.LANGBOT_E2E_FAILURE_SIGNALS, /qa-plugin-smoke:mcp-ok-local-agent/); + assert.match(run.automation.env_defaults.LANGBOT_E2E_FAILURE_SIGNALS, /model_not_found/); +}); + +test("MCP stdio tool-call case setups pipeline and registered MCP server", () => { + const result = capture(() => commandTestRun(ctx([ + "test", + "run", + "mcp-stdio-tool-call", + "--dry-run", + "--json", + ]))); + assert.equal(result.code, 0); + const run = JSON.parse(result.output); + assert.deepEqual(run.setup_automation.map((item: { entry: string }) => item.entry), [ + "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env", + "case:mcp-stdio-register", + ]); + + const planResult = capture(() => commandTestPlan(ctx(["test", "plan", "mcp-stdio-tool-call", "--json"]))); + assert.equal(planResult.code, 0); + const plan = JSON.parse(planResult.output); + assert.deepEqual(plan.setup_provides_env, [ + "LANGBOT_LOCAL_AGENT_PIPELINE_URL", + "LANGBOT_LOCAL_AGENT_PIPELINE_NAME", + ]); + assert.ok(!plan.preconditions.some((item: string) => item.includes("points to the local-agent pipeline"))); +}); + +test("generic pipeline automation can still use the shared pipeline env", () => { + const result = capture(() => commandTestRun(ctx([ + "test", + "run", + "pipeline-debug-chat", + "--dry-run", + "--json", + ]))); + assert.equal(result.code, 0); + const run = JSON.parse(result.output); + assert.equal(run.automation.pipeline_env_required, false); + assert.deepEqual(run.automation.env_aliases, []); + assert.ok(run.automation.required_env.includes("LANGBOT_PIPELINE_URL|LANGBOT_PIPELINE_NAME")); +}); + +test("AgentRunner live install case exposes package automation defaults", () => { + const result = capture(() => commandTestRun(ctx([ + "test", + "run", + "agent-runner-live-install", + "--dry-run", + "--json", + ]))); + assert.equal(result.code, 0); + const run = JSON.parse(result.output); + assert.equal( + run.automation.env_defaults.LANGBOT_E2E_PLUGIN_PACKAGE, + "skills/langbot-testing/fixtures/plugins/qa-agent-runner/dist/qa-agent-runner-0.1.0.lbpkg", + ); + assert.equal(run.automation.env_defaults.LANGBOT_E2E_EXPECTED_PLUGIN_ID, "qa/agent-runner"); + assert.equal(run.automation.env_defaults.LANGBOT_E2E_EXPECTED_RUNNER_ID, "plugin:qa/agent-runner/default"); +}); + +test("QA plugin live install checks the fixture package before installed state", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-install-qa-plugin-")); + try { + const result = spawnSync( + process.execPath, + [join(root, "scripts/e2e/install-qa-plugin-smoke.mjs")], + { + cwd: root, + env: { + ...process.env, + LBS_RUN_ID: "missing-package", + LBS_EVIDENCE_DIR: join(tmp, "evidence"), + LANGBOT_BACKEND_URL: "http://127.0.0.1:59999", + LANGBOT_E2E_LOGIN_USER: "qa@example.test", + LANGBOT_E2E_PLUGIN_PACKAGE: join(tmp, "missing.lbpkg"), + }, + encoding: "utf8", + }, + ); + assert.equal(result.status, 1); + const output = JSON.parse(result.stdout); + assert.match(output.reason, /missing\.lbpkg/); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("AgentRunner QA Debug Chat case uses dedicated pipeline env", () => { + const result = capture(() => commandTestRun(ctx([ + "test", + "run", + "agent-runner-qa-debug-chat", + "--dry-run", + "--json", + ]))); + assert.equal(result.code, 0); + const run = JSON.parse(result.output); + assert.equal(run.automation.script, "scripts/e2e/pipeline-debug-chat.mjs"); + assert.equal(run.automation.pipeline_env_required, true); + assert.equal(run.automation.env_defaults.LANGBOT_E2E_EXPECTED_RUNNER_ID, "plugin:qa/agent-runner/default"); + assert.deepEqual( + run.setup_automation.map((item: { entry: string }) => item.entry), + [ + "case:agent-runner-live-install", + "node:scripts/e2e/ensure-qa-agent-runner-pipeline.mjs --write-env", + ], + ); + assert.ok(run.automation.env_aliases.some((alias: { target: string; source: string }) => ( + alias.target === "LANGBOT_E2E_PIPELINE_URL" && alias.source === "LANGBOT_QA_AGENT_RUNNER_PIPELINE_URL" + ))); +}); + +test("AgentRunner QA Debug Chat setup automation removes manual readiness", () => { + const planResult = capture(() => commandTestPlan(ctx(["test", "plan", "agent-runner-qa-debug-chat", "--json"]))); + assert.equal(planResult.code, 0); + const plan = JSON.parse(planResult.output); + assert.equal(plan.manual_readiness.status, "not_required"); + assert.deepEqual(plan.setup_provides_env, [ + "LANGBOT_QA_AGENT_RUNNER_PIPELINE_URL", + "LANGBOT_QA_AGENT_RUNNER_PIPELINE_NAME", + ]); + assert.equal(plan.automation_readiness.status, "ready"); + + const suiteResult = capture(() => commandSuitePlan(ctx(["suite", "plan", "agent-runner-release-gate", "--json"]))); + assert.equal(suiteResult.code, 0); + const suite = JSON.parse(suiteResult.output); + assert.ok(!suite.readiness.manual_check_cases.includes("agent-runner-qa-debug-chat")); +}); + +test("ACP AgentRunner Debug Chat case setups the ACP pipeline env", () => { + const result = capture(() => commandTestRun(ctx([ + "test", + "run", + "acp-agent-runner-debug-chat", + "--dry-run", + "--json", + ]))); + assert.equal(result.code, 0); + const run = JSON.parse(result.output); + assert.deepEqual(run.setup_automation.map((item: { entry: string }) => item.entry), [ + "node:scripts/e2e/ensure-acp-agent-runner-pipeline.mjs --write-env", + ]); + assert.ok(run.automation.env_aliases.some((alias: { target: string; source: string }) => ( + alias.target === "LANGBOT_E2E_PIPELINE_URL" && alias.source === "LANGBOT_ACP_AGENT_RUNNER_PIPELINE_URL" + ))); + + const planResult = capture(() => commandTestPlan(ctx(["test", "plan", "acp-agent-runner-debug-chat", "--json"]))); + assert.equal(planResult.code, 0); + const plan = JSON.parse(planResult.output); + assert.deepEqual(plan.setup_provides_env, [ + "LANGBOT_ACP_AGENT_RUNNER_PIPELINE_URL", + "LANGBOT_ACP_AGENT_RUNNER_PIPELINE_NAME", + ]); + assert.ok(!plan.preconditions.some((item: string) => item.includes("pipeline AI runner"))); +}); + +test("local-agent plugin cases setup the QA plugin smoke fixture", () => { + const result = capture(() => commandTestRun(ctx([ + "test", + "run", + "local-agent-plugin-tool-call-debug-chat", + "--dry-run", + "--json", + ]))); + assert.equal(result.code, 0); + const run = JSON.parse(result.output); + assert.deepEqual(run.setup_automation.map((item: { entry: string }) => item.entry), [ + "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env", + "case:qa-plugin-smoke-live-install", + ]); +}); + +test("local-agent RAG case only requires the KB fixture env", () => { + const result = capture(() => commandTestRun(ctx([ + "test", + "run", + "local-agent-rag-debug-chat", + "--dry-run", + "--json", + ]))); + assert.equal(result.code, 0); + const run = JSON.parse(result.output); + assert.ok(run.automation.required_env.includes("LANGBOT_LOCAL_AGENT_RAG_KB_UUID")); + assert.ok(!run.automation.required_env.includes("LANGBOT_LOCAL_AGENT_RAG_TEXT_MODEL_UUID")); + assert.equal( + run.automation.env_defaults.LANGBOT_E2E_RUNNER_CONFIG_PATCH_JSON, + JSON.stringify({ + "knowledge-bases": [ + loadEnv(root).LANGBOT_LOCAL_AGENT_RAG_KB_UUID || "", + ], + }), + ); +}); + +test("LangRAG retrieve readiness requires a KB UUID alternative", () => { + const result = capture(() => commandTestPlan(ctx(["test", "plan", "langrag-kb-retrieve", "--json"]))); + assert.equal(result.code, 0); + const plan = JSON.parse(result.output); + assert.ok(plan.automation_readiness.required.includes("LANGBOT_LOCAL_AGENT_RAG_KB_UUID|LANGBOT_RAG_KB_UUID")); +}); + +test("local-agent RAG multimodal case setups the KB fixture env", () => { + const result = capture(() => commandTestRun(ctx([ + "test", + "run", + "local-agent-rag-multimodal-debug-chat", + "--dry-run", + "--json", + ]))); + assert.equal(result.code, 0); + const run = JSON.parse(result.output); + assert.ok(run.automation.required_env.includes("LANGBOT_LOCAL_AGENT_RAG_KB_UUID")); + assert.equal( + run.automation.env_defaults.LANGBOT_E2E_RUNNER_CONFIG_PATCH_JSON, + JSON.stringify({ + "knowledge-bases": [ + loadEnv(root).LANGBOT_LOCAL_AGENT_RAG_KB_UUID || "", + ], + }), + ); + assert.deepEqual(run.setup_automation.map((item: { entry: string }) => item.entry), [ + "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env", + "node:scripts/e2e/ensure-langrag-sentinel-kb.mjs --write-env", + ]); +}); + +test("test report renders a reusable evidence template", () => { + const result = capture(() => commandTestReport(ctx(["test", "report", "pipeline-debug-chat", "--no-auto-log"]))); + assert.equal(result.code, 0); + assert.match(result.output, /^# Test Report: pipeline-debug-chat/m); + assert.match(result.output, /result: pass \| fail \| blocked \| env_issue \| flaky/); + assert.match(result.output, /## Log Guard/); + assert.match(result.output, /## Automation Result/); + assert.match(result.output, /## Required Evidence/); + assert.match(result.output, /no log files provided/); +}); + +test("validate rejects dangling case references and missing automation scripts", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-validate-strict-")); + try { + const schemasDir = join(tmp, "schemas"); + const skillsDir = join(tmp, "skills"); + const envSetupDir = join(skillsDir, "langbot-env-setup"); + const testingDir = join(skillsDir, "langbot-testing"); + mkdirSync(schemasDir, { recursive: true }); + mkdirSync(join(testingDir, "cases"), { recursive: true }); + mkdirSync(join(testingDir, "fixtures"), { recursive: true }); + mkdirSync(join(testingDir, "suites"), { recursive: true }); + mkdirSync(envSetupDir, { recursive: true }); + for (const schemaName of ["case.schema.json", "suite.schema.json", "troubleshooting.schema.json", "skill-index.schema.json"]) { + writeFileSync(join(schemasDir, schemaName), "{}"); + } + writeFileSync(join(envSetupDir, "SKILL.md"), "---\nname: langbot-env-setup\ndescription: Env setup.\n---\n\n# Env\n"); + writeFileSync(join(testingDir, "SKILL.md"), "---\nname: langbot-testing\ndescription: Testing.\n---\n\n# Testing\n"); + writeFileSync( + join(skillsDir, ".env"), + [ + "LANGBOT_FRONTEND_URL=http://127.0.0.1:3000", + "LANGBOT_BACKEND_URL=http://127.0.0.1:5300", + "LANGBOT_DEV_FRONTEND_URL=http://127.0.0.1:3000", + "LANGBOT_REPO=/tmp/langbot", + "LANGBOT_WEB_REPO=/tmp/langbot/web", + "LANGBOT_BROWSER_PROFILE=/tmp/browser", + "LANGBOT_CHROMIUM_EXECUTABLE=/tmp/chromium", + "LANGBOT_PROXY_HTTP=http://127.0.0.1:7890", + "LANGBOT_PROXY_SOCKS=socks5://127.0.0.1:7890", + "LANGBOT_NO_PROXY=localhost,127.0.0.1,::1", + ].join("\n"), + ); + writeFileSync( + join(testingDir, "cases", "bad.yaml"), + [ + "id: bad", + "title: Bad", + "mode: agent-browser", + "area: pipeline", + "type: smoke", + "priority: p9", + "risk: medium", + "ci_eligible: false", + "tags:", + " - smoke", + "skills:", + " - langbot-env-setup", + " - langbot-testing", + "env:", + " - LANGBOT_FRONTEND_URL", + "automation: scripts/e2e/missing.mjs", + "setup_provides_env:", + " - LANGBOT_PIPELINE_URL", + "steps:", + " - Open UI.", + "checks:", + " - UI works.", + "evidence_required:", + " - ui", + "troubleshooting:", + " - missing-trouble", + ].join("\n"), + ); + for (const [id, target] of [["cycle-a", "cycle-b"], ["cycle-b", "cycle-a"]]) { + writeFileSync( + join(testingDir, "cases", `${id}.yaml`), + [ + `id: ${id}`, + `title: ${id}`, + "mode: probe", + "area: qa", + "type: smoke", + "priority: p2", + "risk: low", + "ci_eligible: true", + "tags:", + " - smoke", + "skills:", + " - langbot-testing", + "setup_automation:", + ` - \"case:${target}\"`, + "steps:", + " - Run probe.", + "checks:", + " - Probe works.", + "evidence_required:", + " - filesystem", + ].join("\n"), + ); + } + writeFileSync( + join(testingDir, "suites", "bad-suite.yaml"), + [ + "id: bad-suite", + "title: Bad Suite", + "description: Bad suite for strict validation.", + "type: release_gate", + "priority: p1", + "tags:", + " - gate", + "cases:", + " - missing-case", + ].join("\n"), + ); + writeFileSync( + join(testingDir, "fixtures", "fixtures.json"), + JSON.stringify([{ id: "bad-fixture", title: "Bad Fixture", path: "fixtures/missing.txt", related_cases: ["missing-case"] }]), + ); + + const result = captureAll(() => commandValidate(tmp)); + + assert.equal(result.code, 1); + assert.match(result.error, /priority/); + assert.match(result.error, /missing-trouble/); + assert.match(result.error, /missing-case/); + assert.match(result.error, /bad-fixture/); + assert.match(result.error, /automation script does not exist/); + assert.match(result.error, /setup_provides_env/); + assert.match(result.error, /setup_automation case cycle detected/); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("test report JSON scans logs and redacts secrets", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-report-")); + try { + const logPath = join(tmp, "backend.log"); + writeFileSync( + logPath, + [ + "INFO request started", + "Action invoke_llm_stream call timed out", + "Traceback (most recent call last):", + "API_KEY=sk-test-secret", + ].join("\n"), + ); + + const result = capture(() => commandTestReport(ctx(["test", "report", "pipeline-debug-chat", "--backend-log", logPath, "--json"]))); + assert.equal(result.code, 0); + assert.doesNotMatch(result.output, /sk-test-secret/); + + const report = JSON.parse(result.output); + assert.equal(report.log_guard.status, "fail"); + assert.ok(report.log_guard.findings.some((finding: { kind: string }) => ( + finding.kind === "case_failure_pattern" + ))); + assert.ok(report.log_guard.findings.some((finding: { troubleshooting_id?: string }) => ( + finding.troubleshooting_id === "plugin-runtime-timeout" + ))); + assert.ok(report.log_guard.findings.some((finding: { kind: string }) => finding.kind === "python_traceback")); + + const secretFinding = report.log_guard.findings.find((finding: { kind: string }) => finding.kind === "secret_leak"); + assert.ok(secretFinding); + assert.match(secretFinding.excerpt, /\[redacted\]/); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("test report does not treat invalid api key wording as a secret leak", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-report-api-key-wording-")); + try { + const logPath = join(tmp, "backend.log"); + writeFileSync( + logPath, + "RequesterError: 模型请求失败: 无效的 api-key: Error code: 401 - invalid api key\n", + ); + + const result = capture(() => commandTestReport(ctx(["test", "report", "mcp-stdio-tool-call", "--backend-log", logPath, "--json"]))); + assert.equal(result.code, 0); + assert.match(result.output, /api-key: Error code/); + + const report = JSON.parse(result.output); + assert.ok(!report.log_guard.findings.some((finding: { kind: string }) => finding.kind === "secret_leak")); + assert.ok(report.log_guard.findings.some((finding: { troubleshooting_id?: string }) => ( + finding.troubleshooting_id === "local-agent-model-route-unavailable" + ))); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("test report records declared success signals from logs", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-report-success-")); + try { + const logPath = join(tmp, "backend.log"); + writeFileSync( + logPath, + [ + "[05-21 10:31:00.000] websocket.py (1) - [INFO] : Processing request from person_websocket", + "[05-21 10:31:01.000] runner.py (2) - [INFO] : Conversation(0) Streaming completed", + ].join("\n"), + ); + + const result = capture(() => commandTestReport(ctx([ + "test", + "report", + "pipeline-debug-chat", + "--backend-log", + logPath, + "--json", + ]))); + assert.equal(result.code, 0); + const report = JSON.parse(result.output); + assert.equal(report.log_guard.status, "pass"); + assert.equal(report.log_guard.success_signals.length, 2); + assert.ok(report.log_guard.success_signals.some((signal: { pattern: string }) => ( + signal.pattern === "Streaming completed" + ))); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("test report warns when declared success signals are missing", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-report-missing-success-")); + try { + const logPath = join(tmp, "backend.log"); + writeFileSync(logPath, "INFO request started\nINFO request ended\n"); + + const result = capture(() => commandTestReport(ctx([ + "test", + "report", + "pipeline-debug-chat", + "--backend-log", + logPath, + "--json", + ]))); + assert.equal(result.code, 0); + const report = JSON.parse(result.output); + assert.equal(report.log_guard.status, "warning"); + assert.ok(report.log_guard.findings.some((finding: { kind: string }) => ( + finding.kind === "missing_success_signal" + ))); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("test report can limit log guard to tail lines", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-report-tail-")); + try { + const logPath = join(tmp, "backend.log"); + writeFileSync( + logPath, + [ + "ERROR old failure outside scan window", + "INFO middle", + "Action invoke_llm_stream call timed out", + "API_KEY=sk-tail-secret", + ].join("\n"), + ); + + const result = capture(() => commandTestReport(ctx([ + "test", + "report", + "pipeline-debug-chat", + "--backend-log", + logPath, + "--tail-lines", + "2", + "--json", + ]))); + assert.equal(result.code, 0); + const report = JSON.parse(result.output); + assert.equal(report.log_guard.scan.mode, "tail-lines"); + assert.equal(report.log_guard.scan.tail_lines, 2); + assert.equal(report.log_guard.sources[0].line_count, 2); + assert.equal(report.log_guard.sources[0].start_line, 3); + assert.ok(report.log_guard.findings.some((finding: { troubleshooting_id?: string }) => ( + finding.troubleshooting_id === "plugin-runtime-timeout" + ))); + assert.ok(!report.log_guard.findings.some((finding: { kind: string; excerpt?: string }) => ( + finding.kind === "error_log" && finding.excerpt?.includes("old failure") + ))); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("test report can limit log guard with since timestamp", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-report-since-")); + try { + const logPath = join(tmp, "backend.log"); + writeFileSync( + logPath, + [ + "[05-21 09:59:00.000] old.py (1) - [ERROR] : old failure outside scan window", + "[05-21 10:31:00.000] runner.py (2) - [ERROR] : Action invoke_llm_stream call timed out", + "Traceback continuation should stay with the matching timestamp block", + "[05-21 10:32:00.000] secrets.py (3) - [INFO] : API_KEY=sk-since-secret", + ].join("\n"), + ); + + const result = capture(() => commandTestReport(ctx([ + "test", + "report", + "pipeline-debug-chat", + "--backend-log", + logPath, + "--since", + "2026-05-21T10:30:00+08:00", + "--json", + ]))); + assert.equal(result.code, 0); + const report = JSON.parse(result.output); + assert.equal(report.log_guard.scan.mode, "since"); + assert.equal(report.log_guard.sources[0].line_count, 3); + assert.equal(report.log_guard.sources[0].start_line, 2); + assert.equal(report.log_guard.sources[0].timestamped_line_count, 3); + assert.ok(report.log_guard.findings.some((finding: { line?: number; troubleshooting_id?: string }) => ( + finding.line === 2 && finding.troubleshooting_id === "plugin-runtime-timeout" + ))); + assert.ok(!report.log_guard.findings.some((finding: { excerpt?: string }) => finding.excerpt?.includes("old failure"))); + assert.doesNotMatch(result.output, /sk-since-secret/); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("test report can limit log guard with since and until timestamps", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-report-window-")); + try { + const logPath = join(tmp, "backend.log"); + writeFileSync( + logPath, + [ + "[05-21 10:29:59.000] old.py (1) - [ERROR] : old failure outside scan window", + "[05-21 10:31:00.000] runner.py (2) - [INFO] : Processing request from person_websocket", + "[05-21 10:31:01.000] runner.py (3) - [INFO] : Conversation(0) Streaming completed", + "[05-21 10:32:01.000] later.py (4) - [ERROR] : later failure outside scan window", + ].join("\n"), + ); + + const result = capture(() => commandTestReport(ctx([ + "test", + "report", + "pipeline-debug-chat", + "--backend-log", + logPath, + "--since", + "2026-05-21T10:30:00+08:00", + "--until", + "2026-05-21T10:32:00+08:00", + "--json", + ]))); + assert.equal(result.code, 0); + const report = JSON.parse(result.output); + assert.equal(report.log_guard.scan.mode, "since+until"); + assert.equal(report.log_guard.sources[0].line_count, 2); + assert.equal(report.log_guard.sources[0].start_line, 2); + assert.equal(report.log_guard.sources[0].end_line, 3); + assert.equal(report.log_guard.status, "pass"); + assert.ok(!report.log_guard.findings.some((finding: { excerpt?: string }) => finding.excerpt?.includes("old failure"))); + assert.ok(!report.log_guard.findings.some((finding: { excerpt?: string }) => finding.excerpt?.includes("later failure"))); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("test report classifies model route failures as env_issue", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-report-env-issue-")); + try { + const logPath = join(tmp, "backend.log"); + writeFileSync( + logPath, + "[05-21 10:31:00.000] runner.py (2) - [ERROR] : runner.llm_error model_not_found no available channel for model gpt-test\n", + ); + + const result = capture(() => commandTestReport(ctx([ + "test", + "report", + "local-agent-plugin-tool-call-debug-chat", + "--backend-log", + logPath, + "--json", + ]))); + assert.equal(result.code, 0); + const report = JSON.parse(result.output); + assert.equal(report.log_guard.status, "env_issue"); + assert.ok(report.log_guard.findings.some((finding: { severity?: string; troubleshooting_id?: string }) => ( + finding.severity === "env_issue" && finding.troubleshooting_id === "local-agent-model-route-unavailable" + ))); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("test report infers scan window from automation result evidence", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-report-evidence-window-")); + try { + const evidenceDir = join(tmp, "evidence", "run-123"); + mkdirSync(evidenceDir, { recursive: true }); + const consoleLog = join(evidenceDir, "console.log"); + writeFileSync( + consoleLog, + [ + "[05-21 10:29:59.000] old.js (1) - [ERROR] : old failure outside scan window", + "[05-21 10:31:00.000] runner.js (2) - [INFO] : Processing request from person_websocket", + "[05-21 10:31:01.000] runner.js (3) - [INFO] : Conversation(0) Streaming completed", + "[05-21 10:32:01.000] later.js (4) - [ERROR] : later failure outside scan window", + ].join("\n"), + ); + writeFileSync( + join(evidenceDir, "automation-result.json"), + JSON.stringify({ + source: "automation", + status: "pass", + reason: "UI sentinel appeared.", + started_at_local: "2026-05-21T10:30:00.000+08:00", + finished_at_local: "2026-05-21T10:32:00.000+08:00", + }), + ); + + const result = capture(() => commandTestReport(ctx([ + "test", + "report", + "pipeline-debug-chat", + "--console-log", + consoleLog, + "--no-auto-log", + "--json", + ]))); + assert.equal(result.code, 0); + const report = JSON.parse(result.output); + assert.equal(report.log_guard.scan.mode, "since+until"); + assert.equal(report.log_guard.scan.since, "2026-05-21T10:30:00.000+08:00"); + assert.equal(report.log_guard.scan.until, "2026-05-21T10:32:00.000+08:00"); + assert.equal(report.log_guard.sources[0].line_count, 2); + assert.equal(report.log_guard.status, "pass"); + assert.equal(report.automation_result.status, "loaded"); + assert.equal(report.automation_result.result, "pass"); + assert.equal(report.automation_result.reason, "UI sentinel appeared."); + assert.ok(!report.log_guard.findings.some((finding: { excerpt?: string }) => finding.excerpt?.includes("old failure"))); + assert.ok(!report.log_guard.findings.some((finding: { excerpt?: string }) => finding.excerpt?.includes("later failure"))); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("test report does not treat final result as automation evidence", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-report-final-result-")); + try { + const evidenceDir = join(tmp, "evidence", "run-final"); + mkdirSync(evidenceDir, { recursive: true }); + const consoleLog = join(evidenceDir, "console.log"); + writeFileSync(consoleLog, "[05-21 10:31:00.000] ui.js (1) - [INFO] : opened\n"); + writeFileSync( + join(evidenceDir, "result.json"), + JSON.stringify({ + source: "final", + status: "pass", + reason: "Final manual decision.", + started_at_local: "2026-05-21T10:30:00.000+08:00", + finished_at_local: "2026-05-21T10:32:00.000+08:00", + evidence_collected: ["ui", "screenshot", "console"], + }), + ); + + const result = capture(() => commandTestReport(ctx([ + "test", + "report", + "webui-login-state", + "--console-log", + consoleLog, + "--no-auto-log", + "--json", + ]))); + + assert.equal(result.code, 0); + const report = JSON.parse(result.output); + assert.equal(report.automation_result.status, "not_provided"); + assert.match(report.automation_result.reason, /only final result\.json is present/); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("test report still scans untimestamped explicit console evidence within an inferred run window", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-report-untimestamped-console-")); + try { + const evidenceDir = join(tmp, "evidence", "run-untimestamped"); + mkdirSync(evidenceDir, { recursive: true }); + const consoleLog = join(evidenceDir, "console.log"); + writeFileSync(consoleLog, "[error] Uncaught TypeError: Cannot read properties of undefined\n"); + writeFileSync( + join(evidenceDir, "result.json"), + JSON.stringify({ + status: "pass", + reason: "UI sentinel appeared.", + started_at_local: "2026-05-21T10:30:00.000+08:00", + finished_at_local: "2026-05-21T10:32:00.000+08:00", + }), + ); + + const result = capture(() => commandTestReport(ctx([ + "test", + "report", + "webui-login-state", + "--console-log", + consoleLog, + "--no-auto-log", + "--json", + ]))); + + assert.equal(result.code, 0); + const report = JSON.parse(result.output); + assert.equal(report.log_guard.scan.mode, "since+until"); + assert.equal(report.log_guard.sources[0].timestamped_line_count, 0); + assert.ok(report.log_guard.sources[0].line_count >= 1); + assert.equal(report.log_guard.status, "fail"); + assert.ok(report.log_guard.findings.some((finding: { kind: string }) => ( + finding.kind === "frontend_uncaught_error" + ))); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("test report can write markdown to an output path", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-report-output-")); + try { + const output = join(tmp, "reports", "pipeline-debug-chat.md"); + const result = capture(() => commandTestReport(ctx(["test", "report", "pipeline-debug-chat", "--output", output]))); + assert.equal(result.code, 0); + assert.match(result.output, /pipeline-debug-chat\.md$/); + assert.match(readFileSync(output, "utf8"), /^# Test Report: pipeline-debug-chat/m); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("log scan reuses case-aware log guard patterns", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-log-scan-")); + try { + const logPath = join(tmp, "backend.log"); + writeFileSync( + logPath, + [ + "[05-21 10:31:00.000] process.py (1) - [INFO] : Processing request from person_websocket", + "[05-21 10:31:01.000] chat.py (2) - [INFO] : Conversation(0) Streaming completed", + "[05-21 10:31:02.000] runner.py (3) - [ERROR] : Action invoke_llm_stream call timed out", + ].join("\n"), + ); + + const result = capture(() => commandLogScan(ctx([ + "log", + "scan", + "--backend-log", + logPath, + "--case", + "pipeline-debug-chat", + "--json", + ]))); + + assert.equal(result.code, 0); + const report = JSON.parse(result.output); + assert.equal(report.status, "fail"); + assert.ok(report.success_signals.some((signal: { pattern: string }) => signal.pattern === "Streaming completed")); + assert.ok(report.findings.some((finding: { kind: string }) => finding.kind === "case_failure_pattern")); + + const strict = capture(() => commandLogScan(ctx([ + "log", + "scan", + "--backend-log", + logPath, + "--case", + "pipeline-debug-chat", + "--strict", + "--json", + ]))); + assert.equal(strict.code, 1); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("log guard start and stop bound a QA log window", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-log-guard-")); + try { + const logPath = join(tmp, "backend.log"); + const outputDir = join(tmp, "guards"); + writeFileSync(logPath, "INFO before guard\n"); + + const start = capture(() => commandLogGuard(ctx([ + "log", + "guard", + "start", + "--run-id", + "qa-run", + "--output-dir", + outputDir, + "--backend-log", + logPath, + "--case", + "pipeline-debug-chat", + "--json", + ]))); + assert.equal(start.code, 0); + const session = JSON.parse(start.output); + assert.equal(session.run_id, "qa-run"); + assert.ok(existsSync(join(outputDir, "qa-run.json"))); + + appendFileSync(logPath, "Traceback (most recent call last):\n"); + const stop = capture(() => commandLogGuard(ctx([ + "log", + "guard", + "stop", + "--run-id", + "qa-run", + "--output-dir", + outputDir, + "--json", + ]))); + + assert.equal(stop.code, 1); + const report = JSON.parse(stop.output); + assert.equal(report.session.run_id, "qa-run"); + assert.equal(report.result.status, "fail"); + assert.ok(report.result.findings.some((finding: { kind: string }) => finding.kind === "python_traceback")); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("log watch observes appended LangBot backend lines", async () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-log-watch-")); + try { + const logPath = join(tmp, "backend.log"); + writeFileSync(logPath, "INFO existing line\n"); + + const watching = captureAsync(() => commandLogWatch(ctx([ + "log", + "watch", + "--backend-log", + logPath, + "--duration-ms", + "220", + "--interval-ms", + "20", + "--strict", + "--json", + ]))); + setTimeout(() => { + appendFileSync(logPath, "Traceback (most recent call last):\n"); + }, 50); + + const result = await watching; + assert.equal(result.code, 1); + const summary = JSON.parse(result.output); + assert.equal(summary.mode, "watch"); + assert.equal(summary.status, "fail"); + assert.ok(summary.bytes_read > 0); + assert.ok(summary.findings.some((finding: { kind: string }) => finding.kind === "python_traceback")); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); + +test("trouble search finds structured troubleshooting entries", () => { + const result = capture(() => commandTroubleSearch(ctx(["trouble", "search", "proxy"]))); + assert.equal(result.code, 0); + assert.match(result.output, /proxy-env-mismatch/); +}); + +test("env local overrides shared env defaults", () => { + const tmp = mkdtempSync(join(tmpdir(), "lbs-env-")); + try { + mkdirSync(join(tmp, "skills")); + writeFileSync(join(tmp, "skills", ".env"), "LANGBOT_REPO=/shared\nLANGBOT_BACKEND_URL=http://127.0.0.1:5300\n"); + writeFileSync(join(tmp, "skills", ".env.local"), "LANGBOT_REPO=/local\n"); + + assert.deepEqual(loadEnv(tmp), { + LANGBOT_REPO: "/local", + LANGBOT_BACKEND_URL: "http://127.0.0.1:5300", + }); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +}); diff --git a/src/langbot/pkg/api/http/controller/main.py b/src/langbot/pkg/api/http/controller/main.py index 2e366c3ef..835617e16 100644 --- a/src/langbot/pkg/api/http/controller/main.py +++ b/src/langbot/pkg/api/http/controller/main.py @@ -17,6 +17,7 @@ from .groups import platform as groups_platform from .groups import pipelines as groups_pipelines from .groups import knowledge as groups_knowledge from .groups import resources as groups_resources +from ...mcp.mount import MCPMount importutil.import_modules_in_pkg(groups) importutil.import_modules_in_pkg(groups_provider) @@ -39,6 +40,10 @@ class HTTPController: # Set maximum content length to prevent large file uploads self.quart_app.config['MAX_CONTENT_LENGTH'] = group.MAX_FILE_SIZE + # MCP server (mounted at /mcp, see ..mcp.mount). Built lazily in + # initialize() so the service layer is ready. + self.mcp_mount: MCPMount | None = None + async def initialize(self) -> None: # Register custom error handler for file size limit @self.quart_app.errorhandler(RequestEntityTooLarge) @@ -52,6 +57,12 @@ class HTTPController: await self.register_routes() + # Build the MCP server and start its session-manager lifespan in the + # background so the streamable-HTTP transport is ready to serve. + self.mcp_mount = MCPMount(self.ap) + await self.mcp_mount.start_session_manager() + self.ap.logger.info('LangBot MCP server mounted at /mcp (API-key authenticated).') + async def run(self) -> None: if True: @@ -61,7 +72,7 @@ class HTTPController: async def exception_handler(*args, **kwargs): try: - await self.quart_app.run_task(*args, **kwargs) + await self._run_task(*args, **kwargs) except Exception as e: self.ap.logger.error(f'Failed to start HTTP service: {e}') @@ -77,6 +88,28 @@ class HTTPController: # await asyncio.sleep(5) + async def _run_task(self, host: str, port: int, shutdown_trigger) -> None: + """Serve the Quart app, fronted by the MCP dispatcher at /mcp. + + Mirrors Quart.run_task() but wraps the ASGI app so MCP requests are + intercepted before Quart's router. Falls back to plain Quart if the + MCP mount failed to build for any reason. + """ + from hypercorn.config import Config as HyperConfig + from hypercorn.asyncio import serve as hypercorn_serve + + config = HyperConfig() + config.access_log_format = '%(h)s %(r)s %(s)s %(b)s %(D)s' + config.accesslog = '-' + config.bind = [f'{host}:{port}'] + config.errorlog = config.accesslog + + asgi_app = self.quart_app + if self.mcp_mount is not None: + asgi_app = self.mcp_mount.wrap(self.quart_app) + + await hypercorn_serve(asgi_app, config, shutdown_trigger=shutdown_trigger) + async def register_routes(self) -> None: @self.quart_app.route('/healthz') async def healthz(): diff --git a/src/langbot/pkg/api/http/service/apikey.py b/src/langbot/pkg/api/http/service/apikey.py index 5e6ff15d5..207254351 100644 --- a/src/langbot/pkg/api/http/service/apikey.py +++ b/src/langbot/pkg/api/http/service/apikey.py @@ -51,8 +51,25 @@ class ApiKeyService: return self.ap.persistence_mgr.serialize_model(apikey.ApiKey, key) async def verify_api_key(self, key: str) -> bool: - """Verify if an API key is valid""" - if not isinstance(key, str) or not key.startswith('lbk_'): + """Verify if an API key is valid. + + A key is accepted if it matches the global API key configured in + ``config.yaml`` (``api.global_api_key``) — which requires no login + session and no database record — or if it matches a key created via + the web UI (stored in the database, prefixed with ``lbk_``). + """ + if not isinstance(key, str) or not key: + return False + + # 1. Global API key from config.yaml (no DB lookup, no login state). + # Note: config completion only backfills top-level keys, so existing + # installs may not have this key — access it defensively. + global_api_key = self.ap.instance_config.data.get('api', {}).get('global_api_key', '') + if global_api_key and secrets.compare_digest(key, global_api_key): + return True + + # 2. Web-UI-created keys are stored in the database and prefixed lbk_. + if not key.startswith('lbk_'): return False result = await self.ap.persistence_mgr.execute_async( diff --git a/src/langbot/pkg/api/mcp/__init__.py b/src/langbot/pkg/api/mcp/__init__.py new file mode 100644 index 000000000..dac07bf4b --- /dev/null +++ b/src/langbot/pkg/api/mcp/__init__.py @@ -0,0 +1,14 @@ +"""LangBot MCP (Model Context Protocol) server. + +This package exposes a subset of LangBot's HTTP service API as MCP tools so +that external AI agents can manage a LangBot instance through the MCP +protocol. The MCP server reuses the same API-key authentication as the HTTP +API (including the global API key from ``config.yaml``). + +See ``server.py`` for the tool surface and ``mount.py`` for the ASGI +integration with the Quart HTTP app. +""" + +from .server import LangBotMCPServer + +__all__ = ['LangBotMCPServer'] diff --git a/src/langbot/pkg/api/mcp/mount.py b/src/langbot/pkg/api/mcp/mount.py new file mode 100644 index 000000000..d113b2501 --- /dev/null +++ b/src/langbot/pkg/api/mcp/mount.py @@ -0,0 +1,112 @@ +"""ASGI integration: serve the LangBot MCP server alongside the Quart HTTP app. + +The Quart app and the MCP server are both ASGI apps. We front them with a small +dispatcher ASGI callable: + +- Requests whose path is (or is under) ``/mcp`` are authenticated with a + LangBot API key (reusing ``apikey_service.verify_api_key``, which also + accepts the global API key from ``config.yaml``) and then handed to the + FastMCP Starlette app. +- Every other request goes to the Quart app unchanged. + +The FastMCP streamable-HTTP transport requires its session manager's lifespan +to be running. Rather than rely on the dispatcher receiving ASGI lifespan +events (Quart owns those), we explicitly run the session manager in a background +task managed by LangBot's task manager. +""" + +from __future__ import annotations + +import contextlib +import typing + +from .server import LangBotMCPServer + +if typing.TYPE_CHECKING: + from ...core import app as app_module + + +# JSON-RPC-ish 401 body returned before the MCP app is reached. +_UNAUTHORIZED_BODY = b'{"error":"unauthorized","message":"A valid LangBot API key is required for MCP access."}' + + +def _extract_api_key(headers: list[tuple[bytes, bytes]]) -> str: + """Pull an API key from ASGI headers (X-API-Key or Authorization: Bearer).""" + header_map = {k.lower(): v for k, v in headers} + api_key = header_map.get(b'x-api-key', b'').decode('latin-1').strip() + if api_key: + return api_key + auth = header_map.get(b'authorization', b'').decode('latin-1').strip() + if auth.lower().startswith('bearer '): + return auth[7:].strip() + return '' + + +class MCPMount: + """Owns the MCP server and produces the dispatcher ASGI app.""" + + MCP_PATH_PREFIX = '/mcp' + + def __init__(self, ap: app_module.Application) -> None: + self.ap = ap + self.server = LangBotMCPServer(ap) + self._mcp_asgi = self.server.streamable_http_app() + self._lifespan_cm: typing.Any = None + + async def start_session_manager(self) -> None: + """Run the MCP session manager lifespan in the background. + + StreamableHTTPSessionManager.run() is a one-shot async context manager + (it may only be entered once). We keep it open for the process lifetime; + it is torn down when the event loop stops. + """ + cm = self.server.session_manager.run() + self._lifespan_cm = cm + await cm.__aenter__() + + async def stop_session_manager(self) -> None: + if self._lifespan_cm is not None: + with contextlib.suppress(Exception): + await self._lifespan_cm.__aexit__(None, None, None) + self._lifespan_cm = None + + def _is_mcp_path(self, path: str) -> bool: + return path == self.MCP_PATH_PREFIX or path.startswith(self.MCP_PATH_PREFIX + '/') + + def wrap(self, quart_asgi: typing.Callable) -> typing.Callable: + """Return a dispatcher ASGI app fronting ``quart_asgi``.""" + mcp_asgi = self._mcp_asgi + verify_api_key = self.ap.apikey_service.verify_api_key + is_mcp_path = self._is_mcp_path + + async def dispatcher(scope, receive, send): # type: ignore[no-untyped-def] + # Pass through non-HTTP scopes (lifespan, websocket) to Quart so its + # own startup/shutdown and websocket routes keep working. + if scope['type'] != 'http' or not is_mcp_path(scope.get('path', '')): + await quart_asgi(scope, receive, send) + return + + # Authenticate MCP HTTP requests with a LangBot API key. + api_key = _extract_api_key(scope.get('headers', [])) + authorized = False + if api_key: + with contextlib.suppress(Exception): + authorized = await verify_api_key(api_key) + + if not authorized: + await send( + { + 'type': 'http.response.start', + 'status': 401, + 'headers': [ + (b'content-type', b'application/json'), + (b'www-authenticate', b'Bearer'), + ], + } + ) + await send({'type': 'http.response.body', 'body': _UNAUTHORIZED_BODY}) + return + + await mcp_asgi(scope, receive, send) + + return dispatcher diff --git a/src/langbot/pkg/api/mcp/server.py b/src/langbot/pkg/api/mcp/server.py new file mode 100644 index 000000000..95630bbaf --- /dev/null +++ b/src/langbot/pkg/api/mcp/server.py @@ -0,0 +1,204 @@ +"""LangBot MCP server definition. + +Wraps a curated subset of LangBot's HTTP service API as MCP tools. Tools call +the existing service layer directly (not the HTTP API over the network), so the +MCP surface stays aligned with the API by construction. + +IMPORTANT: when you add, remove, or change an HTTP API endpoint that should be +agent-accessible, update the corresponding MCP tool here AND the skills under +``skills/`` (see AGENTS.md). The MCP tool surface and the API must stay aligned. + +Scope (first version): core read operations plus the most common writes for +bots, pipelines, LLM/embedding models, knowledge bases, MCP servers, skills, +and read-only system info. This intentionally does NOT expose every one of the +~25 HTTP route groups — that keeps the agent surface small, safe, and +maintainable. Extend deliberately. +""" + +from __future__ import annotations + +import json +import typing + +from mcp.server.fastmcp import FastMCP + +if typing.TYPE_CHECKING: + from ...core import app as app_module + + +INSTRUCTIONS = """\ +This MCP server manages a LangBot instance. LangBot is an LLM-native instant +messaging bot platform. Use these tools to inspect and manage bots, pipelines, +models, knowledge bases, MCP servers, and skills. + +Authentication uses a LangBot API key (web-UI-created `lbk_...` key or the +global API key from config.yaml), passed as the `X-API-Key` header or +`Authorization: Bearer `. + +Prefer the `list_*` / `get_*` tools to discover resources before mutating. All +identifiers are UUIDs unless noted. Mutating tools take JSON objects matching +the same shape as the LangBot HTTP API request bodies. +""" + + +def _dump(value: typing.Any) -> str: + """Serialize a tool result to a compact JSON string for the agent.""" + return json.dumps(value, ensure_ascii=False, default=str) + + +class LangBotMCPServer: + """Builds and owns the FastMCP instance for LangBot.""" + + def __init__(self, ap: app_module.Application) -> None: + self.ap = ap + # Stateless HTTP so the server does not need sticky sessions behind a + # load balancer; json_response keeps responses simple (no SSE stream + # required for unary tool calls). + self.mcp = FastMCP( + name='LangBot', + instructions=INSTRUCTIONS, + stateless_http=True, + json_response=True, + ) + self._register_tools() + + # ------------------------------------------------------------------ # + # Tool registration + # ------------------------------------------------------------------ # + def _register_tools(self) -> None: + ap = self.ap + mcp = self.mcp + + # ----- System (read-only) -------------------------------------- # + @mcp.tool(description='Get basic LangBot system/runtime information (version, edition).') + async def get_system_info() -> str: + version = None + try: + version = ap.ver_mgr.get_current_version() + except Exception: + pass + data = { + 'version': version, + 'edition': ap.instance_config.data.get('system', {}).get('edition'), + 'instance_id': ap.instance_config.data.get('system', {}).get('instance_id'), + } + return _dump(data) + + # ----- Bots ---------------------------------------------------- # + @mcp.tool(description='List all messaging-platform bots. Secrets are redacted.') + async def list_bots() -> str: + return _dump(await ap.bot_service.get_bots(include_secret=False)) + + @mcp.tool(description='Get a single bot by its UUID. Secrets are redacted.') + async def get_bot(bot_uuid: str) -> str: + return _dump(await ap.bot_service.get_bot(bot_uuid, include_secret=False)) + + @mcp.tool( + description=( + 'Create a bot. `bot_data` is a JSON object matching the LangBot ' + 'POST /api/v1/platform/bots body (e.g. name, adapter, config). ' + 'Returns the new bot UUID.' + ) + ) + async def create_bot(bot_data: dict) -> str: + return _dump({'uuid': await ap.bot_service.create_bot(bot_data)}) + + @mcp.tool(description='Update a bot by UUID. `bot_data` matches the PUT bot body.') + async def update_bot(bot_uuid: str, bot_data: dict) -> str: + await ap.bot_service.update_bot(bot_uuid, bot_data) + return _dump({'ok': True}) + + @mcp.tool(description='Delete a bot by UUID.') + async def delete_bot(bot_uuid: str) -> str: + await ap.bot_service.delete_bot(bot_uuid) + return _dump({'ok': True}) + + # ----- Pipelines ----------------------------------------------- # + @mcp.tool(description='List all pipelines.') + async def list_pipelines() -> str: + return _dump(await ap.pipeline_service.get_pipelines()) + + @mcp.tool(description='Get a single pipeline by UUID.') + async def get_pipeline(pipeline_uuid: str) -> str: + return _dump(await ap.pipeline_service.get_pipeline(pipeline_uuid)) + + @mcp.tool( + description=( + 'Create a pipeline. `pipeline_data` matches the LangBot POST ' + '/api/v1/pipelines body. Returns the new pipeline UUID.' + ) + ) + async def create_pipeline(pipeline_data: dict) -> str: + return _dump({'uuid': await ap.pipeline_service.create_pipeline(pipeline_data)}) + + @mcp.tool(description='Update a pipeline by UUID. `pipeline_data` matches the PUT body.') + async def update_pipeline(pipeline_uuid: str, pipeline_data: dict) -> str: + await ap.pipeline_service.update_pipeline(pipeline_uuid, pipeline_data) + return _dump({'ok': True}) + + @mcp.tool(description='Delete a pipeline by UUID.') + async def delete_pipeline(pipeline_uuid: str) -> str: + await ap.pipeline_service.delete_pipeline(pipeline_uuid) + return _dump({'ok': True}) + + # ----- Models -------------------------------------------------- # + @mcp.tool(description='List all configured LLM models. Secrets are redacted.') + async def list_llm_models() -> str: + return _dump(await ap.llm_model_service.get_llm_models(include_secret=False)) + + @mcp.tool(description='Get a single LLM model by UUID.') + async def get_llm_model(model_uuid: str) -> str: + return _dump(await ap.llm_model_service.get_llm_model(model_uuid)) + + @mcp.tool(description='List all configured embedding models.') + async def list_embedding_models() -> str: + return _dump(await ap.embedding_models_service.get_embedding_models()) + + @mcp.tool(description='List all model providers (OpenAI-compatible, Anthropic, etc.).') + async def list_model_providers() -> str: + return _dump(await ap.provider_service.get_providers()) + + # ----- Knowledge bases ----------------------------------------- # + @mcp.tool(description='List all knowledge bases (RAG).') + async def list_knowledge_bases() -> str: + return _dump(await ap.knowledge_service.get_knowledge_bases()) + + @mcp.tool(description='Get a single knowledge base by UUID.') + async def get_knowledge_base(kb_uuid: str) -> str: + return _dump(await ap.knowledge_service.get_knowledge_base(kb_uuid)) + + @mcp.tool( + description=('Retrieve (semantic search) from a knowledge base. Returns the matched chunks for `query`.') + ) + async def retrieve_knowledge_base(kb_uuid: str, query: str) -> str: + return _dump(await ap.knowledge_service.retrieve_knowledge_base(kb_uuid, query)) + + # ----- MCP servers (LangBot as MCP client) --------------------- # + @mcp.tool( + description=( + 'List external MCP servers registered in LangBot (the servers LangBot itself connects to as a client).' + ) + ) + async def list_mcp_servers() -> str: + return _dump(await ap.mcp_service.get_mcp_servers()) + + # ----- Skills -------------------------------------------------- # + @mcp.tool(description='List installed skills.') + async def list_skills() -> str: + return _dump(await ap.skill_service.list_skills()) + + @mcp.tool(description='Get a single skill by name.') + async def get_skill(skill_name: str) -> str: + return _dump(await ap.skill_service.get_skill(skill_name)) + + # ------------------------------------------------------------------ # + # ASGI app + # ------------------------------------------------------------------ # + def streamable_http_app(self): # type: ignore[no-untyped-def] + """Return the Starlette ASGI app serving MCP over streamable HTTP at /mcp.""" + return self.mcp.streamable_http_app() + + @property + def session_manager(self): # type: ignore[no-untyped-def] + """Expose the session manager so its lifespan can be run by the host.""" + return self.mcp.session_manager diff --git a/src/langbot/templates/config.yaml b/src/langbot/templates/config.yaml index c0002e8f6..24874fa6a 100644 --- a/src/langbot/templates/config.yaml +++ b/src/langbot/templates/config.yaml @@ -3,6 +3,12 @@ api: port: 5300 webhook_prefix: 'http://127.0.0.1:5300' extra_webhook_prefix: '' + # Global API key for the HTTP service API and the MCP server. When set to a + # non-empty string, this key is accepted anywhere a web-UI-created API key is + # accepted (X-API-Key header or "Authorization: Bearer "), WITHOUT any + # login session and without a database record. Leave empty to disable. + # Keep this value secret; only enable it on trusted/internal deployments. + global_api_key: '' command: enable: true prefix: diff --git a/tests/manual/mcp_smoke.py b/tests/manual/mcp_smoke.py new file mode 100644 index 000000000..197724fff --- /dev/null +++ b/tests/manual/mcp_smoke.py @@ -0,0 +1,132 @@ +"""End-to-end test: boot the real MCPMount on a port and drive it with an MCP client. + +Exercises the ASGI dispatcher (auth + /mcp routing), the FastMCP streamable-HTTP +transport, and a real tool call against the (mocked) service layer. + +Run: uv run --no-sync python tests/manual/mcp_smoke.py +""" + +from __future__ import annotations + +import asyncio +import contextlib +from types import SimpleNamespace +from unittest.mock import AsyncMock + +from hypercorn.asyncio import serve +from hypercorn.config import Config +from quart import Quart + +from langbot.pkg.api.mcp.mount import MCPMount + +PORT = 5399 +GLOBAL_KEY = 'test-global-key-123' + + +def build_ap() -> SimpleNamespace: + ap = SimpleNamespace() + ap.instance_config = SimpleNamespace( + data={'api': {'global_api_key': GLOBAL_KEY}, 'system': {'edition': 'community', 'instance_id': 'inst-1'}} + ) + ap.ver_mgr = SimpleNamespace(get_current_version=lambda: '4.5.0-test') + ap.logger = SimpleNamespace(info=print, error=print, warning=print) + + # API key verification: reuse real logic shape (global key match) + async def verify_api_key(key: str) -> bool: + return bool(key) and key == GLOBAL_KEY + + ap.apikey_service = SimpleNamespace(verify_api_key=verify_api_key) + ap.bot_service = SimpleNamespace( + get_bots=AsyncMock(return_value=[{'uuid': 'bot-1', 'name': 'Demo Bot', 'adapter': 'telegram'}]) + ) + ap.pipeline_service = SimpleNamespace(get_pipelines=AsyncMock(return_value=[{'uuid': 'pl-1', 'name': 'default'}])) + ap.llm_model_service = SimpleNamespace(get_llm_models=AsyncMock(return_value=[])) + ap.embedding_models_service = SimpleNamespace(get_embedding_models=AsyncMock(return_value=[])) + ap.provider_service = SimpleNamespace(get_providers=AsyncMock(return_value=[])) + ap.knowledge_service = SimpleNamespace(get_knowledge_bases=AsyncMock(return_value=[])) + ap.mcp_service = SimpleNamespace(get_mcp_servers=AsyncMock(return_value=[])) + ap.skill_service = SimpleNamespace(list_skills=AsyncMock(return_value=[{'name': 'demo-skill'}])) + return ap + + +async def run_server(mount: MCPMount, shutdown: asyncio.Event) -> None: + quart_app = Quart(__name__) + + @quart_app.route('/healthz') + async def healthz(): + return {'code': 0, 'msg': 'ok'} + + config = Config() + config.bind = [f'127.0.0.1:{PORT}'] + config.accesslog = None + asgi = mount.wrap(quart_app) + await serve(asgi, config, shutdown_trigger=shutdown.wait) + + +async def main() -> int: + from mcp.client.session import ClientSession + from mcp.client.streamable_http import streamablehttp_client + + ap = build_ap() + mount = MCPMount(ap) + await mount.start_session_manager() + + shutdown = asyncio.Event() + server_task = asyncio.create_task(run_server(mount, shutdown)) + await asyncio.sleep(1.0) # let the server bind + + url = f'http://127.0.0.1:{PORT}/mcp' + failures = [] + + # 1. Unauthorized request is rejected. + import httpx + + async with httpx.AsyncClient() as client: + r = await client.post(url, json={'jsonrpc': '2.0', 'id': 1, 'method': 'ping'}) + if r.status_code != 401: + failures.append(f'expected 401 without key, got {r.status_code}') + else: + print('PASS: unauthorized request rejected (401)') + + # 2. Authorized MCP session: list tools + call two. + headers = {'X-API-Key': GLOBAL_KEY} + async with streamablehttp_client(url, headers=headers) as (read, write, _): + async with ClientSession(read, write) as session: + await session.initialize() + tools = await session.list_tools() + names = [t.name for t in tools.tools] + print(f'PASS: listed {len(names)} tools') + for required in ('list_bots', 'get_system_info', 'list_skills'): + if required not in names: + failures.append(f'missing tool {required}') + + res = await session.call_tool('list_bots', {}) + text = res.content[0].text if res.content else '' + if 'Demo Bot' not in text: + failures.append(f'list_bots did not return expected data: {text!r}') + else: + print('PASS: list_bots returned bot data') + + res2 = await session.call_tool('get_system_info', {}) + text2 = res2.content[0].text if res2.content else '' + if '4.5.0-test' not in text2: + failures.append(f'get_system_info wrong: {text2!r}') + else: + print('PASS: get_system_info returned version') + + shutdown.set() + with contextlib.suppress(Exception): + await asyncio.wait_for(server_task, timeout=5) + await mount.stop_session_manager() + + if failures: + print('\nFAILURES:') + for f in failures: + print(' -', f) + return 1 + print('\nALL MCP SMOKE CHECKS PASSED') + return 0 + + +if __name__ == '__main__': + raise SystemExit(asyncio.run(main())) diff --git a/tests/unit_tests/api/service/test_apikey_service.py b/tests/unit_tests/api/service/test_apikey_service.py index 287a21bad..9726888eb 100644 --- a/tests/unit_tests/api/service/test_apikey_service.py +++ b/tests/unit_tests/api/service/test_apikey_service.py @@ -255,16 +255,22 @@ class TestApiKeyServiceGetApiKey: class TestApiKeyServiceVerifyApiKey: """Tests for verify_api_key method.""" + @staticmethod + def _make_ap(db_key=None, global_api_key=''): + """Build a mock Application with persistence + instance_config.""" + ap = SimpleNamespace() + ap.persistence_mgr = SimpleNamespace() + mock_result = Mock() + mock_result.first = Mock(return_value=db_key) + ap.persistence_mgr.execute_async = AsyncMock(return_value=mock_result) + ap.instance_config = SimpleNamespace(data={'api': {'global_api_key': global_api_key}}) + return ap + async def test_verify_api_key_valid(self): """Returns True for valid API key.""" # Setup - ap = SimpleNamespace() - ap.persistence_mgr = SimpleNamespace() - key = Mock(spec=ApiKey) - mock_result = Mock() - mock_result.first = Mock(return_value=key) - ap.persistence_mgr.execute_async = AsyncMock(return_value=mock_result) + ap = self._make_ap(db_key=key) service = ApiKeyService(ap) @@ -277,12 +283,7 @@ class TestApiKeyServiceVerifyApiKey: async def test_verify_api_key_invalid(self): """Returns False for invalid API key.""" # Setup - ap = SimpleNamespace() - ap.persistence_mgr = SimpleNamespace() - - mock_result = Mock() - mock_result.first = Mock(return_value=None) - ap.persistence_mgr.execute_async = AsyncMock(return_value=mock_result) + ap = self._make_ap(db_key=None) service = ApiKeyService(ap) @@ -295,12 +296,7 @@ class TestApiKeyServiceVerifyApiKey: async def test_verify_api_key_empty_string(self): """Returns False for empty key string.""" # Setup - ap = SimpleNamespace() - ap.persistence_mgr = SimpleNamespace() - - mock_result = Mock() - mock_result.first = Mock(return_value=None) - ap.persistence_mgr.execute_async = AsyncMock(return_value=mock_result) + ap = self._make_ap(db_key=None) service = ApiKeyService(ap) @@ -313,12 +309,7 @@ class TestApiKeyServiceVerifyApiKey: async def test_verify_api_key_unknown_key(self): """Returns False when the key is not present in persistence.""" # Setup - ap = SimpleNamespace() - ap.persistence_mgr = SimpleNamespace() - - mock_result = Mock() - mock_result.first = Mock(return_value=None) - ap.persistence_mgr.execute_async = AsyncMock(return_value=mock_result) + ap = self._make_ap(db_key=None) service = ApiKeyService(ap) @@ -328,6 +319,70 @@ class TestApiKeyServiceVerifyApiKey: # Verify assert result is False + async def test_verify_global_api_key_match(self): + """Returns True when key matches the config.yaml global API key (no DB lookup).""" + # Setup: no DB record, but a global key is configured + ap = self._make_ap(db_key=None, global_api_key='my-global-secret') + + service = ApiKeyService(ap) + + # Execute + result = await service.verify_api_key('my-global-secret') + + # Verify: accepted purely on config match + assert result is True + # DB should not have been consulted for the global-key path + ap.persistence_mgr.execute_async.assert_not_called() + + async def test_verify_global_api_key_no_prefix_required(self): + """Global API key is accepted even without the lbk_ prefix.""" + ap = self._make_ap(db_key=None, global_api_key='plainsecret123') + + service = ApiKeyService(ap) + + result = await service.verify_api_key('plainsecret123') + + assert result is True + + async def test_verify_global_api_key_mismatch_falls_back_to_db(self): + """A non-matching key still falls through to the DB lookup.""" + # Global key set, but request uses a different lbk_ key that IS in DB + key = Mock(spec=ApiKey) + ap = self._make_ap(db_key=key, global_api_key='my-global-secret') + + service = ApiKeyService(ap) + + result = await service.verify_api_key('lbk_db_key') + + assert result is True + ap.persistence_mgr.execute_async.assert_called_once() + + async def test_verify_empty_global_api_key_disabled(self): + """An empty global_api_key must never authenticate an empty/blank request.""" + ap = self._make_ap(db_key=None, global_api_key='') + + service = ApiKeyService(ap) + + # Empty request key is rejected, and a blank global key never matches + assert await service.verify_api_key('') is False + assert await service.verify_api_key(' ') is False + + async def test_verify_api_key_missing_global_config_key(self): + """Works even when api.global_api_key is absent (existing installs).""" + # instance_config without the global_api_key field at all + ap = SimpleNamespace() + ap.persistence_mgr = SimpleNamespace() + mock_result = Mock() + mock_result.first = Mock(return_value=None) + ap.persistence_mgr.execute_async = AsyncMock(return_value=mock_result) + ap.instance_config = SimpleNamespace(data={'api': {}}) + + service = ApiKeyService(ap) + + result = await service.verify_api_key('lbk_some_key') + + assert result is False + class TestApiKeyServiceDeleteApiKey: """Tests for delete_api_key method.""" diff --git a/tests/unit_tests/api/test_apikey_service.py b/tests/unit_tests/api/test_apikey_service.py index 67b6737ba..2065ae3b7 100644 --- a/tests/unit_tests/api/test_apikey_service.py +++ b/tests/unit_tests/api/test_apikey_service.py @@ -12,7 +12,8 @@ from langbot.pkg.api.http.service.apikey import ApiKeyService @pytest.mark.parametrize('api_key', [None, 123, b'lbk_bytes', '', 'plain_key', ' LBK_bad', 'sk-lbk_fake']) async def test_verify_api_key_rejects_non_lbk_keys_without_db_query(api_key): persistence_mgr = SimpleNamespace(execute_async=AsyncMock()) - service = ApiKeyService(SimpleNamespace(persistence_mgr=persistence_mgr)) + instance_config = SimpleNamespace(data={'api': {'global_api_key': ''}}) + service = ApiKeyService(SimpleNamespace(persistence_mgr=persistence_mgr, instance_config=instance_config)) result = await service.verify_api_key(api_key) @@ -32,7 +33,8 @@ async def test_verify_api_key_keeps_db_validation_for_lbk_keys(db_row, expected) query_result = Mock() query_result.first.return_value = db_row persistence_mgr = SimpleNamespace(execute_async=AsyncMock(return_value=query_result)) - service = ApiKeyService(SimpleNamespace(persistence_mgr=persistence_mgr)) + instance_config = SimpleNamespace(data={'api': {'global_api_key': ''}}) + service = ApiKeyService(SimpleNamespace(persistence_mgr=persistence_mgr, instance_config=instance_config)) result = await service.verify_api_key('lbk_valid_format') diff --git a/web/src/app/home/components/api-integration-dialog/ApiIntegrationPanel.tsx b/web/src/app/home/components/api-integration-dialog/ApiIntegrationPanel.tsx index 9cb14a696..7aa3634a2 100644 --- a/web/src/app/home/components/api-integration-dialog/ApiIntegrationPanel.tsx +++ b/web/src/app/home/components/api-integration-dialog/ApiIntegrationPanel.tsx @@ -86,6 +86,11 @@ export default function ApiIntegrationPanel({ ); const [copiedKey, setCopiedKey] = useState(null); + // MCP server endpoint, derived from the current origin. The backend serves + // the MCP server at /mcp on the same host/port as the HTTP API + web UI. + const mcpEndpoint = + typeof window !== 'undefined' ? `${window.location.origin}/mcp` : '/mcp'; + // 清理 body 样式，防止嵌套对话框关闭后页面无法交互 useEffect(() => { if (!deleteKeyId && !deleteWebhookId) { @@ -262,6 +267,7 @@ export default function ApiIntegrationPanel({ {t('common.apiKeys')} {t('common.webhooks')} + {t('common.mcpTab')} {activeTab === 'apikeys' ? ( - ) : ( + ) : activeTab === 'webhooks' ? ( - )} + ) : null} {/* API Keys Tab */} @@ -455,6 +461,71 @@ export default function ApiIntegrationPanel({ )} + + {/* MCP Tab */} + +

{t('common.mcpHint')}

+ +
+ +
+ + {mcpEndpoint} + + +
+
+ +
+ +

+ {t('common.mcpAuthDesc')} +

+ 
+              {`X-API-Key: 
+# or
+Authorization: Bearer `}
+
+

+ {t('common.mcpGlobalKeyNote')} +

+
+ +
+ + 
+              {`{
+  "mcpServers": {
+    "langbot": {
+      "url": "${mcpEndpoint}",
+      "headers": { "X-API-Key": "" }
+    }
+  }
+}`}
+
+
+ {/* Create API Key Dialog */} diff --git a/web/src/i18n/locales/en-US.ts b/web/src/i18n/locales/en-US.ts index 902c749b1..c4a79c589 100644 --- a/web/src/i18n/locales/en-US.ts +++ b/web/src/i18n/locales/en-US.ts @@ -140,6 +140,16 @@ const enUS = { noApiKeys: 'No API keys configured', apiKeyHint: 'API keys allow external systems to access LangBot Service APIs', + mcpTab: 'MCP', + mcpHint: + 'LangBot exposes an MCP (Model Context Protocol) server so AI agents can manage this instance. It uses the same API keys as the Service API.', + mcpEndpoint: 'MCP Endpoint', + mcpAuthTitle: 'Authentication', + mcpAuthDesc: + 'Authenticate with any API key from the API Keys tab, sent as a header:', + mcpGlobalKeyNote: + 'You can also set a global API key in config.yaml (api.global_api_key) to use without logging in.', + mcpClientConfigTitle: 'Example MCP client config', webhooks: 'Webhooks', createWebhook: 'Create Webhook', webhookName: 'Webhook Name', diff --git a/web/src/i18n/locales/es-ES.ts b/web/src/i18n/locales/es-ES.ts index b91486168..019b48f2d 100644 --- a/web/src/i18n/locales/es-ES.ts +++ b/web/src/i18n/locales/es-ES.ts @@ -144,6 +144,16 @@ const esES = { noApiKeys: 'No hay claves API configuradas', apiKeyHint: 'Las claves API permiten a sistemas externos acceder a las API del servicio LangBot', + mcpTab: 'MCP', + mcpHint: + 'LangBot exposes an MCP (Model Context Protocol) server so AI agents can manage this instance. It uses the same API keys as the Service API.', + mcpEndpoint: 'MCP Endpoint', + mcpAuthTitle: 'Authentication', + mcpAuthDesc: + 'Authenticate with any API key from the API Keys tab, sent as a header:', + mcpGlobalKeyNote: + 'You can also set a global API key in config.yaml (api.global_api_key) to use without logging in.', + mcpClientConfigTitle: 'Example MCP client config', webhooks: 'Webhooks', createWebhook: 'Crear Webhook', webhookName: 'Nombre del Webhook', diff --git a/web/src/i18n/locales/ja-JP.ts b/web/src/i18n/locales/ja-JP.ts index d61d65a01..898011b0b 100644 --- a/web/src/i18n/locales/ja-JP.ts +++ b/web/src/i18n/locales/ja-JP.ts @@ -142,6 +142,16 @@ const jaJP = { noApiKeys: 'API キーが設定されていません', apiKeyHint: 'API キーを使用すると、外部システムが LangBot Service API にアクセスできます', + mcpTab: 'MCP', + mcpHint: + 'LangBot は MCP（Model Context Protocol）サーバーを提供し、AI エージェントがこのインスタンスを管理できます。Service API と同じ API キーを使用します。', + mcpEndpoint: 'MCP エンドポイント', + mcpAuthTitle: '認証', + mcpAuthDesc: + '「API キー」タブの任意のキーを、ヘッダーで送信して認証します：', + mcpGlobalKeyNote: + 'config.yaml にグローバル API キー（api.global_api_key）を設定すると、ログインなしで使用できます。', + mcpClientConfigTitle: 'MCP クライアント設定の例', webhooks: 'Webhooks', createWebhook: 'Webhook を作成', webhookName: 'Webhook 名', diff --git a/web/src/i18n/locales/ru-RU.ts b/web/src/i18n/locales/ru-RU.ts index 988ab7ca6..8aec863fa 100644 --- a/web/src/i18n/locales/ru-RU.ts +++ b/web/src/i18n/locales/ru-RU.ts @@ -140,6 +140,16 @@ const ruRU = { noApiKeys: 'API-ключи не настроены', apiKeyHint: 'API-ключи позволяют внешним системам получать доступ к API сервисов LangBot', + mcpTab: 'MCP', + mcpHint: + 'LangBot exposes an MCP (Model Context Protocol) server so AI agents can manage this instance. It uses the same API keys as the Service API.', + mcpEndpoint: 'MCP Endpoint', + mcpAuthTitle: 'Authentication', + mcpAuthDesc: + 'Authenticate with any API key from the API Keys tab, sent as a header:', + mcpGlobalKeyNote: + 'You can also set a global API key in config.yaml (api.global_api_key) to use without logging in.', + mcpClientConfigTitle: 'Example MCP client config', webhooks: 'Вебхуки', createWebhook: 'Создать вебхук', webhookName: 'Имя вебхука', diff --git a/web/src/i18n/locales/th-TH.ts b/web/src/i18n/locales/th-TH.ts index b172e420a..d5c331f2e 100644 --- a/web/src/i18n/locales/th-TH.ts +++ b/web/src/i18n/locales/th-TH.ts @@ -138,6 +138,16 @@ const thTH = { noApiKeys: 'ยังไม่มีคีย์ API ที่กำหนดค่า', apiKeyHint: 'คีย์ API ช่วยให้ระบบภายนอกสามารถเข้าถึง API บริการของ LangBot ได้', + mcpTab: 'MCP', + mcpHint: + 'LangBot exposes an MCP (Model Context Protocol) server so AI agents can manage this instance. It uses the same API keys as the Service API.', + mcpEndpoint: 'MCP Endpoint', + mcpAuthTitle: 'Authentication', + mcpAuthDesc: + 'Authenticate with any API key from the API Keys tab, sent as a header:', + mcpGlobalKeyNote: + 'You can also set a global API key in config.yaml (api.global_api_key) to use without logging in.', + mcpClientConfigTitle: 'Example MCP client config', webhooks: 'Webhooks', createWebhook: 'สร้าง Webhook', webhookName: 'ชื่อ Webhook', diff --git a/web/src/i18n/locales/vi-VN.ts b/web/src/i18n/locales/vi-VN.ts index 504cfb512..d28f8a47a 100644 --- a/web/src/i18n/locales/vi-VN.ts +++ b/web/src/i18n/locales/vi-VN.ts @@ -141,6 +141,16 @@ const viVN = { noApiKeys: 'Chưa cấu hình khóa API nào', apiKeyHint: 'Khóa API cho phép các hệ thống bên ngoài truy cập API dịch vụ LangBot', + mcpTab: 'MCP', + mcpHint: + 'LangBot exposes an MCP (Model Context Protocol) server so AI agents can manage this instance. It uses the same API keys as the Service API.', + mcpEndpoint: 'MCP Endpoint', + mcpAuthTitle: 'Authentication', + mcpAuthDesc: + 'Authenticate with any API key from the API Keys tab, sent as a header:', + mcpGlobalKeyNote: + 'You can also set a global API key in config.yaml (api.global_api_key) to use without logging in.', + mcpClientConfigTitle: 'Example MCP client config', webhooks: 'Webhooks', createWebhook: 'Tạo Webhook', webhookName: 'Tên Webhook', diff --git a/web/src/i18n/locales/zh-Hans.ts b/web/src/i18n/locales/zh-Hans.ts index bb72de0fb..9fa90be9f 100644 --- a/web/src/i18n/locales/zh-Hans.ts +++ b/web/src/i18n/locales/zh-Hans.ts @@ -132,6 +132,15 @@ const zhHans = { apiKeyCopied: 'API 密钥已复制到剪贴板', noApiKeys: '暂无 API 密钥', apiKeyHint: 'API 密钥允许外部系统访问 LangBot 的 Service API', + mcpTab: 'MCP', + mcpHint: + 'LangBot 提供 MCP（Model Context Protocol）服务，让 AI Agent 管理本实例，复用与 Service API 相同的 API 密钥。', + mcpEndpoint: 'MCP 接入地址', + mcpAuthTitle: '鉴权方式', + mcpAuthDesc: '使用「API 密钥」标签页中的任意密钥，通过请求头传入：', + mcpGlobalKeyNote: + '也可在 config.yaml 中设置全局 API 密钥（api.global_api_key），无需登录即可使用。', + mcpClientConfigTitle: 'MCP 客户端配置示例', webhooks: 'Webhooks', createWebhook: '创建 Webhook', webhookName: 'Webhook 名称', diff --git a/web/src/i18n/locales/zh-Hant.ts b/web/src/i18n/locales/zh-Hant.ts index 9dcbb0851..09568d6b9 100644 --- a/web/src/i18n/locales/zh-Hant.ts +++ b/web/src/i18n/locales/zh-Hant.ts @@ -132,6 +132,15 @@ const zhHant = { apiKeyCopied: 'API 金鑰已複製到剪貼簿', noApiKeys: '暫無 API 金鑰', apiKeyHint: 'API 金鑰允許外部系統訪問 LangBot 的 Service API', + mcpTab: 'MCP', + mcpHint: + 'LangBot 提供 MCP（Model Context Protocol）服務，讓 AI Agent 管理本實例，複用與 Service API 相同的 API 金鑰。', + mcpEndpoint: 'MCP 接入位址', + mcpAuthTitle: '驗證方式', + mcpAuthDesc: '使用「API 金鑰」標籤頁中的任意金鑰，透過請求標頭傳入：', + mcpGlobalKeyNote: + '也可在 config.yaml 中設定全域 API 金鑰（api.global_api_key），無需登入即可使用。', + mcpClientConfigTitle: 'MCP 用戶端設定範例', webhooks: 'Webhooks', createWebhook: '建立 Webhook', webhookName: 'Webhook 名稱',