diff --git a/docs/agent-runner-pluginization/STATUS.md b/docs/agent-runner-pluginization/STATUS.md index 4692c5bf1..17a7f1f03 100644 --- a/docs/agent-runner-pluginization/STATUS.md +++ b/docs/agent-runner-pluginization/STATUS.md @@ -15,7 +15,7 @@ | Result payload validation | Done | Wire 保持 `{type, data}`；Host 对投递/副作用类 payload 严格校验，tool-call telemetry 宽松，未知 type 忽略并 warning。 | | Old built-in runners | Done | 旧 `src/langbot/pkg/provider/runners/*` 与 `RequestRunner` 路径已从本分支删除。 | | Official runner manifests | Done | `local-agent`、ACP / Claude Code / Codex 外部 harness runner、外部服务 runner 已重新声明真实生效的 LangBot resource permissions。 | -| Skill 链路 | Unit-pass; WebUI E2E pass | 已按 **skill 全 tool 化** 收敛：发现走 `list_skills` / `langbot_list_assets` 和 skill resources；`activate` / `register_skill` 走统一 tool 授权；`skill_authoring` capability 降级为便捷开关。`activate` 会 best-effort 写入 conversation-scope `host.activated_skills`，后续 run 通过当前 pipeline-visible skill cache 恢复。新注册 Skill 在当前 Query 内立即获得临时可见性；Docker `exec` 产生的宿主侧不可写文件由 `write` / `edit` 回退到 Box 执行。2026-07-15 真实 LocalAgent Debug Chat 已完成创建、注册、同 Query 激活、编辑和执行闭环。 | +| Skill 链路 | Unit-pass; WebUI E2E pass | 已按 **skill 全 tool 化** 收敛：发现走 `list_skills` / `langbot_list_assets` 和 skill resources；`activate` / `register_skill` 走统一 tool 授权；`skill_authoring` capability 降级为便捷开关。`activate` 会 best-effort 写入 conversation-scope `host.activated_skills`，后续 run 通过当前 pipeline-visible skill cache 恢复。新注册 Skill 在当前 Query 内立即获得临时可见性；Docker `exec` 产生的宿主侧不可写文件由 `write` / `edit` 回退到 Box 执行。2026-07-15 真实 LocalAgent Debug Chat 已完成创建、注册、同 Query 激活、编辑和执行闭环；非流式 runner turn 只向下游 Pipeline 产出一次，工具中间结果不再拆成额外 Bot 气泡。 | | Runtime Control Plane v2 foundation | Partial | Host-owned `AgentRun` / `AgentRunEvent` ledger、orchestrator 自动建账、result event persistence、run get/list/event page/cancel/append/finalize actions 已落地；`agent_run:admin` / `runtime:admin` 控制权限、最小 runtime register/heartbeat/list/reconcile 和 run claim/renew/release 原语已落地。完整 Agent Platform 产品形态、daemon supervisor、任务唤醒/长轮询/WebSocket、分布式 runtime 管控仍未完成。 | | Security boundary | Done | 当前口径降级为轻量边界：LangBot 保护自身持有资源；external harness 的 OS / process / network / workspace 风险由用户或部署环境承担；managed sandbox 不是当前承诺。 | | Steering control path | Done | claim 异常不再逃逸 consumer loop；queue 有上限；未 pull 的 claimed 输入在 run 结束时写 `steering.dropped` 审计终态。 | @@ -37,7 +37,7 @@ | Runner | 状态 | 最近证据 | | --- | --- | --- | -| `plugin:langbot-team/LocalAgent/default` | Unit-pass; Marketplace UI pass; Debug Chat E2E pass | 2026-07-12 隔离 first-run 实例从真实 AgentRunner catalog 安装 `langbot-team/LocalAgent` 0.1.0，Host 注册 `plugin:langbot-team/LocalAgent/default`，Wizard 自动选中并解锁后续操作。2026-07-15 `local-agent-functional-20260715-skill-edit-final` 使用真实 `gpt-5.5` 完成 Skill 创建、注册、同 Query 激活、已激活包编辑与脚本执行；三阶段 UI、浏览器诊断和结构化文件系统检查全部通过，p95 18.8 秒、错误率 0。 | +| `plugin:langbot-team/LocalAgent/default` | Unit-pass; Marketplace UI pass; Debug Chat E2E pass | 2026-07-12 隔离 first-run 实例从真实 AgentRunner catalog 安装 `langbot-team/LocalAgent` 0.1.0，Host 注册 `plugin:langbot-team/LocalAgent/default`，Wizard 自动选中并解锁后续操作。2026-07-15 `2026-07-15-08-44-10-770-08-00-sandbox-skill-authoring-edit-existing-e2e` 使用真实 `gpt-5.5` 完成 Skill 创建、注册、同 Query 激活、已激活包编辑与脚本执行；三阶段 UI、浏览器诊断和结构化文件系统检查全部通过，每阶段恰好新增一个 Bot 气泡，p95 14.6 秒、错误率 0。 | | `plugin:langbot-team/ACPAgentRunner/default` / `plugin:langbot-team/ClaudeCodeAgent/default` / `plugin:langbot-team/CodexAgent/default` | Unit-pass; E2E pending | 通过 runner 仓库单测覆盖 session、run_id 注入和 LangBot MCP gateway；真实 harness E2E 取决于对应运行环境、CLI/daemon 可用性和 provider 登录态。 | | Dify / n8n / Coze / DashScope / Langflow / Tbox / DeerFlow / WeKnora | Unit-pass; credential smoke optional | 2026-06-13 plugin layout / parser tests 通过；真实服务凭据 smoke 非每轮必跑。 | @@ -46,7 +46,7 @@ | 范围 | 状态 | 最近证据 | | --- | --- | --- | | LangBot Runtime Control Plane v2 foundation | Unit-pass; EBA release gate 5/5 pass | 2026-07-12 `eba-functional-20260712-release-gate-rerun` 通过 Quick Start 场景筛选、隔离实例 Runner Marketplace 安装、Runner 健康状态、事件路由 dry-run / 合成派发，以及真实 OneBot `group.member_joined` → Agent → `send_group_msg` 链路。 | -| Host Skill / native tool integration | Unit-pass; WebUI E2E pass | 2026-07-15 provider / native / Skill / monitoring 定向测试 67 项通过，Skills CLI 104 项通过；真实 Debug Chat 验证 `register_skill` 后同 Query `activate` 成功，监控工具调用不再把 SQL 行误取为字符串，结构化 JSON 文件检查不依赖格式空格。 | +| Host Skill / native tool integration | Unit-pass; WebUI E2E pass | 2026-07-15 provider / native / Skill / monitoring 定向测试 67 项通过，Pipeline / Chat / Wrapper 定向测试 61 项通过，Skills CLI 105 项通过；真实 Debug Chat 验证 `register_skill` 后同 Query `activate` 成功，监控工具调用不再把 SQL 行误取为字符串，结构化 JSON 文件检查不依赖格式空格，非流式多阶段 runner 结果只生成一个最终 Bot 气泡。 | | SDK AgentRunner control entities / proxy | Unit-pass | 2026-06-23 SDK `tests/api/entities/builtin/agent_runner`、`tests/api/proxies`、`tests/api/test_agent_tools_mcp_bridge.py`、`tests/runtime/plugin/test_mgr_agent_runner.py`、`tests/runtime/test_pull_api_handlers.py`、`tests/runtime/io/handlers/test_plugin_handler.py`、EBA event entities 和 message tests 通过，覆盖 typed entities、AgentRunAPIProxy、MCP bridge、runtime manager 与 pull API handlers。 | ## 历史高价值记录 diff --git a/skills/scripts/e2e/lib/debug-chat.mjs b/skills/scripts/e2e/lib/debug-chat.mjs index 7d44d00c9..3f2903410 100755 --- a/skills/scripts/e2e/lib/debug-chat.mjs +++ b/skills/scripts/e2e/lib/debug-chat.mjs @@ -91,6 +91,7 @@ export function classifyDebugChatResult({ beforeMessages = null, afterMessages = null, latestAssistantText = "", + maxNewAssistantMessages = null, failureSignals = DEBUG_CHAT_FAILURE_SIGNALS, }) { const minExpectedCount = minExpectedOccurrences(beforeText, expectedText, prompt); @@ -107,6 +108,20 @@ export function classifyDebugChatResult({ const assistantExpectedIncreased = hasMessageEvidence ? afterAssistantExpectedCount > beforeAssistantExpectedCount : false; + const beforeAssistantMessageCount = hasMessageEvidence + ? beforeMessages.filter((message) => message.role === "assistant").length + : null; + const afterAssistantMessageCount = hasMessageEvidence + ? afterMessages.filter((message) => message.role === "assistant").length + : null; + const newAssistantMessageCount = hasMessageEvidence + ? afterAssistantMessageCount - beforeAssistantMessageCount + : null; + const assistantMessageEvidence = { + before_assistant_message_count: beforeAssistantMessageCount, + after_assistant_message_count: afterAssistantMessageCount, + new_assistant_message_count: newAssistantMessageCount, + }; if (hasMessageEvidence) { const latestAssistantFailure = findFailureSignalInText(latestAssistantText, failureSignals); @@ -119,6 +134,18 @@ export function classifyDebugChatResult({ failure_signal: latestAssistantFailure, before_assistant_expected_count: beforeAssistantExpectedCount, after_assistant_expected_count: afterAssistantExpectedCount, + ...assistantMessageEvidence, + }; + } + if (maxNewAssistantMessages !== null && newAssistantMessageCount > maxNewAssistantMessages) { + return { + status: "fail", + reason: `Debug Chat created ${newAssistantMessageCount} assistant messages; expected at most ${maxNewAssistantMessages}.`, + min_expected_count: minExpectedCount, + final_count: finalCount, + before_assistant_expected_count: beforeAssistantExpectedCount, + after_assistant_expected_count: afterAssistantExpectedCount, + ...assistantMessageEvidence, }; } if (assistantExpectedIncreased && String(latestAssistantText).includes(expectedText)) { @@ -129,6 +156,7 @@ export function classifyDebugChatResult({ final_count: finalCount, before_assistant_expected_count: beforeAssistantExpectedCount, after_assistant_expected_count: afterAssistantExpectedCount, + ...assistantMessageEvidence, }; } if (failureText) { @@ -140,6 +168,7 @@ export function classifyDebugChatResult({ failure_signal: failureText, before_assistant_expected_count: beforeAssistantExpectedCount, after_assistant_expected_count: afterAssistantExpectedCount, + ...assistantMessageEvidence, }; } return { @@ -149,6 +178,7 @@ export function classifyDebugChatResult({ final_count: finalCount, before_assistant_expected_count: beforeAssistantExpectedCount, after_assistant_expected_count: afterAssistantExpectedCount, + ...assistantMessageEvidence, }; } if (failureText) { @@ -345,7 +375,14 @@ export async function sendDebugChatPrompt(page, prompt, imagePath = "") { return true; } -export async function runDebugChatPrompt(page, { prompt, expectedText, responseTimeoutMs, imagePath = "", failureSignals = DEBUG_CHAT_FAILURE_SIGNALS }) { +export async function runDebugChatPrompt(page, { + prompt, + expectedText, + responseTimeoutMs, + imagePath = "", + maxNewAssistantMessages = null, + failureSignals = DEBUG_CHAT_FAILURE_SIGNALS, +}) { const beforeText = await bodyText(page); const beforeMessages = await visibleDebugChatMessages(page); const minExpectedCount = minExpectedOccurrences(beforeText, expectedText, prompt); @@ -380,6 +417,7 @@ export async function runDebugChatPrompt(page, { prompt, expectedText, responseT beforeMessages, afterMessages, latestAssistantText, + maxNewAssistantMessages, failureSignals, }); } diff --git a/skills/scripts/e2e/pipeline-debug-chat.mjs b/skills/scripts/e2e/pipeline-debug-chat.mjs index fffd303c0..a66590b70 100755 --- a/skills/scripts/e2e/pipeline-debug-chat.mjs +++ b/skills/scripts/e2e/pipeline-debug-chat.mjs @@ -989,6 +989,7 @@ try { expectedText: step.expectedText, responseTimeoutMs: step.responseTimeoutMs, imagePath: index === 0 ? imagePath : "", + maxNewAssistantMessages: streamOutput === false ? 1 : null, failureSignals: failureSignals.length > 0 ? failureSignals : undefined, }); const promptDurationMs = Date.now() - promptStartedAt; @@ -1002,6 +1003,9 @@ try { final_count: chatResult.final_count, before_assistant_expected_count: chatResult.before_assistant_expected_count, after_assistant_expected_count: chatResult.after_assistant_expected_count, + before_assistant_message_count: chatResult.before_assistant_message_count, + after_assistant_message_count: chatResult.after_assistant_message_count, + new_assistant_message_count: chatResult.new_assistant_message_count, failure_signal: chatResult.failure_signal || "", }); result.status = chatResult.status; diff --git a/skills/test/lbs-cli.test.ts b/skills/test/lbs-cli.test.ts index 221f2b2a5..84a2b4001 100644 --- a/skills/test/lbs-cli.test.ts +++ b/skills/test/lbs-cli.test.ts @@ -2006,6 +2006,33 @@ test("debug chat classifier passes when expected text appears in a new assistant assert.match(result.reason, /new assistant message/); assert.equal(result.before_assistant_expected_count, 0); assert.equal(result.after_assistant_expected_count, 1); + assert.equal(result.new_assistant_message_count, 1); +}); + +test("debug chat classifier rejects split non-streaming assistant messages", () => { + const expectedText = "E2E_OK:skill"; + const result = classifyDebugChatResult({ + beforeText: "", + afterText: `Bot: intermediate one\nBot: intermediate two\nBot: ${expectedText}`, + expectedText, + prompt: "Run a tool and return the result", + latestExpectedLeaf: expectedText, + latestFailureLeaf: "", + beforeMessages: [], + afterMessages: [ + { role: "assistant", text: "intermediate one" }, + { role: "assistant", text: "intermediate two" }, + { role: "assistant", text: expectedText }, + ], + latestAssistantText: expectedText, + maxNewAssistantMessages: 1, + }); + + assert.equal(result.status, "fail"); + assert.match(result.reason, /created 3 assistant messages/); + assert.equal(result.before_assistant_message_count, 0); + assert.equal(result.after_assistant_message_count, 3); + assert.equal(result.new_assistant_message_count, 3); }); test("debug chat classifier allows a recovered failure when latest assistant is successful", () => { diff --git a/src/langbot/pkg/pipeline/process/handlers/chat.py b/src/langbot/pkg/pipeline/process/handlers/chat.py index 51ea2aa04..7454a6cf0 100644 --- a/src/langbot/pkg/pipeline/process/handlers/chat.py +++ b/src/langbot/pkg/pipeline/process/handlers/chat.py @@ -113,10 +113,12 @@ class ChatMessageHandler(handler.MessageHandler): # Create a single resp_message_id for the entire streaming response resp_message_id = uuid.uuid4() chunk_count = 0 + has_result = False # Use AgentRunOrchestrator to run the agent # This replaces direct runner lookup and PluginAgentRunnerWrapper async for result in self.ap.agent_run_orchestrator.run_from_query(query): + has_result = True result.resp_message_id = str(resp_message_id) # For streaming mode, pop previous response before adding new chunk @@ -156,7 +158,8 @@ class ChatMessageHandler(handler.MessageHandler): if result.content is not None: text_length += len(result.content) - yield entities.StageProcessResult(result_type=entities.ResultType.CONTINUE, new_query=query) + if is_stream: + yield entities.StageProcessResult(result_type=entities.ResultType.CONTINUE, new_query=query) # Log final summary after streaming completes if is_stream: @@ -170,6 +173,12 @@ class ChatMessageHandler(handler.MessageHandler): # history source for this path. await self._ensure_conversation_for_history(query) + # Non-streaming adapters should receive only the completed runner turn. + # Keep every normalized result on the query for diagnostics and history, + # while running downstream response stages once with the final result. + if not is_stream and has_result: + yield entities.StageProcessResult(result_type=entities.ResultType.CONTINUE, new_query=query) + except Exception as e: # Import orchestrator errors for specific handling from ....agent.runner.errors import ( diff --git a/tests/unit_tests/agent/test_chat_handler.py b/tests/unit_tests/agent/test_chat_handler.py index db50984d1..62cc892df 100644 --- a/tests/unit_tests/agent/test_chat_handler.py +++ b/tests/unit_tests/agent/test_chat_handler.py @@ -384,8 +384,8 @@ class TestChatHandlerAsyncBehavior: assert query.resp_messages[0].content == 'Hello World!' @pytest.mark.asyncio - async def test_non_streaming_no_pop(self): - """Non-streaming mode: all chunks should remain.""" + async def test_non_streaming_keeps_results_and_yields_once(self): + """Non-streaming mode keeps runner results but runs downstream stages once.""" from langbot.pkg.pipeline.process.handlers.chat import ChatMessageHandler from langbot.pkg.pipeline import entities @@ -424,6 +424,7 @@ class TestChatHandlerAsyncBehavior: assert len(query.resp_messages) == 2 assert query.resp_messages[0].content == 'Response 1' assert query.resp_messages[1].content == 'Response 2' + assert len(results) == 1 @pytest.mark.asyncio async def test_agent_turn_recreates_conversation_if_tool_resets_it(self):