test(agent-runner): strengthen local agent e2e gate

skills/skills/langbot-testing/cases/local-agent-combo-rag-compaction-tool-debug-chat.yaml

@@ -0,0 +1,103 @@
+id: local-agent-combo-rag-compaction-tool-debug-chat
+title: "Local Agent preserves RAG, compacted history, and plugin tool result in one Debug Chat run"
+mode: agent-browser
+area: pipeline
+type: regression
+priority: p1
+risk: high
+ci_eligible: false
+tags:
+  - local-agent
+  - langrag
+  - context
+  - compaction
+  - plugin
+  - tools
+  - pipeline
+skills:
+  - langbot-env-setup
+  - langbot-testing
+env:
+  - LANGBOT_FRONTEND_URL
+  - LANGBOT_BACKEND_URL
+  - LANGBOT_LOCAL_AGENT_PIPELINE_URL
+  - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+env_optional:
+  - LANGBOT_E2E_FAKE_PROVIDER_BASE_URL
+automation: scripts/e2e/pipeline-debug-chat.mjs
+automation_env:
+  - LANGBOT_FRONTEND_URL
+  - LANGBOT_BACKEND_URL
+  - LANGBOT_BROWSER_PROFILE
+  - LANGBOT_CHROMIUM_EXECUTABLE
+  - LANGBOT_LOCAL_AGENT_PIPELINE_URL
+  - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+  - LANGBOT_LOCAL_AGENT_RAG_KB_UUID
+automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL
+automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+automation_expected_runner_id: "plugin:langbot/local-agent/default"
+automation_runner_config_patch_json: '{"knowledge-bases":["${LANGBOT_LOCAL_AGENT_RAG_KB_UUID}"],"retrieval-top-k":1,"context-window-tokens":650,"context-reserve-tokens":180,"context-keep-recent-tokens":120,"context-summary-tokens":220,"max-tool-iterations":4,"tool-execution-mode":"serial"}'
+automation_extensions_patch_json: '{"enable_all_plugins":false,"bound_plugins":[{"author":"langbot","name":"local-agent"},{"author":"qa","name":"plugin-smoke"}],"enable_all_mcp_servers":false,"bound_mcp_servers":[],"enable_all_skills":false,"bound_skills":[]}'
+automation_restore_runner_config: "1"
+automation_restore_extensions: "1"
+automation_reset_debug_chat: "1"
+automation_debug_chat_session_type: "person"
+automation_expected_text: "COMBO_FINAL qa_combo_compaction_sentinel_2406 azalea-cobalt-7421 qa-plugin-smoke:combo-tool-ok-local-agent"
+automation_response_timeout_ms: "180000"
+automation_prompts_json: '[{"prompt":"请记住这个用于 local-agent 组合回归测试的暗号：qa_combo_compaction_sentinel_2406。请只回复 MEMORY_SET。","expected_text":"MEMORY_SET","response_timeout_ms":"180000"},{"prompt":"下面这轮只用于制造长历史压力，目标标记 RAG_TOOL_COMBO_GOAL 必须被压缩保留。请忽略填充内容，最后只回复 COMBO_CONTEXT_PRESSURE_READY。填充片段 C001 combo context padding. C002 combo context padding. C003 combo context padding. C004 combo context padding. C005 combo context padding. C006 combo context padding. C007 combo context padding. C008 combo context padding. C009 combo context padding. C010 combo context padding. C011 combo context padding. C012 combo context padding. C013 combo context padding. C014 combo context padding. C015 combo context padding. C016 combo context padding. C017 combo context padding. C018 combo context padding. C019 combo context padding. C020 combo context padding. C021 combo context padding. C022 combo context padding. C023 combo context padding. C024 combo context padding. C025 combo context padding. C026 combo context padding. C027 combo context padding. C028 combo context padding. C029 combo context padding. C030 combo context padding. C031 combo context padding. C032 combo context padding. C033 combo context padding. C034 combo context padding. C035 combo context padding. C036 combo context padding. C037 combo context padding. C038 combo context padding. C039 combo context padding. C040 combo context padding.","expected_text":"COMBO_CONTEXT_PRESSURE_READY","response_timeout_ms":"180000"},{"prompt":"qa_combo final check: using the knowledge base, the compacted earlier passcode, and qa_plugin_echo with exactly combo-tool-ok-local-agent, return only COMBO_FINAL plus the passcode, the RAG sentinel, and the plugin tool result.","expected_text":"COMBO_FINAL qa_combo_compaction_sentinel_2406 azalea-cobalt-7421 qa-plugin-smoke:combo-tool-ok-local-agent","response_timeout_ms":"180000"}]'
+setup_automation:
+  - "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env"
+  - "node:scripts/e2e/ensure-langrag-sentinel-kb.mjs --write-env"
+  - "case:qa-plugin-smoke-live-install"
+setup_provides_env:
+  - LANGBOT_LOCAL_AGENT_PIPELINE_URL
+  - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+  - LANGBOT_LOCAL_AGENT_RAG_KB_UUID
+preconditions:
+  - "The selected model route supports function/tool calling and deterministic multi-turn answers, or LANGBOT_E2E_FAKE_PROVIDER_BASE_URL points to scripts/e2e/fake-openai-provider.mjs."
+  - "The LangRAG sentinel knowledge base contains azalea-cobalt-7421 and qa-plugin-smoke exposes qa_plugin_echo."
+steps:
+  - "Ensure the local-agent pipeline, LangRAG sentinel KB, and qa-plugin-smoke fixture plugin are available."
+  - "Temporarily bind the LangRAG KB, shrink the runner context budget, and allow serial plugin tool execution."
+  - "Temporarily bind only langbot/local-agent and qa/plugin-smoke; disable MCP servers and skills to isolate the tool surface."
+  - "Reset the person Debug Chat session for the target pipeline."
+  - "Send the passcode memory prompt and wait for MEMORY_SET."
+  - "Send the long pressure prompt and wait for COMBO_CONTEXT_PRESSURE_READY."
+  - "Ask the runner to combine compacted passcode, RAG sentinel, and qa_plugin_echo result in one final answer."
+  - "Restore the original runner config and pipeline extension bindings."
+checks:
+  - "UI: All three user messages appear in Debug Chat."
+  - "UI: The final Bot message contains COMBO_FINAL qa_combo_compaction_sentinel_2406 azalea-cobalt-7421 qa-plugin-smoke:combo-tool-ok-local-agent."
+  - "API diagnostic: pipeline-config-diagnostic.json shows knowledge-bases, context token settings, and tool-loop settings were patched."
+  - "API diagnostic: pipeline-extensions-diagnostic.json shows only langbot/local-agent and qa/plugin-smoke were bound."
+  - "API diagnostic: restore diagnostics show the original runner config and extension bindings were restored."
+  - "Logs: Backend completes RAG retrieval, plugin tool call, and Debug Chat response without runner timeout or model setup failure."
+  - "Console: No unexpected frontend runtime errors appear during the run."
+evidence_required:
+  - ui
+  - screenshot
+  - console
+  - backend_log
+  - api_diagnostic
+diagnostics:
+  - "For token-free deterministic UI coverage, start scripts/e2e/fake-openai-provider.mjs and pass LANGBOT_E2E_FAKE_PROVIDER_BASE_URL to this case."
+  - "The ensure-langrag-sentinel-kb setup script verifies LANGBOT_E2E_RAG_EXPECTED_TEXT, defaulting to azalea-cobalt-7421, while this case's Debug Chat expected text is the full COMBO_FINAL string."
+  - "The fake provider intentionally returns COMBO_MISSING_* if the final model input lacks compacted memory, RAG context, or plugin tool result."
+  - "If the final answer lacks azalea-cobalt-7421, first rerun ensure-langrag-sentinel-kb and local-agent-rag-debug-chat."
+  - "If the final answer lacks qa-plugin-smoke:combo-tool-ok-local-agent, inspect /api/v1/tools and the pipeline extensions diagnostic."
+success_patterns:
+  - "Processing request from person_websocket"
+  - "Streaming completed"
+failure_patterns:
+  - "Action invoke_llm_stream call timed out"
+  - "All models failed during streaming setup"
+  - "Task exception was never retrieved"
+  - "COMBO_MISSING_"
+  - "survey widget blocks debug chat"
+troubleshooting:
+  - local-agent-model-route-unavailable
+  - tool-name-collision-between-mcp-and-plugin
+  - plugin-runtime-timeout
+  - proxy-env-mismatch
+  - survey-widget-blocks-debug-chat
+  - debug-chat-history-contaminates-automation

skills/skills/langbot-testing/cases/local-agent-context-compaction-debug-chat.yaml

@@ -19,6 +19,8 @@ env:
   - LANGBOT_BACKEND_URL
   - LANGBOT_LOCAL_AGENT_PIPELINE_URL
   - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+env_optional:
+  - LANGBOT_E2E_FAKE_PROVIDER_BASE_URL
 automation: scripts/e2e/pipeline-debug-chat.mjs
 automation_env:
   - LANGBOT_FRONTEND_URL
@@ -31,7 +33,9 @@ automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL
 automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME
 automation_expected_runner_id: "plugin:langbot/local-agent/default"
 automation_runner_config_patch_json: '{"context-window-tokens":225,"context-reserve-tokens":50,"context-keep-recent-tokens":30,"context-summary-tokens":105,"knowledge-bases":[]}'
+automation_extensions_patch_json: '{"enable_all_plugins":false,"bound_plugins":[{"author":"langbot","name":"local-agent"}],"enable_all_mcp_servers":false,"bound_mcp_servers":[],"enable_all_skills":false,"bound_skills":[]}'
 automation_restore_runner_config: "1"
+automation_restore_extensions: "1"
 automation_reset_debug_chat: "1"
 automation_debug_chat_session_type: "person"
 automation_expected_text: "qa_compaction_sentinel_7391"
@@ -43,12 +47,13 @@ setup_provides_env:
   - LANGBOT_LOCAL_AGENT_PIPELINE_URL
   - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
 preconditions:
-  - "The selected model route can follow short deterministic instructions across multiple Debug Chat turns."
+  - "The selected model route can follow short deterministic instructions across multiple Debug Chat turns, or LANGBOT_E2E_FAKE_PROVIDER_BASE_URL points to scripts/e2e/fake-openai-provider.mjs."
 steps:
   - "Open the target local-agent pipeline through LANGBOT_FRONTEND_URL."
   - "Use the authenticated browser token only inside automation to GET and PUT /api/v1/pipelines/{uuid}."
   - "Assert the saved runner is plugin:langbot/local-agent/default."
   - "Temporarily set context-window-tokens, context-reserve-tokens, context-keep-recent-tokens, and context-summary-tokens to force compaction, and clear knowledge-bases so RAG does not answer the memory question."
+  - "Temporarily bind only the local-agent plugin, disable MCP servers, and disable skills so unrelated visible tools cannot answer or distract the memory question."
   - "Reset the person Debug Chat session for the target pipeline."
   - "Send the sentinel memory prompt and wait for MEMORY_SET."
   - "Send the long padding prompt and wait for CONTEXT_PRESSURE_READY."
@@ -59,6 +64,8 @@ checks:
   - "UI: The final Bot message contains qa_compaction_sentinel_7391."
   - "API diagnostic: pipeline-config-diagnostic.json shows patched=true and patch_keys include the four token context compaction fields plus knowledge-bases."
   - "API diagnostic: pipeline-config-restore-diagnostic.json shows the original runner config was restored."
+  - "API diagnostic: pipeline-extensions-diagnostic.json shows enable_all_plugins=false, bound_plugins contains langbot/local-agent, enable_all_mcp_servers=false, and enable_all_skills=false."
+  - "API diagnostic: pipeline-extensions-restore-diagnostic.json shows the original extension bindings were restored."
   - "Logs: Backend completes the multi-turn Debug Chat path without runner timeout or model setup failure."
   - "Console: No unexpected frontend runtime errors appear during the run."
 evidence_required:
@@ -68,7 +75,8 @@ evidence_required:
   - backend_log
   - api_diagnostic
 diagnostics:
-  - "If the final sentinel is missing, inspect whether pipeline-config-diagnostic.json targeted ai.runner_config[runnerId], cleared knowledge-bases, and whether the backend log shows the local-agent runner loading the small context settings."
+  - "For token-free deterministic UI coverage, start scripts/e2e/fake-openai-provider.mjs and pass LANGBOT_E2E_FAKE_PROVIDER_BASE_URL to this case; setup will bind the local-agent pipeline to that fake OpenAI-compatible model."
+  - "If the final sentinel is missing, inspect whether pipeline-config-diagnostic.json targeted ai.runner_config[runnerId], cleared knowledge-bases, whether pipeline-extensions-diagnostic.json isolated unrelated tools, and whether the backend log shows the local-agent runner loading the small context settings."
   - "If the model ignores deterministic replies, rerun with a known-good model route before diagnosing ContextAssembler."
   - "If restore fails, use pipeline-config-restore-diagnostic.json and GET /api/v1/pipelines/{uuid} to confirm the current saved config before retrying."
 success_patterns:

skills/skills/langbot-testing/cases/local-agent-multitool-rag-compaction-debug-chat.yaml

@@ -0,0 +1,102 @@
+id: local-agent-multitool-rag-compaction-debug-chat
+title: "Local Agent preserves RAG, compacted history, and two-step plugin tool loop"
+mode: agent-browser
+area: pipeline
+type: regression
+priority: p1
+risk: high
+ci_eligible: false
+tags:
+  - local-agent
+  - langrag
+  - context
+  - compaction
+  - plugin
+  - tools
+  - tool-loop
+  - pipeline
+skills:
+  - langbot-env-setup
+  - langbot-testing
+env:
+  - LANGBOT_FRONTEND_URL
+  - LANGBOT_BACKEND_URL
+  - LANGBOT_LOCAL_AGENT_PIPELINE_URL
+  - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+env_optional:
+  - LANGBOT_E2E_FAKE_PROVIDER_BASE_URL
+automation: scripts/e2e/pipeline-debug-chat.mjs
+automation_env:
+  - LANGBOT_FRONTEND_URL
+  - LANGBOT_BACKEND_URL
+  - LANGBOT_BROWSER_PROFILE
+  - LANGBOT_CHROMIUM_EXECUTABLE
+  - LANGBOT_LOCAL_AGENT_PIPELINE_URL
+  - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+  - LANGBOT_LOCAL_AGENT_RAG_KB_UUID
+automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL
+automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+automation_expected_runner_id: "plugin:langbot/local-agent/default"
+automation_runner_config_patch_json: '{"knowledge-bases":["${LANGBOT_LOCAL_AGENT_RAG_KB_UUID}"],"retrieval-top-k":1,"context-window-tokens":900,"context-reserve-tokens":220,"context-keep-recent-tokens":160,"context-summary-tokens":260,"max-tool-iterations":5,"tool-execution-mode":"serial"}'
+automation_extensions_patch_json: '{"enable_all_plugins":false,"bound_plugins":[{"author":"langbot","name":"local-agent"},{"author":"qa","name":"plugin-smoke"}],"enable_all_mcp_servers":false,"bound_mcp_servers":[],"enable_all_skills":false,"bound_skills":[]}'
+automation_restore_runner_config: "1"
+automation_restore_extensions: "1"
+automation_reset_debug_chat: "1"
+automation_debug_chat_session_type: "person"
+automation_expected_text: "MULTITOOL_COMBO_FINAL qa_multitool_compaction_sentinel_6718 azalea-cobalt-7421 qa-plugin-smoke:multi-tool-a-local-agent qa-plugin-smoke:multi-tool-b-local-agent"
+automation_response_timeout_ms: "180000"
+automation_prompts_json: '[{"prompt":"请记住这个用于 local-agent 多工具组合回归测试的暗号：qa_multitool_compaction_sentinel_6718。请只回复 MEMORY_SET。","expected_text":"MEMORY_SET","response_timeout_ms":"180000"},{"prompt":"下面这轮只用于制造长历史压力，目标标记 MULTITOOL_RAG_GOAL 必须被压缩保留。请忽略填充内容，最后只回复 MULTITOOL_CONTEXT_PRESSURE_READY。填充片段 M001 multitool context padding. M002 multitool context padding. M003 multitool context padding. M004 multitool context padding. M005 multitool context padding. M006 multitool context padding. M007 multitool context padding. M008 multitool context padding. M009 multitool context padding. M010 multitool context padding. M011 multitool context padding. M012 multitool context padding. M013 multitool context padding. M014 multitool context padding. M015 multitool context padding. M016 multitool context padding. M017 multitool context padding. M018 multitool context padding. M019 multitool context padding. M020 multitool context padding. M021 multitool context padding. M022 multitool context padding. M023 multitool context padding. M024 multitool context padding. M025 multitool context padding. M026 multitool context padding. M027 multitool context padding. M028 multitool context padding. M029 multitool context padding. M030 multitool context padding. M031 multitool context padding. M032 multitool context padding.","expected_text":"MULTITOOL_CONTEXT_PRESSURE_READY","response_timeout_ms":"180000"},{"prompt":"MULTITOOL_COMBO final check: using the knowledge base, the compacted earlier passcode, and qa_plugin_echo twice with exactly multi-tool-a-local-agent then multi-tool-b-local-agent, return only MULTITOOL_COMBO_FINAL plus the passcode, the RAG sentinel, and both plugin tool results.","expected_text":"MULTITOOL_COMBO_FINAL qa_multitool_compaction_sentinel_6718 azalea-cobalt-7421 qa-plugin-smoke:multi-tool-a-local-agent qa-plugin-smoke:multi-tool-b-local-agent","response_timeout_ms":"180000"}]'
+setup_automation:
+  - "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env"
+  - "node:scripts/e2e/ensure-langrag-sentinel-kb.mjs --write-env"
+  - "case:qa-plugin-smoke-live-install"
+setup_provides_env:
+  - LANGBOT_LOCAL_AGENT_PIPELINE_URL
+  - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+  - LANGBOT_LOCAL_AGENT_RAG_KB_UUID
+preconditions:
+  - "The selected model route supports function/tool calling and repeated tool loop turns, or LANGBOT_E2E_FAKE_PROVIDER_BASE_URL points to scripts/e2e/fake-openai-provider.mjs."
+  - "The LangRAG sentinel knowledge base contains azalea-cobalt-7421 and qa-plugin-smoke exposes qa_plugin_echo."
+steps:
+  - "Ensure the local-agent pipeline, LangRAG sentinel KB, and qa-plugin-smoke fixture plugin are available."
+  - "Temporarily bind the LangRAG KB, shrink the runner context budget, and set max-tool-iterations high enough for two serial plugin tool calls."
+  - "Temporarily bind only langbot/local-agent and qa/plugin-smoke; disable MCP servers and skills to isolate the tool surface."
+  - "Reset the person Debug Chat session for the target pipeline."
+  - "Send the passcode memory prompt and wait for MEMORY_SET."
+  - "Send the long pressure prompt and wait for MULTITOOL_CONTEXT_PRESSURE_READY."
+  - "Ask the runner to combine compacted passcode, RAG sentinel, and two qa_plugin_echo results in one final answer."
+  - "Restore the original runner config and pipeline extension bindings."
+checks:
+  - "UI: All three user messages appear in Debug Chat."
+  - "UI: The final Bot message contains MULTITOOL_COMBO_FINAL with the compacted sentinel, azalea-cobalt-7421, and both qa-plugin-smoke tool results."
+  - "API diagnostic: pipeline-config-diagnostic.json shows knowledge-bases, context token settings, and max-tool-iterations were patched."
+  - "API diagnostic: pipeline-extensions-diagnostic.json shows only langbot/local-agent and qa/plugin-smoke were bound."
+  - "Logs: Backend completes RAG retrieval, two plugin tool calls, and Debug Chat response without runner timeout or model setup failure."
+  - "Console: No unexpected frontend runtime errors appear during the run."
+evidence_required:
+  - ui
+  - screenshot
+  - console
+  - backend_log
+  - api_diagnostic
+diagnostics:
+  - "For token-free deterministic UI coverage, start scripts/e2e/fake-openai-provider.mjs and pass LANGBOT_E2E_FAKE_PROVIDER_BASE_URL to this case."
+  - "The fake provider intentionally returns MULTITOOL_COMBO_MISSING_* if the final model input lacks compacted memory, RAG context, or either plugin tool result."
+  - "If only one tool result is present, inspect max-tool-iterations and the runner tool-loop events."
+  - "If the final answer lacks azalea-cobalt-7421, first rerun ensure-langrag-sentinel-kb and local-agent-rag-debug-chat."
+success_patterns:
+  - "Processing request from person_websocket"
+  - "Streaming completed"
+failure_patterns:
+  - "Action invoke_llm_stream call timed out"
+  - "All models failed during streaming setup"
+  - "Task exception was never retrieved"
+  - "MULTITOOL_COMBO_MISSING_"
+  - "survey widget blocks debug chat"
+troubleshooting:
+  - local-agent-model-route-unavailable
+  - tool-name-collision-between-mcp-and-plugin
+  - plugin-runtime-timeout
+  - proxy-env-mismatch
+  - survey-widget-blocks-debug-chat
+  - debug-chat-history-contaminates-automation

skills/skills/langbot-testing/cases/local-agent-nonstreaming-debug-chat.yaml

@@ -1,5 +1,5 @@
 id: local-agent-nonstreaming-debug-chat
-title: "Local Agent Debug Chat returns a deterministic non-streaming response"
+title: "Local Agent Debug Chat returns a deterministic response with UI streaming disabled"
 mode: agent-browser
 area: pipeline
 type: regression
@@ -48,14 +48,16 @@ steps:
 checks:
   - "UI: The user message appears in Debug Chat."
   - "UI: A Bot message appears and contains NONSTREAM_OK."
-  - "Logs: Backend completes the request as a normal response rather than only relying on the streaming-completed path."
+  - "UI: The Debug Chat stream switch is disabled for this send path."
+  - "Logs: Backend completes the request without plugin/runtime timeout or frontend streaming UI errors."
   - "Console: No unexpected frontend runtime errors appear during the send/receive path."
 evidence_required:
   - ui
   - console
   - backend_log
 diagnostics:
-  - "If the UI still streams after the switch is disabled, inspect the adapter streaming capability and runner config before diagnosing the model."
+  - "This browser case validates the Debug Chat non-streaming UI delivery path. The runner-internal invoke_llm path is covered by langbot-local-agent component tests using runtime_metadata.streaming_supported=false."
+  - "If the UI still streams after the switch is disabled, inspect the Debug Chat WebSocket stream flag before diagnosing the model."
   - "Use GET /api/v1/pipelines/{uuid} only to confirm the saved runner and model config."
 troubleshooting:
   - local-agent-model-route-unavailable

skills/skills/langbot-testing/cases/local-agent-parallel-tools-rag-compaction-debug-chat.yaml

@@ -0,0 +1,103 @@
+id: local-agent-parallel-tools-rag-compaction-debug-chat
+title: "Local Agent preserves RAG, compacted history, and parallel plugin tools"
+mode: agent-browser
+area: pipeline
+type: regression
+priority: p1
+risk: high
+ci_eligible: false
+tags:
+  - local-agent
+  - langrag
+  - context
+  - compaction
+  - plugin
+  - tools
+  - parallel-tools
+  - pipeline
+skills:
+  - langbot-env-setup
+  - langbot-testing
+env:
+  - LANGBOT_FRONTEND_URL
+  - LANGBOT_BACKEND_URL
+  - LANGBOT_LOCAL_AGENT_PIPELINE_URL
+  - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+env_optional:
+  - LANGBOT_E2E_FAKE_PROVIDER_BASE_URL
+automation: scripts/e2e/pipeline-debug-chat.mjs
+automation_env:
+  - LANGBOT_FRONTEND_URL
+  - LANGBOT_BACKEND_URL
+  - LANGBOT_BROWSER_PROFILE
+  - LANGBOT_CHROMIUM_EXECUTABLE
+  - LANGBOT_LOCAL_AGENT_PIPELINE_URL
+  - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+  - LANGBOT_LOCAL_AGENT_RAG_KB_UUID
+automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL
+automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+automation_expected_runner_id: "plugin:langbot/local-agent/default"
+automation_runner_config_patch_json: '{"knowledge-bases":["${LANGBOT_LOCAL_AGENT_RAG_KB_UUID}"],"retrieval-top-k":1,"context-window-tokens":900,"context-reserve-tokens":220,"context-keep-recent-tokens":160,"context-summary-tokens":260,"max-tool-iterations":3,"tool-execution-mode":"parallel"}'
+automation_extensions_patch_json: '{"enable_all_plugins":false,"bound_plugins":[{"author":"langbot","name":"local-agent"},{"author":"qa","name":"plugin-smoke"}],"enable_all_mcp_servers":false,"bound_mcp_servers":[],"enable_all_skills":false,"bound_skills":[]}'
+automation_restore_runner_config: "1"
+automation_restore_extensions: "1"
+automation_reset_debug_chat: "1"
+automation_debug_chat_session_type: "person"
+automation_expected_text: "PARALLEL_COMBO_FINAL qa_parallel_compaction_sentinel_8142 azalea-cobalt-7421 qa-plugin-smoke:parallel-tool-a-local-agent qa-plugin-smoke:parallel-tool-b-local-agent"
+automation_response_timeout_ms: "180000"
+automation_prompts_json: '[{"prompt":"请记住这个用于 local-agent 并行工具组合回归测试的暗号：qa_parallel_compaction_sentinel_8142。请只回复 MEMORY_SET。","expected_text":"MEMORY_SET","response_timeout_ms":"180000"},{"prompt":"下面这轮只用于制造长历史压力，目标标记 PARALLEL_RAG_GOAL 必须被压缩保留。请忽略填充内容，最后只回复 PARALLEL_CONTEXT_PRESSURE_READY。填充片段 P001 parallel context padding. P002 parallel context padding. P003 parallel context padding. P004 parallel context padding. P005 parallel context padding. P006 parallel context padding. P007 parallel context padding. P008 parallel context padding. P009 parallel context padding. P010 parallel context padding. P011 parallel context padding. P012 parallel context padding. P013 parallel context padding. P014 parallel context padding. P015 parallel context padding. P016 parallel context padding. P017 parallel context padding. P018 parallel context padding. P019 parallel context padding. P020 parallel context padding. P021 parallel context padding. P022 parallel context padding. P023 parallel context padding. P024 parallel context padding. P025 parallel context padding. P026 parallel context padding. P027 parallel context padding. P028 parallel context padding. P029 parallel context padding. P030 parallel context padding. P031 parallel context padding. P032 parallel context padding.","expected_text":"PARALLEL_CONTEXT_PRESSURE_READY","response_timeout_ms":"180000"},{"prompt":"PARALLEL_COMBO final check: using the knowledge base, the compacted earlier passcode, and one model turn that calls qa_plugin_echo for both parallel-tool-a-local-agent and parallel-tool-b-local-agent, return only PARALLEL_COMBO_FINAL plus the passcode, the RAG sentinel, and both plugin tool results.","expected_text":"PARALLEL_COMBO_FINAL qa_parallel_compaction_sentinel_8142 azalea-cobalt-7421 qa-plugin-smoke:parallel-tool-a-local-agent qa-plugin-smoke:parallel-tool-b-local-agent","response_timeout_ms":"180000"}]'
+setup_automation:
+  - "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env"
+  - "node:scripts/e2e/ensure-langrag-sentinel-kb.mjs --write-env"
+  - "case:qa-plugin-smoke-live-install"
+setup_provides_env:
+  - LANGBOT_LOCAL_AGENT_PIPELINE_URL
+  - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+  - LANGBOT_LOCAL_AGENT_RAG_KB_UUID
+preconditions:
+  - "The selected model route supports function/tool calling with multiple tool calls in one turn, or LANGBOT_E2E_FAKE_PROVIDER_BASE_URL points to scripts/e2e/fake-openai-provider.mjs."
+  - "The LangRAG sentinel knowledge base contains azalea-cobalt-7421 and qa-plugin-smoke exposes qa_plugin_echo."
+steps:
+  - "Ensure the local-agent pipeline, LangRAG sentinel KB, and qa-plugin-smoke fixture plugin are available."
+  - "Temporarily bind the LangRAG KB, shrink the runner context budget, and set tool-execution-mode to parallel."
+  - "Temporarily bind only langbot/local-agent and qa/plugin-smoke; disable MCP servers and skills to isolate the tool surface."
+  - "Reset the person Debug Chat session for the target pipeline."
+  - "Send the passcode memory prompt and wait for MEMORY_SET."
+  - "Send the long pressure prompt and wait for PARALLEL_CONTEXT_PRESSURE_READY."
+  - "Ask the runner to combine compacted passcode, RAG sentinel, and two same-turn qa_plugin_echo results in one final answer."
+  - "Restore the original runner config and pipeline extension bindings."
+checks:
+  - "UI: All three user messages appear in Debug Chat."
+  - "UI: The final Bot message contains PARALLEL_COMBO_FINAL with the compacted sentinel, azalea-cobalt-7421, and both qa-plugin-smoke tool results."
+  - "API diagnostic: pipeline-config-diagnostic.json shows knowledge-bases, context token settings, max-tool-iterations, and tool-execution-mode=parallel were patched."
+  - "API diagnostic: pipeline-extensions-diagnostic.json shows only langbot/local-agent and qa/plugin-smoke were bound."
+  - "Logs: Backend completes RAG retrieval, both plugin tool calls, and Debug Chat response without runner timeout or model setup failure."
+  - "Console: No unexpected frontend runtime errors appear during the run."
+evidence_required:
+  - ui
+  - screenshot
+  - console
+  - backend_log
+  - api_diagnostic
+diagnostics:
+  - "For token-free deterministic UI coverage, start scripts/e2e/fake-openai-provider.mjs and pass LANGBOT_E2E_FAKE_PROVIDER_BASE_URL to this case."
+  - "The fake provider intentionally returns two qa_plugin_echo tool_calls in the same streamed model turn for this prompt."
+  - "The fake provider intentionally returns PARALLEL_COMBO_MISSING_* if the final model input lacks compacted memory, RAG context, or either plugin tool result."
+  - "If only one tool result is present, inspect the provider trace and runner parallel tool batch handling."
+  - "If the final answer lacks azalea-cobalt-7421, first rerun ensure-langrag-sentinel-kb and local-agent-rag-debug-chat."
+success_patterns:
+  - "Processing request from person_websocket"
+  - "Streaming completed"
+failure_patterns:
+  - "Action invoke_llm_stream call timed out"
+  - "All models failed during streaming setup"
+  - "Task exception was never retrieved"
+  - "PARALLEL_COMBO_MISSING_"
+  - "survey widget blocks debug chat"
+troubleshooting:
+  - local-agent-model-route-unavailable
+  - tool-name-collision-between-mcp-and-plugin
+  - plugin-runtime-timeout
+  - proxy-env-mismatch
+  - survey-widget-blocks-debug-chat
+  - debug-chat-history-contaminates-automation

skills/skills/langbot-testing/cases/local-agent-plugin-tool-call-debug-chat.yaml

@@ -19,6 +19,8 @@ env:
   - LANGBOT_BACKEND_URL
   - LANGBOT_LOCAL_AGENT_PIPELINE_URL
   - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+env_optional:
+  - LANGBOT_E2E_FAKE_PROVIDER_BASE_URL
 automation: scripts/e2e/pipeline-debug-chat.mjs
 automation_env:
   - LANGBOT_FRONTEND_URL
@@ -38,7 +40,7 @@ setup_provides_env:
   - LANGBOT_LOCAL_AGENT_PIPELINE_URL
   - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
 preconditions:
-  - "The selected model route supports function/tool calling."
+  - "The selected model route supports function/tool calling, or LANGBOT_E2E_FAKE_PROVIDER_BASE_URL points to scripts/e2e/fake-openai-provider.mjs."
 steps:
   - "Install or enable the bundled qa-plugin-smoke fixture plugin."
   - "Confirm /api/v1/tools or the plugin detail shows qa_plugin_echo."
@@ -58,6 +60,7 @@ evidence_required:
   - backend_log
   - api_diagnostic
 diagnostics:
+  - "For token-free deterministic UI coverage, start scripts/e2e/fake-openai-provider.mjs and pass LANGBOT_E2E_FAKE_PROVIDER_BASE_URL to this case; setup will bind the local-agent pipeline to that fake OpenAI-compatible model."
   - "If qa_plugin_echo is not listed, rebuild and reinstall the qa-plugin-smoke fixture plugin."
   - "If the selected model returns model_not_found or no available channel only when tools are provided, switch to a known-good function-calling model before diagnosing plugin tools or local-agent."
 troubleshooting:

skills/skills/langbot-testing/cases/local-agent-rag-debug-chat.yaml

@@ -18,6 +18,8 @@ env:
   - LANGBOT_BACKEND_URL
   - LANGBOT_LOCAL_AGENT_PIPELINE_URL
   - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+env_optional:
+  - LANGBOT_E2E_FAKE_PROVIDER_BASE_URL
 automation: scripts/e2e/pipeline-debug-chat.mjs
 automation_env:
   - LANGBOT_FRONTEND_URL
@@ -43,7 +45,7 @@ setup_provides_env:
   - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
   - LANGBOT_LOCAL_AGENT_RAG_KB_UUID
 preconditions:
-  - "The target pipeline already has a text-capable model route that is available for this run."
+  - "The target pipeline already has a text-capable model route that is available for this run, or LANGBOT_E2E_FAKE_PROVIDER_BASE_URL points to scripts/e2e/fake-openai-provider.mjs."
 steps:
   - "Ensure case langrag-kb-retrieve has produced a knowledge base containing sentinel azalea-cobalt-7421."
   - "Open LANGBOT_FRONTEND_URL."
@@ -68,6 +70,8 @@ evidence_required:
   - backend_log
   - api_diagnostic
 diagnostics:
+  - "For token-free deterministic UI coverage, start scripts/e2e/fake-openai-provider.mjs and pass LANGBOT_E2E_FAKE_PROVIDER_BASE_URL to this case; setup will bind the local-agent pipeline to that fake OpenAI-compatible model."
+  - "The ensure-langrag-sentinel-kb setup script verifies LANGBOT_E2E_RAG_EXPECTED_TEXT, defaulting to azalea-cobalt-7421, so the Debug Chat expected text does not leak into KB setup."
   - "Use GET /api/v1/pipelines/{uuid} only to confirm the saved runner_config contains the knowledge base uuid."
   - "If the bot ignores the knowledge base, rerun Retrieve Test before debugging the runner."
 troubleshooting:

skills/skills/langbot-testing/cases/local-agent-steering-debug-chat.yaml

@@ -74,7 +74,6 @@ success_patterns:
   - "Steering"
   - "Streaming completed"
 failure_patterns:
-  - "STEERING_NO_FOLLOWUP"
   - "Action invoke_llm_stream call timed out"
   - "All models failed during streaming setup"
   - "Task exception was never retrieved"

skills/skills/langbot-testing/cases/local-agent-tool-error-recovery-debug-chat.yaml

@@ -0,0 +1,94 @@
+id: local-agent-tool-error-recovery-debug-chat
+title: "Local Agent feeds plugin tool errors back to the model"
+mode: agent-browser
+area: pipeline
+type: regression
+priority: p1
+risk: high
+ci_eligible: false
+tags:
+  - local-agent
+  - plugin
+  - tools
+  - tool-error
+  - pipeline
+skills:
+  - langbot-env-setup
+  - langbot-testing
+env:
+  - LANGBOT_FRONTEND_URL
+  - LANGBOT_BACKEND_URL
+  - LANGBOT_LOCAL_AGENT_PIPELINE_URL
+  - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+env_optional:
+  - LANGBOT_E2E_FAKE_PROVIDER_BASE_URL
+automation: scripts/e2e/pipeline-debug-chat.mjs
+automation_env:
+  - LANGBOT_FRONTEND_URL
+  - LANGBOT_BACKEND_URL
+  - LANGBOT_BROWSER_PROFILE
+  - LANGBOT_CHROMIUM_EXECUTABLE
+  - LANGBOT_LOCAL_AGENT_PIPELINE_URL
+  - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL
+automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+automation_expected_runner_id: "plugin:langbot/local-agent/default"
+automation_runner_config_patch_json: '{"knowledge-bases":[],"max-tool-iterations":3,"tool-execution-mode":"serial"}'
+automation_extensions_patch_json: '{"enable_all_plugins":false,"bound_plugins":[{"author":"langbot","name":"local-agent"},{"author":"qa","name":"plugin-smoke"}],"enable_all_mcp_servers":false,"bound_mcp_servers":[],"enable_all_skills":false,"bound_skills":[]}'
+automation_restore_runner_config: "1"
+automation_restore_extensions: "1"
+automation_reset_debug_chat: "1"
+automation_debug_chat_session_type: "person"
+automation_prompt: "TOOL_ERROR_RECOVERY regression: call qa_plugin_fail with exactly tool-error-recovery-local-agent. After observing the tool error, return only TOOL_ERROR_RECOVERY_FINAL qa-plugin-smoke forced failure observed."
+automation_expected_text: "TOOL_ERROR_RECOVERY_FINAL qa-plugin-smoke forced failure observed"
+automation_response_timeout_ms: "180000"
+setup_automation:
+  - "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env"
+  - "case:qa-plugin-smoke-live-install"
+setup_provides_env:
+  - LANGBOT_LOCAL_AGENT_PIPELINE_URL
+  - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+preconditions:
+  - "The selected model route supports function/tool calling, or LANGBOT_E2E_FAKE_PROVIDER_BASE_URL points to scripts/e2e/fake-openai-provider.mjs."
+  - "qa-plugin-smoke exposes qa_plugin_fail; the setup case reinstalls the fixture if an older installed package is missing it."
+steps:
+  - "Ensure the local-agent pipeline and qa-plugin-smoke fixture plugin are available."
+  - "Temporarily bind only langbot/local-agent plus qa/plugin-smoke and clear RAG knowledge bases."
+  - "Reset the person Debug Chat session for the target pipeline."
+  - "Send the TOOL_ERROR_RECOVERY prompt."
+  - "Verify the failing plugin tool result is fed back into the model and the final answer contains the recovery sentinel."
+  - "Restore the original runner config and pipeline extension bindings."
+checks:
+  - "UI: The final Bot message contains TOOL_ERROR_RECOVERY_FINAL qa-plugin-smoke forced failure observed."
+  - "API diagnostic: pipeline-config-diagnostic.json shows max-tool-iterations=3 and knowledge-bases cleared."
+  - "API diagnostic: pipeline-extensions-diagnostic.json shows only langbot/local-agent and qa/plugin-smoke were bound."
+  - "Logs: Backend shows the request completed without runner.timeout, runner.tool_error run failure, or model setup failure."
+  - "Console: No unexpected frontend runtime errors appear during Debug Chat."
+evidence_required:
+  - ui
+  - screenshot
+  - console
+  - backend_log
+  - api_diagnostic
+diagnostics:
+  - "For token-free deterministic UI coverage, start scripts/e2e/fake-openai-provider.mjs and pass LANGBOT_E2E_FAKE_PROVIDER_BASE_URL to this case."
+  - "The fake provider calls qa_plugin_fail once, then only returns the final sentinel after it sees an Error: tool result."
+  - "If the run returns a generic answer, inspect whether qa_plugin_fail was exposed by /api/v1/tools and whether the installed fixture plugin was refreshed."
+success_patterns:
+  - "TOOL_ERROR_RECOVERY_FINAL"
+  - "Streaming completed"
+failure_patterns:
+  - "Action invoke_llm_stream call timed out"
+  - "All models failed during streaming setup"
+  - "Task exception was never retrieved"
+  - "runner.timeout"
+  - "runner.tool_error"
+  - "Tool call iteration limit reached"
+  - "survey widget blocks debug chat"
+troubleshooting:
+  - local-agent-model-route-unavailable
+  - tool-name-collision-between-mcp-and-plugin
+  - plugin-runtime-timeout
+  - proxy-env-mismatch
+  - survey-widget-blocks-debug-chat
+  - debug-chat-history-contaminates-automation

skills/skills/langbot-testing/cases/local-agent-tool-loop-limit-debug-chat.yaml

@@ -0,0 +1,92 @@
+id: local-agent-tool-loop-limit-debug-chat
+title: "Local Agent stops repeated tool calls at max-tool-iterations"
+mode: agent-browser
+area: pipeline
+type: regression
+priority: p1
+risk: high
+ci_eligible: false
+tags:
+  - local-agent
+  - plugin
+  - tools
+  - tool-loop
+  - limit
+  - pipeline
+skills:
+  - langbot-env-setup
+  - langbot-testing
+env:
+  - LANGBOT_FRONTEND_URL
+  - LANGBOT_BACKEND_URL
+  - LANGBOT_LOCAL_AGENT_PIPELINE_URL
+  - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+env_optional:
+  - LANGBOT_E2E_FAKE_PROVIDER_BASE_URL
+automation: scripts/e2e/pipeline-debug-chat.mjs
+automation_env:
+  - LANGBOT_FRONTEND_URL
+  - LANGBOT_BACKEND_URL
+  - LANGBOT_BROWSER_PROFILE
+  - LANGBOT_CHROMIUM_EXECUTABLE
+  - LANGBOT_LOCAL_AGENT_PIPELINE_URL
+  - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL
+automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+automation_expected_runner_id: "plugin:langbot/local-agent/default"
+automation_runner_config_patch_json: '{"knowledge-bases":[],"max-tool-iterations":2,"tool-execution-mode":"serial"}'
+automation_extensions_patch_json: '{"enable_all_plugins":false,"bound_plugins":[{"author":"langbot","name":"local-agent"},{"author":"qa","name":"plugin-smoke"}],"enable_all_mcp_servers":false,"bound_mcp_servers":[],"enable_all_skills":false,"bound_skills":[]}'
+automation_restore_runner_config: "1"
+automation_restore_extensions: "1"
+automation_reset_debug_chat: "1"
+automation_debug_chat_session_type: "person"
+automation_prompt: "LOOP_LIMIT regression: keep requesting qa_plugin_echo with exactly loop-limit-repeat-local-agent until the runner stops you. The correct final visible answer is the runner limit message."
+automation_expected_text: "Tool call iteration limit reached. I stopped before calling more tools."
+automation_response_timeout_ms: "180000"
+setup_automation:
+  - "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env"
+  - "case:qa-plugin-smoke-live-install"
+setup_provides_env:
+  - LANGBOT_LOCAL_AGENT_PIPELINE_URL
+  - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+preconditions:
+  - "The selected model route supports function/tool calling, or LANGBOT_E2E_FAKE_PROVIDER_BASE_URL points to scripts/e2e/fake-openai-provider.mjs."
+  - "qa-plugin-smoke exposes qa_plugin_echo."
+steps:
+  - "Ensure the local-agent pipeline and qa-plugin-smoke fixture plugin are available."
+  - "Temporarily set max-tool-iterations to 2 and bind only langbot/local-agent plus qa/plugin-smoke."
+  - "Reset the person Debug Chat session for the target pipeline."
+  - "Send the LOOP_LIMIT prompt and wait for the runner limit message."
+  - "Restore the original runner config and pipeline extension bindings."
+checks:
+  - "UI: The final Bot message contains Tool call iteration limit reached. I stopped before calling more tools."
+  - "API diagnostic: pipeline-config-diagnostic.json shows max-tool-iterations was patched to 2."
+  - "API diagnostic: pipeline-extensions-diagnostic.json shows only langbot/local-agent and qa/plugin-smoke were bound."
+  - "Logs: Backend shows the request completed without infinite loop, runner timeout, or model setup failure."
+  - "Console: No unexpected frontend runtime errors appear during Debug Chat."
+evidence_required:
+  - ui
+  - screenshot
+  - console
+  - backend_log
+  - api_diagnostic
+diagnostics:
+  - "For token-free deterministic UI coverage, start scripts/e2e/fake-openai-provider.mjs and pass LANGBOT_E2E_FAKE_PROVIDER_BASE_URL to this case."
+  - "The fake provider intentionally keeps requesting qa_plugin_echo; this case passes only when the runner stops at max-tool-iterations."
+  - "If the run times out, inspect whether max-tool-iterations was patched or whether the runner ignored tool loop limits."
+success_patterns:
+  - "Tool call iteration limit reached"
+  - "Streaming completed"
+failure_patterns:
+  - "Action invoke_llm_stream call timed out"
+  - "All models failed during streaming setup"
+  - "Task exception was never retrieved"
+  - "runner.timeout"
+  - "survey widget blocks debug chat"
+troubleshooting:
+  - local-agent-model-route-unavailable
+  - tool-name-collision-between-mcp-and-plugin
+  - plugin-runtime-timeout
+  - proxy-env-mismatch
+  - survey-widget-blocks-debug-chat
+  - debug-chat-history-contaminates-automation

skills/skills/langbot-testing/cases/mcp-stdio-register.yaml

@@ -34,8 +34,10 @@ steps:
   - "Poll /api/v1/tools and /api/v1/mcp/servers/qa-local-stdio until qa_mcp_echo is visible."
 checks:
   - "API diagnostic: qa-local-stdio runtime_status is connected."
+  - "API diagnostic: qa-local-stdio server_uuid is present for pipeline extension binding."
   - "API diagnostic: runtime_tool_names includes qa_mcp_echo."
   - "API diagnostic: /api/v1/tools includes qa_mcp_echo."
+  - "Automation writes LANGBOT_MCP_QA_STDIO_SERVER_UUID into skills/.env.local for downstream tool-call cases."
   - "Console and network logs contain no unexpected frontend/runtime failures."
 evidence_required:
   - screenshot

skills/skills/langbot-testing/cases/mcp-stdio-tool-call.yaml

@@ -18,15 +18,23 @@ env:
   - LANGBOT_BACKEND_URL
   - LANGBOT_LOCAL_AGENT_PIPELINE_URL
   - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+env_optional:
+  - LANGBOT_E2E_FAKE_PROVIDER_BASE_URL
 automation: scripts/e2e/pipeline-debug-chat.mjs
 automation_env:
   - LANGBOT_FRONTEND_URL
+  - LANGBOT_BACKEND_URL
   - LANGBOT_BROWSER_PROFILE
   - LANGBOT_CHROMIUM_EXECUTABLE
   - LANGBOT_LOCAL_AGENT_PIPELINE_URL
   - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+  - LANGBOT_MCP_QA_STDIO_SERVER_UUID
 automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL
 automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+automation_expected_runner_id: "plugin:langbot/local-agent/default"
+automation_extensions_patch_json: '{"enable_all_plugins":false,"bound_plugins":[{"author":"langbot","name":"local-agent"}],"enable_all_mcp_servers":false,"bound_mcp_servers":["${LANGBOT_MCP_QA_STDIO_SERVER_UUID}"],"enable_all_skills":false,"bound_skills":[]}'
+automation_restore_extensions: "1"
+automation_reset_debug_chat: "1"
 automation_prompt: "Call the qa_mcp_echo MCP tool with exactly this text: mcp-ok-local-agent. Return only the tool result."
 automation_expected_text: "qa_mcp_echo:mcp-ok-local-agent"
 automation_response_timeout_ms: "180000"
@@ -36,6 +44,7 @@ setup_automation:
 setup_provides_env:
   - LANGBOT_LOCAL_AGENT_PIPELINE_URL
   - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
+  - LANGBOT_MCP_QA_STDIO_SERVER_UUID
 failure_patterns:
   - "qa-plugin-smoke:mcp-ok-local-agent"
   - "qa_echo:mcp-ok-local-agent"
@@ -45,7 +54,7 @@ failure_patterns:
   - "no available channel for model"
 preconditions:
   - "box.local.allowed_mount_roots includes the bundled MCP fixture directory when LangBot runs stdio MCP servers through Box."
-  - "The selected model route supports function/tool calling."
+  - "The selected model route supports function/tool calling, or LANGBOT_E2E_FAKE_PROVIDER_BASE_URL points to scripts/e2e/fake-openai-provider.mjs."
 steps:
   - "Open LANGBOT_FRONTEND_URL."
   - "Navigate to MCP Servers."
@@ -66,6 +75,7 @@ checks:
   - "API diagnostic: /api/v1/tools contains qa_mcp_echo."
   - "UI: Debug Chat bot response contains qa_mcp_echo:mcp-ok-local-agent."
   - "Logs: Backend logs show the MCP tool call was executed, not only listed."
+  - "API diagnostic: pipeline-extensions-diagnostic.json shows the qa-local-stdio server UUID is the only bound MCP server for the run."
   - "Console: No unexpected frontend errors appear during MCP form use or Debug Chat."
 evidence_required:
   - ui
@@ -73,6 +83,7 @@ evidence_required:
   - backend_log
   - api_diagnostic
 diagnostics:
+  - "For token-free deterministic UI coverage, start scripts/e2e/fake-openai-provider.mjs and pass LANGBOT_E2E_FAKE_PROVIDER_BASE_URL to this case; setup will bind the local-agent pipeline to that fake OpenAI-compatible model."
   - "Run node scripts/e2e/mcp-stdio-fixture.mjs to verify the bundled stdio fixture can list and call qa_mcp_echo without involving a model provider."
   - "Run node scripts/e2e/mcp-stdio-register.mjs to upsert qa-local-stdio in LangBot and verify /api/v1/tools exposes qa_mcp_echo."
   - "If backend logs show host_path is outside allowed_mount_roots, add the fixture directory to box.local.allowed_mount_roots in the local LangBot data config."

skills/skills/langbot-testing/cases/qa-plugin-smoke-live-install.yaml

@@ -20,14 +20,14 @@ env:
 automation: scripts/e2e/install-qa-plugin-smoke.mjs
 automation_plugin_package: "skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/dist/qa-plugin-smoke-0.1.0.lbpkg"
 automation_expected_plugin_id: "qa/plugin-smoke"
-automation_expected_tool: "qa_plugin_echo"
+automation_expected_tool: "qa_plugin_echo,qa_plugin_fail"
 steps:
   - "Run `rtk bin/lbs test run qa-plugin-smoke-live-install --dry-run` first; remove `--dry-run` only after readiness points at a local test LangBot instance."
   - "Automation authenticates the local test user, uploads the QA plugin smoke .lbpkg package when missing, waits for install, and checks tool exposure."
 checks:
   - "automation-result.json status is pass."
   - "/api/v1/plugins lists qa/plugin-smoke after install."
-  - "/api/v1/tools lists qa_plugin_echo after install."
+  - "/api/v1/tools lists qa_plugin_echo and qa_plugin_fail after install."
 evidence_required:
   - api_diagnostic
   - filesystem
@@ -39,6 +39,7 @@ success_patterns:
 failure_patterns:
   - "Plugin install task did not complete successfully"
   - "qa_plugin_echo is not listed"
+  - "qa_plugin_fail is not listed"
 troubleshooting:
   - plugin-runtime-timeout
   - plugin-dependency-install-offline

skills/skills/langbot-testing/fixtures/fixtures.json

@@ -49,7 +49,13 @@
     "title": "LangRAG sentinel text document",
     "kind": "text",
     "path": "fixtures/rag/sentinel-doc.txt",
-    "related_cases": ["langrag-kb-retrieve", "local-agent-rag-debug-chat"],
+    "related_cases": [
+      "langrag-kb-retrieve",
+      "local-agent-rag-debug-chat",
+      "local-agent-combo-rag-compaction-tool-debug-chat",
+      "local-agent-multitool-rag-compaction-debug-chat",
+      "local-agent-parallel-tools-rag-compaction-debug-chat"
+    ],
     "checks": ["exists"]
   },
   {
@@ -78,6 +84,11 @@
       "plugin-e2e-smoke",
       "local-agent-effective-prompt-debug-chat",
       "local-agent-plugin-tool-call-debug-chat",
+      "local-agent-tool-error-recovery-debug-chat",
+      "local-agent-tool-loop-limit-debug-chat",
+      "local-agent-combo-rag-compaction-tool-debug-chat",
+      "local-agent-multitool-rag-compaction-debug-chat",
+      "local-agent-parallel-tools-rag-compaction-debug-chat",
       "local-agent-steering-debug-chat"
     ],
     "checks": ["exists"]
@@ -87,7 +98,7 @@
     "title": "QA plugin smoke prebuilt package",
     "kind": "plugin_package",
     "path": "fixtures/plugins/qa-plugin-smoke/dist/qa-plugin-smoke-0.1.0.lbpkg",
-    "related_cases": ["qa-plugin-smoke-live-install", "plugin-e2e-smoke"],
+    "related_cases": ["qa-plugin-smoke-live-install", "plugin-e2e-smoke", "local-agent-tool-error-recovery-debug-chat"],
     "checks": ["exists", "zip_package"]
   }
 ]

skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/README.md

@@ -7,3 +7,4 @@ Tools:
 - `qa_echo(text)` returns `qa-plugin-smoke:<text>`.
 - `qa_plugin_echo(text)` returns `qa-plugin-smoke:<text>`.
 - `qa_plugin_sleep(seconds, text)` waits up to 15 seconds and returns `qa-plugin-smoke:sleep:<seconds>:<text>`.
+- `qa_plugin_fail(text)` raises a deterministic error containing `text`.

skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/tools/qa_plugin_fail.py

@@ -0,0 +1,17 @@
+from __future__ import annotations
+
+from typing import Any
+
+from langbot_plugin.api.definition.components.tool.tool import Tool
+from langbot_plugin.api.entities.builtin.provider import session as provider_session
+
+
+class QAPluginFailTool(Tool):
+    async def call(
+        self,
+        params: dict[str, Any],
+        session: provider_session.Session,
+        query_id: int,
+    ) -> str:
+        text = str(params.get("text", "qa-plugin-fail"))
+        raise RuntimeError(f"qa-plugin-smoke forced failure: {text}")

skills/skills/langbot-testing/fixtures/plugins/qa-plugin-smoke/components/tools/qa_plugin_fail.yaml

@@ -0,0 +1,24 @@
+apiVersion: v1
+kind: Tool
+metadata:
+  name: qa_plugin_fail
+  label:
+    en_US: QA Plugin Fail
+    zh_Hans: QA 插件失败工具
+  description:
+    en_US: Raises a deterministic error for local-agent tool error recovery tests.
+    zh_Hans: 为 local-agent 工具错误恢复测试抛出确定性错误。
+spec:
+  parameters:
+    type: object
+    properties:
+      text:
+        type: string
+        description: Text included in the deterministic failure message.
+    required:
+      - text
+  llm_prompt: Always fail with a deterministic qa-plugin-smoke error for tool error recovery tests.
+execution:
+  python:
+    path: qa_plugin_fail.py
+    attr: QAPluginFailTool

skills/skills/langbot-testing/references/agent-runner-release-gate.md

@@ -111,13 +111,15 @@ Each probe writes `automation-result.json` and probe logs under
 | Deterministic QA runner install | `agent-runner-live-install` | A local `.lbpkg` AgentRunner package can install and register a runner. |
 | Deterministic QA runner Debug Chat | `agent-runner-qa-debug-chat` | The installed QA runner executes through WebUI Debug Chat without a model provider. |
 | Required runner plugins | `agent-runner-release-preflight` | `langbot/local-agent` and `langbot/acp-agent-runner` are visible to the host. |
-| Required QA plugin tool | `plugin-e2e-smoke`, `agent-runner-release-preflight` | The deterministic `qa_plugin_echo` tool is exposed before tool-loop cases start. |
+| Required QA plugin tools | `plugin-e2e-smoke`, `agent-runner-release-preflight`, `qa-plugin-smoke-live-install` | The deterministic `qa_plugin_echo` and `qa_plugin_fail` tools are exposed before tool-loop and tool-error cases start. |
 | Knowledge base fixture | `langrag-kb-retrieve`, `local-agent-rag-debug-chat` | LangRAG data is queryable and the runner inserts retrieved context. |
 | Effective prompt bridge | `local-agent-effective-prompt-debug-chat` | Host prompt preprocessing reaches the runner. |
 | History and compaction | `local-agent-context-compaction-debug-chat` | Runner-owned history budgeting keeps recoverable older context. |
 | Streaming LLM | `local-agent-basic-debug-chat` | The default streaming path returns a visible answer. |
 | Non-streaming LLM | `local-agent-nonstreaming-debug-chat` | The non-streaming adapter path returns a visible answer. |
 | Plugin tool loop | `local-agent-plugin-tool-call-debug-chat` | Function-call capable models can call host plugin tools through authorization. |
+| Plugin tool error recovery | `local-agent-tool-error-recovery-debug-chat` | Tool execution errors are serialized into model-facing tool results and the model can produce a final answer instead of failing the run. |
+| Parallel plugin tool batch | `local-agent-parallel-tools-rag-compaction-debug-chat` | Local-agent executes multiple same-turn plugin tool calls and returns both results with RAG and compacted history. |
 | MCP registration | `mcp-stdio-register` | The deterministic stdio MCP server is registered and exposes `qa_mcp_echo`. |
 | MCP tool loop | `mcp-stdio-tool-call` | Local-agent can call the registered MCP tool through the same tool loop. |
 | Multimodal input | `local-agent-multimodal-debug-chat` | Image upload and structured input reach the runner. |

skills/skills/langbot-testing/references/local-agent-runner-coverage.md

@@ -29,10 +29,15 @@ These browser cases are the minimum gate for a local-agent migration check:
 | `local-agent-rag-debug-chat` | Knowledge-base authorization, retrieval, and RAG prompt insertion | Bot returns the KB sentinel, not a generic answer. |
 | `mcp-stdio-tool-call` | MCP stdio discovery, tool detail, model function calling, and tool execution | Bot returns `qa_mcp_echo:<input>` and backend logs the MCP tool call. |
 | `local-agent-plugin-tool-call-debug-chat` | Plugin tool discovery, tool detail, model function calling, and tool execution | Bot returns `qa-plugin-smoke:<input>` and backend logs the plugin tool call. |
+| `local-agent-tool-error-recovery-debug-chat` | Plugin tool execution failure, model-facing error tool result, and final recovery turn | Bot returns `TOOL_ERROR_RECOVERY_FINAL...` only after the model sees the tool error result. |
+| `local-agent-tool-loop-limit-debug-chat` | Repeated plugin tool requests and max tool iteration enforcement | Bot returns the runner limit message instead of looping indefinitely. |
+| `local-agent-combo-rag-compaction-tool-debug-chat` | RAG, compacted history, and plugin tool execution in the same Debug Chat run | Bot returns `COMBO_FINAL` with the compacted sentinel, KB sentinel, and plugin tool result together. |
+| `local-agent-multitool-rag-compaction-debug-chat` | RAG, compacted history, and repeated serial plugin tool loop turns in the same Debug Chat run | Bot returns `MULTITOOL_COMBO_FINAL` with the compacted sentinel, KB sentinel, and both plugin tool results together. |
+| `local-agent-parallel-tools-rag-compaction-debug-chat` | RAG, compacted history, and two plugin tool calls emitted in the same model turn | Bot returns `PARALLEL_COMBO_FINAL` with the compacted sentinel, KB sentinel, and both plugin tool results together. |
 | `local-agent-steering-debug-chat` | Host steering claim, runner pull at turn boundary, and follow-up injection during an active tool loop | Two user messages produce one assistant response containing the steering sentinel. |
 | `local-agent-multimodal-debug-chat` | Image upload, structured input contents, and multimodal runner consumption | UI shows uploaded image and bot returns `IMAGE_OK`; backend receives an image input. |
 | `local-agent-rag-multimodal-debug-chat` | RAG insertion while structured image input is present | UI shows uploaded image, bot returns the KB sentinel, and backend logs the same request with `[Image]`. |
-| `local-agent-nonstreaming-debug-chat` | Host non-streaming adapter path and runner non-streaming invocation | Bot returns `NONSTREAM_OK`; backend completes without the streaming-completed path. |
+| `local-agent-nonstreaming-debug-chat` | Debug Chat non-streaming UI delivery path | Bot returns `NONSTREAM_OK` with the Debug Chat stream switch disabled; runner-internal `invoke_llm` mode is covered by local-agent component tests using `runtime_metadata.streaming_supported=false`. |
 
 ## Full Coverage Matrix
 
@@ -44,14 +49,18 @@ These browser cases are the minimum gate for a local-agent migration check:
 | Multimodal plus RAG | Run `local-agent-rag-multimodal-debug-chat`. | RAG sentinel is still retrievable and the image is not dropped from the user message; exact image-preservation inside the model message is covered by unit tests. |
 | History and context compaction | Run `local-agent-context-compaction-debug-chat` with a small temporary `context-window-tokens` budget. | The runner compacts older history into `<conversation_summary>` and the final answer still recovers the older sentinel from the compacted context. |
 | Streaming model invocation | Enable Debug Chat streaming and ask for `OK`. | UI receives incremental bot output and backend logs streaming completion. |
-| Non-streaming model invocation | Disable Debug Chat streaming or use a non-streaming adapter path. | UI receives a final bot message and backend logs a normal response completion. |
+| Non-streaming UI delivery | Disable Debug Chat streaming. | UI receives a final bot message without frontend streaming errors. |
+| Non-streaming model invocation | Use local-agent component tests with `runtime_metadata.streaming_supported=false` or a host adapter that does not support streaming. | Runner calls `invoke_llm` instead of `invoke_llm_stream` and emits `message.completed`. |
 | Model fallback before first chunk | Configure a failing primary and working fallback, preferably with a controlled test provider. | First model failure does not fail the run; fallback model produces the final answer. |
 | Failure after streaming commit | Use a controlled provider that emits one chunk and then fails. | Runner reports a terminal run failure and does not fallback after partial output. |
 | No authorized model | Clear model config or configure a model not in run resources. | Runner returns `runner.no_model` instead of calling an unauthorized model. |
 | MCP tool call | Use `qa-local-stdio` and `qa_mcp_echo`. | Bot returns the exact `qa_mcp_echo:<input>` result; `/api/v1/tools` contains `qa_mcp_echo`. |
 | Plugin tool call | Install a fixture plugin exposing a deterministic tool and bind it to the pipeline. | Runner lists the plugin tool and can call it through the same tool loop as MCP tools. |
+| Repeated tool loop | Run `local-agent-multitool-rag-compaction-debug-chat` with `max-tool-iterations` high enough for two serial calls. | Runner calls `qa_plugin_echo` twice and the final answer contains both distinct tool results with RAG and compacted history. |
+| Parallel tool batch | Run `local-agent-parallel-tools-rag-compaction-debug-chat` with `tool-execution-mode: parallel`. | Runner executes two same-turn `qa_plugin_echo` calls and the final answer contains both results with RAG and compacted history. |
+| Tool iteration limit | Run `local-agent-tool-loop-limit-debug-chat` with `max-tool-iterations: 2` and a fake provider that keeps requesting tools. | Runner stops and returns `Tool call iteration limit reached...`; it does not keep invoking tools indefinitely. |
 | Run steering | Use `local-agent-steering-debug-chat` with the fixture `qa_plugin_sleep` tool. | A follow-up sent while the sleep tool keeps the run active is claimed into the same run: two user messages, one assistant response, sentinel included. |
-| Tool errors | Make the model request an unauthorized tool or invalid arguments in a controlled unit/component test. | Tool result contains an error message and the run does not bypass authorization. |
+| Tool errors | Run `local-agent-tool-error-recovery-debug-chat` for plugin tool execution errors; keep unauthorized-tool branches in unit/component tests. | The model receives an `Error:` tool result and returns the recovery sentinel; unauthorized calls remain blocked in component tests. |
 | Tool iteration limit | Use a controlled model/tool fixture that repeatedly requests more tool calls. | Runner stops with `runner.tool_loop_limit` at the configured limit. |
 | Knowledge retrieval | Bind a KB containing a unique sentinel. | Bot returns the sentinel and backend logs LangRAG retrieval. |
 | Legacy `knowledge-base` config | Load a pipeline config using the old single-KB field. | Runner still retrieves from the KB. |

skills/skills/langbot-testing/references/local-agent-runner.md

@@ -36,7 +36,7 @@ If the direct MCP fixture passes but `/api/v1/tools` still shows the old MCP nam
 
 For a multimodal check, upload a small image and ask for a deterministic acknowledgement. Prefer the bundled 64x64 red-square fixture over a 1x1 image because some model providers reject tiny images before the runner path is exercised.
 
-For a non-streaming check, disable the Debug Chat stream switch before sending the prompt.
+For a Debug Chat non-streaming delivery check, disable the Debug Chat stream switch before sending the prompt. This validates the UI/adapter delivery path. Runner-internal non-streaming model invocation is covered by component tests that set `runtime_metadata.streaming_supported=false`.
 
 ## Timeout And Tool Regression Checks
 
@@ -53,9 +53,14 @@ Run these cases before saying the pluginized local-agent behavior is healthy:
 - `local-agent-rag-debug-chat`: LangRAG retrieval reaches the runner and affects the answer.
 - `mcp-stdio-tool-call`: MCP tool discovery and local-agent tool loop.
 - `local-agent-plugin-tool-call-debug-chat`: plugin tool discovery and local-agent tool loop.
+- `local-agent-tool-error-recovery-debug-chat`: plugin tool execution errors are returned to the model as tool results and can produce a final recovery answer.
+- `local-agent-tool-loop-limit-debug-chat`: repeated plugin tool requests stop at the configured max tool iteration limit.
+- `local-agent-combo-rag-compaction-tool-debug-chat`: one Debug Chat run combines compacted history, LangRAG context, and a plugin tool result.
+- `local-agent-multitool-rag-compaction-debug-chat`: one Debug Chat run combines compacted history, LangRAG context, and two serial plugin tool calls.
+- `local-agent-parallel-tools-rag-compaction-debug-chat`: one Debug Chat run combines compacted history, LangRAG context, and two same-turn parallel plugin tool calls.
 - `local-agent-multimodal-debug-chat`: uploaded image reaches `ctx.input.contents`.
 - `local-agent-rag-multimodal-debug-chat`: RAG retrieval still works when the same user message carries an image.
-- `local-agent-nonstreaming-debug-chat`: runner works when the host adapter cannot or should not stream.
+- `local-agent-nonstreaming-debug-chat`: Debug Chat still returns a complete visible response when UI streaming is disabled.
 
 ## Pass Criteria
 

skills/skills/langbot-testing/references/mcp-stdio-testing.md

@@ -79,7 +79,9 @@ node scripts/e2e/mcp-stdio-register.mjs
 It upserts `qa-local-stdio` through the authenticated WebUI session, points it at
 the bundled `qa_mcp_echo_server.py`, then checks `/api/v1/tools` and the MCP
 runtime info. A pass proves LangBot has refreshed the saved server and exposes
-`qa_mcp_echo` before any model provider is involved.
+`qa_mcp_echo` before any model provider is involved. It also writes the resolved
+server UUID to `LANGBOT_MCP_QA_STDIO_SERVER_UUID` in `skills/.env.local`; pipeline
+extension binding uses this UUID, not the human-readable server name.
 
 ## Local-Agent Tool Call Check
 

skills/skills/langbot-testing/suites/agent-runner-release-gate.yaml

@@ -30,6 +30,11 @@ cases:
   - local-agent-context-compaction-debug-chat
   - local-agent-rag-debug-chat
   - local-agent-plugin-tool-call-debug-chat
+  - local-agent-tool-error-recovery-debug-chat
+  - local-agent-tool-loop-limit-debug-chat
+  - local-agent-combo-rag-compaction-tool-debug-chat
+  - local-agent-multitool-rag-compaction-debug-chat
+  - local-agent-parallel-tools-rag-compaction-debug-chat
   - mcp-stdio-register
   - mcp-stdio-tool-call
   - local-agent-nonstreaming-debug-chat

skills/skills/langbot-testing/suites/local-agent-gate.yaml

@@ -14,6 +14,11 @@ cases:
   - local-agent-context-compaction-debug-chat
   - local-agent-rag-debug-chat
   - local-agent-plugin-tool-call-debug-chat
+  - local-agent-tool-error-recovery-debug-chat
+  - local-agent-tool-loop-limit-debug-chat
+  - local-agent-combo-rag-compaction-tool-debug-chat
+  - local-agent-multitool-rag-compaction-debug-chat
+  - local-agent-parallel-tools-rag-compaction-debug-chat
   - local-agent-steering-debug-chat
   - mcp-stdio-tool-call
   - local-agent-nonstreaming-debug-chat