mirror of
https://github.com/langbot-app/LangBot.git
synced 2026-08-17 07:50:59 +00:00
feat(agent-runner): finalize 4.x processor integration
This commit is contained in:
huanghuoguoguo
@@ -34,7 +34,7 @@ automation_env:
|
||||
- LANGBOT_E2E_RESPONSE_TIMEOUT_MS
|
||||
automation_pipeline_url_env: LANGBOT_ACP_AGENT_RUNNER_PIPELINE_URL
|
||||
automation_pipeline_name_env: LANGBOT_ACP_AGENT_RUNNER_PIPELINE_NAME
|
||||
automation_expected_runner_id: "plugin:langbot/acp-agent-runner/default"
|
||||
automation_expected_runner_id: "plugin:langbot-team/ACPAgentRunner/default"
|
||||
automation_prompt: "Use the injected LangBot MCP server tool langbot_get_current_event once. If the MCP call succeeds, reply with exactly ACP_AGENT_RUNNER_E2E_OK."
|
||||
automation_expected_text: "ACP_AGENT_RUNNER_E2E_OK"
|
||||
automation_response_timeout_ms: "300000"
|
||||
@@ -49,7 +49,7 @@ preconditions:
|
||||
steps:
|
||||
- "Open LANGBOT_FRONTEND_URL."
|
||||
- "Open the ACP AgentRunner QA pipeline."
|
||||
- "Confirm the pipeline AI runner is plugin:langbot/acp-agent-runner/default."
|
||||
- "Confirm the pipeline AI runner is plugin:langbot-team/ACPAgentRunner/default."
|
||||
- "Open Debug Chat."
|
||||
- "Ask the real remote Claude ACP agent to call langbot_get_current_event and return ACP_AGENT_RUNNER_E2E_OK exactly."
|
||||
checks:
|
||||
@@ -72,7 +72,7 @@ success_patterns:
|
||||
failure_patterns:
|
||||
- "acp.command_not_found"
|
||||
- "acp.process_exited"
|
||||
- "Agent runner plugin:langbot/acp-agent-runner/default execution failed"
|
||||
- "Agent runner plugin:langbot-team/ACPAgentRunner/default execution failed"
|
||||
troubleshooting:
|
||||
- backend-not-listening
|
||||
- plugin-runtime-timeout
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
id: agent-runner-health-visibility
|
||||
title: "Agent configuration shows whether its selected runner is usable"
|
||||
mode: agent-browser
|
||||
area: agent
|
||||
type: feature
|
||||
priority: p0
|
||||
risk: medium
|
||||
ci_eligible: false
|
||||
tags:
|
||||
- agent-runner
|
||||
- agent
|
||||
- productization
|
||||
- release-gate
|
||||
skills:
|
||||
- langbot-env-setup
|
||||
- langbot-testing
|
||||
env:
|
||||
- LANGBOT_FRONTEND_URL
|
||||
- LANGBOT_BACKEND_URL
|
||||
- LANGBOT_BROWSER_PROFILE
|
||||
automation: scripts/e2e/agent-runner-health-visibility.mjs
|
||||
automation_env:
|
||||
- LANGBOT_FRONTEND_URL
|
||||
- LANGBOT_BACKEND_URL
|
||||
- LANGBOT_BROWSER_PROFILE
|
||||
- LANGBOT_CHROMIUM_EXECUTABLE
|
||||
preconditions:
|
||||
- "The plugin runtime is enabled and connected, with at least one AgentRunner registered."
|
||||
- "The target is a local test instance where a temporary Agent may be created and deleted."
|
||||
steps:
|
||||
- "Read the live plugin runtime status and select a registered runner from Agent metadata."
|
||||
- "Create a temporary Agent using that runner and open its Runner settings in the WebUI."
|
||||
- "Confirm the UI reports that the runner is ready."
|
||||
- "Replace the temporary Agent binding with a syntactically valid but unregistered runner id."
|
||||
- "Reload Runner settings and confirm the UI reports that the selected runner is unavailable."
|
||||
checks:
|
||||
- "UI: A registered runner and connected plugin runtime produce a visible ready status."
|
||||
- "UI: A stale runner binding produces a visible unavailable status with recovery actions."
|
||||
- "API: Plugin runtime status is connected and Agent metadata contains the registered runner."
|
||||
- "Console: No unexpected frontend errors appear during the flow."
|
||||
- "Cleanup: The temporary Agent is deleted after evidence is collected."
|
||||
evidence_required:
|
||||
- ui
|
||||
- screenshot
|
||||
- console
|
||||
- api_diagnostic
|
||||
diagnostics:
|
||||
- "A disconnected plugin runtime is env_issue, not a passing unavailable-runner product check."
|
||||
- "A ready status inferred only from API metadata without opening the Runner UI is not a pass."
|
||||
troubleshooting:
|
||||
- backend-not-listening
|
||||
- plugin-runtime-timeout
|
||||
- proxy-env-mismatch
|
||||
@@ -45,7 +45,7 @@ steps:
|
||||
- "Run the local-agent primary model test endpoint unless LANGBOT_PREFLIGHT_TEST_MODELS=0."
|
||||
checks:
|
||||
- "API diagnostic: api-diagnostic.json has no blockers and no env_issues."
|
||||
- "API diagnostic: required pipelines resolve to plugin:langbot/local-agent/default and plugin:langbot/acp-agent-runner/default."
|
||||
- "API diagnostic: required pipelines resolve to plugin:langbot-team/LocalAgent/default and plugin:langbot-team/ACPAgentRunner/default."
|
||||
- "API diagnostic: qa_plugin_echo is exposed by /api/v1/tools."
|
||||
- "API diagnostic: local-agent model check catches invalid credentials or missing func_call/vision before release E2E starts."
|
||||
- "Secret safety: token values, api keys, and provider secrets are not printed."
|
||||
|
||||
@@ -38,7 +38,7 @@ steps:
|
||||
- "Open LANGBOT_FRONTEND_URL."
|
||||
- "Navigate to Pipelines and open the target local-agent pipeline."
|
||||
- "Open Configuration > AI."
|
||||
- "Use runner Default or the pluginized langbot/local-agent runner."
|
||||
- "Use runner Default or the pluginized langbot-team/LocalAgent runner."
|
||||
- "Select a model that is known to answer Debug Chat in the current environment."
|
||||
- "Clear Knowledge Bases unless this run intentionally combines RAG with the smoke path."
|
||||
- "Save the pipeline."
|
||||
|
||||
+3
-3
@@ -35,7 +35,7 @@ automation_env:
|
||||
- LANGBOT_LOCAL_AGENT_RAG_KB_UUID
|
||||
automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL
|
||||
automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME
|
||||
automation_expected_runner_id: "plugin:langbot/local-agent/default"
|
||||
automation_expected_runner_id: "plugin:langbot-team/LocalAgent/default"
|
||||
automation_runner_config_patch_json: '{"knowledge-bases":["${LANGBOT_LOCAL_AGENT_RAG_KB_UUID}"],"retrieval-top-k":1,"context-window-tokens":650,"context-reserve-tokens":180,"context-keep-recent-tokens":120,"context-summary-tokens":220,"max-tool-iterations":4,"tool-execution-mode":"serial"}'
|
||||
automation_extensions_patch_json: '{"enable_all_plugins":false,"bound_plugins":[{"author":"langbot","name":"local-agent"},{"author":"qa","name":"plugin-smoke"}],"enable_all_mcp_servers":false,"bound_mcp_servers":[],"enable_all_skills":false,"bound_skills":[]}'
|
||||
automation_restore_runner_config: "1"
|
||||
@@ -59,7 +59,7 @@ preconditions:
|
||||
steps:
|
||||
- "Ensure the local-agent pipeline, LangRAG sentinel KB, and qa-plugin-smoke fixture plugin are available."
|
||||
- "Temporarily bind the LangRAG KB, shrink the runner context budget, and allow serial plugin tool execution."
|
||||
- "Temporarily bind only langbot/local-agent and qa/plugin-smoke; disable MCP servers and skills to isolate the tool surface."
|
||||
- "Temporarily bind only langbot-team/LocalAgent and qa/plugin-smoke; disable MCP servers and skills to isolate the tool surface."
|
||||
- "Reset the person Debug Chat session for the target pipeline."
|
||||
- "Send the passcode memory prompt and wait for MEMORY_SET."
|
||||
- "Send the long pressure prompt and wait for COMBO_CONTEXT_PRESSURE_READY."
|
||||
@@ -69,7 +69,7 @@ checks:
|
||||
- "UI: All three user messages appear in Debug Chat."
|
||||
- "UI: The final Bot message contains COMBO_FINAL qa_combo_compaction_sentinel_2406 azalea-cobalt-7421 qa-plugin-smoke:combo-tool-ok-local-agent."
|
||||
- "API diagnostic: pipeline-config-diagnostic.json shows knowledge-bases, context token settings, and tool-loop settings were patched."
|
||||
- "API diagnostic: pipeline-extensions-diagnostic.json shows only langbot/local-agent and qa/plugin-smoke were bound."
|
||||
- "API diagnostic: pipeline-extensions-diagnostic.json shows only langbot-team/LocalAgent and qa/plugin-smoke were bound."
|
||||
- "API diagnostic: restore diagnostics show the original runner config and extension bindings were restored."
|
||||
- "Logs: Backend completes RAG retrieval, plugin tool call, and Debug Chat response without runner timeout or model setup failure."
|
||||
- "Console: No unexpected frontend runtime errors appear during the run."
|
||||
|
||||
@@ -31,7 +31,7 @@ automation_env:
|
||||
- LANGBOT_LOCAL_AGENT_PIPELINE_NAME
|
||||
automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL
|
||||
automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME
|
||||
automation_expected_runner_id: "plugin:langbot/local-agent/default"
|
||||
automation_expected_runner_id: "plugin:langbot-team/LocalAgent/default"
|
||||
automation_runner_config_patch_json: '{"context-window-tokens":225,"context-reserve-tokens":50,"context-keep-recent-tokens":30,"context-summary-tokens":105,"knowledge-bases":[]}'
|
||||
automation_extensions_patch_json: '{"enable_all_plugins":false,"bound_plugins":[{"author":"langbot","name":"local-agent"}],"enable_all_mcp_servers":false,"bound_mcp_servers":[],"enable_all_skills":false,"bound_skills":[]}'
|
||||
automation_restore_runner_config: "1"
|
||||
@@ -51,7 +51,7 @@ preconditions:
|
||||
steps:
|
||||
- "Open the target local-agent pipeline through LANGBOT_FRONTEND_URL."
|
||||
- "Use the authenticated browser token only inside automation to GET and PUT /api/v1/pipelines/{uuid}."
|
||||
- "Assert the saved runner is plugin:langbot/local-agent/default."
|
||||
- "Assert the saved runner is plugin:langbot-team/LocalAgent/default."
|
||||
- "Temporarily set context-window-tokens, context-reserve-tokens, context-keep-recent-tokens, and context-summary-tokens to force compaction, and clear knowledge-bases so RAG does not answer the memory question."
|
||||
- "Temporarily bind only the local-agent plugin, disable MCP servers, and disable skills so unrelated visible tools cannot answer or distract the memory question."
|
||||
- "Reset the person Debug Chat session for the target pipeline."
|
||||
@@ -64,7 +64,7 @@ checks:
|
||||
- "UI: The final Bot message contains qa_compaction_sentinel_7391."
|
||||
- "API diagnostic: pipeline-config-diagnostic.json shows patched=true and patch_keys include the four token context compaction fields plus knowledge-bases."
|
||||
- "API diagnostic: pipeline-config-restore-diagnostic.json shows the original runner config was restored."
|
||||
- "API diagnostic: pipeline-extensions-diagnostic.json shows enable_all_plugins=false, bound_plugins contains langbot/local-agent, enable_all_mcp_servers=false, and enable_all_skills=false."
|
||||
- "API diagnostic: pipeline-extensions-diagnostic.json shows enable_all_plugins=false, bound_plugins contains langbot-team/LocalAgent, enable_all_mcp_servers=false, and enable_all_skills=false."
|
||||
- "API diagnostic: pipeline-extensions-restore-diagnostic.json shows the original extension bindings were restored."
|
||||
- "Logs: Backend completes the multi-turn Debug Chat path without runner timeout or model setup failure."
|
||||
- "Console: No unexpected frontend runtime errors appear during the run."
|
||||
|
||||
@@ -44,7 +44,7 @@ steps:
|
||||
- "Confirm the fixture plugin is bound to the target pipeline through Extensions."
|
||||
- "Open the target local-agent pipeline."
|
||||
- "Open Configuration > AI."
|
||||
- "Use runner Default or the pluginized langbot/local-agent runner."
|
||||
- "Use runner Default or the pluginized langbot-team/LocalAgent runner."
|
||||
- "Select a model that is known to follow system prompts in Debug Chat."
|
||||
- "Save the pipeline."
|
||||
- "Open Debug Chat."
|
||||
|
||||
@@ -42,7 +42,7 @@ steps:
|
||||
- "Open LANGBOT_FRONTEND_URL."
|
||||
- "Navigate to Pipelines and open the target local-agent pipeline."
|
||||
- "Open Configuration > AI."
|
||||
- "Use runner Default or the pluginized langbot/local-agent runner."
|
||||
- "Use runner Default or the pluginized langbot-team/LocalAgent runner."
|
||||
- "Select a model that supports image input in the current environment, or use a known model that at least accepts the uploaded image payload."
|
||||
- "Save the pipeline."
|
||||
- "Open Debug Chat."
|
||||
|
||||
+3
-3
@@ -36,7 +36,7 @@ automation_env:
|
||||
- LANGBOT_LOCAL_AGENT_RAG_KB_UUID
|
||||
automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL
|
||||
automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME
|
||||
automation_expected_runner_id: "plugin:langbot/local-agent/default"
|
||||
automation_expected_runner_id: "plugin:langbot-team/LocalAgent/default"
|
||||
automation_runner_config_patch_json: '{"knowledge-bases":["${LANGBOT_LOCAL_AGENT_RAG_KB_UUID}"],"retrieval-top-k":1,"context-window-tokens":900,"context-reserve-tokens":220,"context-keep-recent-tokens":160,"context-summary-tokens":260,"max-tool-iterations":5,"tool-execution-mode":"serial"}'
|
||||
automation_extensions_patch_json: '{"enable_all_plugins":false,"bound_plugins":[{"author":"langbot","name":"local-agent"},{"author":"qa","name":"plugin-smoke"}],"enable_all_mcp_servers":false,"bound_mcp_servers":[],"enable_all_skills":false,"bound_skills":[]}'
|
||||
automation_restore_runner_config: "1"
|
||||
@@ -60,7 +60,7 @@ preconditions:
|
||||
steps:
|
||||
- "Ensure the local-agent pipeline, LangRAG sentinel KB, and qa-plugin-smoke fixture plugin are available."
|
||||
- "Temporarily bind the LangRAG KB, shrink the runner context budget, and set max-tool-iterations high enough for two serial plugin tool calls."
|
||||
- "Temporarily bind only langbot/local-agent and qa/plugin-smoke; disable MCP servers and skills to isolate the tool surface."
|
||||
- "Temporarily bind only langbot-team/LocalAgent and qa/plugin-smoke; disable MCP servers and skills to isolate the tool surface."
|
||||
- "Reset the person Debug Chat session for the target pipeline."
|
||||
- "Send the passcode memory prompt and wait for MEMORY_SET."
|
||||
- "Send the long pressure prompt and wait for MULTITOOL_CONTEXT_PRESSURE_READY."
|
||||
@@ -70,7 +70,7 @@ checks:
|
||||
- "UI: All three user messages appear in Debug Chat."
|
||||
- "UI: The final Bot message contains MULTITOOL_COMBO_FINAL with the compacted sentinel, azalea-cobalt-7421, and both qa-plugin-smoke tool results."
|
||||
- "API diagnostic: pipeline-config-diagnostic.json shows knowledge-bases, context token settings, and max-tool-iterations were patched."
|
||||
- "API diagnostic: pipeline-extensions-diagnostic.json shows only langbot/local-agent and qa/plugin-smoke were bound."
|
||||
- "API diagnostic: pipeline-extensions-diagnostic.json shows only langbot-team/LocalAgent and qa/plugin-smoke were bound."
|
||||
- "Logs: Backend completes RAG retrieval, two plugin tool calls, and Debug Chat response without runner timeout or model setup failure."
|
||||
- "Console: No unexpected frontend runtime errors appear during the run."
|
||||
evidence_required:
|
||||
|
||||
@@ -39,7 +39,7 @@ steps:
|
||||
- "Open LANGBOT_FRONTEND_URL."
|
||||
- "Navigate to Pipelines and open the target local-agent pipeline."
|
||||
- "Open Configuration > AI."
|
||||
- "Use runner Default or the pluginized langbot/local-agent runner."
|
||||
- "Use runner Default or the pluginized langbot-team/LocalAgent runner."
|
||||
- "Select a model that is known to answer Debug Chat in the current environment."
|
||||
- "Save the pipeline."
|
||||
- "Open Debug Chat."
|
||||
|
||||
+3
-3
@@ -36,7 +36,7 @@ automation_env:
|
||||
- LANGBOT_LOCAL_AGENT_RAG_KB_UUID
|
||||
automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL
|
||||
automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME
|
||||
automation_expected_runner_id: "plugin:langbot/local-agent/default"
|
||||
automation_expected_runner_id: "plugin:langbot-team/LocalAgent/default"
|
||||
automation_runner_config_patch_json: '{"knowledge-bases":["${LANGBOT_LOCAL_AGENT_RAG_KB_UUID}"],"retrieval-top-k":1,"context-window-tokens":900,"context-reserve-tokens":220,"context-keep-recent-tokens":160,"context-summary-tokens":260,"max-tool-iterations":3,"tool-execution-mode":"parallel"}'
|
||||
automation_extensions_patch_json: '{"enable_all_plugins":false,"bound_plugins":[{"author":"langbot","name":"local-agent"},{"author":"qa","name":"plugin-smoke"}],"enable_all_mcp_servers":false,"bound_mcp_servers":[],"enable_all_skills":false,"bound_skills":[]}'
|
||||
automation_restore_runner_config: "1"
|
||||
@@ -60,7 +60,7 @@ preconditions:
|
||||
steps:
|
||||
- "Ensure the local-agent pipeline, LangRAG sentinel KB, and qa-plugin-smoke fixture plugin are available."
|
||||
- "Temporarily bind the LangRAG KB, shrink the runner context budget, and set tool-execution-mode to parallel."
|
||||
- "Temporarily bind only langbot/local-agent and qa/plugin-smoke; disable MCP servers and skills to isolate the tool surface."
|
||||
- "Temporarily bind only langbot-team/LocalAgent and qa/plugin-smoke; disable MCP servers and skills to isolate the tool surface."
|
||||
- "Reset the person Debug Chat session for the target pipeline."
|
||||
- "Send the passcode memory prompt and wait for MEMORY_SET."
|
||||
- "Send the long pressure prompt and wait for PARALLEL_CONTEXT_PRESSURE_READY."
|
||||
@@ -70,7 +70,7 @@ checks:
|
||||
- "UI: All three user messages appear in Debug Chat."
|
||||
- "UI: The final Bot message contains PARALLEL_COMBO_FINAL with the compacted sentinel, azalea-cobalt-7421, and both qa-plugin-smoke tool results."
|
||||
- "API diagnostic: pipeline-config-diagnostic.json shows knowledge-bases, context token settings, max-tool-iterations, and tool-execution-mode=parallel were patched."
|
||||
- "API diagnostic: pipeline-extensions-diagnostic.json shows only langbot/local-agent and qa/plugin-smoke were bound."
|
||||
- "API diagnostic: pipeline-extensions-diagnostic.json shows only langbot-team/LocalAgent and qa/plugin-smoke were bound."
|
||||
- "Logs: Backend completes RAG retrieval, both plugin tool calls, and Debug Chat response without runner timeout or model setup failure."
|
||||
- "Console: No unexpected frontend runtime errors appear during the run."
|
||||
evidence_required:
|
||||
|
||||
@@ -46,7 +46,7 @@ steps:
|
||||
- "Confirm /api/v1/tools or the plugin detail shows qa_plugin_echo."
|
||||
- "Confirm the fixture plugin is bound to the target pipeline through Extensions, or that all plugins are enabled."
|
||||
- "Open the target local-agent pipeline."
|
||||
- "Use runner Default or the pluginized langbot/local-agent runner."
|
||||
- "Use runner Default or the pluginized langbot-team/LocalAgent runner."
|
||||
- "Select a model with function-calling ability that is known to work with tools in the current environment."
|
||||
- "Open Debug Chat."
|
||||
- "Send: Call the qa_plugin_echo tool with exactly this text: plugin-tool-ok-local-agent. Return only the tool result."
|
||||
|
||||
@@ -31,7 +31,7 @@ automation_env:
|
||||
- LANGBOT_LOCAL_AGENT_RAG_KB_UUID
|
||||
automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL
|
||||
automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME
|
||||
automation_expected_runner_id: "plugin:langbot/local-agent/default"
|
||||
automation_expected_runner_id: "plugin:langbot-team/LocalAgent/default"
|
||||
automation_runner_config_patch_json: '{"knowledge-bases":["${LANGBOT_LOCAL_AGENT_RAG_KB_UUID}"]}'
|
||||
automation_restore_runner_config: "1"
|
||||
automation_reset_debug_chat: "1"
|
||||
|
||||
@@ -29,7 +29,7 @@ automation_env:
|
||||
- LANGBOT_LOCAL_AGENT_PIPELINE_NAME
|
||||
automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL
|
||||
automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME
|
||||
automation_expected_runner_id: "plugin:langbot/local-agent/default"
|
||||
automation_expected_runner_id: "plugin:langbot-team/LocalAgent/default"
|
||||
automation_reset_debug_chat: "1"
|
||||
automation_stream_output: "1"
|
||||
automation_prompt: "You are running the LangBot steering E2E test. First call the qa_plugin_sleep tool with seconds=8 and text=steering-e2e-anchor. Do not answer before the tool result is available. After the tool returns, answer the latest user follow-up. If no follow-up was injected, reply only STEERING_NO_FOLLOWUP."
|
||||
@@ -45,7 +45,7 @@ preconditions:
|
||||
- "The selected model route supports function/tool calling and can call qa_plugin_sleep."
|
||||
steps:
|
||||
- "Open the target local-agent pipeline."
|
||||
- "Assert the saved runner is plugin:langbot/local-agent/default."
|
||||
- "Assert the saved runner is plugin:langbot-team/LocalAgent/default."
|
||||
- "Reset the person Debug Chat session for the target pipeline."
|
||||
- "Assert /api/v1/tools contains qa_plugin_sleep."
|
||||
- "Send the first prompt that instructs the model to call qa_plugin_sleep for 8 seconds."
|
||||
|
||||
@@ -32,7 +32,7 @@ automation_env:
|
||||
- LANGBOT_LOCAL_AGENT_PIPELINE_NAME
|
||||
automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL
|
||||
automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME
|
||||
automation_expected_runner_id: "plugin:langbot/local-agent/default"
|
||||
automation_expected_runner_id: "plugin:langbot-team/LocalAgent/default"
|
||||
automation_runner_config_patch_json: '{"knowledge-bases":[],"max-tool-iterations":3,"tool-execution-mode":"serial"}'
|
||||
automation_extensions_patch_json: '{"enable_all_plugins":false,"bound_plugins":[{"author":"langbot","name":"local-agent"},{"author":"qa","name":"plugin-smoke"}],"enable_all_mcp_servers":false,"bound_mcp_servers":[],"enable_all_skills":false,"bound_skills":[]}'
|
||||
automation_restore_runner_config: "1"
|
||||
@@ -53,7 +53,7 @@ preconditions:
|
||||
- "qa-plugin-smoke exposes qa_plugin_fail; the setup case reinstalls the fixture if an older installed package is missing it."
|
||||
steps:
|
||||
- "Ensure the local-agent pipeline and qa-plugin-smoke fixture plugin are available."
|
||||
- "Temporarily bind only langbot/local-agent plus qa/plugin-smoke and clear RAG knowledge bases."
|
||||
- "Temporarily bind only langbot-team/LocalAgent plus qa/plugin-smoke and clear RAG knowledge bases."
|
||||
- "Reset the person Debug Chat session for the target pipeline."
|
||||
- "Send the TOOL_ERROR_RECOVERY prompt."
|
||||
- "Verify the failing plugin tool result is fed back into the model and the final answer contains the recovery sentinel."
|
||||
@@ -61,7 +61,7 @@ steps:
|
||||
checks:
|
||||
- "UI: The final Bot message contains TOOL_ERROR_RECOVERY_FINAL qa-plugin-smoke forced failure observed."
|
||||
- "API diagnostic: pipeline-config-diagnostic.json shows max-tool-iterations=3 and knowledge-bases cleared."
|
||||
- "API diagnostic: pipeline-extensions-diagnostic.json shows only langbot/local-agent and qa/plugin-smoke were bound."
|
||||
- "API diagnostic: pipeline-extensions-diagnostic.json shows only langbot-team/LocalAgent and qa/plugin-smoke were bound."
|
||||
- "Logs: Backend shows the request completed without runner.timeout, runner.tool_error run failure, or model setup failure."
|
||||
- "Console: No unexpected frontend runtime errors appear during Debug Chat."
|
||||
evidence_required:
|
||||
|
||||
@@ -33,7 +33,7 @@ automation_env:
|
||||
- LANGBOT_LOCAL_AGENT_PIPELINE_NAME
|
||||
automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL
|
||||
automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME
|
||||
automation_expected_runner_id: "plugin:langbot/local-agent/default"
|
||||
automation_expected_runner_id: "plugin:langbot-team/LocalAgent/default"
|
||||
automation_runner_config_patch_json: '{"knowledge-bases":[],"max-tool-iterations":2,"tool-execution-mode":"serial"}'
|
||||
automation_extensions_patch_json: '{"enable_all_plugins":false,"bound_plugins":[{"author":"langbot","name":"local-agent"},{"author":"qa","name":"plugin-smoke"}],"enable_all_mcp_servers":false,"bound_mcp_servers":[],"enable_all_skills":false,"bound_skills":[]}'
|
||||
automation_restore_runner_config: "1"
|
||||
@@ -54,14 +54,14 @@ preconditions:
|
||||
- "qa-plugin-smoke exposes qa_plugin_echo."
|
||||
steps:
|
||||
- "Ensure the local-agent pipeline and qa-plugin-smoke fixture plugin are available."
|
||||
- "Temporarily set max-tool-iterations to 2 and bind only langbot/local-agent plus qa/plugin-smoke."
|
||||
- "Temporarily set max-tool-iterations to 2 and bind only langbot-team/LocalAgent plus qa/plugin-smoke."
|
||||
- "Reset the person Debug Chat session for the target pipeline."
|
||||
- "Send the LOOP_LIMIT prompt and wait for the runner limit message."
|
||||
- "Restore the original runner config and pipeline extension bindings."
|
||||
checks:
|
||||
- "UI: The final Bot message contains Tool call iteration limit reached. I stopped before calling more tools."
|
||||
- "API diagnostic: pipeline-config-diagnostic.json shows max-tool-iterations was patched to 2."
|
||||
- "API diagnostic: pipeline-extensions-diagnostic.json shows only langbot/local-agent and qa/plugin-smoke were bound."
|
||||
- "API diagnostic: pipeline-extensions-diagnostic.json shows only langbot-team/LocalAgent and qa/plugin-smoke were bound."
|
||||
- "Logs: Backend shows the request completed without infinite loop, runner timeout, or model setup failure."
|
||||
- "Console: No unexpected frontend runtime errors appear during Debug Chat."
|
||||
evidence_required:
|
||||
|
||||
@@ -31,7 +31,7 @@ automation_env:
|
||||
- LANGBOT_MCP_QA_STDIO_SERVER_UUID
|
||||
automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL
|
||||
automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME
|
||||
automation_expected_runner_id: "plugin:langbot/local-agent/default"
|
||||
automation_expected_runner_id: "plugin:langbot-team/LocalAgent/default"
|
||||
automation_extensions_patch_json: '{"enable_all_plugins":false,"bound_plugins":[{"author":"langbot","name":"local-agent"}],"enable_all_mcp_servers":false,"bound_mcp_servers":["${LANGBOT_MCP_QA_STDIO_SERVER_UUID}"],"enable_all_skills":false,"bound_skills":[]}'
|
||||
automation_restore_extensions: "1"
|
||||
automation_reset_debug_chat: "1"
|
||||
@@ -65,7 +65,7 @@ steps:
|
||||
- "Submit the server."
|
||||
- "Confirm the server detail page shows Tools: 1 and qa_mcp_echo."
|
||||
- "Open the target local-agent pipeline."
|
||||
- "Use runner Default or the pluginized langbot/local-agent runner."
|
||||
- "Use runner Default or the pluginized langbot-team/LocalAgent runner."
|
||||
- "Select a model with function-calling ability that is known to work with tools in the current environment."
|
||||
- "Open Debug Chat."
|
||||
- "Send: Call the qa_mcp_echo tool with exactly this text: mcp-ok-local-agent. Return only the tool result."
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
id: wizard-runner-marketplace-catalog
|
||||
title: "Quick Start discovers AgentRunner extensions on a clean instance"
|
||||
mode: agent-browser
|
||||
area: wizard
|
||||
type: feature
|
||||
priority: p0
|
||||
risk: high
|
||||
ci_eligible: false
|
||||
tags:
|
||||
- wizard
|
||||
- agent-runner
|
||||
- marketplace
|
||||
- clean-install
|
||||
- productization
|
||||
skills:
|
||||
- langbot-env-setup
|
||||
- langbot-testing
|
||||
env:
|
||||
- LANGBOT_FRONTEND_URL
|
||||
- LANGBOT_BACKEND_URL
|
||||
- LANGBOT_E2E_LOGIN_USER
|
||||
automation: scripts/e2e/wizard-runner-marketplace-catalog.mjs
|
||||
automation_env:
|
||||
- LANGBOT_FRONTEND_URL
|
||||
- LANGBOT_BACKEND_URL
|
||||
- LANGBOT_E2E_LOGIN_USER
|
||||
- LANGBOT_BROWSER_PROFILE
|
||||
- LANGBOT_CHROMIUM_EXECUTABLE
|
||||
preconditions:
|
||||
- "The target is a newly initialized local instance with wizard_status=none."
|
||||
- "No plugins or AgentRunner components are installed."
|
||||
steps:
|
||||
- "Confirm the backend reports zero installed plugins and zero registered runners."
|
||||
- "Resume Quick Start at the AI Engine step with a temporary disabled Bot."
|
||||
- "Confirm the browser requests Marketplace plugins with component_filter=AgentRunner."
|
||||
- "When Space has not published Runner plugins, confirm the safe empty state and Runner Extensions link."
|
||||
- "Confirm Create & Deploy remains disabled until a Runner is installed and selected."
|
||||
- "Verify the layout at desktop and mobile widths."
|
||||
checks:
|
||||
- "API: The instance has zero installed plugins and zero registered runners."
|
||||
- "API: The instance wizard status is none."
|
||||
- "Network: Marketplace search uses component_filter=AgentRunner and type_filter=plugin."
|
||||
- "UI: The AI Engine step remains usable when the Runner catalog is empty."
|
||||
- "UI: Browse Runner Extensions links to the AgentRunner-filtered market."
|
||||
- "UI: Create & Deploy remains disabled without a selected Runner."
|
||||
- "Console: No unexpected frontend errors appear during the flow."
|
||||
- "Cleanup: Wizard progress and the temporary Bot are removed."
|
||||
evidence_required:
|
||||
- ui
|
||||
- screenshot
|
||||
- console
|
||||
- api_diagnostic
|
||||
diagnostics:
|
||||
- "This case validates the clean-instance discovery and empty state. Run the install continuation after Space publishes at least one AgentRunner plugin."
|
||||
- "A Marketplace API response alone is not a pass; the empty or installable Runner UI must be visible."
|
||||
troubleshooting:
|
||||
- backend-not-listening
|
||||
- plugin-runtime-timeout
|
||||
- proxy-env-mismatch
|
||||
@@ -110,7 +110,7 @@ Each probe writes `automation-result.json` and probe logs under
|
||||
| Generic Pipeline Debug Chat | `pipeline-debug-chat` | The WebUI Debug Chat path itself works before runner-specific failures are diagnosed. |
|
||||
| Deterministic QA runner install | `agent-runner-live-install` | A local `.lbpkg` AgentRunner package can install and register a runner. |
|
||||
| Deterministic QA runner Debug Chat | `agent-runner-qa-debug-chat` | The installed QA runner executes through WebUI Debug Chat without a model provider. |
|
||||
| Required runner plugins | `agent-runner-release-preflight` | `langbot/local-agent` and `langbot/acp-agent-runner` are visible to the host. |
|
||||
| Required runner plugins | `agent-runner-release-preflight` | `langbot-team/LocalAgent` and `langbot-team/ACPAgentRunner` are visible to the host. |
|
||||
| Required QA plugin tools | `plugin-e2e-smoke`, `agent-runner-release-preflight`, `qa-plugin-smoke-live-install` | The deterministic `qa_plugin_echo` and `qa_plugin_fail` tools are exposed before tool-loop and tool-error cases start. |
|
||||
| Knowledge base fixture | `langrag-kb-retrieve`, `local-agent-rag-debug-chat` | LangRAG data is queryable and the runner inserts retrieved context. |
|
||||
| Effective prompt bridge | `local-agent-effective-prompt-debug-chat` | Host prompt preprocessing reaches the runner. |
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Dify AgentRunner
|
||||
|
||||
Use this reference when validating `langbot/dify-agent` through LangBot WebUI.
|
||||
Use this reference when validating `langbot-team/DifyAgent` through LangBot WebUI.
|
||||
|
||||
## Prepare Dify
|
||||
|
||||
@@ -38,7 +38,7 @@ Pass only when:
|
||||
|
||||
## Diagnostics
|
||||
|
||||
- `GET /api/v1/pipelines/{uuid}` can confirm the saved runner id is `plugin:langbot/dify-agent/default` and runner config contains `app-type`, `base-url`, and `api-key`.
|
||||
- `GET /api/v1/pipelines/{uuid}` can confirm the saved runner id is `plugin:langbot-team/DifyAgent/default` and runner config contains `app-type`, `base-url`, and `api-key`.
|
||||
- Direct Dify streaming API calls are useful only to distinguish invalid Dify credentials from LangBot runner issues.
|
||||
- If Debug Chat returns `Agent runner execution failed`, inspect backend logs before changing UI settings.
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Local Agent Runner Coverage
|
||||
|
||||
Use this matrix when judging whether the external `langbot/local-agent` plugin still behaves like the old built-in local-agent runner.
|
||||
Use this matrix when judging whether the external `langbot-team/LocalAgent` plugin still behaves like the old built-in local-agent runner.
|
||||
|
||||
The QA target is end-to-end behavior. UI cases prove the host, SDK, plugin runtime, and WebUI work together. Unit or component tests are still needed for negative branches that are hard to trigger reliably through a live provider.
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Local Agent Runner
|
||||
|
||||
Use this reference when validating the pluginized `langbot/local-agent` runner through the WebUI.
|
||||
Use this reference when validating the pluginized `langbot-team/LocalAgent` runner through the WebUI.
|
||||
|
||||
The goal is behavior parity with the old built-in local-agent runner. The code does not need to be identical, but the visible behavior should match: effective prompt, current input, history, model selection and fallback, tool calling, knowledge retrieval, multimodal input, streaming and non-streaming output all have to reach the runner through the host and SDK.
|
||||
|
||||
|
||||
@@ -87,7 +87,7 @@ extension binding uses this UUID, not the human-readable server name.
|
||||
|
||||
1. Open the target pipeline.
|
||||
2. Confirm `Extensions` allows the MCP server, or that all MCP servers are enabled.
|
||||
3. Use runner `Default` or the pluginized `langbot/local-agent` runner.
|
||||
3. Use runner `Default` or the pluginized `langbot-team/LocalAgent` runner.
|
||||
4. Select a model with function-calling ability that is known to work with tools in the current environment.
|
||||
5. Open `Debug Chat`.
|
||||
6. Ask:
|
||||
|
||||
@@ -20,7 +20,7 @@ An old `langbot_plugin` runtime process survived a backend restart, or multiple
|
||||
|
||||
### Fix
|
||||
|
||||
Stop the LangBot backend and any orphaned `langbot_plugin.cli` runtime processes, confirm the configured backend URL is free/reachable as appropriate, then start LangBot again. A healthy startup logs `Connected to plugin runtime`, mounts `langbot/local-agent`, and initializes the default agent runner.
|
||||
Stop the LangBot backend and any orphaned `langbot_plugin.cli` runtime processes, confirm the configured backend URL is free/reachable as appropriate, then start LangBot again. A healthy startup logs `Connected to plugin runtime`, mounts `langbot-team/LocalAgent`, and initializes the default agent runner.
|
||||
|
||||
### Verification
|
||||
|
||||
|
||||
@@ -10,5 +10,7 @@ tags:
|
||||
- release-gate
|
||||
cases:
|
||||
- wizard-scenario-routing
|
||||
- wizard-runner-marketplace-catalog
|
||||
- agent-runner-health-visibility
|
||||
- bot-event-routing-product-flow
|
||||
- wizard-onebot-agent-runtime
|
||||
|
||||
@@ -16,7 +16,7 @@ fix_steps:
|
||||
- "Change metadata.label to the provider or runner family name, such as Dify, Local Agent, Coze, DashScope, Langflow, n8n, or Tbox."
|
||||
- "Restart LangBot or plugin runtime to refresh runner metadata cache."
|
||||
- "Check /api/v1/pipelines/_/metadata and the WebUI runner selector."
|
||||
verification: "Runner options are distinguishable by visible label while their ids remain stable, such as plugin:langbot/dify-agent/default."
|
||||
verification: "Runner options are distinguishable by visible label while their ids remain stable, such as plugin:langbot-team/DifyAgent/default."
|
||||
related_cases:
|
||||
- dify-agent-debug-chat
|
||||
- local-agent-rag-debug-chat
|
||||
|
||||
@@ -22,7 +22,7 @@ fix_steps:
|
||||
- "Stop the LangBot backend."
|
||||
- "Stop orphaned langbot_plugin.cli runtime processes."
|
||||
- "Confirm the configured backend URL is free or reachable as appropriate."
|
||||
- "Start LangBot again and wait for Connected to plugin runtime plus langbot/local-agent initialization."
|
||||
- "Start LangBot again and wait for Connected to plugin runtime plus langbot-team/LocalAgent initialization."
|
||||
- "For disposable test data only, move suspected fixture plugins out of data/plugins and restart to isolate plugin discovery failures."
|
||||
verification: "Open Installed Plugins and confirm the plugin list loads, then run pipeline-debug-chat and confirm the UI shows a Bot response while backend logs include Streaming completed."
|
||||
related_cases:
|
||||
|
||||
Reference in New Issue
Block a user