# Local Agent Runner Coverage

Use this matrix when judging whether the external `langbot-team/LocalAgent` plugin still behaves like the old built-in local-agent runner.

The QA target is end-to-end behavior. UI cases prove the host, SDK, plugin runtime, and WebUI work together. Unit or component tests are still needed for negative branches that are hard to trigger reliably through a live provider.

## Code Path Basis

- `LangBot/src/langbot/pkg/agent/runner/context_builder.py` builds the Protocol v1 context from the event envelope: `ctx.input.text`, `ctx.input.contents`, attachments, state, resources, and runtime metadata.
- `LangBot/src/langbot/pkg/agent/runner/pipeline_adapter.py` adapts Pipeline-only fields into `ctx.adapter.extra.prompt`, `ctx.adapter.extra.params`, and optional `ctx.bootstrap.messages`.
- `LangBot/src/langbot/pkg/agent/runner/resource_builder.py` authorizes models, fallback models, rerank models, tools, and knowledge bases for the current run.
- `LangBot/src/langbot/pkg/plugin/handler.py` validates run-scoped model/tool/rerank access and calls the host model provider or tool manager with the current query.
- `langbot-local-agent/components/agent_runner/default.py` selects streaming or non-streaming execution, retrieves RAG context, builds messages, invokes models with fallback, and runs tool loops.
- `langbot-local-agent/pkg/messages.py` prefers the host effective prompt from `ctx.adapter.extra.prompt`, uses `ctx.bootstrap.messages` only as a small bootstrap window, and preserves structured/multimodal input while inserting RAG context.

TODO: Treat `ctx.adapter.extra.prompt` as a temporary Pipeline bridge for old
local-agent behavior parity. It is not the final answer for how user plugins or
host hooks should influence agent behavior after Pipeline is replaced.

## Minimum UI Gate

These browser cases are the minimum gate for a local-agent migration check:

| Case | Path Covered | Expected Behavior |
| --- | --- | --- |
| `local-agent-basic-debug-chat` | Streaming LLM invocation with effective host context | Bot returns deterministic `OK`; backend logs streaming completion. |
| `local-agent-model-fallback-before-first-chunk-debug-chat` | Primary/fallback selection before stream commitment | Provider records failed primary requests and a successful fallback request; Debug Chat completes. |
| `local-agent-streaming-post-commit-failure-debug-chat` | Terminal provider failure after stream commitment | Provider records the primary error event, fallback request count stays zero, and Debug Chat reports a controlled failure. |
| `local-agent-effective-prompt-debug-chat` | PromptPreProcessing and host effective prompt handoff through `ctx.adapter.extra.prompt` | Bot returns `PROMPT_PREPROCESS_OK` from the fixture prompt probe. |
| `local-agent-context-compaction-debug-chat` | Runner-owned context budgeting and old-history compaction | Automation temporarily shrinks the runner context window, sends multi-turn Debug Chat history, and the bot still recovers the older sentinel. |
| `local-agent-rag-debug-chat` | Knowledge-base authorization, retrieval, and RAG prompt insertion | Bot returns the KB sentinel, not a generic answer. |
| `mcp-stdio-tool-call` | MCP stdio discovery, tool detail, model function calling, and tool execution | Bot returns `qa_mcp_echo:<input>` and backend logs the MCP tool call. |
| `local-agent-plugin-tool-call-debug-chat` | Plugin tool discovery, tool detail, model function calling, and tool execution | Bot returns `qa-plugin-smoke:<input>` and backend logs the plugin tool call. |
| `local-agent-tool-error-recovery-debug-chat` | Plugin tool execution failure, model-facing error tool result, and final recovery turn | Bot returns `TOOL_ERROR_RECOVERY_FINAL...` only after the model sees the tool error result. |
| `local-agent-tool-loop-limit-debug-chat` | Repeated plugin tool requests and max tool iteration enforcement | Bot returns the runner limit message instead of looping indefinitely. |
| `local-agent-combo-rag-compaction-tool-debug-chat` | RAG, compacted history, and plugin tool execution in the same Debug Chat run | Bot returns `COMBO_FINAL` with the compacted sentinel, KB sentinel, and plugin tool result together. |
| `local-agent-multitool-rag-compaction-debug-chat` | RAG, compacted history, and repeated serial plugin tool loop turns in the same Debug Chat run | Bot returns `MULTITOOL_COMBO_FINAL` with the compacted sentinel, KB sentinel, and both plugin tool results together. |
| `local-agent-parallel-tools-rag-compaction-debug-chat` | RAG, compacted history, and two plugin tool calls emitted in the same model turn | Bot returns `PARALLEL_COMBO_FINAL` with the compacted sentinel, KB sentinel, and both plugin tool results together. |
| `local-agent-steering-debug-chat` | Host steering claim, runner pull at turn boundary, and follow-up injection during an active tool loop | Two user messages produce one assistant response containing the steering sentinel. |
| `local-agent-multimodal-debug-chat` | Image upload, structured input contents, and multimodal runner consumption | UI shows uploaded image and bot returns `IMAGE_OK`; backend receives an image input. |
| `local-agent-rag-multimodal-debug-chat` | RAG insertion while structured image input is present | UI shows uploaded image, bot returns the KB sentinel, and backend logs the same request with `[Image]`. |
| `local-agent-nonstreaming-debug-chat` | Debug Chat non-streaming UI delivery path | Bot returns `NONSTREAM_OK` with the Debug Chat stream switch disabled; runner-internal `invoke_llm` mode is covered by local-agent component tests using `runtime_metadata.streaming_supported=false`. |
| `local-agent-complex-coding-task-debug-chat` | Sustained native-tool planning, multi-file edits, iterative test recovery, and artifact production | Bot returns the completion sentinel only after host verification reruns 12 tests, acceptance, and protected-file checks. |

## Full Coverage Matrix

| Area | How To Cover | Pass Signal |
| --- | --- | --- |
| Effective prompt | Use the `qa-plugin-smoke` prompt probe and send `qa-effective-prompt`. | The answer follows `query.prompt.messages` and returns `PROMPT_PREPROCESS_OK`; plugin-local fallback config prompt is not used when host prompt exists. |
| Current text input | Send a deterministic text-only Debug Chat prompt. | `ctx.input.text` becomes the user text and the bot answers the text request. |
| Structured input contents | Upload an image with text in Debug Chat. | User message shows the image; backend log or request payload contains image content; model can acknowledge it. |
| Multimodal plus RAG | Run `local-agent-rag-multimodal-debug-chat`. | RAG sentinel is still retrievable and the image is not dropped from the user message; exact image-preservation inside the model message is covered by unit tests. |
| History and context compaction | Run `local-agent-context-compaction-debug-chat` with a small temporary `context-window-tokens` budget. | The runner compacts older history into `<conversation_summary>` and the final answer still recovers the older sentinel from the compacted context. |
| Streaming model invocation | Enable Debug Chat streaming and ask for `OK`. | UI receives incremental bot output and backend logs streaming completion. |
| Non-streaming UI delivery | Disable Debug Chat streaming. | UI receives a final bot message without frontend streaming errors. |
| Non-streaming model invocation | Use local-agent component tests with `runtime_metadata.streaming_supported=false` or a host adapter that does not support streaming. | Runner calls `invoke_llm` instead of `invoke_llm_stream` and emits `message.completed`. |
| Model fallback before first chunk | Run `local-agent-model-fallback-before-first-chunk-debug-chat`. | Provider records both the failed primary and successful fallback model requests; the final user request succeeds. |
| Failure after streaming commit | Run `local-agent-streaming-post-commit-failure-debug-chat`. | Provider records a post-content error event, Runner reports terminal failure, and fallback request count remains zero. |
| No authorized model | Clear model config or configure a model not in run resources. | Runner returns `runner.no_model` instead of calling an unauthorized model. |
| MCP tool call | Use `qa-local-stdio` and `qa_mcp_echo`. | Bot returns the exact `qa_mcp_echo:<input>` result; `/api/v1/tools` contains `qa_mcp_echo`. |
| Plugin tool call | Install a fixture plugin exposing a deterministic tool and bind it to the pipeline. | Runner lists the plugin tool and can call it through the same tool loop as MCP tools. |
| Repeated tool loop | Run `local-agent-multitool-rag-compaction-debug-chat` with `max-tool-iterations` high enough for two serial calls. | Runner calls `qa_plugin_echo` twice and the final answer contains both distinct tool results with RAG and compacted history. |
| Parallel tool batch | Run `local-agent-parallel-tools-rag-compaction-debug-chat` with `tool-execution-mode: parallel`. | Runner executes two same-turn `qa_plugin_echo` calls and the final answer contains both results with RAG and compacted history. |
| Tool iteration limit | Run `local-agent-tool-loop-limit-debug-chat` with `max-tool-iterations: 2` and a fake provider that keeps requesting tools. | Runner stops and returns `Tool call iteration limit reached...`; it does not keep invoking tools indefinitely. |
| Run steering | Use `local-agent-steering-debug-chat` with the fixture `qa_plugin_sleep` tool. | A follow-up sent while the sleep tool keeps the run active is claimed into the same run: two user messages, one assistant response, sentinel included. |
| Tool errors | Run `local-agent-tool-error-recovery-debug-chat` for plugin tool execution errors; keep unauthorized-tool branches in unit/component tests. | The model receives an `Error:` tool result and returns the recovery sentinel; unauthorized calls remain blocked in component tests. |
| Tool iteration limit | Use a controlled model/tool fixture that repeatedly requests more tool calls. | Runner stops with `runner.tool_loop_limit` at the configured limit. |
| Knowledge retrieval | Bind a KB containing a unique sentinel. | Bot returns the sentinel and backend logs LangRAG retrieval. |
| Legacy `knowledge-base` config | Load a pipeline config using the old single-KB field. | Runner still retrieves from the KB. |
| Rerank | Configure `rerank-model` and `rerank-top-k` with a working rerank provider. | Retrieval order follows rerank output; unauthorized or failing rerank falls back to original retrieval order. |
| Remove-think | Enable output `remove-think` on a model that emits think tags. | Final visible output omits think content on both streaming and non-streaming paths. |
| Model extra args | Configure provider/model extra args and run Debug Chat. | Host merges persisted model extra args before provider invocation. |
| Query-aware tools | Call a tool that needs the current Query/session context. | Tool receives the active query and behaves the same as it did under the built-in runner. |
| Params filtering | Add public and secret-like variables before the run. | Public params are visible to the runner; `_internal`, token, key, password, and credential fields are filtered. |
| Actor/session context | Run through Debug Chat and at least one platform adapter path. | `conversation`, `actor`, `subject`, and state scopes contain stable IDs for the current launcher and sender. |

## Reporting Rules

When reporting a local-agent QA result, separate these categories:

- `Passed by UI`: path was verified through browser-visible behavior and backend/network evidence.
- `Covered by unit/component tests`: path is deterministic in tests but not practical as a live UI case.
- `Not covered`: path still needs a fixture or provider setup.
- `Environment issue`: provider channel, proxy, OAuth, or external marketplace/network problem outside the runner path.

Do not mark the whole runner healthy based only on a single text Debug Chat response.
