LangBot/docs/LLM_REQUEST_E2E.md

Request-chain acceptance (2026-10-15)

The acceptance suite starts an isolated Core process, a real SDK Plugin Runtime process, and a packaged Runner plugin. It installs the plugin and creates the provider, model and Agent through the same authenticated HTTP APIs used by the application. Agent Debug requests then traverse:

HTTP -> Core Agent orchestration -> Plugin Runtime -> plugin SDK -> Runtime -> Core model manager/request policy -> LiteLLM/provider SDK -> HTTP/SSE provider.

Assertions cover plugin return values, captured provider requests, TCP disconnects and persisted SQLite monitoring records. No requester, action transport, model manager or monitoring implementation is replaced with a mock.

The remote provider is a deterministic local HTTP server. It injects failures and delays reproducibly; these are real process/network integration tests, not a claim that every external provider has been tested. See the separate latency benchmark for the live NewAPI experiment.

Matrix

Measured results on Windows / Python 3.12.7 / LiteLLM 1.88.1:

• Core with SDK companion commit c25ff9b: 31 passed, strict stream closure, including all four historical API cases (106.30 seconds).
• Core with the existing SDK 0.7.11rc5 environment: 27 passed, 4 XFAIL for the reproduced SDK cancellation defect (107.61 seconds).
• Published stable SDK 0.7.11 also reproduced the cancellation defect in the preceding 23-case run. The old and stable wrappers were verified identical.
• Focused Core regressions: 371 passed on Python 3.13.7, ResourceWarnings fatal.
• SDK API and stream flow-control regressions: 507 passed.
• Existing real Runner discovery/orchestration/event-processor E2E: 3 passed.

Area	Cases and evidence
Public APIs	Runner and invocation-bound plugin APIs: message, message with usage, chunk stream and event stream; unchanged SDK entities and nested usage
Historical plugin APIs	Verbatim SDK 0.5.8 proxy and provider-message entities, all four APIs, ordinary calls without run_id
Request configuration	Persisted defaults, call overrides, OpenAI cache key/retention, OpenRouter numeric reasoning budget and Anthropic cache marker
Headers	Explicit application attribution and scoped conversation ID reach the provider
Multimodal/tools	Image data URL and function schema survive plugin serialization and provider conversion
Retry/failure	503 then success, num_retries=0, max_retries=0, 401 without retry, first-event timeout, total timeout, idle timeout and connection loss after content
Recovery	Each fault followed by a successful request through the same running processes
Cancellation	Early close of Runner and plugin APIs, both chunk and event streams; provider socket closes and the next invocation succeeds
Reasoning	Reasoning content shown/hidden with the existing remove_think option; unsupported numeric budget rejected before HTTP
Monitoring	Full cache/reasoning usage, transport timing and structured errors persisted

Historical API files come from SDK tag 0.5.8, commit 41e320b68e899a7b5e74cb48c5246f6ccbf9d73f. They use their own message serializers and the current Runtime transport. This checks old plugin call/response compatibility; it is not a test of running the entire old Runtime against the current Core.

Defects exposed by the full chain

• The plugin bridge flattened model and call policy layers, allowing persisted request_policy.retries to override a call's legacy zero-retry option. LLM policy layers are now resolved separately; unrelated argument merging remains.
• Ordinary plugin calls without a Runner/query returned content but lost usage. Core now creates a trusted, invocation-local query for accounting, using the existing plugin-action query builder. No public payload/signature changes.
• The shipped OpenRouter requester uses OpenAI metadata. Numeric budgets selected the wrong mapping, and LiteLLM's OpenAI chat adapter stripped cache markers. OpenRouter chat now uses its own protocol adapter. Embedding/rerank routing is unchanged, and a regression test reads the actual shipped requester YAML.
• Published SDK 0.7.11 leaves inner streaming generators unowned in several public proxy wrappers. Core's direct TCP close test passes, but the complete plugin chain does not promptly cancel its upstream request. The companion SDK PR #177 adds explicit generator ownership without changing signatures, permissions, action payloads or error translation.

Immediate remote cancellation requires the SDK companion fix. Core's pinned SDK has not been silently replaced or upgraded. The E2E suite executes the close checks against published SDKs and reports their known failure as XFAIL. Set LLM_E2E_REQUIRE_STREAM_CLOSE=1 to require strict passing with the patched SDK; newer SDK versions must pass without this override. Historical API checks are explicitly skipped unless their released source files are supplied.

Reproduction

python -m pytest tests/e2e/test_llm_request_chain.py -q --tb=short

For the historical API matrix, export the two unmodified files below from SDK tag 0.5.8 into a temporary directory, named legacy_api.py and legacy_message.py, then set LLM_E2E_LEGACY_API_DIR to its absolute path:

• src/langbot_plugin/api/proxies/langbot_api.py
• src/langbot_plugin/api/entities/builtin/provider/message.py

For strict cancellation acceptance, install the companion SDK into an isolated test environment and set LLM_E2E_REQUIRE_STREAM_CLOSE=1 before running the suite. The existing CI E2E job discovers this test module automatically. Tests use random localhost ports, test-only credentials and pytest temporary data directories; they leave application data and running user instances untouched.

Limits

The suite does not exercise IM delivery, browser rendering, external provider authentication/billing, native Anthropic/Gemini network services, Codex OAuth, or complete third-party Runner business logic. It validates multimodal/tool request transport rather than image understanding or actual tool execution. Native Codex, retry, reasoning, transport and plugin regressions also run in the focused Core test suite. A Python 3.13 Windows full-app attempt timed out during startup before any LLM request; full process acceptance therefore uses Python 3.12.7, while the focused Core regressions run on Python 3.13.7.