mirror of
https://github.com/langbot-app/LangBot.git
synced 2026-10-27 02:45:24 +08:00
perf(llm): harden request policy and optimize connections, reasoning and prompt caching (#2661)
* fix(llm): centralize request policy and preserve stream diagnostics * test(llm): dispose SQLite engine after migration regression * perf(llm): reuse connections and unify reasoning and prompt cache policy * fix(llm): preserve explicit native prompt cache options * perf(llm): fix measured TLS setup and streaming reuse overhead * fix(llm): resolve plugin request-chain regressions found by E2E --------- Co-authored-by: Hyu <[email protected]>
This commit is contained in:
RockChinQ
@@ -0,0 +1,107 @@
|
||||
# Request-chain acceptance (2026-10-15)
|
||||
|
||||
The acceptance suite starts an isolated Core process, a real SDK Plugin Runtime
|
||||
process, and a packaged Runner plugin. It installs the plugin and creates the
|
||||
provider, model and Agent through the same authenticated HTTP APIs used by the
|
||||
application. Agent Debug requests then traverse:
|
||||
|
||||
HTTP -> Core Agent orchestration -> Plugin Runtime -> plugin SDK -> Runtime ->
|
||||
Core model manager/request policy -> LiteLLM/provider SDK -> HTTP/SSE provider.
|
||||
|
||||
Assertions cover plugin return values, captured provider requests, TCP disconnects
|
||||
and persisted SQLite monitoring records. No requester, action transport, model
|
||||
manager or monitoring implementation is replaced with a mock.
|
||||
|
||||
The remote provider is a deterministic local HTTP server. It injects failures and
|
||||
delays reproducibly; these are real process/network integration tests, **not** a
|
||||
claim that every external provider has been tested. See the separate
|
||||
[latency benchmark](LLM_LATENCY_BENCHMARK.md) for the live NewAPI experiment.
|
||||
|
||||
## Matrix
|
||||
|
||||
Measured results on Windows / Python 3.12.7 / LiteLLM 1.88.1:
|
||||
|
||||
- Core with SDK companion commit `c25ff9b`: **31 passed**, strict stream closure,
|
||||
including all four historical API cases (106.30 seconds).
|
||||
- Core with the existing SDK 0.7.11rc5 environment: **27 passed, 4 XFAIL** for
|
||||
the reproduced SDK cancellation defect (107.61 seconds).
|
||||
- Published stable SDK 0.7.11 also reproduced the cancellation defect in the
|
||||
preceding 23-case run. The old and stable wrappers were verified identical.
|
||||
- Focused Core regressions: **371 passed** on Python 3.13.7, ResourceWarnings fatal.
|
||||
- SDK API and stream flow-control regressions: **507 passed**.
|
||||
- Existing real Runner discovery/orchestration/event-processor E2E: **3 passed**.
|
||||
|
||||
| Area | Cases and evidence |
|
||||
| --- | --- |
|
||||
| Public APIs | Runner and invocation-bound plugin APIs: message, message with usage, chunk stream and event stream; unchanged SDK entities and nested usage |
|
||||
| Historical plugin APIs | Verbatim SDK 0.5.8 proxy and provider-message entities, all four APIs, ordinary calls without `run_id` |
|
||||
| Request configuration | Persisted defaults, call overrides, OpenAI cache key/retention, OpenRouter numeric reasoning budget and Anthropic cache marker |
|
||||
| Headers | Explicit application attribution and scoped conversation ID reach the provider |
|
||||
| Multimodal/tools | Image data URL and function schema survive plugin serialization and provider conversion |
|
||||
| Retry/failure | 503 then success, `num_retries=0`, `max_retries=0`, 401 without retry, first-event timeout, total timeout, idle timeout and connection loss after content |
|
||||
| Recovery | Each fault followed by a successful request through the same running processes |
|
||||
| Cancellation | Early close of Runner and plugin APIs, both chunk and event streams; provider socket closes and the next invocation succeeds |
|
||||
| Reasoning | Reasoning content shown/hidden with the existing `remove_think` option; unsupported numeric budget rejected before HTTP |
|
||||
| Monitoring | Full cache/reasoning usage, transport timing and structured errors persisted |
|
||||
|
||||
Historical API files come from SDK tag `0.5.8`, commit
|
||||
`41e320b68e899a7b5e74cb48c5246f6ccbf9d73f`.
|
||||
They use their own message serializers and the current Runtime transport. This
|
||||
checks old plugin call/response compatibility; it is not a test of running the
|
||||
entire old Runtime against the current Core.
|
||||
|
||||
## Defects exposed by the full chain
|
||||
|
||||
- The plugin bridge flattened model and call policy layers, allowing persisted
|
||||
`request_policy.retries` to override a call's legacy zero-retry option. LLM
|
||||
policy layers are now resolved separately; unrelated argument merging remains.
|
||||
- Ordinary plugin calls without a Runner/query returned content but lost usage.
|
||||
Core now creates a trusted, invocation-local query for accounting, using the
|
||||
existing plugin-action query builder. No public payload/signature changes.
|
||||
- The shipped OpenRouter requester uses OpenAI metadata. Numeric budgets selected
|
||||
the wrong mapping, and LiteLLM's OpenAI chat adapter stripped cache markers.
|
||||
OpenRouter chat now uses its own protocol adapter. Embedding/rerank routing is
|
||||
unchanged, and a regression test reads the actual shipped requester YAML.
|
||||
- Published SDK 0.7.11 leaves inner streaming generators unowned in several public
|
||||
proxy wrappers. Core's direct TCP close test passes, but the complete plugin
|
||||
chain does not promptly cancel its upstream request. The companion
|
||||
[SDK PR #177](https://github.com/langbot-app/langbot-plugin-sdk/pull/177) adds
|
||||
explicit generator ownership without changing signatures, permissions, action
|
||||
payloads or error translation.
|
||||
|
||||
**Immediate remote cancellation requires the SDK companion fix.** Core's pinned
|
||||
SDK has not been silently replaced or upgraded. The E2E suite executes the close
|
||||
checks against published SDKs and reports their known failure as XFAIL. Set
|
||||
`LLM_E2E_REQUIRE_STREAM_CLOSE=1` to require strict passing with the patched SDK;
|
||||
newer SDK versions must pass without this override. Historical API checks are
|
||||
explicitly skipped unless their released source files are supplied.
|
||||
|
||||
## Reproduction
|
||||
|
||||
```sh
|
||||
python -m pytest tests/e2e/test_llm_request_chain.py -q --tb=short
|
||||
```
|
||||
|
||||
For the historical API matrix, export the two unmodified files below from SDK tag
|
||||
`0.5.8` into a temporary directory, named `legacy_api.py` and `legacy_message.py`,
|
||||
then set `LLM_E2E_LEGACY_API_DIR` to its absolute path:
|
||||
|
||||
- `src/langbot_plugin/api/proxies/langbot_api.py`
|
||||
- `src/langbot_plugin/api/entities/builtin/provider/message.py`
|
||||
|
||||
For strict cancellation acceptance, install the companion SDK into an isolated
|
||||
test environment and set `LLM_E2E_REQUIRE_STREAM_CLOSE=1` before running the suite.
|
||||
The existing CI E2E job discovers this test module automatically. Tests use random
|
||||
localhost ports, test-only credentials and pytest temporary data directories;
|
||||
they leave application data and running user instances untouched.
|
||||
|
||||
## Limits
|
||||
|
||||
The suite does not exercise IM delivery, browser rendering, external provider
|
||||
authentication/billing, native Anthropic/Gemini network services, Codex OAuth, or
|
||||
complete third-party Runner business logic. It validates multimodal/tool request
|
||||
transport rather than image understanding or actual tool execution. Native Codex,
|
||||
retry, reasoning, transport and plugin regressions also run in the focused Core
|
||||
test suite. A Python 3.13 Windows full-app attempt timed out during startup before
|
||||
any LLM request; full process acceptance therefore uses Python 3.12.7, while the
|
||||
focused Core regressions run on Python 3.13.7.
|
||||
Reference in New Issue
Block a user