mirror of
https://github.com/langbot-app/LangBot.git
synced 2026-10-22 16:35:20 +08:00
RockChinQandHyu
616981b0cf
* fix(llm): centralize request policy and preserve stream diagnostics * test(llm): dispose SQLite engine after migration regression * perf(llm): reuse connections and unify reasoning and prompt cache policy * fix(llm): preserve explicit native prompt cache options * perf(llm): fix measured TLS setup and streaming reuse overhead * fix(llm): resolve plugin request-chain regressions found by E2E --------- Co-authored-by: Hyu <[email protected]>
101 lines
5.5 KiB
Markdown
101 lines
5.5 KiB
Markdown
# LLM request-layer benchmark (2026-10-15)
|
|
|
|
This controlled test did not reproduce a large LangBot slowdown. Warm request
|
|
preparation was sub-millisecond; most measured latency occurred upstream.
|
|
It did identify two local transport problems, now fixed in this PR.
|
|
|
|
## Method
|
|
|
|
One Windows host, Python 3.12.7, LiteLLM 1.88.1 and OpenCode 2.0.16 used the
|
|
same user-provided NewAPI gateway and requested `claude-haiku-4.5`. Each client
|
|
made one warmup and six measured streaming calls, alternating all six client
|
|
orders. The synthetic system prefix was about 6,000 tokens. Parameters were
|
|
`temperature=0`, `max_tokens=256`, `stream=true`, `include_usage=true`, with an
|
|
ephemeral cache marker on the system text. No reasoning budget was requested.
|
|
|
|
The measurement relay deliberately forwards identical serialized requests and
|
|
headers for all three clients. It replaces OpenCode's added agent/environment
|
|
instructions and normalizes SDK request differences. The canonical SHA-256 is
|
|
`7834a66a188e70b4eb1b6234b3a14be4c66b8c72dd8a2c06395fe2505c1c371b`.
|
|
OpenCode runs as an isolated resident server, using its
|
|
[versioned session API](https://github.com/anomalyco/opencode/blob/v2.0.16/packages/protocol/openapi.json).
|
|
Process startup and session creation are outside the warm dispatch measurement.
|
|
|
|
All clients share the relay's upstream connection pool. Therefore this experiment
|
|
isolates local request dispatch and upstream response timing; it does **not**
|
|
compare their independent direct-to-provider TLS/proxy behavior. It also excludes
|
|
the complete bot pipeline, Runner/plugin RPC and OpenCode UI rendering. It cannot
|
|
establish which application is faster in every production deployment.
|
|
|
|
## Results
|
|
|
|
Medians, milliseconds; six measured samples per client:
|
|
|
|
| Client | Dispatch to relay | First content | Complete upstream response |
|
|
| --- | ---: | ---: | ---: |
|
|
| Direct HTTPX | 3.952 | 1119.541 | 2199.413 |
|
|
| LangBot Core requester | 6.802 | 1089.092 | 2135.935 |
|
|
| OpenCode | 41.158 | 1104.020 | 2148.265 |
|
|
|
|
Dispatch includes client preparation and local HTTP dispatch. First content and
|
|
completion use the same relay boundary for all clients, measured from dispatch;
|
|
they are not UI latency. LangBot's separately instrumented preparation median
|
|
was **0.316 ms**, maximum **0.497 ms**. These small samples and upstream variation
|
|
do not justify interpreting the small TTFT differences as a speed ranking.
|
|
|
|
Every measured response reported 5,990 input tokens, 5,987 cache-read tokens and
|
|
109 output tokens. This demonstrates equal warm cache accounting under the
|
|
normalized upstream request, not native cache-marker forwarding by all clients.
|
|
No cache-miss latency comparison or independent provider-route verification was
|
|
performed. Missing cache accounting is represented as null, never assumed zero.
|
|
The gateway may perform its own routing, retries and cache translation.
|
|
|
|
Sanitized individual samples and summaries are in
|
|
[llm-latency-2026-10-15.json](llm-latency-2026-10-15.json).
|
|
|
|
## Bottlenecks fixed
|
|
|
|
1. The OpenAI SDK stops consuming SSE at `[DONE]` before HTTP framing reaches
|
|
EOF. The healthy connection was then discarded. In the baseline relay run,
|
|
LangBot reused zero of six measured client connections; after bounded terminal
|
|
draining it reused six of six. A real TCP regression test also reproduces this
|
|
with split terminal markers and late usage. OpenRouter's generic adapter
|
|
already consumed EOF. Early close and cancellation still close immediately.
|
|
2. On this host, constructing default HTTPX clients took 1,761 / 1,443 / 1,435 ms.
|
|
Loading one TLS context per requester pool and sharing it across that client's
|
|
proxy mounts reduced measured pool setup to 666 / 430 / 431 ms. Certificate
|
|
loading now runs off the event loop. These are local setup measurements, not
|
|
a claim of equivalent percentage improvement in LLM latency. Runtime lazy
|
|
initialization and process startup remain separate cold costs.
|
|
3. High-resolution monotonic timing replaces the low-resolution clock used on
|
|
Python 3.12 Windows. Preparation no longer misleadingly rounds to zero.
|
|
`transport_setup_ms` exposes pool initialization separately.
|
|
|
|
The relay socket observations concern client-to-relay connections. HTTPX and
|
|
OpenCode have different keepalive policies, so those counts alone are not a
|
|
cross-client pooling ranking. The shared relay also prevents quantifying the
|
|
provider-side handshake savings from the SSE fix in this run.
|
|
|
|
## Reproduce
|
|
|
|
From a development environment with LangBot dependencies installed, set
|
|
`LLM_BENCHMARK_API_KEY` securely in the environment, then run:
|
|
|
|
```sh
|
|
python scripts/llm_latency_probe.py --opencode /path/to/opencode \
|
|
--opencode-version 2.0.16 --base-url https://gateway.example/v1 \
|
|
--model claude-haiku-4.5 --long-prefix --cache-control --rounds 6 \
|
|
--output /path/outside/repository/latency.json
|
|
```
|
|
|
|
This makes 21 billable requests by default. `--mock` uses synthetic local SSE
|
|
instead; it still needs OpenCode. `--baseline-no-drain` disables terminal draining
|
|
for a transport regression comparison. The harness records only hashes and
|
|
message sizes, not prompt bodies or credentials, and terminates its own isolated
|
|
OpenCode process. Keep raw results outside the repository because they include
|
|
the configured gateway URL. Other OpenCode versions may need API adjustments.
|
|
|
|
The next diagnostic for a reported slow production session should capture the
|
|
actual pipeline/Runner timing, streaming mode, prompt size, reasoning settings,
|
|
cache usage and gateway routing alongside these requester-level measurements.
|