LangBot/docs/LLM_LATENCY_BENCHMARK.md

LLM request-layer benchmark (2026-10-15)

This controlled test did not reproduce a large LangBot slowdown. Warm request preparation was sub-millisecond; most measured latency occurred upstream. It did identify two local transport problems, now fixed in this PR.

Method

One Windows host, Python 3.12.7, LiteLLM 1.88.1 and OpenCode 2.0.16 used the same user-provided NewAPI gateway and requested claude-haiku-4.5. Each client made one warmup and six measured streaming calls, alternating all six client orders. The synthetic system prefix was about 6,000 tokens. Parameters were temperature=0, max_tokens=256, stream=true, include_usage=true, with an ephemeral cache marker on the system text. No reasoning budget was requested.

The measurement relay deliberately forwards identical serialized requests and headers for all three clients. It replaces OpenCode's added agent/environment instructions and normalizes SDK request differences. The canonical SHA-256 is 7834a66a188e70b4eb1b6234b3a14be4c66b8c72dd8a2c06395fe2505c1c371b. OpenCode runs as an isolated resident server, using its versioned session API. Process startup and session creation are outside the warm dispatch measurement.

All clients share the relay's upstream connection pool. Therefore this experiment isolates local request dispatch and upstream response timing; it does not compare their independent direct-to-provider TLS/proxy behavior. It also excludes the complete bot pipeline, Runner/plugin RPC and OpenCode UI rendering. It cannot establish which application is faster in every production deployment.

Results

Medians, milliseconds; six measured samples per client:

Client	Dispatch to relay	First content	Complete upstream response
Direct HTTPX	3.952	1119.541	2199.413
LangBot Core requester	6.802	1089.092	2135.935
OpenCode	41.158	1104.020	2148.265

Dispatch includes client preparation and local HTTP dispatch. First content and completion use the same relay boundary for all clients, measured from dispatch; they are not UI latency. LangBot's separately instrumented preparation median was 0.316 ms, maximum 0.497 ms. These small samples and upstream variation do not justify interpreting the small TTFT differences as a speed ranking.

Every measured response reported 5,990 input tokens, 5,987 cache-read tokens and 109 output tokens. This demonstrates equal warm cache accounting under the normalized upstream request, not native cache-marker forwarding by all clients. No cache-miss latency comparison or independent provider-route verification was performed. Missing cache accounting is represented as null, never assumed zero. The gateway may perform its own routing, retries and cache translation.

Sanitized individual samples and summaries are in llm-latency-2026-10-15.json.

Bottlenecks fixed

1. The OpenAI SDK stops consuming SSE at [DONE] before HTTP framing reaches EOF. The healthy connection was then discarded. In the baseline relay run, LangBot reused zero of six measured client connections; after bounded terminal draining it reused six of six. A real TCP regression test also reproduces this with split terminal markers and late usage. OpenRouter's generic adapter already consumed EOF. Early close and cancellation still close immediately.
2. On this host, constructing default HTTPX clients took 1,761 / 1,443 / 1,435 ms. Loading one TLS context per requester pool and sharing it across that client's proxy mounts reduced measured pool setup to 666 / 430 / 431 ms. Certificate loading now runs off the event loop. These are local setup measurements, not a claim of equivalent percentage improvement in LLM latency. Runtime lazy initialization and process startup remain separate cold costs.
3. High-resolution monotonic timing replaces the low-resolution clock used on Python 3.12 Windows. Preparation no longer misleadingly rounds to zero. transport_setup_ms exposes pool initialization separately.

The relay socket observations concern client-to-relay connections. HTTPX and OpenCode have different keepalive policies, so those counts alone are not a cross-client pooling ranking. The shared relay also prevents quantifying the provider-side handshake savings from the SSE fix in this run.

Reproduce

From a development environment with LangBot dependencies installed, set LLM_BENCHMARK_API_KEY securely in the environment, then run:

python scripts/llm_latency_probe.py --opencode /path/to/opencode \
  --opencode-version 2.0.16 --base-url https://gateway.example/v1 \
  --model claude-haiku-4.5 --long-prefix --cache-control --rounds 6 \
  --output /path/outside/repository/latency.json

This makes 21 billable requests by default. --mock uses synthetic local SSE instead; it still needs OpenCode. --baseline-no-drain disables terminal draining for a transport regression comparison. The harness records only hashes and message sizes, not prompt bodies or credentials, and terminates its own isolated OpenCode process. Keep raw results outside the repository because they include the configured gateway URL. Other OpenCode versions may need API adjustments.

The next diagnostic for a reported slow production session should capture the actual pipeline/Runner timing, streaming mode, prompt size, reasoning settings, cache usage and gateway routing alongside these requester-level measurements.