xinyin025/LangBot
1
0
Fork 0
mirror of https://github.com/langbot-app/LangBot.git synced 2026-10-22 08:25:21 +08:00
Code Issues Packages Projects Releases Wiki Activity

perf(llm): harden request policy and optimize connections, reasoning and prompt caching (#2661)

Browse Source 
* fix(llm): centralize request policy and preserve stream diagnostics

* test(llm): dispose SQLite engine after migration regression

* perf(llm): reuse connections and unify reasoning and prompt cache policy

* fix(llm): preserve explicit native prompt cache options

* perf(llm): fix measured TLS setup and streaming reuse overhead

* fix(llm): resolve plugin request-chain regressions found by E2E

---------

Co-authored-by: Hyu <[email protected]>
This commit is contained in:
RockChinQ
2026-10-15 19:48:35 +08:00
committed by GitHub
co-authored by Hyu
parent 502b38130f
commit 616981b0cf
30 changed files with 4556 additions and 405 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/LLM_LATENCY_BENCHMARK.md
+100
View File
@@ -0,0 +1,100 @@
# LLM request-layer benchmark (2026-10-15)
This controlled test did not reproduce a large LangBot slowdown. Warm request
preparation was sub-millisecond; most measured latency occurred upstream.
It did identify two local transport problems, now fixed in this PR.
## Method
One Windows host, Python 3.12.7, LiteLLM 1.88.1 and OpenCode 2.0.16 used the
same user-provided NewAPI gateway and requested `claude-haiku-4.5`. Each client
made one warmup and six measured streaming calls, alternating all six client
orders. The synthetic system prefix was about 6,000 tokens. Parameters were
`temperature=0`, `max_tokens=256`, `stream=true`, `include_usage=true`, with an
ephemeral cache marker on the system text. No reasoning budget was requested.
The measurement relay deliberately forwards identical serialized requests and
headers for all three clients. It replaces OpenCode's added agent/environment
instructions and normalizes SDK request differences. The canonical SHA-256 is
`7834a66a188e70b4eb1b6234b3a14be4c66b8c72dd8a2c06395fe2505c1c371b`.
OpenCode runs as an isolated resident server, using its
[versioned session API](https://github.com/anomalyco/opencode/blob/v2.0.16/packages/protocol/openapi.json).
Process startup and session creation are outside the warm dispatch measurement.
All clients share the relay's upstream connection pool. Therefore this experiment
isolates local request dispatch and upstream response timing; it does **not**
compare their independent direct-to-provider TLS/proxy behavior. It also excludes
the complete bot pipeline, Runner/plugin RPC and OpenCode UI rendering. It cannot
establish which application is faster in every production deployment.
## Results
Medians, milliseconds; six measured samples per client:
| Client | Dispatch to relay | First content | Complete upstream response |
| --- | ---: | ---: | ---: |
| Direct HTTPX | 3.952 | 1119.541 | 2199.413 |
| LangBot Core requester | 6.802 | 1089.092 | 2135.935 |
| OpenCode | 41.158 | 1104.020 | 2148.265 |
Dispatch includes client preparation and local HTTP dispatch. First content and
completion use the same relay boundary for all clients, measured from dispatch;
they are not UI latency. LangBot's separately instrumented preparation median
was **0.316 ms**, maximum **0.497 ms**. These small samples and upstream variation
do not justify interpreting the small TTFT differences as a speed ranking.
Every measured response reported 5,990 input tokens, 5,987 cache-read tokens and
109 output tokens. This demonstrates equal warm cache accounting under the
normalized upstream request, not native cache-marker forwarding by all clients.
No cache-miss latency comparison or independent provider-route verification was
performed. Missing cache accounting is represented as null, never assumed zero.
The gateway may perform its own routing, retries and cache translation.
Sanitized individual samples and summaries are in
[llm-latency-2026-10-15.json](llm-latency-2026-10-15.json).
## Bottlenecks fixed
1. The OpenAI SDK stops consuming SSE at `[DONE]` before HTTP framing reaches
EOF. The healthy connection was then discarded. In the baseline relay run,
LangBot reused zero of six measured client connections; after bounded terminal
draining it reused six of six. A real TCP regression test also reproduces this
with split terminal markers and late usage. OpenRouter's generic adapter
already consumed EOF. Early close and cancellation still close immediately.
2. On this host, constructing default HTTPX clients took 1,761 / 1,443 / 1,435 ms.
Loading one TLS context per requester pool and sharing it across that client's
proxy mounts reduced measured pool setup to 666 / 430 / 431 ms. Certificate
loading now runs off the event loop. These are local setup measurements, not
a claim of equivalent percentage improvement in LLM latency. Runtime lazy
initialization and process startup remain separate cold costs.
3. High-resolution monotonic timing replaces the low-resolution clock used on
Python 3.12 Windows. Preparation no longer misleadingly rounds to zero.
`transport_setup_ms` exposes pool initialization separately.
The relay socket observations concern client-to-relay connections. HTTPX and
OpenCode have different keepalive policies, so those counts alone are not a
cross-client pooling ranking. The shared relay also prevents quantifying the
provider-side handshake savings from the SSE fix in this run.
## Reproduce
From a development environment with LangBot dependencies installed, set
`LLM_BENCHMARK_API_KEY` securely in the environment, then run:
```sh
python scripts/llm_latency_probe.py --opencode /path/to/opencode \
--opencode-version 2.0.16 --base-url https://gateway.example/v1 \
--model claude-haiku-4.5 --long-prefix --cache-control --rounds 6 \
--output /path/outside/repository/latency.json
```
This makes 21 billable requests by default. `--mock` uses synthetic local SSE
instead; it still needs OpenCode. `--baseline-no-drain` disables terminal draining
for a transport regression comparison. The harness records only hashes and
message sizes, not prompt bodies or credentials, and terminates its own isolated
OpenCode process. Keep raw results outside the repository because they include
the configured gateway URL. Other OpenCode versions may need API adjustments.
The next diagnostic for a reported slow production session should capture the
actual pipeline/Runner timing, streaming mode, prompt size, reasoning settings,
cache usage and gateway routing alongside these requester-level measurements.
docs/LLM_REQUEST_E2E.md
+107
View File
@@ -0,0 +1,107 @@
# Request-chain acceptance (2026-10-15)
The acceptance suite starts an isolated Core process, a real SDK Plugin Runtime
process, and a packaged Runner plugin. It installs the plugin and creates the
provider, model and Agent through the same authenticated HTTP APIs used by the
application. Agent Debug requests then traverse:
HTTP -> Core Agent orchestration -> Plugin Runtime -> plugin SDK -> Runtime ->
Core model manager/request policy -> LiteLLM/provider SDK -> HTTP/SSE provider.
Assertions cover plugin return values, captured provider requests, TCP disconnects
and persisted SQLite monitoring records. No requester, action transport, model
manager or monitoring implementation is replaced with a mock.
The remote provider is a deterministic local HTTP server. It injects failures and
delays reproducibly; these are real process/network integration tests, **not** a
claim that every external provider has been tested. See the separate
[latency benchmark](LLM_LATENCY_BENCHMARK.md) for the live NewAPI experiment.
## Matrix
Measured results on Windows / Python 3.12.7 / LiteLLM 1.88.1:
- Core with SDK companion commit `c25ff9b`: **31 passed**, strict stream closure,
including all four historical API cases (106.30 seconds).
- Core with the existing SDK 0.7.11rc5 environment: **27 passed, 4 XFAIL** for
the reproduced SDK cancellation defect (107.61 seconds).
- Published stable SDK 0.7.11 also reproduced the cancellation defect in the
preceding 23-case run. The old and stable wrappers were verified identical.
- Focused Core regressions: **371 passed** on Python 3.13.7, ResourceWarnings fatal.
- SDK API and stream flow-control regressions: **507 passed**.
- Existing real Runner discovery/orchestration/event-processor E2E: **3 passed**.
| Area | Cases and evidence |
| --- | --- |
| Public APIs | Runner and invocation-bound plugin APIs: message, message with usage, chunk stream and event stream; unchanged SDK entities and nested usage |
| Historical plugin APIs | Verbatim SDK 0.5.8 proxy and provider-message entities, all four APIs, ordinary calls without `run_id` |
| Request configuration | Persisted defaults, call overrides, OpenAI cache key/retention, OpenRouter numeric reasoning budget and Anthropic cache marker |
| Headers | Explicit application attribution and scoped conversation ID reach the provider |
| Multimodal/tools | Image data URL and function schema survive plugin serialization and provider conversion |
| Retry/failure | 503 then success, `num_retries=0`, `max_retries=0`, 401 without retry, first-event timeout, total timeout, idle timeout and connection loss after content |
| Recovery | Each fault followed by a successful request through the same running processes |
| Cancellation | Early close of Runner and plugin APIs, both chunk and event streams; provider socket closes and the next invocation succeeds |
| Reasoning | Reasoning content shown/hidden with the existing `remove_think` option; unsupported numeric budget rejected before HTTP |
| Monitoring | Full cache/reasoning usage, transport timing and structured errors persisted |
Historical API files come from SDK tag `0.5.8`, commit
`41e320b68e899a7b5e74cb48c5246f6ccbf9d73f`.
They use their own message serializers and the current Runtime transport. This
checks old plugin call/response compatibility; it is not a test of running the
entire old Runtime against the current Core.
## Defects exposed by the full chain
- The plugin bridge flattened model and call policy layers, allowing persisted
`request_policy.retries` to override a call's legacy zero-retry option. LLM
policy layers are now resolved separately; unrelated argument merging remains.
- Ordinary plugin calls without a Runner/query returned content but lost usage.
Core now creates a trusted, invocation-local query for accounting, using the
existing plugin-action query builder. No public payload/signature changes.
- The shipped OpenRouter requester uses OpenAI metadata. Numeric budgets selected
the wrong mapping, and LiteLLM's OpenAI chat adapter stripped cache markers.
OpenRouter chat now uses its own protocol adapter. Embedding/rerank routing is
unchanged, and a regression test reads the actual shipped requester YAML.
- Published SDK 0.7.11 leaves inner streaming generators unowned in several public
proxy wrappers. Core's direct TCP close test passes, but the complete plugin
chain does not promptly cancel its upstream request. The companion
[SDK PR #177](https://github.com/langbot-app/langbot-plugin-sdk/pull/177) adds
explicit generator ownership without changing signatures, permissions, action
payloads or error translation.
**Immediate remote cancellation requires the SDK companion fix.** Core's pinned
SDK has not been silently replaced or upgraded. The E2E suite executes the close
checks against published SDKs and reports their known failure as XFAIL. Set
`LLM_E2E_REQUIRE_STREAM_CLOSE=1` to require strict passing with the patched SDK;
newer SDK versions must pass without this override. Historical API checks are
explicitly skipped unless their released source files are supplied.
## Reproduction
```sh
python -m pytest tests/e2e/test_llm_request_chain.py -q --tb=short
```
For the historical API matrix, export the two unmodified files below from SDK tag
`0.5.8` into a temporary directory, named `legacy_api.py` and `legacy_message.py`,
then set `LLM_E2E_LEGACY_API_DIR` to its absolute path:
- `src/langbot_plugin/api/proxies/langbot_api.py`
- `src/langbot_plugin/api/entities/builtin/provider/message.py`
For strict cancellation acceptance, install the companion SDK into an isolated
test environment and set `LLM_E2E_REQUIRE_STREAM_CLOSE=1` before running the suite.
The existing CI E2E job discovers this test module automatically. Tests use random
localhost ports, test-only credentials and pytest temporary data directories;
they leave application data and running user instances untouched.
## Limits
The suite does not exercise IM delivery, browser rendering, external provider
authentication/billing, native Anthropic/Gemini network services, Codex OAuth, or
complete third-party Runner business logic. It validates multimodal/tool request
transport rather than image understanding or actual tool execution. Native Codex,
retry, reasoning, transport and plugin regressions also run in the focused Core
test suite. A Python 3.13 Windows full-app attempt timed out during startup before
any LLM request; full process acceptance therefore uses Python 3.12.7, while the
focused Core regressions run on Python 3.13.7.
docs/LLM_REQUEST_POLICY.md
+254
View File
@@ -0,0 +1,254 @@
# LLM request policy and diagnostics
The Core owns chat-call deadlines and retry decisions. The existing SDK
`Message`, `MessageChunk`, invoke-with-usage and stream-usage shapes are unchanged.
No SDK or Runner upgrade is required. This policy does not add automatic tool
replay, model fallback, key rotation, or embedding/rerank retries.
The public API remains compatible with existing plugins. Full-chain testing
identified a pre-existing SDK <=0.7.11 early-close defect: prompt remote
cancellation through public plugin/Runner wrappers requires the companion
[SDK fix](https://github.com/langbot-app/langbot-plugin-sdk/pull/177).
See [E2E acceptance results](LLM_REQUEST_E2E.md) for tested versions and scope.
## Configuration
Instance defaults live in `data/config.yaml` under `llm_request_policy`; restart
Core after editing them. Persist model-specific overrides in the existing model
advanced parameters (`extra_args`). Runners can override them through their
existing call-level `extra_args` argument:
```json
{
"request_policy": {
"retries": 1,
"total_timeout": 180,
"first_event_timeout": 60,
"stream_idle_timeout": 45,
"connect_timeout": 10,
"read_timeout": 120,
"write_timeout": 30,
"pool_timeout": 10,
"backoff_base": 0.5,
"backoff_max": 8,
"attribution": true,
"conversation_header": null
}
}
```
Precedence is instance/requester defaults, model extra arguments, then call extra
arguments. Within each layer, `request_policy.retries` overrides legacy
`num_retries` / `max_retries`. Zero disables retries. Legacy numeric `timeout`
sets read, first-event and idle timeouts; it does not extend the total deadline.
Timeouts are positive finite seconds; retries must be an integer from 0 to 5.
Unknown policy keys are rejected. Policy keys are removed before calling upstream.
Default retries are **zero**, total timeout is 300 seconds, and first-event and
idle timeouts are 120 seconds. Retry backoff uses exponential full jitter;
`Retry-After` seconds or HTTP dates are respected only within the remaining total
budget. Quota, authentication and invalid-request failures are not retried.
Core disables both LiteLLM and provider-SDK retries for managed calls. A gateway
may still retry internally; configure its budget separately.
Retries stop after any raw upstream event (including reasoning/control/usage)
or, for legacy adapters without raw instrumentation, the first yielded chunk.
Cancellation is propagated and the owned stream is closed. Total deadlines also
include downstream consumption time, but Core does not cancel a consumer while
the generator is suspended at a yield; expiration is enforced on the next pull.
Transport connect/read/write/pool limits apply to LiteLLM and native Codex HTTP.
Legacy third-party requesters receive Host-level deadlines but remain responsible
for their internal transport configuration and internal retries.
## Diagnostics and usage
HTTP monitoring and `get_monitoring_records(kind="llm_calls")` return additive
`details` containing `timing`, complete provider `usage`, and structured `error`.
Older records have null details. Existing top-level token counters remain intact.
Migration `0038_llm_call_details` adds a nullable JSON column and preserves rows.
Timing offsets use a high-resolution monotonic clock, in milliseconds from invocation start:
| Field | Meaning |
| --- | --- |
| `request_prepared_ms` | First attempt finished building its request |
| `request_started_ms` | First adapter call into the network SDK; not a socket-level timestamp |
| `response_open_ms` | SDK returned a response/stream object |
| `first_event_ms` | First parsed upstream stream event, including reasoning/control |
| `first_reasoning_ms` | First upstream reasoning delta when supported |
| `first_content_ms` | First upstream content delta; inline think tags may still be content |
| `first_visible_chunk_ms` | First nonempty normalized chunk; may include displayed reasoning |
| `response_completed_ms` | Host finished consuming the response |
| `total_ms` | Invocation duration, excluding the subsequent monitoring write |
| `attempts`, `retry_wait_ms` | Host attempts and total backoff wait |
| `preparation_ms` | Cumulative LiteLLM request-building time across attempts |
| `transport_setup_ms` | Cumulative managed HTTP pool acquisition, including first-use TLS setup |
| `preparation_cache_hits` | Message preparation reused within a retrying invocation |
Missing fields mean not observed or not instrumented, not zero latency. The Core
does not label these measurements as network-only TTFT. Usage preserves nested
cache and reasoning counts, costs and provider-specific counters when supplied;
missing upstream accounting cannot be reconstructed. Partial usage received
before cancellation/failure is retained. No fabricated usage event is sent early.
`RequesterError` retains its legacy string representation and adds `code`,
`status_code`, `retryable`, `retry_after`, `request_id`, `phase`, and `as_dict()`.
Native Codex errors retain their existing ValueError/HTTP error contract.
Legacy SDK error transport remains unchanged; structured diagnostics are available
in Core monitoring. The metadata never copies upstream error bodies or headers
wholesale. Requester-specific raw error strings are not included by the LiteLLM
error translator.
## Application and conversation headers
Direct OpenRouter requests automatically receive:
```text
HTTP-Referer: https://langbot.app
X-OpenRouter-Title: LangBot
x-session-id: lb-<opaque scoped hash>
```
Existing headers in model/call `extra_headers` win, case-insensitively. Legacy
`X-Title` is respected. Set `attribution: false` to disable automatic application
attribution (explicit headers remain untouched). Detection uses the exact
`openrouter.ai` hostname, or an OpenRouter requester without an overridden base
URL; arbitrary gateway hosts do not inherit OpenRouter identity headers.
The conversation hash includes Workspace, bot, pipeline and session identity;
it is stable across turns and differs across those scopes. It is pseudonymous,
not a claim of cryptographic anonymization. Calls without conversation context
omit generated conversation headers. No prompt, raw user ID or group ID is sent.
`x-langbot-conversation-id` is a custom convention, **not a documented universal
provider standard**. OpenRouter documents `x-session-id` (or body `session_id`,
which takes precedence). LiteLLM Proxy documents `x-litellm-session-id` and
vendor session headers. Configure `conversation_header` explicitly for gateways:
```json
{
"request_policy": {"conversation_header": "x-litellm-session-id"},
"extra_headers": {
"HTTP-Referer": "https://langbot.app",
"X-OpenRouter-Title": "LangBot"
}
}
```
Supported generated header names are `x-session-id`, `x-litellm-session-id` and
`x-langbot-conversation-id`. The latter is intended for a gateway configured to
consume that custom header; do not expect automatic routing or caching benefits.
New API may supply its own OpenRouter attribution and may filter/override client
headers. Configure explicit channel forwarding/override rules and verify the
outgoing request. Sending a header to New API is not proof OpenRouter received it.
The local `langbot-new-api` snapshot inspected during implementation overwrites
attribution with New API; upstream main uses defaults when its outgoing headers
are empty. This change does not modify or deploy either gateway.
Sources checked on 2026-10-15:
- https://openrouter.ai/docs/app-attribution
- https://openrouter.ai/docs/guides/best-practices/prompt-caching
- https://docs.litellm.ai/docs/proxy/request_headers
- https://docs.litellm.ai/docs/routing
- https://github.com/QuantumNous/new-api/blob/main/relay/channel/openai/adaptor.go
- https://github.com/QuantumNous/new-api/blob/main/relay/channel/api_request.go
## Connection and preparation reuse
Initialized OpenAI-compatible, OpenRouter and native Codex requesters own a lazy
HTTP pool per runtime Provider. Keys, headers and timeout budgets stay request-local.
The pool holds at most 100 connections, with 20 idle keepalive connections and a
30-second idle expiry. Provider retirement rejects new leases and closes its pool
after existing streams finish or are cancelled. Native LiteLLM providers retain
their own transport handling; custom LiteLLM transport/TLS settings, explicit caller clients and redirected model routes
are preserved. Bare OpenAI names on OpenRouter retain LiteLLM's alternate routing;
use OpenRouter's fully qualified names such as `openai/gpt-4o`.
Each pool loads its TLS context once in a worker thread, preserving HTTPX
certificate and proxy environment settings. Concurrent initialization is serialized;
cancellation or retirement during certificate loading creates no HTTP client.
OpenAI-compatible streams drain HTTP framing after the SSE `[DONE]` record
(at most 50 ms / 64 KiB) so the SDK can return healthy connections to the pool.
Early close and cancellation never trigger this drain; late usage preceding
`[DONE]` remains available. OpenRouter already consumes EOF in its generic adapter.
Codex briefly drains the HTTP tail after a completion event (at most 50 ms) to
release a healthy connection for reuse. Incomplete/cancelled streams are closed.
Capability lookups use a bounded requester-local cache (128 entries, five minutes).
Message/tool conversion is reused only within an invocation that permits retries,
with independent copies for each attempt. No conversation content is cached across
invocations. Token counting runs in the application's executor instead of blocking
the event loop. These reduce local overhead; they do not guarantee faster upstream
inference or a cache hit.
## Unified thinking budgets
Existing effort levels and Runner/SDK signatures remain unchanged. Model
`reasoning_config` accepts an optional positive integer `budget_tokens`:
```json
{"reasoning_config": {"level": "enabled", "budget_tokens": 2048}}
```
Old SDKs can send the same object inside their existing call `extra_args`.
Precedence is persisted model configuration, model `extra_args.reasoning_config`,
call `extra_args.reasoning_config`, then the existing Runner override. Each object
replaces the previous one; a Runner effort override clears a lower-priority budget.
Omitting `level` with a budget means `enabled`. Budgets cannot be combined with an
effort level, `disabled`, or native thinking arguments. Unsupported budgets fail
before the network call; they are never silently converted into an effort level.
Numeric budgets map to Anthropic `thinking.budget_tokens`, Gemini 2.5 thinking
configuration, supported Qwen `thinking_budget`, or OpenRouter
`reasoning.max_tokens`. Direct OpenAI/Codex and adaptive-only Claude models keep
effort-based control. Query `reasoning_capabilities.budget_tokens` for the Core's
accepted bounds; upstream models/gateways may impose additional limits. OpenRouter
may translate numeric budgets to effort for models without native numeric support.
Direct Anthropic calls require an explicit `max_tokens` greater than the budget;
Core never silently raises the output limit. Existing UI effort selectors are
unchanged; numeric budgets can be configured through the model API or extra args.
## Prompt caching
Default behavior remains provider-controlled. `extra_args.prompt_caching` is a
Core-only object and is stripped before sending the request. Call-level objects
replace model-level objects. There is no local generated-response cache.
```json
{"prompt_caching": {"mode": "openai", "retention": "in_memory"}}
```
OpenAI mode adds an opaque conversation-and-model-scoped `prompt_cache_key` when
conversation context exists. `retention` is optional (`in_memory` or `24h`) and
must be supported by the destination model/gateway. Native explicit
`prompt_cache_key`, `prompt_cache_retention`, and newer `prompt_cache_options`
take precedence. Routing keys are hints, not cache-hit guarantees; current models
may route caching automatically or use newer native options instead. This mode
supports OpenAI-compatible/OpenRouter OpenAI models; forwarding through gateways
must be verified separately.
```json
{"prompt_caching": {"mode": "anthropic", "ttl": "5m"}}
```
Anthropic mode supports native Claude and OpenRouter Claude. It adds one ephemeral
breakpoint to the last text block in the leading system/developer prefix, without
reordering tools/history or marking user messages. TTL is `5m` (default) or `1h`.
If any native cache-control marker exists, Core preserves it without adding more.
With no eligible prefix, no marker is inserted. Provider minimum prefix lengths,
TTL pricing and eligibility still apply. Explicit `provider_default` adds no fields.
Codex's restricted Responses body does not accept this caching object; its existing
server-managed caching behavior is preserved. Gemini explicit cache resource
creation/lifecycle and general Responses migration are outside this batch.
References checked on 2026-10-15:
- https://developers.openai.com/api/docs/guides/prompt-caching
- https://platform.claude.com/docs/en/build-with-claude/prompt-caching
- https://platform.claude.com/docs/en/build-with-claude/extended-thinking
- https://ai.google.dev/gemini-api/docs/generate-content/thinking
- https://openrouter.ai/docs/guides/best-practices/reasoning-tokens
See [the latency benchmark](LLM_LATENCY_BENCHMARK.md) for measured results and limitations.
docs/llm-latency-2026-10-15.json
+411
View File
@@ -0,0 +1,411 @@
{
"model": "claude-haiku-4.5",
"litellm": "1.88.1",
"opencode": "2.0.16",
"canonical_sha256": "7834a66a188e70b4eb1b6234b3a14be4c66b8c72dd8a2c06395fe2505c1c371b",
"canonical_bytes": 31785,
"summary": {
"direct": {
"samples": 6,
"dispatch_to_wire_ms": {
"median": 3.952,
"max": 5.44
},
"upstream_ttft_ms": {
"median": 1115.744,
"max": 1158.609
},
"ttft_ms": {
"median": 1119.541,
"max": 1164.048
},
"total_ms": {
"median": 2199.413,
"max": 2219.7
},
"cache_read_tokens": [
5987,
5987,
5987,
5987,
5987,
5987
]
},
"langbot": {
"samples": 6,
"dispatch_to_wire_ms": {
"median": 6.802,
"max": 10.028
},
"upstream_ttft_ms": {
"median": 1079.658,
"max": 1113.394
},
"ttft_ms": {
"median": 1089.092,
"max": 1120.475
},
"total_ms": {
"median": 2135.935,
"max": 2175.882
},
"preparation_ms": {
"median": 0.316,
"max": 0.497
},
"cache_read_tokens": [
5987,
5987,
5987,
5987,
5987,
5987
]
},
"opencode": {
"samples": 6,
"dispatch_to_wire_ms": {
"median": 41.158,
"max": 50.283
},
"upstream_ttft_ms": {
"median": 1063.875,
"max": 1158.768
},
"ttft_ms": {
"median": 1104.02,
"max": 1201.731
},
"total_ms": {
"median": 2148.265,
"max": 2244.956
},
"cache_read_tokens": [
5987,
5987,
5987,
5987,
5987,
5987
]
}
},
"records": [
{
"client": "direct",
"round": -1,
"warmup": true,
"dispatch_to_wire_ms": 3.500899998471141,
"upstream_ttft_ms": 2094.1667999722995,
"ttft_ms": 2097.6676999707706,
"total_ms": 3145.2215999597684,
"client_socket_reused": false,
"consumer_first_content_ms": 2098.3635999727994,
"consumer_total_ms": 3146.0422999807633,
"input_tokens": 5990,
"output_tokens": 109,
"cache_read_tokens": 5987
},
{
"client": "langbot",
"round": -1,
"warmup": true,
"dispatch_to_wire_ms": 1452.2201999789104,
"upstream_ttft_ms": 1122.500299999956,
"ttft_ms": 2574.7204999788664,
"total_ms": 3640.2128000045195,
"preparation_ms": 0.373,
"transport_setup_ms": 441.243,
"client_socket_reused": false,
"consumer_first_content_ms": 2576.894700003322,
"consumer_total_ms": 3646.833000006154,
"input_tokens": 5990,
"output_tokens": 109,
"cache_read_tokens": 5987
},
{
"client": "opencode",
"round": -1,
"warmup": true,
"dispatch_to_wire_ms": 846.3009999832138,
"upstream_ttft_ms": 1082.7391999773681,
"ttft_ms": 1929.040199960582,
"total_ms": 3569.3477999884635,
"client_socket_reused": false,
"input_tokens": 5990,
"output_tokens": 109,
"cache_read_tokens": 5987
},
{
"client": "direct",
"round": 0,
"warmup": false,
"dispatch_to_wire_ms": 3.5812999703921378,
"upstream_ttft_ms": 1012.8218000172637,
"ttft_ms": 1016.4030999876559,
"total_ms": 2067.0370999723673,
"client_socket_reused": false,
"consumer_first_content_ms": 1017.0787000097334,
"consumer_total_ms": 2067.508200008888,
"input_tokens": 5990,
"output_tokens": 109,
"cache_read_tokens": 5987
},
{
"client": "langbot",
"round": 0,
"warmup": false,
"dispatch_to_wire_ms": 6.439700024202466,
"upstream_ttft_ms": 1067.0202999608591,
"ttft_ms": 1073.4599999850616,
"total_ms": 2151.4558999915607,
"preparation_ms": 0.281,
"transport_setup_ms": 0.017,
"client_socket_reused": true,
"consumer_first_content_ms": 1074.6371999848634,
"consumer_total_ms": 2156.441500002984,
"input_tokens": 5990,
"output_tokens": 109,
"cache_read_tokens": 5987
},
{
"client": "opencode",
"round": 0,
"warmup": false,
"dispatch_to_wire_ms": 50.28290004702285,
"upstream_ttft_ms": 1151.4483999926597,
"ttft_ms": 1201.7313000396825,
"total_ms": 2244.956100010313,
"client_socket_reused": false,
"input_tokens": 5990,
"output_tokens": 109,
"cache_read_tokens": 5987
},
{
"client": "direct",
"round": 1,
"warmup": false,
"dispatch_to_wire_ms": 3.269899985753,
"upstream_ttft_ms": 1149.7626000200398,
"ttft_ms": 1153.0325000057928,
"total_ms": 2198.293499997817,
"client_socket_reused": true,
"consumer_first_content_ms": 1153.3279999857768,
"consumer_total_ms": 2198.7193999812007,
"input_tokens": 5990,
"output_tokens": 109,
"cache_read_tokens": 5987
},
{
"client": "opencode",
"round": 1,
"warmup": false,
"dispatch_to_wire_ms": 40.65959999570623,
"upstream_ttft_ms": 1158.767800021451,
"ttft_ms": 1199.4274000171572,
"total_ms": 2242.273700016085,
"client_socket_reused": false,
"input_tokens": 5990,
"output_tokens": 109,
"cache_read_tokens": 5987
},
{
"client": "langbot",
"round": 1,
"warmup": false,
"dispatch_to_wire_ms": 6.523299962282181,
"upstream_ttft_ms": 1070.9955000202172,
"ttft_ms": 1077.5187999824993,
"total_ms": 2109.3437999952585,
"preparation_ms": 0.497,
"transport_setup_ms": 0.015,
"client_socket_reused": true,
"consumer_first_content_ms": 1081.5072999685071,
"consumer_total_ms": 2113.767499977257,
"input_tokens": 5990,
"output_tokens": 109,
"cache_read_tokens": 5987
},
{
"client": "langbot",
"round": 2,
"warmup": false,
"dispatch_to_wire_ms": 7.08060001488775,
"upstream_ttft_ms": 1113.394399988465,
"ttft_ms": 1120.4750000033528,
"total_ms": 2175.882300012745,
"preparation_ms": 0.221,
"transport_setup_ms": 0.009,
"client_socket_reused": true,
"consumer_first_content_ms": 1122.9100999771617,
"consumer_total_ms": 2187.183100031689,
"input_tokens": 5990,
"output_tokens": 109,
"cache_read_tokens": 5987
},
{
"client": "direct",
"round": 2,
"warmup": false,
"dispatch_to_wire_ms": 5.439500033389777,
"upstream_ttft_ms": 1158.6087999748997,
"ttft_ms": 1164.0483000082895,
"total_ms": 2219.7004000190645,
"client_socket_reused": false,
"consumer_first_content_ms": 1164.5996000152081,
"consumer_total_ms": 2220.197000016924,
"input_tokens": 5990,
"output_tokens": 109,
"cache_read_tokens": 5987
},
{
"client": "opencode",
"round": 2,
"warmup": false,
"dispatch_to_wire_ms": 41.65710002416745,
"upstream_ttft_ms": 1063.7803999707103,
"ttft_ms": 1105.4374999948777,
"total_ms": 2145.6969000282697,
"client_socket_reused": false,
"input_tokens": 5990,
"output_tokens": 109,
"cache_read_tokens": 5987
},
{
"client": "langbot",
"round": 3,
"warmup": false,
"dispatch_to_wire_ms": 10.027599986642599,
"upstream_ttft_ms": 1080.2179000456817,
"ttft_ms": 1090.2455000323243,
"total_ms": 2105.4020000156015,
"preparation_ms": 0.406,
"transport_setup_ms": 0.012,
"client_socket_reused": true,
"consumer_first_content_ms": 1092.152299999725,
"consumer_total_ms": 2109.886699996423,
"input_tokens": 5990,
"output_tokens": 109,
"cache_read_tokens": 5987
},
{
"client": "opencode",
"round": 3,
"warmup": false,
"dispatch_to_wire_ms": 38.63289998844266,
"upstream_ttft_ms": 1063.9694000128657,
"ttft_ms": 1102.6023000013083,
"total_ms": 2139.4490000093356,
"client_socket_reused": false,
"input_tokens": 5990,
"output_tokens": 109,
"cache_read_tokens": 5987
},
{
"client": "direct",
"round": 3,
"warmup": false,
"dispatch_to_wire_ms": 4.3889000080525875,
"upstream_ttft_ms": 1157.878800004255,
"ttft_ms": 1162.2677000123076,
"total_ms": 2214.4845000002533,
"client_socket_reused": false,
"consumer_first_content_ms": 1162.9892000346445,
"consumer_total_ms": 2214.923900028225,
"input_tokens": 5990,
"output_tokens": 109,
"cache_read_tokens": 5987
},
{
"client": "opencode",
"round": 4,
"warmup": false,
"dispatch_to_wire_ms": 34.960800025146455,
"upstream_ttft_ms": 1052.52319999272,
"ttft_ms": 1087.4840000178665,
"total_ms": 2137.3740999842994,
"client_socket_reused": true,
"input_tokens": 5990,
"output_tokens": 109,
"cache_read_tokens": 5987
},
{
"client": "direct",
"round": 4,
"warmup": false,
"dispatch_to_wire_ms": 2.0186000037938356,
"upstream_ttft_ms": 987.3963000136428,
"ttft_ms": 989.4149000174366,
"total_ms": 2200.532699993346,
"client_socket_reused": true,
"consumer_first_content_ms": 989.9038000148721,
"consumer_total_ms": 2201.0204999824055,
"input_tokens": 5990,
"output_tokens": 109,
"cache_read_tokens": 5987
},
{
"client": "langbot",
"round": 4,
"warmup": false,
"dispatch_to_wire_ms": 6.363800028339028,
"upstream_ttft_ms": 1112.4532999820076,
"ttft_ms": 1118.8171000103466,
"total_ms": 2150.362300046254,
"preparation_ms": 0.325,
"transport_setup_ms": 0.011,
"client_socket_reused": true,
"consumer_first_content_ms": 1120.592100021895,
"consumer_total_ms": 2154.264000011608,
"input_tokens": 5990,
"output_tokens": 109,
"cache_read_tokens": 5987
},
{
"client": "opencode",
"round": 5,
"warmup": false,
"dispatch_to_wire_ms": 41.722599999047816,
"upstream_ttft_ms": 1059.4015000388026,
"ttft_ms": 1101.1241000378504,
"total_ms": 2150.8331000222825,
"client_socket_reused": false,
"input_tokens": 5990,
"output_tokens": 109,
"cache_read_tokens": 5987
},
{
"client": "langbot",
"round": 5,
"warmup": false,
"dispatch_to_wire_ms": 8.840300026349723,
"upstream_ttft_ms": 1079.098400019575,
"ttft_ms": 1087.9387000459246,
"total_ms": 2121.5078000095673,
"preparation_ms": 0.307,
"transport_setup_ms": 0.021,
"client_socket_reused": true,
"consumer_first_content_ms": 1091.6245000553317,
"consumer_total_ms": 2126.561200013384,
"input_tokens": 5990,
"output_tokens": 109,
"cache_read_tokens": 5987
},
{
"client": "direct",
"round": 5,
"warmup": false,
"dispatch_to_wire_ms": 4.323699977248907,
"upstream_ttft_ms": 1081.7263000062667,
"ttft_ms": 1086.0499999835156,
"total_ms": 2126.9287999602966,
"client_socket_reused": false,
"consumer_first_content_ms": 1086.7821999709122,
"consumer_total_ms": 2127.6911000022665,
"input_tokens": 5990,
"output_tokens": 109,
"cache_read_tokens": 5987
}
]
}