mirror of
https://github.com/langbot-app/LangBot.git
synced 2026-08-14 22:40:59 +00:00
fix(provider): capture streaming token usage; add token observability
The LiteLLM streaming requester only captured usage when a chunk had an empty `choices` list. Many OpenAI-compatible gateways (e.g. new-api) and providers send the final usage payload in a chunk that still carries an empty-delta choice, so streamed calls always recorded 0 tokens in the monitoring logs/dashboard (non-streaming worked). - Capture stream usage whenever a chunk carries it, regardless of choices - Add robust _normalize_usage (dict/obj shapes, derive missing total_tokens) - Register litellm in bootutils/deps.py (was in pyproject only) - Add MonitoringService.get_token_statistics + /monitoring/token-statistics endpoint: summary, per-model breakdown, token timeseries, and a zero-token-success data-quality signal - Add TokenMonitoring dashboard tab (summary tiles, stacked token chart, per-model table) + i18n (en/zh) - Regression tests for stream usage capture and usage normalization Verified end-to-end against a real OpenAI-compatible endpoint with gpt-5.5 and claude-opus-4-8: tokens now recorded non-zero for both streaming and non-streaming paths.
This commit is contained in:
@@ -1196,6 +1196,7 @@ const enUS = {
|
||||
llmCalls: 'LLM Calls',
|
||||
embeddingCalls: 'Embedding Calls',
|
||||
modelCalls: 'Model Calls',
|
||||
tokens: 'Token Monitoring',
|
||||
feedback: 'User Feedback',
|
||||
sessions: 'Session Analysis',
|
||||
errors: 'Error Logs',
|
||||
@@ -1234,6 +1235,30 @@ const enUS = {
|
||||
avgDuration: 'Avg Duration',
|
||||
calls: 'Calls',
|
||||
},
|
||||
tokens: {
|
||||
totalTokens: 'Total Tokens',
|
||||
inputTokens: 'Input Tokens',
|
||||
outputTokens: 'Output Tokens',
|
||||
avgPerCall: 'Avg / Call',
|
||||
throughput: 'Throughput',
|
||||
tokensPerSec: 'tokens/sec',
|
||||
errorCalls: 'Failed Calls',
|
||||
acrossCalls: 'across {{count}} calls',
|
||||
ofTotal: 'of {{count}} total',
|
||||
usageOverTime: 'Token Usage Over Time',
|
||||
byModel: 'By Model',
|
||||
model: 'Model',
|
||||
calls: 'Calls',
|
||||
avgLatency: 'Avg Latency',
|
||||
noData: 'No token usage in the selected time range',
|
||||
loadError: 'Failed to load token statistics: {{error}}',
|
||||
zeroTokenWarning:
|
||||
'{{count}} successful call(s) reported zero token usage. This usually means the upstream provider did not return usage info — check the model provider configuration.',
|
||||
bucket: {
|
||||
hour: 'Hourly',
|
||||
day: 'Daily',
|
||||
},
|
||||
},
|
||||
embeddingCalls: {
|
||||
title: 'Embedding Calls',
|
||||
model: 'Model',
|
||||
|
||||
Reference in New Issue
Block a user