fix(provider): capture streaming token usage; add token observability

The LiteLLM streaming requester only captured usage when a chunk had an
empty `choices` list. Many OpenAI-compatible gateways (e.g. new-api) and
providers send the final usage payload in a chunk that still carries an
empty-delta choice, so streamed calls always recorded 0 tokens in the
monitoring logs/dashboard (non-streaming worked).

- Capture stream usage whenever a chunk carries it, regardless of choices
- Add robust _normalize_usage (dict/obj shapes, derive missing total_tokens)
- Register litellm in bootutils/deps.py (was in pyproject only)
- Add MonitoringService.get_token_statistics + /monitoring/token-statistics
  endpoint: summary, per-model breakdown, token timeseries, and a
  zero-token-success data-quality signal
- Add TokenMonitoring dashboard tab (summary tiles, stacked token chart,
  per-model table) + i18n (en/zh)
- Regression tests for stream usage capture and usage normalization

Verified end-to-end against a real OpenAI-compatible endpoint with
gpt-5.5 and claude-opus-4-8: tokens now recorded non-zero for both
streaming and non-streaming paths.
This commit is contained in:
RockChinQ
2026-06-05 09:13:57 -04:00
parent d450226701
commit 39673444d2
10 changed files with 986 additions and 15 deletions
+25
View File
@@ -1196,6 +1196,7 @@ const enUS = {
llmCalls: 'LLM Calls',
embeddingCalls: 'Embedding Calls',
modelCalls: 'Model Calls',
tokens: 'Token Monitoring',
feedback: 'User Feedback',
sessions: 'Session Analysis',
errors: 'Error Logs',
@@ -1234,6 +1235,30 @@ const enUS = {
avgDuration: 'Avg Duration',
calls: 'Calls',
},
tokens: {
totalTokens: 'Total Tokens',
inputTokens: 'Input Tokens',
outputTokens: 'Output Tokens',
avgPerCall: 'Avg / Call',
throughput: 'Throughput',
tokensPerSec: 'tokens/sec',
errorCalls: 'Failed Calls',
acrossCalls: 'across {{count}} calls',
ofTotal: 'of {{count}} total',
usageOverTime: 'Token Usage Over Time',
byModel: 'By Model',
model: 'Model',
calls: 'Calls',
avgLatency: 'Avg Latency',
noData: 'No token usage in the selected time range',
loadError: 'Failed to load token statistics: {{error}}',
zeroTokenWarning:
'{{count}} successful call(s) reported zero token usage. This usually means the upstream provider did not return usage info — check the model provider configuration.',
bucket: {
hour: 'Hourly',
day: 'Daily',
},
},
embeddingCalls: {
title: 'Embedding Calls',
model: 'Model',