xinyin025/LangBot
1
0
Fork 0
mirror of https://github.com/langbot-app/LangBot.git synced 2026-07-17 09:56:06 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor(provider): use LiteLLM as unified LLM requester backend (#2150)

Browse Source 
* refactor(provider): use LiteLLM as unified LLM requester backend

  - Replace 23+ individual requester implementations with unified litellmchat.py
  - Add litellm_provider field to 27 YAML manifests for provider routing
  - Delete redundant requester subclasses
  - Add unit tests for LiteLLMRequester (29 tests)
  - Fix num_retries parameter name (was max_retries)
  - Fix exception handling order for subclass exceptions

  LiteLLM provides unified API for 100+ providers, eliminating need for
  provider-specific requesters.

* fix: ruff format provider.py

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* refactor(provider): simplify LiteLLM requester usage handling

  - Remove unused Anthropic-specific tool schema generation
  - Share completion argument construction between normal and streaming calls
  - Use LiteLLM/OpenAI native usage fields for monitoring
  - Collect stream token usage from LiteLLM stream_options
  - Update LiteLLM requester tests for unified usage fields

* restore: restore deleted provider requester files

Restore individual provider requester implementations that were
removed in de61b5d3. These files coexist with the unified
litellmchat.py backend.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat: update requesters and improve provider selection UI

- Added `litellm_provider` field to various requesters' YAML configurations.
- Removed obsolete Python requester files for OpenRouter, PPIO, QHAIGC, ShengSuanYun, SiliconFlow, Space, TokenPony, VolcArk, and Xai.
- Introduced new requesters for Tencent and Together AI with corresponding YAML configurations and SVG icons.
- Enhanced the ProviderForm component to include a searchable dropdown for selecting providers, improving user experience.
- Updated localization files to include search provider text for both English and Chinese.

* fix(provider): align litellm rebase with master

* fix(provider): capture streaming token usage; add token observability

The LiteLLM streaming requester only captured usage when a chunk had an
empty `choices` list. Many OpenAI-compatible gateways (e.g. new-api) and
providers send the final usage payload in a chunk that still carries an
empty-delta choice, so streamed calls always recorded 0 tokens in the
monitoring logs/dashboard (non-streaming worked).

- Capture stream usage whenever a chunk carries it, regardless of choices
- Add robust _normalize_usage (dict/obj shapes, derive missing total_tokens)
- Register litellm in bootutils/deps.py (was in pyproject only)
- Add MonitoringService.get_token_statistics + /monitoring/token-statistics
  endpoint: summary, per-model breakdown, token timeseries, and a
  zero-token-success data-quality signal
- Add TokenMonitoring dashboard tab (summary tiles, stacked token chart,
  per-model table) + i18n (en/zh)
- Regression tests for stream usage capture and usage normalization

Verified end-to-end against a real OpenAI-compatible endpoint with
gpt-5.5 and claude-opus-4-8: tokens now recorded non-zero for both
streaming and non-streaming paths.

* refactor(provider): simplify litellm capabilities

* style: simplify wrapped expressions

* feat(models): persist context metadata

* fix(provider): handle dict embeddings and openai-compatible rerank in LiteLLMRequester

- invoke_embedding: support both object- and dict-shaped response.data
  entries (OpenAI-compatible gateways like new-api return dicts)
- invoke_rerank: litellm.arerank rejects the 'openai' provider, so for
  openai-compatible (or unspecified) providers call the standard
  Jina/Cohere-style POST /v1/rerank endpoint directly over HTTP
- accept both 'relevance_score' and 'score' fields in rerank results
- add unit tests for the openai-compatible HTTP rerank path

* feat(provider): enforce requester support_type when adding models

- frontend: AddModelPopover only shows model-type tabs (llm/embedding/
  rerank) that the provider's requester declares in its manifest
  support_type; ModelsDialog fetches requester manifests and maps
  requester -> support_type, passed down through ProviderCard
- backend: add _validate_provider_supports guard in create_llm_model /
  create_embedding_model / create_rerank_model so a model cannot be
  attached to a provider whose requester does not support that type,
  even if the frontend restriction is bypassed (manifests without
  support_type are allowed for backward compatibility)
- manifests: correct support_type for providers that do not offer all
  three model types:
  - llm only: anthropic, deepseek, groq, moonshot, openrouter, xai
  - llm + text-embedding: openai, gemini, mistral
  - add rerank to new-api (verified working via /v1/rerank)
  - set llm + text-embedding + rerank for aggregator/unknown gateways

* feat(provider): add searchable alias to requester manifests

- add a free-text 'alias' field to every requester manifest spec,
  containing the vendor's English/Chinese names, pinyin, common
  nicknames and flagship model-series names (e.g. moonshot -> kimi,
  月之暗面; zhipu -> glm, 智谱清言)
- frontend: ProviderForm requester search now also matches against
  alias (substring/contains), so searching 'kimi' surfaces Moonshot,
  '硅基' surfaces SiliconFlow, etc.
- also fix support_type: openrouter (relay) supports embedding+rerank;
  LangBot Space gains rerank (coming soon)

* fix(provider): make support_type guard defensive against incomplete model_mgr

- _validate_provider_supports now uses getattr to gracefully skip when
  model_mgr / provider_dict / manifest lookup is unavailable, instead of
  raising AttributeError (fixes unit tests that mock ap.model_mgr as a
  bare SimpleNamespace)
- add TestValidateProviderSupports covering: allow supported type,
  reject unsupported type, allow when support_type missing, allow when
  provider unknown, degrade safely when model_mgr is incomplete

* fix(persistence): guard 0004 migration against missing llm_models table

The 0004_add_llm_model_context_length migration called
inspector.get_columns('llm_models') unconditionally, raising
NoSuchTableError when the table does not exist (e.g. migrating a
fresh/empty DB, as exercised by the integration tests where
create_all() registers no tables because the ORM models are not
imported). Every other migration guards with a table-existence check
first; add the same guard here for both upgrade and downgrade.

Also restore the test head assertion to 0004 (it had been lowered to
0003 to mask this failure).

* Merge branch 'master' into feat/litellm

Resolve conflicts:
- uv.lock: regenerated via 'uv lock' to reconcile litellm/fastuuid
  (ours) with openai bump (master).
- Alembic migrations: master added 0004_add_mcp_readme while this
  branch added 0004_add_llm_model_context_length, both as children of
  0003 (would create multiple heads). Re-chain the litellm migration as
  0005_add_llm_model_context_length with down_revision=0004_add_mcp_readme
  for a single linear head. Update test head assertion accordingly.

* fix(persistence): shorten migration revision id to fit varchar(32)

PostgreSQL stores alembic_version.version_num as varchar(32).
'0005_add_llm_model_context_length' (33 chars) overflowed it, raising
StringDataRightTruncationError in the PG migration tests. Rename the
revision (and file) to '0005_add_llm_context_length' (27 chars) and
update the head assertions in both SQLite and PostgreSQL migration
tests.

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: fdc310 <2213070223@qq.com>
Co-authored-by: RockChinQ <rockchinq@gmail.com>
This commit is contained in:
huanghuoguoguo
2026-06-13 16:59:48 +08:00
committed by GitHub
parent 7965d333ac
commit 9ecb587ac0
123 changed files with 4098 additions and 4513 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/langbot/pkg/provider/runners/localagent.py
+75 -90
View File
@@ -42,6 +42,64 @@ SANDBOX_EXEC_SYSTEM_GUIDANCE = (
MAX_TOOL_CALL_ROUNDS = 128
def _model_has_ability(model: modelmgr_requester.RuntimeLLMModel, ability: str) -> bool:
return ability in (model.model_entity.abilities or [])
class _StreamAccumulator:
"""Accumulate streamed content and fragmented OpenAI-style tool calls."""
def __init__(self, msg_sequence: int = 0, initial_content: str | None = None):
self.tool_calls_map: dict[str, provider_message.ToolCall] = {}
self.msg_idx = 0
self.accumulated_content = initial_content or ''
self.last_role = 'assistant'
self.msg_sequence = msg_sequence
def add(self, msg: provider_message.MessageChunk) -> provider_message.MessageChunk | None:
self.msg_idx += 1
if msg.role:
self.last_role = msg.role
if msg.content:
self.accumulated_content += msg.content
if msg.tool_calls:
for tool_call in msg.tool_calls:
if tool_call.id not in self.tool_calls_map:
self.tool_calls_map[tool_call.id] = provider_message.ToolCall(
id=tool_call.id,
type=tool_call.type,
function=provider_message.FunctionCall(
name=tool_call.function.name if tool_call.function else '',
arguments='',
),
)
if tool_call.function and tool_call.function.arguments:
self.tool_calls_map[tool_call.id].function.arguments += tool_call.function.arguments
if self.msg_idx % 8 == 0 or msg.is_final:
self.msg_sequence += 1
return provider_message.MessageChunk(
role=self.last_role,
content=self.accumulated_content,
tool_calls=list(self.tool_calls_map.values()) if (self.tool_calls_map and msg.is_final) else None,
is_final=msg.is_final,
msg_sequence=self.msg_sequence,
)
return None
def final_message(self) -> provider_message.MessageChunk:
return provider_message.MessageChunk(
role=self.last_role,
content=self.accumulated_content,
tool_calls=list(self.tool_calls_map.values()) if self.tool_calls_map else None,
msg_sequence=self.msg_sequence,
)
@runner.runner_class('local-agent')
class LocalAgentRunner(runner.RequestRunner):
"""Local agent request runner"""
@@ -106,7 +164,7 @@ class LocalAgentRunner(runner.RequestRunner):
query,
model,
messages,
funcs if model.model_entity.abilities.__contains__('func_call') else [],
funcs if _model_has_ability(model, 'func_call') else [],
extra_args=model.model_entity.extra_args,
remove_think=remove_think,
)
@@ -136,7 +194,7 @@ class LocalAgentRunner(runner.RequestRunner):
query,
model,
messages,
funcs if model.model_entity.abilities.__contains__('func_call') else [],
funcs if _model_has_ability(model, 'func_call') else [],
extra_args=model.model_entity.extra_args,
remove_think=remove_think,
)
@@ -322,11 +380,7 @@ class LocalAgentRunner(runner.RequestRunner):
final_msg = msg
else:
# Streaming: invoke with fallback
tool_calls_map: dict[str, provider_message.ToolCall] = {}
msg_idx = 0
accumulated_content = ''
last_role = 'assistant'
msg_sequence = 1
stream_accumulator = _StreamAccumulator(msg_sequence=1)
stream_src, use_llm_model = await self._invoke_stream_with_fallback(
query,
@@ -336,44 +390,12 @@ class LocalAgentRunner(runner.RequestRunner):
remove_think,
)
async for msg in stream_src:
msg_idx = msg_idx + 1
if msg.role:
last_role = msg.role
if msg.content:
accumulated_content += msg.content
if msg.tool_calls:
for tool_call in msg.tool_calls:
if tool_call.id not in tool_calls_map:
tool_calls_map[tool_call.id] = provider_message.ToolCall(
id=tool_call.id,
type=tool_call.type,
function=provider_message.FunctionCall(
name=tool_call.function.name if tool_call.function else '', arguments=''
),
)
if tool_call.function and tool_call.function.arguments:
tool_calls_map[tool_call.id].function.arguments += tool_call.function.arguments
if msg_idx % 8 == 0 or msg.is_final:
msg_sequence += 1
yield provider_message.MessageChunk(
role=last_role,
content=accumulated_content,
tool_calls=list(tool_calls_map.values()) if (tool_calls_map and msg.is_final) else None,
is_final=msg.is_final,
msg_sequence=msg_sequence,
)
chunk = stream_accumulator.add(msg)
if chunk:
yield chunk
initial_response_emitted = True
final_msg = provider_message.MessageChunk(
role=last_role,
content=accumulated_content,
tool_calls=list(tool_calls_map.values()) if tool_calls_map else None,
msg_sequence=msg_sequence,
)
final_msg = stream_accumulator.final_message()
pending_tool_calls = final_msg.tool_calls
first_content = final_msg.content
@@ -459,69 +481,32 @@ class LocalAgentRunner(runner.RequestRunner):
)
if is_stream:
tool_calls_map = {}
msg_idx = 0
accumulated_content = ''
last_role = 'assistant'
msg_sequence = first_end_sequence
stream_accumulator = _StreamAccumulator(
msg_sequence=first_end_sequence,
initial_content=first_content,
)
tool_stream_src = use_llm_model.provider.invoke_llm_stream(
query,
use_llm_model,
req_messages,
query.use_funcs if use_llm_model.model_entity.abilities.__contains__('func_call') else [],
query.use_funcs if _model_has_ability(use_llm_model, 'func_call') else [],
extra_args=use_llm_model.model_entity.extra_args,
remove_think=remove_think,
)
async for msg in tool_stream_src:
msg_idx += 1
chunk = stream_accumulator.add(msg)
if chunk:
yield chunk
if msg.role:
last_role = msg.role
# Prepend first-round content on first chunk of tool-call round
if msg_idx == 1:
accumulated_content = first_content if first_content is not None else accumulated_content
if msg.content:
accumulated_content += msg.content
if msg.tool_calls:
for tool_call in msg.tool_calls:
if tool_call.id not in tool_calls_map:
tool_calls_map[tool_call.id] = provider_message.ToolCall(
id=tool_call.id,
type=tool_call.type,
function=provider_message.FunctionCall(
name=tool_call.function.name if tool_call.function else '', arguments=''
),
)
if tool_call.function and tool_call.function.arguments:
tool_calls_map[tool_call.id].function.arguments += tool_call.function.arguments
if msg_idx % 8 == 0 or msg.is_final:
msg_sequence += 1
yield provider_message.MessageChunk(
role=last_role,
content=accumulated_content,
tool_calls=list(tool_calls_map.values()) if (tool_calls_map and msg.is_final) else None,
is_final=msg.is_final,
msg_sequence=msg_sequence,
)
final_msg = provider_message.MessageChunk(
role=last_role,
content=accumulated_content,
tool_calls=list(tool_calls_map.values()) if tool_calls_map else None,
msg_sequence=msg_sequence,
)
final_msg = stream_accumulator.final_message()
else:
# Non-streaming: use committed model directly (no fallback in tool loop)
msg = await use_llm_model.provider.invoke_llm(
query,
use_llm_model,
req_messages,
query.use_funcs if use_llm_model.model_entity.abilities.__contains__('func_call') else [],
query.use_funcs if _model_has_ability(use_llm_model, 'func_call') else [],
extra_args=use_llm_model.model_entity.extra_args,
remove_think=remove_think,
)