mirror of
https://github.com/langbot-app/LangBot.git
synced 2026-06-13 17:26:04 +00:00
* refactor(provider): use LiteLLM as unified LLM requester backend
- Replace 23+ individual requester implementations with unified litellmchat.py
- Add litellm_provider field to 27 YAML manifests for provider routing
- Delete redundant requester subclasses
- Add unit tests for LiteLLMRequester (29 tests)
- Fix num_retries parameter name (was max_retries)
- Fix exception handling order for subclass exceptions
LiteLLM provides unified API for 100+ providers, eliminating need for
provider-specific requesters.
* fix: ruff format provider.py
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(provider): simplify LiteLLM requester usage handling
- Remove unused Anthropic-specific tool schema generation
- Share completion argument construction between normal and streaming calls
- Use LiteLLM/OpenAI native usage fields for monitoring
- Collect stream token usage from LiteLLM stream_options
- Update LiteLLM requester tests for unified usage fields
* restore: restore deleted provider requester files
Restore individual provider requester implementations that were
removed in de61b5d3. These files coexist with the unified
litellmchat.py backend.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat: update requesters and improve provider selection UI
- Added `litellm_provider` field to various requesters' YAML configurations.
- Removed obsolete Python requester files for OpenRouter, PPIO, QHAIGC, ShengSuanYun, SiliconFlow, Space, TokenPony, VolcArk, and Xai.
- Introduced new requesters for Tencent and Together AI with corresponding YAML configurations and SVG icons.
- Enhanced the ProviderForm component to include a searchable dropdown for selecting providers, improving user experience.
- Updated localization files to include search provider text for both English and Chinese.
* fix(provider): align litellm rebase with master
* fix(provider): capture streaming token usage; add token observability
The LiteLLM streaming requester only captured usage when a chunk had an
empty `choices` list. Many OpenAI-compatible gateways (e.g. new-api) and
providers send the final usage payload in a chunk that still carries an
empty-delta choice, so streamed calls always recorded 0 tokens in the
monitoring logs/dashboard (non-streaming worked).
- Capture stream usage whenever a chunk carries it, regardless of choices
- Add robust _normalize_usage (dict/obj shapes, derive missing total_tokens)
- Register litellm in bootutils/deps.py (was in pyproject only)
- Add MonitoringService.get_token_statistics + /monitoring/token-statistics
endpoint: summary, per-model breakdown, token timeseries, and a
zero-token-success data-quality signal
- Add TokenMonitoring dashboard tab (summary tiles, stacked token chart,
per-model table) + i18n (en/zh)
- Regression tests for stream usage capture and usage normalization
Verified end-to-end against a real OpenAI-compatible endpoint with
gpt-5.5 and claude-opus-4-8: tokens now recorded non-zero for both
streaming and non-streaming paths.
* refactor(provider): simplify litellm capabilities
* style: simplify wrapped expressions
* feat(models): persist context metadata
* fix(provider): handle dict embeddings and openai-compatible rerank in LiteLLMRequester
- invoke_embedding: support both object- and dict-shaped response.data
entries (OpenAI-compatible gateways like new-api return dicts)
- invoke_rerank: litellm.arerank rejects the 'openai' provider, so for
openai-compatible (or unspecified) providers call the standard
Jina/Cohere-style POST /v1/rerank endpoint directly over HTTP
- accept both 'relevance_score' and 'score' fields in rerank results
- add unit tests for the openai-compatible HTTP rerank path
* feat(provider): enforce requester support_type when adding models
- frontend: AddModelPopover only shows model-type tabs (llm/embedding/
rerank) that the provider's requester declares in its manifest
support_type; ModelsDialog fetches requester manifests and maps
requester -> support_type, passed down through ProviderCard
- backend: add _validate_provider_supports guard in create_llm_model /
create_embedding_model / create_rerank_model so a model cannot be
attached to a provider whose requester does not support that type,
even if the frontend restriction is bypassed (manifests without
support_type are allowed for backward compatibility)
- manifests: correct support_type for providers that do not offer all
three model types:
- llm only: anthropic, deepseek, groq, moonshot, openrouter, xai
- llm + text-embedding: openai, gemini, mistral
- add rerank to new-api (verified working via /v1/rerank)
- set llm + text-embedding + rerank for aggregator/unknown gateways
* feat(provider): add searchable alias to requester manifests
- add a free-text 'alias' field to every requester manifest spec,
containing the vendor's English/Chinese names, pinyin, common
nicknames and flagship model-series names (e.g. moonshot -> kimi,
月之暗面; zhipu -> glm, 智谱清言)
- frontend: ProviderForm requester search now also matches against
alias (substring/contains), so searching 'kimi' surfaces Moonshot,
'硅基' surfaces SiliconFlow, etc.
- also fix support_type: openrouter (relay) supports embedding+rerank;
LangBot Space gains rerank (coming soon)
* fix(provider): make support_type guard defensive against incomplete model_mgr
- _validate_provider_supports now uses getattr to gracefully skip when
model_mgr / provider_dict / manifest lookup is unavailable, instead of
raising AttributeError (fixes unit tests that mock ap.model_mgr as a
bare SimpleNamespace)
- add TestValidateProviderSupports covering: allow supported type,
reject unsupported type, allow when support_type missing, allow when
provider unknown, degrade safely when model_mgr is incomplete
* fix(persistence): guard 0004 migration against missing llm_models table
The 0004_add_llm_model_context_length migration called
inspector.get_columns('llm_models') unconditionally, raising
NoSuchTableError when the table does not exist (e.g. migrating a
fresh/empty DB, as exercised by the integration tests where
create_all() registers no tables because the ORM models are not
imported). Every other migration guards with a table-existence check
first; add the same guard here for both upgrade and downgrade.
Also restore the test head assertion to 0004 (it had been lowered to
0003 to mask this failure).
* Merge branch 'master' into feat/litellm
Resolve conflicts:
- uv.lock: regenerated via 'uv lock' to reconcile litellm/fastuuid
(ours) with openai bump (master).
- Alembic migrations: master added 0004_add_mcp_readme while this
branch added 0004_add_llm_model_context_length, both as children of
0003 (would create multiple heads). Re-chain the litellm migration as
0005_add_llm_model_context_length with down_revision=0004_add_mcp_readme
for a single linear head. Update test head assertion accordingly.
* fix(persistence): shorten migration revision id to fit varchar(32)
PostgreSQL stores alembic_version.version_num as varchar(32).
'0005_add_llm_model_context_length' (33 chars) overflowed it, raising
StringDataRightTruncationError in the PG migration tests. Rename the
revision (and file) to '0005_add_llm_context_length' (27 chars) and
update the head assertions in both SQLite and PostgreSQL migration
tests.
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: fdc310 <2213070223@qq.com>
Co-authored-by: RockChinQ <rockchinq@gmail.com>
217 lines
6.8 KiB
Python
217 lines
6.8 KiB
Python
"""
|
|
PostgreSQL migration integration tests.
|
|
|
|
Tests real Alembic migration behavior using PostgreSQL database.
|
|
Marked as slow - requires external PostgreSQL service.
|
|
|
|
Run locally (requires PostgreSQL):
|
|
TEST_POSTGRES_URL=postgresql+asyncpg://user:pass@localhost:5432/test_db \
|
|
uv run pytest tests/integration/persistence/test_migrations_postgres.py -q
|
|
|
|
CI runs automatically with PostgreSQL service container.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
import pytest
|
|
from sqlalchemy.ext.asyncio import create_async_engine
|
|
from sqlalchemy import text
|
|
|
|
from langbot.pkg.entity.persistence.base import Base
|
|
from langbot.pkg.persistence.alembic_runner import (
|
|
run_alembic_upgrade,
|
|
run_alembic_stamp,
|
|
get_alembic_current,
|
|
)
|
|
|
|
|
|
pytestmark = [pytest.mark.integration, pytest.mark.slow]
|
|
|
|
|
|
@pytest.fixture
|
|
def postgres_url():
|
|
"""Get PostgreSQL URL from environment."""
|
|
url = os.environ.get('TEST_POSTGRES_URL')
|
|
if not url:
|
|
pytest.skip("TEST_POSTGRES_URL not set")
|
|
return url
|
|
|
|
|
|
@pytest.fixture
|
|
async def postgres_engine(postgres_url):
|
|
"""Create async PostgreSQL engine."""
|
|
engine = create_async_engine(postgres_url, isolation_level="AUTOCOMMIT")
|
|
yield engine
|
|
await engine.dispose()
|
|
|
|
|
|
@pytest.fixture
|
|
async def clean_tables(postgres_engine):
|
|
"""Drop all tables before and after each test for isolation."""
|
|
# Drop all tables before test
|
|
async with postgres_engine.begin() as conn:
|
|
await conn.run_sync(Base.metadata.drop_all)
|
|
|
|
yield
|
|
|
|
# Drop all tables after test
|
|
async with postgres_engine.begin() as conn:
|
|
await conn.run_sync(Base.metadata.drop_all)
|
|
|
|
|
|
@pytest.fixture
|
|
async def clean_alembic_version(postgres_engine):
|
|
"""Drop alembic_version table before and after each test."""
|
|
async with postgres_engine.begin() as conn:
|
|
# Drop alembic_version table if exists
|
|
try:
|
|
await conn.execute(text("DROP TABLE IF EXISTS alembic_version"))
|
|
except Exception:
|
|
pass
|
|
|
|
yield
|
|
|
|
async with postgres_engine.begin() as conn:
|
|
try:
|
|
await conn.execute(text("DROP TABLE IF EXISTS alembic_version"))
|
|
except Exception:
|
|
pass
|
|
|
|
|
|
class TestPostgreSQLMigrationBaseline:
|
|
"""Tests for baseline stamp workflow on PostgreSQL."""
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_postgres_baseline_stamp_sets_revision(
|
|
self, postgres_engine, clean_tables, clean_alembic_version
|
|
):
|
|
"""
|
|
Stamp baseline on existing tables sets correct revision.
|
|
|
|
Workflow:
|
|
1. Create tables via Base.metadata.create_all
|
|
2. Stamp with '0001_baseline'
|
|
3. Verify current revision is '0001_baseline'
|
|
"""
|
|
# Create all tables (simulates existing DB created by ORM)
|
|
async with postgres_engine.begin() as conn:
|
|
await conn.run_sync(Base.metadata.create_all)
|
|
|
|
# Stamp baseline
|
|
await run_alembic_stamp(postgres_engine, '0001_baseline')
|
|
|
|
# Verify revision
|
|
rev = await get_alembic_current(postgres_engine)
|
|
assert rev == '0001_baseline', f"Expected '0001_baseline', got {rev}"
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_postgres_baseline_stamp_on_empty_db(
|
|
self, postgres_engine, clean_tables, clean_alembic_version
|
|
):
|
|
"""
|
|
Stamp on empty database (no tables) still sets revision.
|
|
|
|
This is an edge case - stamping without tables.
|
|
"""
|
|
# Don't create tables - stamp directly
|
|
await run_alembic_stamp(postgres_engine, '0001_baseline')
|
|
|
|
rev = await get_alembic_current(postgres_engine)
|
|
assert rev == '0001_baseline'
|
|
|
|
|
|
class TestPostgreSQLMigrationUpgrade:
|
|
"""Tests for upgrade to head workflow on PostgreSQL."""
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_postgres_upgrade_from_baseline_to_head(
|
|
self, postgres_engine, clean_tables, clean_alembic_version
|
|
):
|
|
"""
|
|
Upgrade from baseline to head applies all migrations.
|
|
|
|
Workflow:
|
|
1. Create tables
|
|
2. Stamp baseline
|
|
3. Upgrade to head
|
|
4. Verify current revision is head
|
|
"""
|
|
# Create tables
|
|
async with postgres_engine.begin() as conn:
|
|
await conn.run_sync(Base.metadata.create_all)
|
|
|
|
# Stamp baseline
|
|
await run_alembic_stamp(postgres_engine, '0001_baseline')
|
|
|
|
# Upgrade to head
|
|
await run_alembic_upgrade(postgres_engine, 'head')
|
|
|
|
# Verify revision
|
|
rev = await get_alembic_current(postgres_engine)
|
|
assert rev is not None, "Expected a revision after upgrade"
|
|
# Head should be the latest migration (0005 for current state)
|
|
assert rev.startswith('0005'), f"Expected head to be 0005_*, got {rev}"
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_postgres_upgrade_idempotent(
|
|
self, postgres_engine, clean_tables, clean_alembic_version
|
|
):
|
|
"""
|
|
Running upgrade to head multiple times is idempotent.
|
|
|
|
Workflow:
|
|
1. Upgrade to head
|
|
2. Get revision
|
|
3. Upgrade to head again
|
|
4. Verify same revision
|
|
"""
|
|
# Create tables
|
|
async with postgres_engine.begin() as conn:
|
|
await conn.run_sync(Base.metadata.create_all)
|
|
|
|
# Stamp and upgrade
|
|
await run_alembic_stamp(postgres_engine, '0001_baseline')
|
|
await run_alembic_upgrade(postgres_engine, 'head')
|
|
|
|
rev1 = await get_alembic_current(postgres_engine)
|
|
|
|
# Upgrade again - should be idempotent
|
|
await run_alembic_upgrade(postgres_engine, 'head')
|
|
|
|
rev2 = await get_alembic_current(postgres_engine)
|
|
assert rev2 == rev1, f"Expected {rev1}, got {rev2}"
|
|
|
|
|
|
class TestPostgreSQLMigrationGetCurrent:
|
|
"""Tests for get_alembic_current behavior on PostgreSQL."""
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_postgres_get_current_on_unstamped_db_returns_none(
|
|
self, postgres_engine, clean_tables, clean_alembic_version
|
|
):
|
|
"""
|
|
get_alembic_current returns None for unstamped database.
|
|
"""
|
|
# Create tables but don't stamp
|
|
async with postgres_engine.begin() as conn:
|
|
await conn.run_sync(Base.metadata.create_all)
|
|
|
|
# No stamp - should return None
|
|
rev = await get_alembic_current(postgres_engine)
|
|
assert rev is None, f"Expected None for unstamped DB, got {rev}"
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_postgres_get_current_after_stamp_returns_revision(
|
|
self, postgres_engine, clean_tables, clean_alembic_version
|
|
):
|
|
"""
|
|
get_alembic_current returns correct revision after stamp.
|
|
"""
|
|
async with postgres_engine.begin() as conn:
|
|
await conn.run_sync(Base.metadata.create_all)
|
|
|
|
await run_alembic_stamp(postgres_engine, '0001_baseline')
|
|
|
|
rev = await get_alembic_current(postgres_engine)
|
|
assert rev == '0001_baseline' |