Commit Graph

351 Commits

Author SHA1 Message Date
huanghuoguoguo
5831198f38 refactor(agent-runner): remove protocol_version from various components and update related documentation 2026-06-13 17:46:31 +08:00
huanghuoguoguo
7675f565ff test(agent): harden runner persistence coverage 2026-06-13 17:46:31 +08:00
huanghuoguoguo
54bba1a1f5 feat(agent-runner): expose skill resources through host context 2026-06-13 17:45:53 +08:00
huanghuoguoguo
a6a90f7d1b test: cover host skill tool scoping 2026-06-13 17:45:53 +08:00
huanghuoguoguo
4a8c1a76d7 refactor(agent-runner): use protocol version field 2026-06-13 17:45:53 +08:00
huanghuoguoguo
2de6d15d07 refactor(provider): formalize tool lookup contract 2026-06-13 17:45:53 +08:00
huanghuoguoguo
d347df02f6 fix(agent-runner): clean plugin review issues 2026-06-13 17:45:53 +08:00
huanghuoguoguo
f1a44ea8a8 refactor agent runner orchestration boundaries 2026-06-13 17:45:53 +08:00
huanghuoguoguo
3773e3dfaf fix(agent-runner): align plugin runner runtime boundaries 2026-06-13 17:45:53 +08:00
huanghuoguoguo
23d3b7c279 feat(agent-runner): add bounded native tool artifacts 2026-06-13 17:45:53 +08:00
huanghuoguoguo
058721cca3 feat(agent-runner): expose effective prompt and transcript history 2026-06-13 17:45:53 +08:00
huanghuoguoguo
e13a3b845c refactor(agent-runner): make agent binding and auth snapshot explicit 2026-06-13 17:45:53 +08:00
huanghuoguoguo
dc4cf5711e refactor(agent-runner): simplify event-first entry path 2026-06-13 17:45:53 +08:00
huanghuoguoguo
1384d328d6 refactor(agent-runner): align config with agent semantics 2026-06-13 17:45:14 +08:00
huanghuoguoguo
16faeca508 refactor(agent-runner): remove host context windowing 2026-06-13 17:45:14 +08:00
huanghuoguoguo
4852b21f9b feat(agent-runner): normalize binding config boundaries 2026-06-13 17:45:14 +08:00
huanghuoguoguo
0b9778abd9 fix: enforce agent run API permissions 2026-06-13 17:45:14 +08:00
huanghuoguoguo
c296c187f4 fix(agent-runner): authorize external runner tools 2026-06-13 17:45:14 +08:00
huanghuoguoguo
94c0adc8a1 docs(agent-runner): align runner protocol boundaries 2026-06-13 17:45:14 +08:00
huanghuoguoguo
fc2dc34ecf fix(agent-runner): stabilize event context and streams 2026-06-13 17:45:14 +08:00
huanghuoguoguo
819a2843e7 refactor(agent-runner): tighten protocol v1 runtime boundaries 2026-06-13 17:44:44 +08:00
huanghuoguoguo
96fa9e1eeb feat(agent-runner): align protocol adapter terminology 2026-06-13 17:44:44 +08:00
huanghuoguoguo
b4ae049c54 feat(agent-runner): route pipeline runs through event-first flow
- run_from_query() now delegates to run(event, binding) instead of maintaining
  a separate legacy execution path
- Pipeline Query is converted to AgentEventEnvelope via PipelineCompatAdapter
- Pipeline config is converted to AgentBinding with StatePolicy
- bound_plugins authorization preserved from Pipeline
- Legacy compatibility fields preserved:
  - query_id → context.runtime.query_id → session registry
  - prompt → context.compatibility.extra.prompt (not top-level)
  - params → context.compatibility.extra.params (with proper filtering)
  - max-round → bootstrap.messages and compatibility.legacy_messages
- Pipeline path gains event-first host capabilities:
  - EventLog and Transcript writing
  - ArtifactStore registration
  - PersistentStateStore for state.updated
- Removed legacy handlers:
  - _handle_artifact_created_query() (replaced by _handle_artifact_created)
  - _handle_state_updated() (replaced by _handle_state_updated_event)

This change unifies the execution path while preserving backward compatibility
for Pipeline-based runners. EventGateway is not implemented in this branch;
only the event-first entry point is reserved.
2026-06-13 17:44:44 +08:00
huanghuoguoguo
d1e49a5b44 feat(agent-runner): add persistent state APIs 2026-06-13 17:44:44 +08:00
huanghuoguoguo
2e0343cb21 feat(agent-runner): scope event-first state by binding 2026-06-13 17:44:44 +08:00
huanghuoguoguo
53c9199df8 feat(agent-runner): persist created artifacts 2026-06-13 17:44:44 +08:00
huanghuoguoguo
bec11e5a18 feat(agent-runner): add artifact store pull APIs 2026-06-13 17:44:44 +08:00
huanghuoguoguo
a31f910f10 feat(agent-runner): add event-first context facts and pull APIs
Add EventLog and Transcript persistence entities for storing auditable
event facts and conversation history projection. Implement event-first
AgentRunContext builder that produces Protocol v1 compliant context
payloads with required fields: event, delivery, context (ContextAccess).

Key changes:
- EventLog ORM: auditable event records with indexes
- Transcript ORM: conversation history projection with composite indexes
- AgentRunContextBuilder: Protocol v1 payload with delivery, context, bootstrap
- EventLogStore/TranscriptStore: async stores for fact sources
- Host action handlers: HISTORY_PAGE, HISTORY_SEARCH, EVENT_GET, EVENT_PAGE
- Context validation: build_context output validates via SDK AgentRunContext
- Alembic migration for event_log and transcript tables
- Alembic env.py imports all ORM models for autogenerate discovery

Legacy compatibility: max-round messages go into bootstrap.messages and
compatibility.legacy_messages, not top-level messages field.
2026-06-13 17:44:44 +08:00
huanghuoguoguo
c1dc5e3970 fix(agent-runner): package context for plugin execution 2026-06-13 17:44:44 +08:00
huanghuoguoguo
d8d98b0838 feat: make agent runner config schema driven 2026-06-13 17:44:44 +08:00
huanghuoguoguo
651e28113e chore(pipeline): clarify preferred default runner 2026-06-13 17:42:59 +08:00
huanghuoguoguo
c97ea27d42 chore(agent): remove v1 wording from runner internals 2026-06-13 17:42:59 +08:00
huanghuoguoguo
bbbbc05201 feat(agent): reserve stable runner event names 2026-06-13 17:42:59 +08:00
huanghuoguoguo
752ac6e9d2 feat(agent-runner): enrich plugin runner host context 2026-06-13 17:42:59 +08:00
huanghuoguoguo
9dfddd4927 fix: log agent runner best-effort failures 2026-06-13 17:42:59 +08:00
huanghuoguoguo
d185712716 fix: stabilize dynamic forms and mcp testing 2026-06-13 17:42:59 +08:00
huanghuoguoguo
c601dc5908 refactor(modelmgr): simplify model sync logic and remove timeout configuration 2026-06-13 17:42:21 +08:00
huanghuoguoguo
54e925daa0 fix(rag): align knowledge engine plugin actions 2026-06-13 17:42:21 +08:00
huanghuoguoguo
6d0e6dcc63 feat: support dynamic agent runner defaults 2026-06-13 17:42:21 +08:00
huanghuoguoguo
3baf899c20 feat(toolmgr): add get_tool_by_name for unified tool lookup
Add unified tool lookup method that searches both plugin and MCP loaders.
Also add _get_tool method to MCPLoader for consistency with PluginToolLoader.
2026-06-13 17:41:37 +08:00
huanghuoguoguo
2123ef5816 feat(plugin): implement INVOKE_RERANK handler with run-scoped authorization
- Add invoke_rerank action handler in plugin handler
- Validate rerank model access via run session
- Cap documents at 64 for API limit
- Return sorted results by relevance score
2026-06-13 17:41:37 +08:00
huanghuoguoguo
811549e1c4 docs(runner): mark legacy runners and add PROGRESS.md
- Add DEPRECATED docstring to all legacy runners in pkg/provider/runners/
- Mark migration target for each runner (local-agent, dify, n8n, coze, dashscope, langflow, tbox)
- Add PROGRESS.md to track agent-runner-pluginization implementation status
- Remove completed PHASE0_INTEGRATION_RECORD.md
2026-06-13 17:41:37 +08:00
huanghuoguoguo
6ef40fbd68 perf(agent-runner): improve session registry and orchestrator efficiency
- Add pre-computed _authorized_ids (frozenset) at session registration for O(1) lookup
- Refactor is_resource_allowed() from linear search to set membership check
- Add thread-safe locking to get_session_registry() singleton
- Cache _session_registry and _state_store references in orchestrator __init__
- Add asyncio.gather() for parallel resource building in AgentResourceBuilder
- Create shared test fixtures in tests/unit_tests/agent/conftest.py
- Update test files to import from shared conftest.py

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-13 17:41:37 +08:00
huanghuoguoguo
45f150da2d feat(agent-runner): integrate AgentRunner Protocol v1 with plugin system
Phase 0 integration complete - verified minimal loop with local-agent stub runner.

Changes:
- Add AgentRunOrchestrator for plugin-based agent execution
- Add AgentResultNormalizer for Protocol v1 result conversion
- Add AgentRunnerDescriptor for runner ID parsing (plugin:author/name/runner)
- Update chat handler to use new orchestrator instead of direct runner lookup
- Add plugin handler methods for list_agent_runners and run_agent
- Add connector methods for AgentRunner protocol forwarding
- Update pipeline API to include runner options in metadata
- Add integration docs and implementation plan

Integration verified:
- Runner: plugin:langbot/local-agent/default
- Input: "你好"
- Output: [stub] Echo: 你好
- Date: 2026-05-10 10:09

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-13 17:41:37 +08:00
Junyan Qin
6d87b7927d chore: stash code 2026-06-13 17:40:45 +08:00
RockChinQ
a97d2040bb fix(i18n,api): backfill missing token-monitoring keys and fix JWT expiry tz
- i18n: add models.searchProviders, monitoring.tabs.tokens and the
  monitoring.tokens.* block (incl. bucket.hour/day) to es-ES, ja-JP,
  ru-RU, th-TH, vi-VN and zh-Hant, which were missing them and failed
  the Check i18n Keys CI.
- api: generate_jwt_token built 'exp' from a naive datetime.now(), which
  PyJWT validates against UTC — in any timezone ahead of UTC the token
  was already expired at issue time. Use datetime.now(timezone.utc).
2026-06-13 05:26:18 -04:00
RockChinQ
a2c6c8201b refactor(persistence): freeze legacy DB migration chain, drop dbm026
The legacy pkg/persistence/migrations (DBMigration / dbmXXX) system now
coexists with Alembic but accepts no new migrations — all new schema
changes go through Alembic.

- remove dbm026_llm_model_context_length (superseded by Alembic
  0005_add_llm_context_length, which makes the identical change)
- cap required_database_version at 25 (legacy chain dbm001-025 kept
  read-only to upgrade pre-existing 3.x DBs to the Alembic baseline)
- add migrations/README.md documenting the freeze
- document the Alembic-only policy and revision-id/idempotency rules in
  AGENTS.md
2026-06-13 05:26:08 -04:00
RockChinQ
672abfe95d refactor(core): remove pre-3.x legacy config migration system
The pkg/core/migrations system (m001-m043 DBMigration-style config
migrations, MigrationStage, and the core.migration base class) only ever
ran when upgrading from LangBot 3.x. The last 3.x release is over a year
old and is no longer supported, so this dead code is removed entirely:

- delete pkg/core/migrations/ (43 mXXX_*.py + __init__)
- delete pkg/core/migration.py (base class + registry)
- delete pkg/core/stages/migrate.py (MigrationStage)
- drop 'MigrationStage' from boot.py stage_order
- delete tests/unit_tests/core/test_migration.py (tested the removed base class)
2026-06-13 05:26:01 -04:00
huanghuoguoguo
9ecb587ac0 refactor(provider): use LiteLLM as unified LLM requester backend (#2150)
* refactor(provider): use LiteLLM as unified LLM requester backend

  - Replace 23+ individual requester implementations with unified litellmchat.py
  - Add litellm_provider field to 27 YAML manifests for provider routing
  - Delete redundant requester subclasses
  - Add unit tests for LiteLLMRequester (29 tests)
  - Fix num_retries parameter name (was max_retries)
  - Fix exception handling order for subclass exceptions

  LiteLLM provides unified API for 100+ providers, eliminating need for
  provider-specific requesters.

* fix: ruff format provider.py

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* refactor(provider): simplify LiteLLM requester usage handling

  - Remove unused Anthropic-specific tool schema generation
  - Share completion argument construction between normal and streaming calls
  - Use LiteLLM/OpenAI native usage fields for monitoring
  - Collect stream token usage from LiteLLM stream_options
  - Update LiteLLM requester tests for unified usage fields

* restore: restore deleted provider requester files

Restore individual provider requester implementations that were
removed in de61b5d3. These files coexist with the unified
litellmchat.py backend.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat: update requesters and improve provider selection UI

- Added `litellm_provider` field to various requesters' YAML configurations.
- Removed obsolete Python requester files for OpenRouter, PPIO, QHAIGC, ShengSuanYun, SiliconFlow, Space, TokenPony, VolcArk, and Xai.
- Introduced new requesters for Tencent and Together AI with corresponding YAML configurations and SVG icons.
- Enhanced the ProviderForm component to include a searchable dropdown for selecting providers, improving user experience.
- Updated localization files to include search provider text for both English and Chinese.

* fix(provider): align litellm rebase with master

* fix(provider): capture streaming token usage; add token observability

The LiteLLM streaming requester only captured usage when a chunk had an
empty `choices` list. Many OpenAI-compatible gateways (e.g. new-api) and
providers send the final usage payload in a chunk that still carries an
empty-delta choice, so streamed calls always recorded 0 tokens in the
monitoring logs/dashboard (non-streaming worked).

- Capture stream usage whenever a chunk carries it, regardless of choices
- Add robust _normalize_usage (dict/obj shapes, derive missing total_tokens)
- Register litellm in bootutils/deps.py (was in pyproject only)
- Add MonitoringService.get_token_statistics + /monitoring/token-statistics
  endpoint: summary, per-model breakdown, token timeseries, and a
  zero-token-success data-quality signal
- Add TokenMonitoring dashboard tab (summary tiles, stacked token chart,
  per-model table) + i18n (en/zh)
- Regression tests for stream usage capture and usage normalization

Verified end-to-end against a real OpenAI-compatible endpoint with
gpt-5.5 and claude-opus-4-8: tokens now recorded non-zero for both
streaming and non-streaming paths.

* refactor(provider): simplify litellm capabilities

* style: simplify wrapped expressions

* feat(models): persist context metadata

* fix(provider): handle dict embeddings and openai-compatible rerank in LiteLLMRequester

- invoke_embedding: support both object- and dict-shaped response.data
  entries (OpenAI-compatible gateways like new-api return dicts)
- invoke_rerank: litellm.arerank rejects the 'openai' provider, so for
  openai-compatible (or unspecified) providers call the standard
  Jina/Cohere-style POST /v1/rerank endpoint directly over HTTP
- accept both 'relevance_score' and 'score' fields in rerank results
- add unit tests for the openai-compatible HTTP rerank path

* feat(provider): enforce requester support_type when adding models

- frontend: AddModelPopover only shows model-type tabs (llm/embedding/
  rerank) that the provider's requester declares in its manifest
  support_type; ModelsDialog fetches requester manifests and maps
  requester -> support_type, passed down through ProviderCard
- backend: add _validate_provider_supports guard in create_llm_model /
  create_embedding_model / create_rerank_model so a model cannot be
  attached to a provider whose requester does not support that type,
  even if the frontend restriction is bypassed (manifests without
  support_type are allowed for backward compatibility)
- manifests: correct support_type for providers that do not offer all
  three model types:
  - llm only: anthropic, deepseek, groq, moonshot, openrouter, xai
  - llm + text-embedding: openai, gemini, mistral
  - add rerank to new-api (verified working via /v1/rerank)
  - set llm + text-embedding + rerank for aggregator/unknown gateways

* feat(provider): add searchable alias to requester manifests

- add a free-text 'alias' field to every requester manifest spec,
  containing the vendor's English/Chinese names, pinyin, common
  nicknames and flagship model-series names (e.g. moonshot -> kimi,
  月之暗面; zhipu -> glm, 智谱清言)
- frontend: ProviderForm requester search now also matches against
  alias (substring/contains), so searching 'kimi' surfaces Moonshot,
  '硅基' surfaces SiliconFlow, etc.
- also fix support_type: openrouter (relay) supports embedding+rerank;
  LangBot Space gains rerank (coming soon)

* fix(provider): make support_type guard defensive against incomplete model_mgr

- _validate_provider_supports now uses getattr to gracefully skip when
  model_mgr / provider_dict / manifest lookup is unavailable, instead of
  raising AttributeError (fixes unit tests that mock ap.model_mgr as a
  bare SimpleNamespace)
- add TestValidateProviderSupports covering: allow supported type,
  reject unsupported type, allow when support_type missing, allow when
  provider unknown, degrade safely when model_mgr is incomplete

* fix(persistence): guard 0004 migration against missing llm_models table

The 0004_add_llm_model_context_length migration called
inspector.get_columns('llm_models') unconditionally, raising
NoSuchTableError when the table does not exist (e.g. migrating a
fresh/empty DB, as exercised by the integration tests where
create_all() registers no tables because the ORM models are not
imported). Every other migration guards with a table-existence check
first; add the same guard here for both upgrade and downgrade.

Also restore the test head assertion to 0004 (it had been lowered to
0003 to mask this failure).

* Merge branch 'master' into feat/litellm

Resolve conflicts:
- uv.lock: regenerated via 'uv lock' to reconcile litellm/fastuuid
  (ours) with openai bump (master).
- Alembic migrations: master added 0004_add_mcp_readme while this
  branch added 0004_add_llm_model_context_length, both as children of
  0003 (would create multiple heads). Re-chain the litellm migration as
  0005_add_llm_model_context_length with down_revision=0004_add_mcp_readme
  for a single linear head. Update test head assertion accordingly.

* fix(persistence): shorten migration revision id to fit varchar(32)

PostgreSQL stores alembic_version.version_num as varchar(32).
'0005_add_llm_model_context_length' (33 chars) overflowed it, raising
StringDataRightTruncationError in the PG migration tests. Rename the
revision (and file) to '0005_add_llm_context_length' (27 chars) and
update the head assertions in both SQLite and PostgreSQL migration
tests.

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: fdc310 <2213070223@qq.com>
Co-authored-by: RockChinQ <rockchinq@gmail.com>
2026-06-13 16:59:48 +08:00
RockChinQ
2b6dcfe9c7 feat(survey): add bot_response_success_100 milestone trigger event
Counts successful non-WebSocket bot responses (persisted in the metadata
table as survey_bot_response_count, survives restarts) and fires the
bot_response_success_100 survey event once the instance reaches 100
responses. Counting stops after the milestone has been triggered.

Existing first_bot_response_success behavior unchanged. 6 new unit tests.
2026-06-12 09:40:07 -04:00