xinyin025/LangBot
1
0
Fork 0
mirror of https://github.com/langbot-app/LangBot.git synced 2026-06-02 03:55:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

Feat/rag plugin (#1995)

Browse Source 
* [issue:1933] RAG engine plugin architecture (#1967)

* refactor: migrate RAG knowledge services to a plugin-oriented host service architecture.

* feat(rag): phase 2 core refactor with RPC Action handlers

* feat: 为 RAG 插件添加知识库创建和删除事件通知，并优化了 RAG 动作的参数传递和枚举使用。

* feat: 统一知识库管理为RAG引擎，支持动态配置并移除旧的外部知识库组件。

* refactor(rag): remove plugin_adapter, inline logic into RuntimeKnowledgeBase

BREAKING CHANGE: RAGPluginAdapter has been removed. All plugin
communication is now handled directly by RuntimeKnowledgeBase.

Architecture change:
- Before: RuntimeKnowledgeBase → RAGPluginAdapter → plugin_connector
- After:  RuntimeKnowledgeBase → plugin_connector (direct)

Changes to kbmgr.py (RuntimeKnowledgeBase):
- Remove RAGPluginAdapter import and usage
- Inline plugin communication methods:
  - _on_kb_create(): Notify plugin when KB is created
  - _on_kb_delete(): Notify plugin when KB is deleted
  - _ingest_document(): Call plugin for document ingestion
  - _retrieve(): Call plugin for retrieval
  - _delete_document(): Call plugin to delete document
- Simplify dispose(): Only notify plugin, no built-in VDB assumption

Changes to base.py (KnowledgeBaseInterface):
- Remove get_type() abstract method (outdated internal/external concept)
- Add get_rag_engine_plugin_id() abstract method

Changes to localagent.py:
- Remove get_type() call
- Simplify top_k retrieval from KB entity

Deleted files:
- pkg/rag/knowledge/plugin_adapter.py

Benefits:
- Reduced abstraction layer, simpler code
- Plugin communication logic centralized in RuntimeKnowledgeBase
- Easier to understand and maintain

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor(api): remove ExternalKnowledgeBase infrastructure

BREAKING CHANGE: ExternalKnowledgeBase has been completely removed.
All knowledge bases are now unified under the single KnowledgeBase model,
differentiated by their rag_engine_plugin_id.

Deleted files:
- pkg/api/http/controller/groups/knowledge/external.py
  (ExternalKBController with /external-bases routes)
- pkg/api/http/service/external_kb.py
  (ExternalKnowledgeBaseService)
- pkg/rag/knowledge/external.py
  (ExternalKnowledgeBase implementation)

Modified files:
- pkg/entity/persistence/rag.py:
  Remove ExternalKnowledgeBase SQLAlchemy table definition
- pkg/core/app.py:
  Remove external_kb_service attribute from LangBotApplication
- pkg/core/stages/build_app.py:
  Remove external_kb_service initialization

Migration notes:
- Existing external knowledge base data should be migrated manually
- API consumers should use /api/v1/knowledge/bases for all KB operations
- Use /api/v1/knowledge/engines to discover available RAG engines

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor(plugin): remove list_knowledge_retrievers from connector

Remove deprecated list_knowledge_retrievers functionality from the
plugin communication layer. This aligns with the SDK change that
removed the LIST_KNOWLEDGE_RETRIEVERS action.

Changes:
- connector.py: Remove list_knowledge_retrievers() method
- handler.py: Remove list_knowledge_retrievers() handler

The functionality is replaced by the new /api/v1/knowledge/engines
endpoint which lists available RAGEngine components with their
capabilities and configuration schemas.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor(service): update knowledge service with capability-based checks

Replace type-based checks with capability-based checks for file
operations, aligning with the unified knowledge base architecture.

Changes to knowledge.py:
- store_file(): Replace get_type() check with doc_ingestion capability check
- delete_file(): Replace get_type() check with doc_ingestion capability check
- list_rag_engines(): Remove list_knowledge_retrievers call, simplify to
  only list RAGEngine components (KnowledgeRetriever type removed)

Changes to pipelines.py:
- Minor cleanup related to knowledge base references

The capability-based approach allows RAG engines to declare their
supported features (doc_ingestion, chunking_config, rerank, hybrid_search)
and the system responds accordingly, rather than hardcoding behavior
based on internal/external type distinction.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(web): unify knowledge base UI, remove external KB components

BREAKING CHANGE: The internal/external knowledge base distinction
has been removed from the frontend. All knowledge bases are now
displayed in a unified list, differentiated by their RAG engine.

Changes to page.tsx:
- Remove Tab component (内置/外置 tabs)
- Remove selectedKbType state
- Unified knowledge base list display
- Single "Create Knowledge Base" button for all types

Changes to KBDetailDialog.tsx:
- Remove kbType prop
- Simplify dialog logic for unified KB handling
- Documents menu item conditionally shown based on doc_ingestion capability

Changes to KBForm.tsx:
- Remove retriever type handling code
- Simplify form for unified KB creation
- Dynamic form rendering based on RAG engine's creation_schema

Changes to KBCardVO.ts:
- Remove 'type' field from KBCardVO interface

Changes to BackendClient.ts:
- Remove all external KB related methods:
  - getExternalKnowledgeBases()
  - getExternalKnowledgeBase()
  - createExternalKnowledgeBase()
  - updateExternalKnowledgeBase()
  - deleteExternalKnowledgeBase()
  - retrieveFromExternalKnowledgeBase()

Changes to api/index.ts:
- Remove ExternalKnowledgeBase interface definition

UI/UX improvements:
- Users no longer need to understand internal vs external distinction
- RAG engine selection is now the primary differentiator
- Documents panel visibility is capability-driven (doc_ingestion)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor(plugin): code review improvements for RAG handlers

- Unify embed_model field naming to embedding_model_uuid only
- Add structured error responses with error_type for RAG actions
- Fix file_size and mime_type detection in _store_file_task
- Improve error handling with detailed error context (error_type, original_error)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor(rag): refactor KB dynamic form and vector manager

- Frontend: Refactor Knowledge Base form using DynamicForm components.
- Frontend: Remove obsolete jsonSchemaConverter utility.
- Backend: Update VectorManager and PluginHandler to support new RAG architecture.
- Chore: Update dependencies in pyproject.toml.

* fix: code review fixes for RAG refactor

- Remove DEBUG stderr outputs in handler.py
- Move repeated `import json` to file top
- Add warning log for unimplemented delete_by_filter

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor(rag): consolidate valid_fields into entity constants

Define MUTABLE_FIELDS, CREATE_FIELDS, ALL_DB_FIELDS as class
constants in KnowledgeBase entity to eliminate duplication.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: 将知识库获取和RAG引擎信息丰富逻辑移至知识库管理器。

* refactor(rag): introduce RAGRuntimeService and clean up plugin handler

- Create RAGRuntimeService to encapsulate RAG capability implementation (Embedding, VectorOps).
- Refactor PluginHandler to delegate RAG actions to RAGRuntimeService.
- Move KnowledgeService enrichment and creation logic to RAGManager.
- Register RAGRuntimeService in Application and BuildAppStage.
- Clean up legacy code in KnowledgeService.

* refactor(rag): standardize logger and fix type hints

- Use self.ap.logger consistently in kbmgr.py and runtime.py, removing module-level loggers.
- Fix type hints for retrieve_knowledge in handler.py and connector.py to match implementation returning dict.

* refactor: 将引擎徽章的样式从 Tailwind CSS 类迁移到 CSS 模块。

* fix(web): resolve React rendering errors in plugins page

- Fix missing key prop in PluginComponentList by using ternary instead of Fragment
- Fix RAGEngine.name type to I18nObject and use extractI18nObject() for rendering
- Preserves multi-language support

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(rag): update runtime service and web components

* refactor: 优化知识库设置结构并增强前端距离显示健壮性。

* fix: 处理前端距离显示中的空值。

* fix(rag): document retrieve ui and kbmgr top_k validation

* 更新 uv.lock 中的 PyPI 镜像源为官方地址。

* fix: address code review issues for RAG engine plugin architecture

P0 fixes:
- Fix ALL_DB_FIELDS missing collection_id and emoji fields
- Move rag_engine_plugin_id to CREATE_FIELDS (immutable after creation)
- Fix creation_settings mutable default value (dict -> None)
- Rename vector delete method to delete_by_file_id for correct semantics
- Fix delete_by_filter to raise NotImplementedError instead of silent no-op
- Add database migration script (dbm019) for new columns and table cleanup

P1 fixes:
- Clean up design-hesitation comments in connector.py
- Add _parse_plugin_id() with format validation for all RAG methods
- Make _retrieve() raise exceptions instead of silently returning empty results
- Extract _make_rag_error_response() helper for clean error formatting
- Remove unused imports from handler.py

P2 fixes:
- Fix runtime.py indentation inconsistencies
- Simplify get_file_stream to use storage abstraction uniformly
- Reduce redundant DB queries in knowledge service (extract _check_doc_capability)
- Fix engines.py URL encoding: use <path:plugin_id> instead of __ replacement
- Add read-only mode for engine settings in KBForm edit mode
- Simplify page.tsx handleKBCardClick to pass only kbId string

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: address code review findings for RAG plugin architecture

- Frontend: add retrieval_settings param to retrieveKnowledgeBase API call
- Backend: return {uuid} from PUT knowledge base to match frontend expectation
- Backend: validate query is non-empty in retrieve endpoint (400 on empty)
- Backend: rename vector_delete ids→file_ids for semantic clarity, keep
  backward compat by accepting both 'file_ids' and 'ids' in RPC handler
- Backend: ensure rag_engine.name fallback is always I18nObject-compatible
  dict, preventing frontend extractI18nObject from receiving plain strings
- Migration: fix misleading docstring about external_kb data migration

Co-authored-by: Cursor <cursoragent@cursor.com>

* Update langbot-plugin version to 0.2.6

* chore: update required database version from 18 to 19

* refactor: remove unused polymorphic component framework

* chore: fix lint and format issues for python and frontend

* fix(plugin): remove legacy `ids` fallback in rag_vector_delete handler

SDK now sends `file_ids` directly, the `ids` backward-compat fallback
is no longer needed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(rag): deep review fixes for critical bugs, security and quality

Critical:
- Fix StorageMgr.load() -> storage_provider.load() (C1, AttributeError)
- Update required_database_version 18 -> 19 (C2, migration never runs)

Security:
- Add path traversal validation in get_file_stream (C11)
- Add vectors/ids/metadata length validation in rag_vector_upsert (C12)

Logic fixes:
- Legacy KBs: set capabilities to [] instead of ['doc_ingestion'] (C4)
- Fix store_file return type int -> str (C5)
- Fix retrieve_knowledge return [] -> {'results': []} when disabled (C6)
- Re-raise exception in _on_kb_create instead of silently swallowing (C7)
- Log warning when KB not found in memory during delete (C8)

API fixes:
- Catch ValueError as 400 in create_knowledge_base endpoint (C15)
- Validate plugin_id format in engines endpoints (C16)

Quality:
- Remove dead if/else in migration with identical branches (C17)
- Fix variable shadowing: rag_context -> rag_context_text (C18)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: remove unused os import to fix ruff lint

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(plugin): remove PolymorphicComponent sync from LangBot side

Remove sync_polymorphic_component_instances() from connector and handler,
and the post-connection sync call in initialize(). This dead code synced
an always-empty list of polymorphic instances that were never created.

Companion change to langbot-plugin-sdk PolymorphicComponent removal.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(rag): fix vector_delete count bug and remove vestigial instance_id parameter

1. vector_delete: assign return value from delete_by_filter to count
   instead of silently returning 0 for filter-based deletion.

2. Remove instance_id parameter from the entire retrieve_knowledge
   call chain (kbmgr → connector → handler → runtime). This parameter
   was a remnant of the PolymorphicComponent mechanism and is no longer
   used — RAGEngine operates as a stateless singleton.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(web): 支持 creation_schema 字段级别的 editable 属性控制编辑模式可修改性

- IDynamicFormItemSchema 添加 editable 可选属性
- DynamicFormItemConfig 透传 editable 属性
- DynamicFormComponent 接收 isEditing prop，按字段 editable 值控制禁用
- KBForm 解析 editable 并传递 isEditing 给动态表单组件
- editable 未指定时默认可编辑，editable: false 时编辑模式下禁用该字段

* feat(storage): 添加 size() 抽象方法及 LocalStorage/S3 实现

支持获取存储对象大小，S3 使用 head_object 避免下载整个文件

* fix(migration): 删除 external_knowledge_bases 表前记录日志警告

- 迁移时如果表中存在数据，先 warning 日志记录避免无感数据丢失
- 添加 chunk 清理注释说明：仅对旧版非插件架构 KB 有效

* fix(web): 修复检索结果长文本撑大容器导致查询按钮不可见

KBDetailDialog 的 main 容器添加 min-w-0 overflow-x-hidden，
限制 flex-1 子容器宽度，防止 Dify RAG 长文本撑出 Dialog 边界

* fix(rag): address code review issues for plugin architecture PR

- Fix SQL injection in migration helpers by using bind parameters
- Move numpy import to module level in vector/mgr.py
- Improve path traversal validation using posixpath.normpath
- Add call_rag_retrieve to connector, eliminating duplicate plugin_id
  parsing in kbmgr.py _retrieve
- Normalize typing style to modern dict/list/None syntax

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* style(web): fix prettier formatting errors

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(rag): update embedding handling in RuntimeConnectionHandler

- Renamed RAG_EMBED_DOCUMENTS and RAG_EMBED_QUERY actions to INVOKE_EMBEDDING for clarity.
- Removed embed_documents and embed_query methods from RuntimeEmbeddingModel and RAGRuntimeService.
- Integrated embedding model retrieval directly in the invoke_embedding method, improving error handling for missing models.
- Updated the embedding invocation logic to streamline the process and enhance error reporting.

* refactor(web): replace KnowledgeRetriever with RAGEngine across frontend and tests

KnowledgeRetriever component type has been removed in favor of the new
RAGEngine architecture. Update all remaining references in i18n locales,
plugin component icon mappings, marketplace filter, and unit tests.

Addresses reviewer notes from RockChinQ on PR #1967.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(rag): address critical bugs found in deep review

- Fix path traversal bypass in runtime.py (check all path components for '..')
- Use normalized path for file loading instead of raw user input
- Change knowledge_bases from list to dict for O(1) lookup and race safety
- Add rollback on KB creation failure (clean up DB + runtime on plugin error)
- Add null check after KB update in knowledge service
- Fix file extension parsing to use os.path.splitext instead of split('.')
  (handles multi-dot filenames like 'report.v2.pdf' correctly)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(rag): address remaining review issues across frontend and backend

Frontend:
- Fix KB delete: use async/await with error handling instead of fire-and-forget
- Fix capabilities null check: add optional chaining to prevent crash
- Add toast.error on KB info load failure instead of silent console.error
- Replace hard-coded Chinese validation message with i18n key
- Replace hard-coded English error messages in DynamicFormItemComponent with i18n
- Optimize document polling: stop when all documents reach terminal state
- Add i18n keys (fieldRequired, loadKnowledgeBaseFailed,
  deleteKnowledgeBaseFailed, getKnowledgeBaseListError) to all 4 locales

Backend:
- Fix KB delete atomicity: delete from DB first, then notify plugin
- Add RAG engine plugin existence validation before creating KB

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* style(rag): fix ruff formatting in kbmgr.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Junyan Qin <rockchinq@gmail.com>

* chore: bump langbot-plugin to 0.3.0 (#1992)

* chore: correct sdk version to 0.3.0a1

* feat: normalize rag related actions' names

* refactor(rag): align IngestionContext fields with SDK changes

Remove redundant `chunking_strategy` field and rename `custom_settings`
to `creation_settings` to match the updated SDK entity definitions
(langbot-plugin-sdk#36).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* style: fix ruff formatting

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(rag): enforce immutability of embedding_model_uuid and non-editable creation_settings fields

Remove embedding_model_uuid from MUTABLE_FIELDS to prevent post-creation
modification via API. Add backend validation for creation_settings to
preserve fields marked editable:false in the plugin's creation schema.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* style(rag): fix ruff formatting in knowledge service

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(rag): split settings into immutable creation_settings and mutable retrieval_settings

- Remove standalone embedding_model_uuid and top_k columns from KB entity
- Add retrieval_settings column; update MUTABLE_FIELDS/CREATE_FIELDS accordingly
- Merge migration logic into dbm019 (add retrieval_settings, migrate top_k
  and embedding_model_uuid into JSON settings, drop old columns on PostgreSQL)
- Remove _filter_creation_settings and per-field editable concept
- Frontend: creation_settings fields are all disabled when editing,
  retrieval_settings fields are always editable via a second DynamicFormComponent
- Remove editable from IDynamicFormItemSchema, DynamicFormItemConfig
- Clean up KBCardVO, KnowledgeBase API type, and localagent runner

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* bugfix: if ingest_document failed,not raise exep

* fix: ruff lint

* refactor(rag): remove unused _get_kb_entity method from RAGRuntimeService

* feat(vector): implement metadata filters for vector_search and vector_delete (#1997)

Add functional metadata filter support across all 5 VDB backends using
Chroma-style where syntax as the canonical format. Previously the filters
parameter existed throughout the stack but was entirely ignored.

- Add filter_utils.py with normalize_filter() and strip_unsupported_fields()
- Implement filter in search() and add delete_by_filter() for all backends:
  Chroma/SeekDB (native passthrough), Qdrant (translated to models.Filter),
  Milvus (translated to expr string), pgvector (translated to SQLAlchemy conditions)
- Milvus/pgvector limited to {text, file_id, chunk_uuid}; other fields logged and ignored
- Replace delete_by_filter() NotImplementedError with backend delegation in mgr.py
- Populate retrieval_context['filters'] from settings in kbmgr._retrieve()
- Pass search_type/query_text/documents through handler and runtime service

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* style(vector): fix ruff formatting

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(vector): remove numpy dependency and fix SeekDB search modes

- Remove numpy array conversion for query vectors; all VDB backends
  accept list[float] directly
- Remove redundant get_or_create_collection call from upsert; backends
  handle collection creation internally in add_embeddings
- Fix SeekDB to raise ValueError when vector dimension is unknown
  instead of defaulting to 384
- Use hybrid_search() for full-text and hybrid search modes in SeekDB,
  since pyseekdb's query() always requires embeddings

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(vector): escape single quotes in SeekDB documents and metadata

Document text containing apostrophes (e.g. "don't", "it's") causes
SQL syntax errors in OceanBase because single quotes were not in the
escape table. Add single-quote escaping and apply the escape table to
the documents parameter in add_embeddings(), not just metadata.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(vector): use standard SQL escaping for single quotes in SeekDB

Change single quote escaping from MySQL-style \' to standard SQL ''
(doubled quote). The backslash escape is not recognized by OceanBase
in NO_BACKSLASH_ESCAPES mode, causing SQL syntax errors when metadata
text contains apostrophes (e.g. O'Shea in academic citations).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(rag): persist retrieval_settings on knowledge base creation

retrieval_settings was not being passed from the service layer to
RAGManager.create_knowledge_base(), causing retrieval schema fields
(e.g. query_rewrite) to be lost on initial KB creation. They only
took effect after a subsequent edit/update.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(web): add show_if conditional rendering for dynamic forms

Support conditional field visibility in plugin-defined forms via
show_if rules (eq, neq, in operators). Fields can depend on values
from the same form or cross-reference between creation and retrieval
settings via externalDependentValues.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(rag): replace base64 with chunked file transfer for get_rag_file_stream

Use send_file() instead of base64 encoding for returning file content
in the GET_RAG_FILE_STREAM handler, avoiding memory issues with large files.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(parser): add parser plugin integration and capability-aware upload UI (#2000)

* feat(parser): add parser plugin integration and capability-aware upload UI

Backend: add parser plugin API endpoints (list/invoke), connector and
handler support for parser actions, and KB manager passthrough.

Frontend: thread ragEngineCapabilities prop to FileUploadZone and use
doc_parsing capability to conditionally show the RAG engine option in
the parser selector. When no parser is available, show a warning
prompting users to install a parser plugin.

Update i18n: rename builtInParser to "Provided by RAG engine" and add
noParserAvailable warning message in all 4 locales.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(parser): replace base64 with chunked file transfer and remove stale cache

- Remove @alru_cache from list_parsers() and list_rag_engines()
- Replace inline base64 file content with send_file/read_local_file
  chunked transfer pattern in parse_document and invoke_parser flows
- Remove unused base64 import from kbmgr.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* feat(web): add Parser component kind to plugin market UI and i18n

Add Parser to kindIconMap, market filter toggle, and all 4 locale files
so parser plugins are properly displayed and filterable in the plugin
market, matching the existing RAGEngine treatment.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* style(web): fix prettier formatting from merge

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: rename RAGEngine to KnowledgeEngine across frontend and backend

* fix(web): fix I18nObject import path in FileUploadZone and KBDoc

* chore: format files involved in RAGEngine to KnowledgeEngine refactor

* refactor: change rag engine to knowledge engine

* fix: update langbot-plugin version to 0.3.0rc1

* chore: disable migration 20 for now

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Junyan Qin <rockchinq@gmail.com>
This commit is contained in:
huanghuoguoguo
2026-03-06 21:54:38 +08:00
committed by GitHub
parent 3e8f47fd97
commit cadcf10047
67 changed files with 2962 additions and 2703 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
pyproject.toml
View File

@@ -64,7 +64,7 @@ dependencies = [
"chromadb>=0.4.24",
"qdrant-client (>=1.15.1,<2.0.0)",
"pyseekdb==1.0.0b7",
"langbot-plugin==0.2.7",
"langbot-plugin==0.3.0rc1",
"asyncpg>=0.30.0",
"line-bot-sdk>=3.19.0",
"tboxsdk>=0.0.10",

23
src/langbot/pkg/api/http/controller/groups/knowledge/base.py
View File

@@ -13,7 +13,10 @@ class KnowledgeBaseRouterGroup(group.RouterGroup):
elif quart.request.method == 'POST':
json_data = await quart.request.json
knowledge_base_uuid = await self.ap.knowledge_service.create_knowledge_base(json_data)
try:
knowledge_base_uuid = await self.ap.knowledge_service.create_knowledge_base(json_data)
except ValueError as e:
return self.http_status(400, -1, str(e))
return self.success(data={'uuid': knowledge_base_uuid})
return self.http_status(405, -1, 'Method not allowed')
@@ -39,7 +42,7 @@ class KnowledgeBaseRouterGroup(group.RouterGroup):
elif quart.request.method == 'PUT':
json_data = await quart.request.json
await self.ap.knowledge_service.update_knowledge_base(knowledge_base_uuid, json_data)
return self.success({})
return self.success(data={'uuid': knowledge_base_uuid})
elif quart.request.method == 'DELETE':
await self.ap.knowledge_service.delete_knowledge_base(knowledge_base_uuid)
@@ -65,8 +68,12 @@ class KnowledgeBaseRouterGroup(group.RouterGroup):
if not file_id:
return self.http_status(400, -1, 'File ID is required')
parser_plugin_id = json_data.get('parser_plugin_id')
# 调用服务层方法将文件与知识库关联
task_id = await self.ap.knowledge_service.store_file(knowledge_base_uuid, file_id)
task_id = await self.ap.knowledge_service.store_file(
knowledge_base_uuid, file_id, parser_plugin_id=parser_plugin_id
)
return self.success(
{
'task_id': task_id,
@@ -90,5 +97,13 @@ class KnowledgeBaseRouterGroup(group.RouterGroup):
async def retrieve_knowledge_base(knowledge_base_uuid: str) -> str:
json_data = await quart.request.json
query = json_data.get('query')
results = await self.ap.knowledge_service.retrieve_knowledge_base(knowledge_base_uuid, query)
if not query or not query.strip():
return self.http_status(400, -1, 'Query is required and cannot be empty')
# Extract retrieval_settings to allow dynamic control over Knowledge Engine behavior (e.g. top_k, filters)
retrieval_settings = json_data.get('retrieval_settings', {})
results = await self.ap.knowledge_service.retrieve_knowledge_base(
knowledge_base_uuid, query, retrieval_settings
)
return self.success(data={'results': results})

45
src/langbot/pkg/api/http/controller/groups/knowledge/engines.py Normal file
View File

@@ -0,0 +1,45 @@
import quart
from urllib.parse import unquote
from ... import group
@group.group_class('knowledge_engines', '/api/v1/knowledge/engines')
class KnowledgeEnginesRouterGroup(group.RouterGroup):
async def initialize(self) -> None:
@self.route('', methods=['GET'], auth_type=group.AuthType.USER_TOKEN_OR_API_KEY)
async def list_knowledge_engines() -> quart.Response:
"""List all available Knowledge Engines from plugins.
Returns a list of Knowledge Engines with their capabilities and configuration schemas.
This is used by the frontend to render the knowledge base creation wizard.
"""
engines = await self.ap.knowledge_service.list_knowledge_engines()
return self.success(data={'engines': engines})
@self.route(
'/<path:plugin_id>/creation-schema', methods=['GET'], auth_type=group.AuthType.USER_TOKEN_OR_API_KEY
)
async def get_engine_creation_schema(plugin_id: str) -> quart.Response:
"""Get creation settings schema for a specific Knowledge Engine.
plugin_id is in 'author/name' format, captured via <path:> converter.
"""
plugin_id = unquote(plugin_id)
if '/' not in plugin_id:
return self.http_status(400, -1, 'Invalid plugin_id format. Expected author/name.')
schema = await self.ap.knowledge_service.get_engine_creation_schema(plugin_id)
return self.success(data={'schema': schema})
@self.route(
'/<path:plugin_id>/retrieval-schema', methods=['GET'], auth_type=group.AuthType.USER_TOKEN_OR_API_KEY
)
async def get_engine_retrieval_schema(plugin_id: str) -> quart.Response:
"""Get retrieval settings schema for a specific Knowledge Engine.
plugin_id is in 'author/name' format, captured via <path:> converter.
"""
plugin_id = unquote(plugin_id)
if '/' not in plugin_id:
return self.http_status(400, -1, 'Invalid plugin_id format. Expected author/name.')
schema = await self.ap.knowledge_service.get_engine_retrieval_schema(plugin_id)
return self.success(data={'schema': schema})

61
src/langbot/pkg/api/http/controller/groups/knowledge/external.py
View File

@@ -1,61 +0,0 @@
import quart
from ... import group
@group.group_class('external_knowledge_base', '/api/v1/knowledge/external-bases')
class ExternalKnowledgeBaseRouterGroup(group.RouterGroup):
async def initialize(self) -> None:
@self.route('/retrievers', methods=['GET'])
async def list_knowledge_retrievers() -> quart.Response:
"""List all available knowledge retrievers from plugins."""
retrievers = await self.ap.plugin_connector.list_knowledge_retrievers()
return self.success(data={'retrievers': retrievers})
@self.route('', methods=['POST', 'GET'])
async def handle_external_knowledge_bases() -> quart.Response:
if quart.request.method == 'GET':
external_kbs = await self.ap.external_kb_service.get_external_knowledge_bases()
return self.success(data={'bases': external_kbs})
elif quart.request.method == 'POST':
json_data = await quart.request.json
kb_uuid = await self.ap.external_kb_service.create_external_knowledge_base(json_data)
return self.success(data={'uuid': kb_uuid})
return self.http_status(405, -1, 'Method not allowed')
@self.route(
'/<kb_uuid>',
methods=['GET', 'DELETE', 'PUT'],
)
async def handle_specific_external_knowledge_base(kb_uuid: str) -> quart.Response:
if quart.request.method == 'GET':
external_kb = await self.ap.external_kb_service.get_external_knowledge_base(kb_uuid)
if external_kb is None:
return self.http_status(404, -1, 'external knowledge base not found')
return self.success(
data={
'base': external_kb,
}
)
elif quart.request.method == 'PUT':
json_data = await quart.request.json
await self.ap.external_kb_service.update_external_knowledge_base(kb_uuid, json_data)
return self.success({})
elif quart.request.method == 'DELETE':
await self.ap.external_kb_service.delete_external_knowledge_base(kb_uuid)
return self.success({})
@self.route(
'/<kb_uuid>/retrieve',
methods=['POST'],
)
async def retrieve_external_knowledge_base(kb_uuid: str) -> str:
json_data = await quart.request.json
query = json_data.get('query')
results = await self.ap.external_kb_service.retrieve_external_knowledge_base(kb_uuid, query)
return self.success(data={'results': results})

16
src/langbot/pkg/api/http/controller/groups/knowledge/parsers.py Normal file
View File

@@ -0,0 +1,16 @@
import quart
from ... import group
@group.group_class('parsers', '/api/v1/knowledge/parsers')
class ParsersRouterGroup(group.RouterGroup):
async def initialize(self) -> None:
@self.route('', methods=['GET'], auth_type=group.AuthType.USER_TOKEN_OR_API_KEY)
async def list_parsers() -> quart.Response:
"""List all available parsers from plugins.
Optional query parameter `mime_type` to filter parsers by supported MIME type.
"""
mime_type = quart.request.args.get('mime_type')
parsers = await self.ap.knowledge_service.list_parsers(mime_type)
return self.success(data={'parsers': parsers})

2
src/langbot/pkg/api/http/controller/groups/pipelines/pipelines.py
View File

@@ -68,7 +68,7 @@ class PipelinesRouterGroup(group.RouterGroup):
return self.http_status(404, -1, 'pipeline not found')
# Only include plugins with pipeline-related components (Command, EventListener, Tool)
# Plugins that only have KnowledgeRetriever components are not suitable for pipeline extensions
# Plugins that only have KnowledgeEngine components are not suitable for pipeline extensions
pipeline_component_kinds = ['Command', 'EventListener', 'Tool']
plugins = await self.ap.plugin_connector.list_plugins(component_kinds=pipeline_component_kinds)
mcp_servers = await self.ap.mcp_service.get_mcp_servers(contain_runtime_info=True)

80
src/langbot/pkg/api/http/service/external_kb.py
View File

@@ -1,80 +0,0 @@
from __future__ import annotations
from ....core import app
import sqlalchemy
from langbot.pkg.entity.persistence import rag as persistence_rag
import uuid
class ExternalKBService:
"""External KB service"""
ap: app.Application
def __init__(self, ap: app.Application) -> None:
self.ap = ap
# External Knowledge Base methods
async def get_external_knowledge_bases(self) -> list[dict]:
result = await self.ap.persistence_mgr.execute_async(sqlalchemy.select(persistence_rag.ExternalKnowledgeBase))
external_kbs = result.all()
return [
self.ap.persistence_mgr.serialize_model(persistence_rag.ExternalKnowledgeBase, external_kb)
for external_kb in external_kbs
]
async def get_external_knowledge_base(self, kb_uuid: str) -> dict | None:
result = await self.ap.persistence_mgr.execute_async(
sqlalchemy.select(persistence_rag.ExternalKnowledgeBase).where(
persistence_rag.ExternalKnowledgeBase.uuid == kb_uuid
)
)
external_kb = result.first()
if external_kb is None:
return None
return self.ap.persistence_mgr.serialize_model(persistence_rag.ExternalKnowledgeBase, external_kb)
async def create_external_knowledge_base(self, kb_data: dict) -> str:
kb_data['uuid'] = str(uuid.uuid4())
await self.ap.persistence_mgr.execute_async(
sqlalchemy.insert(persistence_rag.ExternalKnowledgeBase).values(kb_data)
)
kb = await self.get_external_knowledge_base(kb_data['uuid'])
await self.ap.rag_mgr.load_external_knowledge_base(kb)
return kb_data['uuid']
async def retrieve_external_knowledge_base(self, kb_uuid: str, query: str) -> list[dict]:
"""Retrieve external knowledge base"""
runtime_kb = await self.ap.rag_mgr.get_knowledge_base_by_uuid(kb_uuid)
if runtime_kb is None:
raise Exception('Knowledge base not found')
return [
result.model_dump() for result in await runtime_kb.retrieve(query, 5)
] # top_k is just a placeholder for external knowledge base
async def update_external_knowledge_base(self, kb_uuid: str, kb_data: dict) -> None:
if 'uuid' in kb_data:
del kb_data['uuid']
await self.ap.persistence_mgr.execute_async(
sqlalchemy.update(persistence_rag.ExternalKnowledgeBase)
.values(kb_data)
.where(persistence_rag.ExternalKnowledgeBase.uuid == kb_uuid)
)
await self.ap.rag_mgr.remove_knowledge_base_from_runtime(kb_uuid)
kb = await self.get_external_knowledge_base(kb_uuid)
await self.ap.rag_mgr.load_external_knowledge_base(kb)
async def delete_external_knowledge_base(self, kb_uuid: str) -> None:
await self.ap.persistence_mgr.execute_async(
sqlalchemy.delete(persistence_rag.ExternalKnowledgeBase).where(
persistence_rag.ExternalKnowledgeBase.uuid == kb_uuid
)
)
await self.ap.rag_mgr.delete_knowledge_base(kb_uuid)

149
src/langbot/pkg/api/http/service/knowledge.py
View File

@@ -1,6 +1,5 @@
from __future__ import annotations
import uuid
import sqlalchemy
from ....core import app
@@ -17,64 +16,77 @@ class KnowledgeService:
async def get_knowledge_bases(self) -> list[dict]:
"""获取所有知识库"""
result = await self.ap.persistence_mgr.execute_async(sqlalchemy.select(persistence_rag.KnowledgeBase))
knowledge_bases = result.all()
return [
self.ap.persistence_mgr.serialize_model(persistence_rag.KnowledgeBase, knowledge_base)
for knowledge_base in knowledge_bases
]
return await self.ap.rag_mgr.get_all_knowledge_base_details()
async def get_knowledge_base(self, kb_uuid: str) -> dict | None:
"""获取知识库"""
result = await self.ap.persistence_mgr.execute_async(
sqlalchemy.select(persistence_rag.KnowledgeBase).where(persistence_rag.KnowledgeBase.uuid == kb_uuid)
)
knowledge_base = result.first()
if knowledge_base is None:
return None
return self.ap.persistence_mgr.serialize_model(persistence_rag.KnowledgeBase, knowledge_base)
return await self.ap.rag_mgr.get_knowledge_base_details(kb_uuid)
async def create_knowledge_base(self, kb_data: dict) -> str:
"""创建知识库"""
kb_data['uuid'] = str(uuid.uuid4())
await self.ap.persistence_mgr.execute_async(sqlalchemy.insert(persistence_rag.KnowledgeBase).values(kb_data))
# In new architecture, we delegate entirely to RAGManager which uses plugins.
# Legacy internal KB creation is removed.
kb = await self.get_knowledge_base(kb_data['uuid'])
knowledge_engine_plugin_id = kb_data.get('knowledge_engine_plugin_id')
if not knowledge_engine_plugin_id:
raise ValueError('knowledge_engine_plugin_id is required')
await self.ap.rag_mgr.load_knowledge_base(kb)
return kb_data['uuid']
kb = await self.ap.rag_mgr.create_knowledge_base(
name=kb_data.get('name', 'Untitled'),
knowledge_engine_plugin_id=knowledge_engine_plugin_id,
creation_settings=kb_data.get('creation_settings', {}),
retrieval_settings=kb_data.get('retrieval_settings', {}),
description=kb_data.get('description', ''),
)
return kb.uuid
async def update_knowledge_base(self, kb_uuid: str, kb_data: dict) -> None:
"""更新知识库"""
if 'uuid' in kb_data:
del kb_data['uuid']
# Filter to only mutable fields
filtered_data = {k: v for k, v in kb_data.items() if k in persistence_rag.KnowledgeBase.MUTABLE_FIELDS}
if 'embedding_model_uuid' in kb_data:
del kb_data['embedding_model_uuid']
if not filtered_data:
return
await self.ap.persistence_mgr.execute_async(
sqlalchemy.update(persistence_rag.KnowledgeBase)
.values(kb_data)
.values(filtered_data)
.where(persistence_rag.KnowledgeBase.uuid == kb_uuid)
)
await self.ap.rag_mgr.remove_knowledge_base_from_runtime(kb_uuid)
kb = await self.get_knowledge_base(kb_uuid)
if kb is None:
raise Exception('Knowledge base not found after update')
await self.ap.rag_mgr.load_knowledge_base(kb)
async def store_file(self, kb_uuid: str, file_id: str) -> int:
async def _check_doc_capability(self, kb_uuid: str, operation: str) -> None:
"""Check if the KB's Knowledge Engine supports document operations.
Args:
kb_uuid: Knowledge base UUID.
operation: Human-readable operation name for error messages.
Raises:
Exception: If the KB does not support doc_ingestion.
"""
kb_info = await self.ap.rag_mgr.get_knowledge_base_details(kb_uuid)
if not kb_info:
raise Exception('Knowledge base not found')
capabilities = kb_info.get('knowledge_engine', {}).get('capabilities', [])
if 'doc_ingestion' not in capabilities:
raise Exception(f'This knowledge base does not support {operation}')
async def store_file(self, kb_uuid: str, file_id: str, parser_plugin_id: str | None = None) -> str:
"""存储文件"""
# await self.ap.persistence_mgr.execute_async(sqlalchemy.insert(persistence_rag.File).values(kb_id=kb_uuid, file_id=file_id))
# await self.ap.rag_mgr.store_file(file_id)
runtime_kb = await self.ap.rag_mgr.get_knowledge_base_by_uuid(kb_uuid)
if runtime_kb is None:
raise Exception('Knowledge base not found')
# Only internal KBs support file storage
if runtime_kb.get_type() != 'internal':
raise Exception('Only internal knowledge bases support file storage')
result = await runtime_kb.store_file(file_id)
await self._check_doc_capability(kb_uuid, 'document upload')
result = await runtime_kb.store_file(file_id, parser_plugin_id=parser_plugin_id)
# Update the KB's updated_at timestamp
await self.ap.persistence_mgr.execute_async(
@@ -85,14 +97,18 @@ class KnowledgeService:
return result
async def retrieve_knowledge_base(self, kb_uuid: str, query: str) -> list[dict]:
async def retrieve_knowledge_base(
self, kb_uuid: str, query: str, retrieval_settings: dict | None = None
) -> list[dict]:
"""检索知识库"""
runtime_kb = await self.ap.rag_mgr.get_knowledge_base_by_uuid(kb_uuid)
if runtime_kb is None:
raise Exception('Knowledge base not found')
return [
result.model_dump() for result in await runtime_kb.retrieve(query, runtime_kb.knowledge_base_entity.top_k)
]
# Pass retrieval_settings
results = await runtime_kb.retrieve(query, settings=retrieval_settings)
return [result.model_dump() for result in results]
async def get_files_by_knowledge_base(self, kb_uuid: str) -> list[dict]:
"""获取知识库文件"""
@@ -107,9 +123,9 @@ class KnowledgeService:
runtime_kb = await self.ap.rag_mgr.get_knowledge_base_by_uuid(kb_uuid)
if runtime_kb is None:
raise Exception('Knowledge base not found')
# Only internal KBs support file deletion
if runtime_kb.get_type() != 'internal':
raise Exception('Only internal knowledge bases support file deletion')
await self._check_doc_capability(kb_uuid, 'document deletion')
await runtime_kb.delete_file(file_id)
# Update the KB's updated_at timestamp
@@ -121,13 +137,14 @@ class KnowledgeService:
async def delete_knowledge_base(self, kb_uuid: str) -> None:
"""删除知识库"""
await self.ap.rag_mgr.delete_knowledge_base(kb_uuid)
# Delete from DB first to commit the deletion, then clean up runtime/plugin (best-effort)
await self.ap.persistence_mgr.execute_async(
sqlalchemy.delete(persistence_rag.KnowledgeBase).where(persistence_rag.KnowledgeBase.uuid == kb_uuid)
)
# delete files
# NOTE: Chunk cleanup is for legacy (pre-plugin) KBs that stored chunks locally.
# For plugin-based Knowledge Engines, the Chunk table is not populated, so this is a no-op.
files = await self.ap.persistence_mgr.execute_async(
sqlalchemy.select(persistence_rag.File).where(persistence_rag.File.kb_id == kb_uuid)
)
@@ -140,3 +157,53 @@ class KnowledgeService:
await self.ap.persistence_mgr.execute_async(
sqlalchemy.delete(persistence_rag.File).where(persistence_rag.File.uuid == file.uuid)
)
# Remove from runtime and notify plugin (best-effort, DB is already cleaned up)
await self.ap.rag_mgr.delete_knowledge_base(kb_uuid)
# ================= Knowledge Engine Discovery =================
async def list_knowledge_engines(self) -> list[dict]:
"""List all available Knowledge Engines from plugins."""
engines = []
if not self.ap.plugin_connector.is_enable_plugin:
return engines
# Get KnowledgeEngine plugins
try:
knowledge_engines = await self.ap.plugin_connector.list_knowledge_engines()
engines.extend(knowledge_engines)
except Exception as e:
self.ap.logger.warning(f'Failed to list Knowledge Engines from plugins: {e}')
return engines
async def list_parsers(self, mime_type: str | None = None) -> list[dict]:
"""List available parsers, optionally filtered by MIME type."""
if not self.ap.plugin_connector.is_enable_plugin:
return []
try:
parsers = await self.ap.plugin_connector.list_parsers()
if mime_type:
parsers = [p for p in parsers if mime_type in p.get('supported_mime_types', [])]
return parsers
except Exception as e:
self.ap.logger.warning(f'Failed to list parsers: {e}')
return []
async def get_engine_creation_schema(self, plugin_id: str) -> dict:
"""Get creation settings schema for a specific Knowledge Engine."""
try:
return await self.ap.plugin_connector.get_rag_creation_schema(plugin_id)
except Exception as e:
self.ap.logger.warning(f'Failed to get creation schema for {plugin_id}: {e}')
return {}
async def get_engine_retrieval_schema(self, plugin_id: str) -> dict:
"""Get retrieval settings schema for a specific Knowledge Engine."""
try:
return await self.ap.plugin_connector.get_rag_retrieval_schema(plugin_id)
except Exception as e:
self.ap.logger.warning(f'Failed to get retrieval schema for {plugin_id}: {e}')
return {}

5
src/langbot/pkg/core/app.py
View File

@@ -29,7 +29,6 @@ from ..api.http.service import knowledge as knowledge_service
from ..api.http.service import mcp as mcp_service
from ..api.http.service import apikey as apikey_service
from ..api.http.service import webhook as webhook_service
from ..api.http.service import external_kb as external_kb_service
from ..api.http.service import monitoring as monitoring_service
from ..discover import engine as discover_engine
from ..storage import mgr as storagemgr
@@ -37,6 +36,7 @@ from ..utils import logcache
from . import taskmgr
from . import entities as core_entities
from ..rag.knowledge import kbmgr as rag_mgr
from ..rag.service import RAGRuntimeService
from ..vector import mgr as vectordb_mgr
from ..telemetry import telemetry as telemetry_module
from ..survey import manager as survey_module
@@ -63,6 +63,7 @@ class Application:
model_mgr: llm_model_mgr.ModelManager = None
rag_mgr: rag_mgr.RAGManager = None
rag_runtime_service: RAGRuntimeService = None
# TODO move to pipeline
tool_mgr: llm_tool_mgr.ToolManager = None
@@ -138,8 +139,6 @@ class Application:
knowledge_service: knowledge_service.KnowledgeService = None
external_kb_service: external_kb_service.ExternalKBService = None
mcp_service: mcp_service.MCPService = None
apikey_service: apikey_service.ApiKeyService = None

8
src/langbot/pkg/core/stages/build_app.py
View File

@@ -12,6 +12,7 @@ from ...provider.session import sessionmgr as llm_session_mgr
from ...provider.modelmgr import modelmgr as llm_model_mgr
from ...provider.tools import toolmgr as llm_tool_mgr
from ...rag.knowledge import kbmgr as rag_mgr
from ...rag.service import RAGRuntimeService
from ...platform import botmgr as im_mgr
from ...platform.webhook_pusher import WebhookPusher
from ...persistence import mgr as persistencemgr
@@ -26,7 +27,6 @@ from ...api.http.service import knowledge as knowledge_service
from ...api.http.service import mcp as mcp_service
from ...api.http.service import apikey as apikey_service
from ...api.http.service import webhook as webhook_service
from ...api.http.service import external_kb as external_kb_service
from ...api.http.service import monitoring as monitoring_service
from ...discover import engine as discover_engine
from ...storage import mgr as storagemgr
@@ -73,9 +73,6 @@ class BuildAppStage(stage.BootingStage):
knowledge_service_inst = knowledge_service.KnowledgeService(ap)
ap.knowledge_service = knowledge_service_inst
external_kb_service_inst = external_kb_service.ExternalKBService(ap)
ap.external_kb_service = external_kb_service_inst
mcp_service_inst = mcp_service.MCPService(ap)
ap.mcp_service = mcp_service_inst
@@ -152,6 +149,9 @@ class BuildAppStage(stage.BootingStage):
await rag_mgr_inst.initialize()
ap.rag_mgr = rag_mgr_inst
# Initialize RAG Runtime Service for plugins
ap.rag_runtime_service = RAGRuntimeService(ap)
# 初始化向量数据库管理器
vectordb_mgr_inst = vectordb_mgr.VectorDBManager(ap)
await vectordb_mgr_inst.initialize()

30
src/langbot/pkg/entity/persistence/rag.py
View File

@@ -10,8 +10,21 @@ class KnowledgeBase(Base):
emoji = sqlalchemy.Column(sqlalchemy.String(10), nullable=True, default='📚')
created_at = sqlalchemy.Column(sqlalchemy.DateTime, default=sqlalchemy.func.now())
updated_at = sqlalchemy.Column(sqlalchemy.DateTime, default=sqlalchemy.func.now(), onupdate=sqlalchemy.func.now())
embedding_model_uuid = sqlalchemy.Column(sqlalchemy.String, default='')
top_k = sqlalchemy.Column(sqlalchemy.Integer, default=5)
# New fields for plugin-based RAG
knowledge_engine_plugin_id = sqlalchemy.Column(sqlalchemy.String, nullable=True)
collection_id = sqlalchemy.Column(sqlalchemy.String, nullable=True)
creation_settings = sqlalchemy.Column(sqlalchemy.JSON, nullable=True, default=None)
retrieval_settings = sqlalchemy.Column(sqlalchemy.JSON, nullable=True, default=None)
# Field sets for different operations
MUTABLE_FIELDS = {'name', 'description', 'retrieval_settings'}
"""Fields that can be updated after creation."""
CREATE_FIELDS = MUTABLE_FIELDS | {'uuid', 'knowledge_engine_plugin_id', 'collection_id', 'creation_settings'}
"""Fields used when creating a new knowledge base."""
ALL_DB_FIELDS = CREATE_FIELDS | {'emoji', 'created_at', 'updated_at'}
"""All fields stored in database (for loading from DB row)."""
class File(Base):
@@ -29,16 +42,3 @@ class Chunk(Base):
uuid = sqlalchemy.Column(sqlalchemy.String(255), primary_key=True, unique=True)
file_id = sqlalchemy.Column(sqlalchemy.String(255), nullable=True)
text = sqlalchemy.Column(sqlalchemy.Text)
class ExternalKnowledgeBase(Base):
__tablename__ = 'external_knowledge_bases'
uuid = sqlalchemy.Column(sqlalchemy.String(255), primary_key=True, unique=True)
name = sqlalchemy.Column(sqlalchemy.String, index=True)
description = sqlalchemy.Column(sqlalchemy.Text)
emoji = sqlalchemy.Column(sqlalchemy.String(10), nullable=True, default='🔗')
plugin_author = sqlalchemy.Column(sqlalchemy.String, nullable=False)
plugin_name = sqlalchemy.Column(sqlalchemy.String, nullable=False)
retriever_name = sqlalchemy.Column(sqlalchemy.String, nullable=False)
retriever_config = sqlalchemy.Column(sqlalchemy.JSON, nullable=False, default={})
created_at = sqlalchemy.Column(sqlalchemy.DateTime, default=sqlalchemy.func.now())

184
src/langbot/pkg/persistence/migrations/dbm020_knowledge_engine_plugin_architecture.py Normal file
View File

@@ -0,0 +1,184 @@
import json
import sqlalchemy
from .. import migration
@migration.migration_class(20)
class DBMigrateKnowledgeEnginePluginArchitecture(migration.DBMigration):
"""Migrate to unified Knowledge Engine plugin architecture.
Changes:
- Add knowledge_engine_plugin_id, collection_id, creation_settings, retrieval_settings columns to knowledge_bases
- Migrate existing top_k values into retrieval_settings JSON
- Migrate existing embedding_model_uuid into creation_settings JSON
- Drop embedding_model_uuid and top_k columns (PostgreSQL only; SQLite leaves them unmapped)
- Drop external_knowledge_bases table (no longer needed; external KB data is not migrated)
"""
async def upgrade(self):
"""Upgrade"""
await self._add_columns_to_knowledge_bases()
await self._migrate_top_k_to_retrieval_settings()
await self._migrate_embedding_model_uuid_to_creation_settings()
await self._drop_old_columns()
await self._drop_external_knowledge_bases_table()
async def _get_table_columns(self, table_name: str) -> list[str]:
"""Get column names from a table (works for both SQLite and PostgreSQL)."""
if self.ap.persistence_mgr.db.name == 'postgresql':
result = await self.ap.persistence_mgr.execute_async(
sqlalchemy.text(
'SELECT column_name FROM information_schema.columns WHERE table_name = :table_name;'
).bindparams(table_name=table_name)
)
return [row[0] for row in result.fetchall()]
else:
# SQLite PRAGMA does not support bind parameters; validate identifier.
if not table_name.isidentifier():
raise ValueError(f'Invalid table name: {table_name}')
result = await self.ap.persistence_mgr.execute_async(sqlalchemy.text(f'PRAGMA table_info({table_name});'))
return [row[1] for row in result.fetchall()]
async def _table_exists(self, table_name: str) -> bool:
"""Check if a table exists."""
if self.ap.persistence_mgr.db.name == 'postgresql':
result = await self.ap.persistence_mgr.execute_async(
sqlalchemy.text(
'SELECT EXISTS (SELECT FROM information_schema.tables WHERE table_name = :table_name);'
).bindparams(table_name=table_name)
)
return result.scalar()
else:
result = await self.ap.persistence_mgr.execute_async(
sqlalchemy.text("SELECT name FROM sqlite_master WHERE type='table' AND name=:table_name;").bindparams(
table_name=table_name
)
)
return result.first() is not None
async def _add_columns_to_knowledge_bases(self):
"""Add new RAG plugin architecture columns to knowledge_bases table."""
columns = await self._get_table_columns('knowledge_bases')
new_columns = {
'knowledge_engine_plugin_id': 'VARCHAR',
'collection_id': 'VARCHAR',
'creation_settings': 'TEXT', # JSON stored as TEXT for SQLite compatibility
'retrieval_settings': 'TEXT',
}
for col_name, col_type in new_columns.items():
if col_name not in columns:
await self.ap.persistence_mgr.execute_async(
sqlalchemy.text(f'ALTER TABLE knowledge_bases ADD COLUMN {col_name} {col_type};')
)
# For existing knowledge bases without knowledge_engine_plugin_id,
# set collection_id = uuid (same default as new KBs)
await self.ap.persistence_mgr.execute_async(
sqlalchemy.text('UPDATE knowledge_bases SET collection_id = uuid WHERE collection_id IS NULL;')
)
async def _migrate_top_k_to_retrieval_settings(self):
"""Migrate existing top_k values into retrieval_settings JSON."""
columns = await self._get_table_columns('knowledge_bases')
if 'top_k' not in columns:
return
result = await self.ap.persistence_mgr.execute_async(
sqlalchemy.text(
'SELECT uuid, top_k FROM knowledge_bases WHERE top_k IS NOT NULL AND retrieval_settings IS NULL;'
)
)
rows = result.fetchall()
for row in rows:
kb_uuid = row[0]
top_k = row[1]
retrieval_settings = json.dumps({'top_k': top_k})
await self.ap.persistence_mgr.execute_async(
sqlalchemy.text('UPDATE knowledge_bases SET retrieval_settings = :rs WHERE uuid = :uuid;').bindparams(
rs=retrieval_settings, uuid=kb_uuid
)
)
async def _migrate_embedding_model_uuid_to_creation_settings(self):
"""Migrate existing embedding_model_uuid into creation_settings JSON."""
columns = await self._get_table_columns('knowledge_bases')
if 'embedding_model_uuid' not in columns:
return
result = await self.ap.persistence_mgr.execute_async(
sqlalchemy.text(
'SELECT uuid, embedding_model_uuid, creation_settings FROM knowledge_bases '
"WHERE embedding_model_uuid IS NOT NULL AND embedding_model_uuid != '';"
)
)
rows = result.fetchall()
for row in rows:
kb_uuid = row[0]
emb_uuid = row[1]
existing_settings = row[2]
if existing_settings and isinstance(existing_settings, str):
try:
settings = json.loads(existing_settings)
except (json.JSONDecodeError, TypeError):
settings = {}
elif isinstance(existing_settings, dict):
settings = existing_settings
else:
settings = {}
if 'embedding_model_uuid' not in settings:
settings['embedding_model_uuid'] = emb_uuid
new_settings = json.dumps(settings)
await self.ap.persistence_mgr.execute_async(
sqlalchemy.text(
'UPDATE knowledge_bases SET creation_settings = :cs WHERE uuid = :uuid;'
).bindparams(cs=new_settings, uuid=kb_uuid)
)
async def _drop_old_columns(self):
"""Drop embedding_model_uuid and top_k columns (PostgreSQL only).
SQLite does not support DROP COLUMN in older versions, so we leave the
columns in place — the SQLAlchemy entity simply won't map them.
"""
if self.ap.persistence_mgr.db.name != 'postgresql':
return
columns = await self._get_table_columns('knowledge_bases')
if 'embedding_model_uuid' in columns:
await self.ap.persistence_mgr.execute_async(
sqlalchemy.text('ALTER TABLE knowledge_bases DROP COLUMN embedding_model_uuid;')
)
if 'top_k' in columns:
await self.ap.persistence_mgr.execute_async(
sqlalchemy.text('ALTER TABLE knowledge_bases DROP COLUMN top_k;')
)
async def _drop_external_knowledge_bases_table(self):
"""Drop the external_knowledge_bases table if it exists."""
if await self._table_exists('external_knowledge_bases'):
# Log existing external KBs before dropping, so users are aware of data loss
rows = await self.ap.persistence_mgr.execute_async(
sqlalchemy.text('SELECT * FROM external_knowledge_bases;')
)
existing = rows.fetchall()
if existing:
self.ap.logger.warning(
'Dropping external_knowledge_bases table with %d existing record(s). '
'These external KB configurations will be removed: %s',
len(existing),
[dict(row._mapping) for row in existing],
)
await self.ap.persistence_mgr.execute_async(sqlalchemy.text('DROP TABLE external_knowledge_bases;'))
async def downgrade(self):
"""Downgrade"""
pass

134
src/langbot/pkg/plugin/connector.py
View File

@@ -7,7 +7,6 @@ import typing
import os
import sys
import httpx
import traceback
import sqlalchemy
from async_lru import alru_cache
from langbot_plugin.api.entities.builtin.pipeline.query import provider_session
@@ -102,12 +101,6 @@ class PluginRuntimeConnector:
self.handler_task = asyncio.create_task(self.handler.run())
_ = await self.handler.ping()
self.ap.logger.info('Connected to plugin runtime.')
# Sync polymorphic component instances after connection
try:
await self.sync_polymorphic_component_instances()
except Exception as e:
traceback.print_exc()
self.ap.logger.error(f'Failed to sync polymorphic component instances: {e}')
await self.handler_task
task: asyncio.Task | None = None
@@ -463,30 +456,18 @@ class PluginRuntimeConnector:
yield cmd_ret
# KnowledgeRetriever methods
async def list_knowledge_retrievers(self, bound_plugins: list[str] | None = None) -> list[dict[str, Any]]:
"""List all available KnowledgeRetriever components."""
if not self.is_enable_plugin:
return []
retrievers_data = await self.handler.list_knowledge_retrievers(include_plugins=bound_plugins)
return retrievers_data
async def retrieve_knowledge(
self,
plugin_author: str,
plugin_name: str,
retriever_name: str,
instance_id: str,
retrieval_context: dict[str, Any],
) -> list[dict[str, Any]]:
"""Retrieve knowledge using a KnowledgeRetriever instance."""
) -> dict[str, Any]:
"""Retrieve knowledge using a KnowledgeEngine instance."""
if not self.is_enable_plugin:
return []
return {'results': []}
return await self.handler.retrieve_knowledge(
plugin_author, plugin_name, retriever_name, instance_id, retrieval_context
)
return await self.handler.retrieve_knowledge(plugin_author, plugin_name, retriever_name, retrieval_context)
def dispose(self):
# No need to consider the shutdown on Windows
@@ -500,41 +481,84 @@ class PluginRuntimeConnector:
self.heartbeat_task.cancel()
self.heartbeat_task = None
async def sync_polymorphic_component_instances(self) -> dict[str, Any]:
"""Sync polymorphic component instances with runtime.
@staticmethod
def _parse_plugin_id(plugin_id: str) -> tuple[str, str]:
"""Parse a plugin ID string into (author, name).
This collects all external knowledge bases from database and sends to runtime
to ensure instance integrity across restarts.
Args:
plugin_id: Plugin ID in 'author/name' format.
Returns:
Tuple of (plugin_author, plugin_name).
Raises:
ValueError: If plugin_id is not in the expected 'author/name' format.
"""
if '/' not in plugin_id:
raise ValueError(
f"Invalid plugin_id format: '{plugin_id}'. Expected 'author/name' format (e.g. 'langbot/rag-engine')."
)
return plugin_id.split('/', 1)
async def call_rag_ingest(self, plugin_id: str, context_data: dict[str, Any]) -> dict[str, Any]:
"""Call plugin to ingest document.
Args:
plugin_id: Target plugin ID (author/name).
context_data: IngestionContext data.
"""
plugin_author, plugin_name = self._parse_plugin_id(plugin_id)
return await self.handler.rag_ingest_document(plugin_author, plugin_name, context_data)
async def call_rag_delete_document(self, plugin_id: str, document_id: str, kb_id: str) -> bool:
plugin_author, plugin_name = self._parse_plugin_id(plugin_id)
return await self.handler.rag_delete_document(plugin_author, plugin_name, document_id, kb_id)
async def get_rag_creation_schema(self, plugin_id: str) -> dict[str, Any]:
plugin_author, plugin_name = self._parse_plugin_id(plugin_id)
return await self.handler.get_rag_creation_schema(plugin_author, plugin_name)
async def get_rag_retrieval_schema(self, plugin_id: str) -> dict[str, Any]:
plugin_author, plugin_name = self._parse_plugin_id(plugin_id)
return await self.handler.get_rag_retrieval_schema(plugin_author, plugin_name)
async def rag_on_kb_create(self, plugin_id: str, kb_id: str, config: dict[str, Any]) -> dict[str, Any]:
"""Notify plugin about KB creation."""
plugin_author, plugin_name = self._parse_plugin_id(plugin_id)
return await self.handler.rag_on_kb_create(plugin_author, plugin_name, kb_id, config)
async def rag_on_kb_delete(self, plugin_id: str, kb_id: str) -> dict[str, Any]:
"""Notify plugin about KB deletion."""
plugin_author, plugin_name = self._parse_plugin_id(plugin_id)
return await self.handler.rag_on_kb_delete(plugin_author, plugin_name, kb_id)
async def call_rag_retrieve(self, plugin_id: str, retrieval_context: dict[str, Any]) -> dict[str, Any]:
"""Call plugin to retrieve knowledge.
Args:
plugin_id: Target plugin ID (author/name).
retrieval_context: RetrievalContext data.
"""
plugin_author, plugin_name = self._parse_plugin_id(plugin_id)
return await self.handler.retrieve_knowledge(plugin_author, plugin_name, '', retrieval_context)
async def list_knowledge_engines(self) -> list[dict[str, Any]]:
"""List all available Knowledge Engines from plugins.
Returns a list of Knowledge Engines with their capabilities and configuration schemas.
"""
if not self.is_enable_plugin:
return {}
return []
# ===== external knowledge bases =====
return await self.handler.list_knowledge_engines()
external_kbs = await self.ap.external_kb_service.get_external_knowledge_bases()
async def list_parsers(self) -> list[dict[str, Any]]:
"""List all available parsers from plugins."""
if not self.is_enable_plugin:
return []
return await self.handler.list_parsers()
# Build required_instances list
required_instances = []
for kb in external_kbs:
required_instances.append(
{
'instance_id': kb['uuid'],
'plugin_author': kb['plugin_author'],
'plugin_name': kb['plugin_name'],
'component_kind': 'KnowledgeRetriever',
'component_name': kb['retriever_name'],
'config': kb['retriever_config'],
}
)
self.ap.logger.info(f'Syncing {len(required_instances)} polymorphic component instances to runtime')
# Send to runtime
sync_result = await self.handler.sync_polymorphic_component_instances(required_instances)
self.ap.logger.info(
f'Sync complete: {len(sync_result.get("success_instances", []))} succeeded, '
f'{len(sync_result.get("failed_instances", []))} failed'
)
return sync_result
async def call_parser(self, plugin_id: str, context_data: dict[str, Any], file_bytes: bytes) -> dict[str, Any]:
"""Call plugin to parse a document."""
plugin_author, plugin_name = self._parse_plugin_id(plugin_id)
return await self.handler.parse_document(plugin_author, plugin_name, context_data, file_bytes)

250
src/langbot/pkg/plugin/handler.py
View File

@@ -26,6 +26,20 @@ from ..core import app
from ..utils import constants
def _make_rag_error_response(error: Exception, error_type: str, **extra_context) -> handler.ActionResponse:
"""Create a clean error response for RAG operations.
Args:
error: The caught exception.
error_type: A category string like 'EmbeddingError', 'VectorStoreError'.
**extra_context: Additional context fields for the error message.
"""
context_parts = [f'{k}={v}' for k, v in extra_context.items()]
context_str = f' [{", ".join(context_parts)}]' if context_parts else ''
message = f'[{error_type}/{type(error).__name__}]{context_str} {str(error)}'
return handler.ActionResponse.error(message=message)
class RuntimeConnectionHandler(handler.Handler):
"""Runtime connection handler"""
@@ -439,7 +453,7 @@ class RuntimeConnectionHandler(handler.Handler):
},
)
@self.action(RuntimeToLangBotAction.GET_CONFIG_FILE)
@self.action(PluginToRuntimeAction.GET_CONFIG_FILE)
async def get_config_file(data: dict[str, Any]) -> handler.ActionResponse:
"""Get a config file by file key"""
file_key = data['file_key']
@@ -458,6 +472,125 @@ class RuntimeConnectionHandler(handler.Handler):
message=f'Failed to load config file {file_key}: {e}',
)
# ================= RAG Capability Handlers =================
@self.action(PluginToRuntimeAction.INVOKE_EMBEDDING)
async def invoke_embedding(data: dict[str, Any]) -> handler.ActionResponse:
embedding_model_uuid = data['embedding_model_uuid']
texts = data['texts']
embedding_model = await self.ap.model_mgr.get_embedding_model_by_uuid(embedding_model_uuid)
if embedding_model is None:
return handler.ActionResponse.error(
message=f'Embedding model with embedding_model_uuid {embedding_model_uuid} not found',
)
try:
vectors = await embedding_model.provider.invoke_embedding(embedding_model, texts)
return handler.ActionResponse.success(data={'vectors': vectors})
except Exception as e:
return _make_rag_error_response(e, 'EmbeddingError', embedding_model_uuid=embedding_model_uuid)
@self.action(PluginToRuntimeAction.VECTOR_UPSERT)
async def vector_upsert(data: dict[str, Any]) -> handler.ActionResponse:
collection_id = data['collection_id']
vectors = data['vectors']
ids = data['ids']
metadata = data.get('metadata')
documents = data.get('documents')
if len(vectors) != len(ids):
return handler.ActionResponse.error(message='vectors and ids must have same length')
if metadata and len(metadata) != len(vectors):
return handler.ActionResponse.error(message='metadata must match vectors length')
if documents and len(documents) != len(vectors):
return handler.ActionResponse.error(message='documents must match vectors length')
try:
await self.ap.rag_runtime_service.vector_upsert(
collection_id,
vectors,
ids,
metadata,
documents,
)
return handler.ActionResponse.success(data={})
except Exception as e:
return _make_rag_error_response(e, 'VectorStoreError', collection_id=collection_id)
@self.action(PluginToRuntimeAction.VECTOR_SEARCH)
async def vector_search(data: dict[str, Any]) -> handler.ActionResponse:
collection_id = data['collection_id']
query_vector = data['query_vector']
top_k = data['top_k']
filters = data.get('filters')
search_type = data.get('search_type', 'vector')
query_text = data.get('query_text', '')
try:
results = await self.ap.rag_runtime_service.vector_search(
collection_id,
query_vector,
top_k,
filters,
search_type,
query_text,
)
return handler.ActionResponse.success(data={'results': results})
except Exception as e:
return _make_rag_error_response(e, 'VectorStoreError', collection_id=collection_id)
@self.action(PluginToRuntimeAction.VECTOR_DELETE)
async def vector_delete(data: dict[str, Any]) -> handler.ActionResponse:
collection_id = data['collection_id']
file_ids = data.get('file_ids')
filters = data.get('filters')
try:
count = await self.ap.rag_runtime_service.vector_delete(collection_id, file_ids, filters)
return handler.ActionResponse.success(data={'count': count})
except Exception as e:
return _make_rag_error_response(e, 'VectorStoreError', collection_id=collection_id)
@self.action(PluginToRuntimeAction.GET_KNOWLEDEGE_FILE_STREAM)
async def get_knowledge_file_stream(data: dict[str, Any]) -> handler.ActionResponse:
storage_path = data['storage_path']
try:
content_bytes = await self.ap.rag_runtime_service.get_file_stream(storage_path)
file_key = await self.send_file(content_bytes, '')
return handler.ActionResponse.success(data={'file_key': file_key})
except Exception as e:
return _make_rag_error_response(e, 'FileServiceError', storage_path=storage_path)
@self.action(PluginToRuntimeAction.INVOKE_PARSER)
async def invoke_parser(data: dict[str, Any]) -> handler.ActionResponse:
"""Plugin requests host to invoke a parser plugin."""
plugin_author = data['plugin_author']
plugin_name = data['plugin_name']
storage_path = data['storage_path']
mime_type = data.get('mime_type', 'application/octet-stream')
filename = data.get('filename', '')
metadata = data.get('metadata', {})
try:
# Read file from storage
file_bytes = await self.ap.rag_runtime_service.get_file_stream(storage_path)
context_data = {
'mime_type': mime_type,
'filename': filename,
'metadata': metadata,
}
result = await self.ap.plugin_connector.call_parser(
f'{plugin_author}/{plugin_name}', context_data, file_bytes
)
return handler.ActionResponse.success(data=result)
except Exception as e:
return _make_rag_error_response(e, 'ParserError')
@self.action(CommonAction.PING)
async def ping(data: dict[str, Any]) -> handler.ActionResponse:
"""Ping"""
return handler.ActionResponse.success(
data={
'pong': 'pong',
},
)
async def ping(self) -> dict[str, Any]:
"""Ping the runtime"""
return await self.call_action(
@@ -717,26 +850,13 @@ class RuntimeConnectionHandler(handler.Handler):
async for ret in gen:
yield ret
# KnowledgeRetriever methods
async def list_knowledge_retrievers(self, include_plugins: list[str] | None = None) -> list[dict[str, Any]]:
"""List knowledge retrievers"""
result = await self.call_action(
LangBotToRuntimeAction.LIST_KNOWLEDGE_RETRIEVERS,
{
'include_plugins': include_plugins,
},
timeout=10,
)
return result['retrievers']
async def retrieve_knowledge(
self,
plugin_author: str,
plugin_name: str,
retriever_name: str,
instance_id: str,
retrieval_context: dict[str, Any],
) -> list[dict[str, Any]]:
) -> dict[str, Any]:
"""Retrieve knowledge"""
result = await self.call_action(
LangBotToRuntimeAction.RETRIEVE_KNOWLEDGE,
@@ -744,22 +864,10 @@ class RuntimeConnectionHandler(handler.Handler):
'plugin_author': plugin_author,
'plugin_name': plugin_name,
'retriever_name': retriever_name,
'instance_id': instance_id,
'retrieval_context': retrieval_context,
},
timeout=30,
)
return result['retrieval_results']
async def sync_polymorphic_component_instances(self, required_instances: list[dict[str, Any]]) -> dict[str, Any]:
"""Sync polymorphic component instances with runtime"""
result = await self.call_action(
LangBotToRuntimeAction.SYNC_POLYMORPHIC_COMPONENT_INSTANCES,
{
'required_instances': required_instances,
},
timeout=30,
)
return result
async def get_debug_info(self) -> dict[str, Any]:
@@ -770,3 +878,91 @@ class RuntimeConnectionHandler(handler.Handler):
timeout=10,
)
return result
# ================= RAG Capability Callers (LangBot -> Runtime) =================
async def rag_ingest_document(
self, plugin_author: str, plugin_name: str, context_data: dict[str, Any]
) -> dict[str, Any]:
"""Send INGEST_DOCUMENT action to runtime."""
result = await self.call_action(
LangBotToRuntimeAction.RAG_INGEST_DOCUMENT,
{'plugin_author': plugin_author, 'plugin_name': plugin_name, 'context': context_data},
timeout=300, # Ingestion can be slow
)
return result
async def rag_delete_document(self, plugin_author: str, plugin_name: str, document_id: str, kb_id: str) -> bool:
result = await self.call_action(
LangBotToRuntimeAction.RAG_DELETE_DOCUMENT,
{'plugin_author': plugin_author, 'plugin_name': plugin_name, 'document_id': document_id, 'kb_id': kb_id},
timeout=30,
)
return result.get('success', False)
async def rag_on_kb_create(
self, plugin_author: str, plugin_name: str, kb_id: str, config: dict[str, Any]
) -> dict[str, Any]:
"""Notify plugin about KB creation."""
result = await self.call_action(
LangBotToRuntimeAction.RAG_ON_KB_CREATE,
{'plugin_author': plugin_author, 'plugin_name': plugin_name, 'kb_id': kb_id, 'config': config},
timeout=30,
)
return result
async def rag_on_kb_delete(self, plugin_author: str, plugin_name: str, kb_id: str) -> dict[str, Any]:
"""Notify plugin about KB deletion."""
result = await self.call_action(
LangBotToRuntimeAction.RAG_ON_KB_DELETE,
{'plugin_author': plugin_author, 'plugin_name': plugin_name, 'kb_id': kb_id},
timeout=30,
)
return result
async def get_rag_creation_schema(self, plugin_author: str, plugin_name: str) -> dict[str, Any]:
return await self.call_action(
LangBotToRuntimeAction.GET_RAG_CREATION_SETTINGS_SCHEMA,
{'plugin_author': plugin_author, 'plugin_name': plugin_name},
timeout=10,
)
async def get_rag_retrieval_schema(self, plugin_author: str, plugin_name: str) -> dict[str, Any]:
return await self.call_action(
LangBotToRuntimeAction.GET_RAG_RETRIEVAL_SETTINGS_SCHEMA,
{'plugin_author': plugin_author, 'plugin_name': plugin_name},
timeout=10,
)
async def list_knowledge_engines(self) -> list[dict[str, Any]]:
"""List all available Knowledge Engines from plugins."""
result = await self.call_action(LangBotToRuntimeAction.LIST_KNOWLEDGE_ENGINES, {}, timeout=60)
return result.get('engines', [])
# ================= Parser Capability Callers (LangBot -> Runtime) =================
async def list_parsers(self) -> list[dict[str, Any]]:
"""List all available parsers from plugins."""
result = await self.call_action(LangBotToRuntimeAction.LIST_PARSERS, {}, timeout=60)
return result.get('parsers', [])
async def parse_document(
self, plugin_author: str, plugin_name: str, context_data: dict[str, Any], file_bytes: bytes
) -> dict[str, Any]:
"""Send PARSE_DOCUMENT action to runtime.
Sends file content via chunked FILE_CHUNK transfer, then invokes
the PARSE_DOCUMENT action with a file_key reference.
"""
# Send file to runtime via chunked transfer
file_key = await self.send_file(file_bytes, '')
# Include file_key in context_data for the runtime to read
context_data['file_key'] = file_key
result = await self.call_action(
LangBotToRuntimeAction.PARSE_DOCUMENT,
{'plugin_author': plugin_author, 'plugin_name': plugin_name, 'context': context_data},
timeout=300,
)
return result

14
src/langbot/pkg/provider/runners/localagent.py
View File

@@ -74,15 +74,7 @@ class LocalAgentRunner(runner.RequestRunner):
self.ap.logger.warning(f'Knowledge base {kb_uuid} not found, skipping')
continue
# Get top_k based on KB type
if kb.get_type() == 'internal':
top_k = kb.knowledge_base_entity.top_k
elif kb.get_type() == 'external':
top_k = 5 # external kb's top_k is managed by plugin config
else:
top_k = 5 # default fallback
result = await kb.retrieve(user_message_text, top_k)
result = await kb.retrieve(user_message_text)
if result:
all_results.extend(result)
@@ -97,9 +89,9 @@ class LocalAgentRunner(runner.RequestRunner):
if content.type == 'text' and content.text is not None:
texts.append(f'[{idx}] {content.text}')
idx += 1
rag_context = '\n\n'.join(texts)
rag_context_text = '\n\n'.join(texts)
final_user_message_text = rag_combined_prompt_template.format(
rag_context=rag_context, user_message=user_message_text
rag_context=rag_context_text, user_message=user_message_text
)
else:

8
src/langbot/pkg/rag/knowledge/base.py
View File

@@ -22,12 +22,12 @@ class KnowledgeBaseInterface(metaclass=abc.ABCMeta):
pass
@abc.abstractmethod
async def retrieve(self, query: str, top_k: int) -> list[rag_context.RetrievalResultEntry]:
async def retrieve(self, query: str, settings: dict | None = None) -> list[rag_context.RetrievalResultEntry]:
"""Retrieve relevant documents from the knowledge base
Args:
query: The query string
top_k: Number of top results to return
settings: Optional per-request retrieval settings overrides
Returns:
List of retrieve result entries
@@ -45,8 +45,8 @@ class KnowledgeBaseInterface(metaclass=abc.ABCMeta):
pass
@abc.abstractmethod
def get_type(self) -> str:
"""Get the type of knowledge base (internal/external)"""
def get_knowledge_engine_plugin_id(self) -> str:
"""Get the Knowledge Engine plugin ID"""
pass
@abc.abstractmethod

85
src/langbot/pkg/rag/knowledge/external.py
View File

@@ -1,85 +0,0 @@
"""External knowledge base implementation"""
from __future__ import annotations
from langbot.pkg.core import app
from langbot.pkg.entity.persistence import rag as persistence_rag
from langbot_plugin.api.entities.builtin.rag import context as rag_context
from .base import KnowledgeBaseInterface
class ExternalKnowledgeBase(KnowledgeBaseInterface):
"""External knowledge base that queries via HTTP API or plugin retriever"""
external_kb_entity: persistence_rag.ExternalKnowledgeBase
# Plugin retriever instance ID
retriever_instance_id: str | None
def __init__(self, ap: app.Application, external_kb_entity: persistence_rag.ExternalKnowledgeBase):
super().__init__(ap)
self.external_kb_entity = external_kb_entity
self.retriever_instance_id = None
async def initialize(self):
"""Initialize the external knowledge base"""
# Use KB UUID as instance ID
# Instance creation is now handled by the unified sync mechanism
# when LangBot connects to runtime
self.retriever_instance_id = self.external_kb_entity.uuid
self.ap.logger.info(
f'Initialized external KB {self.external_kb_entity.uuid}, instance will be created by sync mechanism'
)
async def retrieve(self, query: str, top_k: int = 5) -> list[rag_context.RetrievalResultEntry]:
"""Retrieve documents from external knowledge base via plugin retriever"""
if not self.retriever_instance_id:
self.ap.logger.error(f'No retriever instance for KB {self.external_kb_entity.uuid}')
return []
try:
results = await self.ap.plugin_connector.retrieve_knowledge(
self.external_kb_entity.plugin_author,
self.external_kb_entity.plugin_name,
self.external_kb_entity.retriever_name,
self.retriever_instance_id,
{'query': query},
)
# Convert plugin results to RetrievalResultEntry
retrieval_entries = []
for result in results:
retrieval_entries.append(rag_context.RetrievalResultEntry(**result))
return retrieval_entries
except Exception as e:
self.ap.logger.error(f'Plugin retriever error: {e}')
import traceback
traceback.print_exc()
return []
def get_uuid(self) -> str:
"""Get the UUID of the external knowledge base"""
return self.external_kb_entity.uuid
def get_name(self) -> str:
"""Get the name of the external knowledge base"""
return self.external_kb_entity.name
def get_type(self) -> str:
"""Get the type of knowledge base"""
return 'external'
async def dispose(self):
"""Clean up resources"""
# Trigger sync to immediately delete the instance from plugin process
# This ensures instance is cleaned up without waiting for next LangBot restart
try:
await self.ap.plugin_connector.sync_polymorphic_component_instances()
self.ap.logger.info(
f'Disposed external KB {self.external_kb_entity.uuid}, triggered sync to delete instance'
)
except Exception as e:
self.ap.logger.error(f'Failed to sync after disposing KB: {e}')

487
src/langbot/pkg/rag/knowledge/kbmgr.py
View File

@@ -1,18 +1,19 @@
from __future__ import annotations
import mimetypes
import os.path
import traceback
import uuid
import zipfile
import io
from .services import parser, chunker
from typing import Any
from langbot.pkg.core import app
from langbot.pkg.rag.knowledge.services.embedder import Embedder
from langbot.pkg.rag.knowledge.services.retriever import Retriever
import sqlalchemy
from langbot.pkg.entity.persistence import rag as persistence_rag
from langbot.pkg.core import taskmgr
from langbot_plugin.api.entities.builtin.rag import context as rag_context
from .base import KnowledgeBaseInterface
from .external import ExternalKnowledgeBase
class RuntimeKnowledgeBase(KnowledgeBaseInterface):
@@ -20,28 +21,16 @@ class RuntimeKnowledgeBase(KnowledgeBaseInterface):
knowledge_base_entity: persistence_rag.KnowledgeBase
parser: parser.FileParser
chunker: chunker.Chunker
embedder: Embedder
retriever: Retriever
def __init__(self, ap: app.Application, knowledge_base_entity: persistence_rag.KnowledgeBase):
super().__init__(ap)
self.knowledge_base_entity = knowledge_base_entity
self.parser = parser.FileParser(ap=self.ap)
self.chunker = chunker.Chunker(ap=self.ap)
self.embedder = Embedder(ap=self.ap)
self.retriever = Retriever(ap=self.ap)
# 传递kb_id给retriever
self.retriever.kb_id = knowledge_base_entity.uuid
async def initialize(self):
pass
async def _store_file_task(self, file: persistence_rag.File, task_context: taskmgr.TaskContext):
async def _store_file_task(
self, file: persistence_rag.File, task_context: taskmgr.TaskContext, parser_plugin_id: str | None = None
):
try:
# set file status to processing
await self.ap.persistence_mgr.execute_async(
@@ -50,31 +39,46 @@ class RuntimeKnowledgeBase(KnowledgeBaseInterface):
.values(status='processing')
)
task_context.set_current_action('Parsing file')
# parse file
text = await self.parser.parse(file.file_name, file.extension)
if not text:
raise Exception(f'No text extracted from file {file.file_name}')
task_context.set_current_action('Processing file')
task_context.set_current_action('Chunking file')
# chunk file
chunks_texts = await self.chunker.chunk(text)
if not chunks_texts:
raise Exception(f'No chunks extracted from file {file.file_name}')
# Get file size from storage
file_size = await self.ap.storage_mgr.storage_provider.size(file.file_name)
task_context.set_current_action('Embedding chunks')
# Detect MIME type from extension
mime_type, _ = mimetypes.guess_type(file.file_name)
if mime_type is None:
mime_type = 'application/octet-stream'
embedding_model = await self.ap.model_mgr.get_embedding_model_by_uuid(
self.knowledge_base_entity.embedding_model_uuid
)
# embed chunks
await self.embedder.embed_and_store(
kb_id=self.knowledge_base_entity.uuid,
file_id=file.uuid,
chunks=chunks_texts,
embedding_model=embedding_model,
# If a parser plugin is specified, call it before ingestion
parsed_content = None
if parser_plugin_id:
task_context.set_current_action('Parsing file')
file_bytes = await self.ap.storage_mgr.storage_provider.load(file.file_name)
parse_context = {
'mime_type': mime_type,
'filename': file.file_name,
'metadata': {},
}
parsed_content = await self.ap.plugin_connector.call_parser(parser_plugin_id, parse_context, file_bytes)
# Call plugin to ingest document
result = await self._ingest_document(
{
'document_id': file.uuid,
'filename': file.file_name,
'extension': file.extension,
'file_size': file_size,
'mime_type': mime_type,
},
file.file_name, # storage path
parsed_content=parsed_content,
)
# Check plugin result status
if result.get('status') == 'failed':
error_msg = result.get('error_message', 'Plugin ingestion returned failed status')
raise Exception(error_msg)
# set file status to completed
await self.ap.persistence_mgr.execute_async(
sqlalchemy.update(persistence_rag.File)
@@ -97,16 +101,17 @@ class RuntimeKnowledgeBase(KnowledgeBaseInterface):
# delete file from storage
await self.ap.storage_mgr.storage_provider.delete(file.file_name)
async def store_file(self, file_id: str) -> str:
async def store_file(self, file_id: str, parser_plugin_id: str | None = None) -> str:
# pre checking
if not await self.ap.storage_mgr.storage_provider.exists(file_id):
raise Exception(f'File {file_id} not found')
file_name = file_id
extension = file_name.split('.')[-1].lower()
_, ext = os.path.splitext(file_name)
extension = ext.lstrip('.').lower() if ext else ''
if extension == 'zip':
return await self._store_zip_file(file_id)
return await self._store_zip_file(file_id, parser_plugin_id=parser_plugin_id)
file_uuid = str(uuid.uuid4())
kb_id = self.knowledge_base_entity.uuid
@@ -126,7 +131,7 @@ class RuntimeKnowledgeBase(KnowledgeBaseInterface):
# run background task asynchronously
ctx = taskmgr.TaskContext.new()
wrapper = self.ap.task_mgr.create_user_task(
self._store_file_task(file_obj, task_context=ctx),
self._store_file_task(file_obj, task_context=ctx, parser_plugin_id=parser_plugin_id),
kind='knowledge-operation',
name=f'knowledge-store-file-{file_id}',
label=f'Store file {file_id}',
@@ -134,7 +139,7 @@ class RuntimeKnowledgeBase(KnowledgeBaseInterface):
)
return wrapper.id
async def _store_zip_file(self, zip_file_id: str) -> str:
async def _store_zip_file(self, zip_file_id: str, parser_plugin_id: str | None = None) -> str:
"""Handle ZIP file by extracting each document and storing them separately."""
self.ap.logger.info(f'Processing ZIP file: {zip_file_id}')
@@ -150,7 +155,8 @@ class RuntimeKnowledgeBase(KnowledgeBaseInterface):
if file_info.is_dir() or file_info.filename.startswith('.'):
continue
file_extension = file_info.filename.split('.')[-1].lower()
_, file_ext = os.path.splitext(file_info.filename)
file_extension = file_ext.lstrip('.').lower()
if file_extension not in supported_extensions:
self.ap.logger.debug(f'Skipping unsupported file in ZIP: {file_info.filename}')
continue
@@ -159,18 +165,18 @@ class RuntimeKnowledgeBase(KnowledgeBaseInterface):
file_content = zip_ref.read(file_info.filename)
base_name = file_info.filename.replace('/', '_').replace('\\', '_')
extension = base_name.split('.')[-1]
file_name = base_name.split('.')[0]
file_stem, file_ext = os.path.splitext(base_name)
extension = file_ext.lstrip('.')
if file_name.startswith('__MACOSX'):
if file_stem.startswith('__MACOSX'):
continue
extracted_file_id = file_name + '_' + str(uuid.uuid4())[:8] + '.' + extension
extracted_file_id = file_stem + '_' + str(uuid.uuid4())[:8] + '.' + extension
# save file to storage
await self.ap.storage_mgr.storage_provider.save(extracted_file_id, file_content)
task_id = await self.store_file(extracted_file_id)
task_id = await self.store_file(extracted_file_id, parser_plugin_id=parser_plugin_id)
stored_file_tasks.append(task_id)
self.ap.logger.info(
@@ -189,21 +195,28 @@ class RuntimeKnowledgeBase(KnowledgeBaseInterface):
return stored_file_tasks[0] if stored_file_tasks else ''
async def retrieve(self, query: str, top_k: int) -> list[rag_context.RetrievalResultEntry]:
embedding_model = await self.ap.model_mgr.get_embedding_model_by_uuid(
self.knowledge_base_entity.embedding_model_uuid
)
return await self.retriever.retrieve(self.knowledge_base_entity.uuid, query, embedding_model, top_k)
async def retrieve(self, query: str, settings: dict | None = None) -> list[rag_context.RetrievalResultEntry]:
# Merge stored retrieval_settings with per-request overrides
stored = self.knowledge_base_entity.retrieval_settings or {}
merged = {**stored, **(settings or {})}
if 'top_k' not in merged:
merged['top_k'] = 5 # fallback default
response = await self._retrieve(query, merged)
results_data = response.get('results', [])
entries = []
for r in results_data:
if isinstance(r, dict):
entries.append(rag_context.RetrievalResultEntry(**r))
elif isinstance(r, rag_context.RetrievalResultEntry):
entries.append(r)
return entries
async def delete_file(self, file_id: str):
# delete vector
await self.ap.vector_db_mgr.vector_db.delete_by_file_id(self.knowledge_base_entity.uuid, file_id)
# delete chunk
await self.ap.persistence_mgr.execute_async(
sqlalchemy.delete(persistence_rag.Chunk).where(persistence_rag.Chunk.file_id == file_id)
)
await self._delete_document(file_id)
# Also cleanup DB record
await self.ap.persistence_mgr.execute_async(
sqlalchemy.delete(persistence_rag.File).where(persistence_rag.File.uuid == file_id)
)
@@ -216,32 +229,289 @@ class RuntimeKnowledgeBase(KnowledgeBaseInterface):
"""Get the name of the knowledge base"""
return self.knowledge_base_entity.name
def get_type(self) -> str:
"""Get the type of knowledge base"""
return 'internal'
def get_knowledge_engine_plugin_id(self) -> str:
"""Get the Knowledge Engine plugin ID"""
return self.knowledge_base_entity.knowledge_engine_plugin_id or ''
async def dispose(self):
await self.ap.vector_db_mgr.vector_db.delete_collection(self.knowledge_base_entity.uuid)
"""Dispose the knowledge base, notifying the plugin to cleanup."""
await self._on_kb_delete()
# ========== Plugin Communication Methods ==========
async def _on_kb_create(self) -> None:
"""Notify plugin about KB creation."""
plugin_id = self.knowledge_base_entity.knowledge_engine_plugin_id
if not plugin_id:
return
try:
config = self.knowledge_base_entity.creation_settings or {}
self.ap.logger.info(
f'Calling RAG plugin {plugin_id}: on_knowledge_base_create(kb_id={self.knowledge_base_entity.uuid})'
)
await self.ap.plugin_connector.rag_on_kb_create(plugin_id, self.knowledge_base_entity.uuid, config)
except Exception as e:
self.ap.logger.error(f'Failed to notify plugin {plugin_id} on KB create: {e}')
raise
async def _on_kb_delete(self) -> None:
"""Notify plugin about KB deletion."""
plugin_id = self.knowledge_base_entity.knowledge_engine_plugin_id
if not plugin_id:
return
try:
self.ap.logger.info(
f'Calling RAG plugin {plugin_id}: on_knowledge_base_delete(kb_id={self.knowledge_base_entity.uuid})'
)
await self.ap.plugin_connector.rag_on_kb_delete(plugin_id, self.knowledge_base_entity.uuid)
except Exception as e:
self.ap.logger.error(f'Failed to notify plugin {plugin_id} on KB delete: {e}')
async def _ingest_document(
self,
file_metadata: dict[str, Any],
storage_path: str,
parsed_content: dict[str, Any] | None = None,
) -> dict[str, Any]:
"""Call plugin to ingest document."""
kb = self.knowledge_base_entity
plugin_id = kb.knowledge_engine_plugin_id
if not plugin_id:
self.ap.logger.error(f'No RAG plugin ID configured for KB {kb.uuid}. Ingestion failed.')
raise ValueError('RAG Plugin ID required')
self.ap.logger.info(f'Calling RAG plugin {plugin_id}: ingest(doc={file_metadata.get("filename")})')
# Inject knowledge_base_id into file metadata as required by SDK schema
file_metadata['knowledge_base_id'] = kb.uuid
context_data = {
'file_object': {
'metadata': file_metadata,
'storage_path': storage_path,
},
'knowledge_base_id': kb.uuid,
'collection_id': kb.collection_id or kb.uuid,
'creation_settings': kb.creation_settings or {},
'parsed_content': parsed_content,
}
try:
result = await self.ap.plugin_connector.call_rag_ingest(plugin_id, context_data)
return result
except Exception as e:
self.ap.logger.error(f'Plugin ingestion failed: {e}')
raise
async def _retrieve(
self,
query: str,
settings: dict[str, Any],
) -> dict[str, Any]:
"""Call plugin to retrieve documents.
Raises:
ValueError: If no RAG plugin is configured for this KB.
Exception: If the plugin retrieval call fails.
"""
kb = self.knowledge_base_entity
plugin_id = kb.knowledge_engine_plugin_id
if not plugin_id:
raise ValueError(f'No RAG plugin ID configured for KB {kb.uuid}. Retrieval failed.')
retrieval_context = {
'query': query,
'knowledge_base_id': kb.uuid,
'collection_id': kb.collection_id or kb.uuid,
'retrieval_settings': settings,
'creation_settings': kb.creation_settings or {},
'filters': settings.pop('filters', {}),
}
result = await self.ap.plugin_connector.call_rag_retrieve(
plugin_id,
retrieval_context,
)
return result
async def _delete_document(self, document_id: str) -> bool:
"""Call plugin to delete document."""
kb = self.knowledge_base_entity
plugin_id = kb.knowledge_engine_plugin_id
if not plugin_id:
return False
self.ap.logger.info(f'Calling RAG plugin {plugin_id}: delete_document(doc_id={document_id})')
try:
return await self.ap.plugin_connector.call_rag_delete_document(plugin_id, document_id, kb.uuid)
except Exception as e:
self.ap.logger.error(f'Plugin document deletion failed: {e}')
return False
class RAGManager:
ap: app.Application
knowledge_bases: list[KnowledgeBaseInterface]
knowledge_bases: dict[str, KnowledgeBaseInterface]
def __init__(self, ap: app.Application):
self.ap = ap
self.knowledge_bases = []
self.knowledge_bases = {}
async def initialize(self):
await self.load_knowledge_bases_from_db()
async def get_all_knowledge_base_details(self) -> list[dict]:
"""Get all knowledge bases with enriched Knowledge Engine details."""
# 1. Get raw KBs from DB
result = await self.ap.persistence_mgr.execute_async(sqlalchemy.select(persistence_rag.KnowledgeBase))
knowledge_bases = result.all()
# 2. Get all available Knowledge Engines for enrichment
engine_map = {}
if self.ap.plugin_connector.is_enable_plugin:
try:
engines = await self.ap.plugin_connector.list_knowledge_engines()
engine_map = {e['plugin_id']: e for e in engines}
except Exception as e:
self.ap.logger.warning(f'Failed to list Knowledge Engines: {e}')
# 3. Serialize and enrich
kb_list = []
for kb in knowledge_bases:
kb_dict = self.ap.persistence_mgr.serialize_model(persistence_rag.KnowledgeBase, kb)
self._enrich_kb_dict(kb_dict, engine_map)
kb_list.append(kb_dict)
return kb_list
async def get_knowledge_base_details(self, kb_uuid: str) -> dict | None:
"""Get specific knowledge base with enriched Knowledge Engine details."""
result = await self.ap.persistence_mgr.execute_async(
sqlalchemy.select(persistence_rag.KnowledgeBase).where(persistence_rag.KnowledgeBase.uuid == kb_uuid)
)
kb = result.first()
if not kb:
return None
kb_dict = self.ap.persistence_mgr.serialize_model(persistence_rag.KnowledgeBase, kb)
# Fetch engines
engine_map = {}
if self.ap.plugin_connector.is_enable_plugin:
try:
engines = await self.ap.plugin_connector.list_knowledge_engines()
engine_map = {e['plugin_id']: e for e in engines}
except Exception as e:
self.ap.logger.warning(f'Failed to list Knowledge Engines: {e}')
self._enrich_kb_dict(kb_dict, engine_map)
return kb_dict
@staticmethod
def _to_i18n_name(name) -> dict:
"""Ensure name is always an I18nObject-compatible dict.
If *name* is already a dict (with ``en_US`` / ``zh_Hans`` keys) it is
returned as-is. A plain string is wrapped into an I18nObject so the
frontend ``extractI18nObject`` helper never receives an unexpected type.
"""
if isinstance(name, dict):
return name
return {'en_US': str(name), 'zh_Hans': str(name)}
def _enrich_kb_dict(self, kb_dict: dict, engine_map: dict) -> None:
"""Helper to inject engine info into KB dict."""
plugin_id = kb_dict.get('knowledge_engine_plugin_id')
# Default fallback structure — name must be I18nObject for frontend compatibility
fallback_name = self._to_i18n_name(plugin_id or 'Internal (Legacy)')
fallback_info = {
'plugin_id': plugin_id,
'name': fallback_name,
'capabilities': [],
}
if not plugin_id:
kb_dict['knowledge_engine'] = fallback_info
return
engine_info = engine_map.get(plugin_id)
if engine_info:
kb_dict['knowledge_engine'] = {
'plugin_id': plugin_id,
'name': self._to_i18n_name(engine_info.get('name', plugin_id)),
'capabilities': engine_info.get('capabilities', []),
}
else:
kb_dict['knowledge_engine'] = fallback_info
async def create_knowledge_base(
self,
name: str,
knowledge_engine_plugin_id: str,
creation_settings: dict,
retrieval_settings: dict | None = None,
description: str = '',
) -> persistence_rag.KnowledgeBase:
"""Create a new knowledge base using a RAG plugin."""
# Validate that the Knowledge Engine plugin exists
if self.ap.plugin_connector.is_enable_plugin:
try:
engines = await self.ap.plugin_connector.list_knowledge_engines()
engine_ids = [e.get('plugin_id') for e in engines]
if knowledge_engine_plugin_id not in engine_ids:
raise ValueError(f'Knowledge Engine plugin {knowledge_engine_plugin_id} not found')
except ValueError:
raise
except Exception as e:
self.ap.logger.warning(f'Failed to validate Knowledge Engine plugin existence: {e}')
kb_uuid = str(uuid.uuid4())
# Use UUID as collection ID by default for isolation
collection_id = kb_uuid
kb_data = {
'uuid': kb_uuid,
'name': name,
'description': description,
'knowledge_engine_plugin_id': knowledge_engine_plugin_id,
'collection_id': collection_id,
'creation_settings': creation_settings,
'retrieval_settings': retrieval_settings or {},
}
# Create Entity
kb = persistence_rag.KnowledgeBase(**kb_data)
# Persist
await self.ap.persistence_mgr.execute_async(sqlalchemy.insert(persistence_rag.KnowledgeBase).values(kb_data))
# Load into Runtime
runtime_kb = await self.load_knowledge_base(kb)
# Notify Plugin — rollback DB record and runtime entry on failure
try:
await runtime_kb._on_kb_create()
except Exception:
self.knowledge_bases.pop(kb_uuid, None)
await self.ap.persistence_mgr.execute_async(
sqlalchemy.delete(persistence_rag.KnowledgeBase).where(persistence_rag.KnowledgeBase.uuid == kb_uuid)
)
raise
self.ap.logger.info(f'Created new Knowledge Base {name} ({kb_uuid}) using plugin {knowledge_engine_plugin_id}')
return kb
async def load_knowledge_bases_from_db(self):
self.ap.logger.info('Loading knowledge bases from db...')
self.knowledge_bases = []
self.knowledge_bases = {}
# Load internal knowledge bases
# Load knowledge bases
result = await self.ap.persistence_mgr.execute_async(sqlalchemy.select(persistence_rag.KnowledgeBase))
knowledge_bases = result.all()
@@ -253,86 +523,37 @@ class RAGManager:
f'Error loading knowledge base {knowledge_base.uuid}: {e}\n{traceback.format_exc()}'
)
# Load external knowledge bases
external_result = await self.ap.persistence_mgr.execute_async(
sqlalchemy.select(persistence_rag.ExternalKnowledgeBase)
)
external_kbs = external_result.all()
for external_kb in external_kbs:
try:
# Don't trigger sync during batch loading - will sync once after LangBot connects to runtime
await self.load_external_knowledge_base(external_kb, trigger_sync=False)
except Exception as e:
self.ap.logger.error(
f'Error loading external knowledge base {external_kb.uuid}: {e}\n{traceback.format_exc()}'
)
async def load_knowledge_base(
self,
knowledge_base_entity: persistence_rag.KnowledgeBase | sqlalchemy.Row | dict,
) -> RuntimeKnowledgeBase:
if isinstance(knowledge_base_entity, sqlalchemy.Row):
# Safe access to _mapping for SQLAlchemy 1.4+
knowledge_base_entity = persistence_rag.KnowledgeBase(**knowledge_base_entity._mapping)
elif isinstance(knowledge_base_entity, dict):
knowledge_base_entity = persistence_rag.KnowledgeBase(**knowledge_base_entity)
# Filter out non-database fields (like knowledge_engine which is computed)
filtered_dict = {
k: v for k, v in knowledge_base_entity.items() if k in persistence_rag.KnowledgeBase.ALL_DB_FIELDS
}
knowledge_base_entity = persistence_rag.KnowledgeBase(**filtered_dict)
runtime_knowledge_base = RuntimeKnowledgeBase(ap=self.ap, knowledge_base_entity=knowledge_base_entity)
await runtime_knowledge_base.initialize()
self.knowledge_bases.append(runtime_knowledge_base)
self.knowledge_bases[runtime_knowledge_base.get_uuid()] = runtime_knowledge_base
return runtime_knowledge_base
async def load_external_knowledge_base(
self,
external_kb_entity: persistence_rag.ExternalKnowledgeBase | sqlalchemy.Row | dict,
trigger_sync: bool = True,
) -> ExternalKnowledgeBase:
"""Load external knowledge base into runtime
Args:
external_kb_entity: External KB entity to load
trigger_sync: Whether to trigger sync after loading (default True for manual creation, False for batch loading)
"""
if isinstance(external_kb_entity, sqlalchemy.Row):
external_kb_entity = persistence_rag.ExternalKnowledgeBase(**external_kb_entity._mapping)
elif isinstance(external_kb_entity, dict):
external_kb_entity = persistence_rag.ExternalKnowledgeBase(**external_kb_entity)
external_kb = ExternalKnowledgeBase(ap=self.ap, external_kb_entity=external_kb_entity)
await external_kb.initialize()
self.knowledge_bases.append(external_kb)
# Trigger sync to create the instance immediately (for manual creation)
# Skip sync during batch loading from DB to avoid multiple sync calls
if trigger_sync:
try:
await self.ap.plugin_connector.sync_polymorphic_component_instances()
self.ap.logger.info(f'Triggered sync after loading external KB {external_kb_entity.uuid}')
except Exception as e:
self.ap.logger.error(f'Failed to sync after loading external KB: {e}')
return external_kb
async def get_knowledge_base_by_uuid(self, kb_uuid: str) -> KnowledgeBaseInterface | None:
for kb in self.knowledge_bases:
if kb.get_uuid() == kb_uuid:
return kb
return None
return self.knowledge_bases.get(kb_uuid)
async def remove_knowledge_base_from_runtime(self, kb_uuid: str):
for kb in self.knowledge_bases:
if kb.get_uuid() == kb_uuid:
self.knowledge_bases.remove(kb)
return
self.knowledge_bases.pop(kb_uuid, None)
async def delete_knowledge_base(self, kb_uuid: str):
for kb in self.knowledge_bases:
if kb.get_uuid() == kb_uuid:
await kb.dispose()
self.knowledge_bases.remove(kb)
return
kb = self.knowledge_bases.pop(kb_uuid, None)
if kb is not None:
await kb.dispose()
else:
self.ap.logger.warning(f'Knowledge base {kb_uuid} not found in runtime, skipping plugin notification')

0
src/langbot/pkg/rag/knowledge/services/__init__.py
View File

15
src/langbot/pkg/rag/knowledge/services/base_service.py
View File

@@ -1,15 +0,0 @@
# 封装异步操作
import asyncio
class BaseService:
def __init__(self):
pass
async def _run_sync(self, func, *args, **kwargs):
"""
在单独的线程中运行同步函数。
如果第一个参数是 session，则在 to_thread 中获取新的 session。
"""
return await asyncio.to_thread(func, *args, **kwargs)

49
src/langbot/pkg/rag/knowledge/services/chunker.py
View File

@@ -1,49 +0,0 @@
from __future__ import annotations
import json
from typing import List
from langbot.pkg.rag.knowledge.services import base_service
from langbot.pkg.core import app
from langchain_text_splitters import RecursiveCharacterTextSplitter
class Chunker(base_service.BaseService):
"""
A class for splitting long texts into smaller, overlapping chunks.
"""
def __init__(self, ap: app.Application, chunk_size: int = 500, chunk_overlap: int = 50):
self.ap = ap
self.chunk_size = chunk_size
self.chunk_overlap = chunk_overlap
if self.chunk_overlap >= self.chunk_size:
self.ap.logger.warning(
'Chunk overlap is greater than or equal to chunk size. This may lead to empty or malformed chunks.'
)
def _split_text_sync(self, text: str) -> List[str]:
"""
Synchronously splits a long text into chunks with specified overlap.
This is a CPU-bound operation, intended to be run in a separate thread.
"""
if not text:
return []
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=self.chunk_size,
chunk_overlap=self.chunk_overlap,
length_function=len,
is_separator_regex=False,
)
return text_splitter.split_text(text)
async def chunk(self, text: str) -> List[str]:
"""
Asynchronously chunks a given text into smaller pieces.
"""
self.ap.logger.info(f'Chunking text (length: {len(text)})...')
# Run the synchronous splitting logic in a separate thread
chunks = await self._run_sync(self._split_text_sync, text)
self.ap.logger.info(f'Text chunked into {len(chunks)} pieces.')
self.ap.logger.debug(f'Chunks: {json.dumps(chunks, indent=4, ensure_ascii=False)}')
return chunks

55
src/langbot/pkg/rag/knowledge/services/embedder.py
View File

@@ -1,55 +0,0 @@
from __future__ import annotations
import uuid
from typing import List
from langbot.pkg.rag.knowledge.services.base_service import BaseService
from langbot.pkg.entity.persistence import rag as persistence_rag
from langbot.pkg.core import app
from langbot.pkg.provider.modelmgr.requester import RuntimeEmbeddingModel
import sqlalchemy
class Embedder(BaseService):
def __init__(self, ap: app.Application) -> None:
super().__init__()
self.ap = ap
async def embed_and_store(
self, kb_id: str, file_id: str, chunks: List[str], embedding_model: RuntimeEmbeddingModel
) -> list[persistence_rag.Chunk]:
# save chunk to db
chunk_entities: list[persistence_rag.Chunk] = []
chunk_ids: list[str] = []
for chunk_text in chunks:
chunk_uuid = str(uuid.uuid4())
chunk_ids.append(chunk_uuid)
chunk_entity = persistence_rag.Chunk(uuid=chunk_uuid, file_id=file_id, text=chunk_text)
chunk_entities.append(chunk_entity)
chunk_dicts = [
self.ap.persistence_mgr.serialize_model(persistence_rag.Chunk, chunk) for chunk in chunk_entities
]
await self.ap.persistence_mgr.execute_async(sqlalchemy.insert(persistence_rag.Chunk).values(chunk_dicts))
# get embeddings (batch size limit: 64 for OpenAI)
MAX_BATCH_SIZE = 64
embeddings_list: list[list[float]] = []
for i in range(0, len(chunks), MAX_BATCH_SIZE):
batch = chunks[i : i + MAX_BATCH_SIZE]
batch_embeddings = await embedding_model.provider.invoke_embedding(
model=embedding_model,
input_text=batch,
extra_args={}, # TODO: add extra args
knowledge_base_id=kb_id,
call_type='embedding',
)
embeddings_list.extend(batch_embeddings)
# save embeddings to vdb
await self.ap.vector_db_mgr.vector_db.add_embeddings(kb_id, chunk_ids, embeddings_list, chunk_dicts)
self.ap.logger.info(f'Successfully saved {len(chunk_entities)} embeddings to Knowledge Base.')
return chunk_entities

291
src/langbot/pkg/rag/knowledge/services/parser.py
View File

@@ -1,291 +0,0 @@
from __future__ import annotations
import PyPDF2
import io
from docx import Document
import chardet
from typing import Union, Callable, Any
import markdown
from bs4 import BeautifulSoup
import re
import asyncio # Import asyncio for async operations
from langbot.pkg.core import app
class FileParser:
"""
A robust file parser class to extract text content from various document formats.
It supports TXT, PDF, DOCX, XLSX, CSV, Markdown, HTML, and EPUB files.
All core file reading operations are designed to be run synchronously in a thread pool
to avoid blocking the asyncio event loop.
"""
def __init__(self, ap: app.Application):
self.ap = ap
async def _run_sync(self, sync_func: Callable, *args: Any, **kwargs: Any) -> Any:
"""
Runs a synchronous function in a separate thread to prevent blocking the event loop.
This is a general utility method for wrapping blocking I/O operations.
"""
try:
return await asyncio.to_thread(sync_func, *args, **kwargs)
except Exception as e:
self.ap.logger.error(f'Error running synchronous function {sync_func.__name__}: {e}')
raise
async def parse(self, file_name: str, extension: str) -> Union[str, None]:
"""
Parses the file based on its extension and returns the extracted text content.
This is the main asynchronous entry point for parsing.
Args:
file_name (str): The name of the file to be parsed, get from ap.storage_mgr
Returns:
Union[str, None]: The extracted text content as a single string, or None if parsing fails.
"""
file_extension = extension.lower()
parser_method = getattr(self, f'_parse_{file_extension}', None)
if parser_method is None:
self.ap.logger.error(f'Unsupported file format: {file_extension} for file {file_name}')
return None
try:
# Pass file_path to the specific parser methods
return await parser_method(file_name)
except Exception as e:
self.ap.logger.error(f'Failed to parse {file_extension} file {file_name}: {e}')
return None
# --- Helper for reading files with encoding detection ---
async def _read_file_content(self, file_name: str) -> Union[str, bytes]:
"""
Reads a file with automatic encoding detection, ensuring the synchronous
file read operation runs in a separate thread.
"""
# def _read_sync():
# with open(file_path, 'rb') as file:
# raw_data = file.read()
# detected = chardet.detect(raw_data)
# encoding = detected['encoding'] or 'utf-8'
# if mode == 'r':
# return raw_data.decode(encoding, errors='ignore')
# return raw_data # For binary mode
# return await self._run_sync(_read_sync)
file_bytes = await self.ap.storage_mgr.storage_provider.load(file_name)
detected = chardet.detect(file_bytes)
encoding = detected['encoding'] or 'utf-8'
return file_bytes.decode(encoding, errors='ignore')
# --- Specific Parser Methods ---
async def _parse_txt(self, file_name: str) -> str:
"""Parses a TXT file and returns its content."""
self.ap.logger.info(f'Parsing TXT file: {file_name}')
return await self._read_file_content(file_name)
async def _parse_pdf(self, file_name: str) -> str:
"""Parses a PDF file and returns its text content."""
self.ap.logger.info(f'Parsing PDF file: {file_name}')
# def _parse_pdf_sync():
# text_content = []
# with open(file_name, 'rb') as file:
# pdf_reader = PyPDF2.PdfReader(file)
# for page in pdf_reader.pages:
# text = page.extract_text()
# if text:
# text_content.append(text)
# return '\n'.join(text_content)
# return await self._run_sync(_parse_pdf_sync)
pdf_bytes = await self.ap.storage_mgr.storage_provider.load(file_name)
def _parse_pdf_sync():
pdf_reader = PyPDF2.PdfReader(io.BytesIO(pdf_bytes))
text_content = []
for page in pdf_reader.pages:
text = page.extract_text()
if text:
text_content.append(text)
return '\n'.join(text_content)
return await self._run_sync(_parse_pdf_sync)
async def _parse_docx(self, file_name: str) -> str:
"""Parses a DOCX file and returns its text content."""
self.ap.logger.info(f'Parsing DOCX file: {file_name}')
docx_bytes = await self.ap.storage_mgr.storage_provider.load(file_name)
def _parse_docx_sync():
doc = Document(io.BytesIO(docx_bytes))
text_content = [paragraph.text for paragraph in doc.paragraphs if paragraph.text.strip()]
return '\n'.join(text_content)
return await self._run_sync(_parse_docx_sync)
async def _parse_doc(self, file_name: str) -> str:
"""Handles .doc files, explicitly stating lack of direct support."""
self.ap.logger.warning(f'Direct .doc parsing is not supported for {file_name}. Please convert to .docx first.')
raise NotImplementedError('Direct .doc parsing not supported. Please convert to .docx first.')
# async def _parse_xlsx(self, file_name: str) -> str:
# """Parses an XLSX file, returning text from all sheets."""
# self.ap.logger.info(f'Parsing XLSX file: {file_name}')
# xlsx_bytes = await self.ap.storage_mgr.storage_provider.load(file_name)
# def _parse_xlsx_sync():
# excel_file = pd.ExcelFile(io.BytesIO(xlsx_bytes))
# all_sheet_content = []
# for sheet_name in excel_file.sheet_names:
# df = pd.read_excel(io.BytesIO(xlsx_bytes), sheet_name=sheet_name)
# sheet_text = f'--- Sheet: {sheet_name} ---\n{df.to_string(index=False)}\n'
# all_sheet_content.append(sheet_text)
# return '\n'.join(all_sheet_content)
# return await self._run_sync(_parse_xlsx_sync)
# async def _parse_csv(self, file_name: str) -> str:
# """Parses a CSV file and returns its content as a string."""
# self.ap.logger.info(f'Parsing CSV file: {file_name}')
# csv_bytes = await self.ap.storage_mgr.storage_provider.load(file_name)
# def _parse_csv_sync():
# # pd.read_csv can often detect encoding, but explicit detection is safer
# # raw_data = self._read_file_content(
# # file_name, mode='rb'
# # ) # Note: this will need to be await outside this sync function
# # _ = raw_data
# # For simplicity, we'll let pandas handle encoding internally after a raw read.
# # A more robust solution might pass encoding directly to pd.read_csv after detection.
# detected = chardet.detect(io.BytesIO(csv_bytes))
# encoding = detected['encoding'] or 'utf-8'
# df = pd.read_csv(io.BytesIO(csv_bytes), encoding=encoding)
# return df.to_string(index=False)
# return await self._run_sync(_parse_csv_sync)
async def _parse_md(self, file_name: str) -> str:
"""Parses a Markdown file, converting it to structured plain text."""
self.ap.logger.info(f'Parsing Markdown file: {file_name}')
md_bytes = await self.ap.storage_mgr.storage_provider.load(file_name)
def _parse_markdown_sync():
md_content = io.BytesIO(md_bytes).read().decode('utf-8', errors='ignore')
html_content = markdown.markdown(
md_content, extensions=['extra', 'codehilite', 'tables', 'toc', 'fenced_code']
)
soup = BeautifulSoup(html_content, 'html.parser')
text_parts = []
for element in soup.children:
if element.name in ['h1', 'h2', 'h3', 'h4', 'h5', 'h6']:
level = int(element.name[1])
text_parts.append('#' * level + ' ' + element.get_text().strip())
elif element.name == 'p':
text = element.get_text().strip()
if text:
text_parts.append(text)
elif element.name in ['ul', 'ol']:
for li in element.find_all('li'):
text_parts.append(f'* {li.get_text().strip()}')
elif element.name == 'pre':
code_block = element.get_text().strip()
if code_block:
text_parts.append(f'```\n{code_block}\n```')
elif element.name == 'table':
table_str = self._extract_table_to_markdown_sync(element) # Call sync helper
if table_str:
text_parts.append(table_str)
elif element.name:
text = element.get_text(separator=' ', strip=True)
if text:
text_parts.append(text)
cleaned_text = re.sub(r'\n\s*\n', '\n\n', '\n'.join(text_parts))
return cleaned_text.strip()
return await self._run_sync(_parse_markdown_sync)
async def _parse_html(self, file_name: str) -> str:
"""Parses an HTML file, extracting structured plain text."""
self.ap.logger.info(f'Parsing HTML file: {file_name}')
html_bytes = await self.ap.storage_mgr.storage_provider.load(file_name)
def _parse_html_sync():
html_content = io.BytesIO(html_bytes).read().decode('utf-8', errors='ignore')
soup = BeautifulSoup(html_content, 'html.parser')
for script_or_style in soup(['script', 'style']):
script_or_style.decompose()
text_parts = []
for element in soup.body.children if soup.body else soup.children:
if element.name in ['h1', 'h2', 'h3', 'h4', 'h5', 'h6']:
level = int(element.name[1])
text_parts.append('#' * level + ' ' + element.get_text().strip())
elif element.name == 'p':
text = element.get_text().strip()
if text:
text_parts.append(text)
elif element.name in ['ul', 'ol']:
for li in element.find_all('li'):
text = li.get_text().strip()
if text:
text_parts.append(f'* {text}')
elif element.name == 'table':
table_str = self._extract_table_to_markdown_sync(element) # Call sync helper
if table_str:
text_parts.append(table_str)
elif element.name:
text = element.get_text(separator=' ', strip=True)
if text:
text_parts.append(text)
cleaned_text = re.sub(r'\n\s*\n', '\n\n', '\n'.join(text_parts))
return cleaned_text.strip()
return await self._run_sync(_parse_html_sync)
def _add_toc_items_sync(self, toc_list: list, text_content: list, level: int):
"""Recursively adds TOC items to text_content (synchronous helper)."""
indent = ' ' * level
for item in toc_list:
if isinstance(item, tuple):
chapter, subchapters = item
text_content.append(f'{indent}- {chapter.title}')
self._add_toc_items_sync(subchapters, text_content, level + 1)
else:
text_content.append(f'{indent}- {item.title}')
def _extract_table_to_markdown_sync(self, table_element: BeautifulSoup) -> str:
"""Helper to convert a BeautifulSoup table element into a Markdown table string (synchronous)."""
headers = [th.get_text().strip() for th in table_element.find_all('th')]
rows = []
for tr in table_element.find_all('tr'):
cells = [td.get_text().strip() for td in tr.find_all('td')]
if cells:
rows.append(cells)
if not headers and not rows:
return ''
table_lines = []
if headers:
table_lines.append(' | '.join(headers))
table_lines.append(' | '.join(['---'] * len(headers)))
for row_cells in rows:
padded_cells = row_cells + [''] * (len(headers) - len(row_cells)) if headers else row_cells
table_lines.append(' | '.join(padded_cells))
return '\n'.join(table_lines)

53
src/langbot/pkg/rag/knowledge/services/retriever.py
View File

@@ -1,53 +0,0 @@
from __future__ import annotations
from . import base_service
from ....core import app
from ....provider.modelmgr.requester import RuntimeEmbeddingModel
from langbot_plugin.api.entities.builtin.rag import context as rag_context
from langbot_plugin.api.entities.builtin.provider.message import ContentElement
class Retriever(base_service.BaseService):
def __init__(self, ap: app.Application):
super().__init__()
self.ap = ap
async def retrieve(
self, kb_id: str, query: str, embedding_model: RuntimeEmbeddingModel, k: int = 5
) -> list[rag_context.RetrievalResultEntry]:
self.ap.logger.info(
f"Retrieving for query: '{query[:10]}' with k={k} using {embedding_model.model_entity.uuid}"
)
query_embedding: list[float] = await embedding_model.provider.invoke_embedding(
model=embedding_model,
input_text=[query],
extra_args={}, # TODO: add extra args
knowledge_base_id=kb_id,
query_text=query,
call_type='retrieve',
)
vector_results = await self.ap.vector_db_mgr.vector_db.search(kb_id, query_embedding[0], k)
# 'ids' shape mirrors the Chroma-style response contract for compatibility
matched_vector_ids = vector_results.get('ids', [[]])[0]
distances = vector_results.get('distances', [[]])[0]
vector_metadatas = vector_results.get('metadatas', [[]])[0]
if not matched_vector_ids:
self.ap.logger.info('No relevant chunks found in vector database.')
return []
result: list[rag_context.RetrievalResultEntry] = []
for i, id in enumerate(matched_vector_ids):
entry = rag_context.RetrievalResultEntry(
id=id,
content=[ContentElement.from_text(vector_metadatas[i].get('text', ''))],
metadata=vector_metadatas[i],
distance=distances[i],
)
result.append(entry)
return result

1
src/langbot/pkg/rag/service/__init__.py Normal file
View File

@@ -0,0 +1 @@
from .runtime import RAGRuntimeService as RAGRuntimeService

89
src/langbot/pkg/rag/service/runtime.py Normal file
View File

@@ -0,0 +1,89 @@
from __future__ import annotations
import posixpath
from typing import Any
from langbot.pkg.core import app
class RAGRuntimeService:
"""Service to handle RAG-related requests from plugins (Runtime).
This service acts as the bridge between plugin RPC requests and
LangBot's infrastructure (embedding models, vector databases, file storage).
"""
def __init__(self, ap: app.Application):
self.ap = ap
async def vector_upsert(
self,
collection_id: str,
vectors: list[list[float]],
ids: list[str],
metadata: list[dict[str, Any]] | None = None,
documents: list[str] | None = None,
) -> None:
"""Handle VECTOR_UPSERT action."""
metadatas = metadata if metadata else [{} for _ in vectors]
await self.ap.vector_db_mgr.upsert(
collection_name=collection_id,
vectors=vectors,
ids=ids,
metadata=metadatas,
documents=documents,
)
async def vector_search(
self,
collection_id: str,
query_vector: list[float],
top_k: int,
filters: dict[str, Any] | None = None,
search_type: str = 'vector',
query_text: str = '',
) -> list[dict[str, Any]]:
"""Handle VECTOR_SEARCH action."""
return await self.ap.vector_db_mgr.search(
collection_name=collection_id,
query_vector=query_vector,
limit=top_k,
filter=filters,
search_type=search_type,
query_text=query_text,
)
async def vector_delete(
self, collection_id: str, file_ids: list[str] | None = None, filters: dict[str, Any] | None = None
) -> int:
"""Handle VECTOR_DELETE action.
Deletes vectors associated with the given file IDs from the collection.
Each file_id corresponds to a document whose vectors will be removed.
Args:
collection_id: The collection to delete from.
file_ids: File IDs whose associated vectors should be deleted.
Each file_id maps to a set of vectors stored with that file_id
in their metadata.
filters: Filter-based deletion (not yet supported, will raise).
"""
count = 0
if file_ids:
await self.ap.vector_db_mgr.delete_by_file_id(collection_name=collection_id, file_ids=file_ids)
count = len(file_ids)
elif filters:
count = await self.ap.vector_db_mgr.delete_by_filter(collection_name=collection_id, filter=filters)
return count
async def get_file_stream(self, storage_path: str) -> bytes:
"""Handle GET_KNOWLEDEGE_FILE_STREAM action.
Uses the storage manager abstraction to load file content,
regardless of the underlying storage provider.
"""
# Validate storage_path to prevent path traversal
normalized = posixpath.normpath(storage_path)
if normalized.startswith('/') or '..' in normalized.split('/'):
raise ValueError('Invalid storage path')
content_bytes = await self.ap.storage_mgr.storage_provider.load(normalized)
return content_bytes if content_bytes else b''

7
src/langbot/pkg/storage/provider.py
View File

@@ -43,6 +43,13 @@ class StorageProvider(abc.ABC):
):
pass
@abc.abstractmethod
async def size(
self,
key: str,
) -> int:
pass
@abc.abstractmethod
async def delete_dir_recursive(
self,

6
src/langbot/pkg/storage/providers/localstorage.py
View File

@@ -47,6 +47,12 @@ class LocalStorageProvider(provider.StorageProvider):
):
os.remove(os.path.join(LOCAL_STORAGE_PATH, f'{key}'))
async def size(
self,
key: str,
) -> int:
return os.path.getsize(os.path.join(LOCAL_STORAGE_PATH, f'{key}'))
async def delete_dir_recursive(
self,
dir_path: str,

15
src/langbot/pkg/storage/providers/s3storage.py
View File

@@ -117,6 +117,21 @@ class S3StorageProvider(provider.StorageProvider):
self.ap.logger.error(f'Failed to delete from S3: {e}')
raise
async def size(
self,
key: str,
) -> int:
"""Get object size from S3 without downloading it"""
try:
response = self.s3_client.head_object(
Bucket=self.bucket_name,
Key=key,
)
return response['ContentLength']
except Exception as e:
self.ap.logger.error(f'Failed to get size from S3: {e}')
raise
async def delete_dir_recursive(
self,
dir_path: str,

69
src/langbot/pkg/vector/filter_utils.py Normal file
View File

@@ -0,0 +1,69 @@
"""Shared utilities for metadata filter handling across VDB backends.
Canonical filter format (Chroma-style ``where`` syntax):
{"file_id": "abc"} # implicit $eq
{"file_id": {"$eq": "abc"}} # explicit $eq
{"created_at": {"$gte": 1700000000}} # comparison
{"file_type": {"$in": ["pdf", "docx"]}} # in-list
Multiple top-level keys are AND-ed. Supported operators:
``$eq``, ``$ne``, ``$gt``, ``$gte``, ``$lt``, ``$lte``, ``$in``, ``$nin``.
"""
from __future__ import annotations
import logging
from typing import Any
SUPPORTED_OPS = frozenset({'$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in', '$nin'})
logger = logging.getLogger(__name__)
def normalize_filter(
raw: dict[str, Any] | None,
) -> list[tuple[str, str, Any]]:
"""Parse a canonical filter dict into ``[(field, op, value)]`` triples.
Returns an empty list when *raw* is ``None`` or empty.
Raises ``ValueError`` on unsupported operators or malformed entries.
"""
if not raw:
return []
triples: list[tuple[str, str, Any]] = []
for field, condition in raw.items():
if isinstance(condition, dict):
for op, value in condition.items():
if op not in SUPPORTED_OPS:
raise ValueError(f'Unsupported filter operator: {op}')
triples.append((field, op, value))
else:
# Bare value -> implicit $eq
triples.append((field, '$eq', condition))
return triples
def strip_unsupported_fields(
triples: list[tuple[str, str, Any]],
supported_fields: set[str],
) -> list[tuple[str, str, Any]]:
"""Return only triples whose field is in *supported_fields*.
Dropped fields are logged at WARNING level so the caller knows they were
silently ignored (useful for Milvus / pgvector which only store a fixed
schema).
"""
kept: list[tuple[str, str, Any]] = []
for field, op, value in triples:
if field in supported_fields:
kept.append((field, op, value))
else:
logger.warning(
'Filter field %r is not supported by this backend and will be ignored (supported: %s)',
field,
', '.join(sorted(supported_fields)),
)
return kept

94
src/langbot/pkg/vector/mgr.py
View File

@@ -1,7 +1,7 @@
from __future__ import annotations
from ..core import app
from .vdb import VectorDatabase
from .vdb import VectorDatabase, SearchType
from .vdbs.chroma import ChromaVectorDatabase
from .vdbs.qdrant import QdrantVectorDatabase
from .vdbs.seekdb import SeekDBVectorDatabase
@@ -65,3 +65,95 @@ class VectorDBManager:
else:
self.vector_db = ChromaVectorDatabase(self.ap)
self.ap.logger.warning('No vector database backend configured, defaulting to Chroma.')
def get_supported_search_types(self) -> list[str]:
"""Return the search types supported by the current VDB backend."""
if self.vector_db is None:
return [SearchType.VECTOR.value]
return [st.value for st in self.vector_db.supported_search_types()]
async def upsert(
self,
collection_name: str,
vectors: list[list[float]],
ids: list[str],
metadata: list[dict] | None = None,
documents: list[str] | None = None,
):
"""Proxy: Upsert vectors"""
await self.vector_db.add_embeddings(
collection=collection_name,
ids=ids,
embeddings_list=vectors,
metadatas=metadata or [{} for _ in vectors],
documents=documents,
)
async def search(
self,
collection_name: str,
query_vector: list[float],
limit: int,
filter: dict | None = None,
search_type: str = 'vector',
query_text: str = '',
) -> list[dict]:
"""Proxy: Search vectors.
Returns a list of dicts with keys: 'id', 'score', 'metadata'.
The underlying VectorDatabase.search returns Chroma-style format:
{ 'ids': [['id1']], 'distances': [[0.1]], 'metadatas': [[{}]] }
"""
results = await self.vector_db.search(
collection=collection_name,
query_embedding=query_vector,
k=limit,
search_type=search_type,
query_text=query_text,
filter=filter,
)
if not results or 'ids' not in results or not results['ids']:
return []
# Flatten nested lists (Chroma returns batch-style: list of lists)
raw_ids = results['ids']
raw_dists = results.get('distances', [])
raw_metas = results.get('metadatas', [])
r_ids = raw_ids[0] if raw_ids and isinstance(raw_ids[0], list) else raw_ids
r_dists = raw_dists[0] if raw_dists and isinstance(raw_dists[0], list) else raw_dists
r_metas = raw_metas[0] if raw_metas and isinstance(raw_metas[0], list) else raw_metas
parsed_results = []
for i, id_val in enumerate(r_ids):
parsed_results.append(
{
'id': id_val,
'score': r_dists[i] if r_dists and i < len(r_dists) else 0.0,
'metadata': r_metas[i] if r_metas and i < len(r_metas) else {},
}
)
return parsed_results
async def delete_by_file_id(self, collection_name: str, file_ids: list[str]):
"""Proxy: Delete vectors by file_id (metadata-level identifier).
This delegates to VectorDatabase.delete_by_file_id which removes
all vectors associated with the given file IDs.
"""
for file_id in file_ids:
await self.vector_db.delete_by_file_id(collection_name, file_id)
async def delete_collection(self, collection_name: str):
"""Proxy: Delete an entire collection."""
await self.vector_db.delete_collection(collection_name)
async def delete_by_filter(self, collection_name: str, filter: dict) -> int:
"""Proxy: Delete vectors by metadata filter.
Returns:
Number of deleted vectors (best-effort; some backends return 0).
"""
return await self.vector_db.delete_by_filter(collection_name, filter)

73
src/langbot/pkg/vector/vdb.py
View File

@@ -1,10 +1,28 @@
from __future__ import annotations
import abc
import enum
from typing import Any, Dict
import numpy as np
class SearchType(str, enum.Enum):
"""Supported search types for vector databases."""
VECTOR = 'vector'
FULL_TEXT = 'full_text'
HYBRID = 'hybrid'
class VectorDatabase(abc.ABC):
@classmethod
def supported_search_types(cls) -> list[SearchType]:
"""Return the search types supported by this VDB backend.
Default: vector search only. Override in subclasses that support
full-text or hybrid search.
"""
return [SearchType.VECTOR]
@abc.abstractmethod
async def add_embeddings(
self,
@@ -12,14 +30,47 @@ class VectorDatabase(abc.ABC):
ids: list[str],
embeddings_list: list[list[float]],
metadatas: list[dict[str, Any]],
documents: list[str],
documents: list[str] | None = None,
) -> None:
"""Add vector data to the specified collection."""
"""Add vector data to the specified collection.
Args:
collection: Collection name.
ids: Unique IDs for each vector.
embeddings_list: List of embedding vectors.
metadatas: List of metadata dicts.
documents: Optional raw text documents. Required for full-text
and hybrid search in backends that support them.
"""
pass
@abc.abstractmethod
async def search(self, collection: str, query_embedding: np.ndarray, k: int = 5) -> Dict[str, Any]:
"""Search for the most similar vectors in the specified collection."""
async def search(
self,
collection: str,
query_embedding: np.ndarray,
k: int = 5,
search_type: str = 'vector',
query_text: str = '',
filter: dict[str, Any] | None = None,
) -> Dict[str, Any]:
"""Search for the most similar vectors in the specified collection.
Args:
collection: Collection name.
query_embedding: Query vector for similarity search.
k: Number of results to return.
search_type: One of 'vector', 'full_text', 'hybrid'.
query_text: Raw query text, used for full_text and hybrid search.
filter: Optional metadata filters using Chroma-style ``where``
syntax. Multiple top-level keys are AND-ed. Supported
operators: ``$eq``, ``$ne``, ``$gt``, ``$gte``, ``$lt``,
``$lte``, ``$in``, ``$nin``. Example::
{"file_id": "abc"}
{"created_at": {"$gte": 1700000000}}
{"file_type": {"$in": ["pdf", "docx"]}}
"""
pass
@abc.abstractmethod
@@ -27,6 +78,20 @@ class VectorDatabase(abc.ABC):
"""Delete vectors from the specified collection by file_id."""
pass
@abc.abstractmethod
async def delete_by_filter(self, collection: str, filter: dict[str, Any]) -> int:
"""Delete vectors matching the given metadata filter.
Args:
collection: Collection name.
filter: Metadata filter dict in canonical format (see ``search``).
Returns:
Number of deleted vectors (best-effort; backends that cannot
report an exact count may return 0).
"""
pass
@abc.abstractmethod
async def get_or_create_collection(self, collection: str):
"""Get or create collection."""

28
src/langbot/pkg/vector/vdbs/chroma.py
View File

@@ -28,19 +28,33 @@ class ChromaVectorDatabase(VectorDatabase):
ids: list[str],
embeddings_list: list[list[float]],
metadatas: list[dict[str, Any]],
documents: list[str] | None = None,
) -> None:
col = await self.get_or_create_collection(collection)
await asyncio.to_thread(col.add, embeddings=embeddings_list, ids=ids, metadatas=metadatas)
kwargs: dict[str, Any] = dict(embeddings=embeddings_list, ids=ids, metadatas=metadatas)
if documents is not None:
kwargs['documents'] = documents
await asyncio.to_thread(col.add, **kwargs)
self.ap.logger.info(f"Added {len(ids)} embeddings to Chroma collection '{collection}'.")
async def search(self, collection: str, query_embedding: list[float], k: int = 5) -> dict[str, Any]:
async def search(
self,
collection: str,
query_embedding: list[float],
k: int = 5,
search_type: str = 'vector',
query_text: str = '',
filter: dict[str, Any] | None = None,
) -> dict[str, Any]:
col = await self.get_or_create_collection(collection)
results = await asyncio.to_thread(
col.query,
query_kwargs: dict[str, Any] = dict(
query_embeddings=query_embedding,
n_results=k,
include=['metadatas', 'distances', 'documents'],
)
if filter:
query_kwargs['where'] = filter
results = await asyncio.to_thread(col.query, **query_kwargs)
self.ap.logger.info(f"Chroma search in '{collection}' returned {len(results.get('ids', [[]])[0])} results.")
return results
@@ -49,6 +63,12 @@ class ChromaVectorDatabase(VectorDatabase):
await asyncio.to_thread(col.delete, where={'file_id': file_id})
self.ap.logger.info(f"Deleted embeddings from Chroma collection '{collection}' with file_id: {file_id}")
async def delete_by_filter(self, collection: str, filter: dict[str, Any]) -> int:
col = await self.get_or_create_collection(collection)
await asyncio.to_thread(col.delete, where=filter)
self.ap.logger.info(f"Deleted embeddings from Chroma collection '{collection}' by filter")
return 0 # Chroma delete does not return a count
async def delete_collection(self, collection: str):
if collection in self._collections:
del self._collections[collection]

78
src/langbot/pkg/vector/vdbs/milvus.py
View File

@@ -4,8 +4,51 @@ from typing import Any, Dict
from pymilvus import MilvusClient, DataType, CollectionSchema, FieldSchema
from pymilvus.milvus_client.index import IndexParams
from langbot.pkg.vector.vdb import VectorDatabase
from langbot.pkg.vector.filter_utils import normalize_filter, strip_unsupported_fields
from langbot.pkg.core import app
# Milvus schema only stores these metadata fields; filter on other fields is
# silently dropped with a warning.
_MILVUS_SUPPORTED_FIELDS = {'text', 'file_id', 'chunk_uuid'}
def _build_milvus_expr(filter_dict: dict[str, Any]) -> str:
"""Translate canonical filter dict into a Milvus boolean expression string."""
triples = normalize_filter(filter_dict)
triples = strip_unsupported_fields(triples, _MILVUS_SUPPORTED_FIELDS)
if not triples:
return ''
parts: list[str] = []
for field, op, value in triples:
if op == '$eq':
parts.append(f'{field} == {_milvus_literal(value)}')
elif op == '$ne':
parts.append(f'{field} != {_milvus_literal(value)}')
elif op == '$gt':
parts.append(f'{field} > {_milvus_literal(value)}')
elif op == '$gte':
parts.append(f'{field} >= {_milvus_literal(value)}')
elif op == '$lt':
parts.append(f'{field} < {_milvus_literal(value)}')
elif op == '$lte':
parts.append(f'{field} <= {_milvus_literal(value)}')
elif op == '$in':
items = ', '.join(_milvus_literal(v) for v in value)
parts.append(f'{field} in [{items}]')
elif op == '$nin':
items = ', '.join(_milvus_literal(v) for v in value)
parts.append(f'{field} not in [{items}]')
return ' and '.join(parts)
def _milvus_literal(value: Any) -> str:
"""Format a Python value as a Milvus expression literal."""
if isinstance(value, str):
escaped = value.replace('\\', '\\\\').replace('"', '\\"')
return f'"{escaped}"'
return str(value)
class MilvusVectorDatabase(VectorDatabase):
"""Milvus vector database implementation"""
@@ -155,6 +198,7 @@ class MilvusVectorDatabase(VectorDatabase):
ids: list[str],
embeddings_list: list[list[float]],
metadatas: list[dict[str, Any]],
documents: list[str] | None = None,
) -> None:
"""Add vector embeddings to Milvus collection
@@ -200,7 +244,15 @@ class MilvusVectorDatabase(VectorDatabase):
self.ap.logger.info(f"Added {len(ids)} embeddings to Milvus collection '{collection}'")
async def search(self, collection: str, query_embedding: list[float], k: int = 5) -> Dict[str, Any]:
async def search(
self,
collection: str,
query_embedding: list[float],
k: int = 5,
search_type: str = 'vector',
query_text: str = '',
filter: dict[str, Any] | None = None,
) -> Dict[str, Any]:
"""Search for similar vectors in Milvus collection
Args:
@@ -217,14 +269,19 @@ class MilvusVectorDatabase(VectorDatabase):
# Perform search
search_params = {'metric_type': 'COSINE', 'params': {}}
results = await asyncio.to_thread(
self.client.search,
search_kwargs: dict[str, Any] = dict(
collection_name=collection,
data=[query_embedding],
limit=k,
search_params=search_params,
output_fields=['text', 'file_id', 'chunk_uuid'],
)
if filter:
expr = _build_milvus_expr(filter)
if expr:
search_kwargs['filter'] = expr
results = await asyncio.to_thread(self.client.search, **search_kwargs)
# Convert results to Chroma-compatible format
# Milvus returns: [[ {id, distance, entity: {...}} ]]
@@ -268,6 +325,21 @@ class MilvusVectorDatabase(VectorDatabase):
await asyncio.to_thread(self.client.delete, collection_name=collection, filter=f'file_id == "{file_id}"')
self.ap.logger.info(f"Deleted embeddings from Milvus collection '{collection}' with file_id: {file_id}")
async def delete_by_filter(self, collection: str, filter: dict[str, Any]) -> int:
collection = self._normalize_collection_name(collection)
await self.get_or_create_collection(collection)
expr = _build_milvus_expr(filter)
if not expr:
self.ap.logger.warning(
f"Milvus delete_by_filter on '{collection}': filter produced empty expression, skipping"
)
return 0
await asyncio.to_thread(self.client.delete, collection_name=collection, filter=expr)
self.ap.logger.info(f"Deleted embeddings from Milvus collection '{collection}' by filter")
return 0 # Milvus delete does not return a count
async def delete_collection(self, collection: str):
"""Delete a Milvus collection

86
src/langbot/pkg/vector/vdbs/pgvector_db.py
View File

@@ -5,10 +5,21 @@ from sqlalchemy.orm import declarative_base
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession, async_sessionmaker
from pgvector.sqlalchemy import Vector
from langbot.pkg.vector.vdb import VectorDatabase
from langbot.pkg.vector.filter_utils import normalize_filter, strip_unsupported_fields
from langbot.pkg.core import app
Base = declarative_base()
# pgvector schema only stores these metadata fields.
_PG_SUPPORTED_FIELDS = {'text', 'file_id', 'chunk_uuid'}
# Map schema field names to SQLAlchemy columns (resolved lazily from PgVectorEntry).
_PG_COLUMN_MAP = {
'text': 'text',
'file_id': 'file_id',
'chunk_uuid': 'chunk_uuid',
}
class PgVectorEntry(Base):
"""SQLAlchemy model for pgvector entries"""
@@ -23,6 +34,33 @@ class PgVectorEntry(Base):
chunk_uuid = Column(String)
def _build_pg_conditions(filter_dict: dict[str, Any]) -> list:
"""Translate canonical filter dict into a list of SQLAlchemy conditions."""
triples = normalize_filter(filter_dict)
triples = strip_unsupported_fields(triples, _PG_SUPPORTED_FIELDS)
conditions = []
for field, op, value in triples:
col = getattr(PgVectorEntry, _PG_COLUMN_MAP[field])
if op == '$eq':
conditions.append(col == value)
elif op == '$ne':
conditions.append(col != value)
elif op == '$gt':
conditions.append(col > value)
elif op == '$gte':
conditions.append(col >= value)
elif op == '$lt':
conditions.append(col < value)
elif op == '$lte':
conditions.append(col <= value)
elif op == '$in':
conditions.append(col.in_(value))
elif op == '$nin':
conditions.append(col.notin_(value))
return conditions
class PgVectorDatabase(VectorDatabase):
"""PostgreSQL with pgvector extension database implementation"""
@@ -109,6 +147,7 @@ class PgVectorDatabase(VectorDatabase):
ids: list[str],
embeddings_list: list[list[float]],
metadatas: list[dict[str, Any]],
documents: list[str] | None = None,
) -> None:
"""Add vector embeddings to pgvector
@@ -142,7 +181,15 @@ class PgVectorDatabase(VectorDatabase):
self.ap.logger.error(f'Error adding embeddings to pgvector: {e}')
raise
async def search(self, collection: str, query_embedding: list[float], k: int = 5) -> Dict[str, Any]:
async def search(
self,
collection: str,
query_embedding: list[float],
k: int = 5,
search_type: str = 'vector',
query_text: str = '',
filter: dict[str, Any] | None = None,
) -> Dict[str, Any]:
"""Search for similar vectors using cosine distance
Args:
@@ -174,6 +221,10 @@ class PgVectorDatabase(VectorDatabase):
.limit(k)
)
if filter:
for cond in _build_pg_conditions(filter):
stmt = stmt.filter(cond)
result = await session.execute(stmt)
rows = result.fetchall()
@@ -225,6 +276,39 @@ class PgVectorDatabase(VectorDatabase):
self.ap.logger.error(f'Error deleting from pgvector: {e}')
raise
async def delete_by_filter(self, collection: str, filter: dict[str, Any]) -> int:
"""Delete vectors matching a metadata filter.
Args:
collection: Collection name
filter: Canonical metadata filter dict
"""
conditions = _build_pg_conditions(filter)
if not conditions:
self.ap.logger.warning(
f"pgvector delete_by_filter on '{collection}': filter produced no conditions, skipping"
)
return 0
await self.get_or_create_collection(collection)
async with self.AsyncSessionLocal() as session:
try:
from sqlalchemy import delete
stmt = delete(PgVectorEntry).where(PgVectorEntry.collection == collection)
for cond in conditions:
stmt = stmt.where(cond)
result = await session.execute(stmt)
await session.commit()
deleted = result.rowcount
self.ap.logger.info(f"Deleted {deleted} embeddings from pgvector collection '{collection}' by filter")
return deleted
except Exception as e:
await session.rollback()
self.ap.logger.error(f'Error deleting from pgvector by filter: {e}')
raise
async def delete_collection(self, collection: str):
"""Delete all vectors in a collection

73
src/langbot/pkg/vector/vdbs/qdrant.py
View File

@@ -5,6 +5,37 @@ from typing import Any, Dict, List
from qdrant_client import AsyncQdrantClient, models
from langbot.pkg.core import app
from langbot.pkg.vector.vdb import VectorDatabase
from langbot.pkg.vector.filter_utils import normalize_filter
def _build_qdrant_filter(filter_dict: dict[str, Any]) -> models.Filter:
"""Translate canonical filter dict into a Qdrant ``models.Filter``."""
triples = normalize_filter(filter_dict)
must: list[models.Condition] = []
must_not: list[models.Condition] = []
for field, op, value in triples:
if op == '$eq':
must.append(models.FieldCondition(key=field, match=models.MatchValue(value=value)))
elif op == '$ne':
must_not.append(models.FieldCondition(key=field, match=models.MatchValue(value=value)))
elif op == '$in':
must.append(models.FieldCondition(key=field, match=models.MatchAny(any=value)))
elif op == '$nin':
must_not.append(models.FieldCondition(key=field, match=models.MatchAny(any=value)))
elif op in ('$gt', '$gte', '$lt', '$lte'):
range_kwargs: dict[str, Any] = {}
if op == '$gt':
range_kwargs['gt'] = value
elif op == '$gte':
range_kwargs['gte'] = value
elif op == '$lt':
range_kwargs['lt'] = value
elif op == '$lte':
range_kwargs['lte'] = value
must.append(models.FieldCondition(key=field, range=models.Range(**range_kwargs)))
return models.Filter(must=must or None, must_not=must_not or None)
class QdrantVectorDatabase(VectorDatabase):
@@ -48,6 +79,7 @@ class QdrantVectorDatabase(VectorDatabase):
ids: List[str],
embeddings_list: List[List[float]],
metadatas: List[Dict[str, Any]],
documents: List[str] | None = None,
) -> None:
if not embeddings_list:
return
@@ -60,19 +92,29 @@ class QdrantVectorDatabase(VectorDatabase):
await self.client.upsert(collection_name=collection, points=points)
self.ap.logger.info(f"Added {len(ids)} embeddings to Qdrant collection '{collection}'.")
async def search(self, collection: str, query_embedding: list[float], k: int = 5) -> dict[str, Any]:
async def search(
self,
collection: str,
query_embedding: list[float],
k: int = 5,
search_type: str = 'vector',
query_text: str = '',
filter: dict[str, Any] | None = None,
) -> dict[str, Any]:
exists = await self.client.collection_exists(collection)
if not exists:
return {'ids': [[]], 'metadatas': [[]], 'distances': [[]]}
hits = (
await self.client.query_points(
collection_name=collection,
query=query_embedding,
limit=k,
with_payload=True,
)
).points
query_kwargs: dict[str, Any] = dict(
collection_name=collection,
query=query_embedding,
limit=k,
with_payload=True,
)
if filter:
query_kwargs['query_filter'] = _build_qdrant_filter(filter)
hits = (await self.client.query_points(**query_kwargs)).points
ids = [str(hit.id) for hit in hits]
metadatas = [hit.payload or {} for hit in hits]
# Qdrant's score is similarity; convert to a pseudo-distance for consistency
@@ -95,6 +137,19 @@ class QdrantVectorDatabase(VectorDatabase):
)
self.ap.logger.info(f"Deleted embeddings from Qdrant collection '{collection}' with file_id: {file_id}")
async def delete_by_filter(self, collection: str, filter: dict[str, Any]) -> int:
exists = await self.client.collection_exists(collection)
if not exists:
return 0
qdrant_filter = _build_qdrant_filter(filter)
await self.client.delete(
collection_name=collection,
points_selector=qdrant_filter,
)
self.ap.logger.info(f"Deleted embeddings from Qdrant collection '{collection}' by filter")
return 0 # Qdrant delete does not return a count
async def delete_collection(self, collection: str):
try:
await self.client.delete_collection(collection)

137
src/langbot/pkg/vector/vdbs/seekdb.py
View File

@@ -5,7 +5,7 @@ from typing import Any, Dict, List
from langbot.pkg.core import app
from langbot.pkg.vector.vdb import VectorDatabase
from langbot.pkg.vector.vdb import VectorDatabase, SearchType
try:
import pyseekdb
@@ -25,9 +25,13 @@ class SeekDBVectorDatabase(VectorDatabase):
SeekDB is an AI-native search database by OceanBase that unifies
relational, vector, text, JSON and GIS in a single engine.
Supports both embedded mode and remote server mode.
Supports embedded mode, remote server mode, and full-text/hybrid search.
"""
@classmethod
def supported_search_types(cls) -> list[SearchType]:
return [SearchType.VECTOR, SearchType.FULL_TEXT, SearchType.HYBRID]
def __init__(self, ap: app.Application):
if not SEEKDB_AVAILABLE:
raise ImportError('pyseekdb is not installed. Install it with: pip install pyseekdb')
@@ -89,6 +93,7 @@ class SeekDBVectorDatabase(VectorDatabase):
{
'\x00': '',
'\\': '\\\\',
"'": "''", # Standard SQL escaping (OceanBase NO_BACKSLASH_ESCAPES)
'"': '\\"',
'\n': '\\n',
'\r': '\\r',
@@ -111,8 +116,10 @@ class SeekDBVectorDatabase(VectorDatabase):
# Collection doesn't exist, create it
if vector_size is None:
# Default dimension if not specified
vector_size = 384
raise ValueError(
f"Cannot create SeekDB collection '{collection}' without knowing the vector dimension. "
'Ensure add_embeddings is called before any standalone get_or_create_collection.'
)
# Create HNSW configuration
config = HNSWConfiguration(dimension=vector_size, distance='cosine')
@@ -147,7 +154,12 @@ class SeekDBVectorDatabase(VectorDatabase):
return await self._get_or_create_collection_internal(collection)
async def add_embeddings(
self, collection: str, ids: List[str], embeddings_list: List[List[float]], metadatas: List[Dict[str, Any]]
self,
collection: str,
ids: List[str],
embeddings_list: List[List[float]],
metadatas: List[Dict[str, Any]],
documents: List[str] | None = None,
) -> None:
"""Add vector embeddings to the specified collection.
@@ -156,6 +168,7 @@ class SeekDBVectorDatabase(VectorDatabase):
ids: List of document IDs
embeddings_list: List of embedding vectors
metadatas: List of metadata dictionaries
documents: Optional raw text documents for full-text search support
"""
if not embeddings_list:
return
@@ -166,17 +179,33 @@ class SeekDBVectorDatabase(VectorDatabase):
cleaned_metadatas = [self._clean_metadata(meta) for meta in metadatas]
await asyncio.to_thread(coll.add, ids=ids, embeddings=embeddings_list, metadatas=cleaned_metadatas)
kwargs: Dict[str, Any] = dict(ids=ids, embeddings=embeddings_list, metadatas=cleaned_metadatas)
if documents is not None:
kwargs['documents'] = [doc.translate(self._escape_table) for doc in documents]
await asyncio.to_thread(coll.add, **kwargs)
self.ap.logger.info(f"Added {len(ids)} embeddings to SeekDB collection '{collection}'")
async def search(self, collection: str, query_embedding: List[float], k: int = 5) -> Dict[str, Any]:
async def search(
self,
collection: str,
query_embedding: List[float],
k: int = 5,
search_type: str = 'vector',
query_text: str = '',
filter: Dict[str, Any] | None = None,
) -> Dict[str, Any]:
"""Search for the most similar vectors in the specified collection.
SeekDB supports vector, full-text, and hybrid search modes.
Args:
collection: Collection name
query_embedding: Query vector
query_embedding: Query vector (used for vector and hybrid modes)
k: Number of results to return
search_type: One of 'vector', 'full_text', 'hybrid'
query_text: Raw query text (used for full_text and hybrid modes)
filter: Optional metadata filters (Chroma-style ``where`` syntax).
Returns:
Dictionary with 'ids', 'metadatas', 'distances' keys
@@ -193,11 +222,73 @@ class SeekDBVectorDatabase(VectorDatabase):
else:
coll = self._collections[collection]
# Perform query
# SeekDB's query() returns: {'ids': [[...]], 'metadatas': [[...]], 'distances': [[...]]}
results = await asyncio.to_thread(coll.query, query_embeddings=query_embedding, n_results=k)
# Route by search type.
# pyseekdb's query() always requires embeddings, so full-text and
# hybrid modes use hybrid_search() which supports text-only queries
# and returns the same nested-list format with distances.
if search_type == SearchType.FULL_TEXT:
if not query_text:
return {'ids': [[]], 'metadatas': [[]], 'distances': [[]]}
self.ap.logger.info(f"SeekDB search in '{collection}' returned {len(results.get('ids', [[]])[0])} results")
query_cfg: Dict[str, Any] = {
'where_document': {'$contains': query_text},
'n_results': k,
}
if filter:
query_cfg['where'] = filter
# TODO: pyseekdb hybrid_search with query-only (no knn) returns None
# for IDs due to column name mismatch (*/_id vs _id).
# See: https://github.com/oceanbase/pyseekdb/issues/171
results = await asyncio.to_thread(
coll.hybrid_search,
query=query_cfg,
knn=None,
n_results=k,
include=['documents', 'metadatas'],
)
elif search_type == SearchType.HYBRID:
if not query_text:
# Fall back to pure vector search when no text is provided
query_kwargs: Dict[str, Any] = {
'n_results': k,
'query_embeddings': query_embedding,
}
if filter:
query_kwargs['where'] = filter
results = await asyncio.to_thread(coll.query, **query_kwargs)
else:
query_cfg = {
'where_document': {'$contains': query_text},
'n_results': k,
}
knn_cfg: Dict[str, Any] = {
'query_embeddings': query_embedding,
'n_results': k,
}
if filter:
query_cfg['where'] = filter
knn_cfg['where'] = filter
results = await asyncio.to_thread(
coll.hybrid_search,
query=query_cfg,
knn=knn_cfg,
rank={'rrf': {}},
n_results=k,
include=['documents', 'metadatas'],
)
else:
# Default: vector search via query()
query_kwargs = {'n_results': k, 'query_embeddings': query_embedding}
if filter:
query_kwargs['where'] = filter
results = await asyncio.to_thread(coll.query, **query_kwargs)
self.ap.logger.info(
f"SeekDB {search_type} search in '{collection}' returned {len(results.get('ids', [[]])[0])} results"
)
return results
@@ -227,6 +318,28 @@ class SeekDBVectorDatabase(VectorDatabase):
self.ap.logger.info(f"Deleted embeddings from SeekDB collection '{collection}' with file_id: {file_id}")
async def delete_by_filter(self, collection: str, filter: Dict[str, Any]) -> int:
"""Delete vectors from the collection by metadata filter.
Args:
collection: Collection name
filter: Chroma-style ``where`` filter dict
"""
exists = await asyncio.to_thread(self.client.has_collection, collection)
if not exists:
self.ap.logger.warning(f"SeekDB collection '{collection}' not found for deletion")
return 0
if collection not in self._collections:
coll = await asyncio.to_thread(self.client.get_collection, collection, embedding_function=None)
self._collections[collection] = coll
else:
coll = self._collections[collection]
await asyncio.to_thread(coll.delete, where=filter)
self.ap.logger.info(f"Deleted embeddings from SeekDB collection '{collection}' by filter")
return 0 # SeekDB delete does not return a count
async def delete_collection(self, collection: str):
"""Delete the entire collection.

22
tests/unit_tests/plugin/test_plugin_component_filtering.py
View File

@@ -38,13 +38,11 @@ async def test_plugin_list_filter_by_component_kinds():
'manifest': {
'metadata': {
'author': 'author2',
'name': 'plugin_with_knowledge_retriever_only',
'name': 'plugin_with_knowledge_engine_only',
}
}
},
'components': [
{'manifest': {'manifest': {'kind': 'KnowledgeRetriever', 'metadata': {'name': 'retriever1'}}}}
],
'components': [{'manifest': {'manifest': {'kind': 'KnowledgeEngine', 'metadata': {'name': 'retriever1'}}}}],
},
{
'debug': False,
@@ -81,7 +79,7 @@ async def test_plugin_list_filter_by_component_kinds():
}
},
'components': [
{'manifest': {'manifest': {'kind': 'KnowledgeRetriever', 'metadata': {'name': 'retriever2'}}}},
{'manifest': {'manifest': {'kind': 'KnowledgeEngine', 'metadata': {'name': 'retriever2'}}}},
{'manifest': {'manifest': {'kind': 'Tool', 'metadata': {'name': 'tool2'}}}},
],
},
@@ -108,8 +106,8 @@ async def test_plugin_list_filter_by_component_kinds():
assert 'plugin_with_command' in plugin_names
assert 'plugin_with_event_listener' in plugin_names
assert 'plugin_with_mixed_components' in plugin_names
# Plugin with only KnowledgeRetriever should NOT be included
assert 'plugin_with_knowledge_retriever_only' not in plugin_names
# Plugin with only KnowledgeEngine should NOT be included
assert 'plugin_with_knowledge_engine_only' not in plugin_names
@pytest.mark.asyncio
@@ -150,9 +148,7 @@ async def test_plugin_list_filter_no_filter():
}
}
},
'components': [
{'manifest': {'manifest': {'kind': 'KnowledgeRetriever', 'metadata': {'name': 'retriever1'}}}}
],
'components': [{'manifest': {'manifest': {'kind': 'KnowledgeEngine', 'metadata': {'name': 'retriever1'}}}}],
},
]
@@ -189,7 +185,7 @@ async def test_plugin_list_filter_empty_result():
connector = PluginRuntimeConnector(mock_app, AsyncMock())
connector.handler = MagicMock()
# Mock plugin data - only KnowledgeRetriever plugins
# Mock plugin data - only KnowledgeEngine plugins
mock_plugins = [
{
'debug': False,
@@ -201,9 +197,7 @@ async def test_plugin_list_filter_empty_result():
}
}
},
'components': [
{'manifest': {'manifest': {'kind': 'KnowledgeRetriever', 'metadata': {'name': 'retriever1'}}}}
],
'components': [{'manifest': {'manifest': {'kind': 'KnowledgeEngine', 'metadata': {'name': 'retriever1'}}}}],
},
]

36
web/pnpm-lock.yaml generated
View File

@@ -508,6 +508,7 @@ packages:
resolution: {integrity: sha512-excjX8DfsIcJ10x1Kzr4RcWe1edC9PquDRRPx3YVCvQv+U5p7Yin2s32ftzikXojb1PIFc/9Mt28/y+iRklkrw==}
cpu: [arm64]
os: [linux]
libc: [glibc]
requiresBuild: true
dev: false
optional: true
@@ -516,6 +517,7 @@ packages:
resolution: {integrity: sha512-bFI7xcKFELdiNCVov8e44Ia4u2byA+l3XtsAj+Q8tfCwO6BQ8iDojYdvoPMqsKDkuoOo+X6HZA0s0q11ANMQ8A==}
cpu: [arm]
os: [linux]
libc: [glibc]
requiresBuild: true
dev: false
optional: true
@@ -524,6 +526,7 @@ packages:
resolution: {integrity: sha512-FMuvGijLDYG6lW+b/UvyilUWu5Ayu+3r2d1S8notiGCIyYU/76eig1UfMmkZ7vwgOrzKzlQbFSuQfgm7GYUPpA==}
cpu: [ppc64]
os: [linux]
libc: [glibc]
requiresBuild: true
dev: false
optional: true
@@ -532,6 +535,7 @@ packages:
resolution: {integrity: sha512-oVDbcR4zUC0ce82teubSm+x6ETixtKZBh/qbREIOcI3cULzDyb18Sr/Wcyx7NRQeQzOiHTNbZFF1UwPS2scyGA==}
cpu: [riscv64]
os: [linux]
libc: [glibc]
requiresBuild: true
dev: false
optional: true
@@ -540,6 +544,7 @@ packages:
resolution: {integrity: sha512-qmp9VrzgPgMoGZyPvrQHqk02uyjA0/QrTO26Tqk6l4ZV0MPWIW6LTkqOIov+J1yEu7MbFQaDpwdwJKhbJvuRxQ==}
cpu: [s390x]
os: [linux]
libc: [glibc]
requiresBuild: true
dev: false
optional: true
@@ -548,6 +553,7 @@ packages:
resolution: {integrity: sha512-tJxiiLsmHc9Ax1bz3oaOYBURTXGIRDODBqhveVHonrHJ9/+k89qbLl0bcJns+e4t4rvaNBxaEZsFtSfAdquPrw==}
cpu: [x64]
os: [linux]
libc: [glibc]
requiresBuild: true
dev: false
optional: true
@@ -556,6 +562,7 @@ packages:
resolution: {integrity: sha512-FVQHuwx1IIuNow9QAbYUzJ+En8KcVm9Lk5+uGUQJHaZmMECZmOlix9HnH7n1TRkXMS0pGxIJokIVB9SuqZGGXw==}
cpu: [arm64]
os: [linux]
libc: [musl]
requiresBuild: true
dev: false
optional: true
@@ -564,6 +571,7 @@ packages:
resolution: {integrity: sha512-+LpyBk7L44ZIXwz/VYfglaX/okxezESc6UxDSoyo2Ks6Jxc4Y7sGjpgU9s4PMgqgjj1gZCylTieNamqA1MF7Dg==}
cpu: [x64]
os: [linux]
libc: [musl]
requiresBuild: true
dev: false
optional: true
@@ -573,6 +581,7 @@ packages:
engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0}
cpu: [arm64]
os: [linux]
libc: [glibc]
requiresBuild: true
optionalDependencies:
'@img/sharp-libvips-linux-arm64': 1.2.4
@@ -584,6 +593,7 @@ packages:
engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0}
cpu: [arm]
os: [linux]
libc: [glibc]
requiresBuild: true
optionalDependencies:
'@img/sharp-libvips-linux-arm': 1.2.4
@@ -595,6 +605,7 @@ packages:
engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0}
cpu: [ppc64]
os: [linux]
libc: [glibc]
requiresBuild: true
optionalDependencies:
'@img/sharp-libvips-linux-ppc64': 1.2.4
@@ -606,6 +617,7 @@ packages:
engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0}
cpu: [riscv64]
os: [linux]
libc: [glibc]
requiresBuild: true
optionalDependencies:
'@img/sharp-libvips-linux-riscv64': 1.2.4
@@ -617,6 +629,7 @@ packages:
engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0}
cpu: [s390x]
os: [linux]
libc: [glibc]
requiresBuild: true
optionalDependencies:
'@img/sharp-libvips-linux-s390x': 1.2.4
@@ -628,6 +641,7 @@ packages:
engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0}
cpu: [x64]
os: [linux]
libc: [glibc]
requiresBuild: true
optionalDependencies:
'@img/sharp-libvips-linux-x64': 1.2.4
@@ -639,6 +653,7 @@ packages:
engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0}
cpu: [arm64]
os: [linux]
libc: [musl]
requiresBuild: true
optionalDependencies:
'@img/sharp-libvips-linuxmusl-arm64': 1.2.4
@@ -650,6 +665,7 @@ packages:
engines: {node: ^18.17.0 || ^20.3.0 || >=21.0.0}
cpu: [x64]
os: [linux]
libc: [musl]
requiresBuild: true
optionalDependencies:
'@img/sharp-libvips-linuxmusl-x64': 1.2.4
@@ -766,6 +782,7 @@ packages:
engines: {node: '>= 10'}
cpu: [arm64]
os: [linux]
libc: [glibc]
requiresBuild: true
dev: false
optional: true
@@ -775,6 +792,7 @@ packages:
engines: {node: '>= 10'}
cpu: [arm64]
os: [linux]
libc: [musl]
requiresBuild: true
dev: false
optional: true
@@ -784,6 +802,7 @@ packages:
engines: {node: '>= 10'}
cpu: [x64]
os: [linux]
libc: [glibc]
requiresBuild: true
dev: false
optional: true
@@ -793,6 +812,7 @@ packages:
engines: {node: '>= 10'}
cpu: [x64]
os: [linux]
libc: [musl]
requiresBuild: true
dev: false
optional: true
@@ -1892,6 +1912,7 @@ packages:
engines: {node: '>= 10'}
cpu: [arm64]
os: [linux]
libc: [glibc]
requiresBuild: true
dev: false
optional: true
@@ -1901,6 +1922,7 @@ packages:
engines: {node: '>= 10'}
cpu: [arm64]
os: [linux]
libc: [musl]
requiresBuild: true
dev: false
optional: true
@@ -1910,6 +1932,7 @@ packages:
engines: {node: '>= 10'}
cpu: [x64]
os: [linux]
libc: [glibc]
requiresBuild: true
dev: false
optional: true
@@ -1919,6 +1942,7 @@ packages:
engines: {node: '>= 10'}
cpu: [x64]
os: [linux]
libc: [musl]
requiresBuild: true
dev: false
optional: true
@@ -2334,6 +2358,7 @@ packages:
resolution: {integrity: sha512-34gw7PjDGB9JgePJEmhEqBhWvCiiWCuXsL9hYphDF7crW7UgI05gyBAi6MF58uGcMOiOqSJ2ybEeCvHcq0BCmQ==}
cpu: [arm64]
os: [linux]
libc: [glibc]
requiresBuild: true
dev: true
optional: true
@@ -2342,6 +2367,7 @@ packages:
resolution: {integrity: sha512-RyMIx6Uf53hhOtJDIamSbTskA99sPHS96wxVE/bJtePJJtpdKGXO1wY90oRdXuYOGOTuqjT8ACccMc4K6QmT3w==}
cpu: [arm64]
os: [linux]
libc: [musl]
requiresBuild: true
dev: true
optional: true
@@ -2350,6 +2376,7 @@ packages:
resolution: {integrity: sha512-D8Vae74A4/a+mZH0FbOkFJL9DSK2R6TFPC9M+jCWYia/q2einCubX10pecpDiTmkJVUH+y8K3BZClycD8nCShA==}
cpu: [ppc64]
os: [linux]
libc: [glibc]
requiresBuild: true
dev: true
optional: true
@@ -2358,6 +2385,7 @@ packages:
resolution: {integrity: sha512-frxL4OrzOWVVsOc96+V3aqTIQl1O2TjgExV4EKgRY09AJ9leZpEg8Ak9phadbuX0BA4k8U5qtvMSQQGGmaJqcQ==}
cpu: [riscv64]
os: [linux]
libc: [glibc]
requiresBuild: true
dev: true
optional: true
@@ -2366,6 +2394,7 @@ packages:
resolution: {integrity: sha512-mJ5vuDaIZ+l/acv01sHoXfpnyrNKOk/3aDoEdLO/Xtn9HuZlDD6jKxHlkN8ZhWyLJsRBxfv9GYM2utQ1SChKew==}
cpu: [riscv64]
os: [linux]
libc: [musl]
requiresBuild: true
dev: true
optional: true
@@ -2374,6 +2403,7 @@ packages:
resolution: {integrity: sha512-kELo8ebBVtb9sA7rMe1Cph4QHreByhaZ2QEADd9NzIQsYNQpt9UkM9iqr2lhGr5afh885d/cB5QeTXSbZHTYPg==}
cpu: [s390x]
os: [linux]
libc: [glibc]
requiresBuild: true
dev: true
optional: true
@@ -2382,6 +2412,7 @@ packages:
resolution: {integrity: sha512-C3ZAHugKgovV5YvAMsxhq0gtXuwESUKc5MhEtjBpLoHPLYM+iuwSj3lflFwK3DPm68660rZ7G8BMcwSro7hD5w==}
cpu: [x64]
os: [linux]
libc: [glibc]
requiresBuild: true
dev: true
optional: true
@@ -2390,6 +2421,7 @@ packages:
resolution: {integrity: sha512-rV0YSoyhK2nZ4vEswT/QwqzqQXw5I6CjoaYMOX0TqBlWhojUf8P94mvI7nuJTeaCkkds3QE4+zS8Ko+GdXuZtA==}
cpu: [x64]
os: [linux]
libc: [musl]
requiresBuild: true
dev: true
optional: true
@@ -4424,6 +4456,7 @@ packages:
engines: {node: '>= 12.0.0'}
cpu: [arm64]
os: [linux]
libc: [glibc]
requiresBuild: true
dev: false
optional: true
@@ -4433,6 +4466,7 @@ packages:
engines: {node: '>= 12.0.0'}
cpu: [arm64]
os: [linux]
libc: [musl]
requiresBuild: true
dev: false
optional: true
@@ -4442,6 +4476,7 @@ packages:
engines: {node: '>= 12.0.0'}
cpu: [x64]
os: [linux]
libc: [glibc]
requiresBuild: true
dev: false
optional: true
@@ -4451,6 +4486,7 @@ packages:
engines: {node: '>= 12.0.0'}
cpu: [x64]
os: [linux]
libc: [musl]
requiresBuild: true
dev: false
optional: true

1
web/src/app/home/bots/components/bot-form/BotForm.tsx
View File

@@ -319,6 +319,7 @@ export default function BotForm({
required: item.required,
type: parseDynamicFormItemType(item.type),
options: item.options,
show_if: item.show_if,
}),
),
);

113
web/src/app/home/components/dynamic-form/DynamicFormComponent.tsx
View File

@@ -13,20 +13,26 @@ import {
import DynamicFormItemComponent from '@/app/home/components/dynamic-form/DynamicFormItemComponent';
import { useCallback, useEffect, useRef } from 'react';
import { extractI18nObject } from '@/i18n/I18nProvider';
import { useTranslation } from 'react-i18next';
export default function DynamicFormComponent({
itemConfigList,
onSubmit,
initialValues,
onFileUploaded,
isEditing,
externalDependentValues,
}: {
itemConfigList: IDynamicFormItemSchema[];
onSubmit?: (val: object) => unknown;
initialValues?: Record<string, object>;
onFileUploaded?: (fileKey: string) => void;
isEditing?: boolean;
externalDependentValues?: Record<string, unknown>;
}) {
const isInitialMount = useRef(true);
const previousInitialValues = useRef(initialValues);
const { t } = useTranslation();
// 根据 itemConfigList 动态生成 zod schema
const formSchema = z.object(
@@ -55,6 +61,9 @@ export default function DynamicFormComponent({
case 'llm-model-selector':
fieldSchema = z.string();
break;
case 'embedding-model-selector':
fieldSchema = z.string();
break;
case 'knowledge-base-selector':
fieldSchema = z.string();
break;
@@ -81,7 +90,9 @@ export default function DynamicFormComponent({
(fieldSchema instanceof z.ZodString ||
fieldSchema instanceof z.ZodArray)
) {
fieldSchema = fieldSchema.min(1, { message: '此字段为必填项' });
fieldSchema = fieldSchema.min(1, {
message: t('common.fieldRequired'),
});
}
return {
@@ -141,6 +152,9 @@ export default function DynamicFormComponent({
}
}, [initialValues, form, itemConfigList]);
// Get reactive form values for conditional rendering
const watchedValues = form.watch();
// Stable ref for onSubmit to avoid re-triggering the effect when the
// parent passes a new closure on every render.
const onSubmitRef = useRef(onSubmit);
@@ -183,34 +197,75 @@ export default function DynamicFormComponent({
return (
<Form {...form}>
<div className="space-y-4">
{itemConfigList.map((config) => (
<FormField
key={config.id}
control={form.control}
name={config.name as keyof FormValues}
render={({ field }) => (
<FormItem>
<FormLabel>
{extractI18nObject(config.label)}{' '}
{config.required && <span className="text-red-500">*</span>}
</FormLabel>
<FormControl>
<DynamicFormItemComponent
config={config}
field={field}
onFileUploaded={onFileUploaded}
/>
</FormControl>
{config.description && (
<p className="text-sm text-muted-foreground">
{extractI18nObject(config.description)}
</p>
)}
<FormMessage />
</FormItem>
)}
/>
))}
{itemConfigList.map((config) => {
if (config.show_if) {
const dependValue =
watchedValues[
config.show_if.field as keyof typeof watchedValues
] !== undefined
? watchedValues[
config.show_if.field as keyof typeof watchedValues
]
: externalDependentValues?.[config.show_if.field];
if (
config.show_if.operator === 'eq' &&
dependValue !== config.show_if.value
) {
return null;
}
if (
config.show_if.operator === 'neq' &&
dependValue === config.show_if.value
) {
return null;
}
if (
config.show_if.operator === 'in' &&
Array.isArray(config.show_if.value) &&
!config.show_if.value.includes(dependValue)
) {
return null;
}
}
// All fields are disabled when editing (creation_settings are immutable)
const isFieldDisabled = !!isEditing;
return (
<FormField
key={config.id}
control={form.control}
name={config.name as keyof FormValues}
render={({ field }) => (
<FormItem>
<FormLabel>
{extractI18nObject(config.label)}{' '}
{config.required && <span className="text-red-500">*</span>}
</FormLabel>
<FormControl>
<div
className={
isFieldDisabled ? 'pointer-events-none opacity-60' : ''
}
>
<DynamicFormItemComponent
config={config}
field={field}
onFileUploaded={onFileUploaded}
/>
</div>
</FormControl>
{config.description && (
<p className="text-sm text-muted-foreground">
{extractI18nObject(config.description)}
</p>
)}
<FormMessage />
</FormItem>
)}
/>
);
})}
</div>
</Form>
);

236
web/src/app/home/components/dynamic-form/DynamicFormItemComponent.tsx
View File

@@ -22,8 +22,7 @@ import {
LLMModel,
Bot,
KnowledgeBase,
ExternalKnowledgeBase,
ApiRespPluginSystemStatus,
EmbeddingModel,
} from '@/app/infra/entities/api';
import { toast } from 'sonner';
import { useTranslation } from 'react-i18next';
@@ -51,16 +50,12 @@ export default function DynamicFormItemComponent({
onFileUploaded?: (fileKey: string) => void;
}) {
const [llmModels, setLlmModels] = useState<LLMModel[]>([]);
const [embeddingModels, setEmbeddingModels] = useState<EmbeddingModel[]>([]);
const [knowledgeBases, setKnowledgeBases] = useState<KnowledgeBase[]>([]);
const [externalKnowledgeBases, setExternalKnowledgeBases] = useState<
ExternalKnowledgeBase[]
>([]);
const [bots, setBots] = useState<Bot[]>([]);
const [uploading, setUploading] = useState<boolean>(false);
const [kbDialogOpen, setKbDialogOpen] = useState(false);
const [tempSelectedKBIds, setTempSelectedKBIds] = useState<string[]>([]);
const [pluginSystemStatus, setPluginSystemStatus] =
useState<ApiRespPluginSystemStatus | null>(null);
const { t } = useTranslation();
const handleFileUpload = async (file: File): Promise<IFileConfig | null> => {
@@ -111,7 +106,20 @@ export default function DynamicFormItemComponent({
setLlmModels(models);
})
.catch((err) => {
toast.error('Failed to get LLM model list: ' + err.msg);
toast.error(t('models.getModelListError') + err.msg);
});
}
}, [config.type]);
useEffect(() => {
if (config.type === DynamicFormItemType.EMBEDDING_MODEL_SELECTOR) {
httpClient
.getProviderEmbeddingModels()
.then((resp) => {
setEmbeddingModels(resp.models);
})
.catch((err) => {
toast.error(t('embedding.getModelListError') + err.msg);
});
}
}, [config.type]);
@@ -127,39 +135,11 @@ export default function DynamicFormItemComponent({
setKnowledgeBases(resp.bases);
})
.catch((err) => {
toast.error('Failed to get knowledge base list: ' + err.msg);
});
// Fetch plugin system status
httpClient
.getPluginSystemStatus()
.then((status) => {
setPluginSystemStatus(status);
})
.catch((err) => {
console.error('Failed to get plugin system status:', err);
toast.error(t('knowledge.getKnowledgeBaseListError') + err.msg);
});
}
}, [config.type]);
useEffect(() => {
if (
(config.type === DynamicFormItemType.KNOWLEDGE_BASE_SELECTOR ||
config.type === DynamicFormItemType.KNOWLEDGE_BASE_MULTI_SELECTOR) &&
pluginSystemStatus?.is_enable &&
pluginSystemStatus?.is_connected
) {
httpClient
.getExternalKnowledgeBases()
.then((resp) => {
setExternalKnowledgeBases(resp.bases);
})
.catch((err) => {
console.error('Failed to get external knowledge base list:', err);
});
}
}, [config.type, pluginSystemStatus]);
useEffect(() => {
if (config.type === DynamicFormItemType.BOT_SELECTOR) {
httpClient
@@ -168,7 +148,7 @@ export default function DynamicFormItemComponent({
setBots(resp.bots);
})
.catch((err) => {
toast.error('Failed to get bot list: ' + err.msg);
toast.error(t('bots.getBotListError') + err.msg);
});
}
}, [config.type]);
@@ -304,7 +284,56 @@ export default function DynamicFormItemComponent({
</Select>
);
case DynamicFormItemType.EMBEDDING_MODEL_SELECTOR:
// Group embedding models by provider
const groupedEmbeddingModels = embeddingModels.reduce(
(acc, model) => {
const providerName = model.provider?.name || 'Unknown';
if (!acc[providerName]) acc[providerName] = [];
acc[providerName].push(model);
return acc;
},
{} as Record<string, EmbeddingModel[]>,
);
return (
<Select value={field.value} onValueChange={field.onChange}>
<SelectTrigger className="bg-[#ffffff] dark:bg-[#2a2a2e]">
<SelectValue placeholder={t('knowledge.selectEmbeddingModel')} />
</SelectTrigger>
<SelectContent>
{Object.entries(groupedEmbeddingModels).map(
([providerName, models]) => (
<SelectGroup key={providerName}>
<SelectLabel>{providerName}</SelectLabel>
{models.map((model) => (
<SelectItem key={model.uuid} value={model.uuid}>
{model.name}
</SelectItem>
))}
</SelectGroup>
),
)}
</SelectContent>
</Select>
);
case DynamicFormItemType.KNOWLEDGE_BASE_SELECTOR:
// Group KBs by Knowledge Engine name
const kbsByEngine = knowledgeBases.reduce(
(acc, kb) => {
const engineName = kb.knowledge_engine?.name
? extractI18nObject(kb.knowledge_engine.name)
: t('knowledge.unknownEngine');
if (!acc[engineName]) {
acc[engineName] = [];
}
acc[engineName].push(kb);
return acc;
},
{} as Record<string, typeof knowledgeBases>,
);
return (
<Select value={field.value} onValueChange={field.onChange}>
<SelectTrigger className="bg-[#ffffff] dark:bg-[#2a2a2e]">
@@ -315,53 +344,45 @@ export default function DynamicFormItemComponent({
<SelectItem value="__none__">{t('knowledge.empty')}</SelectItem>
</SelectGroup>
{knowledgeBases.length > 0 && (
<SelectGroup>
<SelectLabel>{t('knowledge.builtIn')}</SelectLabel>
{knowledgeBases.map((base) => (
{Object.entries(kbsByEngine).map(([engineName, kbs]) => (
<SelectGroup key={engineName}>
<SelectLabel>{engineName}</SelectLabel>
{kbs.map((base) => (
<SelectItem key={base.uuid} value={base.uuid ?? ''}>
{base.name}
</SelectItem>
))}
</SelectGroup>
)}
{externalKnowledgeBases.length > 0 && (
<SelectGroup>
<SelectLabel>{t('knowledge.external')}</SelectLabel>
{externalKnowledgeBases.map((base) => (
<SelectItem key={base.uuid} value={base.uuid ?? ''}>
<div className="flex items-center gap-2">
<img
src={httpClient.getPluginIconURL(
base.plugin_author,
base.plugin_name,
)}
alt="plugin icon"
className="w-4 h-4 rounded-[8%] flex-shrink-0"
/>
<span>{base.name}</span>
</div>
</SelectItem>
))}
</SelectGroup>
)}
))}
</SelectContent>
</Select>
);
case DynamicFormItemType.KNOWLEDGE_BASE_MULTI_SELECTOR:
// Group KBs by Knowledge Engine name for multi-selector
const multiKbsByEngine = knowledgeBases.reduce(
(acc, kb) => {
const engineName = kb.knowledge_engine?.name
? extractI18nObject(kb.knowledge_engine.name)
: t('knowledge.unknownEngine');
if (!acc[engineName]) {
acc[engineName] = [];
}
acc[engineName].push(kb);
return acc;
},
{} as Record<string, typeof knowledgeBases>,
);
return (
<>
<div className="space-y-2">
{field.value && field.value.length > 0 ? (
<div className="space-y-2">
{field.value.map((kbId: string) => {
const kb = knowledgeBases.find((base) => base.uuid === kbId);
const externalKb = externalKnowledgeBases.find(
const currentKb = knowledgeBases.find(
(base) => base.uuid === kbId,
);
const currentKb = kb || externalKb;
if (!currentKb) return null;
return (
@@ -370,18 +391,17 @@ export default function DynamicFormItemComponent({
className="flex items-center justify-between rounded-lg border p-3 hover:bg-accent"
>
<div className="flex items-center gap-2 flex-1">
{externalKb && (
<img
src={httpClient.getPluginIconURL(
externalKb.plugin_author,
externalKb.plugin_name,
)}
alt="plugin icon"
className="w-8 h-8 rounded-[8%] flex-shrink-0"
/>
)}
<div className="flex-1 min-w-0">
<div className="font-medium">{currentKb.name}</div>
<div className="font-medium flex items-center gap-2">
{currentKb.name}
{currentKb.knowledge_engine?.name && (
<span className="text-xs px-2 py-0.5 rounded-full bg-purple-100 text-purple-700 dark:bg-purple-900 dark:text-purple-300">
{extractI18nObject(
currentKb.knowledge_engine.name,
)}
</span>
)}
</div>
{currentKb.description && (
<div className="text-sm text-muted-foreground">
{currentKb.description}
@@ -435,13 +455,12 @@ export default function DynamicFormItemComponent({
<DialogTitle>{t('knowledge.selectKnowledgeBases')}</DialogTitle>
</DialogHeader>
<div className="flex-1 overflow-y-auto space-y-4 pr-2">
{/* Built-in Knowledge Bases */}
{knowledgeBases.length > 0 && (
<div className="space-y-2">
{Object.entries(multiKbsByEngine).map(([engineName, kbs]) => (
<div key={engineName} className="space-y-2">
<div className="text-sm font-semibold text-muted-foreground px-2">
{t('knowledge.builtIn')}
{engineName}
</div>
{knowledgeBases.map((base) => {
{kbs.map((base) => {
const isSelected = tempSelectedKBIds.includes(
base.uuid ?? '',
);
@@ -474,56 +493,7 @@ export default function DynamicFormItemComponent({
);
})}
</div>
)}
{/* External Knowledge Bases */}
{externalKnowledgeBases.length > 0 && (
<div className="space-y-2">
<div className="text-sm font-semibold text-muted-foreground px-2">
{t('knowledge.external')}
</div>
{externalKnowledgeBases.map((base) => {
const isSelected = tempSelectedKBIds.includes(
base.uuid ?? '',
);
return (
<div
key={base.uuid}
className="flex items-center gap-3 rounded-lg border p-3 hover:bg-accent cursor-pointer"
onClick={() => {
const kbId = base.uuid ?? '';
setTempSelectedKBIds((prev) =>
prev.includes(kbId)
? prev.filter((id) => id !== kbId)
: [...prev, kbId],
);
}}
>
<Checkbox
checked={isSelected}
aria-label={`Select ${base.name}`}
/>
<img
src={httpClient.getPluginIconURL(
base.plugin_author,
base.plugin_name,
)}
alt="plugin icon"
className="w-8 h-8 rounded-[8%] flex-shrink-0"
/>
<div className="flex-1">
<div className="font-medium">{base.name}</div>
{base.description && (
<div className="text-sm text-muted-foreground">
{base.description}
</div>
)}
</div>
</div>
);
})}
</div>
)}
))}
</div>
<DialogFooter>
<Button

3
web/src/app/home/components/dynamic-form/DynamicFormItemConfig.ts
View File

@@ -2,6 +2,7 @@ import {
IDynamicFormItemSchema,
DynamicFormItemType,
IDynamicFormItemOption,
IShowIfCondition,
} from '@/app/infra/entities/form/dynamic';
import { I18nObject } from '@/app/infra/entities/common';
@@ -14,6 +15,7 @@ export class DynamicFormItemConfig implements IDynamicFormItemSchema {
type: DynamicFormItemType;
description?: I18nObject;
options?: IDynamicFormItemOption[];
show_if?: IShowIfCondition;
constructor(params: IDynamicFormItemSchema) {
this.id = params.id;
@@ -24,6 +26,7 @@ export class DynamicFormItemConfig implements IDynamicFormItemSchema {
this.type = params.type;
this.description = params.description;
this.options = params.options;
this.show_if = params.show_if;
}
}

178
web/src/app/home/knowledge/KBDetailDialog.tsx
View File

@@ -21,18 +21,16 @@ import {
import { Button } from '@/components/ui/button';
import { useTranslation } from 'react-i18next';
import { httpClient } from '@/app/infra/http/HttpClient';
// import { KnowledgeBase } from '@/app/infra/entities/api';
import { KnowledgeBase } from '@/app/infra/entities/api';
import { toast } from 'sonner';
import KBForm from '@/app/home/knowledge/components/kb-form/KBForm';
import KBDoc from '@/app/home/knowledge/components/kb-docs/KBDoc';
import KBRetrieve from '@/app/home/knowledge/components/kb-retrieve/KBRetrieve';
import ExternalKBForm from '@/app/home/knowledge/components/external-kb-form/ExternalKBForm';
import ExternalKBRetrieve from '@/app/home/knowledge/components/kb-retrieve/ExternalKBRetrieve';
import KBRetrieveGeneric from '@/app/home/knowledge/components/kb-retrieve/KBRetrieveGeneric';
interface KBDetailDialogProps {
open: boolean;
onOpenChange: (open: boolean) => void;
kbId?: string;
kbType: 'builtin' | 'external';
onFormCancel: () => void;
onKbDeleted: () => void;
onNewKbCreated: (kbId: string) => void;
@@ -43,7 +41,6 @@ export default function KBDetailDialog({
open,
onOpenChange,
kbId: propKbId,
kbType,
onFormCancel,
onKbDeleted,
onNewKbCreated,
@@ -53,13 +50,39 @@ export default function KBDetailDialog({
const [kbId, setKbId] = useState<string | undefined>(propKbId);
const [activeMenu, setActiveMenu] = useState('metadata');
const [showDeleteConfirm, setShowDeleteConfirm] = useState(false);
const [kbInfo, setKbInfo] = useState<KnowledgeBase | null>(null);
useEffect(() => {
setKbId(propKbId);
setActiveMenu('metadata');
if (propKbId) {
loadKbInfo(propKbId);
} else {
setKbInfo(null);
}
}, [propKbId, open]);
// Build menu based on KB type
async function loadKbInfo(id: string) {
try {
const resp = await httpClient.getKnowledgeBase(id);
setKbInfo(resp.base);
} catch (e) {
console.error('Failed to load KB info:', e);
toast.error(t('knowledge.loadKnowledgeBaseFailed'));
}
}
// Check if this KB supports document management
const hasDocumentCapability = (): boolean => {
if (!kbInfo || !kbInfo.knowledge_engine) {
return false;
}
return (
kbInfo.knowledge_engine.capabilities?.includes('doc_ingestion') ?? false
);
};
// Build menu based on KB capabilities
const menu = [
{
key: 'metadata',
@@ -74,8 +97,8 @@ export default function KBDetailDialog({
</svg>
),
},
// Only show documents for builtin KB
...(kbType === 'builtin'
// Show documents only if capability is present
...(hasDocumentCapability()
? [
{
key: 'documents',
@@ -107,66 +130,49 @@ export default function KBDetailDialog({
},
];
const confirmDelete = () => {
const deletePromise =
kbType === 'builtin'
? httpClient.deleteKnowledgeBase(kbId ?? '')
: httpClient.deleteExternalKnowledgeBase(kbId ?? '');
deletePromise.then(() => {
const confirmDelete = async () => {
try {
await httpClient.deleteKnowledgeBase(kbId ?? '');
onKbDeleted();
});
setShowDeleteConfirm(false);
} catch (e) {
console.error('Failed to delete KB:', e);
toast.error(t('knowledge.deleteKnowledgeBaseFailed'));
} finally {
setShowDeleteConfirm(false);
}
};
// Retrieve function
const retrieveFunction = async (id: string, query: string) => {
return await httpClient.retrieveKnowledgeBase(id, query);
};
if (!kbId) {
// new kb
// New KB creation
return (
<Dialog open={open} onOpenChange={onOpenChange}>
<DialogContent className="overflow-hidden p-0 !max-w-[40vw] max-h-[70vh] flex">
<main className="flex flex-1 flex-col h-[70vh]">
<DialogHeader className="px-6 pt-6 pb-4 shrink-0">
<DialogTitle>
{kbType === 'builtin'
? t('knowledge.createKnowledgeBase')
: t('knowledge.addExternal')}
</DialogTitle>
<DialogTitle>{t('knowledge.createKnowledgeBase')}</DialogTitle>
</DialogHeader>
<div className="flex-1 overflow-y-auto px-6 pb-6">
{kbType === 'builtin' ? (
<KBForm
initKbId={undefined}
onNewKbCreated={onNewKbCreated}
onKbUpdated={onKbUpdated}
/>
) : (
<ExternalKBForm
initKBId={undefined}
onFormSubmit={() => onOpenChange(false)}
onKBDeleted={() => {}}
onNewKBCreated={onNewKbCreated}
/>
)}
<KBForm
initKbId={undefined}
onNewKbCreated={onNewKbCreated}
onKbUpdated={onKbUpdated}
/>
</div>
{activeMenu === 'metadata' && (
<DialogFooter className="px-6 py-4 border-t shrink-0">
<div className="flex justify-end gap-2">
<Button
type="submit"
form={kbType === 'builtin' ? 'kb-form' : 'external-kb-form'}
>
{t('common.save')}
</Button>
<Button
type="button"
variant="outline"
onClick={onFormCancel}
>
{t('common.cancel')}
</Button>
</div>
</DialogFooter>
)}
<DialogFooter className="px-6 py-4 border-t shrink-0">
<div className="flex justify-end gap-2">
<Button type="submit" form="kb-form">
{t('common.save')}
</Button>
<Button type="button" variant="outline" onClick={onFormCancel}>
{t('common.cancel')}
</Button>
</div>
</DialogFooter>
</main>
</DialogContent>
</Dialog>
@@ -205,7 +211,7 @@ export default function KBDetailDialog({
</SidebarGroup>
</SidebarContent>
</Sidebar>
<main className="flex flex-1 flex-col h-[75vh]">
<main className="flex flex-1 flex-col h-[75vh] min-w-0 overflow-x-hidden">
<DialogHeader className="px-6 pt-6 pb-4 shrink-0">
<DialogTitle>
{activeMenu === 'metadata'
@@ -216,33 +222,28 @@ export default function KBDetailDialog({
</DialogTitle>
</DialogHeader>
<div className="flex-1 overflow-y-auto px-6 pb-6">
{activeMenu === 'metadata' &&
(kbType === 'builtin' ? (
<KBForm
initKbId={kbId}
onNewKbCreated={onNewKbCreated}
onKbUpdated={onKbUpdated}
/>
) : (
<ExternalKBForm
initKBId={kbId}
onFormSubmit={() => onOpenChange(false)}
onKBDeleted={() => {
onKbDeleted();
onOpenChange(false);
}}
onNewKBCreated={onNewKbCreated}
/>
))}
{activeMenu === 'documents' && kbType === 'builtin' && (
<KBDoc kbId={kbId} />
{activeMenu === 'metadata' && (
<KBForm
initKbId={kbId}
onNewKbCreated={onNewKbCreated}
onKbUpdated={onKbUpdated}
/>
)}
{activeMenu === 'documents' && hasDocumentCapability() && (
<KBDoc
kbId={kbId}
ragEngineName={kbInfo?.knowledge_engine?.name}
ragEngineCapabilities={
kbInfo?.knowledge_engine?.capabilities
}
/>
)}
{activeMenu === 'retrieve' && (
<KBRetrieveGeneric
kbId={kbId}
retrieveFunction={retrieveFunction}
/>
)}
{activeMenu === 'retrieve' &&
(kbType === 'builtin' ? (
<KBRetrieve kbId={kbId} />
) : (
<ExternalKBRetrieve kbId={kbId} />
))}
</div>
{activeMenu === 'metadata' && (
<DialogFooter className="px-6 py-4 border-t shrink-0">
@@ -254,12 +255,7 @@ export default function KBDetailDialog({
>
{t('common.delete')}
</Button>
<Button
type="submit"
form={
kbType === 'builtin' ? 'kb-form' : 'external-kb-form'
}
>
<Button type="submit" form="kb-form">
{t('common.save')}
</Button>
<Button
@@ -277,7 +273,7 @@ export default function KBDetailDialog({
</DialogContent>
</Dialog>
{/* 删除确认对话框 */}
{/* Delete confirmation dialog */}
<Dialog open={showDeleteConfirm} onOpenChange={setShowDeleteConfirm}>
<DialogContent>
<DialogHeader>

59
web/src/app/home/knowledge/components/external-kb-card/ExternalKBCard.tsx
View File

@@ -1,59 +0,0 @@
import { ExternalKBCardVO } from '@/app/home/knowledge/components/external-kb-card/ExternalKBCardVO';
import { useTranslation } from 'react-i18next';
import styles from '../kb-card/KBCard.module.css';
import { httpClient } from '@/app/infra/http/HttpClient';
export default function ExternalKBCard({
kbCardVO,
}: {
kbCardVO: ExternalKBCardVO;
}) {
const { t } = useTranslation();
return (
<div className={`${styles.cardContainer}`}>
<div className={`${styles.basicInfoContainer}`}>
<div className={`${styles.iconBasicInfoContainer}`}>
{/* Emoji with plugin icon badge */}
<div className="relative">
<div className={`${styles.iconEmoji}`}>
{kbCardVO.emoji || '🔗'}
</div>
{/* Plugin icon badge at bottom right */}
<img
src={httpClient.getPluginIconURL(
kbCardVO.pluginAuthor,
kbCardVO.pluginName,
)}
alt="plugin icon"
className="absolute -bottom-1 -right-1 w-5 h-5 rounded-[20%]"
/>
</div>
<div className={`${styles.basicInfoNameContainer}`}>
<div className={`${styles.basicInfoNameText} ${styles.bigText}`}>
{kbCardVO.name}
</div>
<div className={`${styles.basicInfoDescriptionText}`}>
{kbCardVO.description}
</div>
</div>
</div>
<div className={`${styles.basicInfoLastUpdatedTimeContainer}`}>
<svg
className={`${styles.basicInfoUpdateTimeIcon}`}
xmlns="http://www.w3.org/2000/svg"
viewBox="0 0 24 24"
fill="currentColor"
>
<path d="M12 22C6.47715 22 2 17.5228 2 12C2 6.47715 6.47715 2 12 2C17.5228 2 22 6.47715 22 12C22 17.5228 17.5228 22 12 22ZM12 20C16.4183 20 20 16.4183 20 12C20 7.58172 16.4183 4 12 4C7.58172 4 4 7.58172 4 12C4 16.4183 7.58172 20 12 20ZM13 12H17V14H11V7H13V12Z"></path>
</svg>
<div className={`${styles.basicInfoUpdateTimeText}`}>
{t('knowledge.updateTime')}
{kbCardVO.lastUpdatedTimeAgo}
</div>
</div>
</div>
</div>
);
}

43
web/src/app/home/knowledge/components/external-kb-card/ExternalKBCardVO.ts
View File

@@ -1,43 +0,0 @@
export class ExternalKBCardVO {
id: string;
name: string;
description: string;
emoji?: string;
retrieverName: string;
retrieverConfig: Record<string, unknown>;
lastUpdatedTimeAgo: string;
pluginAuthor: string;
pluginName: string;
constructor({
id,
name,
description,
emoji,
retrieverName,
retrieverConfig,
lastUpdatedTimeAgo,
pluginAuthor,
pluginName,
}: {
id: string;
name: string;
description: string;
emoji?: string;
retrieverName: string;
retrieverConfig: Record<string, unknown>;
lastUpdatedTimeAgo: string;
pluginAuthor: string;
pluginName: string;
}) {
this.id = id;
this.name = name;
this.description = description;
this.emoji = emoji;
this.retrieverName = retrieverName;
this.retrieverConfig = retrieverConfig;
this.lastUpdatedTimeAgo = lastUpdatedTimeAgo;
this.pluginAuthor = pluginAuthor;
this.pluginName = pluginName;
}
}

593
web/src/app/home/knowledge/components/external-kb-form/ExternalKBForm.tsx
View File

@@ -1,593 +0,0 @@
import { useEffect, useState } from 'react';
import { zodResolver } from '@hookform/resolvers/zod';
import { useForm } from 'react-hook-form';
import { z } from 'zod';
import { toast } from 'sonner';
import { useTranslation } from 'react-i18next';
import { UUID } from 'uuidjs';
import {
DynamicFormItemConfig,
getDefaultValues,
parseDynamicFormItemType,
} from '@/app/home/components/dynamic-form/DynamicFormItemConfig';
import { IDynamicFormItemSchema } from '@/app/infra/entities/form/dynamic';
import DynamicFormComponent from '@/app/home/components/dynamic-form/DynamicFormComponent';
import { httpClient } from '@/app/infra/http/HttpClient';
import { ExternalKnowledgeBase } from '@/app/infra/entities/api';
import EmojiPicker from '@/components/ui/emoji-picker';
import {
Dialog,
DialogContent,
DialogDescription,
DialogHeader,
DialogTitle,
DialogFooter,
} from '@/components/ui/dialog';
import { Button } from '@/components/ui/button';
import {
Form,
FormControl,
FormField,
FormItem,
FormLabel,
FormMessage,
} from '@/components/ui/form';
import { Input } from '@/components/ui/input';
import {
Select,
SelectContent,
SelectGroup,
SelectItem,
SelectTrigger,
SelectValue,
} from '@/components/ui/select';
import {
HoverCard,
HoverCardContent,
HoverCardTrigger,
} from '@/components/ui/hover-card';
import { extractI18nObject } from '@/i18n/I18nProvider';
import { I18nObject } from '@/app/infra/entities/common';
// Form schema
const getFormSchema = (t: (key: string) => string) =>
z.object({
name: z.string().min(1, { message: t('knowledge.nameRequired') }),
description: z.string().optional(),
emoji: z.string().optional(),
plugin_author: z.string().min(1, { message: 'Please select a retriever' }),
plugin_name: z.string().min(1, { message: 'Please select a retriever' }),
retriever_name: z.string().min(1, { message: 'Please select a retriever' }),
retriever_config: z.record(z.string(), z.any()),
});
// Retriever information interface
interface RetrieverInfo {
plugin_author: string;
plugin_name: string;
retriever_name: string;
retriever_description: I18nObject;
manifest: {
manifest?: {
metadata?: {
label?: I18nObject;
description?: I18nObject;
};
spec?: {
config?: IDynamicFormItemSchema[];
};
};
};
}
interface ExternalKBFormProps {
initKBId?: string;
onFormSubmit: (value: z.infer<ReturnType<typeof getFormSchema>>) => void;
onKBDeleted: () => void;
onNewKBCreated: (kbId: string) => void;
}
export default function ExternalKBForm({
initKBId,
onFormSubmit,
onKBDeleted,
onNewKBCreated,
}: ExternalKBFormProps) {
const { t } = useTranslation();
const formSchema = getFormSchema(t);
// Form setup
const form = useForm<z.infer<typeof formSchema>>({
resolver: zodResolver(formSchema),
defaultValues: {
name: '',
description: '',
emoji: '🔗',
plugin_author: '',
plugin_name: '',
retriever_name: '',
retriever_config: {},
},
});
// State management
const [showDeleteConfirmModal, setShowDeleteConfirmModal] = useState(false);
const [availableRetrievers, setAvailableRetrievers] = useState<
RetrieverInfo[]
>([]);
const [retrieverNameToConfigMap, setRetrieverNameToConfigMap] = useState(
new Map<string, IDynamicFormItemSchema[]>(),
);
const [showDynamicForm, setShowDynamicForm] = useState<boolean>(false);
const [dynamicFormConfigList, setDynamicFormConfigList] = useState<
IDynamicFormItemSchema[]
>([]);
// Initialize form when initKBId changes
useEffect(() => {
loadFormData();
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [initKBId]);
/**
* Load form data: initialize retrievers list and load KB config if editing
*/
async function loadFormData() {
const configMap = await loadAvailableRetrievers();
if (initKBId) {
// Edit mode: load existing KB configuration
try {
const kbConfig = await loadKBConfig(initKBId);
// Set form values
form.setValue('name', kbConfig.name);
form.setValue('description', kbConfig.description || '');
form.setValue('emoji', kbConfig.emoji || '🔗');
form.setValue('plugin_author', kbConfig.plugin_author);
form.setValue('plugin_name', kbConfig.plugin_name);
form.setValue('retriever_name', kbConfig.retriever_name);
form.setValue('retriever_config', kbConfig.retriever_config);
// Load dynamic form for the selected retriever
const fullName = `${kbConfig.plugin_author}/${kbConfig.plugin_name}/${kbConfig.retriever_name}`;
loadDynamicFormConfig(fullName, configMap);
} catch (err) {
toast.error('Failed to load KB config: ' + (err as Error).message);
}
} else {
// Create mode: reset form
form.reset();
}
}
/**
* Load available retrievers from API and build config map
*/
async function loadAvailableRetrievers(): Promise<
Map<string, IDynamicFormItemSchema[]>
> {
const retrieversRes = await httpClient.listKnowledgeRetrievers();
setAvailableRetrievers((retrieversRes.retrievers || []) as RetrieverInfo[]);
// Build retriever name to config map
const configMap = new Map<string, IDynamicFormItemSchema[]>();
((retrieversRes.retrievers || []) as RetrieverInfo[]).forEach(
(retriever) => {
const fullName = `${retriever.plugin_author}/${retriever.plugin_name}/${retriever.retriever_name}`;
const configSchema = retriever.manifest?.manifest?.spec?.config || [];
configMap.set(
fullName,
configSchema.map(
(item) =>
new DynamicFormItemConfig({
default: item.default,
id: UUID.generate(),
label: item.label,
description: item.description,
name: item.name,
required: item.required,
type: parseDynamicFormItemType(item.type),
options: item.options,
}),
),
);
},
);
setRetrieverNameToConfigMap(configMap);
return configMap;
}
/**
* Load KB configuration from API
*/
async function loadKBConfig(
kbId: string,
): Promise<z.infer<typeof formSchema>> {
const res = await httpClient.getExternalKnowledgeBase(kbId);
const kb = res.base;
return {
name: kb.name,
description: kb.description,
emoji: kb.emoji || '🔗',
plugin_author: kb.plugin_author,
plugin_name: kb.plugin_name,
retriever_name: kb.retriever_name,
retriever_config: kb.retriever_config || {},
};
}
/**
* Load dynamic form configuration for selected retriever
* @param fullRetrieverName - Full retriever name in format: plugin_author/plugin_name/retriever_name
* @param configMapOverride - Optional config map to use (for initial load)
*/
function loadDynamicFormConfig(
fullRetrieverName: string,
configMapOverride?: Map<string, IDynamicFormItemSchema[]>,
) {
if (!fullRetrieverName) {
setShowDynamicForm(false);
return;
}
// Use provided config map or fall back to state
const configMap = configMapOverride || retrieverNameToConfigMap;
const configList = configMap.get(fullRetrieverName);
if (configList && configList.length > 0) {
setDynamicFormConfigList(configList);
setShowDynamicForm(true);
// Only reset to default values when manually selecting (not initial load)
if (!configMapOverride) {
form.setValue('retriever_config', getDefaultValues(configList));
}
} else {
setShowDynamicForm(false);
if (!configMapOverride) {
form.setValue('retriever_config', {});
}
}
}
/**
* Handle retriever selection change
*/
function handleRetrieverSelect(fullRetrieverName: string) {
if (!fullRetrieverName) {
setShowDynamicForm(false);
return;
}
// Parse and update form fields
const parts = fullRetrieverName.split('/');
if (parts.length === 3) {
form.setValue('plugin_author', parts[0]);
form.setValue('plugin_name', parts[1]);
form.setValue('retriever_name', parts[2]);
}
// Load dynamic form configuration
loadDynamicFormConfig(fullRetrieverName);
}
/**
* Handle form submission (create or update)
*/
function handleFormSubmit() {
const formData: ExternalKnowledgeBase = {
name: form.getValues().name,
description: form.getValues().description || '',
emoji: form.getValues().emoji,
plugin_author: form.getValues().plugin_author,
plugin_name: form.getValues().plugin_name,
retriever_name: form.getValues().retriever_name,
retriever_config: form.getValues().retriever_config,
};
if (initKBId) {
// Update existing KB
httpClient
.updateExternalKnowledgeBase(initKBId, { ...formData, uuid: initKBId })
.then(() => {
onFormSubmit(form.getValues());
toast.success(t('knowledge.updateExternalSuccess'));
})
.catch((err) => {
toast.error('Failed to update KB: ' + err.msg);
});
} else {
// Create new KB
httpClient
.createExternalKnowledgeBase(formData)
.then((res) => {
toast.success(t('knowledge.createExternalSuccess'));
onNewKBCreated(res.uuid);
form.reset();
})
.catch((err) => {
toast.error('Failed to create KB: ' + err.msg);
});
}
}
/**
* Handle KB deletion
*/
function handleDelete() {
if (!initKBId) return;
httpClient
.deleteExternalKnowledgeBase(initKBId)
.then(() => {
onKBDeleted();
toast.success(t('knowledge.deleteExternalSuccess'));
})
.catch((err) => {
toast.error('Failed to delete KB: ' + err.msg);
});
}
/**
* Get retriever label with i18n support
*/
function getRetrieverLabel(fullName: string): string {
const retriever = availableRetrievers.find(
(r) =>
`${r.plugin_author}/${r.plugin_name}/${r.retriever_name}` === fullName,
);
return retriever?.manifest?.manifest?.metadata?.label
? extractI18nObject(retriever.manifest.manifest.metadata.label)
: fullName;
}
// Compute full retriever name for display
const currentRetrieverFullName =
form.watch('plugin_author') &&
form.watch('plugin_name') &&
form.watch('retriever_name')
? `${form.watch('plugin_author')}/${form.watch(
'plugin_name',
)}/${form.watch('retriever_name')}`
: '';
return (
<div>
{/* Delete Confirmation Dialog */}
<Dialog
open={showDeleteConfirmModal}
onOpenChange={setShowDeleteConfirmModal}
>
<DialogContent>
<DialogHeader>
<DialogTitle>{t('common.confirmDelete')}</DialogTitle>
</DialogHeader>
<DialogDescription>
{t('knowledge.deleteConfirmation')}
</DialogDescription>
<DialogFooter>
<Button
variant="outline"
onClick={() => setShowDeleteConfirmModal(false)}
>
{t('common.cancel')}
</Button>
<Button
variant="destructive"
onClick={() => {
handleDelete();
setShowDeleteConfirmModal(false);
}}
>
{t('common.confirmDelete')}
</Button>
</DialogFooter>
</DialogContent>
</Dialog>
{/* Main Form */}
<Form {...form}>
<form
id="external-kb-form"
onSubmit={form.handleSubmit(handleFormSubmit)}
className="space-y-8"
>
<div className="space-y-4">
{/* KB Name and Emoji in same row */}
<div className="flex gap-4 items-start">
<FormField
control={form.control}
name="name"
render={({ field }) => (
<FormItem className="flex-1">
<FormLabel>
{t('knowledge.kbName')}
<span className="text-red-500">*</span>
</FormLabel>
<FormControl>
<Input {...field} />
</FormControl>
<FormMessage />
</FormItem>
)}
/>
<FormField
control={form.control}
name="emoji"
render={({ field }) => (
<FormItem>
<FormLabel>{t('common.icon')}</FormLabel>
<FormControl>
<EmojiPicker
value={field.value}
onChange={field.onChange}
/>
</FormControl>
<FormMessage />
</FormItem>
)}
/>
</div>
{/* KB Description */}
<FormField
control={form.control}
name="description"
render={({ field }) => (
<FormItem>
<FormLabel>{t('knowledge.kbDescription')}</FormLabel>
<FormControl>
<Input {...field} />
</FormControl>
<FormMessage />
</FormItem>
)}
/>
{/* Retriever Selector */}
<FormField
control={form.control}
name="retriever_name"
render={() => (
<FormItem>
<FormLabel>
{t('knowledge.retriever')}
<span className="text-red-500">*</span>
</FormLabel>
<FormControl>
<Select
onValueChange={handleRetrieverSelect}
value={currentRetrieverFullName}
>
<SelectTrigger className="w-full bg-[#ffffff] dark:bg-[#2a2a2e]">
<SelectValue
placeholder={t('knowledge.selectRetriever')}
/>
</SelectTrigger>
<SelectContent className="fixed z-[1000]">
<SelectGroup>
{availableRetrievers.map((retriever) => {
const fullName = `${retriever.plugin_author}/${retriever.plugin_name}/${retriever.retriever_name}`;
const label = retriever.manifest?.manifest?.metadata
?.label
? extractI18nObject(
retriever.manifest.manifest.metadata.label,
)
: retriever.retriever_name;
const description = extractI18nObject(
retriever.retriever_description,
);
return (
<HoverCard
key={fullName}
openDelay={0}
closeDelay={0}
>
<HoverCardTrigger asChild>
<SelectItem value={fullName}>
{label}
</SelectItem>
</HoverCardTrigger>
<HoverCardContent
className="w-80 data-[state=open]:animate-none"
align="end"
side="right"
sideOffset={10}
>
<div className="space-y-2">
<div className="flex items-start gap-3">
<img
src={httpClient.getPluginIconURL(
retriever.plugin_author,
retriever.plugin_name,
)}
alt="plugin icon"
className="w-10 h-10 rounded-[8%] flex-shrink-0"
/>
<div className="flex flex-col gap-1 flex-1 min-w-0">
<h4 className="font-medium text-sm">
{label}
</h4>
<p className="text-xs text-muted-foreground">
{retriever.plugin_author} /{' '}
{retriever.plugin_name}
</p>
</div>
</div>
{description && (
<p className="text-sm text-muted-foreground">
{description}
</p>
)}
</div>
</HoverCardContent>
</HoverCard>
);
})}
</SelectGroup>
</SelectContent>
</Select>
</FormControl>
<FormMessage />
<p className="text-sm text-muted-foreground">
{t('knowledge.retrieverInstallInfo')}{' '}
<a
href="https://space.langbot.app/market?category=KnowledgeRetriever"
target="_blank"
rel="noopener noreferrer"
className="text-primary underline hover:no-underline"
>
{t('knowledge.retrieverMarketLink')}
</a>
</p>
</FormItem>
)}
/>
{/* Selected Retriever Card */}
{currentRetrieverFullName && (
<div className="flex items-start gap-3 p-4 rounded-lg border">
<img
src={httpClient.getPluginIconURL(
form.watch('plugin_author'),
form.watch('plugin_name'),
)}
alt="plugin icon"
className="w-12 h-12 rounded-[8%] flex-shrink-0"
/>
<div className="flex flex-col gap-1">
<div className="font-medium">
{getRetrieverLabel(currentRetrieverFullName)}
</div>
<div className="text-sm text-gray-500">
{form.watch('plugin_author')} / {form.watch('plugin_name')}
</div>
</div>
</div>
)}
{/* Dynamic Retriever Configuration Form */}
{showDynamicForm && dynamicFormConfigList.length > 0 && (
<div className="space-y-4">
<div className="text-lg font-medium">
{t('knowledge.retrieverConfiguration')}
</div>
<DynamicFormComponent
itemConfigList={dynamicFormConfigList}
initialValues={form.watch('retriever_config')}
onSubmit={(values) => {
form.setValue('retriever_config', values);
}}
/>
</div>
)}
</div>
</form>
</Form>
</div>
);
}

15
web/src/app/home/knowledge/components/kb-card/KBCard.module.css
View File

@@ -169,3 +169,18 @@
width: 1.2rem;
height: 1.2rem;
}
.engineBadge {
font-size: 0.75rem;
line-height: 1rem;
padding: 0.125rem 0.5rem;
border-radius: 9999px;
background-color: #f3e8ff;
color: #7e22ce;
white-space: nowrap;
}
:global(.dark) .engineBadge {
background-color: #581c87;
color: #d8b4fe;
}

11
web/src/app/home/knowledge/components/kb-card/KBCard.tsx
View File

@@ -4,14 +4,21 @@ import styles from './KBCard.module.css';
export default function KBCard({ kbCardVO }: { kbCardVO: KnowledgeBaseVO }) {
const { t } = useTranslation();
return (
<div className={`${styles.cardContainer}`}>
<div className={`${styles.basicInfoContainer}`}>
<div className={`${styles.iconBasicInfoContainer}`}>
<div className={`${styles.iconEmoji}`}>{kbCardVO.emoji || '📚'}</div>
<div className={`${styles.basicInfoNameContainer}`}>
<div className={`${styles.basicInfoNameText} ${styles.bigText}`}>
{kbCardVO.name}
<div className="flex items-center gap-2">
<div className={`${styles.basicInfoNameText} ${styles.bigText}`}>
{kbCardVO.name}
</div>
{/* Engine badge */}
<span className={styles.engineBadge}>
{kbCardVO.getEngineName()}
</span>
</div>
<div className={`${styles.basicInfoDescriptionText}`}>
{kbCardVO.description}

35
web/src/app/home/knowledge/components/kb-card/KBCardVO.ts
View File

@@ -1,29 +1,52 @@
import { KnowledgeEngineInfo } from '@/app/infra/entities/api';
import { extractI18nObject } from '@/i18n/I18nProvider';
export interface IKnowledgeBaseVO {
id: string;
name: string;
description: string;
embeddingModelUUID: string;
top_k: number;
lastUpdatedTimeAgo: string;
emoji?: string;
ragEngine?: KnowledgeEngineInfo;
ragEnginePluginId?: string;
}
export class KnowledgeBaseVO implements IKnowledgeBaseVO {
id: string;
name: string;
description: string;
embeddingModelUUID: string;
top_k: number;
lastUpdatedTimeAgo: string;
emoji?: string;
ragEngine?: KnowledgeEngineInfo;
ragEnginePluginId?: string;
constructor(props: IKnowledgeBaseVO) {
this.id = props.id;
this.name = props.name;
this.description = props.description;
this.embeddingModelUUID = props.embeddingModelUUID;
this.top_k = props.top_k;
this.lastUpdatedTimeAgo = props.lastUpdatedTimeAgo;
this.emoji = props.emoji;
this.ragEngine = props.ragEngine;
this.ragEnginePluginId = props.ragEnginePluginId;
}
/**
* Check if this KB supports document management
*/
hasDocumentCapability(): boolean {
if (!this.ragEngine) {
return false;
}
return this.ragEngine.capabilities.includes('doc_ingestion');
}
/**
* Get display name for the Knowledge Engine
*/
getEngineName(): string {
if (!this.ragEngine) {
return 'Unknown';
}
return extractI18nObject(this.ragEngine.name);
}
}

307
web/src/app/home/knowledge/components/kb-docs/FileUploadZone.tsx
View File

@@ -1,17 +1,32 @@
import React, { useCallback, useState } from 'react';
import React, { useCallback, useEffect, useState } from 'react';
import { Card, CardContent } from '@/components/ui/card';
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from '@/components/ui/select';
import { Button } from '@/components/ui/button';
import { httpClient } from '@/app/infra/http/HttpClient';
import { toast } from 'sonner';
import { useTranslation } from 'react-i18next';
import { ParserInfo } from '@/app/infra/entities/api';
import { I18nObject } from '@/app/infra/entities/common';
import { extractI18nObject } from '@/i18n/I18nProvider';
interface FileUploadZoneProps {
kbId: string;
ragEngineName?: I18nObject;
ragEngineCapabilities?: string[];
onUploadSuccess: () => void;
onUploadError: (error: string) => void;
}
export default function FileUploadZone({
kbId,
ragEngineName,
ragEngineCapabilities,
onUploadSuccess,
onUploadError,
}: FileUploadZoneProps) {
@@ -19,17 +34,49 @@ export default function FileUploadZone({
const [isDragOver, setIsDragOver] = useState(false);
const [isUploading, setIsUploading] = useState(false);
const handleUpload = useCallback(
async (file: File) => {
if (isUploading) return;
// Parser selection state
const [pendingFile, setPendingFile] = useState<File | null>(null);
const [availableParsers, setAvailableParsers] = useState<ParserInfo[]>([]);
const [selectedParser, setSelectedParser] = useState<string>('builtin');
const [loadingParsers, setLoadingParsers] = useState(false);
// Check file size (10MB limit)
const MAX_FILE_SIZE = 10 * 1024 * 1024; // 10MB
if (file.size > MAX_FILE_SIZE) {
toast.error(t('knowledge.documentsTab.fileSizeExceeded'));
return;
}
// Whether the Knowledge Engine natively supports document parsing.
// This is a coarse-grained capability check rather than per-MIME-type filtering.
// Fine-grained MIME type declaration (e.g. supported_parse_mime_types on the engine)
// would require changes across the SDK, backend, and frontend prop chain;
// using an engine-level capability flag keeps the change minimal.
const ragEngineCanParse =
ragEngineCapabilities?.includes('doc_parsing') ?? false;
// When a file is selected, check for available parsers
useEffect(() => {
if (!pendingFile) return;
const mimeType = pendingFile.type || undefined;
setLoadingParsers(true);
httpClient
.listParsers(mimeType)
.then((resp) => {
const parsers = resp.parsers || [];
setAvailableParsers(parsers);
if (ragEngineCanParse) {
setSelectedParser('builtin');
} else if (parsers.length > 0) {
setSelectedParser(parsers[0].plugin_id);
} else {
setSelectedParser('');
}
})
.catch(() => {
setAvailableParsers([]);
})
.finally(() => {
setLoadingParsers(false);
});
}, [pendingFile, ragEngineCanParse]);
const doUpload = useCallback(
async (file: File, parserPluginId?: string) => {
setIsUploading(true);
const toastId = toast.loading(t('knowledge.documentsTab.uploadingFile'));
@@ -37,8 +84,12 @@ export default function FileUploadZone({
// Step 1: Upload file to server
const uploadResult = await httpClient.uploadDocumentFile(file);
// Step 2: Associate file with knowledge base
await httpClient.uploadKnowledgeBaseFile(kbId, uploadResult.file_id);
// Step 2: Associate file with knowledge base (with optional parser)
await httpClient.uploadKnowledgeBaseFile(
kbId,
uploadResult.file_id,
parserPluginId,
);
toast.success(t('knowledge.documentsTab.uploadSuccess'), {
id: toastId,
@@ -51,11 +102,65 @@ export default function FileUploadZone({
onUploadError(errorMessage);
} finally {
setIsUploading(false);
setPendingFile(null);
setAvailableParsers([]);
setSelectedParser('builtin');
}
},
[kbId, isUploading, onUploadSuccess, onUploadError, t],
[kbId, onUploadSuccess, onUploadError, t],
);
const handleFileSelected = useCallback(
async (file: File) => {
if (isUploading) return;
// Check file size (10MB limit)
const MAX_FILE_SIZE = 10 * 1024 * 1024; // 10MB
if (file.size > MAX_FILE_SIZE) {
toast.error(t('knowledge.documentsTab.fileSizeExceeded'));
return;
}
// Set loadingParsers=true BEFORE pendingFile so both state updates
// batch together in the same render. This prevents the auto-upload
// effect from firing before parser fetch completes.
setLoadingParsers(true);
setPendingFile(file);
},
[isUploading, t],
);
// Auto-upload if Knowledge Engine can parse and no external parsers available
useEffect(() => {
if (
pendingFile &&
!loadingParsers &&
ragEngineCanParse &&
availableParsers.length === 0
) {
doUpload(pendingFile);
}
}, [
pendingFile,
loadingParsers,
ragEngineCanParse,
availableParsers,
doUpload,
]);
const handleConfirmUpload = useCallback(() => {
if (!pendingFile) return;
const parserPluginId =
selectedParser === 'builtin' ? undefined : selectedParser;
doUpload(pendingFile, parserPluginId);
}, [pendingFile, selectedParser, doUpload]);
const handleCancelUpload = useCallback(() => {
setPendingFile(null);
setAvailableParsers([]);
setSelectedParser('builtin');
}, []);
const handleDragOver = useCallback((e: React.DragEvent) => {
e.preventDefault();
setIsDragOver(true);
@@ -73,79 +178,145 @@ export default function FileUploadZone({
const files = Array.from(e.dataTransfer.files);
if (files.length > 0) {
handleUpload(files[0]);
handleFileSelected(files[0]);
}
},
[handleUpload],
[handleFileSelected],
);
const handleFileSelect = useCallback(
(e: React.ChangeEvent<HTMLInputElement>) => {
const files = e.target.files;
if (files && files.length > 0) {
handleUpload(files[0]);
handleFileSelected(files[0]);
}
// Reset the input so the same file can be selected again
e.target.value = '';
},
[handleUpload],
[handleFileSelected],
);
// Show parser selection UI when there are choices to make, or when no parser is available
const showParserSelector =
pendingFile &&
!loadingParsers &&
(availableParsers.length > 0 || !ragEngineCanParse);
const noParserAvailable = !ragEngineCanParse && availableParsers.length === 0;
return (
<Card className="mb-4">
<CardContent className="p-4">
<div
className={`
relative border-2 border-dashed rounded-lg p-4 text-center transition-colors
${
isDragOver
? 'border-blue-500 bg-blue-50'
: 'border-gray-300 hover:border-gray-400'
}
${isUploading ? 'opacity-50 pointer-events-none' : ''}
`}
onDragOver={handleDragOver}
onDragLeave={handleDragLeave}
onDrop={handleDrop}
>
<input
type="file"
id="file-upload"
className="hidden"
onChange={handleFileSelect}
accept=".pdf,.doc,.docx,.txt,.md,.html,.zip"
disabled={isUploading}
/>
<label htmlFor="file-upload" className="cursor-pointer block">
<div className="space-y-2">
<div className="mx-auto w-10 h-10 bg-gray-100 rounded-full flex items-center justify-center">
<svg
className="w-5 h-5 text-gray-400"
fill="none"
stroke="currentColor"
viewBox="0 0 24 24"
{showParserSelector ? (
<div className="space-y-3">
<p className="text-sm font-medium text-gray-900 dark:text-gray-100">
{pendingFile.name}
</p>
{noParserAvailable ? (
<div className="rounded-md bg-yellow-50 dark:bg-yellow-900/20 border border-yellow-200 dark:border-yellow-800 p-3">
<p className="text-sm text-yellow-800 dark:text-yellow-200">
{t('knowledge.documentsTab.noParserAvailable')}
</p>
</div>
) : (
<div className="space-y-2">
<label className="text-sm text-gray-600 dark:text-gray-400">
{t('knowledge.documentsTab.selectParser')}
</label>
<Select
value={selectedParser}
onValueChange={setSelectedParser}
>
<path
strokeLinecap="round"
strokeLinejoin="round"
strokeWidth={2}
d="M7 16a4 4 0 01-.88-7.903A5 5 0 1115.9 6L16 6a5 5 0 011 9.9M15 13l-3-3m0 0l-3 3m3-3v12"
/>
</svg>
</div>
<div>
<p className="text-base font-medium text-gray-900 dark:text-gray-100">
{isUploading
? t('knowledge.documentsTab.uploading')
: t('knowledge.documentsTab.dragAndDrop')}
</p>
<p className="text-xs text-gray-500 mt-1 dark:text-gray-400">
{t('knowledge.documentsTab.supportedFormats')}
</p>
<SelectTrigger className="w-full">
<SelectValue />
</SelectTrigger>
<SelectContent>
{ragEngineCanParse && (
<SelectItem value="builtin">
{ragEngineName
? extractI18nObject(ragEngineName)
: t('knowledge.documentsTab.builtInParser')}
</SelectItem>
)}
{availableParsers.map((parser) => (
<SelectItem
key={parser.plugin_id}
value={parser.plugin_id}
>
{extractI18nObject(parser.name)}
</SelectItem>
))}
</SelectContent>
</Select>
</div>
)}
<div className="flex justify-end gap-2">
<Button variant="outline" size="sm" onClick={handleCancelUpload}>
{t('knowledge.documentsTab.cancelUpload')}
</Button>
{!noParserAvailable && (
<Button size="sm" onClick={handleConfirmUpload}>
{t('knowledge.documentsTab.confirmUpload')}
</Button>
)}
</div>
</label>
</div>
</div>
) : (
<div
className={`
relative border-2 border-dashed rounded-lg p-4 text-center transition-colors
${
isDragOver
? 'border-blue-500 bg-blue-50'
: 'border-gray-300 hover:border-gray-400'
}
${isUploading || loadingParsers ? 'opacity-50 pointer-events-none' : ''}
`}
onDragOver={handleDragOver}
onDragLeave={handleDragLeave}
onDrop={handleDrop}
>
<input
type="file"
id="file-upload"
className="hidden"
onChange={handleFileSelect}
accept=".pdf,.doc,.docx,.txt,.md,.html,.zip"
disabled={isUploading || loadingParsers}
/>
<label htmlFor="file-upload" className="cursor-pointer block">
<div className="space-y-2">
<div className="mx-auto w-10 h-10 bg-gray-100 rounded-full flex items-center justify-center">
<svg
className="w-5 h-5 text-gray-400"
fill="none"
stroke="currentColor"
viewBox="0 0 24 24"
>
<path
strokeLinecap="round"
strokeLinejoin="round"
strokeWidth={2}
d="M7 16a4 4 0 01-.88-7.903A5 5 0 1115.9 6L16 6a5 5 0 011 9.9M15 13l-3-3m0 0l-3 3m3-3v12"
/>
</svg>
</div>
<div>
<p className="text-base font-medium text-gray-900 dark:text-gray-100">
{isUploading
? t('knowledge.documentsTab.uploading')
: t('knowledge.documentsTab.dragAndDrop')}
</p>
<p className="text-xs text-gray-500 mt-1 dark:text-gray-400">
{t('knowledge.documentsTab.supportedFormats')}
</p>
</div>
</div>
</label>
</div>
)}
</CardContent>
</Card>
);

86
web/src/app/home/knowledge/components/kb-docs/KBDoc.tsx
View File

@@ -1,48 +1,80 @@
import { useEffect, useState } from 'react';
import { useCallback, useEffect, useRef, useState } from 'react';
import { httpClient } from '@/app/infra/http/HttpClient';
import { KnowledgeBaseFile } from '@/app/infra/entities/api';
import { I18nObject } from '@/app/infra/entities/common';
import { columns, DocumentFile } from './documents/columns';
import { DataTable } from './documents/data-table';
import FileUploadZone from './FileUploadZone';
import { toast } from 'sonner';
import { useTranslation } from 'react-i18next';
export default function KBDoc({ kbId }: { kbId: string }) {
export default function KBDoc({
kbId,
ragEngineName,
ragEngineCapabilities,
}: {
kbId: string;
ragEngineName?: I18nObject;
ragEngineCapabilities?: string[];
}) {
const [documentsList, setDocumentsList] = useState<DocumentFile[]>([]);
const { t } = useTranslation();
const intervalRef = useRef<ReturnType<typeof setInterval> | null>(null);
useEffect(() => {
getDocumentsList();
const intervalId = setInterval(() => {
getDocumentsList();
}, 5000);
return () => {
clearInterval(intervalId);
};
const getDocumentsList = useCallback(async () => {
const resp = await httpClient.getKnowledgeBaseFiles(kbId);
const files = resp.files.map((file: KnowledgeBaseFile) => ({
uuid: file.uuid,
name: file.file_name,
status: file.status,
}));
setDocumentsList(files);
return files;
}, [kbId]);
async function getDocumentsList() {
const resp = await httpClient.getKnowledgeBaseFiles(kbId);
setDocumentsList(
resp.files.map((file: KnowledgeBaseFile) => {
return {
uuid: file.uuid,
name: file.file_name,
status: file.status,
};
}),
);
}
const startPolling = useCallback(() => {
if (intervalRef.current) return;
intervalRef.current = setInterval(() => {
getDocumentsList().then((files) => {
const allDone =
files.length > 0 &&
files.every(
(doc: DocumentFile) =>
doc.status === 'completed' || doc.status === 'failed',
);
if (allDone && intervalRef.current) {
clearInterval(intervalRef.current);
intervalRef.current = null;
}
});
}, 5000);
}, [getDocumentsList]);
useEffect(() => {
getDocumentsList().then((files) => {
const hasProcessing = files.some(
(doc: DocumentFile) =>
doc.status !== 'completed' && doc.status !== 'failed',
);
if (hasProcessing) {
startPolling();
}
});
return () => {
if (intervalRef.current) {
clearInterval(intervalRef.current);
intervalRef.current = null;
}
};
}, [kbId, getDocumentsList, startPolling]);
const handleUploadSuccess = () => {
// Refresh document list after successful upload
getDocumentsList();
startPolling();
};
const handleUploadError = (error: string) => {
// Error messages are already handled by toast in FileUploadZone component
console.error('Upload failed:', error);
};
@@ -63,6 +95,8 @@ export default function KBDoc({ kbId }: { kbId: string }) {
<div className="container mx-auto py-2">
<FileUploadZone
kbId={kbId}
ragEngineName={ragEngineName}
ragEngineCapabilities={ragEngineCapabilities}
onUploadSuccess={handleUploadSuccess}
onUploadError={handleUploadError}
/>

423
web/src/app/home/knowledge/components/kb-form/KBForm.tsx
View File

@@ -14,18 +14,25 @@ import {
FormMessage,
FormDescription,
} from '@/components/ui/form';
import { httpClient, systemInfo, userInfo } from '@/app/infra/http';
import { httpClient } from '@/app/infra/http/HttpClient';
import {
Select,
SelectContent,
SelectGroup,
SelectItem,
SelectLabel,
SelectTrigger,
SelectValue,
} from '@/components/ui/select';
import { KnowledgeBase, EmbeddingModel } from '@/app/infra/entities/api';
import { KnowledgeBase, KnowledgeEngine } from '@/app/infra/entities/api';
import { toast } from 'sonner';
import { extractI18nObject } from '@/i18n/I18nProvider';
import DynamicFormComponent from '@/app/home/components/dynamic-form/DynamicFormComponent';
import { IDynamicFormItemSchema } from '@/app/infra/entities/form/dynamic';
import {
DynamicFormItemConfig,
getDefaultValues,
parseDynamicFormItemType,
} from '@/app/home/components/dynamic-form/DynamicFormItemConfig';
import { UUID } from 'uuidjs';
const getFormSchema = (t: (key: string) => string) =>
z.object({
@@ -34,15 +41,42 @@ const getFormSchema = (t: (key: string) => string) =>
.string()
.min(1, { message: t('knowledge.kbDescriptionRequired') }),
emoji: z.string().optional(),
embeddingModelUUID: z
ragEngineId: z
.string()
.min(1, { message: t('knowledge.embeddingModelUUIDRequired') }),
top_k: z
.number()
.min(1, { message: t('knowledge.topKRequired') })
.max(30, { message: t('knowledge.topKMax') }),
.min(1, { message: t('knowledge.knowledgeEngineRequired') }),
});
/**
* Parse creation schema from Knowledge Engine to IDynamicFormItemSchema[]
* Same pattern as ExternalKBForm uses for retriever config
*/
function parseCreationSchema(
// eslint-disable-next-line @typescript-eslint/no-explicit-any
schemaItems: any | any[] | undefined,
): IDynamicFormItemSchema[] {
if (!schemaItems) return [];
// Handle wrapped schema (e.g. { schema: [...] }) which might be returned by the API
const items = Array.isArray(schemaItems) ? schemaItems : schemaItems.schema;
if (!items || !Array.isArray(items)) return [];
return items.map(
(item) =>
new DynamicFormItemConfig({
default: item.default,
id: UUID.generate(),
label: item.label,
description: item.description,
name: item.name,
required: item.required,
type: parseDynamicFormItemType(item.type),
options: item.options,
show_if: item.show_if,
}),
);
}
export default function KBForm({
initKbId,
onNewKbCreated,
@@ -53,6 +87,17 @@ export default function KBForm({
onKbUpdated: (kbId: string) => void;
}) {
const { t } = useTranslation();
const [ragEngines, setRagEngines] = useState<KnowledgeEngine[]>([]);
const [selectedEngineId, setSelectedEngineId] = useState<string>('');
const [configSettings, setConfigSettings] = useState<Record<string, unknown>>(
{},
);
const [retrievalSettings, setRetrievalSettings] = useState<
Record<string, unknown>
>({});
const [isEditing, setIsEditing] = useState(false);
const [loading, setLoading] = useState(true);
const formSchema = getFormSchema(t);
const form = useForm<z.infer<typeof formSchema>>({
@@ -61,70 +106,111 @@ export default function KBForm({
name: '',
description: t('knowledge.defaultDescription'),
emoji: '📚',
embeddingModelUUID: '',
top_k: 5,
ragEngineId: '',
},
});
const [embeddingModels, setEmbeddingModels] = useState<EmbeddingModel[]>([]);
// Get selected engine details
const selectedEngine = ragEngines.find(
(e) => e.plugin_id === selectedEngineId,
);
useEffect(() => {
getEmbeddingModelNameList().then(() => {
loadRagEngines().then(() => {
if (initKbId) {
getKbConfig(initKbId).then((val) => {
form.setValue('name', val.name);
form.setValue('description', val.description);
form.setValue('emoji', val.emoji);
form.setValue('embeddingModelUUID', val.embeddingModelUUID);
form.setValue('top_k', val.top_k || 5);
});
loadKbConfig(initKbId);
}
});
}, []);
const getKbConfig = async (
kbId: string,
): Promise<z.infer<typeof formSchema>> => {
return new Promise((resolve) => {
httpClient.getKnowledgeBase(kbId).then((res) => {
resolve({
name: res.base.name,
description: res.base.description,
emoji: res.base.emoji || '📚',
embeddingModelUUID: res.base.embedding_model_uuid,
top_k: res.base.top_k || 5,
});
});
});
// Auto-select first engine when engines are loaded and no selection
useEffect(() => {
if (ragEngines.length > 0 && !selectedEngineId && !isEditing) {
const firstEngine = ragEngines[0];
setSelectedEngineId(firstEngine.plugin_id);
form.setValue('ragEngineId', firstEngine.plugin_id);
// Initialize config settings with defaults
const formItems = parseCreationSchema(firstEngine.creation_schema);
if (formItems.length > 0) {
setConfigSettings(getDefaultValues(formItems));
}
const retrievalItems = parseCreationSchema(firstEngine.retrieval_schema);
if (retrievalItems.length > 0) {
setRetrievalSettings(getDefaultValues(retrievalItems));
}
}
}, [ragEngines, selectedEngineId, isEditing]);
const loadRagEngines = async () => {
setLoading(true);
try {
const resp = await httpClient.getKnowledgeEngines();
setRagEngines(resp.engines);
} catch (err) {
console.error('Failed to load Knowledge Engines:', err);
} finally {
setLoading(false);
}
};
const getEmbeddingModelNameList = async () => {
const resp = await httpClient.getProviderEmbeddingModels();
let models = resp.models;
// Filter out space-chat-completions models when not logged in with space account or when models service is disabled
if (
systemInfo.disable_models_service ||
userInfo?.account_type !== 'space'
) {
models = models.filter(
(m) => m.provider?.requester !== 'space-chat-completions',
);
const loadKbConfig = async (kbId: string) => {
try {
setIsEditing(true);
const res = await httpClient.getKnowledgeBase(kbId);
const kb = res.base;
const engineId = kb.knowledge_engine_plugin_id || '';
setSelectedEngineId(engineId);
form.setValue('name', kb.name);
form.setValue('description', kb.description);
form.setValue('emoji', kb.emoji || '📚');
form.setValue('ragEngineId', engineId);
setConfigSettings(kb.creation_settings || {});
setRetrievalSettings(kb.retrieval_settings || {});
} catch (err) {
console.error('Failed to load KB config:', err);
}
};
const handleEngineChange = (engineId: string) => {
setSelectedEngineId(engineId);
form.setValue('ragEngineId', engineId);
// Find engine and initialize config settings with defaults from schema
const engine = ragEngines.find((e) => e.plugin_id === engineId);
if (engine) {
const formItems = parseCreationSchema(engine.creation_schema);
if (formItems.length > 0) {
setConfigSettings(getDefaultValues(formItems));
} else {
setConfigSettings({});
}
const retrievalItems = parseCreationSchema(engine.retrieval_schema);
if (retrievalItems.length > 0) {
setRetrievalSettings(getDefaultValues(retrievalItems));
} else {
setRetrievalSettings({});
}
}
setEmbeddingModels(models);
};
const onSubmit = (data: z.infer<typeof formSchema>) => {
const kbData: KnowledgeBase = {
name: data.name,
description: data.description,
emoji: data.emoji,
knowledge_engine_plugin_id: selectedEngineId,
creation_settings: configSettings,
retrieval_settings: retrievalSettings,
};
if (initKbId) {
// update knowledge base
const updateKb: KnowledgeBase = {
name: data.name,
description: data.description,
emoji: data.emoji,
embedding_model_uuid: data.embeddingModelUUID,
top_k: data.top_k,
};
// Update knowledge base
httpClient
.updateKnowledgeBase(initKbId, updateKb)
.updateKnowledgeBase(initKbId, kbData)
.then((res) => {
onKbUpdated(res.uuid);
toast.success(t('knowledge.updateKnowledgeBaseSuccess'));
@@ -134,25 +220,50 @@ export default function KBForm({
toast.error(t('knowledge.updateKnowledgeBaseFailed'));
});
} else {
// create knowledge base
const newKb: KnowledgeBase = {
name: data.name,
description: data.description,
emoji: data.emoji,
embedding_model_uuid: data.embeddingModelUUID,
top_k: data.top_k,
};
// Create knowledge base
httpClient
.createKnowledgeBase(newKb)
.createKnowledgeBase(kbData)
.then((res) => {
onNewKbCreated(res.uuid);
})
.catch((err) => {
console.error('create knowledge base failed', err);
toast.error(t('knowledge.createKnowledgeBaseFailed'));
});
}
};
// Convert creation schema to dynamic form items (same as ExternalKBForm)
const configFormItems = parseCreationSchema(selectedEngine?.creation_schema);
// Convert retrieval schema to dynamic form items
const retrievalFormItems = parseCreationSchema(
selectedEngine?.retrieval_schema,
);
// Show loading state
if (loading) {
return (
<div className="flex items-center justify-center py-8">
<p className="text-muted-foreground">{t('common.loading')}</p>
</div>
);
}
// Show message if no engines available
if (ragEngines.length === 0) {
return (
<div className="flex flex-col items-center justify-center py-8 space-y-4">
<p className="text-muted-foreground">
{t('knowledge.noEnginesAvailable')}
</p>
<p className="text-sm text-muted-foreground">
{t('knowledge.installEngineHint')}
</p>
</div>
);
}
return (
<>
<Form {...form}>
@@ -162,6 +273,57 @@ export default function KBForm({
className="space-y-8"
>
<div className="space-y-4">
{/* Knowledge Engine Selector */}
<FormField
control={form.control}
name="ragEngineId"
render={({ field }) => (
<FormItem>
<FormLabel>
{t('knowledge.knowledgeEngine')}
<span className="text-red-500">*</span>
</FormLabel>
<FormControl>
<Select
disabled={isEditing}
onValueChange={(value) => {
field.onChange(value);
handleEngineChange(value);
}}
value={field.value}
>
<SelectTrigger className="w-full bg-[#ffffff] dark:bg-[#2a2a2e]">
<SelectValue
placeholder={t('knowledge.selectKnowledgeEngine')}
/>
</SelectTrigger>
<SelectContent className="fixed z-[1000]">
{ragEngines.map((engine) => (
<SelectItem
key={engine.plugin_id}
value={engine.plugin_id}
>
{extractI18nObject(engine.name)}
</SelectItem>
))}
</SelectContent>
</Select>
</FormControl>
{selectedEngine?.description && (
<FormDescription>
{extractI18nObject(selectedEngine.description)}
</FormDescription>
)}
{isEditing && (
<FormDescription>
{t('knowledge.cannotChangeKnowledgeEngine')}
</FormDescription>
)}
<FormMessage />
</FormItem>
)}
/>
{/* Name and Emoji in same row */}
<div className="flex gap-4 items-start">
<FormField
@@ -197,6 +359,8 @@ export default function KBForm({
)}
/>
</div>
{/* Description */}
<FormField
control={form.control}
name="description"
@@ -213,96 +377,45 @@ export default function KBForm({
</FormItem>
)}
/>
<FormField
control={form.control}
name="embeddingModelUUID"
render={({ field }) => (
<FormItem>
<FormLabel>
{t('knowledge.embeddingModelUUID')}
<span className="text-red-500">*</span>
</FormLabel>
<FormControl>
<div className="relative">
<Select
disabled={!!initKbId}
onValueChange={(value) => {
field.onChange(value);
}}
value={field.value}
>
<SelectTrigger className="w-[180px] bg-[#ffffff] dark:bg-[#2a2a2e]">
<SelectValue
placeholder={t('knowledge.selectEmbeddingModel')}
/>
</SelectTrigger>
<SelectContent className="fixed z-[1000]">
{(() => {
const grouped = embeddingModels.reduce(
(acc, model) => {
const providerName =
model.provider?.name ||
model.provider?.requester ||
'Unknown';
if (!acc[providerName]) acc[providerName] = [];
acc[providerName].push(model);
return acc;
},
{} as Record<string, EmbeddingModel[]>,
);
return Object.entries(grouped).map(
([providerName, models]) => (
<SelectGroup key={providerName}>
<SelectLabel>{providerName}</SelectLabel>
{models.map((model) => (
<SelectItem
key={model.uuid}
value={model.uuid}
>
{model.name}
</SelectItem>
))}
</SelectGroup>
),
);
})()}
</SelectContent>
</Select>
</div>
</FormControl>
<FormDescription>
{initKbId
? t('knowledge.cannotChangeEmbeddingModel')
: t('knowledge.embeddingModelDescription')}
</FormDescription>
<FormMessage />
</FormItem>
)}
/>
<FormField
control={form.control}
name="top_k"
render={({ field }) => (
<FormItem>
<FormLabel>
{t('knowledge.topK')}
<span className="text-red-500">*</span>
</FormLabel>
<FormControl>
<Input
type="number"
{...field}
onChange={(e) => field.onChange(Number(e.target.value))}
className="w-[180px] h-10 text-base appearance-none"
/>
</FormControl>
<FormDescription>
{t('knowledge.topKdescription')}
</FormDescription>
<FormMessage />
</FormItem>
)}
/>
{/* Engine specific fields (dynamic form from creation_schema) */}
{configFormItems.length > 0 && (
<div className="space-y-4 pt-2 border-t">
<div className="text-sm font-medium text-muted-foreground">
{t('knowledge.engineSettings')}
</div>
<div>
<DynamicFormComponent
itemConfigList={configFormItems}
initialValues={configSettings as Record<string, object>}
onSubmit={(val) =>
setConfigSettings(val as Record<string, unknown>)
}
isEditing={isEditing}
externalDependentValues={retrievalSettings}
/>
</div>
</div>
)}
{/* Retrieval settings (dynamic form from retrieval_schema) */}
{retrievalFormItems.length > 0 && (
<div className="space-y-4 pt-2 border-t">
<div className="text-sm font-medium text-muted-foreground">
{t('knowledge.retrievalSettings')}
</div>
<div>
<DynamicFormComponent
itemConfigList={retrievalFormItems}
initialValues={retrievalSettings as Record<string, object>}
onSubmit={(val) =>
setRetrievalSettings(val as Record<string, unknown>)
}
externalDependentValues={configSettings}
/>
</div>
</div>
)}
</div>
</form>
</Form>

35
web/src/app/home/knowledge/components/kb-retrieve/ExternalKBRetrieve.tsx
View File

@@ -1,35 +0,0 @@
'use client';
import React from 'react';
import { httpClient } from '@/app/infra/http/HttpClient';
import { RetrieveResult } from '@/app/infra/entities/api';
import KBRetrieveGeneric from './KBRetrieveGeneric';
interface ExternalKBRetrieveProps {
kbId: string;
}
/**
* External knowledge base retrieve component
* Uses the generic retrieve component with external KB API
*/
export default function ExternalKBRetrieve({ kbId }: ExternalKBRetrieveProps) {
const getResultTitle = (result: RetrieveResult): string => {
// For external KB, try to get document_name or use a generic title
return (
(result.metadata.document_name as string) ||
(result.metadata.source as string) ||
result.id
);
};
return (
<KBRetrieveGeneric
kbId={kbId}
retrieveFunction={httpClient.retrieveExternalKnowledgeBase.bind(
httpClient,
)}
getResultTitle={getResultTitle}
/>
);
}

124
web/src/app/home/knowledge/components/kb-retrieve/KBRetrieve.tsx
View File

@@ -1,124 +0,0 @@
'use client';
import React, { useState, useEffect } from 'react';
import { Card, CardContent, CardHeader, CardTitle } from '@/components/ui/card';
import { Button } from '@/components/ui/button';
import { Input } from '@/components/ui/input';
import { useTranslation } from 'react-i18next';
import { httpClient } from '@/app/infra/http/HttpClient';
import { RetrieveResult, KnowledgeBaseFile } from '@/app/infra/entities/api';
import { toast } from 'sonner';
interface KBRetrieveProps {
kbId: string;
}
export default function KBRetrieve({ kbId }: KBRetrieveProps) {
const { t } = useTranslation();
const [query, setQuery] = useState('');
const [results, setResults] = useState<RetrieveResult[]>([]);
const [files, setFiles] = useState<KnowledgeBaseFile[]>([]);
const [loading, setLoading] = useState(false);
useEffect(() => {
const loadFiles = async () => {
try {
const response = await httpClient.getKnowledgeBaseFiles(kbId);
setFiles(response.files);
} catch (error) {
console.error('Failed to load files:', error);
}
};
loadFiles();
}, [kbId]);
const handleRetrieve = async () => {
if (!query.trim()) return;
setLoading(true);
try {
setResults([]);
const response = await httpClient.retrieveKnowledgeBase(kbId, query);
setResults(response.results);
} catch (error) {
console.error('Retrieve failed:', error);
toast.error(t('knowledge.retrieveError'));
} finally {
setLoading(false);
}
};
const getFileName = (fileId?: string) => {
if (!fileId) return '';
const file = files.find((f) => f.uuid === fileId);
return file?.file_name || fileId;
};
/**
* Extract text content from the content array
* The content array may contain multiple items with type 'text'
*/
const extractTextFromContent = (result: RetrieveResult): string => {
// First try to get content from the new format
if (result.content && Array.isArray(result.content)) {
const textParts = result.content
.filter((item) => item.type === 'text' && item.text)
.map((item) => item.text);
if (textParts.length > 0) {
return textParts.join('\n\n');
}
}
// Fallback to metadata.text for backward compatibility
if (result.metadata?.text) {
return result.metadata.text as string;
}
return '';
};
return (
<div className="space-y-4">
<div className="flex gap-2">
<Input
value={query}
onChange={(e) => setQuery(e.target.value)}
placeholder={t('knowledge.queryPlaceholder')}
onKeyPress={(e) => e.key === 'Enter' && handleRetrieve()}
/>
<Button onClick={handleRetrieve} disabled={loading || !query.trim()}>
{t('knowledge.query')}
</Button>
</div>
<div className="space-y-3">
{results.length === 0 && !loading && (
<p className="text-muted-foreground">{t('knowledge.noResults')}</p>
)}
{loading ? (
<p className="text-muted-foreground">{t('common.loading')}</p>
) : (
results.map((result) => (
<Card key={result.id} className="w-full">
<CardHeader className="pb-3">
<CardTitle className="text-sm font-medium flex justify-between items-center">
<span>{getFileName(result.metadata.file_id)}</span>
<span className="text-xs text-muted-foreground">
{t('knowledge.distance')}: {result.distance.toFixed(4)}
</span>
</CardTitle>
</CardHeader>
<CardContent>
<p className="text-sm whitespace-pre-wrap">
{extractTextFromContent(result)}
</p>
</CardContent>
</Card>
))
)}
</div>
</div>
);
}

3
web/src/app/home/knowledge/components/kb-retrieve/KBRetrieveGeneric.tsx
View File

@@ -106,7 +106,8 @@ export default function KBRetrieveGeneric({
<CardTitle className="text-sm font-medium flex justify-between items-center">
<span>{getTitle(result)}</span>
<span className="text-xs text-muted-foreground">
{t('knowledge.distance')}: {result.distance.toFixed(4)}
{t('knowledge.distance')}:{' '}
{(result.distance ?? 0).toFixed(4)}
</span>
</CardTitle>
</CardHeader>

224
web/src/app/home/knowledge/page.tsx
View File

@@ -5,139 +5,64 @@ import styles from './knowledgeBase.module.css';
import { useTranslation } from 'react-i18next';
import { useEffect, useState } from 'react';
import { KnowledgeBaseVO } from '@/app/home/knowledge/components/kb-card/KBCardVO';
import { ExternalKBCardVO } from '@/app/home/knowledge/components/external-kb-card/ExternalKBCardVO';
import KBCard from '@/app/home/knowledge/components/kb-card/KBCard';
import ExternalKBCard from '@/app/home/knowledge/components/external-kb-card/ExternalKBCard';
import KBDetailDialog from '@/app/home/knowledge/KBDetailDialog';
import { httpClient } from '@/app/infra/http/HttpClient';
import {
KnowledgeBase,
ExternalKnowledgeBase,
ApiRespPluginSystemStatus,
} from '@/app/infra/entities/api';
import { Tabs, TabsContent, TabsList, TabsTrigger } from '@/components/ui/tabs';
import { KnowledgeBase } from '@/app/infra/entities/api';
export default function KnowledgePage() {
const { t } = useTranslation();
const [activeTab, setActiveTab] = useState('builtin');
const [knowledgeBaseList, setKnowledgeBaseList] = useState<KnowledgeBaseVO[]>(
[],
);
const [externalKBList, setExternalKBList] = useState<ExternalKBCardVO[]>([]);
const [selectedKbId, setSelectedKbId] = useState<string>('');
const [selectedKbType, setSelectedKbType] = useState<'builtin' | 'external'>(
'builtin',
);
const [detailDialogOpen, setDetailDialogOpen] = useState(false);
const [pluginSystemStatus, setPluginSystemStatus] =
useState<ApiRespPluginSystemStatus | null>(null);
useEffect(() => {
getKnowledgeBaseList();
getExternalKBList();
fetchPluginSystemStatus();
}, []);
async function fetchPluginSystemStatus() {
try {
const status = await httpClient.getPluginSystemStatus();
setPluginSystemStatus(status);
} catch (error) {
console.error('Failed to fetch plugin system status:', error);
}
}
async function getKnowledgeBaseList() {
const resp = await httpClient.getKnowledgeBases();
setKnowledgeBaseList(
resp.bases.map((kb: KnowledgeBase) => {
const currentTime = new Date();
const lastUpdatedTimeAgo = Math.floor(
(currentTime.getTime() -
new Date(kb.updated_at ?? currentTime.getTime()).getTime()) /
1000 /
60 /
60 /
24,
);
const lastUpdatedTimeAgoText =
lastUpdatedTimeAgo > 0
? ` ${lastUpdatedTimeAgo} ${t('knowledge.daysAgo')}`
: t('knowledge.today');
const currentTime = new Date();
return new KnowledgeBaseVO({
id: kb.uuid || '',
name: kb.name,
description: kb.description,
emoji: kb.emoji,
embeddingModelUUID: kb.embedding_model_uuid,
top_k: kb.top_k ?? 5,
lastUpdatedTimeAgo: lastUpdatedTimeAgoText,
});
}),
);
}
async function getExternalKBList() {
try {
const resp = await httpClient.getExternalKnowledgeBases();
setExternalKBList(
resp.bases.map((kb: ExternalKnowledgeBase) => {
const currentTime = new Date();
const lastUpdatedTimeAgo = Math.floor(
(currentTime.getTime() -
new Date(kb.created_at ?? currentTime.getTime()).getTime()) /
1000 /
60 /
60 /
24,
);
const lastUpdatedTimeAgoText =
lastUpdatedTimeAgo > 0
? ` ${lastUpdatedTimeAgo} ${t('knowledge.daysAgo')}`
: t('knowledge.today');
return new ExternalKBCardVO({
id: kb.uuid || '',
name: kb.name,
description: kb.description,
emoji: kb.emoji,
retrieverName: `${kb.plugin_author}/${kb.plugin_name}/${kb.retriever_name}`,
retrieverConfig: kb.retriever_config || {},
lastUpdatedTimeAgo: lastUpdatedTimeAgoText,
pluginAuthor: kb.plugin_author,
pluginName: kb.plugin_name,
});
}),
const kbs = resp.bases.map((kb: KnowledgeBase) => {
const lastUpdatedTimeAgo = Math.floor(
(currentTime.getTime() -
new Date(kb.updated_at ?? currentTime.getTime()).getTime()) /
1000 /
60 /
60 /
24,
);
} catch (error) {
console.error('Failed to load external knowledge bases:', error);
}
const lastUpdatedTimeAgoText =
lastUpdatedTimeAgo > 0
? ` ${lastUpdatedTimeAgo} ${t('knowledge.daysAgo')}`
: t('knowledge.today');
return new KnowledgeBaseVO({
id: kb.uuid || '',
name: kb.name,
description: kb.description,
emoji: kb.emoji,
lastUpdatedTimeAgo: lastUpdatedTimeAgoText,
ragEngine: kb.knowledge_engine,
ragEnginePluginId: kb.knowledge_engine_plugin_id,
});
});
setKnowledgeBaseList(kbs);
}
const handleKBCardClick = (kbId: string) => {
setSelectedKbId(kbId);
setSelectedKbType('builtin');
setDetailDialogOpen(true);
};
const handleCreateKBClick = () => {
setSelectedKbId('');
setSelectedKbType('builtin');
setDetailDialogOpen(true);
};
const handleExternalKBCardClick = (kbId: string) => {
setSelectedKbId(kbId);
setSelectedKbType('external');
setDetailDialogOpen(true);
};
const handleCreateExternalKB = () => {
setSelectedKbId('');
setSelectedKbType('external');
setDetailDialogOpen(true);
};
@@ -146,30 +71,18 @@ export default function KnowledgePage() {
};
const handleKbDeleted = () => {
if (selectedKbType === 'builtin') {
getKnowledgeBaseList();
} else {
getExternalKBList();
}
getKnowledgeBaseList();
setDetailDialogOpen(false);
};
const handleNewKbCreated = (newKbId: string) => {
if (selectedKbType === 'builtin') {
getKnowledgeBaseList();
} else {
getExternalKBList();
}
getKnowledgeBaseList();
setSelectedKbId(newKbId);
setDetailDialogOpen(true);
};
const handleKbUpdated = () => {
if (selectedKbType === 'builtin') {
getKnowledgeBaseList();
} else {
getExternalKBList();
}
getKnowledgeBaseList();
};
return (
@@ -178,73 +91,28 @@ export default function KnowledgePage() {
open={detailDialogOpen}
onOpenChange={setDetailDialogOpen}
kbId={selectedKbId || undefined}
kbType={selectedKbType}
onFormCancel={handleFormCancel}
onKbDeleted={handleKbDeleted}
onNewKbCreated={handleNewKbCreated}
onKbUpdated={handleKbUpdated}
/>
<Tabs value={activeTab} onValueChange={setActiveTab} className="w-full">
<div className="flex flex-row justify-between items-center px-[0.8rem]">
<TabsList className="shadow-md py-5 bg-[#f0f0f0] dark:bg-[#2a2a2e]">
<TabsTrigger value="builtin" className="px-6 py-4 cursor-pointer">
{t('knowledge.builtIn')}
</TabsTrigger>
{/* Only show external tab if plugin system is enabled and connected */}
{pluginSystemStatus?.is_enable &&
pluginSystemStatus?.is_connected && (
<TabsTrigger
value="external"
className="px-6 py-4 cursor-pointer"
>
{t('knowledge.external')}
</TabsTrigger>
)}
</TabsList>
</div>
<div className={styles.knowledgeListContainer}>
<CreateCardComponent
width={'100%'}
height={'10rem'}
plusSize={'90px'}
onClick={handleCreateKBClick}
/>
<TabsContent value="builtin">
<div className={styles.knowledgeListContainer}>
<CreateCardComponent
width={'100%'}
height={'10rem'}
plusSize={'90px'}
onClick={handleCreateKBClick}
/>
{knowledgeBaseList.map((kb) => {
return (
<div key={kb.id} onClick={() => handleKBCardClick(kb.id)}>
<KBCard kbCardVO={kb} />
</div>
);
})}
</div>
</TabsContent>
<TabsContent value="external">
<div className={styles.knowledgeListContainer}>
<CreateCardComponent
width={'100%'}
height={'10rem'}
plusSize={'90px'}
onClick={handleCreateExternalKB}
/>
{externalKBList.map((kb) => {
return (
<div
key={kb.id}
onClick={() => handleExternalKBCardClick(kb.id)}
>
<ExternalKBCard kbCardVO={kb} />
</div>
);
})}
</div>
</TabsContent>
</Tabs>
{knowledgeBaseList.map((kb) => {
return (
<div key={kb.id} onClick={() => handleKBCardClick(kb.id)}>
<KBCard kbCardVO={kb} />
</div>
);
})}
</div>
</div>
);
}

73
web/src/app/home/plugins/components/plugin-installed/PluginComponentList.tsx
View File

@@ -1,5 +1,5 @@
import { TFunction } from 'i18next';
import { Wrench, AudioWaveform, Hash, Book } from 'lucide-react';
import { Wrench, AudioWaveform, Hash, Book, FileText } from 'lucide-react';
import { Badge } from '@/components/ui/badge';
export default function PluginComponentList({
@@ -21,7 +21,8 @@ export default function PluginComponentList({
Tool: <Wrench className="w-5 h-5" />,
EventListener: <AudioWaveform className="w-5 h-5" />,
Command: <Hash className="w-5 h-5" />,
KnowledgeRetriever: <Book className="w-5 h-5" />,
KnowledgeEngine: <Book className="w-5 h-5" />,
Parser: <FileText className="w-5 h-5" />,
};
const componentKindList = Object.keys(components || {});
@@ -32,45 +33,39 @@ export default function PluginComponentList({
{componentKindList.length > 0 && (
<>
{componentKindList.map((kind) => {
return (
<>
{useBadge && (
<Badge
key={kind}
variant="outline"
className="flex items-center gap-1"
>
{kindIconMap[kind]}
{/* 响应式显示组件名称：在中等屏幕以上显示 */}
{responsive ? (
<span className="hidden md:inline">
{t('plugins.componentName.' + kind)}
</span>
) : (
showComponentName && t('plugins.componentName.' + kind)
)}
<span className="ml-1">{components[kind]}</span>
</Badge>
return useBadge ? (
<Badge
key={kind}
variant="outline"
className="flex items-center gap-1"
>
{kindIconMap[kind]}
{/* 响应式显示组件名称：在中等屏幕以上显示 */}
{responsive ? (
<span className="hidden md:inline">
{t('plugins.componentName.' + kind)}
</span>
) : (
showComponentName && t('plugins.componentName.' + kind)
)}
{!useBadge && (
<div
key={kind}
className="flex flex-row items-center justify-start gap-[0.2rem]"
>
{kindIconMap[kind]}
{/* 响应式显示组件名称：在中等屏幕以上显示 */}
{responsive ? (
<span className="hidden md:inline">
{t('plugins.componentName.' + kind)}
</span>
) : (
showComponentName && t('plugins.componentName.' + kind)
)}
<span className="ml-1">{components[kind]}</span>
</div>
<span className="ml-1">{components[kind]}</span>
</Badge>
) : (
<div
key={kind}
className="flex flex-row items-center justify-start gap-[0.2rem]"
>
{kindIconMap[kind]}
{/* 响应式显示组件名称：在中等屏幕以上显示 */}
{responsive ? (
<span className="hidden md:inline">
{t('plugins.componentName.' + kind)}
</span>
) : (
showComponentName && t('plugins.componentName.' + kind)
)}
</>
<span className="ml-1">{components[kind]}</span>
</div>
);
})}
</>

23
web/src/app/home/plugins/components/plugin-market/PluginMarketComponent.tsx
View File

@@ -17,7 +17,14 @@ import {
SelectValue,
} from '@/components/ui/select';
import { ToggleGroup, ToggleGroupItem } from '@/components/ui/toggle-group';
import { Search, Wrench, AudioWaveform, Hash, Book } from 'lucide-react';
import {
Search,
Wrench,
AudioWaveform,
Hash,
Book,
FileText,
} from 'lucide-react';
import PluginMarketCardComponent from './plugin-market-card/PluginMarketCardComponent';
import { PluginMarketCardVO } from './plugin-market-card/PluginMarketCardVO';
import { getCloudServiceClientSync } from '@/app/infra/http';
@@ -499,12 +506,20 @@ function MarketPageContent({
{t('plugins.componentName.EventListener')}
</ToggleGroupItem>
<ToggleGroupItem
value="KnowledgeRetriever"
aria-label="KnowledgeRetriever"
value="KnowledgeEngine"
aria-label="KnowledgeEngine"
className="text-xs sm:text-sm cursor-pointer"
>
<Book className="h-4 w-4 mr-1" />
{t('plugins.componentName.KnowledgeRetriever')}
{t('plugins.componentName.KnowledgeEngine')}
</ToggleGroupItem>
<ToggleGroupItem
value="Parser"
aria-label="Parser"
className="text-xs sm:text-sm cursor-pointer"
>
<FileText className="h-4 w-4 mr-1" />
{t('plugins.componentName.Parser')}
</ToggleGroupItem>
</ToggleGroup>
</div>

4
web/src/app/home/plugins/components/plugin-market/plugin-market-card/PluginMarketCardComponent.tsx
View File

@@ -8,6 +8,7 @@ import {
Download,
ExternalLink,
Book,
FileText,
} from 'lucide-react';
import { useState } from 'react';
import { Button } from '@/components/ui/button';
@@ -41,7 +42,8 @@ export default function PluginMarketCardComponent({
Tool: <Wrench className="w-4 h-4" />,
EventListener: <AudioWaveform className="w-4 h-4" />,
Command: <Hash className="w-4 h-4" />,
KnowledgeRetriever: <Book className="w-4 h-4" />,
KnowledgeEngine: <Book className="w-4 h-4" />,
Parser: <FileText className="w-4 h-4" />,
};
return (

59
web/src/app/infra/entities/api/index.ts
View File

@@ -70,17 +70,6 @@ export interface LLMModel {
extra_args?: object;
}
export interface KnowledgeBase {
uuid?: string;
name: string;
description: string;
embedding_model_uuid: string;
created_at?: string;
updated_at?: string;
top_k: number;
emoji?: string;
}
export interface ApiRespProviderEmbeddingModels {
models: EmbeddingModel[];
}
@@ -166,31 +155,47 @@ export interface KnowledgeBase {
uuid?: string;
name: string;
description: string;
embedding_model_uuid: string;
top_k: number;
created_at?: string;
updated_at?: string;
emoji?: string;
// New unified fields
knowledge_engine_plugin_id?: string;
creation_settings?: Record<string, unknown>;
retrieval_settings?: Record<string, unknown>;
knowledge_engine?: KnowledgeEngineInfo;
}
export interface ExternalKnowledgeBase {
uuid?: string;
name: string;
description: string;
created_at?: string;
plugin_author: string;
plugin_name: string;
retriever_name: string;
retriever_config?: Record<string, unknown>;
emoji?: string;
// Knowledge Engine types
export interface KnowledgeEngineInfo {
plugin_id: string | null;
name: I18nObject;
capabilities: string[];
}
export interface ApiRespExternalKnowledgeBases {
bases: ExternalKnowledgeBase[];
export interface KnowledgeEngine {
plugin_id: string;
name: I18nObject;
description?: I18nObject;
capabilities: string[];
// Schema format: Array of form field definitions (IDynamicFormItemSchema-like)
// Each item: { name, label, type, required, default, description?, options? }
creation_schema?: unknown[];
retrieval_schema?: unknown[];
}
export interface ApiRespExternalKnowledgeBase {
base: ExternalKnowledgeBase;
export interface ApiRespKnowledgeEngines {
engines: KnowledgeEngine[];
}
export interface ParserInfo {
plugin_id: string;
name: I18nObject;
description?: I18nObject;
supported_mime_types: string[];
}
export interface ApiRespParsers {
parsers: ParserInfo[];
}
export interface ApiRespKnowledgeBaseFiles {

9
web/src/app/infra/entities/form/dynamic.ts
View File

@@ -1,5 +1,12 @@
import { I18nObject } from '@/app/infra/entities/common';
export interface IShowIfCondition {
field: string;
operator: 'eq' | 'neq' | 'in';
// eslint-disable-next-line @typescript-eslint/no-explicit-any
value: any;
}
export interface IDynamicFormItemSchema {
id: string;
default: string | number | boolean | Array<unknown>;
@@ -9,6 +16,7 @@ export interface IDynamicFormItemSchema {
type: DynamicFormItemType;
description?: I18nObject;
options?: IDynamicFormItemOption[];
show_if?: IShowIfCondition;
/** when type is PLUGIN_SELECTOR, the scopes is the scopes of components(plugin contains), the default is all */
scopes?: string[];
@@ -26,6 +34,7 @@ export enum DynamicFormItemType {
FILE_ARRAY = 'array[file]',
SELECT = 'select',
LLM_MODEL_SELECTOR = 'llm-model-selector',
EMBEDDING_MODEL_SELECTOR = 'embedding-model-selector',
PROMPT_EDITOR = 'prompt-editor',
UNKNOWN = 'unknown',
KNOWLEDGE_BASE_SELECTOR = 'knowledge-base-selector',

57
web/src/app/infra/http/BackendClient.ts
View File

@@ -35,12 +35,11 @@ import {
ApiRespMCPServers,
ApiRespMCPServer,
MCPServer,
ExternalKnowledgeBase,
ApiRespExternalKnowledgeBases,
ApiRespExternalKnowledgeBase,
ApiRespModelProviders,
ApiRespModelProvider,
ModelProvider,
ApiRespKnowledgeEngines,
ApiRespParsers,
} from '@/app/infra/entities/api';
import { Plugin } from '@/app/infra/entities/plugin';
import { GetBotLogsRequest } from '@/app/infra/http/requestParam/bots/GetBotLogsRequest';
@@ -435,9 +434,11 @@ export class BackendClient extends BaseHttpClient {
public uploadKnowledgeBaseFile(
uuid: string,
file_id: string,
parserPluginId?: string,
): Promise<object> {
return this.post(`/api/v1/knowledge/bases/${uuid}/files`, {
file_id,
parser_plugin_id: parserPluginId,
});
}
@@ -461,49 +462,23 @@ export class BackendClient extends BaseHttpClient {
public retrieveKnowledgeBase(
uuid: string,
query: string,
retrievalSettings?: Record<string, unknown>,
): Promise<ApiRespKnowledgeBaseRetrieve> {
return this.post(`/api/v1/knowledge/bases/${uuid}/retrieve`, { query });
}
// ============ External Knowledge Base API ============
public getExternalKnowledgeBases(): Promise<ApiRespExternalKnowledgeBases> {
return this.get('/api/v1/knowledge/external-bases');
}
public getExternalKnowledgeBase(
uuid: string,
): Promise<ApiRespExternalKnowledgeBase> {
return this.get(`/api/v1/knowledge/external-bases/${uuid}`);
}
public createExternalKnowledgeBase(
base: ExternalKnowledgeBase,
): Promise<{ uuid: string }> {
return this.post('/api/v1/knowledge/external-bases', base);
}
public updateExternalKnowledgeBase(
uuid: string,
base: ExternalKnowledgeBase,
): Promise<{ uuid: string }> {
return this.put(`/api/v1/knowledge/external-bases/${uuid}`, base);
}
public deleteExternalKnowledgeBase(uuid: string): Promise<object> {
return this.delete(`/api/v1/knowledge/external-bases/${uuid}`);
}
public retrieveExternalKnowledgeBase(
uuid: string,
query: string,
): Promise<ApiRespKnowledgeBaseRetrieve> {
return this.post(`/api/v1/knowledge/external-bases/${uuid}/retrieve`, {
return this.post(`/api/v1/knowledge/bases/${uuid}/retrieve`, {
query,
retrieval_settings: retrievalSettings ?? {},
});
}
public listKnowledgeRetrievers(): Promise<{ retrievers: unknown[] }> {
return this.get('/api/v1/knowledge/external-bases/retrievers');
// ============ Knowledge Engines API ============
public getKnowledgeEngines(): Promise<ApiRespKnowledgeEngines> {
return this.get('/api/v1/knowledge/engines');
}
// ============ Parsers API ============
public listParsers(mimeType?: string): Promise<ApiRespParsers> {
const params = mimeType ? `?mime_type=${encodeURIComponent(mimeType)}` : '';
return this.get(`/api/v1/knowledge/parsers${params}`);
}
// ============ Plugins API ============

30
web/src/i18n/locales/en-US.ts
View File

@@ -48,6 +48,7 @@ const enUS = {
test: 'Test',
forgotPassword: 'Forgot Password?',
loading: 'Loading...',
fieldRequired: 'This field is required',
or: 'or',
loginWithSpace: 'Login with Space',
spaceLoginRecommended:
@@ -371,7 +372,8 @@ const enUS = {
Tool: 'Tool',
EventListener: 'Event Listener',
Command: 'Command',
KnowledgeRetriever: 'Knowledge Retriever',
KnowledgeEngine: 'Knowledge Engine',
Parser: 'Parser',
},
uploadLocal: 'Upload Local',
debugging: 'Debugging',
@@ -726,6 +728,12 @@ const enUS = {
processing: 'Processing',
completed: 'Completed',
failed: 'Failed',
selectParser: 'Select Parser',
builtInParser: 'Provided by Knowledge engine',
noParserAvailable:
'No parser supports this file type. Please install a parser plugin that can handle this format.',
confirmUpload: 'Upload',
cancelUpload: 'Cancel',
},
deleteKnowledgeBaseConfirmation:
'Are you sure you want to delete this knowledge base? All documents in this knowledge base will be deleted.',
@@ -738,8 +746,24 @@ const enUS = {
fileName: 'File Name',
noResults: 'No results',
retrieveError: 'Retrieve failed',
builtIn: 'Built-in',
external: 'External',
unknownEngine: 'Unknown Engine',
knowledgeEngine: 'Knowledge Engine',
knowledgeEngineRequired: 'Knowledge engine is required',
selectKnowledgeEngine: 'Select Knowledge Engine',
builtInEngine: 'Built-in Engine',
cannotChangeKnowledgeEngine:
'Knowledge engine cannot be changed after creation',
engineSettings: 'Engine Settings',
engineSettingsReadonly: 'read-only in edit mode',
retrievalSettings: 'Retrieval Settings',
noEnginesAvailable: 'No knowledge base engines available',
installEngineHint: 'Please install a knowledge base plugin first',
createKnowledgeBaseFailed: 'Failed to create knowledge base',
loadKnowledgeBaseFailed: 'Failed to load knowledge base',
deleteKnowledgeBaseFailed: 'Failed to delete knowledge base',
getKnowledgeBaseListError: 'Failed to get knowledge base list: ',
embeddingModel: 'Embedding Model',
embeddingModelRequired: 'Embedding model is required for this engine',
addExternal: 'Add External Knowledge Base',
createExternalSuccess: 'External knowledge base created successfully',
updateExternalSuccess: 'External knowledge base updated successfully',

16
web/src/i18n/locales/ja-JP.ts
View File

@@ -49,6 +49,7 @@ const jaJP = {
test: 'テスト',
forgotPassword: 'パスワードを忘れた？',
loading: '読み込み中...',
fieldRequired: 'この項目は必須です',
or: 'または',
loginWithSpace: 'Space でログイン',
spaceLoginRecommended:
@@ -371,7 +372,8 @@ const jaJP = {
Tool: 'ツール',
EventListener: 'イベント監視器',
Command: 'コマンド',
KnowledgeRetriever: '知識検索',
KnowledgeEngine: '知識エンジン',
Parser: 'パーサー',
},
uploadLocal: 'ローカルアップロード',
debugging: 'デバッグ中',
@@ -729,6 +731,12 @@ const jaJP = {
processing: '処理中',
completed: '完了',
failed: '失敗',
selectParser: 'パーサーを選択',
builtInParser: '知識エンジンが提供',
noParserAvailable:
'このファイル形式に対応するパーサーがありません。対応するパーサープラグインをインストールしてください。',
confirmUpload: 'アップロード',
cancelUpload: 'キャンセル',
},
deleteKnowledgeBaseConfirmation:
'本当にこの知識ベースを削除しますか？この知識ベースに紐付けられたドキュメントは削除されます。',
@@ -741,8 +749,10 @@ const jaJP = {
fileName: 'ファイル名',
noResults: '検索結果がありません',
retrieveError: '検索に失敗しました',
builtIn: '内蔵',
external: '外部ナレッジベース',
unknownEngine: '不明なエンジン',
loadKnowledgeBaseFailed: 'ナレッジベースの読み込みに失敗しました',
deleteKnowledgeBaseFailed: 'ナレッジベースの削除に失敗しました',
getKnowledgeBaseListError: 'ナレッジベース一覧の取得に失敗しました：',
addExternal: '外部ナレッジベースを追加',
createExternalSuccess: '外部ナレッジベースが正常に作成されました',
updateExternalSuccess: '外部ナレッジベースが正常に更新されました',

29
web/src/i18n/locales/zh-Hans.ts
View File

@@ -48,6 +48,7 @@ const zhHans = {
test: '测试',
forgotPassword: '忘记密码？',
loading: '加载中...',
fieldRequired: '此字段为必填项',
or: '或',
loginWithSpace: '通过 Space 登录',
spaceLoginRecommended: '推荐：使用官方提供的稳定模型 API 和云服务',
@@ -353,7 +354,8 @@ const zhHans = {
Tool: '工具',
EventListener: '事件监听器',
Command: '命令',
KnowledgeRetriever: '知识检索',
KnowledgeEngine: '知识引擎',
Parser: '解析器',
},
uploadLocal: '本地上传',
debugging: '调试中',
@@ -696,6 +698,12 @@ const zhHans = {
processing: '处理中',
completed: '完成',
failed: '失败',
selectParser: '选择解析器',
builtInParser: '由知识引擎提供',
noParserAvailable:
'没有解析器支持此文件类型，请安装支持该格式的解析器插件。',
confirmUpload: '上传',
cancelUpload: '取消',
},
deleteKnowledgeBaseConfirmation:
'你确定要删除这个知识库吗？此知识库下的所有文档将被删除。',
@@ -708,8 +716,23 @@ const zhHans = {
fileName: '文件名',
noResults: '暂无结果',
retrieveError: '检索失败',
builtIn: '内置',
external: '外部知识库',
unknownEngine: '未知引擎',
knowledgeEngine: '知识引擎',
knowledgeEngineRequired: '知识引擎不能为空',
selectKnowledgeEngine: '选择知识引擎',
builtInEngine: '内置引擎',
cannotChangeKnowledgeEngine: '知识库创建后不可修改知识引擎',
engineSettings: '引擎设置',
engineSettingsReadonly: '编辑模式下不可修改',
retrievalSettings: '检索设置',
noEnginesAvailable: '没有可用的知识库引擎',
installEngineHint: '请先安装知识库插件',
createKnowledgeBaseFailed: '知识库创建失败',
loadKnowledgeBaseFailed: '知识库加载失败',
deleteKnowledgeBaseFailed: '知识库删除失败',
getKnowledgeBaseListError: '获取知识库列表失败：',
embeddingModel: '嵌入模型',
embeddingModelRequired: '此引擎需要选择嵌入模型',
addExternal: '添加外部知识库',
createExternalSuccess: '外部知识库创建成功',
updateExternalSuccess: '外部知识库更新成功',

16
web/src/i18n/locales/zh-Hant.ts
View File

@@ -48,6 +48,7 @@ const zhHant = {
test: '測試',
forgotPassword: '忘記密碼？',
loading: '載入中...',
fieldRequired: '此欄位為必填',
or: '或',
loginWithSpace: '透過 Space 登入',
spaceLoginRecommended: '推薦：使用官方提供的穩定模型 API 和雲服務',
@@ -347,7 +348,8 @@ const zhHant = {
Tool: '工具',
EventListener: '事件監聽器',
Command: '命令',
KnowledgeRetriever: '知識檢索',
KnowledgeEngine: '知識引擎',
Parser: '解析器',
},
uploadLocal: '本地上傳',
debugging: '調試中',
@@ -689,6 +691,12 @@ const zhHant = {
processing: '處理中',
completed: '完成',
failed: '失敗',
selectParser: '選擇解析器',
builtInParser: '由知識引擎提供',
noParserAvailable:
'沒有解析器支援此檔案類型，請安裝支援該格式的解析器插件。',
confirmUpload: '上傳',
cancelUpload: '取消',
},
deleteKnowledgeBaseConfirmation:
'您確定要刪除這個知識庫嗎？此知識庫下的所有文檔將被刪除。',
@@ -701,8 +709,10 @@ const zhHant = {
fileName: '文檔名稱',
noResults: '暫無結果',
retrieveError: '檢索失敗',
builtIn: '內置',
external: '外部知識庫',
unknownEngine: '未知引擎',
loadKnowledgeBaseFailed: '知識庫載入失敗',
deleteKnowledgeBaseFailed: '知識庫刪除失敗',
getKnowledgeBaseListError: '取得知識庫列表失敗：',
addExternal: '添加外部知識庫',
createExternalSuccess: '外部知識庫創建成功',
updateExternalSuccess: '外部知識庫更新成功',