mirror of
https://github.com/langbot-app/LangBot.git
synced 2026-06-13 01:06:03 +00:00
Feat/rag plugin (#1995)
* [issue:1933] RAG engine plugin architecture (#1967) * refactor: migrate RAG knowledge services to a plugin-oriented host service architecture. * feat(rag): phase 2 core refactor with RPC Action handlers * feat: 为 RAG 插件添加知识库创建和删除事件通知，并优化了 RAG 动作的参数传递和枚举使用。 * feat: 统一知识库管理为RAG引擎，支持动态配置并移除旧的外部知识库组件。 * refactor(rag): remove plugin_adapter, inline logic into RuntimeKnowledgeBase BREAKING CHANGE: RAGPluginAdapter has been removed. All plugin communication is now handled directly by RuntimeKnowledgeBase. Architecture change: - Before: RuntimeKnowledgeBase → RAGPluginAdapter → plugin_connector - After: RuntimeKnowledgeBase → plugin_connector (direct) Changes to kbmgr.py (RuntimeKnowledgeBase): - Remove RAGPluginAdapter import and usage - Inline plugin communication methods: - _on_kb_create(): Notify plugin when KB is created - _on_kb_delete(): Notify plugin when KB is deleted - _ingest_document(): Call plugin for document ingestion - _retrieve(): Call plugin for retrieval - _delete_document(): Call plugin to delete document - Simplify dispose(): Only notify plugin, no built-in VDB assumption Changes to base.py (KnowledgeBaseInterface): - Remove get_type() abstract method (outdated internal/external concept) - Add get_rag_engine_plugin_id() abstract method Changes to localagent.py: - Remove get_type() call - Simplify top_k retrieval from KB entity Deleted files: - pkg/rag/knowledge/plugin_adapter.py Benefits: - Reduced abstraction layer, simpler code - Plugin communication logic centralized in RuntimeKnowledgeBase - Easier to understand and maintain 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(api): remove ExternalKnowledgeBase infrastructure BREAKING CHANGE: ExternalKnowledgeBase has been completely removed. All knowledge bases are now unified under the single KnowledgeBase model, differentiated by their rag_engine_plugin_id. Deleted files: - pkg/api/http/controller/groups/knowledge/external.py (ExternalKBController with /external-bases routes) - pkg/api/http/service/external_kb.py (ExternalKnowledgeBaseService) - pkg/rag/knowledge/external.py (ExternalKnowledgeBase implementation) Modified files: - pkg/entity/persistence/rag.py: Remove ExternalKnowledgeBase SQLAlchemy table definition - pkg/core/app.py: Remove external_kb_service attribute from LangBotApplication - pkg/core/stages/build_app.py: Remove external_kb_service initialization Migration notes: - Existing external knowledge base data should be migrated manually - API consumers should use /api/v1/knowledge/bases for all KB operations - Use /api/v1/knowledge/engines to discover available RAG engines 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(plugin): remove list_knowledge_retrievers from connector Remove deprecated list_knowledge_retrievers functionality from the plugin communication layer. This aligns with the SDK change that removed the LIST_KNOWLEDGE_RETRIEVERS action. Changes: - connector.py: Remove list_knowledge_retrievers() method - handler.py: Remove list_knowledge_retrievers() handler The functionality is replaced by the new /api/v1/knowledge/engines endpoint which lists available RAGEngine components with their capabilities and configuration schemas. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(service): update knowledge service with capability-based checks Replace type-based checks with capability-based checks for file operations, aligning with the unified knowledge base architecture. Changes to knowledge.py: - store_file(): Replace get_type() check with doc_ingestion capability check - delete_file(): Replace get_type() check with doc_ingestion capability check - list_rag_engines(): Remove list_knowledge_retrievers call, simplify to only list RAGEngine components (KnowledgeRetriever type removed) Changes to pipelines.py: - Minor cleanup related to knowledge base references The capability-based approach allows RAG engines to declare their supported features (doc_ingestion, chunking_config, rerank, hybrid_search) and the system responds accordingly, rather than hardcoding behavior based on internal/external type distinction. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(web): unify knowledge base UI, remove external KB components BREAKING CHANGE: The internal/external knowledge base distinction has been removed from the frontend. All knowledge bases are now displayed in a unified list, differentiated by their RAG engine. Changes to page.tsx: - Remove Tab component (内置/外置 tabs) - Remove selectedKbType state - Unified knowledge base list display - Single "Create Knowledge Base" button for all types Changes to KBDetailDialog.tsx: - Remove kbType prop - Simplify dialog logic for unified KB handling - Documents menu item conditionally shown based on doc_ingestion capability Changes to KBForm.tsx: - Remove retriever type handling code - Simplify form for unified KB creation - Dynamic form rendering based on RAG engine's creation_schema Changes to KBCardVO.ts: - Remove 'type' field from KBCardVO interface Changes to BackendClient.ts: - Remove all external KB related methods: - getExternalKnowledgeBases() - getExternalKnowledgeBase() - createExternalKnowledgeBase() - updateExternalKnowledgeBase() - deleteExternalKnowledgeBase() - retrieveFromExternalKnowledgeBase() Changes to api/index.ts: - Remove ExternalKnowledgeBase interface definition UI/UX improvements: - Users no longer need to understand internal vs external distinction - RAG engine selection is now the primary differentiator - Documents panel visibility is capability-driven (doc_ingestion) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(plugin): code review improvements for RAG handlers - Unify embed_model field naming to embedding_model_uuid only - Add structured error responses with error_type for RAG actions - Fix file_size and mime_type detection in _store_file_task - Improve error handling with detailed error context (error_type, original_error) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(rag): refactor KB dynamic form and vector manager - Frontend: Refactor Knowledge Base form using DynamicForm components. - Frontend: Remove obsolete jsonSchemaConverter utility. - Backend: Update VectorManager and PluginHandler to support new RAG architecture. - Chore: Update dependencies in pyproject.toml. * fix: code review fixes for RAG refactor - Remove DEBUG stderr outputs in handler.py - Move repeated `import json` to file top - Add warning log for unimplemented delete_by_filter 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(rag): consolidate valid_fields into entity constants Define MUTABLE_FIELDS, CREATE_FIELDS, ALL_DB_FIELDS as class constants in KnowledgeBase entity to eliminate duplication. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor: 将知识库获取和RAG引擎信息丰富逻辑移至知识库管理器。 * refactor(rag): introduce RAGRuntimeService and clean up plugin handler - Create RAGRuntimeService to encapsulate RAG capability implementation (Embedding, VectorOps). - Refactor PluginHandler to delegate RAG actions to RAGRuntimeService. - Move KnowledgeService enrichment and creation logic to RAGManager. - Register RAGRuntimeService in Application and BuildAppStage. - Clean up legacy code in KnowledgeService. * refactor(rag): standardize logger and fix type hints - Use self.ap.logger consistently in kbmgr.py and runtime.py, removing module-level loggers. - Fix type hints for retrieve_knowledge in handler.py and connector.py to match implementation returning dict. * refactor: 将引擎徽章的样式从 Tailwind CSS 类迁移到 CSS 模块。 * fix(web): resolve React rendering errors in plugins page - Fix missing key prop in PluginComponentList by using ternary instead of Fragment - Fix RAGEngine.name type to I18nObject and use extractI18nObject() for rendering - Preserves multi-language support 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(rag): update runtime service and web components * refactor: 优化知识库设置结构并增强前端距离显示健壮性。 * fix: 处理前端距离显示中的空值。 * fix(rag): document retrieve ui and kbmgr top_k validation * 更新 uv.lock 中的 PyPI 镜像源为官方地址。 * fix: address code review issues for RAG engine plugin architecture P0 fixes: - Fix ALL_DB_FIELDS missing collection_id and emoji fields - Move rag_engine_plugin_id to CREATE_FIELDS (immutable after creation) - Fix creation_settings mutable default value (dict -> None) - Rename vector delete method to delete_by_file_id for correct semantics - Fix delete_by_filter to raise NotImplementedError instead of silent no-op - Add database migration script (dbm019) for new columns and table cleanup P1 fixes: - Clean up design-hesitation comments in connector.py - Add _parse_plugin_id() with format validation for all RAG methods - Make _retrieve() raise exceptions instead of silently returning empty results - Extract _make_rag_error_response() helper for clean error formatting - Remove unused imports from handler.py P2 fixes: - Fix runtime.py indentation inconsistencies - Simplify get_file_stream to use storage abstraction uniformly - Reduce redundant DB queries in knowledge service (extract _check_doc_capability) - Fix engines.py URL encoding: use <path:plugin_id> instead of __ replacement - Add read-only mode for engine settings in KBForm edit mode - Simplify page.tsx handleKBCardClick to pass only kbId string Co-authored-by: Cursor <cursoragent@cursor.com> * fix: address code review findings for RAG plugin architecture - Frontend: add retrieval_settings param to retrieveKnowledgeBase API call - Backend: return {uuid} from PUT knowledge base to match frontend expectation - Backend: validate query is non-empty in retrieve endpoint (400 on empty) - Backend: rename vector_delete ids→file_ids for semantic clarity, keep backward compat by accepting both 'file_ids' and 'ids' in RPC handler - Backend: ensure rag_engine.name fallback is always I18nObject-compatible dict, preventing frontend extractI18nObject from receiving plain strings - Migration: fix misleading docstring about external_kb data migration Co-authored-by: Cursor <cursoragent@cursor.com> * Update langbot-plugin version to 0.2.6 * chore: update required database version from 18 to 19 * refactor: remove unused polymorphic component framework * chore: fix lint and format issues for python and frontend * fix(plugin): remove legacy `ids` fallback in rag_vector_delete handler SDK now sends `file_ids` directly, the `ids` backward-compat fallback is no longer needed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(rag): deep review fixes for critical bugs, security and quality Critical: - Fix StorageMgr.load() -> storage_provider.load() (C1, AttributeError) - Update required_database_version 18 -> 19 (C2, migration never runs) Security: - Add path traversal validation in get_file_stream (C11) - Add vectors/ids/metadata length validation in rag_vector_upsert (C12) Logic fixes: - Legacy KBs: set capabilities to [] instead of ['doc_ingestion'] (C4) - Fix store_file return type int -> str (C5) - Fix retrieve_knowledge return [] -> {'results': []} when disabled (C6) - Re-raise exception in _on_kb_create instead of silently swallowing (C7) - Log warning when KB not found in memory during delete (C8) API fixes: - Catch ValueError as 400 in create_knowledge_base endpoint (C15) - Validate plugin_id format in engines endpoints (C16) Quality: - Remove dead if/else in migration with identical branches (C17) - Fix variable shadowing: rag_context -> rag_context_text (C18) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: remove unused os import to fix ruff lint Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(plugin): remove PolymorphicComponent sync from LangBot side Remove sync_polymorphic_component_instances() from connector and handler, and the post-connection sync call in initialize(). This dead code synced an always-empty list of polymorphic instances that were never created. Companion change to langbot-plugin-sdk PolymorphicComponent removal. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(rag): fix vector_delete count bug and remove vestigial instance_id parameter 1. vector_delete: assign return value from delete_by_filter to count instead of silently returning 0 for filter-based deletion. 2. Remove instance_id parameter from the entire retrieve_knowledge call chain (kbmgr → connector → handler → runtime). This parameter was a remnant of the PolymorphicComponent mechanism and is no longer used — RAGEngine operates as a stateless singleton. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(web): 支持 creation_schema 字段级别的 editable 属性控制编辑模式可修改性 - IDynamicFormItemSchema 添加 editable 可选属性 - DynamicFormItemConfig 透传 editable 属性 - DynamicFormComponent 接收 isEditing prop，按字段 editable 值控制禁用 - KBForm 解析 editable 并传递 isEditing 给动态表单组件 - editable 未指定时默认可编辑，editable: false 时编辑模式下禁用该字段 * feat(storage): 添加 size() 抽象方法及 LocalStorage/S3 实现 支持获取存储对象大小，S3 使用 head_object 避免下载整个文件 * fix(migration): 删除 external_knowledge_bases 表前记录日志警告 - 迁移时如果表中存在数据，先 warning 日志记录避免无感数据丢失 - 添加 chunk 清理注释说明：仅对旧版非插件架构 KB 有效 * fix(web): 修复检索结果长文本撑大容器导致查询按钮不可见 KBDetailDialog 的 main 容器添加 min-w-0 overflow-x-hidden， 限制 flex-1 子容器宽度，防止 Dify RAG 长文本撑出 Dialog 边界 * fix(rag): address code review issues for plugin architecture PR - Fix SQL injection in migration helpers by using bind parameters - Move numpy import to module level in vector/mgr.py - Improve path traversal validation using posixpath.normpath - Add call_rag_retrieve to connector, eliminating duplicate plugin_id parsing in kbmgr.py _retrieve - Normalize typing style to modern dict/list/None syntax Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * style(web): fix prettier formatting errors Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(rag): update embedding handling in RuntimeConnectionHandler - Renamed RAG_EMBED_DOCUMENTS and RAG_EMBED_QUERY actions to INVOKE_EMBEDDING for clarity. - Removed embed_documents and embed_query methods from RuntimeEmbeddingModel and RAGRuntimeService. - Integrated embedding model retrieval directly in the invoke_embedding method, improving error handling for missing models. - Updated the embedding invocation logic to streamline the process and enhance error reporting. * refactor(web): replace KnowledgeRetriever with RAGEngine across frontend and tests KnowledgeRetriever component type has been removed in favor of the new RAGEngine architecture. Update all remaining references in i18n locales, plugin component icon mappings, marketplace filter, and unit tests. Addresses reviewer notes from RockChinQ on PR #1967. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(rag): address critical bugs found in deep review - Fix path traversal bypass in runtime.py (check all path components for '..') - Use normalized path for file loading instead of raw user input - Change knowledge_bases from list to dict for O(1) lookup and race safety - Add rollback on KB creation failure (clean up DB + runtime on plugin error) - Add null check after KB update in knowledge service - Fix file extension parsing to use os.path.splitext instead of split('.') (handles multi-dot filenames like 'report.v2.pdf' correctly) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(rag): address remaining review issues across frontend and backend Frontend: - Fix KB delete: use async/await with error handling instead of fire-and-forget - Fix capabilities null check: add optional chaining to prevent crash - Add toast.error on KB info load failure instead of silent console.error - Replace hard-coded Chinese validation message with i18n key - Replace hard-coded English error messages in DynamicFormItemComponent with i18n - Optimize document polling: stop when all documents reach terminal state - Add i18n keys (fieldRequired, loadKnowledgeBaseFailed, deleteKnowledgeBaseFailed, getKnowledgeBaseListError) to all 4 locales Backend: - Fix KB delete atomicity: delete from DB first, then notify plugin - Add RAG engine plugin existence validation before creating KB Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * style(rag): fix ruff formatting in kbmgr.py Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Junyan Qin <rockchinq@gmail.com> * chore: bump langbot-plugin to 0.3.0 (#1992) * chore: correct sdk version to 0.3.0a1 * feat: normalize rag related actions' names * refactor(rag): align IngestionContext fields with SDK changes Remove redundant `chunking_strategy` field and rename `custom_settings` to `creation_settings` to match the updated SDK entity definitions (langbot-plugin-sdk#36). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * style: fix ruff formatting Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(rag): enforce immutability of embedding_model_uuid and non-editable creation_settings fields Remove embedding_model_uuid from MUTABLE_FIELDS to prevent post-creation modification via API. Add backend validation for creation_settings to preserve fields marked editable:false in the plugin's creation schema. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * style(rag): fix ruff formatting in knowledge service Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(rag): split settings into immutable creation_settings and mutable retrieval_settings - Remove standalone embedding_model_uuid and top_k columns from KB entity - Add retrieval_settings column; update MUTABLE_FIELDS/CREATE_FIELDS accordingly - Merge migration logic into dbm019 (add retrieval_settings, migrate top_k and embedding_model_uuid into JSON settings, drop old columns on PostgreSQL) - Remove _filter_creation_settings and per-field editable concept - Frontend: creation_settings fields are all disabled when editing, retrieval_settings fields are always editable via a second DynamicFormComponent - Remove editable from IDynamicFormItemSchema, DynamicFormItemConfig - Clean up KBCardVO, KnowledgeBase API type, and localagent runner Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * bugfix: if ingest_document failed,not raise exep * fix: ruff lint * refactor(rag): remove unused _get_kb_entity method from RAGRuntimeService * feat(vector): implement metadata filters for vector_search and vector_delete (#1997) Add functional metadata filter support across all 5 VDB backends using Chroma-style where syntax as the canonical format. Previously the filters parameter existed throughout the stack but was entirely ignored. - Add filter_utils.py with normalize_filter() and strip_unsupported_fields() - Implement filter in search() and add delete_by_filter() for all backends: Chroma/SeekDB (native passthrough), Qdrant (translated to models.Filter), Milvus (translated to expr string), pgvector (translated to SQLAlchemy conditions) - Milvus/pgvector limited to {text, file_id, chunk_uuid}; other fields logged and ignored - Replace delete_by_filter() NotImplementedError with backend delegation in mgr.py - Populate retrieval_context['filters'] from settings in kbmgr._retrieve() - Pass search_type/query_text/documents through handler and runtime service Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * style(vector): fix ruff formatting Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(vector): remove numpy dependency and fix SeekDB search modes - Remove numpy array conversion for query vectors; all VDB backends accept list[float] directly - Remove redundant get_or_create_collection call from upsert; backends handle collection creation internally in add_embeddings - Fix SeekDB to raise ValueError when vector dimension is unknown instead of defaulting to 384 - Use hybrid_search() for full-text and hybrid search modes in SeekDB, since pyseekdb's query() always requires embeddings Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(vector): escape single quotes in SeekDB documents and metadata Document text containing apostrophes (e.g. "don't", "it's") causes SQL syntax errors in OceanBase because single quotes were not in the escape table. Add single-quote escaping and apply the escape table to the documents parameter in add_embeddings(), not just metadata. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(vector): use standard SQL escaping for single quotes in SeekDB Change single quote escaping from MySQL-style \' to standard SQL '' (doubled quote). The backslash escape is not recognized by OceanBase in NO_BACKSLASH_ESCAPES mode, causing SQL syntax errors when metadata text contains apostrophes (e.g. O'Shea in academic citations). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(rag): persist retrieval_settings on knowledge base creation retrieval_settings was not being passed from the service layer to RAGManager.create_knowledge_base(), causing retrieval schema fields (e.g. query_rewrite) to be lost on initial KB creation. They only took effect after a subsequent edit/update. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(web): add show_if conditional rendering for dynamic forms Support conditional field visibility in plugin-defined forms via show_if rules (eq, neq, in operators). Fields can depend on values from the same form or cross-reference between creation and retrieval settings via externalDependentValues. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(rag): replace base64 with chunked file transfer for get_rag_file_stream Use send_file() instead of base64 encoding for returning file content in the GET_RAG_FILE_STREAM handler, avoiding memory issues with large files. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(parser): add parser plugin integration and capability-aware upload UI (#2000) * feat(parser): add parser plugin integration and capability-aware upload UI Backend: add parser plugin API endpoints (list/invoke), connector and handler support for parser actions, and KB manager passthrough. Frontend: thread ragEngineCapabilities prop to FileUploadZone and use doc_parsing capability to conditionally show the RAG engine option in the parser selector. When no parser is available, show a warning prompting users to install a parser plugin. Update i18n: rename builtInParser to "Provided by RAG engine" and add noParserAvailable warning message in all 4 locales. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(parser): replace base64 with chunked file transfer and remove stale cache - Remove @alru_cache from list_parsers() and list_rag_engines() - Replace inline base64 file content with send_file/read_local_file chunked transfer pattern in parse_document and invoke_parser flows - Remove unused base64 import from kbmgr.py Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * feat(web): add Parser component kind to plugin market UI and i18n Add Parser to kindIconMap, market filter toggle, and all 4 locale files so parser plugins are properly displayed and filterable in the plugin market, matching the existing RAGEngine treatment. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * style(web): fix prettier formatting from merge Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: rename RAGEngine to KnowledgeEngine across frontend and backend * fix(web): fix I18nObject import path in FileUploadZone and KBDoc * chore: format files involved in RAGEngine to KnowledgeEngine refactor * refactor: change rag engine to knowledge engine * fix: update langbot-plugin version to 0.3.0rc1 * chore: disable migration 20 for now --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Junyan Qin <rockchinq@gmail.com>
This commit is contained in:
huanghuoguoguo
69
src/langbot/pkg/vector/filter_utils.py
Normal file
69
src/langbot/pkg/vector/filter_utils.py
Normal file
@@ -0,0 +1,69 @@
|
||||
"""Shared utilities for metadata filter handling across VDB backends.
|
||||
|
||||
Canonical filter format (Chroma-style ``where`` syntax):
|
||||
|
||||
{"file_id": "abc"} # implicit $eq
|
||||
{"file_id": {"$eq": "abc"}} # explicit $eq
|
||||
{"created_at": {"$gte": 1700000000}} # comparison
|
||||
{"file_type": {"$in": ["pdf", "docx"]}} # in-list
|
||||
|
||||
Multiple top-level keys are AND-ed. Supported operators:
|
||||
``$eq``, ``$ne``, ``$gt``, ``$gte``, ``$lt``, ``$lte``, ``$in``, ``$nin``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
SUPPORTED_OPS = frozenset({'$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in', '$nin'})
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def normalize_filter(
|
||||
raw: dict[str, Any] | None,
|
||||
) -> list[tuple[str, str, Any]]:
|
||||
"""Parse a canonical filter dict into ``[(field, op, value)]`` triples.
|
||||
|
||||
Returns an empty list when *raw* is ``None`` or empty.
|
||||
|
||||
Raises ``ValueError`` on unsupported operators or malformed entries.
|
||||
"""
|
||||
if not raw:
|
||||
return []
|
||||
|
||||
triples: list[tuple[str, str, Any]] = []
|
||||
for field, condition in raw.items():
|
||||
if isinstance(condition, dict):
|
||||
for op, value in condition.items():
|
||||
if op not in SUPPORTED_OPS:
|
||||
raise ValueError(f'Unsupported filter operator: {op}')
|
||||
triples.append((field, op, value))
|
||||
else:
|
||||
# Bare value -> implicit $eq
|
||||
triples.append((field, '$eq', condition))
|
||||
return triples
|
||||
|
||||
|
||||
def strip_unsupported_fields(
|
||||
triples: list[tuple[str, str, Any]],
|
||||
supported_fields: set[str],
|
||||
) -> list[tuple[str, str, Any]]:
|
||||
"""Return only triples whose field is in *supported_fields*.
|
||||
|
||||
Dropped fields are logged at WARNING level so the caller knows they were
|
||||
silently ignored (useful for Milvus / pgvector which only store a fixed
|
||||
schema).
|
||||
"""
|
||||
kept: list[tuple[str, str, Any]] = []
|
||||
for field, op, value in triples:
|
||||
if field in supported_fields:
|
||||
kept.append((field, op, value))
|
||||
else:
|
||||
logger.warning(
|
||||
'Filter field %r is not supported by this backend and will be ignored (supported: %s)',
|
||||
field,
|
||||
', '.join(sorted(supported_fields)),
|
||||
)
|
||||
return kept
|
||||
@@ -1,7 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from ..core import app
|
||||
from .vdb import VectorDatabase
|
||||
from .vdb import VectorDatabase, SearchType
|
||||
from .vdbs.chroma import ChromaVectorDatabase
|
||||
from .vdbs.qdrant import QdrantVectorDatabase
|
||||
from .vdbs.seekdb import SeekDBVectorDatabase
|
||||
@@ -65,3 +65,95 @@ class VectorDBManager:
|
||||
else:
|
||||
self.vector_db = ChromaVectorDatabase(self.ap)
|
||||
self.ap.logger.warning('No vector database backend configured, defaulting to Chroma.')
|
||||
|
||||
def get_supported_search_types(self) -> list[str]:
|
||||
"""Return the search types supported by the current VDB backend."""
|
||||
if self.vector_db is None:
|
||||
return [SearchType.VECTOR.value]
|
||||
return [st.value for st in self.vector_db.supported_search_types()]
|
||||
|
||||
async def upsert(
|
||||
self,
|
||||
collection_name: str,
|
||||
vectors: list[list[float]],
|
||||
ids: list[str],
|
||||
metadata: list[dict] | None = None,
|
||||
documents: list[str] | None = None,
|
||||
):
|
||||
"""Proxy: Upsert vectors"""
|
||||
await self.vector_db.add_embeddings(
|
||||
collection=collection_name,
|
||||
ids=ids,
|
||||
embeddings_list=vectors,
|
||||
metadatas=metadata or [{} for _ in vectors],
|
||||
documents=documents,
|
||||
)
|
||||
|
||||
async def search(
|
||||
self,
|
||||
collection_name: str,
|
||||
query_vector: list[float],
|
||||
limit: int,
|
||||
filter: dict | None = None,
|
||||
search_type: str = 'vector',
|
||||
query_text: str = '',
|
||||
) -> list[dict]:
|
||||
"""Proxy: Search vectors.
|
||||
|
||||
Returns a list of dicts with keys: 'id', 'score', 'metadata'.
|
||||
The underlying VectorDatabase.search returns Chroma-style format:
|
||||
{ 'ids': [['id1']], 'distances': [[0.1]], 'metadatas': [[{}]] }
|
||||
"""
|
||||
results = await self.vector_db.search(
|
||||
collection=collection_name,
|
||||
query_embedding=query_vector,
|
||||
k=limit,
|
||||
search_type=search_type,
|
||||
query_text=query_text,
|
||||
filter=filter,
|
||||
)
|
||||
|
||||
if not results or 'ids' not in results or not results['ids']:
|
||||
return []
|
||||
|
||||
# Flatten nested lists (Chroma returns batch-style: list of lists)
|
||||
raw_ids = results['ids']
|
||||
raw_dists = results.get('distances', [])
|
||||
raw_metas = results.get('metadatas', [])
|
||||
|
||||
r_ids = raw_ids[0] if raw_ids and isinstance(raw_ids[0], list) else raw_ids
|
||||
r_dists = raw_dists[0] if raw_dists and isinstance(raw_dists[0], list) else raw_dists
|
||||
r_metas = raw_metas[0] if raw_metas and isinstance(raw_metas[0], list) else raw_metas
|
||||
|
||||
parsed_results = []
|
||||
for i, id_val in enumerate(r_ids):
|
||||
parsed_results.append(
|
||||
{
|
||||
'id': id_val,
|
||||
'score': r_dists[i] if r_dists and i < len(r_dists) else 0.0,
|
||||
'metadata': r_metas[i] if r_metas and i < len(r_metas) else {},
|
||||
}
|
||||
)
|
||||
|
||||
return parsed_results
|
||||
|
||||
async def delete_by_file_id(self, collection_name: str, file_ids: list[str]):
|
||||
"""Proxy: Delete vectors by file_id (metadata-level identifier).
|
||||
|
||||
This delegates to VectorDatabase.delete_by_file_id which removes
|
||||
all vectors associated with the given file IDs.
|
||||
"""
|
||||
for file_id in file_ids:
|
||||
await self.vector_db.delete_by_file_id(collection_name, file_id)
|
||||
|
||||
async def delete_collection(self, collection_name: str):
|
||||
"""Proxy: Delete an entire collection."""
|
||||
await self.vector_db.delete_collection(collection_name)
|
||||
|
||||
async def delete_by_filter(self, collection_name: str, filter: dict) -> int:
|
||||
"""Proxy: Delete vectors by metadata filter.
|
||||
|
||||
Returns:
|
||||
Number of deleted vectors (best-effort; some backends return 0).
|
||||
"""
|
||||
return await self.vector_db.delete_by_filter(collection_name, filter)
|
||||
|
||||
@@ -1,10 +1,28 @@
|
||||
from __future__ import annotations
|
||||
import abc
|
||||
import enum
|
||||
from typing import Any, Dict
|
||||
import numpy as np
|
||||
|
||||
|
||||
class SearchType(str, enum.Enum):
|
||||
"""Supported search types for vector databases."""
|
||||
|
||||
VECTOR = 'vector'
|
||||
FULL_TEXT = 'full_text'
|
||||
HYBRID = 'hybrid'
|
||||
|
||||
|
||||
class VectorDatabase(abc.ABC):
|
||||
@classmethod
|
||||
def supported_search_types(cls) -> list[SearchType]:
|
||||
"""Return the search types supported by this VDB backend.
|
||||
|
||||
Default: vector search only. Override in subclasses that support
|
||||
full-text or hybrid search.
|
||||
"""
|
||||
return [SearchType.VECTOR]
|
||||
|
||||
@abc.abstractmethod
|
||||
async def add_embeddings(
|
||||
self,
|
||||
@@ -12,14 +30,47 @@ class VectorDatabase(abc.ABC):
|
||||
ids: list[str],
|
||||
embeddings_list: list[list[float]],
|
||||
metadatas: list[dict[str, Any]],
|
||||
documents: list[str],
|
||||
documents: list[str] | None = None,
|
||||
) -> None:
|
||||
"""Add vector data to the specified collection."""
|
||||
"""Add vector data to the specified collection.
|
||||
|
||||
Args:
|
||||
collection: Collection name.
|
||||
ids: Unique IDs for each vector.
|
||||
embeddings_list: List of embedding vectors.
|
||||
metadatas: List of metadata dicts.
|
||||
documents: Optional raw text documents. Required for full-text
|
||||
and hybrid search in backends that support them.
|
||||
"""
|
||||
pass
|
||||
|
||||
@abc.abstractmethod
|
||||
async def search(self, collection: str, query_embedding: np.ndarray, k: int = 5) -> Dict[str, Any]:
|
||||
"""Search for the most similar vectors in the specified collection."""
|
||||
async def search(
|
||||
self,
|
||||
collection: str,
|
||||
query_embedding: np.ndarray,
|
||||
k: int = 5,
|
||||
search_type: str = 'vector',
|
||||
query_text: str = '',
|
||||
filter: dict[str, Any] | None = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Search for the most similar vectors in the specified collection.
|
||||
|
||||
Args:
|
||||
collection: Collection name.
|
||||
query_embedding: Query vector for similarity search.
|
||||
k: Number of results to return.
|
||||
search_type: One of 'vector', 'full_text', 'hybrid'.
|
||||
query_text: Raw query text, used for full_text and hybrid search.
|
||||
filter: Optional metadata filters using Chroma-style ``where``
|
||||
syntax. Multiple top-level keys are AND-ed. Supported
|
||||
operators: ``$eq``, ``$ne``, ``$gt``, ``$gte``, ``$lt``,
|
||||
``$lte``, ``$in``, ``$nin``. Example::
|
||||
|
||||
{"file_id": "abc"}
|
||||
{"created_at": {"$gte": 1700000000}}
|
||||
{"file_type": {"$in": ["pdf", "docx"]}}
|
||||
"""
|
||||
pass
|
||||
|
||||
@abc.abstractmethod
|
||||
@@ -27,6 +78,20 @@ class VectorDatabase(abc.ABC):
|
||||
"""Delete vectors from the specified collection by file_id."""
|
||||
pass
|
||||
|
||||
@abc.abstractmethod
|
||||
async def delete_by_filter(self, collection: str, filter: dict[str, Any]) -> int:
|
||||
"""Delete vectors matching the given metadata filter.
|
||||
|
||||
Args:
|
||||
collection: Collection name.
|
||||
filter: Metadata filter dict in canonical format (see ``search``).
|
||||
|
||||
Returns:
|
||||
Number of deleted vectors (best-effort; backends that cannot
|
||||
report an exact count may return 0).
|
||||
"""
|
||||
pass
|
||||
|
||||
@abc.abstractmethod
|
||||
async def get_or_create_collection(self, collection: str):
|
||||
"""Get or create collection."""
|
||||
|
||||
@@ -28,19 +28,33 @@ class ChromaVectorDatabase(VectorDatabase):
|
||||
ids: list[str],
|
||||
embeddings_list: list[list[float]],
|
||||
metadatas: list[dict[str, Any]],
|
||||
documents: list[str] | None = None,
|
||||
) -> None:
|
||||
col = await self.get_or_create_collection(collection)
|
||||
await asyncio.to_thread(col.add, embeddings=embeddings_list, ids=ids, metadatas=metadatas)
|
||||
kwargs: dict[str, Any] = dict(embeddings=embeddings_list, ids=ids, metadatas=metadatas)
|
||||
if documents is not None:
|
||||
kwargs['documents'] = documents
|
||||
await asyncio.to_thread(col.add, **kwargs)
|
||||
self.ap.logger.info(f"Added {len(ids)} embeddings to Chroma collection '{collection}'.")
|
||||
|
||||
async def search(self, collection: str, query_embedding: list[float], k: int = 5) -> dict[str, Any]:
|
||||
async def search(
|
||||
self,
|
||||
collection: str,
|
||||
query_embedding: list[float],
|
||||
k: int = 5,
|
||||
search_type: str = 'vector',
|
||||
query_text: str = '',
|
||||
filter: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
col = await self.get_or_create_collection(collection)
|
||||
results = await asyncio.to_thread(
|
||||
col.query,
|
||||
query_kwargs: dict[str, Any] = dict(
|
||||
query_embeddings=query_embedding,
|
||||
n_results=k,
|
||||
include=['metadatas', 'distances', 'documents'],
|
||||
)
|
||||
if filter:
|
||||
query_kwargs['where'] = filter
|
||||
results = await asyncio.to_thread(col.query, **query_kwargs)
|
||||
self.ap.logger.info(f"Chroma search in '{collection}' returned {len(results.get('ids', [[]])[0])} results.")
|
||||
return results
|
||||
|
||||
@@ -49,6 +63,12 @@ class ChromaVectorDatabase(VectorDatabase):
|
||||
await asyncio.to_thread(col.delete, where={'file_id': file_id})
|
||||
self.ap.logger.info(f"Deleted embeddings from Chroma collection '{collection}' with file_id: {file_id}")
|
||||
|
||||
async def delete_by_filter(self, collection: str, filter: dict[str, Any]) -> int:
|
||||
col = await self.get_or_create_collection(collection)
|
||||
await asyncio.to_thread(col.delete, where=filter)
|
||||
self.ap.logger.info(f"Deleted embeddings from Chroma collection '{collection}' by filter")
|
||||
return 0 # Chroma delete does not return a count
|
||||
|
||||
async def delete_collection(self, collection: str):
|
||||
if collection in self._collections:
|
||||
del self._collections[collection]
|
||||
|
||||
@@ -4,8 +4,51 @@ from typing import Any, Dict
|
||||
from pymilvus import MilvusClient, DataType, CollectionSchema, FieldSchema
|
||||
from pymilvus.milvus_client.index import IndexParams
|
||||
from langbot.pkg.vector.vdb import VectorDatabase
|
||||
from langbot.pkg.vector.filter_utils import normalize_filter, strip_unsupported_fields
|
||||
from langbot.pkg.core import app
|
||||
|
||||
# Milvus schema only stores these metadata fields; filter on other fields is
|
||||
# silently dropped with a warning.
|
||||
_MILVUS_SUPPORTED_FIELDS = {'text', 'file_id', 'chunk_uuid'}
|
||||
|
||||
|
||||
def _build_milvus_expr(filter_dict: dict[str, Any]) -> str:
|
||||
"""Translate canonical filter dict into a Milvus boolean expression string."""
|
||||
triples = normalize_filter(filter_dict)
|
||||
triples = strip_unsupported_fields(triples, _MILVUS_SUPPORTED_FIELDS)
|
||||
if not triples:
|
||||
return ''
|
||||
|
||||
parts: list[str] = []
|
||||
for field, op, value in triples:
|
||||
if op == '$eq':
|
||||
parts.append(f'{field} == {_milvus_literal(value)}')
|
||||
elif op == '$ne':
|
||||
parts.append(f'{field} != {_milvus_literal(value)}')
|
||||
elif op == '$gt':
|
||||
parts.append(f'{field} > {_milvus_literal(value)}')
|
||||
elif op == '$gte':
|
||||
parts.append(f'{field} >= {_milvus_literal(value)}')
|
||||
elif op == '$lt':
|
||||
parts.append(f'{field} < {_milvus_literal(value)}')
|
||||
elif op == '$lte':
|
||||
parts.append(f'{field} <= {_milvus_literal(value)}')
|
||||
elif op == '$in':
|
||||
items = ', '.join(_milvus_literal(v) for v in value)
|
||||
parts.append(f'{field} in [{items}]')
|
||||
elif op == '$nin':
|
||||
items = ', '.join(_milvus_literal(v) for v in value)
|
||||
parts.append(f'{field} not in [{items}]')
|
||||
return ' and '.join(parts)
|
||||
|
||||
|
||||
def _milvus_literal(value: Any) -> str:
|
||||
"""Format a Python value as a Milvus expression literal."""
|
||||
if isinstance(value, str):
|
||||
escaped = value.replace('\\', '\\\\').replace('"', '\\"')
|
||||
return f'"{escaped}"'
|
||||
return str(value)
|
||||
|
||||
|
||||
class MilvusVectorDatabase(VectorDatabase):
|
||||
"""Milvus vector database implementation"""
|
||||
@@ -155,6 +198,7 @@ class MilvusVectorDatabase(VectorDatabase):
|
||||
ids: list[str],
|
||||
embeddings_list: list[list[float]],
|
||||
metadatas: list[dict[str, Any]],
|
||||
documents: list[str] | None = None,
|
||||
) -> None:
|
||||
"""Add vector embeddings to Milvus collection
|
||||
|
||||
@@ -200,7 +244,15 @@ class MilvusVectorDatabase(VectorDatabase):
|
||||
|
||||
self.ap.logger.info(f"Added {len(ids)} embeddings to Milvus collection '{collection}'")
|
||||
|
||||
async def search(self, collection: str, query_embedding: list[float], k: int = 5) -> Dict[str, Any]:
|
||||
async def search(
|
||||
self,
|
||||
collection: str,
|
||||
query_embedding: list[float],
|
||||
k: int = 5,
|
||||
search_type: str = 'vector',
|
||||
query_text: str = '',
|
||||
filter: dict[str, Any] | None = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Search for similar vectors in Milvus collection
|
||||
|
||||
Args:
|
||||
@@ -217,14 +269,19 @@ class MilvusVectorDatabase(VectorDatabase):
|
||||
# Perform search
|
||||
search_params = {'metric_type': 'COSINE', 'params': {}}
|
||||
|
||||
results = await asyncio.to_thread(
|
||||
self.client.search,
|
||||
search_kwargs: dict[str, Any] = dict(
|
||||
collection_name=collection,
|
||||
data=[query_embedding],
|
||||
limit=k,
|
||||
search_params=search_params,
|
||||
output_fields=['text', 'file_id', 'chunk_uuid'],
|
||||
)
|
||||
if filter:
|
||||
expr = _build_milvus_expr(filter)
|
||||
if expr:
|
||||
search_kwargs['filter'] = expr
|
||||
|
||||
results = await asyncio.to_thread(self.client.search, **search_kwargs)
|
||||
|
||||
# Convert results to Chroma-compatible format
|
||||
# Milvus returns: [[ {id, distance, entity: {...}} ]]
|
||||
@@ -268,6 +325,21 @@ class MilvusVectorDatabase(VectorDatabase):
|
||||
await asyncio.to_thread(self.client.delete, collection_name=collection, filter=f'file_id == "{file_id}"')
|
||||
self.ap.logger.info(f"Deleted embeddings from Milvus collection '{collection}' with file_id: {file_id}")
|
||||
|
||||
async def delete_by_filter(self, collection: str, filter: dict[str, Any]) -> int:
|
||||
collection = self._normalize_collection_name(collection)
|
||||
await self.get_or_create_collection(collection)
|
||||
|
||||
expr = _build_milvus_expr(filter)
|
||||
if not expr:
|
||||
self.ap.logger.warning(
|
||||
f"Milvus delete_by_filter on '{collection}': filter produced empty expression, skipping"
|
||||
)
|
||||
return 0
|
||||
|
||||
await asyncio.to_thread(self.client.delete, collection_name=collection, filter=expr)
|
||||
self.ap.logger.info(f"Deleted embeddings from Milvus collection '{collection}' by filter")
|
||||
return 0 # Milvus delete does not return a count
|
||||
|
||||
async def delete_collection(self, collection: str):
|
||||
"""Delete a Milvus collection
|
||||
|
||||
|
||||
@@ -5,10 +5,21 @@ from sqlalchemy.orm import declarative_base
|
||||
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession, async_sessionmaker
|
||||
from pgvector.sqlalchemy import Vector
|
||||
from langbot.pkg.vector.vdb import VectorDatabase
|
||||
from langbot.pkg.vector.filter_utils import normalize_filter, strip_unsupported_fields
|
||||
from langbot.pkg.core import app
|
||||
|
||||
Base = declarative_base()
|
||||
|
||||
# pgvector schema only stores these metadata fields.
|
||||
_PG_SUPPORTED_FIELDS = {'text', 'file_id', 'chunk_uuid'}
|
||||
|
||||
# Map schema field names to SQLAlchemy columns (resolved lazily from PgVectorEntry).
|
||||
_PG_COLUMN_MAP = {
|
||||
'text': 'text',
|
||||
'file_id': 'file_id',
|
||||
'chunk_uuid': 'chunk_uuid',
|
||||
}
|
||||
|
||||
|
||||
class PgVectorEntry(Base):
|
||||
"""SQLAlchemy model for pgvector entries"""
|
||||
@@ -23,6 +34,33 @@ class PgVectorEntry(Base):
|
||||
chunk_uuid = Column(String)
|
||||
|
||||
|
||||
def _build_pg_conditions(filter_dict: dict[str, Any]) -> list:
|
||||
"""Translate canonical filter dict into a list of SQLAlchemy conditions."""
|
||||
triples = normalize_filter(filter_dict)
|
||||
triples = strip_unsupported_fields(triples, _PG_SUPPORTED_FIELDS)
|
||||
|
||||
conditions = []
|
||||
for field, op, value in triples:
|
||||
col = getattr(PgVectorEntry, _PG_COLUMN_MAP[field])
|
||||
if op == '$eq':
|
||||
conditions.append(col == value)
|
||||
elif op == '$ne':
|
||||
conditions.append(col != value)
|
||||
elif op == '$gt':
|
||||
conditions.append(col > value)
|
||||
elif op == '$gte':
|
||||
conditions.append(col >= value)
|
||||
elif op == '$lt':
|
||||
conditions.append(col < value)
|
||||
elif op == '$lte':
|
||||
conditions.append(col <= value)
|
||||
elif op == '$in':
|
||||
conditions.append(col.in_(value))
|
||||
elif op == '$nin':
|
||||
conditions.append(col.notin_(value))
|
||||
return conditions
|
||||
|
||||
|
||||
class PgVectorDatabase(VectorDatabase):
|
||||
"""PostgreSQL with pgvector extension database implementation"""
|
||||
|
||||
@@ -109,6 +147,7 @@ class PgVectorDatabase(VectorDatabase):
|
||||
ids: list[str],
|
||||
embeddings_list: list[list[float]],
|
||||
metadatas: list[dict[str, Any]],
|
||||
documents: list[str] | None = None,
|
||||
) -> None:
|
||||
"""Add vector embeddings to pgvector
|
||||
|
||||
@@ -142,7 +181,15 @@ class PgVectorDatabase(VectorDatabase):
|
||||
self.ap.logger.error(f'Error adding embeddings to pgvector: {e}')
|
||||
raise
|
||||
|
||||
async def search(self, collection: str, query_embedding: list[float], k: int = 5) -> Dict[str, Any]:
|
||||
async def search(
|
||||
self,
|
||||
collection: str,
|
||||
query_embedding: list[float],
|
||||
k: int = 5,
|
||||
search_type: str = 'vector',
|
||||
query_text: str = '',
|
||||
filter: dict[str, Any] | None = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Search for similar vectors using cosine distance
|
||||
|
||||
Args:
|
||||
@@ -174,6 +221,10 @@ class PgVectorDatabase(VectorDatabase):
|
||||
.limit(k)
|
||||
)
|
||||
|
||||
if filter:
|
||||
for cond in _build_pg_conditions(filter):
|
||||
stmt = stmt.filter(cond)
|
||||
|
||||
result = await session.execute(stmt)
|
||||
rows = result.fetchall()
|
||||
|
||||
@@ -225,6 +276,39 @@ class PgVectorDatabase(VectorDatabase):
|
||||
self.ap.logger.error(f'Error deleting from pgvector: {e}')
|
||||
raise
|
||||
|
||||
async def delete_by_filter(self, collection: str, filter: dict[str, Any]) -> int:
|
||||
"""Delete vectors matching a metadata filter.
|
||||
|
||||
Args:
|
||||
collection: Collection name
|
||||
filter: Canonical metadata filter dict
|
||||
"""
|
||||
conditions = _build_pg_conditions(filter)
|
||||
if not conditions:
|
||||
self.ap.logger.warning(
|
||||
f"pgvector delete_by_filter on '{collection}': filter produced no conditions, skipping"
|
||||
)
|
||||
return 0
|
||||
|
||||
await self.get_or_create_collection(collection)
|
||||
|
||||
async with self.AsyncSessionLocal() as session:
|
||||
try:
|
||||
from sqlalchemy import delete
|
||||
|
||||
stmt = delete(PgVectorEntry).where(PgVectorEntry.collection == collection)
|
||||
for cond in conditions:
|
||||
stmt = stmt.where(cond)
|
||||
result = await session.execute(stmt)
|
||||
await session.commit()
|
||||
deleted = result.rowcount
|
||||
self.ap.logger.info(f"Deleted {deleted} embeddings from pgvector collection '{collection}' by filter")
|
||||
return deleted
|
||||
except Exception as e:
|
||||
await session.rollback()
|
||||
self.ap.logger.error(f'Error deleting from pgvector by filter: {e}')
|
||||
raise
|
||||
|
||||
async def delete_collection(self, collection: str):
|
||||
"""Delete all vectors in a collection
|
||||
|
||||
|
||||
@@ -5,6 +5,37 @@ from typing import Any, Dict, List
|
||||
from qdrant_client import AsyncQdrantClient, models
|
||||
from langbot.pkg.core import app
|
||||
from langbot.pkg.vector.vdb import VectorDatabase
|
||||
from langbot.pkg.vector.filter_utils import normalize_filter
|
||||
|
||||
|
||||
def _build_qdrant_filter(filter_dict: dict[str, Any]) -> models.Filter:
|
||||
"""Translate canonical filter dict into a Qdrant ``models.Filter``."""
|
||||
triples = normalize_filter(filter_dict)
|
||||
must: list[models.Condition] = []
|
||||
must_not: list[models.Condition] = []
|
||||
|
||||
for field, op, value in triples:
|
||||
if op == '$eq':
|
||||
must.append(models.FieldCondition(key=field, match=models.MatchValue(value=value)))
|
||||
elif op == '$ne':
|
||||
must_not.append(models.FieldCondition(key=field, match=models.MatchValue(value=value)))
|
||||
elif op == '$in':
|
||||
must.append(models.FieldCondition(key=field, match=models.MatchAny(any=value)))
|
||||
elif op == '$nin':
|
||||
must_not.append(models.FieldCondition(key=field, match=models.MatchAny(any=value)))
|
||||
elif op in ('$gt', '$gte', '$lt', '$lte'):
|
||||
range_kwargs: dict[str, Any] = {}
|
||||
if op == '$gt':
|
||||
range_kwargs['gt'] = value
|
||||
elif op == '$gte':
|
||||
range_kwargs['gte'] = value
|
||||
elif op == '$lt':
|
||||
range_kwargs['lt'] = value
|
||||
elif op == '$lte':
|
||||
range_kwargs['lte'] = value
|
||||
must.append(models.FieldCondition(key=field, range=models.Range(**range_kwargs)))
|
||||
|
||||
return models.Filter(must=must or None, must_not=must_not or None)
|
||||
|
||||
|
||||
class QdrantVectorDatabase(VectorDatabase):
|
||||
@@ -48,6 +79,7 @@ class QdrantVectorDatabase(VectorDatabase):
|
||||
ids: List[str],
|
||||
embeddings_list: List[List[float]],
|
||||
metadatas: List[Dict[str, Any]],
|
||||
documents: List[str] | None = None,
|
||||
) -> None:
|
||||
if not embeddings_list:
|
||||
return
|
||||
@@ -60,19 +92,29 @@ class QdrantVectorDatabase(VectorDatabase):
|
||||
await self.client.upsert(collection_name=collection, points=points)
|
||||
self.ap.logger.info(f"Added {len(ids)} embeddings to Qdrant collection '{collection}'.")
|
||||
|
||||
async def search(self, collection: str, query_embedding: list[float], k: int = 5) -> dict[str, Any]:
|
||||
async def search(
|
||||
self,
|
||||
collection: str,
|
||||
query_embedding: list[float],
|
||||
k: int = 5,
|
||||
search_type: str = 'vector',
|
||||
query_text: str = '',
|
||||
filter: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
exists = await self.client.collection_exists(collection)
|
||||
if not exists:
|
||||
return {'ids': [[]], 'metadatas': [[]], 'distances': [[]]}
|
||||
|
||||
hits = (
|
||||
await self.client.query_points(
|
||||
collection_name=collection,
|
||||
query=query_embedding,
|
||||
limit=k,
|
||||
with_payload=True,
|
||||
)
|
||||
).points
|
||||
query_kwargs: dict[str, Any] = dict(
|
||||
collection_name=collection,
|
||||
query=query_embedding,
|
||||
limit=k,
|
||||
with_payload=True,
|
||||
)
|
||||
if filter:
|
||||
query_kwargs['query_filter'] = _build_qdrant_filter(filter)
|
||||
|
||||
hits = (await self.client.query_points(**query_kwargs)).points
|
||||
ids = [str(hit.id) for hit in hits]
|
||||
metadatas = [hit.payload or {} for hit in hits]
|
||||
# Qdrant's score is similarity; convert to a pseudo-distance for consistency
|
||||
@@ -95,6 +137,19 @@ class QdrantVectorDatabase(VectorDatabase):
|
||||
)
|
||||
self.ap.logger.info(f"Deleted embeddings from Qdrant collection '{collection}' with file_id: {file_id}")
|
||||
|
||||
async def delete_by_filter(self, collection: str, filter: dict[str, Any]) -> int:
|
||||
exists = await self.client.collection_exists(collection)
|
||||
if not exists:
|
||||
return 0
|
||||
|
||||
qdrant_filter = _build_qdrant_filter(filter)
|
||||
await self.client.delete(
|
||||
collection_name=collection,
|
||||
points_selector=qdrant_filter,
|
||||
)
|
||||
self.ap.logger.info(f"Deleted embeddings from Qdrant collection '{collection}' by filter")
|
||||
return 0 # Qdrant delete does not return a count
|
||||
|
||||
async def delete_collection(self, collection: str):
|
||||
try:
|
||||
await self.client.delete_collection(collection)
|
||||
|
||||
@@ -5,7 +5,7 @@ from typing import Any, Dict, List
|
||||
|
||||
|
||||
from langbot.pkg.core import app
|
||||
from langbot.pkg.vector.vdb import VectorDatabase
|
||||
from langbot.pkg.vector.vdb import VectorDatabase, SearchType
|
||||
|
||||
try:
|
||||
import pyseekdb
|
||||
@@ -25,9 +25,13 @@ class SeekDBVectorDatabase(VectorDatabase):
|
||||
SeekDB is an AI-native search database by OceanBase that unifies
|
||||
relational, vector, text, JSON and GIS in a single engine.
|
||||
|
||||
Supports both embedded mode and remote server mode.
|
||||
Supports embedded mode, remote server mode, and full-text/hybrid search.
|
||||
"""
|
||||
|
||||
@classmethod
|
||||
def supported_search_types(cls) -> list[SearchType]:
|
||||
return [SearchType.VECTOR, SearchType.FULL_TEXT, SearchType.HYBRID]
|
||||
|
||||
def __init__(self, ap: app.Application):
|
||||
if not SEEKDB_AVAILABLE:
|
||||
raise ImportError('pyseekdb is not installed. Install it with: pip install pyseekdb')
|
||||
@@ -89,6 +93,7 @@ class SeekDBVectorDatabase(VectorDatabase):
|
||||
{
|
||||
'\x00': '',
|
||||
'\\': '\\\\',
|
||||
"'": "''", # Standard SQL escaping (OceanBase NO_BACKSLASH_ESCAPES)
|
||||
'"': '\\"',
|
||||
'\n': '\\n',
|
||||
'\r': '\\r',
|
||||
@@ -111,8 +116,10 @@ class SeekDBVectorDatabase(VectorDatabase):
|
||||
|
||||
# Collection doesn't exist, create it
|
||||
if vector_size is None:
|
||||
# Default dimension if not specified
|
||||
vector_size = 384
|
||||
raise ValueError(
|
||||
f"Cannot create SeekDB collection '{collection}' without knowing the vector dimension. "
|
||||
'Ensure add_embeddings is called before any standalone get_or_create_collection.'
|
||||
)
|
||||
|
||||
# Create HNSW configuration
|
||||
config = HNSWConfiguration(dimension=vector_size, distance='cosine')
|
||||
@@ -147,7 +154,12 @@ class SeekDBVectorDatabase(VectorDatabase):
|
||||
return await self._get_or_create_collection_internal(collection)
|
||||
|
||||
async def add_embeddings(
|
||||
self, collection: str, ids: List[str], embeddings_list: List[List[float]], metadatas: List[Dict[str, Any]]
|
||||
self,
|
||||
collection: str,
|
||||
ids: List[str],
|
||||
embeddings_list: List[List[float]],
|
||||
metadatas: List[Dict[str, Any]],
|
||||
documents: List[str] | None = None,
|
||||
) -> None:
|
||||
"""Add vector embeddings to the specified collection.
|
||||
|
||||
@@ -156,6 +168,7 @@ class SeekDBVectorDatabase(VectorDatabase):
|
||||
ids: List of document IDs
|
||||
embeddings_list: List of embedding vectors
|
||||
metadatas: List of metadata dictionaries
|
||||
documents: Optional raw text documents for full-text search support
|
||||
"""
|
||||
if not embeddings_list:
|
||||
return
|
||||
@@ -166,17 +179,33 @@ class SeekDBVectorDatabase(VectorDatabase):
|
||||
|
||||
cleaned_metadatas = [self._clean_metadata(meta) for meta in metadatas]
|
||||
|
||||
await asyncio.to_thread(coll.add, ids=ids, embeddings=embeddings_list, metadatas=cleaned_metadatas)
|
||||
kwargs: Dict[str, Any] = dict(ids=ids, embeddings=embeddings_list, metadatas=cleaned_metadatas)
|
||||
if documents is not None:
|
||||
kwargs['documents'] = [doc.translate(self._escape_table) for doc in documents]
|
||||
await asyncio.to_thread(coll.add, **kwargs)
|
||||
|
||||
self.ap.logger.info(f"Added {len(ids)} embeddings to SeekDB collection '{collection}'")
|
||||
|
||||
async def search(self, collection: str, query_embedding: List[float], k: int = 5) -> Dict[str, Any]:
|
||||
async def search(
|
||||
self,
|
||||
collection: str,
|
||||
query_embedding: List[float],
|
||||
k: int = 5,
|
||||
search_type: str = 'vector',
|
||||
query_text: str = '',
|
||||
filter: Dict[str, Any] | None = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Search for the most similar vectors in the specified collection.
|
||||
|
||||
SeekDB supports vector, full-text, and hybrid search modes.
|
||||
|
||||
Args:
|
||||
collection: Collection name
|
||||
query_embedding: Query vector
|
||||
query_embedding: Query vector (used for vector and hybrid modes)
|
||||
k: Number of results to return
|
||||
search_type: One of 'vector', 'full_text', 'hybrid'
|
||||
query_text: Raw query text (used for full_text and hybrid modes)
|
||||
filter: Optional metadata filters (Chroma-style ``where`` syntax).
|
||||
|
||||
Returns:
|
||||
Dictionary with 'ids', 'metadatas', 'distances' keys
|
||||
@@ -193,11 +222,73 @@ class SeekDBVectorDatabase(VectorDatabase):
|
||||
else:
|
||||
coll = self._collections[collection]
|
||||
|
||||
# Perform query
|
||||
# SeekDB's query() returns: {'ids': [[...]], 'metadatas': [[...]], 'distances': [[...]]}
|
||||
results = await asyncio.to_thread(coll.query, query_embeddings=query_embedding, n_results=k)
|
||||
# Route by search type.
|
||||
# pyseekdb's query() always requires embeddings, so full-text and
|
||||
# hybrid modes use hybrid_search() which supports text-only queries
|
||||
# and returns the same nested-list format with distances.
|
||||
if search_type == SearchType.FULL_TEXT:
|
||||
if not query_text:
|
||||
return {'ids': [[]], 'metadatas': [[]], 'distances': [[]]}
|
||||
|
||||
self.ap.logger.info(f"SeekDB search in '{collection}' returned {len(results.get('ids', [[]])[0])} results")
|
||||
query_cfg: Dict[str, Any] = {
|
||||
'where_document': {'$contains': query_text},
|
||||
'n_results': k,
|
||||
}
|
||||
if filter:
|
||||
query_cfg['where'] = filter
|
||||
|
||||
# TODO: pyseekdb hybrid_search with query-only (no knn) returns None
|
||||
# for IDs due to column name mismatch (*/_id vs _id).
|
||||
# See: https://github.com/oceanbase/pyseekdb/issues/171
|
||||
results = await asyncio.to_thread(
|
||||
coll.hybrid_search,
|
||||
query=query_cfg,
|
||||
knn=None,
|
||||
n_results=k,
|
||||
include=['documents', 'metadatas'],
|
||||
)
|
||||
|
||||
elif search_type == SearchType.HYBRID:
|
||||
if not query_text:
|
||||
# Fall back to pure vector search when no text is provided
|
||||
query_kwargs: Dict[str, Any] = {
|
||||
'n_results': k,
|
||||
'query_embeddings': query_embedding,
|
||||
}
|
||||
if filter:
|
||||
query_kwargs['where'] = filter
|
||||
results = await asyncio.to_thread(coll.query, **query_kwargs)
|
||||
else:
|
||||
query_cfg = {
|
||||
'where_document': {'$contains': query_text},
|
||||
'n_results': k,
|
||||
}
|
||||
knn_cfg: Dict[str, Any] = {
|
||||
'query_embeddings': query_embedding,
|
||||
'n_results': k,
|
||||
}
|
||||
if filter:
|
||||
query_cfg['where'] = filter
|
||||
knn_cfg['where'] = filter
|
||||
|
||||
results = await asyncio.to_thread(
|
||||
coll.hybrid_search,
|
||||
query=query_cfg,
|
||||
knn=knn_cfg,
|
||||
rank={'rrf': {}},
|
||||
n_results=k,
|
||||
include=['documents', 'metadatas'],
|
||||
)
|
||||
else:
|
||||
# Default: vector search via query()
|
||||
query_kwargs = {'n_results': k, 'query_embeddings': query_embedding}
|
||||
if filter:
|
||||
query_kwargs['where'] = filter
|
||||
results = await asyncio.to_thread(coll.query, **query_kwargs)
|
||||
|
||||
self.ap.logger.info(
|
||||
f"SeekDB {search_type} search in '{collection}' returned {len(results.get('ids', [[]])[0])} results"
|
||||
)
|
||||
|
||||
return results
|
||||
|
||||
@@ -227,6 +318,28 @@ class SeekDBVectorDatabase(VectorDatabase):
|
||||
|
||||
self.ap.logger.info(f"Deleted embeddings from SeekDB collection '{collection}' with file_id: {file_id}")
|
||||
|
||||
async def delete_by_filter(self, collection: str, filter: Dict[str, Any]) -> int:
|
||||
"""Delete vectors from the collection by metadata filter.
|
||||
|
||||
Args:
|
||||
collection: Collection name
|
||||
filter: Chroma-style ``where`` filter dict
|
||||
"""
|
||||
exists = await asyncio.to_thread(self.client.has_collection, collection)
|
||||
if not exists:
|
||||
self.ap.logger.warning(f"SeekDB collection '{collection}' not found for deletion")
|
||||
return 0
|
||||
|
||||
if collection not in self._collections:
|
||||
coll = await asyncio.to_thread(self.client.get_collection, collection, embedding_function=None)
|
||||
self._collections[collection] = coll
|
||||
else:
|
||||
coll = self._collections[collection]
|
||||
|
||||
await asyncio.to_thread(coll.delete, where=filter)
|
||||
self.ap.logger.info(f"Deleted embeddings from SeekDB collection '{collection}' by filter")
|
||||
return 0 # SeekDB delete does not return a count
|
||||
|
||||
async def delete_collection(self, collection: str):
|
||||
"""Delete the entire collection.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user