mirror of
https://github.com/langbot-app/LangBot.git
synced 2026-07-16 09:26:07 +00:00
Compare commits
8 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Junyan Qin
|
ffd4143672 | ||
|
Junyan Qin
|
516e85f9e3 | ||
|
RockChinQ
|
940234a0d8 | ||
|
Daria Korenieva
|
0c405901d2 | ||
|
Hyu
|
00e2103873 | ||
|
Hyu
|
209706b0b9 | ||
|
Hyu
|
205404e3da | ||
|
Hyu
|
3e93ccfb45 |
@@ -51,7 +51,7 @@ LangBot is an **open-source, production-grade platform** for building AI-powered
|
|||||||
|
|
||||||
[→ Learn more about all features](https://link.langbot.app/en/docs/features)
|
[→ Learn more about all features](https://link.langbot.app/en/docs/features)
|
||||||
|
|
||||||
📍 Practical guides: [deploy a multi-platform AI bot in 5 minutes](https://blog.langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [connect DeepSeek to WeChat, Discord, and Telegram](https://blog.langbot.app/en/blog/connect-deepseek-to-wechat/), [run a Dify Agent in Discord, Telegram, and Slack](https://blog.langbot.app/en/blog/dify-agent-discord-telegram-slack/), and [build an n8n-powered chatbot](https://blog.langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
|
📍 Practical guides: [deploy a multi-platform AI bot in 5 minutes](https://langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [connect DeepSeek to WeChat, Discord, and Telegram](https://langbot.app/en/blog/connect-deepseek-to-wechat/), [run a Dify Agent in Discord, Telegram, and Slack](https://langbot.app/en/blog/dify-agent-discord-telegram-slack/), and [build an n8n-powered chatbot](https://langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -51,7 +51,7 @@ LangBot 是一个**开源的生产级平台**，用于构建 AI 驱动的即时
|
|||||||
|
|
||||||
[→ 了解更多功能特性](https://link.langbot.app/zh/docs/features)
|
[→ 了解更多功能特性](https://link.langbot.app/zh/docs/features)
|
||||||
|
|
||||||
📍 实践指南：[5 分钟部署多平台 AI 机器人](https://blog.langbot.app/zh/blog/deploy-ai-bot-in-5-minutes/)、[将 DeepSeek 接入微信、企业微信与 Discord](https://blog.langbot.app/zh/blog/connect-deepseek-to-wechat/)、[让 Dify Agent 跑在 Discord、Telegram 和 Slack 上](https://blog.langbot.app/zh/blog/dify-agent-discord-telegram-slack/)，以及[用 n8n 构建多平台 AI 聊天机器人](https://blog.langbot.app/zh/blog/n8n-multi-platform-ai-chatbot/)。
|
📍 实践指南：[5 分钟部署多平台 AI 机器人](https://langbot.app/zh/blog/deploy-ai-bot-in-5-minutes/)、[将 DeepSeek 接入微信、企业微信与 Discord](https://langbot.app/zh/blog/connect-deepseek-to-wechat/)、[让 Dify Agent 跑在 Discord、Telegram 和 Slack 上](https://langbot.app/zh/blog/dify-agent-discord-telegram-slack/)，以及[用 n8n 构建多平台 AI 聊天机器人](https://langbot.app/zh/blog/n8n-multi-platform-ai-chatbot/)。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -50,7 +50,7 @@ LangBot es una **plataforma de código abierto y grado de producción** para con
|
|||||||
|
|
||||||
[→ Conocer más sobre todas las funcionalidades](https://link.langbot.app/en/docs/features)
|
[→ Conocer más sobre todas las funcionalidades](https://link.langbot.app/en/docs/features)
|
||||||
|
|
||||||
📍 Guías prácticas: [desplegar un bot de IA multiplataforma en 5 minutos](https://blog.langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [conectar DeepSeek a WeChat, Discord y Telegram](https://blog.langbot.app/en/blog/connect-deepseek-to-wechat/), [ejecutar un Dify Agent en Discord, Telegram y Slack](https://blog.langbot.app/en/blog/dify-agent-discord-telegram-slack/) y [crear un chatbot con n8n](https://blog.langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
|
📍 Guías prácticas: [desplegar un bot de IA multiplataforma en 5 minutos](https://langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [conectar DeepSeek a WeChat, Discord y Telegram](https://langbot.app/en/blog/connect-deepseek-to-wechat/), [ejecutar un Dify Agent en Discord, Telegram y Slack](https://langbot.app/en/blog/dify-agent-discord-telegram-slack/) y [crear un chatbot con n8n](https://langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -50,7 +50,7 @@ LangBot est une **plateforme open-source de niveau production** pour créer des
|
|||||||
|
|
||||||
[→ En savoir plus sur toutes les fonctionnalités](https://link.langbot.app/en/docs/features)
|
[→ En savoir plus sur toutes les fonctionnalités](https://link.langbot.app/en/docs/features)
|
||||||
|
|
||||||
📍 Guides pratiques : [déployer un bot IA multiplateforme en 5 minutes](https://blog.langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [connecter DeepSeek à WeChat, Discord et Telegram](https://blog.langbot.app/en/blog/connect-deepseek-to-wechat/), [exécuter un Dify Agent dans Discord, Telegram et Slack](https://blog.langbot.app/en/blog/dify-agent-discord-telegram-slack/) et [créer un chatbot avec n8n](https://blog.langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
|
📍 Guides pratiques : [déployer un bot IA multiplateforme en 5 minutes](https://langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [connecter DeepSeek à WeChat, Discord et Telegram](https://langbot.app/en/blog/connect-deepseek-to-wechat/), [exécuter un Dify Agent dans Discord, Telegram et Slack](https://langbot.app/en/blog/dify-agent-discord-telegram-slack/) et [créer un chatbot avec n8n](https://langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -50,7 +50,7 @@ LangBot は、AI搭載のインスタントメッセージングボットを構
|
|||||||
|
|
||||||
[→ すべての機能について詳しく見る](https://link.langbot.app/ja/docs/features)
|
[→ すべての機能について詳しく見る](https://link.langbot.app/ja/docs/features)
|
||||||
|
|
||||||
📍 実践ガイド: [5分でマルチプラットフォームAIボットをデプロイ](https://blog.langbot.app/en/blog/deploy-ai-bot-in-5-minutes/)、[DeepSeekをWeChat・Discord・Telegramに接続](https://blog.langbot.app/en/blog/connect-deepseek-to-wechat/)、[Dify AgentをDiscord・Telegram・Slackで動かす](https://blog.langbot.app/en/blog/dify-agent-discord-telegram-slack/)、[n8n連携チャットボットを構築](https://blog.langbot.app/en/blog/n8n-multi-platform-ai-chatbot/)。
|
📍 実践ガイド: [5分でマルチプラットフォームAIボットをデプロイ](https://langbot.app/en/blog/deploy-ai-bot-in-5-minutes/)、[DeepSeekをWeChat・Discord・Telegramに接続](https://langbot.app/en/blog/connect-deepseek-to-wechat/)、[Dify AgentをDiscord・Telegram・Slackで動かす](https://langbot.app/en/blog/dify-agent-discord-telegram-slack/)、[n8n連携チャットボットを構築](https://langbot.app/en/blog/n8n-multi-platform-ai-chatbot/)。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -50,7 +50,7 @@ LangBot은 AI 기반 인스턴트 메시징 봇을 구축하기 위한 **오픈
|
|||||||
|
|
||||||
[→ 모든 기능 자세히 보기](https://link.langbot.app/en/docs/features)
|
[→ 모든 기능 자세히 보기](https://link.langbot.app/en/docs/features)
|
||||||
|
|
||||||
📍 실전 가이드: [5분 만에 멀티 플랫폼 AI 봇 배포하기](https://blog.langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [DeepSeek를 WeChat, Discord, Telegram에 연결하기](https://blog.langbot.app/en/blog/connect-deepseek-to-wechat/), [Dify Agent를 Discord, Telegram, Slack에서 실행하기](https://blog.langbot.app/en/blog/dify-agent-discord-telegram-slack/), [n8n 기반 챗봇 만들기](https://blog.langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
|
📍 실전 가이드: [5분 만에 멀티 플랫폼 AI 봇 배포하기](https://langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [DeepSeek를 WeChat, Discord, Telegram에 연결하기](https://langbot.app/en/blog/connect-deepseek-to-wechat/), [Dify Agent를 Discord, Telegram, Slack에서 실행하기](https://langbot.app/en/blog/dify-agent-discord-telegram-slack/), [n8n 기반 챗봇 만들기](https://langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -50,7 +50,7 @@ LangBot — это **платформа с открытым исходным к
|
|||||||
|
|
||||||
[→ Подробнее обо всех возможностях](https://link.langbot.app/en/docs/features)
|
[→ Подробнее обо всех возможностях](https://link.langbot.app/en/docs/features)
|
||||||
|
|
||||||
📍 Практические руководства: [развернуть мультиплатформенного ИИ-бота за 5 минут](https://blog.langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [подключить DeepSeek к WeChat, Discord и Telegram](https://blog.langbot.app/en/blog/connect-deepseek-to-wechat/), [запустить Dify Agent в Discord, Telegram и Slack](https://blog.langbot.app/en/blog/dify-agent-discord-telegram-slack/) и [создать чат-бота на n8n](https://blog.langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
|
📍 Практические руководства: [развернуть мультиплатформенного ИИ-бота за 5 минут](https://langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [подключить DeepSeek к WeChat, Discord и Telegram](https://langbot.app/en/blog/connect-deepseek-to-wechat/), [запустить Dify Agent в Discord, Telegram и Slack](https://langbot.app/en/blog/dify-agent-discord-telegram-slack/) и [создать чат-бота на n8n](https://langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -52,7 +52,7 @@ LangBot 是一個**開源的生產級平台**，用於建構 AI 驅動的即時
|
|||||||
|
|
||||||
[→ 了解更多功能特性](https://link.langbot.app/zh/docs/features)
|
[→ 了解更多功能特性](https://link.langbot.app/zh/docs/features)
|
||||||
|
|
||||||
📍 實踐指南：[5 分鐘部署多平台 AI 機器人](https://blog.langbot.app/zh/blog/deploy-ai-bot-in-5-minutes/)、[將 DeepSeek 接入微信、企業微信與 Discord](https://blog.langbot.app/zh/blog/connect-deepseek-to-wechat/)、[讓 Dify Agent 跑在 Discord、Telegram 和 Slack 上](https://blog.langbot.app/zh/blog/dify-agent-discord-telegram-slack/)，以及[用 n8n 建構多平台 AI 聊天機器人](https://blog.langbot.app/zh/blog/n8n-multi-platform-ai-chatbot/)。
|
📍 實踐指南：[5 分鐘部署多平台 AI 機器人](https://langbot.app/zh/blog/deploy-ai-bot-in-5-minutes/)、[將 DeepSeek 接入微信、企業微信與 Discord](https://langbot.app/zh/blog/connect-deepseek-to-wechat/)、[讓 Dify Agent 跑在 Discord、Telegram 和 Slack 上](https://langbot.app/zh/blog/dify-agent-discord-telegram-slack/)，以及[用 n8n 建構多平台 AI 聊天機器人](https://langbot.app/zh/blog/n8n-multi-platform-ai-chatbot/)。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -50,7 +50,7 @@ LangBot là một **nền tảng mã nguồn mở, cấp sản xuất** để x
|
|||||||
|
|
||||||
[→ Tìm hiểu thêm về tất cả tính năng](https://link.langbot.app/en/docs/features)
|
[→ Tìm hiểu thêm về tất cả tính năng](https://link.langbot.app/en/docs/features)
|
||||||
|
|
||||||
📍 Hướng dẫn thực hành: [triển khai bot AI đa nền tảng trong 5 phút](https://blog.langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [kết nối DeepSeek với WeChat, Discord và Telegram](https://blog.langbot.app/en/blog/connect-deepseek-to-wechat/), [chạy Dify Agent trên Discord, Telegram và Slack](https://blog.langbot.app/en/blog/dify-agent-discord-telegram-slack/) và [xây dựng chatbot với n8n](https://blog.langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
|
📍 Hướng dẫn thực hành: [triển khai bot AI đa nền tảng trong 5 phút](https://langbot.app/en/blog/deploy-ai-bot-in-5-minutes/), [kết nối DeepSeek với WeChat, Discord và Telegram](https://langbot.app/en/blog/connect-deepseek-to-wechat/), [chạy Dify Agent trên Discord, Telegram và Slack](https://langbot.app/en/blog/dify-agent-discord-telegram-slack/) và [xây dựng chatbot với n8n](https://langbot.app/en/blog/n8n-multi-platform-ai-chatbot/).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,169 @@
|
|||||||
|
# Valkey Search Vector Database Integration
|
||||||
|
|
||||||
|
This document describes how to use **Valkey Search** (the search/vector module bundled in
|
||||||
|
`valkey/valkey-bundle`) as the vector database backend for LangBot's knowledge base (RAG)
|
||||||
|
feature.
|
||||||
|
|
||||||
|
## What is Valkey Search?
|
||||||
|
|
||||||
|
**Valkey Search** is a module that adds vector similarity search and full-text search to
|
||||||
|
[Valkey](https://valkey.io/), the open-source, BSD-licensed in-memory data store forked from
|
||||||
|
Redis OSS. It is distributed in the `valkey/valkey-bundle` image alongside other modules
|
||||||
|
(JSON, Bloom, LDAP).
|
||||||
|
|
||||||
|
LangBot talks to Valkey through the official [`valkey-glide`](https://pypi.org/project/valkey-glide/)
|
||||||
|
client (Rust core + async Python wrapper), using its native `ft` (search) command namespace.
|
||||||
|
|
||||||
|
### Key Features
|
||||||
|
|
||||||
|
- **Vector search**: ANN via HNSW or exact via FLAT, with COSINE / L2 / IP distance metrics
|
||||||
|
- **Full-text search**: term, prefix and phrase matching over indexed text fields
|
||||||
|
- **Hybrid search**: a metadata/text filter pre-selects candidates, then KNN ranks them
|
||||||
|
- **In-memory speed**: vectors and documents are stored as Valkey HASH keys
|
||||||
|
- **Auth + TLS**: optional username/password and TLS for production (toB / SaaS) deployments
|
||||||
|
|
||||||
|
### Licensing
|
||||||
|
|
||||||
|
- Valkey core and the Search module are **BSD-3-Clause**.
|
||||||
|
- The `valkey-glide` client is **Apache-2.0**.
|
||||||
|
|
||||||
|
Both are compatible with LangBot.
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
Valkey Search support is included when you install LangBot — the `valkey-glide` dependency is
|
||||||
|
declared in `pyproject.toml`. To install manually:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install 'valkey-glide>=2.4.1,<3.0.0'
|
||||||
|
```
|
||||||
|
|
||||||
|
You also need a running Valkey server with the Search module loaded. The simplest way is the
|
||||||
|
bundled image:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run valkey-bundle (includes the Search module) on host port 6380
|
||||||
|
podman run -d --name valkey-test-langbot -p 6380:6379 valkey/valkey-bundle:9.1.0
|
||||||
|
# (docker run ... works identically)
|
||||||
|
```
|
||||||
|
|
||||||
|
`valkey-bundle` ships multi-arch images (linux/amd64 + linux/arm64), so it runs on both CI
|
||||||
|
(x86_64) and Apple-silicon dev machines.
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
Valkey Search is **opt-in and disabled by default** — the default `vdb.use` stays `chroma`,
|
||||||
|
so existing single-process deployments are unaffected. To enable it, edit your `config.yaml`:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
vdb:
|
||||||
|
use: valkey_search
|
||||||
|
valkey_search:
|
||||||
|
host: 'localhost'
|
||||||
|
port: 6379 # use 6380 if you started the container as shown above
|
||||||
|
db: 0
|
||||||
|
password: '' # optional (ACL / requirepass) — never logged
|
||||||
|
username: '' # optional (ACL user)
|
||||||
|
tls: false # optional (toB / SaaS)
|
||||||
|
index_algorithm: 'HNSW' # HNSW | FLAT
|
||||||
|
distance_metric: 'COSINE' # COSINE | L2 | IP
|
||||||
|
request_timeout: 5000 # per-request timeout in ms
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Default | Description |
|
||||||
|
|--------|---------|-------------|
|
||||||
|
| `host` | `localhost` | Valkey host |
|
||||||
|
| `port` | `6379` | Valkey port |
|
||||||
|
| `db` | `0` | Logical database id |
|
||||||
|
| `password` | `''` | Optional auth password (empty = no auth). Never logged. |
|
||||||
|
| `username` | `''` | Optional ACL username. Configuring a username without a password fails closed (raises) rather than connecting unauthenticated. |
|
||||||
|
| `tls` | `false` | Enable TLS for the connection |
|
||||||
|
| `index_algorithm` | `HNSW` | `HNSW` (approximate) or `FLAT` (exact) |
|
||||||
|
| `distance_metric` | `COSINE` | `COSINE`, `L2`, or `IP` |
|
||||||
|
| `request_timeout` | `5000` | Per-request timeout in milliseconds. The valkey-glide default (250ms) is too low for vector KNN under load; raise it further for remote/cross-AZ Valkey. |
|
||||||
|
|
||||||
|
### Connection behavior
|
||||||
|
|
||||||
|
The backend uses a **lazy** connection (`lazy_connect=True`): the client is created on first
|
||||||
|
use and the connection is deferred to the first command. A misconfigured or unreachable Valkey
|
||||||
|
server therefore does **not** block LangBot from booting — knowledge-base operations will error
|
||||||
|
at call time instead, and you can recover by switching `vdb.use` back to another backend.
|
||||||
|
|
||||||
|
The connection sets a fixed `client_name` of `langbot_vector_client` so it is identifiable in
|
||||||
|
`CLIENT LIST` and monitoring dashboards.
|
||||||
|
|
||||||
|
## Supported search types
|
||||||
|
|
||||||
|
| Type | Behavior |
|
||||||
|
|------|----------|
|
||||||
|
| `vector` | Pure KNN over the embedding field |
|
||||||
|
| `full_text` | Term/phrase match over the indexed `document` text field |
|
||||||
|
| `hybrid` | Metadata/text filter **pre-selects** candidates, then KNN ranks them |
|
||||||
|
|
||||||
|
### ⚠️ Important: `vector_weight` is NOT honored
|
||||||
|
|
||||||
|
Valkey Search hybrid queries follow a **filter-then-KNN** model: the filter (and/or full-text
|
||||||
|
clause) narrows the candidate set, and the KNN stage ranks the survivors by vector distance.
|
||||||
|
There is **no native weighted score fusion** (unlike, e.g., SeekDB's RRF boost).
|
||||||
|
|
||||||
|
For interface compatibility the backend still accepts a `vector_weight` argument, but it is
|
||||||
|
**ignored** — passing different weights does not change result ordering. The first time a
|
||||||
|
non-default weight is supplied, the backend logs a one-time warning.
|
||||||
|
|
||||||
|
If weighted hybrid ranking is needed in the future, it can be added **application-side** (run
|
||||||
|
vector KNN and full-text search separately and blend the scores). That is intentionally out of
|
||||||
|
scope for this integration.
|
||||||
|
|
||||||
|
## Metadata & filtering
|
||||||
|
|
||||||
|
Documents are stored as Valkey HASH keys under the prefix `kb:{collection}:{id}` with fields:
|
||||||
|
|
||||||
|
- `vector` — the embedding, packed as little-endian FLOAT32
|
||||||
|
- `document` — the raw text (indexed as TEXT for full-text/hybrid search)
|
||||||
|
- `file_id` — promoted to an indexed TAG field so it is filterable
|
||||||
|
- `metadata_json` — the full metadata dict, preserved verbatim as JSON
|
||||||
|
|
||||||
|
Only **indexed** fields are filterable. Currently that is `file_id`. Filters referencing
|
||||||
|
non-indexed metadata keys are dropped with a warning (the same pragmatism used by the Milvus
|
||||||
|
and pgvector backends). All other metadata still round-trips intact via `metadata_json`.
|
||||||
|
|
||||||
|
Supported filter operators (canonical Chroma-style `where` syntax): `$eq`, `$ne`, `$gt`,
|
||||||
|
`$gte`, `$lt`, `$lte`, `$in`, `$nin`. Multiple top-level keys are AND-ed.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
Unit tests (filter mapping, float32 packing, reply parsing, import guard) run in the fast lane
|
||||||
|
with no server:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run pytest tests/unit_tests/vector/test_valkey_search_filter.py -q
|
||||||
|
```
|
||||||
|
|
||||||
|
Integration tests are **slow-gated** on `TEST_VALKEY_URL` and require a running server:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
podman run -d --name valkey-test-langbot -p 6380:6379 valkey/valkey-bundle:9.1.0
|
||||||
|
TEST_VALKEY_URL=valkey://localhost:6380 \
|
||||||
|
uv run pytest tests/integration/vector/test_valkey_search.py -m slow -q
|
||||||
|
```
|
||||||
|
|
||||||
|
The default upstream fast CI lane (`-m "not slow"`) skips these, matching the existing
|
||||||
|
PostgreSQL migration-test precedent.
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
| Symptom | Cause / fix |
|
||||||
|
|---------|-------------|
|
||||||
|
| Tests skip with "Valkey Search module not available" | The server is plain Valkey without the Search module. Use the `valkey/valkey-bundle` image. |
|
||||||
|
| `ConnectionError` at call time | Check `host`/`port`/auth; remember `lazy_connect` defers errors to first use. |
|
||||||
|
| Empty search results right after insert | The Search indexer is asynchronous; results become visible within a short delay. The integration tests poll/retry to account for this. |
|
||||||
|
| Hybrid ranking ignores `vector_weight` | Expected — see the caveat above. |
|
||||||
|
|
||||||
|
## Production considerations
|
||||||
|
|
||||||
|
- **Cluster mode**: Valkey Search in cluster mode uses an additional coordination port. This
|
||||||
|
integration targets standalone mode; cluster support is a future consideration.
|
||||||
|
- **Persistence**: configure Valkey RDB/AOF persistence if the knowledge base must survive
|
||||||
|
restarts; otherwise an in-memory store is ephemeral.
|
||||||
|
- **Security**: set `password`/`username` and `tls: true` for any non-local deployment.
|
||||||
|
Credentials are never written to logs.
|
||||||
@@ -0,0 +1,858 @@
|
|||||||
|
# LangBot 多租户与多用户改造方案
|
||||||
|
|
||||||
|
## 目标
|
||||||
|
|
||||||
|
本方案面向 LangBot 从“单实例单管理员”演进到 SaaS 友好的“多 workspace、多账户、多权限”架构。
|
||||||
|
|
||||||
|
核心定义：
|
||||||
|
|
||||||
|
- Account：登录主体。一个自然人或服务账号，可加入多个 workspace。
|
||||||
|
- Workspace：租户边界。一个 workspace 内可拥有多个用户、机器人、流水线、模型、知识库、扩展、监控数据与 API Key。
|
||||||
|
- Membership：账户与 workspace 的关系，承载角色与权限。
|
||||||
|
- Role/Permission：workspace 内权限，不再用“是否是当前唯一用户”来决定访问能力。
|
||||||
|
|
||||||
|
目标体验：
|
||||||
|
|
||||||
|
- 新用户登录后可以创建 workspace、加入 workspace、切换 workspace。
|
||||||
|
- 同一个账户可加入多个 workspace，每个 workspace 权限不同。
|
||||||
|
- 一个 workspace 可邀请多个用户协作，并分别设置 owner/admin/editor/viewer 等权限。
|
||||||
|
- 所有业务资源默认属于某个 workspace，所有 API 默认在当前 workspace 下工作。
|
||||||
|
- Plugin SDK、MCP、知识库、模型调用、监控日志都能拿到稳定的 workspace 上下文，并且不跨租户泄露数据。
|
||||||
|
|
||||||
|
## 调研结论
|
||||||
|
|
||||||
|
### 当前 LangBot 的单用户假设
|
||||||
|
|
||||||
|
LangBot 现在已经有 `users` 表和 JWT 登录，但仍是单用户/单租户模型：
|
||||||
|
|
||||||
|
- `src/langbot/pkg/entity/persistence/user.py` 的 `User` 只保存 `user/password/account_type/space_*`，没有角色、状态、workspace 关系。
|
||||||
|
- `src/langbot/pkg/api/http/service/user.py` 通过 `is_initialized()` 判断系统是否已有用户；`create_or_update_space_user()` 在系统已初始化且邮箱不匹配时拒绝新用户，这直接限制了多用户登录。
|
||||||
|
- `src/langbot/pkg/api/http/controller/group.py` 的 `AuthType.USER_TOKEN` 验证后只向 handler 注入 `user_email`；JWT payload 也只有 `user`，没有 `account_id`、`workspace_id`、`role`、`permissions`。
|
||||||
|
- `src/langbot/pkg/api/http/service/apikey.py` 的 API Key 只验证 key 是否存在，没有 owner、scope、workspace。
|
||||||
|
- `web/src/app/infra/http/BaseHttpClient.ts` 从 `localStorage.token` 读取单个 token，并在所有请求上加 `Authorization`；前端没有 workspace selector，也没有当前 workspace 上下文。
|
||||||
|
|
||||||
|
当前登录流程更像“初始化一个本地管理账号”，而不是 SaaS 账户体系。要支持多用户，必须把“初始化状态”和“首个 workspace 创建”拆开。
|
||||||
|
|
||||||
|
### 业务资源当前都是全局资源
|
||||||
|
|
||||||
|
主要持久化表没有租户字段：
|
||||||
|
|
||||||
|
- Bot：`bots`
|
||||||
|
- Pipeline：`legacy_pipelines`、`pipeline_run_records`
|
||||||
|
- Model：`model_providers`、`llm_models`、`embedding_models`、`rerank_models`
|
||||||
|
- Plugin：`plugin_settings`
|
||||||
|
- MCP：`mcp_servers`
|
||||||
|
- RAG：`knowledge_bases`、`knowledge_base_files`、`knowledge_base_chunks`
|
||||||
|
- Monitoring：`monitoring_messages`、`monitoring_llm_calls`、`monitoring_sessions`、`monitoring_errors`、`monitoring_embedding_calls`、`monitoring_feedback`
|
||||||
|
- API Key：`api_keys`
|
||||||
|
- Webhook：`webhooks`
|
||||||
|
- Metadata：`metadata`
|
||||||
|
- Binary storage：`binary_storages`
|
||||||
|
|
||||||
|
对应服务也直接 select 全表，例如：
|
||||||
|
|
||||||
|
- `BotService.get_bots()` 返回所有 bot。
|
||||||
|
- `PipelineService.get_pipelines()` 返回所有 pipeline。
|
||||||
|
- `ModelProviderService.get_providers()` 返回所有 provider。
|
||||||
|
- `MCPService.get_mcp_servers()` 返回所有 MCP server。
|
||||||
|
- 插件和二进制存储没有 workspace 维度，插件 workspace storage 在 SDK 里还硬编码为 `default`。
|
||||||
|
|
||||||
|
所以改造重点不是只给用户表加字段，而是给资源访问层统一加入 workspace scope。
|
||||||
|
|
||||||
|
### 运行时也存在全局单例假设
|
||||||
|
|
||||||
|
`src/langbot/pkg/core/stages/build_app.py` 初始化的是一个全局 `Application`，其中包含单例：
|
||||||
|
|
||||||
|
- `platform_mgr`
|
||||||
|
- `pipeline_mgr`
|
||||||
|
- `model_mgr`
|
||||||
|
- `tool_mgr`
|
||||||
|
- `plugin_connector`
|
||||||
|
- `sess_mgr`
|
||||||
|
- `rag_mgr`
|
||||||
|
- `vector_db_mgr`
|
||||||
|
|
||||||
|
当前运行时把所有 bot、pipeline、model、plugin、MCP 都加载到同一套内存管理器。多租户改造需要决定：是共享运行时并在对象上带 workspace 过滤，还是每个 workspace 拆 runtime shard。第一阶段建议共享进程、强制 workspace-aware；等规模变大后再演进为按 workspace 分片。
|
||||||
|
|
||||||
|
### Plugin SDK 对 workspace 的假设
|
||||||
|
|
||||||
|
SDK 当前只认识 bot/pipeline/query/session，不认识租户：
|
||||||
|
|
||||||
|
- `src/langbot_plugin/api/entities/builtin/pipeline/query.py` 的 `Query` 有 `query_id/launcher_type/launcher_id/sender_id/bot_uuid/pipeline_uuid`，没有 `workspace_id/account_id`。
|
||||||
|
- `src/langbot_plugin/api/entities/builtin/provider/session.py` 的 `Session` 只按 `launcher_type + launcher_id` 表达会话。
|
||||||
|
- `src/langbot_plugin/api/proxies/langbot_api.py` 暴露 `get_bots/get_llm_models/invoke_llm/list_tools/vector_*` 等 Host API，都是全局语义。
|
||||||
|
- `src/langbot_plugin/runtime/io/handlers/plugin.py` 的 `set_workspace_storage/get_workspace_storage` 把 `owner_type` 设为 `workspace`，但 `owner` 固定为 `default`。
|
||||||
|
- LangBot 侧 `src/langbot/pkg/plugin/handler.py` 处理插件请求时，会把 `GET_BOTS`、`GET_LLM_MODELS`、`VECTOR_*` 等转到全局服务。
|
||||||
|
|
||||||
|
这意味着多租户落地时，不能只在 Web API 层过滤；插件可以通过 Host API 访问全局资源，所以 SDK/Runtime 通信也必须传递 workspace context。
|
||||||
|
|
||||||
|
## 开源版与商业版产品边界
|
||||||
|
|
||||||
|
LangBot 是开源软件，但多 workspace 能力本质上是 SaaS 控制面能力。如果把完整多 workspace、成员协作、订阅权益和配额代码都放进开源仓库，只靠本地 feature flag 或本地 license check，无法有效避免第三方 fork 后自建 SaaS。因此建议采用 open-core 架构：开源版保留单 workspace 执行能力，账户、订阅、权益和多 workspace 协作能力放到 LangBot Space/Cloud Control Plane 和商业模块中。
|
||||||
|
|
||||||
|
### 版本边界
|
||||||
|
|
||||||
|
推荐拆成三层：
|
||||||
|
|
||||||
|
- `LangBot Core OSS`：开源，自托管，默认只有一个隐式 `default workspace`。它可以在数据结构上兼容 workspace，但产品能力上不提供创建多个 workspace、切换 workspace、成员邀请、成员权限管理、审计和多租户配额。
|
||||||
|
- `LangBot Space / Cloud Control Plane`：托管控制面，负责 account、workspace、membership、subscription、billing、entitlement、license token、workspace quota、marketplace 权益等能力。
|
||||||
|
- `LangBot Commercial Module`：商业闭源或私有包，承载多 workspace、团队协作、RBAC、自定义角色、审计日志、SAML/SSO、企业私有化授权等能力。
|
||||||
|
|
||||||
|
企业私有化版本可以采用 `LangBot Core + Commercial Module + License Token` 的形式交付。开源 Core 仍然可独立运行，但只能作为单 workspace 自托管产品，不提供 SaaS 多租户控制面。
|
||||||
|
|
||||||
|
### OSS 中如何保留兼容但不开放多 workspace
|
||||||
|
|
||||||
|
为了让后续商业版复用同一套资源隔离模型，OSS 代码里可以保留 `workspace_uuid` 相关字段和默认 workspace 迁移，但应限制为单 workspace：
|
||||||
|
|
||||||
|
- 首次初始化时创建一个 `Default Workspace`。
|
||||||
|
- 所有资源自动归属这个 default workspace。
|
||||||
|
- 不暴露 `POST /api/v1/workspaces`。
|
||||||
|
- 不暴露 workspace switcher。
|
||||||
|
- 不暴露成员邀请和成员角色管理。
|
||||||
|
- 不支持一个 account 加入多个 workspace。
|
||||||
|
- 不支持 workspace 数量大于 1。
|
||||||
|
- 前端不展示 workspace selector。
|
||||||
|
- API 层如果收到非 default workspace 的 `X-Workspace-Id`，直接拒绝。
|
||||||
|
|
||||||
|
也就是说，OSS 可以是 workspace-aware，但不是 multi-workspace-enabled。这样做的价值是：代码结构提前适配租户隔离，未来商业版不用重写所有资源模型；同时开源版用户无法直接通过 UI/API 获得 SaaS 型多租户能力。
|
||||||
|
|
||||||
|
### 账户、订阅和权益抽到 Space
|
||||||
|
|
||||||
|
账户和订阅体系建议从 LangBot Core 中抽出，交给 Space 控制面：
|
||||||
|
|
||||||
|
```text
|
||||||
|
LangBot Space
|
||||||
|
-> Account
|
||||||
|
-> Workspace
|
||||||
|
-> Membership
|
||||||
|
-> Subscription
|
||||||
|
-> Entitlement
|
||||||
|
-> License Token
|
||||||
|
|
||||||
|
LangBot Core
|
||||||
|
-> Validate entitlement / license
|
||||||
|
-> Run bots, pipelines, plugins, MCP, RAG
|
||||||
|
-> Enforce local resource scope
|
||||||
|
-> Report usage
|
||||||
|
```
|
||||||
|
|
||||||
|
这样做有几个原因：
|
||||||
|
|
||||||
|
- 账号体系如果完全在本地，第三方容易直接改库创建 workspace/membership。
|
||||||
|
- 订阅、配额和商业权益如果完全在本地，容易绕过。
|
||||||
|
- Space 可以统一处理 OAuth、组织、邀请、付款、发票、套餐、权益、Marketplace 分发权限。
|
||||||
|
- LangBot Core 只作为执行面消费 Space 下发的 entitlement，减少商业规则暴露。
|
||||||
|
|
||||||
|
### Entitlement 设计
|
||||||
|
|
||||||
|
Space 向 LangBot Core 下发签名权益，可以是在线校验，也可以为企业版提供短期/长期离线 license token。
|
||||||
|
|
||||||
|
示例：
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"edition": "oss",
|
||||||
|
"workspace_limit": 1,
|
||||||
|
"member_limit": 1,
|
||||||
|
"multi_workspace": false,
|
||||||
|
"rbac": false,
|
||||||
|
"audit_log": false,
|
||||||
|
"custom_roles": false,
|
||||||
|
"sso": false,
|
||||||
|
"commercial_use": false,
|
||||||
|
"expires_at": 1893456000
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
OSS 默认权益：
|
||||||
|
|
||||||
|
- `workspace_limit = 1`
|
||||||
|
- `member_limit = 1`
|
||||||
|
- `multi_workspace = false`
|
||||||
|
- `rbac = false`
|
||||||
|
- `audit_log = false`
|
||||||
|
- `sso = false`
|
||||||
|
|
||||||
|
Cloud/Pro/Enterprise 权益：
|
||||||
|
|
||||||
|
- `workspace_limit > 1`
|
||||||
|
- `member_limit > 1`
|
||||||
|
- `multi_workspace = true`
|
||||||
|
- `rbac = true`
|
||||||
|
- 可按套餐打开 audit、custom roles、SSO、usage reporting、enterprise support 等能力。
|
||||||
|
|
||||||
|
Core 执行面需要在关键入口强制校验 entitlement：
|
||||||
|
|
||||||
|
- 创建 workspace。
|
||||||
|
- 邀请成员。
|
||||||
|
- 修改成员角色。
|
||||||
|
- 切换 workspace。
|
||||||
|
- 创建超过 quota 的资源。
|
||||||
|
- 开启商业模块功能。
|
||||||
|
|
||||||
|
### 商业模块边界
|
||||||
|
|
||||||
|
以下能力不建议进入 OSS 仓库的完整实现：
|
||||||
|
|
||||||
|
- 多 workspace 创建和切换。
|
||||||
|
- Workspace 成员邀请。
|
||||||
|
- Workspace RBAC 和自定义角色。
|
||||||
|
- Workspace 审计日志。
|
||||||
|
- Workspace 级用量和配额管理。
|
||||||
|
- 订阅、账单、发票。
|
||||||
|
- 企业 SSO/SAML/OIDC。
|
||||||
|
- 在线/离线 license 管理。
|
||||||
|
- 多租户 SaaS 运营控制台。
|
||||||
|
|
||||||
|
OSS 仓库可以保留接口占位、默认 workspace 兼容和必要的数据隔离字段，但完整交互、管理 UI、权益校验器和多 workspace policy engine 应由 Space 或商业模块提供。
|
||||||
|
|
||||||
|
### 防自建 SaaS 的现实边界
|
||||||
|
|
||||||
|
技术上无法 100% 阻止别人 fork 开源代码后自行改造。更可靠的策略是组合：
|
||||||
|
|
||||||
|
- 不把完整商业多 workspace 实现放进 OSS。
|
||||||
|
- Space 控制面提供账号、订阅、权益、Marketplace 和官方托管能力。
|
||||||
|
- 商业模块闭源或私有分发。
|
||||||
|
- 使用商标、云服务条款和商业 license 限制“自称官方 LangBot SaaS”或未经授权商用托管。
|
||||||
|
- 如果当前开源 license 对托管商用限制不足，需要单独评估 license 策略，必要时引入 open-core license 或新增商业附加条款。具体 license 调整需要法律评审。
|
||||||
|
|
||||||
|
结论：多 workspace 的底层 schema 可以在 OSS 中以 default workspace 兼容方式铺路，但多 workspace 产品能力、账户订阅权益、协作管理和 SaaS 控制面应放到 Space/商业模块，不作为开源版可直接使用的能力。
|
||||||
|
|
||||||
|
## 推荐总体架构
|
||||||
|
|
||||||
|
采用“单实例多 workspace，资源行级隔离，运行时上下文隔离”的架构：
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TD
|
||||||
|
A["Account"] --> B["WorkspaceMembership"]
|
||||||
|
B --> C["Workspace"]
|
||||||
|
C --> D["Bots"]
|
||||||
|
C --> E["Pipelines"]
|
||||||
|
C --> F["Models & Providers"]
|
||||||
|
C --> G["Knowledge Bases"]
|
||||||
|
C --> H["Extensions: Plugins / MCP"]
|
||||||
|
C --> I["API Keys & Webhooks"]
|
||||||
|
C --> J["Monitoring"]
|
||||||
|
D --> K["Runtime Query"]
|
||||||
|
E --> K
|
||||||
|
K --> L["Plugin Runtime"]
|
||||||
|
K --> M["MCP Runtime"]
|
||||||
|
L --> N["Workspace-scoped Host APIs"]
|
||||||
|
```
|
||||||
|
|
||||||
|
原则：
|
||||||
|
|
||||||
|
- 账户全局唯一，workspace 是所有业务资源的归属边界。
|
||||||
|
- 所有 HTTP handler 在进入业务服务前解析出 `RequestContext(account, workspace, membership, permissions)`。
|
||||||
|
- 所有 service 方法显式接收 `ctx` 或 `workspace_id`，禁止在业务服务里无条件 select 全表。
|
||||||
|
- 运行时对象的 key 从 `uuid` 扩展为 `(workspace_id, uuid)` 或使用全局唯一 uuid 但必须记录 workspace_id 并校验。
|
||||||
|
- 插件/MCP/知识库/模型调用都按 query 所属 workspace 过滤可用资源。
|
||||||
|
|
||||||
|
## 数据模型设计
|
||||||
|
|
||||||
|
### Account
|
||||||
|
|
||||||
|
替代现有 `users` 的语义，建议保留表名但升级字段，避免过大迁移：
|
||||||
|
|
||||||
|
字段建议：
|
||||||
|
|
||||||
|
- `id`
|
||||||
|
- `uuid`
|
||||||
|
- `email`
|
||||||
|
- `password_hash`
|
||||||
|
- `display_name`
|
||||||
|
- `avatar_url`
|
||||||
|
- `account_type`: `local | space`
|
||||||
|
- `status`: `active | disabled | deleted`
|
||||||
|
- `space_account_uuid`
|
||||||
|
- `space_access_token`
|
||||||
|
- `space_refresh_token`
|
||||||
|
- `space_access_token_expires_at`
|
||||||
|
- `space_api_key`
|
||||||
|
- `created_at`
|
||||||
|
- `updated_at`
|
||||||
|
|
||||||
|
兼容策略：
|
||||||
|
|
||||||
|
- 旧字段 `user` 迁移为 `email`，可以短期保留 alias。
|
||||||
|
- 旧 `password` 迁移为 `password_hash`，也可先保持列名不变，service 层改命名。
|
||||||
|
- JWT 中不要继续只放 email，应放 `sub=account_uuid`。
|
||||||
|
|
||||||
|
### Workspace
|
||||||
|
|
||||||
|
新增 `workspaces`：
|
||||||
|
|
||||||
|
- `uuid`
|
||||||
|
- `name`
|
||||||
|
- `slug`
|
||||||
|
- `avatar_url`
|
||||||
|
- `type`: `personal | team`
|
||||||
|
- `status`: `active | suspended | deleted`
|
||||||
|
- `default_language`
|
||||||
|
- `created_by_account_uuid`
|
||||||
|
- `created_at`
|
||||||
|
- `updated_at`
|
||||||
|
|
||||||
|
每个账户首次登录时自动创建一个 personal workspace。旧单用户实例迁移时创建一个 `Default Workspace`。
|
||||||
|
|
||||||
|
### WorkspaceMembership
|
||||||
|
|
||||||
|
新增 `workspace_memberships`：
|
||||||
|
|
||||||
|
- `workspace_uuid`
|
||||||
|
- `account_uuid`
|
||||||
|
- `role`: `owner | admin | developer | operator | viewer`
|
||||||
|
- `status`: `active | invited | disabled`
|
||||||
|
- `invited_by_account_uuid`
|
||||||
|
- `joined_at`
|
||||||
|
- `created_at`
|
||||||
|
- `updated_at`
|
||||||
|
|
||||||
|
唯一索引：
|
||||||
|
|
||||||
|
- `(workspace_uuid, account_uuid)`
|
||||||
|
|
||||||
|
### WorkspaceInvitation
|
||||||
|
|
||||||
|
新增 `workspace_invitations`：
|
||||||
|
|
||||||
|
- `uuid`
|
||||||
|
- `workspace_uuid`
|
||||||
|
- `email`
|
||||||
|
- `role`
|
||||||
|
- `token_hash`
|
||||||
|
- `expires_at`
|
||||||
|
- `accepted_at`
|
||||||
|
- `created_by_account_uuid`
|
||||||
|
- `created_at`
|
||||||
|
|
||||||
|
用于邀请外部用户加入 workspace。Space OAuth 登录时可以根据 email 自动匹配未接受邀请。
|
||||||
|
|
||||||
|
### Role 与 Permission
|
||||||
|
|
||||||
|
先用固定角色，后续再做自定义角色。
|
||||||
|
|
||||||
|
建议权限：
|
||||||
|
|
||||||
|
- `workspace.manage`
|
||||||
|
- `member.view`
|
||||||
|
- `member.invite`
|
||||||
|
- `member.update_role`
|
||||||
|
- `member.remove`
|
||||||
|
- `bot.view`
|
||||||
|
- `bot.manage`
|
||||||
|
- `pipeline.view`
|
||||||
|
- `pipeline.manage`
|
||||||
|
- `model.view`
|
||||||
|
- `model.manage`
|
||||||
|
- `knowledge.view`
|
||||||
|
- `knowledge.manage`
|
||||||
|
- `extension.view`
|
||||||
|
- `extension.manage`
|
||||||
|
- `monitoring.view`
|
||||||
|
- `apikey.manage`
|
||||||
|
- `webhook.manage`
|
||||||
|
- `billing.view`
|
||||||
|
- `billing.manage`
|
||||||
|
|
||||||
|
角色映射：
|
||||||
|
|
||||||
|
| Role | 说明 | 权限 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| owner | workspace 拥有者 | 全部权限；可转让 owner；不可被其他角色移除 |
|
||||||
|
| admin | 管理员 | 除 owner 转让和删除 workspace 外的全部权限 |
|
||||||
|
| developer | 构建者 | 管理 bot、pipeline、model、knowledge、extension、webhook，可看监控 |
|
||||||
|
| operator | 运营者 | 查看和启停 bot、查看监控、查看配置，不可改密钥和删除资源 |
|
||||||
|
| viewer | 只读成员 | 只读资源和监控 |
|
||||||
|
|
||||||
|
### 业务资源加 workspace_uuid
|
||||||
|
|
||||||
|
以下表需要新增 `workspace_uuid`：
|
||||||
|
|
||||||
|
- `bots`
|
||||||
|
- `legacy_pipelines`
|
||||||
|
- `pipeline_run_records`
|
||||||
|
- `model_providers`
|
||||||
|
- `llm_models`
|
||||||
|
- `embedding_models`
|
||||||
|
- `rerank_models`
|
||||||
|
- `plugin_settings`
|
||||||
|
- `mcp_servers`
|
||||||
|
- `knowledge_bases`
|
||||||
|
- `knowledge_base_files`
|
||||||
|
- `knowledge_base_chunks`
|
||||||
|
- `monitoring_*`
|
||||||
|
- `api_keys`
|
||||||
|
- `webhooks`
|
||||||
|
- `binary_storages`
|
||||||
|
- `metadata`
|
||||||
|
|
||||||
|
索引建议：
|
||||||
|
|
||||||
|
- 所有资源表加 `(workspace_uuid, created_at)` 或 `(workspace_uuid, updated_at)`。
|
||||||
|
- 资源唯一键从单列改为 workspace 复合唯一：
|
||||||
|
- `bots.uuid` 可保持全局唯一，但查询仍必须带 workspace。
|
||||||
|
- `plugin_settings` 主键从 `(plugin_author, plugin_name)` 改为 `(workspace_uuid, plugin_author, plugin_name)`。
|
||||||
|
- `mcp_servers.name` 如果未来要求唯一，必须是 `(workspace_uuid, name)`。
|
||||||
|
- `metadata.key` 改为 `(workspace_uuid, key)`，系统级 metadata 单独放 `system_metadata` 或使用 `workspace_uuid=NULL`。
|
||||||
|
- `binary_storages.unique_key` 建议改为 `workspace_uuid + owner_type + owner + key` 的 hash。
|
||||||
|
|
||||||
|
### API Key
|
||||||
|
|
||||||
|
API Key 必须归属于 workspace：
|
||||||
|
|
||||||
|
- `workspace_uuid`
|
||||||
|
- `created_by_account_uuid`
|
||||||
|
- `scopes`
|
||||||
|
- `expires_at`
|
||||||
|
- `last_used_at`
|
||||||
|
- `status`
|
||||||
|
|
||||||
|
验证 API Key 后生成 `RequestContext`：
|
||||||
|
|
||||||
|
- `account_uuid=None` 或 service account uuid
|
||||||
|
- `workspace_uuid=key.workspace_uuid`
|
||||||
|
- `permissions=key.scopes`
|
||||||
|
|
||||||
|
这样 `/api/v1/platform/bots/<uuid>/send_message` 之类接口不会跨 workspace 操作 bot。
|
||||||
|
|
||||||
|
## 后端改造方案
|
||||||
|
|
||||||
|
### RequestContext
|
||||||
|
|
||||||
|
新增统一上下文对象，例如：
|
||||||
|
|
||||||
|
```python
|
||||||
|
class RequestContext:
|
||||||
|
account_uuid: str | None
|
||||||
|
workspace_uuid: str
|
||||||
|
role: str | None
|
||||||
|
permissions: set[str]
|
||||||
|
auth_type: Literal["user_token", "api_key"]
|
||||||
|
```
|
||||||
|
|
||||||
|
改造 `RouterGroup.route()`：
|
||||||
|
|
||||||
|
- `USER_TOKEN`：验证 JWT，读取 `account_uuid`，再从 header/query/cookie 中解析 current workspace。
|
||||||
|
- `API_KEY`：验证 API Key，直接得到 workspace。
|
||||||
|
- `USER_TOKEN_OR_API_KEY`：两者都返回同一种 `RequestContext`。
|
||||||
|
- handler 参数从可选 `user_email` 升级为可选 `ctx`；兼容期同时支持 `user_email`。
|
||||||
|
|
||||||
|
当前 workspace 传递方式：
|
||||||
|
|
||||||
|
- 推荐 header：`X-Workspace-Id: <workspace_uuid>`
|
||||||
|
- Web 前端同时把当前 workspace 存在 localStorage。
|
||||||
|
- 如果未传，后端用账户最近使用 workspace 或第一个 active membership。
|
||||||
|
|
||||||
|
JWT payload：
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"sub": "account_uuid",
|
||||||
|
"email": "user@example.com",
|
||||||
|
"iss": "LangBot-...",
|
||||||
|
"exp": 1234567890
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
不要把 workspace 写死在 JWT 里，否则切换 workspace 需要刷新 token。可以额外支持短 TTL workspace token，但第一阶段不必。
|
||||||
|
|
||||||
|
### 服务层改造模式
|
||||||
|
|
||||||
|
所有 service 方法增加 `ctx` 或 `workspace_uuid`：
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def get_bots(self, ctx: RequestContext, include_secret: bool = True):
|
||||||
|
require(ctx, "bot.view")
|
||||||
|
query = sqlalchemy.select(Bot).where(Bot.workspace_uuid == ctx.workspace_uuid)
|
||||||
|
```
|
||||||
|
|
||||||
|
需要改造的服务：
|
||||||
|
|
||||||
|
- `UserService`：拆成 AccountService + WorkspaceService 更清晰。
|
||||||
|
- `ApiKeyService`：按 workspace 管理 key。
|
||||||
|
- `BotService`：所有 bot 查询/创建/更新/删除按 workspace。
|
||||||
|
- `PipelineService`：所有 pipeline 查询/默认 pipeline 按 workspace。
|
||||||
|
- `ModelProviderService` 和 model services：按 workspace 隔离 provider 和 model。
|
||||||
|
- `MCPService`：按 workspace 管理 MCP server，运行时按 workspace host。
|
||||||
|
- `KnowledgeService/RAGRuntimeService`：按 workspace 管理 KB、文件、collection。
|
||||||
|
- `MonitoringService`：记录和查询都带 workspace。
|
||||||
|
- `WebhookService`：按 workspace 管理 webhook。
|
||||||
|
- `PluginRuntimeConnector`：插件安装、设置、配置按 workspace。
|
||||||
|
|
||||||
|
### HTTP API 形态
|
||||||
|
|
||||||
|
保留现有路径，靠 `X-Workspace-Id` 表示当前 workspace，可减少前端和 SDK 破坏：
|
||||||
|
|
||||||
|
- `GET /api/v1/workspaces`
|
||||||
|
- `POST /api/v1/workspaces`
|
||||||
|
- `GET /api/v1/workspaces/current`
|
||||||
|
- `PUT /api/v1/workspaces/current`
|
||||||
|
- `GET /api/v1/workspaces/<workspace_uuid>/members`
|
||||||
|
- `POST /api/v1/workspaces/<workspace_uuid>/invitations`
|
||||||
|
- `PUT /api/v1/workspaces/<workspace_uuid>/members/<account_uuid>`
|
||||||
|
- `DELETE /api/v1/workspaces/<workspace_uuid>/members/<account_uuid>`
|
||||||
|
|
||||||
|
现有资源 API：
|
||||||
|
|
||||||
|
- `/api/v1/platform/bots`
|
||||||
|
- `/api/v1/pipelines`
|
||||||
|
- `/api/v1/provider/*`
|
||||||
|
- `/api/v1/plugins`
|
||||||
|
- `/api/v1/mcp`
|
||||||
|
- `/api/v1/knowledge`
|
||||||
|
|
||||||
|
继续保留，但必须从 `RequestContext.workspace_uuid` 过滤。
|
||||||
|
|
||||||
|
对外 API Key 也使用相同路径，只是由 key 决定 workspace。
|
||||||
|
|
||||||
|
### 初始化流程
|
||||||
|
|
||||||
|
现有 `/api/v1/user/init` 含义改为“创建首个账号和首个 workspace”：
|
||||||
|
|
||||||
|
1. 如果系统没有任何 account：
|
||||||
|
- 创建 account。
|
||||||
|
- 创建 personal/team workspace。
|
||||||
|
- 创建 owner membership。
|
||||||
|
- 创建默认 pipeline。
|
||||||
|
- 标记 wizard status 到该 workspace metadata。
|
||||||
|
2. 如果系统已有 account：
|
||||||
|
- 禁止无邀请注册，除非配置允许公开注册。
|
||||||
|
- Space OAuth 登录后，如果没有 membership，引导创建 workspace 或接受邀请。
|
||||||
|
|
||||||
|
`/api/v1/user/account-info` 不应再只返回 first user，应返回：
|
||||||
|
|
||||||
|
- `initialized`
|
||||||
|
- `registration_mode`
|
||||||
|
- `space_enabled`
|
||||||
|
- `default_login_methods`
|
||||||
|
|
||||||
|
登录成功后前端调用 `/api/v1/workspaces` 选择 workspace。
|
||||||
|
|
||||||
|
### 运行时隔离
|
||||||
|
|
||||||
|
第一阶段采用共享进程 + workspace-aware runtime：
|
||||||
|
|
||||||
|
- `RuntimeBot` 增加 `workspace_uuid`。
|
||||||
|
- `RuntimePipeline` 增加 `workspace_uuid`。
|
||||||
|
- `Query` 增加 `workspace_uuid`，从 bot/pipeline 派生。
|
||||||
|
- `SessionManager.get_session()` key 从 `(launcher_type, launcher_id)` 改为 `(workspace_uuid, bot_uuid, launcher_type, launcher_id)`。
|
||||||
|
- `PipelineManager.pipeline_dict` key 可保持 pipeline uuid，但所有 load/get 都校验 workspace；如果 uuid 不是全局唯一则改为 `(workspace_uuid, pipeline_uuid)`。
|
||||||
|
- `ModelManager` provider/model 加 workspace 过滤；`get_model_by_uuid` 必须确保 query workspace 可访问。
|
||||||
|
- `ToolManager` 中 MCP tools、plugin tools 按 query workspace 过滤。
|
||||||
|
|
||||||
|
后续规模化时可演进：
|
||||||
|
|
||||||
|
- workspace runtime shard：每个 workspace 一套 plugin runtime/MCP runtime。
|
||||||
|
- 大客户独立进程或独立数据库。
|
||||||
|
|
||||||
|
## Plugin SDK 与 Runtime 改造
|
||||||
|
|
||||||
|
### Query/Event 增加 workspace context
|
||||||
|
|
||||||
|
SDK `Query` 增加：
|
||||||
|
|
||||||
|
- `workspace_uuid: str`
|
||||||
|
- `workspace_slug: str | None`
|
||||||
|
- `account_uuid: str | None`，仅 Web/API 触发时可能有，聊天平台消息通常为空。
|
||||||
|
|
||||||
|
Event 模型通过 `event.query.workspace_uuid` 可拿到租户上下文；序列化时也应包含这些字段。
|
||||||
|
|
||||||
|
向后兼容：
|
||||||
|
|
||||||
|
- 字段可选，默认 `None`。
|
||||||
|
- 老插件不感知这些字段也能跑。
|
||||||
|
- 新插件可通过 `ctx.event.query.workspace_uuid` 或新增 `ctx.get_workspace()` 访问。
|
||||||
|
|
||||||
|
### Host API 默认按当前 workspace 限制
|
||||||
|
|
||||||
|
`LangBotAPIProxy` 的以下方法必须由 Host 端按 workspace 过滤：
|
||||||
|
|
||||||
|
- `get_bots`
|
||||||
|
- `get_bot_info`
|
||||||
|
- `send_message`
|
||||||
|
- `get_llm_models`
|
||||||
|
- `invoke_llm`
|
||||||
|
- `list_plugins_manifest`
|
||||||
|
- `list_commands`
|
||||||
|
- `list_tools`
|
||||||
|
- `call_tool`
|
||||||
|
- `invoke_embedding`
|
||||||
|
- `vector_*`
|
||||||
|
- `list_knowledge_bases`
|
||||||
|
- `retrieve_knowledge`
|
||||||
|
|
||||||
|
建议新增显式方法：
|
||||||
|
|
||||||
|
- `get_workspace_info()`
|
||||||
|
- `get_current_account()`
|
||||||
|
- `get_workspace_storage(...)`
|
||||||
|
|
||||||
|
但不要让插件传入任意 workspace id 来越权访问。插件请求的 workspace 应由 Runtime 根据当前 query/plugin connection 填充。
|
||||||
|
|
||||||
|
### Workspace storage 修复
|
||||||
|
|
||||||
|
当前 SDK runtime 中：
|
||||||
|
|
||||||
|
```python
|
||||||
|
data["owner_type"] = "workspace"
|
||||||
|
data["owner"] = "default"
|
||||||
|
```
|
||||||
|
|
||||||
|
必须改为：
|
||||||
|
|
||||||
|
- 如果请求来自 query/event：owner 为 `workspace_uuid`。
|
||||||
|
- 如果请求来自后台插件任务：owner 为 plugin 安装所属 workspace。
|
||||||
|
- Host 侧 `binary_storages` 加 `workspace_uuid`，并在 unique key 中包含 workspace。
|
||||||
|
|
||||||
|
Plugin storage 建议也同时加 workspace：
|
||||||
|
|
||||||
|
- 现在 plugin storage owner 是 `author/name`，这会导致同一插件在不同 workspace 的私有数据冲突。
|
||||||
|
- 应改为 `(workspace_uuid, plugin_id, key)`。
|
||||||
|
|
||||||
|
### 插件安装与配置
|
||||||
|
|
||||||
|
`plugin_settings` 从全局变为 workspace-scoped：
|
||||||
|
|
||||||
|
- 同一个插件可安装到多个 workspace。
|
||||||
|
- 每个 workspace 有自己的 enabled、priority、config、install_source、install_info。
|
||||||
|
- 插件 runtime 列表需要能按 workspace 过滤。
|
||||||
|
|
||||||
|
实现路线有两种：
|
||||||
|
|
||||||
|
1. 共享插件进程，插件代码只加载一份，设置和调用时附带 workspace。
|
||||||
|
2. 每个 workspace 一个插件容器实例，隔离最彻底但资源占用更高。
|
||||||
|
|
||||||
|
推荐第一阶段采用方案 1，但要求：
|
||||||
|
|
||||||
|
- 所有 RuntimeToLangBot/PluginToRuntime action 都能携带 `workspace_uuid`。
|
||||||
|
- 插件 config 获取时按 workspace 返回。
|
||||||
|
- 插件 page API 请求必须校验当前用户在该 workspace 有访问权限。
|
||||||
|
|
||||||
|
### MCP
|
||||||
|
|
||||||
|
MCP server 是租户资源：
|
||||||
|
|
||||||
|
- `mcp_servers.workspace_uuid`。
|
||||||
|
- MCP session key 从 `server_name` 改为 `(workspace_uuid, server_name)` 或使用全局 uuid。
|
||||||
|
- Pipeline extension preferences 中绑定 MCP server uuid 时，只能绑定同 workspace 的 server。
|
||||||
|
- MCP tool 列表在 query 执行时按 query.workspace_uuid + pipeline 绑定关系过滤。
|
||||||
|
|
||||||
|
## 前端改造
|
||||||
|
|
||||||
|
### Workspace selector
|
||||||
|
|
||||||
|
Home layout 顶部或 sidebar 增加 workspace selector：
|
||||||
|
|
||||||
|
- 当前 workspace 名称和头像。
|
||||||
|
- 切换 workspace 后写入 `localStorage.currentWorkspaceId`。
|
||||||
|
- 所有请求自动带 `X-Workspace-Id`。
|
||||||
|
- 切换后刷新 sidebar 数据和页面缓存。
|
||||||
|
|
||||||
|
`BaseHttpClient` request interceptor 增加：
|
||||||
|
|
||||||
|
```ts
|
||||||
|
const workspaceId = localStorage.getItem("currentWorkspaceId");
|
||||||
|
if (workspaceId) config.headers["X-Workspace-Id"] = workspaceId;
|
||||||
|
```
|
||||||
|
|
||||||
|
### 用户与成员管理页面
|
||||||
|
|
||||||
|
新增页面：
|
||||||
|
|
||||||
|
- `/home/workspace/settings`
|
||||||
|
- `/home/workspace/members`
|
||||||
|
- `/home/workspace/invitations`
|
||||||
|
|
||||||
|
能力：
|
||||||
|
|
||||||
|
- owner/admin 邀请成员。
|
||||||
|
- owner/admin 修改成员角色。
|
||||||
|
- owner 移除成员、转让 owner。
|
||||||
|
- 所有人可切换 workspace。
|
||||||
|
- viewer/operator 在 UI 上隐藏不可操作按钮，但后端仍做权限校验。
|
||||||
|
|
||||||
|
### 登录与注册
|
||||||
|
|
||||||
|
登录后流程：
|
||||||
|
|
||||||
|
1. `authUser` 拿 token。
|
||||||
|
2. `initializeUserInfo()` 获取 account info。
|
||||||
|
3. `GET /api/v1/workspaces`。
|
||||||
|
4. 如果没有 workspace：进入创建 workspace 向导。
|
||||||
|
5. 如果有多个 workspace：默认进入最近使用 workspace，可切换。
|
||||||
|
|
||||||
|
注册页不再表达“初始化管理员账号”，而是：
|
||||||
|
|
||||||
|
- 首次系统启动：创建首个 owner + default workspace。
|
||||||
|
- 后续：根据配置允许公开注册，或只能接受邀请。
|
||||||
|
|
||||||
|
### 旧页面影响
|
||||||
|
|
||||||
|
需要逐个检查这些页面的数据加载是否都依赖当前 workspace：
|
||||||
|
|
||||||
|
- Bots
|
||||||
|
- Pipelines
|
||||||
|
- Plugins/Market/MCP
|
||||||
|
- Knowledge
|
||||||
|
- Monitoring
|
||||||
|
- Models dialog
|
||||||
|
- API integration dialog
|
||||||
|
- Wizard
|
||||||
|
|
||||||
|
## 迁移方案
|
||||||
|
|
||||||
|
### 迁移阶段 0：准备
|
||||||
|
|
||||||
|
- 引入 `workspaces`、`workspace_memberships`、`workspace_invitations`。
|
||||||
|
- 给 `users` 增加 `uuid/status/display_name` 等字段。
|
||||||
|
- 创建 `RequestContext`，但先不强制所有服务改完。
|
||||||
|
|
||||||
|
### 迁移阶段 1：默认 workspace
|
||||||
|
|
||||||
|
对现有实例执行迁移：
|
||||||
|
|
||||||
|
1. 创建 `Default Workspace`。
|
||||||
|
2. 找到现有第一个 user，设为 owner。
|
||||||
|
3. 所有已有资源写入 `workspace_uuid=default_workspace_uuid`。
|
||||||
|
4. `metadata` 迁入 default workspace；确实全局的配置放到 `system_metadata`。
|
||||||
|
5. `binary_storages` 中 `owner_type=workspace, owner=default` 改为 owner 为 default workspace uuid。
|
||||||
|
6. 插件 `plugin_settings` 归入 default workspace。
|
||||||
|
|
||||||
|
### 迁移阶段 2：服务层强制 scope
|
||||||
|
|
||||||
|
- 改所有 service 查询，必须要求 `workspace_uuid`。
|
||||||
|
- API Key 迁移为 workspace key。
|
||||||
|
- 所有写操作必须检查权限。
|
||||||
|
- 监控和任务查询按 workspace 过滤。
|
||||||
|
|
||||||
|
### 迁移阶段 3：运行时上下文
|
||||||
|
|
||||||
|
- `Query`、`Session`、`RuntimeBot`、`RuntimePipeline` 增加 workspace。
|
||||||
|
- Plugin/MCP/Model/RAG runtime 全部按 workspace 过滤。
|
||||||
|
- 修复 SDK workspace storage。
|
||||||
|
|
||||||
|
### 迁移阶段 4：前端多 workspace
|
||||||
|
|
||||||
|
- 登录后 workspace 选择。
|
||||||
|
- Header/sidebar workspace switcher。
|
||||||
|
- 成员和邀请管理。
|
||||||
|
- 所有 API 请求带 `X-Workspace-Id`。
|
||||||
|
|
||||||
|
### 迁移阶段 5：安全收敛
|
||||||
|
|
||||||
|
- 添加跨 workspace 越权测试。
|
||||||
|
- 添加 API Key scope 测试。
|
||||||
|
- 添加插件 Host API 过滤测试。
|
||||||
|
- 添加 MCP 和 RAG 隔离测试。
|
||||||
|
|
||||||
|
## 安全边界
|
||||||
|
|
||||||
|
必须防的场景：
|
||||||
|
|
||||||
|
- 用户 A 修改 URL 中 bot uuid，访问用户 B workspace 的 bot。
|
||||||
|
- API Key 来自 workspace A，但调用 workspace B 的 bot。
|
||||||
|
- 插件通过 `get_bots()` 枚举所有 workspace 的 bot。
|
||||||
|
- 插件通过 `workspace_storage` 读取其它 workspace 的数据。
|
||||||
|
- MCP server 名称相同导致 session 复用。
|
||||||
|
- monitoring session_id 相同导致数据串租户。
|
||||||
|
- Space OAuth 登录时，同 email 账户被错误绑定到已有本地 account。
|
||||||
|
|
||||||
|
建议策略：
|
||||||
|
|
||||||
|
- 所有资源访问都使用 `workspace_uuid + resource_id`。
|
||||||
|
- 所有 service 方法入口做权限检查。
|
||||||
|
- 插件 Host API 的 workspace 不信任插件入参，只信任 query/runtime connection 上下文。
|
||||||
|
- API Key 只授予最小 scope，默认不允许成员管理。
|
||||||
|
- owner 角色不能被普通 admin 移除或降权。
|
||||||
|
|
||||||
|
## 实施优先级
|
||||||
|
|
||||||
|
### P0：基础租户骨架
|
||||||
|
|
||||||
|
- Account uuid/status。
|
||||||
|
- Workspace / Membership / Invitation。
|
||||||
|
- RequestContext。
|
||||||
|
- JWT 改为 account uuid。
|
||||||
|
- 前端 current workspace header。
|
||||||
|
|
||||||
|
### P1：资源行级隔离
|
||||||
|
|
||||||
|
- Bots、Pipelines、Models、MCP、Plugins、Knowledge、Monitoring、API Keys 全部加 workspace_uuid。
|
||||||
|
- service 查询统一加 workspace filter。
|
||||||
|
- 权限矩阵落地。
|
||||||
|
|
||||||
|
### P2：运行时隔离
|
||||||
|
|
||||||
|
- Query、Session、RuntimeBot、RuntimePipeline 加 workspace。
|
||||||
|
- Plugin Host API 和 MCP tools 按 workspace 过滤。
|
||||||
|
- SDK workspace storage 从 `default` 改为真实 workspace。
|
||||||
|
|
||||||
|
### P3：协作体验
|
||||||
|
|
||||||
|
- 邀请成员。
|
||||||
|
- 成员列表和角色管理。
|
||||||
|
- workspace switcher。
|
||||||
|
- 最近使用 workspace。
|
||||||
|
|
||||||
|
### P4：SaaS 运维增强
|
||||||
|
|
||||||
|
- Workspace 级用量统计。
|
||||||
|
- Workspace 级限额：max_bots/max_pipelines/max_extensions/tokens/storage。
|
||||||
|
- 审计日志。
|
||||||
|
- workspace suspend/delete。
|
||||||
|
- 可选自定义角色。
|
||||||
|
|
||||||
|
## 测试计划
|
||||||
|
|
||||||
|
后端测试：
|
||||||
|
|
||||||
|
- 账户可加入多个 workspace。
|
||||||
|
- 同账户在不同 workspace 权限不同。
|
||||||
|
- viewer 不能创建/修改资源。
|
||||||
|
- API Key 只能访问所属 workspace。
|
||||||
|
- 所有资源 list/get/update/delete 都不能跨 workspace。
|
||||||
|
- 默认 workspace 迁移后旧数据可用。
|
||||||
|
|
||||||
|
运行时测试：
|
||||||
|
|
||||||
|
- 两个 workspace 使用相同 `launcher_id` 不共享 session。
|
||||||
|
- 两个 workspace 使用相同 MCP server name 不共享 MCP session。
|
||||||
|
- 插件 `get_bots()` 只能看到当前 workspace bot。
|
||||||
|
- 插件 `workspace_storage` 在不同 workspace 读写隔离。
|
||||||
|
- Pipeline 只调用当前 workspace 绑定的插件和 MCP tools。
|
||||||
|
|
||||||
|
前端测试：
|
||||||
|
|
||||||
|
- 登录后自动进入最近 workspace。
|
||||||
|
- 切换 workspace 后列表数据变化。
|
||||||
|
- 无权限按钮隐藏，直接调用 API 也被后端拒绝。
|
||||||
|
- 邀请成员流程完整。
|
||||||
|
|
||||||
|
迁移测试：
|
||||||
|
|
||||||
|
- SQLite 老实例迁移。
|
||||||
|
- PostgreSQL 老实例迁移。
|
||||||
|
- 已有 local account 迁移为 default workspace owner。
|
||||||
|
- 已有 Space account token 和 Space model provider API key 不丢失。
|
||||||
|
|
||||||
|
## 关键实现注意事项
|
||||||
|
|
||||||
|
- 不建议在第一版做数据库 schema-per-tenant。LangBot 当前 ORM 和运行时均以单库单表为主，先做 shared schema + workspace_uuid 成本更低。
|
||||||
|
- 不建议每个 workspace 立即启动独立 plugin runtime。先共享 runtime，强制 action 带 workspace；大客户隔离可作为后续部署形态。
|
||||||
|
- 不要只在前端过滤 workspace。插件、API Key、MCP、RAG 都能绕过前端，必须在后端和运行时层过滤。
|
||||||
|
- `metadata` 要拆清楚：wizard status 属于 workspace，系统版本/迁移信息属于 system。
|
||||||
|
- `users.user` 用 email 当主键语义不稳，应尽快引入 `account_uuid` 并让 JWT 以 uuid 为准。
|
||||||
|
- `plugin_settings` 当前主键没有 workspace，改造时要先改主键/唯一约束，否则同插件无法在多个 workspace 配不同配置。
|
||||||
|
|
||||||
|
## 建议落地顺序
|
||||||
|
|
||||||
|
1. 新增 workspace/account/membership 表和 RequestContext。
|
||||||
|
2. 迁移旧数据到 default workspace。
|
||||||
|
3. 改 auth 和前端请求头，让每个请求都有 current workspace。
|
||||||
|
4. 从最核心资源开始逐个加 scope：bot -> pipeline -> provider/model -> plugin/MCP -> knowledge -> monitoring。
|
||||||
|
5. 改 SDK Query/Event 和 runtime storage。
|
||||||
|
6. 上成员管理 UI 和邀请。
|
||||||
|
7. 做越权测试和迁移测试。
|
||||||
|
|
||||||
|
这个顺序的好处是可以较早让主 UI 在一个 workspace 下继续工作，同时把最危险的跨租户泄露面逐步收紧。
|
||||||
+2
-1
@@ -70,7 +70,7 @@ dependencies = [
|
|||||||
"chromadb>=1.0.0,<2.0.0",
|
"chromadb>=1.0.0,<2.0.0",
|
||||||
"qdrant-client (>=1.15.1,<2.0.0)",
|
"qdrant-client (>=1.15.1,<2.0.0)",
|
||||||
"pyseekdb==1.1.0.post3",
|
"pyseekdb==1.1.0.post3",
|
||||||
"langbot-plugin==0.4.12",
|
"langbot-plugin==0.4.13",
|
||||||
"asyncpg>=0.30.0",
|
"asyncpg>=0.30.0",
|
||||||
"line-bot-sdk>=3.19.0",
|
"line-bot-sdk>=3.19.0",
|
||||||
"matrix-nio>=0.25.2",
|
"matrix-nio>=0.25.2",
|
||||||
@@ -80,6 +80,7 @@ dependencies = [
|
|||||||
"pgvector>=0.4.1",
|
"pgvector>=0.4.1",
|
||||||
"botocore>=1.42.39",
|
"botocore>=1.42.39",
|
||||||
"litellm>=1.0.0",
|
"litellm>=1.0.0",
|
||||||
|
"valkey-glide>=2.4.1,<3.0.0",
|
||||||
]
|
]
|
||||||
keywords = [
|
keywords = [
|
||||||
"bot",
|
"bot",
|
||||||
|
|||||||
@@ -57,6 +57,23 @@ class MCPSessionErrorPhase(enum.Enum):
|
|||||||
BOX_UNAVAILABLE = 'box_unavailable'
|
BOX_UNAVAILABLE = 'box_unavailable'
|
||||||
|
|
||||||
|
|
||||||
|
def _get_default_memory_mb(ap) -> int:
|
||||||
|
"""Read box.default_memory_mb from instance config (env: BOX__DEFAULT_MEMORY_MB).
|
||||||
|
|
||||||
|
Falls back to 1536 MB — a safe floor for Node.js V8 + WASM under nsjail.
|
||||||
|
Operators running memory-constrained hosts can lower this; those with large
|
||||||
|
machines can raise it. Individual MCP servers can still override via their
|
||||||
|
own box.memory_mb setting.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
data = getattr(getattr(ap, 'instance_config', None), 'data', None)
|
||||||
|
if isinstance(data, dict):
|
||||||
|
return int(data.get('box', {}).get('default_memory_mb', 1536))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
pass
|
||||||
|
return 1536
|
||||||
|
|
||||||
|
|
||||||
class MCPServerBoxConfig(pydantic.BaseModel):
|
class MCPServerBoxConfig(pydantic.BaseModel):
|
||||||
"""Structured configuration for running an MCP server inside a Box container."""
|
"""Structured configuration for running an MCP server inside a Box container."""
|
||||||
|
|
||||||
@@ -146,7 +163,13 @@ class BoxStdioSessionRuntime:
|
|||||||
# load WebAssembly modules (llhttp) on startup; the default 512 MB
|
# load WebAssembly modules (llhttp) on startup; the default 512 MB
|
||||||
# cgroup_mem_max is too small and causes OOM kills (return_code=137).
|
# cgroup_mem_max is too small and causes OOM kills (return_code=137).
|
||||||
# Auto-bump to 1024 MB when the runner is npx/bunx/pnpm dlx.
|
# Auto-bump to 1024 MB when the runner is npx/bunx/pnpm dlx.
|
||||||
memory_mb=self.config.memory_mb or 1024,
|
# Per-server override wins; global default comes from
|
||||||
|
# config.yaml box.default_memory_mb (env: BOX__DEFAULT_MEMORY_MB).
|
||||||
|
# Hard floor of 1536 MB: enough for Node.js V8 + WASM without OOM.
|
||||||
|
# Per-server override wins; global default from config.yaml
|
||||||
|
# box.default_memory_mb (env: BOX__DEFAULT_MEMORY_MB), hard floor
|
||||||
|
# of 1536 MB so Node.js V8 + WASM never OOM under nsjail.
|
||||||
|
memory_mb=(self.config.memory_mb or _get_default_memory_mb(self.ap)),
|
||||||
pids_limit=self.config.pids_limit,
|
pids_limit=self.config.pids_limit,
|
||||||
persistent=True,
|
persistent=True,
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -33,6 +33,12 @@ class VectorDBManager:
|
|||||||
self.vector_db = SeekDBVectorDatabase(self.ap)
|
self.vector_db = SeekDBVectorDatabase(self.ap)
|
||||||
self.ap.logger.info('Initialized SeekDB vector database backend.')
|
self.ap.logger.info('Initialized SeekDB vector database backend.')
|
||||||
|
|
||||||
|
elif vdb_type == 'valkey_search':
|
||||||
|
from .vdbs.valkey_search import ValkeySearchVectorDatabase
|
||||||
|
|
||||||
|
self.vector_db = ValkeySearchVectorDatabase(self.ap)
|
||||||
|
self.ap.logger.info('Initialized Valkey Search vector database backend.')
|
||||||
|
|
||||||
elif vdb_type == 'milvus':
|
elif vdb_type == 'milvus':
|
||||||
from .vdbs.milvus import MilvusVectorDatabase
|
from .vdbs.milvus import MilvusVectorDatabase
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,828 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
import struct
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from langbot.pkg.core import app
|
||||||
|
from langbot.pkg.vector.vdb import VectorDatabase, SearchType
|
||||||
|
from langbot.pkg.vector.filter_utils import normalize_filter, strip_unsupported_fields
|
||||||
|
|
||||||
|
try:
|
||||||
|
from glide import (
|
||||||
|
Batch,
|
||||||
|
GlideClient,
|
||||||
|
GlideClientConfiguration,
|
||||||
|
NodeAddress,
|
||||||
|
RequestError,
|
||||||
|
ServerCredentials,
|
||||||
|
ft,
|
||||||
|
VectorField,
|
||||||
|
VectorFieldAttributesHnsw,
|
||||||
|
VectorFieldAttributesFlat,
|
||||||
|
VectorAlgorithm,
|
||||||
|
VectorType,
|
||||||
|
DistanceMetricType,
|
||||||
|
TagField,
|
||||||
|
TextField,
|
||||||
|
FtCreateOptions,
|
||||||
|
DataType,
|
||||||
|
FtSearchOptions,
|
||||||
|
FtSearchLimit,
|
||||||
|
ReturnField,
|
||||||
|
)
|
||||||
|
|
||||||
|
VALKEY_SEARCH_AVAILABLE = True
|
||||||
|
except ImportError:
|
||||||
|
VALKEY_SEARCH_AVAILABLE = False
|
||||||
|
|
||||||
|
# Default per-request timeout (ms) for the glide client. The glide library
|
||||||
|
# default is 250ms, which is too low for vector KNN (``FT.SEARCH ... =>[KNN]``)
|
||||||
|
# under moderate load or with large indexes and yields spurious TimeoutErrors.
|
||||||
|
# Overridable via the ``vdb.valkey_search.request_timeout`` config option.
|
||||||
|
_DEFAULT_REQUEST_TIMEOUT_MS = 5000
|
||||||
|
|
||||||
|
# Safety cap on the number of SCAN rounds when purging a collection's keys, so
|
||||||
|
# a cursor-handling bug or pathological keyspace can never spin forever.
|
||||||
|
_MAX_SCAN_ROUNDS = 100000
|
||||||
|
|
||||||
|
|
||||||
|
# Mandatory client name for production observability (CLIENT LIST / dashboards).
|
||||||
|
VALKEY_CLIENT_NAME = 'langbot_vector_client'
|
||||||
|
|
||||||
|
# Fixed, indexed metadata schema. LangBot's RAG layer stores ``file_id`` on
|
||||||
|
# every chunk; it is the only metadata field we promote to a first-class
|
||||||
|
# (filterable) index field. All other metadata is preserved verbatim inside
|
||||||
|
# the ``metadata_json`` field so it survives a round-trip, but is NOT
|
||||||
|
# filterable (the established Milvus / pgvector pragmatism).
|
||||||
|
_INDEXED_TAG_FIELDS = {'file_id'}
|
||||||
|
_SUPPORTED_FILTER_FIELDS = set(_INDEXED_TAG_FIELDS)
|
||||||
|
|
||||||
|
# Hash field names used for stored documents.
|
||||||
|
_FIELD_VECTOR = 'vector'
|
||||||
|
_FIELD_DOCUMENT = 'document'
|
||||||
|
_FIELD_FILE_ID = 'file_id'
|
||||||
|
_FIELD_METADATA = 'metadata_json'
|
||||||
|
_VEC_SCORE_ALIAS = '__vec_score'
|
||||||
|
|
||||||
|
# Valkey Search has no bare "match everything" token for non-vector queries
|
||||||
|
# (a standalone ``*`` is a syntax error). A negated match on a sentinel tag
|
||||||
|
# value that can never exist matches every key, which is the canonical
|
||||||
|
# match-all idiom for FT.SEARCH.
|
||||||
|
_MATCH_ALL = '-@file_id:{__langbot_match_all_sentinel__}'
|
||||||
|
|
||||||
|
# Page size used when enumerating matching keys for deletion. Deletes
|
||||||
|
# paginate through the full result set in batches of this size so that
|
||||||
|
# files/filters matching more than one page of chunks are fully removed
|
||||||
|
# (no silent truncation / orphaned vectors).
|
||||||
|
_DELETE_SCAN_BATCH = 10000
|
||||||
|
|
||||||
|
# Characters Valkey Search's TAG query parser cannot handle even when
|
||||||
|
# backslash-escaped (the brace delimiters and the wildcard). file_id TAG
|
||||||
|
# values are percent-encoded over this set (plus '%' itself, so the encoding
|
||||||
|
# is reversible/unambiguous) before being stored or queried, so an arbitrary
|
||||||
|
# file_id round-trips instead of producing an unparseable query. For normal
|
||||||
|
# UUID/hash file_ids none of these characters occur, so the encoding is a
|
||||||
|
# no-op and the stored value is unchanged. The original file_id is always
|
||||||
|
# preserved verbatim inside ``metadata_json``.
|
||||||
|
_FT_UNSAFE_TAG_CHARS = frozenset('{}*%')
|
||||||
|
|
||||||
|
|
||||||
|
class ValkeySearchVectorDatabase(VectorDatabase):
|
||||||
|
"""Valkey Search (valkey-bundle) vector database adapter for LangBot.
|
||||||
|
|
||||||
|
Backed by the Valkey Search module shipped in ``valkey/valkey-bundle``,
|
||||||
|
accessed through the official ``valkey-glide`` client's native ``ft``
|
||||||
|
(search) command namespace. Documents are stored as Valkey HASH keys
|
||||||
|
under a per-collection prefix and indexed by one ``FT.CREATE`` index per
|
||||||
|
collection.
|
||||||
|
|
||||||
|
Supported search types: ``VECTOR``, ``FULL_TEXT`` and ``HYBRID``.
|
||||||
|
|
||||||
|
Hybrid search semantics (IMPORTANT)
|
||||||
|
-----------------------------------
|
||||||
|
Valkey Search hybrid queries follow a *filter-then-KNN* model: the text /
|
||||||
|
metadata filter pre-selects candidate keys and the KNN stage ranks them by
|
||||||
|
vector distance. This backend does **NOT** implement application-side
|
||||||
|
weighted score fusion. The ``vector_weight`` argument is therefore
|
||||||
|
accepted for interface compatibility but is **not honored** — passing
|
||||||
|
different weights does not change result ordering. A one-time warning is
|
||||||
|
emitted the first time a non-default weight is supplied. App-side score
|
||||||
|
fusion can be layered on later if weighted hybrid ranking is required.
|
||||||
|
"""
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def supported_search_types(cls) -> list[SearchType]:
|
||||||
|
return [SearchType.VECTOR, SearchType.FULL_TEXT, SearchType.HYBRID]
|
||||||
|
|
||||||
|
def __init__(self, ap: app.Application):
|
||||||
|
if not VALKEY_SEARCH_AVAILABLE:
|
||||||
|
raise ImportError(
|
||||||
|
"valkey-glide is not installed. Install it with: pip install 'valkey-glide>=2.4.1,<3.0.0'"
|
||||||
|
)
|
||||||
|
|
||||||
|
self.ap = ap
|
||||||
|
config = self.ap.instance_config.data['vdb']['valkey_search']
|
||||||
|
|
||||||
|
self._host = config.get('host', 'localhost')
|
||||||
|
self._port = int(config.get('port', 6379))
|
||||||
|
self._db = int(config.get('db', 0))
|
||||||
|
# Auth / TLS are optional (toB / SaaS). Never logged.
|
||||||
|
self._password = config.get('password', '') or None
|
||||||
|
self._username = config.get('username', '') or None
|
||||||
|
self._tls = bool(config.get('tls', False))
|
||||||
|
self._request_timeout = int(config.get('request_timeout', _DEFAULT_REQUEST_TIMEOUT_MS))
|
||||||
|
|
||||||
|
algorithm = str(config.get('index_algorithm', 'HNSW')).upper()
|
||||||
|
self._algorithm = VectorAlgorithm.FLAT if algorithm == 'FLAT' else VectorAlgorithm.HNSW
|
||||||
|
|
||||||
|
metric = str(config.get('distance_metric', 'COSINE')).upper()
|
||||||
|
self._distance_metric = {
|
||||||
|
'COSINE': DistanceMetricType.COSINE,
|
||||||
|
'L2': DistanceMetricType.L2,
|
||||||
|
'IP': DistanceMetricType.IP,
|
||||||
|
}.get(metric, DistanceMetricType.COSINE)
|
||||||
|
|
||||||
|
# Lazily-created client (created on first use so a down Valkey does not
|
||||||
|
# block LangBot boot).
|
||||||
|
self._client: GlideClient | None = None
|
||||||
|
# Serializes lazy client creation so concurrent first-use callers do not
|
||||||
|
# each construct (and leak) a separate GlideClient.
|
||||||
|
self._client_lock = asyncio.Lock()
|
||||||
|
# Index names we have already ensured this process lifetime.
|
||||||
|
self._ensured_indexes: set[str] = set()
|
||||||
|
# Whether we have already warned about the non-honored vector_weight.
|
||||||
|
self._vector_weight_warned = False
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# Client lifecycle
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
async def _ensure_client(self) -> GlideClient:
|
||||||
|
"""Create the glide client on first use (lazy, non-blocking boot)."""
|
||||||
|
if self._client is not None:
|
||||||
|
return self._client
|
||||||
|
# Double-checked locking: serialize creation so two concurrent
|
||||||
|
# first-use callers don't both build a client and leak one.
|
||||||
|
async with self._client_lock:
|
||||||
|
if self._client is not None:
|
||||||
|
return self._client
|
||||||
|
|
||||||
|
credentials = None
|
||||||
|
if self._password is not None:
|
||||||
|
# username is optional alongside a password (ACL "user" vs default user).
|
||||||
|
credentials = ServerCredentials(password=self._password, username=self._username)
|
||||||
|
elif self._username is not None:
|
||||||
|
# A username without a password is not a valid credential pair, and silently
|
||||||
|
# connecting unauthenticated to a potentially shared Valkey instance is a
|
||||||
|
# security footgun (e.g. an env var that failed to resolve). Fail closed.
|
||||||
|
raise ValueError(
|
||||||
|
'Valkey Search: a username was configured without a password. '
|
||||||
|
'Set both username and password to use ACL authentication, or remove both.'
|
||||||
|
)
|
||||||
|
|
||||||
|
conf = GlideClientConfiguration(
|
||||||
|
addresses=[NodeAddress(self._host, self._port)],
|
||||||
|
client_name=VALKEY_CLIENT_NAME,
|
||||||
|
database_id=self._db,
|
||||||
|
use_tls=self._tls,
|
||||||
|
lazy_connect=True,
|
||||||
|
credentials=credentials,
|
||||||
|
request_timeout=self._request_timeout,
|
||||||
|
)
|
||||||
|
self._client = await GlideClient.create(conf)
|
||||||
|
self.ap.logger.info(
|
||||||
|
f'Initialized Valkey Search client to {self._host}:{self._port} (db={self._db}, tls={self._tls})'
|
||||||
|
)
|
||||||
|
return self._client
|
||||||
|
|
||||||
|
async def close(self) -> None:
|
||||||
|
"""Close the glide client and reset state.
|
||||||
|
|
||||||
|
Safe to call when no client was created. After ``close`` the next
|
||||||
|
operation transparently re-creates the client (``_ensure_client``
|
||||||
|
guards on ``self._client is None``).
|
||||||
|
"""
|
||||||
|
if self._client is not None:
|
||||||
|
try:
|
||||||
|
await self._client.close()
|
||||||
|
except Exception:
|
||||||
|
self.ap.logger.warning('Valkey Search: error while closing client (ignored)')
|
||||||
|
finally:
|
||||||
|
self._client = None
|
||||||
|
self._ensured_indexes.clear()
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# Naming helpers
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
@staticmethod
|
||||||
|
def _index_name(collection: str) -> str:
|
||||||
|
return f'idx:{collection}'
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _key_prefix(collection: str) -> str:
|
||||||
|
return f'kb:{collection}:'
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _pack_vector(vec: list[float]) -> bytes:
|
||||||
|
"""Pack a float vector into little-endian float32 bytes.
|
||||||
|
|
||||||
|
Valkey Search stores and queries vectors as FLOAT32 little-endian
|
||||||
|
blobs (per the search query-language spec).
|
||||||
|
"""
|
||||||
|
return struct.pack(f'<{len(vec)}f', *[float(x) for x in vec])
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _escape_tag(value: str) -> str:
|
||||||
|
"""Escape characters that are special inside a TAG ``{...}`` clause.
|
||||||
|
|
||||||
|
The backslash is escaped first so it cannot consume a following
|
||||||
|
escape. This neutralises injection-style values (quotes, parens,
|
||||||
|
``|``, ``@``, ``:``, spaces, dashes) so a crafted ``file_id`` cannot
|
||||||
|
break out of the clause.
|
||||||
|
|
||||||
|
Note: Valkey Search's TAG query parser cannot handle a literal brace
|
||||||
|
(``{`` / ``}``) or ``*`` even when backslash-escaped. Callers that pass
|
||||||
|
a ``file_id`` route it through ``_encode_and_escape_tag`` /
|
||||||
|
``_encode_file_id`` first, which percent-encodes exactly those
|
||||||
|
characters, so an arbitrary ``file_id`` round-trips safely. This raw
|
||||||
|
escaper is still correct for all other special characters.
|
||||||
|
"""
|
||||||
|
out = []
|
||||||
|
for ch in str(value):
|
||||||
|
if ch in '\\,.<>{}[]"\':;!@#$%^&*()-+=~| ':
|
||||||
|
out.append('\\')
|
||||||
|
out.append(ch)
|
||||||
|
return ''.join(out)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _encode_file_id(value: str) -> str:
|
||||||
|
"""Make a ``file_id`` safe to use as an FT TAG token AND query value.
|
||||||
|
|
||||||
|
Percent-encodes the characters Valkey Search's TAG parser cannot handle
|
||||||
|
even when backslash-escaped (``{``, ``}``, ``*``) plus ``%`` itself for
|
||||||
|
reversibility. Applied identically at write time (the stored TAG field)
|
||||||
|
and query time (filters / ``delete_by_file_id``) so any value matches
|
||||||
|
itself. For normal UUID/hash ids none of these characters occur, so
|
||||||
|
this is a no-op. The original value is always kept verbatim in
|
||||||
|
``metadata_json``; this encoded form is only ever used for the indexed
|
||||||
|
TAG.
|
||||||
|
"""
|
||||||
|
out = []
|
||||||
|
for ch in str(value):
|
||||||
|
if ch in _FT_UNSAFE_TAG_CHARS:
|
||||||
|
out.append('%{:02X}'.format(ord(ch)))
|
||||||
|
else:
|
||||||
|
out.append(ch)
|
||||||
|
return ''.join(out)
|
||||||
|
|
||||||
|
def _encode_and_escape_tag(self, value: str) -> str:
|
||||||
|
"""Encode an FT-unsafe ``file_id`` then escape TAG special chars."""
|
||||||
|
return self._escape_tag(self._encode_file_id(value))
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# Filter mapping (canonical triples -> FT query fragment)
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
def _triples_to_ft(self, filter: dict[str, Any] | None) -> str:
|
||||||
|
"""Translate a canonical filter dict into an FT filter expression.
|
||||||
|
|
||||||
|
Only indexed fields (``file_id``) are filterable; unsupported fields
|
||||||
|
are dropped with a warning (matching the Milvus / pgvector pattern).
|
||||||
|
Returns an empty string when there is no usable filter.
|
||||||
|
"""
|
||||||
|
triples = normalize_filter(filter)
|
||||||
|
if not triples:
|
||||||
|
return ''
|
||||||
|
triples = strip_unsupported_fields(triples, _SUPPORTED_FILTER_FIELDS)
|
||||||
|
|
||||||
|
fragments: list[str] = []
|
||||||
|
for field, op, value in triples:
|
||||||
|
# All currently-indexed fields are TAG fields; file_id values are
|
||||||
|
# encoded (FT-unsafe chars) then escaped so any value round-trips.
|
||||||
|
if op == '$eq':
|
||||||
|
fragments.append(f'@{field}:{{{self._encode_and_escape_tag(value)}}}')
|
||||||
|
elif op == '$ne':
|
||||||
|
fragments.append(f'-@{field}:{{{self._encode_and_escape_tag(value)}}}')
|
||||||
|
elif op == '$in':
|
||||||
|
joined = '|'.join(self._encode_and_escape_tag(v) for v in value)
|
||||||
|
fragments.append(f'@{field}:{{{joined}}}')
|
||||||
|
elif op == '$nin':
|
||||||
|
joined = '|'.join(self._encode_and_escape_tag(v) for v in value)
|
||||||
|
fragments.append(f'-@{field}:{{{joined}}}')
|
||||||
|
elif op == '$gt':
|
||||||
|
fragments.append(f'@{field}:[({float(value)} +inf]')
|
||||||
|
elif op == '$gte':
|
||||||
|
fragments.append(f'@{field}:[{float(value)} +inf]')
|
||||||
|
elif op == '$lt':
|
||||||
|
fragments.append(f'@{field}:[-inf ({float(value)}]')
|
||||||
|
elif op == '$lte':
|
||||||
|
fragments.append(f'@{field}:[-inf {float(value)}]')
|
||||||
|
else:
|
||||||
|
# normalize_filter() already rejects unknown operators, so this
|
||||||
|
# only triggers if SUPPORTED_OPS grows without this chain being
|
||||||
|
# updated. Fail closed (rather than silently dropping the
|
||||||
|
# condition, which would widen delete_by_filter's match set).
|
||||||
|
raise ValueError(f'Valkey Search: unhandled filter operator {op!r} on field {field!r}')
|
||||||
|
|
||||||
|
return ' '.join(fragments)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _build_text_clause(text: str) -> str:
|
||||||
|
"""Build a field-scoped full-text clause for the ``document`` field.
|
||||||
|
|
||||||
|
Each whitespace-delimited word becomes a ``@document:<term>`` term and
|
||||||
|
the terms are AND-ed (space separated). FT special characters in each
|
||||||
|
term are escaped. Returns an empty string when *text* has no words.
|
||||||
|
"""
|
||||||
|
words = [w for w in str(text).split() if w]
|
||||||
|
if not words:
|
||||||
|
return ''
|
||||||
|
terms = [f'@{_FIELD_DOCUMENT}:{ValkeySearchVectorDatabase._escape_text(w)}' for w in words]
|
||||||
|
return ' '.join(terms)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _escape_text(text: str) -> str:
|
||||||
|
"""Escape FT full-text special characters in a single term."""
|
||||||
|
out = []
|
||||||
|
for ch in str(text):
|
||||||
|
if ch in '@!{}[]()|-"~*:\\':
|
||||||
|
out.append('\\')
|
||||||
|
out.append(ch)
|
||||||
|
return ''.join(out)
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# Index management
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
async def _ensure_index(self, client: GlideClient, collection: str, dim: int) -> None:
|
||||||
|
index = self._index_name(collection)
|
||||||
|
if index in self._ensured_indexes:
|
||||||
|
return
|
||||||
|
|
||||||
|
# ft.info is O(1) and raises RequestError when the index is absent —
|
||||||
|
# cheaper than ft.list (O(n) over all indexes) and it closes the
|
||||||
|
# check-then-create TOCTOU window.
|
||||||
|
try:
|
||||||
|
await ft.info(client, index)
|
||||||
|
self._ensured_indexes.add(index)
|
||||||
|
return
|
||||||
|
except RequestError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
if self._algorithm == VectorAlgorithm.FLAT:
|
||||||
|
vector_attrs = VectorFieldAttributesFlat(
|
||||||
|
dimensions=dim,
|
||||||
|
distance_metric=self._distance_metric,
|
||||||
|
type=VectorType.FLOAT32,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
vector_attrs = VectorFieldAttributesHnsw(
|
||||||
|
dimensions=dim,
|
||||||
|
distance_metric=self._distance_metric,
|
||||||
|
type=VectorType.FLOAT32,
|
||||||
|
)
|
||||||
|
|
||||||
|
schema = [
|
||||||
|
VectorField(name=_FIELD_VECTOR, algorithm=self._algorithm, attributes=vector_attrs),
|
||||||
|
TagField(name=_FIELD_FILE_ID),
|
||||||
|
TextField(name=_FIELD_DOCUMENT),
|
||||||
|
]
|
||||||
|
options = FtCreateOptions(data_type=DataType.HASH, prefixes=[self._key_prefix(collection)])
|
||||||
|
await ft.create(client, index, schema, options)
|
||||||
|
self._ensured_indexes.add(index)
|
||||||
|
self.ap.logger.info(
|
||||||
|
f"Valkey Search index '{index}' created (dim={dim}, algo={self._algorithm.value}, "
|
||||||
|
f'metric={self._distance_metric.value})'
|
||||||
|
)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _decode(value: Any) -> str:
|
||||||
|
if isinstance(value, (bytes, bytearray, memoryview)):
|
||||||
|
return bytes(value).decode('utf-8', errors='replace')
|
||||||
|
return str(value)
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# VectorDatabase ABC implementation
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
async def get_or_create_collection(self, collection: str):
|
||||||
|
"""Ensure a client exists.
|
||||||
|
|
||||||
|
The index itself requires the vector dimension, which is only known at
|
||||||
|
first ``add_embeddings`` (same constraint as Qdrant / SeekDB), so this
|
||||||
|
is a best-effort no-op when the index does not yet exist.
|
||||||
|
"""
|
||||||
|
await self._ensure_client()
|
||||||
|
|
||||||
|
async def add_embeddings(
|
||||||
|
self,
|
||||||
|
collection: str,
|
||||||
|
ids: list[str],
|
||||||
|
embeddings_list: list[list[float]],
|
||||||
|
metadatas: list[dict[str, Any]],
|
||||||
|
documents: list[str] | None = None,
|
||||||
|
) -> None:
|
||||||
|
if not embeddings_list:
|
||||||
|
return
|
||||||
|
|
||||||
|
client = await self._ensure_client()
|
||||||
|
dim = len(embeddings_list[0])
|
||||||
|
# The index schema is fixed to the first embedding's dimension. A later
|
||||||
|
# embedding of a different length would be packed into a wrong-sized
|
||||||
|
# blob that Valkey stores silently but that yields garbage KNN
|
||||||
|
# distances, so reject mixed dimensions up-front.
|
||||||
|
if any(len(e) != dim for e in embeddings_list[1:]):
|
||||||
|
raise ValueError(f'All embeddings must have dimension {dim}; got mixed lengths')
|
||||||
|
await self._ensure_index(client, collection, dim)
|
||||||
|
|
||||||
|
prefix = self._key_prefix(collection)
|
||||||
|
|
||||||
|
batch = Batch(is_atomic=False)
|
||||||
|
for i, _id in enumerate(ids):
|
||||||
|
key = prefix + str(_id)
|
||||||
|
metadata = metadatas[i] if i < len(metadatas) else {}
|
||||||
|
mapping: dict[str, Any] = {
|
||||||
|
_FIELD_VECTOR: self._pack_vector(embeddings_list[i]),
|
||||||
|
_FIELD_METADATA: json.dumps(metadata, ensure_ascii=False),
|
||||||
|
}
|
||||||
|
file_id = metadata.get('file_id')
|
||||||
|
if file_id is not None:
|
||||||
|
mapping[_FIELD_FILE_ID] = self._encode_file_id(str(file_id))
|
||||||
|
if documents is not None and i < len(documents) and documents[i] is not None:
|
||||||
|
mapping[_FIELD_DOCUMENT] = documents[i]
|
||||||
|
|
||||||
|
batch.hset(key, mapping)
|
||||||
|
|
||||||
|
# Pipeline all HSETs into a single round-trip (non-atomic) instead of
|
||||||
|
# one await per embedding, which is N sequential round-trips for N
|
||||||
|
# chunks.
|
||||||
|
await client.exec(batch, raise_on_error=True)
|
||||||
|
|
||||||
|
self.ap.logger.info(f"Added {len(ids)} embeddings to Valkey Search collection '{collection}'")
|
||||||
|
|
||||||
|
async def search(
|
||||||
|
self,
|
||||||
|
collection: str,
|
||||||
|
query_embedding: list[float],
|
||||||
|
k: int = 5,
|
||||||
|
search_type: str = 'vector',
|
||||||
|
query_text: str = '',
|
||||||
|
filter: dict[str, Any] | None = None,
|
||||||
|
vector_weight: float | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
client = await self._ensure_client()
|
||||||
|
index = self._index_name(collection)
|
||||||
|
|
||||||
|
if not await self._index_exists(client, index):
|
||||||
|
return {'ids': [[]], 'metadatas': [[]], 'distances': [[]]}
|
||||||
|
|
||||||
|
# vector_weight is accepted for interface parity but NOT honored by this
|
||||||
|
# backend (filter-then-KNN, no weighted fusion). Warn once.
|
||||||
|
if vector_weight is not None and not self._vector_weight_warned:
|
||||||
|
self.ap.logger.warning(
|
||||||
|
'Valkey Search backend does not honor vector_weight: hybrid search uses '
|
||||||
|
'filter-then-KNN without weighted score fusion. The vector_weight value '
|
||||||
|
'is ignored. See docs/VALKEY_SEARCH_INTEGRATION.md.'
|
||||||
|
)
|
||||||
|
self._vector_weight_warned = True
|
||||||
|
|
||||||
|
filter_expr = self._triples_to_ft(filter)
|
||||||
|
|
||||||
|
if search_type == SearchType.FULL_TEXT:
|
||||||
|
if not query_text:
|
||||||
|
return {'ids': [[]], 'metadatas': [[]], 'distances': [[]]}
|
||||||
|
text_clause = self._build_text_clause(query_text)
|
||||||
|
if not text_clause:
|
||||||
|
return {'ids': [[]], 'metadatas': [[]], 'distances': [[]]}
|
||||||
|
query = f'{filter_expr} {text_clause}'.strip() if filter_expr else text_clause
|
||||||
|
return await self._run_text_search(client, index, query, k)
|
||||||
|
|
||||||
|
if search_type == SearchType.HYBRID:
|
||||||
|
# Filter / text pre-selects candidates; KNN ranks. No fusion.
|
||||||
|
pre = filter_expr
|
||||||
|
if query_text:
|
||||||
|
text_clause = self._build_text_clause(query_text)
|
||||||
|
if text_clause:
|
||||||
|
pre = f'{pre} {text_clause}'.strip() if pre else text_clause
|
||||||
|
pre = pre or '*'
|
||||||
|
query = f'{self._wrap_pre(pre)}=>[KNN {k} @{_FIELD_VECTOR} $BLOB AS {_VEC_SCORE_ALIAS}]'
|
||||||
|
return await self._run_knn_search(client, index, query, query_embedding, k)
|
||||||
|
|
||||||
|
# Default: pure VECTOR search.
|
||||||
|
pre = filter_expr or '*'
|
||||||
|
query = f'{self._wrap_pre(pre)}=>[KNN {k} @{_FIELD_VECTOR} $BLOB AS {_VEC_SCORE_ALIAS}]'
|
||||||
|
return await self._run_knn_search(client, index, query, query_embedding, k)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _wrap_pre(pre: str) -> str:
|
||||||
|
"""Parenthesize a multi-condition pre-filter before the ``=>`` KNN clause.
|
||||||
|
|
||||||
|
When ``pre`` combines several terms (e.g. ``@file_id:{x} @document:term``)
|
||||||
|
the Valkey Search parser can otherwise mis-associate only the last term
|
||||||
|
with the KNN clause. Wrapping the whole expression forces correct
|
||||||
|
grouping. A bare ``*`` (match-all) and single-term expressions are left
|
||||||
|
untouched.
|
||||||
|
"""
|
||||||
|
if pre and pre != '*' and ' ' in pre.strip():
|
||||||
|
return f'({pre})'
|
||||||
|
return pre
|
||||||
|
|
||||||
|
async def _run_knn_search(
|
||||||
|
self,
|
||||||
|
client: GlideClient,
|
||||||
|
index: str,
|
||||||
|
query: str,
|
||||||
|
query_embedding: list[float],
|
||||||
|
k: int,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
options = FtSearchOptions(
|
||||||
|
params={'BLOB': self._pack_vector(list(query_embedding))},
|
||||||
|
return_fields=[
|
||||||
|
ReturnField(field_identifier=_VEC_SCORE_ALIAS, alias='distance'),
|
||||||
|
ReturnField(field_identifier=_FIELD_DOCUMENT),
|
||||||
|
ReturnField(field_identifier=_FIELD_METADATA),
|
||||||
|
],
|
||||||
|
limit=FtSearchLimit(0, k),
|
||||||
|
dialect=2,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
reply = await ft.search(client, index, query, options)
|
||||||
|
except Exception as exc:
|
||||||
|
if self._is_missing_index_error(exc):
|
||||||
|
return {'ids': [[]], 'metadatas': [[]], 'distances': [[]]}
|
||||||
|
raise
|
||||||
|
return self._reply_to_chroma(index, reply, has_distance=True)
|
||||||
|
|
||||||
|
async def _run_text_search(
|
||||||
|
self,
|
||||||
|
client: GlideClient,
|
||||||
|
index: str,
|
||||||
|
query: str,
|
||||||
|
k: int,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
options = FtSearchOptions(
|
||||||
|
return_fields=[
|
||||||
|
ReturnField(field_identifier=_FIELD_DOCUMENT),
|
||||||
|
ReturnField(field_identifier=_FIELD_METADATA),
|
||||||
|
],
|
||||||
|
limit=FtSearchLimit(0, k),
|
||||||
|
dialect=2,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
reply = await ft.search(client, index, query, options)
|
||||||
|
except Exception as exc:
|
||||||
|
if self._is_missing_index_error(exc):
|
||||||
|
return {'ids': [[]], 'metadatas': [[]], 'distances': [[]]}
|
||||||
|
raise
|
||||||
|
return self._reply_to_chroma(index, reply, has_distance=False)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _is_missing_index_error(exc: Exception) -> bool:
|
||||||
|
"""Return True if *exc* indicates the FT index does not exist.
|
||||||
|
|
||||||
|
``FT.DROPINDEX`` is applied eventually, so an index can briefly still
|
||||||
|
appear in ``FT._LIST`` after being dropped; a follow-up search then
|
||||||
|
fails with a "not found" error which we treat as an empty result.
|
||||||
|
"""
|
||||||
|
message = str(exc).lower()
|
||||||
|
return 'not found' in message and 'index' in message
|
||||||
|
|
||||||
|
def _iter_reply_docs(self, reply: Any, prefix: str):
|
||||||
|
"""Yield ``(doc_id, decoded_fields)`` pairs from an FT.SEARCH reply.
|
||||||
|
|
||||||
|
glide returns ``[total, {key: {field: value}, ...}]``. This shared
|
||||||
|
iterator decodes each key, strips the per-collection prefix to recover
|
||||||
|
the original document id, and decodes the field map — the logic both
|
||||||
|
``_reply_to_chroma`` and ``list_by_filter`` need.
|
||||||
|
"""
|
||||||
|
docs = reply[1] if reply and len(reply) >= 2 and isinstance(reply[1], dict) else {}
|
||||||
|
for key, fields in docs.items():
|
||||||
|
key_str = self._decode(key)
|
||||||
|
doc_id = key_str[len(prefix) :] if prefix and key_str.startswith(prefix) else key_str
|
||||||
|
decoded_fields = {self._decode(fk): fv for fk, fv in fields.items()} if isinstance(fields, dict) else {}
|
||||||
|
yield doc_id, decoded_fields
|
||||||
|
|
||||||
|
def _reply_to_chroma(self, index: str, reply: Any, has_distance: bool) -> dict[str, Any]:
|
||||||
|
"""Convert an FT.SEARCH reply into Chroma-style nested lists.
|
||||||
|
|
||||||
|
The KNN score field (aliased ``distance``) is a COSINE/L2 distance
|
||||||
|
directly, so no inversion is needed (unlike Qdrant).
|
||||||
|
"""
|
||||||
|
ids: list[str] = []
|
||||||
|
distances: list[float] = []
|
||||||
|
metadatas: list[dict[str, Any]] = []
|
||||||
|
|
||||||
|
if not reply or len(reply) < 2:
|
||||||
|
return {'ids': [ids], 'metadatas': [metadatas], 'distances': [distances]}
|
||||||
|
|
||||||
|
prefix = self._key_prefix(index[len('idx:') :]) if index.startswith('idx:') else ''
|
||||||
|
|
||||||
|
for doc_id, decoded_fields in self._iter_reply_docs(reply, prefix):
|
||||||
|
ids.append(doc_id)
|
||||||
|
|
||||||
|
if has_distance and 'distance' in decoded_fields:
|
||||||
|
try:
|
||||||
|
distances.append(float(self._decode(decoded_fields['distance'])))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
distances.append(0.0)
|
||||||
|
else:
|
||||||
|
distances.append(0.0)
|
||||||
|
|
||||||
|
metadata: dict[str, Any] = {}
|
||||||
|
raw_meta = decoded_fields.get(_FIELD_METADATA)
|
||||||
|
if raw_meta is not None:
|
||||||
|
try:
|
||||||
|
metadata = json.loads(self._decode(raw_meta))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
metadata = {}
|
||||||
|
metadatas.append(metadata)
|
||||||
|
|
||||||
|
return {'ids': [ids], 'metadatas': [metadatas], 'distances': [distances]}
|
||||||
|
|
||||||
|
async def delete_by_file_id(self, collection: str, file_id: str) -> None:
|
||||||
|
client = await self._ensure_client()
|
||||||
|
index = self._index_name(collection)
|
||||||
|
if not await self._index_exists(client, index):
|
||||||
|
self.ap.logger.warning(f"Valkey Search collection '{collection}' not found for deletion")
|
||||||
|
return
|
||||||
|
|
||||||
|
query = f'@{_FIELD_FILE_ID}:{{{self._encode_and_escape_tag(file_id)}}}'
|
||||||
|
keys = await self._search_keys(client, index, query)
|
||||||
|
if keys:
|
||||||
|
await client.delete(keys)
|
||||||
|
self.ap.logger.info(
|
||||||
|
f"Deleted {len(keys)} embeddings from Valkey Search collection '{collection}' with file_id: {file_id}"
|
||||||
|
)
|
||||||
|
|
||||||
|
async def delete_by_filter(self, collection: str, filter: dict[str, Any]) -> int:
|
||||||
|
client = await self._ensure_client()
|
||||||
|
index = self._index_name(collection)
|
||||||
|
if not await self._index_exists(client, index):
|
||||||
|
self.ap.logger.warning(f"Valkey Search collection '{collection}' not found for deletion")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
# Guard against accidental mass deletion: a non-empty filter that maps
|
||||||
|
# to no usable (indexed) conditions must NOT fall back to match-all and
|
||||||
|
# wipe the whole collection. Skip instead (matching Milvus / pgvector).
|
||||||
|
query = self._triples_to_ft(filter)
|
||||||
|
if not query:
|
||||||
|
self.ap.logger.warning(
|
||||||
|
"Valkey Search delete_by_filter on '%s': filter produced no usable conditions, skipping",
|
||||||
|
collection,
|
||||||
|
)
|
||||||
|
return 0
|
||||||
|
keys = await self._search_keys(client, index, query)
|
||||||
|
if keys:
|
||||||
|
await client.delete(keys)
|
||||||
|
self.ap.logger.info(f"Deleted {len(keys)} embeddings from Valkey Search collection '{collection}' by filter")
|
||||||
|
return len(keys)
|
||||||
|
|
||||||
|
async def list_by_filter(
|
||||||
|
self,
|
||||||
|
collection: str,
|
||||||
|
filter: dict[str, Any] | None = None,
|
||||||
|
limit: int = 20,
|
||||||
|
offset: int = 0,
|
||||||
|
) -> tuple[list[dict[str, Any]], int]:
|
||||||
|
client = await self._ensure_client()
|
||||||
|
index = self._index_name(collection)
|
||||||
|
if not await self._index_exists(client, index):
|
||||||
|
return [], 0
|
||||||
|
|
||||||
|
query = self._triples_to_ft(filter) or _MATCH_ALL
|
||||||
|
options = FtSearchOptions(
|
||||||
|
return_fields=[
|
||||||
|
ReturnField(field_identifier=_FIELD_DOCUMENT),
|
||||||
|
ReturnField(field_identifier=_FIELD_METADATA),
|
||||||
|
],
|
||||||
|
limit=FtSearchLimit(offset, limit),
|
||||||
|
dialect=2,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
reply = await ft.search(client, index, query, options)
|
||||||
|
except Exception as exc:
|
||||||
|
if self._is_missing_index_error(exc):
|
||||||
|
return [], 0
|
||||||
|
raise
|
||||||
|
|
||||||
|
total = 0
|
||||||
|
if reply:
|
||||||
|
try:
|
||||||
|
total = int(reply[0])
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
total = 0
|
||||||
|
|
||||||
|
prefix = self._key_prefix(collection)
|
||||||
|
items: list[dict[str, Any]] = []
|
||||||
|
for doc_id, decoded_fields in self._iter_reply_docs(reply, prefix):
|
||||||
|
document = decoded_fields.get(_FIELD_DOCUMENT)
|
||||||
|
metadata: dict[str, Any] = {}
|
||||||
|
raw_meta = decoded_fields.get(_FIELD_METADATA)
|
||||||
|
if raw_meta is not None:
|
||||||
|
try:
|
||||||
|
metadata = json.loads(self._decode(raw_meta))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
metadata = {}
|
||||||
|
|
||||||
|
items.append(
|
||||||
|
{
|
||||||
|
'id': doc_id,
|
||||||
|
'document': self._decode(document) if document is not None else None,
|
||||||
|
'metadata': metadata,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
return items, total
|
||||||
|
|
||||||
|
async def delete_collection(self, collection: str):
|
||||||
|
client = await self._ensure_client()
|
||||||
|
index = self._index_name(collection)
|
||||||
|
self._ensured_indexes.discard(index)
|
||||||
|
|
||||||
|
if await self._index_exists(client, index):
|
||||||
|
try:
|
||||||
|
await ft.dropindex(client, index)
|
||||||
|
except RequestError:
|
||||||
|
# The index was already dropped (e.g. by a concurrent process)
|
||||||
|
# between the existence check and this call — benign. Other
|
||||||
|
# errors (connection / auth) must propagate so the caller knows
|
||||||
|
# the operation failed rather than silently SCAN-deleting next.
|
||||||
|
pass
|
||||||
|
|
||||||
|
# DROPINDEX does not remove the underlying hashes; delete them too.
|
||||||
|
prefix = self._key_prefix(collection)
|
||||||
|
cursor = b'0'
|
||||||
|
deleted = 0
|
||||||
|
for _ in range(_MAX_SCAN_ROUNDS):
|
||||||
|
cursor, keys = await client.scan(cursor, match=f'{prefix}*', count=500)
|
||||||
|
if keys:
|
||||||
|
await client.delete(keys)
|
||||||
|
deleted += len(keys)
|
||||||
|
if cursor in (b'0', '0', 0):
|
||||||
|
break
|
||||||
|
self.ap.logger.info(f"Valkey Search collection '{collection}' deleted ({deleted} keys removed)")
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# Internal search helpers
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
async def _index_exists(self, client: GlideClient, index: str) -> bool:
|
||||||
|
if index in self._ensured_indexes:
|
||||||
|
return True
|
||||||
|
# ft.info is O(1) and raises RequestError when the index does not
|
||||||
|
# exist, vs ft.list which is O(n) over every index on the server and
|
||||||
|
# was being paid on the first query to each collection.
|
||||||
|
try:
|
||||||
|
await ft.info(client, index)
|
||||||
|
self._ensured_indexes.add(index)
|
||||||
|
return True
|
||||||
|
except RequestError:
|
||||||
|
return False
|
||||||
|
|
||||||
|
async def _search_keys(self, client: GlideClient, index: str, query: str) -> list[str]:
|
||||||
|
"""Return all matching document keys for a query (NOCONTENT).
|
||||||
|
|
||||||
|
Paginates through the full result set in pages of ``_DELETE_SCAN_BATCH``
|
||||||
|
so that queries matching more than one page of chunks are fully
|
||||||
|
enumerated (avoids silently truncating deletes and leaving orphaned
|
||||||
|
vectors).
|
||||||
|
"""
|
||||||
|
keys: list[str] = []
|
||||||
|
offset = 0
|
||||||
|
while True:
|
||||||
|
options = FtSearchOptions(
|
||||||
|
nocontent=True,
|
||||||
|
limit=FtSearchLimit(offset, _DELETE_SCAN_BATCH),
|
||||||
|
dialect=2,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
reply = await ft.search(client, index, query, options)
|
||||||
|
except Exception as exc:
|
||||||
|
if self._is_missing_index_error(exc):
|
||||||
|
return keys
|
||||||
|
raise
|
||||||
|
|
||||||
|
if not reply or len(reply) < 2:
|
||||||
|
break
|
||||||
|
|
||||||
|
# reply[0] is the total match count; reply[1] holds this page.
|
||||||
|
total = 0
|
||||||
|
try:
|
||||||
|
total = int(reply[0])
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
total = 0
|
||||||
|
|
||||||
|
docs = reply[1]
|
||||||
|
if isinstance(docs, dict):
|
||||||
|
page = [self._decode(k) for k in docs.keys()]
|
||||||
|
elif isinstance(docs, (list, tuple)):
|
||||||
|
page = [self._decode(k) for k in docs]
|
||||||
|
else:
|
||||||
|
page = []
|
||||||
|
|
||||||
|
if not page:
|
||||||
|
break
|
||||||
|
keys.extend(page)
|
||||||
|
|
||||||
|
offset += len(page)
|
||||||
|
if offset >= total or len(page) < _DELETE_SCAN_BATCH:
|
||||||
|
break
|
||||||
|
|
||||||
|
return keys
|
||||||
@@ -87,6 +87,16 @@ vdb:
|
|||||||
database: 'langbot'
|
database: 'langbot'
|
||||||
user: 'postgres'
|
user: 'postgres'
|
||||||
password: 'postgres'
|
password: 'postgres'
|
||||||
|
valkey_search:
|
||||||
|
host: 'localhost'
|
||||||
|
port: 6379 # integration tests use 6380 -> valkey/valkey-bundle:9.1.0
|
||||||
|
db: 0
|
||||||
|
password: '' # optional (toB auth)
|
||||||
|
username: '' # optional (ACL user, toB)
|
||||||
|
tls: false # optional (toB/SaaS)
|
||||||
|
index_algorithm: 'HNSW' # HNSW | FLAT
|
||||||
|
distance_metric: 'COSINE' # COSINE | L2 | IP
|
||||||
|
request_timeout: 5000 # per-request timeout in ms (glide default 250ms is too low for KNN)
|
||||||
storage:
|
storage:
|
||||||
use: local
|
use: local
|
||||||
cleanup:
|
cleanup:
|
||||||
@@ -143,6 +153,15 @@ box:
|
|||||||
- './data/box'
|
- './data/box'
|
||||||
- '/tmp'
|
- '/tmp'
|
||||||
workspace_quota_mb: null # Optional disk quota override (>= 0). null = profile default.
|
workspace_quota_mb: null # Optional disk quota override (>= 0). null = profile default.
|
||||||
|
# Default nsjail cgroup memory limit for each MCP stdio server process, in MB.
|
||||||
|
# Node.js MCP servers (npx/bunx) need more memory than Python ones because V8
|
||||||
|
# and WebAssembly modules (e.g. undici llhttp) reserve large virtual address
|
||||||
|
# space at startup. Setting this too low causes processes to be killed with
|
||||||
|
# return_code=137 (OOM kill); the symptom is "Box managed process exited
|
||||||
|
# unexpectedly" in the logs. Raise on machines with ample RAM; lower only if
|
||||||
|
# you run exclusively Python (uvx) MCP servers.
|
||||||
|
# Can also be set via BOX__DEFAULT_MEMORY_MB. Default: 1536.
|
||||||
|
default_memory_mb: 1536
|
||||||
docker:
|
docker:
|
||||||
cpu_limit_enabled: true # When false, Docker sandbox containers are started without --cpus. Memory and PID limits still apply.
|
cpu_limit_enabled: true # When false, Docker sandbox containers are started without --cpus. Memory and PID limits still apply.
|
||||||
e2b:
|
e2b:
|
||||||
|
|||||||
@@ -104,6 +104,17 @@ def create_minimal_config(tmpdir: Path, port: int = 15300) -> Path:
|
|||||||
'user': 'postgres',
|
'user': 'postgres',
|
||||||
'password': 'postgres',
|
'password': 'postgres',
|
||||||
},
|
},
|
||||||
|
'valkey_search': {
|
||||||
|
'host': 'localhost',
|
||||||
|
'port': 6379,
|
||||||
|
'db': 0,
|
||||||
|
'password': '',
|
||||||
|
'username': '',
|
||||||
|
'tls': False,
|
||||||
|
'index_algorithm': 'HNSW',
|
||||||
|
'distance_metric': 'COSINE',
|
||||||
|
'request_timeout': 5000,
|
||||||
|
},
|
||||||
},
|
},
|
||||||
'storage': {
|
'storage': {
|
||||||
'use': 'local',
|
'use': 'local',
|
||||||
|
|||||||
@@ -0,0 +1,343 @@
|
|||||||
|
"""Integration tests for the Valkey Search VDB backend.
|
||||||
|
|
||||||
|
These are SLOW, real-server tests. They are gated on ``TEST_VALKEY_URL`` and
|
||||||
|
skipped when it is unset (same precedent as the PostgreSQL migration tests).
|
||||||
|
|
||||||
|
Run locally against valkey/valkey-bundle:9.1.0::
|
||||||
|
|
||||||
|
podman run -d --name valkey-test-langbot -p 6380:6379 valkey/valkey-bundle:9.1.0
|
||||||
|
TEST_VALKEY_URL=valkey://localhost:6380 \\
|
||||||
|
uv run pytest tests/integration/vector/test_valkey_search.py -m slow -q
|
||||||
|
|
||||||
|
The default upstream fast CI lane (``-m "not slow"``) skips these; the local
|
||||||
|
supervisor validator MUST run them.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import os
|
||||||
|
import uuid
|
||||||
|
from types import SimpleNamespace
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
pytestmark = [pytest.mark.integration, pytest.mark.slow]
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_valkey_url(url: str) -> tuple[str, int, int]:
|
||||||
|
"""Parse ``valkey://host:port/db`` into ``(host, port, db)``."""
|
||||||
|
parsed = urlparse(url)
|
||||||
|
host = parsed.hostname or 'localhost'
|
||||||
|
port = parsed.port or 6379
|
||||||
|
db = 0
|
||||||
|
if parsed.path and parsed.path.strip('/'):
|
||||||
|
try:
|
||||||
|
db = int(parsed.path.strip('/'))
|
||||||
|
except ValueError:
|
||||||
|
db = 0
|
||||||
|
return host, port, db
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def valkey_config():
|
||||||
|
url = os.environ.get('TEST_VALKEY_URL')
|
||||||
|
if not url:
|
||||||
|
pytest.skip('TEST_VALKEY_URL not set')
|
||||||
|
host, port, db = _parse_valkey_url(url)
|
||||||
|
return {
|
||||||
|
'host': host,
|
||||||
|
'port': port,
|
||||||
|
'db': db,
|
||||||
|
'password': '',
|
||||||
|
'username': '',
|
||||||
|
'tls': False,
|
||||||
|
'index_algorithm': 'HNSW',
|
||||||
|
'distance_metric': 'COSINE',
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _make_ap(valkey_config):
|
||||||
|
"""Build a minimal fake ``ap`` with the config + a no-op logger."""
|
||||||
|
logger = SimpleNamespace(
|
||||||
|
info=lambda *a, **k: None,
|
||||||
|
warning=lambda *a, **k: None,
|
||||||
|
error=lambda *a, **k: None,
|
||||||
|
debug=lambda *a, **k: None,
|
||||||
|
)
|
||||||
|
instance_config = SimpleNamespace(data={'vdb': {'valkey_search': valkey_config}})
|
||||||
|
return SimpleNamespace(instance_config=instance_config, logger=logger)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
async def backend(valkey_config):
|
||||||
|
"""Create a Valkey Search backend, skip if module/server unavailable."""
|
||||||
|
from langbot.pkg.vector.vdbs.valkey_search import (
|
||||||
|
ValkeySearchVectorDatabase,
|
||||||
|
VALKEY_SEARCH_AVAILABLE,
|
||||||
|
)
|
||||||
|
from glide import ft
|
||||||
|
|
||||||
|
if not VALKEY_SEARCH_AVAILABLE:
|
||||||
|
pytest.skip('valkey-glide not installed')
|
||||||
|
|
||||||
|
ap = _make_ap(valkey_config)
|
||||||
|
db = ValkeySearchVectorDatabase(ap)
|
||||||
|
client = await db._ensure_client()
|
||||||
|
|
||||||
|
# Module-presence gate: FT.LIST must be available (Search module loaded).
|
||||||
|
try:
|
||||||
|
await ft.list(client)
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
await client.close()
|
||||||
|
pytest.skip(f'Valkey Search module not available: {exc}')
|
||||||
|
|
||||||
|
collection = f'test_{uuid.uuid4().hex[:12]}'
|
||||||
|
yield db, collection
|
||||||
|
|
||||||
|
# Cleanup
|
||||||
|
try:
|
||||||
|
await db.delete_collection(collection)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
if db._client is not None:
|
||||||
|
await db._client.close()
|
||||||
|
|
||||||
|
|
||||||
|
async def _poll_until(coro_factory, predicate, timeout=5.0, interval=0.2):
|
||||||
|
"""Poll an async result until predicate is true (indexer is async)."""
|
||||||
|
deadline = asyncio.get_event_loop().time() + timeout
|
||||||
|
result = await coro_factory()
|
||||||
|
while not predicate(result) and asyncio.get_event_loop().time() < deadline:
|
||||||
|
await asyncio.sleep(interval)
|
||||||
|
result = await coro_factory()
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _sample_docs():
|
||||||
|
ids = ['d1', 'd2', 'd3']
|
||||||
|
embeddings = [
|
||||||
|
[1.0, 0.0, 0.0, 0.0],
|
||||||
|
[0.0, 1.0, 0.0, 0.0],
|
||||||
|
[0.9, 0.1, 0.0, 0.0],
|
||||||
|
]
|
||||||
|
metadatas = [
|
||||||
|
{'file_id': 'fileA', 'topic': 'cats'},
|
||||||
|
{'file_id': 'fileB', 'topic': 'dogs'},
|
||||||
|
{'file_id': 'fileA', 'topic': 'cats'},
|
||||||
|
]
|
||||||
|
documents = [
|
||||||
|
'the quick brown fox',
|
||||||
|
'lazy dogs sleeping',
|
||||||
|
'foxes and cats playing',
|
||||||
|
]
|
||||||
|
return ids, embeddings, metadatas, documents
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_add_and_vector_search(backend):
|
||||||
|
db, collection = backend
|
||||||
|
ids, embeddings, metadatas, documents = _sample_docs()
|
||||||
|
await db.add_embeddings(collection, ids, embeddings, metadatas, documents)
|
||||||
|
|
||||||
|
result = await _poll_until(
|
||||||
|
lambda: db.search(collection, [1.0, 0.0, 0.0, 0.0], k=3, search_type='vector'),
|
||||||
|
lambda r: len(r['ids'][0]) >= 1,
|
||||||
|
)
|
||||||
|
assert len(result['ids'][0]) >= 1
|
||||||
|
# Closest to [1,0,0,0] should be d1.
|
||||||
|
assert result['ids'][0][0] == 'd1'
|
||||||
|
assert all(isinstance(d, float) for d in result['distances'][0])
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_full_text_search(backend):
|
||||||
|
db, collection = backend
|
||||||
|
ids, embeddings, metadatas, documents = _sample_docs()
|
||||||
|
await db.add_embeddings(collection, ids, embeddings, metadatas, documents)
|
||||||
|
|
||||||
|
result = await _poll_until(
|
||||||
|
lambda: db.search(collection, [0.0, 0.0, 0.0, 0.0], k=5, search_type='full_text', query_text='dogs'),
|
||||||
|
lambda r: len(r['ids'][0]) >= 1,
|
||||||
|
)
|
||||||
|
assert 'd2' in result['ids'][0]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_hybrid_filter_then_knn(backend):
|
||||||
|
db, collection = backend
|
||||||
|
ids, embeddings, metadatas, documents = _sample_docs()
|
||||||
|
await db.add_embeddings(collection, ids, embeddings, metadatas, documents)
|
||||||
|
|
||||||
|
result = await _poll_until(
|
||||||
|
lambda: db.search(
|
||||||
|
collection,
|
||||||
|
[1.0, 0.0, 0.0, 0.0],
|
||||||
|
k=5,
|
||||||
|
search_type='hybrid',
|
||||||
|
query_text='cats',
|
||||||
|
filter={'file_id': 'fileA'},
|
||||||
|
),
|
||||||
|
lambda r: len(r['ids'][0]) >= 1,
|
||||||
|
)
|
||||||
|
# Only fileA docs (d1, d3) should be candidates.
|
||||||
|
assert set(result['ids'][0]).issubset({'d1', 'd3'})
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_vector_weight_not_honored(backend):
|
||||||
|
"""Passing different vector_weight values must NOT change ranking."""
|
||||||
|
db, collection = backend
|
||||||
|
ids, embeddings, metadatas, documents = _sample_docs()
|
||||||
|
await db.add_embeddings(collection, ids, embeddings, metadatas, documents)
|
||||||
|
|
||||||
|
common = dict(
|
||||||
|
collection=collection, query_embedding=[1.0, 0.0, 0.0, 0.0], k=3, search_type='hybrid', query_text='cats'
|
||||||
|
)
|
||||||
|
await _poll_until(lambda: db.search(**common), lambda r: len(r['ids'][0]) >= 1)
|
||||||
|
|
||||||
|
r_low = await db.search(**common, vector_weight=0.1)
|
||||||
|
r_high = await db.search(**common, vector_weight=0.9)
|
||||||
|
assert r_low['ids'][0] == r_high['ids'][0]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_filter_operators(backend):
|
||||||
|
db, collection = backend
|
||||||
|
ids, embeddings, metadatas, documents = _sample_docs()
|
||||||
|
await db.add_embeddings(collection, ids, embeddings, metadatas, documents)
|
||||||
|
|
||||||
|
# Wait for indexing.
|
||||||
|
await _poll_until(
|
||||||
|
lambda: db.list_by_filter(collection, limit=10),
|
||||||
|
lambda r: r[1] >= 3,
|
||||||
|
)
|
||||||
|
|
||||||
|
# $eq
|
||||||
|
items, total = await db.list_by_filter(collection, filter={'file_id': 'fileA'})
|
||||||
|
assert total == 2
|
||||||
|
assert {it['id'] for it in items} == {'d1', 'd3'}
|
||||||
|
|
||||||
|
# $ne
|
||||||
|
items, total = await db.list_by_filter(collection, filter={'file_id': {'$ne': 'fileA'}})
|
||||||
|
assert {it['id'] for it in items} == {'d2'}
|
||||||
|
|
||||||
|
# $in
|
||||||
|
items, total = await db.list_by_filter(collection, filter={'file_id': {'$in': ['fileA', 'fileB']}})
|
||||||
|
assert total == 3
|
||||||
|
|
||||||
|
# $nin
|
||||||
|
items, total = await db.list_by_filter(collection, filter={'file_id': {'$nin': ['fileB']}})
|
||||||
|
assert {it['id'] for it in items} == {'d1', 'd3'}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_delete_by_file_id(backend):
|
||||||
|
db, collection = backend
|
||||||
|
ids, embeddings, metadatas, documents = _sample_docs()
|
||||||
|
await db.add_embeddings(collection, ids, embeddings, metadatas, documents)
|
||||||
|
await _poll_until(lambda: db.list_by_filter(collection, limit=10), lambda r: r[1] >= 3)
|
||||||
|
|
||||||
|
await db.delete_by_file_id(collection, 'fileA')
|
||||||
|
items, total = await _poll_until(
|
||||||
|
lambda: db.list_by_filter(collection, limit=10),
|
||||||
|
lambda r: r[1] <= 1,
|
||||||
|
)
|
||||||
|
assert {it['id'] for it in items} == {'d2'}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_delete_by_filter_returns_count(backend):
|
||||||
|
db, collection = backend
|
||||||
|
ids, embeddings, metadatas, documents = _sample_docs()
|
||||||
|
await db.add_embeddings(collection, ids, embeddings, metadatas, documents)
|
||||||
|
await _poll_until(lambda: db.list_by_filter(collection, limit=10), lambda r: r[1] >= 3)
|
||||||
|
|
||||||
|
deleted = await db.delete_by_filter(collection, filter={'file_id': 'fileA'})
|
||||||
|
assert deleted == 2
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_list_by_filter_pagination(backend):
|
||||||
|
db, collection = backend
|
||||||
|
ids, embeddings, metadatas, documents = _sample_docs()
|
||||||
|
await db.add_embeddings(collection, ids, embeddings, metadatas, documents)
|
||||||
|
await _poll_until(lambda: db.list_by_filter(collection, limit=10), lambda r: r[1] >= 3)
|
||||||
|
|
||||||
|
page1, total = await db.list_by_filter(collection, limit=2, offset=0)
|
||||||
|
assert total == 3
|
||||||
|
assert len(page1) == 2
|
||||||
|
|
||||||
|
page2, total = await db.list_by_filter(collection, limit=2, offset=2)
|
||||||
|
assert total == 3
|
||||||
|
assert len(page2) == 1
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_delete_collection(backend):
|
||||||
|
db, collection = backend
|
||||||
|
ids, embeddings, metadatas, documents = _sample_docs()
|
||||||
|
await db.add_embeddings(collection, ids, embeddings, metadatas, documents)
|
||||||
|
await _poll_until(lambda: db.list_by_filter(collection, limit=10), lambda r: r[1] >= 3)
|
||||||
|
|
||||||
|
await db.delete_collection(collection)
|
||||||
|
|
||||||
|
# After dropping, search on a missing index returns empty.
|
||||||
|
result = await db.search(collection, [1.0, 0.0, 0.0, 0.0], k=3, search_type='vector')
|
||||||
|
assert result['ids'][0] == []
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_adversarial_filter_and_query_input(backend):
|
||||||
|
"""Crafted FT special chars in file_id / query_text must not break out.
|
||||||
|
|
||||||
|
Guarantees locked in here:
|
||||||
|
* A file_id full of injection-style chars (quotes, parens, ``|``, ``@``,
|
||||||
|
``:``, spaces, dashes) only ever matches its own row — the payload is
|
||||||
|
escaped to literal TAG content, never interpreted as extra clauses.
|
||||||
|
* A query_text full of FT operators does not raise and does not widen the
|
||||||
|
result set.
|
||||||
|
* A file_id containing FT-unsafe chars (``{`` / ``}`` / ``*``) is
|
||||||
|
percent-encoded, so it round-trips correctly: an exact match returns ONLY
|
||||||
|
its own row and never widens to an unrelated row, and the query does not
|
||||||
|
raise.
|
||||||
|
"""
|
||||||
|
db, collection = backend
|
||||||
|
|
||||||
|
# Injection-style file_id WITHOUT FT-unsafe chars (the realistic surface).
|
||||||
|
injection_fid = 'evil") @file_id (".id|x-y:z'
|
||||||
|
# file_id WITH FT-unsafe chars that previously could not be queried.
|
||||||
|
brace_fid = 'x} @file_id:{*'
|
||||||
|
ids = ['adv1', 'benign2', 'brace3']
|
||||||
|
embeddings = [[1.0, 0.0, 0.0, 0.0], [0.0, 1.0, 0.0, 0.0], [0.0, 0.0, 1.0, 0.0]]
|
||||||
|
metadatas = [{'file_id': injection_fid}, {'file_id': 'plainB'}, {'file_id': brace_fid}]
|
||||||
|
documents = ['payload row content', 'unrelated benign content', 'brace row content']
|
||||||
|
await db.add_embeddings(collection, ids, embeddings, metadatas, documents)
|
||||||
|
await _poll_until(lambda: db.list_by_filter(collection, limit=10), lambda r: r[1] >= 3)
|
||||||
|
|
||||||
|
# Exact-match on the crafted file_id returns ONLY its own row.
|
||||||
|
items, total = await db.list_by_filter(collection, filter={'file_id': injection_fid})
|
||||||
|
assert total == 1
|
||||||
|
assert {it['id'] for it in items} == {'adv1'}
|
||||||
|
|
||||||
|
# A query_text packed with FT operators must not raise and must not match
|
||||||
|
# the benign row (escaped to literal terms, none of which it contains).
|
||||||
|
result = await db.search(
|
||||||
|
collection,
|
||||||
|
[0.0, 0.0, 0.0, 0.0],
|
||||||
|
k=5,
|
||||||
|
search_type='full_text',
|
||||||
|
query_text='@document:{*} | -()~ "evil"',
|
||||||
|
)
|
||||||
|
assert 'benign2' not in result['ids'][0]
|
||||||
|
|
||||||
|
# The brace/star-bearing file_id is encoded, so it round-trips: exact match
|
||||||
|
# returns ONLY its own row and never widens. No RequestError is raised.
|
||||||
|
b_items, b_total = await db.list_by_filter(collection, filter={'file_id': brace_fid})
|
||||||
|
assert b_total == 1
|
||||||
|
assert {it['id'] for it in b_items} == {'brace3'}
|
||||||
|
|
||||||
|
# And deletion by that file_id removes exactly its own row.
|
||||||
|
deleted = await db.delete_by_filter(collection, filter={'file_id': brace_fid})
|
||||||
|
assert deleted == 1
|
||||||
@@ -27,7 +27,7 @@
|
|||||||
|
|
||||||
### 4. 向量数据库 (`vector/vdbs/`)
|
### 4. 向量数据库 (`vector/vdbs/`)
|
||||||
- **路径**: `src/langbot/pkg/vector/vdbs/`
|
- **路径**: `src/langbot/pkg/vector/vdbs/`
|
||||||
- **模块**: chroma, milvus, pgvector, qdrant, seekdb
|
- **模块**: chroma, milvus, pgvector, qdrant, seekdb, valkey_search
|
||||||
- **排除原因**: 需要真实向量数据库实例运行
|
- **排除原因**: 需要真实向量数据库实例运行
|
||||||
- **测试方式**: 需要 Docker 启动测试数据库或 mock
|
- **测试方式**: 需要 Docker 启动测试数据库或 mock
|
||||||
- **状态**: 后续可补充 mock 测试
|
- **状态**: 后续可补充 mock 测试
|
||||||
|
|||||||
@@ -417,7 +417,7 @@ class TestBuildBoxSessionPayload:
|
|||||||
payload = s._build_box_session_payload('session-123')
|
payload = s._build_box_session_payload('session-123')
|
||||||
assert payload['image'] == 'node:20'
|
assert payload['image'] == 'node:20'
|
||||||
assert payload['cpus'] == 2.0
|
assert payload['cpus'] == 2.0
|
||||||
assert payload['memory_mb'] == 1024
|
assert payload["memory_mb"] == 1024
|
||||||
assert payload['pids_limit'] == 256
|
assert payload['pids_limit'] == 256
|
||||||
|
|
||||||
def test_none_fields_excluded(self, mcp_module):
|
def test_none_fields_excluded(self, mcp_module):
|
||||||
|
|||||||
@@ -33,7 +33,7 @@ class TestVectorDBManagerInitialization:
|
|||||||
mocks['langbot.pkg.core.app'] = MagicMock()
|
mocks['langbot.pkg.core.app'] = MagicMock()
|
||||||
|
|
||||||
# Mock all VDB backend implementations
|
# Mock all VDB backend implementations
|
||||||
for backend in ['chroma', 'qdrant', 'seekdb', 'milvus', 'pgvector_db']:
|
for backend in ['chroma', 'qdrant', 'seekdb', 'milvus', 'pgvector_db', 'valkey_search']:
|
||||||
mocks[f'langbot.pkg.vector.vdbs.{backend}'] = MagicMock()
|
mocks[f'langbot.pkg.vector.vdbs.{backend}'] = MagicMock()
|
||||||
|
|
||||||
return mocks
|
return mocks
|
||||||
@@ -123,6 +123,25 @@ class TestVectorDBManagerInitialization:
|
|||||||
|
|
||||||
mock_seekdb_class.assert_called_once_with(mock_app)
|
mock_seekdb_class.assert_called_once_with(mock_app)
|
||||||
|
|
||||||
|
def test_initialize_valkey_search_backend(self):
|
||||||
|
"""Valkey Search config uses ValkeySearchVectorDatabase backend."""
|
||||||
|
vdb_config = {'use': 'valkey_search'}
|
||||||
|
mock_app = self._create_mock_app(vdb_config)
|
||||||
|
|
||||||
|
mocks = self._make_vector_import_mocks()
|
||||||
|
mock_valkey_class = MagicMock()
|
||||||
|
mocks['langbot.pkg.vector.vdbs.valkey_search'].ValkeySearchVectorDatabase = mock_valkey_class
|
||||||
|
|
||||||
|
with isolated_sys_modules(mocks):
|
||||||
|
from langbot.pkg.vector.mgr import VectorDBManager
|
||||||
|
|
||||||
|
mgr = VectorDBManager(mock_app)
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
asyncio.get_event_loop().run_until_complete(mgr.initialize())
|
||||||
|
|
||||||
|
mock_valkey_class.assert_called_once_with(mock_app)
|
||||||
|
|
||||||
def test_initialize_milvus_backend_with_uri(self):
|
def test_initialize_milvus_backend_with_uri(self):
|
||||||
"""Milvus config with custom URI."""
|
"""Milvus config with custom URI."""
|
||||||
vdb_config = {
|
vdb_config = {
|
||||||
|
|||||||
@@ -0,0 +1,388 @@
|
|||||||
|
"""Unit tests for the Valkey Search VDB backend's pure helpers.
|
||||||
|
|
||||||
|
These tests exercise the filter-to-FT mapping, float32 packing, tag/text
|
||||||
|
escaping, FT.SEARCH reply parsing and the import guard. They run in the fast
|
||||||
|
CI lane and require NO running Valkey server.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import struct
|
||||||
|
from importlib import import_module
|
||||||
|
from unittest.mock import AsyncMock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
|
||||||
|
def get_valkey_module():
|
||||||
|
"""Lazy import of the valkey_search backend module."""
|
||||||
|
return import_module('langbot.pkg.vector.vdbs.valkey_search')
|
||||||
|
|
||||||
|
|
||||||
|
def make_backend():
|
||||||
|
"""Construct a backend instance without running its __init__.
|
||||||
|
|
||||||
|
The constructor needs a live ``ap`` + config; for pure-helper tests we
|
||||||
|
only need a bare instance with the attributes the helpers touch.
|
||||||
|
"""
|
||||||
|
mod = get_valkey_module()
|
||||||
|
backend = object.__new__(mod.ValkeySearchVectorDatabase)
|
||||||
|
# _ensure_client serializes creation through this lock; set it here since
|
||||||
|
# __init__ (which normally creates it) is bypassed.
|
||||||
|
backend._client_lock = asyncio.Lock()
|
||||||
|
return backend
|
||||||
|
|
||||||
|
|
||||||
|
class TestFloat32Packing:
|
||||||
|
"""Tests for _pack_vector little-endian float32 packing."""
|
||||||
|
|
||||||
|
def test_pack_round_trips(self):
|
||||||
|
mod = get_valkey_module()
|
||||||
|
vec = [0.1, -2.5, 3.0, 4.25]
|
||||||
|
packed = mod.ValkeySearchVectorDatabase._pack_vector(vec)
|
||||||
|
assert isinstance(packed, bytes)
|
||||||
|
assert len(packed) == 4 * len(vec)
|
||||||
|
unpacked = list(struct.unpack(f'<{len(vec)}f', packed))
|
||||||
|
for original, restored in zip(vec, unpacked):
|
||||||
|
assert restored == pytest.approx(original, rel=1e-6)
|
||||||
|
|
||||||
|
def test_pack_is_little_endian(self):
|
||||||
|
mod = get_valkey_module()
|
||||||
|
packed = mod.ValkeySearchVectorDatabase._pack_vector([1.0])
|
||||||
|
assert packed == struct.pack('<f', 1.0)
|
||||||
|
|
||||||
|
|
||||||
|
class TestTagEscaping:
|
||||||
|
"""Tests for _escape_tag."""
|
||||||
|
|
||||||
|
def test_escapes_special_chars(self):
|
||||||
|
mod = get_valkey_module()
|
||||||
|
escaped = mod.ValkeySearchVectorDatabase._escape_tag('a-b c.d')
|
||||||
|
assert '\\-' in escaped
|
||||||
|
assert '\\ ' in escaped
|
||||||
|
assert '\\.' in escaped
|
||||||
|
|
||||||
|
def test_plain_value_unchanged(self):
|
||||||
|
mod = get_valkey_module()
|
||||||
|
assert mod.ValkeySearchVectorDatabase._escape_tag('abc123') == 'abc123'
|
||||||
|
|
||||||
|
|
||||||
|
class TestFileIdEncoding:
|
||||||
|
"""Tests for _encode_file_id (FT-unsafe char percent-encoding)."""
|
||||||
|
|
||||||
|
def test_uuid_is_noop(self):
|
||||||
|
mod = get_valkey_module()
|
||||||
|
fid = '550e8400-e29b-41d4-a716-446655440000'
|
||||||
|
assert mod.ValkeySearchVectorDatabase._encode_file_id(fid) == fid
|
||||||
|
|
||||||
|
def test_encodes_braces_star_and_percent(self):
|
||||||
|
mod = get_valkey_module()
|
||||||
|
enc = mod.ValkeySearchVectorDatabase._encode_file_id('a{b}c*d%e')
|
||||||
|
# '{'=7B '}'=7D '*'=2A '%'=25
|
||||||
|
assert enc == 'a%7Bb%7Dc%2Ad%25e'
|
||||||
|
# No raw FT-unsafe char survives.
|
||||||
|
assert all(ch not in enc for ch in '{}*') or '%' in enc
|
||||||
|
|
||||||
|
def test_encoding_is_deterministic_and_collision_safe(self):
|
||||||
|
mod = get_valkey_module()
|
||||||
|
enc = mod.ValkeySearchVectorDatabase._encode_file_id
|
||||||
|
# A literal "%7B" must not collide with an encoded "{".
|
||||||
|
assert enc('{') != enc('%7B')
|
||||||
|
assert enc('{') == '%7B'
|
||||||
|
assert enc('%7B') == '%257B'
|
||||||
|
|
||||||
|
def test_filter_encodes_unsafe_chars_in_tag_query(self):
|
||||||
|
backend = make_backend()
|
||||||
|
# The emitted TAG query must contain the encoded form, never raw braces.
|
||||||
|
frag = backend._triples_to_ft({'file_id': 'x}y{z*'})
|
||||||
|
assert '7D' in frag and '7B' in frag and '2A' in frag
|
||||||
|
# No raw '*' from the value, and exactly one opening/closing brace (the
|
||||||
|
# TAG-clause delimiters) — the value's own braces were encoded away.
|
||||||
|
assert '*' not in frag
|
||||||
|
assert frag.count('{') == 1 and frag.count('}') == 1
|
||||||
|
assert frag.startswith('@file_id:{') and frag.endswith('}')
|
||||||
|
|
||||||
|
def test_filter_in_operator_encodes_each_value(self):
|
||||||
|
backend = make_backend()
|
||||||
|
frag = backend._triples_to_ft({'file_id': {'$in': ['a*b', 'c}d']}})
|
||||||
|
assert '2A' in frag and '7D' in frag
|
||||||
|
assert '*' not in frag
|
||||||
|
|
||||||
|
|
||||||
|
class TestFilterToFt:
|
||||||
|
"""Tests for _triples_to_ft filter mapping (all 8 operators)."""
|
||||||
|
|
||||||
|
def test_empty_filter_returns_empty_string(self):
|
||||||
|
backend = make_backend()
|
||||||
|
assert backend._triples_to_ft(None) == ''
|
||||||
|
assert backend._triples_to_ft({}) == ''
|
||||||
|
|
||||||
|
def test_eq_tag(self):
|
||||||
|
backend = make_backend()
|
||||||
|
assert backend._triples_to_ft({'file_id': 'abc'}) == '@file_id:{abc}'
|
||||||
|
|
||||||
|
def test_explicit_eq_tag(self):
|
||||||
|
backend = make_backend()
|
||||||
|
assert backend._triples_to_ft({'file_id': {'$eq': 'abc'}}) == '@file_id:{abc}'
|
||||||
|
|
||||||
|
def test_ne_tag(self):
|
||||||
|
backend = make_backend()
|
||||||
|
assert backend._triples_to_ft({'file_id': {'$ne': 'abc'}}) == '-@file_id:{abc}'
|
||||||
|
|
||||||
|
def test_in_tag(self):
|
||||||
|
backend = make_backend()
|
||||||
|
assert backend._triples_to_ft({'file_id': {'$in': ['a', 'b']}}) == '@file_id:{a|b}'
|
||||||
|
|
||||||
|
def test_nin_tag(self):
|
||||||
|
backend = make_backend()
|
||||||
|
assert backend._triples_to_ft({'file_id': {'$nin': ['a', 'b']}}) == '-@file_id:{a|b}'
|
||||||
|
|
||||||
|
def test_numeric_range_operators(self):
|
||||||
|
backend = make_backend()
|
||||||
|
# file_id is the only indexed field; numeric ops still render via the
|
||||||
|
# generic range fragment, so use file_id to keep the field supported.
|
||||||
|
# Values are cast to float (defensive against non-numeric input and a
|
||||||
|
# future NUMERIC field becoming an injection surface).
|
||||||
|
assert backend._triples_to_ft({'file_id': {'$gt': 5}}) == '@file_id:[(5.0 +inf]'
|
||||||
|
assert backend._triples_to_ft({'file_id': {'$gte': 5}}) == '@file_id:[5.0 +inf]'
|
||||||
|
assert backend._triples_to_ft({'file_id': {'$lt': 5}}) == '@file_id:[-inf (5.0]'
|
||||||
|
assert backend._triples_to_ft({'file_id': {'$lte': 5}}) == '@file_id:[-inf 5.0]'
|
||||||
|
|
||||||
|
def test_numeric_range_rejects_non_numeric(self):
|
||||||
|
backend = make_backend()
|
||||||
|
# A non-numeric range value fails closed rather than interpolating raw.
|
||||||
|
with pytest.raises((ValueError, TypeError)):
|
||||||
|
backend._triples_to_ft({'file_id': {'$gt': 'not-a-number'}})
|
||||||
|
|
||||||
|
def test_unsupported_field_dropped(self):
|
||||||
|
backend = make_backend()
|
||||||
|
# Non-indexed fields are dropped (returns empty expression).
|
||||||
|
assert backend._triples_to_ft({'some_other_field': 'x'}) == ''
|
||||||
|
|
||||||
|
def test_multiple_supported_keys_anded(self):
|
||||||
|
backend = make_backend()
|
||||||
|
# Two conditions on the same indexed field are joined with a space (AND).
|
||||||
|
result = backend._triples_to_ft({'file_id': {'$in': ['a', 'b']}})
|
||||||
|
assert result == '@file_id:{a|b}'
|
||||||
|
|
||||||
|
|
||||||
|
class TestTextEscaping:
|
||||||
|
"""Tests for _escape_text full-text escaping."""
|
||||||
|
|
||||||
|
def test_escapes_ft_special_chars(self):
|
||||||
|
mod = get_valkey_module()
|
||||||
|
escaped = mod.ValkeySearchVectorDatabase._escape_text('hello@world|test')
|
||||||
|
assert '\\@' in escaped
|
||||||
|
assert '\\|' in escaped
|
||||||
|
|
||||||
|
|
||||||
|
class TestReplyToChroma:
|
||||||
|
"""Tests for _reply_to_chroma FT.SEARCH reply parsing."""
|
||||||
|
|
||||||
|
def test_parses_knn_reply(self):
|
||||||
|
backend = make_backend()
|
||||||
|
# glide returns [total, {key: {field: value}}]
|
||||||
|
reply = [
|
||||||
|
2,
|
||||||
|
{
|
||||||
|
b'kb:col1:id1': {
|
||||||
|
b'distance': b'0.10',
|
||||||
|
b'document': b'hello',
|
||||||
|
b'metadata_json': b'{"file_id": "f1"}',
|
||||||
|
},
|
||||||
|
b'kb:col1:id2': {
|
||||||
|
b'distance': b'0.25',
|
||||||
|
b'document': b'world',
|
||||||
|
b'metadata_json': b'{"file_id": "f2"}',
|
||||||
|
},
|
||||||
|
},
|
||||||
|
]
|
||||||
|
result = backend._reply_to_chroma('idx:col1', reply, has_distance=True)
|
||||||
|
assert result['ids'][0] == ['id1', 'id2']
|
||||||
|
assert result['distances'][0] == [pytest.approx(0.10), pytest.approx(0.25)]
|
||||||
|
assert result['metadatas'][0][0] == {'file_id': 'f1'}
|
||||||
|
assert result['metadatas'][0][1] == {'file_id': 'f2'}
|
||||||
|
|
||||||
|
def test_empty_reply(self):
|
||||||
|
backend = make_backend()
|
||||||
|
result = backend._reply_to_chroma('idx:col1', [0, {}], has_distance=True)
|
||||||
|
assert result == {'ids': [[]], 'metadatas': [[]], 'distances': [[]]}
|
||||||
|
|
||||||
|
def test_malformed_reply(self):
|
||||||
|
backend = make_backend()
|
||||||
|
result = backend._reply_to_chroma('idx:col1', [], has_distance=True)
|
||||||
|
assert result == {'ids': [[]], 'metadatas': [[]], 'distances': [[]]}
|
||||||
|
|
||||||
|
def test_text_search_reply_no_distance(self):
|
||||||
|
backend = make_backend()
|
||||||
|
reply = [
|
||||||
|
1,
|
||||||
|
{
|
||||||
|
b'kb:col1:id1': {
|
||||||
|
b'document': b'hello',
|
||||||
|
b'metadata_json': b'{"file_id": "f1"}',
|
||||||
|
},
|
||||||
|
},
|
||||||
|
]
|
||||||
|
result = backend._reply_to_chroma('idx:col1', reply, has_distance=False)
|
||||||
|
assert result['ids'][0] == ['id1']
|
||||||
|
assert result['distances'][0] == [0.0]
|
||||||
|
|
||||||
|
|
||||||
|
class TestImportGuard:
|
||||||
|
"""Tests for the ImportError guard when glide is unavailable."""
|
||||||
|
|
||||||
|
def test_constructor_raises_when_unavailable(self, monkeypatch):
|
||||||
|
mod = get_valkey_module()
|
||||||
|
monkeypatch.setattr(mod, 'VALKEY_SEARCH_AVAILABLE', False)
|
||||||
|
with pytest.raises(ImportError, match='valkey-glide'):
|
||||||
|
mod.ValkeySearchVectorDatabase(ap=None)
|
||||||
|
|
||||||
|
|
||||||
|
class TestSupportedSearchTypes:
|
||||||
|
"""Tests for supported_search_types."""
|
||||||
|
|
||||||
|
def test_supports_vector_full_text_hybrid(self):
|
||||||
|
mod = get_valkey_module()
|
||||||
|
from langbot.pkg.vector.vdb import SearchType
|
||||||
|
|
||||||
|
types = mod.ValkeySearchVectorDatabase.supported_search_types()
|
||||||
|
assert SearchType.VECTOR in types
|
||||||
|
assert SearchType.FULL_TEXT in types
|
||||||
|
assert SearchType.HYBRID in types
|
||||||
|
|
||||||
|
|
||||||
|
class TestDeleteByFilterGuard:
|
||||||
|
"""Regression tests for the delete_by_filter mass-deletion guard.
|
||||||
|
|
||||||
|
A non-empty filter referencing only non-indexed fields must NOT fall back
|
||||||
|
to match-all and wipe the whole collection: it must skip and return 0.
|
||||||
|
"""
|
||||||
|
|
||||||
|
async def test_unsupported_only_filter_skips_and_returns_zero(self):
|
||||||
|
backend = make_backend()
|
||||||
|
# Make the client/index lookups succeed without a real server.
|
||||||
|
backend._client = AsyncMock()
|
||||||
|
backend.ap = type('Ap', (), {'logger': AsyncMock()})()
|
||||||
|
backend._ensure_client = AsyncMock(return_value=backend._client)
|
||||||
|
backend._index_exists = AsyncMock(return_value=True)
|
||||||
|
# _search_keys must never be reached for an unusable filter.
|
||||||
|
backend._search_keys = AsyncMock(
|
||||||
|
side_effect=AssertionError('_search_keys must not be called for an unusable filter')
|
||||||
|
)
|
||||||
|
|
||||||
|
# Filter references only a non-indexed field -> maps to no FT conditions.
|
||||||
|
deleted = await backend.delete_by_filter('col1', {'some_other_field': 'x'})
|
||||||
|
|
||||||
|
assert deleted == 0
|
||||||
|
backend._client.delete.assert_not_called()
|
||||||
|
|
||||||
|
async def test_supported_filter_deletes_matching_keys(self):
|
||||||
|
backend = make_backend()
|
||||||
|
backend._client = AsyncMock()
|
||||||
|
backend.ap = type('Ap', (), {'logger': AsyncMock()})()
|
||||||
|
backend._ensure_client = AsyncMock(return_value=backend._client)
|
||||||
|
backend._index_exists = AsyncMock(return_value=True)
|
||||||
|
backend._search_keys = AsyncMock(return_value=['kb:col1:id1', 'kb:col1:id2'])
|
||||||
|
|
||||||
|
deleted = await backend.delete_by_filter('col1', {'file_id': 'f1'})
|
||||||
|
|
||||||
|
assert deleted == 2
|
||||||
|
backend._client.delete.assert_awaited_once_with(['kb:col1:id1', 'kb:col1:id2'])
|
||||||
|
|
||||||
|
|
||||||
|
class TestClose:
|
||||||
|
"""Tests for the close() teardown."""
|
||||||
|
|
||||||
|
async def test_close_resets_client_and_indexes(self):
|
||||||
|
backend = make_backend()
|
||||||
|
client = AsyncMock()
|
||||||
|
backend._client = client
|
||||||
|
backend.ap = type('Ap', (), {'logger': AsyncMock()})()
|
||||||
|
backend._ensured_indexes = {'idx:col1'}
|
||||||
|
|
||||||
|
await backend.close()
|
||||||
|
|
||||||
|
client.close.assert_awaited_once()
|
||||||
|
assert backend._client is None
|
||||||
|
assert backend._ensured_indexes == set()
|
||||||
|
|
||||||
|
async def test_close_is_noop_when_no_client(self):
|
||||||
|
backend = make_backend()
|
||||||
|
backend._client = None
|
||||||
|
backend.ap = type('Ap', (), {'logger': AsyncMock()})()
|
||||||
|
backend._ensured_indexes = set()
|
||||||
|
# Should not raise.
|
||||||
|
await backend.close()
|
||||||
|
assert backend._client is None
|
||||||
|
|
||||||
|
|
||||||
|
class TestCredentialsBuild:
|
||||||
|
"""Tests for the auth-credential construction in _ensure_client."""
|
||||||
|
|
||||||
|
def _prep_backend(self, mod, monkeypatch, *, username, password):
|
||||||
|
backend = make_backend()
|
||||||
|
backend._client = None
|
||||||
|
backend._host = 'localhost'
|
||||||
|
backend._port = 6379
|
||||||
|
backend._db = 0
|
||||||
|
backend._tls = False
|
||||||
|
backend._username = username
|
||||||
|
backend._password = password
|
||||||
|
backend._request_timeout = 5000
|
||||||
|
backend._ensured_indexes = set()
|
||||||
|
warnings: list[str] = []
|
||||||
|
backend.ap = type(
|
||||||
|
'Ap',
|
||||||
|
(),
|
||||||
|
{
|
||||||
|
'logger': type(
|
||||||
|
'L', (), {'info': lambda self, *a, **k: None, 'warning': lambda s, m, *a, **k: warnings.append(m)}
|
||||||
|
)()
|
||||||
|
},
|
||||||
|
)()
|
||||||
|
|
||||||
|
created = {}
|
||||||
|
|
||||||
|
class _FakeClient:
|
||||||
|
@staticmethod
|
||||||
|
async def create(conf):
|
||||||
|
created['conf'] = conf
|
||||||
|
return AsyncMock()
|
||||||
|
|
||||||
|
cred_calls: list[dict] = []
|
||||||
|
|
||||||
|
def _fake_credentials(**kwargs):
|
||||||
|
cred_calls.append(kwargs)
|
||||||
|
return ('CRED', kwargs)
|
||||||
|
|
||||||
|
monkeypatch.setattr(mod, 'GlideClient', _FakeClient)
|
||||||
|
monkeypatch.setattr(mod, 'ServerCredentials', _fake_credentials)
|
||||||
|
monkeypatch.setattr(mod, 'GlideClientConfiguration', lambda **kw: kw)
|
||||||
|
monkeypatch.setattr(mod, 'NodeAddress', lambda *a, **k: ('node', a, k))
|
||||||
|
return backend, created, cred_calls, warnings
|
||||||
|
|
||||||
|
async def test_username_without_password_fails_closed(self, monkeypatch):
|
||||||
|
mod = get_valkey_module()
|
||||||
|
backend, created, cred_calls, warnings = self._prep_backend(mod, monkeypatch, username='acluser', password=None)
|
||||||
|
|
||||||
|
# A username without a password must fail closed rather than silently
|
||||||
|
# connecting unauthenticated to a (potentially shared) Valkey instance.
|
||||||
|
with pytest.raises(ValueError, match='without a password'):
|
||||||
|
await backend._ensure_client()
|
||||||
|
|
||||||
|
assert cred_calls == [] # ServerCredentials NOT constructed
|
||||||
|
assert 'conf' not in created # client never created
|
||||||
|
|
||||||
|
async def test_password_builds_credentials(self, monkeypatch):
|
||||||
|
mod = get_valkey_module()
|
||||||
|
backend, created, cred_calls, warnings = self._prep_backend(
|
||||||
|
mod, monkeypatch, username='acluser', password='secret'
|
||||||
|
)
|
||||||
|
|
||||||
|
await backend._ensure_client()
|
||||||
|
|
||||||
|
assert len(cred_calls) == 1
|
||||||
|
assert cred_calls[0] == {'password': 'secret', 'username': 'acluser'}
|
||||||
|
assert created['conf']['credentials'] == ('CRED', {'password': 'secret', 'username': 'acluser'})
|
||||||
@@ -2084,6 +2084,7 @@ dependencies = [
|
|||||||
{ name = "tiktoken" },
|
{ name = "tiktoken" },
|
||||||
{ name = "urllib3" },
|
{ name = "urllib3" },
|
||||||
{ name = "uv" },
|
{ name = "uv" },
|
||||||
|
{ name = "valkey-glide" },
|
||||||
{ name = "websockets" },
|
{ name = "websockets" },
|
||||||
]
|
]
|
||||||
|
|
||||||
@@ -2123,7 +2124,7 @@ requires-dist = [
|
|||||||
{ name = "ebooklib", specifier = ">=0.18" },
|
{ name = "ebooklib", specifier = ">=0.18" },
|
||||||
{ name = "gewechat-client", specifier = ">=0.1.5" },
|
{ name = "gewechat-client", specifier = ">=0.1.5" },
|
||||||
{ name = "html2text", specifier = ">=2024.2.26" },
|
{ name = "html2text", specifier = ">=2024.2.26" },
|
||||||
{ name = "langbot-plugin", specifier = "==0.4.12" },
|
{ name = "langbot-plugin", specifier = "==0.4.13" },
|
||||||
{ name = "langchain", specifier = ">=1.3.9" },
|
{ name = "langchain", specifier = ">=1.3.9" },
|
||||||
{ name = "langchain-core", specifier = ">=1.3.3" },
|
{ name = "langchain-core", specifier = ">=1.3.3" },
|
||||||
{ name = "langchain-text-splitters", specifier = ">=1.1.2" },
|
{ name = "langchain-text-splitters", specifier = ">=1.1.2" },
|
||||||
@@ -2172,6 +2173,7 @@ requires-dist = [
|
|||||||
{ name = "tiktoken", specifier = ">=0.9.0" },
|
{ name = "tiktoken", specifier = ">=0.9.0" },
|
||||||
{ name = "urllib3", specifier = ">=2.7.0" },
|
{ name = "urllib3", specifier = ">=2.7.0" },
|
||||||
{ name = "uv", specifier = ">=0.11.15" },
|
{ name = "uv", specifier = ">=0.11.15" },
|
||||||
|
{ name = "valkey-glide", specifier = ">=2.4.1,<3.0.0" },
|
||||||
{ name = "websockets", specifier = ">=15.0.1" },
|
{ name = "websockets", specifier = ">=15.0.1" },
|
||||||
]
|
]
|
||||||
|
|
||||||
@@ -2187,7 +2189,7 @@ dev = [
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "langbot-plugin"
|
name = "langbot-plugin"
|
||||||
version = "0.4.12"
|
version = "0.4.13"
|
||||||
source = { registry = "https://pypi.org/simple" }
|
source = { registry = "https://pypi.org/simple" }
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "aiofiles" },
|
{ name = "aiofiles" },
|
||||||
@@ -2208,9 +2210,9 @@ dependencies = [
|
|||||||
{ name = "watchdog" },
|
{ name = "watchdog" },
|
||||||
{ name = "websockets" },
|
{ name = "websockets" },
|
||||||
]
|
]
|
||||||
sdist = { url = "https://files.pythonhosted.org/packages/9c/84/78054f9caff83acb96d472c09c1345beed9241adf05afd372972f1dd8d1c/langbot_plugin-0.4.12.tar.gz", hash = "sha256:5344a2280c7d99d18379ea9e5ce224ad573bb875aa5f06ec7da0e1ec16e0200c", size = 334903, upload-time = "2026-07-04T01:27:08.609Z" }
|
sdist = { url = "https://files.pythonhosted.org/packages/40/a6/1eaf77c3b81e9de3390c504c5f627dc41f43bff6df9aff0e1e31d796b6f0/langbot_plugin-0.4.13.tar.gz", hash = "sha256:f936340e67679c21f1e7e7f1447339f31a0a2c965db060ecfbd9d0c51bb0d6fe", size = 334887, upload-time = "2026-07-04T05:38:59.942Z" }
|
||||||
wheels = [
|
wheels = [
|
||||||
{ url = "https://files.pythonhosted.org/packages/4f/1e/bc020fb1b5ec656b7b742ead68a88db5396055af88302d105b0420e2657f/langbot_plugin-0.4.12-py3-none-any.whl", hash = "sha256:68606de0c305c823e7e2a399a07b1c0116f90fae664627b7059761ca318d69c0", size = 221886, upload-time = "2026-07-04T01:27:07.352Z" },
|
{ url = "https://files.pythonhosted.org/packages/e9/bf/fc9671a7afbd933440c38403c84d918c1022fdeed16e22a6ab3b2aec83ff/langbot_plugin-0.4.13-py3-none-any.whl", hash = "sha256:9d45ebc7a7ee0413d6db9baa009fcbf0ad07e2e1753a6f0a27f37b8b665cd1ee", size = 221884, upload-time = "2026-07-04T05:38:58.525Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
@@ -5984,6 +5986,39 @@ wheels = [
|
|||||||
{ url = "https://files.pythonhosted.org/packages/e4/16/c1fd27e9549f3c4baf1dc9c20c456cd2f822dbf8de9f463824b0c0357e06/uvloop-0.22.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:6cde23eeda1a25c75b2e07d39970f3374105d5eafbaab2a4482be82f272d5a5e", size = 4296730, upload-time = "2025-10-16T22:17:00.744Z" },
|
{ url = "https://files.pythonhosted.org/packages/e4/16/c1fd27e9549f3c4baf1dc9c20c456cd2f822dbf8de9f463824b0c0357e06/uvloop-0.22.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:6cde23eeda1a25c75b2e07d39970f3374105d5eafbaab2a4482be82f272d5a5e", size = 4296730, upload-time = "2025-10-16T22:17:00.744Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "valkey-glide"
|
||||||
|
version = "2.4.1"
|
||||||
|
source = { registry = "https://pypi.org/simple" }
|
||||||
|
dependencies = [
|
||||||
|
{ name = "anyio" },
|
||||||
|
{ name = "protobuf" },
|
||||||
|
{ name = "sniffio" },
|
||||||
|
]
|
||||||
|
sdist = { url = "https://files.pythonhosted.org/packages/72/a2/582b34c6acc8dc857c537f6007459cba48dfa0dc404789a657e5c1a998c0/valkey_glide-2.4.1.tar.gz", hash = "sha256:f1155d84156d11b90488aa67e90102f0bf98a45314f5b99308ac9074c05f7241", size = 898030, upload-time = "2026-05-28T21:41:55.881Z" }
|
||||||
|
wheels = [
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/60/60/961ce40492a56ef831a905dfe03df4a81c0705152f6a8e49c541c634f49e/valkey_glide-2.4.1-cp311-cp311-macosx_10_7_x86_64.whl", hash = "sha256:d7285d03c2df040f26874b7f4ae96f040da2daecc9a34fa99da6f4e6ce5149c8", size = 7482152, upload-time = "2026-05-28T21:41:02.205Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/a4/b2/5a05567f0fc385dcbbbf6ab1061f0bc00443d51c2996e95eed45feaedda9/valkey_glide-2.4.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:5d2e82b74127897ccb7a957ad455787816a75fdc8c60a5e8004aef65ea93e99c", size = 6928601, upload-time = "2026-05-28T21:41:04.543Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/c5/d9/7ea2b47cff0a2f99921eb0db404215f828ced7814bd09ede9c93b65d20bc/valkey_glide-2.4.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4094128cb07e06e87013b7afab1e9388f8f5aeebe48ea6cbd54de15bd772e644", size = 7236977, upload-time = "2026-05-28T21:41:06.055Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/00/7a/6cda6b42156ed260e765e4ad2d6ab831607775e218a00fbb0d93411c4e8f/valkey_glide-2.4.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9f8dc0f3a36adb1cbe4e167972ca4758acdfed6baf58a4db94bbb713df56c8f5", size = 7691446, upload-time = "2026-05-28T21:41:07.833Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/c5/b4/da8c058baaee414a6bb2450742359f3b3b6993b23281bf227c5089f0099c/valkey_glide-2.4.1-cp312-cp312-macosx_10_7_x86_64.whl", hash = "sha256:5f8df64f6a4f0fd7203113103101fdf0aaa7ff0e7557312611de11ab89c6db75", size = 7472646, upload-time = "2026-05-28T21:41:09.451Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/f5/94/e1e311cb56597272b9cb69afb3fe8e2e7dd3371f88c92836015deddc6f49/valkey_glide-2.4.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:b45e35f44c17e88f8cd8082f8d8061a9763238c44ef20b11b615f6d87235864a", size = 6943375, upload-time = "2026-05-28T21:41:11.079Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/76/00/0e42e2f6866ebf0de552e076dc585a487b488b5b818c52460d28b50de65b/valkey_glide-2.4.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cf812b498925a30abab6e1a9f82f5eb821e967904fe7724729b2c82c47e29edf", size = 7237469, upload-time = "2026-05-28T21:41:12.733Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/f5/4c/c5dd9a1ed995453b0d9ca75a5af87e881c14e6eebdbf5a5fa78c3bae23fc/valkey_glide-2.4.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:214e2faca98966eea3eaf9e09de616862423815a5059843a9884125e2427a344", size = 7678744, upload-time = "2026-05-28T21:41:14.634Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/6a/2f/3df5702fc68684cef3e09f9cb6ed85578ddb08dc43593b1694c977f396fa/valkey_glide-2.4.1-cp313-cp313-macosx_10_7_x86_64.whl", hash = "sha256:c18976553ba663c03f7cc18c7e6075f4cbd2236c18b051e3d55bb213c6c44cb4", size = 7472972, upload-time = "2026-05-28T21:41:16.063Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/54/a3/6a74c6f996fa9e411e66b6f0e645fead2e0a341f1371e4cf3212efa54412/valkey_glide-2.4.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:43006e19cd63d66051263fa34a8ad47ba7d08a199585689b3f12f56ed6c9a005", size = 6943012, upload-time = "2026-05-28T21:41:17.492Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/fc/e7/d10ec41dca703f8c5dcbcba2b905e660c1cf56be53c4d5e368d7aa23d220/valkey_glide-2.4.1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b652a2a62aad87738e8f0e0aa5bf660ba91449c9fdb88550ccbc42e5fec08fe7", size = 7237842, upload-time = "2026-05-28T21:41:18.995Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/0a/a3/8916a9ed9e871686db444c86e601773245852ba1ad451ce1bb06f7aed91d/valkey_glide-2.4.1-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fbd27d26947fd9f1b6e9eaf0abce4bccfde779c1e618b310c4d725424b609793", size = 7678919, upload-time = "2026-05-28T21:41:20.502Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/05/35/6d39ec3cbd24d85ad8e1051e29e6509c0999f760aff5af7851c1a1981471/valkey_glide-2.4.1-cp314-cp314-macosx_10_7_x86_64.whl", hash = "sha256:91fb7ff97acdabc8f641255b548a48627bb731e65037b1126745bf8a0022e87d", size = 7471906, upload-time = "2026-05-28T21:41:22.135Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/ab/fc/3c28f794b7d35e13101598669c1d249c0a9f0408c545c87212e364c6ee4e/valkey_glide-2.4.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:d49a2537c2de44b0fc57691b1ae6c3d6f481e6f7f7eb879c0d28921d0aaec67d", size = 6943495, upload-time = "2026-05-28T21:41:23.783Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/2e/15/fb884631f5df78dc538c56bca9391165e40906b9b63ca65633d1be5bf980/valkey_glide-2.4.1-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cded9f14e448da5a96f61c066395f2c7e2846f2afe74cacc8634da0ae0c3425f", size = 7257720, upload-time = "2026-05-28T21:41:25.361Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/73/79/0b881017194386d21812b929a81dd8afd51d6b8d92280895b45913854785/valkey_glide-2.4.1-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5f249ab5bd0d69befe35897cf51a8fc9e01e9c8c9fe03087a68e6fe6d3e31d0d", size = 7682318, upload-time = "2026-05-28T21:41:26.996Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/7a/4d/f2b4e508692fcd21e76c7cbdc4f988bec7f4675e60f4f35ef482a826f6ae/valkey_glide-2.4.1-pp311-pypy311_pp73-macosx_10_7_x86_64.whl", hash = "sha256:775df9c7421a187c41caf003e4af5f073ed7e4b8abe50f8b9bec712cb03e12bf", size = 7479155, upload-time = "2026-05-28T21:41:42.399Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/52/d8/8a3495f5582dccb4c8e7faf6a73baf3dbc4580701923f06d8abf210ff22d/valkey_glide-2.4.1-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:0d87f21c77004240189cc3c5aab156966487afd81ffdee04225a52c7bd7132e4", size = 6938571, upload-time = "2026-05-28T21:41:44.078Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/f3/5a/a70077f76c2f18e94ec4309857b248beb7a8c7a3a50e30242abde2c3827d/valkey_glide-2.4.1-pp311-pypy311_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:44376ef5fe7a25287095b073d8abde510a50b1ead0143662394b3da9717863ef", size = 7260021, upload-time = "2026-05-28T21:41:45.837Z" },
|
||||||
|
{ url = "https://files.pythonhosted.org/packages/aa/12/72d31522e06fcc9b391118c1f69a09002224e78114b1db0d01b96008dc59/valkey_glide-2.4.1-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a59cc0a21d7a8b1b3caeb299f23817429b5fe6579bd4cb016382e6b7a10de984", size = 7693093, upload-time = "2026-05-28T21:41:47.617Z" },
|
||||||
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "virtualenv"
|
name = "virtualenv"
|
||||||
version = "20.36.1"
|
version = "20.36.1"
|
||||||
|
|||||||
Reference in New Issue
Block a user