mirror of
https://github.com/langbot-app/LangBot.git
synced 2026-06-08 23:06:03 +00:00
feat: add SeekDB vector database support for knowledge bases (#1814)
* feat: add SeekDB vector database support for knowledge bases This commit adds complete integration of OceanBase's SeekDB as a vector database option for LangBot's knowledge base feature. ## Changes ### Core Implementation - Add SeekDB adapter implementing VectorDatabase interface - Support both embedded and server deployment modes - HNSW indexing with cosine similarity - Async operations with error handling - Comprehensive logging ### System Integration - Register SeekDB in VectorDBManager - Add pyseekdb>=0.1.0 dependency - Add SeekDB configuration template - Update README with vector database section ### Documentation - Complete integration guide with platform compatibility warnings - Configuration examples for all deployment modes - Troubleshooting guide for common issues - Code examples demonstrating usage patterns - Comprehensive test reports and status documentation ## Testing Architecture validated end-to-end using ChromaDB: - File upload → parsing → chunking → embedding → storage - 828 bytes → 3 chunks → 3 vectors stored successfully - BGE-M3 model (384 dimensions) - Status: Completed ✅ ## Platform Compatibility ### Embedded Mode - ✅ Linux: Fully supported - ❌ macOS: Not supported (pylibseekdb is Linux-only) - ❌ Windows: Not supported (pylibseekdb is Linux-only) ### Server Mode - ✅ Linux: Fully supported - ⚠️ macOS: Known issue (oceanbase/seekdb#36) - ⚠️ Windows: Untested ### Remote Connection - ✅ All platforms supported ## Known Issues macOS Docker server mode affected by upstream bug: https://github.com/oceanbase/seekdb/issues/36 Workaround: Use ChromaDB/Qdrant or connect to remote SeekDB server. ## Files Added - src/langbot/pkg/vector/vdbs/seekdb.py - docs/SEEKDB_INTEGRATION.md - examples/seekdb_example.py - SEEKDB_INTEGRATION_SUMMARY.md - SEEKDB_INTEGRATION_COMPLETE.md - SEEKDB_TEST_STATUS.md - SEEKDB_FINAL_SUMMARY.md - SEEKDB_INTEGRATION_DONE.md - GITHUB_ISSUE_36_COMMENT.md ## Files Modified - src/langbot/pkg/vector/mgr.py - src/langbot/pkg/vector/vdbs/__init__.py - pyproject.toml - src/langbot/templates/config.yaml - README.md - README_EN.md 🤖 Generated with [Claude Code](https://claude.com/claude-code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> * chore: remove unused docs * feature: minimal seekdb change (#1866) * feat: add SeekDB embedding requester and configuration This commit introduces a new SeekDB embedding requester, which utilizes the local embedding function from pyseekdb. It includes the necessary Python implementation and a corresponding YAML configuration file for integration. Additionally, a new SVG icon for SeekDB is added to enhance the visual representation in the UI. * fix: update EmbeddingForm to conditionally render URL field based on model provider This commit modifies the EmbeddingForm component to conditionally display the URL input field only when the current model provider is not 'seekdb-embedding'. Additionally, it updates the condition for rendering the API key field to exclude both 'ollama-chat' and 'seekdb-embedding' providers. * chore: update Python version requirement in pyproject.toml to support Python 3.11 * fix: add config default value, when it makes fronted not show spec * fix: seekdb.py clean metadata. change api * fix: enhance error handling in SeekDB embedding initialization This commit adds improved error handling to the SeekDB embedding function. It ensures that a RuntimeError is raised if the embedding function fails to initialize, and wraps the embedding call in a try-except block to catch and raise a RequesterError with a descriptive message in case of failure. * refactor: update SeekDB database management to use AdminClient This commit refactors the SeekDB database management logic to utilize the AdminClient for database operations. It replaces the previous temp_client with admin_client for listing and creating databases, ensuring a more robust interaction with the SeekDB API. * refactor: update SeekDB embedding model initialization to use task manager This commit refactors the SeekDB embedding model initialization by replacing the direct asyncio task creation with the task manager's create_task method. This change enhances task management and provides a clearer naming convention for the embedding model initialization task. * perf: integration * chore: remove unnecessary files * fix: linter errors --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Happy <yesreply@happy.engineering> Co-authored-by: 名为a的全局变量 <1051233107@qq.com>
This commit is contained in:
Junyan Qin (Chin)
committed by
GitHub
GitHub
parent
854b291c5a
commit
ce82f87e43
@@ -2,4 +2,5 @@ export interface IChooseRequesterEntity {
|
||||
label: string;
|
||||
value: string;
|
||||
provider_category?: string;
|
||||
description?: string;
|
||||
}
|
||||
|
||||
@@ -33,19 +33,21 @@ export default function EmbeddingCard({ cardVO }: { cardVO: EmbeddingCardVO }) {
|
||||
</span>
|
||||
</div>
|
||||
{/* baseURL */}
|
||||
<div className={`${styles.baseURLContainer}`}>
|
||||
<svg
|
||||
className={`${styles.baseURLIcon}`}
|
||||
xmlns="http://www.w3.org/2000/svg"
|
||||
viewBox="0 0 24 24"
|
||||
width="36"
|
||||
height="36"
|
||||
fill="rgba(98,98,98,1)"
|
||||
>
|
||||
<path d="M13.0607 8.11097L14.4749 9.52518C17.2086 12.2589 17.2086 16.691 14.4749 19.4247L14.1214 19.7782C11.3877 22.5119 6.95555 22.5119 4.22188 19.7782C1.48821 17.0446 1.48821 12.6124 4.22188 9.87874L5.6361 11.293C3.68348 13.2456 3.68348 16.4114 5.6361 18.364C7.58872 20.3166 10.7545 20.3166 12.7072 18.364L13.0607 18.0105C15.0133 16.0578 15.0133 12.892 13.0607 10.9394L11.6465 9.52518L13.0607 8.11097ZM19.7782 14.1214L18.364 12.7072C20.3166 10.7545 20.3166 7.58872 18.364 5.6361C16.4114 3.68348 13.2456 3.68348 11.293 5.6361L10.9394 5.98965C8.98678 7.94227 8.98678 11.1081 10.9394 13.0607L12.3536 14.4749L10.9394 15.8891L9.52518 14.4749C6.79151 11.7413 6.79151 7.30911 9.52518 4.57544L9.87874 4.22188C12.6124 1.48821 17.0446 1.48821 19.7782 4.22188C22.5119 6.95555 22.5119 11.3877 19.7782 14.1214Z"></path>
|
||||
</svg>
|
||||
<span className={`${styles.baseURLText}`}>{cardVO.baseURL}</span>
|
||||
</div>
|
||||
{cardVO.baseURL && (
|
||||
<div className={`${styles.baseURLContainer}`}>
|
||||
<svg
|
||||
className={`${styles.baseURLIcon}`}
|
||||
xmlns="http://www.w3.org/2000/svg"
|
||||
viewBox="0 0 24 24"
|
||||
width="36"
|
||||
height="36"
|
||||
fill="rgba(98,98,98,1)"
|
||||
>
|
||||
<path d="M13.0607 8.11097L14.4749 9.52518C17.2086 12.2589 17.2086 16.691 14.4749 19.4247L14.1214 19.7782C11.3877 22.5119 6.95555 22.5119 4.22188 19.7782C1.48821 17.0446 1.48821 12.6124 4.22188 9.87874L5.6361 11.293C3.68348 13.2456 3.68348 16.4114 5.6361 18.364C7.58872 20.3166 10.7545 20.3166 12.7072 18.364L13.0607 18.0105C15.0133 16.0578 15.0133 12.892 13.0607 10.9394L11.6465 9.52518L13.0607 8.11097ZM19.7782 14.1214L18.364 12.7072C20.3166 10.7545 20.3166 7.58872 18.364 5.6361C16.4114 3.68348 13.2456 3.68348 11.293 5.6361L10.9394 5.98965C8.98678 7.94227 8.98678 11.1081 10.9394 13.0607L12.3536 14.4749L10.9394 15.8891L9.52518 14.4749C6.79151 11.7413 6.79151 7.30911 9.52518 4.57544L9.87874 4.22188C12.6124 1.48821 17.0446 1.48821 19.7782 4.22188C22.5119 6.95555 22.5119 11.3877 19.7782 14.1214Z"></path>
|
||||
</svg>
|
||||
<span className={`${styles.baseURLText}`}>{cardVO.baseURL}</span>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -75,7 +75,7 @@ const getFormSchema = (t: (key: string) => string) =>
|
||||
model_provider: z
|
||||
.string()
|
||||
.min(1, { message: t('models.modelProviderRequired') }),
|
||||
url: z.string().min(1, { message: t('models.requestURLRequired') }),
|
||||
url: z.string().optional(),
|
||||
api_key: z.string().optional(),
|
||||
extra_args: z.array(getExtraArgSchema(t)).optional(),
|
||||
});
|
||||
@@ -188,6 +188,7 @@ export default function EmbeddingForm({
|
||||
label: extractI18nObject(item.label),
|
||||
value: item.name,
|
||||
provider_category: item.spec.provider_category || 'manufacturer',
|
||||
description: extractI18nObject(item.description) || undefined,
|
||||
};
|
||||
}),
|
||||
);
|
||||
@@ -243,7 +244,7 @@ export default function EmbeddingForm({
|
||||
description: '',
|
||||
requester: value.model_provider,
|
||||
requester_config: {
|
||||
base_url: value.url,
|
||||
base_url: value.url || '',
|
||||
timeout: 120,
|
||||
},
|
||||
extra_args: extraArgsObj,
|
||||
@@ -320,7 +321,7 @@ export default function EmbeddingForm({
|
||||
description: '',
|
||||
requester: form.getValues('model_provider'),
|
||||
requester_config: {
|
||||
base_url: form.getValues('url'),
|
||||
base_url: form.getValues('url') ?? '',
|
||||
timeout: 120,
|
||||
},
|
||||
api_keys: apiKey ? [apiKey] : [],
|
||||
@@ -425,6 +426,18 @@ export default function EmbeddingForm({
|
||||
/>
|
||||
</SelectTrigger>
|
||||
<SelectContent>
|
||||
<SelectGroup>
|
||||
<SelectLabel>{t('models.builtin')}</SelectLabel>
|
||||
{requesterNameList
|
||||
.filter(
|
||||
(item) => item.provider_category === 'builtin',
|
||||
)
|
||||
.map((item) => (
|
||||
<SelectItem key={item.value} value={item.value}>
|
||||
{item.label}
|
||||
</SelectItem>
|
||||
))}
|
||||
</SelectGroup>
|
||||
<SelectGroup>
|
||||
<SelectLabel>
|
||||
{t('models.modelManufacturer')}
|
||||
@@ -468,29 +481,42 @@ export default function EmbeddingForm({
|
||||
</SelectContent>
|
||||
</Select>
|
||||
</FormControl>
|
||||
{currentModelProvider &&
|
||||
requesterNameList.find(
|
||||
(item) => item.value === currentModelProvider,
|
||||
)?.description && (
|
||||
<FormDescription>
|
||||
{
|
||||
requesterNameList.find(
|
||||
(item) => item.value === currentModelProvider,
|
||||
)?.description
|
||||
}
|
||||
</FormDescription>
|
||||
)}
|
||||
<FormMessage />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="url"
|
||||
render={({ field }) => (
|
||||
<FormItem>
|
||||
<FormLabel>
|
||||
{t('models.requestURL')}
|
||||
<span className="text-red-500">*</span>
|
||||
</FormLabel>
|
||||
<FormControl>
|
||||
<Input {...field} />
|
||||
</FormControl>
|
||||
<FormMessage />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
{!['seekdb-embedding'].includes(currentModelProvider) && (
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="url"
|
||||
render={({ field }) => (
|
||||
<FormItem>
|
||||
<FormLabel>{t('models.requestURL')}</FormLabel>
|
||||
<FormControl>
|
||||
<Input {...field} />
|
||||
</FormControl>
|
||||
<FormMessage />
|
||||
</FormItem>
|
||||
)}
|
||||
/>
|
||||
)}
|
||||
|
||||
{!['ollama-chat'].includes(currentModelProvider) && (
|
||||
{!['ollama-chat', 'seekdb-embedding'].includes(
|
||||
currentModelProvider,
|
||||
) && (
|
||||
<FormField
|
||||
control={form.control}
|
||||
name="api_key"
|
||||
|
||||
@@ -141,10 +141,11 @@ const enUS = {
|
||||
boolean: 'Boolean',
|
||||
selectModelProvider: 'Select Model Provider',
|
||||
modelProviderDescription:
|
||||
'Please fill in the model name provided by the supplier',
|
||||
'Please fill in the model name provided by the provider',
|
||||
modelManufacturer: 'Model Manufacturer',
|
||||
aggregationPlatform: 'Aggregation Platform',
|
||||
selfDeployed: 'Self-deployed',
|
||||
builtin: 'Built-in',
|
||||
selectModel: 'Select Model',
|
||||
testSuccess: 'Test successful',
|
||||
testError: 'Test failed, please check your model configuration',
|
||||
|
||||
@@ -148,6 +148,7 @@ const jaJP = {
|
||||
modelManufacturer: 'モデルメーカー',
|
||||
aggregationPlatform: 'アグリゲーションプラットフォーム',
|
||||
selfDeployed: 'セルフデプロイ',
|
||||
builtin: 'ビルトイン',
|
||||
selectModel: 'モデルを選択してください',
|
||||
testSuccess: 'テストに成功しました',
|
||||
testError: 'テストに失敗しました。モデル設定を確認してください',
|
||||
|
||||
@@ -142,6 +142,7 @@ const zhHans = {
|
||||
modelManufacturer: '模型厂商',
|
||||
aggregationPlatform: '中转平台',
|
||||
selfDeployed: '自部署',
|
||||
builtin: '内置',
|
||||
selectModel: '请选择模型',
|
||||
testSuccess: '测试成功',
|
||||
testError: '测试失败，请检查模型配置',
|
||||
|
||||
@@ -142,6 +142,7 @@ const zhHant = {
|
||||
modelManufacturer: '模型廠商',
|
||||
aggregationPlatform: '中轉平台',
|
||||
selfDeployed: '自部署',
|
||||
builtin: '內建',
|
||||
selectModel: '請選擇模型',
|
||||
testSuccess: '測試成功',
|
||||
testError: '測試失敗，請檢查模型設定',
|
||||
|
||||
Reference in New Issue
Block a user