mirror of
https://github.com/langbot-app/LangBot.git
synced 2026-10-31 04:45:23 +08:00
feat(plugins): consolidate official plugins with independent releases (LANG-180) (#2797)
Co-authored-by: multica-agent <[email protected]>
This commit is contained in:
RockChinQ
co-authored by
multica-agent
multica-agent
parent
d099033d79
commit
3a77306c3c
@@ -0,0 +1,3 @@
|
||||
# This is a .env file example, please copy it to .env file.
|
||||
# Please refer to https://langbot.app/docs/en/plugin/dev/tutor.html for more details.
|
||||
DEBUG_RUNTIME_WS_URL=ws://127.0.0.1:5401/debug/ws
|
||||
@@ -0,0 +1,176 @@
|
||||
# Byte-compiled / optimized / DLL files
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
|
||||
# C extensions
|
||||
*.so
|
||||
|
||||
# Distribution / packaging
|
||||
.Python
|
||||
build/
|
||||
develop-eggs/
|
||||
dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
lib/
|
||||
lib64/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
wheels/
|
||||
share/python-wheels/
|
||||
*.egg-info/
|
||||
.installed.cfg
|
||||
*.egg
|
||||
MANIFEST
|
||||
|
||||
# PyInstaller
|
||||
# Usually these files are written by a python script from a template
|
||||
# before PyInstaller builds the exe, so as to inject date/other infos into it.
|
||||
*.manifest
|
||||
*.spec
|
||||
|
||||
# Installer logs
|
||||
pip-log.txt
|
||||
pip-delete-this-directory.txt
|
||||
|
||||
# Unit test / coverage reports
|
||||
htmlcov/
|
||||
.tox/
|
||||
.nox/
|
||||
.coverage
|
||||
.coverage.*
|
||||
.cache
|
||||
nosetests.xml
|
||||
coverage.xml
|
||||
*.cover
|
||||
*.py,cover
|
||||
.hypothesis/
|
||||
.pytest_cache/
|
||||
cover/
|
||||
|
||||
# Translations
|
||||
*.mo
|
||||
*.pot
|
||||
|
||||
# Django stuff:
|
||||
*.log
|
||||
local_settings.py
|
||||
db.sqlite3
|
||||
db.sqlite3-journal
|
||||
|
||||
# Flask stuff:
|
||||
instance/
|
||||
.webassets-cache
|
||||
|
||||
# Scrapy stuff:
|
||||
.scrapy
|
||||
|
||||
# Sphinx documentation
|
||||
docs/_build/
|
||||
|
||||
# PyBuilder
|
||||
.pybuilder/
|
||||
target/
|
||||
|
||||
# Jupyter Notebook
|
||||
.ipynb_checkpoints
|
||||
|
||||
# IPython
|
||||
profile_default/
|
||||
ipython_config.py
|
||||
|
||||
# pyenv
|
||||
# For a library or package, you might want to ignore these files since the code is
|
||||
# intended to run in multiple environments; otherwise, check them in:
|
||||
# .python-version
|
||||
|
||||
# pipenv
|
||||
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
|
||||
# However, in case of collaboration, if having platform-specific dependencies or dependencies
|
||||
# having no cross-platform support, pipenv may install dependencies that don't work, or not
|
||||
# install all needed dependencies.
|
||||
#Pipfile.lock
|
||||
|
||||
# UV
|
||||
# Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
|
||||
# This is especially recommended for binary packages to ensure reproducibility, and is more
|
||||
# commonly ignored for libraries.
|
||||
#uv.lock
|
||||
|
||||
# poetry
|
||||
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
|
||||
# This is especially recommended for binary packages to ensure reproducibility, and is more
|
||||
# commonly ignored for libraries.
|
||||
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
|
||||
#poetry.lock
|
||||
|
||||
# pdm
|
||||
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
|
||||
#pdm.lock
|
||||
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
|
||||
# in version control.
|
||||
# https://pdm.fming.dev/latest/usage/project/#working-with-version-control
|
||||
.pdm.toml
|
||||
.pdm-python
|
||||
.pdm-build/
|
||||
|
||||
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
|
||||
__pypackages__/
|
||||
|
||||
# Celery stuff
|
||||
celerybeat-schedule
|
||||
celerybeat.pid
|
||||
|
||||
# SageMath parsed files
|
||||
*.sage.py
|
||||
|
||||
# Environments
|
||||
.env
|
||||
.venv
|
||||
env/
|
||||
venv/
|
||||
ENV/
|
||||
env.bak/
|
||||
venv.bak/
|
||||
|
||||
# Spyder project settings
|
||||
.spyderproject
|
||||
.spyproject
|
||||
|
||||
# Rope project settings
|
||||
.ropeproject
|
||||
|
||||
# mkdocs documentation
|
||||
/site
|
||||
|
||||
# mypy
|
||||
.mypy_cache/
|
||||
.dmypy.json
|
||||
dmypy.json
|
||||
|
||||
# Pyre type checker
|
||||
.pyre/
|
||||
|
||||
# pytype static type analyzer
|
||||
.pytype/
|
||||
|
||||
# Cython debug symbols
|
||||
cython_debug/
|
||||
|
||||
# PyCharm
|
||||
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
|
||||
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
|
||||
# and can be added to the global gitignore or merged into this file. For a more nuclear
|
||||
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
|
||||
#.idea/
|
||||
|
||||
# Ruff stuff:
|
||||
.ruff_cache/
|
||||
|
||||
# PyPI configuration file
|
||||
.pypirc
|
||||
|
||||
/data/
|
||||
@@ -0,0 +1,65 @@
|
||||
# DifyDatasetsConnector
|
||||
|
||||
Retrieve knowledge from or store files into Dify knowledge bases using the Dify API.
|
||||
|
||||
## Configuration
|
||||
|
||||
Please add an external knowledge base in LangBot and select "DifyDatasetsConnector" as the knowledge retriever type.
|
||||
|
||||
### Creation Settings (set when creating a knowledge base)
|
||||
|
||||
- **api_base_url**: Base URL for Dify API
|
||||
- For Dify Cloud: `https://api.dify.ai/v1` (default)
|
||||
- For self-hosted instances: Your server URL (e.g., `http://localhost/api` or `https://your-domain.com/api`)
|
||||
- **dify_apikey**: Your Dify API key from your Dify instance
|
||||
- **dataset_id**: The ID of your Dify knowledge base/dataset
|
||||
|
||||
### Retrieval Settings (adjustable per query)
|
||||
|
||||
- **search_method** (default: semantic_search): The search method to use
|
||||
- `keyword_search`: Keyword-based search
|
||||
- `semantic_search`: Semantic similarity search (default)
|
||||
- `full_text_search`: Full-text search
|
||||
- `hybrid_search`: Hybrid search combining semantic and full-text
|
||||
- **top_k** (default: 5): Maximum number of retrieved results
|
||||
- **score_threshold_enabled** (default: false): Whether to enable score threshold filtering
|
||||
- **score_threshold** (default: 0.5): Minimum relevance score (0-1), only shown when score threshold is enabled
|
||||
- **reranking_enable** (default: false): Enable reranking to improve result quality. The reranking model is automatically fetched from your Dify dataset settings — please configure the reranking model in the Dify console first
|
||||
|
||||
## How to Get Configuration Values
|
||||
|
||||
### Getting your Dify API Key
|
||||
|
||||
1. Go to https://cloud.dify.ai/
|
||||
2. Navigate to your knowledge base page
|
||||
3. Click on "API ACCESS" in the left sidebar
|
||||
4. Create or copy your API key from the "API Keys" section
|
||||
|
||||
### Getting your Dataset ID
|
||||
|
||||
1. In the Dify knowledge base list, click on your knowledge base
|
||||
2. The dataset ID is in the URL: `https://cloud.dify.ai/datasets/{dataset_id}`
|
||||
3. Or you can find it in the API documentation page of your knowledge base
|
||||
|
||||
### Configuring Reranking
|
||||
|
||||
1. In the Dify console, go to your dataset settings
|
||||
2. Enable reranking and select a reranking model (e.g., `cohere/rerank-v3.5`)
|
||||
3. Save the settings
|
||||
4. In LangBot, enable the "Enable Reranking" toggle — the plugin will automatically use the model configured in Dify
|
||||
|
||||
## API Reference
|
||||
|
||||
This plugin uses the Dify Dataset API:
|
||||
- Retrieval: `POST /v1/datasets/{dataset_id}/retrieve`
|
||||
- Dataset info: `GET /v1/datasets/{dataset_id}`
|
||||
- Document upload: `POST /v1/datasets/{dataset_id}/document/create-by-file`
|
||||
- Document delete: `DELETE /v1/datasets/{dataset_id}/documents/{document_id}`
|
||||
- Documentation: https://docs.dify.ai/
|
||||
|
||||
## Runtime placement
|
||||
|
||||
This version remains dedicated. SDK 0.7.4 shared singleton placement requires a
|
||||
signed `stateless-v1` component model, not just `shared-runtime-v1`.
|
||||
The existing per-installation mutation state and cancellation model have
|
||||
not passed that contract. See [the audit](../SHARED_RUNTIME.md).
|
||||
@@ -0,0 +1 @@
|
||||
<svg height="1em" style="flex:none;line-height:1" viewBox="0 0 24 24" width="1em" xmlns="http://www.w3.org/2000/svg"><title>Dify</title><path d="M7.043 6.487c1.635 0 2.241-1.003 2.241-2.243S8.681 2 7.044 2C5.405 2 4.801 3.003 4.801 4.244c0 1.24.604 2.243 2.241 2.243z" fill="#03F"></path><path d="M14.883 6.97v1.443h-3.679v3.203h3.68v8.012H8.801V8.41h-8v3.203h4.48v8.012H0v3.203h24v-3.203h-5.6v-8.012H24V8.41h-5.6V5.206H24V2.003h-4.161a4.97 4.97 0 00-4.961 4.967h.005z" fill="#03F"></path></svg>
|
||||
|
After Width: | Height: | Size: 496 B |
@@ -0,0 +1,136 @@
|
||||
apiVersion: v1
|
||||
kind: KnowledgeEngine
|
||||
metadata:
|
||||
name: dify-connector
|
||||
label:
|
||||
en_US: Dify Knowledge Base
|
||||
zh_Hans: Dify 知识库
|
||||
ja_JP: Dify 知識ベース
|
||||
description:
|
||||
en_US: Retrieve knowledge from or store files into Dify knowledge bases
|
||||
zh_Hans: 从 Dify 知识库中检索知识或向其中存入文件
|
||||
ja_JP: Dify 知識ベースから知識を検索または保存
|
||||
spec:
|
||||
creation_schema:
|
||||
- name: api_base_url
|
||||
label:
|
||||
en_US: API Base URL
|
||||
zh_Hans: API 基础地址
|
||||
description:
|
||||
en_US: 'Base URL for Dify API (e.g., https://api.dify.ai/v1 for cloud, or your self-hosted URL)'
|
||||
zh_Hans: 'Dify API 基础地址（云端版本使用 https://api.dify.ai/v1，自托管版本使用您的服务器地址）'
|
||||
ja_JP: 'Dify API ベース URL（クラウド版は https://api.dify.ai/v1、セルフホストの場合はサーバーの URL）'
|
||||
type: string
|
||||
default: https://api.dify.ai/v1
|
||||
required: true
|
||||
- name: dify_apikey
|
||||
label:
|
||||
en_US: API Key
|
||||
zh_Hans: API 密钥
|
||||
description:
|
||||
en_US: 'API key from your Dify instance'
|
||||
zh_Hans: '从您的 Dify 实例获取的 API Key'
|
||||
ja_JP: 'Dify インスタンスから取得した API Key'
|
||||
type: password
|
||||
required: true
|
||||
- name: dataset_id
|
||||
label:
|
||||
en_US: Dataset ID
|
||||
zh_Hans: 数据集 ID
|
||||
ja_JP: データセット ID
|
||||
description:
|
||||
en_US: 'The UUID of the Dify Dataset to retrieve from'
|
||||
zh_Hans: '要检索的 Dify 数据集 UUID'
|
||||
ja_JP: 'Dify ナレッジベース/データセットの ID'
|
||||
type: string
|
||||
required: true
|
||||
retrieval_schema:
|
||||
- name: search_method
|
||||
label:
|
||||
en_US: Search Method
|
||||
zh_Hans: 检索模式
|
||||
ja_JP: 検索方法
|
||||
description:
|
||||
en_US: 'The search method to use'
|
||||
zh_Hans: '使用的搜索方法'
|
||||
ja_JP: '使用する検索方法'
|
||||
type: select
|
||||
default: semantic_search
|
||||
options:
|
||||
- name: keyword_search
|
||||
label:
|
||||
en_US: Keyword Search
|
||||
zh_Hans: 关键词检索
|
||||
ja_JP: キーワード検索
|
||||
- name: semantic_search
|
||||
label:
|
||||
en_US: Semantic Search
|
||||
zh_Hans: 语义检索
|
||||
ja_JP: セマンティック検索
|
||||
- name: full_text_search
|
||||
label:
|
||||
en_US: Full Text Search
|
||||
zh_Hans: 全文检索
|
||||
ja_JP: 全文検索
|
||||
- name: hybrid_search
|
||||
label:
|
||||
en_US: Hybrid Search
|
||||
zh_Hans: 混合检索
|
||||
ja_JP: ハイブリッド検索
|
||||
required: false
|
||||
- name: top_k
|
||||
label:
|
||||
en_US: Top K
|
||||
zh_Hans: 返回数量
|
||||
ja_JP: 取得件数
|
||||
description:
|
||||
en_US: 'Maximum number of results to return'
|
||||
zh_Hans: '返回的最大结果数量'
|
||||
ja_JP: '返す結果の最大数'
|
||||
type: integer
|
||||
default: 5
|
||||
required: false
|
||||
- name: score_threshold_enabled
|
||||
label:
|
||||
en_US: Enable Score Threshold
|
||||
zh_Hans: 启用分数阈值
|
||||
ja_JP: スコア閾値を有効化
|
||||
description:
|
||||
en_US: 'Filter results below the minimum score'
|
||||
zh_Hans: '过滤低于最低分数的结果'
|
||||
ja_JP: '最小スコア以下の結果をフィルタリング'
|
||||
type: boolean
|
||||
default: false
|
||||
required: false
|
||||
- name: score_threshold
|
||||
label:
|
||||
en_US: Score Threshold
|
||||
zh_Hans: 分数阈值
|
||||
ja_JP: スコア閾値
|
||||
description:
|
||||
en_US: 'Minimum relevance score (0-1)'
|
||||
zh_Hans: '最小相关性分数 (0-1)'
|
||||
ja_JP: '最小関連性スコア (0-1)'
|
||||
type: float
|
||||
default: 0.5
|
||||
required: false
|
||||
show_if:
|
||||
field: score_threshold_enabled
|
||||
operator: eq
|
||||
value: true
|
||||
- name: reranking_enable
|
||||
label:
|
||||
en_US: Enable Reranking
|
||||
zh_Hans: 启用重排序
|
||||
ja_JP: リランキングを有効化
|
||||
description:
|
||||
en_US: 'Enable reranking using the model configured in your Dify dataset settings (configure it in Dify console first)'
|
||||
zh_Hans: '启用重排序，将自动使用 Dify 数据集中配置的重排序模型（请先在 Dify 控制台配置）'
|
||||
ja_JP: 'Dify データセット設定で構成されたリランキングモデルを使用（先に Dify コンソールで設定してください）'
|
||||
type: boolean
|
||||
default: false
|
||||
required: false
|
||||
execution:
|
||||
python:
|
||||
path: engine.py
|
||||
attr: DifyDatasetsConnector
|
||||
@@ -0,0 +1,373 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from urllib.parse import urlsplit, urlunsplit
|
||||
|
||||
from components.shared_state import (
|
||||
AmbiguousMutationError,
|
||||
ConfigStore,
|
||||
serialized,
|
||||
)
|
||||
|
||||
import httpx
|
||||
|
||||
from langbot_plugin.api.definition.components.knowledge_engine import KnowledgeEngine, KnowledgeEngineCapability
|
||||
from langbot_plugin.api.entities.builtin.rag import (
|
||||
IngestionContext,
|
||||
IngestionResult,
|
||||
DocumentStatus,
|
||||
RetrievalContext,
|
||||
RetrievalResultEntry,
|
||||
RetrievalResponse,
|
||||
)
|
||||
from langbot_plugin.api.entities.builtin.provider.message import ContentElement
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _normalize_api_base_url(url: str) -> str:
|
||||
"""Canonical provider target: ``scheme://host[:port]/path``, no trailing slash.
|
||||
|
||||
Used to bind a stored document mapping to the upstream instance that created
|
||||
it, so a later configuration change is detected before any remote call.
|
||||
"""
|
||||
parts = urlsplit(url or '')
|
||||
return urlunsplit((parts.scheme.lower(), parts.netloc.lower(), parts.path.rstrip('/'), '', ''))
|
||||
|
||||
|
||||
class DifyDatasetsConnector(ConfigStore, KnowledgeEngine):
|
||||
"""Knowledge Engine powered by Dify Datasets.
|
||||
|
||||
Supports retrieval, document ingestion (file upload), and deletion
|
||||
via the Dify Dataset API.
|
||||
"""
|
||||
|
||||
@classmethod
|
||||
def get_capabilities(cls) -> list[str]:
|
||||
return [KnowledgeEngineCapability.DOC_INGESTION, KnowledgeEngineCapability.DOC_PARSING]
|
||||
|
||||
# ========== Lifecycle Hooks ==========
|
||||
|
||||
@serialized
|
||||
async def on_knowledge_base_create(self, kb_id: str, config: dict) -> None:
|
||||
"""Persist the knowledge-base config for deletion after worker restart."""
|
||||
await self._save_config(kb_id, config)
|
||||
logger.info(f"[DifyDatasetsConnector] Knowledge base created: {kb_id}")
|
||||
|
||||
@serialized(allow_fenced=True)
|
||||
async def on_knowledge_base_delete(self, kb_id: str) -> None:
|
||||
"""Tombstone the stored config and clear any fence for this knowledge base."""
|
||||
await self._save_config(kb_id, None)
|
||||
# Deleting the knowledge base is the operator path out of a fence: the
|
||||
# pending ambiguity is moot once it no longer exists.
|
||||
await self._state.clear_fence(self.plugin, kb_id)
|
||||
logger.info(f"[DifyDatasetsConnector] Knowledge base deleted: {kb_id}")
|
||||
|
||||
# ========== Helper Methods ==========
|
||||
|
||||
async def _fetch_dataset_reranking_config(
|
||||
self, api_base_url: str, api_key: str, dataset_id: str
|
||||
) -> dict:
|
||||
"""Fetch the reranking model config from the Dify dataset settings."""
|
||||
url = f"{api_base_url}/datasets/{dataset_id}"
|
||||
headers = {"Authorization": f"Bearer {api_key}"}
|
||||
try:
|
||||
async with self.http_client() as client:
|
||||
response = await client.get(url, headers=headers, timeout=15.0)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
retrieval_model = data.get("retrieval_model_dict") or data.get("retrieval_model") or {}
|
||||
reranking = retrieval_model.get("reranking_model", {})
|
||||
mode = retrieval_model.get("reranking_mode")
|
||||
logger.info(
|
||||
f"[DifyDatasetsConnector] Fetched dataset reranking config: "
|
||||
f"mode={mode}, provider={reranking.get('reranking_provider_name')}, "
|
||||
f"model={reranking.get('reranking_model_name')}"
|
||||
)
|
||||
return {
|
||||
"reranking_mode": mode or "reranking_model",
|
||||
"reranking_model": reranking,
|
||||
}
|
||||
except Exception:
|
||||
logger.exception("[DifyDatasetsConnector] Failed to fetch dataset reranking config")
|
||||
return {
|
||||
"reranking_mode": "reranking_model",
|
||||
"reranking_model": {"reranking_provider_name": "", "reranking_model_name": ""},
|
||||
}
|
||||
|
||||
# ========== Core Methods ==========
|
||||
|
||||
async def retrieve(self, context: RetrievalContext) -> RetrievalResponse:
|
||||
"""Execute retrieval against Dify Dataset API."""
|
||||
config = context.creation_settings
|
||||
retrieval = context.retrieval_settings
|
||||
|
||||
api_base_url = config.get("api_base_url", "https://api.dify.ai/v1").rstrip("/")
|
||||
api_key = config.get("dify_apikey")
|
||||
dataset_id = config.get("dataset_id")
|
||||
top_k = retrieval.get("top_k", 5)
|
||||
score_threshold_enabled = retrieval.get("score_threshold_enabled", False)
|
||||
score_threshold = retrieval.get("score_threshold", 0.5)
|
||||
search_method = retrieval.get("search_method", "semantic_search")
|
||||
reranking_enable = retrieval.get("reranking_enable", False)
|
||||
|
||||
if not api_key or not dataset_id:
|
||||
logger.error(
|
||||
f"[DifyDatasetsConnector] Missing required configuration. "
|
||||
f"Config keys: {list(config.keys())}"
|
||||
)
|
||||
return RetrievalResponse(results=[], total_found=0)
|
||||
|
||||
# Fetch reranking model config from Dify dataset when reranking is enabled
|
||||
reranking_mode = None
|
||||
reranking_model = {"reranking_provider_name": "", "reranking_model_name": ""}
|
||||
if reranking_enable:
|
||||
reranking_config = await self._fetch_dataset_reranking_config(
|
||||
api_base_url, api_key, dataset_id
|
||||
)
|
||||
reranking_mode = reranking_config["reranking_mode"]
|
||||
reranking_model = reranking_config["reranking_model"]
|
||||
|
||||
url = f"{api_base_url}/datasets/{dataset_id}/retrieve"
|
||||
headers = {
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
payload = {
|
||||
"query": context.query,
|
||||
"retrieval_model": {
|
||||
"search_method": search_method,
|
||||
"reranking_enable": reranking_enable,
|
||||
"reranking_mode": reranking_mode,
|
||||
"reranking_model": reranking_model,
|
||||
"weights": None,
|
||||
"top_k": int(top_k),
|
||||
"score_threshold_enabled": score_threshold_enabled,
|
||||
"score_threshold": float(score_threshold),
|
||||
},
|
||||
}
|
||||
|
||||
results: list[RetrievalResultEntry] = []
|
||||
try:
|
||||
async with self.http_client() as client:
|
||||
response = await client.post(url, json=payload, headers=headers, timeout=30.0)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
logger.info(
|
||||
f"[DifyDatasetsConnector] Request: search_method={search_method}, top_k={top_k}, "
|
||||
f"score_threshold_enabled={score_threshold_enabled}, score_threshold={score_threshold}, "
|
||||
f"reranking_enable={reranking_enable}, reranking_mode={reranking_mode}, "
|
||||
f"reranking_model={reranking_model}, "
|
||||
f"Response records count: {len(data.get('records', []))}"
|
||||
)
|
||||
|
||||
for record in data.get("records", []):
|
||||
segment = record.get("segment", {})
|
||||
document = segment.get("document", {})
|
||||
|
||||
score = record.get("score")
|
||||
if score is None:
|
||||
score = 0.0
|
||||
|
||||
results.append(
|
||||
RetrievalResultEntry(
|
||||
id=segment.get("id", ""),
|
||||
content=[ContentElement.from_text(segment.get("content", ""))],
|
||||
metadata={
|
||||
"document_id": segment.get("document_id", ""),
|
||||
"document_name": document.get("name", ""),
|
||||
"segment_id": segment.get("id", ""),
|
||||
"keywords": segment.get("keywords", []),
|
||||
"answer": segment.get("answer"),
|
||||
},
|
||||
distance=1.0 - float(score),
|
||||
score=float(score),
|
||||
)
|
||||
)
|
||||
|
||||
logger.info(f"[DifyDatasetsConnector] Retrieved {len(results)} chunks from Dify.")
|
||||
except Exception:
|
||||
logger.exception("[DifyDatasetsConnector] Error during retrieval")
|
||||
|
||||
return RetrievalResponse(results=results, total_found=len(results))
|
||||
|
||||
@serialized
|
||||
async def ingest(self, context: IngestionContext) -> IngestionResult:
|
||||
"""Upload a file to a Dify dataset for indexing."""
|
||||
doc_id = context.file_object.metadata.document_id
|
||||
filename = context.file_object.metadata.filename
|
||||
config = context.creation_settings
|
||||
|
||||
api_base_url = config.get("api_base_url", "https://api.dify.ai/v1").rstrip("/")
|
||||
upstream_target = _normalize_api_base_url(api_base_url)
|
||||
api_key = config.get("dify_apikey")
|
||||
dataset_id = config.get("dataset_id")
|
||||
|
||||
if not api_key or not dataset_id:
|
||||
logger.error(
|
||||
f"[DifyDatasetsConnector] Missing required configuration for ingestion. "
|
||||
f"Config keys: {list(config.keys())}"
|
||||
)
|
||||
return IngestionResult(
|
||||
document_id=doc_id,
|
||||
status=DocumentStatus.FAILED,
|
||||
error_message="Missing required config: dify_apikey or dataset_id.",
|
||||
)
|
||||
|
||||
# Persist config for deletion after worker restart
|
||||
kb_id = context.get_collection_id()
|
||||
await self._save_config(kb_id, config)
|
||||
if await self._load_document(kb_id, doc_id) is not None:
|
||||
raise RuntimeError('Existing upload intent; reconcile before retry')
|
||||
|
||||
# 1. Read file content from Host
|
||||
try:
|
||||
file_bytes = await self.plugin.get_knowledge_file_stream(context.file_object.storage_path)
|
||||
except Exception as e:
|
||||
logger.error(f"[DifyDatasetsConnector] Failed to read file content: {e}")
|
||||
return IngestionResult(
|
||||
document_id=doc_id,
|
||||
status=DocumentStatus.FAILED,
|
||||
error_message=f"Could not read file: {e}",
|
||||
)
|
||||
|
||||
# Intent is durable before remote I/O; an ambiguous timeout cannot create
|
||||
# a second document on retry without explicit upstream reconciliation.
|
||||
await self._save_document(kb_id, doc_id, {'upstream_id': '', 'dataset_id': dataset_id,
|
||||
'status': 'pending'})
|
||||
# 2. Upload to Dify dataset
|
||||
url = f"{api_base_url}/datasets/{dataset_id}/document/create-by-file"
|
||||
headers = {
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
}
|
||||
|
||||
try:
|
||||
data_payload = json.dumps({
|
||||
"indexing_technique": "high_quality",
|
||||
"process_rule": {"mode": "automatic"},
|
||||
})
|
||||
|
||||
async def upload():
|
||||
async with self.http_client() as client:
|
||||
response = await client.post(
|
||||
url,
|
||||
headers=headers,
|
||||
files={"file": (filename, file_bytes)},
|
||||
data={"data": data_payload},
|
||||
timeout=120.0,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
|
||||
resp_data = await self._state.dispatched(
|
||||
self.plugin, kb_id, upload, 'Dify upload outcome unknown')
|
||||
|
||||
dify_document = resp_data.get("document", {})
|
||||
dify_doc_id = dify_document.get("id")
|
||||
if not isinstance(dify_doc_id, str) or not dify_doc_id:
|
||||
raise AmbiguousMutationError("Dify upload omitted upstream document ID; outcome requires reconciliation")
|
||||
await self._save_document(kb_id, doc_id, {'upstream_id': dify_doc_id,
|
||||
'dataset_id': dataset_id, 'status': 'created',
|
||||
'api_base_url': upstream_target})
|
||||
|
||||
logger.info(
|
||||
f"[DifyDatasetsConnector] File uploaded: {filename} -> "
|
||||
f"dify_doc_id={dify_doc_id}, status={dify_document.get('indexing_status')}"
|
||||
)
|
||||
|
||||
return IngestionResult(
|
||||
document_id=dify_doc_id,
|
||||
status=DocumentStatus.PROCESSING,
|
||||
)
|
||||
|
||||
except httpx.HTTPStatusError as e:
|
||||
# The provider answered, so this rejection is deterministic: no fence.
|
||||
error_body = e.response.text
|
||||
logger.error(
|
||||
f"[DifyDatasetsConnector] Dify API error during ingestion: "
|
||||
f"status={e.response.status_code}, body={error_body}"
|
||||
)
|
||||
return IngestionResult(
|
||||
document_id=doc_id,
|
||||
status=DocumentStatus.FAILED,
|
||||
error_message=f"Dify API error {e.response.status_code}: {error_body}",
|
||||
)
|
||||
except AmbiguousMutationError:
|
||||
# Already-dispatched upload with an unknown outcome: fence this KB.
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"[DifyDatasetsConnector] Ingestion failed for {filename}: {e}")
|
||||
return IngestionResult(
|
||||
document_id=doc_id,
|
||||
status=DocumentStatus.FAILED,
|
||||
error_message=str(e),
|
||||
)
|
||||
|
||||
@serialized
|
||||
async def delete_document(self, kb_id: str, document_id: str) -> bool:
|
||||
"""Delete a document from a Dify dataset."""
|
||||
mapping = await self._load_document(kb_id, document_id)
|
||||
if not mapping or not mapping['upstream_id'] or mapping['status'] != 'created':
|
||||
return False
|
||||
config = await self._load_config(kb_id)
|
||||
if not config:
|
||||
logger.warning(
|
||||
f"[DifyDatasetsConnector] No stored config for kb_id={kb_id}. "
|
||||
"Cannot delete document without API credentials."
|
||||
)
|
||||
return False
|
||||
|
||||
api_base_url = config.get("api_base_url", "https://api.dify.ai/v1").rstrip("/")
|
||||
api_key = config.get("dify_apikey")
|
||||
dataset_id = config.get("dataset_id")
|
||||
|
||||
if not api_key or not dataset_id:
|
||||
logger.error(
|
||||
f"[DifyDatasetsConnector] Missing required configuration for deletion. "
|
||||
f"Config keys: {list(config.keys())}"
|
||||
)
|
||||
return False
|
||||
|
||||
if mapping['dataset_id'] != dataset_id:
|
||||
raise RuntimeError('Dataset changed; reconcile before deletion')
|
||||
# The mapping is bound to the upstream instance that created it: a changed
|
||||
# base URL (or a legacy mapping without one) must not be sent to a
|
||||
# different provider. Reject before dispatching any remote request.
|
||||
if mapping.get('api_base_url') != _normalize_api_base_url(api_base_url):
|
||||
raise RuntimeError('Upstream target changed; reconcile before deletion')
|
||||
url = f"{api_base_url}/datasets/{dataset_id}/documents/{mapping['upstream_id']}"
|
||||
headers = {
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
}
|
||||
|
||||
async def remove():
|
||||
async with self.http_client() as client:
|
||||
response = await client.delete(url, headers=headers, timeout=30.0)
|
||||
if response.status_code != 204:
|
||||
# Some Dify versions may return 200 instead of 204.
|
||||
response.raise_for_status()
|
||||
return response.status_code
|
||||
|
||||
try:
|
||||
status_code = await self._state.dispatched(
|
||||
self.plugin, kb_id, remove, 'Dify delete outcome unknown')
|
||||
await self._save_document(kb_id, mapping['host_document_id'], {**mapping, 'status': 'deleted'})
|
||||
logger.info(
|
||||
f"[DifyDatasetsConnector] Document deleted: {document_id} "
|
||||
f"from dataset {dataset_id} (status={status_code})"
|
||||
)
|
||||
return True
|
||||
except httpx.HTTPStatusError as e:
|
||||
# The provider answered, so the delete was rejected deterministically.
|
||||
logger.error(
|
||||
f"[DifyDatasetsConnector] Failed to delete document {document_id}: {e}"
|
||||
)
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"[DifyDatasetsConnector] Failed to delete document {document_id}: {e}"
|
||||
)
|
||||
return False
|
||||
|
||||
@@ -0,0 +1,381 @@
|
||||
"""Installation- and knowledge-base-scoped state using the SDK-bound Host proxy.
|
||||
|
||||
Bundled in each independent archive. No filesystem/global tenant registry.
|
||||
|
||||
Locks and fences are keyed by ``(installation binding, knowledge-base identity)``
|
||||
so an ambiguous failure for one knowledge base never blocks the other knowledge
|
||||
bases that share the same installation; the platform storage key is the same for
|
||||
a whole installation, hence the knowledge-base id is always folded into our keys.
|
||||
Fences are persisted in installation-bound Host storage (key includes the
|
||||
knowledge-base id) with an in-process cache in front, so a restarted worker still
|
||||
recognises them. Only dispatched mutations whose outcome is unknown fence; a
|
||||
deterministic validation error raised before any remote call and ordinary
|
||||
cancellation before a call is dispatched do not.
|
||||
"""
|
||||
import asyncio
|
||||
import hashlib
|
||||
import inspect
|
||||
import json
|
||||
import logging
|
||||
from functools import wraps
|
||||
from contextlib import asynccontextmanager
|
||||
import httpx
|
||||
from weakref import WeakValueDictionary
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
HTTP_TOTAL_TIMEOUT = 150.0
|
||||
FENCE_KEY_PREFIX = "ke.fence.v1."
|
||||
# Fallback fence scope when the wrapped method carries no knowledge-base identity.
|
||||
_INSTALLATION_FENCE_KEY = FENCE_KEY_PREFIX + "installation"
|
||||
_IDENTITY_ARGUMENTS = ('kb_id', 'collection_id', 'knowledge_base_id')
|
||||
|
||||
|
||||
class AmbiguousMutationError(RuntimeError):
|
||||
"""A dispatched mutation's outcome is unknown and requires reconciliation.
|
||||
|
||||
Raise this from a call site when a remote mutation may have taken effect but
|
||||
its result (or its durable record) is unknown. Only these failures fence a
|
||||
knowledge base; deterministic errors raised before any remote call, and
|
||||
ordinary cancellation before a call is dispatched, do not fence.
|
||||
"""
|
||||
|
||||
|
||||
class FenceStateUnavailableError(RuntimeError):
|
||||
"""Persisted fence state could not be read, so the fence cannot be confirmed.
|
||||
|
||||
A failed read is not evidence that no fence exists, so the operation fails
|
||||
closed rather than proceeding as if the knowledge base were clear. The read
|
||||
is retried on the next call instead of being cached as loaded.
|
||||
"""
|
||||
|
||||
|
||||
def is_ambiguous_http_failure(exc):
|
||||
"""Whether a failed provider call had an unknown outcome.
|
||||
|
||||
A response status means the provider answered, so the failure is
|
||||
deterministic. A connection that was never established means nothing was
|
||||
dispatched. Everything else (timeout, lost connection, unparsable response)
|
||||
was dispatched with an unknown outcome.
|
||||
"""
|
||||
if isinstance(exc, httpx.HTTPStatusError):
|
||||
return False
|
||||
if isinstance(exc, (httpx.ConnectError, httpx.ConnectTimeout)):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
async def _settle_commit(task):
|
||||
"""Wait for a dispatched commit and report its outcome.
|
||||
|
||||
Repeated caller cancellation never abandons the wait, so a Host commit that
|
||||
was already dispatched settles before its lock is released. Returns
|
||||
``(caller_cancelled, failure)``; ``failure`` is the commit's exception, or
|
||||
``asyncio.CancelledError`` when the commit itself was cancelled.
|
||||
"""
|
||||
cancelled = False
|
||||
while not task.done():
|
||||
try:
|
||||
await asyncio.shield(task)
|
||||
except asyncio.CancelledError:
|
||||
cancelled = True
|
||||
except Exception:
|
||||
# The commit raised; its outcome is read from the state below.
|
||||
pass
|
||||
if task.cancelled():
|
||||
return cancelled, asyncio.CancelledError()
|
||||
return cancelled, task.exception()
|
||||
|
||||
|
||||
def _kb_identity(method, instance, args, kwargs):
|
||||
"""Extract the knowledge-base identity from a serialized method's arguments.
|
||||
|
||||
Reads a ``kb_id`` / ``collection_id`` / ``knowledge_base_id`` argument first,
|
||||
then an argument exposing ``get_collection_id()`` or ``knowledge_base_id``
|
||||
(ingestion passes the whole context). Returns ``None`` when the wrapped
|
||||
method carries no knowledge-base identity, in which case the caller falls
|
||||
back to an installation-wide lock and fence.
|
||||
"""
|
||||
try:
|
||||
bound = inspect.signature(method).bind_partial(instance, *args, **kwargs)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
for name in _IDENTITY_ARGUMENTS:
|
||||
value = bound.arguments.get(name)
|
||||
if isinstance(value, str) and value:
|
||||
return value
|
||||
for value in bound.arguments.values():
|
||||
getter = getattr(value, 'get_collection_id', None)
|
||||
if callable(getter):
|
||||
try:
|
||||
identity = getter()
|
||||
except Exception:
|
||||
continue
|
||||
if isinstance(identity, str) and identity:
|
||||
return identity
|
||||
identity = getattr(value, 'knowledge_base_id', None)
|
||||
if isinstance(identity, str) and identity:
|
||||
return identity
|
||||
return None
|
||||
|
||||
|
||||
def _kb_fence_key(kb_identity):
|
||||
if kb_identity is None:
|
||||
return _INSTALLATION_FENCE_KEY
|
||||
return FENCE_KEY_PREFIX + hashlib.sha256(kb_identity.encode()).hexdigest()
|
||||
|
||||
|
||||
class SerialState:
|
||||
def __init__(self):
|
||||
self._locks = WeakValueDictionary()
|
||||
# binding -> set of fence keys; cache in front of persisted fences.
|
||||
self._fenced = {}
|
||||
self._fence_loaded = set()
|
||||
|
||||
@staticmethod
|
||||
def binding(plugin):
|
||||
try:
|
||||
binding = plugin.get_installation_binding()
|
||||
except Exception:
|
||||
# Revocation has no invocation context; fall back to dedicated scope.
|
||||
binding = None
|
||||
return binding if binding is not None else ('dedicated',)
|
||||
|
||||
@staticmethod
|
||||
def _scope(binding, kb_identity):
|
||||
return (binding, kb_identity)
|
||||
|
||||
def _fenced_keys(self, binding):
|
||||
return self._fenced.setdefault(binding, set())
|
||||
|
||||
def is_fenced(self, binding, kb_identity):
|
||||
fenced = self._fenced_keys(binding)
|
||||
return (
|
||||
_kb_fence_key(kb_identity) in fenced
|
||||
or _INSTALLATION_FENCE_KEY in fenced
|
||||
)
|
||||
|
||||
async def _load_fences(self, plugin, binding):
|
||||
"""Read persisted fences once per binding.
|
||||
|
||||
A failed read cannot be treated as "no fence": it means the fence state
|
||||
is unknown, so the caller fails closed and retries on the next call
|
||||
rather than caching the failure as loaded.
|
||||
"""
|
||||
if binding in self._fence_loaded:
|
||||
return
|
||||
try:
|
||||
keys = await plugin.get_plugin_storage_keys()
|
||||
fenced = self._fenced_keys(binding)
|
||||
for key in keys:
|
||||
if not key.startswith(FENCE_KEY_PREFIX):
|
||||
continue
|
||||
try:
|
||||
marker = json.loads(await plugin.get_plugin_storage(key))
|
||||
except Exception:
|
||||
# Unreadable marker: fail closed on this knowledge base.
|
||||
marker = {'reason': 'unreadable fence record'}
|
||||
if marker is not None:
|
||||
fenced.add(key)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
logger.warning("Could not read persisted fences for %r", binding, exc_info=True)
|
||||
raise FenceStateUnavailableError(
|
||||
f"Could not confirm persisted fence state for {binding!r}: {exc!r}"
|
||||
) from exc
|
||||
self._fence_loaded.add(binding)
|
||||
|
||||
async def run(self, plugin, kb_identity, operation, *, allow_fenced=False):
|
||||
binding = self.binding(plugin)
|
||||
key = self._scope(binding, kb_identity)
|
||||
lock = self._locks.get(key)
|
||||
if lock is None:
|
||||
lock = self._locks[key] = asyncio.Lock()
|
||||
async with lock:
|
||||
await self._load_fences(plugin, binding)
|
||||
if not allow_fenced and self.is_fenced(binding, kb_identity):
|
||||
raise RuntimeError(
|
||||
f"Knowledge base {kb_identity!r} fenced after an ambiguous mutation; "
|
||||
"reconcile with the provider before retrying"
|
||||
)
|
||||
try:
|
||||
return await operation()
|
||||
except AmbiguousMutationError:
|
||||
await self.fence(plugin, kb_identity, 'ambiguous remote mutation')
|
||||
raise
|
||||
|
||||
async def dispatched(self, plugin, kb_identity, operation, reason):
|
||||
"""Dispatch a remote mutation under a pre-persisted intent.
|
||||
|
||||
The fence is written before the request goes out, so a cancellation after
|
||||
dispatch (or an ambiguous failure) cannot leave a mutation with no marker
|
||||
for reconciliation; a deterministic answer clears it again.
|
||||
"""
|
||||
await self.fence(plugin, kb_identity, reason)
|
||||
try:
|
||||
result = await operation()
|
||||
except asyncio.CancelledError:
|
||||
# Dispatched, outcome unknown: keep the fence and let the caller see it.
|
||||
raise
|
||||
except Exception as exc:
|
||||
if not is_ambiguous_http_failure(exc):
|
||||
await self.clear_fence(plugin, kb_identity)
|
||||
raise
|
||||
await self.clear_fence(plugin, kb_identity)
|
||||
return result
|
||||
|
||||
async def fence(self, plugin, kb_identity, reason):
|
||||
"""Mark one knowledge base as requiring reconciliation.
|
||||
|
||||
The in-process marker is set synchronously; persisting it to Host storage
|
||||
is best effort so that a caller cancellation cannot lose the fence.
|
||||
"""
|
||||
binding = self.binding(plugin)
|
||||
fence_key = _kb_fence_key(kb_identity)
|
||||
self._fenced_keys(binding).add(fence_key)
|
||||
payload = json.dumps({'kb_id': kb_identity, 'reason': reason}).encode()
|
||||
cancelled, failure = await _settle_commit(
|
||||
asyncio.create_task(plugin.set_plugin_storage(fence_key, payload))
|
||||
)
|
||||
if failure is not None:
|
||||
logger.warning("Could not persist fence for %r: %r", kb_identity, failure)
|
||||
if cancelled:
|
||||
raise asyncio.CancelledError
|
||||
|
||||
async def clear_fence(self, plugin, kb_identity):
|
||||
"""Drop the fence for one knowledge base, in process and in storage.
|
||||
|
||||
Used when a knowledge base is deleted: its pending ambiguity is moot, so
|
||||
the operator path out of a fence is deletion. A lingering persisted
|
||||
marker is inert because the configuration tombstone is already written.
|
||||
"""
|
||||
binding = self.binding(plugin)
|
||||
fence_key = _kb_fence_key(kb_identity)
|
||||
self._fenced_keys(binding).discard(fence_key)
|
||||
cancelled, failure = await _settle_commit(
|
||||
asyncio.create_task(plugin.set_plugin_storage(fence_key, b'null'))
|
||||
)
|
||||
if failure is not None:
|
||||
logger.warning("Could not clear fence for %r: %r", kb_identity, failure)
|
||||
if cancelled:
|
||||
raise asyncio.CancelledError
|
||||
|
||||
async def write(self, plugin, kb_identity, key, value):
|
||||
# A caller cancellation must not abandon a dispatched Host commit while
|
||||
# its lock is released; settle it and fence only a real ambiguity.
|
||||
task = asyncio.create_task(plugin.set_plugin_storage(key, value))
|
||||
cancelled, failure = await _settle_commit(task)
|
||||
if failure is not None:
|
||||
await self.fence(plugin, kb_identity, f'Host write outcome unknown: {failure!r}')
|
||||
if cancelled:
|
||||
raise asyncio.CancelledError
|
||||
if failure is not None:
|
||||
raise failure
|
||||
return task.result()
|
||||
|
||||
def release_binding(self, binding):
|
||||
"""Drop process-local state keyed by one installation binding."""
|
||||
for key in [k for k in self._locks if k[0] == binding]:
|
||||
self._locks.pop(key, None)
|
||||
self._fenced.pop(binding, None)
|
||||
self._fence_loaded.discard(binding)
|
||||
|
||||
|
||||
def serialized(method=None, *, allow_fenced=False):
|
||||
def decorate(func):
|
||||
@wraps(func)
|
||||
async def call(self, *args, **kwargs):
|
||||
return await self._state.run(
|
||||
self.plugin,
|
||||
_kb_identity(func, self, args, kwargs),
|
||||
lambda: func(self, *args, **kwargs),
|
||||
allow_fenced=allow_fenced,
|
||||
)
|
||||
return call
|
||||
return decorate if method is None else decorate(method)
|
||||
|
||||
|
||||
class ConfigStore:
|
||||
"""No credential cache: every delete resolves durable installation storage.
|
||||
|
||||
A JSON null tombstone makes missing KBs explicit without ambiguous delete RPCs.
|
||||
The one mutation lock also orders ingest/delete/create inside this worker.
|
||||
"""
|
||||
def __init__(self, *args, **kwargs):
|
||||
super().__init__(*args, **kwargs)
|
||||
self._state = SerialState()
|
||||
self._http_slots = asyncio.Semaphore(4)
|
||||
|
||||
async def initialize(self) -> None:
|
||||
await super().initialize()
|
||||
# The runtime dispatches installation revocation to the plugin instance,
|
||||
# which holds no back-reference to its components; hand it this
|
||||
# component's state so the revocation hook can drop per-binding locks and
|
||||
# fences. Without the hook (langbot-plugin 0.7.4) this attribute is simply
|
||||
# never read.
|
||||
plugin = getattr(self, 'plugin', None)
|
||||
if plugin is not None:
|
||||
plugin.knowledge_engine_serial_state = self._state
|
||||
|
||||
@asynccontextmanager
|
||||
async def http_client(self):
|
||||
# Include queueing and the whole response body in the deadline, not just
|
||||
# individual socket reads. Clients/credentials never survive a call.
|
||||
async with asyncio.timeout(HTTP_TOTAL_TIMEOUT):
|
||||
async with self._http_slots:
|
||||
async with httpx.AsyncClient() as client:
|
||||
yield client
|
||||
|
||||
@staticmethod
|
||||
def _key(kb_id):
|
||||
return "ke.config.v1." + hashlib.sha256(kb_id.encode()).hexdigest()
|
||||
|
||||
async def _save_config(self, kb_id, config):
|
||||
value = json.dumps(config, ensure_ascii=False).encode()
|
||||
if len(value) > 65536:
|
||||
raise ValueError("Knowledge-base configuration exceeds 64 KiB")
|
||||
await self._state.write(self.plugin, kb_id, self._key(kb_id), value)
|
||||
|
||||
async def _load_config(self, kb_id):
|
||||
key = self._key(kb_id)
|
||||
# Only an authoritative keys response establishes absence; RPC errors propagate.
|
||||
if key not in await self.plugin.get_plugin_storage_keys():
|
||||
return None
|
||||
value = json.loads(await self.plugin.get_plugin_storage(key))
|
||||
if value is not None and not isinstance(value, dict):
|
||||
raise ValueError("Invalid persisted knowledge-base configuration")
|
||||
return value
|
||||
|
||||
@staticmethod
|
||||
def _document_key(kb_id, host_document_id):
|
||||
return "ke.document.v1." + hashlib.sha256(
|
||||
json.dumps([kb_id, host_document_id]).encode()
|
||||
).hexdigest()
|
||||
|
||||
async def _load_document(self, kb_id, host_document_id):
|
||||
key = self._document_key(kb_id, host_document_id)
|
||||
keys = await self.plugin.get_plugin_storage_keys()
|
||||
if key in keys:
|
||||
value = json.loads(await self.plugin.get_plugin_storage(key))
|
||||
else:
|
||||
# Current Host passes the durable upstream ID on delete; older Host
|
||||
# revisions pass its file UUID. Resolve either without trusting an
|
||||
# arbitrary unrecorded identifier.
|
||||
matches = []
|
||||
for candidate in keys:
|
||||
if candidate.startswith('ke.document.v1.'):
|
||||
item = json.loads(await self.plugin.get_plugin_storage(candidate))
|
||||
if item.get('kb_id') == kb_id and item.get('upstream_id') == host_document_id:
|
||||
matches.append(item)
|
||||
if len(matches) > 1:
|
||||
raise ValueError('Ambiguous upstream document ID')
|
||||
return matches[0] if matches else None
|
||||
if not isinstance(value, dict) or not isinstance(value.get('upstream_id'), str):
|
||||
raise ValueError('Invalid upstream document mapping; reconcile installation')
|
||||
return value
|
||||
|
||||
async def _save_document(self, kb_id, host_document_id, value):
|
||||
value = {**value, 'kb_id': kb_id, 'host_document_id': host_document_id}
|
||||
await self._state.write(self.plugin, kb_id, self._document_key(kb_id, host_document_id),
|
||||
json.dumps(value).encode())
|
||||
@@ -0,0 +1,30 @@
|
||||
# Auto generated by LangBot Plugin SDK.
|
||||
# Please refer to https://langbot.app/docs/en/plugin/dev/tutor.html for more details.
|
||||
from __future__ import annotations
|
||||
|
||||
from langbot_plugin.api.definition.plugin import BasePlugin
|
||||
|
||||
class DifyDatasetsConnector(BasePlugin):
|
||||
|
||||
async def initialize(self) -> None:
|
||||
# Will be called when plugin is launching
|
||||
pass
|
||||
|
||||
async def on_installation_revoked(self, binding) -> None:
|
||||
"""Release process-local state keyed by this binding.
|
||||
|
||||
Dispatched by SDK revisions that implement installation revocation
|
||||
(``BasePlugin.on_installation_revoked``); on langbot-plugin 0.7.4 the
|
||||
hook does not exist and is never called, so older SDKs are unaffected.
|
||||
There is no invocation context, so Host APIs are unavailable: only the
|
||||
in-process lock table and fence cache for this binding are dropped.
|
||||
Persisted fences stay in Host storage and are re-read on the next
|
||||
invocation.
|
||||
"""
|
||||
state = getattr(self, 'knowledge_engine_serial_state', None)
|
||||
if state is not None:
|
||||
state.release_binding(binding)
|
||||
|
||||
def __del__(self) -> None:
|
||||
# Will be called when plugin is terminating
|
||||
pass
|
||||
@@ -0,0 +1,39 @@
|
||||
apiVersion: v1
|
||||
kind: Plugin
|
||||
metadata:
|
||||
author: langbot-team
|
||||
name: DifyDatasetsConnector
|
||||
repository: https://github.com/langbot-app/LangBot/tree/master/plugins/KnowledgeEngine/DifyDatasetsConnector
|
||||
version: 0.2.2
|
||||
description:
|
||||
en_US: Retrieve knowledge from or store files into Dify knowledge bases
|
||||
zh_Hans: 从 Dify 知识库中检索知识或向其中存入文件
|
||||
ja_JP: Difyナレッジベースコネクタプラグイン。Difyのナレッジベース検索をLangBot Toolとして公開します。
|
||||
zh_Hant: Dify 知識庫連接器插件，將 Dify 知識庫檢索暴露為 LangBot Tool。
|
||||
vi_VN: Plugin kết nối cơ sở tri thức Dify, cung cấp khả năng truy xuất cơ sở tri
|
||||
thức Dify dưới dạng LangBot Tool.
|
||||
th_TH: ปลั๊กอินเชื่อมต่อฐานความรู้ Dify เปิดการค้นหาฐานความรู้ Dify เป็น LangBot
|
||||
Tool
|
||||
es_ES: Plugin conector de bases de conocimiento Dify, que expone la búsqueda de
|
||||
conocimiento de Dify como una herramienta LangBot Tool.
|
||||
label:
|
||||
en_US: DifyKnowledgeBase
|
||||
zh_Hans: Dify 知识库
|
||||
ja_JP: Dify 知識ベース
|
||||
zh_Hant: DifyKnowledgeBase
|
||||
vi_VN: DifyKnowledgeBase
|
||||
th_TH: DifyKnowledgeBase
|
||||
es_ES: DifyKnowledgeBase
|
||||
icon: assets/icon.svg
|
||||
spec:
|
||||
config: []
|
||||
components:
|
||||
KnowledgeEngine:
|
||||
fromFiles:
|
||||
- components/knowledge_engine/dify.yaml
|
||||
execution:
|
||||
python:
|
||||
path: main.py
|
||||
attr: DifyDatasetsConnector
|
||||
sharedRuntime: shared-runtime-v1
|
||||
componentModel: stateless-v1
|
||||
@@ -0,0 +1,58 @@
|
||||
# DifyDatasetsConnector
|
||||
|
||||
Recupere conocimientos o almacene archivos en las bases de conocimientos de Dify utilizando la API de Dify.
|
||||
|
||||
## Configuración
|
||||
|
||||
Agregue una base de conocimientos externa en LangBot y seleccione "DifyDatasetsConnector" como tipo de recuperador de conocimientos.
|
||||
|
||||
### Configuración de creación (establecida al crear una base de conocimientos)
|
||||
|
||||
-**api_base_url**: URL base para la API de Dify
|
||||
- Para Dify Cloud:`https://api.dify.ai/v1`(predeterminado)
|
||||
- Para instancias autohospedadas: la URL de su servidor (por ejemplo,`http://localhost/api`o`https://your-domain.com/api`)
|
||||
-**dify_apikey**: Su clave API de Dify de su instancia de Dify
|
||||
-**dataset_id**: el ID de su base de conocimientos/conjunto de datos de Dify
|
||||
|
||||
### Configuración de recuperación (ajustable por consulta)
|
||||
|
||||
-**search_method**(predeterminado: semantic_search): el método de búsqueda a utilizar
|
||||
-`keyword_search`: búsqueda basada en palabras clave
|
||||
-`semantic_search`: búsqueda de similitud semántica (predeterminado)
|
||||
-`full_text_search`: búsqueda de texto completo
|
||||
-`hybrid_search`: búsqueda híbrida que combina semántica y texto completo
|
||||
-**top_k**(predeterminado: 5): número máximo de resultados recuperados
|
||||
-**score_threshold_enabled**(predeterminado: falso): si se habilita el filtrado de umbral de puntuación
|
||||
-**score_threshold**(predeterminado: 0,5): puntuación de relevancia mínima (0-1), solo se muestra cuando el umbral de puntuación está habilitado
|
||||
-**reranking_enable**(predeterminado: falso): habilita la reclasificación para mejorar la calidad de los resultados. El modelo de reclasificación se obtiene automáticamente de la configuración de su conjunto de datos de Dify; primero configure el modelo de reclasificación en la consola de Dify.
|
||||
|
||||
## Cómo obtener valores de configuración
|
||||
|
||||
### Obteniendo su clave API de Dify
|
||||
|
||||
1. Vaya a https://cloud.dify.ai/
|
||||
2. Navegue a la página de su base de conocimientos.
|
||||
3. Haga clic en "ACCESO API" en la barra lateral izquierda
|
||||
4. Cree o copie su clave API desde la sección "Claves API"
|
||||
|
||||
### Obteniendo su ID de conjunto de datos
|
||||
|
||||
1. En la lista de la base de conocimientos de Dify, haga clic en su base de conocimientos.
|
||||
2. El ID del conjunto de datos está en la URL:`https://cloud.dify.ai/datasets/{dataset_id}`
|
||||
3. O puede encontrarlo en la página de documentación de API de su base de conocimientos.
|
||||
|
||||
### Configuración de reclasificación
|
||||
|
||||
1. En la consola Dify, vaya a la configuración de su conjunto de datos.
|
||||
2. Habilite la reclasificación y seleccione un modelo de reclasificación (por ejemplo,`cohere/rerank-v3.5`)
|
||||
3. Guarde la configuración
|
||||
4. En LangBot, habilite la opción "Habilitar reclasificación": el complemento utilizará automáticamente el modelo configurado en Dify.
|
||||
|
||||
## Referencia de API
|
||||
|
||||
Este complemento utiliza la API Dify Dataset:
|
||||
- Recuperación:`POST /v1/datasets/{dataset_id}/retrieve`
|
||||
- Información del conjunto de datos:`GET /v1/datasets/{dataset_id}`
|
||||
- Carga de documentos:`POST /v1/datasets/{dataset_id}/document/create-by-file`
|
||||
- Eliminación de documentos:`DELETE /v1/datasets/{dataset_id}/documents/{document_id}`
|
||||
- Documentación: https://docs.dify.ai/
|
||||
@@ -0,0 +1,58 @@
|
||||
# DifyDatasetsConnector
|
||||
|
||||
Dify APIを使用してDifyナレッジベースから知識を取得またはファイルを保存します。
|
||||
|
||||
## 設定
|
||||
|
||||
LangBotで外部ナレッジベースを追加し、ナレッジリトリーバータイプとして「DifyDatasetsConnector」を選択してください。
|
||||
|
||||
### 作成設定（ナレッジベース作成時に設定）
|
||||
|
||||
- **api_base_url**：Dify APIのベースURL
|
||||
- Dify Cloudの場合：`https://api.dify.ai/v1`（デフォルト）
|
||||
- セルフホストインスタンスの場合：サーバーのURL（例：`http://localhost/api` または `https://your-domain.com/api`）
|
||||
- **dify_apikey**：DifyインスタンスのAPIキー
|
||||
- **dataset_id**：Difyナレッジベース/データセットのID
|
||||
|
||||
### 検索設定（クエリごとに調整可能）
|
||||
|
||||
- **search_method**（デフォルト：semantic_search）：使用する検索方法
|
||||
- `keyword_search`：キーワードベースの検索
|
||||
- `semantic_search`：セマンティック類似度検索（デフォルト）
|
||||
- `full_text_search`：全文検索
|
||||
- `hybrid_search`：ハイブリッド検索（セマンティック検索と全文検索の組み合わせ）
|
||||
- **top_k**（デフォルト：5）：取得する結果の最大数
|
||||
- **score_threshold_enabled**（デフォルト：オフ）：スコア閾値フィルタリングを有効にするかどうか
|
||||
- **score_threshold**（デフォルト：0.5）：最小関連度スコア（0-1）、スコア閾値が有効な場合のみ表示
|
||||
- **reranking_enable**（デフォルト：オフ）：リランキングを有効にして結果の品質を向上。リランキングモデルはDifyデータセット設定から自動的に取得されます。先にDifyコンソールでリランキングモデルを設定してください
|
||||
|
||||
## 設定値の取得方法
|
||||
|
||||
### Dify APIキーの取得
|
||||
|
||||
1. https://cloud.dify.ai/ にアクセス
|
||||
2. ナレッジベースページに移動
|
||||
3. 左サイドバーの「API ACCESS」をクリック
|
||||
4. 「API Keys」セクションからAPIキーを作成またはコピー
|
||||
|
||||
### データセットIDの取得
|
||||
|
||||
1. Difyナレッジベース一覧でナレッジベースをクリック
|
||||
2. データセットIDはURLに含まれています：`https://cloud.dify.ai/datasets/{dataset_id}`
|
||||
3. またはナレッジベースのAPIドキュメントページで確認できます
|
||||
|
||||
### リランキングの設定
|
||||
|
||||
1. Difyコンソールでデータセット設定に移動
|
||||
2. リランキングを有効にし、リランキングモデルを選択（例：`cohere/rerank-v3.5`）
|
||||
3. 設定を保存
|
||||
4. LangBotで「リランキングを有効化」トグルをオンにすると、プラグインがDifyで設定されたモデルを自動的に使用します
|
||||
|
||||
## APIリファレンス
|
||||
|
||||
このプラグインはDify Dataset APIを使用します：
|
||||
- 検索：`POST /v1/datasets/{dataset_id}/retrieve`
|
||||
- データセット情報：`GET /v1/datasets/{dataset_id}`
|
||||
- ドキュメントアップロード：`POST /v1/datasets/{dataset_id}/document/create-by-file`
|
||||
- ドキュメント削除：`DELETE /v1/datasets/{dataset_id}/documents/{document_id}`
|
||||
- ドキュメント：https://docs.dify.ai/
|
||||
@@ -0,0 +1,58 @@
|
||||
# DifyDatasetsConnector
|
||||
|
||||
ดึงความรู้จากหรือจัดเก็บไฟล์ลงในฐานความรู้ของ Dify โดยใช้ Dify API
|
||||
|
||||
## การกำหนดค่า
|
||||
|
||||
โปรดเพิ่มฐานความรู้ภายนอกใน LangBot และเลือก "DifyDatasetsConnector" เป็นประเภทการดึงความรู้
|
||||
|
||||
### การตั้งค่าการสร้าง (ตั้งค่าเมื่อสร้างฐานความรู้)
|
||||
|
||||
-**api_base_url**: URL ฐานสำหรับ Dify API
|
||||
- สำหรับ Dify Cloud:`https://api.dify.ai/v1`(ค่าเริ่มต้น)
|
||||
- สำหรับอินสแตนซ์ที่โฮสต์เอง: URL เซิร์ฟเวอร์ของคุณ (เช่น`http://localhost/api`หรือ`https://your-domain.com/api`)
|
||||
-**dify_apikey**: คีย์ Dify API ของคุณจากอินสแตนซ์ Dify ของคุณ
|
||||
-**dataset_id**: ID ของฐานความรู้/ชุดข้อมูล Dify ของคุณ
|
||||
|
||||
### การตั้งค่าการดึงข้อมูล (ปรับได้ตามแบบสอบถาม)
|
||||
|
||||
-**search_method**(ค่าเริ่มต้น: semantic_search): วิธีค้นหาที่จะใช้
|
||||
-`keyword_search`: การค้นหาตามคำหลัก
|
||||
-`semantic_search`: การค้นหาความคล้ายคลึงทางความหมาย (ค่าเริ่มต้น)
|
||||
-`full_text_search`: ค้นหาข้อความแบบเต็ม
|
||||
-`hybrid_search`: การค้นหาแบบไฮบริดที่รวมความหมายและข้อความแบบเต็ม
|
||||
-**top_k**(ค่าเริ่มต้น: 5): จำนวนผลลัพธ์ที่ดึงได้สูงสุด
|
||||
-**score_threshold_enabled**(ค่าเริ่มต้น: false): ว่าจะเปิดใช้งานการกรองเกณฑ์คะแนนหรือไม่
|
||||
-**score_threshold**(ค่าเริ่มต้น: 0.5): คะแนนความเกี่ยวข้องขั้นต่ำ (0-1) จะแสดงเฉพาะเมื่อมีการเปิดใช้งานเกณฑ์คะแนน
|
||||
-**reranking_enable**(ค่าเริ่มต้น: false): เปิดใช้งานการจัดอันดับใหม่เพื่อปรับปรุงคุณภาพผลลัพธ์ โมเดลการจัดอันดับใหม่จะถูกดึงมาจากการตั้งค่าชุดข้อมูล Dify ของคุณโดยอัตโนมัติ โปรดกำหนดค่าโมเดลการจัดอันดับใหม่ในคอนโซล Dify ก่อน
|
||||
|
||||
## วิธีรับค่าการกำหนดค่า
|
||||
|
||||
### รับคีย์ Dify API ของคุณ
|
||||
|
||||
1. ไปที่ https://cloud.dify.ai/
|
||||
2. ไปที่หน้าฐานความรู้ของคุณ
|
||||
3. คลิกที่ "API ACCESS" ในแถบด้านข้างซ้าย
|
||||
4. สร้างหรือคัดลอกคีย์ API ของคุณจากส่วน "คีย์ API"
|
||||
|
||||
### รับรหัสชุดข้อมูลของคุณ
|
||||
|
||||
1. ในรายการฐานความรู้ Dify ให้คลิกที่ฐานความรู้ของคุณ
|
||||
2. รหัสชุดข้อมูลอยู่ใน URL:`https://cloud.dify.ai/datasets/{dataset_id}`
|
||||
3. หรือคุณสามารถค้นหาได้ในหน้าเอกสารประกอบ API ของฐานความรู้ของคุณ
|
||||
|
||||
### การกำหนดค่าการจัดอันดับใหม่
|
||||
|
||||
1. ในคอนโซล Dify ให้ไปที่การตั้งค่าชุดข้อมูลของคุณ
|
||||
2. เปิดใช้งานการจัดอันดับใหม่และเลือกโมเดลการจัดอันดับใหม่ (เช่น`cohere/rerank-v3.5`)
|
||||
3. บันทึกการตั้งค่า
|
||||
4. ใน LangBot เปิดใช้งานการสลับ "เปิดใช้งานการจัดอันดับใหม่" — ปลั๊กอินจะใช้โมเดลที่กำหนดค่าใน Dify โดยอัตโนมัติ
|
||||
|
||||
## การอ้างอิง API
|
||||
|
||||
ปลั๊กอินนี้ใช้ Dify Dataset API:
|
||||
- การดึงข้อมูล:`POST /v1/datasets/{dataset_id}/retrieve`
|
||||
- ข้อมูลชุดข้อมูล:`GET /v1/datasets/{dataset_id}`
|
||||
- การอัปโหลดเอกสาร:`POST /v1/datasets/{dataset_id}/document/create-by-file`
|
||||
- ลบเอกสาร:`DELETE /v1/datasets/{dataset_id}/documents/{document_id}`
|
||||
- เอกสารประกอบ: https://docs.dify.ai/
|
||||
@@ -0,0 +1,58 @@
|
||||
#DifyDatasetsConnector
|
||||
|
||||
Truy xuất kiến thức từ hoặc lưu trữ tệp vào cơ sở kiến thức Dify bằng API Dify.
|
||||
|
||||
## Cấu hình
|
||||
|
||||
Vui lòng thêm cơ sở kiến thức bên ngoài vào LangBot và chọn "DifyDatasetsConnector" làm loại trình truy xuất kiến thức.
|
||||
|
||||
### Cài đặt tạo (được đặt khi tạo cơ sở kiến thức)
|
||||
|
||||
-**api_base_url**: URL cơ sở cho API Dify
|
||||
- Đối với Dify Cloud:`https://api.dify.ai/v1`(mặc định)
|
||||
- Đối với các phiên bản tự lưu trữ: URL máy chủ của bạn (ví dụ:`http://localhost/api`hoặc`https://your-domain.com/api`)
|
||||
-**dify_apikey**: Khóa API Dify từ phiên bản Dify của bạn
|
||||
-**dataset_id**: ID của cơ sở dữ liệu/cơ sở kiến thức Dify của bạn
|
||||
|
||||
### Cài đặt truy xuất (có thể điều chỉnh cho mỗi truy vấn)
|
||||
|
||||
-**search_method**(mặc định: ngữ nghĩa_search): Phương thức tìm kiếm sẽ sử dụng
|
||||
-`keyword_search`: Tìm kiếm dựa trên từ khóa
|
||||
-`semantic_search`: Tìm kiếm tương tự về mặt ngữ nghĩa (mặc định)
|
||||
-`full_text_search`: Tìm kiếm toàn văn
|
||||
-`hybrid_search`: Tìm kiếm kết hợp ngữ nghĩa và toàn văn
|
||||
-**top_k**(mặc định: 5): Số lượng kết quả truy xuất tối đa
|
||||
-**score_threshold_enabled**(mặc định: false): Có bật tính năng lọc ngưỡng điểm hay không
|
||||
-**score_threshold**(mặc định: 0,5): Điểm phù hợp tối thiểu (0-1), chỉ hiển thị khi bật ngưỡng điểm
|
||||
-**reranking_enable**(mặc định: false): Bật tính năng sắp xếp lại để cải thiện chất lượng kết quả. Mô hình sắp xếp lại được tự động tìm nạp từ cài đặt tập dữ liệu Dify của bạn - trước tiên hãy định cấu hình mô hình sắp xếp lại trong bảng điều khiển Dify
|
||||
|
||||
## Cách lấy giá trị cấu hình
|
||||
|
||||
### Lấy khóa API Dify của bạn
|
||||
|
||||
1. Truy cập https://cloud.dify.ai/
|
||||
2. Điều hướng đến trang cơ sở kiến thức của bạn
|
||||
3. Nhấp vào "TRUY CẬP API" ở thanh bên trái
|
||||
4. Tạo hoặc sao chép khóa API của bạn từ phần "Khóa API"
|
||||
|
||||
### Lấy ID tập dữ liệu của bạn
|
||||
|
||||
1. Trong danh sách cơ sở kiến thức Dify, nhấp vào cơ sở kiến thức của bạn
|
||||
2. ID tập dữ liệu có trong URL:`https://cloud.dify.ai/datasets/{dataset_id}`
|
||||
3. Hoặc bạn có thể tìm thấy nó trong trang tài liệu API trong cơ sở kiến thức của bạn
|
||||
|
||||
### Cấu hình sắp xếp lại
|
||||
|
||||
1. Trong bảng điều khiển Dify, hãy chuyển đến cài đặt tập dữ liệu của bạn
|
||||
2. Kích hoạt tính năng sắp xếp lại và chọn mô hình sắp xếp lại (ví dụ:`cohere/rerank-v3.5`)
|
||||
3. Lưu cài đặt
|
||||
4. Trong LangBot, bật nút chuyển đổi "Bật xếp hạng lại" - plugin sẽ tự động sử dụng mô hình được định cấu hình trong Dify
|
||||
|
||||
## Tham chiếu API
|
||||
|
||||
Plugin này sử dụng API bộ dữ liệu Dify:
|
||||
- Truy xuất:`POST /v1/datasets/{dataset_id}/retrieve`
|
||||
- Thông tin tập dữ liệu:`GET /v1/datasets/{dataset_id}`
|
||||
- Tải lên tài liệu:`POST /v1/datasets/{dataset_id}/document/create-by-file`
|
||||
- Xóa tài liệu:`DELETE /v1/datasets/{dataset_id}/documents/{document_id}`
|
||||
- Tài liệu: https://docs.dify.ai/
|
||||
@@ -0,0 +1,58 @@
|
||||
# DifyDatasetsConnector
|
||||
|
||||
使用 Dify API 从 Dify 知识库中检索知识或向其中存入文件。
|
||||
|
||||
## 配置
|
||||
|
||||
请在 LangBot 中添加外部知识库，并选择"DifyDatasetsConnector"作为知识检索器类型。
|
||||
|
||||
### 创建设置（创建知识库时配置）
|
||||
|
||||
- **api_base_url**：Dify API 的基础 URL
|
||||
- Dify Cloud：`https://api.dify.ai/v1`（默认）
|
||||
- 自托管实例：您的服务器 URL（例如 `http://localhost/api` 或 `https://your-domain.com/api`）
|
||||
- **dify_apikey**：您的 Dify 实例的 API 密钥
|
||||
- **dataset_id**：您的 Dify 知识库/数据集的 ID
|
||||
|
||||
### 检索设置（每次查询可调整）
|
||||
|
||||
- **search_method**（默认：semantic_search）：使用的搜索方法
|
||||
- `keyword_search`：基于关键词的搜索
|
||||
- `semantic_search`：语义相似度搜索（默认）
|
||||
- `full_text_search`：全文搜索
|
||||
- `hybrid_search`：混合搜索，结合语义检索和全文检索
|
||||
- **top_k**（默认：5）：返回的最大检索结果数量
|
||||
- **score_threshold_enabled**（默认：关闭）：是否启用分数阈值过滤
|
||||
- **score_threshold**（默认：0.5）：最低相关度分数（0-1），仅在启用分数阈值时显示
|
||||
- **reranking_enable**（默认：关闭）：启用重排序以提升结果质量。重排序模型将自动从 Dify 数据集设置中获取，请先在 Dify 控制台中配置重排序模型
|
||||
|
||||
## 如何获取配置值
|
||||
|
||||
### 获取您的 Dify API 密钥
|
||||
|
||||
1. 访问 https://cloud.dify.ai/
|
||||
2. 导航到您的知识库页面
|
||||
3. 点击左侧边栏中的"API ACCESS"
|
||||
4. 从"API Keys"部分创建或复制您的 API 密钥
|
||||
|
||||
### 获取您的数据集 ID
|
||||
|
||||
1. 在 Dify 知识库列表中，点击您的知识库
|
||||
2. 数据集 ID 在 URL 中：`https://cloud.dify.ai/datasets/{dataset_id}`
|
||||
3. 或者您可以在知识库的 API 文档页面中找到它
|
||||
|
||||
### 配置重排序
|
||||
|
||||
1. 在 Dify 控制台中，进入数据集设置
|
||||
2. 启用重排序并选择重排序模型（如 `cohere/rerank-v3.5`）
|
||||
3. 保存设置
|
||||
4. 在 LangBot 中开启"启用重排序"开关，插件将自动使用 Dify 中配置的模型
|
||||
|
||||
## API 参考
|
||||
|
||||
本插件使用 Dify Dataset API：
|
||||
- 检索：`POST /v1/datasets/{dataset_id}/retrieve`
|
||||
- 数据集信息：`GET /v1/datasets/{dataset_id}`
|
||||
- 文档上传：`POST /v1/datasets/{dataset_id}/document/create-by-file`
|
||||
- 文档删除：`DELETE /v1/datasets/{dataset_id}/documents/{document_id}`
|
||||
- 文档：https://docs.dify.ai/
|
||||
@@ -0,0 +1,58 @@
|
||||
# DifyDatasets連接器
|
||||
|
||||
使用 Dify API 從 Dify 知識庫檢索知識或將文件儲存到 Dify 知識庫中。
|
||||
|
||||
＃＃ 配置
|
||||
|
||||
請在 LangBot 中新增外部知識庫，並選擇「DifyDatasetsConnector」作為知識擷取器類型。
|
||||
|
||||
### 建立設定（建立知識庫時設定）
|
||||
|
||||
-**api_base_url**：Dify API 的基本 URL
|
||||
- 對於 Dify Cloud：「https://api.dify.ai/v1」（預設）
|
||||
- 對於自架執行個體：您的伺服器 URL（例如「http://localhost/api」或「https://your-domain.com/api」）
|
||||
-**dify_apikey**：您的 Dify 實例中的 Dify API 金鑰
|
||||
-**dataset_id**：您的 Dify 知識庫/資料集的 ID
|
||||
|
||||
### 檢索設定（每個查詢均可調整）
|
||||
|
||||
-**search_method**（預設值：semantic_search）：要使用的搜尋方法
|
||||
-`keyword_search`：基於關鍵字的搜索
|
||||
-`semantic_search`：語意相似性搜尋（預設）
|
||||
-`full_text_search`：全文搜尋
|
||||
-`hybrid_search`：結合語意和全文的混合搜尋
|
||||
-**top_k**（預設值：5）：檢索結果的最大數量
|
||||
-**score_threshold_enabled**(預設: false): 是否啟用分數閾值過濾
|
||||
-**score_threshold**（預設值：0.5）：最小相關性分數（0-1），僅在啟用分數閾值時顯示
|
||||
-**reranking_enable**（預設值：false）：啟用重新排名以提高結果品質。重新排名模型會自動從您的 Dify 資料集設定中取得 - 請先在 Dify 控制台中設定重新排名模型
|
||||
|
||||
## 如何取得設定值
|
||||
|
||||
### 取得您的 Dify API 金鑰
|
||||
|
||||
1. 造訪https://cloud.dify.ai/
|
||||
2. 導覽至您的知識庫頁面
|
||||
3. 點選左側邊欄中的“API 存取”
|
||||
4. 從「API 金鑰」部分建立或複製您的 API 金鑰
|
||||
|
||||
### 取得您的資料集 ID
|
||||
|
||||
1. 在 Dify 知識庫清單中，按一下您的知識庫
|
||||
2. 資料集 ID 在 URL 中：`https://cloud.dify.ai/datasets/{dataset_id}`
|
||||
3.或者您可以在知識庫的API文檔頁面中找到它
|
||||
|
||||
### 配置重新排名
|
||||
|
||||
1. 在 Dify 控制台中，前往您的資料集設置
|
||||
2. 啟用重新排名並選擇重新排名模型（例如“cohere/rerank-v3.5”）
|
||||
3. 儲存設定
|
||||
4. 在 LangBot 中，啟用「啟用重新排名」開關 - 外掛程式將自動使用 Dify 中配置的模型
|
||||
|
||||
## API 參考
|
||||
|
||||
該外掛程式使用 Dify 資料集 API：
|
||||
- 檢索：`POST /v1/datasets/{dataset_id}/retrieve`
|
||||
- 資料集資訊：`GET /v1/datasets/{dataset_id}`
|
||||
- 文件上傳：`POST /v1/datasets/{dataset_id}/document/create-by-file`
|
||||
- 文件刪除：`DELETE /v1/datasets/{dataset_id}/documents/{document_id}`
|
||||
- 文件：https://docs.dify.ai/
|
||||
@@ -0,0 +1,2 @@
|
||||
langbot-plugin>=0.7.4,<0.8
|
||||
httpx
|
||||
Reference in New Issue
Block a user