mirror of
https://github.com/langbot-app/LangBot.git
synced 2026-07-22 12:26:08 +00:00
feat: add supports for dify hitl (#2226)
* feat: Implement workflow form handling for paused workflows - Added module-level storage for pending forms to manage state across sessions. - Introduced functions to set, get, and clear pending forms with expiration handling. - Enhanced DifyServiceAPIRunner to support resuming paused workflows via form actions. - Implemented logic to yield human input requests and display appropriate messages. - Updated workflow submission methods to handle paused states and resume actions. - Ensured proper merging of pending form actions with user inputs for seamless interaction. * feat: Add '_routed_by_rule' variable to form action in Lark and Telegram adapters * feat: Enhance Lark and Telegram adapters with new form handling for paused workflows * feat: Enhance TelegramAdapter to handle form action buttons and message threading * feat: Improve TelegramAdapter message handling with enhanced error management and draft message support * feat: Add the function for formatting human input text to support adapters without rich UI. * feat(dingtalk): implement human input card support and card action handling - Add a new module `card_callback.py` to handle card action button clicks from DingTalk. - Introduce `DingTalkCardActionHandler` to process card action callbacks and extract parameters. - Update `DingTalkAdapter` to manage card state and handle form input through a single card template. - Add configuration for `human_input_card_template_id` in `dingtalk.yaml` to specify the template for human input. - Create a new card template `dingtalk_human_input_card.json` for rendering human input prompts and buttons. * feat(dingtalk): enhance human input card functionality with streaming support and active turn management - Updated the DingTalk card template to enable streaming mode and multi-update configuration. - Removed the obsolete delete_card method from DingTalkClient to streamline card management. - Enhanced DingTalkAdapter to manage active turn cards and accumulated streaming text, ensuring a seamless user experience during human input prompts. - Modified the create_message_card method to utilize existing active cards for resumed workflows, preventing duplication. - Improved the _paint_form_on_card method to update existing cards with human input prompts and buttons dynamically. - Updated the dingtalk_human_input_card.json template to reflect the new streaming capabilities and configuration options. * feat(wecom): implement Dify human input pause handling with button interaction support * feat(qqofficial): implement Dify human input button interaction handling and markdown keyboard support * feat(qqofficial): implement one-click QR binding and enhance localization support * feat(discord): implement Discord form view with button interactions for Dify actions * fix(telegram): correct group chat type check and handle oversized callback data for Telegram actions fix(difysvapi): ensure safe access to remove-think configuration in pipeline settings * feat(dify): add support for chatflow app type and enhance human input handling * feat(telegram): add action title feedback for user selections in Telegram messages * feat(lark): enhance LarkAdapter to store form content for resume notices * feat(dingtalk): update display formatting for card content with HTML line breaks * feat(dingtalk): add feedback functionality to cards with 👍/👎 buttons - Implemented feedback state management for cards, allowing users to provide feedback via thumbs up/down buttons. - Enhanced card rendering to include feedback buttons when appropriate. - Registered feedback listeners to handle feedback events and update card states accordingly. - Updated the card template to support dynamic button rendering for feedback actions. - Improved error handling and logging for feedback actions and card updates. * fix: add Avatar component to dingtalk_human_input_card.json for enhanced user interaction * feat(wecom): add optional source block to interactive template cards for enhanced branding * feat(wecom): add functions for template card action extraction and update, enhance button interaction handling * feat(qqofficial): synchronize passive-reply counter with inbound message sequence * feat(qqofficial): add method to identify invisible form placeholder chunks in messages * feat(dingtalk): add download link for human input card template and enhance dynamic form configuration * feat(telegram): enhance message handling with group stream deletion and form placeholder detection * Add unit tests for DingTalk, Lark, WeComBot, and Dify service API runners - Implement tests for DingTalk adapter helper functions including form content cleaning, input extraction, and completed input lines. - Create unit tests for Lark adapter helper functions focusing on input extraction and completed input lines. - Add tests for WeComBot template card functionalities, including event extraction and payload building for human input. - Enhance Dify service API runner tests to cover human input forms, including input collection, action handling, and form snapshot extraction. * feat: Enhance Telegram and QQ Official adapters with select field handling and form action processing - Added support for select fields in Telegram adapter, including option extraction and callback handling. - Implemented form action processing for Telegram callbacks, improving user interaction feedback. - Introduced new helper functions for building keyboards and resolving select button actions in QQ Official adapter. - Enhanced DifyServiceAPIRunner to handle cumulative streaming responses and improve error handling during workflow resumes. - Added unit tests for new functionalities in Telegram and QQ Official adapters, ensuring robust behavior for select fields and form actions. * feat(lark): add functions for current input definitions and visible form content handling feat(qqofficial): update fallback text handling for non-streaming scenarios feat(difysvapi): enhance form content processing for interactive fields and actions test: add unit tests for Lark and QQ Official adapter functionalities * Add tests for DingTalk adapter content processing and markdown formatting - Updated the assertion in `test_dingtalk_completed_input_lines_include_text_and_select_values` to remove unnecessary markdown formatting. - Added new tests to verify that `_dingtalk_clean_form_content` maintains the order of prompts and completed values in various scenarios. - Introduced `test_dingtalk_card_markdown_preserves_internal_line_breaks` to ensure internal line breaks are correctly converted to HTML line breaks. * feat: Refactor input handling and feedback messages across multiple adapters * feat: Update the human-computer interaction template cards, and optimize the prompt information and content display. * feat: Refactor pending form handling to isolate by bot and pipeline * feat: Enhance error handling and caching for Dify and WeCom interactions * feat: Enhance select input handling and validation in Dify API runner and Telegram adapter * feat: Add missing completed input lines handling in DingTalk adapter * feat: Add pipeline_uuid handling across multiple adapters and update related tests
This commit is contained in:
Dongchuan Fu
@@ -5,6 +5,29 @@ from ... import group
|
||||
from langbot.pkg.utils import importutil
|
||||
|
||||
|
||||
def _decrypt_qqofficial_secret(encrypted_b64: str, key: bytes) -> str:
|
||||
"""Decrypt the AppSecret returned by the QQ Official QR binding endpoint.
|
||||
|
||||
The base64 payload is laid out as `nonce (12 B) | ciphertext | tag (16 B)`.
|
||||
`key` is the 32-byte AES-256 key locally generated when the bind task
|
||||
was created and submitted as `key` to `q.qq.com/lite/create_bind_task`.
|
||||
"""
|
||||
import base64
|
||||
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
|
||||
|
||||
try:
|
||||
raw = base64.b64decode(encrypted_b64)
|
||||
except Exception as exc:
|
||||
raise ValueError('Malformed encrypted credential') from exc
|
||||
if len(key) != 32 or len(raw) <= 28:
|
||||
raise ValueError('Invalid encrypted credential layout')
|
||||
nonce, ciphertext, tag = raw[:12], raw[12:-16], raw[-16:]
|
||||
try:
|
||||
return AESGCM(key).decrypt(nonce, ciphertext + tag, None).decode('utf-8')
|
||||
except Exception as exc:
|
||||
raise ValueError('Failed to decrypt credential') from exc
|
||||
|
||||
|
||||
@group.group_class('adapters', '/api/v1/platform/adapters')
|
||||
class AdaptersRouterGroup(group.RouterGroup):
|
||||
async def initialize(self) -> None:
|
||||
@@ -37,6 +60,15 @@ class AdaptersRouterGroup(group.RouterGroup):
|
||||
importutil.read_resource_file_bytes(icon_path), mimetype=mimetypes.guess_type(icon_path)[0]
|
||||
)
|
||||
|
||||
@self.route('/dingtalk/human-input-card-template', methods=['GET'], auth_type=group.AuthType.NONE)
|
||||
async def _() -> quart.Response:
|
||||
filename = 'dingtalk_human_input_card.json'
|
||||
response = quart.Response(
|
||||
importutil.read_resource_file_bytes(f'templates/{filename}'), mimetype='application/json'
|
||||
)
|
||||
response.headers['Content-Disposition'] = f'attachment; filename={filename}'
|
||||
return response
|
||||
|
||||
# In-memory session store for active registrations
|
||||
_create_app_sessions: dict = {}
|
||||
_SESSION_TTL = 900 # 15 minutes
|
||||
@@ -650,3 +682,220 @@ class AdaptersRouterGroup(group.RouterGroup):
|
||||
if session and session.get('task') and not session['task'].done():
|
||||
session['task'].cancel()
|
||||
return self.success(data={})
|
||||
|
||||
# -----------------------------------------------------------------------
|
||||
# QQ Official QR Binding
|
||||
# -----------------------------------------------------------------------
|
||||
|
||||
_qqofficial_sessions: dict = {}
|
||||
_QQOFFICIAL_SESSION_TTL = 300 # 5 minutes (QQ bind QR validity window)
|
||||
|
||||
def _cleanup_expired_qqofficial_sessions():
|
||||
import time
|
||||
|
||||
now = time.time()
|
||||
expired = [
|
||||
sid for sid, s in _qqofficial_sessions.items() if now - s.get('created_at', 0) > _QQOFFICIAL_SESSION_TTL
|
||||
]
|
||||
for sid in expired:
|
||||
session = _qqofficial_sessions.pop(sid, None)
|
||||
if session and session.get('task') and not session['task'].done():
|
||||
session['task'].cancel()
|
||||
|
||||
@self.route('/qqofficial/bind', methods=['POST'])
|
||||
async def _() -> str:
|
||||
"""Start QQ Official QR binding. Returns session_id + QR URL.
|
||||
|
||||
Flow: generate a local AES-256 key, register it with
|
||||
`q.qq.com/lite/create_bind_task`, then poll
|
||||
`q.qq.com/lite/poll_bind_result` until the user authorizes the
|
||||
bind inside the QQ Bot Assistant on mobile QQ. The encrypted
|
||||
AppSecret returned by the poll endpoint is decrypted with the
|
||||
same key. The key never leaves this process.
|
||||
"""
|
||||
import uuid
|
||||
import time
|
||||
import secrets
|
||||
import base64
|
||||
import aiohttp
|
||||
|
||||
QQ_BIND_BASE = 'https://q.qq.com'
|
||||
_cleanup_expired_qqofficial_sessions()
|
||||
|
||||
bind_key_bytes = secrets.token_bytes(32)
|
||||
bind_key = base64.b64encode(bind_key_bytes).decode('ascii')
|
||||
|
||||
session_id = str(uuid.uuid4())
|
||||
session = {
|
||||
'status': 'pending',
|
||||
'qr_url': None,
|
||||
'expire_at': None,
|
||||
'appid': None,
|
||||
'secret': None,
|
||||
'user_openid': None,
|
||||
'error': None,
|
||||
'created_at': time.time(),
|
||||
'task_id': None,
|
||||
'bind_key_bytes': bind_key_bytes,
|
||||
'interval': 2,
|
||||
}
|
||||
_qqofficial_sessions[session_id] = session
|
||||
|
||||
async def run_qr_binding():
|
||||
try:
|
||||
timeout = aiohttp.ClientTimeout(total=10)
|
||||
async with aiohttp.ClientSession(timeout=timeout) as http:
|
||||
# Step 1: create_bind_task — register our AES key, get task_id
|
||||
async with http.post(
|
||||
f'{QQ_BIND_BASE}/lite/create_bind_task',
|
||||
json={'key': bind_key},
|
||||
headers={'Accept': 'application/json'},
|
||||
) as resp:
|
||||
try:
|
||||
data = await resp.json(content_type=None)
|
||||
except (aiohttp.ContentTypeError, ValueError):
|
||||
session['status'] = 'error'
|
||||
session['error'] = 'Invalid response from QQ bind service'
|
||||
return
|
||||
if int(data.get('retcode', -1)) != 0:
|
||||
session['status'] = 'error'
|
||||
session['error'] = (
|
||||
data.get('msg') or data.get('message') or 'Failed to create bind task'
|
||||
)
|
||||
return
|
||||
task_id = str((data.get('data') or {}).get('task_id') or '').strip()
|
||||
if not task_id:
|
||||
session['status'] = 'error'
|
||||
session['error'] = 'Missing task_id in QQ response'
|
||||
return
|
||||
|
||||
# The QR encodes a URL that mobile QQ opens inside the QQ Bot Assistant.
|
||||
# `source=langbot` is a courtesy attribution parameter so Tencent
|
||||
# can see LangBot adoption metrics, matching the convention used by
|
||||
# other third-party integrations (e.g. hermes-agent uses `source=hermes`).
|
||||
qr_url = f'{QQ_BIND_BASE}/qqbot/openclaw/connect.html?task_id={task_id}&_wv=2&source=langbot'
|
||||
session['task_id'] = task_id
|
||||
session['qr_url'] = qr_url
|
||||
session['expire_at'] = time.time() + _QQOFFICIAL_SESSION_TTL
|
||||
session['status'] = 'waiting'
|
||||
|
||||
# Step 2: poll_bind_result until completed (status=2) or expired (3).
|
||||
deadline = time.time() + _QQOFFICIAL_SESSION_TTL
|
||||
while time.time() < deadline:
|
||||
await asyncio.sleep(session['interval'])
|
||||
|
||||
async with http.post(
|
||||
f'{QQ_BIND_BASE}/lite/poll_bind_result',
|
||||
json={'task_id': task_id},
|
||||
headers={'Accept': 'application/json'},
|
||||
) as poll_resp:
|
||||
try:
|
||||
poll_data = await poll_resp.json(content_type=None)
|
||||
except (aiohttp.ContentTypeError, ValueError):
|
||||
continue
|
||||
|
||||
if int(poll_data.get('retcode', -1)) != 0:
|
||||
session['status'] = 'error'
|
||||
session['error'] = poll_data.get('msg') or poll_data.get('message') or 'Poll failed'
|
||||
return
|
||||
|
||||
payload = poll_data.get('data') or {}
|
||||
try:
|
||||
raw_status = int(payload.get('status', 0))
|
||||
except (TypeError, ValueError):
|
||||
raw_status = 0
|
||||
|
||||
if raw_status == 2:
|
||||
appid = str(payload.get('bot_appid') or '').strip()
|
||||
encrypted = str(payload.get('bot_encrypt_secret') or '').strip()
|
||||
if not appid or not encrypted:
|
||||
session['status'] = 'error'
|
||||
session['error'] = 'Incomplete credential payload'
|
||||
return
|
||||
try:
|
||||
session['secret'] = _decrypt_qqofficial_secret(
|
||||
encrypted,
|
||||
bind_key_bytes,
|
||||
)
|
||||
except ValueError as exc:
|
||||
session['status'] = 'error'
|
||||
session['error'] = str(exc)
|
||||
return
|
||||
session['appid'] = appid
|
||||
# The scanner's OpenID is returned alongside the credentials —
|
||||
# surfaced to the dashboard for audit / "bound by" display.
|
||||
session['user_openid'] = str(payload.get('user_openid') or '').strip() or None
|
||||
session['status'] = 'success'
|
||||
return
|
||||
|
||||
if raw_status == 3:
|
||||
session['status'] = 'expired'
|
||||
session['error'] = 'QR code expired'
|
||||
return
|
||||
# status 0 / 1: still pending, continue polling
|
||||
|
||||
session['status'] = 'expired'
|
||||
session['error'] = 'QR code expired'
|
||||
|
||||
except asyncio.CancelledError:
|
||||
return
|
||||
except Exception as e:
|
||||
session['status'] = 'error'
|
||||
session['error'] = str(e)
|
||||
|
||||
task = asyncio.create_task(run_qr_binding())
|
||||
session['task'] = task
|
||||
|
||||
# Wait up to 10s for the QR URL to be ready before responding.
|
||||
for _ in range(20):
|
||||
if session['qr_url'] or session['error']:
|
||||
break
|
||||
await asyncio.sleep(0.5)
|
||||
|
||||
if session['error']:
|
||||
task.cancel()
|
||||
return self.http_status(502, -1, session['error'])
|
||||
|
||||
if not session['qr_url']:
|
||||
task.cancel()
|
||||
session['status'] = 'error'
|
||||
session['error'] = 'Timeout waiting for QR code'
|
||||
return self.http_status(504, -1, 'Timeout waiting for QR code')
|
||||
|
||||
return self.success(
|
||||
data={
|
||||
'session_id': session_id,
|
||||
'qr_url': session['qr_url'],
|
||||
'expire_at': session['expire_at'],
|
||||
}
|
||||
)
|
||||
|
||||
@self.route('/qqofficial/bind/status/<session_id>', methods=['GET'])
|
||||
async def _(session_id: str) -> str:
|
||||
"""Poll QQ Official QR binding status."""
|
||||
_cleanup_expired_qqofficial_sessions()
|
||||
session = _qqofficial_sessions.get(session_id)
|
||||
if not session:
|
||||
return self.http_status(404, -1, 'Session not found')
|
||||
|
||||
data = {'status': session['status']}
|
||||
|
||||
if session['status'] == 'success':
|
||||
data['appid'] = session['appid']
|
||||
data['secret'] = session['secret']
|
||||
if session.get('user_openid'):
|
||||
data['user_openid'] = session['user_openid']
|
||||
_qqofficial_sessions.pop(session_id, None)
|
||||
elif session['status'] in ('error', 'expired'):
|
||||
data['error'] = session['error']
|
||||
_qqofficial_sessions.pop(session_id, None)
|
||||
|
||||
return self.success(data=data)
|
||||
|
||||
@self.route('/qqofficial/bind/<session_id>', methods=['DELETE'])
|
||||
async def _(session_id: str) -> str:
|
||||
"""Cancel and clean up a QQ Official QR binding session."""
|
||||
session = _qqofficial_sessions.pop(session_id, None)
|
||||
if session and session.get('task') and not session['task'].done():
|
||||
session['task'].cancel()
|
||||
return self.success(data={})
|
||||
|
||||
Reference in New Issue
Block a user