mirror of
https://github.com/langbot-app/LangBot.git
synced 2026-08-09 04:40:57 +00:00
feat: add supports for dify hitl (#2226)
* feat: Implement workflow form handling for paused workflows - Added module-level storage for pending forms to manage state across sessions. - Introduced functions to set, get, and clear pending forms with expiration handling. - Enhanced DifyServiceAPIRunner to support resuming paused workflows via form actions. - Implemented logic to yield human input requests and display appropriate messages. - Updated workflow submission methods to handle paused states and resume actions. - Ensured proper merging of pending form actions with user inputs for seamless interaction. * feat: Add '_routed_by_rule' variable to form action in Lark and Telegram adapters * feat: Enhance Lark and Telegram adapters with new form handling for paused workflows * feat: Enhance TelegramAdapter to handle form action buttons and message threading * feat: Improve TelegramAdapter message handling with enhanced error management and draft message support * feat: Add the function for formatting human input text to support adapters without rich UI. * feat(dingtalk): implement human input card support and card action handling - Add a new module `card_callback.py` to handle card action button clicks from DingTalk. - Introduce `DingTalkCardActionHandler` to process card action callbacks and extract parameters. - Update `DingTalkAdapter` to manage card state and handle form input through a single card template. - Add configuration for `human_input_card_template_id` in `dingtalk.yaml` to specify the template for human input. - Create a new card template `dingtalk_human_input_card.json` for rendering human input prompts and buttons. * feat(dingtalk): enhance human input card functionality with streaming support and active turn management - Updated the DingTalk card template to enable streaming mode and multi-update configuration. - Removed the obsolete delete_card method from DingTalkClient to streamline card management. - Enhanced DingTalkAdapter to manage active turn cards and accumulated streaming text, ensuring a seamless user experience during human input prompts. - Modified the create_message_card method to utilize existing active cards for resumed workflows, preventing duplication. - Improved the _paint_form_on_card method to update existing cards with human input prompts and buttons dynamically. - Updated the dingtalk_human_input_card.json template to reflect the new streaming capabilities and configuration options. * feat(wecom): implement Dify human input pause handling with button interaction support * feat(qqofficial): implement Dify human input button interaction handling and markdown keyboard support * feat(qqofficial): implement one-click QR binding and enhance localization support * feat(discord): implement Discord form view with button interactions for Dify actions * fix(telegram): correct group chat type check and handle oversized callback data for Telegram actions fix(difysvapi): ensure safe access to remove-think configuration in pipeline settings * feat(dify): add support for chatflow app type and enhance human input handling * feat(telegram): add action title feedback for user selections in Telegram messages * feat(lark): enhance LarkAdapter to store form content for resume notices * feat(dingtalk): update display formatting for card content with HTML line breaks * feat(dingtalk): add feedback functionality to cards with 👍/👎 buttons - Implemented feedback state management for cards, allowing users to provide feedback via thumbs up/down buttons. - Enhanced card rendering to include feedback buttons when appropriate. - Registered feedback listeners to handle feedback events and update card states accordingly. - Updated the card template to support dynamic button rendering for feedback actions. - Improved error handling and logging for feedback actions and card updates. * fix: add Avatar component to dingtalk_human_input_card.json for enhanced user interaction * feat(wecom): add optional source block to interactive template cards for enhanced branding * feat(wecom): add functions for template card action extraction and update, enhance button interaction handling * feat(qqofficial): synchronize passive-reply counter with inbound message sequence * feat(qqofficial): add method to identify invisible form placeholder chunks in messages * feat(dingtalk): add download link for human input card template and enhance dynamic form configuration * feat(telegram): enhance message handling with group stream deletion and form placeholder detection * Add unit tests for DingTalk, Lark, WeComBot, and Dify service API runners - Implement tests for DingTalk adapter helper functions including form content cleaning, input extraction, and completed input lines. - Create unit tests for Lark adapter helper functions focusing on input extraction and completed input lines. - Add tests for WeComBot template card functionalities, including event extraction and payload building for human input. - Enhance Dify service API runner tests to cover human input forms, including input collection, action handling, and form snapshot extraction. * feat: Enhance Telegram and QQ Official adapters with select field handling and form action processing - Added support for select fields in Telegram adapter, including option extraction and callback handling. - Implemented form action processing for Telegram callbacks, improving user interaction feedback. - Introduced new helper functions for building keyboards and resolving select button actions in QQ Official adapter. - Enhanced DifyServiceAPIRunner to handle cumulative streaming responses and improve error handling during workflow resumes. - Added unit tests for new functionalities in Telegram and QQ Official adapters, ensuring robust behavior for select fields and form actions. * feat(lark): add functions for current input definitions and visible form content handling feat(qqofficial): update fallback text handling for non-streaming scenarios feat(difysvapi): enhance form content processing for interactive fields and actions test: add unit tests for Lark and QQ Official adapter functionalities * Add tests for DingTalk adapter content processing and markdown formatting - Updated the assertion in `test_dingtalk_completed_input_lines_include_text_and_select_values` to remove unnecessary markdown formatting. - Added new tests to verify that `_dingtalk_clean_form_content` maintains the order of prompts and completed values in various scenarios. - Introduced `test_dingtalk_card_markdown_preserves_internal_line_breaks` to ensure internal line breaks are correctly converted to HTML line breaks. * feat: Refactor input handling and feedback messages across multiple adapters * feat: Update the human-computer interaction template cards, and optimize the prompt information and content display. * feat: Refactor pending form handling to isolate by bot and pipeline * feat: Enhance error handling and caching for Dify and WeCom interactions * feat: Enhance select input handling and validation in Dify API runner and Telegram adapter * feat: Add missing completed input lines handling in DingTalk adapter * feat: Add pipeline_uuid handling across multiple adapters and update related tests
This commit is contained in:
Dongchuan Fu
@@ -12,6 +12,142 @@ import traceback
|
||||
from cryptography.hazmat.primitives.asymmetric import ed25519
|
||||
|
||||
|
||||
QQ_SELECT_ACTION_PREFIX = '__langbot_select__:'
|
||||
|
||||
|
||||
def get_select_field_options(form_data: dict) -> tuple[str, list[str]]:
|
||||
"""Return the active select field name and its display/submission values."""
|
||||
field_name = str(form_data.get('_current_input_field') or '').strip()
|
||||
if not field_name:
|
||||
return '', []
|
||||
|
||||
field = next(
|
||||
(
|
||||
item
|
||||
for item in form_data.get('input_defs') or []
|
||||
if str(item.get('output_variable_name') or '').strip() == field_name
|
||||
),
|
||||
None,
|
||||
)
|
||||
if not field or str(field.get('type') or '').strip().lower() != 'select':
|
||||
return '', []
|
||||
|
||||
source = field.get('option_source') or {}
|
||||
source_value = source.get('value') if isinstance(source, dict) else None
|
||||
if isinstance(source_value, list):
|
||||
return field_name, [str(item) for item in source_value]
|
||||
if isinstance(source_value, str):
|
||||
return field_name, [part.strip() for part in source_value.splitlines() if part.strip()]
|
||||
|
||||
options = field.get('options')
|
||||
if not isinstance(options, list):
|
||||
return field_name, []
|
||||
values = []
|
||||
for item in options:
|
||||
if isinstance(item, dict):
|
||||
values.append(str(item.get('label') or item.get('value') or ''))
|
||||
else:
|
||||
values.append(str(item))
|
||||
return field_name, [value for value in values if value]
|
||||
|
||||
|
||||
def build_keyboard_from_select_field(form_data: dict, *, buttons_per_row: int | None = None) -> dict:
|
||||
"""Build callback buttons for the currently active Dify select field."""
|
||||
_, options = get_select_field_options(form_data)
|
||||
visible_options = options[:25]
|
||||
if buttons_per_row is None:
|
||||
# Keep small choices readable while fitting up to QQ's 5x5 limit.
|
||||
buttons_per_row = min(5, max(2, (len(visible_options) + 4) // 5))
|
||||
selection_actions = [
|
||||
{
|
||||
'id': f'{QQ_SELECT_ACTION_PREFIX}{idx}',
|
||||
'title': option,
|
||||
'button_style': 'secondary',
|
||||
}
|
||||
for idx, option in enumerate(visible_options)
|
||||
]
|
||||
return build_keyboard_from_form({'actions': selection_actions}, buttons_per_row=buttons_per_row)
|
||||
|
||||
|
||||
def resolve_select_button_action(form_data: dict, action_id: str) -> tuple[str, str] | None:
|
||||
"""Resolve a select-button callback to ``(field_name, option_value)``."""
|
||||
if not action_id.startswith(QQ_SELECT_ACTION_PREFIX):
|
||||
return None
|
||||
try:
|
||||
option_index = int(action_id[len(QQ_SELECT_ACTION_PREFIX) :])
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
field_name, options = get_select_field_options(form_data)
|
||||
if not field_name or option_index < 0 or option_index >= len(options) or option_index >= 25:
|
||||
return None
|
||||
return field_name, options[option_index]
|
||||
|
||||
|
||||
def build_keyboard_from_form(form_data: dict, *, buttons_per_row: int = 2) -> dict:
|
||||
"""Build a QQ keyboard JSON payload from a Dify human-input form_data.
|
||||
|
||||
Each Dify ``action`` becomes a callback button (``action.type=1``)
|
||||
whose ``data`` is set directly to the Dify ``action_id``. The
|
||||
INTERACTION_CREATE event carries this back as
|
||||
``data.resolved.button_data`` so the adapter can match the click to
|
||||
the originating form.
|
||||
|
||||
Layout limits per spec: max 5 rows, max 5 buttons per row. We default
|
||||
to 2 buttons per row for legibility; oversized button lists wrap
|
||||
onto additional rows and overflow gets dropped (max 25 visible).
|
||||
|
||||
Args:
|
||||
form_data: Dify ``{"actions": [{"id", "title", "button_style"}, ...]}``.
|
||||
buttons_per_row: 1..5. Mobile UI looks best at 2.
|
||||
|
||||
Returns:
|
||||
``{"content": {"rows": [{"buttons": [...]}]}}``.
|
||||
"""
|
||||
actions = list(form_data.get('actions') or [])[:25] # 5×5 hard cap
|
||||
buttons_per_row = max(1, min(5, buttons_per_row))
|
||||
|
||||
def _button(idx: int, action: dict) -> dict:
|
||||
action_id = str(action.get('id') or '')
|
||||
label = str(action.get('title') or action_id or f'选项 {idx + 1}')
|
||||
style_raw = (action.get('button_style') or '').lower()
|
||||
# QQ: 0 灰色线框, 1 蓝色线框. Highlight the primary / first action.
|
||||
if style_raw == 'primary' or (style_raw == '' and idx == 0):
|
||||
style = 1
|
||||
else:
|
||||
style = 0
|
||||
return {
|
||||
'id': str(idx + 1),
|
||||
'render_data': {
|
||||
'label': label,
|
||||
# Shown after the user clicks — gives local "已选择" feedback
|
||||
# without a follow-up message. Style mimics DingTalk/Lark's
|
||||
# in-card selection state.
|
||||
'visited_label': f'✓ {label}',
|
||||
'style': style,
|
||||
},
|
||||
'action': {
|
||||
'type': 1, # callback button
|
||||
'permission': {'type': 2}, # everyone can click
|
||||
'data': action_id,
|
||||
'unsupport_tips': '当前客户端版本不支持此按钮，请升级 QQ',
|
||||
},
|
||||
}
|
||||
|
||||
rows = []
|
||||
for row_start in range(0, len(actions), buttons_per_row):
|
||||
row_actions = actions[row_start : row_start + buttons_per_row]
|
||||
rows.append(
|
||||
{
|
||||
'buttons': [_button(row_start + j, a) for j, a in enumerate(row_actions)],
|
||||
}
|
||||
)
|
||||
if len(rows) >= 5:
|
||||
break
|
||||
|
||||
return {'content': {'rows': rows}}
|
||||
|
||||
|
||||
class QQOfficialClient:
|
||||
def __init__(self, secret: str, token: str, app_id: str, logger: None, unified_mode: bool = False):
|
||||
self.unified_mode = unified_mode
|
||||
@@ -30,6 +166,10 @@ class QQOfficialClient:
|
||||
self.token = token
|
||||
self.app_id = app_id
|
||||
self._message_handlers = {}
|
||||
# Single optional handler for INTERACTION_CREATE (button click). We
|
||||
# don't multiplex like message handlers — only the adapter cares,
|
||||
# and the click<->resume path needs a single source of truth.
|
||||
self._interaction_handler: Optional[Callable[[Dict[str, Any], Optional[str]], Any]] = None
|
||||
self.base_url = 'https://api.sgroup.qq.com'
|
||||
self.access_token = ''
|
||||
self.access_token_expiry_time = None
|
||||
@@ -107,6 +247,23 @@ class QQOfficialClient:
|
||||
return response, 200
|
||||
|
||||
if payload.get('op') == 0:
|
||||
# INTERACTION_CREATE (button click) skips ``get_message`` —
|
||||
# that helper only flattens message-event fields and would
|
||||
# drop ``data.resolved.button_data`` / ``data.button_id``.
|
||||
if payload.get('t') == 'INTERACTION_CREATE':
|
||||
if self._interaction_handler:
|
||||
try:
|
||||
d = payload.get('d') or {}
|
||||
# Top-level ``id`` is the ws/event id used as
|
||||
# ``event_id`` for passive replies. ``d.id``
|
||||
# is the interaction id used for ACK. Do not
|
||||
# confuse the two — QQ rejects misuse with
|
||||
# 40034025.
|
||||
ws_event_id = payload.get('id')
|
||||
await self._interaction_handler(d, ws_event_id)
|
||||
except Exception:
|
||||
await self.logger.error(f'Error in interaction handler: {traceback.format_exc()}')
|
||||
return {'code': 0, 'message': 'success'}
|
||||
message_data = await self.get_message(payload)
|
||||
if message_data:
|
||||
event = QQOfficialEvent.from_payload(message_data)
|
||||
@@ -133,6 +290,21 @@ class QQOfficialClient:
|
||||
|
||||
return decorator
|
||||
|
||||
def on_interaction(self):
|
||||
"""Register a single handler for INTERACTION_CREATE events.
|
||||
|
||||
The handler receives ``(data_dict, interaction_id)`` — the raw
|
||||
``d`` payload plus the top-level ``id`` field (the interaction
|
||||
id, needed for the PUT /interactions/{id} ack and for reuse as
|
||||
an ``event_id`` on the resumed reply within 30 minutes).
|
||||
"""
|
||||
|
||||
def decorator(func: Callable[[Dict[str, Any], Optional[str]], Any]):
|
||||
self._interaction_handler = func
|
||||
return func
|
||||
|
||||
return decorator
|
||||
|
||||
async def _handle_message(self, event: QQOfficialEvent):
|
||||
"""处理消息事件"""
|
||||
msg_type = event.t
|
||||
@@ -177,8 +349,20 @@ class QQOfficialClient:
|
||||
content_type = attachment.get('content_type', '')
|
||||
return content_type.startswith('image/')
|
||||
|
||||
async def send_private_text_msg(self, user_openid: str, content: str, msg_id: str):
|
||||
"""发送私聊消息"""
|
||||
async def send_private_text_msg(
|
||||
self,
|
||||
user_openid: str,
|
||||
content: str,
|
||||
msg_id: Optional[str] = None,
|
||||
event_id: Optional[str] = None,
|
||||
msg_seq: int = 1,
|
||||
):
|
||||
"""Send a c2c text message.
|
||||
|
||||
Either ``msg_id`` (inbound user msg, free passive reply) or
|
||||
``event_id`` (e.g. INTERACTION_CREATE id, valid 30 min) is
|
||||
required. Without either, the call costs the proactive-send quota.
|
||||
"""
|
||||
if not await self.check_access_token():
|
||||
await self.get_access_token()
|
||||
|
||||
@@ -188,11 +372,15 @@ class QQOfficialClient:
|
||||
'Authorization': f'QQBot {self.access_token}',
|
||||
'Content-Type': 'application/json',
|
||||
}
|
||||
data = {
|
||||
data: dict[str, Any] = {
|
||||
'content': content,
|
||||
'msg_type': 0,
|
||||
'msg_id': msg_id,
|
||||
'msg_seq': msg_seq,
|
||||
}
|
||||
if msg_id:
|
||||
data['msg_id'] = msg_id
|
||||
if event_id:
|
||||
data['event_id'] = event_id
|
||||
response = await client.post(url, headers=headers, json=data)
|
||||
response_data = response.json()
|
||||
if response.status_code == 200:
|
||||
@@ -201,8 +389,19 @@ class QQOfficialClient:
|
||||
await self.logger.error(f'Failed to send private message: {response_data}')
|
||||
raise ValueError(response)
|
||||
|
||||
async def send_group_text_msg(self, group_openid: str, content: str, msg_id: str):
|
||||
"""发送群聊消息"""
|
||||
async def send_group_text_msg(
|
||||
self,
|
||||
group_openid: str,
|
||||
content: str,
|
||||
msg_id: Optional[str] = None,
|
||||
event_id: Optional[str] = None,
|
||||
msg_seq: int = 1,
|
||||
):
|
||||
"""Send a group text message.
|
||||
|
||||
Either ``msg_id`` or ``event_id`` is required (see
|
||||
:meth:`send_private_text_msg` for the distinction).
|
||||
"""
|
||||
if not await self.check_access_token():
|
||||
await self.get_access_token()
|
||||
|
||||
@@ -212,11 +411,15 @@ class QQOfficialClient:
|
||||
'Authorization': f'QQBot {self.access_token}',
|
||||
'Content-Type': 'application/json',
|
||||
}
|
||||
data = {
|
||||
data: dict[str, Any] = {
|
||||
'content': content,
|
||||
'msg_type': 0,
|
||||
'msg_id': msg_id,
|
||||
'msg_seq': msg_seq,
|
||||
}
|
||||
if msg_id:
|
||||
data['msg_id'] = msg_id
|
||||
if event_id:
|
||||
data['event_id'] = event_id
|
||||
response = await client.post(url, headers=headers, json=data)
|
||||
if response.status_code == 200:
|
||||
return
|
||||
@@ -485,6 +688,107 @@ class QQOfficialClient:
|
||||
raise Exception(f'Failed to send stream message: HTTP {response.status_code} {response.text}')
|
||||
return response.json()
|
||||
|
||||
async def send_markdown_keyboard(
|
||||
self,
|
||||
target_type: str,
|
||||
target_id: str,
|
||||
markdown_content: str,
|
||||
keyboard: Optional[dict] = None,
|
||||
msg_id: Optional[str] = None,
|
||||
event_id: Optional[str] = None,
|
||||
msg_seq: int = 1,
|
||||
) -> dict:
|
||||
"""Send a ``msg_type=2`` (markdown) message carrying a keyboard.
|
||||
|
||||
The keyboard ride-along is the only documented way to attach
|
||||
buttons in QQ official; pure keyboard-only messages are not
|
||||
accepted by the server (markdown content is required).
|
||||
|
||||
Args:
|
||||
target_type: 'c2c' (single chat), 'group', 'channel' (text
|
||||
channel — uses POST /channels/{id}/messages instead of v2).
|
||||
target_id: openid for c2c/group, channel_id for channel.
|
||||
markdown_content: Plain markdown text shown above the buttons.
|
||||
keyboard: ``{'content': {'rows': [{'buttons': [...]}]}}`` per
|
||||
the official spec. Use :func:`build_keyboard_from_form`
|
||||
to construct from Dify form_data.
|
||||
msg_id: Inbound user message id; turns this into a passive
|
||||
reply (preferred — no monthly quota cost).
|
||||
event_id: Use ``INTERACTION_CREATE`` event id from a prior
|
||||
button click to keep within the 30-minute passive window
|
||||
without an inbound msg_id.
|
||||
msg_seq: De-dup counter when reusing msg_id.
|
||||
"""
|
||||
if not await self.check_access_token():
|
||||
await self.get_access_token()
|
||||
|
||||
if target_type == 'c2c':
|
||||
url = f'{self.base_url}/v2/users/{target_id}/messages'
|
||||
elif target_type == 'group':
|
||||
url = f'{self.base_url}/v2/groups/{target_id}/messages'
|
||||
elif target_type == 'channel':
|
||||
url = f'{self.base_url}/channels/{target_id}/messages'
|
||||
else:
|
||||
raise ValueError(f'Unsupported target_type for markdown+keyboard: {target_type}')
|
||||
|
||||
body: dict[str, Any] = {
|
||||
'msg_type': 2,
|
||||
'markdown': {'content': markdown_content},
|
||||
'msg_seq': msg_seq,
|
||||
}
|
||||
if keyboard and keyboard.get('content', {}).get('rows'):
|
||||
body['keyboard'] = keyboard
|
||||
if msg_id:
|
||||
body['msg_id'] = msg_id
|
||||
if event_id:
|
||||
body['event_id'] = event_id
|
||||
|
||||
async with httpx.AsyncClient(timeout=30) as client:
|
||||
headers = {
|
||||
'Authorization': f'QQBot {self.access_token}',
|
||||
'Content-Type': 'application/json',
|
||||
}
|
||||
response = await client.post(url, headers=headers, json=body)
|
||||
if response.status_code != 200:
|
||||
await self.logger.error(
|
||||
f'Failed to send markdown+keyboard: HTTP {response.status_code} {response.text}'
|
||||
)
|
||||
raise Exception(f'Failed to send markdown+keyboard: HTTP {response.status_code} {response.text}')
|
||||
return response.json()
|
||||
|
||||
async def ack_interaction(self, interaction_id: str, code: int = 0) -> None:
|
||||
"""Acknowledge a button-click INTERACTION_CREATE event.
|
||||
|
||||
QQ keeps the client in a loading spinner until this ack is
|
||||
received. Should be called as soon as the click is parsed, before
|
||||
any heavier downstream work (the actual workflow resume can run
|
||||
async).
|
||||
|
||||
Args:
|
||||
interaction_id: The ``id`` field from the INTERACTION_CREATE event.
|
||||
code: 0=success, 1=fail, 2=rate-limited, 3=duplicate, 4=no
|
||||
permission, 5=admin only. Default 0.
|
||||
"""
|
||||
if not interaction_id:
|
||||
return
|
||||
if not await self.check_access_token():
|
||||
await self.get_access_token()
|
||||
|
||||
url = f'{self.base_url}/interactions/{interaction_id}'
|
||||
async with httpx.AsyncClient(timeout=10) as client:
|
||||
headers = {
|
||||
'Authorization': f'QQBot {self.access_token}',
|
||||
'Content-Type': 'application/json',
|
||||
}
|
||||
try:
|
||||
response = await client.put(url, headers=headers, json={'code': code})
|
||||
if response.status_code >= 400:
|
||||
await self.logger.warning(
|
||||
f'ack_interaction non-success: HTTP {response.status_code} {response.text}'
|
||||
)
|
||||
except Exception as e:
|
||||
await self.logger.warning(f'ack_interaction error (non-fatal): {e}')
|
||||
|
||||
async def is_token_expired(self):
|
||||
"""检查token是否过期"""
|
||||
if self.access_token_expiry_time is None:
|
||||
@@ -653,6 +957,12 @@ class QQOfficialClient:
|
||||
d = payload.get('d', {})
|
||||
s = payload.get('s')
|
||||
t = payload.get('t')
|
||||
# Top-level event id, distinct from `d.id`. Per QQ
|
||||
# spec this is the only value accepted as ``event_id``
|
||||
# in subsequent passive-reply send-message calls
|
||||
# (``d.id`` for INTERACTION_CREATE is the interaction
|
||||
# id, used solely for PUT /interactions/{id} ack).
|
||||
ws_event_id = payload.get('id')
|
||||
|
||||
if not isinstance(d, dict):
|
||||
d = {}
|
||||
@@ -731,7 +1041,22 @@ class QQOfficialClient:
|
||||
|
||||
else:
|
||||
await self.logger.debug(f'Received event: {t}, seq={s}')
|
||||
if on_event:
|
||||
# INTERACTION_CREATE bypasses the regular
|
||||
# on_event dispatcher so the adapter sees the
|
||||
# top-level ws_event_id (needed as event_id
|
||||
# for the resumed reply) — same shape as the
|
||||
# webhook handler.
|
||||
if t == 'INTERACTION_CREATE':
|
||||
if self._interaction_handler:
|
||||
try:
|
||||
result = self._interaction_handler(d, ws_event_id)
|
||||
if asyncio.iscoroutine(result):
|
||||
await result
|
||||
except Exception:
|
||||
await self.logger.error(
|
||||
f'Error in interaction handler (ws): {traceback.format_exc()}'
|
||||
)
|
||||
elif on_event:
|
||||
try:
|
||||
result = on_event(t, d)
|
||||
if asyncio.iscoroutine(result):
|
||||
|
||||
Reference in New Issue
Block a user