feat: add supports for dify hitl (#2226)

* feat: Implement workflow form handling for paused workflows

- Added module-level storage for pending forms to manage state across sessions.
- Introduced functions to set, get, and clear pending forms with expiration handling.
- Enhanced DifyServiceAPIRunner to support resuming paused workflows via form actions.
- Implemented logic to yield human input requests and display appropriate messages.
- Updated workflow submission methods to handle paused states and resume actions.
- Ensured proper merging of pending form actions with user inputs for seamless interaction.

* feat: Add '_routed_by_rule' variable to form action in Lark and Telegram adapters

* feat: Enhance Lark and Telegram adapters with new form handling for paused workflows

* feat: Enhance TelegramAdapter to handle form action buttons and message threading

* feat: Improve TelegramAdapter message handling with enhanced error management and draft message support

* feat: Add the function for formatting human input text to support adapters without rich UI.

* feat(dingtalk): implement human input card support and card action handling

- Add a new module `card_callback.py` to handle card action button clicks from DingTalk.
- Introduce `DingTalkCardActionHandler` to process card action callbacks and extract parameters.
- Update `DingTalkAdapter` to manage card state and handle form input through a single card template.
- Add configuration for `human_input_card_template_id` in `dingtalk.yaml` to specify the template for human input.
- Create a new card template `dingtalk_human_input_card.json` for rendering human input prompts and buttons.

* feat(dingtalk): enhance human input card functionality with streaming support and active turn management

- Updated the DingTalk card template to enable streaming mode and multi-update configuration.
- Removed the obsolete delete_card method from DingTalkClient to streamline card management.
- Enhanced DingTalkAdapter to manage active turn cards and accumulated streaming text, ensuring a seamless user experience during human input prompts.
- Modified the create_message_card method to utilize existing active cards for resumed workflows, preventing duplication.
- Improved the _paint_form_on_card method to update existing cards with human input prompts and buttons dynamically.
- Updated the dingtalk_human_input_card.json template to reflect the new streaming capabilities and configuration options.

* feat(wecom): implement Dify human input pause handling with button interaction support

* feat(qqofficial): implement Dify human input button interaction handling and markdown keyboard support

* feat(qqofficial): implement one-click QR binding and enhance localization support

* feat(discord): implement Discord form view with button interactions for Dify actions

* fix(telegram): correct group chat type check and handle oversized callback data for Telegram actions
fix(difysvapi): ensure safe access to remove-think configuration in pipeline settings

* feat(dify): add support for chatflow app type and enhance human input handling

* feat(telegram): add action title feedback for user selections in Telegram messages

* feat(lark): enhance LarkAdapter to store form content for resume notices

* feat(dingtalk): update display formatting for card content with HTML line breaks

* feat(dingtalk): add feedback functionality to cards with 👍/👎 buttons

- Implemented feedback state management for cards, allowing users to provide feedback via thumbs up/down buttons.
- Enhanced card rendering to include feedback buttons when appropriate.
- Registered feedback listeners to handle feedback events and update card states accordingly.
- Updated the card template to support dynamic button rendering for feedback actions.
- Improved error handling and logging for feedback actions and card updates.

* fix: add Avatar component to dingtalk_human_input_card.json for enhanced user interaction

* feat(wecom): add optional source block to interactive template cards for enhanced branding

* feat(wecom): add functions for template card action extraction and update, enhance button interaction handling

* feat(qqofficial): synchronize passive-reply counter with inbound message sequence

* feat(qqofficial): add method to identify invisible form placeholder chunks in messages

* feat(dingtalk): add download link for human input card template and enhance dynamic form configuration

* feat(telegram): enhance message handling with group stream deletion and form placeholder detection

* Add unit tests for DingTalk, Lark, WeComBot, and Dify service API runners

- Implement tests for DingTalk adapter helper functions including form content cleaning, input extraction, and completed input lines.
- Create unit tests for Lark adapter helper functions focusing on input extraction and completed input lines.
- Add tests for WeComBot template card functionalities, including event extraction and payload building for human input.
- Enhance Dify service API runner tests to cover human input forms, including input collection, action handling, and form snapshot extraction.

* feat: Enhance Telegram and QQ Official adapters with select field handling and form action processing

- Added support for select fields in Telegram adapter, including option extraction and callback handling.
- Implemented form action processing for Telegram callbacks, improving user interaction feedback.
- Introduced new helper functions for building keyboards and resolving select button actions in QQ Official adapter.
- Enhanced DifyServiceAPIRunner to handle cumulative streaming responses and improve error handling during workflow resumes.
- Added unit tests for new functionalities in Telegram and QQ Official adapters, ensuring robust behavior for select fields and form actions.

* feat(lark): add functions for current input definitions and visible form content handling
feat(qqofficial): update fallback text handling for non-streaming scenarios
feat(difysvapi): enhance form content processing for interactive fields and actions
test: add unit tests for Lark and QQ Official adapter functionalities

* Add tests for DingTalk adapter content processing and markdown formatting

- Updated the assertion in `test_dingtalk_completed_input_lines_include_text_and_select_values` to remove unnecessary markdown formatting.
- Added new tests to verify that `_dingtalk_clean_form_content` maintains the order of prompts and completed values in various scenarios.
- Introduced `test_dingtalk_card_markdown_preserves_internal_line_breaks` to ensure internal line breaks are correctly converted to HTML line breaks.

* feat: Refactor input handling and feedback messages across multiple adapters

* feat: Update the human-computer interaction template cards, and optimize the prompt information and content display.

* feat: Refactor pending form handling to isolate by bot and pipeline

* feat: Enhance error handling and caching for Dify and WeCom interactions

* feat: Enhance select input handling and validation in Dify API runner and Telegram adapter

* feat: Add missing completed input lines handling in DingTalk adapter

* feat: Add pipeline_uuid handling across multiple adapters and update related tests
This commit is contained in:
Dongchuan Fu
2026-07-13 00:42:46 +08:00
committed by GitHub
parent 3ddebd26ae
commit 0755beebcd
43 changed files with 12320 additions and 221 deletions
+424 -5
View File
@@ -11,7 +11,13 @@ import langbot_plugin.api.entities.builtin.platform.events as platform_events
import langbot_plugin.api.entities.builtin.platform.entities as platform_entities
from ..logger import EventLogger
from langbot.libs.wecom_ai_bot_api.wecombotevent import WecomBotEvent
from langbot.libs.wecom_ai_bot_api.api import WecomBotClient
from langbot.libs.wecom_ai_bot_api.api import (
WecomBotClient,
extract_template_card_action,
extract_template_card_event_payload,
extract_template_card_selections,
parse_select_button_action,
)
from langbot.libs.wecom_ai_bot_api.ws_client import WecomBotWsClient
@@ -296,6 +302,7 @@ class WecomBotAdapter(abstract_platform_adapter.AbstractMessagePlatformAdapter):
listeners: dict = {}
_stream_to_monitoring_msg: dict = {} # Maps stream_id to (monitoring_message_id, timestamp)
_STREAM_MAPPING_TTL = 600 # 10 minutes
ap: typing.Any = None
def __init__(self, config: dict, logger: EventLogger):
enable_webhook = config.get('enable-webhook', False)
@@ -336,6 +343,25 @@ class WecomBotAdapter(abstract_platform_adapter.AbstractMessagePlatformAdapter):
_stream_to_monitoring_msg={},
)
# Both WecomBotClient (webhook) and WecomBotWsClient (ws long-conn)
# expose ``set_card_action_callback``. Wire the click handler so
# Dify human-input button taps resume the workflow on either mode.
if hasattr(self.bot, 'set_card_action_callback'):
self.bot.set_card_action_callback(self._on_card_action)
# Hand the client a `source` block so every interactive
# template_card it emits carries the LangBot logo + name at the
# top — the WeCom analogue of DingTalk's Avatar header.
# Always on; icon_url accepts plain HTTPS URLs (no upload needed).
if hasattr(self.bot, 'set_card_source'):
self.bot.set_card_source(
{
'icon_url': 'https://raw.githubusercontent.com/RockChinQ/LangBot/master/res/logo-blue.png',
'desc': 'LangBot',
'desc_color': 0,
}
)
async def reply_message(
self,
message_source: platform_events.MessageEvent,
@@ -345,15 +371,37 @@ class WecomBotAdapter(abstract_platform_adapter.AbstractMessagePlatformAdapter):
content = await self.message_converter.yiri2target(message)
_ws_mode = not self.config.get('enable-webhook', False)
event = message_source.source_platform_object
# Synthetic events (button-click resume queries) have no inbound
# platform object. Fall back to a proactive send so error
# messages and one-shot replies still reach the user.
if event is None:
if _ws_mode:
if isinstance(message_source, platform_events.GroupMessage):
chat_id = str(message_source.group.id)
else:
chat_id = str(message_source.sender.id)
try:
await self.bot.send_message(chat_id, content)
except Exception:
await self.logger.error(
f'WeComBot: proactive reply for synthetic event failed: {traceback.format_exc()}'
)
else:
await self.logger.warning(
'WeComBot webhook mode cannot reply to a synthetic event '
'(no req_id and no proactive-send credentials); dropping.'
)
return
if _ws_mode:
event = message_source.source_platform_object
req_id = event.get('req_id', '')
req_id = event.get('req_id', '') if isinstance(event, dict) else getattr(event, 'req_id', '')
if req_id:
await self.bot.reply_text(req_id, content)
else:
await self.bot.set_message(event.message_id, content)
else:
await self.bot.set_message(message_source.source_platform_object.message_id, content)
await self.bot.set_message(event.message_id, content)
async def reply_message_chunk(
self,
@@ -364,9 +412,56 @@ class WecomBotAdapter(abstract_platform_adapter.AbstractMessagePlatformAdapter):
is_final: bool = False,
):
content = await self.message_converter.yiri2target(message)
msg_id = message_source.source_platform_object.message_id
_ws_mode = not self.config.get('enable-webhook', False)
# Synthetic events (e.g. button-click triggered form resume) have
# no inbound platform message — no msg_id, no req_id, no stream
# session. The output must go via the proactive-send path instead
# of the stream/reply path.
spo = message_source.source_platform_object
if spo is None:
return await self._handle_synthetic_chunk(message_source, bot_message, content, is_final, _ws_mode)
msg_id = spo.message_id
# Dify human-input pause: when the runner attaches `_form_data` to
# the final chunk, hand the button_interaction card off to the
# underlying client. In webhook mode the card is queued for the
# next followup poll; in ws mode it's sent as a reply frame
# immediately. Falls back to plain text when the bot has no active
# stream session for this msg_id (rare).
form_data = getattr(bot_message, '_form_data', None)
if form_data and is_final:
if hasattr(self.bot, 'push_form_pause'):
ok, stream_id, task_id = await self.bot.push_form_pause(msg_id, form_data)
if ok:
await self.logger.info(
f'WeComBot: pending button_interaction registered '
f'stream_id={stream_id} task_id={task_id} ws_mode={_ws_mode}'
)
return {'stream': True, 'form': True, 'task_id': task_id}
await self.logger.warning(
'WeComBot: cannot register form pause (no active stream session); falling back to plain text'
)
try:
from langbot.pkg.provider.runners.difysvapi import _format_human_input_text
fallback = _format_human_input_text(
form_data.get('node_title', ''),
form_data.get('form_content', ''),
form_data.get('actions', []) or [],
)
except Exception:
fallback = content or '(人工输入)'
if _ws_mode:
event = message_source.source_platform_object
req_id = event.get('req_id', '') if isinstance(event, dict) else getattr(event, 'req_id', '')
if req_id:
await self.bot.reply_text(req_id, fallback)
else:
await self.bot.set_message(msg_id, fallback)
return {'stream': False, 'form': True, 'fallback': True}
if _ws_mode:
success = await self.bot.push_stream_chunk(msg_id, content, is_final=is_final)
if not success and is_final:
@@ -385,6 +480,142 @@ class WecomBotAdapter(abstract_platform_adapter.AbstractMessagePlatformAdapter):
"""Whether streaming output is enabled for this bot instance."""
return self.config.get('enable-stream-reply', True)
async def _handle_synthetic_chunk(
self,
message_source: platform_events.MessageEvent,
bot_message,
content: str,
is_final: bool,
ws_mode: bool,
) -> dict:
"""Handle reply_message_chunk for synthetic events (button clicks).
Synthetic events have no inbound message → no msg_id, no req_id,
no stream session. We can't do incremental streaming, so we
buffer chunks per-conversation and flush on ``is_final`` via the
proactive send path.
Buffer keyed by ``(launcher_type, launcher_id)`` from the
synthetic event itself. Only ws mode has a usable proactive-send
path right now (``ws_client.send_message`` /
``ws_client.send_template_card``); webhook mode requires a
corpid/secret we don't have, so it logs and drops.
"""
if isinstance(message_source, platform_events.GroupMessage):
chat_id = str(message_source.group.id)
else:
chat_id = str(message_source.sender.id)
form_data = getattr(bot_message, '_form_data', None)
# Buffer streaming content until is_final.
buf_key = chat_id
if not hasattr(self, '_synthetic_buffers'):
# Attribute-not-declared trick: pydantic forbids dynamic attrs
# on the model, but plain instance dicts via object.__setattr__
# do work. Lazy-create on first call.
object.__setattr__(self, '_synthetic_buffers', {})
buffers: dict[str, str] = self._synthetic_buffers
if content and not form_data:
previous = buffers.get(buf_key, '')
if previous and content.startswith(previous):
buffers[buf_key] = content
elif previous and previous.endswith(content):
buffers[buf_key] = previous
else:
buffers[buf_key] = previous + content
if not is_final:
return {'stream': True, 'synthetic': True, 'buffered': True}
final_content = buffers.pop(buf_key, '')
if content:
if final_content and content.startswith(final_content):
final_content = content
elif final_content and final_content.endswith(content):
pass
else:
final_content = final_content + content
if not ws_mode:
await self.logger.warning(
'WeComBot webhook mode cannot proactively push synthetic-event '
'output (no corpid/secret); the resume reply is dropped. '
f'content_len={len(final_content)} form_data_present={form_data is not None}'
)
return {'stream': False, 'synthetic': True, 'dropped': True}
# ws mode: proactive send.
try:
if form_data:
# Determine user_id / chat_id for the routing context of any
# subsequent click on this card.
if isinstance(message_source, platform_events.GroupMessage):
routing_chat_id = str(message_source.group.id)
routing_user_id = str(message_source.sender.id)
else:
routing_chat_id = ''
routing_user_id = str(message_source.sender.id)
payload = self._build_button_interaction_payload_from_form(
form_data,
user_id=routing_user_id,
chat_id=routing_chat_id,
)
await self.bot.send_template_card(chat_id, payload)
await self.logger.info(
f'WeComBot ws: proactively sent template_card for synthetic event '
f'chat_id={chat_id} form_token={form_data.get("form_token")!r} '
f'workflow_run_id={form_data.get("workflow_run_id")!r}'
)
elif final_content:
await self.bot.send_message(chat_id, final_content)
await self.logger.info(
f'WeComBot ws: proactively sent text for synthetic event chat_id={chat_id} len={len(final_content)}'
)
except Exception:
await self.logger.error(f'WeComBot: synthetic event proactive send failed: {traceback.format_exc()}')
return {'stream': False, 'synthetic': True, 'error': True}
return {'stream': True, 'synthetic': True}
def _build_button_interaction_payload_from_form(
self, form_data: dict, *, user_id: str = '', chat_id: str = ''
) -> dict:
"""Build a button_interaction payload + track task_id for click resolution.
Unlike the inbound-event path (where push_form_pause registers the
task_id with the active stream session), proactive sends still
need the task_id registered so button clicks find pending_form.
For ws mode we stash it directly on the ws_client's pending dict.
"""
from langbot.libs.wecom_ai_bot_api.api import build_human_input_template_card_payload
import secrets as _secrets
task_id = f'dify-{_secrets.token_hex(12)}'
source = getattr(self.bot, 'card_source', None)
payload = build_human_input_template_card_payload(
form_data,
task_id,
source=source,
select_as_buttons=not self.config.get('enable-webhook', False),
)
# Register task_id → form_data so the click callback can find it.
# user_id / chat_id are required so _on_card_action can route the
# resulting synthetic query back to the right user. msg_id / req_id
# / stream_id are intentionally empty — synthetic cards have no
# inbound message to anchor on.
if hasattr(self.bot, '_pending_forms_by_task'):
self.bot._pending_forms_by_task[task_id] = {
'form_data': form_data,
'msg_id': '',
'user_id': user_id,
'chat_id': chat_id,
'stream_id': '',
'req_id': '',
}
return payload
async def send_message(self, target_type, target_id, message):
_ws_mode = not self.config.get('enable-webhook', False)
if _ws_mode:
@@ -531,3 +762,191 @@ class WecomBotAdapter(abstract_platform_adapter.AbstractMessagePlatformAdapter):
async def is_muted(self, group_id: int) -> bool:
pass
# ------------------------------------------------------------------
# Dify human-input button-interaction click handling
# ------------------------------------------------------------------
async def _on_card_action(self, session, action_id: str, task_id: str, raw_event: dict) -> None:
"""Translate a button click on a button_interaction card into a
synthetic ``_dify_form_action`` query enqueued on the pool.
Pattern mirrors DingTalk / Lark / Telegram so the runner's
``_merge_pending_form_action`` path resumes the workflow.
"""
import langbot_plugin.api.entities.builtin.provider.session as provider_session
form = session.pending_form or {}
await self.logger.info(
f'WeComBot _on_card_action: task_id={task_id} action_id={action_id!r} '
f'form_token={form.get("form_token")!r} workflow_run_id={form.get("workflow_run_id")!r} '
f'session.user_id={session.user_id!r} session.chat_id={session.chat_id!r}'
)
actions = form.get('actions') or []
tce = extract_template_card_event_payload(raw_event) if isinstance(raw_event, dict) else {}
_, _, card_type = extract_template_card_action(tce)
selections = extract_template_card_selections(tce, form)
if not selections:
selections = parse_select_button_action(action_id, form)
await self.logger.info(
f'WeComBot template_card selections: task_id={task_id} card_type={card_type} selections={selections}'
)
if card_type == 'multiple_interaction' and not selections:
await self.logger.warning(
f'WeComBot: multiple_interaction callback has no parseable selections; raw={str(tce)[:1000]}'
)
return
is_select_submit = card_type == 'multiple_interaction' or bool(selections)
clean_action_id = '' if is_select_submit else (action_id or '').strip()
action_title = clean_action_id
for a in actions:
if str(a.get('id', '')) == clean_action_id:
action_title = a.get('title') or clean_action_id
break
inputs = dict(form.get('inputs') or {})
inputs.update(selections)
def _missing_fields_after_select() -> list[str]:
missing: list[str] = []
for field in form.get('input_defs') or form.get('all_input_defs') or []:
field_name = str(field.get('output_variable_name') or '').strip()
if not field_name:
continue
if inputs.get(field_name) in (None, '', []):
missing.append(field_name)
return missing
input_progress = False
if is_select_submit:
missing_fields = _missing_fields_after_select()
if not missing_fields and len(actions) == 1:
action = actions[0]
clean_action_id = str(action.get('id') or '').strip()
action_title = action.get('title') or clean_action_id
elif not missing_fields and len(actions) > 1:
if not self.config.get('enable-webhook', False):
action_form_data = {
'form_content': form.get('raw_form_content') or form.get('form_content') or '',
'raw_form_content': form.get('raw_form_content') or form.get('form_content') or '',
'input_defs': [],
'all_input_defs': form.get('all_input_defs') or form.get('input_defs') or [],
'inputs': inputs,
'actions': actions,
'node_title': form.get('node_title', ''),
'workflow_run_id': form.get('workflow_run_id', ''),
'form_token': form.get('form_token', ''),
'pipeline_uuid': form.get('pipeline_uuid', ''),
'_action_select_only': True,
}
target_chat_id = session.chat_id or session.user_id or ''
try:
payload = self._build_button_interaction_payload_from_form(
action_form_data,
user_id=session.user_id or '',
chat_id=session.chat_id or '',
)
await self.bot.send_template_card(target_chat_id, payload)
await self.logger.info(
f'WeComBot: sent action-select button card after select submit '
f'task_id={task_id} action_count={len(actions)}'
)
except Exception:
await self.logger.error(
f'WeComBot: failed to send action-select button card: {traceback.format_exc()}'
)
return
await self.logger.warning(
'WeComBot webhook mode cannot proactively send action-select button card after select submit'
)
return
else:
input_progress = True
action_title = 'Submit'
launcher_id = session.user_id or session.chat_id or ''
sender_user_id = session.user_id or launcher_id
# WeCom AI bot has both single-chat and group-chat; chat_id present
# indicates group context.
if session.chat_id:
launcher_type = provider_session.LauncherTypes.GROUP
launcher_id = session.chat_id
else:
launcher_type = provider_session.LauncherTypes.PERSON
launcher_id = session.user_id or ''
form_action_data = {
'form_token': form.get('form_token', ''),
'workflow_run_id': form.get('workflow_run_id', ''),
'action_id': clean_action_id,
'action_title': action_title,
'node_title': form.get('node_title', ''),
'user': f'{launcher_type.value}_{launcher_id}',
'inputs': inputs,
}
if input_progress:
form_action_data['_input_progress'] = True
message_chain = platform_message.MessageChain([platform_message.Plain(text=f'[Form Action: {action_title}]')])
if launcher_type == provider_session.LauncherTypes.GROUP:
synthetic_event = platform_events.GroupMessage(
sender=platform_entities.GroupMember(
id=sender_user_id,
member_name='',
permission=platform_entities.Permission.Member,
group=platform_entities.Group(
id=launcher_id,
name='',
permission=platform_entities.Permission.Member,
),
special_title='',
),
message_chain=message_chain,
time=int(time.time()),
source_platform_object=None,
)
else:
synthetic_event = platform_events.FriendMessage(
sender=platform_entities.Friend(
id=sender_user_id,
nickname='',
remark='',
),
message_chain=message_chain,
time=int(time.time()),
source_platform_object=None,
)
if self.ap is None:
await self.logger.error('WeComBot: ap not injected; cannot enqueue button-click query')
return
bot_uuid = ''
pipeline_uuid = form.get('pipeline_uuid') or None
for bot in self.ap.platform_mgr.bots:
if bot.adapter is self:
bot_uuid = bot.bot_entity.uuid
pipeline_uuid = pipeline_uuid or bot.bot_entity.use_pipeline_uuid
break
try:
await self.ap.query_pool.add_query(
bot_uuid=bot_uuid,
launcher_type=launcher_type,
launcher_id=launcher_id,
sender_id=sender_user_id,
message_event=synthetic_event,
message_chain=message_chain,
adapter=self,
pipeline_uuid=pipeline_uuid,
variables={
'_dify_form_action': form_action_data,
'_routed_by_rule': True,
},
)
await self.logger.info(f'WeComBot: button-click query enqueued action_id={clean_action_id!r}')
except Exception:
await self.logger.error(f'WeComBot: enqueue button-click query failed: {traceback.format_exc()}')