fix(agent-runner): close sandbox skill authoring flow

This commit is contained in:
huanghuoguoguo
2026-07-15 08:35:52 +08:00
parent d871baab68
commit 730d52bcd0
12 changed files with 313 additions and 31 deletions
+4 -3
View File
@@ -2,7 +2,7 @@
本文档是 `docs/agent-runner-pluginization/` 的状态事实源。协议 schema 仍以 [PROTOCOL_V1.md](./PROTOCOL_V1.md) 为准;测试步骤以 [AGENT_RUNNER_QA_GUIDE.md](./AGENT_RUNNER_QA_GUIDE.md) 为准;安全发布门槛以 [SECURITY_HARDENING.md](./SECURITY_HARDENING.md) 为准。 本文档是 `docs/agent-runner-pluginization/` 的状态事实源。协议 schema 仍以 [PROTOCOL_V1.md](./PROTOCOL_V1.md) 为准;测试步骤以 [AGENT_RUNNER_QA_GUIDE.md](./AGENT_RUNNER_QA_GUIDE.md) 为准;安全发布门槛以 [SECURITY_HARDENING.md](./SECURITY_HARDENING.md) 为准。
状态快照日期:2026-07-12 状态快照日期:2026-07-15
## 实现状态 ## 实现状态
@@ -15,7 +15,7 @@
| Result payload validation | Done | Wire 保持 `{type, data}`Host 对投递/副作用类 payload 严格校验,tool-call telemetry 宽松,未知 type 忽略并 warning。 | | Result payload validation | Done | Wire 保持 `{type, data}`Host 对投递/副作用类 payload 严格校验,tool-call telemetry 宽松,未知 type 忽略并 warning。 |
| Old built-in runners | Done | 旧 `src/langbot/pkg/provider/runners/*``RequestRunner` 路径已从本分支删除。 | | Old built-in runners | Done | 旧 `src/langbot/pkg/provider/runners/*``RequestRunner` 路径已从本分支删除。 |
| Official runner manifests | Done | `local-agent`、ACP / Claude Code / Codex 外部 harness runner、外部服务 runner 已重新声明真实生效的 LangBot resource permissions。 | | Official runner manifests | Done | `local-agent`、ACP / Claude Code / Codex 外部 harness runner、外部服务 runner 已重新声明真实生效的 LangBot resource permissions。 |
| Skill 链路 | Unit-pass; WebUI smoke pending | 已按 **skill 全 tool 化** 收敛:发现走 `list_skills` / `langbot_list_assets` 和 skill resources`activate` / `register_skill` 走统一 tool 授权;`skill_authoring` capability 降级为便捷开关。`activate` 现在会 best-effort 写入 conversation-scope `host.activated_skills`,后续 run 通过当前 pipeline-visible skill cache 恢复,语义为 last-write-wins。 | | Skill 链路 | Unit-pass; WebUI E2E pass | 已按 **skill 全 tool 化** 收敛:发现走 `list_skills` / `langbot_list_assets` 和 skill resources`activate` / `register_skill` 走统一 tool 授权;`skill_authoring` capability 降级为便捷开关。`activate` 会 best-effort 写入 conversation-scope `host.activated_skills`,后续 run 通过当前 pipeline-visible skill cache 恢复。新注册 Skill 在当前 Query 内立即获得临时可见性;Docker `exec` 产生的宿主侧不可写文件由 `write` / `edit` 回退到 Box 执行。2026-07-15 真实 LocalAgent Debug Chat 已完成创建、注册、同 Query 激活、编辑和执行闭环。 |
| Runtime Control Plane v2 foundation | Partial | Host-owned `AgentRun` / `AgentRunEvent` ledger、orchestrator 自动建账、result event persistence、run get/list/event page/cancel/append/finalize actions 已落地;`agent_run:admin` / `runtime:admin` 控制权限、最小 runtime register/heartbeat/list/reconcile 和 run claim/renew/release 原语已落地。完整 Agent Platform 产品形态、daemon supervisor、任务唤醒/长轮询/WebSocket、分布式 runtime 管控仍未完成。 | | Runtime Control Plane v2 foundation | Partial | Host-owned `AgentRun` / `AgentRunEvent` ledger、orchestrator 自动建账、result event persistence、run get/list/event page/cancel/append/finalize actions 已落地;`agent_run:admin` / `runtime:admin` 控制权限、最小 runtime register/heartbeat/list/reconcile 和 run claim/renew/release 原语已落地。完整 Agent Platform 产品形态、daemon supervisor、任务唤醒/长轮询/WebSocket、分布式 runtime 管控仍未完成。 |
| Security boundary | Done | 当前口径降级为轻量边界:LangBot 保护自身持有资源;external harness 的 OS / process / network / workspace 风险由用户或部署环境承担;managed sandbox 不是当前承诺。 | | Security boundary | Done | 当前口径降级为轻量边界:LangBot 保护自身持有资源;external harness 的 OS / process / network / workspace 风险由用户或部署环境承担;managed sandbox 不是当前承诺。 |
| Steering control path | Done | claim 异常不再逃逸 consumer loopqueue 有上限;未 pull 的 claimed 输入在 run 结束时写 `steering.dropped` 审计终态。 | | Steering control path | Done | claim 异常不再逃逸 consumer loopqueue 有上限;未 pull 的 claimed 输入在 run 结束时写 `steering.dropped` 审计终态。 |
@@ -37,7 +37,7 @@
| Runner | 状态 | 最近证据 | | Runner | 状态 | 最近证据 |
| --- | --- | --- | | --- | --- | --- |
| `plugin:langbot-team/LocalAgent/default` | Unit-pass; Marketplace UI pass | 2026-07-12 隔离 first-run 实例从真实 AgentRunner catalog 安装 `langbot-team/LocalAgent` 0.1.0Host 注册 `plugin:langbot-team/LocalAgent/default`,Wizard 自动选中并解锁后续操作。此项不替代带真实模型的 Debug Chat 能力验收。 | | `plugin:langbot-team/LocalAgent/default` | Unit-pass; Marketplace UI pass; Debug Chat E2E pass | 2026-07-12 隔离 first-run 实例从真实 AgentRunner catalog 安装 `langbot-team/LocalAgent` 0.1.0Host 注册 `plugin:langbot-team/LocalAgent/default`,Wizard 自动选中并解锁后续操作。2026-07-15 `local-agent-functional-20260715-skill-edit-final` 使用真实 `gpt-5.5` 完成 Skill 创建、注册、同 Query 激活、已激活包编辑与脚本执行;三阶段 UI、浏览器诊断和结构化文件系统检查全部通过,p95 18.8 秒、错误率 0。 |
| `plugin:langbot-team/ACPAgentRunner/default` / `plugin:langbot-team/ClaudeCodeAgent/default` / `plugin:langbot-team/CodexAgent/default` | Unit-pass; E2E pending | 通过 runner 仓库单测覆盖 session、run_id 注入和 LangBot MCP gateway;真实 harness E2E 取决于对应运行环境、CLI/daemon 可用性和 provider 登录态。 | | `plugin:langbot-team/ACPAgentRunner/default` / `plugin:langbot-team/ClaudeCodeAgent/default` / `plugin:langbot-team/CodexAgent/default` | Unit-pass; E2E pending | 通过 runner 仓库单测覆盖 session、run_id 注入和 LangBot MCP gateway;真实 harness E2E 取决于对应运行环境、CLI/daemon 可用性和 provider 登录态。 |
| Dify / n8n / Coze / DashScope / Langflow / Tbox / DeerFlow / WeKnora | Unit-pass; credential smoke optional | 2026-06-13 plugin layout / parser tests 通过;真实服务凭据 smoke 非每轮必跑。 | | Dify / n8n / Coze / DashScope / Langflow / Tbox / DeerFlow / WeKnora | Unit-pass; credential smoke optional | 2026-06-13 plugin layout / parser tests 通过;真实服务凭据 smoke 非每轮必跑。 |
@@ -46,6 +46,7 @@
| 范围 | 状态 | 最近证据 | | 范围 | 状态 | 最近证据 |
| --- | --- | --- | | --- | --- | --- |
| LangBot Runtime Control Plane v2 foundation | Unit-pass; EBA release gate 5/5 pass | 2026-07-12 `eba-functional-20260712-release-gate-rerun` 通过 Quick Start 场景筛选、隔离实例 Runner Marketplace 安装、Runner 健康状态、事件路由 dry-run / 合成派发,以及真实 OneBot `group.member_joined` → Agent → `send_group_msg` 链路。 | | LangBot Runtime Control Plane v2 foundation | Unit-pass; EBA release gate 5/5 pass | 2026-07-12 `eba-functional-20260712-release-gate-rerun` 通过 Quick Start 场景筛选、隔离实例 Runner Marketplace 安装、Runner 健康状态、事件路由 dry-run / 合成派发,以及真实 OneBot `group.member_joined` → Agent → `send_group_msg` 链路。 |
| Host Skill / native tool integration | Unit-pass; WebUI E2E pass | 2026-07-15 provider / native / Skill / monitoring 定向测试 67 项通过,Skills CLI 104 项通过;真实 Debug Chat 验证 `register_skill` 后同 Query `activate` 成功,监控工具调用不再把 SQL 行误取为字符串,结构化 JSON 文件检查不依赖格式空格。 |
| SDK AgentRunner control entities / proxy | Unit-pass | 2026-06-23 SDK `tests/api/entities/builtin/agent_runner``tests/api/proxies``tests/api/test_agent_tools_mcp_bridge.py``tests/runtime/plugin/test_mgr_agent_runner.py``tests/runtime/test_pull_api_handlers.py``tests/runtime/io/handlers/test_plugin_handler.py`、EBA event entities 和 message tests 通过,覆盖 typed entities、AgentRunAPIProxy、MCP bridge、runtime manager 与 pull API handlers。 | | SDK AgentRunner control entities / proxy | Unit-pass | 2026-06-23 SDK `tests/api/entities/builtin/agent_runner``tests/api/proxies``tests/api/test_agent_tools_mcp_bridge.py``tests/runtime/plugin/test_mgr_agent_runner.py``tests/runtime/test_pull_api_handlers.py``tests/runtime/io/handlers/test_plugin_handler.py`、EBA event entities 和 message tests 通过,覆盖 typed entities、AgentRunAPIProxy、MCP bridge、runtime manager 与 pull API handlers。 |
## 历史高价值记录 ## 历史高价值记录
+26 -2
View File
@@ -239,6 +239,7 @@ async function runFilesystemChecks(checks) {
} }
const contains = textList(check.contains); const contains = textList(check.contains);
const notContains = textList(check.not_contains || check.notContains); const notContains = textList(check.not_contains || check.notContains);
const expectedJson = check.json_equals ?? check.jsonEquals;
const expectedExitCode = Number.isInteger(check.exit_code) const expectedExitCode = Number.isInteger(check.exit_code)
? check.exit_code ? check.exit_code
: Number.isInteger(check.expected_exit_code) : Number.isInteger(check.expected_exit_code)
@@ -257,18 +258,31 @@ async function runFilesystemChecks(checks) {
} }
const missing = contains.filter((needle) => !text.includes(needle)); const missing = contains.filter((needle) => !text.includes(needle));
const forbidden = notContains.filter((needle) => text.includes(needle)); const forbidden = notContains.filter((needle) => text.includes(needle));
let jsonError = "";
let jsonMatches = null;
if (expectedJson !== undefined) {
try {
jsonMatches = JSON.stringify(sortJson(JSON.parse(text))) === JSON.stringify(sortJson(expectedJson));
if (!jsonMatches) jsonError = "JSON content does not match json_equals.";
} catch (error) {
jsonMatches = false;
jsonError = `Invalid JSON: ${error.message}`;
}
}
const failed = missing.length > 0 || forbidden.length > 0 || jsonMatches === false;
results.push({ results.push({
index, index,
status: missing.length || forbidden.length ? "fail" : "pass", status: failed ? "fail" : "pass",
type: "file", type: "file",
path, path,
missing, missing,
forbidden, forbidden,
json_matches: jsonMatches,
reason: missing.length reason: missing.length
? `Missing expected text: ${missing.join(", ")}` ? `Missing expected text: ${missing.join(", ")}`
: forbidden.length : forbidden.length
? `Found forbidden text: ${forbidden.join(", ")}` ? `Found forbidden text: ${forbidden.join(", ")}`
: "", : jsonError,
}); });
continue; continue;
} }
@@ -310,6 +324,16 @@ async function runFilesystemChecks(checks) {
}; };
} }
function sortJson(value) {
if (Array.isArray(value)) return value.map(sortJson);
if (!value || typeof value !== "object") return value;
return Object.fromEntries(
Object.keys(value)
.sort()
.map((key) => [key, sortJson(value[key])]),
);
}
function pipelineIdFromUrl(url) { function pipelineIdFromUrl(url) {
if (!url) return ""; if (!url) return "";
try { try {
@@ -0,0 +1,76 @@
#!/usr/bin/env node
import { env } from "node:process";
import {
apiJson,
ensureEvidence,
evidencePaths,
loadEnvFiles,
resetAndAuthLocalUser,
writeResult,
} from "./lib/langbot-e2e.mjs";
const DEFAULT_LOCAL_PASSWORD = "LangBotE2ELocalPass!2026";
const caseId = "reset-sandbox-skill";
await loadEnvFiles();
const paths = evidencePaths(caseId);
await ensureEvidence(paths);
const nameArgument = process.argv.find((argument) => argument.startsWith("--name="));
const skillName = nameArgument?.slice("--name=".length).trim() || "";
const backendUrl = env.LANGBOT_BACKEND_URL || "";
const result = {
source: "setup_automation",
case_id: caseId,
run_id: paths.runId,
status: "fail",
reason: "",
skill_name: skillName,
existed: false,
deleted: false,
evidence_collected: ["api_diagnostic"],
};
try {
if (!backendUrl) throw new Error("LANGBOT_BACKEND_URL is not configured.");
if (!skillName || !/^[A-Za-z0-9_-]+$/.test(skillName)) {
throw new Error("--name must contain only letters, numbers, hyphens, or underscores.");
}
const user = env.LANGBOT_E2E_LOGIN_USER || "";
if (!user) throw new Error("LANGBOT_E2E_LOGIN_USER is required.");
const password = env.LANGBOT_E2E_LOGIN_PASSWORD || DEFAULT_LOCAL_PASSWORD;
const auth = await resetAndAuthLocalUser({ backendUrl, user, password });
const listed = await apiJson(backendUrl, "/api/v1/skills", { token: auth.token });
if (listed.status >= 400 || listed.json.code !== 0) {
throw new Error(listed.json.msg || `Skill listing failed with HTTP ${listed.status}.`);
}
result.existed = (listed.json.data?.skills || []).some((skill) => skill?.name === skillName);
if (result.existed) {
const deleted = await apiJson(
backendUrl,
`/api/v1/skills/${encodeURIComponent(skillName)}`,
{ method: "DELETE", token: auth.token },
);
if (deleted.status >= 400 || deleted.json.code !== 0) {
throw new Error(deleted.json.msg || `Skill deletion failed with HTTP ${deleted.status}.`);
}
result.deleted = true;
}
result.status = "pass";
result.reason = result.deleted
? `Removed stale sandbox skill ${skillName}.`
: `Sandbox skill ${skillName} was already absent.`;
} catch (error) {
result.status = /ECONNREFUSED|fetch failed|not configured|required/.test(error.message)
? "env_issue"
: "fail";
result.reason = error.message;
}
await writeResult(paths, result);
console.log(JSON.stringify(result, null, 2));
process.exit(result.status === "pass" ? 0 : result.status === "env_issue" ? 2 : 1);
+8 -2
View File
@@ -1621,8 +1621,14 @@
"edit" "edit"
], ],
"automation": "scripts/e2e/pipeline-debug-chat.mjs", "automation": "scripts/e2e/pipeline-debug-chat.mjs",
"setup_automation": [], "setup_automation": [
"setup_provides_env": [], "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env",
"node:scripts/e2e/reset-sandbox-skill.mjs --name=lb-skill-edit-regression"
],
"setup_provides_env": [
"LANGBOT_LOCAL_AGENT_PIPELINE_URL",
"LANGBOT_LOCAL_AGENT_PIPELINE_NAME"
],
"evidence_required": [ "evidence_required": [
"ui", "ui",
"backend_log", "backend_log",
@@ -32,9 +32,15 @@ automation_env:
- LANGBOT_LOCAL_AGENT_PIPELINE_NAME - LANGBOT_LOCAL_AGENT_PIPELINE_NAME
automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL automation_pipeline_url_env: LANGBOT_LOCAL_AGENT_PIPELINE_URL
automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME automation_pipeline_name_env: LANGBOT_LOCAL_AGENT_PIPELINE_NAME
automation_expected_text: "UPDATED_MARKER_EDIT_REGRESSION" automation_expected_text: "E2E_OK:lb-skill-edit-regression:SANDBOX_SKILL_MODIFIED_OK"
automation_prompts_json: '[{"prompt":"Skill name is lb-skill-edit-regression. Use exactly one exec tool call with workdir=/workspace. In that single command, remove old /workspace/lb-skill-edit-regression, create SKILL.md, scripts/use.py, and data/input.json. The JSON must contain numbers [1,2,3,4], factors [2,3,4], and marker CREATE_MARKER_EDIT_REGRESSION. The Python script must read data/input.json relative to __file__ and print exactly SANDBOX_SKILL_CREATE_OK sum=10 product=24 marker=CREATE_MARKER_EDIT_REGRESSION. Run python3 /workspace/lb-skill-edit-regression/scripts/use.py in the same command. After the exec succeeds, final answer exactly CREATE_OK:lb-skill-edit-regression.","expected_text":"CREATE_MARKER_EDIT_REGRESSION","response_timeout_ms":"180000"},{"prompt":"Continue with the same skill name lb-skill-edit-regression. Strictly call register_skill with path=/workspace/lb-skill-edit-regression and name=lb-skill-edit-regression, then call activate with skill_name=lb-skill-edit-regression. Do not call exec, read, write, edit, grep, or ls in this step. After both tools succeed, final answer exactly REGISTER_ACTIVATE_OK:lb-skill-edit-regression.","expected_text":"REGISTER_ACTIVATE_OK:lb-skill-edit-regression","response_timeout_ms":"180000"},{"prompt":"Continue with the same activated skill lb-skill-edit-regression. Use exactly one exec tool call with workdir=/workspace/.skills/lb-skill-edit-regression. In that single exec command, run exactly this shell logic: set -e; overwrite SKILL.md with a heading and UPDATED_MARKER_EDIT_REGRESSION; overwrite data/input.json with numbers [5,6], factors [7,8], and marker UPDATED_MARKER_EDIT_REGRESSION; overwrite scripts/use.py so it reads data/input.json relative to __file__ and prints exactly SANDBOX_SKILL_MODIFIED_OK sum=11 product=56 marker=UPDATED_MARKER_EDIT_REGRESSION; run python3 scripts/use.py. Do not write exit(1). Do not call find, ls, read, grep as separate tools. If the single exec exits successfully and prints SANDBOX_SKILL_MODIFIED_OK, do not call more tools. Final answer exactly E2E_OK:lb-skill-edit-regression:SANDBOX_SKILL_MODIFIED_OK.","expected_text":"UPDATED_MARKER_EDIT_REGRESSION","response_timeout_ms":"240000"}]' automation_prompts_json: '[{"prompt":"Skill name is lb-skill-edit-regression. Use exactly one exec tool call with workdir=/workspace. In that single command, remove old /workspace/lb-skill-edit-regression, create SKILL.md, scripts/use.py, and data/input.json. The JSON must contain numbers [1,2,3,4], factors [2,3,4], and marker CREATE_MARKER_EDIT_REGRESSION. The Python script must read data/input.json relative to __file__ and print exactly SANDBOX_SKILL_CREATE_OK sum=10 product=24 marker=CREATE_MARKER_EDIT_REGRESSION. Run python3 /workspace/lb-skill-edit-regression/scripts/use.py in the same command. After the exec succeeds, final answer exactly CREATE_OK:lb-skill-edit-regression.","expected_text":"CREATE_OK:lb-skill-edit-regression","response_timeout_ms":"180000"},{"prompt":"Continue with the same skill name lb-skill-edit-regression. Strictly call register_skill with path=/workspace/lb-skill-edit-regression and name=lb-skill-edit-regression, then call activate with skill_name=lb-skill-edit-regression. Do not call exec, read, write, edit, grep, or ls in this step. After both tools succeed, final answer exactly REGISTER_ACTIVATE_OK:lb-skill-edit-regression.","expected_text":"REGISTER_ACTIVATE_OK:lb-skill-edit-regression","response_timeout_ms":"180000"},{"prompt":"Continue with the same activated skill lb-skill-edit-regression. Use exactly one exec tool call with workdir=/workspace/.skills/lb-skill-edit-regression. In that single exec command, run exactly this shell logic: set -e; overwrite SKILL.md with a heading and UPDATED_MARKER_EDIT_REGRESSION; overwrite data/input.json with numbers [5,6], factors [7,8], and marker UPDATED_MARKER_EDIT_REGRESSION; overwrite scripts/use.py so it reads data/input.json relative to __file__ and prints exactly SANDBOX_SKILL_MODIFIED_OK sum=11 product=56 marker=UPDATED_MARKER_EDIT_REGRESSION; run python3 scripts/use.py. Do not write exit(1). Do not call find, ls, read, grep as separate tools. If the single exec exits successfully and prints SANDBOX_SKILL_MODIFIED_OK, do not call more tools. Final answer exactly E2E_OK:lb-skill-edit-regression:SANDBOX_SKILL_MODIFIED_OK.","expected_text":"E2E_OK:lb-skill-edit-regression:SANDBOX_SKILL_MODIFIED_OK","response_timeout_ms":"240000"}]'
automation_filesystem_checks_json: '[{"path":"${LANGBOT_REPO}/data/box/skills/lb-skill-edit-regression/SKILL.md","contains":"UPDATED_MARKER_EDIT_REGRESSION"},{"path":"${LANGBOT_REPO}/data/box/skills/lb-skill-edit-regression/data/input.json","contains":["UPDATED_MARKER_EDIT_REGRESSION","\"numbers\": [5, 6]","\"factors\": [7, 8]"]},{"path":"${LANGBOT_REPO}/data/box/skills/lb-skill-edit-regression/scripts/use.py","contains":["SANDBOX_SKILL_MODIFIED_OK"],"not_contains":["SANDBOX_SKILL_CREATE_OK","exit(1)"]},{"argv":["python3","scripts/use.py"],"cwd":"${LANGBOT_REPO}/data/box/skills/lb-skill-edit-regression","stdout_contains":"SANDBOX_SKILL_MODIFIED_OK sum=11 product=56 marker=UPDATED_MARKER_EDIT_REGRESSION","exit_code":0}]' setup_automation:
- "node:scripts/e2e/ensure-local-agent-pipeline.mjs --write-env"
- "node:scripts/e2e/reset-sandbox-skill.mjs --name=lb-skill-edit-regression"
setup_provides_env:
- LANGBOT_LOCAL_AGENT_PIPELINE_URL
- LANGBOT_LOCAL_AGENT_PIPELINE_NAME
automation_filesystem_checks_json: '[{"path":"${LANGBOT_REPO}/data/box/skills/lb-skill-edit-regression/SKILL.md","contains":"UPDATED_MARKER_EDIT_REGRESSION"},{"path":"${LANGBOT_REPO}/data/box/skills/lb-skill-edit-regression/data/input.json","json_equals":{"numbers":[5,6],"factors":[7,8],"marker":"UPDATED_MARKER_EDIT_REGRESSION"}},{"path":"${LANGBOT_REPO}/data/box/skills/lb-skill-edit-regression/scripts/use.py","contains":["SANDBOX_SKILL_MODIFIED_OK"],"not_contains":["SANDBOX_SKILL_CREATE_OK","exit(1)"]},{"argv":["python3","scripts/use.py"],"cwd":"${LANGBOT_REPO}/data/box/skills/lb-skill-edit-regression","stdout_contains":"SANDBOX_SKILL_MODIFIED_OK sum=11 product=56 marker=UPDATED_MARKER_EDIT_REGRESSION","exit_code":0}]'
automation_stream_output: "0" automation_stream_output: "0"
automation_reset_debug_chat: "1" automation_reset_debug_chat: "1"
automation_response_timeout_ms: "420000" automation_response_timeout_ms: "420000"
@@ -48,9 +54,9 @@ steps:
- "Run the automation prompt or the equivalent prompt pattern from references/sandbox-skill-authoring.md." - "Run the automation prompt or the equivalent prompt pattern from references/sandbox-skill-authoring.md."
- "Capture the visible final response, backend logs, and the registered skill-store files." - "Capture the visible final response, backend logs, and the registered skill-store files."
checks: checks:
- "UI: A Bot message, not only the user prompt, contains UPDATED_MARKER_EDIT_REGRESSION." - "UI: A Bot message, not only the user prompt, contains E2E_OK:lb-skill-edit-regression:SANDBOX_SKILL_MODIFIED_OK."
- "Logs: The model called exec, register_skill, activate, then a second exec whose workdir is /workspace/.skills/lb-skill-edit-regression." - "Logs: The model called exec, register_skill, activate, then a second exec whose workdir is /workspace/.skills/lb-skill-edit-regression."
- "Logs: The second exec stdout contains SANDBOX_SKILL_MODIFIED_OK and GREP_ALL_OK or equivalent grep success." - "Logs: The second exec stdout contains SANDBOX_SKILL_MODIFIED_OK."
- "Automation: Filesystem checks pass for the registered skill-store files under LANGBOT_REPO/data/box/skills/lb-skill-edit-regression." - "Automation: Filesystem checks pass for the registered skill-store files under LANGBOT_REPO/data/box/skills/lb-skill-edit-regression."
- "Skill store: SKILL.md and data/input.json contain UPDATED_MARKER_EDIT_REGRESSION." - "Skill store: SKILL.md and data/input.json contain UPDATED_MARKER_EDIT_REGRESSION."
- "Skill store: scripts/use.py is the modified script and prints SANDBOX_SKILL_MODIFIED_OK sum=11 product=56 marker=UPDATED_MARKER_EDIT_REGRESSION." - "Skill store: scripts/use.py is the modified script and prints SANDBOX_SKILL_MODIFIED_OK sum=11 product=56 marker=UPDATED_MARKER_EDIT_REGRESSION."
@@ -169,7 +169,7 @@ class MonitoringService:
) )
row = result.first() row = result.first()
if row: if row:
return row[0] return row
if not session_id: if not session_id:
return None return None
@@ -188,7 +188,7 @@ class MonitoringService:
result = await self.ap.persistence_mgr.execute_async(user_query) result = await self.ap.persistence_mgr.execute_async(user_query)
row = result.first() row = result.first()
if row: if row:
return row[0] return row
any_query = ( any_query = (
sqlalchemy.select(persistence_monitoring.MonitoringMessage) sqlalchemy.select(persistence_monitoring.MonitoringMessage)
@@ -198,7 +198,7 @@ class MonitoringService:
) )
result = await self.ap.persistence_mgr.execute_async(any_query) result = await self.ap.persistence_mgr.execute_async(any_query)
row = result.first() row = result.first()
return row[0] if row else None return row
# ========== Recording Methods ========== # ========== Recording Methods ==========
@@ -574,9 +574,13 @@ else:
) )
if self._should_use_box_workspace_files(selected_skill): if self._should_use_box_workspace_files(selected_skill):
return await self._write_workspace_via_box(path, content, parameters, query) return await self._write_workspace_via_box(path, content, parameters, query)
os.makedirs(os.path.dirname(host_path), exist_ok=True)
try: try:
os.makedirs(os.path.dirname(host_path), exist_ok=True)
self._write_host_file(host_path, content, parameters) self._write_host_file(host_path, content, parameters)
except PermissionError:
if selected_skill is None and hasattr(self.ap.box_service, 'execute_tool'):
return await self._write_workspace_via_box(path, content, parameters, query)
raise
except ValueError as exc: except ValueError as exc:
return {'ok': False, 'error': str(exc)} return {'ok': False, 'error': str(exc)}
self._refresh_skill_from_disk(selected_skill) self._refresh_skill_from_disk(selected_skill)
@@ -625,18 +629,23 @@ else:
) )
if self._should_use_box_workspace_files(selected_skill): if self._should_use_box_workspace_files(selected_skill):
return await self._edit_workspace_via_box(path, old_string, new_string, query) return await self._edit_workspace_via_box(path, old_string, new_string, query)
if not os.path.isfile(host_path): try:
return {'ok': False, 'error': f'File not found: {path}'} if not os.path.isfile(host_path):
with open(host_path, 'r', encoding='utf-8', errors='replace') as f: return {'ok': False, 'error': f'File not found: {path}'}
content = f.read() with open(host_path, 'r', encoding='utf-8', errors='replace') as f:
count = content.count(old_string) content = f.read()
if count == 0: count = content.count(old_string)
return {'ok': False, 'error': 'old_string not found in file.'} if count == 0:
if count > 1: return {'ok': False, 'error': 'old_string not found in file.'}
return {'ok': False, 'error': f'old_string matches {count} locations; provide a more unique string.'} if count > 1:
new_content = content.replace(old_string, new_string, 1) return {'ok': False, 'error': f'old_string matches {count} locations; provide a more unique string.'}
with open(host_path, 'w', encoding='utf-8') as f: new_content = content.replace(old_string, new_string, 1)
f.write(new_content) with open(host_path, 'w', encoding='utf-8') as f:
f.write(new_content)
except PermissionError:
if selected_skill is None and hasattr(self.ap.box_service, 'execute_tool'):
return await self._edit_workspace_via_box(path, old_string, new_string, query)
raise
self._refresh_skill_from_disk(selected_skill) self._refresh_skill_from_disk(selected_skill)
return {'ok': True, 'path': path} return {'ok': True, 'path': path}
@@ -49,6 +49,18 @@ def get_visible_skill(ap: app.Application, query: pipeline_query.Query, skill_na
return get_visible_skills(ap, query).get(skill_name) return get_visible_skills(ap, query).get(skill_name)
def register_created_skill_visibility(query: pipeline_query.Query, skill_name: str) -> None:
"""Make a newly registered skill visible for the current Query only."""
if not skill_name:
return
if getattr(query, 'variables', None) is None:
query.variables = {}
bound_skills = query.variables.get(PIPELINE_BOUND_SKILLS_KEY)
if isinstance(bound_skills, list) and skill_name not in bound_skills:
bound_skills.append(skill_name)
def get_activated_skills(query: pipeline_query.Query) -> dict[str, dict]: def get_activated_skills(query: pipeline_query.Query) -> dict[str, dict]:
if query.variables is None: if query.variables is None:
return {} return {}
@@ -67,7 +67,7 @@ class SkillToolLoader(loader.ToolLoader):
if name == ACTIVATE_SKILL_TOOL_NAME: if name == ACTIVATE_SKILL_TOOL_NAME:
return await self._invoke_activate_skill(parameters, query) return await self._invoke_activate_skill(parameters, query)
if name == REGISTER_SKILL_TOOL_NAME: if name == REGISTER_SKILL_TOOL_NAME:
return await self._invoke_register_skill(parameters) return await self._invoke_register_skill(parameters, query)
raise ValueError(f'Unknown skill tool: {name}') raise ValueError(f'Unknown skill tool: {name}')
async def shutdown(self): async def shutdown(self):
@@ -121,8 +121,10 @@ class SkillToolLoader(loader.ToolLoader):
'content': result_content, 'content': result_content,
} }
async def _invoke_register_skill(self, parameters: dict) -> typing.Any: async def _invoke_register_skill(self, parameters: dict, query) -> typing.Any:
"""Register a skill from sandbox directory to data/skills/.""" """Register a skill from sandbox directory to data/skills/."""
from . import skill as skill_loader
sandbox_path = str(parameters.get('path', '') or '').strip() sandbox_path = str(parameters.get('path', '') or '').strip()
if not sandbox_path: if not sandbox_path:
raise ValueError('path is required') raise ValueError('path is required')
@@ -153,6 +155,7 @@ class SkillToolLoader(loader.ToolLoader):
'package_root': host_path, 'package_root': host_path,
} }
) )
skill_loader.register_created_skill_visibility(query, skill_name)
return { return {
'registered': True, 'registered': True,
@@ -0,0 +1,48 @@
from __future__ import annotations
from types import SimpleNamespace
from unittest.mock import AsyncMock, Mock
import pytest
from langbot.pkg.api.http.service.monitoring import MonitoringService
class _MonitoringMessageRow:
"""Match the Row shape returned by Core connections for ORM selects."""
id = 'message-1'
bot_id = 'bot-1'
bot_name = 'QA Bot'
pipeline_id = 'pipeline-1'
pipeline_name = 'QA Pipeline'
session_id = 'person_1'
def __getitem__(self, index: int) -> str:
assert index == 0
return self.id
@pytest.mark.asyncio
async def test_record_tool_call_uses_full_monitoring_message_row_for_context():
message_row = _MonitoringMessageRow()
select_result = Mock(first=Mock(return_value=message_row))
insert_result = Mock()
persistence_mgr = SimpleNamespace(
execute_async=AsyncMock(side_effect=[select_result, insert_result]),
)
service = MonitoringService(SimpleNamespace(persistence_mgr=persistence_mgr))
await service.record_tool_call(
tool_name='exec',
tool_source='native',
duration=12,
message_id='message-1',
)
insert_statement = persistence_mgr.execute_async.await_args_list[1].args[0]
values = insert_statement.compile().params
assert values['bot_id'] == 'bot-1'
assert values['pipeline_id'] == 'pipeline-1'
assert values['session_id'] == 'person_1'
assert values['message_id'] == 'message-1'
@@ -440,6 +440,48 @@ class TestSkillToolLoader:
assert result['skill_name'] == 'cloned-skill' assert result['skill_name'] == 'cloned-skill'
assert result['source_path'] == '/workspace/repo' assert result['source_path'] == '/workspace/repo'
@pytest.mark.asyncio
async def test_registered_skill_can_be_activated_in_same_query(self):
from langbot.pkg.provider.tools.loaders.skill import PIPELINE_BOUND_SKILLS_KEY
from langbot.pkg.provider.tools.loaders.skill_authoring import (
ACTIVATE_SKILL_TOOL_NAME,
REGISTER_SKILL_TOOL_NAME,
SkillToolLoader,
)
with tempfile.TemporaryDirectory() as tmpdir:
repo_dir = os.path.join(tmpdir, 'repo')
os.makedirs(repo_dir)
created = _make_skill_data(name='cloned-skill', package_root=os.path.realpath(repo_dir))
ap = _make_ap()
ap.box_service = SimpleNamespace(default_workspace=tmpdir, available=True)
ap.skill_mgr = SimpleNamespace(skills={'existing': _make_skill_data(name='existing')})
async def create_skill(_data):
ap.skill_mgr.skills['cloned-skill'] = created
return created
ap.skill_service = SimpleNamespace(
scan_directory_async=AsyncMock(return_value=created),
create_skill=AsyncMock(side_effect=create_skill),
)
query = SimpleNamespace(variables={PIPELINE_BOUND_SKILLS_KEY: ['existing']})
loader = SkillToolLoader(ap)
await loader.invoke_tool(
REGISTER_SKILL_TOOL_NAME,
{'path': '/workspace/repo', 'name': 'cloned-skill'},
query,
)
activated = await loader.invoke_tool(
ACTIVATE_SKILL_TOOL_NAME,
{'skill_name': 'cloned-skill'},
query,
)
assert query.variables[PIPELINE_BOUND_SKILLS_KEY] == ['existing', 'cloned-skill']
assert activated['activated'] is True
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_register_skill_rejects_workspace_escape(self): async def test_register_skill_rejects_workspace_escape(self):
from langbot.pkg.provider.tools.loaders.skill_authoring import ( from langbot.pkg.provider.tools.loaders.skill_authoring import (
@@ -4,7 +4,7 @@ import base64
import os import os
import tempfile import tempfile
from types import SimpleNamespace from types import SimpleNamespace
from unittest.mock import AsyncMock, Mock from unittest.mock import AsyncMock, Mock, patch
import pytest import pytest
@@ -236,6 +236,32 @@ async def test_write_creates_subdirectories():
assert f.read() == 'nested' assert f.read() == 'nested'
@pytest.mark.asyncio
async def test_write_falls_back_to_box_for_container_owned_workspace_file():
with tempfile.TemporaryDirectory() as tmpdir:
box_service = SimpleNamespace(
available=True,
default_workspace=tmpdir,
execute_tool=AsyncMock(
return_value={
'ok': True,
'stdout': '{"ok": true, "path": "/workspace/root-owned.txt"}',
}
),
)
loader = NativeToolLoader(SimpleNamespace(box_service=box_service, logger=Mock()))
loader._write_host_file = Mock(side_effect=PermissionError)
result = await loader.invoke_tool(
'write',
{'path': '/workspace/root-owned.txt', 'content': 'updated'},
_make_query(),
)
assert result == {'ok': True, 'path': '/workspace/root-owned.txt'}
box_service.execute_tool.assert_awaited_once()
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_read_binary_file_as_base64_chunk(): async def test_read_binary_file_as_base64_chunk():
with tempfile.TemporaryDirectory() as tmpdir: with tempfile.TemporaryDirectory() as tmpdir:
@@ -326,6 +352,35 @@ async def test_edit_replaces_unique_string():
assert f.read() == 'def foo():\n return 42\n' assert f.read() == 'def foo():\n return 42\n'
@pytest.mark.asyncio
async def test_edit_falls_back_to_box_for_container_owned_workspace_file():
with tempfile.TemporaryDirectory() as tmpdir:
path = os.path.join(tmpdir, 'root-owned.txt')
with open(path, 'w') as f:
f.write('before')
box_service = SimpleNamespace(
available=True,
default_workspace=tmpdir,
execute_tool=AsyncMock(
return_value={
'ok': True,
'stdout': '{"ok": true, "path": "/workspace/root-owned.txt"}',
}
),
)
loader = NativeToolLoader(SimpleNamespace(box_service=box_service, logger=Mock()))
with patch('builtins.open', side_effect=PermissionError):
result = await loader.invoke_tool(
'edit',
{'path': '/workspace/root-owned.txt', 'old_string': 'before', 'new_string': 'after'},
_make_query(),
)
assert result == {'ok': True, 'path': '/workspace/root-owned.txt'}
box_service.execute_tool.assert_awaited_once()
@pytest.mark.asyncio @pytest.mark.asyncio
async def test_edit_rejects_ambiguous_match(): async def test_edit_rejects_ambiguous_match():
with tempfile.TemporaryDirectory() as tmpdir: with tempfile.TemporaryDirectory() as tmpdir: