LangBot/tools

LangBot regression tools

Run the deterministic OpenAI Chat Completions endpoint with Python 3.11+, without API credentials or third-party dependencies:

python3 tools/mock_llm.py --port 18080 --record /tmp/langbot-mock-requests.jsonl

In LangBot, create an openai-chat-completions Provider whose base URL is http://127.0.0.1:18080/v1 and whose API key is any test-only value. Use a model from GET /v1/models. The loopback address works when Core and this tool run on the same host; containers must use an address reachable from their own network.

Model	Behavior
mock-chat	Returns LANGBOT_CHAT_OK, including SSE streaming
mock-rag	Requires the golden document sentinel in the model input, then returns LANGBOT_RAG_OK
mock-embedding	Returns deterministic normalized token-hash vectors from /v1/embeddings
mock-box	Calls exec twice to write/read /workspace/mock-llm.txt, then returns LANGBOT_BOX_OK
mock-401, mock-429, mock-503	Returns the selected HTTP failure
mock-slow	Delays the response by two seconds
mock-slow-stream	Delays each SSE chunk by 0.2 seconds

mock-rag requires aurora-parser-rag-9137 in the submitted messages. The golden HTML fixture contains this marker; retrieve it into the model context without including it in the user's question.

For mock-box, enable Box in the Runner and expose the native exec tool. Each tool result must contain LANGBOT_BOX_SENTINEL before the next step is returned. Tool-call IDs and fragmented streaming arguments use the normal OpenAI protocol. Scenario progress comes from each request's current conversation turn, so concurrent sessions and retries do not consume a shared script counter. Tagged group traffic additionally requires the previous user turn and assistant acknowledgement in the model input, and rejects another group's history. Final responses carry a per-turn acknowledgement checked by the traffic driver.

Add fixed response/tool sequences with --scenarios path/to/scenarios.json, using mock_llm_scenarios.json as the example. expect_tool_output checks a substring in the previous tool results; expect_prompt checks the submitted messages. The server returns tool calls; it never executes their commands. Request recordings contain prompts, tool results, responses, status, and disconnects, but no HTTP headers. Use test data only.

Run the focused HTTP checks on the Linux test host:

python3 -m unittest discover -s tools -p test_mock_llm.py -v

These checks validate the mock server. A LangBot regression must additionally verify real tool outputs and retrieved document IDs/content. Token usage is synthetic, and this endpoint does not evaluate answer quality or embedding models.

mock-embedding accepts string/batch inputs, float/base64 encoding, and optional dimensions (default 384). It tests the embedding protocol and RAG plumbing; token hashes do not establish semantic retrieval quality.

Normal traffic and resource collection

regression_traffic.py executes a JSON list of normal lbctl cases at a selected rate. Each case has a name, CLI arguments, and an expected response substring:

[{"name":"chat","args":["pipeline","run","PIPELINE_UUID","--message","Hello"],"expect":"LANGBOT_CHAT_OK"}]

Prepare a Workspace-bound CLI context and its API key environment variable, then run on the Linux test host:

python3 tools/regression_traffic.py \
  --lbctl /path/to/lbctl --config /path/to/lbctl.yaml \
  --cases /path/to/cases.json --output /path/to/new-results-directory \
  --duration 600 --rps 2 --workers 4 --cooldown 30 \
  --pid core=CORE_PID --pid mock=MOCK_PID \
  --health core=http://127.0.0.1:5300/healthz \
  --health mock=http://127.0.0.1:18080/healthz

Add --container CONTAINER_ID for Docker CPU/memory/PID statistics. The executing account must be able to inspect the selected process trees and Docker daemon. Resource collection reuses scripts/cloud_runtime_soak.py; it does not require cgroup v2. PID resource totals include descendants, so select disjoint trees.

The result directory contains requests.jsonl, resources.jsonl, progress.json, and summary.json. The requested rate is a target: four workers bound concurrency, and slow requests lower the achieved rate. Requests with unknown outcomes are recorded without retry. A command fails its business check if it returns an error or omits the expected response. Health/resource collection failures also make the run fail. API key values are redacted from recorded CLI output.

Latency percentiles include the entire CLI invocation and use successful requests only. Achieved request rate is requests / load_elapsed_seconds. health_samples counts sampling rounds; health_failures counts rounds with any failed health probe or process/container resource read.

Configured regression suites

regression_suite.py turns a JSON profile into a repeatable resource preparation, traffic, observation, verdict, and cleanup sequence. Start with regression_profiles/groups.json for continuous group conversations. The earlier normal.json profile retains independent CLI requests; parallel CLI workers alone do not simulate persistent groups.

Use a dedicated QA Workspace. First provision the instance with lbctl install or your source deployment, enable Docker Box, start the mock and embedding backends, and prepare a Workspace-bound lbctl config. Set the profile's CLI config path, backend URLs, and three plugin archive paths. The sample embedding endpoint must serve all-MiniLM-L6-v2; the mock serves generation only. Relative paths resolve against the profile directory, so adjust them if you move it.

Run on the Linux test host:

python3 tools/regression_suite.py validate --profile tools/regression_profiles/groups.json
python3 tools/regression_suite.py run \
  --profile tools/regression_profiles/groups.json --output .qa/regression/run-001

validate checks profile syntax and references without contacting the instance. The output directory for run must be new. CLI options --duration, --rps, and --workers override those values in every phase. No additional Python dependency is needed.

The profile controls:

• plugins: local archives or GitHub/marketplace request files, installed with lbctl plugin install ... --wait.
• resources: Providers, models, knowledge bases, document ingestion, Pipelines, and Bots, created in dependency order. ${resource_id} refers to a UUID returned by an earlier creation; ${workspace_uuid} is the current Workspace. Pipeline config patches merge with actual server defaults. Independent CLI cases can use existing resources supplied through vars; group mode creates its own signed HTTP Bots.
• cases: CLI argument arrays or http_group settings and expected business text. Integer weight values set the mix; 2/1/1/1 means 40/20/20/20. An optional body object becomes the case's request file. All workers share the weighted sequence.
• phases: each phase has a name, duration, target RPS, worker limit, and optional cooldown. Multiple phases provide a step load. In-flight requests drain at the end; the concurrency limit can reduce achieved RPS.
• observe: health URLs, host PIDs, PID files, and Docker container names/IDs. Set PIDs for disjoint process trees to avoid counting descendants twice. box_data_root discovers Box containers mounted below that deployment's data directory on every sample, including sandboxes created as new groups speak. The complete Job sets it automatically; on an existing instance, set it to the instance's Box data directory when sampling dynamic Docker sandboxes.
• thresholds: minimum achieved/target RPS ratio and maximum P95 seconds per case, in addition to the existing business and health/resource checks. The sample limits are starting acceptance settings, not established product SLOs.
• cleanup: on-success (default), always, or never. Only recorded resources created by this run are deleted, in reverse order. Plugins and the existing instance remain installed. Failed/unknown creations are not automatically retried; inspect the instance if a creation outcome is unknown.

Failed runs retain resources by default. After inspection, clean them with the same profile and credentials:

python3 tools/regression_suite.py cleanup \
  --profile tools/regression_profiles/groups.json --output .qa/regression/run-001

Cleanup checks the Workspace against owned.json and records remaining items after partial failures. Keep owned.json until cleanup completes. Each run also contains profile.json, resolved resources.json, and one directory per phase with the original request/resource records. The root summary.json includes target/achieved RPS, phase results, threshold failures, and the final verdict.

The regression_suite.py entry reuses an existing instance and backend processes. Use the complete lifecycle Job below to own deployment, credentials and backends. The independent normal.json cases use a new-session CLI entry. Group traffic uses the signed HTTP Bot adapter instead, as described below. Real IM adapter acceptance and resource-leak verdicts need separate scenarios. Test the suite plumbing on Linux with:

python3 -m unittest discover -s tools -p test_regression_suite.py -v

Persistent group traffic

The group profile creates three HTTP Bots routed to chat, Box, and RAG Pipelines. Their signed /bots/<uuid>/sync ingress produces real GroupMessage events and uses Core's session manager. Four groups follow a 2/1/1 case assignment; members take turns within each group. The same member IDs appear in different groups to exercise group isolation. Box session selection uses the group launcher ID.

In each group phase, groups selects the number of stable sessions and members_per_group selects the rotating member identities. rps remains the aggregate target, so a group's target speaking interval is groups / rps seconds. workers bounds total in-flight requests independently of group count. For example, 8 groups at 4 RPS target one message every 2 seconds per group. Both the Suite and Job accept --groups and --members-per-group overrides. The group count must cover the full weighted case sequence (4 in the sample).

Only one request is in flight per group. A failure or unknown outcome stops that group without retry while other groups continue. Every selected group must complete at least two turns in total for the phase to pass. The fake model checks that the preceding user message and assistant acknowledgement survived in actual model input, including when a different member speaks next. Cross-group markers fail the oracle. The normal Box/tool-output and RAG-context checks still apply.

group-state.json carries group IDs, turn numbers and acknowledgements across phases; existing groups keep their conversations when later phases add groups. It refuses changed cases or a state left in progress by interruption. Do not clear that marker to blindly resume: inspect the run and use a fresh output directory when the last turn's outcome is unknown. Each phase summary includes group-level turn counts and member IDs alongside case latency and health data.

The Suite generates the HTTP Bot signing secret and passes it through LANGBOT_QA_GROUP_SECRET. Resource request bodies containing credentials are stored under .private/; do not share that directory. Cases specify http_group with url, secret_env, and message instead of CLI args. Core needs no code changes. The /sync response collapses streaming parts: this tests multi-turn group processing, not real IM delivery or callback streaming order. Every selected group must also receive traffic in each phase. Group latency measures the signed HTTP request through its completed reply; independent CLI latency includes CLI startup. Compare results within the same traffic mode.

lbctl v0.3 may report Bot verification failure after a successful creation because its safe projection omits event bindings. The Suite records the returned Bot UUID before reading the full Core Bot API with the endpoint, Workspace and environment credential confirmed by lbctl whoami. It then checks the masked signing secret and effective route with a signed warmup turn before load. The creation is not retried.

Complete lifecycle Job

regression_job.py owns the deployment as well as the suite. Use regression_profiles/job.json to pin the Core Release and select the suite and plugin input directory. The job runs this chain:

1. lbctl install --profile all creates a fresh deployment inside the run directory.
2. A Compose override starts our mock backend on the deployment's private network and injects a generated test-only API key. The Job binds its own CLI context.
3. The suite installs plugins, creates resources and ingests documents. Each case runs once as warmup before measured traffic starts.
4. Configured load phases run with health, process-tree and container sampling.
5. Service-log tails and raw request/resource records are retained. Successful Jobs stop the deployment, remove only sandboxes mounted under their own data root, and use lbctl uninstall --purge-data to reclaim the deployment.

Both generation and embedding use our mock backend, with no external model service or real model credentials. Plugin archives, the pinned Core image, lbctl, Docker and Compose v2 are Job inputs/prerequisites. This wrapper uses PyYAML, already included in LangBot's dependencies, to verify the CLI's deployment record and recover ownership after an interrupted install.

Run on a Linux host with Docker access and permission to inspect the selected container processes. Use an idle dedicated Docker daemon: current Box runtime startup cleanup can remove sandboxes from another runtime. The Job refuses to start when existing Box sandboxes are present. The current lbctl installer also reserves host port 5401 and 2280–2285, independently of its HTTP port.

Set the plugin directory and the lbctl executable in job.json, then run:

uv run --no-sync python tools/regression_job.py validate --profile tools/regression_profiles/job.json
uv run --no-sync python tools/regression_job.py run \
  --profile tools/regression_profiles/job.json --output .qa/regression/job-001

--duration, --rps and --workers override every load phase. Each output directory must be new. Job cleanup policy is on-success, always, or never; it governs the whole deployment, whereas suite cleanup governs API resources when a suite runs on an existing deployment. purge_data: false preserves the deployment data after removing containers. Failed Jobs retain their environment by default; after inspection use:

uv run --no-sync python tools/regression_job.py cleanup \
  --output .qa/regression/job-001

The root summary.json is the final Job verdict; suite/ contains the measured results, mock/requests.jsonl contains backend recordings, and services.log contains up to 10,000 log lines per service. Generated credentials and deployment inputs stay in .private/; share the result files, not that directory. A Job has its own CLI locator/config and records its deployment identity before cleanup. Changing that identity makes cleanup fail closed. Cleanup uses the saved Job state and does not require the original profile or plugin input files.

The Job replaces the real embedding model with mock-embedding. This measures LangBot/plugin/Box/RAG overhead, not real-model inference cost or retrieval quality. Run the Job plumbing checks on Linux with:

uv run --no-sync python -m unittest discover -s tools -p 'test_*.py' -v

Finite core smoke

Use regression_profiles/smoke.json for a finite functional check on an existing Core. It prepares 35 resources through lbctl, ingests HTML and plain-text fixtures through GeneralParsers, then executes 28 cases / 43 requests. Reports group the verdicts by stage:

Stage	Cases and independent checks
Configuration	Identity and plugins; provider, model, Pipeline and knowledge-base readback; document completed status and Workspace/KB ownership
Ingress	Missing and incorrect signatures return HTTP 401 / code 40101 without creating a group AgentRun; valid signatures are exercised by group conversations
Conversation	Ordinary and Unicode chat; chat, Box and RAG groups each use two interleaved groups with three turns and rotating members; model input must retain prior user/reply history without cross-group history
Model	Slow SSE reply; HTTP 401, 429 and 503 primary failures followed by successful fallback; recorded model/status/stream sequence must match
Tools	Docker write/read across rounds, two tool calls in one model response, and expected exit code 7 with separate stdout/stderr returned to the model
Delivery	ASCII and Unicode filenames/content delivered as one attachment with exact bytes
Retrieval	HTML and text parsing/ingestion, RAG model input, retrieval from two knowledge bases with distinct sentinels and KB ownership, and an empty knowledge base returning an empty list

Cleanup is checked separately: delete journaled API resources and confirm their UUIDs are absent from CLI list readback. Evidence goes to cleanup-checks.json; cleanup failures affect the overall verdict.

Start a dedicated mock_llm.py with the bundled scenarios and --record as shown above. Copy the smoke profile and configure the CLI executable/config, service URLs, plugin archives, vars.mock_records to that mock recording file, and vars.core_db to the target Core's existing SQLite langbot.db. The suite must read both local files. Keep fixture paths valid relative to the copied profile. Do not share the mock with other traffic during this run: fallback checks compare requests appended during each case.

Set plugins: [] to reuse installed plugins; LocalAgent, LangRAG and GeneralParsers must still be enabled and initialized. The embedding endpoint must serve the configured model (all-MiniLM-L6-v2 by default). Generation is deterministic mock traffic; Core, plugins, Box, document parsing and retrieval execute normally.

uv run --no-sync python tools/regression_suite.py validate --profile /path/to/smoke.json
uv run --no-sync python tools/regression_suite.py run \
  --profile /path/to/smoke.json --output .qa/smoke-001

A correct reply alone does not pass. Each Pipeline/group request must match one new AgentRun in the authenticated workspace with completed status and a persisted run.completed event. The ledger is opened read-only. This version requires local SQLite access; remote-only Core and PostgreSQL ledgers are not supported. The observation timeout does not extend the Runner deadline or retry requests. Unknown outcomes fail the suite; later turns of that group are skipped while independent cases continue. An expected tool exit code of 7 passes only when its output is correct and the agent handles the result and completes normally.

summary.json contains the overall and per-stage verdicts; exit code 0 means all checks and required cleanup passed. cases.json records per-request evidence, environment.json records identity/capabilities/plugins, and owned.json tracks created resources. Successful runs delete those API resources. Failures retain them by default; after inspection use:

uv run --no-sync python tools/regression_suite.py cleanup \
  --profile /path/to/smoke.json --output .qa/smoke-001

Box sessions and AgentRun records follow the reused service's normal retention policy. Load override flags are rejected for smoke. stage labels cases; fields compares exact response values by dotted path (including array indices), forbid rejects response substrings, and mock_requests checks the actual model/status/stream sequence. box_exec defaults to exit code 0; error-handling cases can specify exit_code and stderr explicitly.

To use a fresh deployment, select this suite in job.json; the Job supplies its SQLite/recording paths and uses mock embedding. Smoke skips load warmup and timed phases. The dedicated Docker daemon requirement above still applies.

PR smoke gate

Core Smoke runs regression-tool tests and a real Core smoke in parallel on GitHub-hosted Linux runners. The smoke starts Core from the checked-out candidate, uses the SDK from uv.lock, the existing official-plugin source pin, lbctl v0.3.0, a Docker Box and mock generation/embedding. It needs no model credentials or access to a developer's running environment.

Dependency installation, service startup, sandbox image download and plugin installation are preparation. After readiness, the gate allows 60 seconds total for API resource creation, 28 cases / 37 requests (two turns per group), verdict checks and API cleanup. A missing verdict, failed check or timeout fails the gate. The ordinary expanded profile keeps three turns / 43 requests. The entire cold workflow takes longer than a minute; it has a separate 15-minute timeout.

On an existing prepared stack, use:

uv run --no-sync python tools/regression_gate.py \
  --profile /path/to/smoke.json --output .qa/gate-001

gate.json records the budget, measured duration, verdict and process resource peaks. resources.jsonl samples configured process trees every second (RSS, cumulative CPU time, threads and file descriptors); suite artifacts contain per-case timings and business evidence. These are diagnostics, not calibrated performance-regression thresholds. Docker sandbox resource sampling remains in the longer load runner. A timeout kills the local Suite/CLI process group without retrying requests; a reused remote service can still have an in-flight run, so inspect its retained ownership journal before cleanup. CI tears down its own service processes and owned sandboxes afterward.

The workflow always creates its two checks on PRs and cancels superseded runs. After the workflow is merged, configure Regression tools and Core smoke (60s) as required status checks in the repository ruleset. A green workflow alone does not enforce merge protection. Longer load and soak tests use the existing Suite and Job entry points with separately calibrated latency, throughput and resource budgets; the one-minute smoke does not replace them.