LangBot/tools/README.md

LangBot regression tools

Run the deterministic OpenAI Chat Completions endpoint with Python 3.11+, without API credentials or third-party dependencies:

python3 tools/mock_llm.py --port 18080 --record /tmp/langbot-mock-requests.jsonl

In LangBot, create an openai-chat-completions Provider whose base URL is http://127.0.0.1:18080/v1 and whose API key is any test-only value. Use a model from GET /v1/models. The loopback address works when Core and this tool run on the same host; containers must use an address reachable from their own network.

Model	Behavior
mock-chat	Returns LANGBOT_CHAT_OK, including SSE streaming
mock-rag	Requires the golden document sentinel in the model input, then returns LANGBOT_RAG_OK
mock-embedding	Returns deterministic normalized token-hash vectors from /v1/embeddings
mock-box	Calls exec twice to write/read /workspace/mock-llm.txt, then returns LANGBOT_BOX_OK
mock-401, mock-429, mock-503	Returns the selected HTTP failure
mock-slow	Delays the response by two seconds
mock-slow-stream	Delays each SSE chunk by 0.2 seconds

mock-rag requires aurora-parser-rag-9137 in the submitted messages. The golden HTML fixture contains this marker; retrieve it into the model context without including it in the user's question.

For mock-box, enable Box in the Runner and expose the native exec tool. Each tool result must contain LANGBOT_BOX_SENTINEL before the next step is returned. Tool-call IDs and fragmented streaming arguments use the normal OpenAI protocol. Scenario progress comes from each request's current conversation turn, so concurrent sessions and retries do not consume a shared script counter. Tagged group traffic additionally requires the previous user turn and assistant acknowledgement in the model input, and rejects another group's history. Final responses carry a per-turn acknowledgement checked by the traffic driver.

Add fixed response/tool sequences with --scenarios path/to/scenarios.json, using mock_llm_scenarios.json as the example. expect_tool_output checks a substring in the previous tool results; expect_prompt checks the submitted messages. The server returns tool calls; it never executes their commands. Request recordings contain prompts, tool results, responses, status, and disconnects, but no HTTP headers. Use test data only.

Run the focused HTTP checks on the Linux test host:

python3 -m unittest discover -s tools -p test_mock_llm.py -v

These checks validate the mock server. A LangBot regression must additionally verify real tool outputs and retrieved document IDs/content. Token usage is synthetic, and this endpoint does not evaluate answer quality or embedding models.

mock-embedding accepts string/batch inputs, float/base64 encoding, and optional dimensions (default 384). It tests the embedding protocol and RAG plumbing; token hashes do not establish semantic retrieval quality.

Normal traffic and resource collection

regression_traffic.py executes a JSON list of normal lbctl cases at a selected rate. Each case has a name, CLI arguments, and an expected response substring:

[{"name":"chat","args":["pipeline","run","PIPELINE_UUID","--message","Hello"],"expect":"LANGBOT_CHAT_OK"}]

Prepare a Workspace-bound CLI context and its API key environment variable, then run on the Linux test host:

python3 tools/regression_traffic.py \
  --lbctl /path/to/lbctl --config /path/to/lbctl.yaml \
  --cases /path/to/cases.json --output /path/to/new-results-directory \
  --duration 600 --rps 2 --workers 4 --cooldown 30 \
  --pid core=CORE_PID --pid mock=MOCK_PID \
  --health core=http://127.0.0.1:5300/healthz \
  --health mock=http://127.0.0.1:18080/healthz

Add --container CONTAINER_ID for Docker CPU/memory/PID statistics. The executing account must be able to inspect the selected process trees and Docker daemon. Resource collection reuses scripts/cloud_runtime_soak.py; it does not require cgroup v2. PID resource totals include descendants, so select disjoint trees.

The result directory contains requests.jsonl, resources.jsonl, progress.json, and summary.json. The requested rate is a target: four workers bound concurrency, and slow requests lower the achieved rate. Requests with unknown outcomes are recorded without retry. A command fails its business check if it returns an error or omits the expected response. Health/resource collection failures also make the run fail. API key values are redacted from recorded CLI output.

Latency percentiles include the entire CLI invocation and use successful requests only. Achieved request rate is requests / load_elapsed_seconds. health_samples counts sampling rounds; health_failures counts rounds with any failed health probe or process/container resource read.

Configured regression suites

regression_suite.py turns a JSON profile into a repeatable resource preparation, traffic, observation, verdict, and cleanup sequence. Start with regression_profiles/groups.json for continuous group conversations. The earlier normal.json profile retains independent CLI requests; parallel CLI workers alone do not simulate persistent groups.

Use a dedicated QA Workspace. First provision the instance with lbctl install or your source deployment, enable Docker Box, start the mock and embedding backends, and prepare a Workspace-bound lbctl config. Set the profile's CLI config path, backend URLs, and three plugin archive paths. The sample embedding endpoint must serve all-MiniLM-L6-v2; the mock serves generation only. Relative paths resolve against the profile directory, so adjust them if you move it.

Run on the Linux test host:

python3 tools/regression_suite.py validate --profile tools/regression_profiles/groups.json
python3 tools/regression_suite.py run \
  --profile tools/regression_profiles/groups.json --output .qa/regression/run-001

validate checks profile syntax and references without contacting the instance. The output directory for run must be new. CLI options --duration, --rps, and --workers override those values in every phase. No additional Python dependency is needed.

The profile controls:

• plugins: local archives or GitHub/marketplace request files, installed with lbctl plugin install ... --wait.
• resources: Providers, models, knowledge bases, document ingestion, Pipelines, and Bots, created in dependency order. ${resource_id} refers to a UUID returned by an earlier creation; ${workspace_uuid} is the current Workspace. Pipeline config patches merge with actual server defaults. Independent CLI cases can use existing resources supplied through vars; group mode creates its own signed HTTP Bots.
• cases: CLI argument arrays or http_group settings and expected business text. Integer weight values set the mix; 2/1/1/1 means 40/20/20/20. An optional body object becomes the case's request file. All workers share the weighted sequence.
• phases: each phase has a name, duration, target RPS, worker limit, and optional cooldown. Multiple phases provide a step load. In-flight requests drain at the end; the concurrency limit can reduce achieved RPS.
• observe: health URLs, host PIDs, PID files, and Docker container names/IDs. Set PIDs for disjoint process trees to avoid counting descendants twice. box_data_root discovers Box containers mounted below that deployment's data directory on every sample, including sandboxes created as new groups speak. The complete Job sets it automatically; on an existing instance, set it to the instance's Box data directory when sampling dynamic Docker sandboxes.
• thresholds: minimum achieved/target RPS ratio and maximum P95 seconds per case, in addition to the existing business and health/resource checks. The sample limits are starting acceptance settings, not established product SLOs.
• cleanup: on-success (default), always, or never. Only recorded resources created by this run are deleted, in reverse order. Plugins and the existing instance remain installed. Failed/unknown creations are not automatically retried; inspect the instance if a creation outcome is unknown.

Failed runs retain resources by default. After inspection, clean them with the same profile and credentials:

python3 tools/regression_suite.py cleanup \
  --profile tools/regression_profiles/groups.json --output .qa/regression/run-001

Cleanup checks the Workspace against owned.json and records remaining items after partial failures. Keep owned.json until cleanup completes. Each run also contains profile.json, resolved resources.json, and one directory per phase with the original request/resource records. The root summary.json includes target/achieved RPS, phase results, threshold failures, and the final verdict.

The regression_suite.py entry reuses an existing instance and backend processes. Use the complete lifecycle Job below to own deployment, credentials and backends. The independent normal.json cases use a new-session CLI entry. Group traffic uses the signed HTTP Bot adapter instead, as described below. Real IM adapter acceptance and resource-leak verdicts need separate scenarios. Test the suite plumbing on Linux with:

python3 -m unittest discover -s tools -p test_regression_suite.py -v

Persistent group traffic

The group profile creates three HTTP Bots routed to chat, Box, and RAG Pipelines. Their signed /bots/<uuid>/sync ingress produces real GroupMessage events and uses Core's session manager. Four groups follow a 2/1/1 case assignment; members take turns within each group. The same member IDs appear in different groups to exercise group isolation. Box session selection uses the group launcher ID.

In each group phase, groups selects the number of stable sessions and members_per_group selects the rotating member identities. rps remains the aggregate target, so a group's target speaking interval is groups / rps seconds. workers bounds total in-flight requests independently of group count. For example, 8 groups at 4 RPS target one message every 2 seconds per group. Both the Suite and Job accept --groups and --members-per-group overrides. The group count must cover the full weighted case sequence (4 in the sample).

Only one request is in flight per group. A failure or unknown outcome stops that group without retry while other groups continue. Every selected group must complete at least two turns in total for the phase to pass. The fake model checks that the preceding user message and assistant acknowledgement survived in actual model input, including when a different member speaks next. Cross-group markers fail the oracle. The normal Box/tool-output and RAG-context checks still apply.

group-state.json carries group IDs, turn numbers and acknowledgements across phases; existing groups keep their conversations when later phases add groups. It refuses changed cases or a state left in progress by interruption. Do not clear that marker to blindly resume: inspect the run and use a fresh output directory when the last turn's outcome is unknown. Each phase summary includes group-level turn counts and member IDs alongside case latency and health data.

The Suite generates the HTTP Bot signing secret and passes it through LANGBOT_QA_GROUP_SECRET. Resource request bodies containing credentials are stored under .private/; do not share that directory. Cases specify http_group with url, secret_env, and message instead of CLI args. Core needs no code changes. The /sync response collapses streaming parts: this tests multi-turn group processing, not real IM delivery or callback streaming order. Every selected group must also receive traffic in each phase. Group latency measures the signed HTTP request through its completed reply; independent CLI latency includes CLI startup. Compare results within the same traffic mode.

lbctl v0.3 may report Bot verification failure after a successful creation because its safe projection omits event bindings. The Suite records the returned Bot UUID before reading the full Core Bot API with the endpoint, Workspace and environment credential confirmed by lbctl whoami. It then checks the masked signing secret and effective route with a signed warmup turn before load. The creation is not retried.

Complete lifecycle Job

regression_job.py owns the deployment as well as the suite. Use regression_profiles/job.json to pin the Core Release and select the suite and plugin input directory. The job runs this chain:

1. lbctl install --profile all creates a fresh deployment inside the run directory.
2. A Compose override starts our mock backend on the deployment's private network and injects a generated test-only API key. The Job binds its own CLI context.
3. The suite installs plugins, creates resources and ingests documents. Each case runs once as warmup before measured traffic starts.
4. Configured load phases run with health, process-tree and container sampling.
5. Service-log tails and raw request/resource records are retained. Successful Jobs stop the deployment, remove only sandboxes mounted under their own data root, and use lbctl uninstall --purge-data to reclaim the deployment.

Both generation and embedding use our mock backend, with no external model service or real model credentials. Plugin archives, the pinned Core image, lbctl, Docker and Compose v2 are Job inputs/prerequisites. This wrapper uses PyYAML, already included in LangBot's dependencies, to verify the CLI's deployment record and recover ownership after an interrupted install.

Run on a Linux host with Docker access and permission to inspect the selected container processes. Use an idle dedicated Docker daemon: current Box runtime startup cleanup can remove sandboxes from another runtime. The Job refuses to start when existing Box sandboxes are present. The current lbctl installer also reserves host port 5401 and 2280–2285, independently of its HTTP port.

Set the plugin directory and the lbctl executable in job.json, then run:

uv run --no-sync python tools/regression_job.py validate --profile tools/regression_profiles/job.json
uv run --no-sync python tools/regression_job.py run \
  --profile tools/regression_profiles/job.json --output .qa/regression/job-001

--duration, --rps and --workers override every load phase. Each output directory must be new. Job cleanup policy is on-success, always, or never; it governs the whole deployment, whereas suite cleanup governs API resources when a suite runs on an existing deployment. purge_data: false preserves the deployment data after removing containers. Failed Jobs retain their environment by default; after inspection use:

uv run --no-sync python tools/regression_job.py cleanup \
  --output .qa/regression/job-001

The root summary.json is the final Job verdict; suite/ contains the measured results, mock/requests.jsonl contains backend recordings, and services.log contains up to 10,000 log lines per service. Generated credentials and deployment inputs stay in .private/; share the result files, not that directory. A Job has its own CLI locator/config and records its deployment identity before cleanup. Changing that identity makes cleanup fail closed. Cleanup uses the saved Job state and does not require the original profile or plugin input files.

The Job replaces the real embedding model with mock-embedding. This measures LangBot/plugin/Box/RAG overhead, not real-model inference cost or retrieval quality. Run the Job plumbing checks on Linux with:

uv run --no-sync python -m unittest discover -s tools -p 'test_*.py' -v