LangBot/tools/README.md

# LangBot regression tools

Run the deterministic OpenAI Chat Completions endpoint with Python 3.11+, without
API credentials or third-party dependencies:

```sh
python3 tools/mock_llm.py --port 18080 --record /tmp/langbot-mock-requests.jsonl
```

In LangBot, create an `openai-chat-completions` Provider whose base URL is
`http://127.0.0.1:18080/v1` and whose API key is any test-only value. Use a model
from `GET /v1/models`. The loopback address works when Core and this tool run on
the same host; containers must use an address reachable from their own network.

| Model | Behavior |
| --- | --- |
| `mock-chat` | Returns `LANGBOT_CHAT_OK`, including SSE streaming |
| `mock-rag` | Requires the golden document sentinel in the model input, then returns `LANGBOT_RAG_OK` |
| `mock-embedding` | Returns deterministic normalized token-hash vectors from `/v1/embeddings` |
| `mock-box` | Calls `exec` twice to write/read `/workspace/mock-llm.txt`, then returns `LANGBOT_BOX_OK` |
| `mock-401`, `mock-429`, `mock-503` | Returns the selected HTTP failure |
| `mock-slow` | Delays the response by two seconds |
| `mock-slow-stream` | Delays each SSE chunk by 0.2 seconds |

`mock-rag` requires `aurora-parser-rag-9137` in the submitted messages. The
[golden HTML fixture](../skills/skills/langbot-testing/fixtures/rag/parser-golden.html)
contains this marker; retrieve it into the model context without including it
in the user's question.

For `mock-box`, enable Box in the Runner and expose the native `exec` tool. Each
tool result must contain `LANGBOT_BOX_SENTINEL` before the next step is returned.
Tool-call IDs and fragmented streaming arguments use the normal OpenAI protocol.
Scenario progress comes from each request's current conversation turn, so
concurrent sessions and retries do not consume a shared script counter.
Tagged group traffic additionally requires the previous user turn and assistant
acknowledgement in the model input, and rejects another group's history. Final
responses carry a per-turn acknowledgement checked by the traffic driver.

Add fixed response/tool sequences with `--scenarios path/to/scenarios.json`, using
`mock_llm_scenarios.json` as the example. `expect_tool_output` checks a substring
in the previous tool results; `expect_prompt` checks the submitted messages.
The server returns tool calls; it never executes
their commands. Request recordings contain prompts, tool results, responses,
status, and disconnects, but no HTTP headers. Use test data only.

Run the focused HTTP checks on the Linux test host:

```sh
python3 -m unittest discover -s tools -p test_mock_llm.py -v
```

These checks validate the mock server. A LangBot regression must additionally
verify real tool outputs and retrieved document IDs/content. Token usage is
synthetic, and this endpoint does not evaluate answer quality or embedding models.

`mock-embedding` accepts string/batch inputs, float/base64 encoding, and optional
dimensions (default 384). It tests the embedding protocol and RAG plumbing;
token hashes do not establish semantic retrieval quality.

## Normal traffic and resource collection

`regression_traffic.py` executes a JSON list of normal `lbctl` cases at a selected
rate. Each case has a name, CLI arguments, and an expected response substring:

```json
[{"name":"chat","args":["pipeline","run","PIPELINE_UUID","--message","Hello"],"expect":"LANGBOT_CHAT_OK"}]
```

Prepare a Workspace-bound CLI context and its API key environment variable, then
run on the Linux test host:

```sh
python3 tools/regression_traffic.py \
  --lbctl /path/to/lbctl --config /path/to/lbctl.yaml \
  --cases /path/to/cases.json --output /path/to/new-results-directory \
  --duration 600 --rps 2 --workers 4 --cooldown 30 \
  --pid core=CORE_PID --pid mock=MOCK_PID \
  --health core=http://127.0.0.1:5300/healthz \
  --health mock=http://127.0.0.1:18080/healthz
```

Add `--container CONTAINER_ID` for Docker CPU/memory/PID statistics. The executing
account must be able to inspect the selected process trees and Docker daemon.
Resource collection reuses `scripts/cloud_runtime_soak.py`; it does not require
cgroup v2. PID resource totals include descendants, so select disjoint trees.

The result directory contains `requests.jsonl`, `resources.jsonl`, `progress.json`,
and `summary.json`. The requested rate is a target: four workers bound concurrency,
and slow requests lower the achieved rate. Requests with unknown outcomes are
recorded without retry. A command fails its business check if it returns an error
or omits the expected response. Health/resource collection failures also make the
run fail. API key values are redacted from recorded CLI output.

Latency percentiles include the entire CLI invocation and use successful
requests only. Achieved request rate is `requests / load_elapsed_seconds`.
`health_samples` counts sampling rounds; `health_failures` counts rounds with
any failed health probe or process/container resource read.

## Configured regression suites

[`regression_suite.py`](regression_suite.py) turns a JSON profile into a repeatable
resource preparation, traffic, observation, verdict, and cleanup sequence. Start
with [`regression_profiles/groups.json`](regression_profiles/groups.json) for
continuous group conversations. The earlier
[`normal.json`](regression_profiles/normal.json) profile retains independent CLI
requests; parallel CLI workers alone do not simulate persistent groups.

Use a dedicated QA Workspace. First provision the instance with `lbctl install`
or your source deployment, enable Docker Box, start the mock and embedding
backends, and prepare a Workspace-bound `lbctl` config. Set the profile's CLI
config path, backend URLs, and three plugin archive paths. The sample embedding
endpoint must serve `all-MiniLM-L6-v2`; the mock serves generation only. Relative
paths resolve against the profile directory, so adjust them if you move it.

Run on the Linux test host:

```sh
python3 tools/regression_suite.py validate --profile tools/regression_profiles/groups.json
python3 tools/regression_suite.py run \
  --profile tools/regression_profiles/groups.json --output .qa/regression/run-001
```

`validate` checks profile syntax and references without contacting the instance.
The output directory for `run` must be new. CLI options `--duration`, `--rps`, and
`--workers` override those values in every phase. No additional Python dependency
is needed.

The profile controls:

- `plugins`: local archives or GitHub/marketplace request files, installed with
  `lbctl plugin install ... --wait`.
- `resources`: Providers, models, knowledge bases, document ingestion, Pipelines,
  and Bots, created in dependency order. `${resource_id}` refers to a UUID
  returned by an earlier creation; `${workspace_uuid}` is the current Workspace.
  Pipeline config patches merge with actual server defaults. Independent CLI
  cases can use existing resources supplied through `vars`; group mode creates
  its own signed HTTP Bots.
- `cases`: CLI argument arrays or `http_group` settings and expected business text. Integer `weight`
  values set the mix; `2/1/1/1` means `40/20/20/20`. An optional `body` object
  becomes the case's request file. All workers share the weighted sequence.
- `phases`: each phase has a name, duration, target RPS, worker limit, and optional
  cooldown. Multiple phases provide a step load. In-flight requests drain at the
  end; the concurrency limit can reduce achieved RPS.
- `observe`: health URLs, host PIDs, PID files, and Docker container names/IDs.
  Set PIDs for disjoint process trees to avoid counting descendants twice.
  `box_data_root` discovers Box containers mounted below that deployment's data
  directory on every sample, including sandboxes created as new groups speak.
  The complete Job sets it automatically; on an existing instance, set it to
  the instance's Box data directory when sampling dynamic Docker sandboxes.
- `thresholds`: minimum achieved/target RPS ratio and maximum P95 seconds per
  case, in addition to the existing business and health/resource checks. The
  sample limits are starting acceptance settings, not established product SLOs.
- `cleanup`: `on-success` (default), `always`, or `never`. Only recorded resources
  created by this run are deleted, in reverse order. Plugins and the existing
  instance remain installed. Failed/unknown creations are not automatically
  retried; inspect the instance if a creation outcome is unknown.

Failed runs retain resources by default. After inspection, clean them with the
same profile and credentials:

```sh
python3 tools/regression_suite.py cleanup \
  --profile tools/regression_profiles/groups.json --output .qa/regression/run-001
```

Cleanup checks the Workspace against `owned.json` and records remaining items
after partial failures. Keep `owned.json` until cleanup completes. Each run also
contains `profile.json`, resolved `resources.json`, and one directory per phase
with the original request/resource records. The root `summary.json` includes
target/achieved RPS, phase results, threshold failures, and the final verdict.

The `regression_suite.py` entry reuses an existing instance and backend processes.
Use the complete lifecycle Job below to own deployment, credentials and backends.
The independent `normal.json` cases use a new-session CLI entry. Group traffic
uses the signed HTTP Bot adapter instead, as described below. Real IM adapter
acceptance and resource-leak verdicts need separate scenarios. Test the suite
plumbing on Linux with:

```sh
python3 -m unittest discover -s tools -p test_regression_suite.py -v
```

### Persistent group traffic

The group profile creates three HTTP Bots routed to chat, Box, and RAG Pipelines.
Their signed `/bots/<uuid>/sync` ingress produces real `GroupMessage` events and
uses Core's session manager. Four groups follow a `2/1/1` case assignment;
members take turns within each group. The same member IDs appear in different
groups to exercise group isolation. Box session selection uses the group launcher ID.

In each group phase, `groups` selects the number of stable sessions and
`members_per_group` selects the rotating member identities. `rps` remains the
aggregate target, so a group's target speaking interval is `groups / rps`
seconds. `workers` bounds total in-flight requests independently of group count.
For example, 8 groups at 4 RPS target one message every 2 seconds per group.
Both the Suite and Job accept `--groups` and `--members-per-group` overrides.
The group count must cover the full weighted case sequence (4 in the sample).

Only one request is in flight per group. A failure or unknown outcome stops that
group without retry while other groups continue. Every selected group must
complete at least two turns in total for the phase to pass. The fake model checks
that the preceding user message and assistant acknowledgement survived in actual
model input, including when a different member speaks next. Cross-group markers
fail the oracle. The normal Box/tool-output and RAG-context checks still apply.

`group-state.json` carries group IDs, turn numbers and acknowledgements across
phases; existing groups keep their conversations when later phases add groups.
It refuses changed cases or a state left in progress by interruption. Do not
clear that marker to blindly resume: inspect the run and use a fresh output
directory when the last turn's outcome is unknown. Each phase summary includes
group-level turn counts and member IDs alongside case latency and health data.

The Suite generates the HTTP Bot signing secret and passes it through
`LANGBOT_QA_GROUP_SECRET`. Resource request bodies containing credentials are
stored under `.private/`; do not share that directory. Cases specify `http_group`
with `url`, `secret_env`, and `message` instead of CLI `args`. Core needs no code
changes. The `/sync` response collapses streaming parts: this tests multi-turn
group processing, not real IM delivery or callback streaming order.
Every selected group must also receive traffic in each phase. Group latency
measures the signed HTTP request through its completed reply; independent CLI
latency includes CLI startup. Compare results within the same traffic mode.

`lbctl` v0.3 may report Bot verification failure after a successful creation
because its safe projection omits event bindings. The Suite records the returned
Bot UUID before reading the full Core Bot API with the endpoint, Workspace and
environment credential confirmed by `lbctl whoami`. It then
checks the masked signing secret and effective route with a signed warmup turn
before load. The creation is not retried.

## Complete lifecycle Job

[`regression_job.py`](regression_job.py) owns the deployment as well as the suite.
Use [`regression_profiles/job.json`](regression_profiles/job.json) to pin the Core
Release and select the suite and plugin input directory. The job runs this chain:

1. `lbctl install --profile all` creates a fresh deployment inside the run directory.
2. A Compose override starts our mock backend on the deployment's private network
   and injects a generated test-only API key. The Job binds its own CLI context.
3. The suite installs plugins, creates resources and ingests documents. Each case
   runs once as warmup before measured traffic starts.
4. Configured load phases run with health, process-tree and container sampling.
5. Service-log tails and raw request/resource records are retained. Successful
   Jobs stop the deployment, remove only sandboxes mounted under their own data
   root, and use `lbctl uninstall --purge-data` to reclaim the deployment.

Both generation and embedding use our mock backend, with no external model
service or real model credentials. Plugin archives, the pinned Core image,
`lbctl`, Docker and Compose v2 are Job inputs/prerequisites. This wrapper uses
PyYAML, already included in LangBot's dependencies, to verify the CLI's deployment
record and recover ownership after an interrupted install.

Run on a Linux host with Docker access and permission to inspect the selected
container processes. Use an idle dedicated Docker daemon: current Box runtime
startup cleanup can remove sandboxes from another runtime. The Job refuses to
start when existing Box sandboxes are present. The current `lbctl` installer
also reserves host port 5401 and 2280–2285, independently of its HTTP port.

Set the plugin directory and the `lbctl` executable in `job.json`, then run:

```sh
uv run --no-sync python tools/regression_job.py validate --profile tools/regression_profiles/job.json
uv run --no-sync python tools/regression_job.py run \
  --profile tools/regression_profiles/job.json --output .qa/regression/job-001
```

`--duration`, `--rps` and `--workers` override every load phase. Each output
directory must be new. Job cleanup policy is `on-success`, `always`, or `never`;
it governs the whole deployment, whereas suite cleanup governs API resources
when a suite runs on an existing deployment. `purge_data: false` preserves the
deployment data after removing containers. Failed Jobs retain their environment
by default; after inspection use:

```sh
uv run --no-sync python tools/regression_job.py cleanup \
  --output .qa/regression/job-001
```

The root `summary.json` is the final Job verdict; `suite/` contains the measured
results, `mock/requests.jsonl` contains backend recordings, and `services.log`
contains up to 10,000 log lines per service. Generated credentials and deployment
inputs stay in `.private/`; share the result files, not that directory. A Job has
its own CLI locator/config and records its deployment identity before cleanup.
Changing that identity makes cleanup fail closed. Cleanup uses the saved Job
state and does not require the original profile or plugin input files.

The Job replaces the real embedding model with `mock-embedding`. This measures
LangBot/plugin/Box/RAG overhead, not real-model inference cost or retrieval
quality. Run the Job plumbing checks on Linux with:

```sh
uv run --no-sync python -m unittest discover -s tools -p 'test_*.py' -v
```

## Finite core smoke

Use [`regression_profiles/smoke.json`](regression_profiles/smoke.json) for a finite
functional check on an existing Core. It prepares 35 resources through `lbctl`,
ingests HTML and plain-text fixtures through GeneralParsers, then executes 28
cases / 43 requests. Reports group the verdicts by stage:

| Stage | Cases and independent checks |
| --- | --- |
| Configuration | Identity and plugins; provider, model, Pipeline and knowledge-base readback; document completed status and Workspace/KB ownership |
| Ingress | Missing and incorrect signatures return HTTP 401 / code 40101 without creating a group AgentRun; valid signatures are exercised by group conversations |
| Conversation | Ordinary and Unicode chat; chat, Box and RAG groups each use two interleaved groups with three turns and rotating members; model input must retain prior user/reply history without cross-group history |
| Model | Slow SSE reply; HTTP 401, 429 and 503 primary failures followed by successful fallback; recorded model/status/stream sequence must match |
| Tools | Docker write/read across rounds, two tool calls in one model response, and expected exit code 7 with separate stdout/stderr returned to the model |
| Delivery | ASCII and Unicode filenames/content delivered as one attachment with exact bytes |
| Retrieval | HTML and text parsing/ingestion, RAG model input, retrieval from two knowledge bases with distinct sentinels and KB ownership, and an empty knowledge base returning an empty list |

Cleanup is checked separately: delete journaled API resources and confirm their
UUIDs are absent from CLI list readback. Evidence goes to `cleanup-checks.json`;
cleanup failures affect the overall verdict.

Start a dedicated `mock_llm.py` with the bundled scenarios and `--record` as
shown above. Copy the smoke profile and configure the CLI executable/config,
service URLs, plugin archives, `vars.mock_records` to that mock recording file,
and `vars.core_db` to the target Core's existing SQLite `langbot.db`. The suite
must read both local files. Keep fixture paths valid relative to the copied
profile. Do not share the mock with other traffic during this run: fallback
checks compare requests appended during each case.

Set `plugins: []` to reuse installed plugins; LocalAgent, LangRAG and
GeneralParsers must still be enabled and initialized. The embedding endpoint
must serve the configured model (`all-MiniLM-L6-v2` by default). Generation is
deterministic mock traffic; Core, plugins, Box, document parsing and retrieval
execute normally.

```sh
uv run --no-sync python tools/regression_suite.py validate --profile /path/to/smoke.json
uv run --no-sync python tools/regression_suite.py run \
  --profile /path/to/smoke.json --output .qa/smoke-001
```

A correct reply alone does not pass. Each Pipeline/group request must match one
new AgentRun in the authenticated workspace with `completed` status and a persisted
`run.completed` event. The ledger is opened read-only. This version requires
local SQLite access; remote-only Core and PostgreSQL ledgers are not supported.
The observation timeout does not extend the Runner deadline or retry requests.
Unknown outcomes fail the suite; later turns of that group are skipped while
independent cases continue. An expected tool exit code of 7 passes only when
its output is correct and the agent handles the result and completes normally.

`summary.json` contains the overall and per-stage verdicts; exit code 0 means all
checks and required cleanup passed. `cases.json` records per-request evidence,
`environment.json` records identity/capabilities/plugins, and `owned.json` tracks
created resources. Successful runs delete those API resources. Failures retain
them by default; after inspection use:

```sh
uv run --no-sync python tools/regression_suite.py cleanup \
  --profile /path/to/smoke.json --output .qa/smoke-001
```

Box sessions and AgentRun records follow the reused service's normal retention
policy. Load override flags are rejected for smoke. `stage` labels cases;
`fields` compares exact response values by dotted path (including array indices),
`forbid` rejects response substrings, and `mock_requests` checks the actual
model/status/stream sequence. `box_exec` defaults to exit code 0; error-handling
cases can specify `exit_code` and `stderr` explicitly.

To use a fresh deployment, select this suite in `job.json`; the Job supplies its
SQLite/recording paths and uses mock embedding. Smoke skips load warmup and timed
phases. The dedicated Docker daemon requirement above still applies.

## PR smoke gate

`Core Smoke` runs regression-tool tests and a real Core smoke in parallel on
GitHub-hosted Linux runners. The smoke starts Core from the checked-out candidate,
uses the SDK from `uv.lock`, the existing official-plugin source pin, `lbctl`
v0.3.0, a Docker Box and mock generation/embedding. It needs no model credentials
or access to a developer's running environment.

Dependency installation, service startup, sandbox image download and plugin
installation are preparation. After readiness, the gate allows **60 seconds total**
for API resource creation, 28 cases / 37 requests (two turns per group), verdict
checks and API cleanup. A missing verdict, failed check or timeout fails the gate.
The ordinary expanded profile keeps three turns / 43 requests. The entire cold
workflow takes longer than a minute; it has a separate 15-minute timeout.

On an existing prepared stack, use:

```sh
uv run --no-sync python tools/regression_gate.py \
  --profile /path/to/smoke.json --output .qa/gate-001
```

`gate.json` records the budget, measured duration, verdict and process resource
peaks. `resources.jsonl` samples configured process trees every second (RSS,
cumulative CPU time, threads and file descriptors); suite artifacts contain
per-case timings and business evidence. These are diagnostics, not calibrated
performance-regression thresholds. Docker sandbox resource sampling remains in
the longer load runner. A timeout kills the local Suite/CLI process group without
retrying requests; a reused remote service can still have an in-flight run, so
inspect its retained ownership journal before cleanup. CI tears down its own
service processes and owned sandboxes afterward.

The workflow always creates its two checks on PRs and cancels superseded runs.
After the workflow is merged, configure `Regression tools` and `Core smoke (60s)`
as required status checks in the repository ruleset. A green workflow alone does
not enforce merge protection. Longer load and soak tests use the existing Suite
and Job entry points with separately calibrated latency, throughput and resource
budgets; the one-minute smoke does not replace them.