LangBot/tests/README.md

LangBot Test Suite

This directory contains the LangBot backend test suite, including unit tests, integration tests, startup E2E tests, and container-backed Box runtime tests.

Quality Gate Layers

LangBot uses a layered quality gate system for developers and CI:

Layer	Command	What it runs	When to use
Quick	make test-quick or bash scripts/test-quick.sh	Ruff lint + Unit tests + Smoke tests	Before every commit
Fast Integration	make test-integration-fast or bash scripts/test-integration-fast.sh	SQLite/API/Pipeline integration (no external services)	Before PR, weekly
Backend E2E	uv run --python 3.12 pytest tests/e2e -q --tb=short	Starts a real LangBot process with minimal config	Before release, CI
Box Integration	uv run --python 3.12 pytest tests/integration_tests -q --tb=short	Real Box sandbox/runtime integration	Before Box/runtime changes, CI
Frontend E2E	cd web && pnpm test:e2e	Playwright smoke tests with mocked backend and Space APIs	Before web changes, CI
Coverage Gate	make test-coverage or bash scripts/test-coverage.sh	All tests with coverage, threshold: 18%	Before merge, CI
Full Local	make test-all-local	Quick + Integration + Coverage	Before major changes

Note: PostgreSQL migration tests and slow tests are NOT in local default gates. They run in separate CI workflows. Frontend Playwright tests live under web/tests/e2e and are documented in web/README.md.

Core handler contracts and the LocalAgent fixture

The handler unit tests execute Core's real ChatMessageHandler with SDK messages/queries and a fake orchestrator. They check plugin-altered input, stream output snapshots, completion, response IDs, and query isolation. They do not require LocalAgent to be installed and do not test its internal agent loop. Assertions on a hand-written UUID/list simulation are not a substitute for exercising the handler.

test_local_runner_fake_provider.py is a separate cross-repository contract test: real Core, real SDK Plugin Runtime, and the official LocalAgent plugin, with deterministic Host-side model/tool/RAG resources. It checks installation, Host action routing, ledger/transcript persistence, and selected plugin journeys without a paid model provider. LocalAgent's own unit tests still belong with the plugin. The Box/file-transfer probe additionally needs Docker.

CI's Local Agent Contract E2E job resolves the latest LocalAgent from the public Space version API on every run, matching Core's version selection. It saves local-agent-resolution.json with the resolved version, exact download URL, archive SHA256, and resolution time. Only the version/checksum-addressed archive is cached; latest metadata is always fetched anew. The script checks every cache hit and preserves the signed archive for normal Runtime admission. Space's checksum is the original archive SHA256; its normalized artifact_digest is different and must not be substituted. No Space token or model API key is needed.

Run the same path locally:

export LANGBOT_E2E_LOCAL_AGENT_PACKAGE="$PWD/.pytest_cache/local-agent/LocalAgent.lbpkg"
uv run python scripts/test-local-agent-fixture.py resolve .pytest_cache/local-agent-resolution.json
uv run python scripts/test-local-agent-fixture.py fetch .pytest_cache/local-agent-resolution.json "$LANGBOT_E2E_LOCAL_AGENT_PACKAGE"
LANGBOT_E2E_REQUIRE_LOCAL_AGENT=1 uv run pytest tests/e2e/test_local_runner_fake_provider.py -q -rs --durations=10

A missing explicitly configured package always fails. Without the environment variables, local runs may package a sibling langbot-plugin-demo/Runner/LocalAgent checkout or skip if it is absent. CI requires the resolved package, checks Docker, and rejects any skipped tests; it never relies on that sibling checkout. A metadata/download/checksum failure appears in provisioning separately from a Core/Runtime test failure. Do not silently fall back to an older release when latest is unavailable or incompatible.

Because this tracks latest, an upstream plugin release can break an unchanged Core commit. Compare the saved resolution artifacts, and replay a failed run using its local-agent-resolution.json with the fetch command before assigning the regression to Core. Replay still validates the saved checksum and therefore fails clearly if Space replaces or removes that artifact. Plugin internals and new SDK compatibility requirements belong to their owning repositories.

The Python 3.12 unit/smoke matrix job also collects line and branch coverage and enforces the unchanged 18% line threshold. The pytest-cov total with --cov-branch combines lines and branches; it is not either individual metric. Python 3.11 and 3.13 retain their compatibility runs. The existing Coverage Gate check aggregates these results without rerunning the suite. The new LocalAgent job is mandatory within this workflow; making its check branch-protection-required remains a repository setting.

Source complexity and coverage reports

The 3.12 job uploads coverage.json, coverage.xml, complexity.json, and complexity.md, and shows the summary in the Actions job summary. The complexity report uses the already-installed Ruff C901 McCabe rule with a threshold of zero to measure every function, without adding dependencies or a new complexity gate. It covers src/langbot/**/*.py, including vendored libs, migrations, and templates. It records the Ruff version so comparisons have an explicit analyzer.

uv run pytest tests/unit_tests tests/smoke --cov=langbot --cov-branch --cov-report=json
uv run python scripts/test-complexity.py --json complexity.json --summary complexity.md \
  --coverage-json coverage.json --base-ref origin/master

The report includes function counts/distribution, hotspots, separate line/branch coverage, and function changes against the PR base (or previous push). It matches qualified names and Git-detected file renames; function renames appear as additions/deletions. Function-range coverage overlaps for nested functions and is a navigation aid, not a sum of independent coverage totals. Missing coverage is unknown, not zero.

Use these to identify changed, complex functions needing stronger tests. Coverage shows execution and complexity estimates branching, but neither proves assertions would detect defects. Keep real behavior assertions, targeted mutation probes, integration contracts, skip reasons, and timings alongside these reports. Do not combine the metrics into a single quality score or gate existing complexity debt with an arbitrary threshold.

Developer Workflow

# Daily: Quick self-test
bash scripts/test-quick.sh

# Before PR: Full local gate
make test-all-local

# Or run each layer separately:
bash scripts/test-quick.sh           # ~2 min
bash scripts/test-integration-fast.sh # ~3 min
bash scripts/test-coverage.sh         # ~8 min
uv run --python 3.12 pytest tests/e2e -q --tb=short
uv run --python 3.12 pytest tests/integration_tests -q --tb=short
cd web && pnpm test:e2e

Coverage Baseline

Current coverage threshold: 18% Actual coverage: 30%

This is a conservative baseline to prevent coverage regression. It does NOT represent the final quality target. Key modules have higher coverage:

• pipeline.preproc.preproc: 53%
• pipeline.process.process: 96%
• pipeline.respback.respback: 88%
• telemetry.telemetry: 87%
• provider.session.sessionmgr: 100%
• provider.tools.toolmgr: 83%
• storage.providers.s3storage: 80%

Important Note

Due to circular import dependencies in the pipeline module structure, the test files use lazy imports via importlib.import_module() instead of direct imports. This ensures tests can run without triggering circular import errors.

Structure

tests/
├── __init__.py
├── factories/                    # Shared test factories
│   ├── __init__.py              # Factory exports
│   ├── app.py                   # FakeApp factory
│   ├── message.py               # Message/query factories
│   ├── provider.py              # FakeProvider factory
│   └── platform.py              # FakePlatform factory
├── integration/                  # Integration tests (real resources)
│   ├── __init__.py
│   ├── api/                     # HTTP API tests
│   │   ├── __init__.py
│   │   └── test_smoke.py        # API smoke tests
│   ├── pipeline/                # Pipeline stage-chain tests
│   │   ├── __init__.py
│   │   └── test_full_flow.py    # Full flow integration
│   └── persistence/             # Database/persistence tests
│       ├── __init__.py
│       └── test_migrations.py   # Alembic migration tests
├── e2e/                          # Real LangBot startup E2E tests
│   ├── conftest.py
│   ├── test_startup.py
│   └── utils/
├── integration_tests/            # Container-backed integration tests
│   └── box/                      # Box runtime and MCP process tests
├── smoke/                        # Smoke tests (quick validation)
│   └── test_fake_message_flow.py
├── unit_tests/                   # Unit tests
│   ├── box/                      # Box module tests
│   ├── config/                   # Configuration tests
│   ├── pipeline/                 # Pipeline stage tests
│   │   └── conftest.py          # Shared fixtures and test infrastructure
│   ├── platform/                 # Platform adapter tests
│   ├── plugin/                   # Plugin system tests
│   │   └── test_handler_actions.py # Action handler tests
│   ├── provider/                 # Provider tests
│   │   ├── test_session_manager.py # SessionManager tests
│   │   └── test_tool_manager.py    # ToolManager tests
│   ├── rag/                      # RAG tests
│   │   └── test_file_storage.py   # File/ZIP storage tests
│   ├── storage/                  # Storage tests
│   │   └── test_s3storage.py      # S3StorageProvider tests
│   ├── vector/                   # Vector tests
│   │   └── test_vdb_filter_conversion.py # VDB filter tests
│   └── telemetry/                # Telemetry tests (rewritten)
├── utils/                        # Test utilities
│   ├── __init__.py
│   └── import_isolation.py      # sys.modules isolation for circular imports
└── README.md                     # This file

Test Factories

The tests/factories/ package provides reusable test factories:

from tests.factories import (
    FakeApp,          # Mock application
    FakeProvider,     # Fake LLM provider
    FakePlatform,     # Fake platform adapter
    text_query,       # Create text query
    group_text_query, # Create group query
    command_query,    # Create command query
)

# Create fake app
app = FakeApp()

# Create query with text
query = text_query("hello world")

# Create fake provider that returns specific response
provider = FakeProvider().returns("test response")

# Create fake platform for outbound capture
platform = FakePlatform()
await platform.reply_message(query.message_event, reply_chain)
outbound = platform.get_outbound_messages()

See tests/factories/__init__.py for all available factories.

Test Architecture

Fixtures (conftest.py)

The test suite uses a centralized fixture system that provides:

• MockApplication: Comprehensive mock of the Application object with all dependencies
• Mock objects: Pre-configured mocks for Session, Conversation, Model, Adapter
• Sample data: Ready-to-use Query objects, message chains, and configurations
• Helper functions: Utilities for creating results and common assertions

Design Principles

1. Isolation: Each test is independent and doesn't rely on external systems
2. Mocking: All external dependencies are mocked to ensure fast, reliable tests
3. Coverage: Tests cover happy paths, edge cases, and error conditions
4. Extensibility: Easy to add new tests by reusing existing fixtures

Running Tests

Quick self-test for developers

For local branch validation without real provider keys:

make test-quick

or

bash scripts/test-quick.sh

This runs:

1. Ruff lint check
2. Unit tests
3. Smoke tests

Suitable for quick validation before committing.

Using the test runner script (recommended for full coverage)

bash run_tests.sh

This script automatically:

• Activates the virtual environment
• Installs test dependencies if needed
• Runs tests with coverage
• Generates HTML coverage report

Manual test execution

Run all unit tests

uv run pytest tests/unit_tests/ --cov=langbot --cov-report=xml --cov-report=term

Run specific test module

uv run pytest tests/unit_tests/pipeline/ -v

Run specific test file

uv run pytest tests/unit_tests/pipeline/test_bansess.py -v

Run with coverage

uv run pytest tests/unit_tests/pipeline/ --cov=langbot --cov-report=html

Run specific test

uv run pytest tests/unit_tests/pipeline/test_bansess.py::test_bansess_whitelist_allow -v

Using markers

# Run only unit tests
uv run pytest tests/unit_tests/ -m unit

# Run only integration tests
uv run pytest tests/integration/ -m integration

# Run integration tests excluding slow ones
uv run pytest tests/integration/ -m "not slow" -q

# Skip slow tests
uv run pytest tests/unit_tests/ -m "not slow"

Running integration tests

Integration tests validate real system behavior with actual database/network resources.

# Run all integration tests (excluding slow ones)
uv run pytest tests/integration/ -m "not slow" -q

# Run SQLite migration integration tests
uv run pytest tests/integration/persistence/test_migrations.py -q --tb=short

# Run API smoke integration tests
uv run pytest tests/integration/api/test_smoke.py -q

# Run pipeline full-flow integration tests
uv run pytest tests/integration/pipeline/test_full_flow.py -q

# Run with verbose output
uv run pytest tests/integration/ -v

Note: Integration tests use:

• Temporary databases (tmp_path) for persistence tests
• Fake app/services for API tests (no real provider/platform)
• Fake runner/provider for pipeline tests (no real LLM API)
• Do not require external services

Running migration tests locally

SQLite migration tests can be run locally without any external dependencies:

# SQLite migration tests (uses tmp_path, no external DB needed)
uv run pytest tests/integration/persistence/test_migrations.py -q --tb=short

PostgreSQL migration tests require an external PostgreSQL database:

# PostgreSQL migration tests (requires PostgreSQL service)
# Tests are marked as slow and skipped if TEST_POSTGRES_URL is not set
TEST_POSTGRES_URL=postgresql+asyncpg://user:pass@localhost:5432/test_db \
    uv run pytest tests/integration/persistence/test_migrations_postgres.py -q --tb=short

# Or skip by default (no PostgreSQL available)
uv run pytest tests/integration/persistence/test_migrations_postgres.py -q --tb=short
# Output: SKIPPED (TEST_POSTGRES_URL not set)

Note: PostgreSQL tests are not included in fast integration gate because they:

• Require external PostgreSQL service
• Are marked with @pytest.mark.slow
• Need TEST_POSTGRES_URL environment variable

CI workflow .github/workflows/test-migrations.yml runs:

• SQLite tests in test-migrations-sqlite job (fast, no external services)
• PostgreSQL tests in test-migrations-postgres job (uses PostgreSQL service container)

Running pipeline integration tests locally

Pipeline full-flow integration tests validate real stage interactions:

# Run pipeline integration tests (uses fake runner, no real LLM API)
uv run pytest tests/integration/pipeline/test_full_flow.py -q --tb=short

# Run with coverage for pipeline modules
uv run pytest tests/integration/pipeline \
    --cov=langbot.pkg.pipeline.preproc.preproc \
    --cov=langbot.pkg.pipeline.process.process \
    --cov=langbot.pkg.pipeline.respback.respback \
    --cov-report=term -q

These tests:

• Use FakeRunner class to simulate LLM responses without real API calls
• Import real PreProcessor, MessageProcessor, SendResponseBackStage stages
• Validate stage chain: PreProcessor → Processor → SendResponseBackStage
• Test prevent_default, exception handling, and full message flow
• Do not require real LLM provider keys

Running backend E2E startup tests

Backend E2E tests start a real LangBot process with a generated minimal data/config.yaml, SQLite database, local storage, and embedded Chroma path. They do not require provider keys or external services.

uv run --python 3.12 pytest tests/e2e -q --tb=short

These tests verify startup orchestration, migrations, API route registration, and the minimal no-LLM startup path. The E2E process manager disables ambient proxy variables for subprocess startup and uses direct localhost HTTP clients, so local proxy settings should not affect the health checks.

Running Box integration tests

Box integration tests exercise the real sandbox runtime path, including command execution, session persistence, managed process WebSocket attachment, and cleanup behavior.

uv run --python 3.12 pytest tests/integration_tests -q --tb=short

These tests require a working Docker or Podman runtime. In CI, the dedicated Box integration job checks Docker availability before running the tests.

Running frontend E2E tests

Frontend E2E tests live in web/tests/e2e and use Playwright. They start Vite and mock the LangBot backend and Space APIs, so no backend process is required.

cd web
pnpm test:e2e

Known Issues

Some tests may encounter circular import errors. This is a known issue with the current module structure. The test infrastructure is designed to work around this using lazy imports, but if you encounter issues:

1. Make sure you're running from the project root directory
2. Ensure dependencies are installed: uv sync --dev
3. Try running a simple test first to verify the test infrastructure works

CI/CD Integration

Tests are automatically run on:

• Pull request opened
• Pull request marked ready for review
• Push to PR branch
• Push to master/develop branches

The workflow runs tests on Python 3.11, 3.12, and 3.13 to ensure compatibility. Startup E2E and Box integration tests run as separate Python 3.12 jobs because they exercise process/container behavior instead of pure Python compatibility. Frontend Playwright smoke tests run in .github/workflows/frontend-tests.yml.

Adding New Tests

1. For a new pipeline stage

Create a new test file test_<stage_name>.py:

"""
<StageName> stage unit tests
"""

import pytest
from langbot.pkg.pipeline.<module>.<stage> import <StageClass>
from langbot.pkg.pipeline import entities as pipeline_entities


@pytest.mark.asyncio
async def test_stage_basic_flow(mock_app, sample_query):
    """Test basic flow"""
    stage = <StageClass>(mock_app)
    await stage.initialize({})

    result = await stage.process(sample_query, '<StageName>')

    assert result.result_type == pipeline_entities.ResultType.CONTINUE

2. For additional fixtures

Add new fixtures to the appropriate conftest.py:

@pytest.fixture
def my_custom_fixture():
    """Description of fixture"""
    return create_test_data()

3. For test data

Use the helper functions in conftest.py:

from tests.unit_tests.pipeline.conftest import create_stage_result, assert_result_continue

result = create_stage_result(
    result_type=pipeline_entities.ResultType.CONTINUE,
    query=sample_query
)

assert_result_continue(result)

Best Practices

1. Test naming: Use descriptive names that explain what's being tested
2. Arrange-Act-Assert: Structure tests clearly with setup, execution, and verification
3. One assertion per test: Focus each test on a single behavior
4. Mock appropriately: Mock external dependencies, not the code under test
5. Use fixtures: Reuse common test data through fixtures
6. Document tests: Add docstrings explaining what each test validates

Troubleshooting

Import errors

Make sure you've installed the package in development mode:

uv sync --dev

Async test failures

Ensure you're using @pytest.mark.asyncio decorator for async tests.

Mock not working

Check that you're mocking at the right level and using AsyncMock for async functions.

Future Enhancements

• Add integration tests for database migrations (SQLite)
• Add PostgreSQL migration integration tests (G-003)
• Add integration tests for full pipeline execution
• Add API smoke integration tests
• Add E2E tests
• Add performance benchmarks
• Add mutation testing for better coverage quality
• Add property-based testing with Hypothesis