ci(regression): consume published plugins and shared daily scenarios

tools/README.md

@@ -367,9 +367,13 @@ phases. The dedicated Docker daemon requirement above still applies.
 
 `Core Smoke` runs regression-tool tests and a real Core smoke in parallel on
 GitHub-hosted Linux runners. The smoke starts Core from the checked-out candidate,
-uses the SDK from `uv.lock`, the existing official-plugin source pin, `lbctl`
+uses the SDK from `uv.lock`, the latest published official plugins, `lbctl`
 v0.3.0, a Docker Box and mock generation/embedding. It needs no model credentials
 or access to a developer's running environment.
+Plugin versions are resolved once per run, installed from the downloaded release
+archives and recorded with hashes and installation readback in `plugins/manifest.json`.
+An unavailable or incompatible published package fails preparation; see
+[published plugin preparation and replay](published-plugins.md).
 
 Dependency installation, service startup, sandbox image download and plugin
 installation are preparation. After readiness, the gate allows **60 seconds total**
@@ -396,13 +400,12 @@ inspect its retained ownership journal before cleanup. CI tears down its own
 service processes and owned sandboxes afterward.
 
 The workflow always creates its two checks on PRs and cancels superseded runs.
-After the workflow is merged, configure `Regression tools` and `Core smoke (60s)`
-as required status checks in the repository ruleset. A green workflow alone does
-not enforce merge protection. Longer load and soak tests use the existing Suite
-and Job entry points with separately calibrated latency, throughput and resource
-budgets; the one-minute smoke does not replace them.
+The existing required check names remain `Regression tools` and `Core smoke (60s)`.
+`Daily Endurance` is independent and is not required for merging. Longer load and
+soak tests use the existing Suite and Job entry points with separately calibrated
+latency, throughput and resource budgets; the one-minute smoke does not replace them.
 
-## Sustained regression and baseline comparison
+## Daily regression
 
 `Daily Endurance` runs independently of PR checks at 19:23 UTC (03:23
 Asia/Shanghai) on the default branch. GitHub may delay scheduled starts. It also
@@ -410,30 +413,50 @@ supports **Run workflow** with a selected branch. The schedule becomes active
 only after the workflow is merged into the default branch; it is not a required
 merge check.
 
-The workflow reuses the disposable smoke stack, locked SDK, pinned official
-plugins and lbctl v0.3.0. Generation and embedding are deterministic mocks.
-It runs the endurance profile below, with a 35-minute execution budget for
-resource preparation, load, drain, cooldown and API cleanup. Service preparation
-is outside that budget; the whole job has a 45-minute limit. Service processes
-and owned Box containers are stopped afterward, including after a failed suite.
+The workflow reuses the disposable smoke stack and latest published plugins.
+`--suite daily` consumes the shared 46-case profile, requires its first pass, then
+repeats ordered cases within one **1800-second** window. The clock starts before
+creating scenario resources and includes a 120-second reserve for drain/API
+cleanup and a 60-second cooldown. Service and plugin preparation precede this
+clock. A separate 1860-second process guard allows startup and final reporting;
+it does not grant extra scenario time. The whole job retains its 45-minute limit.
 
-Actions summaries show throughput, per-case P95 and retained resource growth.
-Verdicts, individual requests, resource samples and service logs are retained
-for 30 days, including failures. Initial QA budget violations make this
-observational job red without blocking PRs. GitHub-hosted runners vary between
-runs: these reports do not establish a controlled baseline or prove a performance
-regression. No automatic cross-run performance comparison is enabled.
+The daily profile uses the existing endurance resource allowances with a
+180-second warmup before baseline sampling. Each pipeline that enables Box uses
+one stable reuse key unique to this CI run, so conversations created by repeated
+cycles reuse that pipeline's sandbox. This measures repeated tool use; it does not test
+cross-group Box file isolation. The profile uses one Workspace and does not
+establish isolation between tenants. Conversation history and AgentRuns still
+accumulate and are reported. Resource excesses remain failures.
+
+Throughput and full-step P95 are observations. Business assertions, unknown
+results, health failures, incomplete evidence and resource limits determine the
+verdict. Summaries include first-pass completion, cycles, plugin installation
+readback, final ledger reconciliation and coverage. Manifests, case results,
+coverage, cycles, execution stages, resource samples and service logs are retained
+for 30 days, including failures. Hosted runners do not provide a controlled
+performance baseline. No automatic comparison between runs is enabled.
 
 To run the same entrypoint on a disposable Linux host with an idle Docker daemon:
 
 ```sh
-uv run --no-sync python tools/regression_ci.py --suite endurance \
-  --lbctl /path/to/lbctl --plugins /path/to/pinned/langbot-plugins \
-  --output .qa/endurance-001
+uv run --no-sync python tools/regression_ci.py --suite daily \
+  --lbctl /path/to/lbctl --output .qa/daily-001
 ```
 
+`--suite smoke` retains the 60-second gate. `--suite scenarios` runs the shared
+expanded profile once under a 300-second guard. `--suite endurance` retains the
+separate ten-minute load profile under a 900-second guard; its CI entry point also
+reports throughput/P95 without applying their thresholds. All four modes use
+published plugins by default. Explicit `--plugins` remains available for source
+development; neither workflow selects it. The suite attempts owned API cleanup
+even after failures. The CI owner then stops its service processes and removes
+its Box containers; a process guard timeout can interrupt API cleanup.
+
+## Sustained regression and baseline comparison
+
 [`regression_profiles/endurance.json`](regression_profiles/endurance.json) runs
-29 minutes of mixed chat/Box/RAG traffic at 1 request/second across four stable
+nine minutes of mixed chat/Box/RAG traffic at 1 request/second across four stable
 groups, followed by one minute without traffic to check recovery. Configure its
 CLI context, plugin inputs, model endpoints, Core SQLite path and Core/Runtime/Box
 PID files as for smoke. Set `plugins: []` to reuse initialized plugins. Configure

tools/regression_ci.py

@@ -15,15 +15,45 @@ import time
 import yaml
 
 from regression_gate import supervise
+from regression_coverage import write_report
 from regression_plugins import install_published_plugins, prepare_published_plugins
 from regression_suite import Suite
 from regression_traffic import owned_box_containers, read_endpoint_snapshot
 
 ROOT = Path(__file__).resolve().parents[1]
+SUITE_BUDGETS = {'smoke': 60, 'scenarios': 300, 'daily': 1860, 'endurance': 900}
 sys.path.insert(0, str(ROOT))
 from tests.e2e.utils.config_factory import create_minimal_config, create_test_directories  # noqa: E402
 
 
+def suite_profile(name, run_id):
+    profile_path = ROOT / 'tools/regression_profiles' / ('endurance.json' if name == 'endurance' else 'smoke.json')
+    if name in ('scenarios', 'daily'):
+        from regression_scenarios import expanded_profile
+
+        profile = expanded_profile()
+    else:
+        profile = json.loads(profile_path.read_text())
+    if name == 'smoke':
+        profile['smoke']['group_turns'] = 2
+    if name == 'daily':
+        limits = json.loads((profile_path.parent / 'endurance.json').read_text())['resource_limits']
+        limits['warmup_seconds'] = 180
+        profile.update(
+            execution={'total_seconds': 1800, 'drain_seconds': 120, 'cooldown_seconds': 60},
+            resource_limits=limits,
+        )
+        for resource in profile['resources']:
+            if resource['kind'] != 'pipeline':
+                continue
+            for config in resource['body']['config']['ai']['runner_config'].values():
+                if config.get('box-enabled'):
+                    # This workload measures repeated tool use; each pipeline owns one sandbox per CI run.
+                    config['box-session-id-template'] = f'qa-{run_id}-{resource["id"]}'
+    profile.update(cleanup='always', performance_mode='observe')
+    return profile, profile_path
+
+
 def ready(url, processes, timeout=120):
     deadline = time.monotonic() + timeout
     while time.monotonic() < deadline:
@@ -82,8 +112,7 @@ def run(args):
             host_root=str(private / 'box'), allowed_mount_roots=[str(private / 'box')], image='python:3.12-alpine'
         )
         config_path.write_text(yaml.safe_dump(config))
-        profile_path = ROOT / f'tools/regression_profiles/{args.suite}.json'
-        profile = json.loads(profile_path.read_text())
+        profile, profile_path = suite_profile(args.suite, secrets.token_hex(8))
         cli_config = private / 'lbctl.yaml'
         cli_config.write_text(
             yaml.safe_dump(
@@ -109,9 +138,6 @@ def run(args):
             mock_records=str(root / 'mock-requests.jsonl'),
         )
         profile['plugins'] = []
-        if args.suite == 'smoke':
-            # Two turns retain the history/isolation oracle within the 60-second budget.
-            profile['smoke']['group_turns'] = 2
         for resource in profile['resources']:
             if resource['kind'] == 'model' and resource.get('type') == 'embedding':
                 resource['body']['name'] = 'mock-embedding'
@@ -166,7 +192,7 @@ def run(args):
             },
             'pids': pids,
         }
-        if args.suite == 'endurance':
+        if args.suite in ('daily', 'endurance'):
             profile['observe']['box_data_root'] = str(private / 'box')
         for url in profile['observe']['health'].values():
             ready(url, processes)
@@ -189,7 +215,7 @@ def run(args):
                 suite.call(['plugin', 'install', 'local', '--file', archive, '--wait'])
         generated = private / f'{args.suite}.json'
         generated.write_text(json.dumps(profile, indent=2))
-        return supervise(
+        result = supervise(
             [
                 sys.executable,
                 str(ROOT / 'tools/regression_suite.py'),
@@ -200,10 +226,15 @@ def run(args):
                 str(root / 'gate/suite'),
             ],
             root / 'gate',
-            60 if args.suite == 'smoke' else 2100,
+            SUITE_BUDGETS[args.suite],
             pids,
             env,
         )
+        coverage = write_report([root / 'gate/suite'], root / 'gate/suite', http_logs=[root / 'core.log'])
+        result['coverage_errors'] = coverage['evidence_errors']
+        result['passed'] = result['passed'] and not result['coverage_errors']
+        (root / 'gate/gate.json').write_text(json.dumps(result, indent=2) + '\n')
+        return result
     finally:
         for process in reversed(processes):
             if process.poll() is None:
@@ -227,7 +258,7 @@ def main():
     plugins = parser.add_mutually_exclusive_group()
     plugins.add_argument('--plugins', type=Path, help='Explicit developer override: package a plugin source checkout')
     plugins.add_argument('--plugin-manifest', type=Path, help='Replay published versions and hashes from a prior run')
-    parser.add_argument('--suite', choices=('smoke', 'endurance'), default='smoke')
+    parser.add_argument('--suite', choices=tuple(SUITE_BUDGETS), default='smoke')
     args = parser.parse_args()
     if args.output.exists():
         parser.error('--output must be a new directory')

tools/regression_profiles/endurance.json

@@ -444,7 +444,7 @@
   "phases": [
     {
       "name": "sustained",
-      "duration_seconds": 1740,
+      "duration_seconds": 540,
       "rps": 1,
       "workers": 4,
       "cooldown_seconds": 60,

tools/test_regression_ci.py

@@ -10,6 +10,59 @@ import unittest
 
 import yaml
 
+from regression_ci import SUITE_BUDGETS, suite_profile
+from regression_suite import validate
+
+
+class CIProfileTest(unittest.TestCase):
+    def test_suite_selection_uses_shared_cases_and_one_daily_budget(self):
+        for name in ('smoke', 'scenarios', 'daily', 'endurance'):
+            with self.subTest(suite=name):
+                profile, path = suite_profile(name, 'test-run')
+                profile['observe'] = {
+                    'health': {'core': 'http://localhost/healthz'},
+                    'pids': {'core': 1, 'runtime': 2, 'box': 3},
+                    'box_data_root': '/tmp/test-box',
+                }
+                validate(profile)
+                self.assertTrue(path.exists())
+                self.assertEqual(profile['cleanup'], 'always')
+                self.assertEqual(profile['performance_mode'], 'observe')
+                if name == 'smoke':
+                    self.assertEqual(profile['smoke']['group_turns'], 2)
+                    self.assertEqual(SUITE_BUDGETS[name], 60)
+                elif name in ('scenarios', 'daily'):
+                    self.assertEqual(len(profile['cases']), 46)
+                    self.assertEqual(profile['coverage_suite'], 'scenarios')
+                    self.assertEqual(profile['smoke']['group_turns'], 3)
+                else:
+                    self.assertEqual(profile['phases'][0]['duration_seconds'], 540)
+                self.assertEqual('execution' in profile, name == 'daily')
+        daily, _ = suite_profile('daily', 'test-run')
+        self.assertEqual(daily['execution'], {'total_seconds': 1800, 'drain_seconds': 120, 'cooldown_seconds': 60})
+        endurance, _ = suite_profile('endurance', 'test-run')
+        self.assertEqual(daily['resource_limits']['warmup_seconds'], 180)
+        self.assertEqual(daily['resource_limits']['targets'], endurance['resource_limits']['targets'])
+        self.assertEqual(daily['resource_limits']['containers'], endurance['resource_limits']['containers'])
+
+    def test_daily_reuses_boxes_per_pipeline_and_separates_runs(self):
+        def keys(name, run_id):
+            profile, _ = suite_profile(name, run_id)
+            return {
+                resource['id']: runner['box-session-id-template']
+                for resource in profile['resources']
+                if resource['kind'] == 'pipeline'
+                for runner in resource['body']['config']['ai']['runner_config'].values()
+                if runner.get('box-enabled')
+            }
+
+        first, second = keys('daily', 'run-a'), keys('daily', 'run-b')
+        self.assertTrue(first)
+        self.assertEqual(len(set(first.values())), len(first))
+        self.assertFalse(set(first.values()) & set(second.values()))
+        self.assertEqual(first, keys('daily', 'run-a'))
+        self.assertTrue(all(value == '{launcher_type}_{launcher_id}' for value in keys('scenarios', 'run-a').values()))
+
 
 class DailySummaryTest(unittest.TestCase):
     def test_summary_reports_observations_without_hiding_outer_failures(self):
@@ -18,47 +71,85 @@ class DailySummaryTest(unittest.TestCase):
         command = next(step['run'] for step in steps if step.get('name') == 'Publish observations')
         suite = {
             'passed': True,
+            'outcome': 'PASS',
             'errors': [],
-            'phases': [
-                {
-                    'achieved_rps': 1.0,
-                    'verified_runs': 540,
-                    'runner_requests': 540,
-                    'ledger_closeout': {'matched': True},
-                    'cases': {'chat': {'requests': 270, 'status': {'ok': 270}, 'p95_seconds': 0.5}},
-                }
-            ],
+            'first_pass_complete': True,
+            'cycles': [{'complete': True}],
+            'requests': 86,
+            'ledger_closeout': {'matched': True},
+            'observations': {'successful_requests_per_second': 1.0, 'step_p95_seconds': {'chat': 0.5}},
         }
-        for gate, expected in [
-            ({'passed': True}, 'Endurance: PASS'),
-            ({'passed': False, 'infrastructure_error': 'teardown failed'}, 'teardown failed'),
-            ({'passed': False, 'timed_out': True}, '2100-second execution budget'),
-            (None, 'Missing verdict'),
+        for gate, actual_suite, expected in [
+            ({'passed': True}, suite, 'Daily regression: PASS'),
+            ({'passed': False, 'infrastructure_error': 'teardown failed'}, suite, 'teardown failed'),
+            ({'passed': False, 'infrastructure_error': 'published plugin install failed'}, None, 'not confirmed'),
+            ({'passed': False, 'timed_out': True}, suite, '1860-second outer process guard'),
+            ({'passed': False, 'coverage_errors': ['missing access records']}, suite, 'missing access records'),
+            ({'passed': False}, {**suite, 'passed': False, 'outcome': 'INCOMPLETE'}, 'Daily regression: INCOMPLETE'),
+            (None, None, 'Missing verdict'),
         ]:
             with self.subTest(gate=gate), tempfile.TemporaryDirectory() as directory:
                 root = Path(directory)
+                target = root / '.qa/daily'
+                (target / 'plugins').mkdir(parents=True)
+                (target / 'plugins/manifest.json').write_text(
+                    json.dumps({'plugins': [{'id': 'langbot-team/LocalAgent', 'version': '1.2.3'}]})
+                )
                 if gate is not None:
-                    target = root / '.qa/endurance/gate'
-                    (target / 'suite').mkdir(parents=True)
-                    (target / 'gate.json').write_text(json.dumps(gate))
-                    (target / 'suite/summary.json').write_text(json.dumps(suite))
+                    (target / 'gate/suite').mkdir(parents=True)
+                    (target / 'gate/gate.json').write_text(json.dumps(gate))
+                    if actual_suite is not None:
+                        (target / 'gate/suite/summary.json').write_text(json.dumps(actual_suite))
                 summary = root / 'summary.md'
                 env = {
                     **os.environ,
                     'PATH': str(Path(sys.executable).parent) + os.pathsep + os.environ['PATH'],
                     'GITHUB_STEP_SUMMARY': str(summary),
                     'GITHUB_SHA': 'core-revision',
-                    'PLUGIN_REVISION': 'plugin-revision',
                 }
                 subprocess.run(['bash', '-e', '-c', command], cwd=root, env=env, check=True, capture_output=True)
                 text = summary.read_text()
                 self.assertIn(expected, text)
+                self.assertIn('langbot-team/[email protected]', text)
+                self.assertIn('not confirmed', text)
                 if gate and gate.get('passed'):
-                    self.assertIn('Completed runs: 540 / 540', text)
+                    self.assertIn('Final ledger reconciliation: True', text)
                     self.assertIn('1.000 RPS', text)
                 else:
-                    self.assertIn('Endurance: FAIL / incomplete', text)
-                    self.assertNotIn('Endurance: PASS', text)
+                    self.assertNotIn('Daily regression: PASS', text)
+
+    def test_core_summary_records_preparation_failure_and_selected_plugins(self):
+        workflow = Path(__file__).resolve().parents[1] / '.github/workflows/core-smoke.yml'
+        steps = yaml.safe_load(workflow.read_text())['jobs']['smoke']['steps']
+        command = next(step['run'] for step in steps if step.get('name') == 'Publish gate result')
+        with tempfile.TemporaryDirectory() as directory:
+            root = Path(directory)
+            target = root / '.qa/core-smoke'
+            (target / 'plugins').mkdir(parents=True)
+            (target / 'gate').mkdir()
+            (target / 'plugins/manifest.json').write_text(
+                json.dumps({'plugins': [{'id': 'langbot-team/LocalAgent', 'version': '1.2.3'}]})
+            )
+            (target / 'gate/gate.json').write_text(
+                json.dumps({'passed': False, 'infrastructure_error': 'incompatible'})
+            )
+            summary = root / 'summary.md'
+            subprocess.run(
+                ['bash', '-e', '-c', command],
+                cwd=root,
+                check=True,
+                capture_output=True,
+                env={
+                    **os.environ,
+                    'PATH': str(Path(sys.executable).parent) + os.pathsep + os.environ['PATH'],
+                    'GITHUB_STEP_SUMMARY': str(summary),
+                },
+            )
+            text = summary.read_text()
+            self.assertIn('Core smoke: FAIL', text)
+            self.assertIn('Preparation or teardown failed', text)
+            self.assertIn('langbot-team/[email protected]', text)
+            self.assertIn('not confirmed', text)
 
 
 if __name__ == '__main__':