xinyin025/LangBot
1
0
Fork 0
mirror of https://github.com/langbot-app/LangBot.git synced 2026-08-09 04:40:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(tenancy): add Workspace multi-tenant foundation (#2353)

Browse Source 
* Document multi-tenant workspace architecture

* Add OSS and commercial workspace boundaries

* docs: redesign multi-tenant workspace architecture

* feat(tenancy): implement workspace isolation

* docs(tenancy): record verification evidence

* docs(tenancy): revise single-instance SaaS topology

* docs(tenancy): refine architecture options

* docs: finalize cloud v2 multi-tenant decisions

* feat(tenancy): establish cloud isolation foundations

* feat(tenancy): harden shared cloud runtime boundaries

* docs(tenancy): record final isolation verification

* fix(tenancy): close isolation and permission gaps

* docs(tenancy): record final isolation verification

* feat(tenancy): connect cloud workspace control plane

* fix(build): install git for pinned SDK

* docs(cloud): update control plane verification

* chore: update multi-tenant SDK pin

* fix(cloud): skip legacy model sync during startup

* test(cloud): preserve minimal model manager fixtures

* fix(cloud): preserve authenticated account context

* fix(cloud): reuse authenticated account for user info

* feat(cloud): complete Workspace settings navigation

* test(web): cover Workspace dropdown menu

* feat(web): place workspace controls in sidebar

* refactor(web): streamline workspace controls

* style(web): format workspace layout test

* fix(cloud): surface runtime and workspace plan status

* fix(plugin): keep runtime identity stable across restarts

* fix(ui): widen and center workspace switcher

* fix(ui): hide roles from workspace switcher

* fix(ui): align workspace switcher with sidebar entries

* feat(workspace): add in-product collaboration and direct Cloud launch

* style: format collaboration changes

* fix(workspace): bind collaboration APIs to tenant UoW

* fix(cloud): preserve Core-owned collaboration state

* test(cloud): require Space identity for invite registration

* feat(cloud): complete secure invitation experience

* style(web): format invitation flows

* fix(cloud): recover box runtime without unscoped skill reload

* feat(oss): enforce invitation account and owner billing flows

* style: format OSS account service

* test(oss): cover invitation logout handoff

* fix(oss): resolve workspace owner in scoped session

* feat(cloud): harden multi-tenant runtime resources

* fix(cloud): bound runtime restart storms

* fix(cloud): eliminate periodic runtime CPU spikes

* fix(cloud): enforce instance capacity ceilings

* fix(cloud): scope public login capability discovery

* fix(cloud): bound tenant maintenance and monitoring work

* fix(runtime): bound tenant resource amplification

* fix(deps): pin green multi-tenant plugin SDK

* fix(cloud): handle unavailable skill capability

* fix(security): require authentication for image file endpoint (H-2)

- Changed /api/v1/files/image from AuthType.NONE to USER_TOKEN_OR_API_KEY
- Added Permission.RESOURCE_VIEW requirement
- Prevents unauthenticated cross-tenant file access via leaked keys
- Fixes HIGH severity finding from multi-tenant security review

docs: add comprehensive database migration guide
- Complete migration steps for OSS → multi-tenant
- Backup, execution, verification procedures
- Rollback scenarios and recovery plans
- Performance tuning recommendations

* test: add comprehensive cross-tenant isolation tests

Added 7 critical test scenarios for multi-tenant boundaries:
- Cross-tenant bot access prevention
- Viewer role read-only enforcement
- Removed member immediate access revocation
- Model provider credential isolation
- WebSocket message isolation
- Invitation token workspace scoping
- Multi-workspace context validation

These tests address P0-2 coverage gaps for:
- workspaces.py (membership & invitation flows)
- user.py (authentication & authorization)
- websocket_chat.py (real-time isolation)
- plugins.py (resource access control)

docs: finalize database migration guide

* fix(security): resolve M-1, M-2, M-3 security findings

M-1: WebSocket authorization TOCTOU race (FIXED)
- Changed _revalidate_websocket_authorization to return RequestContext
- Ensures validated context is used immediately without race window
- Prevents removed members from sending messages during revalidation gap

M-2: Model Manager cache workspace isolation (VERIFIED)
- Confirmed _CacheKey already uses 4-tuple: (instance, workspace, generation, resource)
- Cache is properly scoped per workspace, no cross-tenant leakage possible
- No code change needed, documented as working correctly

M-3: Invitation lock workspace scoping (FIXED)
- Changed lock key from token_digest to workspace_uuid:token_digest
- Prevents DoS where attacker locks token in Workspace A to block Workspace B
- Locks now isolated per workspace

All MEDIUM severity findings from security review now resolved.

* fix(cloud): unblock tenant CI and enforce knowledge quotas

* fix(tenancy): scope rerank model sync

---------

Co-authored-by: dadachann <185672915+dadachann@users.noreply.github.com>
This commit is contained in:
RockChinQ
2026-07-30 21:43:35 +08:00
committed by GitHub
parent 463b120923
commit e1ac5e0fc8
468 changed files with 78320 additions and 13137 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/langbot/pkg/core/app.py
+281 -55
View File
@@ -2,9 +2,9 @@ from __future__ import annotations
import logging
import asyncio
import contextlib
import traceback
import os
import contextlib
from ..platform import botmgr as im_mgr
from ..platform.webhook_pusher import WebhookPusher
@@ -19,7 +19,7 @@ from ..plugin import connector as plugin_connector
from ..pipeline import pool
from ..pipeline import controller, pipelinemgr
from ..pipeline import aggregator as message_aggregator
from ..utils import version as version_mgr, proxy as proxy_mgr
from ..utils import version as version_mgr, proxy as proxy_mgr, httpclient
from ..persistence import mgr as persistencemgr
from ..api.http.controller import main as http_controller
from ..api.http.service import user as user_service
@@ -37,7 +37,7 @@ from ..api.http.service import skill as skill_service
from ..api.http.service import maintenance as maintenance_service
from ..discover import engine as discover_engine
from ..storage import mgr as storagemgr
from ..utils import logcache
from ..utils import bounded_executor, event_loop_monitor, logcache
from . import taskmgr
from . import entities as core_entities
from ..rag.knowledge import kbmgr as rag_mgr
@@ -46,6 +46,14 @@ from ..vector import mgr as vectordb_mgr
from ..telemetry import telemetry as telemetry_module
from ..survey import manager as survey_module
from ..skill import manager as skill_mgr
from ..workspace import service as workspace_service_module
from ..workspace import collaboration as workspace_collaboration_module
from ..workspace import invitation_delivery as invitation_delivery_module
from ..cloud import bootstrap as cloud_bootstrap_module
from ..cloud import launch as cloud_launch_module
from ..cloud import directory_projection as cloud_directory_projection_module
from ..cloud import entitlements as cloud_entitlements_module
from ..api.http.context import ExecutionContext, PrincipalContext, PrincipalType
class Application:
@@ -120,6 +128,24 @@ class Application:
persistence_mgr: persistencemgr.PersistenceManager = None
workspace_service: workspace_service_module.WorkspaceService = None
workspace_collaboration_service: workspace_collaboration_module.WorkspaceCollaborationService = None
invitation_delivery_service: invitation_delivery_module.InvitationDeliveryService = None
space_launch_service: cloud_launch_module.SpaceLaunchService = None
deployment: cloud_bootstrap_module.OpenSourceDeployment | cloud_bootstrap_module.VerifiedCloudDeployment = None
deployment_admission: cloud_bootstrap_module.DeploymentAdmissionGuard = None
manifest_refresh_service: cloud_bootstrap_module.CloudManifestRefreshService | None = None
entitlement_resolver: cloud_entitlements_module.EntitlementResolver | None = None
directory_projection_service: cloud_directory_projection_module.DirectoryProjectionService | None = None
vector_db_mgr: vectordb_mgr.VectorDBManager = None
http_ctrl: http_controller.HTTPController = None
@@ -166,15 +192,123 @@ class Application:
maintenance_service: maintenance_service.MaintenanceService = None
blocking_executor: bounded_executor.BoundedThreadPoolExecutor | None = None
event_loop_monitor: event_loop_monitor.EventLoopLagMonitor
def __init__(self):
self._shutdown_lock = asyncio.Lock()
self._shutdown_complete = False
self._shutdown_task: asyncio.Task | None = None
self.event_loop_monitor = event_loop_monitor.EventLoopLagMonitor()
def get_runtime_resource_stats(self) -> dict[str, object]:
"""Return aggregate O(1) counters for liveness and soak validation."""
try:
asyncio_tasks = len(asyncio.all_tasks(self.event_loop))
except (RuntimeError, TypeError):
asyncio_tasks = 0
task_stats = self.task_mgr.get_stats() if self.task_mgr is not None else {}
query_pool_stats = {}
if self.query_pool is not None:
query_pool_stats = {
'queued': len(self.query_pool.queries),
'cached': len(self.query_pool.cached_queries),
'active_workspaces': len(self.query_pool.active_query_count_by_workspace),
}
model_stats = {}
if self.model_mgr is not None:
model_stats = {
'providers': len(self.model_mgr.provider_dict),
'llms': len(self.model_mgr.llm_model_dict),
'embeddings': len(self.model_mgr.embedding_model_dict),
'rerankers': len(self.model_mgr.rerank_model_dict),
}
runtime_stats = {
'bots': len(getattr(self.platform_mgr, '_bots_by_key', {})),
'pipelines': len(getattr(self.pipeline_mgr, '_pipelines_by_key', {})),
'knowledge_bases': len(getattr(self.rag_mgr, 'knowledge_bases', {})),
'message_aggregation_buffers': len(getattr(self.msg_aggregator, 'buffers', {})),
'message_aggregation_scopes': len(
getattr(
self.msg_aggregator,
'_buffer_counts_by_scope',
{},
)
),
'plugin_installations': len(
getattr(
self.plugin_connector,
'_known_desired_states',
{},
)
),
}
mcp_loader = getattr(self.tool_mgr, 'mcp_tool_loader', None)
runtime_stats.update(
{
'mcp_sessions': len(getattr(mcp_loader, '_sessions', {})),
'mcp_host_tasks': len(getattr(mcp_loader, '_hosted_mcp_tasks', ())),
'mcp_dispatch_tasks': len(getattr(mcp_loader, '_host_dispatch_tasks', ())),
'mcp_projection_retirements': len(getattr(mcp_loader, '_pending_projection_retirements', ())),
'mcp_projection_reconcile_active': int(
(
projection_task := getattr(
mcp_loader,
'_projection_reconcile_task',
None,
)
)
is not None
and not projection_task.done()
),
}
)
directory_stats = {}
directory_snapshot = getattr(self.directory_projection_service, 'resource_snapshot', None)
if callable(directory_snapshot):
directory_stats = directory_snapshot()
database_stats = {}
database_snapshot = getattr(self.persistence_mgr, 'get_resource_stats', None)
if callable(database_snapshot):
database_stats = database_snapshot()
return {
'asyncio_tasks': asyncio_tasks,
'event_loop': self.event_loop_monitor.snapshot(),
'blocking_executor': (self.blocking_executor.snapshot() if self.blocking_executor is not None else {}),
'application_tasks': task_stats,
'database_pool': database_stats,
'directory': directory_stats,
'query_pool': query_pool_stats,
'models': model_stats,
'runtimes': runtime_stats,
'telemetry_tasks': len(getattr(self.telemetry, 'send_tasks', ())),
}
async def initialize(self):
pass
async def run(self):
self.event_loop_monitor.start()
try:
if self.directory_projection_service is not None:
self.task_mgr.create_task(
self.directory_projection_service.run(),
name='cloud-directory-projection',
scopes=[core_entities.LifecycleControlScope.APPLICATION],
)
if self.manifest_refresh_service is not None:
self.task_mgr.create_task(
self.manifest_refresh_service.run(),
name='cloud-manifest-refresh',
scopes=[core_entities.LifecycleControlScope.APPLICATION],
)
await self.plugin_connector.initialize_plugins()
# 后续可能会允许动态重启其他任务
@@ -213,74 +347,128 @@ class Application:
scopes=[core_entities.LifecycleControlScope.APPLICATION],
)
# Start monitoring data cleanup task if enabled
monitoring_cfg = self.instance_config.data.get('monitoring', {})
auto_cleanup_cfg = monitoring_cfg.get('auto_cleanup', {})
if auto_cleanup_cfg.get('enabled', True):
retention_days = self._get_positive_int_config(
auto_cleanup_cfg.get('retention_days', 30),
default=30,
name='monitoring.auto_cleanup.retention_days',
)
delete_batch_size = self._get_positive_int_config(
auto_cleanup_cfg.get('delete_batch_size', 1000),
default=1000,
name='monitoring.auto_cleanup.delete_batch_size',
)
check_interval_hours = self._get_positive_float_config(
monitoring_enabled = auto_cleanup_cfg.get('enabled', True)
retention_days = self._get_positive_int_config(
auto_cleanup_cfg.get('retention_days', 30),
default=30,
name='monitoring.auto_cleanup.retention_days',
)
delete_batch_size = self._get_positive_int_config(
auto_cleanup_cfg.get('delete_batch_size', 1000),
default=1000,
name='monitoring.auto_cleanup.delete_batch_size',
)
monitoring_interval_seconds = (
self._get_positive_float_config(
auto_cleanup_cfg.get('check_interval_hours', 1),
default=1,
name='monitoring.auto_cleanup.check_interval_hours',
)
* 3600
)
async def monitoring_cleanup_loop():
check_interval_seconds = check_interval_hours * 3600
while True:
try:
deleted = await self.monitoring_service.cleanup_expired_records(
retention_days,
batch_size=delete_batch_size,
)
total_deleted = sum(deleted.values())
if total_deleted > 0:
self.logger.info(
f'Monitoring auto-cleanup: deleted {total_deleted} expired records '
f'(retention={retention_days}d): {deleted}'
)
except Exception as e:
self.logger.warning(f'Monitoring auto-cleanup error: {e}')
await asyncio.sleep(check_interval_seconds)
self.task_mgr.create_task(
monitoring_cleanup_loop(),
name='monitoring-cleanup',
scopes=[core_entities.LifecycleControlScope.APPLICATION],
)
# Start storage/log maintenance task if enabled
storage_cleanup_cfg = self.instance_config.data.get('storage', {}).get('cleanup', {})
if storage_cleanup_cfg.get('enabled', True) and self.maintenance_service is not None:
check_interval_hours = self._get_positive_float_config(
storage_enabled = storage_cleanup_cfg.get('enabled', True) and self.maintenance_service is not None
storage_interval_seconds = (
self._get_positive_float_config(
storage_cleanup_cfg.get('check_interval_hours', 1),
default=1,
name='storage.cleanup.check_interval_hours',
)
* 3600
)
async def storage_cleanup_loop():
check_interval_seconds = check_interval_hours * 3600
maintenance_intervals: dict[str, float] = {}
if monitoring_enabled:
maintenance_intervals['monitoring'] = monitoring_interval_seconds
if storage_enabled:
maintenance_intervals['storage'] = storage_interval_seconds
if self.workspace_collaboration_service is not None:
maintenance_intervals['invitations'] = 3600.0
if maintenance_intervals:
async def resource_maintenance_loop():
"""Share tenant discovery and serialize periodic maintenance."""
loop = asyncio.get_running_loop()
started_at = loop.time()
next_due = {name: started_at + interval for name, interval in maintenance_intervals.items()}
while True:
await asyncio.sleep(max(min(next_due.values()) - loop.time(), 0.0))
observed_at = loop.time()
due = {name for name, due_at in next_due.items() if due_at <= observed_at}
if not due:
continue
try:
deleted = await self.maintenance_service.cleanup_expired_files()
total_deleted = sum(deleted.values())
if total_deleted > 0:
self.logger.info(f'Storage maintenance: deleted expired files: {deleted}')
except Exception as e:
self.logger.warning(f'Storage maintenance error: {e}')
await asyncio.sleep(check_interval_seconds)
bindings = await self.workspace_service.list_active_execution_bindings()
except asyncio.CancelledError:
raise
except Exception as exc:
self.logger.warning(f'Resource maintenance Workspace discovery failed: {exc}')
else:
for binding in bindings:
context = ExecutionContext(
instance_uuid=binding.instance_uuid,
workspace_uuid=binding.workspace_uuid,
placement_generation=binding.placement_generation,
trigger_principal=PrincipalContext(PrincipalType.SYSTEM),
)
if 'monitoring' in due:
try:
deleted = await self.monitoring_service.cleanup_expired_records(
context,
retention_days,
batch_size=delete_batch_size,
)
total_deleted = sum(deleted.values())
if total_deleted > 0:
self.logger.info(
f'Monitoring auto-cleanup: deleted {total_deleted} expired records '
f'for Workspace {context.workspace_uuid} '
f'(retention={retention_days}d): {deleted}'
)
except asyncio.CancelledError:
raise
except Exception as exc:
self.logger.warning(
f'Monitoring auto-cleanup failed for '
f'Workspace {context.workspace_uuid}: {exc}'
)
if 'storage' in due:
try:
deleted = await self.maintenance_service.cleanup_expired_files(context)
total_deleted = sum(deleted.values())
if total_deleted > 0:
self.logger.info(
f'Storage maintenance for Workspace {context.workspace_uuid}: '
f'deleted expired files: {deleted}'
)
except asyncio.CancelledError:
raise
except Exception as exc:
self.logger.warning(
f'Storage maintenance failed for Workspace {context.workspace_uuid}: {exc}'
)
if 'invitations' in due:
try:
await self.workspace_collaboration_service.cleanup_expired_invitations(
active_bindings=bindings,
)
except asyncio.CancelledError:
raise
except Exception as exc:
self.logger.warning(f'Expired Workspace invitation cleanup failed: {exc}')
completed_at = loop.time()
for name in due:
next_due[name] = completed_at + maintenance_intervals[name]
self.task_mgr.create_task(
storage_cleanup_loop(),
name='storage-maintenance',
resource_maintenance_loop(),
name='resource-maintenance',
scopes=[core_entities.LifecycleControlScope.APPLICATION],
)
@@ -328,30 +516,68 @@ class Application:
if self.task_mgr is not None:
self.task_mgr.cancel_by_scope(core_entities.LifecycleControlScope.APPLICATION)
with contextlib.suppress(Exception):
await self.event_loop_monitor.stop()
mcp_mount = getattr(self.http_ctrl, 'mcp_mount', None)
if mcp_mount is not None:
with contextlib.suppress(Exception):
await mcp_mount.stop_session_manager()
if self.platform_mgr is not None:
with contextlib.suppress(Exception):
await self.platform_mgr.shutdown()
if self.tool_mgr is not None:
with contextlib.suppress(Exception):
await self.tool_mgr.shutdown()
if self.model_mgr is not None:
with contextlib.suppress(Exception):
await self.model_mgr.shutdown()
if self.box_service is not None:
with contextlib.suppress(Exception):
await self.box_service.shutdown()
if self.plugin_connector is not None:
with contextlib.suppress(Exception):
await self.plugin_connector.aclose()
if self.telemetry is not None:
with contextlib.suppress(Exception):
await self.telemetry.shutdown()
if self.vector_db_mgr is not None:
with contextlib.suppress(Exception):
await self.vector_db_mgr.shutdown()
if self.storage_mgr is not None:
with contextlib.suppress(Exception):
await self.storage_mgr.shutdown()
manifest_provider = getattr(self.deployment, 'manifest_provider', None)
if manifest_provider is not None:
with contextlib.suppress(Exception):
await manifest_provider.aclose()
if self.task_mgr is not None:
tasks = [wrapper.task for wrapper in self.task_mgr.tasks if not wrapper.task.done()]
if tasks:
await asyncio.gather(*tasks, return_exceptions=True)
with contextlib.suppress(Exception):
await httpclient.close_all()
persistence_shutdown = getattr(self.persistence_mgr, 'shutdown', None)
if callable(persistence_shutdown):
with contextlib.suppress(Exception):
await persistence_shutdown()
else:
# Compatibility for lightweight test/application doubles.
persistence_db = getattr(self.persistence_mgr, 'db', None)
persistence_engine = getattr(persistence_db, 'engine', None)
if persistence_engine is not None:
with contextlib.suppress(Exception):
await persistence_engine.dispose()
self._shutdown_complete = True
def dispose(self):
"""Compatibility wrapper for callers that cannot await shutdown."""
if self._shutdown_complete:
return
loop = self.event_loop
if loop is not None and not loop.is_closed():
loop.create_task(self.shutdown())
if self._shutdown_task is None or self._shutdown_task.done():
self._shutdown_task = loop.create_task(self.shutdown())
return
if self.plugin_connector is not None:
self.plugin_connector.dispose()
src/langbot/pkg/core/boot.py
+15 -6
View File
@@ -2,6 +2,7 @@ from __future__ import annotations
import traceback
import asyncio
import contextlib
import os
from . import app
@@ -32,14 +33,22 @@ async def make_app(loop: asyncio.AbstractEventLoop) -> app.Application:
ap.event_loop = loop
# Execute startup stage
for stage_name in stage_order:
stage_cls = stage.preregistered_stages[stage_name]
stage_inst = stage_cls()
try:
# Execute startup stage
for stage_name in stage_order:
stage_cls = stage.preregistered_stages[stage_name]
stage_inst = stage_cls()
await stage_inst.run(ap)
await stage_inst.run(ap)
await ap.initialize()
await ap.initialize()
except BaseException:
# ``main()`` cannot clean up a partially built application because
# ``make_app()`` has not returned it yet. Release managers, pools and
# child processes that earlier startup stages already attached.
with contextlib.suppress(BaseException):
await ap.shutdown()
raise
return ap
src/langbot/pkg/core/errors.py
+2
View File
@@ -0,0 +1,2 @@
class TaskCapacityError(RuntimeError):
"""Raised when the configured user-task admission limit is exhausted."""
src/langbot/pkg/core/stages/build_app.py
+107 -14
View File
@@ -1,7 +1,7 @@
from __future__ import annotations
from .. import stage, app
from ...utils import version, proxy
from ...utils import version, proxy, constants
from ...pipeline import pool, controller, pipelinemgr
from ...pipeline import aggregator as message_aggregator
from ...box import service as box_service
@@ -37,6 +37,16 @@ from ...vector import mgr as vectordb_mgr
from .. import taskmgr
from ...telemetry import telemetry as telemetry_module
from ...survey import manager as survey_module
from ...workspace import service as workspace_service_module
from ...workspace import collaboration as workspace_collaboration_module
from ...workspace import invitation_delivery as invitation_delivery_module
from ...cloud import bootstrap as cloud_bootstrap
from ...cloud import launch as cloud_launch_module
from ...cloud.directory import directory_projection_limits_from_config
from ...cloud.directory_projection import DirectoryProjectionService
from ...cloud.entitlements import EntitlementResolver
from ...api.http.context import ExecutionContext, PrincipalContext, PrincipalType
from ...api.http.authz import WorkspaceRequiredError
@stage.stage_class('BuildAppStage')
@@ -45,15 +55,43 @@ class BuildAppStage(stage.BootingStage):
async def run(self, ap: app.Application):
"""Build LangBot application"""
# Multi-Workspace mode is selected only by an installed closed
# bootstrap that returns a verified Manifest receipt. Mutable values
# such as system.edition are intentionally absent from this boundary.
deployment = await cloud_bootstrap.resolve_deployment(
instance_uuid=constants.instance_id,
instance_config=ap.instance_config.data,
)
ap.deployment = deployment
ap.deployment_admission = cloud_bootstrap.DeploymentAdmissionGuard(
constants.instance_id,
deployment,
)
ap.manifest_refresh_service = (
cloud_bootstrap.CloudManifestRefreshService(
ap.deployment_admission,
deployment.manifest_provider,
ap.logger,
)
if deployment.multi_workspace_enabled
else None
)
ap.entitlement_resolver = (
EntitlementResolver(
constants.instance_id,
deployment.entitlement_provider,
deployment_admission=ap.deployment_admission.require_active,
)
if deployment.multi_workspace_enabled
else None
)
ap.task_mgr = taskmgr.AsyncTaskManager(ap)
discover = discover_engine.ComponentDiscoveryEngine(ap)
discover.discover_blueprint('templates/components.yaml')
ap.discover = discover
user_service_inst = user_service.UserService(ap)
ap.user_service = user_service_inst
space_service_inst = space_service.SpaceService(ap)
ap.space_service = space_service_inst
@@ -98,23 +136,77 @@ class BuildAppStage(stage.BootingStage):
await ver_mgr.initialize()
ap.ver_mgr = ver_mgr
ap.query_pool = pool.QueryPool()
log_cache = logcache.LogCache()
ap.log_cache = log_cache
storage_mgr_inst = storagemgr.StorageMgr(ap)
await storage_mgr_inst.initialize()
ap.storage_mgr = storage_mgr_inst
await storage_mgr_inst.initialize()
persistence_mgr_inst = persistencemgr.PersistenceManager(ap)
persistence_mgr_inst = persistencemgr.PersistenceManager(
ap,
mode=persistencemgr.PersistenceMode(deployment.persistence_mode),
)
ap.persistence_mgr = persistence_mgr_inst
await persistence_mgr_inst.initialize()
if deployment.multi_workspace_enabled:
directory_projection_service = DirectoryProjectionService(
ap,
deployment.directory_provider,
constants.instance_id,
limits=directory_projection_limits_from_config(ap.instance_config.data),
)
await directory_projection_service.initialize()
ap.directory_projection_service = directory_projection_service
workspace_policy = deployment.workspace_policy
workspace_service_inst = workspace_service_module.WorkspaceService(
ap,
policy=workspace_policy,
)
if not workspace_policy.multi_workspace_enabled:
await workspace_service_inst.ensure_singleton_workspace()
ap.workspace_service = workspace_service_inst
if workspace_policy.multi_workspace_enabled:
# Directory refresh starts in Application.run(), after this serial
# build graph. Share one validated immutable binding snapshot
# across model/platform/pipeline/RAG/plugin initialization instead
# of repeating tenant validation for every manager.
await workspace_service_inst.prime_startup_execution_bindings()
ap.workspace_collaboration_service = workspace_collaboration_module.WorkspaceCollaborationService(
ap,
workspace_service_inst,
)
ap.invitation_delivery_service = invitation_delivery_module.InvitationDeliveryService(ap)
ap.space_launch_service = cloud_launch_module.SpaceLaunchService(ap)
user_service_inst = user_service.UserService(ap)
ap.user_service = user_service_inst
async def resolve_singleton_execution_context() -> ExecutionContext:
if workspace_policy.multi_workspace_enabled:
raise WorkspaceRequiredError('Cloud runtime work requires an explicit Workspace context')
binding = await workspace_service_inst.get_local_execution_binding()
return ExecutionContext(
instance_uuid=binding.instance_uuid,
workspace_uuid=binding.workspace_uuid,
placement_generation=binding.placement_generation,
trigger_principal=PrincipalContext(PrincipalType.SYSTEM),
)
concurrency_config = ap.instance_config.data.get('concurrency', {})
ap.query_pool = pool.QueryPool(
singleton_context_resolver=resolve_singleton_execution_context,
max_queries=int(concurrency_config.get('pending_queries', 1000)),
max_queries_per_workspace=int(concurrency_config.get('pending_queries_per_workspace', 100)),
)
# Telemetry manager: attach to app so other components can call via self.ap.telemetry
telemetry_inst = telemetry_module.TelemetryManager(ap)
await telemetry_inst.initialize()
ap.telemetry = telemetry_inst
await telemetry_inst.initialize()
# Survey manager
survey_inst = survey_module.SurveyManager(ap)
@@ -134,16 +226,16 @@ class BuildAppStage(stage.BootingStage):
ap.sess_mgr = llm_session_mgr_inst
box_service_inst = box_service.BoxService(ap)
await box_service_inst.initialize()
ap.box_service = box_service_inst
await box_service_inst.initialize()
llm_tool_mgr_inst = llm_tool_mgr.ToolManager(ap)
await llm_tool_mgr_inst.initialize()
ap.tool_mgr = llm_tool_mgr_inst
await llm_tool_mgr_inst.initialize()
im_mgr_inst = im_mgr.PlatformManager(ap=ap)
await im_mgr_inst.initialize()
ap.platform_mgr = im_mgr_inst
await im_mgr_inst.initialize()
# Initialize webhook pusher
webhook_pusher_inst = WebhookPusher(ap)
@@ -171,12 +263,12 @@ class BuildAppStage(stage.BootingStage):
# 初始化向量数据库管理器
vectordb_mgr_inst = vectordb_mgr.VectorDBManager(ap)
await vectordb_mgr_inst.initialize()
ap.vector_db_mgr = vectordb_mgr_inst
await vectordb_mgr_inst.initialize()
http_ctrl = http_controller.HTTPController(ap)
await http_ctrl.initialize()
ap.http_ctrl = http_ctrl
await http_ctrl.initialize()
monitoring_service_inst = monitoring_service.MonitoringService(ap)
ap.monitoring_service = monitoring_service_inst
@@ -196,6 +288,7 @@ class BuildAppStage(stage.BootingStage):
ap.logger.warning(f'Plugin runtime unavailable during startup; reconnecting in background: {exc}')
plugin_connector_inst.schedule_reconnect()
ap.plugin_connector = plugin_connector_inst
workspace_service_inst.release_startup_execution_bindings()
ctrl = controller.Controller(ap)
ap.ctrl = ctrl
src/langbot/pkg/core/stages/load_config.py
+120 -3
View File
@@ -1,8 +1,9 @@
from __future__ import annotations
import os
import copy
from typing import Any
from langbot.pkg.utils import constants
from langbot.pkg.utils import bounded_executor, constants
import yaml
import importlib.resources as resources
import uuid
@@ -12,6 +13,102 @@ from .. import stage, app
from ..bootutils import config
_RUNTIME_POLICY_DEFAULTS = {
'cloud': {
'directory': {
'max_active_workspaces': 1000,
'max_snapshot_workspaces': 1000,
'max_snapshot_memberships': 20000,
'max_response_bytes': 33554432,
}
},
'database': {
'postgresql': {
'pool_size': 10,
'max_overflow': 10,
'pool_timeout_seconds': 30,
'pool_recycle_seconds': 1800,
'statement_timeout_ms': 60000,
'lock_timeout_ms': 5000,
'idle_in_transaction_session_timeout_ms': 60000,
}
},
'system': {
'blocking_executor': {
'max_workers': bounded_executor.DEFAULT_MAX_WORKERS,
'max_pending': bounded_executor.DEFAULT_MAX_PENDING,
'max_inflight_per_scope': (bounded_executor.DEFAULT_MAX_INFLIGHT_PER_SCOPE),
}
},
'plugin': {
'worker': {
'max_cpus': 1.0,
'max_memory_mb': 512,
'max_pids': 128,
'max_open_files': 256,
'max_file_size_mb': 512,
'max_workers': 16,
'max_total_cpus': 8.0,
'max_total_memory_mb': 8192,
'max_installations': 10000,
'max_concurrent_restarts': 1,
'restart_failure_threshold': 8,
'restart_failure_window_seconds': 30.0,
'restart_circuit_open_seconds': 60.0,
'require_hard_limits': False,
}
},
'mcp': {'stdio': {'enabled': True}},
'monitoring': {
'query_limits': {
'page_rows': 1000,
'export_rows': 10000,
'detail_rows': 2000,
'timeseries_buckets': 1000,
'max_offset': 1000000,
},
'auto_cleanup': {'max_batches_per_table_per_run': 4},
},
'storage': {
'max_object_read_bytes': 10485760,
'cleanup': {'max_files_per_run': 1000},
},
'webhooks': {
'max_per_workspace': 16,
'max_inflight_requests': 16,
},
'box': {
'limits': {
'max_workspace_entries': 100000,
}
},
}
def _complete_runtime_policy_defaults(cfg: dict) -> dict:
"""Backfill typed security-policy leaves before applying env overrides.
The historic config loader intentionally does not deep-complete the whole
template. These fields are different: their native env overrides must
retain boolean/numeric types on upgraded instances, so their defaults must
exist before ``CLOUD__...``, ``PLUGIN__...`` and ``MCP__...`` are parsed.
"""
def merge(target: dict, defaults: dict, path: tuple[str, ...] = ()) -> None:
for key, default in defaults.items():
if key not in target:
target[key] = copy.deepcopy(default)
continue
if isinstance(default, dict):
if not isinstance(target[key], dict):
dotted_path = '.'.join((*path, key))
raise ValueError(f'{dotted_path} must be a mapping')
merge(target[key], default, (*path, key))
merge(cfg, _RUNTIME_POLICY_DEFAULTS)
return cfg
def _apply_env_overrides_to_config(cfg: dict) -> dict:
"""Apply environment variable overrides to data/config.yaml
@@ -64,11 +161,19 @@ def _apply_env_overrides_to_config(cfg: dict) -> dict:
if '__' not in env_key:
continue
print(f'apply env overrides to config: env_key: {env_key}, env_value: {env_value}')
# Convert environment variable name to config path
# e.g., CONCURRENCY__PIPELINE -> ['concurrency', 'pipeline']
keys = [key.lower() for key in env_key.split('__')]
# macOS and some launchers expose variables such as
# ``__CF_USER_TEXT_ENCODING``. They are not LangBot config paths and
# must not create an empty top-level YAML key when config is dumped.
if any(not key for key in keys):
continue
# Values may contain database passwords, runtime control tokens, or
# provider credentials. Keep the useful audit breadcrumb without ever
# copying the secret into startup logs.
print(f'apply env override to config: env_key: {env_key}')
# Navigate to the target value and validate the path
current = cfg
@@ -150,9 +255,21 @@ class LoadConfigStage(stage.BootingStage):
ap.instance_config = await config.load_yaml_config('data/config.yaml', 'config.yaml', completion=False)
# Deep-complete only typed execution-policy fields. This keeps native
# env coercion reliable for existing data/config.yaml files.
ap.instance_config.data = _complete_runtime_policy_defaults(ap.instance_config.data)
# Apply environment variable overrides to data/config.yaml
ap.instance_config.data = _apply_env_overrides_to_config(ap.instance_config.data)
blocking_config = ap.instance_config.data['system']['blocking_executor']
ap.blocking_executor = bounded_executor.configure_bounded_default_executor(
ap.event_loop,
max_workers=blocking_config['max_workers'],
max_pending=blocking_config['max_pending'],
max_inflight_per_scope=blocking_config['max_inflight_per_scope'],
)
await ap.instance_config.dump_config()
# load or generate instance id
src/langbot/pkg/core/stages/show_notes.py
+8 -3
View File
@@ -1,8 +1,7 @@
from __future__ import annotations
import asyncio
from .. import stage, app, note
from .. import entities as core_entities
from ...utils import importutil
from .. import notes
@@ -31,6 +30,12 @@ class ShowNotesStage(stage.BootingStage):
if msg:
ap.logger.log(level, msg)
asyncio.create_task(ayield_note(note_inst))
ap.task_mgr.create_task(
ayield_note(note_inst),
kind='launch-note',
name=f'launch-note-{note_cls.__name__}',
scopes=[core_entities.LifecycleControlScope.APPLICATION],
instance_uuid=ap.workspace_service.instance_uuid,
)
except Exception:
continue
src/langbot/pkg/core/task_boundary.py
+82
View File
@@ -0,0 +1,82 @@
from __future__ import annotations
import asyncio
import contextvars
import typing
from ..utils import bounded_executor
T = typing.TypeVar('T')
def create_detached_task(
coro: typing.Coroutine[typing.Any, typing.Any, T],
*,
loop: asyncio.AbstractEventLoop | None = None,
name: str | None = None,
after_commit_manager: typing.Any | None = None,
workspace_uuid: str | None = None,
) -> asyncio.Task[T]:
"""Create a task that inherits no request-local ContextVars.
A normal ``asyncio.create_task`` copies the caller's context. That is
unsafe for work which outlives an HTTP request because it can copy the
request's active database transaction or trusted tenant scope into a
different asyncio task. Detached work must receive durable identity such
as ``ExecutionContext`` through explicit arguments and establish its own
tenant scope or unit of work whenever it accesses persistence.
"""
task_loop = loop or asyncio.get_running_loop()
gate: asyncio.Future[None] | None = None
# Inspect the type so dynamic Mock/AsyncMock attributes do not turn into a
# fake gate in lightweight tests or embedders.
gate_factory = getattr(type(after_commit_manager), 'create_after_commit_gate', None)
if callable(gate_factory):
gate = gate_factory(after_commit_manager)
task_coro = _wait_for_commit(coro, gate) if gate is not None else coro
if workspace_uuid is not None:
task_coro = bounded_executor.run_in_blocking_work_scope(
task_coro,
workspace_uuid,
)
return task_loop.create_task(task_coro, name=name, context=contextvars.Context())
async def _wait_for_commit(
coro: typing.Coroutine[typing.Any, typing.Any, T],
gate: asyncio.Future[None],
) -> T:
try:
await gate
except BaseException:
coro.close()
raise
return await coro
async def run_in_workspace_uow(
ap: typing.Any,
workspace_uuid: str,
operation: typing.Callable[[], typing.Awaitable[T]],
) -> T:
"""Run one short persistence section in a detached Cloud task scope.
This helper deliberately scopes only the supplied operation. Callers
should not wrap long-running network or runtime work in a database
transaction.
"""
persistence_mgr = getattr(ap, 'persistence_mgr', None)
if persistence_mgr is None:
return await operation()
cloud_runtime = getattr(getattr(persistence_mgr, 'mode', None), 'value', None) == 'cloud_runtime'
if not cloud_runtime:
return await operation()
tenant_uow = getattr(persistence_mgr, 'tenant_uow', None)
if not callable(tenant_uow):
raise RuntimeError('Detached Cloud tasks require an explicit tenant unit of work')
async with tenant_uow(workspace_uuid):
return await operation()
src/langbot/pkg/core/taskmgr.py
+127 -7
View File
@@ -7,6 +7,8 @@ import time
from . import app
from . import entities as core_entities
from .errors import TaskCapacityError
from .task_boundary import create_detached_task
class TaskContext:
@@ -21,13 +23,18 @@ class TaskContext:
metadata: dict
"""Structured metadata for progress reporting"""
def __init__(self):
def __init__(self, max_log_chars: int = 200000):
self.current_action = 'default'
self.log = ''
self.metadata = {}
self.max_log_chars = max(int(max_log_chars), 1)
def _log(self, msg: str):
self.log += msg + '\n'
if len(self.log) > self.max_log_chars:
marker = '[older task output truncated]\n'
keep = max(self.max_log_chars - len(marker), 0)
self.log = marker + (self.log[-keep:] if keep else '')
def set_current_action(self, action: str):
self.current_action = action
@@ -98,6 +105,15 @@ class TaskWrapper:
scopes: list[core_entities.LifecycleControlScope]
"""Task scope"""
instance_uuid: str | None
"""Owning LangBot instance for a tenant user task."""
workspace_uuid: str | None
"""Owning Workspace for a tenant user task."""
placement_generation: int | None
"""Workspace execution fence captured when the task was created."""
def __init__(
self,
ap: app.Application,
@@ -108,18 +124,30 @@ class TaskWrapper:
label: str = '',
context: TaskContext = None,
scopes: list[core_entities.LifecycleControlScope] = [core_entities.LifecycleControlScope.APPLICATION],
instance_uuid: str | None = None,
workspace_uuid: str | None = None,
placement_generation: int | None = None,
):
self.id = TaskWrapper._id_index
TaskWrapper._id_index += 1
self.ap = ap
self.task_context = context or TaskContext()
self.task = self.ap.event_loop.create_task(coro)
self.task = create_detached_task(
coro,
loop=self.ap.event_loop,
name=name or None,
after_commit_manager=getattr(self.ap, 'persistence_mgr', None),
workspace_uuid=workspace_uuid,
)
self.task_type = task_type
self.kind = kind
self.name = name
self.label = label if label != '' else name
self.task.set_name(name)
self.scopes = scopes
self.instance_uuid = instance_uuid
self.workspace_uuid = workspace_uuid
self.placement_generation = placement_generation
self.created_at = time.time()
def assume_exception(self):
@@ -155,6 +183,8 @@ class TaskWrapper:
'kind': self.kind,
'name': self.name,
'label': self.label,
'workspace_uuid': self.workspace_uuid,
'placement_generation': self.placement_generation,
'scopes': [scope.value for scope in self.scopes],
'created_at': self.created_at,
'task_context': self.task_context.to_dict(),
@@ -184,6 +214,39 @@ class AsyncTaskManager:
self.ap = ap
self.tasks = []
def _task_log_limit(self) -> int:
value = self.ap.instance_config.data.get('system', {}).get('task_retention', {}).get('max_log_chars', 200000)
try:
value = int(value)
except (TypeError, ValueError):
value = 200000
return max(value, 1)
def _user_task_limit(self, name: str, default: int) -> int:
value = self.ap.instance_config.data.get('system', {}).get('task_retention', {}).get(name, default)
try:
value = int(value)
except (TypeError, ValueError):
value = default
return max(value, 1)
def _admit_user_task(self, coro: typing.Coroutine, workspace_uuid: str | None) -> None:
active_user_tasks = [
wrapper for wrapper in self.tasks if wrapper.task_type == 'user' and not wrapper.task.done()
]
global_limit = self._user_task_limit('max_active_user_tasks', 256)
if len(active_user_tasks) >= global_limit:
coro.close()
raise TaskCapacityError('The instance has too many active user operations')
if workspace_uuid is None:
return
workspace_limit = self._user_task_limit('max_active_user_tasks_per_workspace', 8)
active_workspace_tasks = sum(1 for wrapper in active_user_tasks if wrapper.workspace_uuid == workspace_uuid)
if active_workspace_tasks >= workspace_limit:
coro.close()
raise TaskCapacityError('The Workspace has too many active user operations')
def create_task(
self,
coro: typing.Coroutine,
@@ -193,8 +256,30 @@ class AsyncTaskManager:
label: str = '',
context: TaskContext = None,
scopes: list[core_entities.LifecycleControlScope] = [core_entities.LifecycleControlScope.APPLICATION],
instance_uuid: str | None = None,
workspace_uuid: str | None = None,
placement_generation: int | None = None,
) -> TaskWrapper:
wrapper = TaskWrapper(self.ap, coro, task_type, kind, name, label, context, scopes)
if context is None:
context = TaskContext(max_log_chars=self._task_log_limit())
else:
context.max_log_chars = self._task_log_limit()
if len(context.log) > context.max_log_chars:
context.log = context.log[-context.max_log_chars :]
wrapper = TaskWrapper(
self.ap,
coro,
task_type,
kind,
name,
label,
context,
scopes,
instance_uuid,
workspace_uuid,
placement_generation,
)
self.tasks.append(wrapper)
wrapper.task.add_done_callback(lambda _: self._prune_completed_tasks())
self._prune_completed_tasks()
@@ -208,8 +293,23 @@ class AsyncTaskManager:
label: str = '',
context: TaskContext = None,
scopes: list[core_entities.LifecycleControlScope] = [core_entities.LifecycleControlScope.APPLICATION],
instance_uuid: str | None = None,
workspace_uuid: str | None = None,
placement_generation: int | None = None,
) -> TaskWrapper:
return self.create_task(coro, 'user', kind, name, label, context, scopes)
self._admit_user_task(coro, workspace_uuid)
return self.create_task(
coro,
'user',
kind,
name,
label,
context,
scopes,
instance_uuid,
workspace_uuid,
placement_generation,
)
async def wait_all(self):
await asyncio.gather(*[t.task for t in self.tasks], return_exceptions=True)
@@ -221,12 +321,20 @@ class AsyncTaskManager:
self,
type: str = None,
kind: str = None,
*,
instance_uuid: str | None = None,
workspace_uuid: str | None = None,
placement_generation: int | None = None,
) -> dict:
return {
'tasks': [
t.to_dict()
for t in self.tasks
if (type is None or t.task_type == type) and (kind is None or t.kind == kind)
if (type is None or t.task_type == type)
and (kind is None or t.kind == kind)
and (instance_uuid is None or t.instance_uuid == instance_uuid)
and (workspace_uuid is None or t.workspace_uuid == workspace_uuid)
and (placement_generation is None or t.placement_generation == placement_generation)
],
'id_index': TaskWrapper._id_index,
}
@@ -240,9 +348,21 @@ class AsyncTaskManager:
'id_index': TaskWrapper._id_index,
}
def get_task_by_id(self, id: int) -> TaskWrapper | None:
def get_task_by_id(
self,
id: int,
*,
instance_uuid: str | None = None,
workspace_uuid: str | None = None,
placement_generation: int | None = None,
) -> TaskWrapper | None:
for t in self.tasks:
if t.id == id:
if (
t.id == id
and (instance_uuid is None or t.instance_uuid == instance_uuid)
and (workspace_uuid is None or t.workspace_uuid == workspace_uuid)
and (placement_generation is None or t.placement_generation == placement_generation)
):
return t
return None