feat(tenancy): add Workspace multi-tenant foundation (#2353)

* Document multi-tenant workspace architecture * Add OSS and commercial workspace boundaries * docs: redesign multi-tenant workspace architecture * feat(tenancy): implement workspace isolation * docs(tenancy): record verification evidence * docs(tenancy): revise single-instance SaaS topology * docs(tenancy): refine architecture options * docs: finalize cloud v2 multi-tenant decisions * feat(tenancy): establish cloud isolation foundations * feat(tenancy): harden shared cloud runtime boundaries * docs(tenancy): record final isolation verification * fix(tenancy): close isolation and permission gaps * docs(tenancy): record final isolation verification * feat(tenancy): connect cloud workspace control plane * fix(build): install git for pinned SDK * docs(cloud): update control plane verification * chore: update multi-tenant SDK pin * fix(cloud): skip legacy model sync during startup * test(cloud): preserve minimal model manager fixtures * fix(cloud): preserve authenticated account context * fix(cloud): reuse authenticated account for user info * feat(cloud): complete Workspace settings navigation * test(web): cover Workspace dropdown menu * feat(web): place workspace controls in sidebar * refactor(web): streamline workspace controls * style(web): format workspace layout test * fix(cloud): surface runtime and workspace plan status * fix(plugin): keep runtime identity stable across restarts * fix(ui): widen and center workspace switcher * fix(ui): hide roles from workspace switcher * fix(ui): align workspace switcher with sidebar entries * feat(workspace): add in-product collaboration and direct Cloud launch * style: format collaboration changes * fix(workspace): bind collaboration APIs to tenant UoW * fix(cloud): preserve Core-owned collaboration state * test(cloud): require Space identity for invite registration * feat(cloud): complete secure invitation experience * style(web): format invitation flows * fix(cloud): recover box runtime without unscoped skill reload * feat(oss): enforce invitation account and owner billing flows * style: format OSS account service * test(oss): cover invitation logout handoff * fix(oss): resolve workspace owner in scoped session * feat(cloud): harden multi-tenant runtime resources * fix(cloud): bound runtime restart storms * fix(cloud): eliminate periodic runtime CPU spikes * fix(cloud): enforce instance capacity ceilings * fix(cloud): scope public login capability discovery * fix(cloud): bound tenant maintenance and monitoring work * fix(runtime): bound tenant resource amplification * fix(deps): pin green multi-tenant plugin SDK * fix(cloud): handle unavailable skill capability * fix(security): require authentication for image file endpoint (H-2) - Changed /api/v1/files/image from AuthType.NONE to USER_TOKEN_OR_API_KEY - Added Permission.RESOURCE_VIEW requirement - Prevents unauthenticated cross-tenant file access via leaked keys - Fixes HIGH severity finding from multi-tenant security review docs: add comprehensive database migration guide - Complete migration steps for OSS → multi-tenant - Backup, execution, verification procedures - Rollback scenarios and recovery plans - Performance tuning recommendations * test: add comprehensive cross-tenant isolation tests Added 7 critical test scenarios for multi-tenant boundaries: - Cross-tenant bot access prevention - Viewer role read-only enforcement - Removed member immediate access revocation - Model provider credential isolation - WebSocket message isolation - Invitation token workspace scoping - Multi-workspace context validation These tests address P0-2 coverage gaps for: - workspaces.py (membership & invitation flows) - user.py (authentication & authorization) - websocket_chat.py (real-time isolation) - plugins.py (resource access control) docs: finalize database migration guide * fix(security): resolve M-1, M-2, M-3 security findings M-1: WebSocket authorization TOCTOU race (FIXED) - Changed _revalidate_websocket_authorization to return RequestContext - Ensures validated context is used immediately without race window - Prevents removed members from sending messages during revalidation gap M-2: Model Manager cache workspace isolation (VERIFIED) - Confirmed _CacheKey already uses 4-tuple: (instance, workspace, generation, resource) - Cache is properly scoped per workspace, no cross-tenant leakage possible - No code change needed, documented as working correctly M-3: Invitation lock workspace scoping (FIXED) - Changed lock key from token_digest to workspace_uuid:token_digest - Prevents DoS where attacker locks token in Workspace A to block Workspace B - Locks now isolated per workspace All MEDIUM severity findings from security review now resolved. * fix(cloud): unblock tenant CI and enforce knowledge quotas * fix(tenancy): scope rerank model sync --------- Co-authored-by: dadachann <185672915+dadachann@users.noreply.github.com>
2026-08-09 04:40:57 +00:00 · 2026-07-30 21:43:35 +08:00
parent 463b120923
commit e1ac5e0fc8
468 changed files with 78320 additions and 13137 deletions
@@ -1,11 +1,30 @@
 from __future__ import annotations

+import hashlib
+import json
+import re
+from pathlib import PurePath

 from ..core import app
+from ..utils import bounded_executor
+from ..api.http.authz import WorkspaceRequiredError
+from ..api.http.context import ExecutionContext, RequestContext
 from . import provider
 from .providers import localstorage


+_SAFE_OWNER_TYPE = re.compile(r'^[a-z][a-z0-9_-]{0,63}$')
+_DEFAULT_OBJECT_READ_BYTES = 10 * 1024 * 1024
+_SCOPED_KEY = re.compile(
+    r'^v1/(?P<instance>[a-f0-9]{24})/'
+    r'(?P<workspace>[0-9a-fA-F-]{36})/'
+    r'(?P<generation>[1-9][0-9]*)/'
+    r'(?P<owner_type>[a-z][a-z0-9_-]{0,63})/'
+    r'(?P<owner>[a-f0-9]{32})/'
+    r'(?P<key>[a-f0-9]{64})(?P<suffix>\.[a-zA-Z0-9]{1,16})?$'
+)
+
+
 class StorageMgr:
    """Storage manager"""

@@ -16,6 +35,323 @@ class StorageMgr:
    def __init__(self, ap: app.Application):
        self.ap = ap

+    def _object_read_limit(self) -> int:
+        config = getattr(getattr(self.ap, 'instance_config', None), 'data', {})
+        try:
+            configured = int(
+                config.get('storage', {}).get(
+                    'max_object_read_bytes',
+                    _DEFAULT_OBJECT_READ_BYTES,
+                )
+            )
+        except (AttributeError, TypeError, ValueError):
+            configured = _DEFAULT_OBJECT_READ_BYTES
+        return min(max(configured, 1), provider.HARD_MAX_STORAGE_OBJECT_BYTES)
+
+    async def _load_object_bounded(self, object_key: str) -> bytes:
+        max_bytes = self._object_read_limit()
+        bounded_loader = getattr(self.storage_provider, 'load_bounded', None)
+        if callable(bounded_loader):
+            return await bounded_loader(object_key, max_bytes=max_bytes)
+
+        # Compatibility for lightweight and third-party providers. Built-in
+        # providers enforce the same bound in the actual read operation.
+        object_size = await self.storage_provider.size(object_key)
+        if object_size > max_bytes:
+            raise ValueError(f'Storage object exceeds the {max_bytes}-byte read limit')
+        value = await self.storage_provider.load(object_key)
+        if len(value) > max_bytes:
+            raise ValueError(f'Storage object exceeds the {max_bytes}-byte read limit')
+        return value
+
+    @staticmethod
+    def _require_execution_scope(
+        context: ExecutionContext | RequestContext,
+    ) -> tuple[str, str, int]:
+        if not isinstance(context, (ExecutionContext, RequestContext)):
+            raise WorkspaceRequiredError('Storage operations require an explicit Workspace context')
+        instance_uuid = context.instance_uuid.strip()
+        workspace_uuid = context.workspace_uuid.strip()
+        generation = context.placement_generation
+        if not instance_uuid or not workspace_uuid:
+            raise WorkspaceRequiredError('Storage operations require an instance and Workspace')
+        if generation <= 0:
+            raise WorkspaceRequiredError('Storage operations require a positive placement generation')
+        return instance_uuid, workspace_uuid, generation
+
+    @staticmethod
+    def _digest(value: str, length: int) -> str:
+        return hashlib.sha256(value.encode('utf-8')).hexdigest()[:length]
+
+    async def _require_active_execution_scope(
+        self,
+        context: ExecutionContext | RequestContext,
+    ) -> None:
+        """Revalidate the captured generation before touching object storage."""
+        instance_uuid, workspace_uuid, generation = self._require_execution_scope(context)
+        workspace_service = getattr(self.ap, 'workspace_service', None)
+        if workspace_service is None:
+            raise WorkspaceRequiredError('Storage execution scope is unavailable')
+        try:
+            binding = await workspace_service.get_execution_binding(
+                workspace_uuid,
+                expected_generation=generation,
+            )
+        except Exception as exc:
+            raise WorkspaceRequiredError('Storage execution scope is unavailable') from exc
+        if (
+            getattr(binding, 'instance_uuid', None) != instance_uuid
+            or getattr(binding, 'workspace_uuid', None) != workspace_uuid
+            or getattr(binding, 'placement_generation', None) != generation
+        ):
+            raise WorkspaceRequiredError('Storage execution scope is unavailable')
+
+    @classmethod
+    def canonical_binary_storage_key(
+        cls,
+        context: ExecutionContext | RequestContext,
+        *,
+        owner_type: str,
+        owner: str,
+        key: str,
+    ) -> str:
+        """Return a bounded canonical key over every BinaryStorage owner dimension."""
+
+        instance_uuid, workspace_uuid, _ = cls._require_execution_scope(context)
+        if not _SAFE_OWNER_TYPE.fullmatch(owner_type):
+            raise ValueError('Invalid storage owner_type')
+        if not owner or not key:
+            raise ValueError('Storage owner and key are required')
+        canonical = json.dumps(
+            [instance_uuid, workspace_uuid, owner_type, owner, key],
+            ensure_ascii=False,
+            separators=(',', ':'),
+        )
+        return f'v1:{cls._digest(instance_uuid, 24)}:{workspace_uuid}:{owner_type}:{hashlib.sha256(canonical.encode()).hexdigest()}'
+
+    @classmethod
+    def scoped_object_key(
+        cls,
+        context: ExecutionContext | RequestContext,
+        *,
+        owner_type: str,
+        owner: str,
+        key: str,
+        preserve_suffix: bool = True,
+    ) -> str:
+        """Build a non-enumerable object key with an explicit tenant boundary."""
+
+        instance_uuid, workspace_uuid, generation = cls._require_execution_scope(context)
+        if not _SAFE_OWNER_TYPE.fullmatch(owner_type):
+            raise ValueError('Invalid storage owner_type')
+        if not owner or not key:
+            raise ValueError('Storage owner and key are required')
+        suffix = PurePath(key).suffix.lower() if preserve_suffix else ''
+        if not re.fullmatch(r'\.[a-z0-9]{1,16}', suffix):
+            suffix = ''
+        return (
+            f'v1/{cls._digest(instance_uuid, 24)}/{workspace_uuid}/{generation}/'
+            f'{owner_type}/{cls._digest(owner, 32)}/{cls._digest(key, 64)}{suffix}'
+        )
+
+    @classmethod
+    def scoped_prefix(
+        cls,
+        context: ExecutionContext | RequestContext,
+        *,
+        owner_type: str | None = None,
+    ) -> str:
+        instance_uuid, workspace_uuid, generation = cls._require_execution_scope(context)
+        prefix = f'v1/{cls._digest(instance_uuid, 24)}/{workspace_uuid}/{generation}/'
+        if owner_type is not None:
+            if not _SAFE_OWNER_TYPE.fullmatch(owner_type):
+                raise ValueError('Invalid storage owner_type')
+            prefix += f'{owner_type}/'
+        return prefix
+
+    async def save_scoped(
+        self,
+        context: ExecutionContext | RequestContext,
+        *,
+        owner_type: str,
+        owner: str,
+        key: str,
+        value: bytes,
+        preserve_suffix: bool = True,
+    ) -> str:
+        await self._require_active_execution_scope(context)
+        max_bytes = self._object_read_limit()
+        if len(value) > max_bytes:
+            raise ValueError(f'Storage object exceeds the {max_bytes}-byte write limit')
+        object_key = self.scoped_object_key(
+            context,
+            owner_type=owner_type,
+            owner=owner,
+            key=key,
+            preserve_suffix=preserve_suffix,
+        )
+        await self.storage_provider.save(object_key, value)
+        return object_key
+
+    async def load_scoped(
+        self,
+        context: ExecutionContext | RequestContext,
+        *,
+        owner_type: str,
+        owner: str,
+        key: str,
+        preserve_suffix: bool = True,
+    ) -> bytes:
+        await self._require_active_execution_scope(context)
+        object_key = self.scoped_object_key(
+            context,
+            owner_type=owner_type,
+            owner=owner,
+            key=key,
+            preserve_suffix=preserve_suffix,
+        )
+        return await self._load_object_bounded(object_key)
+
+    async def delete_scoped(
+        self,
+        context: ExecutionContext | RequestContext,
+        *,
+        owner_type: str,
+        owner: str,
+        key: str,
+        preserve_suffix: bool = True,
+    ) -> None:
+        await self._require_active_execution_scope(context)
+        object_key = self.scoped_object_key(
+            context,
+            owner_type=owner_type,
+            owner=owner,
+            key=key,
+            preserve_suffix=preserve_suffix,
+        )
+        await self.storage_provider.delete(object_key)
+
+    async def resolve_public_object(
+        self,
+        object_key: str,
+        *,
+        expected_owner_type: str,
+    ) -> bytes | None:
+        """Load an opaque public object after validating its trusted scope."""
+
+        match = _SCOPED_KEY.fullmatch(object_key)
+        if match is None or match.group('owner_type') != expected_owner_type:
+            return None
+        if match.group('instance') != self._digest(self.ap.workspace_service.instance_uuid, 24):
+            return None
+        workspace_uuid = match.group('workspace')
+        generation = int(match.group('generation'))
+        with bounded_executor.blocking_work_scope(workspace_uuid):
+            try:
+                await self.ap.workspace_service.get_execution_binding(
+                    workspace_uuid,
+                    expected_generation=generation,
+                )
+            except Exception:
+                return None
+            if not await self.storage_provider.exists(object_key):
+                return None
+            return await self._load_object_bounded(object_key)
+
+    @classmethod
+    def require_scoped_object_key(
+        cls,
+        context: ExecutionContext | RequestContext,
+        object_key: str,
+        *,
+        expected_owner_type: str,
+    ) -> None:
+        """Validate an opaque object key against every captured scope field."""
+
+        instance_uuid, workspace_uuid, generation = cls._require_execution_scope(context)
+        match = _SCOPED_KEY.fullmatch(object_key)
+        if (
+            match is None
+            or match.group('instance') != cls._digest(instance_uuid, 24)
+            or match.group('workspace') != workspace_uuid
+            or int(match.group('generation')) != generation
+            or match.group('owner_type') != expected_owner_type
+        ):
+            raise WorkspaceRequiredError('Object key does not belong to the execution scope')
+
+    async def exists_scoped_object_key(
+        self,
+        context: ExecutionContext | RequestContext,
+        object_key: str,
+        *,
+        expected_owner_type: str,
+    ) -> bool:
+        await self._require_active_execution_scope(context)
+        self.require_scoped_object_key(
+            context,
+            object_key,
+            expected_owner_type=expected_owner_type,
+        )
+        return await self.storage_provider.exists(object_key)
+
+    async def load_scoped_object_key(
+        self,
+        context: ExecutionContext | RequestContext,
+        object_key: str,
+        *,
+        expected_owner_type: str,
+    ) -> bytes:
+        await self._require_active_execution_scope(context)
+        self.require_scoped_object_key(
+            context,
+            object_key,
+            expected_owner_type=expected_owner_type,
+        )
+        return await self._load_object_bounded(object_key)
+
+    async def size_scoped_object_key(
+        self,
+        context: ExecutionContext | RequestContext,
+        object_key: str,
+        *,
+        expected_owner_type: str,
+    ) -> int:
+        await self._require_active_execution_scope(context)
+        self.require_scoped_object_key(
+            context,
+            object_key,
+            expected_owner_type=expected_owner_type,
+        )
+        return await self.storage_provider.size(object_key)
+
+    async def delete_scoped_object_key(
+        self,
+        context: ExecutionContext | RequestContext,
+        object_key: str,
+        *,
+        expected_owner_type: str,
+    ) -> None:
+        """Delete a previously returned key only inside the captured scope."""
+
+        await self._require_active_execution_scope(context)
+        self.require_scoped_object_key(
+            context,
+            object_key,
+            expected_owner_type=expected_owner_type,
+        )
+        if await self.storage_provider.exists(object_key):
+            await self.storage_provider.delete(object_key)
+
+    @classmethod
+    def is_scoped_object_key(
+        cls,
+        object_key: str,
+        *,
+        expected_owner_type: str | None = None,
+    ) -> bool:
+        match = _SCOPED_KEY.fullmatch(object_key)
+        return match is not None and (expected_owner_type is None or match.group('owner_type') == expected_owner_type)
+
    async def initialize(self):
        storage_config = self.ap.instance_config.data.get('storage', {})
        storage_type = storage_config.get('use', 'local')
@@ -30,3 +366,8 @@ class StorageMgr:
            self.ap.logger.info('Initialized local storage backend.')

        await self.storage_provider.initialize()
+
+    async def shutdown(self) -> None:
+        storage_provider = getattr(self, 'storage_provider', None)
+        if storage_provider is not None:
+            await storage_provider.shutdown()
@@ -5,6 +5,19 @@ import abc
 from ..core import app


+HARD_MAX_STORAGE_OBJECT_BYTES = 64 * 1024 * 1024
+
+
+def normalize_read_limit(max_bytes: int) -> int:
+    """Validate a provider read limit without allowing callers to bypass the hard cap."""
+
+    try:
+        normalized = int(max_bytes)
+    except (TypeError, ValueError):
+        normalized = HARD_MAX_STORAGE_OBJECT_BYTES
+    return min(max(normalized, 1), HARD_MAX_STORAGE_OBJECT_BYTES)
+
+
 class StorageProvider(abc.ABC):
    ap: app.Application

@@ -14,6 +27,11 @@ class StorageProvider(abc.ABC):
    async def initialize(self):
        pass

+    async def shutdown(self) -> None:
+        """Release provider-owned clients or pools."""
+
+        return None
+
    @abc.abstractmethod
    async def save(
        self,
@@ -29,6 +47,23 @@ class StorageProvider(abc.ABC):
    ) -> bytes:
        pass

+    async def load_bounded(self, key: str, *, max_bytes: int) -> bytes:
+        """Fallback for third-party providers that have not implemented streaming bounds.
+
+        Built-in providers override this method so the byte limit is enforced by
+        the actual read. The size check still protects compatible providers from
+        downloading a known oversized object.
+        """
+
+        max_bytes = normalize_read_limit(max_bytes)
+        object_size = await self.size(key)
+        if object_size > max_bytes:
+            raise ValueError(f'Storage object exceeds the {max_bytes}-byte read limit')
+        value = await self.load(key)
+        if len(value) > max_bytes:
+            raise ValueError(f'Storage object exceeds the {max_bytes}-byte read limit')
+        return value
+
    @abc.abstractmethod
    async def exists(
        self,
@@ -1,5 +1,6 @@
 from __future__ import annotations

+import asyncio
 import os
 import aiofiles
 import shutil
@@ -40,10 +41,9 @@ class LocalStorageProvider(provider.StorageProvider):
        key: str,
        value: bytes,
    ):
-        resolved = _safe_resolve(LOCAL_STORAGE_PATH, key)
+        resolved = await asyncio.to_thread(_safe_resolve, LOCAL_STORAGE_PATH, key)
        parent = os.path.dirname(resolved)
-        if not os.path.exists(parent):
-            os.makedirs(parent)
+        await asyncio.to_thread(os.makedirs, parent, exist_ok=True)
        async with aiofiles.open(resolved, 'wb') as f:
            await f.write(value)

@@ -51,36 +51,51 @@ class LocalStorageProvider(provider.StorageProvider):
        self,
        key: str,
    ) -> bytes:
-        resolved = _safe_resolve(LOCAL_STORAGE_PATH, key)
+        return await self.load_bounded(key, max_bytes=provider.HARD_MAX_STORAGE_OBJECT_BYTES)
+
+    async def load_bounded(
+        self,
+        key: str,
+        *,
+        max_bytes: int,
+    ) -> bytes:
+        max_bytes = provider.normalize_read_limit(max_bytes)
+        resolved = await asyncio.to_thread(_safe_resolve, LOCAL_STORAGE_PATH, key)
        async with aiofiles.open(resolved, 'rb') as f:
-            return await f.read()
+            value = await f.read(max_bytes + 1)
+        if len(value) > max_bytes:
+            raise ValueError(f'Storage object exceeds the {max_bytes}-byte read limit')
+        return value

    async def exists(
        self,
        key: str,
    ) -> bool:
-        resolved = _safe_resolve(LOCAL_STORAGE_PATH, key)
-        return os.path.exists(resolved)
+        resolved = await asyncio.to_thread(_safe_resolve, LOCAL_STORAGE_PATH, key)
+        return await asyncio.to_thread(os.path.exists, resolved)

    async def delete(
        self,
        key: str,
    ):
-        resolved = _safe_resolve(LOCAL_STORAGE_PATH, key)
-        os.remove(resolved)
+        resolved = await asyncio.to_thread(_safe_resolve, LOCAL_STORAGE_PATH, key)
+        await asyncio.to_thread(os.remove, resolved)

    async def size(
        self,
        key: str,
    ) -> int:
-        resolved = _safe_resolve(LOCAL_STORAGE_PATH, key)
-        return os.path.getsize(resolved)
+        resolved = await asyncio.to_thread(_safe_resolve, LOCAL_STORAGE_PATH, key)
+        return await asyncio.to_thread(os.path.getsize, resolved)

    async def delete_dir_recursive(
        self,
        dir_path: str,
    ):
-        resolved = _safe_resolve(LOCAL_STORAGE_PATH, dir_path)
-        # 直接删除整个目录
-        if os.path.exists(resolved):
-            shutil.rmtree(resolved)
+        resolved = await asyncio.to_thread(
+            _safe_resolve,
+            LOCAL_STORAGE_PATH,
+            dir_path,
+        )
+        if await asyncio.to_thread(os.path.exists, resolved):
+            await asyncio.to_thread(shutil.rmtree, resolved)
@@ -1,9 +1,12 @@
 from __future__ import annotations

+import asyncio
+
 import boto3
 from botocore.exceptions import ClientError

 from ...core import app
+from ...utils import bounded_executor
 from .. import provider


@@ -14,6 +17,7 @@ class S3StorageProvider(provider.StorageProvider):
        super().__init__(ap)
        self.s3_client = None
        self.bucket_name = None
+        self._io_semaphore = asyncio.Semaphore(16)

    async def initialize(self):
        """Initialize S3 client with configuration from config.yaml"""
@@ -26,6 +30,11 @@ class S3StorageProvider(provider.StorageProvider):
        secret_access_key = s3_config.get('secret_access_key', '')
        region_name = s3_config.get('region', 'us-east-1')
        self.bucket_name = s3_config.get('bucket', 'langbot-storage')
+        try:
+            max_concurrency = int(s3_config.get('max_concurrency', 16))
+        except (TypeError, ValueError):
+            max_concurrency = 16
+        self._io_semaphore = asyncio.Semaphore(max(1, min(max_concurrency, 128)))

        # Initialize S3 client
        session = boto3.session.Session()
@@ -37,7 +46,25 @@ class S3StorageProvider(provider.StorageProvider):
            aws_secret_access_key=secret_access_key,
        )

-        # Ensure bucket exists
+        await self._run_io(self._ensure_bucket)
+
+    async def shutdown(self) -> None:
+        """Close the botocore HTTP connection pool without blocking the loop."""
+
+        client = self.s3_client
+        self.s3_client = None
+        if client is not None:
+            await bounded_executor.run_blocking_cleanup(client.close)
+
+    async def _run_io(self, operation, /, *args, **kwargs):
+        """Run one blocking boto3 operation behind a bounded concurrency gate."""
+
+        async with self._io_semaphore:
+            return await asyncio.to_thread(operation, *args, **kwargs)
+
+    def _ensure_bucket(self) -> None:
+        """Probe/create the bucket without blocking the application event loop."""
+
        try:
            self.s3_client.head_bucket(Bucket=self.bucket_name)
        except ClientError as e:
@@ -61,7 +88,8 @@ class S3StorageProvider(provider.StorageProvider):
    ):
        """Save bytes to S3"""
        try:
-            self.s3_client.put_object(
+            await self._run_io(
+                self.s3_client.put_object,
                Bucket=self.bucket_name,
                Key=key,
                Body=value,
@@ -73,25 +101,48 @@ class S3StorageProvider(provider.StorageProvider):
    async def load(
        self,
        key: str,
+    ) -> bytes:
+        return await self.load_bounded(key, max_bytes=provider.HARD_MAX_STORAGE_OBJECT_BYTES)
+
+    async def load_bounded(
+        self,
+        key: str,
+        *,
+        max_bytes: int,
    ) -> bytes:
        """Load bytes from S3"""
+        max_bytes = provider.normalize_read_limit(max_bytes)
        try:
-            response = self.s3_client.get_object(
-                Bucket=self.bucket_name,
-                Key=key,
-            )
-            return response['Body'].read()
+            return await self._run_io(self._load_sync, key, max_bytes)
        except Exception as e:
            self.ap.logger.error(f'Failed to load from S3: {e}')
            raise

+    def _load_sync(self, key: str, max_bytes: int) -> bytes:
+        response = self.s3_client.get_object(
+            Bucket=self.bucket_name,
+            Key=key,
+        )
+        body = response['Body']
+        try:
+            declared_size = response.get('ContentLength')
+            if declared_size is not None and declared_size > max_bytes:
+                raise ValueError(f'Storage object exceeds the {max_bytes}-byte read limit')
+            value = body.read(max_bytes + 1)
+            if len(value) > max_bytes:
+                raise ValueError(f'Storage object exceeds the {max_bytes}-byte read limit')
+            return value
+        finally:
+            body.close()
+
    async def exists(
        self,
        key: str,
    ) -> bool:
        """Check if object exists in S3"""
        try:
-            self.s3_client.head_object(
+            await self._run_io(
+                self.s3_client.head_object,
                Bucket=self.bucket_name,
                Key=key,
            )
@@ -109,7 +160,8 @@ class S3StorageProvider(provider.StorageProvider):
    ):
        """Delete object from S3"""
        try:
-            self.s3_client.delete_object(
+            await self._run_io(
+                self.s3_client.delete_object,
                Bucket=self.bucket_name,
                Key=key,
            )
@@ -123,7 +175,8 @@ class S3StorageProvider(provider.StorageProvider):
    ) -> int:
        """Get object size from S3 without downloading it"""
        try:
-            response = self.s3_client.head_object(
+            response = await self._run_io(
+                self.s3_client.head_object,
                Bucket=self.bucket_name,
                Key=key,
            )
@@ -138,23 +191,23 @@ class S3StorageProvider(provider.StorageProvider):
    ):
        """Delete all objects with the given prefix (directory)"""
        try:
-            # Ensure dir_path ends with /
-            if not dir_path.endswith('/'):
-                dir_path = dir_path + '/'
-
-            # List all objects with the prefix
-            paginator = self.s3_client.get_paginator('list_objects_v2')
-            pages = paginator.paginate(Bucket=self.bucket_name, Prefix=dir_path)
-
-            # Delete all objects
-            for page in pages:
-                if 'Contents' in page:
-                    objects_to_delete = [{'Key': obj['Key']} for obj in page['Contents']]
-                    if objects_to_delete:
-                        self.s3_client.delete_objects(
-                            Bucket=self.bucket_name,
-                            Delete={'Objects': objects_to_delete},
-                        )
+            await self._run_io(self._delete_dir_recursive_sync, dir_path)
        except Exception as e:
            self.ap.logger.error(f'Failed to delete directory from S3: {e}')
            raise
+
+    def _delete_dir_recursive_sync(self, dir_path: str) -> None:
+        if not dir_path.endswith('/'):
+            dir_path = dir_path + '/'
+
+        paginator = self.s3_client.get_paginator('list_objects_v2')
+        pages = paginator.paginate(Bucket=self.bucket_name, Prefix=dir_path)
+        for page in pages:
+            if 'Contents' not in page:
+                continue
+            objects_to_delete = [{'Key': obj['Key']} for obj in page['Contents']]
+            if objects_to_delete:
+                self.s3_client.delete_objects(
+                    Bucket=self.bucket_name,
+                    Delete={'Objects': objects_to_delete},
+                )