xinyin025/LangBot
1
0
Fork 0
mirror of https://github.com/langbot-app/LangBot.git synced 2026-10-25 01:45:23 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat(telemetry)!: upload one record per execution node and one closure per chain

Browse Source 
Core no longer assembles a whole trace locally. Every workflow node is uploaded
the moment it completes as its own `execution_node` record, and each execution
closes with exactly one `execution_chain` record; all records of one execution
carry the same `event_id`, so Space groups them back into a single chain.

- every execution starts at an inbound event: event-less Pipeline lanes and
  WebUI/API debug runs synthesize their virtual inbound event at the boundary
  and are marked debug=true, so a Runner is never a trace origin
- sampling is decided once per event from a stable hash of the event id (the
  previous `int(id[:8], 16)` raised on every `platform:`-prefixed id and so
  dropped every successful inbound-event chain) and is reported on each record
  as sample{mode,denominator}; the default mode is now `all`
- the local trace buffer, the 120s TTL, the per-process trace cap and the shared
  open-node mirror are gone, which also removes the truncated-chain upload and
  the re-buffered-closed-trace leak they caused
- a broken chain is always uploaded, and its reason falls back to the first
  failed node so the chain explains itself
- flush keeps records until the sender accepts them and counts what it drops;
  telemetry health replaces the unused send-task queue
- heartbeat reports bound Agent / plugin-processor / event-binding counts
- bound Pipeline and Agent resources (plugins, MCP servers, knowledge bases,
  tools with their Host source, skills, models and the Runner schema selectors
  that picked them) are reported on the chain record
- failure text is bounded and redacted instead of passing exception text through
- the dead per-query feature counters and their call sites are removed

Wire contract: schema 2 (`execution_node` / `execution_chain`).
This commit is contained in:
Hyu
2026-10-07 09:41:41 +08:00
parent dcc7e17d77
commit 4482e8d876
24 changed files with 1532 additions and 1060 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/langbot/pkg/telemetry/execution.py
+333 -255
View File
@@ -1,18 +1,16 @@
"""Content-free execution records for the existing telemetry sender.
"""Point-wise execution records for the existing telemetry sender.
This module reports observations only. Coverage catalogs and acceptance rules
belong to Space.
One execution is reported as a chain of independent records that all carry the
same ``event_id``: one ``execution_node`` per workflow node, emitted the moment
that node completes, plus exactly one ``execution_chain`` emitted when the
execution closes. The sender never assembles a trace locally, so there is no
per-trace buffer, no TTL and no per-process trace cap; memory is bounded by the
outbound record buffer, the flush batch size and the small cross-task registry.
One complete execution is one record: the execution identity (Pipeline query,
inbound platform event or Agent run), its bounded ordered stages and its
terminal outcome. Nothing is aggregated or counted across executions — every
record carries the id of the execution it belongs to, so Space can fetch the
whole chain by primary identity. Records are buffered whole and flushed as
``{"records": [...]}`` on the configured cadence and on shutdown; an in-flight
trace is never flushed half-written.
Memory is bounded in stages per trace, buffered traces, buffered records, the
cross-task execution registry and flush batch size.
Sampling is decided exactly once per event, at ingress, from a stable hash of
the execution identity, so every record of one execution carries the same
decision. A chain that broke is always emitted even when sampling excluded it,
which is why a chain may arrive with zero or partial nodes.
"""
from __future__ import annotations
@@ -21,29 +19,48 @@ import asyncio
import contextlib
import contextvars
import json
import re
import time
import zlib
from datetime import datetime, timezone
from . import trace as trace_mod
from .identity import workspace_identity
from .resources import snapshot as resources_snapshot
from .resources import begin as resources_begin
from .resources import end as resources_end
from .trace import TraceState
MODES = frozenset({'pipeline', 'agent', 'event_processor', 'none'})
OUTCOMES = frozenset({'success', 'failed', 'cancelled', 'timeout', 'skipped', 'unknown'})
FAMILIES = frozenset({'platform_event', 'event_route', 'pipeline', 'runner', 'platform_api'})
ORIGINS = frozenset({'platform', 'webui', 'api'})
CLOSED_BY = frozenset({'event_done', 'pipeline_done', 'runner_done', 'timeout'})
CHAIN_FAILURES = frozenset({'failed', 'cancelled', 'timeout'})
# Trace bounds: stages per trace, traces buffered per process, trace lifetime.
MAX_TRACES = 64
TRACE_TTL_SECONDS = 120
SCHEMA = 2
TRACE_MODES = frozenset({'off', 'failures', 'sampled', 'all'})
DEFAULT_TRACE_MODE = 'sampled'
DEFAULT_TRACE_MODE = 'all'
DEFAULT_TRACE_SAMPLE = 20
# Record bounds: buffered records, one flush batch, flush cadence, registry.
# Sampling bounds: outbound record buffer, one flush batch, flush cadence.
MAX_BUFFERED_RECORDS = 512
MAX_RECORDS_PER_FLUSH = 128
MAX_FLUSH_BYTES = 200 * 1024
DEFAULT_FLUSH_SECONDS = 180
MAX_FLUSH_ATTEMPTS = 3
FLUSH_RETRY_BACKOFF_SECONDS = 0.2
# Cross-task registry: execution id -> owning execution, bounded and time-limited.
MAX_REGISTRY_KEYS = 256
REGISTRY_TTL_SECONDS = 300
# Failure details are operator-facing: keep the failure class and a short
# machine-ish prefix, never the free text a user or a model produced.
ERROR_LIMIT = 200
ERROR_PREFIX_CHARS = 120
ERROR_PREFIX_TOKENS = 12
_ERROR_CLASS = re.compile(r'[A-Za-z_][\w.]*')
# Set by cross-task call sites (plugin/RPC actions) that run without the
# ingress context but know which execution they belong to.
@@ -68,16 +85,36 @@ def current_execution_id() -> str:
return _execution_id.get()
def _bounded_error(value, limit: int = 400) -> str:
"""Bound and de-fang one failure detail before it leaves the process.
def _short_prefix(text: str) -> str:
"""Keep at most a short machine-ish prefix of one failure message."""
if len(text) > ERROR_PREFIX_CHARS:
text = text[:ERROR_PREFIX_CHARS]
parts = text.split(' ')
if len(parts) > ERROR_PREFIX_TOKENS:
text = ' '.join(parts[:ERROR_PREFIX_TOKENS])
return text
Error text is operator-facing: keep it single-line, printable and short so a
noisy failure cannot smuggle control characters or bloat the payload.
def _bounded_error(value, limit: int = ERROR_LIMIT) -> str:
"""Bound and redact one failure detail before it leaves the process.
The failure class and a short machine-ish prefix survive; control characters
are stripped, whitespace is collapsed and the result is capped, so a noisy
failure cannot smuggle user or model content into the payload.
"""
if value is None:
return ''
text = ''.join(ch if ch.isprintable() else ' ' for ch in str(value))
text = ' '.join(text.split())
if not text:
return ''
head, separator, rest = text.partition(':')
head = head.strip()
if separator and len(head) <= 80 and _ERROR_CLASS.fullmatch(head):
prefix = _short_prefix(rest.strip())
text = f'{head}: {prefix}' if prefix else head
else:
text = _short_prefix(text)
return text[:limit]
@@ -91,18 +128,21 @@ def _duration_ms(started_at: str, ended_at: str) -> int:
class ExecutionCounters:
"""Point-wise execution reporting: one record per node, one per closure."""
def __init__(self, manager):
self.manager = manager
self.records: list[dict] = []
self.task: asyncio.Task | None = None
# Exposed health: records dropped or accepted by the sender.
self.dropped = 0
self.traces: dict[str, TraceState] = {}
self.trace_deadlines: dict[str, float] = {}
# Execution identity -> in-flight trace, for stages recorded by tasks
self.sent = 0
# Execution identity -> owning execution, for nodes recorded by tasks
# that no longer share the ingress context (plugin/RPC platform calls).
# It resolves identity and the sampling decision only; node payloads are
# never accumulated here.
self.registry: dict[str, TraceState] = {}
self.registry_deadlines: dict[str, float] = {}
self.dropped_traces = 0
# ------------------------------------------------------------------ config
@@ -130,6 +170,63 @@ class ExecutionCounters:
seconds = DEFAULT_FLUSH_SECONDS
return seconds if 1 <= seconds <= 86400 else DEFAULT_FLUSH_SECONDS
def health(self) -> dict:
"""Real telemetry health for the instance liveness snapshot."""
return {
'records_buffered': len(self.records),
'records_sent': self.sent,
'records_dropped': self.dropped,
'executions_in_flight': len({id(state) for state in self.registry.values()}),
}
# ----------------------------------------------------------------- sampling
def _sampling_decision(self, event_id: str) -> tuple[str, int, bool]:
"""Decide once, deterministically, whether this event is reported.
The identity is hashed instead of being parsed, so every id shape
(``platform:...``, uuids, debug ids) samples the same way.
"""
mode = self.trace_mode()
if mode == 'sampled':
denominator = self.trace_sample()
try:
sampled_in = zlib.crc32(str(event_id).encode('utf-8')) % denominator == 0
except Exception:
sampled_in = False
return mode, denominator, sampled_in
# 'off' reports nothing, 'failures' keeps nodes out but still reports a
# broken chain, 'all' reports everything; denominator is 1 for all three.
return mode, 1, mode == 'all'
def configure(
self,
state: TraceState,
context=None,
*,
debug: bool = False,
origin: str = 'platform',
identity=None,
) -> None:
"""Bind the execution identity, the sampling decision and the flags."""
if state.configured:
return
state.configured = True
state.debug = bool(debug)
state.origin = origin if origin in ORIGINS else 'platform'
mode, denominator, sampled_in = self._sampling_decision(state.event_id)
state.mode = mode
state.denominator = denominator
# Debug chains are always emitted (unless reporting is off entirely).
state.emit_nodes = mode != 'off' and (state.debug or sampled_in)
if identity is not None:
state.identity = dict(identity)
elif context is not None:
try:
state.identity = workspace_identity(context)
except Exception:
pass
# --------------------------------------------------------------- recording
def record(
@@ -148,102 +245,96 @@ class ExecutionCounters:
node: str = '',
):
try:
cfg = self.manager.telemetry_config
if not cfg or cfg.get('disable_telemetry', False) or not cfg.get('url'):
if not self._enabled():
return
if family not in {'platform_event', 'event_route', 'pipeline', 'runner', 'platform_api'}:
return
if mode not in MODES or outcome not in OUTCOMES:
if family not in FAMILIES or mode not in MODES or outcome not in OUTCOMES:
return
# Only code-defined identifiers are accepted; never pass user values.
if any(not isinstance(v, str) or len(v) > 160 for v in (operation, adapter, runner, node)):
return
state = trace_mod.current()
if state is None:
# Cross-task work has no inherited trace; resolve the owning
# execution through the registry instead.
# Cross-task work has no inherited context; resolve the owning
# execution through the registry. It has no open step here, so
# its node becomes a root of that execution instead of being
# mirrored from another task's stack.
lookup = str(execution_id or '').strip() or current_execution_id()
if lookup:
state = self.registry.get(lookup)
if state is None:
state = self._resolve(lookup)
if state is None or state.closed:
# Every observation belongs to exactly one execution chain.
return
self.configure(state)
try:
identity = workspace_identity(context)
except Exception:
# A cross-task caller may only carry the Workspace id; reuse the
# identity already bound to the owning trace.
# identity already bound to the owning execution.
identity = state.identity or {}
if not identity.get('instance_id') or not identity.get('workspace_uuid'):
return
self._record_trace_stage(
state,
identity=identity,
family=family,
operation=operation,
mode=mode,
adapter=adapter,
runner=runner,
outcome=outcome,
synthetic=synthetic,
error=_bounded_error(error),
node=node,
state.identity = identity
ended_at = datetime.now(timezone.utc).isoformat()
node_id, seq, started_at, parent = trace_mod.resolve_node(state, str(node or ''), ended_at)
root = parent == '' and not state.root_seen
if root:
state.root_seen = True
bounded_error = _bounded_error(error)
state.note_node(outcome=outcome, adapter=adapter, runner=runner, family=family, error=bounded_error)
if not state.emit_nodes:
# Sampling excluded this execution: the node is never buffered,
# only counted so the terminal chain can explain the gap.
state.dropped_nodes += 1
return
self._enqueue_record(
{
'event_type': 'execution_node',
'schema': SCHEMA,
**self._common_fields(state, ended_at),
'node_id': node_id,
'parent_node_id': parent,
'root': root,
'seq': seq,
'family': family,
'operation': operation,
'mode': mode,
'adapter': adapter,
'runner': runner,
'outcome': outcome,
'error': bounded_error,
'route_ref': trace_mod.current_route(),
'run_id': trace_mod.current_run(),
'synthetic': bool(synthetic),
'started_at': started_at,
'ended_at': ended_at,
}
)
self._ensure_loop()
except Exception:
# Observability must never change execution behavior.
return
def _record_trace_stage(
self,
state: TraceState,
*,
identity,
family,
operation,
mode,
adapter,
runner,
outcome,
synthetic,
error: str = '',
node: str = '',
) -> None:
if state.abandoned:
return
if trace_mod.current() is state:
# Recorded by the task that owns the step: its step stack is exact.
# A step's own record sits under its enclosing step, never itself.
parent = trace_mod.current_parent()
if node and parent == node:
parent = trace_mod.enclosing_parent()
else:
# A cross-task observation (plugin/RPC) cannot see the step
# contextvar; attach it to the innermost step the owning task has
# open on the trace.
parent = state.open_nodes[-1] if state.open_nodes else ''
if state.trace_id not in self.traces:
# Never buffer traces this configuration would discard anyway.
if self.trace_mode() == 'off' or len(self.traces) >= MAX_TRACES:
self.dropped += 1
state.abandoned = True
self._unregister_state(state)
return
self.traces[state.trace_id] = state
self.trace_deadlines[state.trace_id] = time.monotonic() + TRACE_TTL_SECONDS
state.identity = identity
state.append(
family=family,
operation=operation,
mode=mode,
adapter=adapter,
runner=runner,
outcome=outcome,
synthetic=bool(synthetic),
error=error,
node=node,
parent=parent,
)
def _common_fields(self, state: TraceState, timestamp: str) -> dict:
identity = state.identity
from ..utils import constants
return {
'event_id': state.event_id,
'instance_id': identity['instance_id'],
'workspace_uuid': identity['workspace_uuid'],
'version': constants.semantic_version,
'edition': constants.edition,
'timestamp': timestamp,
'debug': bool(state.debug),
'sample': {'mode': state.mode, 'denominator': state.denominator},
}
def _enabled(self) -> bool:
try:
cfg = self.manager.telemetry_config
except Exception:
return False
return bool(cfg) and not cfg.get('disable_telemetry', False) and bool(cfg.get('url'))
def _ensure_loop(self) -> None:
if self.task is None or self.task.done():
@@ -252,20 +343,27 @@ class ExecutionCounters:
# --------------------------------------------------------------- registry
def register_trace(self, state: TraceState, execution_id) -> None:
"""Alias one more execution identity onto an in-flight trace."""
"""Alias one more execution identity onto an in-flight execution."""
try:
exec_id = str(execution_id or '').strip()
if not exec_id or state.abandoned:
if not exec_id:
return
self.configure(state)
now = time.monotonic()
self._sweep_registry(now)
if exec_id not in self.registry and len(self.registry) >= MAX_REGISTRY_KEYS:
oldest = next(iter(self.registry))
self.registry.pop(oldest, None)
self.registry_deadlines.pop(oldest, None)
self.registry[exec_id] = state
self.registry_deadlines[exec_id] = time.monotonic() + TRACE_TTL_SECONDS
self.registry_deadlines[exec_id] = now + REGISTRY_TTL_SECONDS
except Exception:
return
def _resolve(self, execution_id: str) -> TraceState | None:
self._sweep_registry(time.monotonic())
return self.registry.get(execution_id)
def _unregister_state(self, state: TraceState) -> None:
for exec_id in [key for key, value in self.registry.items() if value is state]:
self.registry.pop(exec_id, None)
@@ -277,26 +375,68 @@ class ExecutionCounters:
self.registry.pop(exec_id, None)
self.registry_deadlines.pop(exec_id, None)
# ------------------------------------------------------------------ traces
# ------------------------------------------------------------------ chains
def close_trace(self, state: TraceState, reason: str = 'event_done', error: str = '') -> None:
"""Buffer one complete execution record when it matches the sample."""
"""Emit the one terminal ``execution_chain`` record of this execution."""
try:
if state.closed:
return
state.closed = True
registered = self.traces.pop(state.trace_id, None) is not None
self.trace_deadlines.pop(state.trace_id, None)
self._unregister_state(state)
if not registered and not state.failure_reason:
if not self._enabled():
# Nothing can be uploaded; never grow the buffer for a record
# the sender would refuse to send.
return
if (not state.stages and not state.failure_reason) or not self._trace_emitted(state):
self.configure(state)
if not self._chain_emitted(state):
return
record = self._build_record(state, reason, error)
if record is not None:
self._enqueue_record(record)
self._enqueue_record(self._build_chain_record(state, reason, error))
except Exception:
return
finally:
# The snapshot was read while building the chain record; stop
# collecting so a late sibling task cannot leak into the next chain.
resources_end()
def _chain_emitted(self, state: TraceState) -> bool:
if state.mode == 'off':
return False
if state.emit_nodes or state.debug:
return True
# A chain that broke is always uploaded, even when sampling excluded it.
return bool(state.failure_reason) or state.outcome() in CHAIN_FAILURES
def _build_chain_record(self, state: TraceState, reason: str, error: str = '') -> dict:
ended_at = datetime.now(timezone.utc).isoformat()
record = {
'event_type': 'execution_chain',
'schema': SCHEMA,
**self._common_fields(state, ended_at),
'closed_by': reason if reason in CLOSED_BY else 'event_done',
'outcome': state.outcome(),
# First failure detail in this chain, so Space can show why it broke.
'error': _bounded_error(error)
or _bounded_error(state.failure_reason)
or _bounded_error(state.first_error)
or '',
'duration_ms': _duration_ms(state.started_at, ended_at),
'stage_count': state.stage_count,
'dropped_nodes': state.dropped_nodes,
'origin': state.origin,
'model_name': state.model_name or '',
'adapter': state.adapter or state.node_adapter,
'runner': state.runner or state.node_runner,
'runner_category': state.runner_category or '',
'pipeline_plugins': state.pipeline_plugins,
}
# Bound resources of this execution, when any lane reported them. The
# key is omitted entirely when nothing was noted, so the payload stays
# byte-identical for executions with no bindings.
resources = resources_snapshot()
if resources:
record['resources'] = resources
return record
def _enqueue_record(self, record: dict) -> None:
if len(self.records) >= MAX_BUFFERED_RECORDS:
@@ -305,137 +445,52 @@ class ExecutionCounters:
self.records.append(record)
self._ensure_loop()
def _trace_emitted(self, state: TraceState) -> bool:
mode = self.trace_mode()
if mode == 'off':
return False
# A chain that failed must always be uploaded, even when no stage was
# recorded before it broke: the reason is the whole point of the trace.
if state.failure_reason:
return True
if mode == 'all':
return True
# A failed, cancelled or timed-out stage is always worth keeping.
if any(stage['outcome'] not in ('success', 'skipped') for stage in state.stages):
return True
# WebUI debug runs are deliberately traced; they stay flagged synthetic.
if state.synthetic:
return True
if mode == 'failures':
return False
try:
return int(state.trace_id[:8], 16) % self.trace_sample() == 0
except ValueError:
return False
def _build_record(self, state: TraceState, reason: str, error: str = '') -> dict | None:
from ..utils import constants
identity = state.identity
if not identity.get('instance_id') or not identity.get('workspace_uuid'):
return None
ended_at = datetime.now(timezone.utc).isoformat()
stages = state.stages
observations = []
for stage in stages:
observation = {
'seq': stage['seq'],
'family': stage['family'],
'operation': stage['operation'],
'mode': stage['mode'],
'adapter': stage['adapter'],
'runner': stage['runner'],
'outcome': stage['outcome'],
'synthetic': stage['synthetic'],
'error': stage['error'],
'route_ref': stage['route_ref'],
'run_id': stage['run_id'],
'first_seen': stage['first_seen'],
'last_seen': stage['last_seen'],
'trace_id': state.trace_id,
}
# Workflow position is optional: records outside any step omit both
# keys entirely, so their payloads stay byte-identical.
if stage.get('node'):
observation['node'] = stage['node']
if stage.get('parent'):
observation['parent'] = stage['parent']
observations.append(observation)
return {
'event_type': 'feature_execution',
# The execution identity: one record == one chain == one Space row.
'query_id': state.trace_id,
'instance_id': identity['instance_id'],
'workspace_uuid': identity['workspace_uuid'],
'version': constants.semantic_version,
'edition': constants.edition,
'timestamp': ended_at,
'trusted': True,
# First failure detail in this chain, so Space can show why it broke.
'error': _bounded_error(error)
or state.failure_reason
or next((stage['error'] for stage in stages if stage['error']), ''),
# Legacy-compatible columns Space already maps; best effort.
'duration_ms': _duration_ms(state.started_at, ended_at),
'model_name': state.model_name or '',
'adapter': state.adapter or next((stage['adapter'] for stage in stages if stage['adapter']), ''),
'runner': state.runner or next((stage['runner'] for stage in stages if stage['runner']), ''),
'runner_category': state.runner_category or '',
'pipeline_plugins': state.pipeline_plugins,
'features': {
'schema': 1,
'observations': observations,
'trace': {
'closed_by': reason,
'started_at': state.started_at,
'ended_at': ended_at,
'route_ref': next((stage['route_ref'] for stage in stages if stage['route_ref']), ''),
'run_id': next((stage['run_id'] for stage in stages if stage['run_id']), ''),
'outcome': state.outcome(),
'dropped_stages': state.dropped_stages,
},
},
}
def _sweep_traces(self, now: float) -> None:
for trace_id in [key for key, deadline in self.trace_deadlines.items() if deadline <= now]:
state = self.traces.get(trace_id)
if state is not None:
self.close_trace(state, 'ttl')
# ----------------------------------------------------------------- flushing
async def _loop(self):
interval = self.flush_seconds()
last_flush = time.monotonic()
while self.records or self.traces or self.registry:
# Traces expire on a short TTL; records keep their own cadence.
await asyncio.sleep(1 if (self.traces or self.registry) else interval)
now = time.monotonic()
self._sweep_traces(now)
self._sweep_registry(now)
if self.records and now - last_flush >= interval:
last_flush = now
try:
# The outbound loop exists only to flush the record buffer.
while self.records:
await asyncio.sleep(interval)
await self.flush()
self.task = None
finally:
self.task = None
async def flush(self):
async def flush(self) -> bool:
"""Upload the buffered prefix; records stay buffered until accepted."""
if not self.records:
return
return True
batch: list[dict] = []
size = 0
while self.records and len(batch) < MAX_RECORDS_PER_FLUSH:
encoded = len(json.dumps(self.records[0], ensure_ascii=False, default=str))
for record in self.records:
if len(batch) >= MAX_RECORDS_PER_FLUSH:
break
encoded = len(json.dumps(record, ensure_ascii=False, default=str))
if batch and size + encoded > MAX_FLUSH_BYTES:
break
batch.append(self.records.pop(0))
batch.append(record)
size += encoded
if not batch:
return
return True
payload = {'records': batch}
if not await self.manager.send(payload):
await asyncio.sleep(1)
await self.manager.send(payload)
for attempt in range(MAX_FLUSH_ATTEMPTS):
try:
if await self.manager.send(payload):
# Peek-then-send: nothing is removed until the sender
# accepted the exact batch it was given.
del self.records[: len(batch)]
self.sent += len(batch)
return True
except asyncio.CancelledError:
raise
except Exception:
pass
if attempt + 1 < MAX_FLUSH_ATTEMPTS:
await asyncio.sleep(FLUSH_RETRY_BACKOFF_SECONDS * (2**attempt))
# The batch stays buffered for the next cycle; a permanently unaccepted
# record is dropped by the buffer bound above and counted there.
return False
async def shutdown(self):
if self.task is not None:
@@ -451,12 +506,8 @@ class ExecutionCounters:
break
except (Exception, asyncio.CancelledError):
pass
self.dropped += len(self.records)
self.records.clear()
# Executions still in flight cannot be completed during shutdown.
self.dropped += len(self.traces)
self.dropped_traces += len(self.traces)
self.traces.clear()
self.trace_deadlines.clear()
self.registry.clear()
self.registry_deadlines.clear()
@@ -477,7 +528,7 @@ def record(ap, context, **observation):
def close_trace(ap, state, reason: str = 'event_done', error: str = '') -> None:
"""Best-effort trace close usable with optional telemetry and test doubles."""
"""Best-effort chain close usable with optional telemetry and test doubles."""
try:
counters = _execution_counters(ap)
if counters is not None:
@@ -487,8 +538,10 @@ def close_trace(ap, state, reason: str = 'event_done', error: str = '') -> None:
def bind_trace(ap, execution_id: str | None = None) -> trace_mod.TraceBinding:
"""Bind an execution trace and alias its identity for cross-task stages."""
"""Bind an execution and alias its identity for cross-task nodes."""
binding = trace_mod.bind(execution_id)
if binding.created:
resources_begin()
if execution_id:
try:
counters = _execution_counters(ap)
@@ -500,36 +553,61 @@ def bind_trace(ap, execution_id: str | None = None) -> trace_mod.TraceBinding:
@contextlib.contextmanager
def ingress(ap, reason: str = 'event_done', context=None, execution_id: str | None = None):
"""Trace one execution boundary; only the owner of the trace closes it.
def ingress(
ap,
reason: str = 'event_done',
context=None,
execution_id: str | None = None,
*,
debug: bool = False,
origin: str = 'platform',
synthetic_event: str = '',
):
"""Open one execution boundary; only the owner of the chain closes it.
Nested boundaries (an event routed into a Pipeline, a Runner invoked from a
dispatch) reuse the in-flight trace instead of starting a second one, and
only alias their own execution id onto it.
dispatch) reuse the in-flight chain instead of starting a second one, and
only alias their own execution id onto it. ``debug``/``origin`` apply when
this boundary starts the chain: a WebUI/API run without a real platform
event synthesizes its virtual inbound event here, so the chain has a real
identity and a node for the event it stands for.
"""
binding = trace_mod.bind(execution_id)
if binding.created and context is not None:
try:
binding.state.identity = workspace_identity(context)
except Exception:
pass
if execution_id and not binding.state.abandoned:
try:
counters = _execution_counters(ap)
if counters is not None:
counters.register_trace(binding.state, execution_id)
except Exception:
pass
state = binding.state
if binding.created:
resources_begin()
counters = _execution_counters(ap)
if counters is not None:
if binding.created:
counters.configure(state, context, debug=debug, origin=origin)
if execution_id:
counters.register_trace(state, execution_id)
scope = None
if binding.created and synthetic_event:
scope = trace_mod.stage_scope()
node = scope.__enter__()
record(
ap,
context,
family='platform_event',
operation=str(synthetic_event)[:160],
outcome='success',
synthetic=True,
node=node,
)
failure = ''
try:
yield binding
except BaseException as exc:
# The boundary itself failed: keep the reason on the trace so the uploaded
# payload explains the failure instead of reporting a bare outcome.
# The boundary itself failed: keep the reason on the chain so the
# uploaded payload explains the failure instead of reporting a bare
# outcome, even when sampling had excluded this execution.
if binding.created:
failure = f'{type(exc).__name__}: {exc}'
binding.state.mark_failure(failure)
state.mark_failure(failure)
raise
finally:
if scope is not None:
scope.__exit__(None, None, None)
if trace_mod.unbind_root(binding):
close_trace(ap, binding.state, reason, failure)
close_trace(ap, state, reason, failure)