feat(telemetry): trace one inbound event end to end

Execution telemetry already reported window counters. This adds one bounded,
content-free chain per inbound platform event, so a single event can be
followed from its source platform through routing and processing to the
platform API calls it caused.

- telemetry/trace.py: ContextVar trace identity with route and run scopes, and
  a 32-stage bound per chain.
- telemetry/execution.py: keeps at most 64 chains for 120s, decides sampling
  from the observed outcome, and sends one payload per chain keyed by
  query_id = trace_id, so Space fetches a chain by primary identity. Window
  counters are unchanged and remain the source of coverage statistics.
- botmgr / pipelinemgr / orchestrator: bind the trace at the ingress boundary,
  scope routing identity per dispatch, and attach the run identity, so nested
  lanes (event -> route -> pipeline/runner -> platform API) reuse one chain. A
  run reached without an ingress (WebUI debug, service API) owns its own chain.
- space.execution_trace selects off | failures | sampled | all (default
  sampled: failures, WebUI debug runs and every N-th success).

Chains carry only code-defined identifiers (adapter and runner types, route
identity as type:uuid, run id); never message content, tool arguments or
platform user identifiers.
This commit is contained in:
Hyu
2026-10-01 02:12:44 +08:00
parent ae516d2290
commit 0efc1fb1cc
9 changed files with 1082 additions and 38 deletions
@@ -37,6 +37,8 @@ from .run_journal import AgentRunJournal
from .session_registry import AgentRunSessionRegistry, get_session_registry
from .state_scope import build_state_context
from ...provider.tools.loaders import skill as skill_loader
from ...telemetry import trace as trace_mod
from ...telemetry.execution import close_trace as close_execution_trace
from ...telemetry.execution import record as record_execution
@@ -211,8 +213,14 @@ class AgentRunOrchestrator:
terminal_reason: str | None = None
terminal_usage: dict[str, typing.Any] | None = None
execution_outcome = 'unknown'
# A run reached without a platform ingress (WebUI debug, service API)
# owns its own chain; a run inside an ingress reuses that chain.
trace_binding: trace_mod.TraceBinding | None = None
run_token: typing.Any = None
try:
trace_binding = trace_mod.bind()
run_token = trace_mod.set_run(run_id)
await self.journal.create_run(
event=event,
binding=binding,
@@ -419,6 +427,9 @@ class AgentRunOrchestrator:
outcome=execution_outcome,
synthetic=event.source == 'webui',
)
trace_mod.reset_run(run_token)
if trace_binding is not None and trace_mod.unbind_root(trace_binding):
close_execution_trace(self.ap, trace_binding.state, 'runner_done')
binding_box = getattr(execution_query, '_box_binding', None)
if binding_box is not None and binding_box.run_id == run_id:
object.__delattr__(execution_query, '_box_binding')