Core no longer assembles a whole trace locally. Every workflow node is uploaded
the moment it completes as its own `execution_node` record, and each execution
closes with exactly one `execution_chain` record; all records of one execution
carry the same `event_id`, so Space groups them back into a single chain.
- every execution starts at an inbound event: event-less Pipeline lanes and
WebUI/API debug runs synthesize their virtual inbound event at the boundary
and are marked debug=true, so a Runner is never a trace origin
- sampling is decided once per event from a stable hash of the event id (the
previous `int(id[:8], 16)` raised on every `platform:`-prefixed id and so
dropped every successful inbound-event chain) and is reported on each record
as sample{mode,denominator}; the default mode is now `all`
- the local trace buffer, the 120s TTL, the per-process trace cap and the shared
open-node mirror are gone, which also removes the truncated-chain upload and
the re-buffered-closed-trace leak they caused
- a broken chain is always uploaded, and its reason falls back to the first
failed node so the chain explains itself
- flush keeps records until the sender accepts them and counts what it drops;
telemetry health replaces the unused send-task queue
- heartbeat reports bound Agent / plugin-processor / event-binding counts
- bound Pipeline and Agent resources (plugins, MCP servers, knowledge bases,
tools with their Host source, skills, models and the Runner schema selectors
that picked them) are reported on the chain record
- failure text is bounded and redacted instead of passing exception text through
- the dead per-query feature counters and their call sites are removed
Wire contract: schema 2 (`execution_node` / `execution_chain`).
The SDK binds one transfer root inode (mode 0700) and picks it per profile: a shared worker uses /tmp/lbp-rpc while a host without the profile env fell back to data/temp/lbp, so plugin icons and assets died with 'Invalid file transfer capability'. The host now pins the shared root, and a plugin absent from the resolved Workspace answers 404 instead of 500.
The connector stored only uuid/code/message, so the ops reporter stamped every entry with its own sampling time and a stale failure looked freshly broken on each report. Failures now carry failed_at and the payload publishes it.
The execution trace rewrite batches records and uploads them from
ExecutionCounters.flush(), replaced the per-trace direct send, dropped the
window counters, made the execution id the record identity and records the
inbound event stage after the routing stage. The ingress tests still asserted
the old mechanism, so they failed on every Python version.
Assert what the pipeline now emits instead: flush the buffered records and read
them back, require the trace registry to be empty afterwards, assert the
deterministic execution id, expect the route stage before the inbound stage,
and give the dispatch stubs the execution_id parameter the ingress passes.
The shared Cloud deployment runs one Plugin Runtime for every Workspace, but
nothing reported its health: the instance only exposed two aggregate numbers on
its own health probe, and the per-installation failure registry, the desired
state registry and the reconcile outcome lived in memory with no reader.
Add RuntimeOpsReporter, which every 30 seconds reads the Runtime ``/healthz``
probe over HTTP (derived from ``plugin.runtime_ws_url``), merges it with the
connector's desired-state, failure and reconcile summaries, and posts one
sample to the Space control plane on the existing authenticated direction.
Delivery is best-effort and bounded: a single in-flight request, a ten second
timeout, dropped samples counted in ``runtime_ops`` on the instance health
probe, and a failed probe recorded as a drop so the dashboard shows staleness
instead of grading a stale sample as healthy.
Cloud-only: a non-cloud deployment never enables the reporter, and the channel
is independent of product telemetry so ``space.disable_telemetry`` cannot blind
the ops dashboard.
Every host step now opens a stage scope, so observations recorded inside it (including cross-task plugin/RPC calls resolved through the execution registry) carry the owning step as parent; the pipeline lane records its own pipeline/run node so acks and replies issued outside the runner still hang off a node.
close_trace() called the coroutine TelemetryManager.start_send_task without
awaiting or scheduling it, so every payload closed on that path was silently
dropped and the runtime logged "coroutine ... was never awaited". The only
other caller awaits it correctly, so both call styles stay supported: call the
manager, schedule the returned coroutine on the running loop, and close it with
a debug log when no loop is available.
Regression case added in tests/unit_tests/telemetry/test_trace.py: it fails
before this change (payload never delivered) and passes after.
Execution telemetry already reported window counters. This adds one bounded,
content-free chain per inbound platform event, so a single event can be
followed from its source platform through routing and processing to the
platform API calls it caused.
- telemetry/trace.py: ContextVar trace identity with route and run scopes, and
a 32-stage bound per chain.
- telemetry/execution.py: keeps at most 64 chains for 120s, decides sampling
from the observed outcome, and sends one payload per chain keyed by
query_id = trace_id, so Space fetches a chain by primary identity. Window
counters are unchanged and remain the source of coverage statistics.
- botmgr / pipelinemgr / orchestrator: bind the trace at the ingress boundary,
scope routing identity per dispatch, and attach the run identity, so nested
lanes (event -> route -> pipeline/runner -> platform API) reuse one chain. A
run reached without an ingress (WebUI debug, service API) owns its own chain.
- space.execution_trace selects off | failures | sampled | all (default
sampled: failures, WebUI debug runs and every N-th success).
Chains carry only code-defined identifiers (adapter and runner types, route
identity as type:uuid, run id); never message content, tool arguments or
platform user identifiers.
scan_skill_directory is disabled for Cloud-managed sandboxes, but the
register_skill tool was still advertised to the model whenever a backend was
available, so every call failed closed without a usable explanation. Advertise
register_skill only when the runtime may scan sandbox directories, and keep
activate available in both modes.
The workspace file tools (read/write/edit/glob/grep) ran a script through
`python`, which a sandbox built from a host rootfs may not provide: those hosts
ship `python3` only, and a read-only /usr prevents creating a `python` shim
inside the jail. Every file tool then failed with `/bin/sh: 1: python: not
found`. Resolve the interpreter with `command -v python3 || command -v python`,
matching the interpreter the Box service already uses for its own scripts.
Cloud runtime refuses unscoped database access and workspace_operation_logs is row-level-secured on the bound Workspace. The background writer, the retention pass and the metadata reads all run outside a request lifetime, so each of them failed and the failure was swallowed by the best-effort handler: enabling tracing recorded nothing at all.
The route wrapper also traces after the handler released its scope, so level resolution has to carry the Workspace itself. Without it the level read back as 0 and the request was dropped before it was even queued.
The startup gate now enumerates the Workspaces through this instance's execution bindings, because a cross-Workspace metadata scan cannot see them under RLS, and it runs after the Workspace service exists. The writer starts with a clean context so it cannot inherit the scope of the request that first enqueued a trace, and a dropped write is reported instead of being logged only at debug.
The panel answered "who, when, what" poorly: the actual before → after diff
was hidden behind a click, the actor was a muted line, and a whole row went to
the route while duration and hash sat in a permanent right column. The data
already arrives with the page, so the panel now reads top-down in the order an
operator actually looks:
* who and when first, then the action and resource, then the diff spelled out
inline (long values collapse to their size so a pipeline config does not
render as two near-identical blobs), with route / method / status / duration
/ evidence hash moved into the expanded diagnostics line;
* the whole card is the expand toggle with a rotating chevron, which removes
the ambiguity of a small text link whose expanded state looked identical;
* records are bucketed into today / yesterday / dated groups so a long history
reads as a timeline;
* a "changes only" toggle (reusing the level filter) answers "who changed
something" in one click -- in the live log 489 of 499 rows are pure views,
so this is the difference between a haystack and the ten rows that matter.
Tracing fixes found while reading the live log:
* PUT /pipelines/<uuid>/extensions was classified as a plugin config change
because the generic ('/extensions') rule ran first; it now maps to a
pipeline-specific action;
* that handler recorded no diff at all, so "modified extension config" could
never say what changed. It now captures the previous bindings and reports the
before → after.
Cache: the integrity result is now keyed by (Workspace, listing filters) so
each view reuses its own verification, and a delete invalidates all views for
the Workspace. Fourteen tests cover the projection, the verifier, the cache
lifecycle and the failure modes.
The integrity scan selected whole rows, so a cold pass paged in the
changes/detail payloads and the client fingerprint for up to
MAX_INTEGRITY_SCAN_ROWS rows the verifier never reads. Project only the hash
columns, and add a lightweight verifier so the scan no longer builds a full
display dict per row.
Repair the read cache so it is a latency shield, not a correctness shortcut:
a result computed within INTEGRITY_CACHE_TTL_SECONDS is served from the
per-Workspace cache (opening, refreshing and paging all land inside that
window and pay nothing), while a cache miss re-verifies the whole window. An
incremental scan that skips previously verified ids can never see an edit to
an already-cached row -- exactly the tampering this feature exists to expose.
Verified against a live 414-row log: 50 content edits and 7 re-signed links
are all detected, including an edit to a row verified on a previous pass.
Also fix two correctness gaps and one maintenance bug:
- age-based prune deleted rows without invalidating the cached prefix;
- the boundary baseline is now read only when the history exceeds the scan
window, which a bounded scan makes the rare case;
- the operation-log retention block had drifted outside the per-binding loop
in the maintenance task, so only the last discovered Workspace was ever
pruned while the others grew unbounded.
Tests: scan projection, verifier parity, TTL cache reuse, cache-miss
re-verification, edit to an already-verified row, hash mismatch, chain break
and prune invalidation.
RockChinQ