LangBot/plugins/misc/WebSearch/SHARED_RUNTIME.md

# Shared runtime source candidate (SDK >=0.7.4)

This source declares `shared-runtime-v1` and `stateless-v1`. It is not certified
and makes no claim of live vendor acceptance. Certification additionally requires
independent review, signer approval, and two-Workspace invocation acceptance.

- `spec.config` is empty; the tool derives everything from its arguments. No
  configuration is read, cached, or retained on the component object.
- The whole `mux.process` call runs in a worker thread, so the blocking
  fetch/parse cannot stall the shared event loop. At most
  `MAX_CONCURRENT_FETCHES` fetches are in flight per event loop, a call waits at
  most `FETCH_SLOT_WAIT_SECONDS` for a free slot, and cancelling the invocation
  sets a stop flag that ends the I/O. The slot has a single owner: the caller
  releases it if the executor submission itself fails, and the worker thread
  releases it in its own `finally` once it has really exited. An
  await-cancellation cannot cancel the queued executor future (it is awaited
  under `asyncio.shield`), so a worker that has not started yet still runs and
  releases the slot exactly once; a worker that is still running is never
  released early.
- No binding-keyed caches or detached tasks exist, so
  `on_installation_revoked()` has nothing to release.

## Site adapters

`components/tools/sites/model.py` keeps the module-level `__site_adapters__`
registry. It holds only artifact-level data registered by the `@site(...)`
decorator at import time: each entry is a URL regex plus a class reference. It
contains no tenant, Workspace, or installation data, which the stateless
component contract permits for a process-level cache.

All adapters remain synchronous — they fetch through
`SiteAdapterBase.get_html`, which delegates to `components/tools/sites/safe_fetch.py`.
That is acceptable here because the entire `mux.process` adapter dispatch
(registry lookup, fetch, and parse) is executed in a worker thread by the tool;
no adapter code runs on the event loop. The adapters:

- `SiteAdapterBase` — generic fallback used for every URL in this revision.
- `GithubRepoSiteAdapter`, `GithubUserSiteAdapter` — defined under `sites/github/`
  with `@site(...)`, but that package is not imported by the current registry, so
  they are not registered at runtime (unchanged pre-existing behaviour).

## Outbound target and size policy (`safe_fetch.py`)

Fetching is not a bare `requests.get` any more; the module enforces the
certification requirements in the plugin request layer:

- only `http`/`https` URLs are accepted;
- the host is resolved by the plugin, every resolved address must be globally
  routable (loopback, link-local, private, multicast, reserved, unspecified,
  NAT64/IPv4-mapped forms are refused), and the connection is made to the
  validated IP so a later DNS answer cannot rebind it;
- redirects are followed manually, with the same validation applied to every hop;
- the body is read in bounded chunks: a declared or streamed body above
  `MAX_RESPONSE_BYTES`, and a body that inflates beyond it, abort the fetch
  without being buffered; the HTML handed to BeautifulSoup is capped at
  `MAX_PARSE_CHARS` and the extracted title/brief lengths are clamped;
- the *encoded* bytes read are bounded too: a cumulative `MAX_RESPONSE_BYTES +
  64 KiB` budget stops a chunked or length-less peer that keeps streaming
  bytes after the first compressed member, and the decoder tracks the zlib
  stream's end explicitly — exactly one gzip/deflate member is supported, bytes
  after it raise `TrailingDataError` instead of sitting in `unused_data`, an
  EOF'd decompressor is never fed again, and a stream that stops before its end
  marker raises `IncompleteCompressedStreamError` so gzip CRC32 / zlib Adler-32
  is validated before any body becomes a document;
- the whole operation has an absolute deadline plus a cooperative stop flag that
  cover DNS, connect, the TLS handshake, status/header parsing and the body: a
  watchdog thread shuts the socket down when either fires, because a socket
  timeout only bounds the idle time between two reads. The socket is closed on
  every exit path;
- every deadline-bounded DNS lookup runs on a fixed resolver worker pool with a
  fixed in-flight budget (`MAX_DNS_WORKERS` threads, `MAX_DNS_OUTSTANDING`
  lookups running + queued): a lookup whose waiter timed out or was cancelled
  keeps its budget slot until `getaddrinfo` really returns, so it can never
  free budget early or accumulate off the pool, and once the budget is
  exhausted a further lookup is refused with `ResolverBusyError` instead of
  starting another thread or queueing without limit;
- an aborted, timed-out or truncated response is never returned as a document:
  every exit path re-checks the stop flag and the deadline (a shut-down socket
  can read as a clean EOF), and a body shorter than its declared
  `Content-Length` fails with `TruncatedResponseError` instead of being parsed;
- the socket is opened directly, so no proxy environment variables or netrc
  credentials are consulted, and TLS keeps the default verification context;
- userinfo, query string and fragment are stripped from every URL before it
  reaches an error result, an exception (including chained causes) or a log
  line.

## Limits

- A DNS lookup that the resolver never answers cannot be killed, so the fetch
  returns at its deadline while the lookup is still running. It runs inside the
  fixed resolver pool above and holds its budget slot until the resolver really
  returns, so repeated deadline/cancel timeouts fill the pool's fixed budget
  instead of growing the process thread count; further lookups are then refused
  with `ResolverBusyError` until a slot frees. Lookups the resolver does answer
  reuse the same bounded workers (an idle pool is at most `MAX_DNS_WORKERS`
  threads).
- A cancelled invocation still cannot un-do I/O that has already been issued,
  but the stop flag and the absolute deadline shut the socket down, so an
  abandoned worker ends at the next blocking-socket return (bounded by the
  watchdog poll interval) instead of waiting for the peer to finish.
- Fetch concurrency is bounded by the plugin's own slot pool, so a tenant cannot
  occupy the default executor without limit.
- Tests drive the real fetch path against a local HTTP server (target refusals,
  redirect re-validation, DNS pinning, TLS context, byte/parse caps, deadline,
  cancellation, a peer that never completes its response headers, the slot
  ownership cases and the redaction of error results/logs); they also prove the
  resolver budget is independent of the waiter (a timed-out lookup does not free
  it, a further lookup is refused while it is exhausted, it comes back only when
  the lookup really ends, and the pool stays within its fixed worker/queue caps
  under repeated timeouts). They do not fetch the public internet.