Observability: Tracing

Document ID: OBS-003
File Path: docs/14-observability/tracing.md
Version: 1.0.0
Status: Draft
Owner: Platform Operations Team
Last Updated: 2026-06-27

1. Purpose

This document defines distributed tracing across the Wovyr AI Platform — how a single request is traced end to end through services, and how traces correlate with logs and metrics.

2. Standard

Tracing uses OpenTelemetry with W3C Trace Context propagation. Every service is instrumented to create and propagate spans, exporting via OTLP to a tracing backend (Tempo/Jaeger).

3. End-to-End Trace

A user request produces one trace spanning every hop:

trace: agent.run
 ├─ api-gateway: authn/authz
 ├─ agent-runtime: plan
 │   ├─ memory-engine: retrieve  (vector + rank)
 │   ├─ llm-gateway: chat        (route → provider)
 │   └─ tool-runtime: execute    (sandbox → tool)
 └─ agent-runtime: respond

This makes “why was this run slow/expensive?” answerable by inspecting span durations and attributes.

4. Propagation

Client → API Gateway   (generates trace_id + request_id)
       → service A → service B → datastores

Context flows via headers on REST/gRPC and metadata on Event Bus messages, so async work (workflow steps, cost events) joins the same trace where applicable.

5. Span Conventions

Attribute	Example
`wovyr.tenant`	`acme`
`wovyr.principal`	`agent:order-assistant`
`wovyr.resource.id`	`run_01H...`
`wovyr.subsystem`	`llm-gateway`
Semantic conventions	`http.`, `rpc.`, `db.*`

Sensitive values are never placed in span attributes (same masking rules as logging).

6. Correlation Across Pillars

trace_id appears in every log line.
Metrics histograms carry trace exemplars.
The dashboard links a metric/alert → trace → logs using the shared IDs.

7. Domain Spans

Subsystems emit meaningful spans, e.g.:

LLM Gateway: routing decision, provider call, failover hops (routing observability)
Tool Runtime: dispatch → authorize → schedule → sandbox → execute (tracing)
Memory Engine: embed → search → rank → compress

8. Sampling

Head-based sampling by default (configurable rate) to bound volume.
Tail-based sampling can retain all error/slow traces regardless of rate.
Errors and high-latency requests are always sampled.

9. Cost Attribution via Traces

Because model and tool calls are spans with cost attributes, a trace shows the cost breakdown of a single request, complementing aggregate cost metrics.

10. Dependencies

12. Revision History

Version	Date	Description
1.0.0	2026-06-27	Initial Tracing specification