Intermediate9 min

Logging, Metrics & Alerting

Building production dashboards for AI agents: structured logging, custom metrics (latency, cost, completion rate), and alerting on anomalies.

Quick Reference

→Structured logging: emit JSON logs with correlation_id, user_id, agent_version, node_name, and duration for every graph step
→Key agent metrics: p50/p95 latency per node, tokens per conversation, cost per conversation, task completion rate, tool error rate
→Use metric cardinality wisely: label by agent_version and node_name, not by user_id or conversation_id (too many labels)
→Alert on rate-of-change: a sudden spike in tool errors or token usage per request often signals a prompt or model regression
→Build a single-pane dashboard showing agent health: request rate, error rate, latency distribution, and cost trend

Structured Logging for Agents

JSON logs, not print statements

Emit structured JSON logs with correlation_id, user_id, node_name, and duration at every graph step. Structured logs are queryable in log aggregation tools (CloudWatch, Datadog Logs, Loki). Unstructured logs are not.

Structured logging with structlog at every graph node boundary

Key Agent Metrics

Metric	What It Tells You	Collection Method
P50/P95 latency per node	Which nodes are bottlenecks	Histogram from node timing
Tokens per conversation	How much context each conversation consumes	Sum LLM token usage per thread
Cost per conversation	Dollar spend per user interaction	tokens × price per model
Task completion rate	% of conversations that reach a successful end state	Counter at terminal nodes
Tool error rate	% of tool calls that fail or timeout	Counter on tool exceptions
Retry count	How often the agent loops or retries	Counter on graph cycles

Building Agent Dashboards

Emit Prometheus metrics from LangGraph nodes

Sign in to read this article

This is a premium article. Sign in with your Google account to continue.