LangSmith Automation Rules
Event-driven triggers for production traces — filter by error, latency, metadata, or feedback scores, and route matches to annotation queues, datasets, webhooks, or online evaluators. Everything is configured in the LangSmith UI.
Quick Reference
- →Automation rules are created in the LangSmith UI — the Python SDK has no create_rule() method
- →Filter operators: eq, neq, gt, lt, gte, lte, has, search, in, and, or — no exists() operator
- →Latency filters use string duration format: gt(latency, "10s") not gt(latency, 10000)
- →Metadata and feedback use key/value pattern: and(eq(metadata_key, "env"), eq(metadata_value, "prod"))
- →Six action types: annotation queue, dataset, webhook, online evaluator, custom code evaluator, alert
- →Execution order is fixed: queue → dataset → webhook → online eval → code eval → alert
- →Rules run asynchronously — they never add latency to your agent's hot path
What Automation Rules Solve
You have production traces arriving continuously. Without automation, acting on them is manual: engineers periodically scan the trace list for errors, copy interesting examples into eval datasets by hand, and watch dashboards for latency spikes. None of this scales past a few hundred traces per day. Automation rules close the loop — they evaluate every incoming trace against a filter and fire an action when matched. Zero polling, zero manual selection.
As of 2026, automation rules are created and managed in the LangSmith web UI under Automations. The Python SDK does not expose a create_rule() method — any code you find with client.create_rule() will fail at runtime with AttributeError. The examples in this article use the correct LangSmith filter syntax, which you enter directly in the UI.
| Problem | Manual Approach | Automation Rule Approach |
|---|---|---|
| Building eval datasets | Engineer manually selects traces weekly | Rule samples 10% of successful traces into a dataset continuously |
| Error triage | Scan trace list, copy run IDs to a spreadsheet | Rule routes every error trace to an annotation queue immediately |
| Latency alerting | Monitor a dashboard, page on-call manually | Rule fires a webhook to PagerDuty when latency exceeds threshold |
| Quality scoring | Run an evaluator script weekly on a sample | Rule triggers an online evaluator on every trace above a traffic threshold |
| Audit trail | Export traces before they expire | Rule extends retention on high-value traces automatically |