Advanced9 min
LangSmith Automation Rules
Trigger automated actions on production traces — route failing runs to annotation queues, auto-build evaluation datasets, and fire webhooks when anomalies appear.
Quick Reference
- →Automation rules evaluate every trace against filter conditions and fire actions when matched
- →Filter by feedback scores, metadata fields, latency thresholds, tool call patterns, or sampling rate
- →Actions include: add to dataset, route to annotation queue, trigger webhook, extend trace retention
- →Combine sampling with filters to build representative eval datasets without manual curation
- →Rules run asynchronously — they never add latency to your agent's hot path
Anatomy of an Automation Rule
An automation rule consists of two parts: a filter that selects which traces to act on, and an action that determines what happens to matched traces. Rules are evaluated asynchronously against every incoming trace in a project, so they add zero latency to your agent. You can stack multiple rules on the same project — each rule is evaluated independently.
Create an automation rule via the SDK
| Component | Purpose | Example |
|---|---|---|
| Filter | Select which traces to match | eq(metadata.environment, "production") |
| Sampling Rate | Percentage of matched traces to act on | 0.1 for 10% sampling |
| Action | What happens to matched traces | Add to dataset, route to queue |
| Project Scope | Which project the rule monitors | "my-agent-prod" |