Node Caching
Cache expensive node results with CachePolicy on add_node(). Cache backends for dev vs. production, custom key_func, and the pickle deserialization CVE you need to patch.
Quick Reference
- →CachePolicy(ttl=N): set per-node on add_node() — NOT on compile()
- →compile(cache=InMemoryCache()): supply the cache backend separately from the checkpointer
- →key_func: custom function to control which state fields form the cache key (default: pickle hash of full input)
- →@task(cache_policy=CachePolicy(ttl=N)): functional API equivalent of per-node caching
- →__metadata__['cached']: True in stream_mode='updates' confirms a cache hit
- →cache.clear(): manually invalidate all entries — required after deploying new node logic
When to Use Node Caching (and When Not To)
Should I cache this node? Pure → Expensive → Repeated inputs → pick backend
| Node Type | Cache? | Why |
|---|---|---|
| LLM calls | Yes | Expensive, often repeated with same input in retry loops |
| Embedding computation | Yes | Deterministic and expensive at scale — same text always produces same vector |
| Idempotent API lookups | Maybe | Only if response is stable — reference data yes, live prices no |
| DB writes | Never | Side effect — caching skips the write, data never gets persisted |
| Time-dependent logic | Never | Stale cache returns wrong results — current time, market prices, live feeds |
| Random/sampling | Never | Caching defeats randomness — same input should produce different outputs |
The cached result replaces execution — the side effect will not happen. If a node sends an email, writes to a database, or charges a payment, caching it means the action silently disappears. The graph sees a successful result; the real-world effect never occurred.
Caching has the highest payoff in two scenarios: iterative agent loops (the same node called multiple times with unchanged input when a tool call fails and retries) and multi-user workloads (thousands of users asking similar questions within the TTL window). Outside those two cases, run the cost math first before adding caching complexity.