Design an AI Email & Calendar Agent
A hellointerview-style system design deep dive into AI email and calendar agents like Google Gemini in Workspace, Microsoft Copilot for M365, and Lindy AI. Covers requirements, core entities, the action planning pipeline, and three production deep dives: intent understanding and multi-turn planning, calendar constraint solving with preference optimization, and progressive autonomy with trust scoring. Each deep dive walks through naive, better, and production-grade approaches with trade-offs.
Quick Reference
- →The core design challenge is graduated autonomy — what can the agent do without asking permission, and how does that expand over time
- →Calendar constraint solving is a multi-variable optimization: availability, timezone, preferences, fatigue, and conflict resolution
- →Multi-step execution: check availability, propose times, get confirmation, send invites, handle responses — each step may require user input
- →Low-stakes actions (draft email, check availability) vs high-stakes actions (send email, cancel meeting) require different permission levels
- →The agent reads all your email — data minimization, access token management, and content isolation are critical privacy requirements
- →Trust scoring expands agent autonomy over time: the agent earns permission by consistently making good decisions
- →Plan preview before execution lets the user correct mistakes cheaply — much better than apologizing after sending a wrong email
- →Disambiguation is essential: 'Schedule a meeting with Sarah' requires knowing which Sarah, when, how long, and on what topic
Understanding the Problem
An AI email and calendar agent manages communication and scheduling on behalf of a user. It drafts and sends emails, schedules meetings by finding mutually available times across participants, manages calendar conflicts, handles meeting responses, and triages incoming messages. This is not a simple assistant that answers questions about your schedule. It acts on your behalf — sending emails from your account, booking meetings in your name, and modifying your calendar. Products like Google Gemini in Workspace, Microsoft Copilot for M365, and Lindy AI have made this a mainstream product category. From a system design perspective, this is a rich problem because it touches natural language understanding (parsing vague requests like 'find time for the team next week'), constraint satisfaction (finding a slot that works for 5 people across 3 timezones with individual preferences), permission modeling (what the agent should do autonomously versus what requires confirmation), and privacy (the agent reads all your email, including confidential HR discussions and personal messages). The trade-offs are sharp: too much autonomy and the agent sends an embarrassing email or cancels an important meeting; too little autonomy and the agent asks for permission so often that it is faster to just do it yourself.
Google Gemini in Workspace can draft emails in a user's writing style, summarize long email threads, and propose meeting times based on calendar analysis. Microsoft Copilot for M365 integrates across Outlook, Teams, and Calendar to handle scheduling, email triage, and meeting preparation. Lindy AI operates as a fully autonomous email and calendar agent that learns user preferences over time and progressively takes on more tasks without asking. The key insight from all products: users want the agent to be proactive (do not ask me if I want to accept a meeting from my manager — just do it) but safe (never send an email to a client without my approval).
This is fundamentally about building a system that acts on behalf of a user with appropriate autonomy. The three hardest sub-problems are: (1) understanding vague and incomplete scheduling requests well enough to build an executable plan, (2) solving the multi-variable constraint satisfaction problem of finding a meeting time that works for all participants, and (3) calibrating how much the agent does autonomously versus how much requires user confirmation, and evolving that boundary over time as trust builds.