Building an AI agent stopped being impressive about a year ago. You can stand one up in an afternoon from just about anywhere. But when you put it in production where it’s making dozens of calls to models, APIs, and internal services, things get interesting. You’re no longer asking “how do we build a good agent?” You’re asking:
How do we run agents reliably when something in the chain breaks?
How do we keep making them better?
It’s a distributed systems problem, now with a model in the loop.
This piece is about the first question: how we evolved our infrastructure at Abnormal to run increasingly sophisticated agents without the whole thing falling over when one call fails.
Every day, Abnormal runs many different AI agents that perform tens of thousands of security tasks. One of those agents classifies email: it investigates a message, weighs the available evidence, and determines whether the message is a threat. The agent supplements our existing detection with a more extensive analysis of each email, helping catch unusual attacks that are difficult to detect through signal-based methods alone.
The first version of this agent was relatively simple. It received an email, made a few model and tool calls, and returned a classification. That approach worked well when the investigation was short and predictable. As models have become better at reasoning over longer periods, using tools, and completing tasks autonomously, the agent has grown more sophisticated. It now makes far more calls to internal services, including Abnormal’s behavioral engine and specialized analysis tools. Instead of reasoning only about the initial email, it can gather additional evidence, consult domain knowledge, and decide what to examine next before reaching a conclusion.
That deeper investigation produces better-informed classifications, but it also changes the shape of the computation. A single run takes longer, crosses more system boundaries, and encounters more opportunities for failure. So as this agent became more autonomous, our existing deployment infrastructure became a bottleneck and handling failures had to become a first-class part of agent development.
To solve for this, we built an agent platform that enables any team to build, evaluate, and deploy agents with confidence.
Before: Kafka delivered jobs, agents managed execution
Our earlier agent infrastructure used technologies familiar to many engineering teams. We used gRPC APIs for synchronous requests and Kafka for asynchronous jobs. When a job failed, we sent it to a retry queue so it could run again. After too many failed attempts, it went to a dead-letter queue (DLQ) for later investigation. This is the same proven pattern described in Uber’s article on reliable reprocessing, and we use it extensively across Abnormal’s microservices stack.
That architecture was effective when agent executions were short and predictable. Kafka reliably delivered work, retry queues gave failed jobs another chance, and gRPC provided a straightforward request-and-response interface.
The important detail was the unit of work. One Kafka message represented one complete agent run. The consumer received the message, ran the agent from beginning to end, and committed the Kafka offset when processing completed successfully.
If the run failed, the message was handed off to a retry topic, and the consumer recorded the original message as handled by committing its offset. A retry consumer would later receive the message and run the entire agent again. After the configured retry attempts were exhausted, the message moved to a DLQ.
Kafka tracked the message as it moved through this process, but it did not track progress within the agent run. If an agent completed several steps and then failed, none of that intermediate progress was checkpointed. The next attempt started the full run again.
That was a reasonable tradeoff for simple agents. As agents became more autonomous, however, a single run began to involve more reasoning, more tool use, and more calls to internal services. Restarting the whole unit of work meant repeating more expensive model calls and service calls. It also forced each agent to take on increasingly complicated retry and recovery logic.
Kafka was still doing its job: delivering messages reliably. The problem was that our unit of work had changed. An agent run was no longer just a request to process. It was a long-running, multi-step workflow whose progress needed to survive failures.
The infrastructure had become a bottleneck in two ways. Operationally, longer runs magnified the cost of every restart and forced agent teams to build their own recovery logic. Developmentally, that bespoke infrastructure made it harder to move an agent safely from an experiment to production. We needed the platform to manage the execution of an investigation, not merely deliver its initial request.
After: Temporal manages agent workflows
We evaluated several durable execution platforms for addressing these challenges and ultimately chose Temporal. Rather than treating it as a standalone piece of infrastructure, we built an integrated agent platform around it. Self-hosted Temporal exposed the right primitives for us to build vertical integrations and tailor our agent platform to the data residency requirements of our regulated deployments. The platform combines Temporal’s execution guarantees with our systems for experimentation, deployment, observability, and production guardrails.
Temporal separates an application into two parts. A workflow coordinates what happens next and must produce the same decisions when given the same recorded history. Activities perform outside work such as model requests, tool calls, and other input/output (I/O). Temporal durably records the workflow’s decisions and activity results. If execution is interrupted, a worker replays that history to reconstruct the workflow’s state. It then resumes with the next incomplete activity instead of repeating the entire run.
Changing the unit of recovery
Temporal solves the unit-of-work problem by separating the lifetime of the overall agent run from that of any individual worker process or service call. In our Kafka architecture, the durable unit was the message, which represented the entire agent run. Kafka knew whether that message had been completed, but it did not know which steps inside the run had already succeeded. A failure near the end therefore caused the whole run to start again.
With Temporal, the overall agent loop becomes the workflow, while model requests, tool calls, and service calls become activities. During recovery, Temporal re-executes the deterministic workflow code against its event history. When that code reaches an activity that already completed, Temporal supplies the recorded result instead of performing the external call again. When it reaches an activity that did not complete, Temporal can run that activity again according to its retry and timeout policy.
This makes an individual activity the practical unit of retry and recovery. A failed API call can be retried according to its own policy without repeating earlier model calls or analyses. If the failure is not recoverable, the workflow can stop, take a fallback path, or surface the execution for investigation. The agent remains one coherent unit from the product’s perspective, while the platform manages its smaller execution steps independently.
For an agent, the workflow contains the sequencing and decision-making: it keeps track of results, determines what should happen next, and decides when the investigation is complete. Each operation that interacts with the outside world becomes an activity, such as:
Calling an LLM
Retrieving information from an external API
Running a specialized analysis
Reading or writing data
Triggering an approved downstream action
This separation matters because external operations fail in different ways, and not every failure should have the same effect on the agent. If a service provides evidence that is essential to the task, the agent may need to withhold its result entirely when that call fails. If a less critical service is unavailable, the agent may instead retry, use another source, or continue with reduced information. That policy is part of the agent’s correctness: the infrastructure must make it possible to distinguish between failures the agent can work around and failures that must stop the workflow.
An LLM request that encounters a temporary provider error may be safe to retry. An API that is rate-limiting us may need a longer delay. An action that modifies external state may require an idempotency key to ensure it is not performed twice.
Temporal gives us a standard place to express those differences through retries, timeouts, and failure-handling policies.
How product teams consume the platform
We expect agents to play a growing role across Abnormal’s products, which means deploying and operating many more of them without asking every product team to build its own execution infrastructure or making the platform team a bottleneck. Our goal is to make agent deployment self-service: Temporal lets us provide a shared foundation for durable execution, retries, recovery, isolation, and observability, while product teams can build and evolve agents independently. The responsibility boundary is deliberate: product teams own each agent’s behavior, domain logic, tools, service-level objectives, and outcomes, while the platform team owns the orchestration primitives and the health, capacity, and reliability of the shared Temporal infrastructure.
Product teams define an agent as regular application code: its instructions, tools, domain logic, and decisions. The platform team provides primitives that turn that code into a durable workflow, route its external operations through activities, and make the workflow executable on a Temporal worker. Teams do not need to become orchestration experts simply to deploy an agent.
Each agent is deployed as its own Temporal worker with a dedicated task queue and identity. The task queue routes work to the correct agent, while the identity restricts which services and resources that agent can access. Together, they limit what each agent can do and create a stronger security boundary between agents.
An agent also does not have to be the entire workflow. Product teams can compose agents with conventional application logic, other services, approval steps, and product-specific processing. They can then reliably trigger downstream actions—such as creating a finding or starting remediation—as part of the same durable workflow, with the appropriate retry, idempotency, and failure policies.
We operate one shared Temporal cluster with three namespaces for agents:
Experiment for offline analysis and evaluation
Dark for shadow production traffic
Prod for live production execution
Namespaces separate these execution environments, while a dedicated task queue separates each individual agent. This gives us the isolation needed to limit the impact of noisy neighbors and creates clear points for throughput control. The platform team can manage capacity and limits across the cluster and namespaces; product teams can tune worker concurrency and throughput for their own agents.
The ownership boundary follows the same model. Product teams monitor their workflows and define the service-level agreements (SLAs) appropriate for their use cases. The platform team owns the overall health, capacity, and reliability of the shared Temporal cluster.
The Experiment namespace is integrated with our agent platform so teams can run offline analyses against curated datasets. Temporal’s workflow history also gives them a detailed record of each execution, and the agent platform surfaces that history for Dark and Prod workflows so teams can inspect how an agent reached a result and diagnose failures.
Early results
We have started migrating several of our email-classification agents to Temporal-based workflows. Before the migration, roughly 0.5% of agent runs were sent to our retry queues, in part because the agents had begun integrating service calls that took longer to complete and were less reliable. Retrying those jobs meant restarting the entire agent run and repeating work that had already succeeded.
With Temporal, we have eliminated that full-run retry waste. External calls can still fail, but Temporal retries the incomplete activity portion without repeating the rest of the investigation. This has also increased our confidence in integrating more of Abnormal’s behavioral engine into the agent: additional service calls no longer make the entire run disproportionately fragile or expensive to recover.
We have also seen promising early results for agents that run for multiple hours. Some product teams had previously built and operated their own state machines to support those long-running executions. Moving responsibility for orchestration to the shared Temporal platform removes the need for specialized infrastructure and lets those teams focus on the agent’s logic, tools, and outcomes.
Most excitingly, Temporal has allowed us to replace much of the internal execution infrastructure previously owned by the platform team with an integrated agent platform, enabling a lean team to support the full range of Abnormal’s agent requirements. That platform bridges the gap between experimental, dark, and production deployments, giving teams a consistent way to move an agent from evaluation to live traffic.
Temporal gives us confidence not only in deploying these agents, but also in observing, iterating on, and improving them. By automatically bringing workflow history and execution data into the agent platform across deployment modes, Temporal helps shorten our development feedback loop. Teams can understand agent behavior, identify opportunities, and ship better versions more quickly.
Final thoughts
At Abnormal, we use Temporal as the durable orchestration foundation for deploying AI agents reliably. It lets us treat agent execution as the distributed systems problem it is: preserving progress across failures, retrying work at the right boundary, isolating agents through namespaces, task queues, workers, and identities, and giving product teams a shared platform for operating workflows from experimentation through production.
This foundation has a significant impact beyond reliable deployment: it shortens the feedback loop between running an agent, understanding its behavior, and improving it. In a future article, we hope to share how Abnormal evaluates these agents, learns from their production behavior, and turns those insights into faster iteration and better security outcomes.
