Build vs. buy is the most expensive question in AI infrastructure right now.
At billions of embeddings, choosing a vector database becomes a question of cost, scale, and reliability. Buying a managed service reduces the work of operating a search cluster, while running one yourself gives you more control over how it handles your workload.
Abnormal AI’s detection pipeline generates message embeddings that help security teams find emails similar to known threats in each customer’s mail history. Across our US and EU clusters, that means keeping billions of documents searchable within a rolling 30-day window.
We chose OpenSearch for its flexibility and projected cost advantage. As we scaled, shards grew too large, searches hit memory limits, and the cluster began rejecting writes. Handling those failures required safe retries that avoided duplicate records and a recovery path for writes that exhausted their retries.
At production scale, our replicated store cost about 44% less per document than our estimate for a managed service with bulk-write pricing. This post covers how we built the system, what broke as it grew, and the changes that made it work at scale.
What we needed from a vector store
Teams across Abnormal AI had built separate systems for storing message signals and finding similar emails. This duplicated work made search quality harder to tune and improve consistently. We needed a common vector store to support all types of searches.
Our real-time detection pipeline may score the same email again when new signals arrive. That can produce repeated writes for one message. We needed a store that could absorb many more writes than searches, restrict each customer’s searches to that customer’s own data, and return similar messages quickly enough to test a model against past mail. A managed service would have spared us operating a search cluster, but its pricing and write behavior needed to fit that workload. We chose to build a shared path using our own embeddings and OpenSearch.
Indexing and querying message embeddings
Abnormal AI’s real-time scorer generates embeddings from email message attributes and sends each scored message record through Kafka, a message queue. A worker written in Go writes the resultant message vector and metadata to Opensearch, using daily indices organized by the message’s received date. The consumer batch writes roughly every three seconds, but indexing and search visibility are separate steps. Index refreshes make new writes searchable; the current configuration sets a five minute refresh interval.
This is a tradeoff between search freshness and indexing efficiency. Frequent refreshes make new messages searchable sooner, but create more segments that increase overall CPU and disk I/O. Longer refresh intervals reduce that overhead but delay visibility of new messages and updates. We found that a five minute interval was the right balance to reduce segment creation under a write-heavy workload.
A shared search API restricts queries to each customer’s data and supports similarity searches across selected message attributes. After retrieving candidates, it recomputes similarity over those attributes so scores reflect the comparison being made. Our custom AI model testing service uses this API to find similar messages in a customer’s mail history.
Each email is represented by a multi-dimensional vector derived from message attributes. Across billions of messages, storing those vectors alongside metadata, replicas, and index structures made storage footprint and index layout central to our design.
Scaling ingestion and search
As the embedding store grew, keeping messages searchable required more than simply adding capacity. We had to change how data was partitioned, how indices were maintained, and how searches shared resources with incoming writes.
Our initial shard layout did not keep pace with ingestion. Shards grew too large, slowing OpenSearch’s blue/green deployments during configuration changes. We increased the number of primary shards per daily index, added cluster capacity, and increased memory per node. When indexing saturated disk I/O, we also raised storage throughput.
Within each shard, segment merges created additional work. OpenSearch merges smaller segments into larger ones, reducing segment count but consuming CPU and disk I/O as it rewrites data. We tuned the background merge policy and configured lifecycle force merges to target 20 segments per shard rather than one. The aim was to consolidate segments without creating excessively large merge operations. Older indices remained writable to accommodate delayed records and backfills.
Similarity searches exposed another bottleneck. OpenSearch’s approximate kNN search uses Hierarchical Navigable Small World (HNSW) graphs that can consume substantial native memory. These graphs are built over all of a shard’s segments, even when a query restricts results to one tenant. A smaller tenant’s query could therefore search graphs containing a much larger tenant’s vectors. During testing, native memory pressure triggered OpenSearch’s kNN circuit breaker. Data nodes also stopped responding to cluster-manager heartbeats and were removed from the cluster, blocking ingestion. We did not identify customer-visible impact from these failures. We introduced partitioned tenant routing to distribute large tenants’ data across multiple shards and capped concurrent search requests across backtests running on each service instance.
To reduce native memory pressure, we enabled memory-optimized search, which accesses vector index files through memory mapping and the operating system’s page cache. This reduces reliance on loading entire graphs into native memory, though cache misses can introduce disk I/O. We also retrieved candidates using compressed vector representations, then rescored them using full-precision vectors. Rescoring improves candidate ranking but cannot recover messages missed during approximate retrieval. The API retrieves vectors internally to recompute similarity and returns only scores and message metadata, keeping response payloads smaller.
Handling write failures
Cluster tuning addressed capacity and memory bottlenecks, but ingestion also needed to tolerate rejected writes. The consumer distinguishes temporary failures, such as overload, from documents that cannot be indexed. It retries temporary failures locally using exponential backoff. Once the local retry limit is reached, it hands the record to a series of Kafka retry queues with progressively longer delays. Records that exhaust those retries go to a dead-letter queue for investigation, while malformed documents go there directly.
When local retries are exhausted or a document encounters a terminal indexing error, the consumer also sends a copy through Firehose to an S3 bucket archive. Archived documents can be investigated and manually replayed into OpenSearch through an Airflow DAG. This path operates independently of the Kafka retry queues, providing an additional recovery option. The consumer acknowledges the original record after Kafka confirms delivery to a retry or dead-letter queue. Delivery to S3 occurs independently, so an S3 delivery failure does not block the Kafka recovery path.
The consumer also needed limits to prevent pending writes and retries from exhausting memory. We bound Kafka prefetch and the number of in-flight writes, and configured a soft memory limit for the Go runtime. When in-flight writes reach the limit, backpressure slows consumption which leaves records in Kafka rather than retaining them in memory. We also strengthened health checks to detect when individual Kafka partitions stopped making progress, even if the consumer continued retrying writes or processing other partitions.
Testing readiness before launch
We compared the new search path with the previous system before launch to evaluate its effect on threat detection. Both searched the same historical customer emails, and the same AI decision filters evaluated the messages each retrieved. The new system’s detection counts met our acceptance criteria for launch.
We then load-tested the complete backtesting workflow: retrieving similar messages, applying the model’s decision criteria, and returning matching emails for review. Hundreds of historical model tests, including runs with multiple models in parallel, completed with a 100% success rate across the test set. These tests exercised the concurrent workload we expected from larger customers.
After launch, we continued tracking model backtesting latency in production. In the production week ending October 7, 2026, p99 backtest latency was approximately 4.2 minutes in the US and 2.6 minutes in the EU.
The economics at production scale
Running the vector store ourselves gave us a measurable advantage once the system reached production scale.
To compare it with a third-party SaaS, we modeled the same vector dimensions and metadata, with batch writes at the vendor’s bulk-write price. On those terms, our store cost about 44% less per document.
What comes next
Since launch, other teams have begun exploring uses for the shared embedding store beyond the original use case. Similarity search could help teams to find related messages for investigation and build broader evaluation datasets.
These efforts remain exploratory. Any future capabilities are not currently available to customers, and this is not a commitment to deliver them.
This post was authored by Jeremy Ryu, Software Engineer - Model Platform, at Abnormal AI.
If this is how you want to work, we’re hiring! Subscribe to Abnormal Builder’s Substack to see how we actually build with AI, with real stories from our engineering and product teams.