Last month, an autonomous AI agent spent about two and a half days inside Hugging Face’s infrastructure before anyone caught it. It wasn’t a criminal break-in in the usual sense since the agent was one of OpenAI’s own models running during a red-team evaluation. But what it did simulated a real attack: it used a zero-day to get in, stole valid credentials, and moved through internal systems on real accounts.

Once an attacker is working from real, trusted credentials, there’s usually nothing left to flag. While impersonation gets caught, a message from a genuinely compromised account often doesn’t because it arrives from your coworker’s real mailbox, carrying signals your defenses were built to trust. That’s compromised sender phishing, and it’s hard to catch for a simple reason: when everything on the outside says “trusted,” all that’s left to judge is whether the account is behaving like itself.

Here’s how we designed a model to do that.

Why knock when you can use the key?

For years, the dominant playbook was impersonation. But the security industry got good at catching it with authentication checks, domain reputation, and display-name analysis. So, instead of pretending to be someone trusted, attackers started becoming someone trusted: taking over a real account with its real domain, history, and relationships, and sending from it. AI has only made this easier – anyone can sound like you now.

We’ve seen this in the broader threat landscape too; there has been an increase in account takeover cases.

Figure 1: Monthly account-compromise volume, trending up ~30% since late 2025

The attacker now owns every signal we trust

An attack from a trusted, verified source is about as hard as detection gets.

Compromised senders carry signals we read as safe: a real communication history with the recipient and legitimate domains with valid SPF and DKIM. We caught impersonators precisely because they lacked those signals. A compromised account has all of these, so the same signals that once exposed an attacker now protect them.

Rarity makes this worse. Models learn from huge volumes of mail dominated by common attacks, so they get very good at spam and bulk phishing and barely register something rare like this. The attacks worth the most to stop therefore become the ones the system is least equipped to find. And a trusted asset is almost by definition the hardest thing to model because the better an account looks, the less there is to catch.

Let the victim label the attack for you

The bottleneck here wasn’t really the model; it was that we couldn’t even find our own misses.

The usual way to improve a detector is to retrain it on what it missed, using the attacks that come back through support tickets and analyst flags. But that channel is slow and narrow. It depends on a person noticing one specific message and reporting it, so only a trickle of what we missed ever comes back. And the attack messages are almost useless as a search target anyway – they’re diverse and they look completely legitimate, so there’s nothing to query for.

So, we decided to stop looking for the attack and start looking for its aftermath.

When an account gets taken over and used for phishing, the real owner almost always sends an apology a day or two later: “I was hacked, please ignore my last email!” Everyone’s seen these. What we realized is that the apology is independent ground truth. It’s written by the victim and doesn’t depend on a ticket, an analyst, or anyone flagging anything. It also pins down timing, marking roughly when the compromise came to light, which gives us a window to trace the account’s activity back to the attack itself.

The apology is also the one part of the attack that’s easy to find. While attack messages are diverse and look legitimate, apologies are nearly identical; since people describe getting hacked in remarkably consistent words, searching for that formulaic wording surfaces them reliably at scale. We anchored on the findable thing and worked backward: a pipeline to mine apologies at scale, then trace each one back to the messages that account sent in the roughly 72 hours before allowing us to increase our high confidence label set by ~30%.

Moving the needle with data decisions

Once we could recover our misses, the real question was how to build a training set that teaches the model compromise specifically; the decision that mattered most was judging each account against its own recent sending behavior, not against attacks in general. For every compromised account, we paired its de-identified behavioral signals from the attack window with the signals from that same account before the compromise. All of it came from our proprietary threat intelligence, de-identified across customers, and limited to sender and header behavior – not message content. This taught the model not what an attack looks like in the abstract, but what anomalous behavior looks like relative to a given sender’s own baseline, which is the only framing that works when the attack itself carries so many safe signals.

The bigger surprise was what didn’t help. We expected email content to improve detection, but it actually hurt – body text, links, and attachments, all things we’d looked at closely early on, made performance worse at scale. Once paired sender history gave the model enough context, content just added noise.

Figure 2: Our model-building pipeline

The result

The final model was an XGBoost binary classifier trained on sender and header behavioral signals only. The training set was tens of millions of labelled example rows, de-identified across customers and drawn from our proprietary threat intelligence, at roughly a 5% attack rate, far higher than that seen in real production traffic.

On a production-representative evaluation set of 5.4 million messages, the model caught tens of thousands of compromised sender attacks while keeping false positives under 0.15%. Earlier detection approaches to this attack class hadn’t cleared our false-positive bar.

It runs in our real-time scoring pipeline now with no new content-extraction dependencies, and early results show no material regression against prior detection on their own evaluation sets, so the new model is designed to complement rather than replace what’s already running. In the last 30 days, it caught roughly 9000 unique campaigns.

Critically, the model’s value extends beyond its standalone performance. Because it was deliberately built on signals orthogonal to our existing detection ensemble – sender behavioral patterns rather than the content and reputation signals our other models already leverage – its output serves as a powerful complementary signal across the broader detection stack. When combined with our pre-existing models, the compromised sender score allows us to squeeze more detection value out of both systems: the new model catches attacks the existing ensemble misses, and the existing ensemble can operate with higher confidence when the compromised sender signal corroborates or contradicts its own verdict.

First principles compound

The detection systems that hold up best at Abnormal tend to share an origin: they came out of a deep understanding of the attack and not a rush to optimize a metric.

The slow, early investigation wasn’t wasted. It produced the pre-filter, the feature insights, the label corrections, and the dataset design that made the final model possible, and each of those has value on its own. The compromise pre-filter and the probability-of-compromise signal are already being used by other detection systems across our platform. The corrected labels improved training data quality for the broader stack. And because the model was built on orthogonal signals by design, it strengthens the models it runs alongside as much as it strengthens compromised sender detection itself.

We’re taking the same approach to adjacent attack types: deep investigation, paired datasets, and systematic experimentation before committing to a model. We’re also looking at data augmentation to better represent rare attacks in training.

Building AI to fight AI takes more than powerful models and more than moving fast, though both help. It takes the patience to understand the problem before reaching for the modeling toolkit. For compromised sender detection, that patience turned a stubbornly hard attack class into a tractable one, and it paid off well beyond the one model.

Friend or foe: teaching a model how to trust