In September, a support engineer at Abnormal filed a bug after a customer’s security dashboard kept coming up empty when they searched it. Seven hours later, the fix was running in production and the only thing a person actually did was decide whether to merge the PR.
That ticket is one of more than 18,000 the Nora Software Factory has researched. It turns raw customer bug reports into reviewed PRs, with people staying above the loop to make the calls while Nora runs the steps. Since late April, it has merged more than 360 PRs from tickets in 39 Jira projects, approved by more than 160 engineers.
Here is how we handled that ticket, and the places where most software factories stall.
How the factory operates
Research reads several sources at once. From Jira it pulls the ticket, its comments, and linked tickets, which together show whether the problem has come up before and how it was handled. From our internal tools it reads transcripts of recorded customer calls, which often describe the problem more clearly than the ticket and catch what the customer was trying to do when it broke. It reads wiki pages and product FAQs, and several code-reading agents search a sandboxed copy of our monorepo in parallel.
Teams that have joined the factory also keep short markdown context files for their Jira projects and components. They record where the important code lives, which product decisions are settled, and which Slack channels and owners to go to. There are 88 of them across 18 projects, written by nearly 40 people, and most are a few lines linking to a runbook or agent skill the team already had. Nora follows these links and loads the files straight from our main branch, so a merged edit reaches the next run within 15 minutes, without a deployment. They give the agent the knowledge a teammate would otherwise carry in their head.
Some teams also opt into automated PRs, where Nora opens a draft PR from the approved plan and a coding agent implements it in an isolated sandbox. The agent is told to run the relevant tests, type checks, and build before each commit, and to list anything it couldn’t verify rather than claim it passed. A required review step checks the change against the same architecture, security, and legal docs as the plan, and the PR stays a draft assigned to someone on the ticket until a person marks it ready.
Humans above the loop
We want people above the loop, setting policy and judging outcomes, rather than in it doing each step by hand. Every production change in our lights-on software factory still needs a person’s approval to merge.
Here’s how the different phases of the factory played out on that ticket.
Solution
A support engineer filed the bug at 08:48.
Research started on its own, and by 09:02 Nora had posted a Jira comment. The comment said “Code Change Required: Yes” and offered three fixes, recommending the smallest, which extended a helper that already carried one parameter so it carried the other two.
Plan
At 12:29, Nora attached a plan to the ticket and posted it on a new draft PR. The plan named the two files, the test command, and the team that should review, and it spelled out what the change would not touch.
Change
A coding agent implemented the plan in a sandbox in under eight minutes, touching two files and adding seven tests that all passed. The on-call engineer marked the PR ready at 12:38 and approved it at 12:40, a second engineer approved at 13:34, and it merged a minute later. Nobody edited Nora’s code.
Verify and Deploy
At 16:06, after the regular deploy, the on-call engineer closed the ticket and marked it resolved.
Any stage can also hand work back to a person. We call that an off-ramp and treat it as a sign that context, tooling, or architecture needs fixing rather than a failure. If the mistake came before implementation we go back to that stage, and if it is in the implementation itself we iterate on the same PR.
Where software factories break
Our first pilot ran one team’s tickets for a week. It opened 19 PRs across 13 of 16 tickets, none of which fixed the ticket on the first try. One of the four gaps it exposed was really about the agent, which had no map of the code. The rest were organizational: product decisions that lived in people’s heads, reviews with no SLA, and services that couldn’t deploy on their own.
Here are four places factories break, and what we do about each.
1. Too many PRs to review
A factory can open PRs faster than anyone can read them. For us, whoever asked for the PR decided its fate. When a person asked for one, nearly half merged. When a rule asked (research flagging a “one-shottable” fix, or a project labeling every new ticket), most never got a human review, and nearly every PR we closed unmerged had never been looked at by a person.
Research is cheap to read and a PR is expensive to review, so we research widely and open PRs sparingly. Every PR opens as a draft and reaches review only when a person marks it ready. The biggest lever is a named owner. From June to September, PRs a person asked to be merged 55% of the time with a GitHub assignee and 28% without.
2. Building code nobody needed
An agent can write code regardless of whether or not code is needed. In our pilot, at least 6 of the 16 tickets needed no code change at all.
To fix this, we made Nora Research answer “Code Change Required: Yes / No“ before anything else.
3. PRs that grow too big
Agent PRs tend to grow. In September, agent PRs that engineers drove from Nora’s CLI had a median of 263 changed lines, against 129 for ordinary human PRs.
So the factory ships exactly one focused PR per plan, and the research prompt rules out stacked or phased PRs. A person approves the plan before any code exists, and the PR is built from that plan. Only 20 of about 1,168 tickets got a second PR, and merged factory PRs had a median of 94 changed lines. Teams can add their own limits on top of this.
4. Tickets a factory shouldn’t take
The input decides the outcome. Our four strongest projects, all from internal engineering teams with at least 10 PRs, merged 61 to 82% of them. The customer-issue project merged 18.6%, because most of its tickets needed a PM to review and refine the requirements first.
We route by kind of work. One team found 80 to 90% of its tickets needed only an explanation to the customer, so they run research alone.
The test we’d give another team:
Does the work usually need code? If not, run research only.
Will a named engineer review the PR within two weeks? If not, don’t open the PR.
Does an engineer already know the fix? Use the agent directly.
Systems and Security
An agent that works through thousands of tickets on its own will fail some runs and get others wrong. These choices make sure a failed run gets noticed and a wrong run can’t do much harm.
We run research and PR creation on Temporal. Our first version was an Airflow job that ran each agent call itself. Today, the poller only claims tickets and hands each one to a workflow on Nora’s Temporal cluster, a durable-workflow engine we wrote about in Agent Reliability Is a Distributed Systems Problem. A failed run retries once, and a final failure labels the ticket and tells the requester. Since July, 0.4% of research tickets have failed for good.
Sandboxes get partial internet access, on purpose. Coding runs happen in a sandbox that holds no AWS credentials. It can reach an allowlist of services, and its only route into our infrastructure is a test environment. There, the agent signs in as a synthetic test tenant to check its change end to end. That gives it enough access to prove its own work and no path to production.
The agent can read widely but write very little. Research reads every source described earlier and writes back a comment and a plan on the ticket. Coding runs get a GitHub token that expires in an hour, and their work lands in a draft PR that a person has to mark ready and approve. A wrong run goes no further than the ticket or a draft PR.
Results
The factory has run in production since late April. As of October, it has merged 360 PRs, and merges grew every month, from 14 in May to 100 in September. In our customer-issue project, each week from late August through September, 41 to 46% of the tickets resolved had Nora’s research behind them and 5 to 12 went through a factory PR.
Improving the factory is the job
Every run that doesn’t ship is a signal. The fix is almost always context and not the model. A team edits its context file, and the next run picks it up. To check an edit first, an engineer runs a CLI command nora factory replay research <ticket>. It reruns the stage on the production code path with the local edit and with Nora’s earlier comments hidden, so the new context has to find the answer on its own. For changes to the instructions all our coding agents share, an eval runs a fixed set of 15 merged PRs through the old and new setup and has a blinded judge compare the results.
To find what to fix, any engineer can run our learn-team-factory skill on a team’s project. It reads the team’s recent Jira tickets, PR review comments, Slack threads, and meeting transcripts, keeps the lessons that would apply to the next ticket, and opens a docs-only PR against context files, developer docs, or skills. Scheduled jobs do the same every week, but few of their PRs merge.
If this is how you want to work, we’re hiring.
This post was authored by Tushar Amrit, Software Engineer - Nora Platform, at Abnormal AI.
Subscribe to Abnormal Builder’s Substack to see how we actually build with AI, with real stories from our engineering and product teams: