Why enterprise AI agents stall when they meet real company work

Most people who have tried enterprise AI agents at work know the feeling. The demo was impressive. The agent filled in forms, searched documents and drafted replies without any help. Then it met an ordinary workday and things started to go sideways.

A customer asks for something slightly unusual. A request needs approval from someone who is on holiday. The policy document says one thing, but everyone on the team knows they actually do something else. A person handles these moments without much thought. An agent often freezes, guesses, or confidently does the wrong thing.

This gap between the demo and the real deployment is one of the hardest problems in enterprise AI right now. We think a large part of it comes down to training data, and specifically to the kind of data most models have simply never seen.

What the benchmarks are starting to show

Researchers have been building tests that look more like real jobs. TheAgentCompany, a benchmark from researchers at Carnegie Mellon and elsewhere, puts agents inside a simulated software company with its own documents, chat and project tools, then asks them to complete the kind of tasks an employee would. In the latest version of the paper, the strongest agent they tested finished about 30 percent of the tasks on its own.

WorkArena does something similar with tasks on a widely used enterprise platform, and OSWorld tests agents on everyday computer work across real applications. The details differ, but the pattern is consistent. Agents do well on short, tidy tasks and struggle once the work gets long, spans several tools, or depends on knowing how things are done in that particular place.

The work that never gets written down

The philosopher Michael Polanyi once wrote that we know more than we can tell. He called this tacit knowledge, the skill and judgment people carry around without ever putting it into words.

Every company runs on it. People know which customers get a little extra flexibility, which form field everyone ignores, when to escalate a problem and when to quietly fix it, and who really makes a decision regardless of what the org chart says.

None of that is in the public text that models learned from. Much of it is not even in the company's own documentation. It shows up in the records of the work itself, in the ticket that was reassigned three times, the Slack thread where someone explained why an exception was fine, and the approval that took a week because the right person was never copied in.

What real company records add

When an agent learns from a real company's records, a few things become possible that are hard to get any other way.

The first is exposure to exceptions. Synthetic tasks tend to follow the happy path, because that is what people picture when they write them. Real records are full of cases that do not fit the template, and those are exactly the cases where agents fall over.

Real records also show how work changes hands. A request might start as an email, turn into a ticket, pick up a discussion in chat and end with an update in the CRM. Following that thread from start to finish is a skill in its own right, and it is hard to learn from anything that does not contain the whole thread.

Then there are outcomes. A resolved ticket, a closed deal or a reversed refund tells a model whether the work actually succeeded. That turns a pile of history into something a model can learn from, because it can compare what was done with how it turned out.

Finally, real records expose the gap between policy and practice. When a company's written procedures sit next to years of real execution, a model can learn both what the rule says and how experienced people apply it.

Why simulations only take you part of the way

A sensible response is to build better simulated companies, and many teams are doing exactly that. Simulated environments are genuinely useful. They are safe, repeatable and easy to score. But they are written by people imagining how work goes, and that imagination is usually tidier than reality. We compare the two approaches in more detail in synthetic company or real archive.

The strongest simulated environments tend to borrow heavily from real records, using them to decide which tasks to include, how often each kind comes up and what can go wrong along the way.

What this means if you build agents

If you are training or evaluating agents for enterprise work, a few practical things follow.

Test on tasks shaped like real work, with several steps, several tools and at least a few exceptions. An agent that passes clean benchmarks can still fail on a perfectly normal Tuesday.

Look for data that pairs documented processes with records of how the work was actually done, ideally with outcomes attached. That pairing teaches judgment in a way neither half manages alone.

Pay attention to where the data comes from. Records from real companies should arrive with consent handled, personal information carefully removed and a clear history of everything that was done to them. Our provenance page describes how we keep that history for every dataset.

The data problem behind the agent problem

It is tempting to treat the struggles of enterprise agents as a modeling problem that the next release will fix. Some of it will be. But a model cannot pick up habits it has never seen, and most models have never seen the inside of a working company.

That is the gap we work on at Synthsis Labs. We partner with companies to license records of real work, with consent and careful de-identification, for teams building agents that have to cope with the real world. If that sounds like what you are building, our page on human behavioral data is a good place to start.