Synthetic company or real archive? What actually trains an enterprise agent

Should you train an enterprise agent on synthetic company data, or on archives from real companies? Robotics teams have argued about a version of this question for years. Simulation is cheap and you can run it forever, while real data is expensive and scarce but undeniably real. Most of them eventually settled on using both.

The same debate is now playing out for agents that do office work. Teams can build simulated workplaces full of invented employees, tickets and documents, or they can train on records from companies that actually exist. This article looks at what each approach is good at, where each one falls short, and how the two fit together.

What a simulated company looks like

A simulated company is a sandbox. It might include a fake email system, a chat tool, a ticket tracker, a CRM and a set of documents, all filled with made up people and made up work. An agent is dropped inside and given tasks, like resolving a customer complaint or preparing a report, and the environment checks whether it succeeded.

Several well known research benchmarks work this way. TheAgentCompany builds a small software company for agents to work in, and WorkArena sets tasks inside a widely used enterprise platform. Teams building agents increasingly use environments like these for training as well as for testing.

Why simulation is attractive

The appeal is easy to understand. For a start, you can make as much of it as you like, since generating another thousand tasks costs very little once the environment exists.

Simulations can also be reset, so every attempt starts from the same state. That makes it easy to compare agents fairly and to use training methods that need many tries at the same problem.

Scoring is simple as well. Because you built the world yourself, you know what the right answer is, and rewarding an agent for getting things right becomes straightforward.

And there is no privacy work to do, because nobody's real messages are involved.

Where simulation falls short

The weakness of a simulated company is that someone had to imagine it.

People who write synthetic tasks tend to write the version of work they picture in their heads, which is usually the clean version. The customer explains the problem clearly, the policy covers the situation and the right person answers on the first try. Real work is full of half explained requests, missing information and situations nobody planned for.

Simulations also have to guess how often things happen. In a real support queue, a handful of problem types might make up most of the volume, while a long tail of odd cases makes up the rest. Whoever designs the simulation has to guess at that mix, and the guess is often wrong in ways that matter.

There is a subtler risk as well. When the synthetic content is itself written by language models, training on it can feed a model's own habits back into itself. Researchers writing in Nature showed that models trained repeatedly on generated data gradually lose the rarer details of the original data. An enterprise simulation is not the same as that experiment, but the lesson carries over. If you want a model to handle the unusual case, it helps to show it real unusual cases.

What real archives bring

Real company archives have almost the opposite profile.

They contain the real mix of work, long tail included. They contain mess, which is exactly what agents need to learn to cope with. They often contain outcomes, so you can see whether a ticket was resolved, a deal was won or a refund was reversed. And over several years they show how a company's processes changed, which teaches something about adapting that a frozen simulation never can.

They come with real costs, too. There is only so much of it, and you cannot generate more. It has to be cleaned carefully before anyone can use it, which we describe in how to de-identify workplace data. And you cannot reset a real company to replay the same afternoon, so it is harder to use with methods that need many attempts at an identical problem.

Side by side

Using both

In practice the strongest setups combine the two, and the combination works best when real data shapes the simulation instead of simply sitting next to it.

Real archives can tell you which tasks to simulate. If a real support queue shows that a few kinds of requests dominate and that certain exceptions keep coming back, those patterns belong in the simulated environment.

Real archives also make good test sets. Training in simulation and then evaluating on tasks drawn from real records gives you a much more honest picture of whether an agent will cope outside the sandbox.

And real archives teach judgment that is hard to fake. Pairing a company's written procedures with records of how people actually followed them shows a model the difference between a rule and the way it gets applied, which is very hard to invent convincingly.

Simulation then does what it does best, giving agents endless safe practice on tasks that look like the real thing.

Questions to ask about your own setup

If you are building enterprise agents, a few questions can show whether your data is balanced.

• Were your simulated tasks designed from real examples, or from someone's idea of the job?
• Do you evaluate on anything drawn from real work?
• How often do your training tasks include missing information, exceptions or handoffs between people?
• Can you trace where your real data came from and how it was cleaned?

If the answers lean toward imagination more than evidence, adding real records is likely to make a noticeable difference. We look at why agents struggle without them in why enterprise AI agents stall when they meet real company work.

At Synthsis Labs we work with companies to license records of real work for exactly this purpose, and you can see the kinds of data we work with in our marketplace.