Independent testing for software bots and AI agents.
An agent that says "done" is not the same as a task that is done. We test bots and AI agents on the business systems they actually run on, many times over, and check the result in the system itself.
First results
Two AI agents, 17 everyday tasks in an open-source ERP and helpdesk (Odoo and osTicket), five runs each, every result checked in the database.
The failures were all one task: marking a project task as done. The cheaper agent moved it into the Done column instead of setting its status, then reported success. On screen it looked finished; in the database it was still open.
How we test
- Every task runs against a disposable copy of the system, reset to the same starting data before each run.
- Every task runs several times, because one run can look like success while five show the pattern.
- Results are checked in the system's database, not judged from screenshots.
- A person does each task by hand once, so the agent's effort can be compared with a human's.
Cases, method and results are open: github.com/Brunel001/bench.
Work with us
If you build AI agents for businesses, or run bots in production, and want to know how reliable they really are, write to founder@brunelconsulting.com.