The distinction worth holding on to: a tool that produces text hands you something to check. A system that takes actions has already done something.
That difference is the whole design problem.
Where it genuinely works
Triage and routing
Reading incoming enquiries, classifying them, and routing to the right person or queue with a suggested priority.
Works because the objective is clear, errors are visible immediately, and the cost of a misrouted enquiry is that somebody forwards it. Consequence is low and volume is high, which is the ideal combination.
Gathering information across systems
A customer rings. Pulling together their recent orders, open tickets, payment position and last conversation into one summary before the call is answered.
The system is reading rather than changing, which removes most of the risk, and the person on the call verifies as they go.
Preparing work for approval
Drafting the reply, populating the form, preparing the quote from the rules, assembling the report. A person reviews and sends.
This is where most of the practical value currently sits: the system does the assembly, the person supplies the judgement.
Monitoring and flagging
Watching for patterns and raising them: an unusual invoice, a customer whose ordering has stopped, a support ticket that has gone quiet, a contract approaching renewal.
Flagging rather than acting. The output is somebody's attention, which is exactly the right consequence for something that may be wrong.
Where it goes badly
Anything financial without review
Payments, credit decisions, pricing changes, refunds. The consequence of being wrong is direct and the recovery is awkward.
Anything reaching a customer unreviewed
A confidently wrong reply sent under your name is worse than a slow reply. Customers forgive delay and remember being told something untrue.
Anything irreversible
Deletion, cancellation, submission to a third party. If it cannot be undone, it needs a person.
Anything where you could not tell it was wrong
The most important category and the least obvious. If the output is plausible and you have no way to verify it, you have added risk rather than capacity.
A person who makes a mistake makes one. A system that makes the same mistake makes four hundred before anybody notices. Speed changes the shape of the risk.
The controls that matter
Least privilege. The system gets access to exactly what it needs and nothing more. If it only needs to read, it does not get permission to write.
Rate limits. Bound how many actions can be taken in a period. This is the control that turns a runaway error from a catastrophe into an incident.
Reversibility. Prefer actions that can be undone. Where they cannot, insert a person.
A complete log. Every action, with what triggered it, what was done, and what the result was. This is not optional — see why audit logging matters.
Named accountability. A person responsible for what the system does. Not the vendor, not the system.
A stop. Somebody must be able to switch it off in seconds, and must know how without looking it up.
Sampling. Regular review of a sample of actions, by a person, even when nothing has gone wrong. This is how degradation is detected before it becomes visible.
The graduated approach
The sequence that works:
- Suggest only. The system proposes; a person decides every time. Run this long enough to measure how often the proposal is accepted.
- Act with approval. The system prepares the action; a person approves with one click. Faster, still fully controlled.
- Act with review after. The system acts; a person reviews a sample. Only for reversible, low-consequence actions with a good measured record.
- Act unsupervised. Only for actions where being wrong genuinely does not matter, and always with logging and rate limits.
Most business processes should stop at stage two. That is not timidity — stage two captures most of the time saving with almost none of the risk.
The honest position
These systems are useful and they are not reliable in the way ordinary software is reliable. Conventional software fails predictably; these fail plausibly, which is a different quality of failure and requires different controls.
They also change. A service updated by its provider can behave differently next month against the same inputs, which means acceptance testing is a periodic activity rather than a one-off.
Treat them as a capable new colleague who is fast, tireless, occasionally confidently wrong, and does not know when they are out of their depth. That framing produces the right supervision.
Our automation team builds these into business processes for UK companies, with the controls above as a starting position rather than an afterthought. Start a conversation, or read the implementation roadmap first.
