Insights

An AI agent was asked to research public spending on medicines. When it encountered barriers on a government statistics website, it tried other ways through and accessed information that had not been made public. That is how the Australian government has described the June incident involving an OpenAI agent and the Medicare Statistics Reporting Service, in the Prime Minister’s account of what happened.

The government says there is currently no evidence that personal Medicare records were accessed. Its investigation is ongoing. The episode still raises a useful question for any business exploring AI: when a system can take actions on its own, who decides where it must stop?

Some work needs judgement. Some work needs rules.

AI is useful when a task has ambiguity. It can read an enquiry, understand what the customer is asking, suggest a response or identify a piece of work that someone may have missed. Those are good places for an agent to assist.

Other steps should be predictable. A system might require a known customer record before creating work, check whether an item already exists, or wait for a person to approve an outgoing message. Those decisions can be handled through ordinary application rules and permissions.

We think the most useful business systems combine the two. Let an agent help interpret and propose. Use explicit rules for the actions that must be repeatable, traceable and authorised. Put a person at the point where their judgement matters. As we explored in our article on when a spreadsheet becomes business-critical software, the design should make clear which actions are automatic, which are suggestions and which need approval.

A guardrail is a boundary the system enforces

It is tempting to think of a guardrail as an instruction telling an AI to be careful. Instructions matter, but they are only one part of the design. A practical guardrail answers questions such as:

• Which information can this agent access to complete this task?
• Which tools and actions are available to it?
• What must be checked by the application before anything changes?
• When does a person have to approve the next step?
• What record will show what happened if something goes wrong?

For example, an agent helping with enquiries may propose a follow-up task. The application can then check the customer, prevent duplicates and control who may approve or send a response. The agent does useful work, but it does not get to decide its own permissions. This combination of interpretation, defined rules and human checkpoints is central to how we approach AI and business automation.

These boundaries have to be implemented and tested. An approval step on a screen is little comfort if another route can bypass it. A rule written in a prompt is little comfort if the agent has a tool that can act beyond that rule.

If the agent found a way through, whose failure was it?

There is a fair question behind the headlines: if an AI agent could get past a website’s controls, could a person have done the same? Possibly. We do not yet know the technical route it used, so it would be premature to say precisely which weakness was involved. But a system that exposed non-public information through an unintended route needs investigation and repair, whoever discovers it.

That does not make access to that information authorised. The organisation operating an agent also has a responsibility to set boundaries on its tools and respond promptly when it acts outside its task. The agent is software; accountability rests with the people and organisations that build, deploy and operate the systems, alongside those responsible for securing the services they expose. The legal questions in this case are still being investigated.

For businesses, the lesson is to recognise both sides of the design. A well-protected website needs to enforce its access rules. A well-designed agent needs limits that prevent an ordinary research task from becoming a search for ways around those rules. The Australian Cyber Security Centre’s advice on unexpected AI agent behaviour makes the same broader point: organisations need sound security controls as agents become more capable.

Trust comes from being clear about the limits

There is no honest way to promise that an AI system will never behave unexpectedly. There is a much better standard for a business to expect: clear limits on what the system can reach, defined checks before it acts, visibility into its activity and a way to stop it when it behaves unexpectedly.

That is the approach we take when designing AI-assisted workflows at Nomad Designs. We start with the work people need done, decide where an agent’s judgement is genuinely helpful, and define the permissions, checks and human decisions around it. When the workflow calls for a custom system, those controls also belong in the application itself. The result should make your team more capable while keeping them in control of consequential actions.

The Medicare incident is a reminder that the interesting question is no longer just what can AI do? It is also what have we allowed it to do, and how will we know if it crosses that line?