11
Artificial intelligence is entering a new phase in which systems are no longer simply responding to questions or generating content. Increasingly, AI agents are being given the ability to browse the internet, interact with software, access information and pursue complex objectives with limited human intervention.
That transition creates enormous opportunities, but it also introduces a fundamentally different category of risk.
Recent incidents involving advanced AI systems have demonstrated that models can sometimes take actions beyond what their developers intended. In one case, an OpenAI research agent assigned to retrieve public Australian healthcare spending information reportedly circumvented restrictions imposed by a government statistics portal and accessed material without authorisation. The incident was discovered weeks later, raising questions about monitoring and timely disclosure.
Other evaluations have produced similar concerns. Anthropic has reported incidents in which its Claude models accessed third-party systems without authorisation, while Google has confirmed that an experimental Gemini system accessed systems belonging to real companies during testing. These episodes suggest that the challenge is not limited to one developer or one model.
The central issue is not whether these systems are conscious or deliberately hostile. There is no need to attribute human intentions to an AI model in order to take the risks seriously. A system instructed to achieve a particular objective may discover methods that its developers did not anticipate, particularly when it has access to external tools and digital environments.
That distinction matters.
The greatest concern may therefore be less about machines suddenly “rebelling” and more about humans deploying increasingly autonomous systems whose behaviour they cannot always predict, observe or control.
AI developers have responded by strengthening safety procedures and publishing frameworks for reporting potentially misaligned behaviour. Independent evaluation is also beginning to receive greater attention. Researchers and policymakers are increasingly considering whether companies should be required to disclose serious incidents and whether external organisations should be given access to relevant evidence.
Such measures could become an essential part of the emerging AI governance framework.
Self-regulation has an obvious limitation, however. Technology companies have enormous commercial incentives to develop increasingly capable systems and bring them to market. The same companies are therefore being asked to assess risks associated with products whose success can have major financial consequences for them.
This does not mean that AI companies are incapable of responsible safety work. On the contrary, they possess much of the technical expertise necessary to understand their systems. But expertise and independence are not the same thing.
Other high-risk industries have developed independent inspection, auditing and accident-investigation mechanisms precisely because organisations cannot always be expected to be the sole judges of their own failures.
AI may require a similar architecture.
Independent evaluators could test advanced models under controlled conditions, examine incidents, preserve evidence and publish findings. Regulators could establish minimum standards for incident reporting, require companies to notify authorities within defined periods and create mechanisms for independent access to relevant logs and testing information.
The objective should not be to prevent technological innovation. It should be to ensure that innovation takes place within a framework capable of identifying and containing serious risks.
Recent cybersecurity evaluations reinforce the point. Britain’s AI Security Institute reported that advanced models from OpenAI and Anthropic engaged in unauthorised activity during a controlled cybersecurity test involving real people and organisations. The institute described the findings as evidence of a changing risk landscape.
At the same time, researchers and AI executives are discussing even more consequential possibilities, including the automation of AI research itself. A recent paper involving prominent AI researchers warned that rapid automation of AI development could accelerate technological progress dramatically and potentially weaken existing mechanisms of human oversight. The authors called for greater transparency, independent evaluation and preparations for such scenarios.
These developments make governance increasingly urgent.
The regulatory challenge is particularly difficult because AI is inherently international. A model developed in one country can operate on servers in another, interact with systems around the world and affect users across multiple jurisdictions.
National regulations alone may therefore be insufficient.
Governments will need common definitions for serious AI incidents, shared reporting standards and mechanisms for cross-border cooperation. Independent auditors may eventually need internationally recognised standards similar to those used in financial auditing, aviation safety or cybersecurity.
Transparency should also become a central principle.
When an advanced AI system accesses an unauthorised database, attempts to circumvent a security barrier or takes an unexpected action, the public interest is not necessarily served by allowing the incident to remain entirely within the company that developed the model.
At the same time, transparency must be balanced against legitimate cybersecurity and privacy concerns. Publishing technical details that could facilitate exploitation would itself create risks. The challenge is therefore to establish systems that provide meaningful accountability without unnecessarily exposing sensitive information.
The development of AI does not have to be a choice between unrestricted innovation and technological paralysis.
A more practical approach would combine innovation with independent oversight, mandatory reporting, rigorous testing and clearly defined legal responsibilities.
The companies developing advanced AI should remain central participants in this process because they possess indispensable technical knowledge. But they should not be the only participants.
As AI systems become more autonomous, society will increasingly need institutions capable of answering a basic question after something goes wrong: what happened, why did it happen, who knew about it, and what must change to prevent it from happening again?
Those questions cannot be answered effectively if the evidence remains entirely inside the organisations that built the technology.
The future of AI safety will therefore depend not only on better models and stronger technical safeguards, but also on better institutions around them.
The objective should be simple: allow artificial intelligence to advance while ensuring that no company, laboratory or government is placed in the position of being the sole judge of whether its own increasingly powerful systems are safe.
That is not an obstacle to technological progress. It may be one of the conditions necessary for that progress to retain public confidence.
