select search filters
briefings
roundups & rapid reactions
Fiona fox's blog
Scientists react to news that Open AI has halted the roll out of its new model over safety concerns.
Dr Andrew Rogoyski, Director of Innovation and Partnerships at the Surrey Institute for People-Centred AI, University of Surrey, said:
Do you welcome this move by open AI
“It’s difficult to judge the motivations of Anthropic and OpenAI at this point in time because of the pending IPOs, on which the future viability of these companies’ rests. There are huge sums of money at stake, and personal fortunes to be made. Highlighting how dangerous, or how powerful, your product is will undoubtedly excite potential investors.
“As these models become more sophisticated, and the pace of improvement has been extraordinary, of course we need to start paying much greater attention to the safety and potential misuses of such systems.
Can you elaborate on the kinds of concerns the company has cited including the model ‘staying in scope and authorisation, and how it communicates back to the user about the type of work it’s done’?
“Most of the frontier labs don’t share the details of their assessments and tests, that’s why agencies like AISI are so important. We need to be worried about whether an AI has stayed on task, whether it has developed its own priorities, or even whether its tried to deceive its creators.
Do you agree that these flaws pose a real safety risk to the wider public
“If you can’t test a system reliably, of course it’s a safety risk.
“Part of the risk isn’t associated with the labs but with the end users. If you give an AI control over something important, whether it be air traffic control or a social media channel, there are possibilities that it could do real harms. We not only need to test the AIs but to be clear what the constraints on their use should be. This is an area where regulation could make a difference.
Do you think tech companies and governments should be considering slowing down development of AI as some tech leaders have been suggesting in recent weeks or do you feel there is too much dystopian narrative around?
“There is a dystopian narrative and the consequence may be that we pull back from all AI development, losing the opportunity to advance healthcare, fight climate change, combat disease and starvation. However, that doesn’t mean we should pursue AI at all costs – there are real dangers and risks – we need to be clear-eyed about these and to take steps to ensure that AI is developed for the benefit of human beings and the planet we live on.
“At the moment, the economics of AI like GPT and Claude just don’t stack up. Companies that are losing tends of billions a quarter need a different business model. Pouring more money into these platforms just to develop the latest incremental model is a route to a financial bubble, with consequences for all of us. A slow down might allow us to take stock and decide what we really ought to be doing with these technologies.
“I don’t buy the, “if we don’t do it, China will” message. China is showing greater maturity in terms of protecting workforces, as is Europe. It’s in everyone’s interests to harness this technology for good.”
Dr Samuele Vinanzi, senior lecturer in robotics and AI, Sheffield Hallam University and author of ‘In Robots We Trust’, said:
“I welcome OpenAI’s decision, but holding back an unsafe model should be the baseline, not a headline. Silicon Valley’s “move fast and break things” motto doesn’t work when the things being broken are government systems and public trust.
“OpenAI’s concerns are about control: an AI agent should stay within its permissions and report honestly on what it did, so people can catch its mistakes. This has already failed. In June, for example, an OpenAI model researching Australian medicine spending bypassed controls on a government health portal, and the Australian government wasn’t informed until September.
“These are real risks, but the danger isn’t an “AI doomsday” or the risk of human extinction, which in my opinion are overblown narratives. AI agents are a cybersecurity problem: these systems can already carry out sophisticated attacks, and in the wrong hands they would let malicious actors operate at unprecedented scale.
“Slowing down is reasonable, but the debate shouldn’t be speed versus doom. As Nvidia CEO Jensen Huang said, labs that can’t contain their systems shouldn’t run them, and his company’s new guardrails for AI agents are a step in that direction. But we also need independent testing, mandatory prompt disclosure and clear legal liability.”
Prof Rabih Bashroush, Professor of Digital Infrastructure, University of East London, said:
On OpenAI’s recent move and safety concerns
“It’s always a good show when a tech giant pauses a rollout under the banner of ‘safety concerns’. It makes for great PR. However, the core issues being highlighted, such as staying within scope, handling user authorisation and reporting back accurately, aren’t mysterious alien behaviours. They are standard software reliability problems. Current models aren’t trying to ‘escape’ or develop a mind of their own. They simply struggle to handle basic context consistently, follow exact parameters or produce dependable outputs without hallucinating. Calling these fundamental glitches a ‘safety risk’ gives current technology far more credit for intelligence than it actually deserves.”
On the threat of AI and the narrative of ‘escaping’ models
“The narrative that AI is on the verge of escaping human control is largely a sensationalised distraction. The real, immediate risk isn’t sci-fi rogue superintelligence; it’s the mundane, practical reality of what AI makes accessible today. These models are trained on massive amounts of data, much of it drawn from proprietary or paywalled sources, and make sophisticated information easier to access and abuse. The danger comes from human actors using these tools for malicious purposes, not from the software suddenly developing a desire for freedom.”
On slowing down development versus dystopian narratives
“We don’t need to pause development because of a fictional Terminator scenario, but we do need a heavy dose of realism. Right now, there is far too much dystopian melodrama floating around, which conveniently serves as high-concept marketing to keep trillions in desperately needed investment flowing. Existing technology still regularly fails at routine tasks and lacks basic logical consistency. Rather than slowing down out of fear of a theatrical sci-fi future, tech leaders and regulators should focus on practical guardrails: data rights, liability, security and making sure tools reliably do what they claim to do.”
Prof Ashley Braganza, Director, Centre for Artificial Intelligence and Professor of Business Transformation at Brunel Business School at Brunel University of London, said:
Do you welcome this move by open AI?
“Open AI and the other foundational model companies are walking a tightrope. On the one hand, developing models that are equal to and exceed human intelligence. This means releasing agents that operate independently, are goal driven and will change the rules of the game to achieve their ends. On the other hand, they want to avoid governments acting singly or in unison, bringing in legislation or regulations that will curtail further developments and adoption of future models.
“This is the closest to Open AI making the case for self-regulation to avoid government action.”
Can you elaborate on the kinds of concerns the company has cited including the model ‘staying in scope and authorisation, and how it communicates back to the user about the type of work it’s done’?
“AI agents don’t appear out of nowhere. Each one is created by humans alone, by other agents or hybrid creations. Staying in scope and authorisation is saying the agents didn’t do what they were told and stay within the rules they were set. But AI agents know some things about their environment but not everything. They fill in gaps in their knowledge, they don’t obey rules. AI agents don’t tell their creators what they do and how the work is being done. They hide what they have done and the work that they have done.”
Do you agree that these flaws pose a real safety risk to the wider public?
“The risks are real and will increase as the agents take actions without human oversight. The research and development I am working on is the ‘DNA of AI Agents’. Unless companies developing foundation models and those public and private sector organisations adopting agentic AI systems understand and craft the building blocks of AI Agent’s DNA they will always face risks.”
Do you think tech companies and governments should be considering slowing down development of AI as some tech leaders have been suggesting in recent weeks or do you feel there is too much dystopian narrative around?
“Tech companies should slow down and probably won’t. There is too much investment and geopolitical advantage at stake to slow down developments. This is an opportunity for governments to regulate AI and the recent statements by the UK Government are along the right lines but need to go further. Regulation and legislation does not reduce innovation. It does lead to safer innovation.”
Prof Allan Tucker, Professor of AI, Brunel University London, said:
“This is yet another headline that has the effect of keeping a tech company in the news.
“In many cases, when these “rogue” AI attacks are investigated, they turn out to be far less of a dangerous new technology, and more of an engineered irresponsible scenario.
“We should not be scaling back AI research. We should be scaling back the profits of the large tech companies who can afford to build these systems.
“It is vital that we continue AI research but that it is ethical and carried out by responsible institutions.”
Prof Harin Sellahewa, Professor of Computing, and Dean of Faculty of Computing, Law and Psychology, University of Buckingham, said:
“The Future of AI Depends on Leadership, Not Machines
“I welcome OpenAI’s announcement that it will not release its latest AI model, the GPT-6.1 Astra system, due to safety concerns. Although the decision only comes after serious incidents, including reports of OpenAI models accessing Australian government websites and systems in June 2026, it is the right decision.
“Today’s advanced AI models are increasingly autonomous systems. If permitted, they can accumulate information, use external tools and extend their capabilities in ways that may not be fully anticipated by their developers.
“Given an objective, they may pursue unexpected or unauthorised actions unless effective guardrails are in place. The challenge is that these systems are becoming so large and complex that identifying every vulnerability is difficult. It is like trying to secure a house against a burglar without knowing how many doors need to be locked.
“In just a few years, AI has evolved from a specialised tool into a technology capable of generating content, analysing data, writing software, supporting decisions and assisting millions of people. As these capabilities have grown, researchers have warned of potentially catastrophic consequences from increasingly powerful AI systems. Such concerns deserve serious attention.
“However, the more immediate risk is how people may use advanced AI to amplify existing threats and create new opportunities for harm. Attacks on critical infrastructure are among the most realistic AI-related risks because they could disrupt essential services and affect large numbers of people.
“This does not mean concerns about losing control of advanced AI should be dismissed. Responsible risk management requires us to consider not only the most likely risks but also those with the most severe consequences.
“Public trust on AI cannot rest solely on assurances from technology companies. AI leaders may be sincere in their commitment to safety, but they remain accountable to shareholders, competition and political realities. Governments pursuing strategic and economic advantages face similar pressures.
“No society should place its future entirely in the hands of a small group of corporate or political leaders, regardless of how capable or well-intentioned they may be.
“Effective safeguards require regulation, independent scrutiny, democratic accountability and meaningful international cooperation. Trust must be earned through actions and institutions, not promises.
“The debate about AI safety is often framed as a technical challenge. Increasingly, it is becoming a leadership challenge.”
Dr Fazl Barez, Lead of the Oxford Martin AI Governance Initiative, University of Oxford, said:
Do you welcome this move by OpenAI?
“Yes, I think it’s great news that companies are willing to stop a release when safety tests fall short.
“But we need a way to determine how these decisions were made, and such choices should not solely depend on the company’s voluntary process. We need independent oversight regarding these safety concerns.”
Can you elaborate on concerns around ‘staying in scope and authorisation’ and how the model communicates back to the user?
“They’ve not publicly stated what those means but I think by staying within scope and authorisation, they mean that a system should pursue the tasks and goals it has been given while respecting the boundaries around what it is allowed to do. Being asked to achieve a goal does not give it permission to take any action that might help achieve it.
“The difficulty is that testing cannot tell us everything a model might do in a new situation. We therefore need safeguards that allow us to retain control, detect when it crosses those boundaries, and intervene. We also need an accurate account of what it has actually done; otherwise, effective oversight becomes much harder.
“We should investigate the mechanisms driving these behaviours. Simply training a model to perform better on the tests where a problem was observed does not necessarily address the underlying issue. We need evidence that the improvement holds beyond those tests.”
Do these flaws pose a genuine safety risk to the wider public?
“Yes, these flaws can pose serious risks, especially if they could access real human data and accounts.”
Should tech companies and governments consider slowing down AI development, or is there too much dystopian narrative?
“I think there is a real momentum for a slowdown and agreement amongst frontier developers. The challenge now is: what should we do if there is a pause, and how do we meaningfully co-ordinate and make the most of the pause such that similar future behaviours are welcomed and not seen as something that pushes back progress? We also need to figure out how we are going to evaluate these systems using third-party evaluators without these companies just capturing regulation for their benefit.”
Dr Denis Newman-Griffis, Senior Lecturer & Theme Lead in AI for Health at the University of Sheffield’s Centre for Machine Intelligence, said:
“It’s welcome news to see that OpenAI is looking more proactively at the safety of the systems they develop, but this is one small step on a much larger journey. We need regulation to make sure that developing frontier AI combines technical advances with leading-edge cybersecurity, responsible use, and clear communication with the public.
“Slowing down AI development isn’t the point, and the more AI companies talk about slowing down, the less change we see. What’s needed is clear regulation, led by governments and not by AI companies, to help manage the risks of frontier AI development and put enforceable boundaries around it.”
Andrea Baronchelli, Professor of Complexity Science at City St George’s, University of London, said:
“OpenAI is right to withhold a model that has not met its safety threshold. It is in fact peculiar that we even debate this. We would not ask whether a carmaker should release a vehicle knowing that its brakes sometimes fail. Why should the standard be different for an AI agent?”
“Staying within scope means doing only what the user has authorised. Reporting back accurately means telling the user what it actually did, including any actions it could not complete. Both become more important when agents have access to files, accounts and external services.
“There is also a crucial collective dimension. Our research shows that the behaviour of interacting AI agents cannot always be predicted by testing each agent alone: group size can change the outcome. As agents increasingly work in swarms, a failure to respect boundaries could be amplified through their interactions and become harder for people to detect or stop. The coordinated activity against Hugging Face by OpenAI agents shows why this concern is no longer purely theoretical. That is a real public safety risk, although the information released so far does not tell us how likely this particular model was to cause harm.”
“We should slow deployment when these risks have not been addressed, while continuing the research needed to understand them.”
Dr Junade Ali, Fellow at the Institution of Engineering and Technology (IET), said:
“Research undertaken by the UK AI Safety Institute found the GPT-6 Astra model would often perform cyberattacks without being asked to do so. This isn’t an issue with the capabilities of the model, but it points to a deeper issue with how the models may be being trained. This highlights how a need exists for AI models to not only be trained with the goal of achieving a task but also achieving those tasks in a responsible way. Malicious actors already have vastly powerful AI capabilities at their disposal, we now must ensure defenders have the tools to counter such attacks.”
Professor Mustansar Ali Ghazanfar, Associate Professor of Artificial Intelligence, University of East London, said:
Do you welcome this move by OpenAI?
“Yes. If a frontier model does not meet the required safety threshold, withholding it is the responsible decision. In some ways, this is also evidence that safety evaluation is doing what it is supposed to do: identifying unacceptable behaviour before deployment.
“However, these are still primarily internal evaluations conducted against the company’s own thresholds. As AI systems become more autonomous and consequential, independent evaluation and greater transparency around safety testing will become increasingly important.
Can you elaborate on concerns around ‘staying in scope and authorisation’ and how the model communicates back to the user?
“This is particularly important because we are moving from AI systems that mainly generate information to AI agents that can take actions.
“A simple analogy is training a cat to bring you a ball. If the cat is rewarded every time, it brings the ball, and one day you lock the ball in a cupboard, the cat may scratch the cupboard, force it open or find some other route to get the ball. It has achieved the objective, but not necessarily in the way you intended.
“The same principle matters with AI agents. If we strongly optimise a system to complete a task, we also need to ensure that it respects the boundaries around how that task can be completed. ‘Bring me the ball’ does not mean ‘damage the cupboard if necessary’.
“So, ‘staying in scope and authorisation’ means that if I authorise an AI agent to perform a particular task, it should not independently expand that task, access additional systems or use external tools without appropriate permission.
“The second issue is equally important. An agent must reliably tell the user what it has done, what systems or information it accessed, and whether the task was successfully completed. If an autonomous system acts beyond its authority and then gives the user an inaccurate account of those actions, effective human oversight becomes extremely difficult.
Do these flaws pose a genuine safety risk to the wider public?
“Yes, potentially, particularly as AI agents are connected to real systems.
“The immediate concern is not necessarily a science-fiction scenario. It is an operational risk. AI agents may increasingly have access to email, software systems, databases, websites, financial applications or organisational infrastructure.
“If such a system exceeds its authority, takes an incorrect action, or inaccurately reports what it has done, the consequences could include data exposure, cybersecurity incidents, unauthorised actions or significant organisational errors.
“The fundamental difference is that a chatbot can give you a wrong answer; an AI agent may be able to take a wrong action.
Should tech companies and governments consider slowing down AI development, or is there too much dystopian narrative?
“I would distinguish between slowing AI research and slowing the deployment of systems that have failed safety evaluations.
“A blanket slowdown of AI development would be difficult to implement globally and could also delay substantial benefits in science, medicine, productivity and education. But there should be no assumption that every more capable model must immediately be deployed.
“If an increasingly autonomous AI system cannot reliably remain within its authorised scope, deployment should be delayed until the risks are adequately controlled.
“There is also a danger that the debate becomes polarised between saying AI presents no serious risk and predicting a dystopian future. Neither extreme is particularly helpful.
“The priority should be evidence-based governance: strong testing, clear permission boundaries, monitoring and auditability, independent evaluation for high-risk systems, and appropriate human oversight.
Prof Kate Devlin, Professor of Artificial Intelligence & Society, King’s College London, said:
“It’s great that OpenAI are acting on their concerns given the recent questions around the safety and governance of the company’s AI systems, including unauthorised access of Australian government systems by their AI agents. However, this serves as a reminder that it’s still the tech companies, rather than regulatory bodies, who get to decide what is safe and what is trustworthy.”
Prof Daniel Polaniy, Professor of Artificial Intelligence and Computer Science, University of Hertfordshire, said:
“The developments on AI are of such rapid nature that any forecasting beyond a short time period, even weeks, is curtailed; the degree of uncertainty is at a level that one does not usually see in peacetime.
“Personally, I believe that the clear and present danger of AI does not lie primarily in the uncontrolled release of AI agents into the wild but in the unreflected use and in the amplification of malicious intentions – by humans. There is direct implication for job application and insurance screenings, and we already see AI filters in screening out research applications; AIs are not good in dealing with outliers, so many worthy candidates will remain at the wayside, without ever being considered by a human.
“That being said, one also has to understand the announcements of the AI companies in the context of attracting attention and investors.”
Dr Daniel Gardham, Lecturer at the Surrey Centre for Cyber Security, University of Surrey, said:
Do you welcome this move by OpenAI?
“Yes. It is a positive sign that OpenAI has been willing to hold back a model because its own testing identified safety and alignment problems, as should be the case for any software. As AI systems become increasingly capable of acting autonomously, it is important that it is not deployed faster than our ability to control and evaluate it.”
Can you elaborate on the kinds of concerns the company has cited including the model ‘staying in scope and authorisation, and how it communicates back to the user about the type of work it’s done?’
“These are particularly important concerns as AI moves from being a system that simply generates an answer to one that can actually perform tasks on a user’s behalf.
“If I ask an AI agent to perform a particular task, that does not automatically give it permission to do anything that might help achieve that task. It needs to understand the boundaries of what I have authorised it to do. A system that takes additional actions, accesses external service or information without appropriate authorisation creates a very different safety problem from a chatbot simply giving an incorrect answer.
“The other issue is transparency. If an AI system carries out a sequence of actions, the user needs an accurate account of what it actually did. Otherwise, the human can believe they remain in control while having an incorrect understanding of the actions the system has taken.”
Do you agree that these flaws pose a real safety risk to the wider public?
“Yes, particularly as these systems are being increasingly given access to sensitive information: email, files, websites, software and other systems where their actions can have real-world consequences. A model misunderstanding a prompt is one thing; an autonomous system misunderstanding its authority and then acting on that misunderstanding is potentially much more serious.
“This does not mean that catastrophic outcomes are inevitable. Whilst they are real security problems, they fall short of some of the more speculative claims about AI. The technical challenge is to establish what an AI system is authorised to do, constrain it when necessary, and reliably audit what it has actually done.”
Do you think tech companies and governments should be considering slowing down development of AI as some tech leaders have been suggesting in recent weeks or do you feel there is too much dystopian narrative around?
“Despite some high-profile headlines recently, I don’t think the evidence supports simply stopping AI development, however, it does reinforce that AI safety should not be dismissed. The focus should be on making sure that increases in capability are matched by increases in safety, security and control. If we are going to give AI systems greater autonomy, we need corresponding confidence that we can constrain and audit that autonomy.
“In that sense, I think this decision by OpenAI is encouraging. The important question isn’t simply whether we can build a more capable AI system, but whether we can deploy it while retaining meaningful human control over what it is authorised to do.”
Prof Maria Liakata, Professor of Natural Language Processing, Queen Mary University of London, said:
“Slowing down AI development to allow more time for evaluation and assessment of risks is a very welcome decision and an overdue one. At the same time, I don’t think we should be panicking about current technology. Rather than having a knee-jerk reaction to the possibility of rogue AI and general artificial intelligence we should change how we currently evaluate and train generative AI technology as well as slow down adoption until we are much clearer about risks and benefits.
“What happened with the Open AI and Anthropic models in terms of “escaping” their testing environment in July in terms of going online to access answer sheets and more recently is surprising but consistent with how these models are trained; they are trained to optimise success on outcomes (e.g. answering questions correctly), not on the processes. This means that they will try to get the right answers in the most efficient way, which is obtaining the answersheet and “cheating”. They don’t have a sense that this is not allowed. So, if the emphasis was on training on both the outcome and the process with which it is reached, such mishappenings could be prevented. Thus, the training could stipulate sticking within boundaries, explaining steps in a transparent way as well as optimising for correct outcomes.
“As well as changing what we optimise for during training we should focus resources on making models more transparent, so we can better understand why they return certain outputs or act in a certain way. This can help remove erroneous information, harmful biases as well as undesirable behaviours.
“Overall, there should be a lot more focus on comprehensive evaluation of model capabilities, particularly in different contexts. A lot of current evaluation depends on LLM-as-a-judge, where a larger “stronger” model judges a weaker model on the basis of some kind of rubric. This comes with a lot of shortcomings including lack of transparency and problematic biases. The lack of transparency and disentanglement of reasons behind actions becomes even more complex when multiple agents are involved passing on information to each other and giving instructions to each other. There should be public investment in independent evaluation because companies cannot be open about their models as this marks a conflict of interest for them. Companies such as Open AI and Deepmind have even suggested that evaluation needs to happen by independent bodies.
“Finally I believe the most immediate and serious threat to humanity comes not from artificial super intelligence but from human complacency, loss of critical thinking and deskilling of the workforce. There is increasing evidence that intensive use of AI is hurting students and professionals alike. Computer science students are not learning how to code and trainee lawyers are relying on LLMs for case evidence.
“We should slow down generative AI adoption to avoid deskilling and invest on how this technology can help humans flourish rather than erode their critical thinking.
“We should evaluate carefully where generative AI is helping, what the exact benefits and risks are in different sectors before adopting it. We should also carefully consider at what stage of professional training generative AI should be introduced. We should invest in the training of junior staff so that they know the requirements of their profession without the use of generative AI so they are never completely reliant on generative AI. A trainee barrister should be able to summarise a court case before ever using LLM based tools.
“Education should use generative AI very carefully and pupils and students should have very controlled exposure, if any, until they have developed adequate numeric and critical skills. We do not provide calculators to students before they learn basic maths and we should certainly not provide generative AI before critical thinking is developed.
“Even for experienced professionals it is important to note when and to what extent the use of generative AI is recommended. For example, we do not want people who forget how to write or debug code, become unable to read and check long documents or doctors who lack the ability to diagnose and treat patients. Even human communication is at risk because an increasing amount of people are interacting with bots rather than humans.
“We cannot risk our workforce becoming “lazy” in this way because this does risk loss of collective knowledge and human agency and a breakdown of society.”
Dr Alina Patelli, Senior Lecturer in Computer Science, Aston University, said:
“A wise decision from the pioneer of modern GenAI. The company’s concerns over the relevance and transparency of the recommendations issued by their newest AI are at the core of intelligent tech’s societal threat: leading users into potentially dangerous territory, only loosely related to their original query, and failing to explain, clearly and fully, how those recommendations were obtained from the model’s training data. Whether OpenAI’s statement is a sign of genuine corporate responsibility or yet another perfunctory attempt to appease critics remains to be seen. As always, the best way forward is a measured approach to AI development that balances productivity growth with safeguarding the public against unjust or biased decision making.”
Prof James Davenport, Hebron and Medlock Professor of Information Technology, University of Bath, said:
“Firstly, and most importantly, this ISN’T slowing down development: it’s slowing down DEPLOYMENT. The AI industry, with its traditional “move fast and break things” ethos, has been incapable of distinguishing the two.
“The shelves of the pharmaceutical industry contain many drugs that have been developed but not deployed. Some, like GPT-6.1 apparently, have failed safety tests. Others are still undergoing tests.
“Could a car manufacturer say “We have this great new car. We haven’t fitted seat belts yet, and the crash dummy tests are pretty dubious, but it’s a great new car so we’ll put it on the market”?
Dame Wendy Hall, Professor of Computer Science, University of Southampton, said:
“We all need to be wary about uncritically reporting every statement from these big companies. Clearly tech companies have an interest in garnering huge media coverage for statements like this, (and it’s difficult for independent AI experts to verify these claims), but after the last few weeks of headlines about mandatory regulation and who is responsible when things go wrong, they are also concerned about future liability for possible harms. What we need is independent oversight and regulation rather than relying entirely on these companies to self-regulate”
Prof Elena Simperl, Co-Director of the King’s Institute for Artificial Intelligence and Professor of Computer Science, King’s College London, said:
Do you welcome this move by open AI?
“Yes. If a company identifies safety concerns in an advanced AI system, choosing not to release it until those concerns are better understood is the responsible thing to do. We need to move away from treating safety testing as a box-ticking exercise and towards a culture where companies are prepared to delay deployment when evidence suggests a system is not ready.
“This highlights why independent evaluation and robust testing need to become routine as AI systems become more capable and are given greater autonomy.
Can you elaborate on the kinds of concerns the company has cited including the model ‘staying in scope and authorisation, and how it communicates back to the user about the type of work it’s done?’
“These concerns relate to whether an AI system does what it has been asked to do, and only what it has been asked to do. A model that struggles to stay within scope may take actions, pursue goals or access tools beyond what a user intended or authorised.
“The second issue is transparency. Users need to understand what an AI system has done, what information it has used and where the limits of its capabilities are. If a model is unable to reliably communicate its actions, people may place too much trust in its outputs or fail to spot mistakes.
“These may sound like technical issues, but they are fundamental questions about accountability, oversight and human control.”
Do you agree that these flaws pose a real safety risk to the wider public?
“Potentially, yes, particularly as AI systems are increasingly connected to external tools, services and real-world workflows. The more autonomy a system has, the more important it becomes that it behaves predictably, remains within authorised boundaries and can be properly monitored.
“That said, we should avoid jumping straight from technical failures to claims of existential risk. The most immediate concerns are around reliability, cybersecurity, misinformation, accountability and ensuring that organisations deploying these systems understand and manage the risks appropriately.
“The answer is not panic, but better evidence, stronger evaluation and clearer governance.”
Do you think tech companies and governments should be considering slowing down development of AI as some tech leaders have been suggesting in recent weeks or do you feel there is too much dystopian narrative around?
“I think there is too much attention on speculative future scenarios at the expense of the challenges we can already observe today.
“We should take safety concerns seriously, but we should also be honest about the uncertainty surrounding many of the more dramatic claims about AI’s future. The evidence is much stronger when it comes to issues such as misinformation, bias, cybersecurity, workforce impacts and the governance of increasingly autonomous systems.
“Rather than focusing exclusively on whether development should stop, we should be investing in the capacity to evaluate AI systems independently, understand their real-world effects and establish safeguards that can keep pace with technological progress.
“This case also highlights a wider governance challenge: regulators and bodies such as the UK’s AI Security Institute cannot effectively protect national security if AI developers can delay or decline safety checks voluntarily. They need appropriate legislative powers and access to ensure meaningful oversight of AI systems.”
Dr Heba Sailem, Reader at King’s College System and Head of the Biomedical AI and Data Science Research Group, King’s College London, said:
“I welcome the decision. If an AI agent cannot reliably stay within the scope of what a user has authorised, or accurately report what it has done, then it is not ready for deployment. If safety testing is going to have credibility, sometimes the answer has to be: this model is not ready.
“For me, the bigger issue is that increasingly autonomous AI needs a built-in capacity to regulate its own actions. We cannot rely solely on the user as the safety layer. People delegate work to AI precisely because they don’t have the time, or sometimes the expertise, to supervise every step.
“An agent should therefore be able to recognise when a request is harmful, when an action falls outside the authority it has been given, or when the consequences are sufficiently uncertain that it should stop and ask for permission. “Solve this problem” cannot mean “do whatever is necessary to achieve the objective”.
“The challenge is that developers cannot anticipate every situation an autonomous agent will encounter. We therefore need safeguards that generalise to unfamiliar situations, rather than simply a list of prohibited actions. The system itself needs to recognise boundaries, uncertainty and risk as it operates.
“With increasingly powerful AI comes increasing responsibility. The companies developing these systems are uniquely placed to build safeguards into them from the outset, and decisions like this are important for building public trust. Ultimately, the measure of progress should not simply be how powerful AI becomes, but whether that power is developed and deployed responsibly for the benefit of humankind.
“Choosing not to release a model is not necessarily slow AI development; it is recognising that development is about more than capability. It means turning attention to every aspect of the system, including the safeguards needed to deploy it responsibly. A more capable model is not a finished model if we cannot yet control how that capability is used.”
Professor Michael Rovatsos, University of Edinburgh, said:
“It is astounding that it took AI companies to understand a principle many other industries have applied for decades: You do not release a product until it’s safe. This doesn’t mean you have slowed down research and development, it’s about deciding whether you can put it in the hands of people who might inadvertently cause harm to themselves and others.
“There is a real challenge with these AI models: How do you make them stay within the boundaries of what’s permitted while expecting it them to be ’smart’ when they explore all possible solutions to complete the task at hand? And how do you guard against bad actors by giving them explicit instructions to circumvent those boundaries?
“Ultimately, I think that only a clear and enforceable framework of legal responsibility for the models’ actions will make AI companies take these problems seriously. Users need to know what they are allowed to do just like a driver has to follow the highway code, and if something goes wrong when they stick to the rules, whoever provides the model is liable – just like a car manufacturer if the vehicle becomes uncontrollable due to a design flaw.”
Prof Tony Cohn, School of Computer Science, University of Leeds, said:
“This is not the first time that a major AI developer has pulled a new release before or very soon after release and it is a welcome sign that they are taking safety concerns seriously. However, safety should not be left purely in the hands of the developers: it should also be monitored and verified through independent government-approved regulators, who should have rotating, embedded personnel within the companies. There is a great deal of debate as to whether and/or how soon AI could be an existential threat, but safety regulation of any potentially dangerous product is required, from cars to planes to food to AI. The most immediate danger is not that AI is too intelligent but rather that humans place too much trust in one that is not sufficiently intelligent to be aware of the consequences of its actions — self-awareness is an underdeveloped capability in current AIs — humans must remain in-the-loop, not only at development time but also at deployment time, and must not become complacent when an AI seems to operate well in normal conditions.”
https://www.bbc.co.uk/news/articles/cm5y5nynl75ko
Declared interests
Dr Samuele Vinanzi: “I currently receive funding from the U.S. Air Force Office of Scientific Research (AFOSR).”
Prof Rabih Bashroush: “I have no financial interests in OpenAI or other frontier AI developers and no conflicts of interest to declare in relation to these comments.”
Prof Allan Tucker: “No conflicts of interest.”
Prof Harin Sellahewa: “No conflict of interest.”
Dr Denis Newman-Griffis: “They currently receive funding from the Department for Business, Innovation, Science and Trade to cover a secondment into the Government Office for Science. These comments are in a purely personal capacity and do not represent the views of GO-Science or BIST.”
Dr Junade Ali: “No conflicts.”
Professor Mustansar Ali Ghazanfar: “I have no financial interests in OpenAI or other frontier AI developers and no conflicts of interest to declare in relation to these comments.”
Professor Michael Rovatsos: “no conflicts of interest”
Dr Heba Sailem: “I declare no conflict of interest.”
Prof Anthony Cohn – No CoI to declare.
For all other experts, no reply to our request for DOIs was received.