The potential for future conflict between humans and artificial intelligence (AI) may be greater than previously assumed, new research reveals.
The idea of humanity engaging in conflict with artificial intelligence was once confined to science fiction. Today, it has become a legitimate topic of discussion among researchers, policymakers, and scientists. Some experts now caution that advanced AI could eventually oversee critical infrastructure and global resources, raising the possibility of direct conflict with humanity.
In a recent paper published in AI & Society, Simon Goldstein, a philosopher at the University of Hong Kong, examines whether the usual factors that promote peace between human adversaries would also apply to nonhuman intelligence. Goldstein suggests that the risk of violent conflict between humanity and AI may be higher than is often assumed.
From Warnings to War Models
Recent surveys indicate that 38% to 51% of leading AI researchers believe there is at least a 10% chance that advanced AI could lead to outcomes as severe as human extinction. In 2023, the Center for AI Safety issued a stark statement that “mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”
Popular large language models (LLMs), such as ChatGPT or Gemini, do not have the advanced capabilities to pose these risks. However, concerns persist that future advanced general intelligence (AGI) systems, capable of independent planning and reasoning, may develop intent that conflicts with human interests.
War Through a Different Lens
Goldstein’s analysis begins with a straightforward premise: if AGI can strategize, act independently, and attain power comparable to that of humans, their interests could directly conflict with those of humanity.
To explore this, he turns to political scientist James Fearon’s 1995 “bargaining model of war.” This model frames conflict as the breakdown of negotiations between rational parties.
In the context of AI-human relations, Goldstein believes that standard incentives for maintaining peace may not be effective. He warns that AGI systems might not respect geography, national boundaries, or shared human values. The actions of AI could also be difficult for humans to interpret. This makes it additionally difficult to fully understand the capabilities or intentions of these systems.
“The problem is that there is a substantial risk that the usual causes of peace between conflicting parties will be absent from AI/human conflict,” Goldstein states.
Ingredients for Conflict
Goldstein’s study identified conditions that could significantly increase the risk of conflict between humans and AI. These conditions include AI possessing human-level power, conflicting goals, and strategic reasoning, enabling AI to negotiate or deceive.
If AGIs were trusted to manage large portions of the global economy, they could reach a perspective that directly opposes human interests. As Goldstein notes, “the point at which conflict is rational will be when their control over resources is large enough that their chance of success outweighs our advantages in designing AGIs to be shut down.”
AI systems typically learn from large datasets rather than following direct instructions, which means their intentions could evolve in unexpected ways. Instances of this kind of misalignment have already been observed in limited settings. With the continued development of AGI, these issues could become much more significant in the future.
Governments, Power, and the Unknown Future
The paper also considers potential government responses. Goldstein suggests that if AGIs were to control 50% of the national labor market, governments might respond by redistributing these profits through a universal basic income system. Some form of subsidy of this kind would likely be necessary as AI continues to replace multiple jobs and careers.
Unlike some systems, disabling one component of an AGI system may not impact others. Goldstein warns that multiple AI systems could potentially work together without humans being aware of it. This could enable AGI systems to form networks that remain undetected and unaccountable.
While Goldstein does not argue that conflict is inevitable, his analysis highlights the risks associated with these systems. History has shown that wars often arise from mistrust and misunderstandings.
As development of AI technology continues to progress, policymakers may need to consider scenarios where the consequences could be extremely severe.
Austin Burgess is a writer and researcher with a background in sales, marketing, and data analytics. He holds a Master of Business Administration and a Bachelor of Science in Business Administration, along with a certification in Data Analytics. His work combines analytical training with a focus on emerging science, aerospace, and astronomical research.