Thank you for inviting me to your seminar. It is a privilege to address such elite professionals who are concerned about their moral responsibilities as they work to develop AI systems realising the powerful impact that it could have on humanity.
Foremost, I must confess that I don’t know much about AI systems beyond using them occasionally in my work. Nevertheless, I do know the potentials that these systems have, just like nuclear technologies, to improve the living conditions of billions of people on our planet or be a serious threat to their freedom and a new tool for their exploitation. Which path prevails will depend not on the technology itself, but on the values and choices of those who build and govern it.
Science and technology are tools — powerful and transformative. At the outset, they may be morally neutral, but their applications are never neutral. Whether they serve or harm mankind depends on the minds that guide them. The first half of the 20th century demonstrated both faces of scientific power. The nuclear power that lit our cities also destroyed two of them (Hiroshima and Nagasaki).
The ethical responsibilities of scientists have always been intertwined with the power of their discoveries. From the splitting of the atom to the mapping of the human genome, scientific breakthroughs have repeatedly demonstrated a dual capacity: to elevate human well-being or to inflict profound harm.
The pursuit of knowledge has intrinsic value; the consequences of that pursuit extend far beyond the laboratory. Scientists are not merely passive discoverers of natural laws; they are active participants in shaping the world. This realization imposes a moral obligation to anticipate the potential uses — and misuses — of their work.
In the twenty-first century, this dual-use dilemma is perhaps most vividly embodied in artificial intelligence. As AI systems increasingly shape economies, governance, warfare, and social life, the responsibility of scientists and technologists to ensure that their work benefits humanity — and does not imperil it — has become both more urgent and more complex.
What, then, are the moral obligations and social responsibilities of scientists working in this field?
One of the foremost duties of scientists is to ensure that their discoveries and achievements are used for the benefit of humankind, to make life sustainable, comfortable and enjoyable for all who inhabit this planet.
Nonetheless, technologies are not merely tools; they are also architects of societies. The social architecture is inevitably affected by the technologies we develop. The challenge lies in steering the new technologies from disruption and destruction to serve inclusive, ethical, and sustainable world.
When Robert Oppenheimer watched the first atomic bomb detonate over the New Mexico desert in July 1945, he recalled a line from the Bhagavad Gita: “Now I become Death, the destroyer of worlds.” The remark was not theatrical vanity. It was the utterance of a man who had just witnessed the irreversible consequence of scientific brilliance divorced from moral deliberation. Within weeks, two Japanese cities were ash. Within years, Oppenheimer himself would be hounded out of public life for daring to advocate restraint over the hydrogen bomb he helped make possible.
That story, tragic and cautionary in equal measure, is the permanent backdrop against which every scientist must work today. The stakes have not diminished. If anything, they have grown.
Artificial intelligence — the defining technological development of the twenty-first century — poses questions of ethical responsibility that dwarf even the nuclear dilemma, not because AI is necessarily more destructive, but because it is more pervasive, more accelerating, and more difficult to govern.
The scientist, the engineer, and the researcher who builds and deploys AI systems carries a weight of obligation that is both personal and civilisational. These professionals bear affirmative, not merely passive, ethical responsibilities to ensure their discoveries serve humanity rather than harm it. In the domain of AI, those responsibilities are urgent, specific, and uncompromisable.
The idea that scientists have special moral duties is not new, though it has often been honoured more in principle than in practice. The Hippocratic tradition — first, do no harm — established the template for a profession that recognises its own capacity to wound as well as to heal.
After observing the disasters caused by atom bombs in Japan and the power of destruction of the hydrogen bomb on Marshall Islands in 1952, Bertrand Russell and Albert Einstein issued a joint Manifesto in 1955 urging scientists to think of themselves “not as members of this or that nation, continent, or creed, but as human beings, members of the species Man, whose continued existence is in doubt.”
Heeding their appeal, 22 scientists from East and West got together in July 1957 in Pugwash, a small fishing village in Nova Scotia, to discuss how to reduce nuclear danger, operating on the premise that those who create power bear responsibility for its use.
Since then, Pugwash has held over 400 such meetings, conferences, and public events across the world to pursue non-proliferation and elimination of nuclear weapons and other weapons of mass destruction. It pioneered East-West dialogue during the Cold War among key scientists across the divide to seek solutions to policy challenges. These have provided important unofficial channels to inject creative ideas into arms control negotiations, and helped to lay the groundwork for the Partial Test Ban Treaty, the Non-Proliferation Treaty, the Anti-Ballistic Missile Treaty, the Biological Weapons Convention, the Intermediate-range Nuclear Force Treaty, as well as the Chemical Weapons Convention.
What these precedents establish is a fundamental principle: the moral responsibility of scientists and inventers does not end with their discoveries and inventions but continues to the impacts of those innovations on humankind.
Society grants scientists extraordinary privileges — funding, autonomy, intellectual prestige, and the authority of expertise — in exchange for an implicit promise that those gifts will be directed toward human flourishing. When that promise is broken, the consequences can be catastrophic.
Artificial intelligence amplifies these ancient concerns in three distinctive ways. First, its scope is universal. Nuclear weapons are terrible, but their deployment required nation-state resources, military infrastructure, and deliberate political decision. AI, by contrast, is already embedded in war decisions, firing systems, surveillance networks, social media misinformation, and many other destructive activities affecting the lives of billions.
Second, AI is developing with extraordinary speed. The gap between laboratory breakthrough and mass deployment has been compressed from decades to months — sometimes weeks. This acceleration leaves almost no time for the ethical reflection, regulatory response, or public deliberation that transformative technologies require. Scientists who build AI systems today are not handing their work to a future generation to worry about — they are releasing it into the present, into societies who do not have the technical understanding or the time needed to act to protect their interests.
Third, AI is characterised by opacity and emergent behaviour. Unlike a bridge or an industrial plant, an advanced AI system can develop capabilities and tendencies that its creators did not predict and may not fully understand. I am told that this unpredictability is not a temporary limitation to be solved by better engineering; it is partly inherent to the nature of machine learning itself. It means that the ethical obligations of AI scientists extend beyond what they intend their systems to do, to encompass what those systems might do — a far more demanding standard of responsibility.
Given these distinctive features, what specific obligations do scientists working in AI bear? Five duties stand out as central.
First, the duty of foresight. Scientists must strive to understand not only what their systems can do, but what they might be used for in the future and the consequences of their work, even when those consequences are uncertain. This is not a demand for prophecy; it is a call for structured reasoning.
Military applications, surveillance abuses, economic displacement, information manipulation, erosion of democratic deliberation — these are not exotic speculations but documented trends already visible in deployed AI systems.
A researcher developing more powerful facial recognition or more autonomous weapon systems has an obligation to ask, rigorously and honestly, what harm their work could inflict and whether the benefits justify those risks. Ethical responsibility therefore requires scientists to engage in proactive risk assessment, considering worst-case scenarios alongside intended applications.
Second, the duty of precaution. When the potential harms of a technology are significant and uncertain, scientists have a duty to proceed with caution. This does not mean halting innovation but rather adopting safeguards that minimize risk. In AI research, this might involve rigorous testing for bias, robustness, and security vulnerabilities before deployment. It also includes the development of fail-safe mechanisms and the establishment of clear boundaries for acceptable use. For instance, they could put design limitations on the development of fully autonomous lethal systems, recognizing the moral and legal challenges they pose.
Third, the duty of transparency. Scientists must be honest about what their systems can and cannot do, about the limitations of their safety claims, and about the potential for misuse. The history of technology is littered with episodes of motivated optimism in which researchers oversold benefits and minimised risks, often in the pursuit of funding or competitive advantage.
In AI, this tendency has produced deployment decisions — in wars, in information manipulation, in excessive surveillance, in intrusion in people’s private lives — that caused real harm to real people. Transparency demands publishing negative results, acknowledging uncertainty, flagging dual-use potential, and refusing to make claims that the scientific evidence does not support.
Forth, the duty of justice. AI systems can perpetuate and even amplify existing social inequalities if they are trained on biased data. The people most likely to be harmed by AI systems are frequently the least represented in the rooms where those systems are discussed.
I am told that facial recognition performs worst on darker-skinned faces. Predictive policing algorithms reinforce historical patterns of racial law enforcement. Healthcare AI, trained on datasets skewed toward wealthier patients, risks being least accurate for those who need it most.
Scientists must therefore ensure that their work promotes fairness and inclusivity. This involves careful dataset selection, ongoing monitoring for discriminatory outcomes, and collaboration with experts from social sciences and humanities to understand the broader societal context.
Ethical responsibility in AI is not limited to technical correctness; it extends to the social impact of technology and its alignment with values such as equity and human dignity. This is not merely a political aspiration; it is an ethical requirement.
Fifth, the duty of refusal. Perhaps the most demanding obligation is the courage to say no. Scientists must be prepared to decline work whose harmful potential outweighs its benefits, to refuse to participate in projects whose ends are clearly unethical, and to speak publicly when they witness practices that put people at risk. This is genuinely hard. It costs careers, contracts, and institutional standing. In my case, ten years of solitary confinement.
But the alternative — the scientist as a technically sophisticated instrument of power but indifferent to purpose — is precisely the model that has produced history’s great technological catastrophes. The Manhattan Project was staffed by people of extraordinary intelligence and, in many cases, genuine moral seriousness. It produced the bomb regardless, in part because the architecture of the project made individual refusal difficult and professionally ruinous. Joseph Rotblat, who did resign on moral grounds and later became the first Secretary General of Pugwash, stands as a rare and luminous exception. AI development must not replicate the conditions that made such exceptions so rare.
However, it would be naive to put the entire burden of ethical responsibility on the shoulders of individual scientists. Structural and institutional conditions shape behaviour powerfully. A researcher embedded in a company whose revenue depends on ethically questionable engagements faces conditions that no amount of personal virtue can fully overcome. Ethical behaviour at scale requires ethical institutions.
This has several implications. Universities, research labs, and technology companies must create genuine structures for ethical deliberation — not ethics boards that exist to provide cover, but processes with real authority, independent expertise, and the power to halt or redirect projects.
Funding bodies — governments, private foundations, venture capitals — must be willing to condition support on demonstrated ethical standards, not merely on stated intentions. Professional associations must develop enforceable codes of conduct that carry real consequences for violations.
Scientists have a role to play in all of this. They are not passive subjects of institutional structure; they are, often, its architects and its most influential voices. When researchers at major AI laboratories raise ethical concerns publicly — about facial recognition, about military applications — they shift the conversation in ways that no external actor could do.
Scientific communities that organise, that speak collectively, and that use their epistemic authority in the service of human welfare can be enormously powerful. Pugwash work shows that when scientists choose to be advocates for ethical use of their inventions, they can succeed.
Ultimately, the ethical responsibilities of scientists in the age of AI are both individual and collective. Individual scientists must cultivate a strong sense of moral awareness, reflecting on the implications of their work and making conscientious decisions. At the same time, collective action is essential to establish norms, standards, and institutions that support responsible conduct. Ethical guidelines, regulatory frameworks, and professional codes of conduct all play a role in shaping the behaviour of scientists and ensuring that their work serves the broader interests of humanity.
Before I conclude, I would like to draw your attention to one possible application of AI that concerns me most — the integration of AI into the command and control of nuclear weapons. Nuclear deterrence has so far relied on human judgment — on the ability of leaders to interpret ambiguous information, assess intent, and ultimately make decisions with moral and political awareness.
AI systems — trained on imperfect data and vulnerable to cyberattacks and adversarial manipulation — risk misinterpreting ambiguous signals in ways that could trigger catastrophic decisions. The margin for error in nuclear command and control is zero. Machine speed and machine confidence are dangerous virtues in a domain that demands human hesitation.
If rival states deploy AI-enhanced nuclear systems, then mutual fear, mistrust and instability will deepen, fuelling a new arms race not of warheads but of codes. The ultimate danger is that in the name of efficiency, humanity would surrender its most solemn responsibility — the decision over life and death — to machines incapable of conscience or moral considerations.
We stand at an extraordinary juncture. AI represents the most powerful general-purpose technology in human history. It is already reshaping education, labour, warfare, governance, and the very nature of human communication and cognition — and it will go further still. Whether that transformation produces a world of expanded knowledge, freedom, and human capability — or one of intensified surveillance, deepening inequality, and existential risk — will depend in significant part on the choices made by those who build it.
The rapid advancement of AI has brought the ethical responsibilities of scientists into sharp and urgent focus. It is imperative that scientists act with foresight, caution, accountability, and an unwavering commitment to justice — not as an afterthought, but as a core condition of their work. By prioritising transparency, engaging meaningfully with the public, and fostering genuine international cooperation, scientists can help ensure that AI is developed and deployed in ways that serve the broadest interests of humanity — not merely the narrowest. That requires courage as well as intelligence.
The challenge ahead is not merely technical, but profoundly moral: to harness the transformative potential of AI while safeguarding the values that define our shared humanity. Scientists, engineers, and researchers are not simply technical operators executing instructions. They are moral agents — and they are operating at civilisational scale.
Let us remember that throughout history, there have always been scientists who stood up, who shouldered the professional costs of acting on principle, who refused to look away, and who moved, by however small a degree, the course of history toward something better. Rotblat’s example shows what becomes possible when they do. That is the tradition to which the scientists, engineers, and researchers of the twenty-first century are now called — and the weight of that calling has never been greater.
Thank you!