I was recently asked to speak briefly before a private screening of the movie "Oppenheimer". The audience contained many people who work on AI, and the organizer asked me to speak because I have worked both on AI and also at Los Alamos National Laboratory (in 1997 and 1998). Although my work at Los Alamos was unrelated to weapons, I did get to know some nuclear physicists a little, including one of the early pioneers of the hydrogen bomb. The following are a cleaned-up and slightly extended version of my remarks. They are, of course, far too short to do justice to the topic.

I was at a party recently, and happened to meet a senior person at a well-known AI startup in the Bay Area. They volunteered that they thought "humanity had about a 50% chance of extinction" caused by artificial intelligence. I asked why they were working at an AI startup if they believed that to be true. They told me that while they thought it was true, "in the meantime I get to have a nice house and car".

This was an unusually stark and confronting version of a conversation I've had several times. Certainly, I often meet people who claim to sincerely believe (or at least seriously worry) that AI may cause significant damage to humanity. And yet they are also working on it, justifying it in ways that sometimes seem sincerely thought out, but which all-too-often seem self-serving or self-deceiving. (I've also, of course, had many conversations with AI people who sincerely believe safety is a minor problem; I won't engage with that point of view today.) People working on AI but deeply worried about AI risks seem to be in a situation akin to participants in the Manhattan Project. What moral choices are available to someone working on a technology they believe may have very destructive consequences for the world?

Part of what makes the Manhattan Project interesting is that we can chart the arcs of moral thinking of multiple participants. It's tempting to adopt a stance of mostly judging those participants, but I think it's more useful to focus first on empathetically understanding what the arc was, from the inside; in the light of such understanding we may be better able to decide our own relationship to science and technology. Here are four caricatures. They are historically incomplete, but nonetheless useful as models to consider in the light of AI.

• Klaus Fuchs and Ted Hall were two Manhattan Project physicists who took it upon themselves to commit espionage, communicating the secret of the bomb to the Soviet Union. It's difficult to know for sure, but both seem to have been deeply morally engaged and trying to do the right thing, willing to risk their lives; they also made, I strongly believe, a terrible error of judgment. I take it as a warning that caring and courage and imagination are not enough; they can, in fact, lead to very bad outcomes. It's easy for us to criticize the espionage agents now as deeply confused; course, they would not have seemed confused to themselves.
• Robert Wilson, the physicist who recruited Richard Feynman to the project. Wilson had thought deeply about Nazi Germany, and the capabilities of German physics and industry, and made a principled commitment to the project on that basis. He half-heartedly considered leaving when Germany surrendered, but opted to continue until the bombings in Japan. He later regretted that choice; immediately after the Trinity Test he was disconsolate, telling an exuberant Feynman: "It's a terrible thing that we made".
• Oppenheimer, who I believe was motivated in part by a genuine fear of the Nazis, but also in part by personal ambition and a desire for "success". It's interesting to ponder his statements after the War: while he seems to have genuinely felt a strong need to work on the bomb in the face of the Nazi threat, his comments about continuing to work up to the bombing of Hiroshima and Nagasaki contain many strained self-exculpatory statements about how you have to work on it as a scientist, that the technical problem is too sweet. It smells, to me, of someone looking for self-justification¹.
• Joseph Rotblat, the one physicist who actually left the project after it became clear the Nazis were not going to make an atomic bomb. He was threatened by the head of Los Alamos security, and falsely accused of having met with Soviet agents. In leaving he was turning his back on his most important professional peers at a crucial time in his career². Doing so must have required tremendous courage and moral imagination. Part of what makes the choice intriguing is that he himself didn't think it would make any difference to the success of the project. I know I personally find it tempting to think about such choices in abstract systems terms: "I, individually, can't change systems outcomes by refusing to participate ['it's inevitable!'], therefore it's okay to participate". And yet while that view seems reasonable, Rotblat's example shows it is incorrect. His private moral thinking, which seemed of small import initially, set a chain of thought in motion that eventually led to Rotblat founding the Pugwash Conferences, a major forum for nuclear arms control, one that both Robert McNamara and Mikhail Gorbachev identified as helping reduce the threat of nuclear weapons. Rotblat ultimately received the Nobel Peace Prize. Moral choices sometimes matter not only for their immediate impact, but because they are seeds for downstream changes in behavior that cannot initially be anticipated.

Today, of course, people often justify the Manhattan Project using after-the-fact reasoning: by appeal to the "success" of MAD³; as saving American or Japanese lives in ending the war; or as simply inevitable for humanity. These arguments are certainly worth consideration. But for most participants in the Manhattan Project these reasons seemed to play little or no role in their early thinking, and so I don't think they're much help in understanding their choices.

I was asked today to comment on connections between the Manhattan Project and AI. I found myself surprisingly uncomfortable with the request: my own thinking is still forming, and I'm not yet confident in what choices to espouse. But that said, I strongly believe it's beneficial to try to understand deeply the many very different choices taken by people involved in nuclear weapons research, and to reflect on what that means for our personal involvement in technology today.

Acknowledgments

Thanks to Jason Benn for the invitation to speak, and to Laura Deming for many generous related conversations, and much insight on this topic.

Footnotes