The Only Way to Prevent Another Nuclear Strike Is to Get Rid of All the Nukes

The cosmologist Stephen Hawking recently suggested that civilization is at a critical juncture in its short—on geological timescales—career. We live in the “most dangerous time for our planet” due to climate change, the sixth mass extinction, and emerging technologies—and if we fail to proceed with great caution and wisdom, our species could follow the dodo into an existential oblivion.

To alert the public of the greatest dangers facing humanity, the Bulletin of the Atomic Scientists created the Doomsday Clock in 1947. Its time represents our collective proximity to a global catastrophe. The closest the clock has ever been to midnight—or doom—was in 1953, after the US and Soviet Union detonated thermonuclear weapons. The furthest away it’s been happened in 1991, when the clock stood at a somewhat reassuring 17 minutes before midnight, following the official end of the Cold War.

Unfortunately, the minute hand has been ominously ticking forward in recent years, and on January 26 the Bulletin announced that it would move the clock forward even more to a mere two and a half minutes before doom. The only time the clock has indicated a higher threat level was in 1953—a fact that should make one uneasy, at the very least, about our collective future on spaceship Earth. While the Bulletin originally considered only the threat of nuclear weapons, it recently added climate change to the list of global, transgenerational risks. Given President Donald Trump’s reckless statements about nuclear proliferation and the Republican’s rampant climate denialism, it should not come as a surprise that “doom-soon” is now more probable than it was a year ago.

To better understand the Bulletin’s decision, I Skyped with Dr. Lawrence Krauss, a theoretical physicist and cosmologist at Arizona State University who also chairs the Bulletin’s Board of Sponsors. He is one of many leading experts—including 15 Nobel laureates—who decide whether the clock should move forward, backwards, or remain frozen. (This interview has been lightly edited with hyperlinks added.)

Mustafa Kibaroglu recently wrote in the Bulletin, “What is overly idealistic is to believe that humanity, if it possesses nuclear weapons indefinitely, will indefinitely manage to avoid nuclear war.” Do you think that nuclear conflict of some sort is more or less inevitable as long as nuclear weapons exist?

I don’t know whether nuclear conflict is inevitable, but I think the use of nuclear weapons is. I think having a nuclear weapon used by accident or on purpose against a civilian population is unavoidable as long as we possess them. Ideally, a world free of nuclear weapons is one that we should aim for.

How might a single nuclear weapon being detonated anywhere change the world?

It would open the floodgates and lower the threshold for using the next one, and the next one. The issue also depends to some extent on where it happens. If a nuclear bomb were detonated in the First World, the tragedy of 9/11 would pale in comparison. The short-term impact on the global economy and the global political situation would dwarf anything that’s happened in recent memory. Would we survive such an attack? Yes. But opening the dam—or better, opening Pandora’s box—would be an extremely dangerous thing to do.

The Iran nuclear deal has been harshly criticized by Republicans. Yet the Bulletin last year identified it as one of two “bright spots in a darker world situation full of potential for catastrophe.” Why do you support this deal, which Trump himself has labeled “the worst … ever negotiated.”

Well, my guess is that he probably hasn’t even read the deal. The point is that Iran had every motivation to develop nuclear weapons and the capability of doing this. What the Iran deal did was remove that capability in a verifiable way. The whole point of non-proliferation efforts is to offer incentives for not creating nuclear weapons, rather than saying, “We don’t want you to have them, even though we do, and are therefore able to control you.” The Iran deal clearly pulled us back from the brink at a time when there was no other solution. I mean, military action would have been a disaster. The nuclear deal was a huge diplomatic win.

Now, can we improve upon it in the long run? Can we try to establish better verification processes down the road? Yes. But this deal was diplomacy at its best. It took us back from the military precipice and it achieved its goal of reducing the nuclear threat from Iran. If you had to pick triumphs, it’s way up there.

When the Bulletin writes that “unless carbon dioxide emissions are dramatically reduced, global warming will threaten the future of humanity,” what does it mean by the “future of humanity”? Are we talking about human extinction, the collapse of global civilization, or the decline of independent societies?

It’s not the same kind of threat as nuclear weapons, which could quickly extinguish humanity—or at least civilization as we know it. However, global warming will create a profound threat to modern civilization, not just locally but globally. For example, if billions of people are displaced from their homes, the result will be immense sociopolitical pressures. If storms regularly produce economic havoc in the First World, this will require a new way of living. And if certain areas of the world become literally uninhabitable, we will have changed the planet in a fundamental way.

These are severe enough, I think, for any one of them to be considered a major challenge to civilization. Could humans go extinct because of global warming? It’s not likely. But the future of human civilization would certainly be jeopardized. Consider that the sea level rise predicted in the near-term is conservative: it could actually be meters. And changing the North Atlantic currents so that Europe becomes colder would have severe and pervasive effects on how people live on that continent. By endangering the future of humanity, the Bulletin means that life will change irrevocably in ways that are highly undesirable. From my point of view, the sociopolitical problems of displacing one-third of the world’s population, which could significantly exacerbate wars and violence, should be taken very seriously. I mean, we only have a few refugees entering the US from a local war in Syria right now and we can barely handle that. Wait until there are 2 billion refugees knocking on the door.

Building on something you just said, do you see climate change as a “conflict multiplier”? To what extent do you worry that it could increase the probability of nuclear conflicts in the future?

I tried to say that at the press conference the other day [held on January 26]. These things are all connected. There are even interrelationships between AI [artificial intelligence], cybertechnology, and nuclear weapons—and when you increase international tensions and countries possess nuclear weapons, you obviously, at some level, increase the likelihood of war as well. Furthermore, when you create a greater disparity between the “haves” and “have-nots” you kind of make the latter want to find ways to gain more political power, and acquiring nuclear weapons are one way to do this. Terrorism is bred in part by ideology, but also by economic disparity and lack of hope. So, yes: climate change is a potential conflict multiplier.

If a climate emergency were to occur—for example, resulting in sudden increases in global surface temperature—would you support the use of stratospheric geoengineering? [Note: stratospheric geoengineering is an idea taken seriously by some scientists. It would involve injecting aerosols into the stratosphere to deflect incoming light from the sun, thereby decreasing global surface temperatures.]

No. Not at this point, because the unknowns outweigh the knowns. I think there are other kinds of geoengineering that are much more relevant, and with which I’m involved. Unfortunately, the total amount of research dollars that’s been devoted to exploring carbon capture and sequestration (not at coal plants but generally) is maybe 10 million versus 60 billion a year for coal and gas exploration. We know what will happen if we reduce the carbon dioxide in the atmosphere, because that’s where we were: the climate will return to a safe equilibrium. But to play games with a complicated system without understanding the implications is to me a great concern.

Why are Trump’s statements about NATO being “obsolete” worrisome? [Note: many scholars see NATO as having played a critical role in keeping the peace in Europe since World War II.]

Trump’s statements about NATO are absurd because they aren’t based on any knowledge. Like monkeys on a typewriter, some of them may eventually be right. But the point is that you have to base such claims on empirical evidence, and you must rely on experts because that’s how you run a responsible government. The statement that NATO is obsolete is just vague and emotional. Whether there could be new allegiances, new strategic frameworks, new cooperations—that’s all possible. But random statements from someone who is obviously ignorant, who hasn’t made any efforts to actually read or learn about the global situation, is worrisome when that person can establish new policies at a whim.

[Although Dr. Krauss adds that he’s fairly convinced that the courts will roll back some of Trump’s executive orders.]

The Bulletin writes, “Wise public officials should act immediately, guiding humanity away from the brink. If they do not, wise citizens must step forward and lead the way.” How should citizens step forward and lead the way?

First of all, they have to become aware of and acknowledge the problems. Then people should phone their congresspeople, participate in public marches, and exercise their power at the voting booth. At this point, we still—I think—live in a democracy, to some extent. Consequently, congresspeople should still react when people say, “This really matters to me.” Unfortunately, in my opinion we’ve become incredibly complaisant when it comes to nuclear weapons. The public would rather not think or worry about them: “they haven’t been used in over 70 years, the reasoning goes, so they won’t be used in the future.”

The only way you’re going to see this [US] Congress ever react to an irrational presidential initiative is if they somehow feel that it’s in their best interest. I know it sounds jaded to say but I’m not convinced that the health and welfare of the nation are their prime concerns right now. Maybe this has always been the case, but the public does have an opportunity to make a difference. Indeed, it’s happened before—for example, with the Vietnam War. You can have an impact. Speak out in your local communities, get people organized, and educate others. It sounds pie in the sky, but it’s the only tool we have right now.

I think most of the public, on many issues, is further away from the brink than governments are, but as long as the public stays quiet, nothing is going to happen.

Steven Pinker and others argue—convincingly, in my view—that the world has never been so peaceful. Yet it seems like the world has also never had so much potential for catastrophe. Are you optimistic about the future?

I try not to be optimistic about the future, in part because I try not to make predictions. As readers might have heard me say, I avoid making predictions about anything less than 2 trillion years in the future. [Laughs.] I’m often guided by a friend of mine, Cormac McCarthy. When I remarked on how pleasant a fellow he was and how that differed from his rather bleak stories, he said: “I’m a pessimist but that’s no reason to be gloomy.” I kind of take that as my mantra. I think there’s great hope and, indeed, opportunities here for both Trump and Putin. I just finished writing a book called The Greatest Story Ever Told… So Far. It’s about, in some ways, humanity at its best. When one looks at what humanity can do, one realizes that there’s always hope. We can overcome the darker side of our nature.

So, we’re not necessarily doomed, but it would be foolish and naive to not recognize the challenges and threats that are present. Louis Pasteur said that “Fortune favors the prepared mind,” and I think that’s the appropriate way to move forward.

---

Phil Torres is the founding Director of the X-Risks Institute. He has published in popular media like the Bulletin of the Atomic Scientists, TIME, Skeptic, Free Inquiry, Truthout, and Salon. His forthcoming book is called Morality, Foresight, and Human Flourishing: An Introduction to Existential Risks. Follow him at @xriskology.

Get six of our favorite Motherboard stories every day by signing up for our newsletter.

from The Only Way to Prevent Another Nuclear Strike Is to Get Rid of All the Nukes

We’re Not Ready For Superintelligence

Artificial Intelligence. Image: Geralt/Pixabay

The problem with the world today isn’t that too many people are afraid—it’s that too many people are afraid of the wrong things. Consider this: what scares you more, that your life could end because of a terrorist attack or because you get crushed to death under a large piece of furniture?

Despite a media environment in which the threat of terrorism is omnipresent and the threat of furniture nonexistent, your gravestone is actually more likely to say, “Died under a couch recently bought from Ikea” than “Perished in a terrorist attack.”

In fact, asteroids are more likely to kill the average person than lightning strikes, and lightning strikes are more dangerous than terrorism. The point is that, as I’ve written elsewhere, our intuitions often fail to track the actual risks around us. We dismiss many of the most likely threats while obsessing over improbable events.

This basic insight forms the basis for a recent TED talk by the neuroscientist Sam Harris about artificial superintelligence. For those who pay attention to the news, superintelligence has been a topic of interest in the popular media at least since the Oxford philosopher Nick Bostrom published a surprise best-seller in 2014 called—you guessed it—Superintelligence.

Major figures like Bill Gates, Elon Musk, and Stephen Hawking subsequently expressed concern about the possibility that a superintelligent machine of some sort could become a less-than-benevolent overlord of humanity, perhaps catapulting us into the eternal grave of extinction.

It isn’t just another “tool” that someone could use to destroy civilization. Rather, superintelligence is an agent in its own right.

Harris is just the most recent public intellectual to wave his arms in the air and shout, “Caution! A machine superintelligence with God-like powers could annihilate humanity.” But is this degree of concern warranted? Is Harris as crazy as he sounds? However fantastical the threat of superintelligence may initially appear, a closer look reveals that it really does constitute perhaps the most formidable challenge that our species will ever encounter in its evolutionary lifetime.

Ask yourself this: what makes nuclear, biological, chemical, and nanotech weapons dangerous? The answer is that an evil or incompetent person could use these weapons to inflict harm on others. But superintelligence isn’t like this. It isn’t just another “tool” that someone could use to destroy civilization. Rather, superintelligence is an agent in its own right.

And, as scholars rightly warn us, a superintelligent mind might not be anything like our minds. It could have a completely different set of goals, motivations, categories of thought, and perhaps even “emotions.” Anthropomorphizing a superintelligence by projecting our own mental properties onto it would be like a grasshopper telling its friends that humans love nothing more than perching atop a blade of grass because that’s what grasshoppers enjoy. Obviously, that’s silly—and simply incorrect.

So, a superintelligence wouldn’t be something that humans use for their own purposes, it would be a unique agent with its own purposes. And what might these purposes be? Since a superintelligence would be our offspring, we could perhaps program certain goals into it, thereby making it our friend rather than foe—that is, making it prefer amity over enmity.

This sounds good in theory, but it raises some serious questions. For example, how exactly could we program human values into a superintelligence? Getting our preferences into computer code poses significant technical challenges. As Bostrom notes, high-level concepts like “happiness” must be defined “in terms that appear in the AI’s programming language, and ultimately in primitives such as mathematical operators and addresses pointing to the contents of individual memory registers.”

Even more, our value systems turn out to be far more complex than most of us realize. For instance, imagine we program a superintelligence to value the well-being of sentient creatures, which Harris himself identifies as the highest moral good. If the resulting superintelligence values well-being, then why wouldn’t it immediately destroy humanity and replace us with a massive warehouse of human brains hooked up to something like the Matrix, except the virtual worlds in which we’d live would be overflowing with constant bliss—unlike the “real” world, which is full of suffering.

A bunch of Matrix brains living in a virtual paradise would produce far more overall well-being in the universe than humans living as we do, yet this would (most would agree) be a catastrophic outcome for humanity.

Adding to this difficulty, there’s the confounding task of figuring out which value system to start with in the first place. Should we choose the values espoused by a particular religion, according to which the aim of moral action is to worship God? Should we borrow the values of contemporary ethicists? If so, which ethicists? (Harris?) There’s a huge range of diverse ethical theories, and almost no consensus among philosophers who study such issues about which theories are correct.

So, not only is there the “technical problem” of embedding values into the superintelligence’s psyche, but there’s the “philosophical problem” of figuring out what the heck those values are.

This being said, one might wonder why exactly it’s so important for a superintelligence to share our values (whatever they are). After all, John prefers chocolate while Sally prefers vanilla, and John and Sally get along just fine. Couldn’t the superintelligence have a different value system and coexist with humanity in peace?

The answer appears to be No. First, consider the fact that intelligence confers power. By “intelligence,” I mean what cognitive scientists, philosophers, and AI researchers mean: the ability to acquire and use effective means to achieve some end, whether that end is solving world poverty or playing tic-tac-toe. Thus, a cockroach is intelligent insofar as it’s able to evade the broom I use to swat it, and humans are intelligent insofar as we’re able to say, “Hey, let’s go to the moon,” and then actually do this.

If intelligence confers power, then a superintelligence would be superpowerful. Don’t picture here a Terminator-like android with a bipedal posture marching through the world with machine guns. This dystopic vision is one of the great myths of AI. Instead, the danger would come from something more like a ghost in the hardware, capable of controlling any device within electronic reach—such as weapon systems, automated laboratory equipment, the stock market, particle accelerators, and future devices like the nanofactory, or some as-yet unknown technology (that it might invent).

Making matters even worse, electrical potentials propagating inside a computer transfer information way, way faster than the action potentials in our puny little brains. A superintelligence could thus think about one million times faster than us—meaning that a single minute of objective time would equal nearly two years of subjective time for the AI. From its perspective, the outside world would be virtually frozen in place, and this would give it ample time to analyze new information, simulate different strategies, and prepare backup plans between every word spoken by a human being in realtime. This could enable it to eventually trick us into hooking it up to the Internet, if researchers initially denied it access.

It could use its power to destroy our species for the same reason that our species destroys ant colonies.

These considerations suggest that a superintelligence could crush humanity with the ease of a child stomping on a spider. But there’s a crucial catch: a superintelligence with the means for destroying humanity need not have the motivation to do this. On the one hand, it’s entirely possible for a superintelligence to be explicitly malicious, and thus try to kill us on purpose. On the other hand, the situation is far more menacing than this: even a superintelligence with no ill-will toward humanity at all could pose a direct and profound existential risk to human civilization.

This is where the issues of power and values collide with nightmarish implications: if the superintelligence’s goals don’t almost completely align with ours, it could use its power to destroy our species for the same reason that our species destroys ant colonies when we convert land into a construction site. It’s not that we hate ants. Rather, they just happen to be in the way, and we don’t really care much about ant genocides. Harris makes this point well in his talk.

For example, imagine that we tell a superintelligence to harvest as much energy from the sun as possible. So what does it do? Obviously, it covers every square inch of land with solar panels, thereby obliterating the biosphere (a “sphere” of which we are a part). The once extant Homo sapiens then goes extinct.

Or imagine that we program the superintelligence to maximize the number of paperclips in the universe. Like the case just mentioned, this appears, at first glance, to be a pretty benign goal for the superintelligence to pursue. After all, a “paperclip maximizer” wouldn’t be hateful, belligerent, sexist, racist, homicidal, genocidal, militaristic, or misanthropic. It would just care a lot about making as many paperclips as possible. (You can think of this as its passion in life.)

So what happens? The superintelligence looks around and notices something relevant to its mission: humans just so happen to be made of the same chemical ingredient that paperclips are made of, namely atoms. It thus proceeds to harvest the atoms contained in every human body—all 7.4 billion of us and counting—thereby converting each individual into a pile of lifeless, twisted steel wire.

These aren’t even all the reasons we should be worried about superintelligence, but they do warrant serious concern about the topic—even if our intuitions fail to sound the emotional alarm in our heads: “Be worried!”

As Harris points out in his talk, superintelligence not only presents a behemoth challenge for the best minds on Earth this century, but we have no idea how long it might take to solve the problems specified above, assuming that they’re soluble at all. It could take only 2 more years of AI research, or require the next 378 years during which billions of work hours are spent ruminating this issue.

This is troublesome because according to a recent survey of AI experts, there’s a very good chance superintelligence will join us by 2075, and 10 percent of respondents claimed that it could arrive by 2022. So, superintelligence could show up before we’ve had enough time to solve the “control problem.” But even if it looms in the far future, it’s not too early to start thinking about these issues—or spreading the word through popular media.

The fact is that once the AI exceeds human-level intelligence, it could be permanently out of our control. Thus, we may have only a single chance to get everything right. If the first superintelligence is motivated by values even slightly incompatible with ours, the game would be over, and humanity will have lost. Perhaps truth is stranger than science fiction.

Author's Note: Thanks to Daniel Kokotajlo for helpful comments on an earlier draft.

from We’re Not Ready For Superintelligence

Our Simulated Universe Is Just One Piece of a Matryoshka Doll of Annihilation

Last June, Elon Musk claimed that we almost certainly live in a simulated universe. In his words, “There’s a one in billions chance [we’re in] base reality. I think it’s one in billions.” He then added that, “We should hope that’s true because otherwise if civilization stops advancing, that could be due to some calamitous event that erases civilization, so maybe we should be hopeful this is a simulation. Otherwise, we will create simulations that are indistinguishable from reality or civilization will cease to exist. Those are the two options.”

The first thing to notice here is that these aren’t the only two options—Musk overlooks a another possibility that we’ll discuss below. And second, Musk is wrong that we should hope we’re in a computer simulation. If we are simulated beings, or sims, living inside a high-resolution simulated reality, then (as I’ve suggested before) we have reasons to expect the probability of doom to be extremely high. Call this the “Simulation Doomsday Hypothesis.”

Consider the original “simulation argument” proposed by the Oxford philosopher Nick Bostrom. It states that there are three—and only three—possible future scenarios that our species could find itself in. First, we could go extinct before reaching a “posthuman” state, or a state in which we become advanced cyborgs so different from contemporary Homo sapiens that we could describe ourselves as a new species: Posthomo cyborgus.

As it happens, the probability of catastrophe is far higher than most people realize. The co-founder of the Centre for the Study of Existential Risk at Cambridge University, Sir Martin Rees, suggests that civilization has a fifty-fifty chance of making it through the twenty-first century. This means that the average American is almost 50 times more likely to witness the collapse of civilization than to die in a motor vehicle accident.

Are you more likely to be a sim or one of the very few “real” individuals at the top level of Ultimate Reality?

The second possibility is what Musk overlooks, and it’s what I believe is by far the most preferable outcome: we advance to a “posthuman” state but decide not to run simulations in which beings similar to us live. Perhaps we decide that doing this would be unethical, given the sadness and sorrow that pervades our world.

Indeed, if we are in a simulation, then we could immediately infer something about our simulators, namely that they aren’t omnibenevolent—or perfectly moral—beings, not by a long shot. This reasoning echoes the “argument from evil,” which points out that moral facts about our world are incompatible with the existence of an omnibenevolent God, therefore God doesn’t exist. So, it could be that we develop into posthumans, and it could even be that we run a large number of simulations in the future, but we refrain from running simulations in which creatures like us live in a world like ours.

Image: shangrilatimes/Flickr

And finally, the third possibility is that we almost certainly exist inside a simulation being run on a supercomputer in some other universe. You can think about it like this: if the first two options are false, then it necessarily follows that we become a posthuman civilization that runs a bunch of “ancestor simulations,” as Bostrom calls them. (If we don’t die out or decide not to run simulations, then we will survive and run simulations. That’s just logic.) By “a bunch” I mean millions, billions, or perhaps a googolplex of simulated universes each crowded with sims completely oblivious of their existence as 1s and 0s in a computer program.

Such sims would be oblivious of this fact because future simulations could be run with sufficient detail to fool even the most observant individual. The sims living inside them would also have conscious experiences just like we do, that is, if our best theory about the nature of mental states, called functionalism, is correct. According to the concept of functionalism, all that’s required for consciousness to arise from a system is for that system to exhibit the right sort of “functional organization.” It follows that whether the system is made of squishy brain stuff or silicon-based hardware is (excuse the pun) completely immaterial.

Minds are defined not by what they are, but by what they do. If we make silicon hardware do the same thing that our brains do, then consciousness will emerge.

This being said, consider the implications of the third possibility. If we simulate billions upon billions (upon billions) of sims in the future, and if we have no way of distinguishing our experiences from theirs, then how sure can we be that we’re not ourselves simulated? In other words, imagine that you could take a “sideways-on” view of every extant universe, whether simulated or not. You then reach into a random universe and choose a random individual. What’s the likelihood that you select a sim? Well, the more simulations that are being run, the more likely the answer will be, “She or he is a sim.”

Imagine that you do this over and over again, randomly plucking individuals from their pockets of reality, and for statistical reasons you almost always win your bet that the selected person is a sim. But then something freaky happens: you reach in and pull out a person who happens to be you. How should you answer the question posed above? Are you more likely to be a sim or one of the very few “real” individuals at the top level of Ultimate Reality?

Well, insofar as you are a typical observer in your universe, you should answer no differently than before: “I am almost certainly a sim.” If you were placing bets with a friend, you’d be far more likely to win a few bucks with this answer than by asserting the much less probable claim that you’re non-simulated. It’s this basic line of reasoning that leads Musk to claim that, “There’s a one in billions chance we’re in base reality.”

If our sims are making their own simulants, at least we're not the base level simulation! Image: Naruso/Deviant Art

But this isn’t the end of the story. Consider the fact that the existence of a sim requires the existence of a simulator. This is an indisputable “genealogical” fact. It follows that if you, me, and Neil deGrasse Tyson (my favorite astrophysicist) are all sims in a simulation, there necessarily exists one or more simulators one level “above” us. Admittedly, it’s odd to think that we may be the involuntary participants in a strange sort of voyeurism, with our simulators looking down on us like the gods of ancient myths and religions, aware of even our most private moments.

But perhaps they should be concerned about the very same invasion of privacy. Why? Well, imagine that after randomly selecting yourself from a random universe, you reach in once more and choose an individual who happens to be—wait for it—one of our simulators. Again, the exact same logic applies: our simulator is much more likely to be a sim than a non-sim. Since sims require simulators, this implies another simulation level above our simulator, at which point the questioning can start over: are these simulators two levels “up” from us more likely to live in a simulation or be one of the few non-simulated beings? For statistical reasons, they’re far more likely to be sims, which implies yet another simulation level above them. And so on.

The result of this line of reasoning, which lies at the heart of Bostrom’s argument, is a vastly tall stack of nested simulations, each embedded in another like Matryoshka dolls. We can call this a simulation hierarchy.

If we inhabit a simulated universe, then it could get shut down at any moment, without warning.

In other words, if Bostrom’s first two options above (i.e., extinction and running no simulations) are false, then human civilization will mature into a posthuman civilization that runs lots of simulations with creatures like us. And if we run lots of simulations with creatures like us, then exactly two things follow: (a) we will almost certainly live in a simulation, and (b) there will almost certainly exist a huge simulation hierarchy.

(As Bostrom himself puts it, if we run lots of simulations in the future, then “we would have to suspect that the posthumans running our simulation are themselves simulated beings; and their creators, in turn, may also be simulated beings.”)

Is this the end of the story? Not quite. There are further questions to be asked and answered. For example, we might wonder where in the simulation hierarchy we live, if indeed we are sims. This may sound like an impossible question, but it’s not. Consider the fact that a simulation could produce many lower-level simulations, but it could not have been produced by more than one higher-level simulation. In other words, there’s a “genealogical asymmetry,” so to speak, between simulation levels in a hierarchy: each simulation could spawn any number of additional simulations below it, and this proliferation of new simulations can only proceed in one direction.

The result is an “inverted tree” shape to the simulation hierarchy, with Ultimate Reality at the very top. This suggests that we are statistically unlikely to find ourselves near the level of Ultimate Reality, since far more simulations will have accumulated at the bottom. As the theoretical physicist Sean Carroll puts it in a recent article about the simulation hypothesis, “We probably live in the lowest-level simulation... [because] that’s where the vast majority of observers are to be found.”

This is where things get really interesting, and scary. Think about the existential implications of living in a simulation: if we inhabit a simulated universe, then it could get shut down at any moment, without warning. This introduces a completely novel existential risk scenario to the list of things we have to worry about, including asteroid and comet impacts, supervolcanic eruptions, nuclear war, and global pandemics.

"Oops, someone spilled soda on the simulation computer." Image: Ben Salter/Flickr

Even more, if our simulators live in a simulated universe, then it could get shut down too, thereby causing the termination of our universe two levels below. This introduces yet another existential risk scenario to the list: we get shut down because our simulators get shut down. Continuing with this line of reasoning, if the simulators of our simulators live in a simulated universe, then they too could get shut down—and so on.

The key idea here is that annihilation is inherited downwards in simulation hierarchies, and the more simulations there are above us, the more ways there will be for our simulation to suddenly vanish into the digital oblivion. Consider a simulation hierarchy that consists of 10 levels. Let's say that on each level, there are 10 scenarios that could lead to all lower-level simulations being shut down. For example, perhaps a posthuman civilization running simulations self-destructs in a nuclear conflagration, or a lab assistant accidentally spills coffee on computer hardware, thereby causing it to malfunction. The possibilities are interminable—and could also be quite exotic, since other simulated universes could have different physical constants and laws of nature.

Mathematically, this would yield 90 distinct ways that a simulation on Level 10 could be terminated. If the simulation hierarchy were to include 1,000 levels, then there would be a staggering 9,990 ways for bottom-level simulations to get shut down. And if the simulation hierarchy were vastly tall—as we established above—then the probability of doom could be virtually certain for those at the bottom. All it would take is a single simulation above our to get shut down and poof!—not even a trace of our civilization would remain.

This is why we should hope that we’re not living in a simulation. The third option that Bostrom outlines—becoming a posthuman civilization that runs lots of simulations—essentially implies his first option: extinction. Musk is therefore incorrect in his sanguine declaration that living in a simulation would be a good thing. The very same reasoning that leads Musk (following Bostrom) to conclude that we’re in a simulation also leads us straight into the jaws of the Simulation Doomsday Hypothesis. If we’re lucky, our Posthomo cyborgus descendants will decide their resources are better spent doing something other than digitally simulating their apeish ancestors.

from Our Simulated Universe Is Just One Piece of a Matryoshka Doll of Annihilation