In Optimism Lies Urgency: Toby Ord and the Future of Humanity

ACCORDING TO TOBY ORD’S The Precipice: Existential Risk and the Future of Humanity, the probability of the annihilation of billions of people and/or the unrecoverable collapse of civilization in the coming century is one in six: “[T]he roll of a die, or Russian roulette.”

The Precipice was released to British audiences on March 5 and reached American shelves by the end of the month. It’s unclear whether the timing of Ord’s intervention, in the midst of the current global pandemic, will damn or consecrate the book — if The Precipice is precisely what the world needs right now or the last book anyone is going to place on their quarantine reading list. Ord includes a section on how we’ve exacerbated the risk of natural pandemics.

In an alternative universe, untouched by COVID-19, whether one would find the title of The Precipice and its conclusion alarming or irritating would depend, at least in part, on what a reader thinks of where Ord works. The Precipice is the latest public missive delivered by an affiliate of Oxford’s Future for Humanity Institute (FHI), founded and directed by Nick Bostrom and dedicated to studying “big picture questions.” FHI shares an address with the Centre for Effective Altruism. The two back up against Westgate, the city’s titanic and improbably open-aired, Golden State–style shopping complex.

Pioneered by Ord and fellow Oxford philosopher William MacAskill, effective altruism is a movement that defines itself as “the use of reason and evidence to help you help others as much as possible.” This sounds especially intriguing to young people hoping to do meaningful work. To those already involved in the endeavor, it can come across as a provocation — as if no one was using reason and evidence before the late 2000s. EA’s advocates often poke at provincial moral intuitions. Famously, their impartial “p-values” suggest that it’s more effective to become a banker and donate well than to work for change in your own neighborhood — happy news for financiers hoping to have their zero-guilt cake and eat it too. The group has been energetically critiqued.

Ord is by no means the first of the effective altruists to focus on existential risk — i.e., dangers that threaten “existential catastrophe,” a class of disasters that includes but is not identical to human extinction. The topic follows in a straightforward way from EA principles. If people matter equally regardless of their position in time, if there will be many more people in the future, and if our current actions can influence how those people live, then in order to maximize the good you can do, your actions should be good for those future people. The greatest good you can do for them is ensuring that they will exist, and that they won’t live in a dystopia. Note that the community once called existential risks “x-risks,” a term with the advantage of combining what many like least about economics, TED Talks, and tech, while also making the shared spirit of global capitalism and EA explicit. Ord calls them “existential risks.”

This is the short chain of logic that leads from the Centre for Effective Altruism to FHI. At its end are some curious alliances: researchers bringing marginal utility to bear on international poverty and futurists chasing after digital minds and cosmic imperium. Ord, who publicly pledged to live on £18,000 a year (adjusted for inflation) and has given away to date over a quarter of his total earnings, works for a center that received a donation from billionaire Elon Musk. A shared interest in artificial intelligence and deep learning is what brings Silicon Valley to FHI’s doorstep. Bostrom and Ord both believe that an AI unaligned with human values is the greatest threat to human potential. And while it is, of course, a good thing for those sounding the AI alarm bells to be within earshot of the technology’s engineers and indecently rich patrons, these alliances haven’t helped to clear the air of suspicion. Their critics are particularly wary of math-intensive products glazed with the user-friendly minimalism favored by the well off.

The Precipice is both an artifact of this institutional context and something altogether different. EA tools are still there: priorities are set with reference to the steepness of curves; there’s an appendix titled “Surprising Effects when Combining Risks.” And EA principles are still there: Ord confirms the overwhelming moral significance of securing humanity’s future, or, in Bostromian terms, “reducing an existential threat by a billionth of a billionth of one per cent” in a way that “would be worth a hundred billion times the value of a billion present-day lives.”

The book’s difference begins with its approach and attitude. Ord suggests that “helping humanity over the long run could be one of the best ways to do good in the world.” Chapter Seven includes a brief section titled “What not to do.” “Don’t be fanatical,” he writes (to his colleagues). Safeguarding humanity isn’t the only priority. “Boring others with endless talk about this cause is counterproductive. Cajoling them about why it is more important than a cause they hold dear is even worse.” In a footnote citing the work of Stuart Russell, Professor of Computer Science at the University of California, Berkeley, Ord suggests that one promising way to address the AI alignment issue is to build machines with a reward function that encourages them to act as if in a state of moral uncertainty. There’s something very touching about the idea of an algorithm being better because it’s not sure of how to do the most good. Ord’s book is certainly more persuasive. In fact, the book persuades its reader that its title comes from a place of deep conviction, not self-importance. The term “effective altruism” appears once in the main body of text. The word “uncertainty” over 60 times.

But Ord is certain that risk levels are unsustainably high, and that there will be resistance to his book’s conclusion — that this is largely the stuff of Marvel comics or science fiction and that many readers, even at a time of global crisis, will stumble over the word “omnicide.” He does his best to anticipate objections. Those looking to challenge Ord’s calculations will have to work through the book’s footnotes and appendices, which, taken together, are as long as the main text, and which synthesize the research of the experts listed in his vast acknowledgments section. “One in six” takes our estimated response into account, so the fatalist should note that, if we do nothing, the odds are closer to two in six. Ord argues that there is insufficient evidence for pessimism. We can’t be sure that the future will be so bad, or whether billions of lives would be better off unlived. Nihilism sounds impressive but isn’t a principle of governance. How can we determine the budget for the Biological Weapons Convention according to the principle that nothing matters? You might agree that the book is alarming but fail to see how it concerns you. Ord compares the issue to climate change and includes a section titled “What you can do.” Concrete policy suggestions are discussed throughout the main text and reiterated in an appendix.

Ord is not simply explaining a probability. He wants to encourage his readers to see ethics from the perspective of humanity. That is, he wants them to imagine the species as a coherent actor and to identify with it. That’s why Ord speaks of “our newfound knowledge” and “our potential,” a decision that seems tone-deaf or wilful, and occasionally completely insane.

Ord writes: “There will be great challenges in getting people to look far enough ahead and see beyond the parochial conflicts of the day.” We need to “accept the fresh responsibilities that come with our unprecedented power.” “Every day we are the beneficiaries of uncountable innovations made by people over hundreds of thousands of years. […] This is a stunning inheritance.” Tonto’s famous riposte to the Lone Ranger comes to mind: “What do you mean we, pale-face?” Those un-ensnared in today’s parochial conflicts have commendable vision. The view from Oxford’s tower is long indeed. From a different perspective, or for those who’ve read Ta-Nehisi Coates, our inheritance may look a little grim. I suspect that the greatest resistance to The Precipice will be to its use of the first-person plural. It isn’t a stylistic objection; it takes us, rather, to the heart of the project. Getting you to occupy this “we,” where Ord’s argument is clarified and its urgency most deeply felt, is the book’s most incredible work. A reader’s resistance won’t be overcome with cost-benefit analyses.

To put a face to this resistance and clarify what’s at issue, let’s imagine a reader. Call her Ann. Ann’s ethical attention is directed to the inequalities between persons and groups. Some people are impermissibly disadvantaged by these fissures. Some are concerned and trying to help. Others resent that they’re finally being roused from their moral stupor. Even if you manage to convince Ann that these inequalities are decreasing, she’ll maintain that our awareness of them has never been greater, and that it needs to continue to grow. Ann is a bit suspicious of universal claims. She looks at “we” and her mind shouts “illusion of consensus!” Ann has read Sontag and Chomsky. Weapons of mass destruction have race, class, and gender aspects. Ann’s background is, moreover, in postcolonial theory. Any call-to-arms on behalf of humanity is going to raise an ugly ghost — a ghost that could forget his principles in order to purportedly spread those principles to the great advantage of his economic agenda. Ann lives in the United Kingdom, or in the United States, or in Europe.

How is Ord going to get Ann to join his “we”? Ord might observe that our moral vision is improving and that Ann herself is evidence of this fact. Our concern is stretching into the past. Why shouldn’t it extend into the future, he might ask, a future that, by all indications, will be more humane? Ann could refuse to listen to the importance of buying us more time. Inequality isn’t a matter of time, she might say, but of moral courage. But there’s a one in six chance that there will be no more time for moral courage or, in fact, anything else, Ord replies. Moral courage is needed to make our future look more democratic. We all need to care to survive. Ann describes racialized capitalism. Ord describes three kinds of unrecoverable dystopia. Ann accuses Ord’s “we” of ignoring systemic injustice. Ord asks Ann, with a nod to Richard Rorty, if she also accuses atheists of blasphemy. Ann thinks this “we” is a dangerous fiction. Ord declares that it’s the most dangerous time in history to throw it away. “Don’t be tribal,” he warns. Ann says that the warning signals his tribal affiliation. “Framing it as a political issue on one side of a contentious divide would be a disaster,” Ord says. Ann insists that there are no politically neutral claims. Ord tells her that she insists that at our collective peril.

The author of The Precipice walks out of the room and takes a few deep breaths.

It’s easy to stay within one’s own moral framework and still prioritize existential risk. We may have obligations to preserve the past or address past evils, or honor past actions by paying it forward. The future could seem particularly valuable to those for whom the past is like a “massive depository that sticks to the present,” to quote philosopher of history Berber Bevernage. Even those who drew humanity’s most bitter stick could be made to care. Perhaps a few readers secretly think that apocalypse wouldn’t be so bad, for at least extinction would be equally distributed. You might find that perspective a little repulsive. You might ask them to think of the enormous waste, or how painful the end would be, or to keep in mind that billionaires are currently building bunkers to weather such a storm. They might be among the few who succeed. Imagine all of humanity’s descendants to be of Bezos.

People from an astonishingly wide range of moral perspectives will agree that existential risk is a great concern. But Ord really wants you to see it in a certain way. “In optimism lies urgency.”

Ord walks back into the room. He invites Ann to take a seat. He turns on the overhead projector.

Philosopher and novelist Iris Murdoch thought that there was something to morality prior to the question of “what should we do.” In her 1964 essay “The Idea of Perfection,” Murdoch suggests that ethical thinking is not simply a matter of “opening one’s eyes” to the relevant facts. Ethical thinking is, rather, the result of “just discernment and exploration.” What characterizes this exploration are “little peering efforts of imagination.” This is a slow and cumulative and often unremarkable process.

Consider two people looking at a piece of art. Certain facts about the object, like its size and material, are listed on the plaque. The first person assesses the sculpture and decides that it’s a little crude. Her companion begins to talk. He points to features of the object and uses words like “bold” and “unexpected.” She listens to him because he’s an authority on art and because she likes his tone, which is soft and probing. Her experience is improved as a result of his descriptions. She wants to go off around the museum and try to see things as he has seen them. There are moments when the face of a thing changes, but not because the face has objectively changed. We see not a different problem, but a different aspect of it.

Both Ann and Ord stand before the following picture: there’s a one in six chance that we will forfeit our potential this century, and that the reality of this “we” is, to quote Ord, “messy and fraught.” He’s not trying to paper over the fissures that preoccupy Ann. But he wants us to ask ourselves what humanity ought to do. Can we imagine ourselves as a collective that responds to an urgent situation? Do we see the person making the call as a man of privilege or of principle? The facts have already been reported. But it matters how they’re being described.

Try to think of humanity, suggests Ord, in terms of the lifespan of a single human. In that life the 20th century is but three days. We went from horse and buggy to Apollo 11 in countable hours. We are, in fact, 16 years old: “[J]ust old enough to get ourselves in serious trouble.” Ord asks us to measure our concern over 20th-century bioweaponry. And yet, 10 decades ago “we had only just discovered viruses and were yet to discover DNA.” How do we feel about “the next hundred years of improvements”? “Would we expect to get through 2,000 centuries like this one?” Did you know that we hold nuclear weapons on hair-trigger alert, the global equivalent of keeping an arsenal of loaded guns within reach of toddlers, just in case of an intruder? Can you imagine existential catastrophe? Climate change will bring “reduced agricultural yields, sea level rises, water scarcity, increased tropical diseases, ocean acidification, and the collapse of the Gulf Stream.” This is tragedy on a scale we’ve never seen. “None of these,” however, “threaten extinction or irrevocable collapse.” The one in six chance we’re facing is something far worse.

Ord includes the following sketch at the beginning of his third chapter, in a section on “Asteroids & Comets”:

An asteroid, ten kilometers across, speeds toward the Earth. The chance of a direct collision is tiny — for millions of years it has swung through the solar system, missing the Earth on every single pass. But given such deep time the chances compound, and this is the day.

It slams into the Earth’s surface off the coast of Mexico at more than 60,000 kilometers an hour. A trillion tons of rock moving so fast strikes with the energy of a hundred times its own weight in TNT. In just seconds, it releases the energy of 10 billion Hiroshima blasts: 10,000 times the entire Cold War nuclear arsenal. It smashes a hole thirty kilometers deep into the Earth’s crust — over sixty times the height of the Empire State Building; three times taller than Everest. Everything living within 1,000 kilometers is killed by heat from the impact fireball. A tsunami devastates the Caribbean. Trillions of tons of rock and dust are thrown far up into the sky. Some of this superheated rock rains down over millions of square kilometers, burning the animals to death and igniting fires that spread the devastation still further. But much more deadly is the dust that stays aloft. A billowing cloud of dust and ash rises all the way to the upper atmosphere, blocking out the Sun’s light.

It’s in the long, cold darkness that “regional catastrophe” turns to “mass extinction.”

What is the effect of narrating the extinction of the dinosaurs in the present tense? It produces a small mental hiccup in the mind of the reader. She recalibrates, adjusting from what she thought was the description of a hypothetical event to that of an event that has already happened. She doesn’t understand 30 kilometers deep, but she can try to imagine 60 Empire State Buildings bored down into her world. She doesn’t understand a trillion tons, or a hundred times that in TNT. But she can try to recall images of the Hiroshima blast and multiply that by 10 billion. The point is not that there’s a real risk of an asteroid striking Earth this century (it’s much, much smaller than the risks we’re engineering ourselves). The point is that, like seeing footage from World War I in color, what was tucked away in history is now brought to our senses. We realize what’s possible. There have been five large mass extinction events in the last 540 million years. The End-Permian extinction, in which 96 percent of species disappeared, was 250 million years ago.

According to the fossil record, mammals last on average about one million years, and species generally last between one and 10 million years. Humans have been around for about 200,000 years. What if we, or the species that follows us “on the evolutionary family tree,” had another 800,000? To put that number into perspective, consider that Stonehenge was built about 5,000 years ago and we’re not even sure why.

Ten million years could be a lower bound for us. We might be like coelacanths and sponges, species that have lasted for hundreds of millions of years essentially unchanged. In 10 million years, Africa will be torn in two along the Rift Valley. In 50 million years, the Mediterranean Sea will be crushed to mountain. In 250 million years, there will be a new Pangaea, a new congress of the continents. Days will stretch to 25 hours. Five hundred million years from now, the land will break apart and the world will see a new configuration. New constellations. Maybe we could live long enough to see it. “If this feels unimaginable, consider that the horseshoe crab has already witnessed such change.”

When we think of humanity through “the slow clock of cosmological expansion,” our future becomes “a canvas vast in time and space.” Perhaps we could live long enough for humans to “heal our society and our planet of the wounds we have caused in our immaturity.” We could prolong life on Earth by saving the biosphere from a brightening sun. When the sun grows too vast, perhaps we could carry seeds and cells to other planets to “make green the barren places of the galaxy,” with a space-fleet called Noah’s Ark. If you think space travel is impossible, try to imagine the “perseverance of the Polynesian sailors who, a thousand years ago, sailed vast stretches of the Pacific.” In the life of our species, we went from discovering that the Earth revolved around the sun to building computers better than us at chess in a matter of weeks. Wait until we’re in our late 30s. Just see what we can do.

Imagine that our descendants will look on the planets and moons of our solar system as I look on the national parks of my home country: “[A]s monuments, jewels. To be explored and treasured. To fill us with wonder and inspire us to journey further.” Shine a beam of light into space and trace it to the edge of influence. Imagine a future human, whose lineage we trace back to you, who can hear “[m]usic that we lack the ears to hear,” journeying beyond halfway to the edge of the affectable universe in a “final diaspora.” Some stars last for trillions of years. Maybe our time is “astonishingly close to the very start of the universe.” Maybe we could last long enough to reach “some external insurmountable limit — perhaps the death of the last stars, the decay of matter into energy.”

The epigraph to Ord’s final chapter is a quote from H. G. Wells: “It is possible to believe that all the past is but the beginning of a beginning. […] It is possible to believe that all that the human mind has ever accomplished is but the dream before the awakening.” That we are all paddling out to some new horizon, with something unwritten before. How much farther can we go? Ord believes “we have barely begun the ascent.” “A vast and extraordinary adulthood awaits.”

Ann blinks and can’t believe that she just had a moment of cosmic optimism. Perhaps humanity needs a common enemy in order to come together. In which case, mankind is engineering its own great problem as well as the conditions for its solution. Ord repeats “we” until she starts to see how it could be true. His nosism may be the most provocative feature of The Precipice. It also signals the book’s real project and serves as a shorthand for its greatest achievement.

Ord may simply deploy these images and this language in order to combat what’s known in behavioral psychology as availability heuristics and scope neglect. An availability heuristic is a mental shortcut that allows us to predict things based on examples that come to mind quickly. Scope neglect is a phenomenon in which our compassion doesn’t scale well with numbers. Together these two phenomena encourage us to undervalue the importance of existential risk. This imaginative work may be an example of a rational man momentarily relying on emotional appeal in order to answer to a meta logic of cold calculus. Perhaps.

And yet, by the end of The Precipice, a reader sees that what Ord is arguing to preserve is itself the stuff of his book. Humans, he suggests, may be cosmically significant. Without us, the universe would lose the buttery sound of a brassy instrument and the noise of New York City on a May afternoon. It would also lose the only things in all its vastness that are capable of “an upwards force,” of imagining horizons of expectation unshaped by our spheres of experience, of imagining justice. The use of this capacity is how an EA doomsday book becomes, against all odds, a vision. The Precipice may be the Silent Spring that the futurists have been waiting for.

¤

Alexa Hazel is a freelance writer.