Mitigating Extinction Risk Isn't The Only Reason to Work on AI
Making AI go well is a lot broader than making sure it doesn't kill us all.
At 80,000 Hours, we write a lot about AI risk. But we also think the moral stakes of the future are incredibly important in other ways, and enjoy Bentham’s Bulldog’s writing about AGI preparedness, so we’re excited to host this guest-post from him.
In AI risk circles, it’s often believed that our primary aim should be avoiding extinction or other catastrophic outcomes. The most central worry about AI is that it will be misaligned—having values radically out of accordance with human interests. Most conceivable values, it is claimed, when taken to the limit entail the destruction of the human race.
This is certainly a serious risk. But it’s not the AI risk scenario that I worry about most in a post-AGI world. Instead, my bigger worry is that a post-AGI world will do terrible things that do not lead to human extinction. Those interested in making a post-AGI world go well would do well to work on these scenarios, and not only on preventing extinction.
Moral errors
Begin with the following fact: nearly all societies in human history have carried out acts of horrendous evil. As Evan Williams notes in his paper, fittingly called The Possibility of an Ongoing Moral Catastrophe:
Show me one society, other than our own, that did not engage in systematic and oppressive discrimination on the basis of race, gender, religion, parentage, or other irrelevancy, that did not launch unnecessary wars or generally treat foreigners as a resource to be mercilessly exploited, and that did not sanction the torturing of criminals, witnesses, and/or POWs as a matter of course. I doubt that there is even one; certainly there are not many.
By default, then, we should expect the future to have atrocities too. It would be suspicious if atrocities were ubiquitous throughout all of history but then ended thirty years ago, particularly when past societies were blind to the atrocities they carried out. And atrocities get more consequential as one gains more power. If one has the ability to transform the cosmos on a galactic scale, moral errors on their part lead to fairly huge loss of value. The atrocities that have occurred recently, continuing through the present day, illustrate that extreme suffering can exist alongside great abundance.
Mere survival and economic prosperity are not themselves enough to ensure a world that is good for its inhabitants.
Factory farming is the most notable modern example. Tens of billions of animals undergo conditions that we wouldn’t hesitate to call torture if inflicted on any person; they’re castrated, caged, selectively bred so they can barely move, mutilated, ground up, transported, and subject to a hundred other profound indignities which, if inflicted on a dog, would uncontroversially merit a long time in jail. Now, there are some reasons to worry that factory farming might continue as we set out and colonize the stars in a post-AGI world.
A more concerning source of potential error, in my view, is the treatment of wild animals. Most animals in nature live short lives that culminate in painful deaths from starvation, disease, predation, and the like. In principle, it should be possible to majorly reduce their suffering by making significant changes to ecosystems so that wild animals suffer less.
Here is why I am more worried about wild animal suffering persisting in the future than farmed animal suffering: nearly everyone is opposed to the suffering of farmed animals. If we could do away with factory farms, most people would favor doing so. It’s possible that we end factory farming, but remain blind to the moral importance of wild animal suffering. After all, most people favor keeping it around. There are many unresolved questions about how transformative AI will impact animals.
In a post-AGI world, we might treat digital minds terribly. Currently, though we’re not yet sure if consciousness is present in AI, few think about this as a subject of much ethical import. There is a serious risk that we will construct digital analogues of factory farms. There could be a vast number of digital minds in the future, and it’s possible that decisions we make in the coming decades could be long-lasting.
Even aside from wholly neglecting the interests of some sizeable class of entities, there are a range of other serious moral errors we might make. For instance suppose, as follows from nearly all of the standard views in population ethics, that it is good to bring well-off people into existence. It is a bit hard to believe that in the far future, we will be particularly concerned with creating maximal numbers of well-off lives. But failure to do this, on standard accounts, could result in the loss of most of the realizable value.
Ordinary decision-making is not made according to the dictates of any particular moral theory, but instead by the dictates of common-sense. While common-sense may be a fine guide in ordinary decision-making, when one must make grand and far-reaching plans for the universe, it is likely to produce wildly suboptimal behavior. In fact, even if we were acting according to a specific moral theory, so long as we pick the wrong one, we’re likely to lose out on most realizable value. The optimal world according to one moral theory might be radically defective or even overall negative according to another.
Thus, no matter what your moral theory is, you should be seriously worried that the future will contain serious moral errors. These errors have the potential both to jeopardize large amounts of value and to lead to direct horrendous catastrophe.
Now, will our greater wisdom in a post-AGI age lead to the automatic dissolution of moral error? It’s far from clear. Selection pressures might produce AIs that conform to the moral dictates of 21st-century humans, rather than the right moral view. Even if we build AIs that get the right answers to moral questions, we are likely to simply ignore their advice. People spend little time thinking seriously about their own misconduct or consulting with moral philosophers on how to behave.
In general, people have relatively little abstract moral motivation. Even when people become aware that, say, they ought to forego meat or give their money to effective charities, they mostly don’t (note: this point is about people’s motivation to follow what is right, so it doesn’t depend on whether people really are obligated to do those things). If the AI advisors suggested that we fill the universe with happy digital minds, or something else comparably weird, it is hard to imagine world leaders following their advice.
Can anything be done to raise the odds that we will not make horrendous errors in the far future? It would be somewhat surprising if there was nothing that could be done on this front. Efforts that improve collective decision-making likely increase the odds of sane ethical decision-making. Attempting to build AIs that engage in sustained moral reflection and possess moral motivation is similarly important. Work to improve the quality of AI character is thus desirable and no more difficult in principle than work on alignment.
Similarly, more work on AI consciousness could help ensure that morally-important AIs’ interests are counted during the time when momentous decisions are being made. It would be worth academics thinking through the kinds of rights AIs should have and the kinds we should be striving to build. Ultimately, this is just a brief summary of a few high-impact ways we can help ensure future catastrophic moral errors do not jeopardize huge amounts of future value. This is a particularly promising area to work on because it’s so neglected.
Power concentration, lock-in, and other ways we could lose the ability to self-correct
There are other ways that large amounts of value could be lost, even without the human race going extinct. One way is through extreme concentration of power. Insofar as a world with advanced AI poses serious risks of AI taking over autonomously, it seems it also poses non-trivial risks of humans using AI to seize unprecedented power. This extreme power concentration could lead to most humans’ interests being neglected; as medieval kings had no need to count the interests of peasants, the same might be true in a post-AGI world. Development of AI in the U.S. and China could leave the rest of the world behind.
Alternatively, a post-AGI world could beget Darwinian selection pressures that lead to the loss of most value. Space contains a great abundance of resources that could be usefully transformed into things of value. Yet in a world of multi-polar space control, actors may be forced to squander their resources on conflict or rapid expansion, thus leading to the loss of most realizable value.
Similarly, once we develop powerful AI, we may have unprecedented ability to preserve the institutions of the present world. This may lead us to lock in current values to horrendous effect. As it would have been bad for ancient civilizations to preserve their civilization’s values in their exact form for all time, the same is true for us. If civilization seriously errs, we should be alarmed by the prospect of it becoming uncorrectable.
In fact, even scenarios of misalignment won’t necessarily lead to human extinction. AI could be misaligned without killing everyone on the planet. Insofar as we partially align AI, humanity may be majorly disempowered without going extinct entirely.
Conclusion
My biggest worries about AI aren’t about it causing extinction, though I consider that a serious risk. In my view, more expected future value is lost from moral errors than extinction risks. There are a number of other serious risks, where large amounts of value could be lost. Those interested in ensuring the post-AGI world goes well shouldn’t just focus on extinction, but should try hard to ensure that the subsequent world of abundant material prosperity is good for the creatures in it.
If you’ve read this far, consider using your career to work on these problems and subscribing to Bentham’s Bulldog on Substack.
 | A guest post by
|