Book Review: Superintelligence - Paths, Dangers, Strategies by Nick Bostrom ★★★★⯪

When I finally invent time-travel, the first thing I'll do is go back in time and give everyone a copy of this book. Published in 2014, it clearly sets out the likely problems with true Artificial Intelligence (not the LLM crap we have now) and what measures need to be put in place before it is created.

It opens with The Unfinished Fable of the Sparrows:

Which, frankly, should be the end of the discussion. Oh Scronkfinkle, why didn't they listen to you?

This book attempts to set out they why and the how of protecting humanity from the (inevitable?) arrival of machines which we would describe as "superintelligent". That is, capable of human-level reasoning and understanding, but unlimited in terms of speed, working memory, and accuracy.

For example, automated trading algorithms caused a "Flash Crash" of the stock market in 2010. Unchecked machines very nearly destabilised the financial work. As Bostrom writes:

[…] while automation contributed to the incident, it also contributed to its resolution. The pre-preprogrammed stop order logic, which suspended trading when prices moved too far out of whack, was set to execute automatically because it had been correctly anticipated that the triggering events could happen on a timescale too swift for humans to respond. The need for pre-installed and automatically executing safety functionality—as opposed to reliance on runtime human supervision—again foreshadows a theme that will be important in our discussion of machine superintelligence.

So where are those safety functions now? Are any of the AI providers building in guardrails to prevent atrocities? We know that some LLMs are restricted from sharing details about devastating weapons of mass destruction - but there seems little else put in place.

The book is mostly accessible but veers wildly between casual language, deep philosophical tracts, pointed snark, and the occasional dive into maths and physics. For anyone with even a passing interest in the progression of any technology, it is a worthwhile read.

Many of the predictions are spot on:

As of 2012, the Zen series of go-playing programs has reached rank 6 dan in fast games (the level of a very strong amateur player), using Monte Carlo tree search and machine learning techniques. Go-playing programs have been improving at a rate of about 1 dan/year in recent years. If this rate of improvement continues, they might beat the human world champion in about a decade.

In fact, AlphaGo achieved mastery at the end of 2016.

In the slightly longer term, the cost of acquiring additional hardware may be driven up as a growing portion of the world’s installed capacity is being used to run digital minds […] as investors bid up the price for existing computing infrastructure to match the return they expect from their investment

As I wrote about in "AI is a NAND Maximiser" this too has come to pass.

While LLMs weren't yet invented when this was written, there's an excellent prediction about how an AI could become a pernicious psychological adversary:

Caution and restraint would be required, however, for us not to ask too many such questions—and not to allow ourselves to partake of too many details of the answers given to the questions we do ask—lest we give the untrustworthy oracle opportunities to work on our psychology (by means of plausible-seeming but subtly manipulative messages). It might not take many bits of communication for an AI with the social manipulation superpower to bend us to its will.

Indeed, I think it is clear that this is already happening. While I don't ascribe malice (or any other motivation) to the AIs, it is clear that their makers have a bias towards obsequiousness.

Other predictions are perhaps a little wide of the mark:

if somebody were to succeed in creating an AI that could understand natural language as well as a human adult, they would in all likelihood also either already have succeeded in creating an AI that could do everything else that human intelligence can do, or they would be but a very short step from such a general capability.

We're a few years in to the LLM revolution and, while we can quibble about what "understand" means, it's clear that natural language can now mostly be interpreted by computers. But that doesn't seem to have made the leap to general intelligence, nor the acceleration of art and science.

Others are hopeful but possibly a bit naïve:

A future superintelligence occupies an epistemically superior vantage point: its beliefs are (probably, on most topics) more likely than ours to be true. We should therefore defer to the superintelligence’s opinion whenever feasible.

Yes, there probably are modern concepts which have more in common with "phlogiston" than reality. But if a scientist were to time-travel back to the early 1700s, how easy would it be for them to disprove the theory? Perhaps AI ought to exist in the "trust but verify" space?

It is slightly over-footnoted, with no distinction between citation and diverting passage. There's also a tendency to go off in fanciful directions - the stuff on genetically enhancing humans goes on a bit too long for my tastes. Similarly, the philosophy of maximising happiness by emulating brains and virtually doping them seemed unconvincing.

That said, some of the thought experiments are both fun and profound - the seminal "Paperclip Maximiser" was introduced in this book.

There are some downsides. An over-reliance on specific individuals like Eliezer Yudkowsky crowds out some of the other important thinkers.

One of the suggestions made has already fallen:

One valuable asset would be a donor network comprising individuals devoted to rational philanthropy, informed about existential risk, and discerning about the means of mitigation. It is especially desirable that the early-day funders be astute and altruistic, because they may have opportunities to shape the field’s culture before the usual venal interests take up position and entrench.

The "Effective Altruism" movement is now hopelessly compromised and seemingly in tatters. Similarly, the cult of rationalism has taken an unfortunate turn to the bizarre and dangerous.

Nevertheless, it's hard to argue with the philosophy. Whether or not "superintelligence" is ever achieved, we should have systems in place now to protect us. It's the same as any other technology - the time to set up nuclear non-proliferation agreements and the systems to monitor them was before we invented them.