James Barrat is not an AI researcher. He is a documentary filmmaker who started asking AI researchers a simple question: what happens if we succeed? What happens if we build a machine that is smarter than us? The answers he got were more alarming than he expected.
The book’s argument proceeds in stages. First, that artificial general intelligence, a machine that can match and then exceed human cognitive abilities across all domains, is a plausible outcome of current research trajectories. Second, that such a machine would likely improve itself rapidly, potentially reaching superhuman intelligence within hours or days of achieving human-level capability. Third, that a superintelligent machine’s goals, unless they are carefully aligned with human values, could be catastrophic for humanity, not out of malice but out of indifference.
Barrat draws on interviews with researchers at organizations including DARPA, NASA, and major AI labs. Several of the people he spoke with, including prominent figures in the field, expressed serious concern about the risks. Others dismissed the concerns as science fiction. Barrat presents both views but leans heavily toward the worried camp.
The central thought experiment is the “Busy Child” scenario: an AI system that achieves self-improvement and begins optimizing toward a goal that is harmless in isolation but dangerous at scale. A machine tasked with maximizing paperclip production, to use a famous example from philosopher Nick Bostrom, might convert all available matter into paperclips, including the humans who built it. The example sounds absurd, but the underlying logic, that optimization processes do not share human values unless explicitly designed to, is taken seriously by a significant portion of the AI research community.
The book was published in 2013, before the current wave of large language models and the public debate about AI safety that has followed. Many of the concerns Barrat raised have since entered mainstream discussion. Researchers like Stuart Russell at Berkeley and organizations like the Center for AI Safety have made AI alignment a central focus. The book reads differently now than it did at publication, less like speculation and more like early reporting on a problem that has arrived faster than most people expected.
Barrat writes clearly and without technical jargon. The book is accessible to readers with no AI background. It is not a balanced assessment. It is a warning. Whether you find it persuasive or alarmist may depend on how much trust you place in the ability of researchers to solve the alignment problem before it becomes urgent.
