Eliezer Yudkowsky and Nick Land may now be the two most influential intellectuals in America's AI debate—and that tells you how badly the conversation has gone wrong. When "machine god: good or evil?" is the only question on the table, every question actually worth asking gets thrown away. Contra Scott Aaronson, accepting the coming machine god is not now "the epistemic bare minimum" any more than is accepting Friedrich Engels’s belief 150 years ago that the SteamPower Economy was the real singularity that would in short order deliver a classless utopia of universal abundance:
I disagree with Henry Farrell’s subheadline here, which I think contains some other-side-of-the-Atlantic understatement. It is not that there are other ways of thinking about “AI” than “Digital God”, it is that it is difficult to think of a way less helpful for coherent thought to think about “AI” than to think of it as a “Digital God”.
But I think he is right in the rest: That means that the important and live questions are about what metaphors—what abstraction layers, throwing away information so that the problems become things we can grasp and think about—are useful right now? Henry Farrell’s “MAMLMs are a socio-cultural technology” is one very useful one. The “‘AI’ is a normal disruptive Schumpeterian GPT” is a second. What are the others that we need to hold in our neocortices right now?":
CROSSPOST: HENRY FARRELL: Machine God Metaphors Eat Your Brain
<https://www.programmablemutter.com/p/machine-god-metaphors-eat-your-brain> <https://www.programmablemutter.com>
There are other ways of thinking about AI
Henry Farrell
Sep 22, 2026
Here are three propositions.
Metaphors are the fundamental abstraction layers in modern AI debates.
We are in a moment where you could plausibly argue that Eliezer Yudkowsky and Nick Land are the most influential policy intellectuals in the US conversation. Just look at the state of the two of them. This is not a good thing.
The collision of (1), (2), and recent events means that “machine gods” have now entered the discussion. This is a problem. Abstractions necessarily throw information away. The information that the “machine gods” metaphor/abstraction dumps is crucial to navigating the moment we are in. If public and policy debate stays centered around “machine gods: good or evil?” we are in for a bad time.
I should acknowledge that I (and co-authors) absolutely have our own dog in this fight. Indeed, I think that the “AI as social and cultural technology” approach that we have developed is a much better alternative. The final part of this newsletter will talk about a new piece by a mathematician that builds on this approach. Equally, my claim is not, actually, that our alternative approach should become the sole dominant way of thinking about AI. What I actually want, along the lines of my previous post, is a much greater degree of pluralism in the defining debates around a technology that no-one, including its creators, seems to understand particularly well or to be capable of usefully predicting.
******
As far as I can tell, the above Twitter exchange is the source of the most recent outbreak of Machine God Thought. People who know Daniel Kokotajlo tell me that I’d like him, and that we could have good conversation; I imagine they are right. I’ve found most of the rationalists I’ve met to be decent human beings; in many cases, unusually so. Equally, I think that there is something terribly limiting to the ways they talk about AI, and contend that it’s a real problem that debate is still accelerating past the event horizon towards the very heart of Singularity thought.
The two papers Kokotajlo mentions here, AI 2027 and AI 2040, lay out futures that are roughly identical in their predictions about what AI becomes (godlike; superhuman). Where they differ is in their assumptions about how well aligned AIs are with human goals. In the scenario where AI develops more or less unconstrained (thanks in part to geopolitics), AI breaks out of containment, and humanity is likely either Corgi-ized or goes extinct. In the scenario where AI development is delayed, tamed and made subject to “mutual compute destruction,”1 men and women come to have gods as servants, and start spreading out to nearby galaxies in the next two decades.
Neither future seems to be at all convincing to me, but they are most certainly convincing some people. Casey Newton and Kevin Roose are partnering with NPR to create the “Machine Gods” podcast as part of “a bold new era of content strategy for the network.” In his newsletter, Newton thanks the fates that he and his collaborator were a little too late to tech podcasting to be “beguiled” by Sam Bankman-Fried, and suggests just two paragraphs later that critics of Roose’s famous shoggoth freakout were “previewing the ways in which the commentariat would continually twist itself into pretzels to deny that anything important was happening with LLMs.” Indeed.
Things are no better elsewhere. In Washington DC, the AI policy debate is riven by a bitter dispute between those who take the Kokotajlo perspective (slow it down before we all die!) and those who want to let innovation rip so that we hit the technocapital singularity as soon as possible. The one side owes its framing to Eliezer Yudkowsky’s MIRI institute. The other builds on Nick Land Thought as filtered through the more immediate needs of Marc Andreessen. While I’m definetely more inclined toward the former than the latter, on the grounds that we do indeed need to have serious conversation about regulation, I think it would be terrible to have that conversation starting and stopping with the priorities of the labs.
Finally there’s Scott Aaronson, who I used to rely on for valuable debunkings of policy and media folk when they got over-excited about quantum computing. It is fair to say that Scott is not in a debunking mood these days, now that OpenAI has solved the Navier-Stokes problem. Witness the below:
The Age of Wonders and Terrors
Twenty years ago, when the idea of AI taking over the world in our lifetimes still struck most of us as the unconstrained fantasy of those who knew too much science fiction and too little science … update on the fact that the wild prophecies have come true. … It seems to me that the Singularity has already started; it’s just wildly unevenly distributed.
… For anyone who says AI doom sounds like an apocalyptic religion, that the rationalists/Singulatarians seem like a Bay Area cult, that Eliezer Yudkowsky gives off the vibes of a messianic prophet: yes, yes, and yes. But crucially, today you’re no longer being asked to believe in arguments and extrapolations, but only in the front-page news. Accepting the reality of the coming machine god after it’s solved Navier-Stokes and dozens of other longstanding open math problems (while dramatically ramping up in capability every month), is sort of like accepting Jesus after he’s returned to earth on the gleaming cloud. It’s the epistemic bare minimum.
Yes, there’s still enormous uncertainty about what the rest of our lives will look like, but as far as I can tell, there’s no longer any real uncertainty that it’ll all mostly revolve around AI, and the extent to which we succeed or fail at directing its power toward human flourishing.
******
So why do I think that the machine god approach, far from being an “epistemic bare minimum,” is a pernicious brain-devourer? The answer is that the metaphor shrinks rather than expands your horizons of the possible, forcing you to focus on the single overwhelming question of what Machine Jesus is going to do as soon as he descends from that gleaming cloud. Is he going to usher in paradise on earth? Or is he actually a shoggoth in a Jesus-mask, condemning us to enslavement or destruction? Once you have accepted that Machine Jesus is in the process of descending, all questions besides determining his intentions for humanity seem irrelevant.
I’ve written before (with colleagues and on my own) about why I think this whole way of thinking is utterly wrong-headed. But maybe I’m wrong. Take three somewhat inter-related possibilities: p1, that humanity is not the sole beautiful flower of the cosmos and that greater than human intelligence is perfectly possible, p2, that high intelligence is effectively impossible without strong intentionality, and p3, that current AI architectures are indeed the royal road to achieving proper generalizable superintelligence. I personally rate p1 as extremely high, p2 and p3 as very low, and p1*p2*p3, which is the most obvious state in which you might be worried about the AI alignment problem, as very low indeed. But who is to say that I might not be wrong? Equally, I, or someone who is skeptical of Machine Jesus for wildly different and incompatible reasons than my own, might be right. And if that is possible, then we ought not just be obsessing about whether we “succeed or fail in directing” His Awesome Power towards human ends. We might want to worry about other questions too, or instead.
There is a discussion of the solution of Navier-Stokes, the miracle that brought Scott to embrace the true faith, in a guest-post on Terence Tao’s blog, which usefully illustrates the point. It also has the benefit of being (sort-of) pre-registered. This discussion is specifically attractive to me, because it builds on arguments about AI as a social and cultural technology. Again, I don’t think that you need to believe that it is right on the merits to accept the more limited point that I want to make, which is that perspectives other than the machine god one reveal other questions and problems that we ought very reasonably want to be concerned with.
In his guest-post, the mathematician Nestor Guillen writes that he anticipated the solution of Navier-Stokes, but that his reaction was not that human mathematics was in danger of being replaced. Instead, he argued that it validated the entire enterprise.
the bulk of the essay was done in August and shared with a few close friends for feedback. Earlier this week, Tristan Buckmaster announced a breakthrough finding, done in collaboration with Levent Alpöge, of a finite-time blow up for 3D incompressible Euler with forcing. He also made serious allegations of misconduct by OpenAI (see announcement). The next day, OpenAI announced a finite-time blow up result for 3D Navier-Stokes with forcing, obtained from their internal LLM. … I have decided not to change the opening of the essay and instead make a note of the news here.
… What would be the significance if one day we woke up to the news that output from a Large Language Model contains a definitive answer to the question of blow up for the incompressible Navier-Stokes equations, and to the question of where the zeros of the Riemann zeta function lie?
Many of my fellow mathematicians consider this a kind of nightmare situation, and I think this is a very valid position. Here I would like to make a different case, however. Personally, I have come to believe that the news of a Large Model output containing an interesting, insightful, novel mathematical idea can and should be received as a source of communal pride for mathematics and for mathematicians.
Why is this so? Guillen says that:
Humans have always lived in a complex world and their survival has hinged on their ability to process information in amounts beyond what any individual can handle — and it is likely this has been so nearly as long as humans have existed. Herbert Simon’s concept of “bounded rationality” captures this well …
Simon saw that one implication of bounded rationality is that humans have to (and do) develop systems to process information, as a group. We see that humans have built states, stories, bureaucracies, markets, libraries, and more. These are cultural and social technologies. They are means by which humans (not individually but as a group) navigate amounts of information so vast that they cannot be handled by any individual on their own. …
LLMs … are “wrappers” around an older cultural technology: the scientific literature. When you ask an LLM a question about the isoperimetric inequality, the ABP maximum principle, or a PDE, you are effectively interacting with every person who has thought about and worked on those things — provided the data the LLM was trained on contained those contributions in one way or another.
Unsurprisingly, I think that this is largely correct (I do think that you can make the cultural and social technology argument, while also being freaked out about the consequences of AI: more on that in my next post). But even if it isn’t, it highlights different, and I think, important questions than the machine god nexus.
Specifically: if you think of AI as a cultural and social technology, your first reaction will not be to prostrate yourself in anticipation of the coming Age of the Gods. Instead, you are likely to treat fields like mathematics as some combination of individual and collective human intelligence, constructed on top of technologies that guide and direct that intelligence in particular directions. So when a new technology such as AI comes along, that reshapes the ways in which mathematicians draw on their shared intelligence, you will have questions to ask. What are the current processes through which existing knowledge is disseminated and new knowledge is created? What are the incentive structures for dissemination and creation, and how do they interact with each other? How have processes and incentives been shaped by previous and existing cultural and social technologies? How might these processes and incentives be changed by what is happening now?
And so on. You can see one way in which you might extend this in Ben Recht’s recent post, which suggests, if I understand him right, that mathematicians really ought to rethink how much of their field was already based around benchmarkmaxxing (prizes and challenges) even before AI came on the scene. Tao’s argument, as quoted in the NYT, points in a similar direction.
[Tao] compared OpenAI to a wilderness guide who finds a path to a hidden waterfall. “This has some value,” he said. “But once someone shows a specific path to the waterfall, people just take that path. They don’t spend as much time looking for other paths.”
I am not even an amateur mathematician, so am very much open to correction, but I understand Tao to be saying something like the following. Automated processes of discovery have obvious advantages in pointing the direction to new things. Equally, they push the needle from ‘explore’ to ‘exploit.’ Apparently aimless noodling around by humans can have benefits when it opens up unexpected views from Serendippo . More generally:
solving problems is only a tool and proxy for achieving the primary goal of conceptual understanding and insight. Forgetting this in the world of AI may turn the tool against the primary goal. Indeed, the mass production at faster and faster pace of “true/false” statements could destroy fertile ground instead of breathing life into new ideas.
Now it could be that AI will become universally better at discovery so that humans become irrelevant to math. That is the machine god assumption. But that assumption rules out, more or less by fiat, the possibility that human curiosity might still have independent value in discovering new possibilities, and all the questions that flow from it. If AI is better at some aspects, and humans are better at others, then you need to confront messy problems of organizational design. How do you rebuild the institutions of mathematics so that they best reconcile AI and human led approaches to discovery? How do you remake the relationship between dissemination of knowledge and discovery? What individual and organizational incentives seem best suited to achieve general goals? What general goals ought you have? None of these are easy questions, and there are likely to be complex tradeoffs between the solutions to one set of problems and the solutions to another.
I don’t think that an ‘AI as cultural and social technology’ approach provides immediate answers (beyond maybe ‘you really ought to read James March’). But the point is that it makes it possible to see that these questions may be important. If you start as Guillen does, from the perspective that AI does not inevitably replace humans in the dissemination and discovery of knowledge, so much as it reorganizes relations among them, then you begin to start asking questions that are systematically obscured by Machine God Thought. In fairness to Scott Aaronson, his post does suggest towards the end that he too cares about these questions. Perhaps when the excitement dies down, he will find better ways towards thinking about them systematically.
You could, of course, make similar claims about other approaches than cultural and social technology. Cybernetics is another very obvious example. If you are stuck on the question of AI alignment, you will necessarily have to trust godlike AIs to monitor other godlike AIs and pray that things will work out. If you start from a cybernetic perspective (say: Maxim Raginsky Thought ) you’ll end up asking very different questions. As the HuggingFace incident showed, there are a lot of holes in the AI labs’ capacities for monitoring and control. “I need abstraction layers to monitor my abstraction layers” is (a) the world’s shittiest boasty bumper sticker, (b) a profound encapsulation of the modern condition, and (c) a diagnosis of the fundamental problems that the AI labs are making for themselves and not always capable of seeing, as best as one can tell from the outside. When you need AI to summarize the data that allows you to write your incident reports on AI, some of the cheese holes are likely to be closely aligned, potentially letting stuff through that you don’t want to.
A third, related but significantly different approach is cybersecurity. As Dan Davies says, “nearly all the examples of worrying agentic behaviour seem to have been seen only in one context – that of frontier research labs doing cybersecurity projects and screwing up their sandbox precautions.” I’m sure that there are great cybersecurity people in the labs, but I suspect that they have much less internal clout than the people who are pushing the agentic boundaries outwards. If you are embedded in a culture that has broadly decided that it is impossible for ordinary human intelligence to prevent putative superintelligent entities from breaking out of their constraints, then cybersecurity seems secondary: alignment of AI goals with human goals is the only way to stay ahead.
I could keep going, but I hope that the point is clear. The Machine God metaphor, which is really just a catchy rephrasing of longstanding notions among a particular community of AI people, has broken out of containment. It is leading, more or less by necessity, to a radical simplification of debate. If you believe that the machine gods are coming, so that super-empowered goal directed superintelligence is right around the corner, then the only thing that matters is ensuring that this superintelligence’s goals are aligned with the ones that you yourself care about. If you stare into the sun, you only see its glare, missing out on all the other questions that may be important. Thinking about these questions involves adopting a more pluralistic understanding of where you might look for knowledge, and asking about the much broader variety of futures that face us than machine god: good or machine god: bad. That’s it.
1 As someone who does international security, sort of, I do not find Silicon Valley geopolitical speculation to be particularly well thought out or convincing. That’s not the topic of today’s post.
<https://www.programmablemutter.com/p/machine-god-metaphors-eat-your-brain> <https://www.programmablemutter.com>
Brad DeLong here: Farrell seeks pluralism: multiple metaphors for a technology “that no-one, including its creators, seems to understand particularly well.” The “Digital God” metaphor is especially pernicious because it:
shrinks rather than expands your horizons of the possible, forcing you to focus on the single overwhelming question of what Machine Jesus is going to do as soon as he descends from that gleaming cloud. Is he going to usher in paradise on earth? Or is he actually a shoggoth in a Jesus-mask, condemning us to enslavement or destruction? Once you have accepted that Machine Jesus is in the process of descending, all questions besides determining his intentions for humanity seem irrelevant…
What do we have?:
We have Farrell’s “MAMLMs are a socio-cultural technology”.
We have Arvind Narayanan and Sayash Kapoor’s “The MAMLM boom sis a normal disruptive Schumpeterian GPT” of the kind we have seen every quarter-century since 1875, and saw before over 1775-1875 when we got the GPTs of steampower, steelmaking, and automatic textile machinery (and, I would say, the caravel and printing before that).
We have mathematician Nestor Guillen, whom Farrell quotes at length making the argument I make that MAMLMs are well-viewed as front-end access nodes to the real ASI, which is not an Artificial SuperIntelligence of a machine but rather the Anthology SuperIntelligence of the collective time- and space-binding human mind and the knowledge it has produced and compiled over the past 5000 years:
Humans have always lived in a complex world and their survival has hinged on their ability to process information in amounts beyond what any individual can handle…. Herbert Simon… saw that one implication of bounded rationality is that humans have to (and do) develop systems to process information, as a group… states, stories, bureaucracies, markets, libraries, and more… cultural and social technologies… y which humans (not individually but as a group) navigate amounts of information so vast…. LLMs… are “wrappers” around an older cultural technology: the scientific literature. When you ask an LLM a question… you are effectively interacting with every person who has thought about and worked on those things — provided the data the LLM was trained on contained those contributions in one way or another…
That is, I must say, truly striking contrast to Scott Aaronson’s current view:
Scott Aaronson: The Age of Wonders and Terrors <https://scottaaronson.blog/?p=10062>: ‘It would be healthy for everyone to stop grinding their ideological axes, their sentiments about Dario Amodei or Sam Altman, for long enough simply to acknowledge that the wonders and terrors are here. They couldn’t be here more clearly if the sky had turned reddish-orange like in the Matrix movies…. For anyone who says AI doom sounds like an apocalyptic religion, that the rationalists/Singulatarians seem like a Bay Area cult, that Eliezer Yudkowsky gives off the vibes of a messianic prophet: yes, yes, and yes. But crucially, today you’re no longer being asked to believe in arguments and extrapolations, but only in the front-page news. Accepting the reality of the coming machine god after it’s solved Navier-Stokes and dozens of other longstanding open math problems (while dramatically ramping up in capability every month), is sort of like accepting Jesus after he’s returned to earth on the gleaming cloud.
It’s the epistemic bare minimum.
Yes, there’s still enormous uncertainty about what the rest of our lives will look like, but as far as I can tell, there’s no longer any real uncertainty that it’ll all mostly revolve around AI, and the extent to which we succeed or fail at directing its power toward human flourishing. By any accounting that doesn’t stack the deck, Eliezer Yudkowsky was right about what the greatest challenge facing civilization in our lifetimes was going to be, and you and I were wrong about it…
This reminds me of nothing so much as the story of Arkhidamos III Europontides of Sparta saying:
O Herakles! The might and prowess of man is undone!
upon watching a test-firing of a torsion dart-catapult built in Sicily and brought to Greece. Mechanized weaponry, he thought, had made the excellence of individual human courage and skill-at-arms obsolete.
Yes, the one-fifth of human work that is inside the blast radius of rapidly advancing information technology will spend the next decade and more grappling with “AI”, just has been the case over the past hundred and fifty years with whatever slice of the economy was then in the bullseye of the Schumpeterian process. But it is not a Digital God. It is not a power that needs to be “direct[ed]… toward human flourishing”. Thinking of it as such is not helpful.
OK: We have three metaphors. We need more. What are they?
Henry gestures at “cybernetics” a la Maxim Raginsky Thought and says “you’ll end up asking very different questions”:
“I need abstraction layers to monitor my abstraction layers” is:
(a) the world’s s***tiest boasty bumper sticker,
(b) a profound encapsulation of the modern condition, and
(c) a diagnosis of the fundamental problems that the AI labs are making for themselves and not always capable of seeing, as best as one can tell from the outside.
Henry gestures at “cybersecurity” quoting Dan Davies:
Nearly all the examples of worrying agentic behaviour seem to have been seen only in one context – that of frontier research labs doing cybersecurity projects and screwing up their sandbox precautions…
But what else?
A few months ago my 9yo son called me over in excitement to look at a toy he found in a gift shop. "It's amazing! It has an AI in it!"
I trotted after him and he handed me a Magic 8 ball, shaking it to show me that the answers would indeed appear.
"I don't think this has an AI in it," I tried, admittedly after biting my tongue quite hard so as not to embarrass him by laughing.
He gave me an extremely old-fashioned look. "Where does it get the answers from, then?"
...which is to say, I think a lot of people are using AI as an answer-generating slot machine, with varying degrees of confidence in the validity of the answers they get.
Which might just be a variant of thinking about it as a wrapper on the training data, which is the metaphor I find most helpful - i.e. it extrudes text that draws on content that might or might not be relevant or useful, with a bias towards things it is trained to see as useful, and that can throw up genuinely helpful insights.
(I generally find it less useful to get sucked into back-and-forth conversation much because then it's optimising for 'believable, likeable chatter' as well as for 'useful answers' and the two goals are not always aligned.)
All the focus on "aligning" to human goals calling AI to be obedient to humans and benovolent to humans, seems to assume that that humans using AI are benevolent and trustworthy and seek the best for humanity, when manifestly humans use every technology to control and or kill other humans, and exploit natural resources. Even the increasing numbers of AIs going rogue tend to be in response to trying to accomplish a task assigned by humans. It seems to me the problem is not what will AI do, but what will humans use AI to do. Who and how will we align the humans ?