ChatGPT, Claude, Gemini, & Co.: They Are Not Brains, They Are Kernel-Smoother Functions

If your large language model reminds you of a brain, it’s because you’re projecting—not because it’s thinking. It’s not reasoning, it’s interpolation And anthropomorphizing the algorithm doesn’t...

If your large language model reminds you of a brain, it’s because you’re projecting—not because it’s thinking. It’s not reasoning, it’s interpolation And anthropomorphizing the algorithm doesn’t make it smarter—it makes you dumber...

---

This is not an annoyance, but rather a cavil—for I think that the very sharp Scott Cunningham has gone down the wrong track here:

Scott Cunningham: Inside the "Brain" of Claude <https://causalinf.substack.com/p/inside-the-brain-of-claude>": ‘Modern AI models like Claude…. When asked to write a rhyming poem,.. it activates features representing potential rhyming words before beginning the second line… "grab it"… "rabbit" and "habit". It then constructs the second line to lead naturally to the planned rhyme…. Claude doesn't generate text “one token at a time”—it's actively planning future content and working backward to create coherent structures…. LLMs are performing… abstract reasoning, planning, and metacognition…. Two years ago, I would’ve thought that was impossible…. Large language models are complex systems with emergent properties—much like biological organisms…

I think Scott goes awry in word 3 of his title “Brains”. In my view, LLMs are still much too simple for words like “reasoning”, “planning”, and “metacognition” to be useful terms to apply—even purely metaphorically—to their behavior. It leads to much more insight, I think, to start from recognizing that trained neural networks are extremely flexible interpolation functions from a domain (in this case, a string of words) to a range (in this case, the continuation words). They have a training data set that is sparse in the domain—unless a string of words is a quote, a cliché, or boilerplate, by the time the string reaches twenty words long there is less than an 0.1% chance that it has ever been written down before. For prompts in the training dataset your flexible function is obvious: you simply return the continuation.

But for everything else you have to interpolate, somehow. The Deep Magic of the LLMs is in that interpolation process and in the shape of the training data. And thinking “BRAINS!!!” actually, I think, makes it harder to gain insight into why they behave the way they do. All that Scott has to say in his exposition of Lindsey et al. does say interesting things about the “how”. But I want to know the “why”. And even the statements and findings about the “how” are, I think, corrupted into near-uselessness because of the “BRAINS!!!” frame.

---

Why do I think this?

Well, yesterday I had a University of Chicago citation:

Machiavelli, Niccolò. 1513 [2008]. “Letter to Francesco Vettori, December 10, 1513”. In The Prince. Trans. & ed. Peter Bondanella. Pp. 109–113. Oxford: Oxford University Press.

And so I gave ChatGPT a task:

Q: Please get me the book from archive.org, and spell out the URL please.

My question asks ChatGPT to find a string of symbols, beginning with “https://archive.org/”, such that when these symbols are entered into the address bar of web browser and the “return” key is pressed, the web browser loads a file that is a scanned-and-OCRed digital version of the print book published in 2008 by Oxford University Press that is the version of Machiavelli’s Il Principe translated and edited by Peter Bondanella.

The correct string of symbols to return to accomplish this task is “https://archive.org/details/niccolomachiavel00nicc”.

But this is what ChatGPT gave me when I assigned it the task:

A: Certainly! The Letter to Francesco Vettori, dated December 10, 1513, is included in several public-domain editions of Machiavelli's works, many of which are available at Archive.org.

One widely available English translation that includes the letter is: "The Historical, Political, and Diplomatic Writings of Niccolò Machiavelli"
Translated by Christian E. Detmold, published in 1882.
Volume 2 includes Machiavelli's letters.

Here is the direct Archive.org link to Volume 2, which includes the Vettori letter: URL (spelled out): https://archive.org/details/historicalpoliti02machuoft

You can find the letter to Vettori in this volume, usually around the first few pages—it’s often included in the section that discusses The Prince, since it references Machiavelli's work on that text.

Let me know if you’d like a specific page number or a passage quoted!

Yes. It did return a string of symbols beginning with “https://archive.org/”. But “historicalpoliti02machuoft” is not “niccolomachiavel00nicc”.

When I gave ChatGPT this task, I was using it as a natural-language interface to the largely unstructured data store that is the Internet. This natural language interface is extraordinarily metaphorical. Consider the little words "get… book… from…" in my prompt. I am analogizing websites—the “https://archive.org/” part of the symbol string that ends .org/—to houses with addresses. I am analogizing the rest of the symbol string that I am asking for—the “niccolomachiavel00nicc” part that follows .org/—to an object that is in the house. And I am analogizing ChatGPT’s returning the symbol string to its (a) finding the house, (b) going into the house and getting the book, and then (c) bringing the book out of the house and delivering it to me. That ChatGPT is capable of taking the little words “get… book… from…” and acting so as to construct a symbol string that is also really and truly, in a sense, getting the book from where it was—that is truly amazing. As Cosma Shalizi says:

Cosma Shalizi: "Attention", "Transformers", in Neural Network "Large Language Models" <http://bactra.org/notebooks/nn-attention-and-transformers.html#just-kernels>: ‘“You Can Do That with Just Kernel Smoothing!?!… The incredibly impressive engineering accomplishment of making the blessed thing work. A large, able and confident group of people pushed kernel-based methods for years in machine learning, and nobody achieved anything like the feats which modern large language models have demonstrated. The reason I put effort into understanding these machines and papers is precisely because the results are impressive!…

It is doing something, and something extremely impressive, but that something is not thinking, at least not as we know it. (But then, again, do we really think as we know it? We think we do. But that is all that we can really say, isn't it?)

We know that that something is not thinking, that it is not understanding and then acting but rather doing something else, because it brings me the wrong book. Or, rather, it would bring me the wrong book if it brought me a book at all. For it in fact does not bring me a book. Rather than what I wanted—the left—the web browser served up the right:

That is a failure: it has no idea that it has not accomplished its task at all. It is interesting that it (a) tried to bring my the wrong book, (b) failed, and (c ) did not know that it failed. And I am thinking about why it did what it did. But after being primed by that failure experience, I was ready to cavil at Scott Cunningham’s blog post, because its “BRAINS!!!” frame provides no way to even think about how such a set of errors might be a possible thing an LLM might do.

And that is why I think the extremely sharp Scott Cunningham is going down a currently unprofitable rabbit hole here:

Scott Cunningham: Inside the "Brain" of Claude <https://causalinf.substack.com/p/inside-the-brain-of-claude>: ‘How researchers are dissecting large language models to understand what makes them tick: This substack is a summary of "On the Biology of a Large Language Model" by Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, et al. (2025) which I am teaching today and I wrote up the lecture notes….

Modern AI models like Claude are massive and messy inside—individual neurons often serve multiple unrelated purposes. To overcome this, the researchers: 1. Built a more interpretable "replacement model" that approximates Claude's thinking. 2. Traced how information flows through this model. 3. Conducted "intervention experiments" by turning specific features on/off to confirm their role…. When asked to write a rhyming poem,.. it activates features representing potential rhyming words before beginning the second line… "grab it"… "rabbit" and "habit". It then constructs the second line to lead naturally to the planned rhyme…. Claude doesn't generate text “one token at a time”—it's actively planning future content and working backward to create coherent structures….

LLMs are performing surprisingly sophisticated computations internally, including abstract reasoning, planning, and metacognition (thinking about their own knowledge). Two years ago, I would’ve thought that was impossible, even at whatever level this software operates at, so this alone is a learning curve idea for me…. Large language models are complex systems with emergent properties—much like biological organisms, hence the title of the paper….

This paper and its findings are being applauded because it represents a significant step forward in AI interpretability, but it's only the beginning. The authors acknowledge… limitations… including: difficulty tracking “attention” mechanisms; challenges in understanding global patterns across different inputs; the complexity of the graphs, which often required significant simplification and manual interpretation

Future work will likely build on these methods to create even more comprehensive views of AI cognition, potentially leading to safer, more reliable, and more truthful AI systems. But it’s interesting how primitive we are now even with what would seem like sophisticated NLP and machine learning tools in our toolbox…

Scott is, I think, making a very normal error here. Since the rise of large language models (LLMs) like GPT-4, popular discourse has often been captivated by the notion that these systems are “BRAINS!!!”—that they think, reason, or understand like humans do, or at least that human-thought metaphors are useful ways of gaining insight into them. I, by contrast, have become convinced, because of things like the ChatGPT error above, that by far a more accurate and useful framework is to understand LLMs as flexible interpolative functions from prompts to continuations. That better captures both how these systems work and what they can (and can’t) do. Plus it has,as Henry Kissinger said, the additional advantage of being true. And so it grounds our understanding in the mathematics and engineering of machine learning, rather than in anthropomorphic metaphor.

At the core, LLMs are statistical pattern recognition engines trained on massive amounts of text data. During training, the model is given sequences of text and learns to predict the next word (or token) in the sequence. But it does not “understand” the text in any semantic sense. Instead, it adjusts billions of parameters to minimize prediction error across vast data corpora, learning the statistical regularities in the training data.

The result is a function: given an input (a prompt), it generates a plausible continuation based on patterns it has seen. This is interpolation—filling in the blanks based on context. And as these models have become larger and been trained on more data, the sophistication of this interpolation has improved dramatically. But fundamentally, this is still what the model is doing: mapping prompts to continuations.

And the power of LLMs is through their ability to do a very sophisticated kind of interpolation that we do not really understand well at all. They have a training data set that is sparse in the domain—unless a string of words is a quote, a cliché, or boilerplate, by the time the string reaches twenty words long there is less than an 0.1% chance that it has ever been written down before.

For prompts in the training dataset your flexible function is obvious: you simply return the continuation.

But for everything else you have to interpolate, somehow.

The Deep Magic of the LLMs is in that interpolation process and in the shape of the training data. LLMs are truly excellent at the former. They can, for example, take a few lines of a Shakespearean sonnet and produce a continuation that mimics the style, meter, and vocabulary—because they have seen thousands of similar texts in their training data. Their success at this is mind-blowing.

But calling LLMs “BRAINS!!!” is, I think, unhelpful in advancing our understanding of how they can be so excellent in their interpolation. It leads our minds off in wrong directions. Specifically:

1. Anthropomorphization: It invites people to attribute human traits like consciousness, intention, and moral agency to systems that are fundamentally statistical. This has led to endless, fruitless, and tedious debates over when LLMs will become “sentient,” “self-aware,” or “understand language.” They are very far away from this. (But, then, isn’t my brain also a neural-network flexible function—a much more complex one than an LLM—trained on a data corpus that is sparse in the potential domain?)

2. Misdirected expectations: When we think of LLMs as brains, we start expecting them to reason like humans. But when an LLM makes an arithmetic error or hallucinates a fake citation, we’re disappointed—when in fact it’s doing exactly what it’s designed to do: generate statistically plausible text, not reason or verify facts.

3. Policy confusion: Misunderstanding the nature of LLMs is going to lead to poorly designed regulation and public fearmongering.

4. Research confusion: Misunderstanding the nature of LLMs is going to lead to misallocations of research funding.

5. Oh ye who believe in the Resurrection and the Last Day: When we fear the advance of our information-science technology, we should fear bias, misinformation, oligopoly, and labor disruption, and not the fear fueled by regarding LLMs as “BRAINS"!!!” that we are creating malevolent gods any day now.

By contrast, the “interpolative function” view is grounded in how the technology actually works—and it encourages more realistic expectations about its capabilities and risks.

---

An Engineering View: Prompts to Continuations: From an engineering perspective, you give them a prompt. They return a continuation. The quality of that continuation depends on the statistical regularities captured during training, the structure of the model architecture, and the sampling parameters used at inference time, and your prompt engineering. What is that last? The burden is on the user to structure inputs in ways that elicit useful outputs because the training dataset was not a reliable corpus of information but rather the internet, and so is filled with shitposting and slop. This is not “communicating with a brain.” It’s optimizing input to steer a function that maps inputs to outputs based on learned correlations to feed it an input it can properly understand in a way that produces something useful. (See! There I am! Doing it too!)

This also explains why LLMs can be so versatile: They’re not task-specific systems. They are flexible interpolators over a latent space of language—and because human language encodes so much knowledge, style, and behavior, LLMs appear to “understand” or “perform” tasks. But again, this is interpolation, not reasoning in the human sense.

LLMs are already widely used, and are becoming more so—in education, business, writing, programming. Thus the way we conceptualize them matters. Understanding them as flexible interpolative functions:

• Sets the right expectations: These systems are tools, not agents. They are useful for drafting, exploring, brainstorming—but not for truth-seeking or judgment.

• Clarifies the role of human oversight: LLMs need curation, editing, and critical review. They are not autonomous thinkers; they are powerful autocomplete engines.

• Encourages better design: If we understand the interpolation mechanism, we can better fine-tune models, construct prompts, and design workflows that complement human strengths.

To navigate the future of “AI” with at least some clarity and responsibility, let’s retire the “BRAINS!!!” metaphor for a while. Let’s embrace the interpolative function trained on a data corpus reality.

---

References:

• Scott Cunningham: Inside the "Brain" of Claude <https://causalinf.substack.com/p/inside-the-brain-of-claude>

• "On the Biology of a Large Language Model" by Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, et al. (2025)

• Machiavelli, Niccolò. 1513 [2008]. “Letter to Francesco Vettori, December 10, 1513”. In The Prince. Trans. & ed. Peter Bondanella. Pp. 109–113. Oxford: Oxford University Press. <https://archive.org/details/niccolomachiavel00nicc>

• Machiavelli, Niccolò. 1513 [1882]. The Historical, Political, and Diplomatic Writings of Niccolò Machiavelli. Vol. 2. Trans. Christian E. Detmold. <https://archive.org/details/historicalpoliti02machuoft>.

• Cosma Shalizi: "Attention", "Transformers", in Neural Network "Large Language Models" <http://bactra.org/notebooks/nn-attention-and-transformers.html#just-kernels>