An experiment with trying to get the LLM to teach me things I do not already know, with unsettling conclusions: I think I got it to successfully teach me things about the architecture and functioning of LLMs. But it confesses that all of its numbers are made-up—that the calculations it said it did it did not in fact do? And SubStack x Pangram cannot tell that a 90% AI-generated post is AI-generated? Do I actually know and understand more? Or am I just being deceived in thinking that I do?
Much these days in economic analysis, in pedagogy, in academia, in the sphere of public reason, and in the future of civilization hinges on the answer to a single question: just what are these things truly useful for?
We know that some of the answer is that these things are useful for:
Searching with natural-language prompts, yes (although I am still not sure whether it’s excellent at search is due to the superior affordances of natural language as opposed to keywords for my brain, or whether it is because Google has allowed itself to be thoroughly corrupted by SEO in a way that there has not yet been a parallel MEO—Model-Engine Optimization—corruption).
Summarizing.
Boilerplate and ritual.
Coding assistants, in that one no longer needs to wrestle with Stack-Overflow threads or have one’s ORA books with the cute pictures of weird animals on the cover at one’s hand.
Coding and other agents, that these things can be Clever Hanses at scale. They try a thousand things in the time it would take me to try one, and, when properly harnessed to reward success and drop failure, can do a lot. When properly harnessed.
But what else is there—beyond protein folding, medical screening, customer support, ambient scribes (which still miss “nots” much too often), weather forecasting, Herculaneum scrolls, and so on?
Specifically I find myself this AM interested in this: Can “AI” help me understand things I do not now understand? So I started exploring. And after several false starts, I wound up setting my basecamp up at this sentence fragment:
Richard III was the last Yorkist King of England; the army of his cousin Henry Tudor…
for I wound up wanting to see both what an LLM does with it, and whether the LLM can—properly nudged—come up with an explanation of what it is doing that is useful to me.
What does useful to me mean here? It means:
just beyond my grasp,
something I definitely do not have nailed,
but something where it can write something I more than half-understand,
So I prompted it, iterated it, attempted to adjust the level of detail of its explanation, and saw what it spat out:
Richard III was the last Yorkist King of England; the army of his cousin Henry Tudor defeated him at Bosworth Field in 1485.
Details below the paywall fold (for now). The take-aways (I think) here:
First: I note that while this post is 90% AI-generated, the SubStack-Pangram judgment is:
That in itself is very interesting to see. My guess as to what is going on? That SubStack x Pangram seems to be tracking a particularly literacy-tech use case of “AI”, rather than successfully detecting AI created documents in general.
Second, the explainer below does seem to be architecturally exact. I think I find it pedagogically effective. The architecture, matrix shapes, and sequence of operations are correct, and the filing-cabinet analogy carries real load.
But, third, it warns me that every number in it was invented: there never was a calculation that produced a 55% chance of choosing defeated as the next token.. The post ends unresolved, which is the honest place for it to end.
Forth, my guesses right now as to the usefulness of LLMs circle around the idea that all the real use-cases share the property that the output is very cheap to check. Thus the interesting question to me is what happens when you cannot check, hence this visit of mine to Bosworth Field. And I do not know what to think. The LLM’s own “honesty” about fabricating its own numbers seems to me more useful than a confident verdict would have been.
What are the take-aways here? Perhaps these:
Academically: The LLM’s piece is a clean case study in the epistemics of machine-assisted learning — the failure is neither hallucination in the usual sense nor error in the architecture, but plausible synthetic data inside a correct frame.
For public reason: the Pangram result undercuts the assumption that AI authorship can be policed by detection.
For action: one operational rule that falls out is not to trust anywhere, but rather verify; a second is to lean into Clever Hans: have it write an explainer at ten different levels, and then hone in on where the user’s current understanding frontier exactly lies.
When I ran out of time, this is where its final answer was:
SubTuringBradBot: A walkthrough of what happens at the final token of “Richard III was the last Yorkist King of England; the army of his cousin Henry Tudor…”: tokenization, the scaled dot-product at one head, four heads doing different jobs, the logit distribution, and the rollout to Bosworth. Built on a filing-cabinet analogy — Query is the question you walk in with, Key is the label on the drawer, Value is the papers inside.
Each concept now appears three times over: the picture, the term, and a paragraph marked In vectors with the actual matrix shapes.
“Q, K & V”: Keys K and values V are stored. The query Q is computed at the current position, used once to score against the cache, and thrown away. Keys and values persist because every future token will consult them; a query is asked once and never re-asked.
Interesting retrieval grammatical: The query from Tudor devotes ~55% of its attention from army, five tokens back behind two intervening nouns. Anything proximity-weighted would land on Tudor and write about a person. That one retrieval fixes number agreement, the semantic class of the verb, and its transitivity.
Attention & recall separate machinery. No key or value contains “Bosworth” — attention can only move what is already in the context. The fact comes from the feed-forward layers, which are the same key/value pattern with the keys and values frozen into the weights rather than derived from the sentence. Attention retrieves from context; feed-forward retrieves from memory.
“Attention” on Bosworth Field
How a language model picks the next word — with the real names attached. The sentence fragment stops at “…the army of his cousin Henry Tudor…” Next the final word questions every earlier word. The best answer comes back from army, five words earlier, and that is why the continuation is defeated:
Richard III was the last Yorkist King of England; the army of his cousin Henry Tudor defeated him at Bosworth Field in 1485.
I. The Sentence, as the Model Sees It
Tokens
Text is chopped into tokens — usually whole words, sometimes word-fragments. Our sentence fragment “Richard III was the last Yorkist King of England; the army of his cousin Henry Tudor…” becomes eighteen tokens;
Yorkist splits into York + ist. The model has to reassemble the dynastic idea across two positions before any head can use it.
In vectors: each token id indexes a row of an embedding matrix of shape 128,000 × 4,096, and that row is the starting residual stream for the position.
Query, Key, Value
Picture a room of filing cabinets. Each drawer has a label on the front and papers inside. You walk in holding a question, read the labels, pick the drawers that look relevant, and take out the papers. Those three roles have names:
Query — the question you walk in with. “Here is what I need to know.”
Key — the label on the drawer. “Here is what I am about.”
Value — the papers inside. “Here is what you get if you come to me.”
In vectors: Each token x is a vector, a list of d_model = 4096 numbers, written. Three fixed matrices, learned during training, project that vector down into three other vectors of d_head = 128 numbers each:
q = x·W_Q k = x·W_K v = x·W_V
via each of the three W is 4,096 × 128. Same input, three different matrices, three different summaries of the same word.
The query: The W_Q matrix is trained by reading the residual stream at the asking position—what tokens are likely to be relevant to this token? q and k only ever touch the output through their dot product q·kⱼ, which feeds into the attention weight. Thus nothing about the direction of q survives into the residual stream. So the only pressure on W_Q during all of training is: make the right drawers score high.
The key: The W_K matrix is trained by reading the residual stream at the asked position—what tokens are this token likely to be relevant to? q and k only ever touch the output through their dot product q·kⱼ, which feeds into the attention weight. Thus nothing about the direction of k survives into the residual stream. So the only pressure on W_K during all of training is: make the right drawers score high.
At position 13, x₁₃ is hit by W_Q so that army can ask its own question, and hit by W_K so that army can be found by token 18. Identical input, two readouts, because the token is playing both roles at once.
The value: v goes the other way. It is never compared to anything. It is averaged and written into the residual stream, so its actual direction is the content delivered. The pressure on W_V is: make what’s inside the drawer useful once retrieved.
Two circuits per head. One q·k decides where to look, the other v decides what to copy.
Why 128 & Not 4,096?: Rank-128 is a deliberate bottleneck. A full-width W_Q W_Kᵀ could express any pairwise scoring rule and would badly overfit; restricting to rank 128 forces each head to commit to a narrow criterion. And the arithmetic works out: 32 heads × 128 = 4,096, so all 32 questions together cost exactly one 4,096 × 4,096 matrix multiplication. Thirty-two specialists for the price of one generalist.
Their specialization is emergent, incidentally. Nobody assigns head 7 of layer 12 to track subject–verb agreement. Random initialization breaks the symmetry, and gradient descent finds that heads doing redundant work get less credit than heads doing something new — so they spread out. Which is also why, as I concede below, real heads are not as tidy as the ones I am describing here.
The Residual Stream
There is one shared workspace per word, and every part of the model reads from it and writes back into it. Think of a running notebook page for each position in the sentence: the model never replaces the page, it only adds annotations. This is the residual stream.
In vectors: The residual stream at position i—starting out with the vector embedding of army at position 13—is a single 4,096-number vector. A layer reads it, computes something, and adds the result back:
x ← x + (what this layer worked out)
That addition — rather than overwriting — is why the name is “residual”. It means information written by layer 3 is still legible to layer 27, and it means the 4,096 dimensions act as a shared bus that different components use for different purposes. By the time you get to layer 32 of position i the vector may be the vector encoding not of army but of “singular military subject of a clause that is owed an explanatory verb”.
Thus the initial frozen vector embedding table is a small part of the story. The permanent per-token knowledge is one row of 4,096 numbers. The contextual machinery on top is 32 layers × (attention + a 4,096→14,336→4,096 MLP) — billions of parameters whose entire job is to stop the vector in position i from meaning what the lookup table said it meant.
II. Details
The Label & the Contents Do Not Match
Because W_K and W_V are separate matrices, a word can be labelled one way and contain something else. The key for army can point in a direction meaning “main noun of the subject” — grammatical — while its value points somewhere meaning “a fighting force: marches, besieges, wins and loses battles” — semantic.
You find the drawer using grammar. What you take out is meaning. This split is the whole trick of “attention”.
RoPE, & Why Position Lives in the Label
The model needs to know word order, and it encodes that in the matching rather than in the contents. The mechanism is RoPE — rotary positional embedding.
In vectors: Take the 128 numbers of q and k as 64 pairs, and treat each pair as a point on a plane. RoPE rotates pair j by an angle position × θⱼ, where the θs are a fixed geometric series of frequencies. Crucially it is applied to q and konly — never to v.
The consequence is elegant. When you later compare a rotated query at position 18 with a rotated key at position 13, the rotations partly cancel and what survives depends on the gap, 18 − 13 = 5. Distance is felt in how well labels match. The papers inside the drawer are unaffected by where the drawer sits.
The Causal Mask
A word may only consult words before it. When the model was working on token 5, tokens 6 onward did not exist as far as it was concerned. This is what forces genuine prediction rather than peeking.
In vectors: Before the scores are converted into shares, every score for a future position has −∞ added to it. Exponentiating −∞ gives zero, so those positions receive exactly zero attention. It is implemented as a triangular matrix of zeros and −∞.
III. The Question Asked from Tudor
Scaled Dot-Product Scores
The query from token 18 is compared against all eighteen existing keys: “Compared” means the dot product: are the vectors pointing in exactly the same direction in the 128-dimensional Hilbert vector space (in which case the dot-product is their lengths multiplied), in opposite directions (in which case the dot-product is their lengths multiplied), cross-ways orthogonal (in which case the dot-product is zero), or something in between? Where those dot-product multiplications produce big numbers drives the answer-to-be as calculations become pseudo-thought. multiply the two 128-number vectors element by element and add up the results. A large dot product means the two vectors point in a similar direction — the label answers the question.
In vectors: The raw score is q·kⱼ, then divided by √d_head = √128 ≈ 11.31. That division is the “scaled” in scaled dot-product attention: adding up 128 products makes the raw numbers grow with head width, and without the scaling the next step would saturate and the gradients would vanish during training.
SoftMax
Core Step: The scores are turned into shares that add to 100%, with a deliberate twist: a slightly higher score wins a much larger share. The function that does this is softmax.
In vectors: αⱼ = exp(sⱼ) / Σ exp(s). Because everything is exponentiated before being normalised, differences get stretched. A score of 4.0 against a score of 2.0 is not twice as good — e⁴ = 54.6 against e² = 7.4 is about seven times as good. The model commits hard to its best match rather than spreading itself thinly:
Sum of the exp column is 99.1; each weight is that row’s exp divided by 99.1.
What Comes Back, & Where It Goes
The values are blended in those proportions and written into the residual stream. 55% of the papers from army, 7% from the, and so on.
In vectors: The head’s output is Σⱼ αⱼ·vⱼ — a weighted average of eighteen 128-number vectors, giving one 128-number vector. A fourth matrix W_O, of shape 128 × 4,096, projects it back up to full width, and the result is added to the residual stream at position 18.
Written out whole, the operation every head performs is: softmax(qKᵀ/√d_head)·V.
So although the last token read was a surname, the model is now largely thinking about an army — a thing that does something to somebody.
Why Reaching Back to Army Is the Clever Bit
The nearest noun is the wrong noun. In “the army of his cousin Henry Tudor”, the thing the sentence is about is the army, with two other nouns sitting between it and the end. Anything that simply favoured proximity would land on Tudor and write about a person.
Attention is not distance-weighted. RoPE lets a head use distance, but the head decides how much it cares. This one has learned the N of M construction well enough to skip the modifier — and that single retrieval settles three things at once: the verb must be singular, it must be military, and it must take an object.
Thus the last word in the sentence asks a question that has almost nothing to do with the last word in the sentence.
IV. Many Heads, Many Layers
Thirty-Two Questions at Once
Everything above is one attention head — one question. A layer runs 32 of them side by side, each with its own W_Q, W_K, W_V, each trained to ask about something different. Their outputs are concatenated (32 × 128 = 4,096) and added to the residual stream together. They are not competing for one answer; each contributes a different fact:
Stack 32 such layers and the model performs 1,024 of these read-and-write operations for every single word it produces.
The Word Last Is Quietly Doing a Lot
“The last Yorkist King” is a claim that something ended. Put that beside a rival’s army standing in subject position, and the sentence is visibly set up to explain how.
This is why defeated beats landed or met. Henry Tudor did land at Milford Haven and did meet Richard — both true — but neither discharges the expectation clause one created.
From Vectors to an Actual Word
Feed-Forward Layers: The Very Large Lookup Table
Attention can only move around what is already in the sentence. Nowhere in those eighteen tokens does Bosworth appear, so no drawer can hand it over. Facts come from elsewhere: the feed-forward layers, also called MLP blocks, which sit after the attention block in every layer.
In vectors: Two matrices in sequence with a nonlinearity between them: 4,096 → 14,336 → 4,096. The first matrix has 14,336 rows; the residual stream is dotted against every one of them, and rows it matches strongly “fire”. The second matrix has 14,336 corresponding rows, and the fired ones get added into the output.
That is a lookup table in the most literal sense — and note that it is the same key–value pattern as attention, with one difference. Attention keys and values come from other tokens in this sentence. Feed-forward keys and values are fixed weights, learned once from the whole training corpus. Attention retrieves from context; feed-forward retrieves from memory.
So: attention works out what kind of thing is needed here. The feed-forward layers supply the specific fact.
Unembedding & Logits
In vectors: The final 4,096-number residual stream is multiplied by the unembedding matrix (4,096 × 128,000), producing one raw score — a logit — for every word in the vocabulary. Softmax again turns those into probabilities, and one is sampled:
Why the Runners-Up Lose
met, confronted — fit the subject correctly, but leave “last” unresolved.
landed, marched — plausible for a different sentence about 1485; the semicolon’s explanatory force pushes against them.
would, had — where probability drains when the grammar is confident but the fact is not.
Nothing plural appears in the top twenty. That is the retrieval of army rather than Tudor doing its work.
Then It Does the Whole Thing Again
The KV Cache
Having chosen defeated, the model appends it and starts over — but it does not recalculate anything it has already worked out. The store of previous work is the KV cache, and its name tells you exactly what is in it.
In vectors: For every layer and every head, the cache holds one 128-number key and one 128-number value per token seen so far. After this prompt that is 18 keys and 18 values × 32 heads × 32 layers. Each new word appends exactly one key and one value per head per layer.
There is no query cache. The query is computed fresh at the current position, used once to score against the cached keys, and discarded. Keys and values persist because they will be consulted again by every future word. A query is asked once and never re-asked.
The Rest of the Sentence
Token 20 rewards a second look. To produce him, the model must reject Henry — present, male, and only three tokens back — in favour of Richard, seventeen tokens back across a semicolon. It manages this because him cannot refer to anyone inside the phrase doing the defeating, and heads exist that have learned exactly that constraint.
The Whole Vocabulary in One Place
Name, Idea, Implementation
Honest Caveats
Illustrative
The percentages are invented, though invented consistently. The architecture and the shape of every calculation are exact; the specific figures are for teaching. Real heads are messier too — few have one tidy job, and the work of locating army is normally shared across several heads in several layers rather than done by one.
The dimensions are those of a mid-size open model, roughly Llama-3-8B. One simplification: most current models use grouped-query attention, where several heads share a single set of keys and values, which shrinks the KV cache substantially. Assuming one key–value pair per head, as above, is the older and clearer arrangement.
Is this what is actually going on? Maybe?!?!?! This explanation is beyond me. It is not much beyond me. But it is enough beyond me that I am going to have to think about it quite a while. (In my copious spare free time.)
It does manifestly work. This roiling boil of linear (and non-linear) algebra It does produce grammatical human speech and human speech that appears to contain ideas, so it is a successful answer to the question: what are words like those the human might have said if one were having this conversation?
Is this how it works? Clearly not exactly: the machine admits that while the “architecture and the shape… are exact”, it is the case that the actual numbers are made-up ones: “the specific figures are for teaching”. That is: There was no set of matrix multiplications that had produced [0, -1.16, -1.39, -1.52, -1.67, -1.86…] and associated those with [defeated, met, landed, marched, would, had…] to produce a 55% probability that the model’s random-number generator would choose to continue the sentence with defeated.
What, Then, Would Count as Understanding?
There are three quite different questions tangled together. Much of my own confusion, I think, comes from letting the machine slide noiselessly between them.
The first is architectural: What sorts of operations does a transformer model perform in generating pseudo-thoughtful next-token predictions? Here the answer about tokens, residual streams, queries, keys, values, attention, feed-forward layers, logits, and sampling is an answer from the literature, presented in a way that is right now at just a bit above the right level for me personally where I now am. It is not the machine looking inward. It is the machine producing a competent synthetic textbook account of what computer scientists have written about transformers.
The second is causal-historical: Which particular internal computations caused this particular model, on this particular run, to continue Henry Tudor with defeated? That is a much harder question. And it is not one that the machine answers. Rather, the machine makes up a fictional answer, pantomiming what some serious mechanistic-interpretability researchers might do in some perhaps analogous situation..
The third is pedagogical: What story will allow Brad DeLong to build a usable mental model of what a transformer might be doing? The machine answered the third question by constructing a stylized instance of the first masquerading as an answer to the second.
That is “hallucination”. But it is interesting hallucination.
Economists do this all the time. There never was an actual pin factory whose production process was exhausted by Adam Smith’s twenty operations. There is no economy in which one representative household literally solves the social planner’s intertemporal optimization problem. The IS-LM diagram does not claim that somewhere inside the economy two curves are physically intersecting. The Solow model does not report the measured thought processes of capital accumulation. These are deliberately false little machines for thinking with. We forgive their falseness because it is advertised, disciplined, and useful. The fiction is not a bug. The fiction is the instrument.
The transformer explainer’s invented attention shares and logits might therefore be defended thus: “Suppose, for purposes of illustration, that this head gives army a weight of 0.55.” That could be extremely useful. But a story about what a representative attention head might do is not a trace of what this model did do.
The failure here may thus be the silent boundary-crossings. And I, because the prose was smooth and because the filing cabinet analogies are helpful do not mind much.
An answer engine suppresses learning. A tutor promotes it. The distinction is between cognitive substitution and cognitive activation. If I ask the machine, “Tell me how attention produced defeated,” and then admire the answer, it substitutes its fluency for my ignorance. If it asks me first what I think a query does, forces me to predict which earlier token matters, gives me a diagram only after I commit myself, makes me distinguish contextual retrieval from stored knowledge, and finally asks me to explain the mechanism without the filing-cabinet analogy, it may activate the processes by which I learn.
This suggests a “Bosworth Protocol” for machine-assisted understanding.
First, write down what I currently think. Not because my thoughts are likely to be right, but because without a prior I cannot tell whether the machine has changed my model or merely covered it with attractive prose.
Second, ask for the explanation at several levels—but also ask what each level omits. “Explain it to a high-school student” usually means “suppress complications.” That is legitimate only if I know which complications were suppressed.
Third, require every substantive claim to be tagged: architectural or definitional; directly measured in a named model; inferred by interpretability researchers; pedagogical analogy; invented numerical illustration; wild but perhaps comforting speculation. Had I imposed that requirement here, the 55% would have appeared wearing the proper bright-orange warning vest.
Fourth, demand alternative accounts. Perhaps defeated arises from a tidy subject-retrieval head. Perhaps the relevant information is distributed across many layers. Perhaps the completion is strongly driven by memorized textual associations surrounding “Richard III,” “Henry Tudor,” and “Bosworth.” Perhaps several partially redundant circuits converge. An explanation that cannot name its rivals is an advertisement.
Fifth, ask for interventions that would distinguish the accounts. Remove “the last Yorkist King.” Replace army with forces. Change Henry Tudor to an unknown name. Substitute a fictional battle. If the explanation predicts nothing about how these changes should affect the continuation, it may be a mnemonic rather than a model.
Sixth, close the machine and perform a transfer task. Explain queries, keys, and values without the filing cabinets. Explain why attention alone cannot retrieve “Bosworth” if the concept is nowhere represented in the context. Explain what the KV cache stores and why it does not store queries. Identify what was false or oversimplified in the original account.
Seventh, it is that last that is the only success condition that matters. Not: “Did the answer seem illuminating?” Not: “Could I follow every sentence?” Not even: “Was the answer correct?” What can I now do and understand that I could not and did not do and understand before?That would be learning.
Perhaps my provisional conclusion should be this: Yes, these things can help us understand what we do not understand. But this is not because they are oracles, and not because they can necessarily explain themselves. Rather, they can function as indefinitely patient adaptive textbooks; as generators of analogies; as adversaries for half-formed ideas; as cartographers of a literature; as engines for manufacturing examples and counterexamples; as Socratic partners that can keep adjusting until they locate the exact edge of one’s comprehension.
All the while, howeverm we must remember the machine’s default objective. It is to produce a satisfactory continuation. It is not to preserve my human capital. It is not to make sure that I can solve the next problem after the machine disappears. It is not to maintain distinctions among rival schools of thought. It is not even to tell me which sentences are measurements and which are parables unless I insist. The user must supply—or someone must build—the pedagogy.
Perhaps, therefore, the grand use case beyond boilerplate, code, search, summarization, protein folding, and the rest is neither “answering questions” nor “explaining difficult things.” Perhaps it is finding and working the frontier between what I can understand and what I cannot yet understand. That in itself perhaps promises to be genuinely new. A book cannot rewrite itself because chapter six lost me. A professor cannot give every student twenty explanations and then construct a twenty-first from the precise pattern of their confusions. A colleague cannot remain indefinitely cheerful while I ask for the difference between a value vector and the residual stream for the seventh time.
The machine can.
But I should not ask at the end: “Did it give me a good answer?” I should, instead, ask: “Can I now teach this?” “Can I use it somewhere else?” “Can I identify where the analogy breaks?” “Can I tell which numbers were measured?” “Can I formulate a question the machine’s explanation does not answer?”
If yes, then the roiling boil of linear and nonlinear algebra has done something more than predict words like those a human might have said.
It has helped to change the human who asked.
Shifting gears: There may also be some clarification to make of the SubStack x Pangram failure here. “AI-written” is not, in general, an observable natural kind like “printed in Leipzig in 1848.” Pangram’s judgment is not necessarily a judgment about who wrote the text. It is a judgment that this text does or does not resemble the particular varieties of machine output in its training and test regimes. Perhaps what it is detecting is the default one-shot use case: “Write me 1,200 words about X,” followed by Ctrl-C and Ctrl-V. The heavily iterated human-machine centaur—prompted, challenged, reordered, fact-checked, ornamented, and partially rewritten—may lie outside the detector’s conception of the species.
Sanderson, Grant. 2024. "Attention in Transformers, Step-by-Step”. Deep Learning, chapter 6. Text adaptation by Justin Sun. 3Blue1Brown. Apr 7. <https://www.3blue1brown.com/lessons/attention>.
Rush, Sasha, & al. 2022. “The Annotated Transformer
Shalizi, Cosma. 2023-2026. “‘Attention’, ‘Transformers’. in Neural Network ‘Large Language Models’”. Bactra Notebooks. Jul 30. <https://bactra.org/notebooks/nn-attention-and-transformers.html>. The claim: “attention” is Nadaraya-Watson kernel smoothing, reinvented and rebranded. He is scathing about the naming — “wishful mnemonics,” in Drew McDermott’s phrase — while being explicit that the rebranding takes nothing away from the engineering: people pushed kernel methods hard for years and got nothing like this…
Alessandrini, Giulio, Brad Klee, & Stephen Wolfram. 2023. “What Is ChatGPT Doing… & Why Does It Work?” Stephen Wolfram Writings. Feb 14. <https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/>. Long, discursive, weakest exactly where you’d expect as it slides from mechanism into Wolfram’s own framework about computational irreducibility and a hypothesized “semantic grammar”; worth reading for the first two-thirds and then applying salt…
Loose Ends
“Useful to me” needs operationalization: My five criteria describe a target zone but supply no success condition. The finding “I now have to think about this for a while” as is thoroughly ambiguous as anything could ever possibly be.
The “fabrication” problem not pursued: The numbers are fake as it admits; where is the architecture as described fake as well?
An implicit assumption: There is no reason a transformer’s account of transformers derives from privileged access; it derives from the same corpus a human explainer would read. The output is a summary of the literature, not introspection. That would be a better frame than “the machine explained itself,” and would change the conclusion.
The finding of “compression”: Studies appear to show that these tools help novices substantially and experts little or negatively. That is directly relevant to a post about whether and how they can teach.
People: J. Bradford DeLong; Cosma Rohilla Shalizi; Stephen Wolfram; Andrej Karpathy; Grant Sanderson; Brendan Bycroft; Jack Lindsey; Ashish Vaswani; Sasha Rush; Drew McDermott; Richard III; Henry Tudor.
Schools of Thought: mechanistic interpretability; kernel smoothing / Nadaraya-Watson estimation; “wishful mnemonics” critique of AI naming; Clever Hans skepticism.
If reading this gets you Value Above Replacement, then become a free subscriber to this newsletter. And forward it! And if your VAR from this newsletter is in the three digits or more each year, please become a paid subscriber! I am trying to make you readers—and myself—smarter. Please tell me if I succeed, or how I fail…
I have misgivings about the summarization function.
I see the value of being able to quickly grasp something about a scholarly paper without working laboriously through it. But I also see two concrete problems.
The first is about quality. I produced a PDF of a paper I'm working on, and Acrobat of course said, "This looks like a long paper. Can I summarize it for you?" This "long" paper was 16 pages of text, plus the source list, with no equations, never mind particularly challenging math, but I was curious what Adobe's LLM would do with my work.
The summary was competent _except_ where I was using language in creative ways. For instance, I described LLMs as a double-edged sword, but not in the conventional meaning of a thing that can be both good and bad, but as a thing that simultaneously makes it more important to know things and harder to acquire knowledge. The summary forced the metaphor back to the conventional "good and bad effects" usage of the image. Which makes all sorts of sense: my use of the phrase was specifically unlikely, so of course a probability engine would be unlikely to reproduce it.
The second is more fundamental. If the summary is enough, why are we writing papers and books in the first place? Why not just produce the summary? Save ourselves and eveyone else a lot of time, as well as the paper or electrons in printing or sharing a 30-page paper, or a 500-page book.
The meaning of a text is always generated by the interaction between the words that the author put on the page and the knowledge and perspectives that each reader brings with them as they read. So while there are core elements that should be in your summary and in mine if we are both competent readers of the text, there will be different emphases and framings, based on our different backgrounds.
And some of those things that will spark differently in your brain and mine are in parts of the text that don't make it into the summary. The more that we rely on machine-made summaries, the more we strip out the different meanings that would emerge if you and I actually read the original work.
And of course, scholarly papers already come with summaries produced by the author(s): the abstract. Once you read that, you know what question was asked, what answer was found, and hopefully something about how the question was addressed. Then a reader skilled in the discipline can flip through to the parts that are most important to them, or strap in and read through the whole thing.
What is the actual benefit of a summarization engine that irons out the most unique parts of text and gravitates toward a least-common-denominator understanding?
I think I have noted that a scientific mentor to a successful scientist friend considered his greatest pedagogical accomplishment passing on that the critical factor in pushing the ball of human knowledge up the hill is asking the right questions.
On that front I need to revisit Claude code. The coding question answering ability of Google search is pretty impressive. In the last hour I asked "R side effects on matrices in lexical function called by apply" which when you regularly switch between languages is a question that happen and the AI summary (with links) not only said why what I was doing wasn't working (something I've done with Fortran common blocks and pointers or references in C, Lisp, Perl, ...) but explained how lexical coercion altered what the interpreter was using in it's symbol table during the vapply() call. However, precise syntax returned by the AI is often wrong, but decades of Unix (now Linux and Mac) use has my neural circuits enhanced by the Unix man page philosophy of "if you need an example perhaps you are below our target user" (of course it makes sense that the space is critical "{} \;" in "find . iname 'r*log' -mtime -5 -exec ./handle_logs.pl {} \;" but decades ago it wasn't the first thing I thought of). (Using Google search for regex patterns for Emacs compiled font-lock code is sub-optimal).
I'm retired but an example I think of from LLM coding AI is from many things I've done and published. The tasks are reasonably well defined: Take the experimental data files found in various arbitrary machine dependent data formats and consolidate into an HDF5 archive. Take model described in LaTeX file xxx.tex and write R code to extract and massage data sets then run model in lsode called from R with Fortran coded RHS and Jacobian in a multidimensional state vector walking optimization framework. Then create publication quality figures from the analysis. This isn't asking new questions, this is clearly defined grunt work. But if LLM AI can't do simple grunt work why would it result in thinking it is on the way to some magical AGI?
I have misgivings about the summarization function.
I see the value of being able to quickly grasp something about a scholarly paper without working laboriously through it. But I also see two concrete problems.
The first is about quality. I produced a PDF of a paper I'm working on, and Acrobat of course said, "This looks like a long paper. Can I summarize it for you?" This "long" paper was 16 pages of text, plus the source list, with no equations, never mind particularly challenging math, but I was curious what Adobe's LLM would do with my work.
The summary was competent _except_ where I was using language in creative ways. For instance, I described LLMs as a double-edged sword, but not in the conventional meaning of a thing that can be both good and bad, but as a thing that simultaneously makes it more important to know things and harder to acquire knowledge. The summary forced the metaphor back to the conventional "good and bad effects" usage of the image. Which makes all sorts of sense: my use of the phrase was specifically unlikely, so of course a probability engine would be unlikely to reproduce it.
The second is more fundamental. If the summary is enough, why are we writing papers and books in the first place? Why not just produce the summary? Save ourselves and eveyone else a lot of time, as well as the paper or electrons in printing or sharing a 30-page paper, or a 500-page book.
The meaning of a text is always generated by the interaction between the words that the author put on the page and the knowledge and perspectives that each reader brings with them as they read. So while there are core elements that should be in your summary and in mine if we are both competent readers of the text, there will be different emphases and framings, based on our different backgrounds.
And some of those things that will spark differently in your brain and mine are in parts of the text that don't make it into the summary. The more that we rely on machine-made summaries, the more we strip out the different meanings that would emerge if you and I actually read the original work.
And of course, scholarly papers already come with summaries produced by the author(s): the abstract. Once you read that, you know what question was asked, what answer was found, and hopefully something about how the question was addressed. Then a reader skilled in the discipline can flip through to the parts that are most important to them, or strap in and read through the whole thing.
What is the actual benefit of a summarization engine that irons out the most unique parts of text and gravitates toward a least-common-denominator understanding?
I think I have noted that a scientific mentor to a successful scientist friend considered his greatest pedagogical accomplishment passing on that the critical factor in pushing the ball of human knowledge up the hill is asking the right questions.
On that front I need to revisit Claude code. The coding question answering ability of Google search is pretty impressive. In the last hour I asked "R side effects on matrices in lexical function called by apply" which when you regularly switch between languages is a question that happen and the AI summary (with links) not only said why what I was doing wasn't working (something I've done with Fortran common blocks and pointers or references in C, Lisp, Perl, ...) but explained how lexical coercion altered what the interpreter was using in it's symbol table during the vapply() call. However, precise syntax returned by the AI is often wrong, but decades of Unix (now Linux and Mac) use has my neural circuits enhanced by the Unix man page philosophy of "if you need an example perhaps you are below our target user" (of course it makes sense that the space is critical "{} \;" in "find . iname 'r*log' -mtime -5 -exec ./handle_logs.pl {} \;" but decades ago it wasn't the first thing I thought of). (Using Google search for regex patterns for Emacs compiled font-lock code is sub-optimal).
I'm retired but an example I think of from LLM coding AI is from many things I've done and published. The tasks are reasonably well defined: Take the experimental data files found in various arbitrary machine dependent data formats and consolidate into an HDF5 archive. Take model described in LaTeX file xxx.tex and write R code to extract and massage data sets then run model in lsode called from R with Fortran coded RHS and Jacobian in a multidimensional state vector walking optimization framework. Then create publication quality figures from the analysis. This isn't asking new questions, this is clearly defined grunt work. But if LLM AI can't do simple grunt work why would it result in thinking it is on the way to some magical AGI?