From clay tablets to GPT LLMs: meditating on minds & machines while knowing that "AI" is not your friend but rather your interface to a very weird library. Mental models, moral panics, why prompting is a form of wizardry, and considering MAMLMs as strange mirrors shaped by the patterns of online discourse…
I am going to try to reserve Mondays to think out loud about Modern Advanced Machine Learning Models—MAMLMs. And I am going to try to shut up the rest of the time.
And now it is Monday. So let me start with three observations I think are relevant to MAMLM’s:
The first is an example of the genus of moral panic that happens when advances in information technology shift our distributed brain processing so that more of the current state is stored in front of rather than behind our eyes:
Jim O'Shaughnessy: <https://newsletter.osv.llc/p/on-ai-writing-thought-design-and>: When I was growing up, calculators were new.... And then for a while, they banned calculators from class. Because that was cheating... [for] four years... until people realized... that moral panic was completely misguided.... I think we're... seeing the same thing with schools... banning AI.... It drives me crazy because it's like... "we just discovered how to consistently start a fire."... :You could get burned by that fire, and therefore, you may not do it."... Hey, right after we invented fire, guess what happened? We invented fire alarms, fire departments, firemen, you know, fire exits, all because fire is that powerful. It is incredibly great for us. It's responsible for our prefrontal cortex. Right. Some speculate that it was only when we started cooking our food that the prefrontal cortex emerged…
The calculator is simply one of the most recent of these. After all, in the real old days, you could not be a real scribe unless you spent at least an hour a day getting your clay into the right consistency and then smoothing it, during which time you meditated on what you were going to write that day.
The second is about how the “ChatBot” nature of our interfaces to MAMLMs is misdirecting us by casting a false illusion about what they really are:
Nathan Baschez: <https://newsletter.osv.llc/p/on-ai-writing-thought-design-and>: ‘Because [AI] takes the form... of a chatbot, it feels... like... a person.... [But] AI[s]... really contain multitudes.... A lot depends on the exact way that you tend to prompt it.... Com[ing] from... an engineering background... I'm used to... tinkering with systems until they give me the results that I want. And—rightfully so—that's not the way that we think about collaborating with other people.... If you come to AI with this mindset of... ["]I'm going to cross my arms and see how good of a job you do... from a distance["]... that's just the wrong way to approach it.... To have a tool that... within... two seconds... give[s]... you more choices that you wouldn't have thought of otherwise.... [is an] incredibly valuable thing.... And... it's about learning little tricks... to prompt it...
The way I think about it is this: You are not talking to a person. You are utilizing a natural-language interface to a summarization engine. What does it summarize? The internet, more or less: it tells you what the typical internet s***poster responds when presented with a call like the one you issue.
Why is this useful?
Because if you can poke it properly by your emitting the right calls you can guide it into a corner of the internet where this call-and-response elicits true and accurate information. And because its responses are much easier to parse for interesting tidbits than the ten blue links.
Why are its responses easier to parse?
I guess: (a) Because our minds are built/trained for call-and-response rather than archives. (b) Because its compression of the entire internet into a 3000-dimension vector space is a much finer and more effective classification scheme than <keywords> + <pagerank> or whatever Google and company are really doing these days. (c) This is especially true now that they have been polluted by so much SEO spam tuned to mimic high-quality results for a <keyword> + <pagerank> search.
How can it be so much finer and more effective a classification scheme?
Because of Deep Magic I do not understand. And I need to understand it if I am to have any chance of having even a semi-accurate Visualization of the Cosmic All going forward.
The third is Dan Davies here, talking not about MAMLMs, but rather about learning history:
Dan Davies: <https://backofmind.substack.com/p/the-valve-amplifier-of-history>: ‘“Recognising that an analogy is no good” is a relatively quick cognitive operation…. Problem solving is based on making mental models of the problem; abstracting some of the detail while hoping that you’re capturing enough of the causal structure so that if you solve the problem “in the model” the same solution will work in real life. Usually, disanalogies are quick to spot…. Creating a mental model from scratch is a very expensive cognitive operation, though. So, if you have a supply of previously existing mental models, it might be a very good strategy to just start going through them one by one, effectively running your thumb through the book going “nope, nope, nope, maybe … nope, nope, nope … nah, doesn’t work … maybe … nope, nope … hang on this might work”. Rather than taking on the expensive task of making a model that you’re certain will work because you’ve constructed that way, you’re making multiple cheap attempts. But where might you get a large supply of ready made mental models to go through in this way?…
He goes on:
Dan Davies: ‘Just filling your head up with stories about how things could happen means that you’ve got a catalogue to riffle through, while the constraint that they are stories about things that actually did happen once should exercise some kind of rudimentary quality control on the library of candidate solutions…
The alternatives that MAMLM GPT LLMs offer us in response to our prompts are, it is true, lacking relative to the analogies offered up to us by history. For history, after all—as long as it is true history rather than fantastic, fabulous, or romantic—is “useful to those inquirers who desire an exact knowledge of the past as an aid to the interpretation of the future, which in the course of human things must resemble if it does not reflect it…”
The alternatives that MAMLM GPT LLMs offer us are lacking in that it is unconstrained by thing like, you know, truth and reality.
But they have an edge. They can generate a huge catalogue of alternative possibilities very very quickly. And those are all clustered around the particular point you choose in the 3000-dimension vector space compressed classification of the internet that lies at their heart when they are considered as summarization engines. Thus, if properly prompted, they can get close enough to serve as a place for you to start, or at least to spark thought.
Remaining questions, at least as I see things:
Why are they so good—or are they not so good, but rather a combination of “Clever Hans” and “cold reading”?
What I call “Clever Hans” is when you put as much work into writing the prompt as you would into doing the thing.
What I call “cold reading” is when positive assessments of MAMLMs is based on their having been trained to deliver shotgunning plus Barnum statements, plus confirmation bias.
(Note that shotgunning and Barnum statements can be very useful even when wrong, if they carry other messages, like “this is close to the Python syntax for that”.)
How good can they get, considered as compression-and-summarization-and-alternative-generation engines?
When does the compression-and-summarization engine frame start to mislead, if it ever does?
What will the next thing be in MAMLMs, that does better than training a simulation of the typical internet s***poster?
If only lawyers would use Dan Davies' method to check that the genAI spitting out a brief to check the output. Maybe a few more expensive fines, perhaps even a disbarment, might change behaviors. In those sort of cases, I am not sure if a better way is to ask the AI to provide cases with references relevant to the case, with a summary of why each case law might be used. Then check everything, use the cases as narrow documents in a RAG, and ask the AI to create a brief only with those curated cases. Then read the brief carefully to make sure it is still correct.
GenAIs should start getting better at retrieving relevant references than the enshittified Google search that is now worse than it was. Surely Google understands that offering a free Gemini AI search summary is undermining their enshittified, ad-supported search results?
It may be somewhat recursive, but can genAI be asked to provide a good prompt for a desired output?
It seems to me that the best way to use genAI is like the way one uses a spreadsheet or code, use it to test out ideas quickly. A spreadsheet removes the slow, repetitive calculation task and quickly shows the result. GenAI should be able to quickly test out how to solve a task, which the user then curates/refines, to produce a good task output. Humans and AIs working together, with the AI acting like a fast go-fer assistant,
I don't know where AI goes next, other than to respond more effectively to prompts. I cannot help but feel that Asimov got there with controlling robots many decades ago when he wrote that the Spacers, with many robots to control, were far better at voice commands than the Earthers, who had little, if any, exposure to robots. We may just learn how best to interact with AIs to get the desired response. It would certainly be nice to use voice input to state whether accuracy or creativity was important in the response, perhaps using a human role as the initial prompt, just as we sort of use "You are a knowledgeable assistant on this subject [X]". I would prefer a catalog of domain specialist AIs, rather than trying to tailor the responses of a general AI. I also think that domain specialists may be able to sell their expertise as an off-the-shelf AI. A BradBot that can answer questions on economic history for a trifling amount might be very useful, with reviewers providing star ratings as feedback. I also would like to stage debates between domain expert AIs to determine which argument[s] are a better way forward, and to expose flawed logic. IOW, different AIs rather than a "mixture of experts" model.
Before I rely on an AI site for any task, I think it’s useful to try an experiment to see how it does on an obscure subject where you already know the answer to the query you’re going to post. An example: I spend a fair amount of time birding in Colombia. When I first tried Claude I posed it a query: What is the most critically endangered bird species in Colombia? I happened to know the answer to the question because I bid in Colombia with a scientist who is studying the species and working on its conservation. The answer is Antioquia Brushfinch, of which there are approximately 125 surviving adult individuals in the world. The identity of the brushfinch is not exactly a secret; there are lots of articles on its status. Yet Claude came back with an answer that just listed a set of widely known, somewhat endangered species. My follow up and their response went like this (paraphrasing : Hey, Bozo, what about Antioquia Brushfinch? Oh, sorry sir, we weren’t trying to answer the question you asked, just give some examples (all of which have populations over 125); after all who even really knows which species is most endangered. But, come to think about it, the brushfinch is pretty endangered and here are some more random examples. The moral of the story: Don’t trust AI unless they cite authority that you can confirm actually exists, is published in a reputable publication, actually says what they say it says and hasn’t been seriously called into question. In short, AI is only useful to the extent that it gets you quickly to some resource that does answer your question. (I’m assessing it only as a research to, not as a tool for writing code, for example.)