On-Device Local-Model "LLM Coherence": TABLE OF THE DAY
Sunday MAMLMs: when the lobster moves In: gemma4 is teaching me about “ai coherence”...
If human language is mostly formulaic stochastic parrotage, you’d expect a good parrot to be cheap wallpaper, not a working research assistant. Yet beneath my dining‑room sidetable a 26‑billion‑parameter open-source model from Google is becoming a disturbingly capable simulacrum of an external brain…
OK. For me at least, modern “AI” is now useful: SubTuringBradBot works—I can get good first-line answers in response to questions about courses or things I have written <https://web.telegram.org/k/#@SubTuringBradBot <https://braddelong.substack.com/p/subturingbradbot-is-finally-liveat>; Exegeticist-Bot works—I can figure out what I, or at least past I, thought about an issue when I have forgotten, because vector-embedding semantic search is a vast improvement over keywords.
Thus I am now questioning what has been my standard line about “AI”, which is, you will recall, this: appearing to work by creating a convincing pantomime simulacrum of a Turing-class entity because much more of human language than we like to think is formulaic stochastic parrotage <https://braddelong.substack.com/p/stochastic-lobsters-token-tsunamis>. But I really do now need an explanation of what is going on, of what is, as Eugene Wigner might say, the unreasonable effectiveness of these systems for so many. The best road into this is to give it a serious chance. But I want to do so without paying a semi-fortune to Anthropic or OpenAI. What is the best road forward>
Well, the championship LLM in terms of “thought quality” of those I have downloaded and now have the capability of running on the on the Maxxxed-Out M5Max MacBookPro burbling under the dining room sidetable.turns out to be
not the 74GB Llama 3.3:70b (q8) from FaceBook,
or the 42GB Deepseek-R1:70b from High-Flyer,
or even the 35GB Qwen 3:32 (q8) from AliBaba,
but rather the 18GB Gemma 4:31b from Google:
With the 17GB Gemma 4:26b as a close second.
Gemma 4 is Google’s bid to own the “serious open‑weights” tier of the stack while still keeping Gemini for the premium, closed models. It is both a technical artifact and a piece of platform strategy. The core line landed March 31, 2026; multi‑token‑prediction (MTP) variants followed April 16; and the unified 12B multimodal model shipped June 3 <https://ai.google.dev/gemma/docs/releases>.
The 31B dense model is meant to be the general workhorse that still fits on a single high‑end GPU (or maxed‑out Apple Silicon) with 256K context.
The 26B A4B MoE (Mixture‑of‑Experts) model that only activates ~4B parameters per token but keeps 26B in memory for routing; it’s tuned for high‑throughput reasoning and agents.
Google’s headline is: “byte‑for‑byte, the most capable open models to date,” purpose‑built for advanced reasoning and agent workflows <https://medium.com/google-cloud/getting-started-with-gemma-4-learning-resources-833ed68fcf2c>.
And they seem, by my experience, to be right.
So now I have a question for any of you similarly trying to figure out this stuff, but wanting to do so on your own devices rather than compete with money for space in hyperscaler clouds: are you seeing anything similar?
More thoughts below the fold here:
What does Google think it is doing with Gemma? Strategically, Gemma4 is doing at least three jobs for Google:
It is cementing an open‑weights flank against Meta (Llama), Alibaba (Qwen), Mistral, and now DeepSeek. This is Google saying: if you want open models for local, edge, or private cloud, you live in our orbit, not Meta’s. https://ai.google.dev/gemma/docs/core
It is a distribution engine for Gemini and Google Cloud. The learning resources pitch is very explicit:
build and experiment locally with Gemma 4,
then “graduate” to Vertex AI, Cloud Run, or GKE when you want scale, governance, and Gemini models,
and wire all of this through Google’s “agentic” tooling—Skills repositories, Interactions APIs, etc. <https://medium.com/google-cloud/getting-started-with-gemma-4-learning-resources-833ed68fcf2c>.
Gemma is the open, developer‑facing on‑ramp; Gemini is the closed, margin‑rich destination.
It is a hardware‑conscious bet on specialization. Google wants to be the house that sells you a coherent family of models tuned to each tier of the hardware stack, with one name and one toolchain.
The 31B dense and 26B MoE are meant to be the largest things you can realistically run on a single high‑end consumer/user box, with a clear migration path to a Gemini endpoint when and if I decide I am tired of ever-whirring fans underneath the dining room sidetable.
And, as far as I am concerned, for now, Google has won the game.
You ask about the other models I have downloaded and played with? You want a crude positioning?
Llama, Qwen, Mistral, Ornith, and DeepSeek are five overlapping but distinct “houses” in the current open-source LLM ecosystem: one Big Tech, two Chinese industrial platforms, one European startup, one ultra-nerd coding boutique, and one cost-optimized upstart that has frightened everyone else:
Llama: default generalist, “Android of LLMs,” huge ecosystem, created and distributed by FaceBook, but now out of favor at the court of Zuckerberg as he pursues “personal-ASI” rather than being satisfied with giving away free models to cripple other companies’ abilities to make money off of natural-language interfaces. Llama models are onventional decoder‑only transformers, aggressively trained and tuned hard for instruction following and software tool use <https://en.wikipedia.org/wiki/Llama_(language_model)>.
Qwen: hyper‑feature‑rich, China‑centered, strong in multilingual and agents, Llama‑inspired, from Alibaba. Multilingual coverage and prominent reasoning‑centric “Thinking” modes, whatever that actually means <https://en.wikipedia.org/wiki/Qwen https://qwen.readthedocs.io/>.
Mistral: compact, efficient European generalists + good code models, with a tidy commercial stack. French startup whose entire business model is “we’ll give you a serious open frontier model, and try to make our money off of a slick commercial API”. <https://docs.mistral.ai/models/overview https://mistral.ai/models/>.
DeepSeek: frontier MoE and reasoning, optimized for cost and performance; open‑source foil to OpenAI/Gemini. TThis cost‑minimization has terrified competitors by showing how much capability you can squeeze out of Mixture-of-Experts plus ruthless engineering and Reinforcement Learning. DeepSeek’s models are shockingly cheap to train and run at scale. <https://www.bentoml.com/blog/the-complete-guide-to-deepseek-models-from-v3-to-r1-and-beyond>.
Ornith: a specialized, coding‑centric line from DeepReinforce betting that you can push code agents much further with Reinforcement Learning and harness- and loop-engineering: Gemma 4 and Qwen 3.5 fine‑tuned heavily. <https://deep-reinforce.com/ornith_1_0.html>.
You ask: why these five categories of “LLM coherence”—logical coherence, structural integrity, information density, information depth, and compliance with directions?
If it cannot do any of those, it is going to be unable to reliably do anything at all. Following directions, diving deep to understand the situation, not blathering pointlessly, remembering that not IV but III is the next step after I and II, and not outputting nonsense are prior to calling tools and setting the right switches on and tasks for them.
You ask: what are the units of these scores? Only God—or, rather, LLM, because I do not think even God—knows. They are LLMs’ products. I asked them to turn verbal descriptions and judgments of models’ answers—descriptions and judgments that I expressed in assessing answers, and then that it expressed in assessing many more answers—into numerical scores.
At least now I understand why when, after finding something unsatisfactory and switching models to see if another one will do better, after a week or two I would look back and find that I had placed all five of my LLM agents on Gemma 4:26b.
You ask: why do I find myself gravitating to the runner-up Gemma 4:26b rather than the champion Gemma 4:31b model? Because, recall from my earlier horseraces, Gemma 4:26b is 2.5 times faster:
That is from:
DeLong, J. Bradford. 2026. “Local LLM Performance on a Maxxxed-Out M5Max MacBookPro: TABLE OF THE DAY”. DeLong’s Grasping Reality Weblog. June 29. <https://braddelong.substack.com/p/local-llm-performance-on-a-maxxxed>.
At least, that and this are how it looks to me now.
I could envision a hit squad coming after you if you keep this up.