On-Device Local-Model "LLM Coherence": TABLE OF THE DAY
Sunday MAMLMs: when the lobster moves In: gemma4 is teaching me about “ai coherence”...
If human language is mostly formulaic stochastic parrotage, you’d expect a good parrot to be cheap wallpaper, not a working research assistant. Yet beneath my dining‑room sidetable a 26‑billion‑parameter open-source model from Google is becoming a disturbingly capable simulacrum of an external brain…
OK. For me at least, modern “AI” is now useful: SubTuringBradBot works—I can get good first-line answers in response to questions about courses or things I have written <https://web.telegram.org/k/#@SubTuringBradBot <https://braddelong.substack.com/p/subturingbradbot-is-finally-liveat>; Exegeticist-Bot works—I can figure out what I, or at least past I, thought about an issue when I have forgotten, because vector-embedding semantic search is a vast improvement over keywords.
Thus I am now questioning what has been my standard line about “AI”, which is, you will recall, this: appearing to work by creating a convincing pantomime simulacrum of a Turing-class entity because much more of human language than we like to think is formulaic stochastic parrotage <https://braddelong.substack.com/p/stochastic-lobsters-token-tsunamis>. But I really do now need an explanation of what is going on, of what is, as Eugene Wigner might say, the unreasonable effectiveness of these systems for so many. The best road into this is to give it a serious chance. But I want to do so without paying a semi-fortune to Anthropic or OpenAI. What is the best road forward>
Well, the championship LLM in terms of “thought quality” of those I have downloaded and now have the capability of running on the on the Maxxxed-Out M5Max MacBookPro burbling under the dining room sidetable.turns out to be
not the 74GB Llama 3.3:70b (q8) from FaceBook,
or the 42GB Deepseek-R1:70b from High-Flyer,
or even the 35GB Qwen 3:32 (q8) from AliBaba,
but rather the 18GB Gemma 4:31b from Google:
With the 17GB Gemma 4:26b as a close second.
Gemma 4 is Google’s bid to own the “serious open‑weights” tier of the stack while still keeping Gemini for the premium, closed models. It is both a technical artifact and a piece of platform strategy. The core line landed March 31, 2026; multi‑token‑prediction (MTP) variants followed April 16; and the unified 12B multimodal model shipped June 3 <https://ai.google.dev/gemma/docs/releases>.
The 31B dense model is meant to be the general workhorse that still fits on a single high‑end GPU (or maxed‑out Apple Silicon) with 256K context.
The 26B A4B MoE (Mixture‑of‑Experts) model that only activates ~4B parameters per token but keeps 26B in memory for routing; it’s tuned for high‑throughput reasoning and agents.
Google’s headline is: “byte‑for‑byte, the most capable open models to date,” purpose‑built for advanced reasoning and agent workflows <https://medium.com/google-cloud/getting-started-with-gemma-4-learning-resources-833ed68fcf2c>.
And they seem, by my experience, to be right.
So now I have a question for any of you similarly trying to figure out this stuff, but wanting to do so on your own devices rather than compete with money for space in hyperscaler clouds: are you seeing anything similar?
More thoughts below the fold here:
