Discussion about this post

User's avatar
David Thomson's avatar

I think this is likely the case, but with some notes. There’s a difference between consumer-facing and the background stack.

First, it’s worth noting that Apple coming out well once the dust settles is largely an accident of historical position, not design. They failed to execute on their own AI strategy. They prefer doing things in house (until their hand is forced) and have been unable to execute on Siri or LLMs in a decade.

On the other hand, I suspect LLM “AI” and this round of hype will go the way of every other AI boom. We’ve had roughly seven machine learning “AI” hype cycles since Babbage and five since the 1960s, giving rise to the oldest joke in computing: if it works it’s computer science; if it doesn’t work it’s AI.

The data centres will largely be useful, maybe not at the scale needed to make untold trillions from digital gods, but everyone will forget about them. Like the fibre optic cables.

One reason you still need the data centres: once you have an LLM doing inference runs to build code and plan things, token processing explodes. It’s not like using a chatbot. Token consumption for coding and similar tasks is 50-100x or more. Then going forward as this gets baked into operating systems and UI, broadly for getting your computer to build more complex things, it’ll go exponentially(?) higher. Even if you stopped at the uses we have today, and it becomes part of the background of coding tools and ui you need a lot of computing power to run systems that operate through natural language - even if most of it is edge.

However, unitary models, one giant general-purpose model handling everything, I don’t think are the way this plays out for most use cases. What you actually want is your existing tools getting smarter within their own domain. Pages doesn’t need a model that can debate philosophy. It needs a constrained model specifically designed for document manipulation, one that understands structure, style, and reorganisation (maybe even constrained ui/ux manipulation). That model can be small, fast, and run on-device most of the time, with access to a more powerful cloud model when the task demands it (eg ui personalisation).

I expect it’ll shake down to an orchestration layer, a router that sits between you and the models which is mostly edge, picking what goes to the cloud. You make a request in natural language and the router decomposes it: some subtasks go to small expert models on-device, some go to domain-specific models in the cloud, and the big planning or research tasks, anything requiring long multi-step reasoning, go to the big frontier general-purpose models in the data centre. The results get stitched back together and you never see the plumbing. This is essentially how cloud computing already works. Microservices, each handling a specific function, with intelligent routing underneath. Nobody builds monolithic applications anymore, and I can’t see why AI deployment will be different.

Why would it not just follow previous patterns… I’ve not seen exceptional evidence for the exceptional claims. So… Edge computing on-device will be good enough for most people most of the time, just as an iPhone is good enough for most things. But as soon as you want anything more powerful that requires building over long autonomous runs, planning, building, testing, you need the big data centres again. Where exactly the lines fall between edge and cloud, between expert models and frontier models, who knows. But the structural principle, orchestration routing to a mix of specialist and generalist models, is just another computer engineering cycle. Custom mainframes to general-purpose servers to virtualised instances to containers to serverless functions… currently we’re pretty much at general purpose servers and edging into virtualisation.

Will O'Neil's avatar

The market for AI products seems sure to be very large, but not all-consuming in the way that enthusiasts assert. It seems to me that it is unlikely to be great enough to provide a good return for the frenzied investments in hyperscaling resources by Microsoft, Alphabet, et al.. It's one of those situations where the winners may wind up envying the losers, and thus Apple's "losers" strategy looks prudent to me. Even if the hyperscalers' strategy works out better than I fear, there are worse things than becoming a well-resourced fast follower. Obviously Apple must keep very well informed about the technology and the market and be ready to jump if it turns to be necessary.

3 more comments...

No posts

Ready for more?