Paul Kedrosky Presents Us with the ECI—the Epoch Capabilities Index for "AI" Frontier Models: CHART OF THE DAY
When every LLM can closely approximate the typical internet s***poster, the money flows not to the competitive model-builders but to those who can build systems that usefully digest your data to...
When every LLM can closely approximate the typical internet s***poster, the money flows not to the competitive model-builders but to those who can build systems that usefully digest your data to reproduce and improve your SOPs. Or so Satya Nadella claims…
If the appropriate metric is the ECI, and the top model’s edge over the pack really has shrunk from 26% to 6%, we are not in “winner-take-all” land any more. We are are, instead, in commodity-silicon-with-fine-tuning land. That’s where pricing power migrates to whoever can bottle organizational procdures and judgment, and make it portable across interchangeable generalist models.
Narrowing in, as models get better but also closer to the same level at any moment in time:
Paul Kedrosky has a gloss:
Paul Kedrosky: Why .400 Hitters Disappeared—& What It Means for AI <https://paulkedrosky.com/why-400-hitters-disappeared-and-what-it-means-for-ai/>: ‘The trend is linear, not exponential, a steady gain of roughly 16 capability units per period, with an R² of 0.73. The exponential fit is worse than the linear one. Progress that gets told as relentless acceleration is, in the data, a straight line (at best). The second thing… is what happens to the spread…. In the early years of Epoch's Index… a top model could score almost fifty points above the mean…. This is the equivalent of Ted Williams, a .400 hitter…. [But today] the best model's premium over the 90th-percentile model, which was 26% in the early era, is now a mere 6%….
When the variance is wide, being the frontier model meant something…. The distance between frontier and good-enough was wide enough that it looked like an early moat…. This is what maturing commodity markets look like…. Price will become the main differentiator… [with] huge implications…. DeepSeek's massive pricing advantage…. Margin pressure on frontier companies, and the implications for said companies' post-IPO performance…
Ben Thompson <https://stratechery.com/aggregation-theory/> talks about the aggregator flywheel: offer a better value proposition, watch demand flow to you, use that demand to learn about how to offer an even better value proposition to your customers, and watch more demand flow to you until the only reason that your competitors are still around is because you are using market power to jack up your margins. (Note: not your prices, but your margins.) But this requires that economies of current scale and economies of learning-by-doing—cumulative scale—are both now and remain strong. In that world, however, OpenAI took its capabilities lead as of the summer of 2023 and its enormous advantage with respect to the number of users running its models to not have its model recursively self-improve itself but rather to give its programmers the insights they could use to pull further and further ahead of their challengers.
It simply did not happen.
And this is why Satya Nadella of Microsoft is confident that market value will flow to companies that can help users curate their own useful data rather than companies that build frontier models:
Satya Nadella: A Frontier without an Ecosystem Is Not Stable <https://twitter.com/satyanadella/article/2066182223213293753/>: ‘We can create a real cognitive loop between people and digital systems…. How [are] organizations [to] continue to learn, build IP, differentiate, and thrive in a world where AI models can continuously absorb the expertise of humans and organizations and commoditize it[?]… Human capital comprises the knowledge, judgment, relationships, ingenuity, and pattern recognition of its people…. Token capital is the firm’s AI capability it builds and owns…. The real opportunity is not in picking the best model but… in building a learning loop… where human capital and token capital compound…. A company should be able to switch out a “generalist” model without losing the “company veteran” expertise built into their learning system. This is the key “test” of your control and sovereignty in the era ahead. Companies need to turn their workflows, domain knowledge, and accumulated judgment into AI systems that improve with each use…. This loop becomes the new IP of the firm…. The last thing any of us want is a world where every company across every sector is ceding value to a few models that eat everything they see…. Employees… [ought to] see their expertise amplified and their judgment become part of systems that make it replicable and scalable and the benefits accrue to the companies and communities around them…
Yes: he is talking his book. But it seems to me that this is a good book to be talking these days. Building bureacracies around knowledge systems of SOPs that are hard to implement and thus to replicate then becomes the defensible IP. Yes, there will be a recursive-improvement loop. But it will be the firm‑specific learning loop, with the underlying model just a swappable, ever‑cheaper input.
I saw this https://www.mercor.com linked from Dwarkesh Patels recent post https://youtu.be/4pG3SJQPAwk?is=Kz8GmSOuYXEi73EF
Definitely some Fifth Generation deja vu vibes. I’m sure these systems will be better but it does look like MS positioning solidly for a spreadsheet style revolution not a jobspocalypse.
With frontier models months not years ahead of open source weights it has to become more effective to distill specialists local models into software rather than outsourcing everything to the big cloud models. Presumably there is still a demand for frontier models but maybe 80%+of what you need can be done on a smaller/distilled open source model. I don’t know, but that feels sensible to me… pure speculation though.
The huge thing I feel these systems solve (and it’s still very powerful and not trivial) is natural language interface with the computer. You don’t need to learn the computer language interface because it has learned how to interpret English.
The argument for swappable models is directly contradicted by the history of Microsoft itself, and of much of current computer infrastructure. But it also argues against high value for cutting edge technology. I was using BSD Unix before MS DOS existed but Mr Market chose the Microsoft stack of often/usually far from leading edge software. One could make a related argument using Google. Their search is good but is it really 26% better than Bing or others? That should be a pure commodity business, bandwidth/servers/storage. In the Greenspan market view then search should have already become a low margin competitive arena, yet margins appear not to have collapsed. I argued this earlier, that there is a business trajectory that supports the XAi IPO (and possibly others). For Musk it is to be the technology backbone of US surveillance and military operations which will also bring in much of the corporate world. If other companies can be threatened to operate by being declared a national security threat like Anthropic then the defensible business decision will be to base your business on the Musk software/ datafarm platform. I can see businesses models for the Musk approach (with government capture) and the Anthropic approach (building focused cutting edge tools that become standard across industries). I don’t see a long term future for OpenAI, but it could be an acquisition target. But it will be interesting.