From two Epoch-Capability points a month back in 2023 to one every six months today, with the spread of assessed model capabilities across frontier labs shrinking by half...
"Yes, writing code is a domain where surface prediction and correct world-modeling are very very close—programs either work or don’t, for there is no underlying reality to which the incantations in their correct form point."
I am a life-long professional software developer. I am under continuous, albeit gentle (no leaderboards) pressure to make use of LLM coding assistance, both chat-based and agentic. I have the benefit of a wide choice of models and professional training from the vendors of these models. I am trying to reserve judgment on the usefulness of LLMs for programming, but the only reason is the proliferation of Silicon Valley programmers telling me how good it is. To be clear, I am not just talking about the "hamburger" model of Kapoor and Narayanan (https://www.normaltech.ai/p/why-ai-hasnt-replaced-software-engineers) - although they are right. In my observation, LLMs are only moderately useful at shrinking the "execute" patty, never mind the bun.
You have (inadvertently) identified what may be the problem: there absolutely is an underlying reference reality for most of the software that makes a difference in your life, and if you fuck that up, then you, too, will be fucked. We are not close to having an LLM that you want to have writing your bank's book-keeping systems, or your hospital's life support systems, or your railroads scheduling system, or basically any other software system of any real importance. I can rely on LLMs to manipulate data ("this .csv file represents a volatility surface in the format blah blah blah, this other file represents a volatility surface in format lorum ipsum, please reformat the .csv file to match the lorum ipsum format.") Using an LLM to write the first cut of a well-known or easily defined algorithm ("please write a function to find the nearest correlation matrix to a given matrix using the algorithm of Higham and Strabic") and it will usually produce a reasonable first cut that is arguably faster than typing in myself. Asking an LLM to find a bug in a complex system will generate suggestions; almost all are wrong and usually none are right. It is doubtful that the cases when it finds a right answer pay for the cost of evaluating the rest. Asking an LLM for a meaningful functional change, well you might as well trying flying to the moon by flapping your arms.
My point is that LLMs work best, when they work at all, on the kind of software that is written by Silicon Valley software developers, where "move fast and break things" is highly prized and the consequences of an error are not too serious. This is also the sort of software written by "part time" programmers, who just want a nice plot of the latest Fed data. But it doesn't represent much of the software that makes modern life possible.
All of the above, exactly as you put it. Well done. The only thing I would add is the exponential growth in difficulty later on when it comes time to enhance and maintain the AI-created code. What was done, exactly how and why, is knowledge that becomes increasingly attenuated, like programming that was written over decades by long gone generations of staff who never bothered to document their work. It’s a technical debt that compounds and compounds.
Another domain would be mathematics. The Xena - Mathematicians learning Lean by doing blog points out that LLMs can be useful in translating mathematics into Lean, a theorem proving language where either the theorem compiles and is correct or it fails to compile and is incorrect. It requires a mathematician to check the Lean to verify that it is actually proving the theorem and not something sort of similar. It also needs the user to verify that it doesn't invoke anything that might delete all of one's files. Apparently, LLMs are also pretty good at finding counterexamples.
Like programming, mathematics requires precise statements, has a form of underlying truth and is highly regular in its use of language. "Not for us rose-fingers, the riches of the Homeric language. Mathematical formulae are the children of poverty."
One can see how LLMs would have problems with legal problems since the law and legal system have been developed to deal with the ambiguity of language. Our current Supreme Court has based numerous decisions on novel interpretations of what was once generally accepted language.
Great post, but I think that post-training for programming adds a lot of value that goes beyond compression or the "sanity" introduced by RLHF. Reinforcement learning with programmatic feedback (usually just referred to as RL) is common; the model learns by doing artificial programming tasks, which look a lot like programming benchmarks, and is rewarded for attempts that result in tests passing. Models can also be rewarded for getting to the answer with fewer tokens. They can quickly learn to do harder and harder programming tasks. The downside is that the RL environments can be quite hard to set up correctly except in rare domains like math where verification is much, much easier than solution. Some companies may also be using captured user feedback (I don't know).
If I had to forecast where progress will continue vs. flatten at around the level of a random human on the internet, I'd look at where programmatic verification is easy. (And where testing is inexpensive -- in robotics, success can be easy to verify, but sometimes the robot breaks or breaks something else, so you can't just try over and over.)
I'm a former economist who now finds myself working in Meta's "Advanced Artificial Intelligence" post-training org as a software engineer FWIW. I am unsure whether Meta's heavy investment in post-training will pay off, but I can see an angle where it makes sense.
It seems that, for everything where verification is much much easier than solution, there are potentially enormous returns for constructing a large database of problems and verified solutions by simply generating plausible solution candidates largely at random. If verification is that much easier, that disk space is cheap gives that a powerful advantage relative to doing some deep analysis that leads to a pre-verified solution.
This is what I think I see with my tame and pet LLMs. They are worse at reading Stack Overflow threads than I am. But they can try a hundred things in the time it would take me to try one, and as long as the harness is such that wrong answers are quickly noted, this is good. It is what I call "Clever Hans, stamping his hoof millions of times a second".
How many and how large are the domains of useful human work that fit this template is something that we will discover:
> **Elizabeth Santorella**: Great post, but I think that post-training for programming adds a lot of value that goes beyond compression or the "sanity" introduced by RLHF. Reinforcement learning with programmatic feedback (usually just referred to as RL) is common; the model learns by doing artificial programming tasks, which look a lot like programming benchmarks, and is rewarded for attempts that result in tests passing. Models can also be rewarded for getting to the answer with fewer tokens. They can quickly learn to do harder and harder programming tasks. The downside is that the RL environments can be quite hard to set up correctly except in rare domains like math where verification is much, much easier than solution. Some companies may also be using captured user feedback (I don't know).
> If I had to forecast where progress will continue vs. flatten at around the level of a random human on the internet, I'd look at where programmatic verification is easy. (And where testing is inexpensive -- in robotics, success can be easy to verify, but sometimes the robot breaks or breaks something else, so you can't just try over and over.)
> I'm a former economist who now finds myself working in Meta's "Advanced Artificial Intelligence" post-training org as a software engineer FWIW. I am unsure whether Meta's heavy investment in post-training will pay off, but I can see an angle where it makes sense.
P.S.: With respect to large open-weight-in-cloud LLMs, has NVIDIA with NemoTron just emerged to squat in the space that Meta used to occupy with Llama? Is that what is going on here? Be well, B.
Regardless whether AGI is real, convergence kills the economic model that justifies hyperscaling. Six months behind the pace at half the cost kills burning money at the bleeding edge
Indeed. I confess that, from the shareholder value point of view, it seems to be obvious that Apple's and Facebook's old strategy of dealing with AI is the best one. Facebook originally had two moves. First was using Modern Advanced Machine-Learning Models for better ad targeting. Second was investing in Llama to keep it close to the frontier, and then giving it away for free, precisely so that nobody could accumulate a war chest by selling AI services that they could then use to try to take a run at Facebook's social media monopoly. Apple wanted to analyze your personal data using private on-device models to produce not a superhuman digital god but well a bicycle for the mind.
Apple seems to have finally figured out how to make this work: pay Google enough that Google is happy to help Apple distill Gemini into on-device models that run on iPhones.
Facebook's new look, I do not understand. It looks as though Mark Zuckerberg set his shareholders' wealth on fire to an unbelievable extent in an attempt to make the metaverse a thing, and that he is now turning around and setting his shareholders' wealth on fire again to an unbelievable extent in an attempt to become the personal superintelligence leader of the world. He has pivoted Facebook to Muse Spark, and is trying to sell it as the affordable frontier model.
I agree that I see no path to hyper profitability for any of these investments. But that they are highly unlikely to be profitable for investors does not mean that these investments are not a worthwhile thing for humanity to attempt right now:
> **John Quggin**: Regardless whether AGI is real, convergence kills the economic model that justifies hyperscaling. Six months behind the pace at half the cost kills burning money at the bleeding edge...
Unlike railway mania and 1990s telecommunications bubble, no equivalent to the notion that excess investment will come in handy later. With adoption (outside coding) still slow, duplcative hyperscaling makes little sense.
Doesn't understanding LLMs need to rest on understanding what the second L comprises? If it comprises the communication of warranted/evidenced information and logic then LLMs have some point. But if language comprises status contests ('winning' arguments etc) then not so much. All the major features of language evolved long before there was much in the way of information to communicate. Language remains a remarkably awkward medium for the transmission of accurate, unambiguous information, (unlike maths) and such use requires substantial and effortful preparation. By contrast it is difficult, if not impossible, to conduct a premeditated as opposed to spontaneous informal conversation. LargeGossip/Bragging/MoaningModels
"Yes, writing code is a domain where surface prediction and correct world-modeling are very very close—programs either work or don’t, for there is no underlying reality to which the incantations in their correct form point."
I am a life-long professional software developer. I am under continuous, albeit gentle (no leaderboards) pressure to make use of LLM coding assistance, both chat-based and agentic. I have the benefit of a wide choice of models and professional training from the vendors of these models. I am trying to reserve judgment on the usefulness of LLMs for programming, but the only reason is the proliferation of Silicon Valley programmers telling me how good it is. To be clear, I am not just talking about the "hamburger" model of Kapoor and Narayanan (https://www.normaltech.ai/p/why-ai-hasnt-replaced-software-engineers) - although they are right. In my observation, LLMs are only moderately useful at shrinking the "execute" patty, never mind the bun.
You have (inadvertently) identified what may be the problem: there absolutely is an underlying reference reality for most of the software that makes a difference in your life, and if you fuck that up, then you, too, will be fucked. We are not close to having an LLM that you want to have writing your bank's book-keeping systems, or your hospital's life support systems, or your railroads scheduling system, or basically any other software system of any real importance. I can rely on LLMs to manipulate data ("this .csv file represents a volatility surface in the format blah blah blah, this other file represents a volatility surface in format lorum ipsum, please reformat the .csv file to match the lorum ipsum format.") Using an LLM to write the first cut of a well-known or easily defined algorithm ("please write a function to find the nearest correlation matrix to a given matrix using the algorithm of Higham and Strabic") and it will usually produce a reasonable first cut that is arguably faster than typing in myself. Asking an LLM to find a bug in a complex system will generate suggestions; almost all are wrong and usually none are right. It is doubtful that the cases when it finds a right answer pay for the cost of evaluating the rest. Asking an LLM for a meaningful functional change, well you might as well trying flying to the moon by flapping your arms.
My point is that LLMs work best, when they work at all, on the kind of software that is written by Silicon Valley software developers, where "move fast and break things" is highly prized and the consequences of an error are not too serious. This is also the sort of software written by "part time" programmers, who just want a nice plot of the latest Fed data. But it doesn't represent much of the software that makes modern life possible.
All of the above, exactly as you put it. Well done. The only thing I would add is the exponential growth in difficulty later on when it comes time to enhance and maintain the AI-created code. What was done, exactly how and why, is knowledge that becomes increasingly attenuated, like programming that was written over decades by long gone generations of staff who never bothered to document their work. It’s a technical debt that compounds and compounds.
That is a good point! And I feel that it is also characteristic of the SV community to discount the burden of maintenance.
Another domain would be mathematics. The Xena - Mathematicians learning Lean by doing blog points out that LLMs can be useful in translating mathematics into Lean, a theorem proving language where either the theorem compiles and is correct or it fails to compile and is incorrect. It requires a mathematician to check the Lean to verify that it is actually proving the theorem and not something sort of similar. It also needs the user to verify that it doesn't invoke anything that might delete all of one's files. Apparently, LLMs are also pretty good at finding counterexamples.
Like programming, mathematics requires precise statements, has a form of underlying truth and is highly regular in its use of language. "Not for us rose-fingers, the riches of the Homeric language. Mathematical formulae are the children of poverty."
One can see how LLMs would have problems with legal problems since the law and legal system have been developed to deal with the ambiguity of language. Our current Supreme Court has based numerous decisions on novel interpretations of what was once generally accepted language.
Touché...
Great post, but I think that post-training for programming adds a lot of value that goes beyond compression or the "sanity" introduced by RLHF. Reinforcement learning with programmatic feedback (usually just referred to as RL) is common; the model learns by doing artificial programming tasks, which look a lot like programming benchmarks, and is rewarded for attempts that result in tests passing. Models can also be rewarded for getting to the answer with fewer tokens. They can quickly learn to do harder and harder programming tasks. The downside is that the RL environments can be quite hard to set up correctly except in rare domains like math where verification is much, much easier than solution. Some companies may also be using captured user feedback (I don't know).
If I had to forecast where progress will continue vs. flatten at around the level of a random human on the internet, I'd look at where programmatic verification is easy. (And where testing is inexpensive -- in robotics, success can be easy to verify, but sometimes the robot breaks or breaks something else, so you can't just try over and over.)
I'm a former economist who now finds myself working in Meta's "Advanced Artificial Intelligence" post-training org as a software engineer FWIW. I am unsure whether Meta's heavy investment in post-training will pay off, but I can see an angle where it makes sense.
It seems that, for everything where verification is much much easier than solution, there are potentially enormous returns for constructing a large database of problems and verified solutions by simply generating plausible solution candidates largely at random. If verification is that much easier, that disk space is cheap gives that a powerful advantage relative to doing some deep analysis that leads to a pre-verified solution.
This is what I think I see with my tame and pet LLMs. They are worse at reading Stack Overflow threads than I am. But they can try a hundred things in the time it would take me to try one, and as long as the harness is such that wrong answers are quickly noted, this is good. It is what I call "Clever Hans, stamping his hoof millions of times a second".
How many and how large are the domains of useful human work that fit this template is something that we will discover:
> **Elizabeth Santorella**: Great post, but I think that post-training for programming adds a lot of value that goes beyond compression or the "sanity" introduced by RLHF. Reinforcement learning with programmatic feedback (usually just referred to as RL) is common; the model learns by doing artificial programming tasks, which look a lot like programming benchmarks, and is rewarded for attempts that result in tests passing. Models can also be rewarded for getting to the answer with fewer tokens. They can quickly learn to do harder and harder programming tasks. The downside is that the RL environments can be quite hard to set up correctly except in rare domains like math where verification is much, much easier than solution. Some companies may also be using captured user feedback (I don't know).
> If I had to forecast where progress will continue vs. flatten at around the level of a random human on the internet, I'd look at where programmatic verification is easy. (And where testing is inexpensive -- in robotics, success can be easy to verify, but sometimes the robot breaks or breaks something else, so you can't just try over and over.)
> I'm a former economist who now finds myself working in Meta's "Advanced Artificial Intelligence" post-training org as a software engineer FWIW. I am unsure whether Meta's heavy investment in post-training will pay off, but I can see an angle where it makes sense.
P.S.: With respect to large open-weight-in-cloud LLMs, has NVIDIA with NemoTron just emerged to squat in the space that Meta used to occupy with Llama? Is that what is going on here? Be well, B.
Regardless whether AGI is real, convergence kills the economic model that justifies hyperscaling. Six months behind the pace at half the cost kills burning money at the bleeding edge
Indeed. I confess that, from the shareholder value point of view, it seems to be obvious that Apple's and Facebook's old strategy of dealing with AI is the best one. Facebook originally had two moves. First was using Modern Advanced Machine-Learning Models for better ad targeting. Second was investing in Llama to keep it close to the frontier, and then giving it away for free, precisely so that nobody could accumulate a war chest by selling AI services that they could then use to try to take a run at Facebook's social media monopoly. Apple wanted to analyze your personal data using private on-device models to produce not a superhuman digital god but well a bicycle for the mind.
Apple seems to have finally figured out how to make this work: pay Google enough that Google is happy to help Apple distill Gemini into on-device models that run on iPhones.
Facebook's new look, I do not understand. It looks as though Mark Zuckerberg set his shareholders' wealth on fire to an unbelievable extent in an attempt to make the metaverse a thing, and that he is now turning around and setting his shareholders' wealth on fire again to an unbelievable extent in an attempt to become the personal superintelligence leader of the world. He has pivoted Facebook to Muse Spark, and is trying to sell it as the affordable frontier model.
I agree that I see no path to hyper profitability for any of these investments. But that they are highly unlikely to be profitable for investors does not mean that these investments are not a worthwhile thing for humanity to attempt right now:
> **John Quggin**: Regardless whether AGI is real, convergence kills the economic model that justifies hyperscaling. Six months behind the pace at half the cost kills burning money at the bleeding edge...
Agreed.
Unlike railway mania and 1990s telecommunications bubble, no equivalent to the notion that excess investment will come in handy later. With adoption (outside coding) still slow, duplcative hyperscaling makes little sense.
I appreciate Ted Chiang references.
Doesn't understanding LLMs need to rest on understanding what the second L comprises? If it comprises the communication of warranted/evidenced information and logic then LLMs have some point. But if language comprises status contests ('winning' arguments etc) then not so much. All the major features of language evolved long before there was much in the way of information to communicate. Language remains a remarkably awkward medium for the transmission of accurate, unambiguous information, (unlike maths) and such use requires substantial and effortful preparation. By contrast it is difficult, if not impossible, to conduct a premeditated as opposed to spontaneous informal conversation. LargeGossip/Bragging/MoaningModels
Microsoft has started their MAi project it seems to cut costs an have more control over the model. Have you looked at what they are doing?
No! I should! Be well, B.