@Ferry Building, SF :: 2024-03-14 Th 18:00 PDT :: Reinvent Future :: Very large-data very high-dimension regression & classification analysis; actual natural-language interfaces to structured...
> (1) Very large-scale very high-dimension regression and classification analysis is going to be, if we can manage to tame and subdue it, truly game-changing: the transformation from the world of the bureaucracy to the world of the algorithm, with not just Peter Drucker’s mass production, not just Bob Reich’s flexible customization, but rather bespoke creation for nearly everything. This is the heart of modern MAMLM-GPTM-LLM technologies.
Maybe I'm too deep in too narrow a trench to see the overall picture, but as a point of technical fact I don't see GPTM-LLM technologies being particularly good at any of that except maybe a small subset of domains (so: protein folding - yes, antibody optimization - no; most science and engineering problems do *not* look like language processing). MAMLM, yes (although the term might be too capacious for clarity), but little of the bubble money and engineering is going to non-GPTM-LLM MAMLM. Hopefully after the bubble bursts and we have the leftover infrastructure to play with that'll help.
To be clear, I enthusiastically agree that (1) is an epoch-shifting change and I'm bullish on current and near-term cutting-edge advanced machine learning and optimization algorithms[1] getting us there; I'm just skeptical of GTPM-LLM being part of the path there except as making it easier to get money from people who want to be into "AI" and not look too deeply into which sort.
[1] In terms of scientific and engineering advances, classification and regression aren't the whole story: optimization and active learning are higher multipliers long-term. Oversimplifying, the former gets you better-than-ever factories (really good!) and the latter gets you better-than-ever science (really really good!). In that sense I suspect there's a general over-estimation of what GTPM can do and a general under-estimation of what MAMLM minus GTPM can do.
Technically they're not even "models" in the conventional sense of the term but that's already a much abused term. Horse gone, barn door etc. At some point toasters will be considered models of bread I suppose.
1) I love MAMLM which I pronounce mammal M. (for Mechanical?)
2) I'm working on a Utopian Hard Sci-Fi trilogy which starts in 2030 when 90% of all white collar jobs have been eliminated by AIs but in response COOP base firms have sprung up with don't have executives, middle managers, HR, or legal departments all of which are handled by AIs. (The protagonist's COOP has named her legal and business AI Learned Digit, LD for short. LD has a sense of humor.
3) Spreadsheets are a good example of your thesis. Pre-spreadsheet a boss would ask a (reasonable) question and a team would labor for a few days, weeks, or months and come back with an answer. The boss would have other questions but not want to spend the effort. With spreadsheets, they'd come back in a few days. He's ask variations on the original and get an answer in the meeting or in hours. So he's ask a lot of questions.
Consider the trend of building curated data. I will use as an example of what might be a common use is Brad's sub-Turing BradBot. Trained on Brad's book "Slouching..." and possibly other works. But Brad is much more than his works. As james Burke reminded us, everything is "Connected". So the BradBot needs to be trained with every reference work Brad's books cite. And in turn, perhaps the works that cite these references too. But as we know from this blog, Brad has wider interests and also includes the posts and writings of others, as with teh current Dune critique. So those must be added in. And on and on. If we are to get to near Turing complete versions of Brad with the fidelity of the dead wife in the movie "margorie Prime", an awful lot of knowledge about Brad needs to be collected. This applies to organizations building curated organizational knowledge. But here is teh rub. bad actors will be purposely trying to infiltrate and render that data untrustworthy, despite the best efforts of curators. Neal Stephenson's "Anathem" introduced teh idea of trust in data, but it is going to be hard to accomplish that. Once scientific papers were considered very trustworthy but now we know that they are polluted with fake datam and even fake papers. How are we going to handle this? It is analogous to spam filters, anti-virus software, etc., but at a greater level of difficulty. Which means more work and more software tools to try to weed out corrupt data and prevent malware corrupting data, not to mention internal human actions.
A world of good, sub-Turing instances of people would be wonderful. Porn is already entering this arena with "AI girlfriends". These instances of people will be very useful...but only if they are not corrupted. A public "J Bradford DeLong Prime" would be very nice as long as it didn't stop to insert an advert every few minutes, or exhibit torettes syndrome due to malware. I think P K Dick had already explored this sort of world.
The idea is that increasing size does lead to new skills and supposedly more accuracy of certain skills. I am not convinced. Why does it take a huge LLM to be able to do simple addition when parsing and extracting the elements and feeding them to a simple function works perfectly? Surely it is not beyond the wit of the companies to incorporate tools that RELIABLY WORK?
Cory Doctorrow has a piece that argues that the whole LLM/ChatGPT model is not likely to keep improving as the training data is getting polluted.
This definitely argues for curated data sets. I see a potential demand for expert curators - jobs, jobs, jobs in every organization and even user to build quality information for interrogation.
What strains credulity in my mind, is that our experience of bad actors in the digital world, will effectively derail much of the value of this form of AI. We already have deepfakes, (text, audio, and even video) used for various purposes. Prompt injection (from websites and malicious code) is going to result in poor quality/biased/disinformation being returned for queries. If you thought spellcheck/grammar check/autocorrect were bad, you haven't seen anything yet. This whole edifice could make online information access a minefield. Technology is always a two-edged sword, and teh best we can hope for is a small net benefit at the cost of complexity and controls to tamp down its use as a weapon against the user.
"1. Very large-data very high-dimension regression and classification analysis" is going to be fun in economic history too! e.g. Julius Koschnick has a nice working paper on pre-industrial British academia which uses transformer models to classify the topic and measure the innovativeness of research.
I actually have found the most recent updates to Apple's iOS auto-complete functionality intensely annoying.
I type regularly in two different languages. It used to be that if I flipped to the Spanish keyboard mode, auto-complete would pretty reliably complete Spanish words. But in the last few months, it's suddenly, and fairly consistently, trying to "correct" my spelling into English, even for the most basic words. So "Es" at the start of a sentence will get turned into "Ed", or I'll be trying to write "Es probable que", and it will want to auto-complete that second word as "probably". These examples are on my mind because they _just_ happened, within the last hour or so. My experience of their newly AI-influenced product is that it has set back the usefulness of auto-complete by about 4-5 years, because they think they are now too smart to have to pay any attention to the user's declaration of what language they're trying to write.
AI is like MOOCs or self driving cars. It offers investors and managers a labor-free fantasy. Since investors and managers rarely have a clue as to how their business works, they won't notice if AI is doing a terrible job. The real AI threat is to workers who will be replaced by AI and customers who will have to put up with a lower quality product. The investors will get bailed out by the government and the managers will have golden parachutes, so the consequences of failure are minimal.
Not just switchboard operators. Copy typists have disappeared. Personal secretaries became Executive Assistants, but the typing pool is no more. Similarly Data Punch Operators, though this was a category created by ITC to replace lots of clerks.
But the obvious response to improved tech productivity, standard until the rise of neoliberalism. is shorter working hours.
I suspect this may be the interesting area. What surprising uses with various AI techniques be with bucketloads of computational brute force behind them? I am interested in how AI algorithms and deployments can be puched to the "edge" rather than centralized. What will that mean for such devices and how will that change things?
One observation I have noted over my lifetime spanning mainframmes to smartphones and smaller devices, is that information technologies don't just replace existing work, but hugely expand it - a sort of Jeavons paradox. Spreadsheets were just becoming available when I diod my MBA in England. What happened was that playing with spreadsheet models replaced carefully thought through simple examples. Spreadsheet-ITIS. Recall how making sacetate slides changed with PowerPoint? Aerospace companies used to just use windtunnels - now aerodynamics are investigated with simulations testing many more cases and variations. And so it goes. The simple becomes more complex. Yes, the solutions can be better, but computers have created a huge demand to use them more to explore the solution space. Even in Math, once the approach was to solve differential and integral equations with calculus. Now you can use a computer algorithm to brute force the answer (and better still, it works with equations that are not amenable to calculus).
Bottom line is that despite the concerns of computers creating unemployment in the early 1980s, the reverse happened. We are seeing the same concern today with the current crop of new AI algorithms. I suspect that as then we will generate more work, not less.
With regard to AI safety, I think you are more or less making the argument that "AI doesn't kill people, people kill people". In a very broad sense, this is correct. We do need to figure out how to regulate it, though, and we need to figure that out pretty quickly. I am pretty sure the constitution does not guarantee the right to bear LLMs.
No. What nearly brought down the world economy in 2008 was the Fed* just not telling everybody [not that they should have needed telling!] that if would NOT allow inflation to fall (very much for very long) below target and unemployment to fall below full and going mano a mano with anyone who did not believe them.
*Yes the ECB was worse, but without a US recession, it would not have had the "occasion for sin."
> (1) Very large-scale very high-dimension regression and classification analysis is going to be, if we can manage to tame and subdue it, truly game-changing: the transformation from the world of the bureaucracy to the world of the algorithm, with not just Peter Drucker’s mass production, not just Bob Reich’s flexible customization, but rather bespoke creation for nearly everything. This is the heart of modern MAMLM-GPTM-LLM technologies.
Maybe I'm too deep in too narrow a trench to see the overall picture, but as a point of technical fact I don't see GPTM-LLM technologies being particularly good at any of that except maybe a small subset of domains (so: protein folding - yes, antibody optimization - no; most science and engineering problems do *not* look like language processing). MAMLM, yes (although the term might be too capacious for clarity), but little of the bubble money and engineering is going to non-GPTM-LLM MAMLM. Hopefully after the bubble bursts and we have the leftover infrastructure to play with that'll help.
To be clear, I enthusiastically agree that (1) is an epoch-shifting change and I'm bullish on current and near-term cutting-edge advanced machine learning and optimization algorithms[1] getting us there; I'm just skeptical of GTPM-LLM being part of the path there except as making it easier to get money from people who want to be into "AI" and not look too deeply into which sort.
[1] In terms of scientific and engineering advances, classification and regression aren't the whole story: optimization and active learning are higher multipliers long-term. Oversimplifying, the former gets you better-than-ever factories (really good!) and the latter gets you better-than-ever science (really really good!). In that sense I suspect there's a general over-estimation of what GTPM can do and a general under-estimation of what MAMLM minus GTPM can do.
Technically they're not even "models" in the conventional sense of the term but that's already a much abused term. Horse gone, barn door etc. At some point toasters will be considered models of bread I suppose.
1) I love MAMLM which I pronounce mammal M. (for Mechanical?)
2) I'm working on a Utopian Hard Sci-Fi trilogy which starts in 2030 when 90% of all white collar jobs have been eliminated by AIs but in response COOP base firms have sprung up with don't have executives, middle managers, HR, or legal departments all of which are handled by AIs. (The protagonist's COOP has named her legal and business AI Learned Digit, LD for short. LD has a sense of humor.
3) Spreadsheets are a good example of your thesis. Pre-spreadsheet a boss would ask a (reasonable) question and a team would labor for a few days, weeks, or months and come back with an answer. The boss would have other questions but not want to spend the effort. With spreadsheets, they'd come back in a few days. He's ask variations on the original and get an answer in the meeting or in hours. So he's ask a lot of questions.
Consider the trend of building curated data. I will use as an example of what might be a common use is Brad's sub-Turing BradBot. Trained on Brad's book "Slouching..." and possibly other works. But Brad is much more than his works. As james Burke reminded us, everything is "Connected". So the BradBot needs to be trained with every reference work Brad's books cite. And in turn, perhaps the works that cite these references too. But as we know from this blog, Brad has wider interests and also includes the posts and writings of others, as with teh current Dune critique. So those must be added in. And on and on. If we are to get to near Turing complete versions of Brad with the fidelity of the dead wife in the movie "margorie Prime", an awful lot of knowledge about Brad needs to be collected. This applies to organizations building curated organizational knowledge. But here is teh rub. bad actors will be purposely trying to infiltrate and render that data untrustworthy, despite the best efforts of curators. Neal Stephenson's "Anathem" introduced teh idea of trust in data, but it is going to be hard to accomplish that. Once scientific papers were considered very trustworthy but now we know that they are polluted with fake datam and even fake papers. How are we going to handle this? It is analogous to spam filters, anti-virus software, etc., but at a greater level of difficulty. Which means more work and more software tools to try to weed out corrupt data and prevent malware corrupting data, not to mention internal human actions.
A world of good, sub-Turing instances of people would be wonderful. Porn is already entering this arena with "AI girlfriends". These instances of people will be very useful...but only if they are not corrupted. A public "J Bradford DeLong Prime" would be very nice as long as it didn't stop to insert an advert every few minutes, or exhibit torettes syndrome due to malware. I think P K Dick had already explored this sort of world.
2 very different views:
1. https://nautil.us/how-quickly-do-large-language-models-learn-unexpected-skills-528047/
The idea is that increasing size does lead to new skills and supposedly more accuracy of certain skills. I am not convinced. Why does it take a huge LLM to be able to do simple addition when parsing and extracting the elements and feeding them to a simple function works perfectly? Surely it is not beyond the wit of the companies to incorporate tools that RELIABLY WORK?
But overall the message is very optimistic.
2. https://pluralistic.net/2024/03/14/inhuman-centipede/#enshittibottification
Cory Doctorrow has a piece that argues that the whole LLM/ChatGPT model is not likely to keep improving as the training data is getting polluted.
This definitely argues for curated data sets. I see a potential demand for expert curators - jobs, jobs, jobs in every organization and even user to build quality information for interrogation.
What strains credulity in my mind, is that our experience of bad actors in the digital world, will effectively derail much of the value of this form of AI. We already have deepfakes, (text, audio, and even video) used for various purposes. Prompt injection (from websites and malicious code) is going to result in poor quality/biased/disinformation being returned for queries. If you thought spellcheck/grammar check/autocorrect were bad, you haven't seen anything yet. This whole edifice could make online information access a minefield. Technology is always a two-edged sword, and teh best we can hope for is a small net benefit at the cost of complexity and controls to tamp down its use as a weapon against the user.
"1. Very large-data very high-dimension regression and classification analysis" is going to be fun in economic history too! e.g. Julius Koschnick has a nice working paper on pre-industrial British academia which uses transformer models to classify the topic and measure the innovativeness of research.
I actually have found the most recent updates to Apple's iOS auto-complete functionality intensely annoying.
I type regularly in two different languages. It used to be that if I flipped to the Spanish keyboard mode, auto-complete would pretty reliably complete Spanish words. But in the last few months, it's suddenly, and fairly consistently, trying to "correct" my spelling into English, even for the most basic words. So "Es" at the start of a sentence will get turned into "Ed", or I'll be trying to write "Es probable que", and it will want to auto-complete that second word as "probably". These examples are on my mind because they _just_ happened, within the last hour or so. My experience of their newly AI-influenced product is that it has set back the usefulness of auto-complete by about 4-5 years, because they think they are now too smart to have to pay any attention to the user's declaration of what language they're trying to write.
AI is like MOOCs or self driving cars. It offers investors and managers a labor-free fantasy. Since investors and managers rarely have a clue as to how their business works, they won't notice if AI is doing a terrible job. The real AI threat is to workers who will be replaced by AI and customers who will have to put up with a lower quality product. The investors will get bailed out by the government and the managers will have golden parachutes, so the consequences of failure are minimal.
Not just switchboard operators. Copy typists have disappeared. Personal secretaries became Executive Assistants, but the typing pool is no more. Similarly Data Punch Operators, though this was a category created by ITC to replace lots of clerks.
But the obvious response to improved tech productivity, standard until the rise of neoliberalism. is shorter working hours.
"6. And more…"
I suspect this may be the interesting area. What surprising uses with various AI techniques be with bucketloads of computational brute force behind them? I am interested in how AI algorithms and deployments can be puched to the "edge" rather than centralized. What will that mean for such devices and how will that change things?
One observation I have noted over my lifetime spanning mainframmes to smartphones and smaller devices, is that information technologies don't just replace existing work, but hugely expand it - a sort of Jeavons paradox. Spreadsheets were just becoming available when I diod my MBA in England. What happened was that playing with spreadsheet models replaced carefully thought through simple examples. Spreadsheet-ITIS. Recall how making sacetate slides changed with PowerPoint? Aerospace companies used to just use windtunnels - now aerodynamics are investigated with simulations testing many more cases and variations. And so it goes. The simple becomes more complex. Yes, the solutions can be better, but computers have created a huge demand to use them more to explore the solution space. Even in Math, once the approach was to solve differential and integral equations with calculus. Now you can use a computer algorithm to brute force the answer (and better still, it works with equations that are not amenable to calculus).
Bottom line is that despite the concerns of computers creating unemployment in the early 1980s, the reverse happened. We are seeing the same concern today with the current crop of new AI algorithms. I suspect that as then we will generate more work, not less.
Hexapodia - change in topic. Perhaps it is time for a Hexapodia on social media. Or on AI. Or on something.
With regard to AI safety, I think you are more or less making the argument that "AI doesn't kill people, people kill people". In a very broad sense, this is correct. We do need to figure out how to regulate it, though, and we need to figure that out pretty quickly. I am pretty sure the constitution does not guarantee the right to bear LLMs.
No. What nearly brought down the world economy in 2008 was the Fed* just not telling everybody [not that they should have needed telling!] that if would NOT allow inflation to fall (very much for very long) below target and unemployment to fall below full and going mano a mano with anyone who did not believe them.
*Yes the ECB was worse, but without a US recession, it would not have had the "occasion for sin."