You know, one can already put a tag in one's website that forbids search engines to crawl it.
One of the staples of the web is teaching/information content. It is a big winner on YouTube, for instance. But the described practice might undermine the attention economy reasons for creating such content.
Perhaps we need a tag that says, "go ahead and index, but you are forbidden to feed this to a LLM or similar AI".
And then I wonder, would this make the web better or worse? Would it make such sites less travelled or more travelled.
I use the web for shopping, yes, but also for generic information about the world. Information to help me program better. Information to help me understand world events better. Sometimes to understand mechanics/strategy in a computer game. And so on.
Sometimes the AI response is helpful, but I can't ever trust it. Not after seeing all the times it makes things up.
This post seems to be missing the known enshittification of Google Search to boost ad revenue. The search was deliberately made worse so that the "best results are to be found on page 1" has been downgraded, so that one needs to dig through more pages. To get more revenue, this implies that just having an ad or paying website on the search page generates revenue for Google. Yes, increasing ad rates boost that further.
But as any business knows, increasing ad costs without a comparable gain just results in advertising elsewhere or through other means. Squeezing businesses and worsening the ad experience is eventually self-destructive.
Despite the increasingly poor quality, I still use Google Search with the AI summaries turned off, but if MAMLMs could deliver what I want, I might end my use of Google Search. However, I don't expect that to happen any time soon.
I switched to Kagi as an improvement on Google, AFAICT it's a clever front end. Bundled with it is FastGP, invoked when you ask a question. It's AI response includes footnoted links for every claim that is made. Massively better than Google.
Roaming through the stacks of Widener Library, the book you wanted was not the one you had the classification code for but the one three to its left—something you could only learn with a stack pass Truer words were never written. This takes me back 60 years, when I spent countless hours in the stacks, a high percentage wasted (I don’t actually believe this for a second) on gaining knowledge and understanding of subjects that had nothing to do with my area of concentration or other course work.
The Southern African Plains Ape has been creating better ways to capture, store and share knowledge and information since the dawn of cave paintings. Written language, mathematical formulae, printing presses, computing machines, networking machines, software abstraction ... each can be seen as new platforms that accelerated the Ape's ability to create new knowledge and technologies. The "business models" for these platforms were key to their proliferation and impact. Aren't LLMs simply the next platform in this evolution, one that will likely mean a yet-to-be-determined new business model? After all, Google Search in it's infancy was a screen devoid of advertising or any other obvious way to make money. It's clear superiority over previous search engines was only part of becoming a going concern; figuring out the business model was the other part.
The Web sites that have lived off of traffic directed to them by Google Search will evolve or perish, like newspapers that lived off of classified advertising revenue before Craig's List.
"Might we see a bifurcation of the web: one layer optimized for human navigation and engagement, another for AI ingestion and synthesis?"
Suggests an LLM friendly equivalent of RSS. A page summarizing the website representing what a Trainer would produce. Assumes a common definition, (some XML definition?)
Anyhow, a techy solution but might be useful in eliminating the duplication of hundreds (thousands?) of web crawlers which currently then have to take the pig home and turn it into bacon to cook.
Google has a brand with tremendous value. Other search engines do basically the same thing. Some even better. But the default search is Google. Its start page will remain on top. Monetizing that in the AI disruption will be its income growth challenge.
You know, one can already put a tag in one's website that forbids search engines to crawl it.
One of the staples of the web is teaching/information content. It is a big winner on YouTube, for instance. But the described practice might undermine the attention economy reasons for creating such content.
Perhaps we need a tag that says, "go ahead and index, but you are forbidden to feed this to a LLM or similar AI".
And then I wonder, would this make the web better or worse? Would it make such sites less travelled or more travelled.
I use the web for shopping, yes, but also for generic information about the world. Information to help me program better. Information to help me understand world events better. Sometimes to understand mechanics/strategy in a computer game. And so on.
Sometimes the AI response is helpful, but I can't ever trust it. Not after seeing all the times it makes things up.
This post seems to be missing the known enshittification of Google Search to boost ad revenue. The search was deliberately made worse so that the "best results are to be found on page 1" has been downgraded, so that one needs to dig through more pages. To get more revenue, this implies that just having an ad or paying website on the search page generates revenue for Google. Yes, increasing ad rates boost that further.
But as any business knows, increasing ad costs without a comparable gain just results in advertising elsewhere or through other means. Squeezing businesses and worsening the ad experience is eventually self-destructive.
Despite the increasingly poor quality, I still use Google Search with the AI summaries turned off, but if MAMLMs could deliver what I want, I might end my use of Google Search. However, I don't expect that to happen any time soon.
I switched to Kagi as an improvement on Google, AFAICT it's a clever front end. Bundled with it is FastGP, invoked when you ask a question. It's AI response includes footnoted links for every claim that is made. Massively better than Google.
Roaming through the stacks of Widener Library, the book you wanted was not the one you had the classification code for but the one three to its left—something you could only learn with a stack pass Truer words were never written. This takes me back 60 years, when I spent countless hours in the stacks, a high percentage wasted (I don’t actually believe this for a second) on gaining knowledge and understanding of subjects that had nothing to do with my area of concentration or other course work.
The Southern African Plains Ape has been creating better ways to capture, store and share knowledge and information since the dawn of cave paintings. Written language, mathematical formulae, printing presses, computing machines, networking machines, software abstraction ... each can be seen as new platforms that accelerated the Ape's ability to create new knowledge and technologies. The "business models" for these platforms were key to their proliferation and impact. Aren't LLMs simply the next platform in this evolution, one that will likely mean a yet-to-be-determined new business model? After all, Google Search in it's infancy was a screen devoid of advertising or any other obvious way to make money. It's clear superiority over previous search engines was only part of becoming a going concern; figuring out the business model was the other part.
The Web sites that have lived off of traffic directed to them by Google Search will evolve or perish, like newspapers that lived off of classified advertising revenue before Craig's List.
"Might we see a bifurcation of the web: one layer optimized for human navigation and engagement, another for AI ingestion and synthesis?"
Suggests an LLM friendly equivalent of RSS. A page summarizing the website representing what a Trainer would produce. Assumes a common definition, (some XML definition?)
Anyhow, a techy solution but might be useful in eliminating the duplication of hundreds (thousands?) of web crawlers which currently then have to take the pig home and turn it into bacon to cook.
Google has a brand with tremendous value. Other search engines do basically the same thing. Some even better. But the default search is Google. Its start page will remain on top. Monetizing that in the AI disruption will be its income growth challenge.