shish_mish@lemmy.world to Technology@lemmy.worldEnglish · 4 months agoGoogle Is the Only Search Engine That Works on Reddit Now Thanks to AI Dealwww.404media.coexternal-linkmessage-square34fedilinkarrow-up1185arrow-down18cross-posted to: reddit@lemmy.worldreddit@lemmy.world
arrow-up1177arrow-down1external-linkGoogle Is the Only Search Engine That Works on Reddit Now Thanks to AI Dealwww.404media.coshish_mish@lemmy.world to Technology@lemmy.worldEnglish · 4 months agomessage-square34fedilinkcross-posted to: reddit@lemmy.worldreddit@lemmy.world
minus-squareBrewchin@lemmy.worldlinkfedilinkEnglisharrow-up70·4 months agoParts of the Internet now only searchable on specific sites now? What next - charging a monthly subscription to use Google? This needs to be regulated before the Internet becomes like streaming TV.
minus-squaretal@lemmy.todaylinkfedilinkEnglisharrow-up28arrow-down3·4 months agoRobots.txt has been around for a long time, and all the major search engines will honor it. Not having a full index of the Web is the norm. That isn’t to say that the practice of signing agreements isn’t potentially a concern.
minus-squarereddig33@lemmy.worldlinkfedilinkEnglisharrow-up10arrow-down1·4 months agoWhat isn’t the norm is to serve one robots.txt to one company, and a different robots.txt to everyone else. Which is what Reddit is doing here.
minus-squareDominusOfMegadeus@sh.itjust.workslinkfedilinkEnglisharrow-up1·4 months agodeleted by creator
Parts of the Internet now only searchable on specific sites now? What next - charging a monthly subscription to use Google?
This needs to be regulated before the Internet becomes like streaming TV.
Robots.txt has been around for a long time, and all the major search engines will honor it. Not having a full index of the Web is the norm.
That isn’t to say that the practice of signing agreements isn’t potentially a concern.
What isn’t the norm is to serve one robots.txt to one company, and a different robots.txt to everyone else. Which is what Reddit is doing here.
deleted by creator