dramaticcat@sh.itjust.works to Lemmy Shitpost@lemmy.world · 11 months agoChad scrapersh.itjust.worksimagemessage-square73fedilinkarrow-up1977arrow-down126
arrow-up1951arrow-down1imageChad scrapersh.itjust.worksdramaticcat@sh.itjust.works to Lemmy Shitpost@lemmy.world · 11 months agomessage-square73fedilink
minus-squareViscloReader@lemmy.worldlinkfedilinkarrow-up66arrow-down2·11 months agoJust a heads up for companies thinking it’s wrong to scrap: if you don’t want info to be scraped, don’t put it on the internet.
minus-squareDAMunzy@lemmy.dbzer0.comlinkfedilinkarrow-up12·11 months agoBut, but, norobots.txt! Chad doesn’t care!
minus-squarePoolloverNathan@programming.devlinkfedilinkEnglisharrow-up5·11 months agouses curl manually instead
Just a heads up for companies thinking it’s wrong to scrap: if you don’t want info to be scraped, don’t put it on the internet.
But, but, norobots.txt!
Chad doesn’t care!
uses curl manually instead