• medem@lemmy.wtf
    link
    fedilink
    arrow-up
    44
    ·
    1 day ago

    Well, now that y’all put it that way, I think it was pretty naive from me to think that these companies, whose business model is basically theft, would honour a lousy robots.txt file…