Didn't know. I would still maintain, however, that alphabet agencies don't care about robots.txt. I mean, as far as I know, the NSA records every single thing that is transmitted on the the internet, much less stored. Seems unbelievable, but that's what I heard. If anyone has the data capacity to store it, it's them.
You heard that from me, as I posted the big researchposts on Voat about so-called 'CALEA Compliance', which mandates by law that everything over the internet be recorded. Passed as law in 1994, effective 1996, so everything since 1996 was monitored.
Problem was that the technology for dragnet spying didn't exist in 1996, so we had to import it from 'our allies' in Israel, a company called Com Infosys. This is the same Com Infosys that implicated in declassified 9/11 papers.
I'll paste the whole posts for you if you want. I did manage to save those before Voat died.
Sure, thanks. I think I read it in wired, but you go too.
(post is archived)