Topic: Robots.txt
From around the web
Nose, face, cut, spite: Blocking Google
November 15, 2009 | BuzzMachine
to do that with robots.txt - so go ahead if you want). I love that The Reach Group (TRG), a German consulting company, has quantified [... ] ...
Rupert Murdoch vows to take all of Newscorp's websites out of Google, abolish fair...
November 09, 2009 | Boing Boing
If you do not like it, just add a robots.txt file that tells Google not to index you. Until you do, stop whining and put it back in your pants. Now Rupert has promised to do...
Google Ransom Note
November 08, 2009 | Fimoculous.com
The idea is that publishers could use their robots.txt as a ransom note, selling it to the highest bidder -- Bing or Google. (I suspect this idea takes fire and gets repeated a...
Google Now Scanning RSS, Atom Feeds, May Experiment with Real- Time Protocols in...
October 30, 2009 | Read/WriteWeb
The post advises website owners who are blocking Google's search bot software known as Googlebot from crawling their RSS/Atom feeds to unblock it via their robots.txt file. If...
The Morning Newsfeed: 05. 15. 09
May 15, 2009 | Media News
Blocking Google is easily done with a simple tag called Robots Exclusion Protocol, or robots.txt protocol. INDenver Times Renegades Plan Rocky Mountain Independent (Westword)...
Rupert Murdoch: Let's Charge for Online Content Again
May 07, 2009 | Read/WriteWeb
a line to their robots.txt file anyway). It's Not About the Future of Newspapers - It's about Journalism We also think that it is important to move away from the question of...
RIAA/MPAA hit men private enforcement site indexed by Google
May 14, 2009 | Boing Boing
Thanks to BayTSP's inability to write a robots.txt file, Google has indexed just about every page on their server. You can get a good idea of the kinds of things that the...
Robots.txt on twitter
Use sparingly, and in a non-discriminatory fashion! RT @radar: Robots.Txt and the .Gov TLD http://bit.ly/7POe1b
by fynali, November 20, 2009
Robots.Txt and the .Gov TLD: The robots.txt file should be used sparingly by government organizations and only .. http://bit.ly/7POe1b
by Telletto, November 20, 2009
Carl Malamud on robots.txt exclusions in .gov domains: http://bit.ly/4pl86g
by cpwillett, November 20, 2009
Robots.Txt and the .Gov TLD http://bit.ly/8Zh5D2
by liorsfeed, November 20, 2009
Wow, I post one link and immediately hit by 10 bots. Only bitlybot checked for robots.txt (really MSFT? really GOOG?) TweetmemeBot hit x 6
by daniellereisch, November 20, 2009
Robots.Txt and the .Gov TLD http://ff.im/-bM0Qs
by briangryth, November 20, 2009
RT @carlmalamud Interesting stats on the use of robots.txt by .gov sites. http://bit.ly/7cWBvg Factual data: http://bit.ly/4E3Jq9
by gilelbaz, November 20, 2009
Robots.Txt and the .Gov TLD: The robots.txt file should be used sparingly by government organizations and only .. http://bit.ly/58t5c0
by andreimpop, November 20, 2009