Comments on: MSNbot 2.0b is ignoring robots.txt and No Index meta tags

By: thorvaldaagaard

thorvaldaagaard — Sat, 25 Jul 2009 12:48:43 +0000

Have you notice a problem with the microsoft bot following a link line http://www.yoursite.con/default.php#someplacein...
Using this in a bowser it will stirp of the name tag, and request the document like http://www.yoursite.con/default.php, but the bot requests the entire url, resulting in a 404

By: Graphic Designer

Graphic Designer — Wed, 03 Jun 2009 08:47:28 +0000

Naughty MSN Bot, bad boy!!!

By: Chewie

Chewie — Tue, 28 Apr 2009 18:21:52 +0000

John: I had no idea that it was also spidering non www, jeez they have more problems than i thought.

By: JohnQPublic

JohnQPublic — Mon, 27 Apr 2009 23:40:14 +0000

Better yet. I’ve notice MSNbot tries to spider domains without a WWW entry in DNS or HTTP server online. Thus I see msnbot trying to spider the company’s SMTP server.

I will give MSNBot this… it’s a persistent little pest.

By: MSNbot using up large amounts of bandwidth

MSNbot using up large amounts of bandwidth — Mon, 20 Apr 2009 23:08:58 +0000

[...] and having read an article last week about the same robot apparently ignoring robots.txt files (MSNbot 2.0b is ignoring robots.txt and No Index meta tags) I went looking in the raw [...]

By: Chewie

Chewie — Fri, 17 Apr 2009 09:43:23 +0000

Richard: Hey man, thanks for the kind words. Let me know what you find in your logs.

Yura: Well however you want to do it is fine, i showed the .htaccess example since it is a quick and easy way to do it.

By: Yura

Yura — Fri, 17 Apr 2009 06:05:10 +0000

I was actually going to ban MSN with robots.txt for spoiling my keyword traffic reports (by doing cloaking checks), but guess I’ll have to use .htaccess. Thanks for the tip.

By: richardbaxterseo

richardbaxterseo — Thu, 16 Apr 2009 20:33:01 +0000

Nice post dean, I’m tempted to go look at my server logs to see what the score is over at ‘gadget. Speak soon!

By: Chewie

Chewie — Thu, 16 Apr 2009 20:26:18 +0000

phaithful: Yeah agree, it is very strange. The thing is, no one will know whats really going on until MS tell us. I don’t expect that happening for some time, if ever.

By: MSNbot 2.0b – ignoriert Robots.txt und noindex tag

MSNbot 2.0b – ignoriert Robots.txt und noindex tag — Thu, 16 Apr 2009 19:13:27 +0000

[...] MSNbot 2.0b is ignoring robots.txt and No Index meta tags | Chewie.co.uk – Now with 100% less Wookie… Tags: Live Search, Meta Tags, MSNbot, Robots.txt Microsoft, Windows Live RSS-Feed Trackback [...]