禁止微软蜘蛛,爬的太疯狂了,还不带流量…
同时降低频率到60秒间隔.
在web根目录下编辑robots.txt
User-agent: Bingbot
Disallow: /
User-agent: Adidxbot
Disallow: /
User-agent: MSNBot
Disallow: /
User-agent: BingPreview
Disallow: /
User-agent: *
Disallow:
Crawl-delay: 60
Disallow: /api/
Disallow: /data/
参考:
http://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0
http://tool.chinaz.com/robots/
No Responses (yet)
Stay in touch with the conversation, subscribe to the RSS feed for comments on this post.