Telling the Bing Network How Often to Crawl Your Website

By default, the Bing Network sets its own crawl rates and times based on load sensed from your server. If you are experiencing bandwidth related server load issues such as too many requests too quickly, you may want to reduce how often the Bing Network and other search engines crawl your site. Too many requests on your site in a very short period can cause your site to load slowly and even cause load issues on the server. This is especially true of very busy websites or websites that are poorly or inefficiently coded.

If you are not experiencing any bandwidth related load issues on your server, it is recommended that you allow Bing to determine the optimal crawl rate for your website.

Changing the Bing Network's Crawl Settings

Microsoft allows you to set the crawl rate for your site and even allows you to specify the crawl rate by hour of the day (a crawl pattern). This allows you to specify that the Bing Network crawl your site more aggressively during non-peak hours and less aggressively while a majority of your visitors are online.

Note: Before specifying a crawl pattern, you may want to view your web statistics in cPanel to see when most of your visitors are online.

To change the Bing Network's crawl settings:

  1. Log into your Bing Network Webmaster Tools.
  2. Add your site to the Bing Network Webmaster Tools if you have not already done so.
  3. Click on the Crawl tab in the top navigation.
  4. Click on Crawl Settings in the left navigation.
  5. Select your website from the drop-down list (if not already selected).
  6. Select the radio button Use the selected crawl rate below to set a custom crawl rate and pattern.
  7. Use the graph to specify the crawl rate and crawl pattern by dragging the graph areas to the desired settings. You can click on individual vertical columns to set the crawl rate individually by hour.
  8. If your website is built in AJAX, select the check box labeled Configure the Bing Network for Ajax site crawling so that the Bing Network handles your website properly.
  9. Click Save to save your settings.