this is to allow all search engines crawl all pages, although Disallow is not allowed, but behind the empty state, it is not allowed to crawl the page No.
After installing the
first to simply say robots.txt file code:
the first two lines mean is not allowed 360 comprehensive search spider crawling any page, the explanation behind the first see. Similarly, if in addition to shield 360 comprehensive search to shield the spider love Shanghai, continue to add at the beginning.
WordPress site on the robots.txt document writing has been a lot of trouble webmaster, Robots.txt File protocol called search engine robots, search engine crawlers when crawling the site, will first look at the root directory of the web site is the robots.txt file, and then follow the robots.txt protocol crawling site owners want the search engine crawling content. The robots.txt file is intended to tell the search engine spiders which pages can crawl, which pages can not crawl, can effectively protect the user’s privacy, but also conducive to save spider bandwidth, so as to let the spider climb take more easily, promote the collection.
return to the point, again said WordPress robots.tx>
we all know, WordPress management background in the root directory of the wp-admin folder inside, plus the /wp-admin behind disallow means the search engine spiders crawling not allowed.
does not allow the search engine grab some one page, here to not allow all search engines crawl management background page take WordPress as an example
2, shielding one or several search engines, with 360 comprehensive search the latest fire as an example
so as not to allow the love to climb from the Shanghai background, allowing other search engines crawl back, or do not allow 360 comprehensive search crawling back, allowing other search engines crawl back and combination, please refer to the above three points are combined.
1, allowing all search crawling caused by any content