Tag Archives: Web crawler

How and when to use robots.txt file

We have heard of crawlers and bots are crawling our sites to scrap content for various reasons like indexing in search engines, identifying content, scanning email addresses, etc. There are all kinds of crawlers/bots which crawl websites. While some are good which should be allowed access to our site, but we might want to restrict some. In this post we will see how we can do this.

Topics Covered:

Read more »