Robots.txt Is usually a file that exists on the basis Listing of every website and can be used to instruct search engines like google and yahoo on which directories/information of the website they might crawl and involve within their index. In line with Rubright, a much better method is hand-picking https://news.rafeeg.ae