https://www.virendrachandak.com/techtalk/how-and-when-to-use-robots-txt-file/