The /robots.txt file is simply a text file on your root directory that tells all “good” robots (the ones that aren’t fishing for email address of other spammy pursuits) what parts of your site to not index. By default all content that is accessible to a search engine spider is considered fair game. Because of [...]