A robots.txt file just isn't an ideal or powerful technique for blocking sensitive or private content. It only instructs well-behaved crawlers that the pages will not be for them, but it doesn't prevent your server from providing These pages to the browser that requests them. 1 rationale is the fact that search engines like google could however ref