Robots.txt

The robots.txt file allows communication with web robots, specifying which parts of the website they are allowed or disallowed to access. It helps control crawling behavior, safeguard sensitive content, and influence how search engines index a website's pages.