# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file User-agent: * Allow: / Disallow: /admin/ Disallow: /*_escaped_fragment_ Disallow: /cms/ Disallow: /api/ Disallow: /cdn-cgi/ User-agent: OAI-SearchBot Allow: / Disallow: /admin/ Disallow: /*_escaped_fragment_ Disallow: /cms/ Disallow: /api/ Disallow: /cdn-cgi/ User-agent: GPTBot Disallow: / User-agent: SEOkicks-Robot Disallow: / Sitemap: https://www.careerhunter.io/sitemap.xml Crawl-delay: 4.5