User Agent Selection
Choose a user agent to identify your crawler to websites.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
Robots.txt Settings
Configure how the crawler respects robots.txt directives.
Sitemap Settings
Configure which pages to include in the crawl.
Response Codes Filter
Select which HTTP response codes to include in results.
Crawl Depth & Timeout
Configure how deep and how long the crawler should work.