Extract sitemap: FAQs

1. What is the expected user input to extract a sitemap?
   • A valid URL is all that is required to extract URLs from a sitemap. Some examples include:
     • XML sitemap URL (example.com/sitemap.xml)
     • Domain URL (example.com)
     • Subdomain URL (subdomain.example.com)
     • File path (example.com/subfolder)
     • Any page on the domain / subdomain (example.com/page.html)
2. How will URLs be extracted?
   • If you input an XML sitemap URL, all URLs present within the sitemap file would be extracted.
   • For other input formats, URLs would be extracted in the following manner:
     • Base domain / subdomain URL would be extracted from the input.
     • Within the base domain / subdomain the extractor would search for robots.txt file contains sitemap URL(s).
     • URLs would then be extracted from the sitemap(s) fetched above.
   • Please note that the count of URLs extracted would be capped at 10k to avoid any computation implications.