Collect URL scans internet like a search engine bot. Starting from the user entered URL, it lists all the links on the URL. Then it recursively scans all the links in the list. User can limit Collect URL to collect single domain only. File extension ignore list is editable. Connection and response timeout values are editable. Also user can change user agent value for HTTP and HTTPS connections.
