Tool for crawling a given domain, and mapping out it's URL's based on scraped links. A play project for learning go.
Installation
go get
Example usage
go run crawler -d 5 -b example.com
go run crawler --help
Output
.txt: textual sitemap text file |.log: log file
Tests
go test
Usage example
go run . -b https://www.katakwe.com -d 2
https://www.katakwe.com/
- https://www.katakwe.com/
- https://www.katakwe.com/
- https://www.katakwe.com/api/auth/login
- https://www.katakwe.com/providers
- https://www.katakwe.com/parents
- https://www.katakwe.com/overview
- https://www.katakwe.com/earnings
- https://www.katakwe.com/social-responsibility
- https://www.katakwe.com/commitment-to-privacy
- https://www.katakwe.com/terms-and-conditions
- https://www.katakwe.com/privacy-policy
- https://www.katakwe.com/competition-terms-and-conditions
- https://www.katakwe.com/api/auth/login
- https://www.katakwe.com/
- https://www.katakwe.com/contact-us
- https://www.katakwe.com/providers
- https://www.katakwe.com/
- https://www.katakwe.com/api/auth/login
- https://www.katakwe.com/providers
- https://www.katakwe.com/parents
- https://www.katakwe.com/overview
- https://www.katakwe.com/earnings
- https://www.katakwe.com/social-responsibility
- https://www.katakwe.com/commitment-to-privacy
- https://www.katakwe.com/terms-and-conditions
- https://www.katakwe.com/privacy-policy
- https://www.katakwe.com/competition-terms-and-conditions
- https://www.katakwe.com/parents
- https://www.katakwe.com/
- https://www.katakwe.com/api/auth/login
- https://www.katakwe.com/providers
- https://www.katakwe.com/parents
- https://www.katakwe.com/overview
- https://www.katakwe.com/earnings
- https://www.katakwe.com/social-responsibility
- https://www.katakwe.com/commitment-to-privacy
- https://www.katakwe.com/terms-and-conditions
- https://www.katakwe.com/privacy-policy
- https://www.katakwe.com/competition-terms-and-conditions
- https://www.katakwe.com/overview
- https://www.katakwe.com/
- https://www.katakwe.com/api/auth/login
- https://www.katakwe.com/providers
- https://www.katakwe.com/parents
- https://www.katakwe.com/overview
- https://www.katakwe.com/earnings
- https://www.katakwe.com/social-responsibility
- https://www.katakwe.com/commitment-to-privacy
- https://www.katakwe.com/terms-and-conditions
- https://www.katakwe.com/privacy-policy
- https://www.katakwe.com/competition-terms-and-conditions
- https://www.katakwe.com/earnings
- https://www.katakwe.com/
- https://www.katakwe.com/api/auth/login
- https://www.katakwe.com/providers
- https://www.katakwe.com/parents
- https://www.katakwe.com/overview
- https://www.katakwe.com/earnings
- https://www.katakwe.com/social-responsibility
- https://www.katakwe.com/commitment-to-privacy
- https://www.katakwe.com/terms-and-conditions
- https://www.katakwe.com/privacy-policy
- https://www.katakwe.com/competition-terms-and-conditions
- https://www.katakwe.com/social-responsibility
- https://www.katakwe.com/
- https://www.katakwe.com/api/auth/login
- https://www.katakwe.com/providers
- https://www.katakwe.com/parents
- https://www.katakwe.com/overview
- https://www.katakwe.com/earnings
- https://www.katakwe.com/social-responsibility
- https://www.katakwe.com/commitment-to-privacy
- https://www.katakwe.com/terms-and-conditions
- https://www.katakwe.com/privacy-policy
- https://www.katakwe.com/competition-terms-and-conditions
- https://www.katakwe.com/commitment-to-privacy
- https://www.katakwe.com/
- https://www.katakwe.com/api/auth/login
- https://www.katakwe.com/providers
- https://www.katakwe.com/parents
- https://www.katakwe.com/overview
- https://www.katakwe.com/earnings
- https://www.katakwe.com/social-responsibility
- https://www.katakwe.com/commitment-to-privacy
- https://www.katakwe.com/terms-and-conditions
- https://www.katakwe.com/privacy-policy
- https://www.katakwe.com/competition-terms-and-conditions
- https://www.katakwe.com/terms-and-conditions
- https://www.katakwe.com/
- https://www.katakwe.com/api/auth/login
- https://www.katakwe.com/providers
- https://www.katakwe.com/parents
- https://www.katakwe.com/overview
- https://www.katakwe.com/earnings
- https://www.katakwe.com/social-responsibility
- https://www.katakwe.com/privacy-policy
- https://www.katakwe.com/commitment-to-privacy
- https://www.katakwe.com/terms-and-conditions
- https://www.katakwe.com/competition-terms-and-conditions
- https://www.katakwe.com/privacy-policy
- https://www.katakwe.com/
- https://www.katakwe.com/api/auth/login
- https://www.katakwe.com/providers
- https://www.katakwe.com/parents
- https://www.katakwe.com/overview
- https://www.katakwe.com/earnings
- https://www.katakwe.com/social-responsibility
- https://www.katakwe.com/commitment-to-privacy
- https://www.katakwe.com/terms-and-conditions
- https://www.katakwe.com/privacy-policy
- https://www.katakwe.com/competition-terms-and-conditions
- https://www.katakwe.com/competition-terms-and-conditions
- https://www.katakwe.com/
- https://www.katakwe.com/api/auth/login
- https://www.katakwe.com/providers
- https://www.katakwe.com/parents
- https://www.katakwe.com/overview
- https://www.katakwe.com/earnings
- https://www.katakwe.com/social-responsibility
- https://www.katakwe.com/commitment-to-privacy
- https://www.katakwe.com/terms-and-conditions
- https://www.katakwe.com/privacy-policy
- https://www.katakwe.com/competition-terms-and-conditions