Commits

Commits on May 13, 2015

Update project.clj & README for pushing to clojars
naiquevin
committedMay 13, 2015
Add user-agent to config
naiquevin
committedMay 13, 2015

Commits on May 10, 2015

Update README
naiquevin
committedMay 10, 2015
Add :default no-op rule if not present
naiquevin
committedMay 10, 2015
Update and fix README
naiquevin
committedMay 10, 2015
Update README
naiquevin
committedMay 10, 2015
Add support for custom follow functions
naiquevin
committedMay 10, 2015

Commits on May 9, 2015

Update docstring
naiquevin
committedMay 9, 2015
Update tests
naiquevin
committedMay 9, 2015
Refactor code for serving mock site
naiquevin
committedMay 9, 2015

Commits on Apr 19, 2015

Use the timbre lib for logging
naiquevin
committedApr 19, 2015
Use humane-test-output for clojure.test
naiquevin
committedApr 19, 2015

Commits on Mar 29, 2015

Fix spider rules in dmoz_lisp example
naiquevin
committedMar 29, 2015
Refactor pipeline code
naiquevin
committedMar 29, 2015
Fix throttling of http requests
naiquevin
committedMar 29, 2015

Commits on Mar 15, 2015

Improve docstring for spider
naiquevin
committedMar 15, 2015
Gracefully shutdown the spider
naiquevin
committedMar 15, 2015
Add docstrings
naiquevin
committedMar 15, 2015

Commits on Mar 8, 2015

Add example spider (dmoz)
naiquevin
committedMar 8, 2015
Throttle/Rate-limit http requests
naiquevin
committedMar 8, 2015

Commits on Mar 6, 2015

Handle anomalies in hrefs extracted from sites
naiquevin
committedMar 6, 2015
Fix project.clj
naiquevin
committedMar 6, 2015

Commits on Feb 22, 2015

Update README
naiquevin
committedFeb 22, 2015
Remove old example spider code and testsite
naiquevin
committedFeb 22, 2015
Add test-site server and tests for spider using it
naiquevin
committedFeb 22, 2015
Handle nil result from the scraper function
naiquevin
committedFeb 22, 2015

Commits on Feb 11, 2015

Use http-kit instead of clj-http
naiquevin
committedFeb 11, 2015

Commits on Feb 4, 2015

Update the example in attercop.core
naiquevin
committedFeb 4, 2015
Skip crawling already visited urls again
naiquevin
committedFeb 4, 2015

Commits on Feb 2, 2015

Update example usage in README
naiquevin
committedFeb 2, 2015
Add support for :follow in rules
naiquevin
committedFeb 2, 2015

Commits on Feb 1, 2015

Update README with more detailed usage
naiquevin
committedFeb 1, 2015
Use thread instead of go block for IO based calls
naiquevin
committedFeb 1, 2015
Add support for allowed-domains
naiquevin
committedFeb 1, 2015

Commits on Jan 14, 2015

Update README
naiquevin
committedJan 14, 2015