DaimonSkycrawlers is a crawler framework.
- Ruby
- RabbitMQ
- RDB
- PostgreSQL (default)
- MySQL
- SQLite3
Add this line to your application's Gemfile:
gem 'daimon_skycrawlers'
And then execute:
$ bundle
Or install it yourself as:
$ gem install daimon_skycrawlers
-
Create project
$ bundle exec daimon_skycrawlers new mycrawlers $ cd mycrawlers
or
$ daimon_skycrawlers new mycrawlers $ cd mycrawlers
-
Install dependencies
$ bundle install
-
Create database
$ bundle exec rake db:create $ bundle exec rake db:migrate
-
Open new terminal and run crawler/processor
$ daimon_skycrawlers exec crawler # on new terminal $ daimon_skycrawlers exec processor # on new terminal
NOTE: Execute step 5 as soon as possible. Because crawler and processor will stop after 10 seconds by default if their queues are empty.
NOTE: You can change
shutdown_interval
using following code in config/init.rb:DaimonSkycrawlers.configure do |config| config.shutdown_interval = 30 end
-
Enqueue task
$ daimon_skycrawlers enqueue url http:https://example.com/
-
You'll see
It works with 'http:https://example.com'
on your terminal which runs your processor! -
You can re-enqueue task for processor
$ daimon_skycrawlers enqueue response http:https://example.com/
Display It works with 'http:https://example.com'
again on your terminal which runs your processor.
-
Create project
$ bundle exec daimon_skycrawlers new mycrawlers $ cd mycrawlers
or
$ daimon_skycrawlers new mycrawlers $ cd mycrawlers
-
Build docker images
$ docker-compose build
-
Run docker containers
$ docker-compose up -d
-
Run a command on docker containers
$ docker-compose exec <service name> <command>
For example,
$ docker-compose exec mycrawlers-db bash $ docker-compose exec mycrawlers-crawler sh $ docker-compose exec mycrawlers-crawler bundle exec daimon_skycrawlers enqueue url http:https://example.com/ $ docker-compose exec mycrawlers-crawler bundle exec daimon_skycrawlers enqueue response http:https://example.com/
-
Shutdown docker containers
$ docker-compose down $ docker-compose down --rmi all # Remove all related images
After checking out the repo, run bundle install
to install dependencies. Then, run bundle exec rake test
to run the tests. You can also run bundle console
for an interactive prompt that will allow you to experiment.
To install this gem onto your local machine, run bundle exec rake install
. To release a new version, update the version number in version.rb
, and then run bundle exec rake release
, which will create a git tag for the version, push git commits and tags, and push the .gem
file to rubygems.org.
Bug reports and pull requests are welcome on GitHub at https://github.com/bm-sms/daimon_skycrawlers.
The gem is available as open source under the terms of the MIT License.