This project is no longer maintained. Please see Mechanize for an alternative.
Arachnid is a fast, soon to be multi-threading capable web crawler for Crystal. It recenty underwent a full rewrite for Crystal 0.35.1, so see the documentation below for updated usage instructions.
-
Add the dependency to your
shard.yml
:dependencies: arachnid: github: watzon/arachnid
-
Run
shards install
First, of course, you need to require arachnid in your project:
require "arachnid"
Agent
is the class that does all the heavy lifting and will be the main one you interact with. To create a new Agent
, use Agent.new
.
agent = Arachnid::Agent.new
The initialize method takes a bunch of optional parameters:
You can, if you wish, supply your own HTTP::Client
instance to the Agent
. This can be useful if you want to use a proxy, provided the proxy client extends HTTP::Client
.
The user agent to be added to every request header. You can override this on a per-host basis with either :host_headers
or :default_headers
.
The default headers to be used in every request.
Headers to be applied on a per-host basis. This is a hash String (host name) => HTTP::Headers
.
The Arachnid::Queue
instance to use for storing links waiting to be processed. The default is a MemoryQueue
(which is the only one for now), but you can easily implement your own Queue
using whatever you want as a backend.
Whether or not to stop running when the queue is empty. This is true by default. If it's made false, the loop will continue when the queue empties, so be sure you have a way to keep adding items to the queue.
Whether or not to follow redirects (add them to the queue).
There are four ways to start your Agent once it's been created. Here are some examples:
#start_at
starts the Agent running on a particular URL. It adds a single URL to the queue and starts there.
agent.start_at("https://crystal-lang.org") do
# ...
end
#site
starts the agent running at the given URL and adds a rule that keeps the agent restricted to the given site. This allows the agent to scan the given domain and any subdomains. For instance:
agent.site("https://crystal-lang.org") do
# ...
end
The above will match crystal-lang.org
and forum.crystal-lang.org
, but not github.com/crystal-lang
or any other site not within the *.crystal-lang.org
space.
#host
is like site, but with the added restriction of just remaining on the current domain path. Subdomains are not included.
agent.host("crystal-lang.org") do
# ...
end
Provided you already have URIs in the queue ready to be scanned, you can also just use #start
to start the Agent running.
agent.enqueue("https://crystal-lang.org")
agent.enqueue("https://kemalcr.com")
agent.start
URI's can be filtered before being enqueued. There are two kinds of filters, accept and reject. Accept filters can be used to ensure that a URI matches before being enqueued. Reject filters do the opposite, keeping URIs from being enqueued if they do match.
For instance:
# This will filter out all sites where the host is not "crystal-lang.org"
agent.accept_filter { |uri| uri.host == "crystal-lang.org" }
If you want to ignore certain parts of the above filter:
# This will ignore paths starting with "/api"
agent.reject_filter { |uri| uri.path.to_s.starts_with?("/api") }
The #site
and #host
methods add a default accept filter in order to keep things in the given site or host.
All the above is useless if you can't do anything with the scanned resources, which is why we have the Resource
class. Every scanned resource is converted into a Resource
(or subclass) based on the content type. For instance, text/html
becomes a Resource::HTML
which is parsed using kostya/myhtml for extra speed.
Each resource has an associated Agent#on_
method so you can do something when one of those resources is scanned:
agent.on_html do |page|
puts typeof(page)
# => Arachnid::Resource::HTML
puts page.title
# => The Title of the Page
end
Currently we have:
#on_html
#on_image
#on_script
#on_stylesheet
#on_xml
There is also #on_resource
which is called for every resource, including ones that don't match the above types. Resources all include, at minimum the URI at which the resource was found, and the response (HTTP::Client::Response
) instance.
- Fork it (https://github.com/watzon/arachnid/fork)
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create a new Pull Request
- your-name-here - creator and maintainer