It's a distributed web crawler that uses nats.io to coordinate work, written in Go.
- CouchDB 1.x (tested on 1.6.1)
- gnatsd
Build the two workers
go build -tags=fetcherExec -o fetcher fetcher.go && \
go build -tags=extractorExec -o extractor extractor.go
- Setup couchdb with at least one admin user, you can follow the instructions here
- create a file
.couchdb.json
and place it in your$HOME
directory
Sample .couchdb.json
```
{ "user": "user-here", "password": "super-secret-password", "url": "https://localhost:5984/owl-crawler" }
```
- create a file
.gnatsd.json
and place it in your$HOME
directory
Sample .gnatsd.json
```
{
"URL": "nats:https://owlcrawler:[email protected]:4222"
}
```
- Start gnatsd with a user and password (use a config file, but for a quick test you can pass parameters):
~/gnatsd --user owlcrawler --pass natsd_password
./extractor -logtostderr=true -v=3
./fetcher -logtostderr=true -v=3
cd webapp
go build && ./webapp -alsologtostderr=true
cd webapp
grunt serve