• Use ops/remote_sync.sh to sync the full repo to the remote server
• For a bare-bones Ubuntu 16.04 server (EC2 tested), use ops/fresh_machine.sh to configure the server as a graphical desktop with VNC connectivity

• Use install.sh to install the system requirements and prep the server for running the repo code

Reads a spreadsheet containing first/last names, State and city values, then scrapes the configured (if supported) website for contact info matching those values as search arguments.

Supports distributed processing by defining what hostname is assigned which rows in a hostname column in the spreadsheet.

Unable to automatically clear Captchas! As such, babysitting is required and remote machines can not be run headless for sites with Captchas.

Recommend to sort the input sheet by owner last-name/first-name to prevent duplicate scrapes beforehand.

~/.virtualenvs/cw/bin/python3 /home/ubuntu/muad-dweeb/cw/scrape/spreadsheet_scrape.py -h

~/.virtualenvs/cw/bin/python3 /home/ubuntu/muad-dweeb/cw/scrape/spreadsheet_scrape.py --config whatcom_duplexes_350_plus --limit-minutes 300 --email-report --site ic --environment ec2

Utility for joining spreadsheets based on a configured primary key.

python ~/muad-dweeb/cw/manage_sheets.py --master-config master_test --child-config child_test --operation merge