Test URL extraction locally:

chmod +x url_parsing/crawl_mapper.rb
chmod +x url_parsing/crawl_reducer.rb

cat sample_input | ./url_parsing/crawl_mapper.rb | ./url_parsing/crawl_reducer.rb > result

• Input location: s3:https://aws-publicdatasets/common-crawl/parse-output/segment/1346823846150/metadata-*
• Output location: s3:https://../output
• Mapper: s3:https://.../crawl_mapper.rb
• Reducer: s3:https://.../crawl_reducer.rb
• Extra args: -inputformat SequenceFileAsTextInputFormat
• Custom Bootstrap action: s3:https://.../setup.sh

Metadata file is about 16MB each. To get the list of valid segments:

aws get aws-publicdatasets/common-crawl/parse-output/valid_segments.txt

Information about https://www.neti.ee/cgi-bin/serverid:

{
  "url": "https://www.neti.ee/cgi-bin/serverid",
  "arcFileParition": 584,
  "arcSourceSegmentId": 1346823846150,
  "arcFileDate": 1346832062332,
  "compressedSize": 524596,
  "arcFileOffset": 8186060
}

chmod +x page_rank/page_rank_mapper.rb
chmod +x page_rank/page_rank_reducer.rb

cat sample_input | ./page_rank/page_rank_mapper.rb | sort | ./page_rank/page_rank_reducer.rb > result