My Name - Summer 2017

To build, run make.

To clean up, run make clean.

The TSE engine is composed of three parts. Crawler.c, indexer.c, and querier.c.. Crawler is is implemented like a breath first search program, but instead of popping in an ordered way out of a queue, it is random out of a bag. The program saves each unique webpage it comes across in a folder, with its URL, depth found, and html in a file.

To test cralwer, cd into the crawler directory. Run crawlertest.sh.

The CS50 playground is in https://old-www.cs.dartmouth.edu/~cs50/data/tse