Implement new Scraper for GitHub Events

[d-Rickyy-b]

Similar to shhgit (repo link) there could be a new parser which clones a repo and checks files with the given analyzers.

For now this is just a random idea with close to no detailled thoughts on how to implement this. There is the GitHub Events API which is also used by shhgit. Maybe also the source code of shhgit can be used to implement some of the code for pastepwn.

Definition of done

1. A new directory called 'github' was created in the scraping directory
2. A new scraper (which is expanding basicscraper) is implemented in the github directory
3. The new scraper works similar to the pastebin scraper and fetches events from the github events API. Currently it seems that it needs to clone the repo before acting on it. You are free to make suggestions how this should work.

[Samyak2]

Would the scraper need to run all analyzers on all files in the cloned repo? Or would it only download the files?

[d-Rickyy-b]

@Samyak2 The scraper would only download the files and put them into a queue similar to the pastebin scraper. Running the analyzers is not the task of the scrapers.

Maybe this architectural illustration can show the inner workings better:

• EDIT: The last box should say 'ActionHandler' and not 'AnalyzerHandler'

The interesting part is indeed what kind of files (and how many) should be inserted into the queue. Maybe starting with files from common programming languages would be a good first step I think. But I think you can come up with a great solution. Downloading would be priority 1. The scanning part can be done later.

[Samyak2]

I understand the flow now. I will try to understand the Pastebin scraper and then start work on this.

[d-Rickyy-b]

I will update the image later. There are a few issues with it...

But the flow is the same. If you need help anywhere, feel free to contact me. I'll be happy to help.