Experimental GitHub project analysis based on GHTorrent.

The goal of this project is to explore the use of GHTorrent to understand better the life of Open Source software projects.

It is in the initial conception phase so probably only useful to exchange ideas and early prototypes.

Don't hesitate to open GitHub issues if you have any kind of problem or you want to feedback or exchange thoughts.

It is needed a machine with 16GB of RAM, 4 cores and 1TB of hard disk to execute all without issues.

You need also a running Elasticsearch and Kibana.

And you will need around 12h of compute time (<1h human time) to execute all processes.

The first step is to download GHTorrent data. The testing has been done with: mysql-2018-09-01.tar.gz.

Now, there are two analysis. The first open loading the projects table and the second one loading projects, users and commits.

To load the GitHub projects data in Elasticsearch:

• Import in MySQL the projects table using ght-restore-mysql-projects
• Import in Elasticsearch the projects data using ght_projects2es.py

ght_projects2es.py -e <elastic_url> -i ghtprojects --db-name ghtorrent_projects

• Use Kibana to visualize the projects data and start analyzing it

You will need around 4h to complete the above steps.

The language detection is based on https://github.com/github/linguist

CSV data

Compare with:

• PYPL PopularitY of Programming Language
• TIOBE Index
• GitHub Octoverse

• Import in MySQL the commits, projects and users tables: it has been tested teh loading of commits only for 2018 (220MM). For doing that:

grep '"2018-' commits.csv > commits-2018.csv

You must load this commits-2018.csv 22GB file instead of the commits.csv which is (100GB).

• Import in Elasticsearch the projects data using ght_commits2es.py

ght_commits2es.py -e <elastic_url> -i ghtcommits --db-name ghtorrent_commits

• Use Kibana to visualize the projects data and start analyzing it

You will need around 12h to complete the above steps.