Skip to content

omer-metin/TweetCollector

Repository files navigation

Tweet Collector

Tweet Collection script that runs with selenium. Collect tweets of tags (#apple), stocks ($AAPL) and words (apple)

To collect large amount of tweet without struggling with Twitter, requires twitter account. (A verified one is strongly recommended.)

Usage

```bash
    python tweet_collector.py --options
```

Options:

  • h, --help

  • -k, --searchKey KEY (required) search key will be searched

  • -a, --search_as [tag, stock, word] (default: tag) search method tag -> $, stock -> $ and word is direct word search

  • -s, --start_date START_DATE (required) starting date for search in YYYY-MM-DD format

  • -e, --end_date END_DATE (required) final date for search in YYYY-MM-DD format

  • -l, --lang SHORTHAND_LANG (default: en) language of tweet to be collected. See supported languages

  • -f, --settings_file BOOL (required) (default: True) use settings.json file (ignore setting parameters)

  • -u, --username USERNAME
    username of twitter account

  • -p, --password PASSWORD
    password of twitter account

  • -d, --chromedriver_path CHROMEDRIVER_PATH (default: chromedriver.exe) chromedriver path that is used

  • -t, --thread_count THREAD_COUNT (default: 0) number of thread that is used in program

  • -m, --missing_run_count MISSING_RUN_COUNT (default: 1) re-run number for missings dates

Examples

  • Using settings.json file

    Example settings file
    {
        "username": "*******",
        "password": "*******",
        "chromedriver_path": "chromedriver.exe",
        "thread_count": 0,
        "missing_run_count": 1
    }
    Running code
    python tweet_collector.py -k AAPL -a stock -s 2020-07-28 -e 2020-08-28 -f True
  • Without settings.json file

    Running code
    ```bash
    python tweet_collector.py -k AAPL -a stock -s 2020-07-28 -e 2020-08-28 -f False -u ******* -p *******
    

Requirements

  • Python 3.6+

    written in 3.7.9

  • Selenium

    You can install by using pip

  • chromedriver

    You can get proper version from chromedriver

License

MIT