Skip to content

Store tweets from a Twitter search results, using browser scraping

License

Notifications You must be signed in to change notification settings

MichaelCurrin/scroll-and-scrape

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

34 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Scroll and Scrape

Store tweets from a Twitter search results, using browser scraping

GitHub tag License

Made with Python Made for Twitter dependency - selenium dependency - beautifulsoup4

Purpose

This application uses Python and the browser (controlled through Selenium) to load a page and scrape the contents and save the data. The approach here loads the DOM (not possible when using plain HTTP request) and also adds in waiting and scrolling logic so that dynamic elements can be pulled in.

This project is aimed at scraping Tweets on a search, where scrolling is needed to load more. It could be used for a user timeline though.

Requirements

Clone

$ git clone [email protected]:MichaelCurrin/scroll-and-scrape.git
$ cd scroll-and-scrape

Installation

Install Python 3.

Create a virtual environment and activate it.

Install project packages:

$ pip install -r requirements.txt
$ pip install -r requirements-dev.txt

Usage

$ cd scrollscrape
$ python main.py

Background

This is a simple Python 3 application, based on existing scripts which are included in the research directory. See also gist.

The goal is to get all the Twitter tweets for a search query, going back as far as possible.

Using the Twitter API is restrictive - it only gives a week worth of data. Note that to keep this application simple, only the tweet ID needs to be stored and none of the tweet or author data. Since, once you have a tweet ID no matter how old, you can use the Twitter API to get all tweet and author data for a given tweet ID.

Future development

So far the main script works okay, but it not getting all known tweets for a particular known so it needs improvement. The format can be improved - for now it is printing to stdout which can be redirected to a text file.

Also, this could be improved for efficiency using a headless browser. It could also use a more modern library such as requests-html.

License

Released under MIT by @MichaelCurrin.