A little (multi-threaded) API wrapper for PullPush.io - the 3rd party replacement API for Reddit. After the 2023 Reddit API controversy, PushShift.io(and also wrappers such as PSAW and PMAW) is now only available to reddit admins and Reddit PRAW is honestly useless when trying to get a lots of data and data from a specific timeframe. PullPush.io thankfully solves this issue and this is the wrapper for that said API. For more info on the API(TOS, Forum, Docs, etc.) go to PullPush.io.

BAScraper(Blue Archive Scraper) was initially made and used for the 2023 recap/wrap up of r/BlueArchive hence the name. It's pretty basic but planning to add some more features as it goes. It uses multithreading to make requests to the PullPush.io endpoint and returns the result as a JSON(dict) object.

currently it can:

• get submissions/comments from a certain subreddit in supported order/sorting methods specified in the PullPush.io API docs
• get comments under the retrieved submissions
• can get all the submissions based on the number of submissions or in a certain timeframe you specify
• can recover(if possible) deleted/removed submission/comments from the returned result

I also have a tool that can organize, clean, and analyze reddit submission and comment data, which I am planning to release it with this or separately after some polishing.

Also, please ask the PullPush.io owner before making large amounts or request and also respect cool-down times. It stresses the server and can cause inconvenience for everyone.

⚠️ As of Feb. 2024, PullPush API implemented ratelimiting!
soft limit will occur after 15 req/min and hard limit after 30 req/min. There's also a long-term (hard) limit of 1000 req/hr.

Recommended request pacing:

• to prevent soft-limit: 4 sec sleep per request
• to prevent hard-limit: 2 sec sleep per request
• for 1000+ requests: 3.6 ~ 4 sec sleep per request

rate limiting will automatically pace your reqeust's response time to meet the following hard limits. Hence making the pace_mode parameter kinda useless (it was made before proper API sice pacing was made). Following the pacing time above is recommended.

you can install the package via pip

pip install BAScraper

Also, python 3.10+ is recommended (3.8 works too)

Example usage

from BAScraper import Pushpull
import json
from datetime import datetime

pp = Pushpull(sleepsec=2, threads=2)
result = pp.get_submissions(after=datetime(2023, 12, 1), before=datetime(2024, 1, 1),
                            subreddit='bluearchive', sort='desc')

# save result as JSON
with open("example.json", "w", encoding='utf-8') as outfile:
    json.dump(result, outfile, indent=4)

all parameters are optional

all parameters are optional returns a dict object

the PushPull.get_submissions & PushPull.get_comments each returns a dict object that is indexed based on its unique ID. it is so that it can be easily saved into a JSON format, also it is sorted in the order you specified when scraping (the sort parameter). So the general structure looks like this (regardless of it being a submission or a comment):

{
  "21jh54" : {
    "approved_at_utc": null,
    "subreddit": "Cars",
    "selftext": "",
    "author_fullname": "t2_culcgvve",
    "saved": false,
    "mod_reason_title": null,
    "gilded": 0,
    "clicked": false,
    "title": "something something",
    ...
  },
  "54jp5i" : {
    "approved_at_utc": null,
    "subreddit": "Cars",
    "selftext": "",
    "author_fullname": "t2_kdbbiwo",
    "saved": false,
    "mod_reason_title": null,
    "gilded": 0,
    "clicked": false,
    "title": "something something",
    ...
  }, 
  ...
}

if the get_comments parameter is set to True the returned result would look like this (for submissions)

{
  "21jh54": {
    "approved_at_utc": null,
    "subreddit": "Cars",
    "selftext": "",
    "author_fullname": "t2_culcgvve",
    "saved": false,
    "mod_reason_title": null,
    "gilded": 0,
    "clicked": false,
    "title": "something something",
    "comments": [
      {
        ...
        info related to comments
        ...      
      },
      ...
    ],
    ...
  },
  ...
}