Twitter searches fail with `blocked (403)` #846

SapiensSuwan · 2023-04-20T22:21:07Z

OS: macOS Venture 13.3.1

Python Version: Python 3.8.3

snscrape Version: snscrape 0.6.2.20230320

snscrape module: snscrape.modules.twitter -> TwitterSearchScraper

Bug log:

Error retrieving https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=bitcoin+since%3A2021-07-01+until%3A2022-12-31&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe: blocked (403)
4 requests to https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=bitcoin+since%3A2021-07-01+until%3A2022-12-31&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe failed, giving up.
Errors: blocked (403), blocked (403), blocked (403), blocked (403)
Traceback (most recent call last):
  File "bitcoin21-22-01.py", line 11, in <module>
    for i, tweet in enumerate(sntwitter.TwitterSearchScraper('bitcoin since:2021-07-01 until:2022-12-31').get_items()):
  File "/Users/opt/anaconda3/lib/python3.8/site-packages/snscrape/modules/twitter.py", line 1661, in get_items
    for obj in self._iter_api_data('https://api.twitter.com/2/search/adaptive.json', _TwitterAPIType.V2, params, paginationParams, cursor = self._cursor):
  File "/Users/opt/anaconda3/lib/python3.8/site-packages/snscrape/modules/twitter.py", line 761, in _iter_api_data
    obj = self._get_api_data(endpoint, apiType, reqParams)
  File "/Users/opt/anaconda3/lib/python3.8/site-packages/snscrape/modules/twitter.py", line 727, in _get_api_data
    r = self._get(endpoint, params = params, headers = self._apiHeaders, responseOkCallback = self._check_api_response)
  File "/Users/opt/anaconda3/lib/python3.8/site-packages/snscrape/base.py", line 251, in _get
    return self._request('GET', *args, **kwargs)
  File "/Users/opt/anaconda3/lib/python3.8/site-packages/snscrape/base.py", line 247, in _request
    raise ScraperException(msg)
snscrape.base.ScraperException: 4 requests to https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=bitcoin+since%3A2021-07-01+until%3A2022-12-31&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe failed, giving up.

Original Code:

import snscrape.modules.twitter as sntwitter
import pandas as pd

# Setting variables to be used below
# maxTweets = 5000

# Creating list to append tweet data to
tweets_list1 = []

# Using TwitterSearchScraper to scrape data and append tweets to list
for i, tweet in enumerate(sntwitter.TwitterSearchScraper('bitcoin since:2021-07-01 until:2022-12-31').get_items()):
    # if i>maxTweets:
    #     break

    tweets_list1.append([tweet.date, tweet.id, tweet.rawContent, tweet.user.username])

    if (i + 1) % 5000 == 0:

        # Creating a dataframe from the tweets list above
        tweet_df1 = pd.DataFrame(tweets_list1, columns=['Datetime', 'Tweet Id', 'Text', 'Username'])
        print(tweet_df1)

        # Export dataframe into a csv
        tweet_df1.to_csv(f'bitcoin-{(i+1)/5000}.csv', sep=',', index=False)
        tweets_list1 = []

Note: The code was working a couple days ago and now it gives me this error again. I did try to search bitcoin as keyword in twitter and it worked fine. Sorry in advance if I made any duplicated issue report. I haven't seen a 403 error here.

The text was updated successfully, but these errors were encountered:

JustAnotherArchivist · 2023-04-20T22:39:27Z

Yeah, the exact error is indeed new, but this is effectively #834.
However, the 403 affects all searches, not only the 'latest' one, so I'll leave it open anyway.

emmanuelva · 2023-04-20T22:41:44Z

The Search page from twitter seems to be gone

JustAnotherArchivist · 2023-04-20T22:53:29Z

There is no need to repeat 'me too!!1!'. Yes, everyone is affected at the moment.

Wouze · 2023-04-20T23:07:49Z

The Search page from twitter seems to be gone

yes for some reason
I've seen tweets from elon about how twitter data is getting ripped of
idk if there is any relation
https://twitter.com/elonmusk/status/1648793947806871556?s=20

Wouze · 2023-04-20T23:49:37Z

so basically twitter search is no longer available, only available to logged in users. Is there a way of doing the same searches while being logged in? maybe by providing tokens/cookies/email&pass?

JustAnotherArchivist · 2023-04-21T00:03:19Z

@Wouze #270

AlekseiShkurin · 2023-04-21T00:09:54Z

Are there any solutions being explored at the moment? Or any temp fixes anyone discovered?

cunninghamx · 2023-04-21T00:49:26Z

:f: in chat for twitter search. Search now redirects to login. @JustAnotherArchivist i hope we as a community find another way in, but thank you so much for the hard work you have put into this

Fa5g · 2023-04-21T02:15:54Z

I think it would be a really good idea to fix snscrape to work once the user logs in and improve it to post auto messages, add follow/unfollow features, etc. taking care of the limits of Twitter, in this way the tool would not be forgotten.

TheTechRobo · 2023-04-21T02:16:49Z

@Fa5g As several people have mentioned in this thread, snscrape can't and won't support authentication, although you are welcome to make a fork of it: #270

Hesko123 · 2023-04-21T05:13:09Z

It seems that it's now coming to an end. Fck elon musk ngl, fcked my journey.

https://nitter.net/search?f=tweets&q=testing&since=&until=&near=

Nitter seems working for searches ! (maybe a hope for a future workaround)

JustAnotherArchivist · 2023-05-10T03:35:12Z

The eagle has landed.
User searches (that is, searching for users by keyword, not searching for a user's tweets) are currently not supported as they use a completely different API.
List searches (including the twitter-list-posts alias) are not possible with the new API.
For any other problems, please file a complete bug report.

engrCoy · 2023-05-10T06:40:31Z

The eagle has landed. User searches (that is, searching for users by keyword, not searching for a user's tweets) are currently not supported as they use a completely different API. List searches (including the twitter-list-posts alias) are not possible with the new API. For any other problems, please file a complete bug report.

Sorry just wanted to as a question. I might just be confused, but what I understand is that we're not able to search by keyword with the new API right? Is there a different way to extract from the API or Twitter isn't allowing any of the search functions?

JustAnotherArchivist · 2023-05-10T06:49:00Z

@engrCoy Yeah, you misread that. Searching for tweets works now except for a few unsupported filters like list:...; there isn't currently a list of what works and doesn't, but keywords are no problem. It's only the search for users (the 'Users' tab on the web search) that's broken because the new API endpoint does not support it.

One general word of caution: the new API is even more unreliable than the old one; you will regularly see scrapes terminate early for no apparent reason.

engrCoy · 2023-05-10T06:57:19Z

@JustAnotherArchivist I appreciate the clarification on that, and also the precaution about the unreliable output due to an abrupt termination.

As for the use of keywords I tried using that on a sample code that I made(please refer to the code attached below) but I think it's still blocked or something. Oh, please let me know if this is a separate problem that I need a bug report. Thanks in advance, and I love your work!

import snscrape.modules.twitter as sntwitter
import pandas as pd

tweets = []

for i,tweet in enumerate(sntwitter.TwitterSearchScraper(query = "#opportunity").get_items()):
    if i>100:
        break
    tweets.append([tweet.user.username, tweet.date, tweet.likeCount, tweet.retweetCount, tweet.replyCount, tweet.lang,
                   tweet.sourceLabel, tweet.content])

JustAnotherArchivist · 2023-05-10T06:59:32Z

That search works fine for me with the latest dev version. If that's what you're using, yes, please file a complete bug report.

engrCoy · 2023-05-10T07:01:06Z

Will do, thank you so much!

anchitshrivastava · 2023-05-10T07:17:54Z

Can you elaborate a bit about the Dev version?
How do I install it?

That search works fine for me with the latest dev version. If that's what you're using, yes, please file a complete bug report.

defaultsettings · 2023-05-10T07:19:07Z

Can you elaborate a bit about the Dev version? How do I install it?

That search works fine for me with the latest dev version. If that's what you're using, yes, please file a complete bug report.

you can see following post - https://stackoverflow.com/questions/19042389/conda-installing-upgrading-directly-from-github

JustAnotherArchivist · 2023-05-10T07:19:35Z

@anchitshrivastava It's explained in the README.

ihabpalamino · 2023-05-10T07:42:58Z

helloo i am sorry but i am using snscrape version 0.6.2.20230320 and still does not work for me the search should i do an update?

oguzhari · 2023-05-10T11:47:55Z

helloo i am sorry but i am using snscrape version 0.6.2.20230320 and still does not work for me the search should i do an update?

Use git version.

pip3 install git+https://github.com/JustAnotherArchivist/snscrape.git

moral-drip · 2023-05-10T11:50:38Z

helloo i am sorry but i am using snscrape version 0.6.2.20230320 and still does not work for me the search should i do an update?

Use git version.

pip3 install git+https://github.com/JustAnotherArchivist/snscrape.git

That worked for me! Issue resolved.

ihabpalamino · 2023-05-10T11:55:27Z

thank you so much guys it works for me now @JustAnotherArchivist Thank you for your work

JustAnotherArchivist · 2023-05-10T14:17:39Z

As already mentioned, the fix is currently only available in the development version. A new release will be made soon if it turns out there are not too many sharp edges. There are also a few unrelated things I want to implement/fix first.

To upgrade your existing installation to the latest development version until then, use:

pip3 install --upgrade git+https://github.com/JustAnotherArchivist/snscrape.git

I will lock this issue now since the bug is fixed. Please file a complete bug report if you still experience problems not described above.

JustAnotherArchivist changed the title ~~sntwitter - Blocked(403) error~~ Twitter searches fail with blocked (403) Apr 20, 2023

JustAnotherArchivist added bug Something isn't working module:twitter upstream labels Apr 20, 2023

This comment was marked as spam.

Sign in to view

JustAnotherArchivist mentioned this issue Apr 21, 2023

Twitter 'latest' search fails with non-200 status code (401) #834

Closed

This comment was marked as off-topic.

Sign in to view

This comment was marked as spam.

Sign in to view

This comment was marked as off-topic.

Sign in to view

Wouze mentioned this issue Apr 21, 2023

Error 403, ScraperException: 4 requests #848

Closed

This comment was marked as off-topic.

Sign in to view

This comment was marked as resolved.

Sign in to view

JustAnotherArchivist closed this as completed in 9c7b888 May 10, 2023

This comment was marked as off-topic.

Sign in to view

oguzhari mentioned this issue May 10, 2023

Twitter searches fail with blocked (403) #883

Closed

This comment was marked as resolved.

Sign in to view

Repository owner locked as resolved and limited conversation to collaborators May 10, 2023

JustAnotherArchivist unpinned this issue Jun 30, 2023

Twitter searches fail with blocked (403) #846

Twitter searches fail with blocked (403) #846

Comments

SapiensSuwan commented Apr 20, 2023 • edited

JustAnotherArchivist commented Apr 20, 2023 • edited

emmanuelva commented Apr 20, 2023

This comment was marked as spam.

JustAnotherArchivist commented Apr 20, 2023

Wouze commented Apr 20, 2023

Wouze commented Apr 20, 2023 • edited

JustAnotherArchivist commented Apr 21, 2023

AlekseiShkurin commented Apr 21, 2023

cunninghamx commented Apr 21, 2023

Fa5g commented Apr 21, 2023

TheTechRobo commented Apr 21, 2023

This comment was marked as off-topic.

This comment was marked as off-topic.

Hesko123 commented Apr 21, 2023 • edited

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as spam.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as resolved.

JustAnotherArchivist commented May 10, 2023 • edited

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

engrCoy commented May 10, 2023

JustAnotherArchivist commented May 10, 2023

engrCoy commented May 10, 2023 • edited

JustAnotherArchivist commented May 10, 2023

engrCoy commented May 10, 2023

anchitshrivastava commented May 10, 2023

defaultsettings commented May 10, 2023

JustAnotherArchivist commented May 10, 2023

ihabpalamino commented May 10, 2023

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

oguzhari commented May 10, 2023

moral-drip commented May 10, 2023

ihabpalamino commented May 10, 2023

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

JustAnotherArchivist commented May 10, 2023

Twitter searches fail with `blocked (403)` #846

Twitter searches fail with `blocked (403)` #846

SapiensSuwan commented Apr 20, 2023 •

edited

JustAnotherArchivist commented Apr 20, 2023 •

edited

Wouze commented Apr 20, 2023 •

edited

Hesko123 commented Apr 21, 2023 •

edited

JustAnotherArchivist commented May 10, 2023 •

edited

engrCoy commented May 10, 2023 •

edited