Twitter search scrapes in default 'latest' order fail with 401 ('non-200 status code')

[JustAnotherArchivist]

Since today, any twitter-search scrapes (i.e. including twitter-user and twitter-hashtags, which are just wrappers around the search) that do not include the --top flag fail. Twitter removed the 'Latest' tab from the search page this afternoon, and since approximately 22:35 UTC, the API seems to fail as well.

[MazenTayseer]

So I have to add top flag for it to work?

[constantin-barbu]

The "Latest" tab seems to be back now on Twitter search pages.

[JustAnotherArchivist]

It's unclear right now whether the functionality is still available at all. If not, it might never get fully solved. You can probably work around it by using --top (e.g. snscrape twitter-search --top 'from:username'), but I'm not sure whether you can get the full feed that way.

[JustAnotherArchivist]

@constantin-barbu They might be doing more changes at the minute. I don't see it. I'm also seeing claims that it is only available when logged in (which would be useless for snscrape).

[constantin-barbu]

You're right, it's only available when logged in :(.

[AhmetcanFR]

Is there a way to log in our access token so we can have access to it?
Would it still be without rate limit?

[JustAnotherArchivist]

@AhmetcanFR No, logging in is not and will not be supported: #270

[AhmetcanFR]

What can we do for now?

[JustAnotherArchivist]

There's currently no known workaround apart from the one I mentioned above.

[MazenTayseer]

@JustAnotherArchivist if I set 'top=False'
Will it work as before?

[JustAnotherArchivist]

@MazenTayseer No, only top = True will work currently.

[Goitsemedi888]

where would this be used top = True???

for i,tweet in enumerate(sntwitter.TwitterSearchScraper('#CRM OR #Salesforce, since:2022-1-1 until:2023-1-10').get_items()):
    if i>250000:
        break
    tweets_list.append([tweet.date, tweet.id, tweet.rawContent, tweet.user.username])
    
# Creating a dataframe from the tweets list above
CRM_tweets_df = pd.DataFrame(tweets_list, columns=['Datetime', 'Tweet Id', 'Text', 'Username'])

[JustAnotherArchivist]

@Goitsemedi888 It's an optional argument to the scraper: TwitterSearchScraper('query', top = True)

[JustAnotherArchivist]

'Top' searches are now broken as well: #647

[JustAnotherArchivist]

Twitter seems to have reverted this change. The 'latest' search scraping works again on the current snscrape version, even though it's still unavailable on the web interface.

[MazenTayseer]

TwitterSearchScraper is still not working for me
UPDATE: I had to change to the latest version for it to work.

[JustAnotherArchivist]

If you (or anyone else) have a problem on the latest snscrape version and no issue exists yet for it, file a complete bug report please.

[MariaPng]

i downloaded the latest version of snscrape but still the error remains,what i have to do?

[JustAnotherArchivist]

@MariaPng You file a complete bug report, as the comment just before yours says.

[Satarupa22-SD]

I am still facing this issue, i have implemented top = True in my code but the error is persisting.