Pagination using offset parameter always return the same statuses

[vvvlll93]

Hi,

I'm having trouble with the pagination. I followed the previous example but each time i'm getting the same result.

Here the code i use :

ArrayList<String> html_statuses_list = new ArrayList<>();
boolean keepSearching = true;
int offset = 20;

while (keepSearching) {
    try {
        ArrayList<String> html_statuses_list_offset = new ArrayList<>();
        Search search_results = mastodonClient.search().searchContent(keyword, null, false, false, false, null, null, null, null, offset).execute();
        search_results.getStatuses().forEach(status -> html_statuses_list_offset.add(status.getContent()));

        if (html_statuses_list_offset.size()<20) {
            keepSearching = false;
        } else {
            html_statuses_list.addAll(html_statuses_list_offset);
            offset += 20;
        }
    } catch (BigBoneRequestException e) {
        logger.info(e.toString());
        keepSearching = false;
    }
}

return html_statuses_list;

When i check the result each time, the offset parameters increase (20, 40, ...) but the results always stay the same.
I tried with the keyword "Paris", and also i have the latest version in my pom.xml : 2.0.0-SNAPSHOT.

Do you have any idea why this happens ?

Thanks.

[MastodonContentMover]

I'm just dipping into this quickly and it looks like you're doing something slightly different to what I was doing (search rather than timelines, for starters), so apologies if this isn't useful, but this pagination code worked for me:

      List<Status> results = new ArrayList();
      Range currentRange = new Range(null, null, STATUSES_PER_PAGE);
      int lastFetched = -1;

      System.out.print("Fetching pages ");

      /**
      *   Note: The last time this runs it will fetch zero new statuses, because when it
      *   fetches the last batch there will still be more than zero so it will run one 
      *   more time.
      */
      do {    
         Pageable<Status> timeline = authenticatedClient.accounts().getStatuses(accountId, false, false, false, currentRange).execute();
         List<Status> page = timeline.getPart();
         results.addAll(page);
         lastFetched = page.size();
         currentRange = timeline.nextRange(STATUSES_PER_PAGE);
         if (Mover.showDebug()) {  System.out.println(Mover.getDebugPrefix() + "Fetched " + results.size() + " statuses in total");  }
         if (Mover.showDebug()) {  System.out.println(Mover.getDebugPrefix() + "Fetched " + page.size() + " statuses this time \n");  }
         if (page.size() > 0) {
            Status last = (Status)page.get(page.size() - 1); 
            if (Mover.showDebug()) {  System.out.println(Mover.getDebugPrefix() + "Last status fetched id: " + last.getId() + " content: " + last.getContent() + " \n");  }
         }
         System.out.print(".");
         try {
            TimeUnit.SECONDS.sleep(API_THROTTLE_STATUS_PAGE_SECONDS + parameterExtraThrottleSeconds);  
         }
         catch (InterruptedException ie) {   }
      } while (lastFetched > 0);
      if (Mover.showDebug()) {  System.out.println(Mover.getDebugPrefix() + "Done - fetched " + results.size() + " statuses \n");  }

Think I had an issue at first as well, until I started using Ranges. Am running with snapshot bigbone-2.0.0-20230818.202222-21 (I need to update).

Also: STATUSES_PER_PAGE is just an int:
private static final int STATUSES_PER_PAGE = 40; // Current maximum

[PattaFeuFeu]

@MastodonContentMover The searchContent function currently doesn’t use the Range parameter, so that won’t work. See #341 (comment) for details. 😊

[bocops]

I just checked, and it seems to be the case that the offset parameter is used only if a specific type is given. If search results are restricted to either statuses, accounts OR hashtags, pagination via the offset parameter does work as expected.

For what it's worth, using a type currently does not work due to a bug on our end. Will create an issue for that part.

[bocops]

The fact that offset can not be used without specifying a type seems to be by design according to Mastodon's search_service.rb, which contains this line:

@offset = options[:type].blank? ? 0 : options[:offset].to_i

[bocops]

See mastodon/documentation#1336.

[vvvlll93]

I tried to replace the second parameters by SearchMethods.SearchType.STATUSES, but right now my search is simply 3 empty arraylists. Does anyone achieve gathering statuses using the pagination feature ?

[bocops]

@vvvlll93 This will start working with the next published snapshot - a bug preventing pagination for single-type searches was just fixed.