Deterministic pagination

[dillonstreator]

Most resources do not have a field to sort against that provides a consistent and deterministic paging. For instance, I'm trying to page through CareTeam's with a particular tag and the end result of the paging does not result in the full set of CareTeam's that exist in the system. Keep in mind that the resource set behind the base filter is not actively changing and yet the pagination does not result in all entries.

const seenCareTeamIds = new Set<string>();
for await (const resources of medplum.searchResourcePages('CareTeam', {
_tag: '...|...',
_count: 1000,
_sort: 'name',
})) {
  for (const resource of resources) {
    if (seenCareTeamIds.has(resource.id!)) {
      console.log('Repeated care team', resource.id);
      continue;
    }
    seenCareTeamIds.add(resource.id!);

    // ...
  }
}

These CareTeam's do not have a date set so that cannot be sorted against. The name's are all the same so this is understandably a bad field to sort against as well.

After updating the script to sort against patient, the results were MUCH better but still resulted in several duplicate records across pages. The fact that there is not a consistent field shared across all resources that can be deterministically sorted against feels like a gap in the spec.

I don't see this covered by the FHIR spec but it would be really valuable to have a shared field across all resources that is either an internal sortable ULID or a 'created at' field. This field could be added by default to the end of whatever sort is provided by the calling client to give more deterministic paging by default. 'created at' could be backfilled based on the first _history entry for the matching resource. Not sure if ULIDs can be easily backfilled but if a seed date can be provided, the derived 'created at' could be used.