Not all indexed objects are checked against the database in case index/database are out of sync

[thomaslow]

There is a code segment that checks whether already indexed objects are still available in the database, and if not, are removed from the index. This may happen only in rare scenarios, e.g., when manually changing the database, or when restoring database/index backups that were not created at the exact same time.

Unfortunately, the implementation only works for a limited amount of objects, because it uses a search query that can only ever retrieve max_result_window (ElasticSearch parameter, usually 10.000) many objects. If the index contains more objects, this check will fail.

Instead, the code should use some method that actually allows to iterate over all indexed objects (independent of some limit).

[matthias-ronge]

This situation happens when there is an exception during process deletion, and it leaves an index with too many objects, and the only work-around is to delete and rebuild the whole index. It doesn’t happen often, but if it happens, for an instance with thousands of processes, it can mean a day of downtime, which is very bad news for a digitization centre. Therefore I add the blocking bug because this is something we should fix really, really soon.