Included resources disappear from search results until $reindex

[dyoung-work]

Hi maintainers! I'm having an issue with resources not showing up in searches that make use of includes. To be clear, these are searches that were returning the included resources, but stopped doing so (i.e. I refresh my browser tab with the search in it and the resources disappear). For example, [tenant_id]/EnrollmentRequest?_include=* will suddenly stop returning all Patient resources (or Coverage, or any other included resource type).

The resources still exist and I can access them directly, but they stop being returned by searches as part of the includes. This is happening on multiple people's machines (everyone has their own instance and DB for testing) but which resource type is affected from instance to instance isn't consistent (other than it's an included resource). Unfortunately we haven't been able to figure out the trigger for this happening, but we've found that the $reindex job restores the full result set. The downside of the $reindex approach is that it can take several hours (or even days, depending on number of records) to complete, during which we can't trust our search results.

The closest thing to reproduction steps I've been able to discern is that it doesn't seem to happen with newly imported data (at least as of yet). The data impacted tends to have been imported at least a week prior to this issue appearing, but there's no guarantee that week-old data will have it happen. The FHIR servers will be spun up and torn down throughout the time between data ingestion and this issue happening, though it's safe to say that they're running more often than not. I should also mention that this is all with individual instances (no clustering).

In case it's relevant, here are some changes we've made to the default config:

• turn off Lucene indexing (we don't need full-text search)
• (delete_)expunge_enabled = false
• we're using tenants

We were originally seeing this issue in 5.7.0, but upgraded to 6.0.1 in the hopes it would be resolved. We thought it had, but this morning we noticed it again. I should also clarify that we weren't seriously testing with HAPI prior to 5.7.0, so there's a chance it existed before that version.

I realize "let it sit a couple of weeks and cross your fingers" is pretty lousy in terms of reproduction steps, but does anyone have any idea of what might be happening here?