Synchronize sync workers

[MJ1998]

IMPORTANT: All PRs must be linked to an issue (except for extremely trivial and straightforward changes).

Fixes #2361

Description
We use the recommended way for mutual exclusion: link

Solution: Wrap the FhirSynchronizer#synchronize with mutex.withLock

mutex.withLock will suspend the successive invocations of FhirSynchronizer#synchronize, without blocking the thread, until the lock is released. Hence all work requests will be processed - and in FIFO order.

Alternative(s) considered
Alternative: One alternative involves using a ReentrantLock within a try-catch-finally block. This approach essentially replicates mutex-like behavior to manage concurrent access to shared resources. It would require maintaining a job queue to handle work requests sequentially. Without a queue, workers would need to return Result.retry, leading to repeated attempts every 30 seconds.

Another alternative was to wrap FhirSyncWorker#doWork with mutex.withLock but based on this comment and also for the fact that FhirSynchronizer#synchronize should run sequentially irrespective of whether FhirSyncWorker uses it or not.

Type
Bug fix

Screenshots (if applicable)

Checklist

• I have read and acknowledged the Code of conduct.
• I have read the Contributing page.
• I have signed the Google Individual CLA, or I am covered by my company's Corporate CLA.
• I have discussed my proposed solution with code owners in the linked issue(s) and we have agreed upon the general approach.
• I have run ./gradlew spotlessApply and ./gradlew spotlessCheck to check my code follows the style guide of this project.
• I have run ./gradlew check and ./gradlew connectedCheck to test my changes locally.
• I have built and run the demo app(s) to verify my change fixes the issue and/or does not break the demo app(s).

[aditya-07]

Please add a test to check the change.

[MJ1998]

I am not sure if automation test for this is a good idea. Because it takes lot of execution time to be able to verify the synchronicity of any concurrent jobs. See how concurrent behavior is verified here.

Such long execution time, however, is very likely in case of long sync time in an actual app

@aditya-07

[jingtang10]

please feel free to merge once tested manually again.

[MJ1998]

I figured how to test the mutex locking.
@aditya-07 @jingtang10