-
Notifications
You must be signed in to change notification settings - Fork 253
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Updated PerResourcePatchGenerator to return ordered PatchMapping to avoid referential integrity issues. #2442
Merged
Merged
Changes from all commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
ea57ccb
Updated PerResourcePatchGenerator to return ordererd PatchMapping to…
aditya-07 24a120e
Fixed the failing test by fixing the palyload json format
aditya-07 558d6c5
Review Comments: Refactored code to find references using LocalChange…
aditya-07 ee9f37b
Merge branch 'master' into ak/patchGeneratorOrder
aditya-07 db4101d
Merge branch 'master' into ak/patchGeneratorOrder
aditya-07 97e8804
Merge branch 'master' into ak/patchGeneratorOrder
aditya-07 75693e6
Review comments: Refactored code
aditya-07 b120218
Added files
aditya-07 cdc2f20
Review comments: Changes to cache PerResourcePatchGenerator instance
aditya-07 575875d
spotless
aditya-07 10e9410
Merge branch 'master' into ak/patchGeneratorOrder
aditya-07 59b7f36
Review comments: Simplified tests and added kdocs
aditya-07 fae4a08
Merge branch 'master' into ak/patchGeneratorOrder
aditya-07 d97a908
Merge branch 'master' into ak/patchGeneratorOrder
aditya-07 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
142 changes: 142 additions & 0 deletions
142
engine/src/main/java/com/google/android/fhir/sync/upload/patch/PatchOrdering.kt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,142 @@ | ||
/* | ||
* Copyright 2024 Google LLC | ||
* | ||
* Licensed under the Apache License, Version 2.0 (the "License"); | ||
* you may not use this file except in compliance with the License. | ||
* You may obtain a copy of the License at | ||
* | ||
* https://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
package com.google.android.fhir.sync.upload.patch | ||
|
||
import androidx.annotation.VisibleForTesting | ||
import com.google.android.fhir.db.Database | ||
import com.google.android.fhir.db.LocalChangeResourceReference | ||
|
||
private typealias Node = String | ||
|
||
/** | ||
* Orders the [PatchMapping]s to maintain referential integrity during upload. | ||
* | ||
* ``` | ||
* Encounter().apply { | ||
* id = "encounter-1" | ||
* subject = Reference("Patient/patient-1") | ||
* } | ||
* | ||
* Observation().apply { | ||
* id = "observation-1" | ||
* subject = Reference("Patient/patient-1") | ||
* encounter = Reference("Encounter/encounter-1") | ||
* } | ||
* ``` | ||
* * The Encounter has an outgoing reference to Patient and the Observation has outgoing references | ||
* to Patient and the Encounter. | ||
* * Now, to maintain the referential integrity of the resources during the upload, | ||
* `Encounter/encounter-1` must go before the `Observation/observation-1`, irrespective of the | ||
* order in which the Encounter and Observation were added to the database. | ||
*/ | ||
internal object PatchOrdering { | ||
|
||
private val PatchMapping.resourceTypeAndId: String | ||
get() = "${generatedPatch.resourceType}/${generatedPatch.resourceId}" | ||
|
||
/** | ||
* Order the [PatchMapping] so that if the resource A has outgoing references {B,C} (CREATE) and | ||
* {D} (UPDATE), then B,C needs to go before the resource A so that referential integrity is | ||
* retained. Order of D shouldn't matter for the purpose of referential integrity. | ||
* | ||
* @return - A ordered list of the [PatchMapping]s based on the references to other [PatchMapping] | ||
* if the mappings are acyclic | ||
* - throws [IllegalStateException] otherwise | ||
*/ | ||
suspend fun List<PatchMapping>.orderByReferences( | ||
database: Database, | ||
): List<PatchMapping> { | ||
val resourceIdToPatchMapping = associateBy { patchMapping -> patchMapping.resourceTypeAndId } | ||
|
||
/* Get LocalChangeResourceReferences for all the local changes. A single LocalChange may have | ||
multiple LocalChangeResourceReference, one for each resource reference in the | ||
LocalChange.payload.*/ | ||
val localChangeIdToResourceReferenceMap: Map<Long, List<LocalChangeResourceReference>> = | ||
database | ||
.getLocalChangeResourceReferences(flatMap { it.localChanges.flatMap { it.token.ids } }) | ||
.groupBy { it.localChangeId } | ||
|
||
val adjacencyList = createAdjacencyListForCreateReferences(localChangeIdToResourceReferenceMap) | ||
return createTopologicalOrderedList(adjacencyList).mapNotNull { resourceIdToPatchMapping[it] } | ||
} | ||
|
||
/** | ||
* @return A map of [PatchMapping] to all the outgoing references to the other [PatchMapping]s of | ||
* type [Patch.Type.INSERT] . | ||
*/ | ||
@VisibleForTesting | ||
internal fun List<PatchMapping>.createAdjacencyListForCreateReferences( | ||
aditya-07 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
localChangeIdToReferenceMap: Map<Long, List<LocalChangeResourceReference>>, | ||
): Map<Node, List<Node>> { | ||
val adjacencyList = mutableMapOf<Node, List<Node>>() | ||
/* if the outgoing reference is to a resource that's just an update and not create, then don't | ||
link to it. This may make the sub graphs smaller and also help avoid cyclic dependencies.*/ | ||
val resourceIdsOfInsertTypeLocalChanges = | ||
asSequence() | ||
.filter { it.generatedPatch.type == Patch.Type.INSERT } | ||
.map { it.resourceTypeAndId } | ||
.toSet() | ||
|
||
forEach { patchMapping -> | ||
adjacencyList[patchMapping.resourceTypeAndId] = | ||
patchMapping.findOutgoingReferences(localChangeIdToReferenceMap).filter { | ||
resourceIdsOfInsertTypeLocalChanges.contains(it) | ||
} | ||
} | ||
return adjacencyList | ||
} | ||
|
||
private fun PatchMapping.findOutgoingReferences( | ||
localChangeIdToReferenceMap: Map<Long, List<LocalChangeResourceReference>>, | ||
): Set<Node> { | ||
val references = mutableSetOf<Node>() | ||
when (generatedPatch.type) { | ||
Patch.Type.INSERT, | ||
Patch.Type.UPDATE, -> { | ||
localChanges.forEach { localChange -> | ||
localChange.token.ids.forEach { id -> | ||
localChangeIdToReferenceMap[id]?.let { | ||
references.addAll(it.map { it.resourceReferenceValue }) | ||
} | ||
} | ||
} | ||
} | ||
Patch.Type.DELETE -> { | ||
// do nothing | ||
} | ||
} | ||
return references | ||
} | ||
|
||
private fun createTopologicalOrderedList(adjacencyList: Map<Node, List<Node>>): List<Node> { | ||
val stack = ArrayDeque<String>() | ||
val visited = mutableSetOf<String>() | ||
val currentPath = mutableSetOf<String>() | ||
|
||
fun dfs(key: String) { | ||
check(currentPath.add(key)) { "Detected a cycle." } | ||
if (visited.add(key)) { | ||
adjacencyList[key]?.forEach { dfs(it) } | ||
stack.addFirst(key) | ||
} | ||
currentPath.remove(key) | ||
} | ||
|
||
adjacencyList.keys.forEach { dfs(it) } | ||
return stack.reversed() | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aditya-07 marked this conversation as resolved.
Show resolved
Hide resolved
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Upload on sync fails here with
Guessing the query should probably use an
IN
instead of=
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the temporary change below worked for me
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes it should be
IN
. Strange the test case didn't track this.Probably the function is being called with one LocalChangeId in the list.
@aditya-07 might wanna add / modify the test case to cover this scenario ?!