Widget Dashboard Queries using extracted Search Parameters #177
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- added fields tracking SearchParameters During record insertion, populating FHIR Resource tables, & populating search param fields using FhirPathjs + GoJa Library Moved the QueryResources function into its own file. Added an automigrate function for the new db models working on query parsing logic (with raw SQL).
…ed files will change every time) make sure simple field types are not json stringified when returned from GoJa.
…ed files will change every time) make sure simple field types are not json stringified when returned from GoJa.
BROKEN - token, quantities, references, etc.
…ize) Working Quantity Json parsing - added support for FROM clauses (used by json_each)
Limited to only a single use (cant be paird with other clauses). added explicit Security annotations for required SQL injection protection.
…FHIRPath expressions. fixing this so it works for complex and primitive types.
…n parameters at the same time. Removing restriction.
…ter, ie. (value = "foo" OR value = "bar") AND (value = "baz")
renamed RawResource field to ResourceRaw to be consistent. Using `resource_raw` json key to be consistent with UI.
dashboard widget where clause changed to support FHIR Search API style querying (server side instead of client side)
… FHIR Table name by resource type. ListResources now uses the FHIR Resource tables.
Make sure we continue to use models.ResourceFhir{} when querying.
Adding Select and Group statements to QueryResources GORM query:
- Select fhir.* is required, otherwise json_each "tables" are included in the result, and mapping to ResourceFhir struct fails (json_each.id is a number not a UUID)
- Group By fhir.id is required otherwise we can have multiple entries for the same resource (json_each can match multiple times).
…oss every table).
fixing condition display in medical history page.
Update all SqliteRepository funtions to correctly search against the individual tables, but use ResourceBase as the query result
…th individual table setup)
regenerate mocked sqlite database struct Adding tests for composition
adding more testing for CustomUser Fixing broken tests removed unnecessary API calls from dashboard component. TODO: storybook tests are still broken.
Codecov Report
❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more. Additional details and impacted files@@ Coverage Diff @@
## sandbox #177 +/- ##
==========================================
Coverage ? 10.26%
==========================================
Files ? 217
Lines ? 25547
Branches ? 450
==========================================
Hits ? 2622
Misses ? 22788
Partials ? 137
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
This was referenced Jul 15, 2023
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As discussed previously, there are multiple issues with the current query syntax used by the Widgets. In short, the query is processed in two places, with a majority of the processing taking place in the user's browser -- which means a large amount of data is retrieved and transmitted from the database.
After a very productive conversation with @cody from Medplum, I've decided to mimic their solution - extract the FHIR SearchParameters for each resource as they are being stored, and store them beside the raw FHIR resource data. The complexity here is that there does not seem to be a working Go implementation of the FHIRPath.js library, so instead we'll be using github.com/dop251/goja to run Javascript code in Go 😬 .
I can confirm that it works, with a reasonable amount of overhead (this Javascript processing only takes place during record storage, so it's a 1 time expense). The benefits are immediate: querying is now much more performant, with a "well documented" syntax (we'll be using a subset of the FHIR search syntax) -- hopefully fully unlocking the widget-based dashboard's potential, allowing developers to create & share their condition specific dashboards with non-developers.
related #21
related #156
Search Safety Checklis - https://hl7.org/fhir/R4/safety.html#search
TODO:
stringsearch needs to support Human Names correctly