SNOMED Hydrating, Filtering, Special MR Generation #1648

jonahkaye · 2024-02-23T18:12:25Z

Refs: #1442

Description

In this PR, we introduce comprehensive deduplication and filtering script as a preprocessing step for generating medical record summaries.
Conditions: We take every single condition a patient has, and to start, remove all duplicate and non-disorder clinical codes. Then, we use SNOMED code lookups to create a hierarchical tree mapping the relationships between all the remaining codes. For every code that is a child of another parent code, we remove that child code. So for example, if we have Diabetes Mellitus and Type II diabetes mellitus without complication (disorder), because the latter is a descendent of Type II diabetes mellitus which is a descendent of Diabetes Mellitus, we remove Type II diabetes mellitus without complication (disorder) from the list of disorders we present
Procedures: We remove all procedures without CPT codes in the surgical range (10004 - 69990)
Medications: We remove all Medications that are not linked to a Condition that made it through the condition filtering phase

Note

This PR introduces yet another bundle-to-html function. A cx has a specific format they want the MR in, and easier (especially since this is just a script) to put that into its own file as opposed to refactoring with backwards compatability

Testing

Local
- local flow for now

Release Plan

Merge this
no prod workflow touched

Refs: #1442

…th same drug diff code case Refs: #1442

Refs: #1442

jonahkaye

Conditions:

deduplication
non disorder conditions removal
non Root code in heirarchy removal

Procedure

deduplication
non CPT surgery range removal

Medications:

removal of all Meds not linked to some value in the condition heirarchy

Refs: #1442

jonahkaye · 2024-04-03T17:44:34Z

packages/fhir-converter/src/templates/cda/Sections/Hospital_Discharge_Medications.hbs

@@ -5,7 +5,7 @@
 {{>Resources/Medication.hbs medication=medAdm.substanceAdministration.consumable.manufacturedProduct.manufacturedMaterial ID=(generateUUID (toJsonString medAdm.substanceAdministration.consumable.manufacturedProduct.manufacturedMaterial))}},
 {{>References/MedicationAdministration/medicationReference.hbs ID=(generateUUID (toJsonString medAdm.substanceAdministration)) REF=(concat 'Medication/' (generateUUID (toJsonString medAdm.substanceAdministration.consumable.manufacturedProduct.manufacturedMaterial)))}},

- {{#with (evaluate 'Utils/GenerateOrganizationId.hbs' obj=edAdm.substanceAdministration.performer.assignedEntity.representedOrganization) as |orgId|}}
+ {{#with (evaluate 'Utils/GenerateOrganizationId.hbs' obj=medAdm.substanceAdministration.performer.assignedEntity.representedOrganization) as |orgId|}}


Ouch, this is stuff that our tooling should catch for us. Isn't there a way we can parse/compile those? At least to check if the variable names are correct and not misspelled?

Also, is there a way we can have unit testing for this? We can def create a ticket to set it up.

Just because w/ unit tests we can also create new tests as we find bugs that would catch those if they were in place.

jonahkaye · 2024-04-03T17:45:55Z

packages/fhir-converter/src/templates/cda/Sections/Medication.hbs

@@ -45,6 +45,14 @@

 {{/each}}
 {{/if}}
+ {{#if medEntry.substanceAdministration.precondition}}


grabbing conditions linked to medications.
This gets us a few more conditions but which empirically seem to not be the most valuable. Optional to keep this, though my preference is to default on the side of getting as much data from the convert and filtering out downstream.

jonahkaye · 2024-04-03T17:47:07Z

packages/utils/src/customer-requests/convert-html-to-csv.ts

@@ -110,3 +111,35 @@ export function convertHtmlTablesToCsv(html: string) {

 return convertedCsv;
 }
+
+function main() {


utility to call in the command line

jonahkaye · 2024-04-03T17:47:56Z

packages/utils/src/customer-requests/medical-records-local.ts

@@ -1,15 +1,21 @@
 import { bundleToHtml } from "@metriport/core/external/aws/lambda-logic/bundle-to-html";
 import fs from "fs";

-// get xml file from this folder and bundle to html
+// Check if a file path is provided
+if (process.argv.length < 3) {


utility to call in command line

Refs: #1442

jonahkaye · 2024-04-03T17:56:54Z

packages/utils/src/terminology-server/precondition-analysis.ts

Analysis done on preconditions on sample patient set for reference:

Total #of Conditions in the directory: 401308 Total #of Conditions generated from <Precondition> in the directory: 685 Total #of Conditions generated from <Precondition> in the directory of type disorder: 56 15802004 with count 1 is a disorder: Dystonia (disorder) 16932000 with count 20 is a disorder: Nausea and vomiting (disorder) 37796009 with count 3 is a disorder: Migraine (disorder) 38341003 with count 3 is a disorder: Hypertensive disorder, systemic arterial (disorder) 39579001 with count 2 is a disorder: Anaphylaxis (disorder) 70153002 with count 3 is a disorder: Hemorrhoids (disorder) 193462001 with count 6 is a disorder: Insomnia (disorder) 242253008 with count 2 is a disorder: Overdose of opiate (disorder) 302866003 with count 4 is a disorder: Hypoglycemia (disorder) 419076005 with count 1 is a disorder: Allergic reaction (disorder) 422400008 with count 10 is a disorder: Vomiting (disorder) 860914002 with count 1 is a disorder: Erectile dysfunction (disorder)

jonahkaye · 2024-04-03T17:58:35Z

packages/utils/src/customer-requests/full-snomed-pipeline.ts

This is the main customer facing output of this PR. A script that for a given customer, gets all their patients consolidated data and performs the filtering and MR summary generation

jonahkaye · 2024-04-22T15:03:21Z

packages/utils/src/customer-requests/full-snomed-pipeline.ts

Full script for running this whole process

Refs: #1442

leite08 · 2024-05-07T16:46:02Z

packages/fhir-converter/src/templates/cda/Sections/Medication.hbs

 {{/with}}
 {{/if}}
+ {{else if (contains (toJsonString entryRelationship.observation.templateId) '2.16.840.1.113883.10.20.22.4.19')}}


I can't tell what the OID is without looking up online. This is a yellow flag, as it increases dependency on the internet and external services. Maybe we can start moving those shared constants on a single file?

leite08 · 2024-05-07T16:48:11Z

packages/fhir-converter/src/templates/cda/Sections/Hospital_Discharge_Medications.hbs

@@ -5,7 +5,7 @@
 {{>Resources/Medication.hbs medication=medAdm.substanceAdministration.consumable.manufacturedProduct.manufacturedMaterial ID=(generateUUID (toJsonString medAdm.substanceAdministration.consumable.manufacturedProduct.manufacturedMaterial))}},
 {{>References/MedicationAdministration/medicationReference.hbs ID=(generateUUID (toJsonString medAdm.substanceAdministration)) REF=(concat 'Medication/' (generateUUID (toJsonString medAdm.substanceAdministration.consumable.manufacturedProduct.manufacturedMaterial)))}},

- {{#with (evaluate 'Utils/GenerateOrganizationId.hbs' obj=edAdm.substanceAdministration.performer.assignedEntity.representedOrganization) as |orgId|}}
+ {{#with (evaluate 'Utils/GenerateOrganizationId.hbs' obj=medAdm.substanceAdministration.performer.assignedEntity.representedOrganization) as |orgId|}}


Ouch, this is stuff that our tooling should catch for us. Isn't there a way we can parse/compile those? At least to check if the variable names are correct and not misspelled?

Also, is there a way we can have unit testing for this? We can def create a ticket to set it up.

Just because w/ unit tests we can also create new tests as we find bugs that would catch those if they were in place.

leite08 · 2024-05-07T16:52:38Z

packages/utils/src/customer-requests/full-snomed-pipeline.ts

+ } catch (error) {
+ console.error(`Error fetching FHIR data for patient ${patientId}:`, error);
+ }


This is failing silently and the code will keep executing. Is that expected?

leite08 · 2024-05-07T16:54:17Z

packages/utils/src/customer-requests/medical-records-local.ts

Let's add documentation explaining what it does and how to use it, please.

leite08 · 2024-05-07T16:55:02Z

packages/utils/src/customer-requests/full-snomed-pipeline.ts

+// This script is used to process all of a customers patients through a snomed filtering pipeline, and
+// then generate MR summaries and CSV files from the resulting bundles
+// The script makes consolidated queries, inserts a patient FHIR resource, and then calls the filtering
+// and MR generation logic.


nit: use multi-line comments

also, please indicate how to use it

leite08 · 2024-05-07T16:55:19Z

packages/utils/src/customer-requests/convert-html-to-csv.ts

Let's add documentation explaining what it does and how to use it, please.

leite08 · 2024-05-07T16:55:47Z

packages/utils/src/fhir-converter/e2e-test.ts

@@ -54,8 +54,8 @@ dayjs.extend(duration);
 * - fhirBaseUrl: the URL of the FHIR server;
 */

-const cdaLocation = ``;
-const converterBaseUrl = "http:https://localhost:8777";
+const cdaLocation = `/Users/jonahkaye/Desktop/2024-01-23T08:02:29.892Z/test-patient-sample`;


leite08 · 2024-05-07T16:56:44Z

packages/utils/src/terminology-server/precondition-analysis.ts

Let's add documentation explaining what it does and how to use it, please.

leite08 · 2024-05-07T16:58:07Z

packages/utils/src/terminology-server/snomed-dedup-and-filter.ts

+async function main() {
+ const [directoryPath] = process.argv.slice(2);
+ if (!directoryPath) {
+ console.error("Please provide a directory path as an argument.");
+ process.exit(1);
+ }
+ await fullProcessing(directoryPath);
+}


nit: It's a pain to have to scroll to the bottom and make the way up instead of just following the text from top to bottom as one scrolls it down

leite08 · 2024-05-07T16:58:19Z

packages/utils/src/terminology-server/snomed-dedup-and-filter.ts

Let's add documentation explaining what it does and how to use it, please.

feat(filter): snomed hydrating

6e2f2cf

Refs: #1442

jonahkaye requested review from Goncharo, leite08, Orta21 and RamilGaripov as code owners February 23, 2024 18:12

feat(snomed): cx specific workflow

9dfda6c

Refs: #1442

jonahkaye marked this pull request as draft March 21, 2024 15:12

jonahkaye self-assigned this Mar 26, 2024

jonahkaye added 3 commits March 26, 2024 18:43

Merge branch 'develop' into 1442-snomed-filtering

f35535f

feat(snomed): snomed hierachies computed and filtered

1565f67

Refs: #1442

feat(snomed): conditions and procedures duplications

0033587

Refs: #1442

Orta21 mentioned this pull request Mar 27, 2024

fix(api): refactor pd status function #1826

Merged

4 tasks

jonahkaye added 3 commits March 27, 2024 18:38

feat(snomed): removing duplicate meds - rxnorm lookups doesnt help wi…

8c132e8

…th same drug diff code case Refs: #1442

feat(snomed): simplified filtering, conditions resource id mapping

b5e9d1a

Refs: #1442

feat(snomed): procedures, medications, conditions good

4df103b

Refs: #1442

jonahkaye commented Mar 28, 2024

View reviewed changes

jonahkaye added 5 commits March 29, 2024 12:57

feat(snomed): more complete filtering by full tree search

bb865d7

Refs: #1442

feat(snomed): precondition analysis

28fb18b

Refs: #1442

feat(snomed): full pipeline done

4f48c40

Refs: #1442

feat(snomed): script header

b823225

Refs: #1442

feat(snomed): filtering for only the resources we need

28b190c

Refs: #1442

jonahkaye commented Apr 3, 2024

View reviewed changes

feat(snomed): adding special snomed bundle to html

9363f91

Refs: #1442

jonahkaye changed the title ~~feat(filter): snomed hydrating~~ SNOMED Hydrating, Filtering, Special MR Generation Apr 3, 2024

jonahkaye marked this pull request as ready for review April 3, 2024 17:59

jonahkaye added 2 commits April 17, 2024 20:52

Merge branch 'develop' into 1442-snomed-filtering

f3fd24f

Merge branch 'develop' into 1442-snomed-filtering

21e3e3f

jonahkaye added 2 commits April 19, 2024 12:13

feat(snomed): medications dates good pray

e7eb48f

Refs: #1442

feat(snomed): dedup med admins too

d0b04b6

Refs: #1442

jonahkaye commented Apr 22, 2024

View reviewed changes

feat(snomed): medadmin to medstatement

960040b

Refs: #1442

leite08 reviewed May 7, 2024

View reviewed changes

jonahkaye marked this pull request as draft May 7, 2024 17:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SNOMED Hydrating, Filtering, Special MR Generation #1648

SNOMED Hydrating, Filtering, Special MR Generation #1648

jonahkaye commented Feb 23, 2024 •

edited

jonahkaye left a comment •

edited

jonahkaye Apr 3, 2024

leite08 May 7, 2024

jonahkaye Apr 3, 2024 •

edited

jonahkaye Apr 3, 2024

jonahkaye Apr 3, 2024

jonahkaye Apr 3, 2024

jonahkaye Apr 3, 2024

jonahkaye Apr 22, 2024

leite08 May 7, 2024

leite08 May 7, 2024

leite08 May 7, 2024

leite08 May 7, 2024

leite08 May 7, 2024

leite08 May 7, 2024

leite08 May 7, 2024

leite08 May 7, 2024

leite08 May 7, 2024

leite08 May 7, 2024

SNOMED Hydrating, Filtering, Special MR Generation #1648

Are you sure you want to change the base?

SNOMED Hydrating, Filtering, Special MR Generation #1648

Conversation

jonahkaye commented Feb 23, 2024 • edited

Description

Note

Testing

Release Plan

jonahkaye left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonahkaye Apr 3, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonahkaye commented Feb 23, 2024 •

edited

jonahkaye left a comment •

edited

jonahkaye Apr 3, 2024 •

edited