SNOMED Hydrating, Filtering, Special MR Generation #1648
+3,355
−29
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
There was a problem hiding this comment. Let's add documentation explaining what it does and how to use it, please. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
There was a problem hiding this comment. This is the main customer facing output of this PR. A script that for a given customer, gets all their patients consolidated data and performs the filtering and MR summary generation There was a problem hiding this comment. Full script for running this whole process |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,114 @@ | ||
| // This script is used to process all of a customers patients through a snomed filtering pipeline, and | ||
| // then generate MR summaries and CSV files from the resulting bundles | ||
| // The script makes consolidated queries, inserts a patient FHIR resource, and then calls the filtering | ||
| // and MR generation logic. | ||
|
Comment on lines
+1
to
+4
There was a problem hiding this comment. nit: use multi-line comments also, please indicate how to use it |
||
|
|
||
| import { bundleToHtml } from "@metriport/core/external/aws/lambda-logic/bundle-to-html-snomed"; | ||
| import fs from "fs/promises"; | ||
| import { convertHtmlTablesToCsv } from "./convert-html-to-csv"; | ||
| import * as path from "path"; | ||
| import { fullProcessing } from "../terminology-server/snomed-dedup-and-filter"; | ||
| import { MetriportMedicalApi } from "@metriport/api-sdk"; | ||
| import { getEnvVarOrFail } from "@metriport/core/util/env-var"; | ||
| import * as dotenv from "dotenv"; | ||
| import axios from "axios"; | ||
| import dayjs from "dayjs"; | ||
| import duration from "dayjs/plugin/duration"; | ||
| dotenv.config(); | ||
| dayjs.extend(duration); | ||
|
|
||
| const apiKey = getEnvVarOrFail("API_KEY"); | ||
| const apiLoadBalancerURL = getEnvVarOrFail("API_URL"); | ||
| const fhirUrl = getEnvVarOrFail("FHIR_URL"); | ||
| const cxId = getEnvVarOrFail("CX_ID"); | ||
| const metriportApi: MetriportMedicalApi = new MetriportMedicalApi(apiKey); | ||
|
|
||
| const timestamp = dayjs().toISOString(); | ||
| const resultsDirectory = `./runs/consolidatedPatients/${timestamp}`; | ||
|
|
||
| async function fetchPatientIds(cxId: string): Promise<string[]> { | ||
| const url = `${apiLoadBalancerURL}/internal/patient/ids?cxId=${cxId}`; | ||
| try { | ||
| const response = await axios.get(url); | ||
| return response.data.patientIds; | ||
| } catch (error) { | ||
| console.error(`Error fetching patient IDs:`, error); | ||
| return []; | ||
| } | ||
| } | ||
|
|
||
| //eslint-disable-next-line | ||
| async function getFhirPatientData(patientId: string): Promise<any> { | ||
| const url = `${fhirUrl}/fhir/${cxId}/Patient/${patientId}/`; | ||
| try { | ||
| const response = await axios.get(url); | ||
| return response.data; | ||
| } catch (error) { | ||
| console.error(`Error fetching FHIR data for patient ${patientId}:`, error); | ||
| } | ||
|
Comment on lines
+46
to
+48
There was a problem hiding this comment. This is failing silently and the code will keep executing. Is that expected? |
||
| } | ||
|
|
||
| export async function ensureDirectory(): Promise<void> { | ||
| await fs.mkdir(resultsDirectory, { recursive: true }); | ||
| } | ||
|
|
||
| async function fetchAndSavePatientData(patientId: string): Promise<void> { | ||
| try { | ||
| const resources = ["Condition, Procedure, MedicationStatement, MedicationAdministration"]; | ||
| const data = await metriportApi.getPatientConsolidated(patientId, resources); | ||
| const fhirPatient = await getFhirPatientData(patientId); | ||
| const resourceWrappedFhirPatient = { resource: fhirPatient }; | ||
| if (!data.entry) { | ||
| data.entry = []; | ||
| } | ||
| data.entry.push(resourceWrappedFhirPatient); | ||
| const filePath = `${resultsDirectory}/${patientId}.json`; | ||
| await fs.writeFile(filePath, JSON.stringify(data, null, 2)); | ||
| console.log(`Data saved for patient ${patientId}`); | ||
| } catch (error) { | ||
| console.error(`Error fetching and saving data for patient ${patientId}:`, error); | ||
| } | ||
| } | ||
|
|
||
| async function processPatients(): Promise<void> { | ||
| console.log("Checking if directory exists..."); | ||
| await ensureDirectory(); | ||
| console.log("Fetching patient IDs..."); | ||
| const patientIds = await fetchPatientIds(cxId); | ||
| for (const patientId of patientIds) { | ||
| await fetchAndSavePatientData(patientId); | ||
| } | ||
| console.log("All patient data processed."); | ||
| } | ||
|
|
||
| async function processFile(filePath: string) { | ||
| await fullProcessing(filePath); | ||
|
|
||
| const bundle = await fs.readFile(filePath, "utf8"); | ||
| const bundleParsed = JSON.parse(bundle); | ||
|
|
||
| const html = bundleToHtml(bundleParsed); | ||
| const htmlOutputFilePath = filePath.replace(".json", ".html"); | ||
| await fs.writeFile(htmlOutputFilePath, html); | ||
| console.log(`HTML file created at ${htmlOutputFilePath}`); | ||
|
|
||
| const csvContent = convertHtmlTablesToCsv(html); | ||
| const csvOutputFilePath = filePath.replace(".json", ".csv"); | ||
| await fs.writeFile(csvOutputFilePath, csvContent); | ||
| console.log(`CSV file created at ${csvOutputFilePath}`); | ||
| } | ||
|
|
||
| async function main() { | ||
| await processPatients(); | ||
|
|
||
| const files = await fs.readdir(resultsDirectory); | ||
| for (const file of files) { | ||
| const fullPath = path.join(resultsDirectory, file); | ||
| if (path.extname(fullPath) === ".json") { | ||
| console.log(`Processing file ${fullPath}`); | ||
| await processFile(fullPath); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| main(); | ||
|
There was a problem hiding this comment. Let's add documentation explaining what it does and how to use it, please. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,15 +1,18 @@ | ||
| import { bundleToHtml } from "@metriport/core/external/aws/lambda-logic/bundle-to-html"; | ||
| import fs from "fs"; | ||
|
|
||
| if (process.argv.length < 3) { | ||
|
There was a problem hiding this comment. utility to call in command line |
||
| console.log("Usage: node medical-records-local.js <path-to-json-file>"); | ||
| process.exit(1); | ||
| } | ||
|
|
||
| const filePath = process.argv[2]; | ||
| const bundle = fs.readFileSync(filePath, "utf8"); | ||
| const bundleParsed = JSON.parse(bundle); | ||
|
|
||
| const html = bundleToHtml(bundleParsed); | ||
|
|
||
| const outputFilePath = filePath.replace(".json", ".html"); | ||
|
|
||
| fs.writeFileSync(outputFilePath, html); | ||
| console.log(`HTML file created at ${outputFilePath}`); | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
There was a problem hiding this comment. Analysis done on preconditions on sample patient set for reference: There was a problem hiding this comment. Let's add documentation explaining what it does and how to use it, please. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| import fs from "fs"; | ||
| import path from "path"; | ||
| import { getCodeDisplay } from "./term-server-api"; | ||
|
|
||
| function processDirectory( | ||
| directoryPath: string, | ||
| hashTable: { [key: string]: number }, | ||
| totalConditionCounts: { count: number } | ||
| ) { | ||
| const filesAndDirectories = fs.readdirSync(directoryPath); | ||
|
|
||
| for (const name of filesAndDirectories) { | ||
| const currentPath = path.join(directoryPath, name); | ||
| const stat = fs.statSync(currentPath); | ||
|
|
||
| if (stat.isDirectory()) { | ||
| processDirectory(currentPath, hashTable, totalConditionCounts); | ||
| } else if (name.endsWith(".json")) { | ||
| processFile(currentPath, hashTable, totalConditionCounts); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| function processFile( | ||
| filePath: string, | ||
| hashTable: { [key: string]: number }, | ||
| totalConditionCounts: { count: number } | ||
| ) { | ||
| const data = JSON.parse(fs.readFileSync(filePath, "utf8")); | ||
| if (data.resourceType === "Bundle" && data.type === "batch") { | ||
| for (const entry of data.entry) { | ||
| if ( | ||
| entry.resource?.resourceType === "Condition" && | ||
| entry.resource.verificationStatus === true | ||
| ) { | ||
| const codeInfo = entry.resource.code; | ||
| const code = codeInfo.coding[0].code; | ||
| if (hashTable[code]) { | ||
| hashTable[code] += 1; | ||
| } else { | ||
| hashTable[code] = 1; | ||
| } | ||
| } | ||
| totalConditionCounts.count += 1; | ||
| } | ||
| } | ||
| } | ||
|
|
||
| async function main() { | ||
| const directoryPath = process.argv[2]; | ||
| if (!directoryPath) { | ||
| console.error("Please provide a directory path as an argument."); | ||
| process.exit(1); | ||
| } | ||
|
|
||
| const hashTable: { [key: string]: number } = {}; | ||
| const totalConditionCount = { count: 0 }; | ||
| processDirectory(directoryPath, hashTable, totalConditionCount); | ||
|
|
||
| let disorderCount = 0; | ||
| for (const code of Object.keys(hashTable)) { | ||
| const codeDetails = await getCodeDisplay(code, "SNOMEDCT_US"); | ||
| if (codeDetails && codeDetails.category === "disorder") { | ||
| console.log( | ||
| `${code} with count ${hashTable[code]} is a disorder: ${codeDetails.display} (${codeDetails.category})` | ||
| ); | ||
| disorderCount += hashTable[code]; | ||
| } | ||
| } | ||
|
|
||
| const totalCount = Object.values(hashTable).reduce((acc, count) => acc + count, 0); | ||
|
|
||
| console.log(`Total #of Conditions in the directory: ${totalConditionCount.count}`); | ||
| console.log(`Total #of Conditions generated from <Precondition> in the directory: ${totalCount}`); | ||
| console.log( | ||
| `Total #of Conditions generated from <Precondition> in the directory of type disorder: ${disorderCount}` | ||
| ); | ||
| } | ||
|
|
||
| if (require.main === module) { | ||
| main(); | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
bug
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ouch, this is stuff that our tooling should catch for us. Isn't there a way we can parse/compile those? At least to check if the variable names are correct and not misspelled?
Also, is there a way we can have unit testing for this? We can def create a ticket to set it up.
Just because w/ unit tests we can also create new tests as we find bugs that would catch those if they were in place.