Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SNOMED Hydrating, Filtering, Special MR Generation #1648

Draft
wants to merge 19 commits into
base: develop
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
feat(snomed): precondition analysis
Refs: #1442
  • Loading branch information
jonahkaye committed Mar 31, 2024
commit 28fb18b1528ccc96e71361ef2e3e09c1cba6b7b1
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
{{>DataType/Identifier.hbs id=this}},
{{/each}}
],
"verificationStatus": {{ProvisionalStatus}},
"clinicalStatus": {
"coding":
[
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,14 @@
{{/with}}
{{/if}}
{{/each}}

{{#if medAdm.substanceAdministration.precondition}}
{{#each (toArray medAdm.substanceAdministration.precondition) as |precondition|}}
{{#if precondition.criterion.value.code}}
{{>Resources/Condition.hbs conditionEntry=precondition.criterion ProvisionalStatus="true" ID=(generateUUID (toJsonString precondition.criterion))}},
{{>References/MedicationAdministration/reasonReference.hbs ID=(generateUUID (toJsonString medAdm.substanceAdministration)) REF=(concat 'Condition/' (generateUUID (toJsonString precondition.criterion)))}},
{{/if}}
{{/each}}
{{/if}}
{{/each}}
{{/each}}
{{/with}}
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,15 @@
{{/with}}
{{/if}}
{{/if}}


{{/each}}
{{/if}}
{{#if medEntry.substanceAdministration.precondition}}
Copy link
Member Author

@jonahkaye jonahkaye Apr 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

grabbing conditions linked to medications.
This gets us a few more conditions but which empirically seem to not be the most valuable. Optional to keep this, though my preference is to default on the side of getting as much data from the convert and filtering out downstream.

{{#each (toArray medEntry.substanceAdministration.precondition) as |precondition|}}
{{#if precondition.criterion.value.code}}
{{>Resources/Condition.hbs conditionEntry=precondition.criterion ProvisionalStatus="true" ID=(generateUUID (toJsonString criterion.value))}},
{{>References/MedicationStatement/reasonReference.hbs ID=(generateUUID (toJsonString medEntry.substanceAdministration)) REF=(concat 'Condition/' (generateUUID (toJsonString criterion.value)))}},
{{/if}}
{{/each}}
{{/if}}

Expand Down
2 changes: 1 addition & 1 deletion packages/utils/src/customer-requests/consolidated-query.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import { getEnvVarOrFail } from "@metriport/core/util/env-var";
const apiKey = getEnvVarOrFail("API_KEY");
const metriportApi: MetriportMedicalApi = new MetriportMedicalApi(apiKey);

const patientIds: string[] = ["patient1", "patient2", "patient3"];
const patientIds: string[] = [""];
const resultsDirectory = "./consolidatedPatients";

const ensureDirectory = async (): Promise<void> => {
Expand Down
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the main customer facing output of this PR. A script that for a given customer, gets all their patients consolidated data and performs the filtering and MR summary generation

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Full script for running this whole process

Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,12 @@ import { bundleToHtml } from "@metriport/core/external/aws/lambda-logic/bundle-t
import fs from "fs";
import { convertHtmlTablesToCsv } from "./convert-html-to-csv";
import * as path from "path";
import { fullProcessing } from "../terminology-server/snomed-dedup-and-filter";

// Function to process each JSON file
function processFile(filePath: string) {
async function processFile(filePath: string) {
await fullProcessing(filePath);

const bundle = fs.readFileSync(filePath, "utf8");
const bundleParsed = JSON.parse(bundle);

Expand All @@ -22,7 +25,7 @@ function processFile(filePath: string) {
}

// Main function to iterate through the directory
function main() {
async function main() {
const targetPath = process.argv[2];

if (!targetPath) {
Expand All @@ -31,12 +34,12 @@ function main() {
}

const files = fs.readdirSync(targetPath);
files.forEach(file => {
for (const file of files) {
const fullPath = path.join(targetPath, file);
if (path.extname(fullPath) === ".json") {
processFile(fullPath);
await processFile(fullPath);
}
});
}
}

main();
80 changes: 80 additions & 0 deletions packages/utils/src/terminology-server/precondition-analysis.ts
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Analysis done on preconditions on sample patient set for reference:

Total #of Conditions in the directory: 401308
Total #of Conditions generated from <Precondition> in the directory: 685
Total #of Conditions generated from <Precondition> in the directory of type disorder: 56

15802004 with count 1 is a disorder: Dystonia (disorder)
16932000 with count 20 is a disorder: Nausea and vomiting (disorder)
37796009 with count 3 is a disorder: Migraine (disorder)
38341003 with count 3 is a disorder: Hypertensive disorder, systemic arterial (disorder)
39579001 with count 2 is a disorder: Anaphylaxis (disorder)
70153002 with count 3 is a disorder: Hemorrhoids (disorder)
193462001 with count 6 is a disorder: Insomnia (disorder)
242253008 with count 2 is a disorder: Overdose of opiate (disorder)
302866003 with count 4 is a disorder: Hypoglycemia (disorder)
419076005 with count 1 is a disorder: Allergic reaction (disorder)
422400008 with count 10 is a disorder: Vomiting (disorder)
860914002 with count 1 is a disorder: Erectile dysfunction (disorder)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add documentation explaining what it does and how to use it, please.

Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
import fs from "fs";
import path from "path";
import { getCodeDisplay } from "./term-server-api";

function processDirectory(
directoryPath: string,
hashTable: { [key: string]: number },
totalConditionCounts: { count: number }
) {
const filesAndDirectories = fs.readdirSync(directoryPath);

for (const name of filesAndDirectories) {
const currentPath = path.join(directoryPath, name);
const stat = fs.statSync(currentPath);

if (stat.isDirectory()) {
processDirectory(currentPath, hashTable, totalConditionCounts);
} else if (name.endsWith(".json")) {
processFile(currentPath, hashTable, totalConditionCounts);
}
}
}

function processFile(
filePath: string,
hashTable: { [key: string]: number },
totalConditionCounts: { count: number }
) {
const data = JSON.parse(fs.readFileSync(filePath, "utf8"));
if (data.resourceType === "Bundle" && data.type === "batch") {
for (const entry of data.entry) {
if (
entry.resource?.resourceType === "Condition" &&
entry.resource.verificationStatus === true
) {
const codeInfo = entry.resource.code;
const code = codeInfo.coding[0].code;
if (hashTable[code]) {
hashTable[code] += 1;
} else {
hashTable[code] = 1;
}
}
totalConditionCounts.count += 1;
}
}
}

async function main() {
const directoryPath = process.argv[2];
if (!directoryPath) {
console.error("Please provide a directory path as an argument.");
process.exit(1);
}

const hashTable: { [key: string]: number } = {};
const totalConditionCount = { count: 0 };
processDirectory(directoryPath, hashTable, totalConditionCount);

let disorderCount = 0;
for (const code of Object.keys(hashTable)) {
const codeDetails = await getCodeDisplay(code, "SNOMEDCT_US");
if (codeDetails && codeDetails.category === "disorder") {
console.log(
`${code} with count ${hashTable[code]} is a disorder: ${codeDetails.display} (${codeDetails.category})`
);
disorderCount += hashTable[code];
}
}

const totalCount = Object.values(hashTable).reduce((acc, count) => acc + count, 0);

console.log(`Total #of Conditions in the directory: ${totalConditionCount.count}`);
console.log(`Total #of Conditions generated from <Precondition> in the directory: ${totalCount}`);
console.log(
`Total #of Conditions generated from <Precondition> in the directory of type disorder: ${disorderCount}`
);
}

main();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add documentation explaining what it does and how to use it, please.

Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,6 @@ async function removeNonRootSnomedCodes(
entries.splice(i, 1);
break;
} else {
console.log("Keeping SNOMED code:", coding.code, "Resource.id:", resource.id);
hashTable[coding.code].inserted = true;
allRemainingEnries.add(resource.id);
break;
Expand Down Expand Up @@ -234,13 +233,7 @@ async function filterMedicationsEntries(
fs.writeFileSync(filePath, JSON.stringify(data, null, 2), "utf8");
}

async function main() {
const [directoryPath] = process.argv.slice(2);
if (!directoryPath) {
console.error("Please provide a directory path as an argument.");
process.exit(1);
}

export async function fullProcessing(directoryPath: string) {
const removalStats = createInitialRemovalStats();

// Removal of non-disorder SNOMED codes, invalid CPT codes, and duplicate CPT codes
Expand Down Expand Up @@ -297,4 +290,13 @@ async function main() {
prettyPrintRemovalStats(removalStats);
}

async function main() {
const [directoryPath] = process.argv.slice(2);
if (!directoryPath) {
console.error("Please provide a directory path as an argument.");
process.exit(1);
}
await fullProcessing(directoryPath);
}
Comment on lines +352 to +359
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: It's a pain to have to scroll to the bottom and make the way up instead of just following the text from top to bottom as one scrolls it down


main();
1 change: 0 additions & 1 deletion packages/utils/src/terminology-server/term-server-api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,6 @@ export async function getCodeDisplay(
console.error("Connection refused. The server is not reachable at the moment.");
return undefined;
} else {
console.error("Error fetching code details. Code Not Found");
return undefined;
}
}
Expand Down
Loading