Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: hapi-fhir-cli: (1) docs outdated, (2) can't import any terminologies #3276

Open
joeflack4 opened this issue Jan 6, 2022 · 12 comments
Open

Comments

@joeflack4
Copy link

joeflack4 commented Jan 6, 2022

1. Docs outdated

https://hapifhir.io/hapi-fhir/docs/tools/hapi_fhir_cli.html

1.

The -f flag should be changed to -v

2.

IC10CM: ./hapi-fhir-cli upload-terminology -d Downloads/LOINC_2.54_MULTI-AXIAL_HIERARCHY.zip -d icd10cm_tabular_2021.xml -f dstu3 -t https://localhost:8080/baseDstu3 -u https://hl7.org/fhir/sid/icd-10-cm
Should Downloads/LOINC_2.54_MULTI-AXIAL_HIERARCHY.zip really be in here?

2. Can't import any terminologies

Steps to reproduce the behavior:

  1. Clone my (ever so slightly changed) hapi-fhir-starter repo: https://github.com/HOT-Ecosystem/hapi-fhir-jpaserver-starter.git
  2. Deploy with docker-compose
  3. Download terminology files mentioned in docs (I can supply if needed): https://hapifhir.io/hapi-fhir/docs/tools/hapi_fhir_cli.html , and put them in the relative path as shown in the makefile
  4. Run make upload-terminology, which runs commands mentioned in docs.

Expected behavior

These code systems / terminologies will be uploaded

Actual behavior

ICD10CM: Succeeds, but doesn't upload any concepts

hapi-fhir-cli upload-terminology -d ./data/value_sets/terminologies/LOINC/Loinc_2.71_MultiAxialHierarchy_3.5.zip -d ./data/value_sets/terminologies/ICD-10-CM/icd10cm_tabular_2021.xml -v r4 -t https://localhost:8080/fhir -u https://hl7.org/fhir/sid/icd-10-cm
------------------------------------------------------------
🔥  HAPI FHIR 5.6.0 - Command Line Tool
------------------------------------------------------------
Process ID                      : [email protected]
Max configured JVM memory (Xmx) : 8.0GB
Detected Java version           : 17.0.1
------------------------------------------------------------
2022-01-05 22:11:04.228 [main] INFO  ca.uhn.fhir.cli.App Logging configuration set from file logback-cli-on.xml
2022-01-05 22:11:05.47 [main] INFO  c.u.f.c.UploadTerminologyCommand Adding ZIP file: ./data/value_sets/terminologies/LOINC/Loinc_2.71_MultiAxialHierarchy_3.5.zip
2022-01-05 22:11:05.57 [main] INFO  c.u.f.c.UploadTerminologyCommand Compressing and adding file: ./data/value_sets/terminologies/ICD-10-CM/icd10cm_tabular_2021.xml
2022-01-05 22:11:05.186 [main] INFO  c.u.f.c.UploadTerminologyCommand Finished compressing ./data/value_sets/terminologies/ICD-10-CM/icd10cm_tabular_2021.xml
2022-01-05 22:11:05.188 [main] INFO  c.u.f.c.UploadTerminologyCommand Compressed 8.8 MB bytes in 0 file(s) into 780.7 kB bytes
2022-01-05 22:11:05.188 [main] INFO  c.u.f.c.UploadTerminologyCommand Beginning upload - This may take a while...
2022-01-05 22:11:07.891 [main] INFO  c.u.f.c.UploadTerminologyCommand Upload complete!
2022-01-05 22:11:07.908 [main] INFO  c.u.f.c.UploadTerminologyCommand Response:
<Parameters xmlns="https://hl7.org/fhir">
   <parameter>
      <name value="success"/>
      <valueBoolean value="true"/>
   </parameter>
   <parameter>
      <name value="conceptCount"/>
      <valueInteger value="45011"/>
   </parameter>
   <parameter>
      <name value="target"/>
      <valueReference>
         <reference value="CodeSystem/1"/>
      </valueReference>
   </parameter>
</Parameters>
2022-01-05 22:11:07.908 [main] INFO  ca.uhn.fhir.cli.App HAPI FHIR is shutting down...

https://20.119.216.32:8080/fhir/CodeSystem/1?_format=json

{
  "resourceType": "CodeSystem",
  "id": "1",
  "meta": {
    "versionId": "1",
    "lastUpdated": "2022-02-16T23:56:06.161+00:00",
    "source": "#hoo7Oyh75NdyGFOP"
  },
  "url": "https://hl7.org/fhir/sid/icd-10-cm",
  "version": "2021",
  "name": "ICD-10-CM",
  "status": "active",
  "content": "not-present"
}

LOINC

After about 2 minutes, got error:

hapi-fhir-cli upload-terminology -d ./data/value_sets/terminologies/LOINC/Loinc_2.71_MultiAxialHierarchy_3.5.zip -d /Users/joeflack4/projects/hapi-fhir-jpaserver-starter/data/value_sets/terminologies/LOINC/_archive/_Full/Loinc_2.71.zip -v r4 -t https://localhost:8080/fhir -u https://loinc.org
------------------------------------------------------------
🔥  HAPI FHIR 5.6.0 - Command Line Tool
------------------------------------------------------------
Process ID                      : [email protected]
Max configured JVM memory (Xmx) : 8.0GB
Detected Java version           : 17.0.1
------------------------------------------------------------
2022-01-05 22:07:23.819 [main] INFO  ca.uhn.fhir.cli.App Logging configuration set from file logback-cli-on.xml
2022-01-05 22:07:24.800 [main] INFO  c.u.f.c.UploadTerminologyCommand Adding ZIP file: ./data/value_sets/terminologies/LOINC/Loinc_2.71_MultiAxialHierarchy_3.5.zip
2022-01-05 22:07:24.812 [main] INFO  c.u.f.c.UploadTerminologyCommand Adding ZIP file: /Users/joeflack4/projects/hapi-fhir-jpaserver-starter/data/value_sets/terminologies/LOINC/_archive/_Full/Loinc_2.71.zip
2022-01-05 22:07:28.577 [main] INFO  c.u.f.c.UploadTerminologyCommand File size is greater than 10 MB - Going to use a local file reference instead of a direct HTTP transfer. Note that this will only work when executing this command on the same server as the FHIR server itself.
2022-01-05 22:07:30.833 [main] INFO  c.u.f.c.UploadTerminologyCommand Beginning upload - This may take a while...
2022-01-05 22:10:00.222 [main] ERROR c.u.f.c.UploadTerminologyCommand Received the following response:
{
  "resourceType": "OperationOutcome",
  "text": {
    "status": "generated",
    "div": "<div xmlns=\"https://www.w3.org/1999/xhtml\"><h1>Operation Outcome</h1><table border=\"0\"><tr><td style=\"font-weight: bold;\">ERROR</td><td>[]</td><td><pre>Unknown file: hapi-fhir-cli12803979877070915728.zip</pre></td>\n\t\t\t</tr>\n\t\t</table>\n\t</div>"
  },
  "issue": [ {
    "severity": "error",
    "code": "processing",
    "diagnostics": "Unknown file: hapi-fhir-cli12803979877070915728.zip"
  } ]
}
2022-01-05 22:10:00.226 [main] ERROR ca.uhn.fhir.cli.App Error during execution:
ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 : Unknown file: hapi-fhir-cli12803979877070915728.zip
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
	at ca.uhn.fhir.rest.server.exceptions.BaseServerResponseException.newInstance(BaseServerResponseException.java:305)
	at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:351)
	at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:540)
	at ca.uhn.fhir.rest.client.impl.GenericClient$OperationInternal.execute(GenericClient.java:1319)
	at ca.uhn.fhir.cli.UploadTerminologyCommand.invokeOperation(UploadTerminologyCommand.java:216)
	at ca.uhn.fhir.cli.UploadTerminologyCommand.run(UploadTerminologyCommand.java:123)
	at ca.uhn.fhir.cli.BaseApp.run(BaseApp.java:253)
	at ca.uhn.fhir.cli.App.main(App.java:43)

SNOMED

make upload-snomed
hapi-fhir-cli upload-terminology -d ./data/value_sets/terminologies/SNOMED/SnomedCT_USEditionRF2_PRODUCTION_20210901T120000Z.zip -v r4 -t https://localhost:8080/fhir -u https://snomed.info/sct
------------------------------------------------------------
🔥  HAPI FHIR 5.6.0 - Command Line Tool
------------------------------------------------------------
Process ID                      : [email protected]
Max configured JVM memory (Xmx) : 8.0GB
Detected Java version           : 17.0.1
------------------------------------------------------------
2022-01-05 22:25:03.428 [main] INFO  ca.uhn.fhir.cli.App Logging configuration set from file logback-cli-on.xml
2022-01-05 22:25:04.332 [main] INFO  c.u.f.c.UploadTerminologyCommand Adding ZIP file: ./data/value_sets/terminologies/SNOMED/SnomedCT_USEditionRF2_PRODUCTION_20210901T120000Z.zip
2022-01-05 22:25:07.231 [main] INFO  c.u.f.c.UploadTerminologyCommand File size is greater than 10 MB - Going to use a local file reference instead of a direct HTTP transfer. Note that this will only work when executing this command on the same server as the FHIR server itself.
2022-01-05 22:25:08.526 [main] INFO  c.u.f.c.UploadTerminologyCommand Beginning upload - This may take a while...
2022-01-05 22:25:09.488 [main] ERROR c.u.f.c.UploadTerminologyCommand Received the following response:
{
  "resourceType": "OperationOutcome",
  "text": {
    "status": "generated",
    "div": "<div xmlns=\"https://www.w3.org/1999/xhtml\"><h1>Operation Outcome</h1><table border=\"0\"><tr><td style=\"font-weight: bold;\">ERROR</td><td>[]</td><td><pre>Unknown file: hapi-fhir-cli2959849804250568432.zip</pre></td>\n\t\t\t</tr>\n\t\t</table>\n\t</div>"
  },
  "issue": [ {
    "severity": "error",
    "code": "processing",
    "diagnostics": "Unknown file: hapi-fhir-cli2959849804250568432.zip"
  } ]
}
2022-01-05 22:25:09.491 [main] ERROR ca.uhn.fhir.cli.App Error during execution:
ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 : Unknown file: hapi-fhir-cli2959849804250568432.zip
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
	at ca.uhn.fhir.rest.server.exceptions.BaseServerResponseException.newInstance(BaseServerResponseException.java:305)
	at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:351)
	at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:540)
	at ca.uhn.fhir.rest.client.impl.GenericClient$OperationInternal.execute(GenericClient.java:1319)
	at ca.uhn.fhir.cli.UploadTerminologyCommand.invokeOperation(UploadTerminologyCommand.java:216)
	at ca.uhn.fhir.cli.UploadTerminologyCommand.run(UploadTerminologyCommand.java:123)
	at ca.uhn.fhir.cli.BaseApp.run(BaseApp.java:253)
	at ca.uhn.fhir.cli.App.main(App.java:43)
2022-01-05 22:25:09.493 [Thread-0] INFO  ca.uhn.fhir.cli.App HAPI FHIR is shutting down...
make: *** [upload-snomed] Error 1

Environment

  • HAPI FHIR Version: R4
  • OS: MacOs 11.4

Possible causes

Docker:
Could be because using Docker, but the terminology files we're trying to import exist outside the container.

Related issues

  1. This is also about LOINC, but a different error: error importing LOINC terminology with hapi-fhir-cli  #1629
  2. Error when uploading terminology in HAPI FHIR R4 server using hapi-fhir-cli tool hapi-fhir-jpaserver-starter#193
  3. Unknown file: hapi-fhir-cli....zip when uploading terminology hapi-fhir-jpaserver-starter#322
@joeflack4
Copy link
Author

ezgif-1-aca0546a61

@mcabello-cens
Copy link

mcabello-cens commented Mar 17, 2022

Why ICD10CM source uses a Loinc_2.71_MultiAxialHierarchy_3.5.zip file in the command line. Is it an error?
https://hapifhir.io/hapi-fhir/docs/tools/hapi_fhir_cli.html shows the same information.

@joeflack4
Copy link
Author

I think it might be a documentation error. I tried with and without, and had an error.

I think my colleague may have been able to successfully import all of these, but only if the upload was executed from the same server that the terminology service is running on. I think that in some cases where we were uploading from our local machine, we would receive an error message telling us that we needed to run on the same server.

Also, I think the SmileCDR documentation might have some better docs for using the hapi-fhir-cli than HAPI does.

@joeflack4
Copy link
Author

joeflack4 commented Apr 22, 2022

I also tried using the SmileCDR docs to try and find a way to use hapi-fhir-cli successfully. For SNOMED, ICD10CM, and LOINC, only had luck w/ ICD10CM (same as before).

I tried doing an extremely simple custom code system (only 2kb, attached), but no luck.

Attempt 1 : hapi-fhir-cli

make upload-custom1
hapi-fhir-cli upload-terminology -d "/Users/joeflack4/projects/hapi-fhir-jpaserver-starter/data/custom1/custom1.zip" -v r4 -t https://20.119.216.32:8080/fhir -u https://20.119.216.32:8080/fhir/CodeSystem/2
------------------------------------------------------------
🔥  HAPI FHIR 5.7.0 - Command Line Tool
------------------------------------------------------------
Process ID                      : [email protected]
Max configured JVM memory (Xmx) : 8.0GB
Detected Java version           : 18
------------------------------------------------------------
2022-04-28 18:59:22.537 [main] INFO  ca.uhn.fhir.cli.App Logging configuration set from file logback-cli-on.xml
2022-04-28 18:59:23.257 [main] INFO  c.u.f.c.UploadTerminologyCommand Adding ZIP file: /Users/joeflack4/projects/hapi-fhir-jpaserver-starter/data/custom1/custom1.zip
2022-04-28 18:59:23.265 [main] INFO  c.u.f.c.UploadTerminologyCommand Beginning upload - This may take a while...
2022-04-28 18:59:23.900 [main] ERROR c.u.f.c.UploadTerminologyCommand Received the following response:
{
  "resourceType": "OperationOutcome",
  "text": {
    "status": "generated",
    "div": "<div xmlns=\"https://www.w3.org/1999/xhtml\"><h1>Operation Outcome</h1><table border=\"0\"><tr><td style=\"font-weight: bold;\">ERROR</td><td>[]</td><td><pre>Failed to call access method: java.lang.IllegalArgumentException: Mapping for CODE not found, expected one of [CODE\tDISPLAY]</pre></td>\n\t\t\t</tr>\n\t\t</table>\n\t</div>"
  },
  "issue": [ {
    "severity": "error",
    "code": "processing",
    "diagnostics": "Failed to call access method: java.lang.IllegalArgumentException: Mapping for CODE not found, expected one of [CODE\tDISPLAY]"
  } ]
}
2022-04-28 18:59:23.901 [main] ERROR ca.uhn.fhir.cli.App Error during execution:
ca.uhn.fhir.rest.server.exceptions.InternalErrorException: HTTP 500 : Failed to call access method: java.lang.IllegalArgumentException: Mapping for CODE not found, expected one of [CODE	DISPLAY]
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:483)
	at ca.uhn.fhir.rest.server.exceptions.BaseServerResponseException.newInstance(BaseServerResponseException.java:310)
	at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:376)
	at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:541)
	at ca.uhn.fhir.rest.client.impl.GenericClient$OperationInternal.execute(GenericClient.java:1320)
	at ca.uhn.fhir.cli.UploadTerminologyCommand.invokeOperation(UploadTerminologyCommand.java:217)
	at ca.uhn.fhir.cli.UploadTerminologyCommand.run(UploadTerminologyCommand.java:124)
	at ca.uhn.fhir.cli.BaseApp.run(BaseApp.java:254)
	at ca.uhn.fhir.cli.App.main(App.java:43)
2022-04-28 18:59:23.902 [Thread-0] INFO  ca.uhn.fhir.cli.App HAPI FHIR is shutting down...
make: *** [upload-custom1] Error 1

custom1.zip

Attempt 2 : curl

command: curl -X POST -H "Content-Type: application/json" -d @data/custom1/params.json "https://20.119.216.32:8080/fhir/CodeSystem/$upload-external-codesystem"

results: Failed to parse request body as JSON resource. Error was: Incorrect resource type found, expected \"CodeSystem\" but found \"Parameters\

curl -X POST -H "Content-Type: application/json" -d @data/custom1/params.json "https://20.119.216.32:8080/fhir/CodeSystem/$upload-external-codesystem"
{
  "resourceType": "OperationOutcome",
  "text": {
    "status": "generated",
    "div": "<div xmlns=\"https://www.w3.org/1999/xhtml\"><h1>Operation Outcome</h1><table border=\"0\"><tr><td style=\"font-weight: bold;\">ERROR</td><td>[]</td><td><pre>Failed to parse request body as JSON resource. Error was: Incorrect resource type found, expected &quot;CodeSystem&quot; but found &quot;Parameters&quot;</pre></td>\n\t\t\t</tr>\n\t\t</table>\n\t</div>"
  },
  "issue": [ {
    "severity": "error",
    "code": "processing",
    "diagnostics": "Failed to parse request body as JSON resource. Error was: Incorrect resource type found, expected \"CodeSystem\" but found \"Parameters\""
  } ]
}% 

custom1.zip
params.json.zip

@mcabello-cens
Copy link

For me, it works the following things:

concepts
CODE,DISPLAY
1,AAAAA
2,BBBBBB

hierarchy
PARENT,CHILD
1,2

Uploading
hapi-fhir-cli upload-terminology -d custom1.zip -v r4 -t https://localhost:8080/fhir -u https://acme.com/my-own-codesystem

@joeflack4
Copy link
Author

That looks like almost exactly like my setup.

Did you include the codesystem.json that is mentioned in the SmileCDR docs (I don't think normal HAPI has docs for this)?

It said to include like so:

{
	"resourceType": "CodeSystem",
	"url": "https://20.119.216.32:8080/fhir/CodeSystem/2",
	"name": "Attempt1",
	"description": "I really hope this works",
	"status": "active",
	"publisher": "TIMS",
	"date": "2022-04-21",
	"content": "not-present"
}

Otherwise my command is exactly the same as yours / basically the same, depending how you look at it. I wonder if it didn't like one of my params, but it didn't give me any error message to indicate that.

hapi-fhir-cli upload-terminology -d "/Users/joeflack4/projects/hapi-fhir-jpaserver-starter/data/custom1/custom1.zip" -v r4 -t https://20.119.216.32:8888/ts1/fhir -u https://20.119.216.32:8080/fhir/CodeSystem/2

Maybe it didn't like me assigning my own canonical URL like that. Maybe htat's why it gave me:
ca.uhn.fhir.rest.server.exceptions.ResourceNotFoundException: HTTP 404 Not Found

Can't check at the moment because I'm not on the right PC, but I suppose I'll try changing that.

@mcabello-cens Do you have a link to your repo/branch? My fork is only from around December 2021, and I haven't updated it since. Perhaps yours is more recent?

@joeflack4
Copy link
Author

@mcabello-cens For uploading custom CSVs the way you did it, I finally got it to work:
hapi-fhir-cli upload-terminology -d "/Users/joeflack4/projects/hapi-fhir-jpaserver-starter/data/custom1/custom1.zip" -v r4 -t https://20.119.216.32:8080/fhir -u https://20.119.216.32:8080/fhir/CodeSystem/2

The problem I had with this particular method of uploading custom CSVs was with my CSVs. Somehow when I saved them from Excel, they somehow became tab-separated instead of comma-separated.

There are still issues I'm having with the hapi-fhir-cli documentation and upload of SNOMED and LOINC, so I'll leave this issue open.

@joeflack4
Copy link
Author

joeflack4 commented May 4, 2022

@mcabello-cens Let me know if you know someone that I should @ on this topic, as nobody else has responded yet.

I believe that @ShahimEssaid has located the issue.

In a private message, he said:

The CLI creates a /tmp/... .file for local reference for large files and then sends this file path to the server for loading. The server needs to be able to find this file. If the sever is running in docker or some other virtual environment, the CLI and all the needed files have to be in that environment, i.e. in the running docker container of the HAPI server if it's in docker. Otherwise, the /tmp/... path won't work.

I think this explains why I'm getting this as my error:

"diagnostics": "Unknown file: hapi-fhir-cli12803979877070915728.zip"

Additional attempts w/ some --debug

With debugging:

shahim@ub2004:/opt/shahim/timsts1$ ./hapi-fhir-cli upload-terminology --debug -d vocabs/Loinc_2.72.zip -v r4 -t https://hm.essaid.com:8888/timsts1/r4 -u https://loinc.org/
------------------------------------------------------------
:fire:  HAPI FHIR 5.7.0 - Command Line Tool
------------------------------------------------------------
Process ID                      : 30081@ub2004
Max configured JVM memory (Xmx) : 3.9GB
Detected Java version           : 11.0.15
------------------------------------------------------------
2022-05-04 05:33:58.928 [main] INFO  ca.uhn.fhir.cli.App Logging configuration set from file logback-cli-on.xml
2022-05-04 05:33:58.941 [main] [BaseApp.java:401] INFO  ca.uhn.fhir.cli.App Logging configuration set from file logback-cli-on-debug.xml
2022-05-04 05:33:58.941 [main] [BaseApp.java:406] INFO  ca.uhn.fhir.cli.App Debug logging is enabled
2022-05-04 05:33:58.952 [main] [FhirContext.java:208] INFO  c.u.f.c.FhirContext Creating new FHIR context for FHIR version [R4]
2022-05-04 05:33:59.168 [main] [DependencyLogImpl.java:73] DEBUG c.u.f.util.XmlUtil FHIR XML procesing will use StAX implementation at jar:file:/opt/shahim/timsts1/hapi-fhir-cli.jar!/
  Title:         null
  Symbolic name: null
  Vendor:        null
  Version:       null
2022-05-04 05:33:59.553 [main] [ModelScanner.java:243] DEBUG c.u.f.c.ModelScanner Scanning datatype class: org.hl7.fhir.r4.model.Dosage
2022-05-04 05:33:59.566 [main] [ModelScanner.java:243] DEBUG c.u.f.c.ModelScanner Scanning datatype class: org.hl7.fhir.r4.model.RelatedArtifact
2022-05-04 05:33:59.567 [main] [ModelScanner.java:243] DEBUG c.u.f.c.ModelScanner Scanning datatype class: org.hl7.fhir.r4.model.Attachment

....

2022-05-04 05:34:00.375 [main] [UploadTerminologyCommand.java:204] INFO  c.u.f.c.UploadTerminologyCommand Beginning upload - This may take a while...
2022-05-04 05:34:00.518 [main] [UploadTerminologyCommand.java:207] INFO  c.u.f.c.UploadTerminologyCommand Submitting parameters: {
  "resourceType": "Parameters",
  "parameter": [ {
    "name": "system",
    "valueUri": "https://loinc.org/"
  }, {
    "name": "file",
    "valueAttachment": {
      "url": "localfile:/tmp/hapi-fhir-cli17837746554575883423.zip"
    }
  } ]
}

....
2022-05-04 05:34:01.577 [main] [UploadTerminologyCommand.java:220] ERROR c.u.f.c.UploadTerminologyCommand Received the following response:
{
  "resourceType": "OperationOutcome",
  "issue": [ {
    "severity": "error",
    "code": "processing",
    "diagnostics": "HAPI-1141: Unknown file: hapi-fhir-cli17837746554575883423.zip"
  } ]
}
2022-05-04 05:34:01.586 [main] [BaseApp.java:278] ERROR ca.uhn.fhir.cli.App Error during execution:
ca.uhn.fhir.rest.server.exceptions.InvalidRequestException: HTTP 400 Bad Request: HAPI-1141: Unknown file: hapi-fhir-cli17837746554575883423.zip
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at ca.uhn.fhir.rest.server.exceptions.BaseServerResponseException.newInstance(BaseServerResponseException.java:310)
        at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:376)
        at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:541)
        at ca.uhn.fhir.rest.client.impl.GenericClient$OperationInternal.execute(GenericClient.java:1320)
        at ca.uhn.fhir.cli.UploadTerminologyCommand.invokeOperation(UploadTerminologyCommand.java:217)
        at ca.uhn.fhir.cli.UploadTerminologyCommand.run(UploadTerminologyCommand.java:124)
        at ca.uhn.fhir.cli.BaseApp.run(BaseApp.java:254)
        at ca.uhn.fhir.cli.App.main(App.java:43)
2022-05-04 05:34:01.587 [Thread-0] [BaseApp.java:364] INFO  ca.uhn.fhir.cli.App HAPI FHIR is shutting down...

Entering the container to run inside, and get an out of memory error:

root@ub2004:/opt/shahim/wf-compose# docker-compose exec wildfly bash
[jboss@edba5cfba059 ~]$ cd /opt/shahim/timsts1/
[jboss@edba5cfba059 timsts1]$ ll
total 1943764
-rw-rw-r-- 1 jboss jboss 655058439 May  4 03:36 drive-download-20220503T173516Z-001.zip
-rw-rw-r-- 1 jboss jboss 534202688 May  4 03:36 hapi-fhir-5.7.0-cli.zip
-r-xr-xr-x 1 jboss jboss      4354 Nov 20  2020 hapi-fhir-cli
-r-xr-xr-x 1 jboss jboss      3861 Nov 20  2020 hapi-fhir-cli.cmd
-rw-r--r-- 1 jboss jboss 551990579 Feb 22 10:15 hapi-fhir-cli.jar
drwxrwxr-x 2 jboss jboss      4096 May  4 05:30 hold
-rw-rw-r-- 1 jboss jboss     30722 May  4 05:34 output.log
-rw-rw-r-- 1 jboss jboss       880 May  4 03:42 timsts1-notes-home.txt
-rw-rw-r-- 1 jboss jboss 249085866 May  4 03:43 timsts1.war
drwxrwxr-x 2 jboss jboss      4096 May  4 05:29 vocabs
[jboss@edba5cfba059 timsts1]$ cat timsts1-notes-home.txt
These worked in the past on Windows:
./hapi-fhir-cli upload-terminology -d Loinc_2.72.zip -v r4 -t https://localhost:8080/fhir -u https://loinc.org/
./hapi-fhir-cli upload-terminology -d icd10cm_tabular_2022.xml -v r4 -t https://localhost:8080/fhir -u https://hl7.org/fhir/sid/icd-10-cm
./hapi-fhir-cli upload-terminology -d SnomedCT_USEditionRF2_PRODUCTION_20210901T120000Z.zip -v r4 -t https://localhost:8080/fhir -u https://snomed.info/sct

For timsts1:
./hapi-fhir-cli upload-terminology -d Loinc_2.72.zip -v r4 -t https://hm.essaid.com:8888/timsts1/r4 -u https://loinc.org/
./hapi-fhir-cli upload-terminology -d icd10cm_tabular_2022.xml -v r4 -t https://hm.essaid.com:8888/timsts1/r4 -u https://hl7.org/fhir/sid/icd-10-cm
./hapi-fhir-cli upload-terminology -d SnomedCT_USEditionRF2_PRODUCTION_20210901T120000Z.zip -v r4 -t https://hm.essaid.com:8888/timsts1/r4 -u https://snomed.info/sct
[jboss@edba5cfba059 timsts1]$ java --version
openjdk 11.0.8 2020-07-14 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.8+10-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.8+10-LTS, mixed mode, sharing)
[jboss@edba5cfba059 timsts1]$ ./hapi-fhir-cli upload-terminology -d vocabs/Loinc_2.72.zip -v r4 -t https://hm.essaid.com:8888/timsts1/r4 -u https://loinc.org/
------------------------------------------------------------
?  HAPI FHIR 5.7.0 - Command Line Tool
------------------------------------------------------------
Process ID                      : 682@edba5cfba059
Max configured JVM memory (Xmx) : 3.9GB
Detected Java version           : 11.0.8
------------------------------------------------------------
2022-05-04 05:43:25.383 [main] INFO  ca.uhn.fhir.cli.App Logging configuration set from file logback-cli-on.xml
2022-05-04 05:43:26.194 [main] INFO  c.u.f.c.UploadTerminologyCommand Adding ZIP file: vocabs/Loinc_2.72.zip
2022-05-04 05:43:26.377 [main] INFO  c.u.f.c.UploadTerminologyCommand File size is greater than 10 MB - Going to use a local file reference instead of a direct HTTP transfer. Note that this will only work when executing this command on the same server as the FHIR server itself.
2022-05-04 05:43:26.863 [main] INFO  c.u.f.c.UploadTerminologyCommand Beginning upload - This may take a while...
2022-05-04 05:44:15.221 [main] ERROR c.u.f.c.UploadTerminologyCommand Received the following response:
{
  "resourceType": "OperationOutcome",
  "issue": [ {
    "severity": "error",
    "code": "processing",
    "diagnostics": "HAPI-0389: Failed to call access method: java.lang.OutOfMemoryError: Java heap space"
  } ]
}
2022-05-04 05:44:15.223 [main] ERROR ca.uhn.fhir.cli.App Error during execution:
ca.uhn.fhir.rest.server.exceptions.InternalErrorException: HTTP 500 Internal Server Error: HAPI-0389: Failed to call access method: java.lang.OutOfMemoryError: Java heap space
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at ca.uhn.fhir.rest.server.exceptions.BaseServerResponseException.newInstance(BaseServerResponseException.java:310)
        at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:376)
        at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:541)
        at ca.uhn.fhir.rest.client.impl.GenericClient$OperationInternal.execute(GenericClient.java:1320)
        at ca.uhn.fhir.cli.UploadTerminologyCommand.invokeOperation(UploadTerminologyCommand.java:217)
        at ca.uhn.fhir.cli.UploadTerminologyCommand.run(UploadTerminologyCommand.java:124)
        at ca.uhn.fhir.cli.BaseApp.run(BaseApp.java:254)
        at ca.uhn.fhir.cli.App.main(App.java:43)
2022-05-04 05:44:15.225 [Thread-0] INFO  ca.uhn.fhir.cli.App HAPI FHIR is shutting down...

Source of the problem

Therefore, the probable source of the problem is:
hapi-fhir-cli is not working well with Docker as explained above.

Possible solutions

If at all possible, hapi-fhir-cli should be able to access the file system outside of the docker container. If that's not possible, then as an alternative: (i) should provide an error message explaining the source of the issue, and how to fix it, (ii) include information about necessary setup (e.g. utilization of docker volumes, if necessary) in the hapi-fhir-cli documentation.

@psavva
Copy link

psavva commented Sep 7, 2022

I have run into this exact same problem.
Solution: Create a dockerfile and COPY the HapiFhir CLI together with the terminology files (or alternatively using a volume mount).

The important thing is that the cli and files are accessible within the container.
then run the command directly within the container.

I created a temporary utility container just for this, and uploading of the latest Snomed worked just fine.

My dockerfile:

FROM maven:3.8-openjdk-17-slim as build-hapi
WORKDIR /tmp/hapi-fhir-jpaserver-starter

ARG OPENTELEMETRY_JAVA_AGENT_VERSION=1.15.0
RUN curl -LSsO https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v${OPENTELEMETRY_JAVA_AGENT_VERSION}/opentelemetry-javaagent.jar

COPY pom.xml .
COPY server.xml .
RUN mvn -ntp dependency:go-offline

COPY src/ /tmp/hapi-fhir-jpaserver-starter/src/
COPY src/main/resources/application.yaml /tmp/hapi-fhir-jpaserver-starter/src/main/resources/application.yaml
#COPY src/main/resources/custom.hapi.properties /tmp/hapi-fhir-jpaserver-starter/src/main/resources/custom.hapi.properties
#RUN rm /tmp/hapi-fhir-jpaserver-starter/src/main/resources/application.yaml

RUN mvn clean install -DskipTests -Djdk.lang.Process.launchMechanism=vfork

FROM build-hapi AS build-distroless
RUN mvn package spring-boot:repackage -Pboot
RUN mkdir /app && cp /tmp/hapi-fhir-jpaserver-starter/target/ROOT.war /app/main.war


########### tomcat version is suitable for debugging and comes with a shell
########### it can be built using eg. `docker build --target tomcat .`
FROM tomcat:8.5-jdk11-openjdk-slim as tomcat

RUN mkdir /opt/hapi-cli
COPY hapi-fhir-6.0.1-cli /opt/hapi-cli
######## Note that the hapi-fhir-6.0.1-cli folder also contains my terminology zip files in there...

RUN mkdir /app
WORKDIR /app

COPY --from=build-hapi /tmp/hapi-fhir-jpaserver-starter/target/ROOT.war /usr/local/tomcat/webapps/
COPY --from=build-hapi /tmp/hapi-fhir-jpaserver-starter/opentelemetry-javaagent.jar /usr/local/tomcat/webapps/

EXPOSE 8080

CMD ["catalina.sh", "run"]

and finally load it:

java -jar hapi-fhir-cli.jar upload-terminology -v r4 -t https://localhost:8080/fhir -u https://loinc.org -d SnomedCT_InternationalRF2_PRODUCTION_20220731T120000Z.zip

@tadgh
Copy link
Collaborator

tadgh commented Sep 17, 2022

Heya, just chiming in here: the HAPI-FHIR CLI loader has a hard-coded 10mb limit to uploads, until it decides to use a local file descriptor instead. When it does this, it sends the local file location to the remote server, which in turn fails to resolve it (because it doesnt exist on the remote system). I'll be writing a pull request to provide a CLI argument to increase the maximum size you can send before it attempts to use a local file descriptor instead.

@joeflack4
Copy link
Author

That would be great.

@brauchlik
Copy link

I was running into the same Problem. In Version 6.4.4 of the hapi-fhir-cli I found a switch to set the max upload filesize

adding -s 10GB to the command did the trick and let me upload Loinc-2.73.zip with no problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants