Python json -> protobuf "cannot be parsed as a reference"

[gitpushdashf]

The latest Synthea gives different output. An older version works fine for me, however the latest results in:

ValueError: String 'Practitioner?identifier=http:https://hl7.org/fhir/sid/us-npi|9999999959' cannot be parsed as a reference.

I can write up a code sample if need be.

This is using Synthea from master: https://github.com/synthetichealth/synthea/releases/download/master-branch-latest/synthea-with-dependencies.jar

It looks like they have some versions released more recently, I will try those.

[Cam2337]

Interesting, yes a code sample would be fantastic!

...An older version works fine for me, however the latest results in...

Just so I'm clear, you're saying an older Synthea works o.k.? What's the version that was working for you?

Thanks for bringing it to our attention!

[gitpushdashf]

Yeah, an older one works fine. Synthea wasn't tagging releases for over a year and the only (easy) option was downloading the .jar made from master.

https://github.com/synthetichealth/synthea/releases

Now it turns out that in September they made releases, but those releases seem to be build for openjdk12 and not openjdk8... synthetichealth/synthea#813

Anyway, here's a snippet of the new output.

            "who": {
              "reference": "Practitioner?identifier=http:https://hl7.org/fhir/sid/us-npi%7C9999948299",
              "display": "Dr. Dorothea248 Huel628"
            },
            "onBehalfOf": {
              "reference": "Organization?identifier=https://github.com/synthetichealth/synthea%7Cde004a50-64ca-37eb-bf70-d5589fa7999f",
              "display": "PCP149816"
            }
          },
          {
            "type": {
              "coding": [
                {
                  "system": "http:https://hl7.org/fhir/us/core/CodeSystem/us-core-provenance-participant-type",
                  "code": "transmitter",
                  "display": "Transmitter"
                }
              ],
              "text": "Transmitter"
            },
            "who": {
              "reference": "Practitioner?identifier=http:https://hl7.org/fhir/sid/us-npi%7C9999948299",
              "display": "Dr. Dorothea248 Huel628"
            },
            "onBehalfOf": {
              "reference": "Organization?identifier=https://github.com/synthetichealth/synthea%7Cde004a50-64ca-37eb-bf70-d5589fa7999f",
              "display": "PCP149816"
            }
          }
        ]
      },
      "request": {
        "method": "POST",
        "url": "Provenance"
      }
    }
  ]
}

Thanks for getting back to me. Let me come up with a code snippet.

[gitpushdashf]

Ok, hopefully this helps. I made/activated a virtualenv, pip3 installed google-fhir and requests, then ran the script in an empty directory.

synthea.py.txt

Thanks!

[gitpushdashf]

I'm having some weird behavior. If I use just the old Synthea JAR file, it seems to generate good output. If I switch to the latest one, it breaks. If I switch back, it's still broken, maybe for hours. It's like Synthea caches something that's throwing it off.

[gitpushdashf]

e5545c13-98db-083f-f580-b22157aa82a8.json.txt

Here's one generated that I'm having issues with.

[gitpushdashf]

Ok, the old .jar actually does work fine -- I was screwing up. But the new jar does produce output that google-fhir can't handle.

I need to find somewhere to host that file.

[gitpushdashf]

Here's a Synthea version that is working.

https://github.com/gitpushdashf/synthea/releases/tag/v2.5.9

[Cam2337]

Thanks for the details! I believe that these patterns started being introduced in Synthea with SHA: b262437bce35bfec4e485e150832242b5a68581a (link).

Since Uri has a really permissive pattern (\S*; source), it seems like we're potentially being too strict in references.py if we're unable to split the relative reference.

Will look into correcting this behavior so that we're taking appropriate action.

[Cam2337]

Circling back on this -- the change was trivial but it required additional work to some things downstream. I have a fix internally now that's going through review.

In the meantime as we discussed you should be able to avoid any problematic Synthea by sticking to versions earlier than b262437bce35bfec4e485e150832242b5a68581a. Additionally, as a temporary workaround I believe you can instruct Synthea to not use transaction_bundle, which should avoid these logical Reference URLs, e.g.:

~/synthea$ ./run_synthea Massachusetts -p 10
~/synthea$ grep -R '"reference":' ./output/fhir/ | grep -E '[A-Z][a-z]+\?.*' | wc -l
6010
~/synthea$ rm ./output/fhir/*

~/synthea$ ./run_synthea Massachusetts --exporter.fhir.transaction_bundle false -p 10
~/synthea$ grep -R '"reference":' ./output/fhir/ | grep -E '[A-Z][a-z]+\?.*' | wc -l
0

See more at: https://github.com/synthetichealth/synthea/blob/master/src/main/resources/synthea.properties#L18

Appreciate the patience and help in identifying this! :)

[gitpushdashf]

Thank you! That's good to know.