Openemr fhir search (openemr#4349)

* Initial FHIR Search implementation.

Built out the FHIR search algorithm and implemented it in
FhirServiceBase and BaseService.

I built the search upon a 3 layer approach.  The FHIR layer that is
aware of FHIR search fields types, FHIR concepts, the OpenEMR Service
layer, and then the database / SQL layer.  I believe I've kept those
layers separate from each other with the bottom layers not making any
calls or dependencies on the top layers.

The FHIR search uses the factory design pattern to build a set of
ISearchField objects that represent the various types of search objects
in both FHIR and now in the OpenEMR service layers.  Currently the type
of search fields that are supported are the String and Date fields which are
almost fully supported with Token kinda working and the others not
supported at this point. The only thing missing in the Date search
type is timezone support which I haven't got a good solution to just
yet.

I also implemented an IPatientCompartmentResourceService that any FHIR
service that touches patient data should implement.  It specifies the
patient field that will bind a FHIR search query into the patient
context.  This will make sure that only a single user's patient data
will be returned when operating within just that patient's context.

I've gone through and added a bunch of tests in the
FhirPatientServiceQueryTest object that deals with the different
modifiers, and comparators that FHIR supports.  I abstracted it away so
that the same kinds of modifiers and comparators (greater than, less
than, etc) can be used in the service classes without knowledge of FHIR.

I built it so we can have composite search fields with nested
heirarchies to support all kinds of search operations that can combine
both unions and intersections of search fields.

There is also a big bug fix as the sqlThrowException operation was only
working on insert/update operations that expected no data back.  I fixed
a bug that prevented the operation from returning data.

* Fix the patient fixtures not cleaning.

* Fix coding standards, full patient search.

Got all of the required us core patient search and the original patient
field searches working and tested in the query search.

Looks like I broke a bunch of unit tests though and I'll have to go
about fixing those.

* Code style fixes, unit test fixes.

Got all of the unit tests back up and running, had to fix a number of
problems with how the search parameters were defined and setup.  Also
fixed some broken tests with the Practitioner service classes.  Put more
logic into the BaseService to handle and be able to override table joins
and selection of data from table joins.

I've partially implemented a very primitive ORM, I keep wondering if itd
be better just to incorporate a pre-existing ORM but it seems like the
project has put those in and ripped them out multiple times so I'm
hedging away from that.

Had to temporarily hard code the Practitioner search values (which we
were doing originally anyways) until I can get the npi:missing stuff
working for a search modifier.  The challenge is that the :missing
identifier overrides the search value to be a boolean true/false which
is going to kill a number of the searchfield data validators.

I'm debating on whether I will under the hood change the type to be a
SearchMissingField class instead of relying on whatever the actual
search type of class should be.  I see pros and cons to that approach.
I'll just have to play around with it and see what I end up with.

Overall this should be a pretty good initial stab at a more robust FHIR
searching solution that will pass ONC requirements.

* Fix php7.3 FHIR date search preg_match bug

php7.3 handles the preg_match criteria different than 7.4+.  Switching
to a !empty clause on each preg_match group that was found resolves this
issue.

* US Core resource endpoints, reference type.

Added a way for each resource to update, map, or change records
retrieved from the data layer.

Added the reference search field type enum.

Added the missing US Core entities and have them return empty responses
for now.

Enabled all of the patient us core entities scopes.

Alphabetized the rest routes for FHIR so we can easily locate the
routes.

* Fix uuid return on Patient Service.

* Initial method stub for reference type creation

* Better logging of Authorization Controller.

* Patient Provenance and US-Core Patient requirements.

Got the Single Patient API ONC Us Core Inferno test suite to pass for
the US Core Patient Profile.  All of the core search criteria is working
as well as the gender, language, and race core extensions.  The
communication is throwing a warning but we use the same data values for
CCDA so I believe this is correct, but the test still passes despite the
warning.

Also implemented the Patient Provenance.  We'll see if the
implementation still works in the future as I'm not sure if our
Provenance record needs to contain EVERY single entity for that
Provenance target... we'll have to see as that could be pretty big at
the organization level.

* Unit test fixes, ONC Patient API implementation.

Got the ONC Patient API implementation working correctly.  Had to fix a
number of unit tests that had broken in the various classes like
CarePlan,Device,DiagnosticReport,Goal,Organization.

Also fixed some code style issues.

Extended the List service to include retrieving records from
list_options table.

* Style fixes, Fix search params.

Had a unit test failing due to the _REWRITE_COMMAND get query string
added the get stuff to the HTTP Request object and exclude the server
openemr get params so we have a clean param list in our API requests.

* Renamed the file to fix psr4 errors.

* Fix PSR12 case statement style problems.

* One last style fix urgh.

* Initial search work for FHIR Allergy Intolerance.

Created a query utils to put common query methods in that can be used by
the project that also throw exceptions and are easier to catch / handle
in the framework (IE returning an operation outcome if something goes
wrong rather than just dieing).

Extended FixtureManager to create fixtures for Allergy Intolerance.  Not
sure I like it as we want something more extensible.

Started the initial underpinnings to support the allergy intolerance
searching with FHIR.  We will be doing a reference search so we will
have that piece supported soon.

* FHIR Reference Field, AllergyIntollerance Search.

Implemented the AllergyIntollerance search on patient.  Also implemented
the initial FHIR reference field.  Wrote unit tests to verify the FHIR
AllergyIntolerance is working properly.

* Finalized AllergyIntolerance and passed ONC tests.

Got the final pieces implemented in the AllergyIntolerance FHIR
resource.

Also fixed a bug in FacilityService with the primary organization query.

Fixed the getOne api signature on a number of the services.

* Fixed unit tests,added logging,query helper method

Added another query helper method.  Fixed the unit tests on allergy
intolerance.  Fixed some other unit tests to support an environment
variable on the admin user for unit tests since jerry's test database
uses a different admin account.

* Fixed lookup codes if codeset is not installed

We had an error on the continuous integration service where the code
sets for allergy-intolerance are not installed.  Fixed it so that it
doesn't throw the error if the codeset is not found.  I'm not sure if
the FHIR unit tests just never checked against snomed codes before, but
I find it strange/bizarre that this never threw errors beforehand.

* Style fixes

* Style fixes

* Fixed multiple value search conditions.

@brady.miller helped identify a problem that was breaking the unit
tests.  Found out that we weren't handling the OR condition on search
parameters that had multiple values.  Fixed the search fields to handle
that properly.

* Force patient binding, fixed missing phone numbers.

Tracked why the patient fixtures would fail the test cases sometimes.
Since the patient is chosen at random in some of the patient fixtures it
was messing up on the phone records.  Found there was a patient that did
not have all of the phone numbers specified.  Fixing that resolves the
test cases that were failing.

Co-authored-by: Stephen Nielson <[email protected]>

custom/code_types.inc.php

@@ -698,83 +698,100 @@ function lookup_code_descriptions($codes, $desc_detail = "code_text")
  }
 
  // added $modifier for HCPCS and other internal codesets so can grab exact entry in codes table
- list($codetype, $code, $modifier) = explode(':', $codestring);
+ $code_parts = explode(':', $codestring);
+ $codetype = $code_parts[0] ?? null;
+ $code = $code_parts[1] ?? null;
+ $modifier = $code_parts[2] ?? null;
+ // if we don't have the code types we can't do much here
+ if (!isset($code_types[$codetype])) {
+ // we can't do much so we will just continue here...
+ continue;
+ }
+
  $table_id = $code_types[$codetype]['external'] ?? '';
- if (isset($code_external_tables[$table_id])) {
- $table_info = $code_external_tables[$table_id];
- $table_name = $table_info[EXT_TABLE_NAME];
- $code_col = $table_info[EXT_COL_CODE];
- $desc_col = $table_info[DISPLAY_DESCRIPTION] == "" ? $table_info[EXT_COL_DESCRIPTION] : $table_info[DISPLAY_DESCRIPTION];
- $desc_col_short = $table_info[DISPLAY_DESCRIPTION] == "" ? $table_info[EXT_COL_DESCRIPTION_BRIEF] : $table_info[DISPLAY_DESCRIPTION];
- $sqlArray = array();
- $sql = "SELECT " . $desc_col . " as code_text," . $desc_col_short . " as code_text_short FROM " . $table_name;
-
- // include the "JOINS" so that we get the preferred term instead of the FullySpecifiedName when appropriate.
- foreach ($table_info[EXT_JOINS] as $join_info) {
- $join_table = $join_info[JOIN_TABLE];
- $check_table = sqlQuery("SHOW TABLES LIKE '" . $join_table . "'");
- if ((empty($check_table))) {
- HelpfulDie("Missing join table in code set search:" . $join_table);
- }
+ if (!isset($code_external_tables[$table_id])) {
+ //using an external code that is not yet supported, so skip.
+ continue;
+ }
+
+ $table_info = $code_external_tables[$table_id];
+ $table_name = $table_info[EXT_TABLE_NAME];
+ $code_col = $table_info[EXT_COL_CODE];
+ $desc_col = $table_info[DISPLAY_DESCRIPTION] == "" ? $table_info[EXT_COL_DESCRIPTION] : $table_info[DISPLAY_DESCRIPTION];
+ $desc_col_short = $table_info[DISPLAY_DESCRIPTION] == "" ? $table_info[EXT_COL_DESCRIPTION_BRIEF] : $table_info[DISPLAY_DESCRIPTION];
+ $sqlArray = array();
+ $sql = "SELECT " . $desc_col . " as code_text," . $desc_col_short . " as code_text_short FROM " . $table_name;
 
-  $sql .= " INNER JOIN " . $join_table;
-  $sql .= " ON ";
-  $not_first = false;
-  foreach ($join_info[JOIN_FIELDS] as $field) {
-  if ($not_first) {
-  $sql .= " AND ";
-  }
+ // include the "JOINS" so that we get the preferred term instead of the FullySpecifiedName when appropriate.
+ foreach ($table_info[EXT_JOINS] as $join_info) {
+ $join_table = $join_info[JOIN_TABLE];
+ $check_table = sqlQuery("SHOW TABLES LIKE '" . $join_table . "'");
+ if ((empty($check_table))) {
+ HelpfulDie("Missing join table in code set search:" . $join_table);
+ }
 
- $sql .= $field;
- $not_first = true;
+ $sql .= " INNER JOIN " . $join_table;
+ $sql .= " ON ";
+ $not_first = false;
+ foreach ($join_info[JOIN_FIELDS] as $field) {
+ if ($not_first) {
+ $sql .= " AND ";
  }
+
+ $sql .= $field;
+ $not_first = true;
  }
+ }
 
-  $sql .= " WHERE ";
+ $sql .= " WHERE ";
 
 
-  // Start building up the WHERE clause
+ // Start building up the WHERE clause
 
-  // When using the external codes table, we have to filter by the code_type. (All the other tables only contain one type)
-  if ($table_id == 0) {
-  $sql .= " code_type = '" . add_escape_custom($code_types[$codetype]['id']) . "' AND ";
-  }
+ // When using the external codes table, we have to filter by the code_type. (All the other tables only contain one type)
+ if ($table_id == 0) {
+ $sql .= " code_type = '" . add_escape_custom($code_types[$codetype]['id']) . "' AND ";
+ }
 
-  // Specify the code in the query.
-  $sql .= $table_name . "." . $code_col . "=? ";
-  array_push($sqlArray, $code);
+ // Specify the code in the query.
+ $sql .= $table_name . "." . $code_col . "=? ";
+ array_push($sqlArray, $code);
 
-  // Add the modifier if necessary for CPT and HCPCS which differentiates code
-  if ($modifier) {
-  $sql .= " AND modifier = ? ";
-  array_push($sqlArray, $modifier);
-  }
+ // Add the modifier if necessary for CPT and HCPCS which differentiates code
+ if ($modifier) {
+ $sql .= " AND modifier = ? ";
+ array_push($sqlArray, $modifier);
+ }
 
-  // We need to include the filter clauses
-  // For SNOMED and SNOMED-CT this ensures that we get the Preferred Term or the Fully Specified Term as appropriate
-  // It also prevents returning "inactive" results
-  foreach ($table_info[EXT_FILTER_CLAUSES] as $filter_clause) {
-  $sql .= " AND " . $filter_clause;
-  }
+ // We need to include the filter clauses
+ // For SNOMED and SNOMED-CT this ensures that we get the Preferred Term or the Fully Specified Term as appropriate
+ // It also prevents returning "inactive" results
+ foreach ($table_info[EXT_FILTER_CLAUSES] as $filter_clause) {
+ $sql .= " AND " . $filter_clause;
+ }
 
-  // END building the WHERE CLAUSE
+ // END building the WHERE CLAUSE
 
 
- if ($table_info[EXT_VERSION_ORDER]) {
- $sql .= " ORDER BY " . $table_info[EXT_VERSION_ORDER];
- }
+ if ($table_info[EXT_VERSION_ORDER]) {
+ $sql .= " ORDER BY " . $table_info[EXT_VERSION_ORDER];
+ }
+
+ // END building the WHERE CLAUSE
 
- $sql .= " LIMIT 1";
- $crow = sqlQuery($sql, $sqlArray);
- if (!empty($crow[$desc_detail])) {
- if ($code_text) {
- $code_text .= '; ';
- }
 
- $code_text .= $crow[$desc_detail];
+ if ($table_info[EXT_VERSION_ORDER]) {
+ $sql .= " ORDER BY " . $table_info[EXT_VERSION_ORDER];
+ }
+
+ $sql .= " LIMIT 1";
+ $crow = sqlQuery($sql, $sqlArray);
+ if (!empty($crow[$desc_detail])) {
+ if ($code_text) {
+ $code_text .= '; ';
  }
- } else {
- //using an external code that is not yet supported, so skip.
+
+ $code_text .= $crow[$desc_detail];
  }
  }
  }

library/sql.inc

@@ -161,8 +161,9 @@ function sqlStatementThrowException($statement, $binds = false)
  // Execute function.
  $recordset = $GLOBALS['adodb']['db']->Execute($statement, $binds, true);
  if ($recordset === false) {
- throw new \OpenEMR\Common\Database\SqlQueryException($statement, "Failed to execute statement. Error: " . getSqlLastError());
+ throw new \OpenEMR\Common\Database\SqlQueryException($statement, "Failed to execute statement. Error: " . getSqlLastError() . " Statement: " . $statement);
  }
+ return $recordset;
 }
 
 /**

src/Common/Auth/OpenIDConnect/Grant/CustomPasswordGrant.php

@@ -21,6 +21,7 @@
 use League\OAuth2\Server\Repositories\UserRepositoryInterface;
 use League\OAuth2\Server\RequestEvent;
 use League\OAuth2\Server\ResponseTypes\ResponseTypeInterface;
+use OpenEMR\Common\Logging\SystemLogger;
 use Psr\Http\Message\ServerRequestInterface;
 
 class CustomPasswordGrant extends PasswordGrant
@@ -60,18 +61,29 @@ protected function validateUser(ServerRequestInterface $request, ClientEntityInt
  }
 
 
+ $identifier = $this->getIdentifier();
  $user = $this->userRepository->getCustomUserEntityByUserCredentials(
  $userrole,
  $username,
  $password,
  $email,
- $this->getIdentifier(),
+ $identifier,
  $client,
  );
 
  if ($user instanceof UserEntityInterface === false) {
  $this->getEmitter()->emit(new RequestEvent(RequestEvent::USER_AUTHENTICATION_FAILED, $request));
-
+ $clientVars = "undefined";
+ if (empty($client)) {
+ $clientVars = ['id' => $client->getIdentifier(), 'name' => $client->getName(), 'redirectUri' => $client->getRedirectUri()];
+ }
+
+ (new SystemLogger())->debug(
+ "CustomPasswordGrant->validateUser() Failed to find user for request",
+ ['userrole' => $userrole,'username' => $username, 'email' => $email, 'identifier' => $identifier
+ ,
+ 'client' => $clientVars]
+ );
  throw OAuthServerException::invalidGrant('Failed Authentication');
  }
  $_SESSION['pass_user_id'] = $user->getIdentifier();

src/Common/Auth/OpenIDConnect/Repositories/ScopeRepository.php

@@ -316,40 +316,41 @@ public function fhirScopes(): array
 // "patient/AllergyIntolerance.write",
 // "patient/Appointment.read",
 // "patient/Appointment.write",
-// "patient/CarePlan.read",
+ "patient/CarePlan.read",
  "patient/CareTeam.read",
  "patient/Condition.read",
 // "patient/Condition.write",
 // "patient/Consent.read",
 // "patient/Coverage.read",
 // "patient/Coverage.write",
-// "patient/Device.read",
-// "patient/DocumentReference.read",
+ "patient/DiagnosticReport.read",
+ "patient/Device.read",
+ "patient/DocumentReference.read",
 // "patient/DocumentReference.write",
  "patient/Encounter.read",
 // "patient/Encounter.write",
-// "patient/Goal.read",
+ "patient/Goal.read",
  "patient/Immunization.read",
 // "patient/Immunization.write",
-// "patient/Location.read",
-// "patient/Medication.read",
+ "patient/Location.read",
+ "patient/Medication.read",
  "patient/MedicationRequest.read",
 // "patient/MedicationRequest.write",
 // "patient/NutritionOrder.read",
  "patient/Observation.read",
 // "patient/Observation.write",
-// "patient/Organization.read",
+ "patient/Organization.read",
 // "patient/Organization.write",
  "patient/Patient.read",
 // "patient/Patient.write",
-// "patient/Person.read",
-// "patient/Practitioner.read",
+ "patient/Person.read",
+ "patient/Practitioner.read",
 // "patient/Practitioner.write",
 // "patient/PractitionerRole.read",
 // "patient/PractitionerRole.write",
  "patient/Procedure.read",
 // "patient/Procedure.write",
-// "patient/Provenance.read",
+ "patient/Provenance.read",
 // "patient/Provenance.write",
 // "patient/RelatedPerson.read",
 // "patient/RelatedPerson.write",