-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* hapi metrics * step 1 * metrics service * adding metrics service * adding documentation * refactor * refactoring * cleanup * changing * adding changelog * some refactoring * format * review fixes part 1 * review points * review fixes * updates for code review * updates for code review * bumping versions --------- Co-authored-by: leif stawnyczy <[email protected]> Co-authored-by: leif stawnyczy <[email protected]>
- Loading branch information
3 people
committed
Sep 29, 2023
1 parent
c3ff3da
commit 7e0fa98
Showing
120 changed files
with
1,955 additions
and
121 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
6 changes: 6 additions & 0 deletions
6
...cs/src/main/resources/ca/uhn/hapi/fhir/changelog/7_0_0/5274-adding-metric-svc-to-mdm.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| --- | ||
| type: add | ||
| issue: 5274 | ||
| title: "Added a service for generating metrics on mdm links and resources. | ||
| This includes JPA queries and updated indices. | ||
| " |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
27 changes: 27 additions & 0 deletions
27
...r-jpaserver-base/src/main/java/ca/uhn/fhir/jpa/dao/data/IMdmLinkJpaMetricsRepository.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| package ca.uhn.fhir.jpa.dao.data; | ||
|
|
||
| import ca.uhn.fhir.jpa.entity.MdmLink; | ||
| import ca.uhn.fhir.mdm.api.MdmLinkSourceEnum; | ||
| import ca.uhn.fhir.mdm.api.MdmMatchResultEnum; | ||
| import org.springframework.data.jpa.repository.JpaRepository; | ||
| import org.springframework.data.jpa.repository.Query; | ||
| import org.springframework.data.repository.query.Param; | ||
| import org.springframework.stereotype.Repository; | ||
|
|
||
| import java.util.List; | ||
|
|
||
| @Repository("metricsRepository") | ||
| public interface IMdmLinkJpaMetricsRepository extends JpaRepository<MdmLink, Long>, IHapiFhirJpaRepository { | ||
|
|
||
| @Query("SELECT ml.myMatchResult AS match_result, ml.myLinkSource AS link_source, count(*) AS c " | ||
| + "FROM MdmLink ml " | ||
| + "WHERE ml.myMdmSourceType = :resourceName " | ||
| + "AND ml.myLinkSource in (:linkSource) " | ||
| + "AND ml.myMatchResult in (:matchResult) " | ||
| + "GROUP BY match_result, link_source " | ||
| + "ORDER BY match_result") | ||
| Object[][] generateMetrics( | ||
| @Param("resourceName") String theResourceType, | ||
| @Param("linkSource") List<MdmLinkSourceEnum> theLinkSources, | ||
| @Param("matchResult") List<MdmMatchResultEnum> theMatchTypes); | ||
| } |
136 changes: 136 additions & 0 deletions
136
hapi-fhir-jpaserver-base/src/main/java/ca/uhn/fhir/jpa/dao/mdm/MdmMetricSvcJpaImpl.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,136 @@ | ||
| package ca.uhn.fhir.jpa.dao.mdm; | ||
|
|
||
| import ca.uhn.fhir.jpa.api.dao.DaoRegistry; | ||
| import ca.uhn.fhir.jpa.dao.data.IMdmLinkJpaMetricsRepository; | ||
| import ca.uhn.fhir.mdm.api.BaseMdmMetricSvc; | ||
| import ca.uhn.fhir.mdm.api.MdmLinkSourceEnum; | ||
| import ca.uhn.fhir.mdm.api.MdmMatchResultEnum; | ||
| import ca.uhn.fhir.mdm.api.params.GenerateMdmMetricsParameters; | ||
| import ca.uhn.fhir.mdm.model.MdmLinkMetrics; | ||
| import ca.uhn.fhir.mdm.model.MdmLinkScoreMetrics; | ||
| import ca.uhn.fhir.mdm.model.MdmMetrics; | ||
| import ca.uhn.fhir.mdm.model.MdmResourceMetrics; | ||
| import org.springframework.transaction.annotation.Transactional; | ||
|
|
||
| import java.math.BigInteger; | ||
| import java.util.Arrays; | ||
| import java.util.List; | ||
| import java.util.stream.Collectors; | ||
| import javax.persistence.EntityManager; | ||
| import javax.persistence.EntityManagerFactory; | ||
| import javax.persistence.Query; | ||
|
|
||
| public class MdmMetricSvcJpaImpl extends BaseMdmMetricSvc { | ||
|
|
||
| private final IMdmLinkJpaMetricsRepository myJpaRepository; | ||
|
|
||
| private final EntityManagerFactory myEntityManagerFactory; | ||
|
|
||
| public MdmMetricSvcJpaImpl( | ||
| IMdmLinkJpaMetricsRepository theRepository, | ||
| DaoRegistry theDaoRegistry, | ||
| EntityManagerFactory theEntityManagerFactory) { | ||
| super(theDaoRegistry); | ||
| myJpaRepository = theRepository; | ||
| myEntityManagerFactory = theEntityManagerFactory; | ||
| } | ||
|
|
||
| protected MdmLinkMetrics generateLinkMetrics(GenerateMdmMetricsParameters theParameters) { | ||
| List<MdmLinkSourceEnum> linkSources = theParameters.getLinkSourceFilters(); | ||
| List<MdmMatchResultEnum> matchResults = theParameters.getMatchResultFilters(); | ||
|
|
||
| if (linkSources.isEmpty()) { | ||
| linkSources = Arrays.asList(MdmLinkSourceEnum.values()); | ||
| } | ||
| if (matchResults.isEmpty()) { | ||
| matchResults = Arrays.asList(MdmMatchResultEnum.values()); | ||
| } | ||
|
|
||
| Object[][] data = myJpaRepository.generateMetrics(theParameters.getResourceType(), linkSources, matchResults); | ||
| MdmLinkMetrics metrics = new MdmLinkMetrics(); | ||
| metrics.setResourceType(theParameters.getResourceType()); | ||
| for (Object[] row : data) { | ||
| MdmMatchResultEnum matchResult = (MdmMatchResultEnum) row[0]; | ||
| MdmLinkSourceEnum source = (MdmLinkSourceEnum) row[1]; | ||
| long count = (Long) row[2]; | ||
| metrics.addMetric(matchResult, source, count); | ||
| } | ||
| return metrics; | ||
| } | ||
|
|
||
| protected MdmLinkScoreMetrics generateLinkScoreMetrics(GenerateMdmMetricsParameters theParameters) { | ||
| String resourceType = theParameters.getResourceType(); | ||
|
|
||
| List<MdmMatchResultEnum> matchResultTypes = theParameters.getMatchResultFilters(); | ||
|
|
||
| // if no result type filter, add all result types | ||
| if (matchResultTypes.isEmpty()) { | ||
| matchResultTypes = Arrays.asList(MdmMatchResultEnum.values()); | ||
| } | ||
|
|
||
| String sql = "SELECT %s FROM MPI_LINK ml WHERE ml.TARGET_TYPE = :resourceType " | ||
| + "AND ml.MATCH_RESULT in (:matchResult)"; | ||
|
|
||
| StringBuilder sb = new StringBuilder(); | ||
| sb.append("sum(case when ml.SCORE is null then 1 else 0 end) as B_" + NULL_VALUE); | ||
|
|
||
| for (int i = 0; i < BUCKETS; i++) { | ||
| double bucket = getBucket(i + 1); | ||
| sb.append(",\n"); | ||
| if (i == 0) { | ||
| // score <= .01 | ||
| sb.append(String.format("sum(case when ml.SCORE <= %.2f then 1 else 0 end) as B%d", bucket, i)); | ||
| } else { | ||
| // score > i/100 && score <= i/100 | ||
| sb.append(String.format( | ||
| "sum(case when ml.score > %.2f and ml.SCORE <= %.2f then 1 else 0 end) as B%d", | ||
| getBucket(i), bucket, i)); | ||
| } | ||
| } | ||
|
|
||
| EntityManager em = myEntityManagerFactory.createEntityManager(); | ||
|
|
||
| Query nativeQuery = em.createNativeQuery(String.format(sql, sb.toString())); | ||
|
|
||
| org.hibernate.query.Query<?> hibernateQuery = (org.hibernate.query.Query<?>) nativeQuery; | ||
|
|
||
| hibernateQuery.setParameter("resourceType", resourceType); | ||
| hibernateQuery.setParameter( | ||
| "matchResult", matchResultTypes.stream().map(Enum::ordinal).collect(Collectors.toList())); | ||
|
|
||
| List<?> results = hibernateQuery.getResultList(); | ||
|
|
||
| em.close(); | ||
|
|
||
| MdmLinkScoreMetrics metrics = new MdmLinkScoreMetrics(); | ||
|
|
||
| // we only get one row back | ||
| Object[] row = (Object[]) results.get(0); | ||
| int length = row.length; | ||
| for (int i = 0; i < length; i++) { | ||
| // if there's nothing in the db, these values will all be null | ||
| BigInteger bi = row[i] != null ? (BigInteger) row[i] : BigInteger.valueOf(0); | ||
| double bucket = getBucket(i); | ||
| if (i == 0) { | ||
| metrics.addScore(NULL_VALUE, bi.longValue()); | ||
| } else if (i == 1) { | ||
| metrics.addScore(String.format(FIRST_BUCKET, bucket), bi.longValue()); | ||
| } else { | ||
| metrics.addScore(String.format(NTH_BUCKET, getBucket(i - 1), bucket), bi.longValue()); | ||
| } | ||
| } | ||
|
|
||
| return metrics; | ||
| } | ||
|
|
||
| @Transactional | ||
| @Override | ||
| public MdmMetrics generateMdmMetrics(GenerateMdmMetricsParameters theParameters) { | ||
| MdmResourceMetrics resourceMetrics = generateResourceMetrics(theParameters); | ||
| MdmLinkMetrics linkMetrics = generateLinkMetrics(theParameters); | ||
| MdmLinkScoreMetrics scoreMetrics = generateLinkScoreMetrics(theParameters); | ||
|
|
||
| MdmMetrics metrics = MdmMetrics.fromSeperableMetrics(resourceMetrics, linkMetrics, scoreMetrics); | ||
| return metrics; | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.