[ BB2-897 ] Nightly global state metrics additional counts (#988)

* Create format_timestamp() utility function

* Create grant counts functions

- Create get_grant_counts() function
  with related top level grant counts and elapsed timings.  The functions return
  a DICT with the mentric items.
  Located in apps/authorization/models.py

- Create get_grant_by_app_counts(application) function
  with related per application grant counts and elapsed timings.  The functions return
  a DICT with the mentric items.
  Located in apps/authorization/models.py

* Create access_token count functions

- Create get_access_token_counts() function
  with related top level token counts and elapsed timings.  The functions return
  a DICT with the metric items.
  Located in apps/dot_ext/models.py

- Create get_access_token_by_app_counts(application) function
  with related single application token counts and elapsed timings.  The functions return
  a DICT with the metric items.
  Located in apps/dot_ext/models.py

* Add elapsed time to check_crosswalks() function

* Add new global state metrics counts to logging

- Add new count and elapsed time metric items to logging that are
  returned by check_crosswalks(), get_grant_counts(),
  get_grant_by_app_counts(), get_grant_counts(),
  get_access_token_counts(), get_access_token_by_app_counts(),
  get_application_counts() functions.
  These are for "global_state_metrics" and "global_state_metrics_per_app"
  type log events in apps/logging/loggers.py

- Update group_timestamp to use the format_timestamp() utility function
  in apps/logging/management/commands/log_global_state_metrics.py

* Create global state metrics logging unit tests

- Update JSON SCHEMA for GLOBAL_STATE_METRICS_LOG_SCHEMA
  and GLOBAL_STATE_METRICS_PER_APP_LOG_SCHEMA
  in apps/logging/tests/audit_logger_schemas.py

- Create a helper method _create_range_users_app_token_grant(self, start_fhir_id, count, app_name)
  hat creates a RANGE of users connected to an application with Crosswalk, Access Token, and Grant for use in tests.
  Returns a DICT of users granted by fhir_id.
  For the BaseApiTest class in apps/test.py

- Create tests to validate log metrics items after creating/removing a variety of
  real/synth users, apps and grants using BaseApiTest helper function.
  This updates the test_management_command_logging() test method
  in apps/logging/tests/test_loggers_management_command.py

* Remove print debugging statements

* Update check_crosswalks() test

* Remove redundant comments

* Update code formatting

* Update queries to handle NULL/empty crosswalks

- Update queries to exclude NULL crosswalk records in
  get_grant_counts() and get_grant_by_app_counts(application)
  functions in apps/authorization/models.py

- Update queries to exclude NULL crosswalk records in
  get_access_token_counts() and get_access_token_by_app_counts(application)
  functions in apps/dot_ext/models.py

- Add test #5 for "Test that crosswalk record with fhir_id = "" is not counted."
  in TestLoggersGlobalMetricsManagementCommand.test_management_command_logging
  method in apps/logging/tests/test_loggers_management_command.py

- Add test #6 for "Test that missing/NULL crosswalk record is not counted"
  in TestLoggersGlobalMetricsManagementCommand.test_management_command_logging
  method in apps/logging/tests/test_loggers_management_command.py

* Refactor namings See confluence doc for details

* Add table total record counts

- Add items crosswalk_table_count & crosswalk_archived_table_count

- Add items grant_table_count & grant_archived_table_count

- Add items token_table_count & token_archived_table_count

- Update log schema for new logging items in apps/logging/tests/audit_logger_schemas.py

- Add tests for new logging items in apps/logging/tests/test_loggers_management_command.py

- Refactor validate methods to be simpler by iterating over a fields list for _validate_global_state_metrics_log() and _validate_global_state_per_app_metrics_log() in apps/logging/tests/test_loggers_management_command.py

* Fix value from testing

apps/authorization/models.py

@@ -1,6 +1,9 @@
+from datetime import datetime
 from django.utils import timezone
 from django.db import models
+from django.db.models import Q
 from django.conf import settings
+
 from oauth2_provider.settings import oauth2_settings
 from oauth2_provider.models import get_access_token_model
 
@@ -70,11 +73,118 @@ def update_grants(*args, **kwargs):
 
 def check_grants():
  AccessToken = get_access_token_model()
- token_count = AccessToken.objects.filter(
- expires__gt=timezone.now(),
- ).values('user', 'application').distinct().count()
+ token_count = (
+ AccessToken.objects.filter(
+ expires__gt=timezone.now(),
+ )
+ .values("user", "application")
+ .distinct()
+ .count()
+ )
  grant_count = DataAccessGrant.objects.all().count()
  return {
  "unique_tokens": token_count,
  "grants": grant_count,
  }
+
+
+def get_grant_bene_counts(application=None):
+ """
+ Get the grant counts for real/synth benes
+
+ If application != None, the counts are for a specific application.
+ """
+ # Init counts dict
+ counts_returned = {}
+
+ # Grant real/synth bene counts (includes granted to multiple apps)
+ start_time = datetime.utcnow().timestamp()
+
+ # Setup base queryset
+ grant_queryset = DataAccessGrant.objects
+
+ if application:
+ grant_queryset = grant_queryset.filter(application=application)
+
+ # Get total table count
+ counts_returned["total"] = grant_queryset.count()
+
+ real_grant_queryset = grant_queryset.filter(
+ ~Q(beneficiary__crosswalk___fhir_id__startswith="-")
+ & ~Q(beneficiary__crosswalk___fhir_id="")
+ & Q(beneficiary__crosswalk___fhir_id__isnull=False)
+ ).values("beneficiary")
+
+ synthetic_grant_queryset = grant_queryset.filter(
+ Q(beneficiary__crosswalk___fhir_id__startswith="-")
+ & ~Q(beneficiary__crosswalk___fhir_id="")
+ & Q(beneficiary__crosswalk___fhir_id__isnull=False)
+ ).values("beneficiary")
+
+ counts_returned["real"] = real_grant_queryset.count()
+ counts_returned["synthetic"] = synthetic_grant_queryset.count()
+ counts_returned["elapsed"] = round(datetime.utcnow().timestamp() - start_time, 3)
+
+ # Grant real/synth bene distinct counts (excludes granted to multiple apps)
+ if application is None:
+ start_time = datetime.utcnow().timestamp()
+
+ counts_returned["real_deduped"] = real_grant_queryset.distinct().count()
+ counts_returned[
+ "synthetic_deduped"
+ ] = synthetic_grant_queryset.distinct().count()
+ counts_returned["deduped_elapsed"] = round(
+ datetime.utcnow().timestamp() - start_time, 3
+ )
+
+ # Archived grant real/synth bene distinct counts (excludes granted to multiple apps and multiple archived records)
+ start_time = datetime.utcnow().timestamp()
+
+ # Setup base queryset
+ archived_queryset = ArchivedDataAccessGrant.objects
+
+ if application:
+ archived_queryset = archived_queryset.filter(application=application)
+
+ # Get total table count
+ counts_returned["archived_total"] = archived_queryset.count()
+
+ real_archived_queryset = archived_queryset.filter(
+ ~Q(beneficiary__crosswalk___fhir_id__startswith="-")
+ & ~Q(beneficiary__crosswalk___fhir_id="")
+ & Q(beneficiary__crosswalk___fhir_id__isnull=False)
+ ).values("beneficiary")
+
+ synthetic_archived_queryset = archived_queryset.filter(
+ Q(beneficiary__crosswalk___fhir_id__startswith="-")
+ & ~Q(beneficiary__crosswalk___fhir_id="")
+ & Q(beneficiary__crosswalk___fhir_id__isnull=False)
+ ).values("beneficiary")
+
+ counts_returned["archived_real_deduped"] = real_archived_queryset.distinct().count()
+ counts_returned[
+ "archived_synthetic_deduped"
+ ] = synthetic_archived_queryset.distinct().count()
+ counts_returned["archived_deduped_elapsed"] = round(
+ datetime.utcnow().timestamp() - start_time, 3
+ )
+
+ # Both Grant and Archived grant (UNION) real/synth bene distinct counts
+ start_time = datetime.utcnow().timestamp()
+
+ real_union_queryset = real_grant_queryset.union(real_archived_queryset)
+ synthetic_union_queryset = synthetic_grant_queryset.union(
+ synthetic_archived_queryset
+ )
+
+ counts_returned[
+ "grant_and_archived_real_deduped"
+ ] = real_union_queryset.distinct().count()
+ counts_returned[
+ "grant_and_archived_synthetic_deduped"
+ ] = synthetic_union_queryset.count()
+ counts_returned["grant_and_archived_deduped_elapsed"] = round(
+ datetime.utcnow().timestamp() - start_time, 3
+ )
+
+ return counts_returned