[ BB2-537 ] Create Middleware-Logging Integration Tests

[JFU-GIT]

JIRA Ticket:
BB2-537

User Story or Bug Summary:

Regression tests should be added for critical request/response middleware logging events.

The request_logging.py middleware is not integrated in to the Django tests. So these can not be tested currently using the same method as the audit type logging.

A/C:

Perform discovery work to determine the best method for validating the logging.

Test coverage of middleware logging (and other specialty logging such as auth flow logging) can be best achieved by validating the real log records collected from close to real life end to end BB2 flows : auth flow and data flow.

In this ticket an integration test approach is adopted to maximize the coverage of various logging components - including log records from logging middleware.

JSONSCHEMA is leveraged in validation logic.

External dependencies:

BB2-545

this PR depends on selenium tests PR

What Does This PR Do?

the implementation leverage selenium tests and bb2 built in testclient to generate a close to real life auth flow + fhir flows, where extensive set of logging middleware log records are generated and these log records are in turn scanned and validated with the extensive set of event json schema.

What Should Reviewers Watch For?

• Are the tests provide good coverage of logging middle ware logging?
• Are there any unhandled and/or untested edge cases you can think of?
• The middleware logging integration tests are launched from the same selenium test script: docker-compose/run_selenium_tests_local_keybase.sh slsx | mslsx | logit
• Make sure selenium tests SLSX vs Mock SLSX (MSLSX) and logging integration tests - logit can be run in a row and give consistent test results
• Make sure the tests can be run on different OS with consistent results

If you're reviewing this PR, please check these things, in particular:

• TODO

The code and scripts introduced can be further used to test other logging areas e.g. auth flow integration tests.
BB2-723

What Security Implications Does This PR Have?

Submitters should complete the following questionnaire:

• If the answer to any of the questions below is Yes, then here's a link to the associated Security Impact Assessment (SIA), security checklist, or other similar document in Confluence: N/A.
  • Does this PR add any new software dependencies? No.
  • Does this PR modify or invalidate any of our security controls? No.
  • Does this PR store or transmit data that was not stored or transmitted before? No.
• If the answer to any of the questions below is Yes, then please add @StewGoin as a reviewer, and note that this PR should not be merged unless/until he also approves it.
  • Do you think this PR requires additional review of its security implications for other reasons? No.

What Needs to Be Merged and Deployed Before this PR?

This PR cannot be either merged or deployed until the following pre-requisite changes have been fully deployed:

• CMSgov/some_repo#42

Submitter Checklist

I have gone through and verified that...:

• This PR is reasonably limited in scope, to help ensure that:
  1. It doesn't unnecessarily tie a bunch of disparate features, fixes, refactorings, etc. together.
  2. There isn't too much of a burden on reviewers.
  3. Any problems it causes have a small "blast radius".
  4. It'll be easier to rollback if that becomes necessary.
• I have named this PR and its branch such that they'll be automatically be linked to the (most) relevant Jira issue, per: https://confluence.atlassian.com/adminjiracloud/integrating-with-development-tools-776636216.html.
• This PR includes any required documentation changes, including README updates and changelog / release notes entries.
• All new and modified code is appropriately commented, such that the what and why of its design would be reasonably clear to engineers, preferably ones unfamiliar with the project.
• All tech debt and/or shortcomings introduced by this PR are detailed in TODO and/or FIXME comments, which include a JIRA ticket ID for any items that require urgent attention.
• Reviews are requested from both:
  • At least two other engineers on this project, at least one of whom is a senior engineer or owns the relevant component(s) here.
  • Any relevant engineers on other projects (e.g. BFD, SLS, etc.).
• Any deviations from the other policies in the DASG Engineering Standards are specifically called out in this PR, above.
  • Please review the standards every few months to ensure you're familiar with them.

[nbragdon]

@JFU-GIT Do you think this one will have a lot of changes after 724 is merged in?

[dtisza1]

@JFU-GIT This is looking good. Only had one change suggestion, if it looks good to do.

I wasn't able to get it working locally. Having the request time-out issue I was having with BB2-545. Assuming I have some kind of issue in my local environment, since this is working for you locally. And also since I can't get the BB2-545 branch to work currently, but it did in the past. I'll work to troubleshoot later, but will feel good if another reviewer can also run this locally without the errors.

[lgtm-com]

This pull request introduces 2 alerts when merging 4088166 into bf17c5b - view on LGTM.com

new alerts:

• 1 for 'import *' may pollute namespace
• 1 for URL redirection from remote source

[lgtm-com]

This pull request introduces 2 alerts when merging a8d6580 into a05bb44 - view on LGTM.com

new alerts:

• 1 for 'import *' may pollute namespace
• 1 for URL redirection from remote source

[lgtm-com]

This pull request introduces 2 alerts when merging ac6e97b into 2d90d4d - view on LGTM.com

new alerts:

• 1 for 'import *' may pollute namespace
• 1 for URL redirection from remote source

[lgtm-com]

This pull request introduces 2 alerts when merging b16dd8e into 2d90d4d - view on LGTM.com

new alerts:

• 1 for 'import *' may pollute namespace
• 1 for URL redirection from remote source

[lgtm-com]

This pull request introduces 2 alerts when merging a1607cf into 2d90d4d - view on LGTM.com

new alerts:

• 1 for 'import *' may pollute namespace
• 1 for URL redirection from remote source

[lgtm-com]

This pull request introduces 2 alerts when merging 1b14c71 into 2d90d4d - view on LGTM.com

new alerts:

• 1 for 'import *' may pollute namespace
• 1 for URL redirection from remote source

[lgtm-com]

This pull request introduces 2 alerts when merging 3c95cd9 into 2d90d4d - view on LGTM.com

new alerts:

• 1 for 'import *' may pollute namespace
• 1 for URL redirection from remote source

[dtisza1]

@JFU-GIT Just a couple changes to suppress the LGTM alerts. Assuming OK since this is for integration tests.

[dtisza1]

@JFU-GIT Same here to suppress LGTM alert:

Suggested change

	from .dev import *
	from .dev import * # lgtm [py/polluting-import]

[lgtm-com]

This pull request introduces 1 alert when merging f045a7b into 2d90d4d - view on LGTM.com

new alerts:

• 1 for 'import *' may pollute namespace

[lgtm-com]

This pull request introduces 1 alert when merging 64c1503 into 2d90d4d - view on LGTM.com

new alerts:

• 1 for 'import *' may pollute namespace

[lgtm-com]

This pull request introduces 1 alert when merging d19564b into 82cce07 - view on LGTM.com

new alerts:

• 1 for 'import *' may pollute namespace

[dtisza1]

@JFU-GIT I just pushed up a commit that fixes the docker timing issue I was having when running locally. You should test that this doesn't break things in your local dev.

@nbragdon @oragame Does this fix the issue for you also, if you were running in to this too?

[nbragdon]

@dtisza1 I'm able to get this working after your change 👍

[lgtm-com]

This pull request introduces 1 alert when merging 10f1c99 into b2c2ab8 - view on LGTM.com

new alerts:

• 1 for 'import *' may pollute namespace

[dtisza1]

LGTM!

The code and local testing is looking good!

@JFU-GIT Great work on this!