[BB2-1283, BB2-1313] Django Upgrade 2.2.28 to 3.2.13 Python37 #1027

JFU-GIT · 2022-06-22T19:51:35Z

User Story or Bug Summary:

Context:

This is the follow up ticket for BB2-1224: Spike: Upgrade Django to 3.2
The SPIKE successfully prototyped BB2 server with Django upgraded from 2.2.28 to 3.2.13 (Latest 3.2), the main functional areas work as expected.

Since Django 2.2.28 is already passed extension LTS (APR 11, 2022), it is highly desirable that BB2 upgrade to 3.2.13 which will keep BB2 good through to 2024.

The next next step might be upgrade to 4.0 for further benefits etc.

AC:

This PR is a re-work of PR 1024: #1024
where the python version is bumped to 3.8.10, in this PR python 3.7.10 is used (to work around uWSGI python plugin issue with python 3.8).

BB2-1283:

BB2 Server Upgraded Django from version 2.2.28 to 3.2.13

All main functionalities tested comprehensively: Admin, Auth Flow, FHIR Flow
All unit tests pass

For BB2-1313:
There is no code changes in bb2 code base, BB2-1313 is about deployment to TEST, SBX (and eventually PROD), the pre-deployment custom migration details are documented in the CF page as shown below:

BB2 Server Upgrade Django from 2.2.28 to 3.2.13

Bumped Django to 3.2.14 security patch for a C vuln: CVE-2022-34265OPEN THIS LINK IN A NEW TAB

Please go to migrations section for details.

Verification and testing instructions and SQL scripts can be found in the CF page.

Note: before go to LLE, it is more convenient for reviewers to perform the customized migrations on a local simulated BB2 Django 2.2 to Django 3.2 migration - details are also documented in the CF page.

What Does This PR Do?

Upgrade Django to 3.2.13

What Should Reviewers Watch For?

If you're reviewing this PR, please check these things, in particular:

Verify that Blue Button server functionalities all work as expected (Admin, Command lines, migrations, authorization flow, FHIR data flow, testclients, swagger doc, loggings etc.)
All tests pass: unittests, integration tests, selenium tests
Dev Local Dockerized BB2 works as expected

Here is the confluence page where more details on testing and verifications can be found:

BB2 Server Upgrade Django from 2.2.28 to 3.2.13

What Security Implications Does This PR Have?

Submitters should complete the following questionnaire:

If the answer to any of the questions below is Yes, then here's a link to the associated Security Impact Assessment (SIA), security checklist, or other similar document in Confluence: N/A.
- Does this PR add any new software dependencies? No.
- Does this PR modify or invalidate any of our security controls? No.
- Does this PR store or transmit data that was not stored or transmitted before? No.
If the answer to any of the questions below is Yes, then please add StewGoin as a reviewer, and note that this PR should not be merged unless/until he also approves it.
- Do you think this PR requires additional review of its security implications for other reasons? No.

What Needs to Be Merged and Deployed Before this PR?

This PR cannot be either merged or deployed until the following pre-requisite changes have been fully deployed:

CMSgov/some_repo#42

Any Migrations?

Yes, there are migrations
- The migrations should be run PRIOR to the code being deployed
- The migrations should be run AFTER the code is deployed
- There is a more complicated migration plan (downtime, etc)
No migrations

Submitter Checklist

I have gone through and verified that...:

This PR is reasonably limited in scope, to help ensure that:
1. It doesn't unnecessarily tie a bunch of disparate features, fixes, refactorings, etc. together.
2. There isn't too much of a burden on reviewers.
3. Any problems it causes have a small "blast radius".
4. It'll be easier to rollback if that becomes necessary.
I have named this PR and its branch such that they'll be automatically be linked to the (most) relevant Jira issue, per: https://confluence.atlassian.com/adminjiracloud/integrating-with-development-tools-776636216.html.
This PR includes any required documentation changes, including README updates and changelog / release notes entries.
All new and modified code is appropriately commented, such that the what and why of its design would be reasonably clear to engineers, preferably ones unfamiliar with the project.
All tech debt and/or shortcomings introduced by this PR are detailed in TODO and/or FIXME comments, which include a JIRA ticket ID for any items that require urgent attention.
Reviews are requested from both:
- At least two other engineers on this project, at least one of whom is a senior engineer or owns the relevant component(s) here.
- Any relevant engineers on other projects (e.g. BFD, SLS, etc.).
Any deviations from the other policies in the DASG Engineering Standards are specifically called out in this PR, above.
- Please review the standards every few months to ensure you're familiar with them.

lgtm-com · 2022-06-22T20:03:20Z

This pull request introduces 1 alert when merging 59f8a76 into d9799a7 - view on LGTM.com

new alerts:

1 for Encoding error

lgtm-com · 2022-06-22T20:56:27Z

This pull request introduces 1 alert when merging 63d2119 into d9799a7 - view on LGTM.com

new alerts:

1 for Encoding error

lgtm-com · 2022-06-22T21:59:14Z

This pull request introduces 1 alert when merging 82e4e30 into d9799a7 - view on LGTM.com

new alerts:

1 for Encoding error

lgtm-com · 2022-06-22T22:22:33Z

This pull request introduces 1 alert when merging daba0ce into d9799a7 - view on LGTM.com

new alerts:

1 for Encoding error

lgtm-com · 2022-06-22T22:33:47Z

This pull request introduces 1 alert when merging 8cb31fe into d9799a7 - view on LGTM.com

new alerts:

1 for Encoding error

…4.manylinux2014_x86_64.whl(the binary used on CI docker) in vendor folder.

ajshred · 2022-06-23T21:37:48Z

Dockerfile.selenium

-RUN pip3 install --upgrade pip
-RUN pip3 install selenium psycopg2-binary==2.8.6 pyyaml==5.4.1 Pillow==9.0.1
+# libpq-dev: ubuntu dev lib for psypsycopg2 sdist build
+RUN apt-get update && apt-get install -yq python3.8 python3-pip git libpq-dev


Is leaving python3.8 here an oversight or intentional?

it was like that on master, note: on local selenium docker the python version does not necessarily need to be the same as those on CI docker or AMI, and it also shows that BB2 is pretty flexible on python versions, a selenium docker with python 3.8 serve the purpose of testing BB2 webUI and the authorize flows and fhir flows.

dtisza1

@JFU-GIT Looking good. Just got an error running the integration tests. See below.

The following is a summary of the local testing:

Tested a fresh PR install.
- Tests and install work OK!
- Python version in docker container shows 3.7

Tested the upgrade/migrations procedure for r88 to PR branch.

Did a fresh install of the r88 tag.
Verified install and tests worked OK.
Switched to the "jfuqian/BB2-1283-1313-Django-Upgrade-2.2.28-to-3.2.13-Python37" branch.
Verified tests DO NOT work anymore.
Ran the docker build for the web container.
- Command: docker-compose up --build --no-deps web

Connected to the DB using the following command:

Command: psql -h localhost -U postgres -d bluebutton
Ran the following SQL:

BEGIN;
insert into django_migrations (app, name, applied) values ('oauth2_provider', '0001_initial', '2022-06-01 10:00:00');
insert into django_migrations (app, name, applied) values ('oauth2_provider', '0002_auto_20190406_1805', '2022-06-01 10:00:00');
insert into django_migrations (app, name, applied) values ('oauth2_provider', '0003_auto_20201211_1314', '2022-06-01 10:00:00');
insert into django_migrations (app, name, applied) values ('oauth2_provider', '0004_auto_20200902_2022', '2022-06-01 10:00:00');
COMMIT;

Verified and Ran the migrations using the following commands:
- Command: docker-compose exec web python manage.py showmigrations
- Command: docker-compose exec web python manage.py migrate
Ran the Django tests. They worked OK!
- Command: docker-compose exec web python runtests.py
Integration tests ran OK!
- Command: docker-compose/run_integration_tests_local_keybase.sh dc

Integration tests using the "cbc" option failed with:

Processing ./vendor/python_stdnum-1.17-py2.py3-none-any.whl
Processing ./vendor/pytz-2022.1-py2.py3-none-any.whl
ERROR: Could not find a version that satisfies the requirement pyyaml==6.0 (from versions: none)
ERROR: No matching distribution found for pyyaml==6.0

Thanks for testing, think cbc is not suppose to run on local, only run dc on local.

ajshred

Ran fine for me locally all tests passing. Great job James!

lgtm-com · 2022-06-30T20:27:37Z

This pull request fixes 3 alerts when merging 7cbca9f into d9799a7 - view on LGTM.com

fixed alerts:

3 for Conflicting attributes in base classes

lgtm-com · 2022-07-05T17:06:08Z

This pull request fixes 3 alerts when merging 5a185b6 into d9799a7 - view on LGTM.com

fixed alerts:

3 for Conflicting attributes in base classes

…reference.

lgtm-com · 2022-07-05T18:53:23Z

This pull request fixes 3 alerts when merging a6f9efa into d9799a7 - view on LGTM.com

fixed alerts:

3 for Conflicting attributes in base classes

lgtm-com · 2022-07-07T18:57:04Z

This pull request fixes 3 alerts when merging c2110e3 into d9799a7 - view on LGTM.com

fixed alerts:

3 for Conflicting attributes in base classes

…13-Django-Upgrade-2.2.28-to-3.2.13-Python37

lgtm-com · 2022-07-07T19:18:19Z

This pull request fixes 3 alerts when merging 153ef08 into 9e823f1 - view on LGTM.com

fixed alerts:

3 for Conflicting attributes in base classes

dtisza1

@JFU-GIT Just had this one change suggestion for now.

All of the testing has been going well!

Just wanting to review some more tomorrow before approving.

dtisza1 · 2022-07-07T20:31:58Z

hhs_oauth_server/settings/base.py

@@ -138,8 +138,7 @@

 ALLOWED_HOSTS = env("DJANGO_ALLOWED_HOSTS", ["*", socket.gethostname()])

-# DEBUG = env("DEBUG", True)
-DEBUG = True
+DEBUG = env("DEBUG", True)


@JFU-GIT I think this original code should be changed to the following to handle any case where the ENV variable might not be set. It should default to False always.

DEBUG = env("DEBUG", False)

lgtm-com · 2022-07-08T21:19:38Z

This pull request fixes 3 alerts when merging 252c553 into 9e823f1 - view on LGTM.com

fixed alerts:

3 for Conflicting attributes in base classes

dtisza1

This looks good to me!

Testing has gone well in our TEST env.
Performed several passes over the code changes. They are looking good. Only having one fringe case type item to discuss with the team on our stand up.
I believe this is ready for deployment out to Sandbox.

@JFU-GIT Thank you for the great work on this!

oragame

Great work on this! As a final sweep over this {PR I find that everything is looking good and ready for deployment.

JAMES FUQIAN added 2 commits June 22, 2022 12:46

first cut p37 django upgrade

332174e

missed whl

59f8a76

JFU-GIT changed the title ~~[BB2-1283-1313] Django Upgrade 2.2.28 to 3.2.13 Python37~~ [BB2-1283, BB2-1313] Django Upgrade 2.2.28 to 3.2.13 Python37 Jun 22, 2022

pip upgrade in CI docker.

63d2119

JAMES FUQIAN added 2 commits June 22, 2022 14:42

remove source dist for cryptography-37.0.2.tar.gz

2496984

store the custom migration sql scripts to be handy on the target EC2

82e4e30

pip upgrade on CI docker venv.

daba0ce

pip install cryptography==37.0.2

8cb31fe

JAMES FUQIAN added 2 commits June 22, 2022 15:50

fix LGTM encoding warning in source code text.

3baccb2

manually provision cryptography-37.0.2-cp36-abi3-manylinux_2_17_x86_6…

ff5f29d

…4.manylinux2014_x86_64.whl(the binary used on CI docker) in vendor folder.

JFU-GIT requested review from dtisza1, ajshred and oragame June 22, 2022 23:41

ajshred reviewed Jun 23, 2022

View reviewed changes

dtisza1 reviewed Jun 24, 2022

View reviewed changes

ajshred approved these changes Jun 24, 2022

View reviewed changes

JAMES FUQIAN added 4 commits June 29, 2022 16:45

added debugging and tracing..

42d0c97

added debugging and tracing..

20e87f0

fix dotenv import ...

c25aad2

clean up.

7cbca9f

bump django to 3.2.14 to address C vuln CVE-2022-34265

5a185b6

bump django_storages from 1.7.1 to 1.12.3 to resolve django.util.six …

a6f9efa

…reference.

restore base settings DEBUG flag setting.

c2110e3

Merge remote-tracking branch 'origin/master' into jfuqian/BB2-1283-13…

153ef08

…13-Django-Upgrade-2.2.28-to-3.2.13-Python37

dtisza1 suggested changes Jul 7, 2022

View reviewed changes

change Django DEBUG flag default to False - to be safer.

252c553

dtisza1 approved these changes Jul 11, 2022

View reviewed changes

JFU-GIT merged commit 160ccb2 into master Jul 11, 2022

JFU-GIT deleted the jfuqian/BB2-1283-1313-Django-Upgrade-2.2.28-to-3.2.13-Python37 branch July 11, 2022 14:51

oragame reviewed Jul 11, 2022

View reviewed changes

oragame mentioned this pull request Jul 12, 2022

add escape double quotes for django logging env CMSgov/bluebutton-web-deployment#1477

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BB2-1283, BB2-1313] Django Upgrade 2.2.28 to 3.2.13 Python37 #1027

[BB2-1283, BB2-1313] Django Upgrade 2.2.28 to 3.2.13 Python37 #1027

JFU-GIT commented Jun 22, 2022 •

edited

Loading

lgtm-com bot commented Jun 22, 2022

lgtm-com bot commented Jun 22, 2022

lgtm-com bot commented Jun 22, 2022

lgtm-com bot commented Jun 22, 2022

lgtm-com bot commented Jun 22, 2022

ajshred Jun 23, 2022

JFU-GIT Jun 26, 2022

dtisza1 left a comment •

edited by JFU-GIT

Loading

ajshred left a comment

lgtm-com bot commented Jun 30, 2022

lgtm-com bot commented Jul 5, 2022

lgtm-com bot commented Jul 5, 2022

lgtm-com bot commented Jul 7, 2022

lgtm-com bot commented Jul 7, 2022

dtisza1 left a comment

dtisza1 Jul 7, 2022

lgtm-com bot commented Jul 8, 2022

dtisza1 left a comment

oragame left a comment

[BB2-1283, BB2-1313] Django Upgrade 2.2.28 to 3.2.13 Python37 #1027

[BB2-1283, BB2-1313] Django Upgrade 2.2.28 to 3.2.13 Python37 #1027

Conversation

JFU-GIT commented Jun 22, 2022 • edited Loading

What Does This PR Do?

What Should Reviewers Watch For?

What Security Implications Does This PR Have?

What Needs to Be Merged and Deployed Before this PR?

Any Migrations?

Submitter Checklist

lgtm-com bot commented Jun 22, 2022

lgtm-com bot commented Jun 22, 2022

lgtm-com bot commented Jun 22, 2022

lgtm-com bot commented Jun 22, 2022

lgtm-com bot commented Jun 22, 2022

ajshred Jun 23, 2022

Choose a reason for hiding this comment

JFU-GIT Jun 26, 2022

Choose a reason for hiding this comment

dtisza1 left a comment • edited by JFU-GIT Loading

Choose a reason for hiding this comment

ajshred left a comment

Choose a reason for hiding this comment

lgtm-com bot commented Jun 30, 2022

lgtm-com bot commented Jul 5, 2022

lgtm-com bot commented Jul 5, 2022

lgtm-com bot commented Jul 7, 2022

lgtm-com bot commented Jul 7, 2022

dtisza1 left a comment

Choose a reason for hiding this comment

dtisza1 Jul 7, 2022

Choose a reason for hiding this comment

lgtm-com bot commented Jul 8, 2022

dtisza1 left a comment

Choose a reason for hiding this comment

oragame left a comment

Choose a reason for hiding this comment

JFU-GIT commented Jun 22, 2022 •

edited

Loading

dtisza1 left a comment •

edited by JFU-GIT

Loading