Research Viability and Implementation of Staging Website

[Aveline-art]

Overview

As a developer, we need to ensure that our work is easy for the team to review and comment on. For this issue we will conduct all the work needed to create a staging version of our product.

Action Items

• Research uploading the website onto a host/server
  • Also research ways to have the repo (this repo) selectively upload to the staging or production website
• Create a configuration proposal in the form of code
  • This includes configurations as part of our frontend, backend, and database
  • This also includes configurations based on infrastructure as code
  • The proposal must make security as its top priority
    • As part of this research how passwords to our various servers are encrypted, and how to prevent client access; make sure that the tools used for our servers provides this kind of protection
• Find out whether we can create a "https://staging.civictechjobs.org" website with the civictechjobs.org domain alone
• Implement the staging website
  • Be sure to discuss with the ops team on how our env variables are protected/encrypted to prevent leaking it out
• Edit the pull-request-template.md to include photos of visual changes.
• Update documentation on devOps details
• Release
  • Research using Service Workers as a caching mechanism #189

Resources/Instructions

Link to current deployment plan
OWASP Top Ten
White-noise

The Ops team at HackForLA manages hosting our various projects. Please make contact with them through Slack under the #ops channel. The stack that they are using are Terraform and AWS, so focus research on those tools.

Concerning the selective upload, GitHub natively have ways of doing this. The slightly more dangerous way is to have two GitHub actions, which runs on dispatch to either the staging or production environment. This is potentially dangerous since reverting is difficult if the the wrong dispatch is accidentally ran without a history of which commit was previously hosted. The better way would be to link it to Github release system, since releases are very purposeful. The final solution might involve a bit of both, but please do broad research on possible options!

[Aveline-art]

Questions:

1. When we say 'staging website' what differentiates it from a production website?
2. What problem is the staging website trying to solve, or what is it about that the production website lacks that can be solved with a staging website?
3. What kind of configurations are most optimal when doing AB testing?
4. What QoL features can we have for a staging environment for the team when we are reviewing the staging website?

Update Jan 6, 2022

@sdimran Will work with stakeholder and PM team to determine the need for a staging website when we do not have a viable production mvp for users to use. Issue might be put on hold until MVP status reached.

[sdimran]

Hi @Aveline-art, as discussed though we may not need the full capabilities of a staging site for our landing page initial, when we get closers to our mvp it will be a need. Bonnie confirmed that the drive space will not time out so lets utilize it and get comfortable with working within the staging environment even for landing page initial.

lets go through with getting that set up

[Aveline-art]

Assigning myself this issue to periodically relay updates on my communications with the ops team

Jan 12, 2022

• spoke with ops team about requirements (thank you @arghmatey for meeting with them at the CoP meeting)
• waiting to hear back from them as they examine our DockerFiles

[Aveline-art]

Jan 20, 2022

• Ops team reported back after examining DockerFiles
• Currently arranging a meeting to go over configuration requirements before staging website can go live

[Aveline-art]

Jan 26, 2022

• Ops team is behind on arranging a meeting due to internal issues. As of now, there is no meeting planned and we are unsure when it will happen.

[sdimran]

thank you for the updates @Aveline-art !

[Aveline-art]

The ops team has gotten back to me with some minimum requirements for deployment:

• production/testing settings are in place
  • configuring django
  • configuring package.json to have correct dependencies vs devdependencies
  • configuring webpack with production script, see Improvements to Webpack Config #50
• the docker container runs when using production settings

Since we are attempting to create a staging environment (with a staging db) before moving onto production (with a production db), we will create a testing environment that tries to mimic a production environment as much as possible, using a temp/testing db.

[Aveline-art]

Update March 22, 2022:

Research stage is complete. The current implementation plan for a staging environment involves

• Set up our technologies for production; this is done since the staging environment should closely follow production, only that the database is not the one used for production
• Using nginex as our webserver
• Using Amazon EC2 to upload our docker container
  • Add other services as needed, although there are no plans for those as of now
• Continue to use PostgreSQL as our database, if only for staging purposes
• Have a framework for writing stress tests to tests the ability of our server
• Give numerical estimates on our storage, memory, and bandwidth needs so that we can compute accurately our costs

The next step is to complete a demo of how the staging environment would look like.

[Aveline-art]

Development proposal for directory setup for dev and staging env pipeline

New Directory Structure

├── .github/
│   └── ISSUE_TEMPLATE/
├── app/
│    ├── config/
│          └── settings.py    
│    ├── frontend/      
│    ├── server/   
│    ├── .babelrc
│    ├── manage.py
│    ├── requirements.txt
│    ├── package.json
│    ├── package-lock.json
│    └── webpack.config.js
├── dev/
│    ├── django (Dockerfile)
│    ├── webpack (Dockerfile)
│    ├── django.env.dev # will be added to .gitignore
│    └── webpack.env.dev # will be added to .gitignore
├── stage/
│    ├── django (Dockerfile)
│    ├── nginx (Dockerfile)
│    ├── postgre.env.stage # will be added to .gitignore
│    ├── django.env.stage # will be added to .gitignore
│    └── nginx.conf # will be added to .gitignore
├── .dockerignore
├── .gitignore
├── CONTRIBUTING.md
├── LICENSE
├── README.md
├── config.json
├── docker-compose.yml
└── docker-compose.stage.yml

Step-by-step (draft) of how our website will be built:

1. Bundle all files using webpack.
2. Database migration (?)
3. Django static collection
4. Start all servers - gunicorn application server (contains Django files), nginx web server, postgreSQL db server
5. PROFIT, if it works, or cry if it fails

Discussion points/possible problems:

1. Where do we output production webpack bundles? How do we have separate webpack configs?
2. How will static be collected when Django goes to staging?
3. Does this directory structure work?
4. Should there be two folders with env configurations, one for stage and one for dev, or is there some other way we can manage our folders?
5. How should we handle databse migrations while in staging (we can get back to this later)?
6. Should we condense the env files into only one for dev and one for prod?

Notes

Docker: Multiple Dockerfiles in project

Update March 29, 2022

• Proposal was approved my development team after discussion.
• Webpack bundles will be done via build step in a multi-stage docker build
• Static is collected via django commands outside of build step. Will work with ops team to see how AWS can allow this (possibly a lambda).
• Migrations will also be handled either via aws or some manual step (automated migrations are dangerous, since they are near impossible to revert in production). The process will be figured out with the ops team.

[Aveline-art]

Update Mar 30, 2022

• Got application layer (gunicorn) and server layer (nginx) to work with docker
• Webpack is not building production files correctly during the dockerfile's build step, will debug
• Added tentative stage and dev env files to 1password as a way of saving work that has been done thus far
• Since documentation is part of this issue, I will put work on this on hold as I work on MKDocs migration #165

[Aveline-art]

Update Mar 31, 2022

• Reconfigured the dockerfile such that webpack properly builds the production files.
• One point of trouble, however is that some js files are still too large due to a lack of lazy-loading. Since I am uncertain how much of an issue this will be at this stage, optimizations will be done as part of Add lazy loading capabilities to our codebase #190.
• Likewise,options for image size optimization will be done if it becomes a problem in the future.
• The only things left to do are:
  • finish the configurations for the django side,
  • splitting requirements.txt into dev and stage dependencies to not bog down the dev env
  • discard our frontend files (src/, node_modiles/, babel.rc, webpack.config.js, package-*.json) after bundling as all of it has been bundled into static/ (theoretically)
  • determine whether our vscode/webpack import paths have been affected by the new app/ directory

[Aveline-art]

Update April 1, 2022

• Finished with everything as far as our side is concerned.
  • One thing I could not test was https settings since I need a valid ssl certification for it (there are options to create a dummy, however, but the process is rather complex)
  • Therefore, ssl certs should be tested once we are live
• The next steps are:
  • Create a demo on what kind of AWS tools we can use and what settings we need to make this work
  • Secure our servers by configuring nginx and gunicorn
  • Consult an experienced devOps developer on the security of our current infrastructure

[Aveline-art]

Update April 14, 2022

• Met with Ops team to discuss our configurations
• Recommendations and improvements:
  • Remove nginx and use whitenoise to reduce server costs
  • (optional) Reduce app size by pooling everything into the server or config directory

@sdimran @sharadgaurav7 After meeting with the Ops team, we have come to the conclusion that we need to cut and reduce costs for our configuration. These measures would take time, however. Below are several plans to move forward. Please select the one we would prefer.

• Do not do a reduction. This means that we can host right now.
  • Pro: No more additional work.
  • Pro: Greater community support.
  • Con: Monetarily costly for little benefit.
  • Con: Adds unnecessary bloat.
• Do a small reduction. This will take an estimated 4 weeks to finish.
  • Pro: Relatively quick to implement.
  • Pro: Removes the costliest component.
  • Pro: Removes bloat.
  • Con: Small community support.
• Do a large reduction. This will take about 8 or more weeks to finish.
  • Pro: Most cost effective.
  • Con: Very difficult to implement and could take a long time.
  • Con: Almost no community support

[sdimran]

@Aveline-art for option 2 when you mention the con as small community support, could you expand on what that means?

[Aveline-art]

Community support means that there is not a large community of developers who uses a certain technology. When something has less community support, it is harder to find solutions to obscure problems or there might not be strict best practices in place.

[Aveline-art]

Reopening this as there is still more work to be done before this can be completely closed

[gjagnoor]

Notes from Ops meeting on 11/9/2022

Tasks -

1. Db - Kevin will set DB instance credentials for CTJ using aws lambda
2. Landing Page - We need to build a coming soon page
3. DNS - Kevin is going to set up an issue for Bonnie using the template they built together

What CTJ needs right now -

We want to focus on the staging environment. So that non-developers on the team can see the site before it goes live. We don’t need anything too heavy in terms of resources. We just need to get this up so people can use it and see it.

Concerns -

• Staging and production cannot use the same tables. Do we know what strategy we have to be able to avoid this, incase it is using the same tables?
  - How does RDS shard data — how do people see it in AWS

[Enzyme3]

• DB instance/creds setup is complete. Creds have been set in GH secrets as well
• DNS is dependant on domain. Waiting for confirmation from broader CTJ team on whether it's civictechjobs.com or civictechjobs.org. If it's .org, ops will have to recreate the R53 entries

Staging and production cannot use the same tables. Do we know what strategy we have to be able to avoid this, incase it is using the same tables?

I don't believe we need or want stage or prod to use the same tables? I envisioned staging as more of our final test environment to ensure things are good before promoting to prod

How does RDS shard data — how do people see it in AWS

FYI, I believe that currently, all DB user accounts (e.g. civictechjobs_stage vs cti_dev) can access any DB in the RDS instance

[Aveline-art]

FYI, I believe that currently, all DB user accounts (e.g. civictechjobs_stage vs cti_dev) can access any DB in the RDS instance

That's not good... Is there a way to limit access so that we lower the risk of cross-contaminating data? The worst situation is if someone added test data from staging into the production database as we'll need to manually verify and delete it.

If not, I think we need to reconsider the value of using our sole RDS for staging. What ways do we have of testing integration without using the sole RDS instance in staging?