T.E.N.I.S. stands for Team Event Notificaton and Interoperability System!
Static manifest is available for demo purposes, it only expects $DOMAIN_NAME variable to be passed with kubectl apply. You can do it this way:
export DOMAIN_NAME=benis.bg502.ru; curl -s https://raw.githubusercontent.com/Ivinco/tenis/774d61c9c0e6f2dbebd8fec5638239650bcb21c9/static-manifest/tenis-0.4.yaml | envsubst '${DOMAIN_NAME}' | kubectl create -f -
It's a little bit more complicated than using werf, but more common one
- helm
- Ready to use k8s cluster with configured kubeconfig
- Registry (for manual deployment you should be logged in if your registry is protected with auth)
- Registry secret (this chart expects a secret called registry-secret)
- Build and publish images in your repo
- You can use
build-and-push.sh
script. Images repo should be available as $REPO env variable - You can also pass $TAG variable, to specify a tag,
latest
is used if no $TAG variable available
- You can use
- Add
mongodb.env.INIT_PASSWORD
andmongodb.env.API_TOKEN
invalues.yaml
. It should be added like that. Note, it's not safe to keep passwords in the repo, you'll have to come up with a secure way on your own, or you can use encryption provided by werf - API token - is a string of ASCII symbols up to 64 bytes
mongodb:
env:
INIT_PASSWORD:
_default: <your default password here>
dev: <example of your password for werf.env=dev>
API_TOKEN:
_default: <Your API token>
To understand more about what kind of password should be used, look [here] (#how-to-generate-new-password-for-mongodb-init-user)
- Add
env_url
. You can add it invalues.yaml
or directly with helm install command - Install chart (example)
$ helm -n <k8s-namespace> install <release-name> .helm --set werf.image.backend="backend image in you repo" --set werf.image.frontend="<frontend image in your repo>" --set werf.image.mongodb="<mongodb image in your repo (or from public repo, right now basic image is used)>" --set werf.env=dev --set env_url="test.example.com"
- werf
- Ready to use k8s cluster with configured kubeconfig
- Registry (for manual deployment you should be logged in if your registry is protected with auth)
- Registry secret (in this app it's called registry-secret)
- Prepare the environment
- Add werf secret key (In repo root)
$ werf helm secret generate-secret-key > .werf_secret_key
- Add INIT_PASSWORD
$ werf helm secret values edit .helm/secret-values.yaml
- Build and push images
$ werf build --repo <registry>/<project>
- Converge
$ werf converge --namespace <your-namespace> --repo <registry>/<project>
For CI/CD purposes, WERF_SECRET_KEY variable should be added to the project variables and made available for use.
We use werf to build, publish, and deploy everything to k8s.
Images building procedure is defined in werf.yaml
file.
For example, backend is built like that:
configVersion: 1 <======= The only version supported currently
project: tenis <======= project name, will be used by werf to get {{ .Chart.Name }}
---
image: backend <======= name of the image. Can be fetched with {{ .Values.werf.images.backend }} in the templates
from: python:alpine3.17
docker:
CMD: [ "/usr/local/bin/python3", "-m", "tenis_backend" ]
git:
- add: /backend
to: /app
excludePaths:
- LICENSE
- "**.sh"
stageDependencies: <====== Defining rules of when image should be rebuilt. In this case - changes in pyproject.toml and also in any file at all
install:
- "**/*"
- "pyproject.toml"
shell:
install:
- cd /app
- pip install .
Werf can also utilize ready Dockerfiles, but this way gives us more flexibility in the building process.
Once definition is ready, images are build with werf build
, see ci-dev.yml
.
We use Github Actions as a CI/CD tool.
-
Deploy to
dev
is happening on once PR to 'dev' branch is closed or manualy.ci-dev.yml
workflow is responsible for Automatically depploying any changes totenis-dev
namespace.
-
Deploy to
dev-2
is happening manualy.ci-dev-2.yml
workflow is responsible for Automatically depploying any changes totenis-dev-2
namespace.
-
Deploy to
prod
is happening on merge to master or manually,ci-prod.yml
workflow deploys the code to nstenis-prod
, available on urltenis.k8s-test.ivinco.com
Deploy job looks simillar in both cases:
converge:
needs: render-and-lint
runs-on: self-hosted
steps:
- name: Checkout code
uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: registry.ivinco.com
username: ${{ secrets.REGISTRY_SECRET_USER }}
password: ${{ secrets.REGISTRY_SECRET_PASSWORD }}
- name: Install werf
uses: werf/actions/[email protected]
- name: Converge
uses: werf/actions/[email protected]
with:
kube-config-base64-data: ${{ secrets.KUBE_CONFIG_BASE64_DATA }}
env: dev
env:
WERF_SET_ENV_URL: "env_url=tenis-dev.k8s-test.ivinco.com"
WERF_NAMESPACE: tenis-dev
WERF_RELEASE: tenis-dev
All the secrets like KUBE_CONFIG_BASE64_DATA,REGISTRY_SECRET_USER, REGISTRY_SECRET_PASSWORD are defined in the repo's secrets. You can find them in repo's settings'
Creating of a new environment is extremely easy. You need to define new job in ci-dev.yml
(or you can make a copy of .github/workflows/ci-dev.yml
file and define your own workflow with minor changes).
For most of the cases it will be enough to change several variables in the converge.
Here's the variables list:
WERF_SET_ENV_URL: "env_url=tenis-dev.k8s-test.ivinco.com" <===== URL on which your app will be available
WERF_NAMESPACE: tenis-dev <===== k8s namespace
WERF_RELEASE: tenis-dev <===== helm release
steps.converge.with.env: dev <===== environment id
Once the deploy is complete, you can access your environment by the URL set in WERF_SET_ENV_URL: "env_url=example.com"
converge-updated:
needs: render-and-lint
runs-on: self-hosted
steps:
- name: Checkout code
uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: registry.ivinco.com
username: ${{ secrets.REGISTRY_SECRET_USER }}
password: ${{ secrets.REGISTRY_SECRET_PASSWORD }}
- name: Install werf
uses: werf/actions/[email protected]
- name: Converge
uses: werf/actions/[email protected]
with:
kube-config-base64-data: ${{ secrets.KUBE_CONFIG_BASE64_DATA }}
env: dev_andrew
env:
WERF_SET_ENV_URL: "env_url=tenis-andrew.k8s-test.ivinco.com"
WERF_NAMESPACE: tenis-andrew
WERF_RELEASE: tenis-andrew
values.yaml
mongodb:
env:
INIT_DB:
_default: "tenis" <==== If `steps.converge.with.env` has no custom value configured, _default is used
dev_andrew: "tenis-andrew-or-whatever" <==== if `steps.converge.with.env` == dev_andrew, this value is used
This behaviour is achieved by the following constriction inside the template files:
containers:
- name: {{ .Chart.Name }}-backend
image: {{ $.Values.werf.image.backend }}
imagePullPolicy: {{ .Values.backend.image.pullPolicy }}
ports:
- name: http
containerPort: 8000
protocol: TCP
env:
- name: MONGO_HOST
value: {{ .Chart.Name }}-mongodb
- name: MONGO_DATABASE <=========== Here
value: {{ pluck .Values.werf.env .Values.mongodb.env.INIT_DB | first | default .Values.mongodb.env.INIT_DB._default }}
Basically, we look for {{ .Values.werf.env }}
in {{ .Values.mongodb.env.INIT_DB }}
. Once we find it - we take it's value. If nothing is found, we default to {{ .Values.mongodb.env.INIT_DB._default }}
On step render-and-lint
there's a step called Render chart
, which contains all all the rendered resources with all the values substituted, except for secret values like passwords.
Password should be hashed with werkzeug.security:
$ python
>>> from werkzeug.security import generate_password_hash
>>> print(generate_password_hash("example-password"), "\n")
scrypt:32768:8:1$wlKfg9o3d9jtl0AU$0aebc1e1d841a46b049a53052ac9c53ccd998975ce84b0dd98089029f072b360316a9c0283e0ef26aa313835b29bf5ad4e8c1674737953ffcc0c2647fe912d64
Here's the hasing alghorithm (scrypt:32768:8:1), salt (wlKfg9o3d9jtl0AU) and hashed password are divided by '$', but you need the whole string:
"scrypt:32768:8:1$wlKfg9o3d9jtl0AU$0aebc1e1d841a46b049a53052ac9c53ccd998975ce84b0dd98089029f072b360316a9c0283e0ef26aa313835b29bf5ad4e8c1674737953ffcc0c2647fe912d64"
Passwords are stored encrypted in .helm/secret-values.yaml
. You need to acquire WERF_SECRET_KEY and add it to .werf_secret_key
file in the root of the repo (it's in .gitignore, so it's safe) or export to variable WERF_SECRET_KEY locally.
export WERF_SECRET_KEY=example-string-qwerty
To add an additional value or variable:
werf helm secret values edit .helm/secret-values.yaml
mongodb:
env:
INIT_PASSWORD:
_default: "scrypt-password-generated-with-werkzeug.security"
dev_andrew: "scrypt:32768:8:1$wlKfg9o3d9jtl0AU$0aebc1e1d841a46b049a53052ac9c53ccd998975ce84b0dd98089029f072b360316a9c0283e0ef26aa313835b29bf5ad4e8c1674737953ffcc0c2647fe912d64" (env should be the same as `steps.converge.with.env` in CI)
To build images manualy:
$ werf build --repo registry.example.ru/tenis
To render a chart:
$ werf render --repo registry.example.ru/tenis
Note, render won't work without repo address since it's calculating resulting image name based on files in git
To lint a chart:
$ werf helm lint .helm
To deploy:
$ werf converge --repo registry.example.ru/tenis
- Registry cleanup is defined in
.github/workflows/cleanup.yml
- CodeQL checks are defined in
.github/workflows/codeql.yml