A fun cheese quizz deployed on OpenShift and illustrating cloud native technologies like Quarkus, Istio Service Mesh, CodeReady Workspaces, Strimzi Kafka Operator, Fuse Online/Syndesis, Tekton pipelines and ArgoCD.
-
Part 1 Try to guess the displayed cheese! Introducing new cheese questions with Canaray Release and making everything resilient and observable using Istio Service Mesh. Deploy supersonic components made in Quarkus,
-
Part 2 Implement a new "Like Cheese" feature in a breeze using CodeReady Workspaces, demonstrate the inner loop development experience and then deploy everything using Tekton.
-
Part 3 Add the "Like Cheese API" using Serverless Knative and make it push new messages to Kafka broker. Use Syndesis or CamelK to deploy new integration services and create lead into Salesforce CRM ;-)
Please initialize and configure following operators in this order:
All the components below can be setup using my
cluster-init.sh
script that you may find here.
-
Istio Service Mesh deployed with
basic-install
onistio-system
project- Take care of removing
LimitRanges
intocheese-quizz
project
- Take care of removing
-
Knative Serving deployed cluster wide
- Create a
KnativeServing
CR intoknative-serving
project, addingimage-registry.openshift-image-registry.svc:5000
intoregistriesSkippingTagResolving
property
- Create a
-
Fuse Online operator deployed into
fuse-online
project- Create a
SyndesisCRD
CR, calling itsyndesis
- Create a
-
AMQ Streams operator deployed cluster wide,
-
OpenShift Pipelines deployed cluster wide,
-
CodeReady Workspaces operator deployed onto
workspaces
project with:quay.io/lbroudoux/che-plugin-registry:master
as thepluginRegistryImage
true
fortlsSupport
CHE_INFRA_KUBERNETES_PVC_WAIT__BOUND: 'false'
asserver.customCheProperties
This is what the CheCluster
custom resource shoud look like:
apiVersion: org.eclipse.che/v1
kind: CheCluster
metadata:
name: codeready-workspaces
namespace: workspaces
spec:
server:
cheImageTag: ''
cheFlavor: codeready
devfileRegistryImage: ''
pluginRegistryImage: 'quay.io/lbroudoux/che-plugin-registry:master'
tlsSupport: true
selfSignedCert: false
customCheProperties:
CHE_INFRA_KUBERNETES_PVC_WAIT__BOUND: 'false'
database:
externalDb: false
chePostgresHostName: ''
chePostgresPort: ''
chePostgresUser: ''
chePostgresPassword: ''
chePostgresDb: ''
auth:
openShiftoAuth: true
identityProviderImage: ''
externalIdentityProvider: false
identityProviderURL: ''
identityProviderRealm: ''
identityProviderClientId: ''
storage:
pvcStrategy: per-workspace
pvcClaimSize: 1Gi
preCreateSubPaths: true
We need to setup the following resources for our demonstration:
-
A
cheese-quizz
project for holding your project component-
Make this project part of the Service Mesh control plane installed into
istio-system
by deploying aServiceMeshMemberRoll
intoistio-system
referencingcheese-quizz
project as member -
Once done, ensure
cheese-quizz
project has the following labels to be sure it is included into Service Mesh:istio.io/member-of=istio-system
kiali.io/member-of=istio-system
-
-
A
cheese-quizz-function
project for holding the serverless part and the Kafka broker -
A
Kafka
broker CR intocheese-quizz-function
letting the default propertiesoc create -f manifests/kafka.yml -n cheese-quizz-function
oc create -f manifests/kafka-topic.yml -n cheese-quizz-function
Start deploying the components needed at the beginning of this demo, we'll deploy the other ones later on.
oc create -f manifests/quizz-question-deployment-v1.yml -n cheese-quizz
oc create -f manifests/quizz-question-deployment-v2.yml -n cheese-quizz
oc create -f manifests/quizz-question-deployment-v3.yml -n cheese-quizz
oc create -f manifests/quizz-question-service.yml -n cheese-quizz
oc apply -f manifests/quizz-question-destinationrule.yml -n cheese-quizz
oc apply -f manifests/quizz-question-virtualservice-v1.yml -n cheese-quizz
oc create -f manifests/quizz-client-buildconfig.yml -n cheese-quizz
oc create -f manifests/quizz-client-deploymentconfig.yml -n cheese-quizz
oc create -f manifests/quizz-client-service.yml -n cheese-quizz
oc create -f manifests/quizz-client-route.yml -n cheese-quizz
oc rollout latest cheese-quizz-client -n cheese-quizz
Once above commands are issued and everything successfully deployed, retrieve the Cheese Quizz route:
$ oc get route/cheese-quizz-client -n cheese-quizz |grep cheese-quizz-client |awk '{print $2}'
cheese-quizz-client-cheese-quizz.apps.cluster-lemans-0014.lemans-0014.example.opentlc.com
and open it into a browser. You should get the following:
Introduce new v2
question using Canary Release and header-matching routing rules:
oc apply -f istiofiles/vs-cheese-quizz-question-v1-v2-canary.yml -n cheese-quizz
Using the hamburger menu on the GUI, you should be able to subscribe the Beta Program
and see the new Emmental question appear ;-)
Now turning on the Auto Refresh
feature, you should be able to visualize everything into Kiali, showing how turning on and off the Beta subscription has influence on the visualization of networks routes.
Once we're confident with the v2
Emmental question, we can turn on Blue-Green deployment process using weighted routes on the Istio VirtualService
. We apply a 70-30 repartition:
oc apply -f istiofiles/vs-cheese-quizz-question-v1-70-v2-30.yml -n cheese-quizz
Of course we can repeat the same kind of process and finally introduce our v3
Camembert question into the game. Finally, we may choose to route evenly to all the different quizz questions, applying a even load-balancer rules on the VirtualService
:
oc apply -f istiofiles/vs-cheese-quizz-question-all.yml -n cheese-quizz
Now let's check some network resiliency features of OpenShift Service Mesh.
Start by simulating some issues on the v2
deployed Pod. For that, we can remote log to shell and invoke an embedded endpoint that will make the pod fail. Here is bellow the sequence of commands you'll need to adapt and run:
$ oc get pods -n cheese-quizz | grep v2
cheese-quizz-question-v2-847df79bd8-9c94t 2/2 Running 0 5d19h
$ oc rsh -n cheese-quizz cheese-quizz-question-v2-847df79bd8-9c94t
----------- TERMINAL MODE: --------------------
Defaulting container name to greeter-service.
Use 'oc describe pod/cheese-quizz-question-v2-847df79bd8-9c94t -n cheese-quizz' to see all of the containers in this pod.
sh-4.4$ curl localhost:8080/api/cheese/flag/misbehave
Following requests to / will return a 503
sh-4.4$ exit
exit
command terminated with exit code 130
Back to the browser window you should now have a little mouse displayed when application tries to reach the v2
question of the quizz.
Using obervability features that comes with OpenShift Service Mesh like Kiali and Jaeger, you are now able to troubleshoot and check where the problem comes from (imagine that we already forgot we did introduce the error ;-))
Thus you can see the Pod causing troubles with Kiali graph:
And inspect Jeager traces to see the details of an error:
In order to make our application more resilient, we have to start by creating new replicas, so scale the v2
deployment.
oc scale deployment/cheese-quizz-question-v2 --replicas=2 -n cheese-quizz
Newly created pod will serve requests without error but we can see in the Kiali console that the service cheese-quizz-question
remains degraded (despite green arrows joinining v2
Pods).
There's still some errors in distributed traces. You can inspect what's going on using Jaeger and may check that there's still some invocations going to the faulty v2
pod.
Istio proxies automatically retry doing the invocation to v2
because a number of conditions are present:
- There's a second replica present,
- It's a HTTP
GET
request that is supposed to be idempotent (so replay is safe), - We're in simple HTTP with no TLS so the headers inspectation allow determine these conditions.
An optimal way of managing this kind of issue would be to declare a CircuitBreaker
for handling this problem more efficiently. Circuit breaker policy will be in charge to detect Pod return ing errors and evict them from the elligible targets pool for a configured time. Then, the endpoint will be re-tried and will re-join the pool if erverything is back to normal.
Let's apply the circuit breaker configuration to our question DestinationRule
:
oc apply -f istiofiles/dr-cheese-quizz-question-cb.yml -n cheese-quizz
Checking the traces once again in Kiali, you should no longer see any errors!
Pursuing with network resiliency features of OpenShift Service Mesh, let's check now how to handle timeouts.
Start by simulating some latencies on the v3
deployed Pod. For that, we can remote log to shell and invoke an embedded endpoint that will make the pod slow. Here is bellow the sequence of commands you'll need to adapt and run:
$ oc get pods -n cheese-quizz | grep v3
cheese-quizz-question-v3-9cfcfb894-tjtln 2/2 Running 0 6d1h
$ oc rsh -n cheese-quizz cheese-quizz-question-v3-9cfcfb894-tjtln
----------- TERMINAL MODE: --------------------
Defaulting container name to greeter-service.
Use 'oc describe pod/cheese-quizz-question-v3-9cfcfb894-tjtln -n cheese-quizz' to see all of the containers in this pod.
sh-4.4$ curl localhost:8080/api/cheese/flag/timeout
Following requests to / will wait 3s
sh-4.4$ exit
exit
Back to the browser window you should now have some moistures displayed when application tries to reach the v3
question of the quizz.
Before digging and solving this issue, let's review the application configuration :
- A 3 seconds timeout is configured within the Pod handling the
v3
question. Let see the question source code - A 1.5 seconds timeout is configured within the Pod handling the client. Let see the client configuration
Checking the distributed traces within Kiali console we can actually see that the request takes 1.5 seconds before returning an error:
In order to make our application more resilient, we have to start by creating new replicas, so scale the v3
deployment.
oc scale deployment/cheese-quizz-question-v3 --replicas=2 -n cheese-quizz
Newly created pod will serve requests without timeout but we can see in the Kiali console that the service cheese-quizz-question
remains degraded (despite green arrows joinining v3
Pods).
However there's still some errors in distributed traces. You can inspect what's going on using Jaeger and may check that there's still some invocations going to the slow v3
pod.
The CircuitBreaker
policy applied previsouly does not do anything here because the issue is not an application problem that can be detected by Istio proxy. The result of a timed out invocation remains uncertain, but we know that in our case - an idempotent GET
HTTP request - we can retry the invocation.
Let's apply for this a new VirtualService
policy that will involve a retry on timeout.
oc apply -f istiofiles/vs-cheese-quizz-question-all-retry-timeout.yml -n cheese-quizz
Once applied, you should not see errors on the GUI anymore. When digging deep dive into the distributed traces offered by OpenShift Service Mesh, you may however see errors traces. Getting into the details, you see that detailed parameters of the VirtualService
are applied: Istio do not wait longer than 100 ms before making another attempt and finally reaching a valid endpoint.
The Kiali console grap allow to check that - from a end user point of view - the service is available and green. We can see that time-to-time the HTTP throughput on v3
may be reduced due to some failing attempts but we have now great SLA even if we've got one v2
Pod failing and one v3
Pod having response time issues:
So far we always used the classical way of entering the application through cheese-quizz-client
pods only. It's now time to see how to use a Gateway
. Gateway configurations are applied to standalone Envoy proxies that are running at the edge of the mesh, rather than sidecar Envoy proxies running alongside your service workloads.
Before creating Gateway
and updating the VirtualService
to be reached by the gateway, you will needs to adapt the full host
name in both resources below. Then apply them:
oc apply -f istiofiles/ga-cheese-quizz-question.yml -n cheese-quizz
oc apply -f istiofiles/vs-cheese-quizz-question-all-gateway.yml -n cheese-quizz
OpenShift takes care of creating a Route
for each and every Gateway
. The new route is created into your mesh control plane namespace (istio-system
here).
$ oc get routes -n istio-system | grep cheese-quizz-question | awk '{print $2}'
cheese-quizz-question.apps.cluster-fdee.fdee.example.opentlc.com
You should now be able to directly call the cheese-quizz-question
VirtualService:
$ curl https://cheese-quizz-question.apps.cluster-fdee.fdee.example.opentlc.com/api/cheese
hello%
When working with Maistra/OpenShift Service Mesh v2.0,
enabledAutoMtls
is set totrue
by default intoistio-config
config map. This made your access wrapped into MTLS communication (between client and question OR between gateway and question) by default.
Let's try applying Mutual TLS on our destinations:
oc apply -f istiofiles/dr-cheese-quizz-question-mtls -n cheese-quizz
This is the beginning of Part 2 of the demonstration. After having installed CodeReady Workspaces as specified in Cluster Setup, start retrieving the route to access it:
oc get route/codeready -n workspaces | grep codeready | awk '{print $2}'
You can use this route to configure the Developer Workspace
badge as I did it at the begining of this README file. Now clicking this badge in a new browser tab or window, you'll ask CodeReady Workspaces to create a new workspace for you.
If it's the first time, you're connecting the service, you'll need to authenticate and approve the reuse of your profile information.
Worskspace creation waiting screen:
After some minutes, the workspace is initialized with the source files coming from a Git clone. You can spend time explaining the relationship between the devfile.yaml
at the root of this repotisotory and the content of the wokspace.
Now let's build and deploy some componentn in order to illustrate the development inner loop.
Using the scripts on the right hand panel, you will have to first locally install the quizz-model
component with the Model - Install script. Here's the terminal results below:
Then, you will be able to launch the quizz-question
module in Quarkus development monde using the Question - Compile (Dev Mode) script. CRW asks if current process should be made available through an OpenShift Route for accessing the pod:
Finally, you can launch the quizz-question
module using the Client - Compile (Dev Mode) script. Here you will have access to the GUI, running in CRW, when launching the preview:
It's time to talk a little bit about Quarkus, demo hot reloading and explain how we're gonna implement the "Like Cheese" screen by modifying src/main/resources/META-INF/index.html
and test it locally:
Before commiting our work, we'd like to talk a bit about how to transition to the outer-loop and trigger deployment pipeline.
Ensure the different custom resources for Tekton / OpenShift Pipelines are installed into the cheese-quizz
project:
oc create -f manifests/oc-deploy-task.yml -n cheese-quizz
oc create -f manifests/oc-ensure-no-triggers.yml -n cheese-quizz
oc create -f manifests/oc-patch-deployment-task.yml -n cheese-quizz
oc create -f manifests/oc-start-build-task.yml -n cheese-quizz
oc create -f manifests/quizz-client-pipeline.yml -n cheese-quizz
oc create -f manifests/quizz-client-pipeline-trigger.yml -n cheese-quizz
oc create -f manifests/quizz-client-pipeline-listener.yml -n cheese-quizz
Configure a Webhook
trigger on your Git repository holding the sources. First you have to retrieve the full URL of the Tekton Trigger
that must be invoked:
oc get route/quizz-client-pipeline-listener -n cheese-quizz | grep quizz-client-pipeline-listener | awk '{print $2}'
Then follow your preferred Git repo documentation to set such a webhook. Here's an example below using GitHub on this repository:
Now that this part is OK, you can finish your work into CodeReady Workspaces by commiting the changed file and pushing to your remote repository:
And this should simply trigger the Tekton / OpenShift Pipeline we just created !
You can display the different task logs in OpenShift console:
And finally ensure that our pipeline is successful.
We can now also demonstrate the new fetaure deployed onto our OpenShift cluster.
This is the beginning of Part 3 of the demonstration. Now you're gonne link the "Like Cheese" feature with a message publication within a Kafka broker. So first, we have to deploy a broker and declare a topic we'll use to advert of new CheeseLike
messages.
oc create -f manifests/kafka.yml -n cheese-quizz-function
oc create -f manifests/kafka-topic.yml -n cheese-quizz-function
Now just deploy our quizz-like-function
module that is a NodeJS app into the cheese-quizz-function
project using the Developer Console, adding a component from Git. Here's the capture of the long form:
Because we use graphical wizard for creating our Knative Service, we do not have the opportunity to set environment variables. Our application should communicate with Kafka broker and also specified it is using a specific 4000 TCP port. You can do this using the kn
command line tool:
kn service update cheese-quizz-like-function -p 4000 -e KAFKA_HOST=my-cluster-kafka-bootstrap.cheese-quizz-function.svc.cluster.local:9092
Still using the kn
tool, we can now see that Knative has created for us 2 revisions. One being the one created at first after form validation, the pther resulting of the environment variable and port modification:
kn revision list -n cheese-quizz-function
NAME SERVICE TRAFFIC TAGS GENERATION AGE CONDITIONS READY REASON
cheese-quizz-like-function-gfdjz-5 cheese-quizz-like-function 2 6m52s 3 OK / 4 True
cheese-quizz-like-function-khzw9 cheese-quizz-like-function 1 37m 0 OK / 4 False ExitCode1 : Container failed with: info using ...
We see that first revision fails to start because of missing environment variable and that latest revision is now ready to receive traffic. We have now to ditribute traffic to this revision. This can be done from the Developer Console when accessing the service resources and hitting the Set Traffic Distribution on the right hand panel:
You'll see now that an arrow indicating that revision receives traffic appears on the Developer Console. Also we can check the traffic distribution using the CLI:
kn revision list -n cheese-quizz-function
NAME SERVICE TRAFFIC TAGS GENERATION AGE CONDITIONS READY REASON
cheese-quizz-like-function-gfdjz-5 cheese-quizz-like-function 100% v1 4 27m 3 OK / 4 True
cheese-quizz-like-function-khzw9 cheese-quizz-like-function 1 58m 0 OK / 4 False ExitCode1 : Container failed with: info using ...
Note: we could have achieved the same result only using the CLI commands below:
kn service update cheese-quizz-like-function --tag v1=cheese-quizz-like-function-gfdjz-5
kn service update cheese-quizz-like-function --traffic v1=100
Now just demo how Pod are dynamically popped and drained when invocation occurs on function route. You may just click on the access link on the Developer Console or retrieve exposed URL from the command line:
kn service describe cheese-quizz-like-function -o yaml -n cheese-quizz-function | yq r - 'status.url'
Now that we also have this URL, we should update the cheese-quizz-client-config
ConfigMap that should hold this value and serve it to our GUI.
$ oc edit cm/cheese-quizz-client-config -n cheese-quizz
----------- TERMINAL MODE: --------------------
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
application.properties: |-
# Configuration file
# key = value
%kube.quizz-like-function.url=https://cheese-quizz-like-function-cheese-quizz-function.apps.cluster-lemans-0014.lemans-0014.example.opentlc.com
kind: ConfigMap
metadata:
creationTimestamp: "2020-06-04T09:55:07Z"
name: cheese-quizz-client-config
namespace: cheese-quizz
resourceVersion: "4656024"
selfLink: /api/v1/namespaces/cheese-quizz/configmaps/cheese-quizz-client-config
uid: 7a97745b-abc3-4b22-aaad-07b566fca3cb
~
~
~
-- INSERT --
:wq
configmap/cheese-quizz-client-config edited
Do not forget to delete the remaining
cheese-quizz-client
pod to ensure reloading of changed ConfigMap.
This is the final part where you'll reuse the events produced within Kafka broker in order to turn into business insights !
First thing first, create a Salesforce connector within your Syndesis/Fuse Online instance. This can be simply done using thet guide.
Then you'll have to create a connector to our Kafka instance located at my-cluster-kafka-bootstrap.cheese-quizz-function.svc.cluster.local:9092
.
You should now have these 2 connectors ready to use and you can create a new integration, we'll call `cheese-quizz-likes to Salesforce'.
When creating a new integration, you shoud select the Kafka connector as a source, subscribing to the topic called cheese-quizz-likes
and filling out this example JSON Instance
:
{
"email": "[email protected]",
"username": "John Doe",
"cheese": "Cheddar"
}
You will be asked to fill out some details about data type and description like below:
Just after that you'll have to select the Salesforce connector as the integration end, picking the New Record action and choosing the Lead data type. Finally, in the next screen, you'll have to add a Data Mapper
intermediary step to allow transformation of the Kafka message data.
We'll realize a mapping between following fields:
username
will be split intoFirstName
andLastName
,email
will remainemail
,cheese
will fedd theDescription
field
We'll add two extras constants on the left hand pane:
Quizz Player
will feed theCompany
field that is required on the Salesforce side,cheese-quizz-app
will feed theLeadSource
field.
You should have something like this:
Hit the Save and Publish button and wait a minute or two that Syndesis built and publish the integration component. Once OK your should be able to fill out the connoisseur form on the app side and hit the Like button. Just see Knative popping out a new pod for processing the HTTP call and producing a message into the Kafka broker. Then the Syndesis integration route will take care of transformaing this message into a Slaesforce Lead.
The result should be something like this on the Slaesforce side:
You can track activity of the integration route, looking at the Activity tab in the route details:
You can use the resources from the gitops/
folder of this repository to deploy a simplest form of the application using GitOps tooling.
We just published a simplest version for sake of simplicity but all the resources involved involved in the complex scenario may also be deployed the same manner 😉
In order to efficiently manage our Kubernetes resources within the Git repository, we have used Kustomize.
The default base
configuration will deploy a quizz question referencing the Cheddar cheese and will present only 1 replica within the question deployment. The overlay cluster1
configuration will override the cheese and will only display Emmental, it will also ask for 2 replicas within the question deployment.
We will apply GitOps deployment for cluster1
configuration.
Open Cluster Management is the upstream community project bringing Multi-cluster Management features for Kubernetes and specially for OpenShift in a flavour called Red Hat Advanced Cluster Management.
We can use RHCAM to ensure our cheese-quizz application is deployed on one or more cluster.
On the Hub cluster, start creating the required namespaces:
kubectl create ns cheese-quizz
kubectl create ns githubcom-lbroudoux-cheese-quizz-ns
Then apply the resource from the ocmfiles/
folder:
oc apply -f ocmfiles/cheese-quizz-channel.yml
oc apply -f ocmfiles/cheese-quizz-application.yml
The application PlacementRules
are such that every registered cluster having the label environment=dev
will received a deployment of the application.
We can check in below screenshot that the customization required have being applied (2 pods present in ReplicaSet
):
Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. The project was admitted into CNCF in April 2020 and since then it attracted more and more users and contributors.
Argo CD can be easily installed on Kubernetes through an Operator and be tighly integrated with OpenShift RBAC system (see this blog). It's really easy to install on OpenShift has shown in this video.
You can reuse the
ArgoCD
custom resource here.
It's really easy to setup Argo for the host Kubernetes cluster deployment. Here below we're going to detail how to do the setup in a multi-cluster environment to compare with OCM/RHACM above.
First login to remote cluster - ie. we one you didn't install ArgoCD on - using a user having cluster:admin
role and define a Kube context:
oc login https://api.cluster1.example.opentlc.com:6443/
oc config rename-context $(oc config current-context) cluster1
Once done, you have to login to ArgoCD using the argocd
CLI tool. Because we have setup the OpenShift login integration, we have to specify --sso
flag.
argocd --insecure login argocd-server-argocd.apps.cluster0.example.opentlc.com:443 --sso
This command should open a browser for authenticating. If like me you encounter trouble with Safari (default browser) not resolving localhost
, you may just have to curl
the callback URL in a terminal for finalizing the login process.
Then, you can now list the clusters:
$ argocd --insecure cluster list
SERVER NAME VERSION STATUS MESSAGE
https://kubernetes.default.svc in-cluster Unknown Cluster has no application and not being monitored.
And add a new one from our Kube context before listing them again:
$ argocd --insecure cluster add cluster1
INFO[0000] ServiceAccount "argocd-manager" created in namespace "kube-system"
INFO[0000] ClusterRole "argocd-manager-role" created
INFO[0000] ClusterRoleBinding "argocd-manager-role-binding" created
Cluster 'https://api.cluster1.example.opentlc.com:6443' added
$ argocd --insecure cluster list
SERVER NAME VERSION STATUS MESSAGE
https://api.cluster1.example.opentlc.com:6443 cluster1 1.19 Successful
https://kubernetes.default.svc in-cluster Unknown Cluster has no application and not being monitored.
Final step, you just have to create the Application
within ArgoCD. You can do that through UI or just by creating the Application
resource within argocd
namespace (where we install ArgoCD on cluster0).
oc apply -f argofiles/cheese-quizz-application.yml
We can check in below screenshot that the customization required have being applied (2 pods present in ReplicaSet
):