Selenium is a browser automation tool used primarily for testing web applications. However when Selenium is used in a CI pipeline to test applications, there is often contention around the use of Selenium resources. This example shows you how to deploy Selenium to Kubernetes in a scalable fashion.

This example assumes you have a working Kubernetes cluster and a properly configured kubectl client. See the Getting Started Guides for details.

Google Container Engine is also a quick way to get Kubernetes up and running: https://cloud.google.com/container-engine/

Your cluster must have 4 CPU and 6 GB of RAM to complete the example up to the scaling portion.

We will be using Selenium Grid Hub to make our Selenium install scalable via a master/worker model. The Selenium Hub is the master, and the Selenium Nodes are the workers(not to be confused with Kubernetes nodes). We only need one hub, but we're using a replication controller to ensure that the hub is always running:

kubectl create --filename=staging/selenium/selenium-hub-deployment.yaml

The Selenium Nodes will need to know how to get to the Hub, let's create a service for the nodes to connect to.

kubectl create --filename=staging/selenium/selenium-hub-svc.yaml

Let's verify our deployment of Selenium hub by connecting to the web console.

If your Kubernetes nodes are reachable from your network, you can verify the hub by hitting it on the nodeport. You can retrieve the nodeport by typing kubectl describe svc selenium-hub, however the snippet below automates that by using kubectl's template functionality:

export NODEPORT=`kubectl get svc --selector='app=selenium-hub' --output=template --template="{{ with index .items 0}}{{with index .spec.ports 0 }}{{.nodePort}}{{end}}{{end}}"`
export NODE=`kubectl get nodes --output=template --template="{{with index .items 0 }}{{.metadata.name}}{{end}}"`

curl https://$NODE:$NODEPORT

If you cannot reach your Kubernetes nodes from your network, you can proxy via kubectl.

export PODNAME=`kubectl get pods --selector="app=selenium-hub" --output=template --template="{{with index .items 0}}{{.metadata.name}}{{end}}"`
kubectl port-forward $PODNAME 4444:4444

In a separate terminal, you can now check the status.

curl https://localhost:4444

If you are using Google Container Engine, you can expose your hub via the internet. This is a bad idea for many reasons, but you can do it as follows:

kubectl expose deployment selenium-hub --name=selenium-hub-external --labels="app=selenium-hub,external=true" --type=LoadBalancer

Then wait a few minutes, eventually your new selenium-hub-external service will be assigned a load balanced IP from gcloud. Once kubectl get svc selenium-hub-external shows two IPs, run this snippet.

export INTERNET_IP=`kubectl get svc --selector="app=selenium-hub,external=true" --output=template --template="{{with index .items 0}}{{with index .status.loadBalancer.ingress 0}}{{.ip}}{{end}}{{end}}"`

curl https://$INTERNET_IP:4444/

You should now be able to hit $INTERNET_IP via your web browser, and so can everyone else on the Internet!

Now that the Hub is up, we can deploy workers.

This will deploy 2 Chrome nodes.

kubectl create --filename=staging/selenium/selenium-node-chrome-deployment.yaml

And 2 Firefox nodes to match.

kubectl create --filename=staging/selenium/selenium-node-firefox-deployment.yaml

Once the pods start, you will see them show up in the Selenium Hub interface.

Let's run a quick Selenium job to validate our setup.

First, we need to start a python container that we can attach to, then get inside this container.

kubectl run selenium-python --tty -i --image=python:slim bash

Once inside, we need to install the Selenium library

pip install selenium

We're all set up, start the python interpreter.

python

And paste in the contents of selenium-test.py.

from selenium import webdriver

def check_browser(browser):
  if browser == "CHROME":
    options = webdriver.ChromeOptions()
  elif browser == "FIREFOX":
    options = webdriver.FirefoxOptions()
  driver = webdriver.Remote(
    command_executor='https://selenium-hub:4444/wd/hub',
    options=options
  )
  driver.get("https://www.google.com")
  assert "google" in driver.page_source
  driver.quit()
  print("Browser %s checks out!" % browser)


check_browser("FIREFOX")
check_browser("CHROME")

You should get

>>> check_browser("FIREFOX")
Browser FIREFOX checks out!
>>> check_browser("CHROME")
Browser CHROME checks out!

Congratulations, your Selenium Hub is up, with Firefox and Chrome nodes!

If you need more Firefox or Chrome nodes, your hardware is the limit:

kubectl scale deployment selenium-node-firefox --replicas=10
kubectl scale deployment selenium-node-chrome --replicas=10

You now have 10 Firefox and 10 Chrome nodes, happy Seleniuming!

Sometimes it is necessary to check on a hung test. Each pod is running VNC. To check on one of the browser nodes via VNC, it's recommended that you proxy, since we don't want to expose a service for every pod, and the containers have a weak VNC password. Replace POD_NAME with the name of the pod you want to connect to.

kubectl port-forward $POD_NAME 5900:5900

Then connect to localhost:5900 with your VNC client using the password "secret"

Enjoy your scalable Selenium Grid!

Adapted from: https://github.com/SeleniumHQ/docker-selenium

To remove all created resources, run the following:

kubectl delete deployment selenium-hub
kubectl delete deployment selenium-node-chrome
kubectl delete deployment selenium-node-firefox
kubectl delete deployment selenium-python
kubectl delete svc selenium-hub