Skip to content

eformat/ocp4-sno-inplace

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

8 Commits
README-4.10.md
README-4.10.md
 
 
README-4.11.md
README-4.11.md
 
 
README.md
README.md
 
 

Repository files navigation

OCP4 SNO Libvirt Bootstrap In-Place

This uses a single instance qemu+libvirt vm for installing OpenShift 4 - SNO in a home lab.

I'm using fc37, AMD gen3 ryzen 5 (6 physical cores), 96 GB RAM on the lab host.

FC37

My fedora core37 has libvirt and hugepages configured based on this

Setup

Create a place to work

mkdir sno-no-ai && cd sno-no-ai

Download the ocp install binaries

Cli and Installer

# the latest OpenShift 4.12 CLI binaries
wget https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/latest-4.12/openshift-client-linux.tar.gz
wget https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/latest-4.12/openshift-install-linux.tar.gz

Unpack

# unpackage them
tar xzf openshift-client-linux.tar.gz
tar xzf openshift-install-linux.tar.gz
# move to the path
chmod 755 kubectl oc openshift-install
VERSION=4.12.1
mv kubectl $HOME/bin/kubectl-$VERSION
mv oc $HOME/bin/oc-$VERSION
mv openshift-install $HOME/bin/openshift-install-$VERSION
ln -sf $HOME/bin/oc-$VERSION $HOME/bin/oc
ln -sf $HOME/kubectl-$VERSION $HOME/bin/kubectl
ln -sf $HOME/bin/openshift-install-$VERSION $HOME/bin/openshift-install
# clean up
rm -f *.gz README.md

Get the coreos-installer binary

wget https://mirror.openshift.com/pub/openshift-v4/clients/coreos-installer/latest/coreos-installer
chmod 755 coreos-installer
mv coreos-installer $HOME/bin

Get the RHCOS ISO from the installer and download

# get the RHCOS live ISO
ISO_URL=$(openshift-install coreos print-stream-json | grep location | grep x86_64 | grep iso | cut -d\" -f4)
curl -L ${ISO_URL} > rhcos-live.x86_64.iso

Create an install config for SNO, adjust ip's, hosts name, secrets to suit

cat << EOF > install-config.yaml
apiVersion: v1
baseDomain: eformat.me
metadata:
  name: sno
networking:
  networkType: OVNKubernetes
  machineNetwork:
  - cidr: 10.1.1.0/24
compute:
- name: worker
  replicas: 0
controlPlane:
  name: master
  replicas: 1
platform:
  none: {}
bootstrapInPlace:
  installationDisk: /dev/vda
pullSecret: '<your pull secret>'
sshKey: |
  <your ssh public key to access host>
EOF

As root, create a libvirt network, adjust to suit

cat <<EOF > /etc/libvirt/qemu/networks/sno.xml
<network>
  <name>sno</name>
  <uuid>fc43091e-de21-4bf5-974b-98711b9f3d9d</uuid>
  <forward mode='nat' size='1500'/>
  <bridge name='tt0' stp='on' delay='0'/>
  <mac address='52:54:00:29:4d:d7'/>
  <domain name='eformat.me'/>
  <dns enable='yes'>
    <host ip='10.1.1.10'>
      <hostname>api.sno.eformat.me</hostname>
      <hostname>api-int.sno.eformat.me</hostname>
      <hostname>console-openshift-console.apps.sno.eformat.me</hostname>
      <hostname>oauth-openshift.apps.sno.eformat.me</hostname>
      <hostname>canary-openshift-ingress-canary.apps.sno.eformat.me</hostname>
    </host>
  </dns>
  <ip family='ipv4' address='10.1.1.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='10.1.1.2' end='10.1.1.254'/>
      <host mac='52:54:00:29:1d:1a' name='sno' ip='10.1.1.10'/>
    </dhcp>
  </ip>
</network>
EOF

Auto start it

virsh net-define /etc/libvirt/qemu/networks/sno.xml
virsh net-start sno
virsh net-autostart sno

Set up dnsmasq pointing to gateway on libvirt network

echo server=/api.sno.eformat.me/10.1.1.1 | sudo tee /etc/NetworkManager/dnsmasq.d/openshift.conf
echo -e "[main]\ndns=dnsmasq" | sudo tee /etc/NetworkManager/conf.d/openshift.conf
sudo systemctl reload NetworkManager.service

Storage - choose between thin-lvm OR qcow2 using a storage pool. i use thin-lvm.

qcow2, setup a storage pool, adjust to suit

cat <<EOF > /root/sno.xml
<pool type='dir'>
  <name>sno</name>
  <uuid>821c97f0-e6f7-403a-8c84-9a67f7607e9f</uuid>
  <capacity unit='bytes'>999581282304</capacity>
  <allocation unit='bytes'>141831270400</allocation>
  <available unit='bytes'>857750011904</available>
  <source>
  </source>
  <target>
    <path>/var/lib/libvirt/openshift-images</path>
    <permissions>
      <mode>0755</mode>
      <owner>0</owner>
      <group>0</group>
      <label>unconfined_u:object_r:virt_image_t:s0</label>
    </permissions>
  </target>
</pool>
EOF
virsh pool-define /root/sno.xml
virsh vol-delete --pool sno sno

OR thin-lvm, create, change names to suit

lvcreate --size 200G --type thin-pool --thinpool thin_pool fedora_fedora
lvcreate --virtualsize 120G --name sno -T fedora_fedora/thin_pool
vgchange -ay -K fedora_fedora

Create all the OCP ignition and iso bits&pieces, set OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE to the version you want to install

rm -rf cluster
mkdir cluster && cp install-config.yaml cluster/
export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=quay.io/openshift-release-dev/ocp-release:4.12.1-x86_64
openshift-install create single-node-ignition-config --dir=cluster
sudo rm -f /var/lib/libvirt/images/rhcos-live.x86_64.iso
sudo $HOME/bin/coreos-installer iso ignition embed -fi cluster/bootstrap-in-place-for-live-iso.ign rhcos-live.x86_64.iso -o /var/lib/libvirt/images/rhcos-live.x86_64.iso
sudo chown qemu:qemu /var/lib/libvirt/images/rhcos-live.x86_64.iso
sudo restorecon -rv /var/lib/libvirt/images/rhcos-live.x86_64.iso

Installation

As root, set these env vars, change to suit

VM_NAME=sno
NET_NAME=sno
POOL_NAME=sno
RAM_MB="32768"
DISK_GB="120"
CPU_CORE="8"
RHCOS_ISO=/var/lib/libvirt/images/rhcos-live.x86_64.iso
MAC=52:54:00:29:1d:1a

Install OpenShift SNO

rm -f nohup.out
nohup virt-install \
    --virt-type kvm \
    --connect qemu:https:///system \
    -n "${VM_NAME}" \
    -r "${RAM_MB}" \
    --memorybacking hugepages=yes \
    --vcpus "${CPU_CORE}" \
    --os-type=linux \
    --cpu=host-passthrough,cache.mode=passthrough \
    --import \
    --network=network:${NET_NAME},mac=${MAC},driver.queues=4 \
    --events on_reboot=restart \
    --cdrom "${RHCOS_ISO}" \
    --disk path=/dev/fedora_fedora/sno,io=io_uring,cache='writeback',discard='unmap' \
    --boot hd,cdrom \
    --wait=-1 &

Use this disk line for qcow2 if you wanted to use that instead of thin-lvm

    --disk pool=sno,size="${DISK_GB},io=io_uring" \

Use this to disable mac check

    --check mac_in_use=off \

Watch the installer

export KUBECONFIG=~/sno-no-ai/cluster/auth/kubeconfig
openshift-install --dir=cluster --log-level debug wait-for install-complete

and oc get co once installer has progressed.

All going well, OpenShift SNO will install, Profit !

Thin LVM Snapshots, Rollbacks

Create snapshot lvms i.e. a restore point (i am using sno-local for openshift local storage here as another disk)

lvcreate -n sno-snap1 -s fedora_fedora/sno
lvcreate -n sno-local-snap1 -s fedora_fedora/sno-local

Stop vm. Restore to snapshots.

lvconvert --merge /dev/fedora_fedora/sno-snap1
lvconvert --merge /dev/fedora_fedora/sno-local-snap1

Restart SNO instance. I have been successfully using this technique to test cluster busting things, then roll back to the thin-lvm snapshot.

Notes

You can ssh to the node after it boots to check out whats going on

ssh [email protected]
sudo crictl ps

The virt install i use is optimized for

  • hugepages
  • host-passthrough cpu
  • more network lanes
  • io_uring kernel io

On my AMD host, i also found setting the Memory Speed to 2667MHz in the bios (rather than its max. 3200Mhz) prevented soft lockup type errors in the kernel

general protection fault, probably for non-canonical address 0x9a158ca82eabad75: 0000 [#1] PREEMPT SMP NOPTI

Some links for the needy

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

4 stars

Watchers

2 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published