How to create a custom Tanzu Kubernetes Grid (TKG) Node OVA based on Photon OS Real Time Kernel?
One really cool feature of Tanzu Kubernetes Grid (TKG) is the ability to bring your custom images (BYOI) which can then be used to deploy TKG Workload Clusters. To do so, customers will need to use Kubernetes (K8s) Image Builder tool to author new OVA images and then make TKG aware by updating the Tanzu Kubernetes Release (TKR) Build of Materials (BOM) configuration. I had played around with Image Builder awhile back during the TKG 1.2 release and it definitely was not very easy to use. I have been meaning to kick the tires on Image Builder again as I know with the latest 1.3.x release, there have been a number of improvements. This week I saw an inquiry from my buddy Alan Renouf who was looking to see if there was a way to use the new Photon OS Real Time Kernel as a base image for a K8s-based application that he was working with that had requirements for the real time kernel. Interestingly enough, there was another inquiry with a similiar customer request for their edge deployment and I thought this would be a good opportunity to try out Image Builder again, which has been overhauled and the build process can be completely consumed as a Docker container, which definitely made things much easier than before. I also had never played with real time version of Photon OS, so this gave me a reason to try that out which was initially introduced with Photon OS 4.0 but it also looks like real time kernel was added to 3.0 recently, which is the version I had used to test. Note: vSphere with Tanzu currently does not support the ability to bring your own image like TKG, I know this is something that has been asked about and is being considered in the future. The BYOI process for TKG is comprised of two steps:  Create Custom TKG OVA  Update TKG with new TKR BOM  Although there are detailed documentation for this process, I still ran into a number of issues which I think the documentation could be improved with a complete working example rather than using generic values which lead to some interpretation, which I did not interpret correctly the first time through. After posting some questions in the Image Builder Slack Channel, I was able to finally connect the dots with the help from Scott Rosenberg, who I also knew, as a customer of our VMware Event Broker Appliance (VEBA) Fling. Putting everything together, I figure it would be useful to document the process I took and hopefully this can benefit other customers looking to build and consume their own OVA images with TKG. Create Custom TKG OVA  In the example below, I will be creating a custom Photon OS 3.0 Real Time OVA supporting the latest TKG K8s release which is v0.20.5. If you wish to build for other K8s versions, you can download the respective image builder configuration files here. Before getting started, you will need the following:  Docker installed on a Linux VM which will run the build process. I used a basic Ubuntu 18.04 installation which I had running in my vSphere environment. I was not able to use Docker on my MacOS desktop and this was the only way I was able to get the network connectivity from the Docker Client to my vSphere setup  vSphere environment with vCenter Server running, you will need to provide credentials for Image Builder to create and install OS  Existing TKG 1.3 environment running in your vSphere environment. If you do not have one, I recommend taking a look at my TKG Demo Appliance Fling, which will help get you setup in just 10 minutes and is also the same setup I am using for my enviornment  Step 1 - Download the Image Builder for k8s v1.20.5 configuration  curl -L https://vdc-download.vmware.com/sampleExchange/v1/downloads/7603 -o TKG-Image-Builder-for-Kubernetes-v1.20.5-master.zip  unzip TKG-Image-Builder-for-Kubernetes-v1.20.5-master.zip  cd TKG-Image-Builder-for-Kubernetes-v1.20.5-master/TKG-Image-Builder-for-Kubernetes-v1.20.5/  Step 2 - Download latest OVFTool for Linux installer and place in the extracted TKG Image Builder folder from Step 1  Step 3 - Create a file called Dockerfile with the following content which will be used to build our Image Builder Container:  FROM k8s.gcr.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.9 USER root ENV LC_CTYPE=POSIX ENV OVFTOOL_FILENAME=VMware-ovftool-4.4.1-16812187-lin.x86_64.bundle ADD $OVFTOOL_FILENAME /tmp/ RUN /bin/sh /tmp/$OVFTOOL_FILENAME --console --required --eulas-agreed && \ rm -f /tmp/$OVFTOOL_FILENAME USER imagebuilder ENV IB_OVFTOOL=1 1 2 3 4 5 6 7 8 9 FROM k8s . gcr . io / scl - image - builder / cluster - node - image - builder - amd64 : v0 . 1.9 USER root ENV LC_CTYPE = POSIX ENV OVFTOOL_FILENAME = VMware - ovftool - 4.4.1 - 16812187 - lin . x86_64 . bundle ADD $ OVFTOOL_FILENAME / tmp / RUN / bin / sh / tmp / $ OVFTOOL_FILENAME -- console -- required -- eulas - agreed && \ rm - f / tmp / $ OVFTOOL_FILENAME USER imagebuilder ENV IB_OVFTOOL = 1  Step 4 - Build the Image Builder Container by running the following command:  docker build . -t projects.registry.vmware.com/tkg/imagebuilder-byoi:v0.1.9  Step 5 - Create a file called vsphere.json that contains the following along with the values of your own environment:  { "vcenter_server": "192.168.30.3", "username": "*protected email*", "password": "VMware1! ", "insecure_connection": "true", "datacenter": "Primp-Datacenter", "cluster": "Supermicro-Cluster", "resource_pool": "", "folder": "Workloads", "datastore": "sm-vsanDatastore", "network": "VM Network", "convert_to_template": "false", "create_snapshot": "false", "linked_clone": "false", "template": "false" } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 { "vcenter_server" : "192.168.30.3" , "username" : "*protected email*" , "password" : "VMware1!" , "insecure_connection" : "true" , "datacenter" : "Primp-Datacenter" , "cluster" : "Supermicro-Cluster" , "resource_pool" : "" , "folder" : "Workloads" , "datastore" : "sm-vsanDatastore" , "network" : "VM Network" , "convert_to_template" : "false" , "create_snapshot" : "false" , "linked_clone" : "false" , "template" : "false" }  Step 6 - Create a file called photon-3.json which will be used to override the Photon OS 3.0 ISO with the Photon OS 3.0 Real-Time ISO and only two lines that should change is iso_checksum and iso_url:  { "boot_command_prefix": "<esc><wait> vmlinuz initrd=initrd.img root/dev/ram0 loglevel=3 photon.media=cdrom ks=", "boot_command_suffix": "/3/ks.json<enter><wait>", "boot_media_path": "http://{{ .HTTPIP }}:{{ .HTTPPort }}", "build_name": "photon-3-rt", "distro_arch": "amd64", "distro_name": "photon", "distro_version": "3", "guest_os_type": "vmware-photon-64", "http_directory": "./packer/ova/linux/{{user `distro_name`}}/http/", "iso_checksum": "bba070a9f491f9cb9c2fb1dff731d628576ae3e3", "iso_checksum_type": "sha1", "iso_url": "https://packages.vmware.com/photon/3.0/Rev3/iso/photon-rt-3.0-a383732.iso", "os_display_name": "VMware Photon OS 64-bit", "shutdown_command": "shutdown now", "vsphere_guest_os_type": "vmwarePhoton64Guest" } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 { "boot_command_prefix" : "<esc><wait> vmlinuz initrd=initrd.img root/dev/ram0 loglevel=3 photon.media=cdrom ks=" , "boot_command_suffix" : "/3/ks.json<enter><wait>" , "boot_media_path" : "http://{{ .HTTPIP }}:{{ .HTTPPort }}" , "build_name" : "photon-3-rt" , "distro_arch" : "amd64" , "distro_name" : "photon" , "distro_version" : "3" , "guest_os_type" : "vmware-photon-64" , "http_directory" : "./packer/ova/linux/{{user `distro_name`}}/http/" , "iso_checksum" : "bba070a9f491f9cb9c2fb1dff731d628576ae3e3" , "iso_checksum_type" : "sha1" , "iso_url" : "https://packages.vmware.com/photon/3.0/Rev3/iso/photon-rt-3.0-a383732.iso" , "os_display_name" : "VMware Photon OS 64-bit" , "shutdown_command" : "shutdown now" , "vsphere_guest_os_type" : "vmwarePhoton64Guest" }  Step 7 - Create a file called ks.json which will be used to override the default kickstart and specify linux-rt kernel flavor along with the tuned package. If you need to make other changes, you can update the package list as this is just the standard Kickstart configuration file for Photon OS. { "disk": "/dev/sda", "hostname": "localhost", "packages": [ "bash", "bc", "bridge-utils", "bzip2", "ca-certificates", "cloud-init", "cpio", "cracklib-dicts", "dbus", "e2fsprogs", "file", "filesystem", "findutils", "gdbm", "grep", "gzip", "iana-etc", "initramfs", "iptables", "iproute2", "iputils", "libtool", "linux-rt", "motd", "net-tools", "openssh-server", "open-vm-tools", "pkg-config", "photon-release", "photon-repos", "procps-ng", "rpm", "sed", "sudo", "tdnf", "tuned", "tzdata", "util-linux", "vim", "which" ], "password": { "age": -1, "crypted": true, "text": "*" }, "postinstall": [ "#!/bin/sh", "useradd -U -d /home/builder -m --groups wheel builder && echo 'builder:builder' | chpasswd", "echo 'builder ALL=(ALL) NOPASSWD: ALL' >/etc/sudoers.d/builder", "chmod 440 /etc/sudoers.d/builder", "systemctl enable sshd", "tdnf clean all", "swapoff -a", "rm -f /swapfile" ] } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 { "disk" : "/dev/sda" , "hostname" : "localhost" , "packages" : [ "bash" , "bc" , "bridge-utils" , "bzip2" , "ca-certificates" , "cloud-init" , "cpio" , "cracklib-dicts" , "dbus" , "e2fsprogs" , "file" , "filesystem" , "findutils" , "gdbm" , "grep" , "gzip" , "iana-etc" , "initramfs" , "iptables" , "iproute2" , "iputils" , "libtool" , "linux-rt" , "motd" , "net-tools" , "openssh-server" , "open-vm-tools" , "pkg-config" , "photon-release" , "photon-repos" , "procps-ng" , "rpm" , "sed" , "sudo" , "tdnf" , "tuned" , "tzdata" , "util-linux" , "vim" , "which" ] , "password" : { "age" : - 1 , "crypted" : true , "text" : "*" } , "postinstall" : [ "#!/bin/sh" , "useradd -U -d /home/builder -m --groups wheel builder && echo 'builder:builder' | chpasswd" , "echo 'builder ALL=(ALL) NOPASSWD: ALL' >/etc/sudoers.d/builder" , "chmod 440 /etc/sudoers.d/builder" , "systemctl enable sshd" , "tdnf clean all" , "swapoff -a" , "rm -f /swapfile" ] }  Step 9 - Create a file called metadata.json which contains the following and update latter part of the string with something more meaningful to your organization (e.g. myorg.0). The content of this file is what will be encoded to our customer TKG OVA and let TKG know which OVA can be used for a given TKR. { "VERSION": "v1.20.5+vmware.2-williamlam.0" } 1 2 3 { "VERSION" : "v1.20.5+vmware.2-williamlam.0" }  Step 10 - Create the following directory which will contain the OVA produced by Image Builder:  mkdir output  Step 11 - Create our custom OVA image using Image Builder by running the following command:  sudo docker run --net=host -it --rm \  -v $(pwd)/vsphere.json:/home/imagebuilder/vsphere.json \  -v $(pwd)/tkg.json:/home/imagebuilder/tkg.json \  -v $(pwd)/tkg:/home/imagebuilder/tkg \  -v $(pwd)/goss/vsphere-photon-1.20.5+vmware.2-goss-spec.yaml:/home/imagebuilder/goss/goss.yaml \  -v $(pwd)/photon-3.json:/home/imagebuilder/packer/ova/photon-3.json \  -v $(pwd)/ks.json:/home/imagebuilder/packer/ova/linux/photon/http/3/ks.json \  -v $(pwd)/metadata.json:/home/imagebuilder/metadata.json \  -v $(pwd)/output:/home/imagebuilder/output \  --env PACKER_VAR_FILES="tkg.json vsphere.json" \  --env OVF_CUSTOM_PROPERTIES=/home/imagebuilder/metadata.json \  projects.registry.vmware.com/tkg/imagebuilder-byoi:v0.1.9 \  build-node-ova-vsphere-photon-3  The build process will take some time and in my environment, it took ~23 minutes to complete and you will find the final OVA stored in output/photon-3-rt-kube-v1.20.5/photon-3-rt-kube-v1.20.5+vmware.2.ova    Step 12 - Finally, for TKG to make use of our custom OVA, we need to upload the OVA produced by Image Builder back into our vSphere environment. You can do this by using the vSphere UI or you can automate using PowerCLI or OVFTool. The example below uses OVFTool (which I also had installed in the Image Builder VM) so that I did not have to download and then re-upload. ovftool --acceptAllEulas --net:nic0=Workload --datastore=sm-vsanDatastore --vmFolder=Templates --importAsTemplate output/photon-3-rt-kube-v1.20.5/photon-3-rt-kube-v1.20.5+vmware.2.ova 'vi://*protected email*:[email protected]/Primp-Datacenter/host/Supermicro-Cluster/'  Note: You will notice that after Image Builder has completed, the VM which was exported as an OVA will continue to exists in the vSphere Inventory. This can be safely deleted as it is no longer needed once the export has completed. Update TKG with new TKR BOM  Step 1 - Make a copy of the latest Tanzu Kubernetes Release (TKR) BOM file which must conform to a specific format. For example, the "+" must be converted into "---" and version syntax should match the original and you can prefix with your own name for readability. In this example, I have named it realtime-v1.20.5---vmware.2-tkg.1.yaml  cp .tanzu/tkg/bom/tkr-bom-v1.20.5+vmware.2-tkg.1.yaml .tanzu/tkg/bom/realtime-v1.20.5---vmware.2-tkg.1.yaml  Step 2 - Edit the duplicated TKR file and under the "ova" section, replace photon definition with the one shown below. It is important to note that the version value to match what was defined in the previous section on Step 9.  ova: - name: ova-photon-3-rt osinfo: name: photon version: "3" arch: amd64 version: v1.20.5+vmware.2-williamlam.0 - name: ova-ubuntu-2004 osinfo: name: ubuntu version: "20.04" arch: amd64 version: v1.20.5+vmware.2-tkg.1-6700972457122900687 1 2 3 4 5 6 7 8 9 10 11 12 13 ova : - name : ova - photon - 3 - rt osinfo : name : photon version : "3" arch : amd64 version : v1 . 20.5 + vmware . 2 - williamlam . 0 - name : ova - ubuntu - 2004 osinfo : name : ubuntu version : "20.04" arch : amd64 version : v1 . 20.5 + vmware . 2 - tkg . 1 - 6700972457122900687  Step 3 - To make TKG aware of our custom TKR, we need to create a new K8s ConfigMap that contains the base64 encoding of our TKR BOM file, you can do so by running the following command which will produce the YAML manifest called realtime-tkr.yaml or any other name you wish to use. TKR_BASE64_ENCODE=$(cat .tanzu/tkg/bom/realtime-v1.20.5---vmware.2-tkg.1.yaml|base64 -w0) cat > realtime-tkr.yaml <<EOF apiVersion: v1 kind: ConfigMap metadata: name: realtime-v1.20.5---vmware.2-tkg.1 labels: tanzuKubernetesRelease: realtime-v1.20.5---vmware.2-tkg.1 binaryData: bomContent: ${TKR_BASE64_ENCODE} EOF 1 2 3 4 5 6 7 8 9 10 11 TKR_BASE64_ENCODE = $ ( cat .tanzu / tkg / bom / realtime - v1 . 20.5 -- - vmware . 2 - tkg . 1.yaml | base64 - w0 ) cat > realtime - tkr .yaml << EOF apiVersion : v1 kind : ConfigMap metadata : name : realtime - v1 . 20.5 -- - vmware . 2 - tkg . 1 labels : tanzuKubernetesRelease : realtime - v1 . 20.5 -- - vmware . 2 - tkg . 1 binaryData : bomContent : $ { TKR_BASE64_ENCODE } EOF  Step 4 - Next, we create our new TKR ConfigMap by running the following command:  kubectl -n tkr-system apply -f realtime-tkr.yaml  Step 5 - For the change to go into effect, we need to delete the existing TKR Controller pod by running the following command:  kubectl -n tkr-system delete pod $(kubectl get pod -n tkr-system | awk '{print $1}' | tail -1)  Step 6 - Once the TKR Controller has completed the discovery of our new TKR BOM, we should now be able to check that our TKR is now available by running the following command:  tanzu kubernetes-release get    Step 7 - Finally, we can now create new TKG Workload Cluster using our new custom TKR. For an example TKG YAML deployment file, please refer this example here and update with your environment configuration. tanzu cluster create -f vsphere-tkg-workload-1-template.yaml --tkr realtime-v1.20.5---vmware.2-tkg.1  tanzu cluster kubeconfig get --admin tkg-cluster-01  kubectl config use-context [email protected]  After switching the context to our new TKG Workload Cluster, we can see the controlplane and worker nodes that were deployed is based on our custom OVA and we can see that it is also running the Photon OS Real-Time kernel and is now ready for your workloads!