Steve Walker

September 17, 2025September 30, 2025

VCF 9 and VVF 9 version mismatch between vCenter and ESX lab installation

This issue appears to have been resolved with the coordinated release of both an updated ESX and vCenter version in the VCF 9.0.1.0 release
Update following GA release of ESX 9.0.1.0.24957456 and vCenter Server 9.0.1.0.24957454

Following on from the excitement of VMware Explore in Las Vegas and receiving my new VMUG Advantage licenses for VCF 9 I have been testing various deployment choices for a new VVF 9 environment in my lab using the GA release of vCenter (9.0.0.0.24755230).

When downloading the various files to populate the offline depot for my cloud installer appliance to consume I noticed that there were two versions of ESX, an older one in both .ISO form (installer) and .ZIP (offline bundle), and a newer one in only the offline bundle.

Here’s what the GA version files look like in the drop down when displaying the original 9.0.0.0 version:

9.0.0.0 offline bundle and ISO downloads

The small drop down box on the top left of the VMware ESX panel can be used to select which version should be displayed, but as you will see below, when selecting the option for 9.0.0.0100 the only file available is an offline bundle.

I had decided to build a custom ISO including some of the VMware Flings (which I thought would also be required to get a working VCF 9 deployment in the future) and used the newer version of ESX (9.0.0.0100) to install my ESX servers prior to executing the VVF installer process.

Mostly this has been a trouble free assumption, other than the warning displayed by the VVF cloud installer that I was using an unexpected version of ESX on my target hosts. I didn’t have any real reason to doubt this process (because the newer version of ESX fixes some critical CVEs) and continued with setting up my environment successfully.

However, during several repeated installations of my Supervisor cluster I ran into issues where the spherelet VIB didn’t always uninstall/reinstall correctly, and was required to remediate the cluster against the ‘autogen-software-spec-1’ lifecycle manager image – which is autogenerated using the version of ESX and any vendor plugins or components used at installation time.

autogen-software-spec-1 lifecycle manager image choices

At this point I noticed that often the remediation might fail because it would skip each of the hosts due to a ‘supposed’ hardware incompatibility and not complete the rest of the remediation. These warnings can be silenced within the vCenter UI under cluster object, Monitor tab, vSAN, Skyline Health.

Silence those alerts if you’re not interested in maintaining compatibility with the HCL (because it’s a lab environment for instance).

In trying to chase down the further cause of the problem I saw the orange ribbon displayed on the Hardware Compatibility tab:

“Requested target version is not supported for the cluster.”

Select the cluster object, choose Updates tab and examine the Hardware Compatibility.

This seems strange, because the remediation will eventually complete anyway because I am not using the remediation option to force hardware compatibility before starting. The following screen shows the same message in a different position.

Successful remediation messages on the cluster tab

Why isn’t the offline-depot from the 9.0.0.0100 version actually supported, even though the auto-gen profile is created automatically and the offline-depot file is imported? It seems that this just comes down to the release dates of vCenter and the subsequent ESX update. You can see this in more detail by examining the following file on the vCenter appliance:

/var/log/vmware/vmware-updatemgr/vum-server/hcl_python_lib.log

Here are the relevant points which called out to me:

Called to discover Hardware Compatibility List
Unknown version 9.0.0-0100.24813472
Cannot find target version (9.0.0-0100.24813472) in VCG

2025-09-17T04:30:13.461Z INFO report.hcl Called to discover HCL for hostId host-34 with target version 9.0.0-0100.24813472, vSanHclConstraints = True.
2025-09-17T04:30:13.462Z ERROR compatibility.releases Got exception while using cache.
Traceback (most recent call last):
  File "/usr/lib/vmware-updatemgr/python/hcl/compatibility/releases.py", line 37, in getEsxiReleaseByVersion
    result = getCacheFactory().getReleasesCache().getByVersion(version)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/vmware-updatemgr/python/hcl/compatibility/cache/release_cache.py", line 72, in getByVersion
    return ProductRelease(releaseId=1, version=version)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/vmware-updatemgr/python/hcl/compatibility/vvs/models/product.py", line 38, in __init__
    raise ValueError("Unknown version %s" % version)
ValueError: Unknown version 9.0.0-0100.24813472
2025-09-17T04:30:13.462Z ERROR report.hcl Cannot find target version (9.0.0-0100.24813472) in VCG.

This would on the face of it seem strange, because the software depot even includes the version which is already deployed to the cluster and already knows the version which it presumably extracted from the host(s) when they were added to the cluster for the first time.

Is it as simple as the vCenter version is older than the ESX release, and that Broadcom haven’t yet released a patch for vCenter which recognises new versions of ESX?

Seems like we just need to get vCenter to agree that it’s not using the latest data and to search online for the updated information.

Eventually I stumbled across the menu option to synchronise the hardware compatibility settings and the release data:

Main vCenter menu dropdown, Lifecycle Manager, Actions drop down, Compatibility Data, Sync

The menu option to update the list of release versions which are supported by vCenter

Returning again to /var/log/vmware/vmware-updatemgr/vum-server/hcl_python_lib.log

2025-09-17T10:32:39.412Z INFO __main__ Loading VVS database from file /storage/updatemgr/patch-store/vvs/vvs-consolidated-bundle-download.json
2025-09-17T10:32:39.966Z INFO __main__ VVS database json loaded in-memory
2025-09-17T10:32:39.966Z INFO compatibility.cache Creating CacheFactory...
2025-09-17T10:32:39.970Z INFO compatibility.cache.executor Datastore locked
2025-09-17T10:32:39.970Z INFO __main__ Loading releases into datastore
2025-09-17T10:32:39.970Z INFO __main__ Releases to be loaded: 9
2025-09-17T10:32:39.971Z INFO __main__ Releases loaded
2025-09-17T10:32:39.971Z INFO __main__ Loading cpuseries into datastore
2025-09-17T10:32:39.971Z INFO __main__ Cpuseries to be loaded: 97
2025-09-17T10:32:39.972Z INFO __main__ Cpuseries loaded
2025-09-17T10:32:39.972Z INFO __main__ Loading servers into datastore
2025-09-17T10:32:40.365Z INFO __main__ Servers to be loaded: 5636
2025-09-17T10:32:40.709Z INFO __main__ Servers loaded
2025-09-17T10:32:40.709Z INFO __main__ Loading devices into datastore
2025-09-17T10:32:41.539Z INFO __main__ Devices to be loaded: 7033
2025-09-17T10:32:42.361Z INFO __main__ Devices loaded
2025-09-17T10:32:42.361Z INFO __main__ Loading OEM vendors into datastore
2025-09-17T10:32:42.361Z INFO __main__ OEM vendors to be loaded: 7
2025-09-17T10:32:42.362Z INFO __main__ OEM vendors loaded
2025-09-17T10:32:42.362Z INFO __main__ Updating datastore time to 1758091862563
2025-09-17T10:32:42.513Z INFO compatibility.cache.executor Datastore unlocked

In this case perhaps the issue isn’t that the versions aren’t being published necessarily but that the compatibility cache doesn’t yet include build ‘9.0.0.0100.24813472’.

Never afraid to reference a William Lam article, I would question why this release is supported for VCF 9 as described here: https://williamlam.com/2025/07/applying-1st-esx-live-patch-using-vcf-9-0-operations.html but not actually within VVF yet?

A quick check through the /storage/updatemgr/patch-store/vvs/vvs-consolidated-bundle-download.json file on my vCenter doesn’t show any release matching that version – so perhaps the initial error was to assume that this patch was also eligible at all for VVF 9 (and not as it seems for VCF 9 only) at this stage.

Perhaps we’ll find out with the next minor or patch release of vCenter if this omission is resolved, but at present it would appear that the best way to avoid these remediation problems in VVF is to only use the GA 9.0.0.0 release of the ESX ISO file.

September 6, 2025September 8, 2025

How much space does VCF 9 offline depot require?

During my own deployment testing of VMware Cloud Foundation 9, and its smaller counterpart VMware vSphere Foundation 9, I came across the need to determine exactly how much space is required if you want to set up an offline depot to install your first fleet member. In offline/airgap or edge locations where connectivity to the internet is either too slow or completely inaccessible VMware provides a great solution to be able to prepare a portable depot which could either be a virtual machine or laptop operating close to where you are going to run the VCF installer.

Previously the Cloud Builder appliance was very large (~27GB) because the image required a lot of space just to store the various files which it will need for the installation. The new combined VMware Cloud Foundation Installer/SDDC Manager .OVA download in VCF 9 replaces this element with a much smaller 2.15GB file because it doesn’t by default include the other components which will be installed later, e.g. VCF Operations, VCF Automation etc.

Instead you can choose to create an offline depot on a laptop or virtual machine and use this to bootstrap the VMware Cloud Foundation Installer with those files instead of having it download everything from the Broadcom site using your download token. This scenario is far more likely in my own experience, as downloading tens of GB when reaching a greenfield site (for any solution) is variously difficult and can take hours.

The product documentation states that the installer appliance requires a minimum of 914GB storage if thick provisioned, but this post will cover what is the bare minimum in order to get a working VM which will deploy the whole VCF suite.

Doing it the William Lam way

William Lam has several excellent articles which describe the process of (1) enabling a simple HTTP server using Python which will serve the files that are imported into the SDDC Manager when it is first deployed, and (2) these will be useful in setting up a depot (perhaps you’ll use a VMUG Advantage entitlement to download the files if it’s for a lab environment)

How to deploy VCF 9 using VMUG Advantage licenses: https://williamlam.com/2025/07/how-to-deploy-vvf-vcf-9-0-using-vmug-advantage-vcp-vcf-certification-entitlement.html

Using HTTP protocol to host the depot files: https://williamlam.com/2025/06/using-http-with-vcf-9-0-installer-for-offline-depot.html

Creating a simple Python web server: https://williamlam.com/2025/01/quick-tip-easily-host-vmware-cloud-foundation-vcf-offline-depot-using-python-simplehttpserver-with-authentication.html

Disable 10Gbit ethernet adapter speed check: https://williamlam.com/2025/06/disable-10gbe-nic-pre-check-in-the-vcf-9-0-installer.html

How much space?!

However rather than duplicating any of this above guidance, this post concentrates on exactly how much space you’ll need for either a VCF 9 or VVF 9 based offline-depot, ideally stored on a laptop or virtual machine acting as a web server. It could be costly in disk space terms to host these files permanently on your laptop, but there’s no reason why you can’t use an external SSD for this purpose – the question is what size will I need?

Also worth considering is where you’re going to put the VCF Installer virtual machine when you’re getting ready to bootstrap a vSAN environment. In this case you’ll need to find a VMFS datastore which is large enough for the installer plus the data which will be imported into the VM itself. The rest of your disks are probably going to be cleared of any partitions so you can’t use those to store data.

A fully populated offline depot which is capable of serving both VVF and VCF products to the SDDC Manager will require ~56GB on disk when combined with the other parts of the depot. The thin provisioned VM will use approximately 80GB when the offline-depot files have been uploaded to it, so if you’ve got a VMFS partition on a 250GB SSD or NVMe disk that is local to your ESXi server this should be sufficient to hold the installer VM before you eventually migrate it to vSAN.

Here’s how the space breaks down into the two products, each is neatly defined within the user interface of the VMware Cloud Foundation Installer:

VMware vSphere Foundation 9

You will need a total of 16.67GB free space to store the three files (.ova and .iso) comprising the three elements stored in the offline-depot.

VVF 9 space requirements in the offline-depot

VMware Cloud Foundation 9

You will need a total of 52GB free space to store the nine files (.ova, .tar, .vlcm and .iso) comprising the seven application elements stored in the offline-depot.

VCF 9 space requirements in the offline-depot

So there you have it, the whole VVF/VCF stack can be installed from these binaries using the VMware Cloud Foundation Installer, and the only element that you’ll need to install onto your target ESXi server initially is the 2GB installer VM (which can optionally become the actual SDDC Manager for the cluster once complete).

Product name	Fully populated depot size
VMware vSphere Foundation (VVF)	~16GB
VMware Cloud Foundation (VCF)	~56GB

August 5, 2025

Connecting Draytek Vigor 2927 to Azure VPN: Step-by-Step Guide

This week, I set out to re-connect my Draytek Vigor 2927 router to an Azure Virtual Network using a site-to-site VPN. While the process had a few challenges, the end result was a simple reliable connection between my on-premises network and Azure.

In this post, I’ll walk through the main steps, highlight a few tips, and share the Powershell commands I used once the Azure Powershell module was installed.

In this brief outline I am only attempting to create an IPsec site-to-site VPN tunnel using a preshared key, alternative steps are required to use an X.509 certificate for authentication.

Today, it is only possible to deploy the lowest cost ‘Basic’ SKU VPN Gateway in Azure using command line tools, and this gateway type requires a Basic public IP address. Coming up, in September 2025 only Standard IPs will be available for deployment, so this guide will likely need to be revised.

NB This guidance should only be followed in a test or lab environment where you can control the networking outcomes caused by reconfiguring your equipment and restarting it where necessary. Use with caution.

Step 1: Install and Connect Azure Powershell

Before running any commands, make sure the Azure Powershell module is installed. If not, you can do it with:

Install-Module -Name Az -AllowClobber -Scope CurrentUser

Once installed, open Powershell and log in to your Azure account:

Connect-AzAccount

Select the correct subscription if you have multiple under your tenant.

Step 2: Define initial Powershell variables

Based on my example values, define the variables which you will need during the process, at minimum you will need to replace the location, subnet prefix, local network prefix and preshared key:

$rg = 'SBC-Infrastructure-PAYG' #This is the resource group where the new resources will be created

$location = 'UK South'

$publicIpName = 'pubip-sbcvpngw1'

$vnetName = 'sbcazurevnet'

$subnetName = 'GatewaySubnet'

$subnetPrefix = '172.20.0.0/24' #This is my Azure VNet’s network range, not the Gateway subnet range

$gatewayName = 'vpngw-sbcvpngw1'

$gatewaySku = 'Basic'

$lngRouterName = 'lng-sbcrouterip1'

$lngRouterIP = '80.xx.yy.zz' #This is my on-premises Draytek router’s public IP

$lngSubnetPrefix = '192.168.0.0/24' #This is my on-premises network range

$preSharedKey = 'yourpresharedkey'

Step 3: Set Up the Azure Virtual Networking elements

After logging in, create the required resource group, virtual network, and gateway subnet. Replace the variable values as needed for your environment.

Create a new resource group for the VPN gateway:

New-AzResourceGroup -Name $rg -Location $location

Create a new public IP address in the resource group:

$gwip = New-AzPublicIpAddress -Name $publicIpName -ResourceGroupName $rg -Location $location -AllocationMethod Dynamic -Sku Basic

Create a subnet in the virtual network for the VPN gateway:

$subnet = New-AzVirtualNetworkSubnetConfig -Name $subnetName -AddressPrefix $subnetPrefix

Create a new virtual network before creating the VPN gateway (if it doesn’t exist):

$vnet = New-AzVirtualNetwork -Name $vnetName -ResourceGroupName $rg -Location $location -AddressPrefix "172.20.0.0/16" -Subnets $subnet

If you have already implemented a VNet, subnet and public IP address and want to retrieve those objects you can follow the sub-step below to populate them using Powershell (this is a good way to validate all of your variables):

Retrieve the virtual network and subnet to ensure they are correctly set up:

$vnet = Get-AzVirtualNetwork -Name $vnetName -ResourceGroupName $rg

$subnet = Get-AzVirtualNetworkSubnetConfig -Name 'GatewaySubnet' -VirtualNetwork $vnet

$gwip = Get-AzPublicIpAddress -Name $publicIpName -ResourceGroupName $rg

Step 4: Create the VPN Gateway

Now you can create the VPN gateway with a basic SKU and route-based IPv4 configuration

$ngwIpConfig = New-AzVirtualNetworkGatewayIpConfig -Name "GatewayIpConfig" -Subnet $subnet -PublicIpAddress $gwip

$azvng = New-AzVirtualNetworkGateway -Name $gatewayName -ResourceGroupName $rg -Location $location -IpConfigurations $ngwIpConfig -GatewayType "Vpn" -VpnType "RouteBased" -GatewaySku $gatewaySku

Note: Creating a gateway can take up to 45 minutes, however in my recent experience this can be something around 10 minutes. Be patient!

Once the command has run to completion you should have a variable containing the detail of the virtual network gateway, however if you need to recreate it run:

$azvng = Get-AzVirtualNetworkGateway -Name $gatewayName -ResourceGroupName $rg

Step 5: Configure the Local Network Gateway

Create a local network gateway which defines the connection target for your on-premises router

$azlng = New-AzLocalNetworkGateway -Name $lngRouterName -ResourceGroupName $rg -Location $location -GatewayIpAddress $lngRouterIP -AddressPrefix $lngSubnetPrefix

Otherwise, you can retrieve an existing local network gateway using

$azlng = Get-AzLocalNetworkGateway -Name $lngRouterName -ResourceGroupName $rg

Step 6: Establish the VPN Connection

Create a VPN connection between the Azure VPN gateway and the local network gateway (this is a multi-line command)

New-AzVirtualNetworkGatewayConnection -Name conn-AZ-to-SBC-vpn -ResourceGroupName $rg ` -Location $location -VirtualNetworkGateway1 $azvng -LocalNetworkGateway2 $azlng ` -ConnectionType IPsec ` -SharedKey $preSharedKey ` -ConnectionProtocol IKEv2

Step 7: Configure the Draytek Vigor 2927

On the Draytek side, log in to the router web interface and configure an IPsec VPN profile.

Input the pre-shared key, remote gateway (Azure’s public IP), and remote subnets as configured above. Use the correct proposal algorithms for compatibility (usually AES256/SHA256 and DH Group 2 for Azure).

These screen elements relate to the latest firmware at time of publishing (V2927_20250804_DrayTek_4462).

It is very important to complete/validate the first step because enabling the IPsec VPN service requires a reboot, and whilst this may have been done previously there’s no way to know if you will be able to make a new tunnel until it is enabled then rebooted.

Here are the exact configuration elements which I finally validated successfully:

Open VPN and Remote Access > Enable IPsec VPN Service (reboot required otherwise no tunnel will be established)
Open VPN and Remote Access > LAN to LAN > Add (new profile)
Set the following Common settings:
- Enable this profile: Yes
- Name: Azure_VPN_tunnel
Set the following Dial-Out settings:
- Call direction: Both
- Dial out through: WAN1
- Dial out settings:
- IPsec tunnel: IKEv2
- Server IP/Host Name: IP address or name of the local network gateway (e.g., lng-sbcrouterip1)
IKE phase 1 settings:
- Authentication method: Pre-shared key
- Pre-shared key: yourpresharedkey
- Proposal Encryption: AES-256
- Proposal DH Group: Group 2 (1024 bit)
- Proposal Authentication: SHA-256
IKE phase 2 settings:
- Security protocol: ESP (High)
- Proposal Encryption: AES-256
- Proposal Authentication: SHA-256
IKE Advanced settings:
- Phase 1 Key lifetime: 28800 seconds
- Phase 2 Key lifetime: 3599 seconds
- Enable Perfect Forward Secret: Not checked
Set the following Dial-In parameters:
- Allowed VPN Type: IPsec Tunnel(IKEv1/IKEv2)
- Specify Remote VPN Gateway:
- Remote IP address: (e.g. public IP of the Azure VPN gateway)
- Allowed IKE Authentication Method:
- Pre-shared key: yourpresharedkey
- Allowed IPsec Security method:
- AH, ESP-DES, ESP-3DES, ESP-AES
TCP/IP Network Settings:

Local Network IP: 192.168.0.0
Local Network mask: 255.255.255.0
Remote Network IP: 172.20.0.0
Remote Network mask: 255.255.0.0

February 13, 2024February 13, 2024

How to clean up NSX Advanced Load Balancer following replacement of a failed Tanzu control plane node

How do I clean up a missing control plane node in the Avi load balancer console?

This post outlines an approach I used to solve a problem which has occurred in several environments I’ve worked in recently. I haven’t seen a similar set of instructions anywhere yet, but it doesn’t mean that they are the only way to solve the problem. Check with VMware Support if you’re having a production problem, don’t follow this guidance without properly understanding the type of problem which you’re experiencing.

If you have found this page because you’re stuck with a similar problem it is probably because one or more of your control plane nodes in a Tanzu Kubernetes Grid (TKG) cluster have failed and been replaced automatically leaving a broken IP pool entry in NSX Advanced Load Balancer user interface.

For example, you log in and find that one of your IP pools which define the control plan endpoints are offline (shown as 3/4 servers up).

Clicking into the cluster will provide further detail of the missing control plane endpoints

In this case, one of the existing control plane nodes (172.20.11.45) became frozen and went offline , eventually losing its DHCP lease before it could be converted into a permanent reservation. Tanzu’s vSphere integration automatically provisioned a new node, and the old IP address now belongs to a new VM somewhere outside of Tanzu.

However, despite this situation occurring some days previously the Avi Kubernetes Operator (ako) has not cleaned up, perhaps expecting that the VM might be recovered eventually.

If you’re in a similar situation you will now know the name of the environment and should be able to determine the IP addresses of your current control plane nodes still:

kubectl config use-context [name of your management cluster context]
kubectl get nodes -o wide

In this case we are only interested in the IP addresses belonging to nodes having the control-plane node (the first three in the output below).

There aren’t any more ‘missing’ control plane endpoints shown above, so Kubernetes appears satisfied that it is in a workable state.

As a validation, check that the endpoints listed within the Kubernetes service map onto the current working list of nodes.

List the endpoints for the Kubernetes service (in default namespace)

kubectl get ep kubernetes -o json

The JSON output above is quite simple to read vertically, and confirms that there are three IP addresses within a subset of endpoints serving the Kubernetes API service on port 6443 (via the Avi Load Balancer vserver) that is defined in your ~/.kube/config file.

These match the output which the NSX Advanced Load Balancer showed previously.

What puzzled me for a very long time now seems obvious, that you cannot edit/remove any defunct entries from the Avi IP pool using the UI because the operator synchronises the list of endpoints for each service. By fixing the condition in Kubernetes the operator will take care of the content of the pool itself.

This is the way.

Obtain the list of services in the tkg-system namespace

kubectl get svc -n tkg-system

Now use the cluster-specific named control plane service to output the list of endpoints for the control plane

Aha, there’s the 172.20.11.45 control plane node which no longer exists in the cluster.

Edit the endpoint and manually remove the missing address from the subset addresses section

kubectl edit ep [tkg-system-tkg-mgmt-projit-control-plane] -n tkg-system

Using the VI editor remove the two lines declaring the ip and nodeName entries for the missing cluster node

Close the file and save the changes, the endpoint will be updated.

Refresh the Avi load balancer UI and if everything is well the pool will be updated dynamically when the ako operator detects the updated list of endpoints.

Further information confirming the status update is reflected in the ako-0 pod logs, which shows that a change has been detected between the cached copy of the virtual server object and the updated relationship which is computed from the graph database.

kubectl get logs ako-0 -n avi-system

It then resynchronises the pool content with Avi.

I’d be very pleased to hear if you run into a similar scenario, as I do not think that this element of ako is described anywhere in the official documentation of either Tanzu or AKO – and the DHCP lease re-issue will often crop up if an admin did not take care of making a permanent reservation after a node is added. Often this is because Tanzu will discover a broken node and intervene without anyone being aware of the problem, but this does not always make sense if addresses are not reserved permanently by default in your subnet.

Thanks for reading –

March 23, 2023March 24, 2023

How much space does an air gap installation of Tanzu TKG 2.1.1 need?

In a follow up post to how-much-space-does-an-air-gap-installation-of-tanzu-tkg-1-6-0-need I thought it would be useful to expand on the initial summary to include an upgrade to TKG 2.1.1.

In the previous 1.6.0 example there was a total of 157 images (881 artifacts) requiring 9.7GB of storage space. However the download process has been modified and doesn’t use a shell script to download files for an air gap registry anymore, but rather a command such as:

tanzu isolated-cluster download-bundle --source-repo projects.registry.vmware.com/tkg --tkg-version v2.1.1

This results in 244 tar files being downloaded for a single version of TKG and 45GB of space needed.

When these tar files are uploaded I experienced several problems caused by a redis bug when using Harbor 1.10.x, and the upload command only succeeded once I had upgraded to Harbor 2.5.0.

tanzu isolated-cluster upload-bundle --source-directory ./ --destination-repo registry.sbcpureconsult.internal/tkg --ca-certificate /tmp/ca.crt

In total (for the combination of both TKG 1.6.0 and 2.1.1 releases) there are a total of 177 repositories requiring 20.58GB of storage space.

If I subtract the two figures from one another it indicates that TKG 2.1.1 requires 10.88GB in total.

March 17, 2023

How much space does an air gap installation of Tanzu TKG 1.6.0 need?

I have implemented several air gapped installations of Tanzu Kubernetes Grid 1.6 now using Harbor registry so thought it might be worth recording how many images are stored and the space required.

Example clean registry with only TKG 1.6 files

Short on time? I should caveat that my results only record the space needed for a single version of Kubernetes (1.23.8). This is the newest supported build of Kubernetes in the TLG 1.6.0 release.

During the air gap installation it is possible to reduce the file set required to be stored in your registry by extracting the Bill of Materials for a specific version only:

export DOWNLOAD_TKRS="v1.23.8_vmware.2-tkg.1"

In total (for this specific release) there are 157 images (881 artifacts) requiring 9.7GB of storage space.

I have tested the deployment of a management and worker cluster from the air gap registry and confirm successful installation.

Over time you may accumulate older versions in your registry which are no longer required, however there’s not information available currently on how you could reduce the number of images stored – so I would recommend keeping the image-copy file produced during each iteration of the air gap registry preparation phase so that you could remove them manually at a later date.

February 10, 2022March 23, 2023

Comparing YAML documents with Beyond Compare

I’m a huge fan of Scooter Software’s magnificent Beyond Compare tool, having used it for years in many different scenarios. It does a brilliant job of lining up the similarities and differences in multiple file types – especially text based file formats.

A Kubernetes container orchestration project likely involves defining multiple YAML manifests for creating different objects in the API so I often spend many happy hours comparing differences between folders full of files in different environments.

Unfortunately within Beyond Compare there isn’t any inbuilt YAML file support so it sometimes makes mistakes when aligning functionally equivalent but structurally different files.

Here’s an example:

Default file comparison using simple everything else text format

By default red text shows differences, blue text means minor differences and hatched areas show where there is a lack of alignment possible. I have defined certain non-important grammar elements to ignore, e.g. secretName: so these are shown in blue, but why is it not possible to line up the other elements?

In discussion with Scooter Software they explained that they would need a YAML parsing algorithm to be developed in order to pre-sort such lists into the correct layout, and this isn’t possible right now.

Alternatively they suggested creating a custom File Type which uses an external program to do the sorting work before it attempts to line up and search for differences.

With that in mind I came across the yq project in Github by Mike Farah which provides amongst other things a sorting tool along the lines of jq (for JSON) for YAML documents instead. The following brief instructions show how to create a custom file type in Beyond Compare to have yq sort the keys and values in a temporary file before showing the comparison view. You can then choose to integrate and save changes on either side.

Follow preparation steps

Download yq_windows_amd64.exe from the release page and rename the file to jq.exe for ease of use
Create a folder called YAML in %USERPROFILE%\AppData\Roaming\Scooter Software\Beyond Compare 4\Helpers
Copy the jq.exe file into the above folder
Create a new file within the same folder yaml_sorted.bat containing:

@echo off Helpers\YAML\yq.exe e -M "sort_keys(…)" %1 >>%2
Open Beyond Compare and configure as follows

Configure Beyond Compare

Choose Tools, File Formats

Choose the [+] button and select Text Format

On the General tab enter a file mask to use when applying the File Format correctly, e.g. *.yaml,*.yml

Click on the Conversion tab and select External program (Unicode filenames) then enter the following path and parameters

Helpers\YAML\yaml_sorted.bat %s %t

Optionally, choose whether to Disable editing (when displaying the resulting comparison) and whether to ‘Trim trailing whitespace’ when saving – I find this helpful sometimes.

Click on the Misc tab and choose ‘Insert spaces instead of tabs’ and tab stop 8.

Click Save and Close.

Now find two YAML files to compare and double click in Beyond Compare. It will use the file type mask to apply the custom file format you just created.

In the background it will run the yq executable twice (via the batch file), once for each input file (left and right comparison). The executable takes the input file (%1) then sorts the keys and values alphabetically in sequence first before outputting the content to a new temporary file (%2).

Now open a YAML file comparison and you’ll find a much improved alignment based upon first the sorting of individual keys, then subsequent scalar values within each.

e.g. yq e -M -P "keys" C:\Users\stwa\desktop\example-ingress1.yaml

Here is the resulting comparison after processing with yq. I think you’ll agree that it’s much easier to spot the differences and also allow standardisation of your files going forward.

Resulting file comparison between left and right – after sorting via yq

There’s much more that you can do with both yq and Beyond Compare to further tune the formatting order, but hopefully this gives you some sort of light at the end of the tunnel when attempting comparisons of YAML documents.

I would like to investigate whether it’s possible to define a template format in yq so that it uses a custom key alignment order so that it matches the specification used by the Kubernetes API. In addition Beyond Compare allows line weighting within the algorithm so that it can assign preferential weights to certain grammar elements, although this has not been particularly fruitful when used according to the example below.

Final thoughts

Custom file types don’t just affect comparisons – If you right click on a file and choose Open With, Text Edit you will also see the sorted version of the file – not the original.

NB – If you save the modified content it will be saved using the new sorted format.

Documentation for yq is available here: https://mikefarah.gitbook.io/yq

October 5, 2021October 8, 2021

NSX-T Manager appliance high-CPU whilst idle

I run VMware NSX-T in a small lab environment based on Intel NUCs, but I’ve noticed recently that even when not being challenged e.g. following initial boot and being essentially idle, the Manager appliance suffers continual high CPU usage which leads eventually to an uncomfortably warm office.

Even though running the correct minimum virtual machine hardware for the appliance has been configured, i.e. 4 vCPU and 16GB RAM, it was regularly using ~4.5GHz of physical CPU.

Here’s a good example of an otherwise idle appliance showing 40% CPU usage.

After connecting over SSH as the ‘admin’ user and entering ‘get process monitor’ it’s quickly apparent from the top output that ‘rngd’ is responsible for the majority of the CPU utilisation:

‘get process monitor’ whilst logged in as NSX-T admin console user

But what is this? A quick search of more general Linux resources informs us that it is a random number generator used in ensuring sufficient ‘entropy’ is available during creation of certificates, SSH keys etc.

In order to discover more about the purpose of this daemon we can inspect the description of the installed version (5-0ubuntu4nn1) under the current Ubuntu 18.04.4 LTS release.

apt show rng-tools/now

Description: Daemon to use a Hardware TRNG
The rngd daemon acts as a bridge between a Hardware TRNG (true random number
generator) such as the ones in some Intel/AMD/VIA chipsets, and the kernel’s
PRNG (pseudo-random number generator).
.
It tests the data received from the TRNG using the FIPS 140-2 (2002-10-10)
tests to verify that it is indeed random, and feeds the random data to the
kernel entropy pool.
.
This increases the bandwidth of the /dev/random device, from a source that
does not depend on outside activity. It may also improve the quality
(entropy) of the randomness of /dev/random.
.
A TRNG kernel module such as hw_random, or some other source of true
entropy that is accessible as a device or fifo, is required to use this
package.
.
This is an unofficial version of rng-tools which has been extensively
modified to add multithreading and a lot of new functionality.

So we know that this is a helper daemon which improves the speed of providing near-truly random numbers when applications ask for them. What version do we currently have installed in the NSX-T 3.1.2 manager appliance?

apt search rng-tools
Sorting... Done
Full Text Search... Done
rng-tools/now 5-0ubuntu4nn1 amd64 [installed,local]
  Daemon to use a Hardware TRNG

This appears to be the latest available version. In order to examine the status of the rngd daemon itself, log in to the appliance console as the root user and use:

systemctl list-units rng-tools.service

The service is shown as running,

Name of rngd random number generator service

root@nsx-manager:~# systemctl status rng-tools.service

rng-tools.service
Loaded: loaded (/etc/init.d/rng-tools; enabled; vendor preset: enabled)
Active: active (running) since Tue 2021-10-05 09:31:59 UTC; 17min ago
Docs: man:systemd-sysv-generator(8)
Process: 886 ExecStart=/etc/init.d/rng-tools start (code=exited, status=0/SUCCESS)
Tasks: 1 (limit: 4915)
CGroup: /system.slice/rng-tools.service
`-934 /usr/sbin/rngd -r /dev/hwrng

Oct 05 09:31:59 nsx-manager systemd[1]: Starting rng-tools.service…
Oct 05 09:31:59 nsx-manager rng-tools[886]: Starting Hardware RNG entropy gatherer daemon: /etc/init.d/rng-tools: assigning /dev/hwrng to access rdrand on cpu
Oct 05 09:31:59 nsx-manager rng-tools[886]: crw-rw-rw- 1 root root 1, 8 Oct 5 09:31 /dev/random
Oct 05 09:31:59 nsx-manager rng-tools[886]: rngd.
Oct 05 09:31:59 nsx-manager systemd[1]: Started rng-tools.service.

What else can you find out about what it is doing in the background?

rngd -v

Two instances of ‘read error’ are output, followed by two further entropy sources being the Intel/AMG hardware random number generator and AES digital random number generator (RNDG). The ‘read error’ issue appears to be normal behaviour as the package attempts to read sources which don’t exist. Both of the displayed sources indicate that the CPU instruction set includes the necessary flags to tell the VM that it can access hardware random number generation.

I must say, at this point it’s not clear whether NSX-T requires this service to be running permanently or whether it’s a component which Linux uses as a background service in order only to optimise the generation of a random number feed. It seems that stopping the service does appear to eventually cause problems in my lab – so please attempt the next section with CAUTION.

systemctl stop rng-tools.service

This leads to a significant reduction in CPU consumption and running temperature of my ESXi nodes.

CPU usage decreases after stopping rngd service

It may also be possible to disable the service permanently, but since I don’t have a full explanation of the purpose of this service from an NSX-T point of view I would stop short currently from doing this.

systemctl disable rng-tools.service

In the meantime I am hoping that I can get someone within the NSX-T development team to investigate these findings and provide some more permanent kind of workaround.

Further investigation

Further reading around the subject led me to find an issue has been reported on certain CPUs leading to activity spikes, https://github.com/nhorman/rng-tools/issues/136 and newer versions promise to fix this problem. The article mentioned suggests adding the -x jitter option to the start command but this is not available in the version installed in NSX-T.

RNGD_OPTS="-x jitter -r /dev/hwrng"

You can locate and edit the startup parameters by altering the service definition:

vi /etc/init.d/rng-tools

and potentially altering the default kernel values which are referenced by:

RNGDOPTIONS=
[ -r /etc/default/rng-tools ] && . /etc/default/rng-tools

Edit using:

vi /etc/default/rng-tools

However until the version of rng-tools used in NSX-T is updated to resolve this apparent issue it remains a personal choice as to whether or not the service can be stopped intermittently when a lab environment is not needed.

https://en.wikipedia.org/wiki/RDRAND
https://www.exoscale.com/syslog/random-numbers-generation-in-virtual-machines/

September 28, 2021October 5, 2021

Citrix Gateway 13.0 Registry value EPA scan examples

If you’re having trouble with getting Citrix Endpoint Analysis scans of client device registry values to work properly (on Citrix Gateway) you may come across the following issue I experienced in the latest versions of firmware.

It appears that the EPA scan functionality in the NS 13.0 GUI (this article relates to 13.0.82.45) has been merged so that the numeric/non-numeric registry scan types now coalesce into one type of scan: REG_PATH; whereas in previous versions string values were interpreted using REG_NON_NUM_PATH.

Here’s a screen shot of the new expression editor drop down for Windows client EPA scans

NS13.0.82.45 drop down for Windows EPA scans

In comparison to the previous version (NS13.0.71.44).

NS13.0.71.44 drop down for Windows EPA scans

Here’s a screenshot of the registry scan entry panel where you can enter registry path and value, plus comparison or presence operators. Note the tooltip box which says that numeric comparisons will be done when using <,>,== etc.

NS13.0 registry scan value/comparison entry GUI

The convergence of these two types of scan into one appears to hide a reduction in comparison functionality, which only emerges once you attempt to use a string based registry value comparison using REG_PATH. You cannot use == anymore with string values such as REG_SZ.

This is a quick summary of the new behaviour following my own testing:

Numeric comparisons

Scans based upon REG_DWORD, REG_QWORD, REG_BINARY values will only work when carrying out boolean comparisons on numeric values with operators such as ==, !=, >=

e.g.

sys.client_expr("sys_0_REG_PATH_==HKEY\\_LOCAL\\_MACHINE\\\\SOFTWARE\\\\Classes\\\\YourRegistryKeyLocation\\\\YourRegistryValueName_VALUE==_12345[COMMENT: Registry]")

will result in a successful scan when YourRegistryValueName == 12345.

String comparisons

However when using the newly merged functionality, scans based upon REG_SZ values will only work when carrying out comparisons on string values using operators such as ‘contains’, ‘notcontains’.

If you try to use == as the operator on a string comparison the EPA scan logs will result in:

2021-09-28 09:25:38.883 Boolean compare failed. Value false operator ==
2021-09-28 09:25:38.883 Scan 'REG_PATH_==_HKEY\_LOCAL\_MACHINE\\SOFTWARE\\Classes\\YourRegistryKeyLocation\\YourRegistryValueName_VALUE_==_12345' failed for method 'VALUE'

Therefore modify your EPA action expression to fit the following example using ‘contains’:

sys.client_expr("sys_0_REG_PATH_==_HKEY\\\\_LOCAL\\\\_MACHINE\\\\\\\\SOFTWARE\\\\\\\\Classes\\\\\\\\YourRegistryKeyLocation\\\\\\\\YourRegistryValueName_VALUE_contains_12345[COMMENT: Registry]")

There are several other comparisons which do not appear to work properly, e.g. a numeric registry comparison of a REG_QWORD value which is longer than that allowed by the Citrix EPA plugin BUT is allowed within the 64 bytes of the Windows Registry value.

So my advice would be to consider whether the version of Citrix ADC you’re currently using actually offers the type of scan which you’re intending to use (REG_NON_NUM_PATH, REG_PATH), and NOT to rely upon documented examples without determining if the operator matches the value type correctly.