I asked myself this question whilst updating a test VVF 9.1 lab cluster recently, and couldn’t be sure whether the new lifecycle management and fleet management services are now instrumental to host remediation?
My simple attempt to prove this scenario is as follows:
Create a new empty host cluster in vCenter and call it ‘esx-upgrade-target’ for instance
Select the new cluster and click on the Updates tab. Edit the cluster, assign it the desired image build and vendor add-ons (if necessary)
Save the image using a name like ‘autogen-software-spec-2’
Now select the cluster which contains the hosts that need remediating, click on the Updates tab and Assign the new image to the cluster
Stage the update to ALL of the hosts so that the source of the image cannot be dependent on any external location
Shut down the VCFMS control-plane VM (it has 10GB RAM and 4 vCPU)
When complete, use Power, Shutdown guest on the remaining three Kubernetes nodes in the VCFMS cluster – because the control-plane VM is powered down already they won’t restart automatically
NB – If you’re having trouble with a shortage of lab resources, now is a good time in the lab to resize them from 16GB RAM to 12GB, and 12 vCPUs to 6 (only attempt this if you don’t want to maintain compliance with the default ‘large’ size, this will then match a ‘medium’ size all the time you maintain the configuration). ONLY do this in a lab environment, not production!
Now remediate the host cluster in the usual way, evacuating workloads via vMotion, entering maintenance mode and rebooting the hosts sequentially
Start up the VCFMS control-plane VM, and then after a couple of minutes start any worker VMs which are still powered down.
At this point we can be sure that even if the source of the downloaded ESXi images is actually located on the VCFMS lifecycle/fleet servers initially that the action of staging the update removes any dependency.
In conclusion, whilst it makes sense that the downloaded ESXi image files could be stored along with the other bundles in the new VCF/VVF 9.1 depot (new fleet and lifecycle services) but it’s not fully clear if this is the case in practice.
I intend to repeat the process without staging the images for remediation and keeping VCFMS shut down in order to test the point, however there seems to be insufficient information around whether the new management services have to remain online – my bet so far (if short on resources during host reboots) is to pre-stage the image and only begin remediation once this has been done.
This issue appears to have been resolved with the coordinated release of both an updated ESX and vCenter version in the VCF 9.0.1.0 release
Update following GA release of ESX 9.0.1.0.24957456 and vCenter Server 9.0.1.0.24957454
Following on from the excitement of VMware Explore in Las Vegas and receiving my new VMUG Advantage licenses for VCF 9 I have been testing various deployment choices for a new VVF 9 environment in my lab using the GA release of vCenter (9.0.0.0.24755230).
When downloading the various files to populate the offline depot for my cloud installer appliance to consume I noticed that there were two versions of ESX, an older one in both .ISO form (installer) and .ZIP (offline bundle), and a newer one in only the offline bundle.
Here’s what the GA version files look like in the drop down when displaying the original 9.0.0.0 version:
9.0.0.0 offline bundle and ISO downloads
The small drop down box on the top left of the VMware ESX panel can be used to select which version should be displayed, but as you will see below, when selecting the option for 9.0.0.0100 the only file available is an offline bundle.
9.0.0.0100 offline bundle displayed
I had decided to build a custom ISO including some of the VMware Flings (which I thought would also be required to get a working VCF 9 deployment in the future) and used the newer version of ESX (9.0.0.0100) to install my ESX servers prior to executing the VVF installer process.
Mostly this has been a trouble free assumption, other than the warning displayed by the VVF cloud installer that I was using an unexpected version of ESX on my target hosts. I didn’t have any real reason to doubt this process (because the newer version of ESX fixes some critical CVEs) and continued with setting up my environment successfully.
However, during several repeated installations of my Supervisor cluster I ran into issues where the spherelet VIB didn’t always uninstall/reinstall correctly, and was required to remediate the cluster against the ‘autogen-software-spec-1’ lifecycle manager image – which is autogenerated using the version of ESX and any vendor plugins or components used at installation time.
At this point I noticed that often the remediation might fail because it would skip each of the hosts due to a ‘supposed’ hardware incompatibility and not complete the rest of the remediation. These warnings can be silenced within the vCenter UI under cluster object, Monitor tab, vSAN, Skyline Health.
Silence those alerts if you’re not interested in maintaining compatibility with the HCL (because it’s a lab environment for instance).
In trying to chase down the further cause of the problem I saw the orange ribbon displayed on the Hardware Compatibility tab:
“Requested target version is not supported for the cluster.”
Select the cluster object, choose Updates tab and examine the Hardware Compatibility.
This seems strange, because the remediation will eventually complete anyway because I am not using the remediation option to force hardware compatibility before starting. The following screen shows the same message in a different position.
Successful remediation messages on the cluster tab
Why isn’t the offline-depot from the 9.0.0.0100 version actually supported, even though the auto-gen profile is created automatically and the offline-depot file is imported? It seems that this just comes down to the release dates of vCenter and the subsequent ESX update. You can see this in more detail by examining the following file on the vCenter appliance:
Here are the relevant points which called out to me:
Called to discover Hardware Compatibility List
Unknown version 9.0.0-0100.24813472
Cannot find target version (9.0.0-0100.24813472) in VCG
2025-09-17T04:30:13.461Z INFO report.hcl Called to discover HCL for hostId host-34 with target version 9.0.0-0100.24813472, vSanHclConstraints = True.
2025-09-17T04:30:13.462Z ERROR compatibility.releases Got exception while using cache.
Traceback (most recent call last):
File "/usr/lib/vmware-updatemgr/python/hcl/compatibility/releases.py", line 37, in getEsxiReleaseByVersion
result = getCacheFactory().getReleasesCache().getByVersion(version)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/vmware-updatemgr/python/hcl/compatibility/cache/release_cache.py", line 72, in getByVersion
return ProductRelease(releaseId=1, version=version)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/vmware-updatemgr/python/hcl/compatibility/vvs/models/product.py", line 38, in __init__
raise ValueError("Unknown version %s" % version)
ValueError: Unknown version 9.0.0-0100.24813472
2025-09-17T04:30:13.462Z ERROR report.hcl Cannot find target version (9.0.0-0100.24813472) in VCG.
This would on the face of it seem strange, because the software depot even includes the version which is already deployed to the cluster and already knows the version which it presumably extracted from the host(s) when they were added to the cluster for the first time.
Is it as simple as the vCenter version is older than the ESX release, and that Broadcom haven’t yet released a patch for vCenter which recognises new versions of ESX?
Seems like we just need to get vCenter to agree that it’s not using the latest data and to search online for the updated information.
Eventually I stumbled across the menu option to synchronise the hardware compatibility settings and the release data:
Main vCenter menu dropdown, Lifecycle Manager, Actions drop down, Compatibility Data, Sync
The menu option to update the list of release versions which are supported by vCenter
Returning again to /var/log/vmware/vmware-updatemgr/vum-server/hcl_python_lib.log
2025-09-17T10:32:39.412Z INFO __main__ Loading VVS database from file /storage/updatemgr/patch-store/vvs/vvs-consolidated-bundle-download.json
2025-09-17T10:32:39.966Z INFO __main__ VVS database json loaded in-memory
2025-09-17T10:32:39.966Z INFO compatibility.cache Creating CacheFactory...
2025-09-17T10:32:39.970Z INFO compatibility.cache.executor Datastore locked
2025-09-17T10:32:39.970Z INFO __main__ Loading releases into datastore
2025-09-17T10:32:39.970Z INFO __main__ Releases to be loaded: 9
2025-09-17T10:32:39.971Z INFO __main__ Releases loaded
2025-09-17T10:32:39.971Z INFO __main__ Loading cpuseries into datastore
2025-09-17T10:32:39.971Z INFO __main__ Cpuseries to be loaded: 97
2025-09-17T10:32:39.972Z INFO __main__ Cpuseries loaded
2025-09-17T10:32:39.972Z INFO __main__ Loading servers into datastore
2025-09-17T10:32:40.365Z INFO __main__ Servers to be loaded: 5636
2025-09-17T10:32:40.709Z INFO __main__ Servers loaded
2025-09-17T10:32:40.709Z INFO __main__ Loading devices into datastore
2025-09-17T10:32:41.539Z INFO __main__ Devices to be loaded: 7033
2025-09-17T10:32:42.361Z INFO __main__ Devices loaded
2025-09-17T10:32:42.361Z INFO __main__ Loading OEM vendors into datastore
2025-09-17T10:32:42.361Z INFO __main__ OEM vendors to be loaded: 7
2025-09-17T10:32:42.362Z INFO __main__ OEM vendors loaded
2025-09-17T10:32:42.362Z INFO __main__ Updating datastore time to 1758091862563
2025-09-17T10:32:42.513Z INFO compatibility.cache.executor Datastore unlocked
In this case perhaps the issue isn’t that the versions aren’t being published necessarily but that the compatibility cache doesn’t yet include build ‘9.0.0.0100.24813472’.
A quick check through the /storage/updatemgr/patch-store/vvs/vvs-consolidated-bundle-download.json file on my vCenter doesn’t show any release matching that version – so perhaps the initial error was to assume that this patch was also eligible at all for VVF 9 (and not as it seems for VCF 9 only) at this stage.
Perhaps we’ll find out with the next minor or patch release of vCenter if this omission is resolved, but at present it would appear that the best way to avoid these remediation problems in VVF is to only use the GA 9.0.0.0 release of the ESX ISO file.
During my own deployment testing of VMware Cloud Foundation 9, and its smaller counterpart VMware vSphere Foundation 9, I came across the need to determine exactly how much space is required if you want to set up an offline depot to install your first fleet member. In offline/airgap or edge locations where connectivity to the internet is either too slow or completely inaccessible VMware provides a great solution to be able to prepare a portable depot which could either be a virtual machine or laptop operating close to where you are going to run the VCF installer.
Previously the Cloud Builder appliance was very large (~27GB) because the image required a lot of space just to store the various files which it will need for the installation. The new combined VMware Cloud Foundation Installer/SDDC Manager .OVA download in VCF 9 replaces this element with a much smaller 2.15GB file because it doesn’t by default include the other components which will be installed later, e.g. VCF Operations, VCF Automation etc.
Instead you can choose to create an offline depot on a laptop or virtual machine and use this to bootstrap the VMware Cloud Foundation Installer with those files instead of having it download everything from the Broadcom site using your download token. This scenario is far more likely in my own experience, as downloading tens of GB when reaching a greenfield site (for any solution) is variously difficult and can take hours.
The product documentation states that the installer appliance requires a minimum of 914GB storage if thick provisioned, but this post will cover what is the bare minimum in order to get a working VM which will deploy the whole VCF suite.
Doing it the William Lam way
William Lam has several excellent articles which describe the process of (1) enabling a simple HTTP server using Python which will serve the files that are imported into the SDDC Manager when it is first deployed, and (2) these will be useful in setting up a depot (perhaps you’ll use a VMUG Advantage entitlement to download the files if it’s for a lab environment)
However rather than duplicating any of this above guidance, this post concentrates on exactly how much space you’ll need for either a VCF 9 or VVF 9 based offline-depot, ideally stored on a laptop or virtual machine acting as a web server. It could be costly in disk space terms to host these files permanently on your laptop, but there’s no reason why you can’t use an external SSD for this purpose – the question is what size will I need?
Also worth considering is where you’re going to put the VCF Installer virtual machine when you’re getting ready to bootstrap a vSAN environment. In this case you’ll need to find a VMFS datastore which is large enough for the installer plus the data which will be imported into the VM itself. The rest of your disks are probably going to be cleared of any partitions so you can’t use those to store data.
A fully populated offline depot which is capable of serving both VVF and VCF products to the SDDC Manager will require ~56GB on disk when combined with the other parts of the depot. The thin provisioned VM will use approximately 80GB when the offline-depot files have been uploaded to it, so if you’ve got a VMFS partition on a 250GB SSD or NVMe disk that is local to your ESXi server this should be sufficient to hold the installer VM before you eventually migrate it to vSAN.
Here’s how the space breaks down into the two products, each is neatly defined within the user interface of the VMware Cloud Foundation Installer:
VMware vSphere Foundation 9
You will need a total of 16.67GB free space to store the three files (.ova and .iso) comprising the three elements stored in the offline-depot.
VVF 9 space requirements in the offline-depot
VMware Cloud Foundation 9
You will need a total of 52GB free space to store the nine files (.ova, .tar, .vlcm and .iso) comprising the seven application elements stored in the offline-depot.
VCF 9 space requirements in the offline-depot
So there you have it, the whole VVF/VCF stack can be installed from these binaries using the VMware Cloud Foundation Installer, and the only element that you’ll need to install onto your target ESXi server initially is the 2GB installer VM (which can optionally become the actual SDDC Manager for the cluster once complete).
This quick post outlines the successful approach for installing Python on Windows in order to use PowerCLI for making a custom ESXi 8 image.
As you’ll know already this requires PowerCLI 13 as a minimum to be able to handle ESXi8 images, however try as I might there were persistent problems installing and configuring Python on my Windows 10 VM (caused by failure to recognise OpenSSL mainly).
Here’s what I know, having been through several troubleshooting steps:
You don’t need to install OpenSSL separately, this is distributed in the pyopenssl package installed by Pip
The -pythonpath parameter in the Set-PowerCLIConfiguration commandlet needs to include python.exe and be surrounded by double-quotes if necessary
You can run pip commands from command prompt or PowerShell, but the VMware instructions have you run the versioned command e.g. pip3.11.exe from within the Python installation
Python 3.12.3 did not work with PowerCLI 13.2.1, even when using the process outlined below. It would never detect the correct version of OpenSSL. I was only able to make Python 3.11.9 work successfully with this release of PowerCLI.
Here is a brief outline of what I did in order to resolve my problems – I feel that starting on a fresh installation was important here.
Outline of installation steps
Built a clean VM in Azure running Windows 11, this was an important point to eliminate any problems which might have been caused by upgrading from previous PowerCLI versions.
Opened PowerShell 5.1, this is pre-installed with Windows and so there is no need to install
$PSVersionTable.PSVersion
Major Minor Build Revision ----- ----- ----- -------- 5 1 22621 2506
The newer versions include the Pip package manager so it’s not necessary to use the later ‘get-pip.py’ script to install the additional packages, these can be obtained without it.
Return to Powershell window and configured the Python path, noting that there are double quotes around the value because of a space character in ‘Program Files’.
Set-PowerCLIConfiguration -PythonPath "C:\Program Files\Python311\python.exe" -Scope User
It was then straight forward to install the necessary additional packages via command line (I used Windows Command Prompt)
cd "C:\Program Files\Python311\Scripts" pip3.11.exe install six psutil lxml pyopenssl
Close Command Prompt and PowerShell, then reopen PowerShell. Test the ability of ImageBuilder to access the Python packages it needs by using:
Get-EsxImageProfile
This command will generate a red error in PowerShell if any of the elements are missing, but don’t be too quick in reading the output as there are several different errors which all start with similar beginnings.
Building custom ESXi image – additional material
Here are some example commands for what you might do next once PowerCLI and Python are working properly. The process here shows what you would do in order to install the VMware Fling for the USB network driver into an ESXi 8.0U2b generic offline depot file.
Only two files are referenced here, ESXi80U2-VMKUSB-NIC-FLING-67561870-component-22416446.zip and VMware-ESXi-8.0U2b-23305546-depot.zip which can be obtained from the VMware Flings page and VMware Customer Connect portal.
Make sure you log in first to access the Flings page otherwise none of the download options will be visible.
Once the .ISO file is generated then you can use a tool like Rufus to write the image to a bootable USB drive for instance.
How do I clean up a missing control plane node in the Avi load balancer console?
This post outlines an approach I used to solve a problem which has occurred in several environments I’ve worked in recently. I haven’t seen a similar set of instructions anywhere yet, but it doesn’t mean that they are the only way to solve the problem. Check with VMware Support if you’re having a production problem, don’t follow this guidance without properly understanding the type of problem which you’re experiencing.
If you have found this page because you’re stuck with a similar problem it is probably because one or more of your control plane nodes in a Tanzu Kubernetes Grid (TKG) cluster have failed and been replaced automatically leaving a broken IP pool entry in NSX Advanced Load Balancer user interface.
For example, you log in and find that one of your IP pools which define the control plan endpoints are offline (shown as 3/4 servers up).
Clicking into the cluster will provide further detail of the missing control plane endpoints
In this case, one of the existing control plane nodes (172.20.11.45) became frozen and went offline , eventually losing its DHCP lease before it could be converted into a permanent reservation. Tanzu’s vSphere integration automatically provisioned a new node, and the old IP address now belongs to a new VM somewhere outside of Tanzu.
However, despite this situation occurring some days previously the Avi Kubernetes Operator (ako) has not cleaned up, perhaps expecting that the VM might be recovered eventually.
If you’re in a similar situation you will now know the name of the environment and should be able to determine the IP addresses of your current control plane nodes still:
kubectl config use-context [name of your management cluster context]
kubectl get nodes -o wide
In this case we are only interested in the IP addresses belonging to nodes having the control-plane node (the first three in the output below).
There aren’t any more ‘missing’ control plane endpoints shown above, so Kubernetes appears satisfied that it is in a workable state.
As a validation, check that the endpoints listed within the Kubernetes service map onto the current working list of nodes.
List the endpoints for the Kubernetes service (in default namespace)
kubectl get ep kubernetes -o json
The JSON output above is quite simple to read vertically, and confirms that there are three IP addresses within a subset of endpoints serving the Kubernetes API service on port 6443 (via the Avi Load Balancer vserver) that is defined in your ~/.kube/config file.
These match the output which the NSX Advanced Load Balancer showed previously.
What puzzled me for a very long time now seems obvious, that you cannot edit/remove any defunct entries from the Avi IP pool using the UI because the operator synchronises the list of endpoints for each service. By fixing the condition in Kubernetes the operator will take care of the content of the pool itself.
This is the way.
Obtain the list of services in the tkg-system namespace
kubectl get svc -n tkg-system
Now use the cluster-specific named control plane service to output the list of endpoints for the control plane
Aha, there’s the 172.20.11.45 control plane node which no longer exists in the cluster.
Edit the endpoint and manually remove the missing address from the subset addresses section
kubectl edit ep [tkg-system-tkg-mgmt-projit-control-plane] -n tkg-system
Using the VI editor remove the two lines declaring the ip and nodeName entries for the missing cluster node
Close the file and save the changes, the endpoint will be updated.
Refresh the Avi load balancer UI and if everything is well the pool will be updated dynamically when the ako operator detects the updated list of endpoints.
Further information confirming the status update is reflected in the ako-0 pod logs, which shows that a change has been detected between the cached copy of the virtual server object and the updated relationship which is computed from the graph database.
kubectl get logs ako-0 -n avi-system
It then resynchronises the pool content with Avi.
I’d be very pleased to hear if you run into a similar scenario, as I do not think that this element of ako is described anywhere in the official documentation of either Tanzu or AKO – and the DHCP lease re-issue will often crop up if an admin did not take care of making a permanent reservation after a node is added. Often this is because Tanzu will discover a broken node and intervene without anyone being aware of the problem, but this does not always make sense if addresses are not reserved permanently by default in your subnet.
In the previous 1.6.0 example there was a total of 157 images (881 artifacts) requiring 9.7GB of storage space. However the download process has been modified and doesn’t use a shell script to download files for an air gap registry anymore, but rather a command such as:
This results in 244 tar files being downloaded for a single version of TKG and 45GB of space needed.
When these tar files are uploaded I experienced several problems caused by a redis bug when using Harbor 1.10.x, and the upload command only succeeded once I had upgraded to Harbor 2.5.0.
I have implemented several air gapped installations of Tanzu Kubernetes Grid 1.6 now using Harbor registry so thought it might be worth recording how many images are stored and the space required.
Example clean registry with only TKG 1.6 files
Short on time? I should caveat that my results only record the space needed for a single version of Kubernetes (1.23.8). This is the newest supported build of Kubernetes in the TLG 1.6.0 release.
During the air gap installation it is possible to reduce the file set required to be stored in your registry by extracting the Bill of Materials for a specific version only:
export DOWNLOAD_TKRS="v1.23.8_vmware.2-tkg.1"
In total (for this specific release) there are 157 images (881 artifacts) requiring 9.7GB of storage space.
I have tested the deployment of a management and worker cluster from the air gap registry and confirm successful installation.
Over time you may accumulate older versions in your registry which are no longer required, however there’s not information available currently on how you could reduce the number of images stored – so I would recommend keeping the image-copy file produced during each iteration of the air gap registry preparation phase so that you could remove them manually at a later date.
I run VMware NSX-T in a small lab environment based on Intel NUCs, but I’ve noticed recently that even when not being challenged e.g. following initial boot and being essentially idle, the Manager appliance suffers continual high CPU usage which leads eventually to an uncomfortably warm office.
Even though running the correct minimum virtual machine hardware for the appliance has been configured, i.e. 4 vCPU and 16GB RAM, it was regularly using ~4.5GHz of physical CPU.
Here’s a good example of an otherwise idle appliance showing 40% CPU usage.
~40% CPU on a 4 vCPU virtual appliance
After connecting over SSH as the ‘admin’ user and entering ‘get process monitor’ it’s quickly apparent from the top output that ‘rngd’ is responsible for the majority of the CPU utilisation:
‘get process monitor’ whilst logged in as NSX-T admin console user
But what is this? A quick search of more general Linux resources informs us that it is a random number generator used in ensuring sufficient ‘entropy’ is available during creation of certificates, SSH keys etc.
In order to discover more about the purpose of this daemon we can inspect the description of the installed version (5-0ubuntu4nn1) under the current Ubuntu 18.04.4 LTS release.
apt show rng-tools/now
Description: Daemon to use a Hardware TRNG The rngd daemon acts as a bridge between a Hardware TRNG (true random number generator) such as the ones in some Intel/AMD/VIA chipsets, and the kernel’s PRNG (pseudo-random number generator). . It tests the data received from the TRNG using the FIPS 140-2 (2002-10-10) tests to verify that it is indeed random, and feeds the random data to the kernel entropy pool. . This increases the bandwidth of the /dev/random device, from a source that does not depend on outside activity. It may also improve the quality (entropy) of the randomness of /dev/random. . A TRNG kernel module such as hw_random, or some other source of true entropy that is accessible as a device or fifo, is required to use this package. . This is an unofficial version of rng-tools which has been extensively modified to add multithreading and a lot of new functionality.
So we know that this is a helper daemon which improves the speed of providing near-truly random numbers when applications ask for them. What version do we currently have installed in the NSX-T 3.1.2 manager appliance?
apt search rng-tools
Sorting... Done
Full Text Search... Done
rng-tools/now 5-0ubuntu4nn1 amd64 [installed,local]
Daemon to use a Hardware TRNG
This appears to be the latest available version. In order to examine the status of the rngd daemon itself, log in to the appliance console as the root user and use:
systemctl list-units rng-tools.service
The service is shown as running,
Name of rngd random number generator service
root@nsx-manager:~# systemctl status rng-tools.service
Oct 05 09:31:59 nsx-manager systemd[1]: Starting rng-tools.service… Oct 05 09:31:59 nsx-manager rng-tools[886]: Starting Hardware RNG entropy gatherer daemon: /etc/init.d/rng-tools: assigning /dev/hwrng to access rdrand on cpu Oct 05 09:31:59 nsx-manager rng-tools[886]: crw-rw-rw- 1 root root 1, 8 Oct 5 09:31 /dev/random Oct 05 09:31:59 nsx-manager rng-tools[886]: rngd. Oct 05 09:31:59 nsx-manager systemd[1]: Started rng-tools.service.
What else can you find out about what it is doing in the background?
rngd -v
Two instances of ‘read error’ are output, followed by two further entropy sources being the Intel/AMG hardware random number generator and AES digital random number generator (RNDG). The ‘read error’ issue appears to be normal behaviour as the package attempts to read sources which don’t exist. Both of the displayed sources indicate that the CPU instruction set includes the necessary flags to tell the VM that it can access hardware random number generation.
Verbose output from rngd daemon
I must say, at this point it’s not clear whether NSX-T requires this service to be running permanently or whether it’s a component which Linux uses as a background service in order only to optimise the generation of a random number feed. It seems that stopping the service does appear to eventually cause problems in my lab – so please attempt the next section with CAUTION.
systemctl stop rng-tools.service
This leads to a significant reduction in CPU consumption and running temperature of my ESXi nodes.
CPU usage decreases after stopping rngd service
It may also be possible to disable the service permanently, but since I don’t have a full explanation of the purpose of this service from an NSX-T point of view I would stop short currently from doing this.
systemctl disable rng-tools.service
In the meantime I am hoping that I can get someone within the NSX-T development team to investigate these findings and provide some more permanent kind of workaround.
Further investigation
Further reading around the subject led me to find an issue has been reported on certain CPUs leading to activity spikes, https://github.com/nhorman/rng-tools/issues/136 and newer versions promise to fix this problem. The article mentioned suggests adding the -x jitter option to the start command but this is not available in the version installed in NSX-T.
RNGD_OPTS="-x jitter -r /dev/hwrng"
You can locate and edit the startup parameters by altering the service definition:
vi /etc/init.d/rng-tools
and potentially altering the default kernel values which are referenced by:
However until the version of rng-tools used in NSX-T is updated to resolve this apparent issue it remains a personal choice as to whether or not the service can be stopped intermittently when a lab environment is not needed.
This is a placeholder posting for ongoing updates as and when new updates/resolutions are found. It isn’t intended to provide any additional detail to the problems outlined but simply to document the areas where bugs or ‘gotchas’ are located.
I have recently acquired several Intel NUC 11th Generation (NUC11TNHv50L) for my lab/testing environment which are being deployed into an existing vSAN/NSX-T environment as a workload domain. The release of these latest NUCs seemed to have generated a lot of interest with different community members discussing the ideal fit with NSX-T (due to the dual 2.5 Gbit/s Intel I225-LM NICs which come in the Pro version), however there are a couple of limitations that make this not a smooth ride currently.
Community networking driver and workarounds
Out of the box these NUCs are not supported with VMware ESXi and rely upon the Community Networking Driver Fling. Therefore before purchasing these devices for your home lab be aware that this fling:
Requires a custom ESXi image to be created which includes the Community Networking Driver
Does not support jumbo frames (e.g. >1500 byte MTU) – which in my view prevents any serious use with the NSX-T Geneve protocol which is typically 1600 byte minimum
Causes the network interface to become disconnected (link layer communication fails) if configured MTU is greater than 1500, which only recovers after a reboot
Seems to cause a purple screen (PSOD) failure when the second NIC is connected (under undefined circumstances currently)
Currently I am overcoming the NSX-T frame size issue by using the Startech USB 3.1 1Gbit/s USB network adapters, but this requires an additional fling to be installed. As a compromise it’s not too bad, since there are two Thunderbolt/USB-C ports on these NUCs allow up to two additional 1Gbit/s interfaces to be attached. So I am configuring my ESXi hosts as:
1 x Onboard Intel I225-LM at 2.5 Gbit/s – dvSwitch 1 (Management, vSAN)
1 x StarTech USB 3.1 adapter at 1Gbit/s – dvSwitch 2 (NSX-T, vMotion)
Power off and shut down
In addition it seems that when ‘Shut down’ of an ESXi host is performed the system ignores the BIOS power setting (e.g. to remain off, or power on etc.) and will immediately restart the operating back to a running condition (almost as if a reboot instead of shut down were chosen). This is strange behaviour which needs further experimentation and makes shutting down your lab a lot more time consuming – however it can be worked around currently by:
Shut down the ESXi instances individually using host UI/vCenter
Watch the power light on the front panel (assuming no screen attached) – when the power light turns off for approximately 0.5s it is initiating the actual power off, prior to becoming turned back on again
At this point pull the power supply out of the back of the NUC and plug it back in a couple of seconds later – it will remain off instead of rebooting (even if the BIOS setting says on loss of power – power on)
It’s getting hot in here
Lack of fan speed and temperature within ESXi hardware sensors. This is not a new issue but despite the integrated 3D graphics which is now on-chip there still seems to be a lack of information exposed to the operating system (presumably by Intel). In my bookcase vSAN/NSX-T environment it’s becoming a ‘hot topic’ to say the least ;-). Both new and older NUCs are doing fine on the Balanced performance/fan speed setting, and do a good job of spinning up and down the fan whenever the CPU turbo feature engages (up to 4.1GHz on my units), but it would be good to be able to view this more empirically than just watching how many windows need to be opened!
With the advent of XenApp 7 and more recently experiencing the higher frequency of VDA cumulative updates I would generally recommend implementing Citrix Machine Creation Services or other imaging mechanism (such as Provisioning Server) when rolling out new versions of the Virtual Desktop Agent to a large number of catalogs.
However, what happens when you only require one XA server per catalog, or when each one of those servers is handled manually when new application code is deployed? This is more common than you might imagine, especially in Citrix deployments which have per-customer or per-app specific catalogs. The work involved in maintaining a master image can be significant and the serviceability of such relies upon someone knowing how to treat image updates in a way that won’t introduce problems that could arise weeks or months later.
One customer of mine has at least 80 catalogs running one or more XenApp VMs and so it simply doesn’t make sense to maintain a single master image for each, especially when application code updates are delivered frequently. So I set about creating a simple PowerShell script which works in a VMware environment to attach the Citrix upgrade ISO and then run the setup installer within the context of a remote PowerShell session.
Using this method you can easily carry out a bulk upgrade of tens (possibly hundreds) of statically assigned VDAs individually by attaching the ISO and installing the update automatically. The advantage of this time saving approach is that it can even be run in a loop so that the upgrade is only attempted when a server is idle and not running any sessions.
NB – as always, please validate the behaviour of the script in a non-production environment and adjust where necessary to meet your own needs.
Here’s a walkthrough of the script, along with the complete example version included at the end.
The script will load the required plugins from both Citrix and VMware PowerShell modules/plugins (I generally run things like this on the Citrix Delivery Controller and install PowerCLI alongside for convenience)
Request credentials and connect to vCenter via a popup
Request credentials for use with WinRM connections to remote Windows servers via a popup
Create a collection of objects (XA servers) which are powered on, do not have any active sessions and don’t already have the target VDA version installed (see $targetvda variable)
For each VM, sequentially:
Attach the specified .iso image file to the resulting VMs
Determine the drive letter where the XA ISO file has been mounted
Create a command line for the setup installer, and save the command into c:\upgrade_vda.cmd on the XA server
Connect via PowerShell remoting session to the remote XA server
Adjust the EUEM registry node permissions (as per https://support.citrix.com/article/CTX215992)
Execute the c:\upgrade_vda.cmd upgrade script on remote machine via PS session
Disconnect the PowerShell remote session
Reboot the VM via vCenter in order to restart the XA services
Review the script and edit the following variables to reflect your use-case:
All servers in my example environment begin with virtual machine names SRV* so this line can be adapted according to the number of VMs which you would like to upgrade, or simply replace with the actual named servers if you want to be more selective:
($_.HostedMachineName -in 'SRV1','SRV2','SRV3')
Finally, consider modifying the following variable from $true to $false in order to actually begin the process of upgrading the selected VMs. I suggest running it in the default $true mode initially in order to validate the initial selection criteria.
$skiprun = $true
Additional work:
I would like additionally to incorporate the disconnection of previous VDA .ISO files from the VM before attempting to upgrade. I have noticed that the attached volume label search e.g. Get-Volume -FileSystemLabel ‘XA and XD*’ that determines the drive letter selection is too wide, and will erroneously detect both XA_7_15_4000.iso and XA_7_15_2000.iso versions without differentiating between them.
I would also like to do further parsing of the installation success result codes in order to decide whether to stop, or simply carry on – however I have used the script on tens of servers without hitting too many roadblocks.
This script could also be adapted to upgrade XenDesktop VDA versions where statically assigned VMs are provided to users.
Final note:
This script does not allow the Citrix installer telemetry to run during the installation because it requires internet access and this generates errors in PowerShell for XenApp servers which can’t talk outbound. You can choose to remove this command line parameter according to your circumstances:
/disableexperiencemetrics
Citrix also optionally collects and uploads anonymised product usage statistics, but again this requires internet access. In order to disable Citrix Telemetry the following setting is used:
/EXCLUDE "Citrix Telemetry Service"
Additionally the Personal vDisk feature is now deprecated, so the script excludes this item in order for it to be removed if it is currently present (so be aware if you’re using PvD):
/EXCLUDE "Personal vDisk"
PowerShell code example:
# Upgrade VDA on remote Citrix servers
if ((Get-PSSnapin -Name "Citrix.Broker.Admin.V2" -ErrorAction SilentlyContinue) -eq $Null){Add-PSSnapin Citrix.Broker.Admin.V2}
if ((Get-PSSnapin -Name "VMware.VimAutomation.Core" -ErrorAction SilentlyContinue) -eq $Null){Add-PSSnapin VMware.VimAutomation.Core}
$vcentersrv = "yourvcentersrv.domain.com"
if ($vmwarecreds -eq $null) {$vmwarecreds = Connect-VIServer -Server $vcentersrv} # Authenticate with vCenter, you should enter using format DOMAIN\username, then password
if ($creds -eq $null) {$creds = Get-Credential -Message 'Enter Windows network credentials'} # Get Windows network credentials
clear
$targetvda = '7.15.4000.653' #Add the target VDA version number - anything which isn't correct will be upgraded
$isopath = "[DATASTORE] ParentFolderName\XenApp_and_XenDesktop_7_15_4000.iso" #Path to ISO image in VMware
$skiprun = $true #Set this variable to false in order to begin processing all listed VMs
$targetvms = Get-BrokerMachine -DesktopKind Shared | Where-Object {($_.AgentVersion -ne $targetvda) -and ($_.PowerState -eq 'On') -and ($_.HostedMachineName -like 'SRV*')}
Write-Host The following XA VMs will be targeted
Write-Host $targetvms.HostedMachineName
if ($skiprun -eq $true) {write-host Skip run is still enabled; exit}
foreach ($i in $targetvms){
if ($i.AgentVersion -ne $targetvda) {
Write-Host Processing $i.HostedMachineName found VDA version $i.AgentVersion
if ($i.sessioncount -ne $null) {Write-Host Processing $i.HostedMachineName found $i.sessioncount users are logged on}
if ($i.sessioncount -eq 0) {#Only continue if there are no logged-on users
Write-Host Processing $i.HostedMachineName verifying attachment of ISO image
$cdstate = Get-VM $i.HostedMachineName | Get-CDDrive
if (($cdstate.IsoPath -ne $isopath) -and ($cdstate -notcontains 'Connected')) { $cdstate | Set-CDDrive -ISOPath $isopath -Confirm:$false -Connected:$true;Write-Host ISO has been attached}
$s = New-PSSession -ComputerName ($i.MachineName.split('\')[1]) -Credential $creds
#Create the upgrade command script using correct drive letters
Write-Host Processing $i.HostedMachineName -NoNewline
invoke-command -Session $s {
$drive = Get-Volume -FileSystemLabel 'XA and XD*'
$workingdir = ($drive.driveletter + ":\x64\XenDesktop Setup\")
$switches = " /COMPONENTS VDA /EXCLUDE `"Citrix Telemetry Service`",`"Personal vDisk`" /disableexperiencemetrics /QUIET"
$cmdscript = "`"$workingdir" + "XenDesktopVDASetup.exe`"" + $switches
Out-File -FilePath c:\upgrade_vda.cmd -InputObject $cmdscript -Force -Encoding ASCII
Write-Host " wrote script using path" $workingdir
}
#Adjust the registry permissions remotely
Write-Host Processing $i.HostedMachineName updating registry permissions
Invoke-Command -Session $s {
$acl = Get-Acl "HKLM:\SOFTWARE\Wow6432Node\Citrix\EUEM\LoggedEvents"
$person = [System.Security.Principal.NTAccount]"Creator Owner"
$access = [System.Security.AccessControl.RegistryRights]"FullControl"
$inheritance = [System.Security.AccessControl.InheritanceFlags]"ContainerInherit,ObjectInherit"
$propagation = [System.Security.AccessControl.PropagationFlags]"None"
$type = [System.Security.AccessControl.AccessControlType]"Allow"}
Invoke-Command -Session $s {$rule = New-Object System.Security.AccessControl.RegistryAccessRule($person,$access,$inheritance,$propagation,$type)}
Invoke-Command -Session $s {$acl.AddAccessRule($rule)}
Invoke-Command -Session $s {$acl |Set-Acl}
#Execute the command script
Write-Host Processing $i.HostedMachineName, executing VDA install script
Invoke-Command -Session $s {& c:\upgrade_vda.cmd} # Runs the upgrade script on remote server
Remove-PSSession $s #Disconnect the remote PS session
Restart-VMGuest -VM $i.HostedMachineName -Confirm:$false #Restart the server following either a successful or unsuccessful upgrade
}
}
}
